RPi: creating a RAM-disk running Linux environment from NFS booted Raspbian

Sometimes you need a very specific use case for your Raspberry Pi.
The durability (and speed) of the SD card used in RPi may vary.
Because of this I always tend to move Raspbian to an NFS share, and boot via NFS (root over NFS).
This way I have the following advantages:
1. The SD card is only necessary to boot (single boot partition is sufficient)
2. Main root filesystem is on fast and durable NFS mount on the server
3. I am not limited by the SD card space in any way
Doing the development/testing work is really nice this way.

The big disadvantage of this approach was the problems when the NFS server was gone for some reason.
Generally the rpi was able to raise up after this and replace the NFS session (eg. after server reboot) with the following kernel messages:

kernel: nfs: server <servername> not responding, still trying
kernel: nfs: server <servername> OK

But in some cases it was unable to do this and stucked:

kernel: nfs: server <servername> not responding, still trying
kernel: nfs: server <servername> not responding, timed out

And in those cases a hard RaspberryPi reset was necessary to boot it up again.

Obviously in my automation use case I wanted the RPi to run independently and stable on its own.
I did not want to put the whole system on the SD card, mainly due the SD durability and speed (and also limited storage space).

So I created a mixed solution, which I want to share below. It is a perfect one for me. My plan was to boot the system from the SD card via NFS (boot partition can be readonly – no writes to the SD card), then move only the necessary files to the RAM disk of the Raspberry Pi.
And this is possible with pivot_root. 🙂

Keep in mind that this method may be not suitable for some bigger projects with a lot of dependencies on external libraries.
In my case the binary which I am running on the RPi is just a C++ program which is doing the automation, so it is really small and not depend on many external libraries (rather – the standard ones).
It’s also important to know that the RAM usage on the RPi has to be monitored (at least after deploying) – more libraries and executables we put into RAM disk, less space is available to running programs. In my case of RPi model B rev 2.0 my RAM usage is about 11MB of the total 118MB available memory.
OK, Let’s start from the beginning:

1. Preparing the SD card

At the time I was doing the work the latest Raspbian version available was:
2016-05-27-raspbian-jessie-lite.img
Of course I choosed the lite version because I don’t need a full distro. The rest of this “howto” is based on this specific version, but if nothing significant changed, I hope it should work with the recent versions as well 🙂
Firstly I had to discover the partition list of downloaded SD card image:

$ fdisk -l 2016-05-27-raspbian-jessie-lite.img
 Disk 2016-05-27-raspbian-jessie-lite.img: 1.3 GiB, 1387266048 bytes, 2709504 sectors
 Units: sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disklabel type: dos
 Disk identifier: 0x6fcf21f3

Device                               Boot  Start     End Sectors  Size Id Type
 2016-05-27-raspbian-jessie-lite.img1        8192  137215  129024   63M  c W95 FAT32 (LBA)
 2016-05-27-raspbian-jessie-lite.img2      137216 2709503 2572288  1.2G 83 Linux

So we have a boot partition started at sector 8192 and a root filesystem at sector 137216.
To get the exact offset in bytes we have to multiply the sector number by sector size (512).

I created a loop device for the boot partition:

losetup --offset 4194304 /dev/loop2 2016-05-27-raspbian-jessie-lite.img

and mounted afterwards into raspbian_boot directory:

mount /dev/loop2 ./raspbian_boot

I have access to original boot files from Raspbian (raspbian_boot), so it was a time to prepare the target location for files. I found the old 16MB SD card (bundled with my canon camera) and since it should be sufficient for boot partition, I just gave it a try.
Indeed, it was working properly, I just had to format the boot partition as FAT16, because there was not enough space for FAT32.
After creating a target partition, I stripped the original Raspbian boot files to the really needed necessary ones for my Raspberry Pi model B.
It was sufficient for me to copy the following files to my target boot partition:
bcm2708-rpi-b.dtb
bootcode.bin
cmdline.txt
config.txt
kernel.img
start.elf

I edited the config.txt and enabled the i2c by commenting out the line:

dtparam=i2c_arm=on

Then I modified the original cmdline.txt and enabled NFS boot:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs ip=dhcp nfsroot=192.168.0.1:/exports/rpi,nfsvers=3,tcp elevator=deadline rootwait

After unmouning, the SD card was ready to insert into Raspberry Pi.

2. Preparing NFS share on server

Then I was able to mount the main filesystem from the raspbian image:

losetup --offset 70254592 /dev/loop2 2016-05-27-raspbian-jessie-lite.img
mount /dev/loop2 ./raspbian_rootfs

I copied all files to my local directory of the NFS export:

rsync -xav raspbian_rootfs/ /exports/rpi/

and added the following line to the /etc/exports:

/exports/rpi 192.168.0.4/32(rw,no_root_squash,async,no_subtree_check)

and reloaded the exports with:

exportfs -ra

I also added in the /etc/fstab in the RPi rootfs the line:

/dev/nfs        /               rootfs  defaults

while also commented out the original rootfs mount (/dev/mmcblk0p2)

At this stage a normal “full” Raspbian should be able to boot from NFS on the Raspberry Pi.

3. Preparing to pivot_root

On the Raspberry, I’ve installed the dropbear and busybox-static packages:

apt install dropbear busybox-static

After installation, all necessary files was in place.
In the root NFS directory I created a three scripts for doing all necessary jobs:

Script called root2ram.sh is for doing early work and pivot_root:

In cmdline.txt on the SD card I changed the initial script run on boot, appending: “– \root2ram.sh”. Full line after this change was:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs ip=dhcp nfsroot=192.168.0.1:/exports/rpi,nfsvers=3,tcp elevator=deadline rootwait -- /root2ram.sh

When the rpi is doing the pivot_root switch it is launching the busybox.
Busybox is in turn starting /etc/init.d/rcS which in my case has the following contents:

The last script is fs_prepare which is called from rcS. It is used for the necessary file copy for the new filesystem. You have to figure out yourself what libraries and binaries are necessary for your configuration (ldd tool is your friend).
In my case the file has the following contents:

After all this steps – my Raspberry Pi successfuly booted from NFS to ramdisk.
I am then able to umount the original NFS with this command:

umount /old_root

And the RPi is still running fine 🙂
I can ssh into it (it has the dropbear running) and it is working really fast, because the whole system is in RAM.
When I needed to do some jobs (eg. compilation) on the “original” NFS system, then I am just mounting it with the following command:

mount -t nfs -o nolock 192.168.0.1:/exports/rpi /old_root

and then chroot into it:

chroot /old_root /bin/bash

Bonus: Running it under QEMU

You can also boot this NFS share under QEMU. This way you can test all changes before running it on the real RaspberryPi.
To be able to run it, you need to install the qemu-system-arm package in your linux distro.
You also need a modified kernel. In my case (for the above Raspbian version) I had to use a kernel named: kernel-qemu-4.4.11-jessie.
Which at the time of writing is available for download from this github repo:
https://github.com/dhruvvyas90/qemu-rpi-kernel

In the NFS share are though required some small changes:
1. in /etc/ld.so.preload you need to comment the line:

/usr/lib/arm-linux-gnueabihf/libarmmem.so

2. in /etc/fstab disable all lines which are for mounting filesystems from SD card (line starting with: /dev/mmcblk0)

After this changes you can run the emulator, eg. like this:

qemu-system-arm -kernel kernel-qemu-4.4.11-jessie -cpu arm1176 -M versatilepb -append "root=/dev/nfs ip=10.0.2.15::10.0.2.1:255.255.255.0 nfsroot=10.0.2.2:/exports/rpi,nfsvers=3,tcp" -clock dynticks -redir tcp:8022::22

The rpi ssh port (22) is redirected to port 8022 on localhost (using a QEMU window as a terminal may be slow and not comfortable).

Here you can find useful information about preparing the Raspberry Pi kernel for the QEMU:
https://github.com/cantora/qemu-arm-rpi-kernel

Comments

  1. Hi, this is great and I’m rather surprised that I’m the first to comment. I would have thought more people would be interested in running from a ramdisk. I’m fairly new to Linux so I wondered if you’d comment on whether this would be suitable for my application. I would like to run Octave from a ramdisk in 64bit Raspberry Pi OS on a Raspi 4 with 4GB of RAM. I have seen the HBFS Robotics article where they run Chromium on a ramdisk and compare with an SD card. But I can’t see what files I’d copy to the ramdisk to do the same with Octave and I wondered if your approach would allow this way of working. Grateful for any tips you may have.

    1. Hi Simon,
      Yeah I think it would be also possible with octave as long as it fits in a ramdisk – but as you have 4GB than you have more then enough. In this case it may be even possible to copy the entire system to the ramdisk 🙂
      I did it for very old raspberry pi – it has very minimal RAM available, now the possibilities are much greater.
      Regarding which files you need to copy – just run:
      ldd octave
      and you’ll see what it is linked against…

  2. Hi Simon,
    Just what I’m looking for. RAM based execution on Raspberry Pi. Will it work on the Pi 4?

    I have been trawling GitHub and come across your hard (home automation rust-daemon) and impressed on the use of threads called from main.rs. I’d like to use the methods for my project with is a controller to control a radio (Ham) repeater.
    I’d like to add a I2C thread to communicate with i2c devices on the bus.
    In addition, how can RS-232 be added to communicate with ardunio for example?
    Furthermore how would turning on device between specific times of the day? On at 06:00 off at 14:00 be coded? I noticed use of floats for timers which gives fine control to timeouts which looks interesting.
    I’m new to rust and so far I’ve enjoyed coding in rust over Python and C..
    Your code is easy to adapt to do what I need.

    Good work and hoped to hear from you.

    Rob

    1. Hi Robert,
      I am not Simon btw 🙂
      Yeah, I don’t see a reason why it should not work on further Pi’s. It should work, definitely.

      Regarding hard: As you can see I am using threads and a mpsc channels to communicate between them. It is one of the many approach available. You can also use async tasks (which I am also using for some of my “threads”). If you want use RS-232, you can just create another task/thread and open a tty device to communicate there. If you need specific RS-232 function I am sure that there is some specific crate for this.
      Turning the device on/off can be done quite easily. In the thread main loop just get hour from the system time of day and check for your range.

      Good luck!

      1. Sorry Manio,
        Must have had a brain f;$t I used the name from the post above mine.

        I have dissected your code and got thinks Woking. Ran into problems with the syntax of
        Std::Threads seems how they are used has change somewhat from your code.
        Let mut threads = vec![] needs to change as well as removing .await from tokio asyn..
        Regards

        Rob

  3. Dear Manio,

    This is really super. I have got it working with a Raspberry pi 4, 3 and 2. I downsize the fs and then I copy the whole fs in memory. Then I adjust the fs with Devuan, so it will deploy with SysVinit. I can run Nodered, Mosquitto, OpenVPN in the Raspberry Pi 4 because of the large memory size, 4GB. In the pi 4 It will sudo apt update and upgrade, pretty awesome. I start the rpi’s with my synology NAS. I wrote several scripts to automate this. With the script I mount the original image file, copy it to a folder and then assign the folder to the specific rpi. If I can buy a Raspberry with 8Gb I can also run Docker. Docker runs from the fs in the NAS but it will give an ‘no memory space’ due to size when starting it from memory. This is the only post on the internet I could find about starting the Raspberry pi in memory. So I am really grateful for it, big thanks!

    Kees

  4. Oh Manio,

    Addition: Raspberry Pi 4 runs with complete LAMP.
    MariaDB, Apache2, PhPmyadmin
    I call it NAMP because I also added NodeRed and Mosquitto.

    Greetings Kees

Leave a Reply to manio Cancel reply

Your email address will not be published. Required fields are marked *