RPi: creating a RAM-disk running Linux environment from NFS booted Raspbian

Sometimes you need a very specific use case for your Raspberry Pi.
The durability (and speed) of the SD card used in RPi may vary.
Because of this I always tend to move Raspbian to an NFS share, and boot via NFS (root over NFS).
This way I have the following advantages:
1. The SD card is only necessary to boot (single boot partition is sufficient)
2. Main root filesystem is on fast and durable NFS mount on the server
3. I am not limited by the SD card space in any way
Doing the development/testing work is really nice this way.

The big disadvantage of this approach was the problems when the NFS server was gone for some reason.
Generally the rpi was able to raise up after this and replace the NFS session (eg. after server reboot) with the following kernel messages:

kernel: nfs: server <servername> not responding, still trying
kernel: nfs: server <servername> OK

But in some cases it was unable to do this and stucked:

kernel: nfs: server <servername> not responding, still trying
kernel: nfs: server <servername> not responding, timed out

And in those cases a hard RaspberryPi reset was necessary to boot it up again.

Obviously in my automation use case I wanted the RPi to run independently and stable on its own.
I did not want to put the whole system on the SD card, mainly due the SD durability and speed (and also limited storage space).

So I created a mixed solution, which I want to share below. It is a perfect one for me. My plan was to boot the system from the SD card via NFS (boot partition can be readonly – no writes to the SD card), then move only the necessary files to the RAM disk of the Raspberry Pi.
And this is possible with pivot_root. 🙂

Keep in mind that this method may be not suitable for some bigger projects with a lot of dependencies on external libraries.
In my case the binary which I am running on the RPi is just a C++ program which is doing the automation, so it is really small and not depend on many external libraries (rather – the standard ones).
It’s also important to know that the RAM usage on the RPi has to be monitored (at least after deploying) – more libraries and executables we put into RAM disk, less space is available to running programs. In my case of RPi model B rev 2.0 my RAM usage is about 11MB of the total 118MB available memory.
OK, Let’s start from the beginning:

1. Preparing the SD card

At the time I was doing the work the latest Raspbian version available was:
2016-05-27-raspbian-jessie-lite.img
Of course I choosed the lite version because I don’t need a full distro. The rest of this “howto” is based on this specific version, but if nothing significant changed, I hope it should work with the recent versions as well 🙂
Firstly I had to discover the partition list of downloaded SD card image:

$ fdisk -l 2016-05-27-raspbian-jessie-lite.img
 Disk 2016-05-27-raspbian-jessie-lite.img: 1.3 GiB, 1387266048 bytes, 2709504 sectors
 Units: sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disklabel type: dos
 Disk identifier: 0x6fcf21f3

Device                               Boot  Start     End Sectors  Size Id Type
 2016-05-27-raspbian-jessie-lite.img1        8192  137215  129024   63M  c W95 FAT32 (LBA)
 2016-05-27-raspbian-jessie-lite.img2      137216 2709503 2572288  1.2G 83 Linux

So we have a boot partition started at sector 8192 and a root filesystem at sector 137216.
To get the exact offset in bytes we have to multiply the sector number by sector size (512).

I created a loop device for the boot partition:

losetup --offset 4194304 /dev/loop2 2016-05-27-raspbian-jessie-lite.img

and mounted afterwards into raspbian_boot directory:

mount /dev/loop2 ./raspbian_boot

I have access to original boot files from Raspbian (raspbian_boot), so it was a time to prepare the target location for files. I found the old 16MB SD card (bundled with my canon camera) and since it should be sufficient for boot partition, I just gave it a try.
Indeed, it was working properly, I just had to format the boot partition as FAT16, because there was not enough space for FAT32.
After creating a target partition, I stripped the original Raspbian boot files to the really needed necessary ones for my Raspberry Pi model B.
It was sufficient for me to copy the following files to my target boot partition:
bcm2708-rpi-b.dtb
bootcode.bin
cmdline.txt
config.txt
kernel.img
start.elf

I edited the config.txt and enabled the i2c by commenting out the line:

dtparam=i2c_arm=on

Then I modified the original cmdline.txt and enabled NFS boot:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs ip=dhcp nfsroot=192.168.0.1:/exports/rpi,nfsvers=3,tcp elevator=deadline rootwait

After unmouning, the SD card was ready to insert into Raspberry Pi.

2. Preparing NFS share on server

Then I was able to mount the main filesystem from the raspbian image:

losetup --offset 70254592 /dev/loop2 2016-05-27-raspbian-jessie-lite.img
mount /dev/loop2 ./raspbian_rootfs

I copied all files to my local directory of the NFS export:

rsync -xav raspbian_rootfs/ /exports/rpi/

and added the following line to the /etc/exports:

/exports/rpi 192.168.0.4/32(rw,no_root_squash,async,no_subtree_check)

and reloaded the exports with:

exportfs -ra

I also added in the /etc/fstab in the RPi rootfs the line:

/dev/nfs        /               rootfs  defaults

while also commented out the original rootfs mount (/dev/mmcblk0p2)

At this stage a normal “full” Raspbian should be able to boot from NFS on the Raspberry Pi.

3. Preparing to pivot_root

On the Raspberry, I’ve installed the dropbear and busybox-static packages:

apt install dropbear busybox-static

After installation, all necessary files was in place.
In the root NFS directory I created a three scripts for doing all necessary jobs:

Script called root2ram.sh is for doing early work and pivot_root:

In cmdline.txt on the SD card I changed the initial script run on boot, appending: “– \root2ram.sh”. Full line after this change was:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs ip=dhcp nfsroot=192.168.0.1:/exports/rpi,nfsvers=3,tcp elevator=deadline rootwait -- /root2ram.sh

When the rpi is doing the pivot_root switch it is launching the busybox.
Busybox is in turn starting /etc/init.d/rcS which in my case has the following contents:

The last script is fs_prepare which is called from rcS. It is used for the necessary file copy for the new filesystem. You have to figure out yourself what libraries and binaries are necessary for your configuration (ldd tool is your friend).
In my case the file has the following contents:

After all this steps – my Raspberry Pi successfuly booted from NFS to ramdisk.
I am then able to umount the original NFS with this command:

umount /old_root

And the RPi is still running fine 🙂
I can ssh into it (it has the dropbear running) and it is working really fast, because the whole system is in RAM.
When I needed to do some jobs (eg. compilation) on the “original” NFS system, then I am just mounting it with the following command:

mount -t nfs -o nolock 192.168.0.1:/exports/rpi /old_root

and then chroot into it:

chroot /old_root /bin/bash

Bonus: Running it under QEMU

You can also boot this NFS share under QEMU. This way you can test all changes before running it on the real RaspberryPi.
To be able to run it, you need to install the qemu-system-arm package in your linux distro.
You also need a modified kernel. In my case (for the above Raspbian version) I had to use a kernel named: kernel-qemu-4.4.11-jessie.
Which at the time of writing is available for download from this github repo:
https://github.com/dhruvvyas90/qemu-rpi-kernel

In the NFS share are though required some small changes:
1. in /etc/ld.so.preload you need to comment the line:

/usr/lib/arm-linux-gnueabihf/libarmmem.so

2. in /etc/fstab disable all lines which are for mounting filesystems from SD card (line starting with: /dev/mmcblk0)

After this changes you can run the emulator, eg. like this:

qemu-system-arm -kernel kernel-qemu-4.4.11-jessie -cpu arm1176 -M versatilepb -append "root=/dev/nfs ip=10.0.2.15::10.0.2.1:255.255.255.0 nfsroot=10.0.2.2:/exports/rpi,nfsvers=3,tcp" -clock dynticks -redir tcp:8022::22

The rpi ssh port (22) is redirected to port 8022 on localhost (using a QEMU window as a terminal may be slow and not comfortable).

Here you can find useful information about preparing the Raspberry Pi kernel for the QEMU:
https://github.com/cantora/qemu-arm-rpi-kernel