This post describes the process of enabling boot-from-USB and then using a USB disk for booting my Raspberry PI 4. It was made from notes I jotted down when I did this, over a week ago, so I apologise if it seems a bit disjointed or bulleted.

Background

Since setting up my Raspberry PI 4, it has failed with filesystem read errors a total of 4 times using 3 different Micro-SD cards (2 of which supposedly have lifetime warranties). I have therefore concluded that the Micro-SD cards are too unreliable and instead looked to the spare M.2 SSD in a USB caddy that I last used while upgrading my laptop’s SSD.

Enable booting from USB

My Raspberry PI is sufficiently old it requires a newer bootloader to boot from USB. Updating this seems to require (or at least all the instructions directed me to) the raspi-config tool, which is included in Raspberry Pi OS (formerly known as Raspbian).

The first stage is to install Raspberry PI OS onto an SD Card (I used Fedora Media Writer, again) and boot the PI.

Once booted, login with default username of pi and password raspberry before running raspi-config (from memory, I think this needs to be run as root - the pi user can sudo it).

In raspi-config:

  1. Go to “8. Update” to make sure everything is up to date.
  2. Go to “6. Advanced Options”, followed by “A7. Bootloader Version” and select “E1. Latest”

We are done with updating the settings, so power off Raspberry PI and remove the SD Card.

Install OS on a USB disk

Next, I needed to install my Debian system onto the USB drive. This is identical to getting it onto an SD card, just copy system image to the USB disk (another job for Fedora Media Writer).

Plug the USB driver into Raspberry PI and insert a blank SD Card or you will find your logs filled with timeouts for a non-existent SD card. There is an alternative (as described in that link), which involves disabling the SD card slot, if there is no card, however that does make it unusable as an extra storage option.

Turn on the Raspberry PI and I then followed the directions from my previous blog post as far as apt-get dist-upgrade.

After doing an apt-get dist-upgrade, which will upgrade the kernel to 5.10.0-0.bpo.7, the OS will be hit by bug 977694 and unable to boot until the reset_raspberrypi module is added to the initial ramdisk (so do this before trying to reboot):

echo "reset_raspberrypi" > /usr/share/initramfs-tools/modules.d/raspi-reset.conf
update-initramfs -uk all

Once these commands have been run, the PI can be rebooted into its new install.

Restore from last backup

After setting up the new OS, I wanted to get it back to the state it was in before it failed (to recover the monitoring data). In doing so, I was reminded that an untested backup maybe as good as no backup and really should make the time to create and test a proper DR plan for all of my kit.

I tried a direct restore but it never completed, so instead I downloaded the last backup (excluding /boot) as a tar and simply extracted it over the root (/). Even after restoring /etc/passwd, tar screwed up mapping ownership during the extract, so I ended up using --numeric-owner which extracted everything with the correct ownership.

I then found out all about Linux file capabilities, which are stored in the extended attributes that BackupPC version 3 (which I am using) does not support:

Long story short, the way ‘ping’ allows unprivileged users to run it was recently changed from suid [to] file capabilities. Taking a filesystem and tarring it does not preserve those capabilities within the archive. Therefore, any extract from such a tar file will result in a system that is missing the capabilities that allow regular users to use ping.

On a similar Debian system I located all files with capabilities and reproduced them on the restored system (as a belts-and-braces, I reinstalled the whole thing and did the same search on the PI before attempting to restore the backup again - I found not other files than on my x86_64 Debian systems). This command finds all files with capabilities (excluding /boot/efi and /boot/firmware makes it work on both x86 based UEFI systems and the Raspberry PI):

sudo find / -type f -not -path '/proc/*' -not -path '/sys/*' -not -path '/run/*' -not -path '/boot/efi/*' -not -path '/boot/firmware/*' -name "*" -exec getcap {} +

Once the capabilities are restored, using setpcap, I rebooted the PI and everything seems to have been working fine since.