After restoring the router, the next step I needed to complete was to restore my configuration management server in order to start orchestrating the rest. On the live network this resides in a virtual machine on my HP Microserver. This means I need to restore the host and the VM to get this up and running.

Network configuration

Since this system is being restored in a working network, now the router has been restored, I began by configuring the network port onto main network:

lab-switch(config)#interface gigabitEthernet 1/0/17
lab-switch(config-if)#description starbase2
lab-switch(config-if)#switchport pvid 20
lab-switch(config-if)#switchport general allowed vlan 20 untagged
lab-switch(config-if)#no switchport general allowed vlan 1
lab-switch(config-if)#exit

Preparing the system

As with the router, I updated and configured the BIOS on the system before I started.

Partitioning the disk

Next I configured the disk using the same process as for the router - in this case, the live system has both (software) RAID 1 and LUKS encryption on its disk which I am currently not mirroring into the lab environment. In the case of encryption, I set this up so I would not have to worry as much about disposing of disks when they fail - a concern that equally applies to the lab kit however the ILO make remotely entering the passphrase to unlock the encryption key very convenient and the same is not true of the very cheap lab kit. I will need to ponder this decision some more, as I do want to turn off the lab when not in use but not have to go down each machine entering a passphrase locally every time I power them on. The RAID is to provide redundancy as this system hosts VMs for both the backup and configuration systems, both of which are amongst the most critical in my home network - the lab environment as a whole is non-critical so this concern does not apply.

The live system also has a BIOS with no EFI support - I reconfigured this particular lab machine to boot ‘legacy only’ (instead of EFI only) and partition accordingly. I made var_lib slightly smaller (190G vs 200G) than the live system due to the lab machine having a smaller disk and it not fitting otherwise:

parted /dev/sda mklabel msdos
parted /dev/sda mkpart primary ext4 1001 5000
parted /dev/sda toggle 1 boot
parted /dev/sda mkpart primary 5000 100%
parted /dev/sda toggle 2 lvm
pvcreate /dev/sda2
vgcreate starbase2 /dev/sda2
lvcreate -L 190G -n var_lib starbase2
lvcreate -L 30G -n swap starbase2
lvcreate -L 20G -n var starbase2
lvcreate -L 20G -n home starbase2
lvcreate -L 10G -n srv starbase2
lvcreate -L 10G -n usr starbase2
lvcreate -L 5G -n root starbase2
lvcreate -L 5G -n tmp starbase2

Then formatting the partitions:

# Create filesystems
mkfs.ext4 -L starbase2-boot /dev/sda1
mkfs.ext4 -L starbase2-home /dev/mapper/starbase2-home
mkfs.ext4 -L starbase2-srv /dev/mapper/starbase2-srv
mkfs.ext4 -L starbase2-var /dev/mapper/starbase2-var
mkfs.ext4 -L starbase2-var_lib /dev/mapper/starbase2-var_lib
mkfs.ext4 -L starbase2-usr /dev/mapper/starbase2-usr
mkfs.ext4 -L starbase2-root /dev/mapper/starbase2-root
mkfs.ext4 -L starbase2-tmp /dev/mapper/starbase2-tmp
mkswap -L starbase2-swap /dev/mapper/starbase2-swap

Mount the formatted devices:

# Mount new devices
swapon /dev/mapper/starbase2-swap
mount -t ext4 /dev/mapper/starbase2-root /mnt
for volume in /dev/mapper/starbase2-*
do
    short_vol=$(basename $volume | sed 's/^[^-]\+-//')
    if [[ $short_vol != 'swap' ]] && [[ $short_vol != 'root' ]] && [[ $short_vol != 'var_lib' ]]
    then
        mkdir /mnt/$short_vol
        mount -t ext4 $volume /mnt/$short_vol
    fi
done
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot
mkdir /mnt/var/lib
mount /dev/mapper/starbase2-var_lib /mnt/var/lib

Restoring the backup

In order to transfer the restored backup file, as I have a working network since restoring the router, I used netcat after (again) downloaded the backup as a tar archive on the system with the DR backup mounted. In the longer term I think I need to investigate creating a custom Debian install image based on the minimal (net install) image that adds (at least) an ssh client, ssh server, rsync and telnet client but remains small enough to fit on my usual Debian install USB drive (which is 1GB in capacity) which will enable the possibility of direct restores and managing the switch.

To start the receiving end:

nc -l -p 9000 | tar -C /mnt

To send (where 192.168.20.154 is the IP address of the receiving host):

cat /path/to/restore_starbase2.home.entek.org.uk_*.tar | busybox nc 192.168.20.154 9000

Post restore configuration

The UUIDs for disks in /mnt/etc/fstab, ``/mnt/boot/grub/grub.cfg need updating - lsblk -o name,uuid,partuuid or ls -l /dev/disk/by-uuid can help find the new ones. I also blanked /mnt/etc/crypttab` as not (currently) using encrypted filesystem on this device.

The files in /mnt/etc/network/interfaces.d were edited and renamed to match the new system’s interface names (in my case bond0, eno1 and eno2 were deleted, enp2s0 created with configuration iface enp2s0 inet manual and br0 updated with bridge_ports enp2s0 instead of bridge_ports bond0 as the new system isn’t dual-homed (and hence not using bonded interfaces).

I then created the missing mount points:

mkdir /mnt/dev /mnt/proc /mnt/sys /mnt/run

Next, to try and pre-empt the boot problem I had with the router, I chrooted into the environment and ran grub-install:

mount -t proc none /mnt/proc
mount -o bind /dev /mnt/dev
mount -o bind /sys /mnt/sys
mount -o bind /run /mnt/run
chroot /mnt /bin/bash
grub-install /dev/sda
exit

Reboot

Finally, I unmounted everything in the new system image and shutdown the machine:

umount /mnt/var/lib
umount /mnt/*
umount /mnt
swapoff /dev/mapper/starbase2-swap
poweroff

After removing the USB boot drive, I powered it back on and tried to boot into the restored system - this time it failed because the encrypted volume that the initial ram disk was trying to unlock could not be found (so the grub-install had worked fine). I booted off the install media again, into rescue mode and ran update-initramfs from within a chroot to the restored system. After this it booted fine.