For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

The original plan was this:

  • iSCSI target for the underlying storage
  • LUKS encrypted partition on top
  • LVM on top of LUKS (purely so we can do snapshotting)
  • BackupPC uses the LVM logical volume for it’s store
  • Make off-site replica:
    • Periodically take snapshot of the LVM volume
    • use clonezilla(this didn’t workout - see below) to duplicate the point-in-time snapshot to an external disk (also encrypted)
    • store off-site (rotating 2 disks to ensure 1 is off-site at all times)
    • remove snapshot

Quick and dirty “how to mount and run backuppc”

If the machine gets rebooted (currently) the BackupPC file-system does not remount automatically (not least because it requires the luks pass-phrase, and I do not want to halt boot on this headless box to wait for it). To manually mount once the system is up:

## Login to iSCSI
sudo iscsiadm --mode node --targetname "iqn.1994-11.com.netgear:isolinear:6349f3fd:backuppc" --login
## Open encrypted filesystem (see lsblk to locate the filesystem)
sudo cryptsetup luksOpen /dev/sdc1 backuppc-pv
## LVM will automagically have found the volume group and logical volume, so it can just be mounted (assuming /etc/fstab is correct)
sudo mount /var/lib/backuppc
## Start the Backuppc service
sudo systemctl start backuppc

How to unmount the partition

If required, the sequence to cleanly unmount and disconnect the iSCSI disk is (usually only needed for planned maintenance on the NAS):

## Stop Backuppc
sudo systemctl stop backuppc
## Unmount the filesystem
sudo umount /var/lib/backuppc
## Shutdown the volume group
sudo vgchange -a n backuppc
## Close the luks volume
sudo cryptsetup luksClose backuppc-pv
## Close the connection to the iSCSI server (target)
sudo iscsiadm --mode node --targetname "iqn.1994-11.com.netgear:isolinear:6349f3fd:backuppc" --logout

Setting up

iSCSI initiator (client)

Firstly install open-iscsi:

apt-get install open-iscsi

Next, run a discovery against the machine with the target we want to use:

iscsiadm -m discovery -t st -p remote_target_hostname_or_ip

Configure credentials:

target="some_target_name_from_discovery"
iscsiadm --mode node --targetname $target --op=update --name node.session.auth.authmethod --value=CHAP
iscsiadm --mode node --targetname $target --op=update --name node.session.auth.username --value=$Id
iscsiadm --mode node --targetname $target --op=update --name node.session.auth.password --value=$MDP

Login (creates device in /dev on success)

iscsiadm --mode node --targetname "some_target_name_from_discovery" --login

Disconnect

iscsiadm --mode node --targetname "some_target_name_from_discovery" --logout

Check if a session exists for a given target

iscsiadm -m session | grep ' some_target_name_from_discovery '

Partition target

The iSCSI target will appear as a plain block device, so use parted to create a single partition starting at 0% and ending at 100%. We will use this for the LUKS encrypted volume:

# parted /dev/sde
GNU Parted 3.2
Using /dev/sde
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) mkpart
Partition name?  []? BackupPC
File system type?  [ext2]? ext4
Start? 0%
End? 100%
(parted) quit

Encryption

Setup a new LUKs encrypted volume

# cryptsetup luksFormat /dev/sde1

WARNING!
========
This will overwrite data on /dev/sde1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:

Open an existing LUKs encrypted volume

# cryptsetup luksOpen /dev/sde1 backuppc-pv
Enter passphrase for /dev/sde1:

LVM

Install the LVM tools (if not already installed):

which pvdisplay || apt-get install lvm2

Set-up the physical volume (decrypted part):

# pvcreate /dev/mapper/backuppc-pv
  Physical volume "/dev/mapper/backuppc-pv" successfully created.

Create the volume group:

# vgcreate backuppc /dev/mapper/backuppc-pv
  Volume group "backuppc" successfully created

Create the logical volume (note only using 4TB (3725GB) of the 4.5TiB volume group, so it will fit on a 4TB external disk (as opposed to 4TiB) and we have space for snapshots):

# lvcreate -n store -l 3725G backuppc
  Logical volume "store" created.

FileSystem

At this point I would created a filesystem on the device, however I have managed to duplicate the fail(ing) old BackupPC disk so instead I will be copying that onto the new device with Clonezilla. Either way the encrypted device is now ready to use.

Clone with partclone

N.B. Cloning with partclone and clonezilla both resulted in corrupt ext4 filesystems on the clone (as reported by fsck). e2image did not do this and seems more reliable, which is important for a backup!

Install if required:

apt-get install partclone

partclone.ext4 -b -I -s /dev/sdd1 -O /dev/backuppc/store

Options I used are:

  • b Local device to device copy mode
  • -I Ignore filesystem check
  • -s Source FILE
  • -O Output FILE, overwriting if exists

Clone with e2image (included with e2fsprogs)

sudo e2image -ra -pc /dev/sdc1 /dev/mapper/backuppc-store

Create a filesystem

mkfs.ext4 /dev/backuppc/store

Mount the filesystem

Add the right path to the block device to fstab:

/dev/mapper/backuppc-store  /var/lib/backuppc   defaults,noauto   0  2

And mount the volume:

mount /var/lib/backuppc

Install backuppc

This needs to be done after mounting /var/lib/backuppc, so the package’s files get dropped in the right place

apt-get install backuppc

Off-site backup (clone of /var/lib/backuppc)

N.B. TODO automate this

copy config to /var/lib/backuppc

In case we want to restore on another machine, we need the contents of /etc/backuppc also in /var/lib/backuppc (which gets cloned).

sudo tar -zcf /var/lib/backuppc/etc-backuppc-new.tgz /etc/backuppc && sudo mv /var/lib/backuppc/etc-backuppc-new.tgz /var/lib/backuppc/etc-backuppc.tgz

Take LVM snapshot

The size of the snapshot is the maximum amount of change that can occur to the volume that has been snapshotted (as it’s a copy-on-write system) during it’s lifetime. The snapshot will automatically be deleted if it becomes full.

lvcreate -L500G -s -n backup /dev/mapper/backuppc-store

Setup receiving volume

Unlock the encrypted USB drive

cryptsetup luksOpen /dev/sdd1 backuppc-backup-removable

N.B. We copy directly to the encrypted container - no need for snapshots on the removable (off-site) backup, so using LVM adds an unnecessary layer (and hence complexity/risk) to the backup.

Clone snapshot

e2image -ra -pc /dev/mapper/backuppc-backup /dev/mapper/backuppc-backup-removable

Check clone (fsck)

fsck -t ext4 /dev/mapper/backuppc-backup-removable -f -n

Eject disk

cryptsetup luksClose /dev/mapper/backuppc-backup-removable
udisksctl power-off -b /dev/sdd

Delete snapshot

lvremove /dev/mapper/backuppc-backup