This post started as an attempt to use a USB drive to unlock LUKS encrypted LVM containing, amongst others, the root filesystem. My idea was to be able to use a USB key to unlock the disks of my Proxmox Virtual Environment cluster, falling back to passphrase if the key was missing. This proved to be more complicated than anticipated, although solvable through the use of a custom script I worried about the fragility of this so I abandoned the idea.

Passing USB devices through to a VM in ProxmoxVE

In order to test this setup, I built a basic VM using my established automated install process in my Proxmox Virtual Environment. To test USB key unlocking, I needed to pass a USB key device through to the VM - I will also need to pass through a USB device when I migrate my BackupPC system from its current VM, which uses an external Icy Box IB-RD3620SU3 enclosure that I bought in March 2023 in anticipation of this migration.

Proxmox has a comprehensive wiki page on using USB Devices in Virtual Machines.

Passing through the USB device can be done using the qm command and, at the time of writing, is not supported by Ansible’s community.general.proxmox_kvm module:

--usb[n] [[host=]<HOSTUSBDEVICE|spice>] [,mapping=<mapping-id>] [,usb3=<1|0>] Configure an USB device (n is 0 to 4, for machine version >= 7.1 and ostype l26 or windows > 7, n can be up to 14).

Locating the VM

The qm command needs to be executed on the host the VM is on. Looking up which host a VM is actually on can be done with a pvesh command I found on the Proxmox forum:

It would be absolutely monumental fantastic if there were a way to export all VM’s and Containers running in a ProxMox cluster. Export to CSV, or even print on screen in console a list of VMs. I think there was a console command in the past but I can’t figure it out. … Hey,

pvesh get /cluster/resources --type vm --output-format [text, json, json-pretty, yaml] should be what you’re looking for.

I began by using whoami (for reasons that will become apparent) on each ProxmoxVE host in order to identify those which are up and contactable and then selected one at random for the pvesh command execution:

- name: Identify a working Proxmox Host to use
  delegate_to: '{{ item }}'
  ansible.builtin.command: /usr/bin/whoami
  register: proxmox_host_test
  loop: '{{ groups.proxmox_virtual_environment_hosts }}'
  loop_control:
    label: '{{ item }}'
  ignore_unreachable: true
  changed_when: false  # Read-only operation
  check_mode: false  # Run in checkmode
- name: Select random Proxmox Host from selection
  ansible.builtin.set_fact:
    proxmox_host_result: >-
      {{
        proxmox_host_test.results
        | selectattr('unreachable', 'undefined')
        | ansible.builtin.random
      }}

Once a random ProxmoxVE host has been chosen, I use that to get the list of VMs in the cluster which includes the nodes they are on then filter the list for the current target VM, ensure that only one matching VM is found (as a VM’s name is not guaranteed to be unique) before promoting the one result in the list to be directly assigned to the fact proxmox_vm_info. Using whoami as the test command allows the output to be used to lookup the appropriate “become” (sudo) password for that host form the vault:

# XXX There seems to be a race condition with this on VM creation
- name: VM resources are known
  delegate_to: '{{ hostvars[inventory_hostname].proxmox_host_result.item }}'
  run_once: true
  become: true
  ansible.builtin.command: /usr/bin/pvesh get /cluster/resources --type vm --output-format=json
  register: pvesh_get_vms_output
  vars:
    ansible_become_password: >-
      {{
        lookup(
          'community.hashi_vault.vault_read',
          'kv/hosts/' + hostvars[inventory_hostname].proxmox_host_result.item + '/users/' + hostvars[inventory_hostname].proxmox_host_result.stdout
        ).data.password
      }}
  changed_when: false  # Read-only operation
  check_mode: false  # Run in checkmode
- name: Proxmox VM info is known
  ansible.builtin.set_fact:
    proxmox_vm_info: >-
      {{
        pvesh_get_vms_output.stdout
        | from_json
        | selectattr('name', 'eq', inventory_hostname)
      }}
- name: One VM has been found
  ansible.builtin.assert:
    that: proxmox_vm_info | length == 1
- name: Flatten Proxmox VM info
  ansible.builtin.set_fact:
    proxmox_vm_info: '{{ proxmox_vm_info | first }}'

Current USB configuration

Once the VM host is available, the VM config can be retrieved and the current USB devices extracted in to a dictionary mapping usb device name to config:

- name: VM config is known
  delegate_to: '{{ proxmox_vm_info.node }}'
  become: true
  ansible.builtin.command: "/usr/sbin/qm config {{ hostvars[inventory_hostname].proxmox_vm_info.vmid | quote }}"
  vars:
    ansible_become_password: >-
      {{
        lookup(
          'community.hashi_vault.vault_read',
          'kv/hosts/' + hostvars[inventory_hostname].proxmox_vm_info.node + '/users/' + hostvars[inventory_hostname].proxmox_host_test.results | selectattr('item', 'eq', hostvars[inventory_hostname].proxmox_vm_info.node) | map(attribute='stdout') | first
        ).data.password
      }}
  register: qm_config_out
  changed_when: false  # Read-only operation
  check_mode: false  # Run in checkmode
- name: USB devices are known
  ansible.builtin.set_fact:
    proxmox_vm_usb_devices: >-
      {{
        qm_config_out.stdout_lines
        | select('match', 'usb[0-9]+: ')
        | map('split', ': ', 1)
        | community.general.dict
      }}

Working out free USB devices

To identify which slots are currently available, I loop over the sequence of all possible device names (0 to 15) and check which are configured (i.e. in use). I originally tried to use Ansible’s ansible.builtin.sequence lookup but was struggling to make it work, which led me to a bug report, which included a comment, that said one should just use Jinja’s range function instead:

While this should probably be fully fixed for consistency’s sake, the sequence lookup is somewhat of a relic from the bad old days and Jinja’s standard range should generally be preferred in modern Ansible content.

# In case none are available, make sure initialised.
- name: USB devices is initialised to empty list
  ansible.builtin.set_fact:
    proxmox_vm_usb_slots: []
# From the `qm` man page:
# > --usb[n] [[host=]<HOSTUSBDEVICE|spice>] [,mapping=<mapping-id>] [,usb3=<1|0>]
# > Configure an USB device (n is 0 to 4, for machine version >= 7.1 and ostype l26 or windows > 7, n can be up to 14).
- name: List of available USB devices is known
  ansible.builtin.set_fact:
    proxmox_vm_usb_slots: >-
      {{
        proxmox_vm_usb_slots
        +
        [usb_slot_name]
      }}
  vars:
    usb_slot_name: usb{{ item }}
  loop: "{{ range(0, 15) }}"
  when: "usb_slot_name not in proxmox_vm_usb_devices.keys()"

Removing devices not specified in inventory

I decided that these VMs USB devices will be fully managed by Ansible, so any USB devices not in the inventory will be assumed to have been removed (as opposed to manually added directly in ProxmoxVE) and therefore removed. I used the qm command to do this, and also added any freed up USB device slots to the start of the list of available devices for reuse when adding new/changed devices:

- name: USB devices not specified in inventory are removed
  delegate_to: '{{ proxmox_vm_info.node }}'
  become: true
  ansible.builtin.command: "/usr/sbin/qm set {{ hostvars[inventory_hostname].proxmox_vm_info.vmid | quote }} --delete {{ item.key }}"
  vars:
    ansible_become_password: >-
      {{
        lookup(
          'community.hashi_vault.vault_read',
          'kv/hosts/' + hostvars[inventory_hostname].proxmox_vm_info.node + '/users/' + hostvars[inventory_hostname].proxmox_host_test.results | selectattr('item', 'eq', hostvars[inventory_hostname].proxmox_vm_info.node) | map(attribute='stdout') | first
        ).data.password
      }}
  loop: '{{ proxmox_vm_usb_devices | dict2items }}'
  when: item.value not in proxmox_vm.usb_devices | default([])
- name: Freed device slots are prepended to available slot list
  ansible.builtin.set_fact:
    proxmox_vm_usb_slots: >-
      {{
        [item.key]
        +
        proxmox_vm_usb_slots
      }}
  loop: '{{ proxmox_vm_usb_devices | dict2items }}'
  when: item.value not in proxmox_vm.usb_devices | default([])

Add new USB devices

Finally, I built a list of USB devices specified in the inventory but not configured according qm then add them (after verifying that there are enough USB device slots available for them):

# Ensure initialised, in case there are none
- name: List of missing USB devices is initialised to empty list
  ansible.builtin.set_fact:
    proxmox_vm_missing_usb_devices: []
- name: List of missing USB devices is known
  ansible.builtin.set_fact:
    proxmox_vm_missing_usb_devices: >-
      {{
        proxmox_vm_missing_usb_devices
        +
        [item]
      }}
  loop: '{{ proxmox_vm.usb_devices | default([]) }}'
  when: item not in proxmox_vm_usb_devices.values()
- name: There are enough USB slots available
  ansible.builtin.assert:
    that: proxmox_vm_usb_slots | length >= proxmox_vm_missing_usb_devices | length
- name: Missing USB devices specified in inventory are added
  delegate_to: '{{ proxmox_vm_info.node }}'
  become: true
  ansible.builtin.command: "/usr/sbin/qm set {{ hostvars[inventory_hostname].proxmox_vm_info.vmid | quote }} --{{ item[1] }} {{ item[0] }}"
  vars:
    ansible_become_password: >-
      {{
        lookup(
          'community.hashi_vault.vault_read',
          'kv/hosts/' + hostvars[inventory_hostname].proxmox_vm_info.node + '/users/' + hostvars[inventory_hostname].proxmox_host_test.results | selectattr('item', 'eq', hostvars[inventory_hostname].proxmox_vm_info.node) | map(attribute='stdout') | first
        ).data.password
      }}
  # Shortest list determines length with zip, so this is fine
  loop: '{{ proxmox_vm_missing_usb_devices | zip(proxmox_vm_usb_slots) }}'

USB unlocking of LUKS

The idea I envisaged was to be able to unlock my LUKS encrypted ProxmoxVE hosts with the current per-host passphrase (stored in HashiCorp Vault) or a USB key physically connected to the host. My thinking was that I might move the Vault onto the ProxmoxVE hosts in the future, increasing resilience from the current single-host deployment. However, this will create a catch-22 if all of the hosts are ever turned off as the passphrase for unlocking the encryption would only exist in a vault that can only be accessed by unlocking the hosts - hence the need for an alternative unlock method.

I tested this by creating a VM in my ProxmoxVE cluster, passing through a USB key and setting up crypttab to use the supplied passdev script to unlock. The USB key’s device ID can be found by running lsusb and it happened that my freebie USB memory stick, which I was using, was the same as the example on the Proxmox Wiki (058f:6387).

Creating a key then adding it to be allowed to unlock the encryption was straight forward:

$ mkfs -t ext4 /dev/sdb1
$ mount /dev/sdb1 /mnt
$ dd if=/dev/urandom of=/mnt/luks.key bs=1M count=4
$ cryptsetup luksAddKey /dev/sda3 /mnt/luks.key
$ umount /mnt

I then modified /etc/crypttab, replacing none with the device path of the USB drive by UUID (determined by running blkid), the path of the key file and adding the passdev key script:

sda3_crypt UUID=.... /dev/disk/by-uuid/...:/luks.key luks,discard,keyscript=passdev

Finally, the kernel’s initial ram filesystem needed updating with the new settings:

$ update-initramfs -u

This worked to unlock the system when the USB key is attached but the passdev script does not fall back to passphrase unlocking if the disk is not attached, so with the keyscript set to passdev the only way to unlock the system is with the USB key - which is not what I wanted (if the vault is available, I want to be able to utilise my existing remote unlock script (e.g. for system updates)).

Researching online, I found various suggestions utilising custom scripts for USB drive unlocking and falling back to asking for a passphrase. One of the ones I liked the best was an unlock script that uses underlying OS-provided scripts, which I thought about combining with detecting if the USB key is present as some of the alternatives do. However, I worried about the fragility of using a custom script with something that might not be regularly tested (e.g. if cryptsetup gets updated, I might not detect if USB unlock became broken until it was too late). So, instead, I added a static additional unlock passphrase that I stored in the vault (for the purpose of provisioning) and my Keepass password safe (for the purpose of remote unlocking without the vault).

Adding a static passphrase

I randomly generated a >30 character passphrase and stored it in the vault. I then added a group called static_luks_passphrase, for hosts that would be configured with the static passphrase and added my ProxmoxVE host group (and test VM) to it:

static_luks_passphrase:
  hosts:
    secureboot-test:
  children:
    proxmox_virtual_environment_hosts:

The tasks to add the static passphrase is modelled on my existing playbook for setting the per-host unlock passphrase during bootstrap:

- name: Static LUKS unlock passphrase
  hosts: static_luks_passphrase
  tasks:
    - name: Block device and filesystem types are known
      ansible.builtin.command: /usr/bin/lsblk -o PATH,FSTYPE -J
      register: block_path_type_json
      # Read only operation - never changes anything
      changed_when: false
    - name: Encrypted block devices are known
      ansible.builtin.set_fact:
        encrypted_block_devices: >-
          {{
              (block_path_type_json.stdout | from_json).blockdevices
              |
              selectattr('fstype', 'eq', 'crypto_LUKS')
              |
              map(attribute='path')
          }}
    - name: Only one encrypted device exists
      ansible.builtin.assert:
        that:
          - encrypted_block_devices | length == 1
    - name: Encrypted device name is stored
      ansible.builtin.set_fact:
        encrypted_block_device: "{{ encrypted_block_devices | first }}"
    - name: Static passphrase is set
      become: true
      community.crypto.luks_device:
        new_passphrase: "{{ lookup('community.hashi_vault.vault_read', 'kv/luks/static_passphrase').data.passphrase }}"
        passphrase: "{{ lookup('community.hashi_vault.vault_read', 'kv/hosts/' + inventory_hostname + '/luks/passphrase').data.passphrase }}"
        device: "{{ encrypted_block_device }}"