Improving iPXE configuration
This post is about migrating my iPXE configuration from SaltStack to Ansible, combining a number of others. Most recently automated Debian install, which I had not realised touched on picking up on using Ansible to configure iPXE from 13 months ago. The configuration of iPXE using Ansible I last left off in March when I migrated my monitoring configuration to Ansible. This also relates to previous posts I made on targeted PXE booting and remote image fetching.
iPXE
My existing SaltStack configuration consists of pillar data for each boot option which is then used to build the iPXE configuration file. I will replicate this with a pxe_options variable but this time I will separate Linux installers from utilities and sort each list into alphabetical order by displayed name. I also intend to make the choice of uefi/bios version of memtest+ automatic (rather than presenting both options) based on which version of iPXE was loaded and automatic loading of host/platform specific sub-options, which I intend to include an option to select the “normal” menu too.
---
pxe_options:
- name: debian_11
display: Debian 11 (Bullseye) installer
menu_class: linux_installer
set:
release: bullsye
goto: debian_installer
- name: debian_11_rescue
display: Debian 11 (Bullseye) installer (rescue)
menu_class: linux_installer
set:
release: bullseye
extra_kernel_options: rescue/enable=true
goto: debian_installer
- name: debian_11_preseed
display: Debian 11 (Bullseye) installer (preseeded)
menu_class: linux_installer
set:
release: bullseye
# Full hostname required for url as the Debian installer might not detect the correct domain name.
extra_kernel_options: netcfg/get_hostname=unconfigured-hostname${sp}netcfg/get_domain=unconfigured-domain${sp}auto-install/enable=true${sp}preseed/url=debian-installer.{{ ansible_facts.domain }}
goto: debian_installer
- name: debian_12
display: Debian 12 (Bookworm) installer
menu_class: linux_installer
set:
release: bookworm
goto: debian_installer
- name: debian_12_rescue
display: Debian 12 (Bookworm) installer (rescue)
menu_class: linux_installer
set:
release: bookworm
extra_kernel_options: rescue/enable=true
goto: debian_installer
- name: debian_12_preseed
display: Debian 12 (Bookworm) installer (preseeded)
menu_class: linux_installer
set:
release: bookworm
# Full hostname required for url as the Debian installer might not detect the correct domain name.
extra_kernel_options: netcfg/get_hostname=unconfigured-hostname${sp}netcfg/get_domain=unconfigured-domain${sp}auto-install/enable=true${sp}preseed/url=debian-installer.{{ ansible_facts.domain }}
goto: debian_installer
- name: debian_installer
initrd: "{{ local_mirror.debian | default('http://deb.debian.org/debian') }}/dists/${release}/main/installer-${deb_arch}/current/images/netboot/debian-installer/${deb_arch}/initrd.gz"
chain: "{{ local_mirror.debian | default('http://deb.debian.org/debian') }}/dists/${release}/main/installer-${deb_arch}/current/images/netboot/debian-installer/${deb_arch}/linux initrd=initrd.gz ${extra_kernel_options}"
- name: memtest
display: Memtest86+
menu_class: utility
required_packages:
- memtest86+
required_symlinks:
memtest.pcbios: /boot/memtest86+x64.bin
memtest.efi: /boot/memtest86+x64.efi
chain: memtest.${platform}
...
I created a pxe role whose main.yaml entry point deploys the main configuration, installs any required packages and creates required symlinks in the tftp_root directory. It’s argument specification looks like this:
---
argument_specs:
main:
tftp_root:
description: Root of the tftp server, all files will be created here.
type: str
default: /srv/tftp
pxe_options:
description: PXE boot options
type: list
required: true
elements: dict
options:
name:
description: Name of the option for label in the iPXE configuration file
required: true
type: str
display_name:
description: Display name of the option for the menu (will not be displayed if absent)
type: str
menu_class:
short_description: Menu class the option will be displayed under (will not be displayed if absent).
description: |
Menu class (which list) the option will be displayed under (will not be displayed if absent).
One of:
* linux_installer
* utility.
type: str
choices:
- linux_installer
- utility
set:
description: Dictionary of variables to set
type: dict
goto:
short_description: Which item to goto after doing everything else.
description: >-
Which item to goto after doing everything else. Note that
options which launch another boot loader are unlikely to
reach this statement.
default: '${menu}'
initrd:
description: Location of the initial ramdisk (mostly used by Linuxes)
chain:
description: Load another bootloader - for Linuxes this can be a kernel image (which can be booted directly).
required_packages:
description: List of additional packages to install on the PXE server.
type: list
elements: str
required_symlinks:
short_description: 'Dictionary of symlinks to create in the form `link: target`'
description: >-
Dictionary of symlinks to create in the form `link: target`.
Relative link names will be created relative to the tftp root directory
type: dict
...
defaults/main.yaml is very simple:
---
tftp_root: /srv/tftp
...
The main.yaml tasks file looks like:
---
- name: iPXE is installed
become: true
ansible.builtin.package:
name: ipxe
state: present
- name: iPXE bootloader exists in tftp root
become: true
# tftp-hpa runs chrooted, so doesn't support out-of-tftp-root symlinks
# dnsmasq's tftp server does
#ansible.builtin.file:
ansible.builtin.copy:
#path: '{{ tftp_root }}/{{ item }}'
dest: '{{ tftp_root }}/{{ item }}'
mode: 00444
src: /usr/lib/ipxe/{{ item }}
#state: link
remote_src: true
owner: root
loop:
- ipxe.efi
- snponly.efi
- undionly.kpxe
- name: Packages for iPXE configuration are installed
become: true
ansible.builtin.package:
name: "{{ pxe_options | selectattr('required_packages', 'defined') | map(attribute='required_packages') | flatten | unique }}"
state: present
- name: Symlinks for iPXE configuration are present
become: true
# tftp-hpa runs chrooted, so doesn't support out-of-tftp-root symlinks
# dnsmasq's tftp server does
#ansible.builtin.file:
ansible.builtin.copy:
#path: '{{ tftp_root }}/{{ item.key }}'
dest: '{{ tftp_root }}/{{ item.key }}'
mode: 00444
src: '{{ item.value }}'
#state: link
remote_src: true
owner: root
loop: "{{ pxe_options | selectattr('required_symlinks', 'defined') | map(attribute='required_symlinks') | map('dict2items') | flatten }}"
- name: iPXE configuration file
become: true
ansible.builtin.template:
owner: root
src: ipxe_main.cfg.j2
mode: 00444
dest: '{{ tftp_root }}/ipxe.cfg'
...
The main iPXE configuration template, ipxe_main.cfg.j2:
#!ipxe
# Detect CPU architecture
cpuid --ext 29 && set arch x86_64 || set arch x86
# For Debian - who refers to x86_64 as amd64
cpuid --ext 29 && set deb_arch amd64 || set deb_arch i386
# See https://forum.ipxe.org/showthread.php?tid=7960 for how to add spaces to variables...
set sp:hex 20 && set sp ${sp:string}
set menu main_menu
# Allow overrides before displaying the default menu
chain serial-${serial}.ipxe ||
chain mac-${mac:hexhyp}.ipxe ||
chain host-${hostmame}.ipxe ||
chain product-${product}.ipxe ||
goto ${menu}
:main_menu
menu Please choose a boot option
item --gap Linux installers:
{% for item in pxe_options | selectattr('menu_class', 'defined') | selectattr('menu_class', 'equalto', 'linux_installer') | sort(attribute='display') %}
item itm_{{ item.name }} {{ item.display }}
{% endfor %}
item --gap Utilities:
{% for item in pxe_options | selectattr('menu_class', 'defined') | selectattr('menu_class', 'equalto', 'utility') | sort(attribute='display') %}
item itm_{{ item.name }} {{ item.display }}
{% endfor %}
item --gap iPXE options:
item shell Start iPXE shell
item exit Exit and try next boot device
choose --default exit option && goto ${option}
goto ${menu}
:shell
shell
goto ${menu}
:exit
exit 1
{# Process all of the options #}
{% for item in pxe_options +%}
:itm_{{ item.name }}
{% for (set_item, value) in (item.set | default({})).items() %}
set {{ set_item }} {{ value }}
{% endfor %}
{% if item.initrd | default(false) %}
initrd {{ item.initrd }}
{% endif %}
{% if item.chain | default(false) %}
chain {{ item.chain }}
{% endif %}
goto {% if item.goto | default(false) %}{% if pxe_options | selectattr('name', 'equalto', item.goto) %}itm_{% endif %}{{ item.goto }}{% else %}${menu}{% endif %}
{% endfor %}
I did have to do some creative poking of the inventory to not apply everything (e.g. firewall) to the server in the live network currently hosting the tftp service, as this is still a SaltStack managed system hosting a lot of services I’ve not setup in Ansible yet.
Debian installer preseed
I migrated the Debian preseed source configuration into being managed by Ansible. This is essentially a set of static files served up by a webserver, albeit with some templating expanding on my previous automating Debian install work. I also modified the old recipe to create a dummy LVM logical volume and a post-install script to delete it so as to leave some unallocated space in the volume group (which the Debian installer will not do - it always expends the logical volumes to fully consume the volume group, even in violation of maximum sizes specified in the preseed configuration).
A new debian-installer role was added. Its meta/argument_specs.yaml looks like:
---
argument_specs:
main:
short_description: Configures a webserver to share pre-seed files for Debian Installer.
description: |
Configures a webserver to share pre-seed files for Debian Installer.
At the time of writing, only x86_64 (amd64 in Debian) is supported - the preseed
is hardcoded to install a 64-bit kernel.
author: Laurence Alexander Hurst
options:
d_i_root:
desciption: Location from which to share `/d-i/` on the webserver
type: str
default: /srv/debian-installed
debian_releases:
short_description: List of Debian releases (codenames) to deploy kickstarts for.
description: |
List of Debian releases (codenames) to deploy kickstarts for.
Note that the default for this option may change between role
releases to remain current with Debian old-stable/stable releases.
type: list
elements: str
default:
- bullseye
- bookworm
initial_crypted_root_password:
description: Encrypted root password (suitable to be passed to preseed's `passwd/root-password-crypted` option).
type: str
required: true
initial_root_ssh_keys:
description: Initial ssh keys to allow root access on initial boot.
type: str
required: true
...
The defaults/main.yaml defaults to old-stable and stable and a sensible (/srv based) location for the web root:
---
d_i_root: /srv/debian-installer
debian_releases:
- bullseye
- bookworm
...
The main tasks (tasks/main.yaml) is concerned with creating the files and ensuring they are served from the right location:
---
- name: Debian installer folder exists
become: true
ansible.builtin.file:
path: '{{ d_i_root }}'
owner: root
group: root
mode: 00755
state: directory
- name: debian-installer site exists
ansible.builtin.include_role:
name: webserver
tasks_from: add_site
vars:
site_name: debian-installer
server_name: debian-installer.{{ ansible_facts.domain }}
nginx:
locations:
- location: /d-i/
configuration: |
alias {{ d_i_root }}/;
autoindex on; # Allow directory listings
- name: Releases directories exist
become: True
ansible.builtin.file:
path: '{{ d_i_root }}/{{ release }}'
owner: root # XXX would www-data be a) better and b) mean become_user: www-data would work (least privilege?)
state: directory
loop: '{{ debian_releases }}'
vars:
release: '{{ item }}'
- name: preseed scripts exist (static)
become: true
ansible.builtin.copy:
dest: '{{ d_i_root }}/{{ release }}/{{ file }}'
owner: root # XXX would www-data be a) better and b) mean become_user: www-data would work (least privilege?)
src: '{{ file }}'
mode: 00444
loop: >-
{{
[
'preseed-script-headers',
'preseed-crypto-key',
'preseed-remove-dummy-lv'
]
| product(debian_releases)
}}
vars:
file: '{{ item[0] }}'
release: '{{ item[1] }}'
- name: preseed scripts exist (dynamic)
become: true
ansible.builtin.template:
dest: '{{ d_i_root }}/{{ release }}/{{ file }}'
owner: root # XXX would www-data be a) better and b) mean become_user: www-data would work (least privilege?)
src: '{{ file }}.j2'
mode: 00444
loop: >-
{{
[
'preseed-ssh-setup',
]
| product(debian_releases)
}}
vars:
file: '{{ item[0] }}'
release: '{{ item[1] }}'
- name: preseed configurations exists
become: true
ansible.builtin.template:
dest: '{{ d_i_root }}/{{ release }}/preseed.cfg'
owner: root # XXX would www-data be a) better and b) mean become_user: www-data would work (least privilege?)
src: preseed.cfg.j2
mode: 00444
loop: '{{ debian_releases }}'
vars:
release: '{{ item }}'
...
The templated files are the preseed configuration itself and the ssh key deployment script. This violates my original intent of having a static base install and doing customisation post-install, but in order to do that we need to be able to login which means having known initial credentials and those need to be stored securely (i.e. not committed to revision control!). This does maintain a single source that has these initial, shared, credentials and the per-host configuration is still done post-install. “Why not just take this a step further and do per-host preseed to cut out a step?” is a good question and I struggle with whether to do this or not.
Firstly the preseed configuration, templates/preseed.cfg.j2:
# Localisation
d-i debian-installer/locale string en_GB.UTF-8
d-i keyboard-configuration/xkb-keymap select gb
# Time
d-i time/zone select Europe/London
# HW clock set to UTC?
d-i clock-setup/utc boolean true
# Use NTP during install
clock-setup clock-setup/ntp boolean true
clock-setup clock-setup/ntp-server string ntp
# Network
#d-i netcfg/get_domain string dev.lahtest.local
#d-i netcfg/get_hostname string unassigned-hostname
# Mirror
#d-i mirror/country string GB
#d-i mirror/http/mirror select ftp.uk.debian.org
d-i mirror/country string manual
d-i mirror/protocol string http
d-i mirror/http/hostname string {{ local_mirror.debian | default('http://deb.debian.org/debian') | urlsplit('hostname') }}
d-i mirror/http/directory string {{ local_mirror.debian | default('http://deb.debian.org/debian') | urlsplit('path') }}
d-i mirror/http/proxy string
# Only enable main
d-i apt-setup/non-free boolean false
d-i apt-setup/non-free-firmware boolean false
d-i apt-setup/contrib boolean false
d-i apt-setup/services-select multiselect security, updates
d-i apt-setup/security_host string {{ local_mirror['debian-security'] | default('http://security.debian.org/debian-security') | urlsplit('hostname') }}
d-i apt-setup/security_path string {{ local_mirror['debian-security'] | default('http://security.debian.org/debian-security') | urlsplit('path') }}
# Users
# Skip creation of a normal user account
d-i passwd/make-user boolean false
# Vault stored initial root password
{# XXX fetch this from vault or an argument (populated from vault) #}
d-i passwd/root-password-crypted password {{ initial_crypted_root_password }}
# Partitioning
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
# Skip doing a full erase on underlying crypto volumes
d-i partman-auto-crypto/erase_disks boolean false
d-i partman-auto/method string crypto
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman-auto-lvm/guided_size string max
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto/expert_recipe string \
lah_local_scheme :: \
500 500 500 free \
$iflabel{ gpt } \
$reusmethod{ } \
method{ efi } \
format{ } . \
500 500 500 ext4 \
$defaultignore{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /boot } . \
4000 4000 4000 linux-swap \
$lvmok{ } \
$reusemethod{ } \
method{ swap } \
lv_name{ swap } \
format{ } . \
4000 4000 4000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ / } . \
1000 1000 1000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /home } . \
1000 1000 1000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /opt } . \
1000 1000 1000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /srv } . \
4000 4000 4000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /var } . \
4000 4000 4000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /var/log } . \
1 1 -1 ext4 \
$lvmok{ } \
lv_name{ deleteme } \
method{ keep } .
# Skip (see, e.g., https://serverfault.com/a/1062217):
# No filesystem is specified for partition #1 of LVM VG <hostname>-vg, LV deleteme.
# If you do not go back to the partitioning menu and assign a file system to this partition, it won't be used at all.
d-i partman-basicmethods/method_only false
d-i preseed/include_command string \
for file in \
preseed-script-headers \
preseed-crypto-key \
preseed-ssh-setup \
preseed-remove-dummy-lv \
; do \
wget -P /tmp $( dirname $( debconf-get preseed/url ) )/$file && \
chmod 500 /tmp/$file && \
/tmp/$file; \
done;
d-i preseed/early_command string \
if [ -f /tmp/early-command-script ]; \
then \
/bin/sh /tmp/early-command-script; \
fi;
d-i preseed/late_command string \
if [ -f /tmp/late-command-script ]; \
then \
/bin/sh /tmp/late-command-script; \
fi;
# Base system install
d-i base-installer/kernel/image select linux-image-amd64
# Package selection
# No 'task' selections
tasksel tasksel/first multiselect
# Don't partake in the popularity contest
popularity-contest popularity-contest/participate boolean false
# Bootloader
# This is fairly safe to set, it makes grub install automatically to the MBR
# if no other operating system is detected on the machine.
d-i grub-installer/only_debian boolean true
# Avoid that last message about the install being complete.
d-i finish-install/reboot_in_progress note
...
The ssh key script, templates/preseed-ssh-setup.j2:
#!/bin/sh
# Install and allow remote-root SSH for post-install config
cat - >>/tmp/late-command-script <<EOF
## BEGIN ADDED BY preseed-ssh-setup preseed/include_command
in-target apt-get install -y openssh-server
# Setup initial root ssh keys
mkdir -m 700 /target/root/.ssh
umask 0077
cat - >/target/root/.ssh/authorized_keys <<KEYS_EOF
{{ initial_root_ssh_keys }}
KEYS_EOF
## END ADDED BY preseed-ssh-setup preseed/include_command
EOF
The static files, to generate a dynamic key, add headers to the preseed scripts, and remove the dummy logical volume are stright files pushed out.
The headers script, files/preseed-script-headers:
#!/bin/sh
cat - >/tmp/late-command-script <<EOF
#!/bin/sh
# Late command script
# Header generated by script-headers preseed/include_command
EOF
cat - >/tmp/early-command-script <<EOF
#!/bin/sh
# Early command script
# Header generated by script-headers preseed/include_command
EOF
The initial random crypt key setup:
#!/bin/sh
# ^^ no bash in installer environment, only BusyBox
# I couldn't get this to work with using debconf-set to set
# the passphrase (so this has to be run via
# `preseed/include_command`)
# Die on error
set -e
# Currently (2023-08-10) the Debian installer only supports
# passphrases, GnuPG key files or random key files which are
# regenerated on each reboot. So, to use a random keyfile, we
# initially have to provide the installer with a passphrase.
# Create a passphrase by pulling only characters in the range
# `!` to `~` (ASCII 0x21 to 0x7e) from /dev/random.
TMP_PASSPHRASE_FILE=$( mktemp )
umask 0077
grep -o '[!-~]' /dev/random | tr -d '\n' | head -c64 > $TMP_PASSPHRASE_FILE
# Create an include file for debian-installer with the passphrase as answers to the questions
DEB_INSTALLER_CRYPT_INC_FILE=$( mktemp )
echo -n "d-i partman-crypto/passphrase string " | \
cat - $TMP_PASSPHRASE_FILE > $DEB_INSTALLER_CRYPT_INC_FILE
# Need a newline between the entries
echo >>$DEB_INSTALLER_CRYPT_INC_FILE
echo -n "d-i partman-crypto/passphrase-again string " | \
cat - $TMP_PASSPHRASE_FILE >>$DEB_INSTALLER_CRYPT_INC_FILE
# Newline probably not needed at end of file, but it is neat to add it.
echo >>$DEB_INSTALLER_CRYPT_INC_FILE
# Echo the file to be included, so debian-installer will do
# that - assuming this command is being run via
# `preseed/include_command`. Without file:// will try and fetch
# from the webserver this preseed was served from.
echo "file://$DEB_INSTALLER_CRYPT_INC_FILE"
# Add extra commands to the file that should be run using
# `preseed/late_command` to ensure passphrase is included in
# new install (or the encryption will be un-unlockable as the
# random passphrase will be lost when the installer reboots).
IN_TARGET_KEY_FILE=/etc/keys/luks-lvm.key
cat - >>/tmp/late-command-script <<LATE_EOF
## BEGIN ADDED BY preseed-crypto-key preseed/include_command
umask 0077
mkdir -p /target$( dirname ${IN_TARGET_KEY_FILE} )
cp $TMP_PASSPHRASE_FILE /target$IN_TARGET_KEY_FILE
# Use /root as /tmp might be noexec
cat - >/target/root/configure-crypt-unlock <<EOF
#!/usr/bin/bash
# Standard bash safety features
set -eufo pipefail
if grep -q UMASK /etc/initramfs-tools/initramfs.conf
then
sed -i 's-^#\?UMASK.*\\\$-UMASK=0077-' /etc/initramfs-tools/initramfs.conf
else
echo -e "# Secure initramfs while it contains unlock keys for root filesystem\nUMASK=0077" >>/etc/initramfs-tools/initramfs.conf
fi
# Include keyfile in initramfs
sed -i 's-^#\?KEYFILE_PATTERN=.*\\\$-KEYFILE_PATTERN=$( dirname ${IN_TARGET_KEY_FILE} )/*.key-' /etc/cryptsetup-initramfs/conf-hook
# Configure crypt to use keyfile to unlock encypted partition(s)
sed -i 's#\(UUID=[^ ]\+\) none#\1 ${IN_TARGET_KEY_FILE}#' /etc/crypttab
# Updater initramfs with key file
update-initramfs -u
EOF
chmod 500 /target/root/configure-crypt-unlock
in-target /root/configure-crypt-unlock
rm /target/root/configure-crypt-unlock
## END ADDED BY preseed-crypto-key preseed/include_command
LATE_EOF
Finally, the script to remove the dummy logical volume:
#!/bin/sh
# ^^ no bash in installer environment, only BusyBox
# Die on error
set -e
# Add extra commands to the file that should be run using
# `preseed/late_command` to remove dummy LVM partition.
cat - >>/tmp/late-command-script <<EOF
## BEGIN ADDED BY preseed-remove-dummy-lv preseed/include_command
lvchange -a n $( hostname -s )-vg/deleteme
lvremove $( hostname -s )-vg/deleteme
## END ADDED BY preseed-remove-dummy-lv preseed/include_command
EOF
Finishing up
I added values to read the security configuration from my hashicorp vault, in group_vars/debian_installer_sources.yaml, the recipe for ansble.builtin.password_hash is stright from the documentation and makes the hash generation idempotent but using a unique seed for each host:
---
initial_crypted_root_password: "{{ lookup('community.hashi_vault.vault_read', 'kv/install/initial_root_password').data.password | ansible.builtin.password_hash('sha512', 65534 | random(seed=inventory_hostname) | string) }}"
initial_root_ssh_keys: "{{ lookup('community.hashi_vault.vault_read', 'kv/install/initial_root_keys').data.ssh_keys }}"
...
For ansible.builtin.password_hash, I need to add passlib to requirements.txt:
# For encryption (e.g. ansible.builtin.password_hash)
passlib
I added both (iPXE Servers and Debian Installer Sources) groups to inventory.yaml:
ipxe_servers:
hosts:
starfleet-command:
debian_installer_sources:
hosts:
starfleet-command:
And specified the roles should be applied to these groups in site.yaml:
- hosts: ipxe_servers
roles:
# Uses pxe_options from ipxe_servers group vars
- pxe
- hosts: debian_installer_sources
roles:
# Uses these variable from from debian_installer_source group vars:
# - initial_crypted_root_password
# - initial_root_ssh_keys
- debian-installer