Jekyll2023-08-31T18:33:51+01:00https://blog.entek.org.uk/feed.xmlLaurence’s BlogPossibly the most boring blog on the web... The random musings, technical notes, life events, etc. of a lapsed *nix sysadmin come research software engineer.LaurenceIcinga2 check for Hashicorp Vault2023-08-31T18:29:27+01:002023-08-31T18:29:27+01:00https://blog.entek.org.uk/notes/2023/08/31/icinga2-check-for-hashicorp-vault<p>When restarted, <a href="https://www.vaultproject.io/">Hasicorp Vault</a> starts “sealed” and has to be unsealed to make the contents accessible. I have, on occasion, forgotten to do this (usually after a reboot for kernel update) and so I want to add checks for this situation to my monitoring. My current monitoring solution is <a href="https://icinga.com/">Icinga</a>, so I am creating a check for this.</p>
<h2 id="the-check-script">The check script</h2>
<p>This is a simple <a href="https://www.gnu.org/software/bash/">bash</a> script that uses Vault’s <code class="language-plaintext highlighter-rouge">vault</code> command and <a href="https://jqlang.github.io/jq/">jq</a> to check the vault at a given URL is contactable and is seal status.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="k">if</span> <span class="o">!</span> which vault &>/dev/null
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Unable to locate vault command line client - cannot continue"</span> <span class="o">></span>&1
<span class="nb">exit </span>3 <span class="c"># Usage/internal error</span>
<span class="k">fi
if</span> <span class="o">!</span> which jq &>/dev/null
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Unable to locate jq (to parse vault output) - cannot continue"</span> <span class="o">></span>&1
<span class="nb">exit </span>3 <span class="c"># Usage/internal error </span>
<span class="k">fi
if</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">==</span> <span class="s2">"--help"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Usage: </span><span class="nv">$0</span><span class="s2"> [-v [-v]] vault_url_to_check"</span>
<span class="nb">echo</span> <span class="s2">"This plugin requires vault and jq commands to be installed, and"</span>
<span class="nb">echo</span> <span class="s2">"network access to the vault."</span>
<span class="nb">echo</span> <span class="s2">"-v: increase the verbosity level one level for each '-v' (up to 2)"</span>
<span class="nb">exit </span>0
<span class="k">fi
</span><span class="nv">VERBOSITY_LEVEL</span><span class="o">=</span>0
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">==</span> <span class="s2">"-v"</span> <span class="o">]]</span>
<span class="k">then
while</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">==</span> <span class="s2">"-v"</span> <span class="o">]]</span>
<span class="k">do
</span><span class="nv">VERBOSITY_LEVEL</span><span class="o">=</span><span class="k">$((</span> VERBOSITY_LEVEL <span class="o">+</span> <span class="m">1</span> <span class="k">))</span>
<span class="nb">shift
</span><span class="k">done
if</span> <span class="o">[[</span> <span class="nv">$VERBOSITY_LEVEL</span> <span class="nt">-gt</span> 2 <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Cannot increase verbosity beyond 2!"</span>
<span class="c"># Could just set it down to 2 but this is a usage error...</span>
<span class="nb">exit </span>3 <span class="c"># Usage/internal error</span>
<span class="k">fi
fi
</span><span class="nv">VAULT_ADDRESS</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nt">-z</span> <span class="s2">"</span><span class="k">${</span><span class="nv">VAULT_ADDRESS</span><span class="k">}</span><span class="s2">"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"No vault url provided - cannot continue"</span> <span class="o">></span>&2
<span class="nb">exit </span>3 <span class="c"># Usage/internal error</span>
<span class="k">fi
if</span> <span class="o">[[</span> <span class="nv">$VERBOSITY_LEVEL</span> <span class="nt">-ge</span> 2 <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Checking vault at </span><span class="k">${</span><span class="nv">VAULT_ADDRESS</span><span class="k">}</span><span class="s2">."</span>
<span class="k">fi
</span><span class="nv">VAULT_OUTPUT</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>vault status <span class="nt">-address</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">VAULT_ADDRESS</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-format</span><span class="o">=</span>json 2>&1<span class="si">)</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">VAULT_OUTPUT</span><span class="k">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"Error"</span><span class="k">*</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"CRITICAL: Error connecting to vault: </span><span class="k">${</span><span class="nv">VAULT_OUTPUT</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">exit </span>2 <span class="c"># Critical</span>
<span class="k">fi
</span><span class="nv">VAULT_SEALED</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">VAULT_OUTPUT</span><span class="k">}</span><span class="s2">"</span> | jq <span class="nt">-r</span> .sealed <span class="si">)</span><span class="s2">"</span>
<span class="nv">VAULT_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">VAULT_OUTPUT</span><span class="k">}</span><span class="s2">"</span> | jq <span class="nt">-r</span> .version <span class="si">)</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$VERBOSITY_LEVEL</span> <span class="nt">-ge</span> 1 <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Vault version is </span><span class="k">${</span><span class="nv">VAULT_VERSION</span><span class="k">}</span><span class="s2">."</span>
<span class="k">fi
if</span> <span class="o">[[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">VAULT_SEALED</span><span class="k">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"false"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"OK: Vault is unsealed (version </span><span class="k">${</span><span class="nv">VAULT_VERSION</span><span class="k">}</span><span class="s2">)"</span>
<span class="nb">exit </span>0 <span class="c"># All good</span>
<span class="k">else</span>
<span class="c"># Vault is sealed...</span>
<span class="nv">VAULT_THRESHOLD</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">VAULT_OUTPUT</span><span class="k">}</span><span class="s2">"</span> | jq <span class="nt">-r</span> .t <span class="si">)</span><span class="s2">"</span>
<span class="nv">VAULT_UNSEAL_PROGRESS</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">VAULT_OUTPUT</span><span class="k">}</span><span class="s2">"</span> | jq <span class="nt">-r</span> .progress <span class="si">)</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"WARNING: Vault is sealed"</span> <span class="se">\</span>
<span class="s2">"(</span><span class="k">${</span><span class="nv">VAULT_UNSEAL_PROGRESS</span><span class="k">}</span><span class="s2">/</span><span class="k">${</span><span class="nv">VAULT_THRESHOLD</span><span class="k">}</span><span class="s2"> unseal keys provided,"</span> <span class="se">\</span>
<span class="s2">"version </span><span class="k">${</span><span class="nv">VAULT_VERSION</span><span class="k">}</span><span class="s2">)"</span>
<span class="nb">exit </span>1 <span class="c"># Warning</span>
<span class="k">fi</span>
</code></pre></div></div>
<h2 id="icinga-configuration">Icinga configuration</h2>
<p>This is modelled on the <a href="/notes/2021/08/25/checking-nextcloud-version.html">Nextcloud version check</a> I previously created.</p>
<p>Firstly the command needs to be defined, which I did in <code class="language-plaintext highlighter-rouge">commands-hashicorp-vault.conf</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>object CheckCommand "hashicorp-vault" {
import "plugin-check-command"
command = [ PluginContribDir + "/check_hashicorp_vault", "$vault_url$" ]
}
</code></pre></div></div>
<p>And the service mapped to hosts with a list of vault urls to check defined, which I did in <code class="language-plaintext highlighter-rouge">services-hashicorp-vault.conf</code> (in addition to the existing <a href="/notes/2023/01/23/adding-a-bastion-host-deploying-hashicorp-vault.html#monitoring-backup">backup age check for the vault servers themselves</a>):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apply Service for (vault_url in host.vars.vault_urls) {
import "generic-service"
check_command = "hashicorp-vault"
vars.vault_url = vault_url
}
</code></pre></div></div>
<p>To monitor, add a list of servers to a host (I attached them to the host that they are served from - as that makes clear where the problem lies):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>object Host "myhost.home.domain.tld" {
// ...
vars.vault_urls = ["https://vault.home.mydomain.tld:8200"]
// ...
}
</code></pre></div></div>
<h2 id="check-screenshots">Check screenshots</h2>
<p>They say a picture paints a thousand words, so here’s some screenshots of the script and configuration (above) in action.</p>
<p>First, everything is fine:</p>
<p><img src="/assets/posts/2023-08-31-icinga2-check-for-hashicorp-vault/Screenshot%202023-08-31%20180423.png" alt="Vault OK" /></p>
<p>Next, the daemon has been restarted and the vault is sealed:</p>
<p><img src="/assets/posts/2023-08-31-icinga2-check-for-hashicorp-vault/Screenshot%202023-08-31%20181348.png" alt="Vault sealed" /></p>
<p>The daemon has been stopped and the vault service cannot be contacted:</p>
<p><img src="/assets/posts/2023-08-31-icinga2-check-for-hashicorp-vault/Screenshot%202023-08-31%20181731.png" alt="Vault dead" /></p>
<p>Finally, the vault is being unsealed but insufficient keys have been provided so far:</p>
<p><img src="/assets/posts/2023-08-31-icinga2-check-for-hashicorp-vault/Screenshot%202023-08-31%20181937.png" alt="Unseal in progress" /></p>LaurenceWhen restarted, Hasicorp Vault starts “sealed” and has to be unsealed to make the contents accessible. I have, on occasion, forgotten to do this (usually after a reboot for kernel update) and so I want to add checks for this situation to my monitoring. My current monitoring solution is Icinga, so I am creating a check for this.Automated Debian install2023-08-22T20:16:47+01:002023-08-22T20:16:47+01:00https://blog.entek.org.uk/notes/2023/08/22/automated-debian-install<p>This is another of <a href="/notes/2022/11/01/adding-a-bastion-host-restructuring-the-network.html">those posts</a> that I started and some time (measured in months) later I split up due to not having completed what I set out to do. My overall goal is to have a fully automated install that results in a system with disk encryption setup up but can be remotely unlocked and managed. The intention is that, in most circumstances, any of my systems could be reinstalled headlessly (that is, without plugging in a keyboard or monitor).</p>
<p>This does have various similarities to <a href="/technology/2020/06/04/kvm-setup.html">previous preseed work</a> and <a href="/notes/2022/10/11/targeted-pxe-booting.html">pxe work</a> I have done.</p>
<h2 id="preparation">Preparation</h2>
<p>I was somewhat surprised (and annoyed by the download time) to find that with the <a href="https://www.debian.org/releases/bookworm/">Bookworm (12) release</a> of Debian, the net-install image has almost doubled in size to 738MB from <a href="https://www.debian.org/releases/bullseye/">Bullseye (11) release</a>’s 389MB. Bullseye’s was a more modest increase from <a href="https://www.debian.org/releases/buster/">Buster (10) release</a>’s 337MB and that was a similar jump from <a href="https://www.debian.org/releases/stretch/">Stretch (9) releases</a>’s 292MB. The <a href="https://www.debian.org/distrib/netinst#verysmall">“mini” iso image</a> has steadily increased in size to 62MB, from 51MB, 48MB, 39MB for Bookworm, Bullseye, Buster and Stretch releases respectively.</p>
<p>In VirtualBox (which I was planning to use to accelerate working on this, rather than swapping keyboard and monitor between physical machines constantly), I was unable to get the Bookworm installer to load, with either the net-install or mini CD images - I got the boot loader menu, a couple of kernel messages then a grey screen with no text at all on screen. I found a <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1036310">bug on Debian</a> which has <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1036310#60">a workaround</a>: “Disabling paravirtualization in virtualbox acceleration or switching it to legacy results in a properly booting installer.”</p>
<p>With a working installer, I did a minimal install on the VM that will be my core system for this work. After installing, I set the VM’s paravirtualisation back to <code class="language-plaintext highlighter-rouge">Default</code> and it booted fine. I installed <a href="https://www.ansible.com/">Ansible</a> to do the rest of the configuration. For simplicity’s sake, I set the core to proxy to Debian’s mirrors as this reflects the setup in my <a href="/notes/2022/02/15/setting-up-cisco-catalyst-switch-for-home-lab.html">air gapped lab</a> which has a local mirror.</p>
<h3 id="networking">Networking</h3>
<p>With ansible, I configured a second interface on an internal-only network (the primary interface is NAT through the host) with <code class="language-plaintext highlighter-rouge">/etc/network/interfaces.d/enp0s8</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auto enp0s8
iface enp0s8 inet static
address 192.168.10.250/24
</code></pre></div></div>
<p>I also installed some extra packages:</p>
<ul>
<li>git</li>
<li>man</li>
<li>tmux</li>
<li>vim</li>
</ul>
<p>Traditionally I have always used <a href="https://www.isc.org/dhcp/">ISC’s DHCP server</a>, although I had just discovered that <a href="https://www.isc.org/dhcp_migration/">went end of life at the end of 2022</a>, for DHCP and <a href="https://www.isc.org/bind/">bind</a> for DNS. For ease I decided to install <a href="https://dnsmasq.org/">Dnsmasq</a>, which provides both DHCP and DNS (and TFTP and PXE) services and <a href="/notes/2021/01/05/split-dns-with-dnsmasq.html">I have previously used it to provide client-side split DNS</a>.</p>
<p>Dnsmasq required relatively little configuration changes:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>interface=enp0s8
bind-interfaces # Do not listen on any other interfaces
domain=dev.internal.lah
# Will not respond to DHCP broadcast requests without this
dhcp-authoritative
dhcp-range=192.168.10.100,192.168.10.200,12h
</code></pre></div></div>
<h3 id="mirror-webserver">Mirror (webserver)</h3>
<p>For the mirror and hosting various configuration files (which I will come on to), I installed <a href="https://nginx.org/en/">nginx</a>. I replaced the default site file with this configuration:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>server {
listen 80 default_server;
listen [::]:80 default_server;
root /srv/mirror;
index index.html index.htm;
server_name _;
location / {
autoindex on; # Enable showing index of directories
try_files $uri $uri/ =404;
}
location /debian/ {
proxy_pass http://deb.debian.org/debian/;
}
location /debian-security/ {
proxy_pass http://security.debian.org/debian-security/;
}
}
</code></pre></div></div>
<p>For the physical environment, the only changes were that a USB hard disk was mounted at <code class="language-plaintext highlighter-rouge">/media/usb-disk</code> and bind-mounted to <code class="language-plaintext highlighter-rouge">/srv/mirror</code>. The proxying was removed from nginx by simply deleting the <code class="language-plaintext highlighter-rouge">location /debian/</code> and <code class="language-plaintext highlighter-rouge">location /debian-security/</code> blocks. This resulted in zero changes from the client perspective regardless of environment.</p>
<p>In both environments, the <code class="language-plaintext highlighter-rouge">preseed.cfg</code> file for the automated install was placed in the web root, <code class="language-plaintext highlighter-rouge">/srv/mirror</code>, to be served up by the webserver.</p>
<h3 id="ntp">NTP</h3>
<p>By default, the Bookworm installer tries to sync the clock via NTP. This fails in a network with no external connectivity but takes a long time to timeout (in the sense that it was longer than I had the patience to wait for it to time out - the progress bar loops, so it it doesn’t time out when it gets to the end of that). However, having clocks in sync (at least with each other) is useful so I installed <code class="language-plaintext highlighter-rouge">chrony</code> and configured it to act as an NTP server for the internal network:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>allow 192.168.10.250/24
bindaddress 192.168.10.250
</code></pre></div></div>
<p>I added <code class="language-plaintext highlighter-rouge">ntp</code> aliases to <code class="language-plaintext highlighter-rouge">/etc/hosts</code> for this system’s IP:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>192.168.10.250 ntp.dev.internal.lah ntp
</code></pre></div></div>
<p>Then I restarted dnsmasq, which made <code class="language-plaintext highlighter-rouge">ntp</code> appear in dnsmasq’s DNS.</p>
<h2 id="automating-the-debian-install">Automating the Debian install</h2>
<p>Lately, my approach to automated installs (of <a href="https://www.debian.org/">Debian</a> or <a href="https://www.redhat.com/">Red Hat</a> based systems) has been to do a minimal install then hand over to the configuration management tool to do the full installation and configuration of any additional software - this is the same tool that will then be used to keep those things in sync and prevent configuration drift. Until recently this has been <a href="https://saltproject.io/">SaltStack</a>, so <a href="/technology/2020/06/04/kvm-setup.html#fully-automated-install-fai">my existing preseed</a> installed the <code class="language-plaintext highlighter-rouge">salt-minion</code> package which then allowed Salt to take over managing the system and do everything else to configure/customise it. However, I am now leaning towards Ansible as my tool of choice for this which does not require a client-side agent (one of its attractions).</p>
<p>My original recipe was to install Ansible into the newly installed OS, fetch a “bootstrap” playbook that then configured the newly installed OS - running it from within the installer. I was hesitant about this approach, as Ansible is not needed on the machines it manages but this method meant the system came up already configured and secured from the first boot. I abandoned it after encountering issues getting Ansible to install packages within the target chroot due to the “CDROM” source entries for the USB media I was using (on the physical hosts) - a solvable problem. I’m wan’t sure doctoring the <code class="language-plaintext highlighter-rouge">/etc/apt/sources.list</code> at this stage, as those same entries get commented out by the installer before the install is finished, would be advisable. I also considered moving the package installations out of Ansible into the preseed itself, but then I will end up maintaining multiple lists of packages for those installed during install and those managed after install. It began to feel like I was jumping through hoops I did not need to.</p>
<p>Instead, I opted to revert to the method of doing the base install and have the new system booted ready to be managed but not yet “adopted” (i.e. configured). This does create a window of opportunity when the not-yet-configured system is waiting to be configured that it could be easier to compromise but fully automating the reinstall (as I plan to do - more on this later…) will minimise this risk, as does the default setup only allowing key-less login (default setting for the OpenSSH server). This can be further mitigated by doing the install in a dedicate, secured, VLAN and only connecting the host to a network with other systems (and internet access) post install - in a way, this is what I am doing with my VirtualMachines during development as only one VM has access to the live network, the rest are on a private “internal only” network with no network route to the live network.</p>
<p>For posterity, the complete preseed recipe to install Ansible, download a playbook and then run it in the target is this (I think it’s pretty short and neat, so worthy of preservation):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Install Ansible to be able to run the bootstrap
d-i pkgsel/include string ansible
# No wget inside the target, use the one in the installer environment.
d-i preseed/late_command string \
wget -P /target/tmp http://debian-installer/bootstrap-playbook.yaml && \
in-target ansible-playbook /tmp/bootstrap-playbook.yaml;
</code></pre></div></div>
<h3 id="initial-preseed-settings">Initial preseed settings</h3>
<p>I added <code class="language-plaintext highlighter-rouge">debian-installer</code> as an aliases to <code class="language-plaintext highlighter-rouge">/etc/hosts</code> for this system’s IP, before restarting dnsmasq, which made <code class="language-plaintext highlighter-rouge">debian-installer</code> appear in dnsmasq’s DNS.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>192.168.10.250 debian-installer.dev.internal.lah debian-installer
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">/srv/mirror/d-i/bookworm</code> I created <code class="language-plaintext highlighter-rouge">preseed.cfg</code>, starting with the general settings:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Localisation
d-i debian-installer/locale string en_GB.UTF-8
d-i keyboard-configuration/xkb-keymap select gb
# Time
d-i time/zone select Europe/London
# HW clock set to UTC?
d-i clock-setup/utc boolean true
# Use NTP during install
clock-setup clock-setup/ntp boolean true
clock-setup clock-setup/ntp-server string ntp
# Mirror
d-i mirror/country string manual
d-i mirror/protocol string http
d-i mirror/http/hostname string 192.168.10.250
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string
# Only enable main
d-i apt-setup/non-free boolean false
d-i apt-setup/non-free-firmware boolean false
d-i apt-setup/contrib boolean false
d-i apt-setup/services-select multiselect security, updates
d-i apt-setup/security_host string 192.168.10.250
# Users
# Skip creation of a normal user account
d-i passwd/make-user boolean false
# Obviously this has an actual password hash here, in the real config
d-i passwd/root-password-crypted password some_crypted_password
# Base system install
d-i base-installer/kernel/image select linux-image-amd64
# Package selection
# No 'task' selections
tasksel tasksel/first multiselect
# Don't partake in the popularity contest
popularity-contest popularity-contest/participate boolean false
# Bootloader
# This is fairly safe to set, it makes grub install automatically to the MBR
# if no other operating system is detected on the machine.
d-i grub-installer/only_debian boolean true
# Avoid that last message about the install being complete.
d-i finish-install/reboot_in_progress note
</code></pre></div></div>
<p>I did setup the <a href="/notes/2022/10/04/debian-network-install-preseed.html">network console</a> in the preseed in the hope that would allow remote progress monitoring but unfortunately it did not easily, so I removed it again. This does have the consequence that SSH will not start listening until after the OS has been fully installed and the device rebooted, making automation a little easier.</p>
<h3 id="stacking-late-command">Stacking late-command</h3>
<p>As I worked through this, I discovered that it was neater to build the <code class="language-plaintext highlighter-rouge">preseed/late_command</code> script at various points - for example, so that <a href="#encryption">the encryption script</a> could add the commands to inject the key it generates - rather than disconnect related bits from a single script. To do this, I added a simple check to see if <code class="language-plaintext highlighter-rouge">/tmp/late-command-script</code> exists and, if it does, run it as the late command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d-i preseed/late_command string \
if [ -f /tmp/late-command-script ]; \
then \
/bin/sh /tmp/late-command-script; \
fi;
</code></pre></div></div>
<p>The script file is started by a script that is fetched from the same location as the preseed file and run through <code class="language-plaintext highlighter-rouge">preseed/include_command</code> (one might be tempted to try using <a href="https://www.debian.org/releases/stable/amd64/apbs05.en.html#preseed-hooks"><code class="language-plaintext highlighter-rouge">preseed/early_command</code>, which is “run as early as possible, just after preseeding is read”</a> but that command runs <em>after</em> the include commands, so after other content might have been added by <code class="language-plaintext highlighter-rouge">preseed/include_command</code>):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d-i preseed/include_command string \
for file in preseed-script-headers; \
do \
wget -P /tmp $( dirname $( debconf-get preseed/url ) )/$file && \
chmod 500 /tmp/$file && \
/tmp/$file; \
done;
</code></pre></div></div>
<p>Note that this is <code class="language-plaintext highlighter-rouge">preseed/include_command</code> command is designed for more, modular, <code class="language-plaintext highlighter-rouge">include_command</code> scripts to be included.</p>
<p>Although I don’t need it immediately, I also added support for the <code class="language-plaintext highlighter-rouge">preseed/early_command</code> to be built up in the same way:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d-i preseed/early_command string \
if [ -f /tmp/early-command-script ]; \
then \
/bin/sh /tmp/early-command-script; \
fi;
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">preseed-script-headers</code> script simply starts these script files:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="nb">cat</span> - <span class="o">></span>/tmp/late-command-script <span class="o"><<</span><span class="no">EOF</span><span class="sh">
#!/bin/sh
# Late command script
# Header generated by script-headers preseed/include_command
</span><span class="no">
EOF
</span><span class="nb">cat</span> - <span class="o">></span>/tmp/early-command-script <span class="o"><<</span><span class="no">EOF</span><span class="sh">
#!/bin/sh
# Early command script
# Header generated by script-headers preseed/include_command
</span><span class="no">
EOF
</span></code></pre></div></div>
<h3 id="partitioning">Partitioning</h3>
<p>It has long been my custom to split the filesystem amongst several partitions on my Linux installs, including a separate <code class="language-plaintext highlighter-rouge">/usr</code> which has caused some issues upgrading systems with <a href="https://wiki.debian.org/UsrMerge">the unification of <code class="language-plaintext highlighter-rouge">/</code> and <code class="language-plaintext highlighter-rouge">/usr</code></a>:</p>
<blockquote>
<p>In February 2021, the Technical Committee has resolved that Debian ‘bookworm’ should support only the merged-usr root filesystem layout, dropping support for the non-merged-usr layout. (<del><a href="https://bugs.debian.org/978636">978636</a></del>)</p>
</blockquote>
<p>When it came to the sizes of partitions, the sizes I selected always “felt” right (and I have almost never had to add more space to one, the exception being my mail server when the mail store grew beyond the originally allocated space). I have listed a number of layouts in previous blog posts, e.g. <a href="/notes/2022/05/06/restoring-the-router-from-dr-backup.html#preparing-the-hard-disk">my home lab router</a>, <a href="/notes/2022/11/03/upgrade-tpm-to-2.0-on-dell-xps-13-9370-and-installing-windows-11-and-debian-linux.html#installing-debian-linux">my laptop</a>, <a href="/notes/2022/07/22/installing-gentoo-on-lenovo-m72e-tiny-pc.html#setup-lvm">test Gentoo install</a> and <a href="/technology/2020/06/02/hp-microservers.html#installing-the-os">my HP Microservers</a>.</p>
<p>Since it is possible to do an <a href="https://ahelpme.com/linux/online-resize-of-a-root-ext4-file-system-increase-the-space">online resize of ext4</a>, my plan is to create a sensible partitioning scheme for a generic Debian base install which can be expanded post install if needed. I did consider doing a per-host preseed configuration with a specific partition scheme based on expected usage but keeping all the per-host customisation in Ansible seemed like it might be cleaner (even if such per-host preseed files might be generated on the fly by Ansible during a (re)install playbook run). This is one of those decisions I think I may revisit in the future as I am not convinced either approach is clearly better.</p>
<p>In order to try and come up with an evidence-based scheme for the generic, base, setup I looked at my existing systems to guage what I typically use currently. From that information, I plan to infer appropriate sizes for the partitions. I looked at a total of 9 systems, 2 of which had desktop environments installed and the rest were “servers” with no gui installed. I used a combination of <code class="language-plaintext highlighter-rouge">df</code> and <code class="language-plaintext highlighter-rouge">du</code> to work out the sizes of various directories, as these systems are not all partitioned the same (in the case of my VMs, there were all single-root filesystems). From these systems, I determined the following usage (“max” indicates the most space consumed across these systems, “min” the least) after adjusting to consider these as though they were separate partitions:</p>
<ul>
<li>swap - max: 4.6G, min: 0B (Only one system, my HP Microserver acting as physical host for my VMs, using using more than 623M and this system will be retired once I have a <a href="https://www.proxmox.com/proxmox-virtual-environment/overview">Proxmox</a> cluster up and running in my live network)</li>
<li><code class="language-plaintext highlighter-rouge">/</code> - (desktop) max: 6.3G, min: 2.7G / (server) max: 2.8G, min: 2.1G</li>
<li><code class="language-plaintext highlighter-rouge">/boot</code> - max: 125M, min: 56M</li>
<li><code class="language-plaintext highlighter-rouge">/boot/efi</code> - max: 38M, min: 6M (only one system, my laptop, out of 5 EFI systems has >6M consumed - suspect due to Windows dual boot)</li>
<li><code class="language-plaintext highlighter-rouge">/boot/firmware</code> - 128M (only my 1 <a href="https://www.raspberrypi.com/">Raspberry Pi</a> has this directory)</li>
<li><code class="language-plaintext highlighter-rouge">/home</code> - (desktop) max: 41G, min: 1.2G / (server) max: 15G, min: 232K (only one “server” system has >825M consumed as it has a copy of my 14G sync-n-share folder for backing up purposes)</li>
<li><code class="language-plaintext highlighter-rouge">/opt</code> - max: 11M, min: 4K (only 2 systems have >1M consumed)</li>
<li><code class="language-plaintext highlighter-rouge">/srv</code> - max: 41G, min: 4K (I ignored my laptop, which has a number of mirrors locally consuming 432G. Only my mail-server has >581M consumed)</li>
<li><code class="language-plaintext highlighter-rouge">/tmp</code> - max: 716K, min: 48K</li>
<li><code class="language-plaintext highlighter-rouge">/var</code> - max: 3.2G, min: 384M</li>
<li><code class="language-plaintext highlighter-rouge">/var/lib</code> - max: 3.3G, min:170M</li>
<li><code class="language-plaintext highlighter-rouge">/var/lib/docker</code> - max: 4G, min: 1.4G (only 3 systems have this directory)</li>
<li><code class="language-plaintext highlighter-rouge">/var/lib/libvirt</code> - 56G (only my HP Microserver acting as physical host for my VMs has this directory)</li>
<li><code class="language-plaintext highlighter-rouge">/var/log</code> - max: 3.1G, min: 723M</li>
</ul>
<p>Based on these values, I settled on this as a starting partitioning scheme for my installs:</p>
<ul>
<li>swap - 4G</li>
<li><code class="language-plaintext highlighter-rouge">/</code> - 4G</li>
<li><code class="language-plaintext highlighter-rouge">/boot</code> - 500M</li>
<li><code class="language-plaintext highlighter-rouge">/boot/efi</code> - 500M</li>
<li><code class="language-plaintext highlighter-rouge">/home</code> - 1G</li>
<li><code class="language-plaintext highlighter-rouge">/opt</code> - 1G</li>
<li><code class="language-plaintext highlighter-rouge">/srv</code> - 1G</li>
<li><code class="language-plaintext highlighter-rouge">/tmp</code> - tmpfs (no longer persistent partition)</li>
<li><code class="language-plaintext highlighter-rouge">/var</code> - 4G</li>
<li><code class="language-plaintext highlighter-rouge">/var/log</code> - 4G</li>
</ul>
<p>Post install, Ansible will make the following changes:</p>
<ul>
<li>On desktop systems, <code class="language-plaintext highlighter-rouge">/</code> will be increased to 8G and <code class="language-plaintext highlighter-rouge">/home</code> to 20G</li>
<li>On systems with docker, <code class="language-plaintext highlighter-rouge">/var/lib/docker</code> will be created with 10G</li>
<li><code class="language-plaintext highlighter-rouge">/srv</code> will be increased on systems that need it (on the server that syncs my sync-n-share locally for backing up, this folder will be moved into <code class="language-plaintext highlighter-rouge">/srv</code>)</li>
</ul>
<p>Documentation for creating preseed partitioning recipes is a bit disjointed. The <a href="https://www.debian.org/releases/bookworm/amd64/apbs04.en.html#preseed-partman">main documentation</a> suggests one reads the <a href="https://salsa.debian.org/installer-team/debian-installer/-/blob/master/doc/devel/partman-auto-recipe.txt">debian-installer source repository</a> which in turn makes reference to <a href="https://salsa.debian.org/installer-team/partman-auto/-/blob/master/recipes-amd64-efi/atomic">partman-auto/recipe-amd64-efi/atomic</a>(“The EFI example above was taken from partman-auto/recipe-amd64-efi/atomic.”) but without explaining how to find this file (<a href="https://www.google.co.uk">Google</a> was how I did). In the end, the Debian installer recipe I created for this is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d-i partman-auto/method string crypto
d-i partman-partitioning/confirm_write_new_label boolean true
# Skip doing a full erase on underlying crypto volumes
d-i partman-auto-crypto/erase_disks boolean false
d-i partman-auto-lvm/guided_size string max
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto/expert_recipe string \
lah_local_scheme :: \
500 500 500 free \
$iflabel{ gpt } \
$reusmethod{ } \
method{ efi } \
format{ } . \
500 500 500 ext4 \
$defaultignore{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /boot } . \
4000 4000 4000 linux-swap \
$lvmok{ } \
$reusemethod{ } \
method{ swap } \
lv_name{ swap } \
format{ } . \
4000 4000 4000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ / } . \
1000 1000 1000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /home } . \
1000 1000 1000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /opt } . \
1000 1000 1000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /srv } . \
4000 4000 4000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /var } . \
4000 4000 4000 ext4 \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /var/log } .
</code></pre></div></div>
<h4 id="encryption">Encryption</h4>
<p>As you can see, I selected the <code class="language-plaintext highlighter-rouge">crypto</code> recipe for this. In order to setup the encryption, the preseed file needs to supply a passphrase for the encryption. The documentation only refers to either passphrase or random key, the latter regenerated on each boot so the previous filesystem’s contents are irretrievable - intended for swap partitions - although there is some reference in the source code to a key file, this seems intended for <a href="https://gnupg.org/">GnuPG</a> keys not generic key files and options for configuring it are absent from the <a href="https://preseed.debian.net/debian-preseed/bookworm/amd64-main-full.txt">reference (full) preseed file</a>.</p>
<p>In the end, I scripted generating a random key locally, early in the install to feed it into the preseed, before injecting it into the installed system (and its initramfs) towards the end of the install so the temporary, random, key will automatically unlock the filesystem on initial boot at which point a post-install configuration tool (although I am using Ansible and migrating away from SaltStack, which tool is used is irrelevant - <a href="https://www.puppet.com/">Puppet</a> or <a href="https://www.chef.io/">Chef</a> would also work perfectly well for this process) takes over, configures a known key via a secure channel, such as <a href="https://en.wikipedia.org/wiki/Secure_Shell">SSH</a>, and removes the random key. In my environment, this known key will be host-specific and manged in <a href="https://www.vaultproject.io/">HasiCorp Vault</a>.</p>
<p>The Debian installer part of this recipe is pretty simple, it just involves using <a href="https://www.debian.org/releases/stable/amd64/apbs05.en.html#preseed-chainload"><code class="language-plaintext highlighter-rouge">preseed/include_command</code></a> to:</p>
<ol>
<li>Download and run the script that generates the random key</li>
<li>Generate the debian installer options in a file</li>
<li>Printing the file name with the debian installer options (so it gets included)</li>
<li>Adding the commands to inject it into the installed OS to the script that <a href="#stacking-late-command">gets run by <code class="language-plaintext highlighter-rouge">preseed/late_command</code></a>.</li>
</ol>
<p>Downloading the script just involves adding it to the list in <a href="(#stacking-late-command)">my modular <code class="language-plaintext highlighter-rouge">preseed/include_command</code> script</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d-i preseed/include_command string \
for file in \
preseed-script-headers \
preseed-crypto-key \
; do \
wget -P /tmp $( dirname $( debconf-get preseed/url ) )/$file && \
chmod 500 /tmp/$file && \
/tmp/$file; \
done;
</code></pre></div></div>
<p>The script itself does all of the rest:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="c"># ^^ no bash in installer environment, only BusyBox</span>
<span class="c"># I couldn't get this to work with using debconf-set to set</span>
<span class="c"># the passphrase (so this has to be run via</span>
<span class="c"># `preseed/include_command`)</span>
<span class="c"># Die on error</span>
<span class="nb">set</span> <span class="nt">-e</span>
<span class="c"># Currently (2023-08-10) the Debian installer only supports</span>
<span class="c"># passphrases, GnuPG key files or random key files which are</span>
<span class="c"># regenerated on each reboot. So, to use a random keyfile, we</span>
<span class="c"># initially have to provide the installer with a passphrase.</span>
<span class="c"># Create a passphrase by pulling only characters in the range</span>
<span class="c"># `!` to `~` (ASCII 0x21 to 0x7e) from /dev/random.</span>
<span class="nv">TMP_PASSPHRASE_FILE</span><span class="o">=</span><span class="si">$(</span> <span class="nb">mktemp</span> <span class="si">)</span>
<span class="nb">umask </span>0077
<span class="nb">grep</span> <span class="nt">-o</span> <span class="s1">'[!-~]'</span> /dev/random | <span class="nb">tr</span> <span class="nt">-d</span> <span class="s1">'\n'</span> | <span class="nb">head</span> <span class="nt">-c64</span> <span class="o">></span> <span class="nv">$TMP_PASSPHRASE_FILE</span>
<span class="c"># Create an include file for debian-installer with the passphrase as answers to the questions</span>
<span class="nv">DEB_INSTALLER_CRYPT_INC_FILE</span><span class="o">=</span><span class="si">$(</span> <span class="nb">mktemp</span> <span class="si">)</span>
<span class="nb">echo</span> <span class="nt">-n</span> <span class="s2">"d-i partman-crypto/passphrase string "</span> | <span class="se">\</span>
<span class="nb">cat</span> - <span class="nv">$TMP_PASSPHRASE_FILE</span> <span class="o">></span> <span class="nv">$DEB_INSTALLER_CRYPT_INC_FILE</span>
<span class="c"># Need a newline between the entries</span>
<span class="nb">echo</span> <span class="o">>></span><span class="nv">$DEB_INSTALLER_CRYPT_INC_FILE</span>
<span class="nb">echo</span> <span class="nt">-n</span> <span class="s2">"d-i partman-crypto/passphrase-again string "</span> | <span class="se">\</span>
<span class="nb">cat</span> - <span class="nv">$TMP_PASSPHRASE_FILE</span> <span class="o">>></span><span class="nv">$DEB_INSTALLER_CRYPT_INC_FILE</span>
<span class="c"># Newline probably not needed at end of file, but it is neat to add it.</span>
<span class="nb">echo</span> <span class="o">>></span><span class="nv">$DEB_INSTALLER_CRYPT_INC_FILE</span>
<span class="c"># Echo the file to be included, so debian-installer will do</span>
<span class="c"># that - assuming this command is being run via</span>
<span class="c"># `preseed/include_command`. Without file:// will try and fetch</span>
<span class="c"># from the webserver this preseed was served from.</span>
<span class="nb">echo</span> <span class="s2">"file://</span><span class="nv">$DEB_INSTALLER_CRYPT_INC_FILE</span><span class="s2">"</span>
<span class="c"># Add extra commands to the file that should be run using</span>
<span class="c"># `preseed/late_command` to ensure passphrase is included in</span>
<span class="c"># new install (or the encryption will be un-unlockable as the</span>
<span class="c"># random passphrase will be lost when the installer reboots).</span>
<span class="nv">IN_TARGET_KEY_FILE</span><span class="o">=</span>/etc/keys/luks-lvm.key
<span class="nb">cat</span> - <span class="o">>></span>/tmp/late-command-script <span class="o"><<</span><span class="no">LATE_EOF</span><span class="sh">
## BEGIN ADDED BY preseed-crypto-key preseed/include_command
umask 0077
mkdir -p /target</span><span class="si">$(</span> <span class="nb">dirname</span> <span class="k">${</span><span class="nv">IN_TARGET_KEY_FILE</span><span class="k">}</span> <span class="si">)</span><span class="sh">
cp </span><span class="nv">$TMP_PASSPHRASE_FILE</span><span class="sh"> /target</span><span class="nv">$IN_TARGET_KEY_FILE</span><span class="sh">
# Use /root as /tmp might be noexec
cat - >/target/root/configure-crypt-unlock <<EOF
#!/usr/bin/bash
# Standard bash safety features
set -eufo pipefail
if grep -q UMASK /etc/initramfs-tools/initramfs.conf
then
sed -i 's-^#</span><span class="se">\?</span><span class="sh">UMASK.*</span><span class="se">\\\$</span><span class="sh">-UMASK=0077-' /etc/initramfs-tools/initramfs.conf
else
echo -e "# Secure initramfs while it contains unlock keys for root filesystem</span><span class="se">\n</span><span class="sh">UMASK=0077" >>/etc/initramfs-tools/initramfs.conf
fi
# Include keyfile in initramfs
sed -i 's-^#</span><span class="se">\?</span><span class="sh">KEYFILE_PATTERN=.*</span><span class="se">\\\$</span><span class="sh">-KEYFILE_PATTERN=</span><span class="si">$(</span> <span class="nb">dirname</span> <span class="k">${</span><span class="nv">IN_TARGET_KEY_FILE</span><span class="k">}</span> <span class="si">)</span><span class="sh">/*.key-' /etc/cryptsetup-initramfs/conf-hook
# Configure crypt to use keyfile to unlock encypted partition(s)
sed -i 's#</span><span class="se">\(</span><span class="sh">UUID=[^ ]</span><span class="se">\+\)</span><span class="sh"> none#</span><span class="se">\1</span><span class="sh"> </span><span class="k">${</span><span class="nv">IN_TARGET_KEY_FILE</span><span class="k">}</span><span class="sh">#' /etc/crypttab
# Updater initramfs with key file
update-initramfs -u
EOF
chmod 500 /target/root/configure-crypt-unlock
in-target /root/configure-crypt-unlock
rm /target/root/configure-crypt-unlock
## END ADDED BY preseed-crypto-key preseed/include_command
</span><span class="no">LATE_EOF
</span></code></pre></div></div>
<h3 id="installing-and-configuring-ssh-server">Installing and configuring SSH server</h3>
<p>Although Ansible doesn’t use an agent, it does use SSH to remotely login to manage targeted systems. So, in place of a agent, one does have to install and configure the SSH server service. I did this <a href="#encryption">like with the encryption</a>, using <code class="language-plaintext highlighter-rouge">preseed/include_command</code> to add to the <code class="language-plaintext highlighter-rouge">preseed/late_command</code> script to do the work. Rather than ask the Debian installer to install the <code class="language-plaintext highlighter-rouge">openssh-server</code> package, by emitting a file to be included and printing its name, I did this with <code class="language-plaintext highlighter-rouge">apt-get</code> in the target OS via the late command script. There can only be one entry in the debian installer for packages to install, so this way avoid potentially conflicting with another.</p>
<p>To include this script, I just added it to the list:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d-i preseed/include_command string \
for file in \
preseed-script-headers \
preseed-crypto-key \
preseed-ssh-setup \
; do \
wget -P /tmp $( dirname $( debconf-get preseed/url ) )/$file && \
chmod 500 /tmp/$file && \
/tmp/$file; \
done;
</code></pre></div></div>
<p>And the script itself is very simple:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="c"># Install and allow remote-root SSH for post-install config</span>
<span class="nb">cat</span> - <span class="o">>></span>/tmp/late-command-script <span class="o"><<</span><span class="no">EOF</span><span class="sh">
## BEGIN ADDED BY preseed-ssh-setup preseed/include_command
in-target apt-get install -y openssh-server
# Setup initial root ssh keys
mkdir -m 700 /target/root/.ssh
umask 0077
cat - >/target/root/.ssh/authorized_keys <<KEYS_EOF
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGELav9hG7S1Kohs5QyEsrBIXLbT18tdTZCFg5rUITwxXg1JDKlzuR7v+8zLmbzWCBs0IR8QA9EBw0099h8QW3A= laurence@core
KEYS_EOF
## END ADDED BY preseed-ssh-setup preseed/include_command
</span><span class="no">EOF
</span></code></pre></div></div>
<h2 id="network-pxe-booting-the-install">Network (PXE) booting the install</h2>
<p>I started by booting from USB (physical machine) or CD (VirtualBox environment) and manually editing the boot options each time. This quickly became tiresome (and in any case, I want to be PXE booting in the live environment). I have set this up numerous times with the ISC DHCP server but this was my first time using dnsmasq (at all). The key parts were enabling TFTP (to allow it to load the boot image) and carefully crafting the options so BIOS devices are given the BIOS version, UEFI devices the UEFI version and iPXE is told where to get its configuration from.</p>
<p>For TFTP, I made a folder <code class="language-plaintext highlighter-rouge">/srv/tftp</code> which is shared. I installed the <code class="language-plaintext highlighter-rouge">ipxe</code> package (from the Debian repositories) and symlinked <code class="language-plaintext highlighter-rouge">ipxe.efi</code>, <code class="language-plaintext highlighter-rouge">undionly.kpxe</code> and <code class="language-plaintext highlighter-rouge">snponly.efi</code> from <code class="language-plaintext highlighter-rouge">/usr/lib/ipxe/</code> to <code class="language-plaintext highlighter-rouge">/tmp/tftp/ipxe/</code>.</p>
<p>I originally tried to use dnsmasq’s built in PXE menu, however it was not working with my UEFI systems so I reverted to directly loading iPXE (without any menu through dnsmasq) and using that to provide the menu. This does have the advantage of making the menu agnostic to the DHCP server solution. I also tried specifying the types by client architecture number (e.g. <code class="language-plaintext highlighter-rouge">7</code>, <code class="language-plaintext highlighter-rouge">9</code>) but did not get it working.</p>
<p>For posterity, this is the non-working configuration for dnsmasq (I get the pxe prompt, just no menu):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Menu options based on https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2020q1/013669.html
# Do not offer menu to ipxe requests
tag-if=set:skipmenu,tag:ipxe
pxe-prompt=tag:!skipmenu,"Select boot option or wait to boot from local disk...",15
# Common menu
pxe-service=tag:!skipmenu,x86PC,"Boot from local disk"
pxe-service=tag:!skipmenu,X86-64_EFI,"Boot from local disk"
# BIOS menu
pxe-service=tag:!skipmenu,x86PC,"ipxe - kpxe",ipxe/undionly.kpxe
# UEFI menu
pxe-service=tag:!skipmenu,X86-64_EFI,"ipxe - UEFI",ipxe/ipxe.efi
</code></pre></div></div>
<p>The working dnsmasq configuration is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enable-tftp
tftp-root=/srv/tftp
# Set the tag "ipxe" if the request has the "iPXE" user class
dhcp-userclass=set:ipxe,iPXE
# Identify VirtualBox VMs (due to buggy Intel 1000 EFI driver)
dhcp-mac=set:virtualbox,08:00:27:*:*:*
# From https://wiki.archilinux.org/title/dnsmasq#PXE_server
dhcp-match=set:efi-x86_64,option:client-arch,7
dhcp-match=set:efi-x86_64,option:client-arch,9
dhcp-match=set:efi-x86,option:client-arch,6
dhcp-match=set:bios,option:client-arch,0
dhcp-boot=tag:efi-x86_64,tag:!virtualbox,ipxe/ipxe.efi
# VirtualBox Intel 1000 EFI driver is buggy, so use
# simple networking version for these hosts. See:
# https://github.com/ipxe/ipxe/issues/977
dhcp-boot=tag:efi-x86_64,tag:virtualbox,ipxe/snponly.efi
dhcp-boot=tag:efi-x86,tag:!virtualbox,ipxe/ipxe.efi
dhcp-boot=tag:efi-x86,tag:virtualbox,ipxe/snponly.efi
dhcp-boot=tag:bios,ipxe/undionly.kpxe
# Tell iPXE where to find its configuration
# Order is important, this must be last to override
# the above options.
dhcp-boot=tag:ipxe,"http://192.168.10.250/ipxe.cfg"
</code></pre></div></div>
<p>The iPXE configuration file I placed in <code class="language-plaintext highlighter-rouge">/srv/mirror</code> as that was <a href="#mirror-webserver">already configured as the web root</a> for this test box. For now (while I am testing in VirtualBox) I had this present a menu:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!ipxe
# Detect CPU architecture
cpuid --ext 29 && set arch x86_64 || set arch x86
# For Debian - who refers to x86_64 as amd64
cpuid --ext 29 && set deb_arch amd64 || set deb_arch i386
menu Please choose an operating system to boot
item debian_12 Debian 12 (Bookworm) installer
item debian_12_preseed Debian 12 (Bookworm) installer (preseeded)
item exit Exit and try next boot device
choose os && goto ${os}
:debian_12
set release bookworm
goto install_debian
:debian_12_preseed
set release bookworm
# See https://forum.ipxe.org/showthread.php?tid=7960 for how to add spaces to variables...
set sp:hex 20 && set sp ${sp:string}
# Full hostname required as the Debian installer might not detect the correct domain name.
set extra_kernel_options netcfg/get_hostname=unconfigured-hostname${sp}netcfg/get_domain=unconfigured-domain${sp}auto-install/enable=true${sp}preseed/url=debian-installer.dev.internal.lah
goto install_debian
:install_debian
initrd http://192.168.10.250/debian/dists/${release}/main/installer-${deb_arch}/current/images/netboot/debian-installer/${deb_arch}/initrd.gz
chain http://192.168.10.250/debian/dists/${release}/main/installer-${deb_arch}/current/images/netboot/debian-installer/${deb_arch}/linux initrd=initrd.gz ${extra_kernel_options}
:exit
exit 1
</code></pre></div></div>
<h2 id="to-be-continued">To be continued…</h2>
<p>This will be followed up by another post that picks up from the point of having the Debian OS installed but not configured.</p>LaurenceThis is another of those posts that I started and some time (measured in months) later I split up due to not having completed what I set out to do. My overall goal is to have a fully automated install that results in a system with disk encryption setup up but can be remotely unlocked and managed. The intention is that, in most circumstances, any of my systems could be reinstalled headlessly (that is, without plugging in a keyboard or monitor).Security anti-patterns and browsing down2023-08-22T11:55:27+01:002023-08-22T11:55:27+01:00https://blog.entek.org.uk/notes/2023/08/22/security-anti-patterns-and-browsing-down<p>While browsing for some information on browsing down, I found some useful resources from the National Cyber Security Centre; <a href="https://www.ncsc.gov.uk/whitepaper/security-architecture-anti-patterns">a whitepaper on Security Architecture Anti-Patterns</a>, <a href="https://www.ncsc.gov.uk/collection/secure-system-administration">guidance on secure system administration</a> and <a href="https://www.ncsc.gov.uk/blog-post/protect-your-management-interfaces">a blog post on protecting management interfaces</a> (which focuses on browsing down).</p>
<p>Reassuringly, my current plans for improving my network’s security align with a lot of this advice - their “small company” example is almost exactly what I am aiming to achieve. While this is still “browsing up”, not having separate uber-secure privileged-access-workstations (“PAW”), my plans incorporate the security controls they describe for this scenario. I am not adding a PAW (at least for now) for the same cost reasons, as well as my agenda to reduce power consumption which adding an PAW runs counter to.</p>LaurenceWhile browsing for some information on browsing down, I found some useful resources from the National Cyber Security Centre; a whitepaper on Security Architecture Anti-Patterns, guidance on secure system administration and a blog post on protecting management interfaces (which focuses on browsing down).New blog posts landing - update2023-05-31T18:11:26+01:002023-05-31T18:11:26+01:00https://blog.entek.org.uk/blog/2023/05/31/new-blog-posts-landing-update<p>I just published <a href="/blog/2023/04/07/new-blog-posts-landing.html">33 new blog posts</a>. Nothing more to say really, other then normal service will hopefully now resume. I have decided that I will adopt a new policy of dating posts when they are published, rather than when I start writing them…</p>LaurenceI just published 33 new blog posts. Nothing more to say really, other then normal service will hopefully now resume. I have decided that I will adopt a new policy of dating posts when they are published, rather than when I start writing them…Efficiently copying git changes across isolating network boundaries2023-05-31T14:41:57+01:002023-05-31T14:41:57+01:00https://blog.entek.org.uk/notes/2023/05/31/efficiently-copying-git-changes-across-isolating-network-boundaries<p>In a previous post I described <a href="/notes/2022/06/17/pushing-git-changes-from-lab-back-upstream.html">copy changes from my air-gapped home lab back upstream</a> by copying the entire tar of the repository back and then pushing the changes. While this works and is fine for small repositories, it is highly inefficient for small changes to large repositories. I wrote the last post in full knowledge there would be a more efficient way, which this post documents. This is useful for other situations where there is some level of network isolation (but not necessarily a full air-gap), such as packing changes to move them via a jump host to another network, for pushing to a remote source.</p>
<h2 id="generating-the-pack">Generating the pack</h2>
<p>To create the <a href="https://git-scm.com/book/en/v2/Git-Internals-Packfiles">pack file</a>, we first need to know the last common commit between the two repositories - if there is none, this is a non-starter. Once we have that commit reference, we can use <a href="https://git-scm.com/docs/git-rev-list"><code class="language-plaintext highlighter-rouge">git rev-list</code></a> to get the list of revisions between that commit and the current <code class="language-plaintext highlighter-rouge">HEAD</code> (this process will also work with another target, but for my purposes I want to use the HEAD of the current branch) and feed that to <a href="https://git-scm.com/docs/git-pack-objects"><code class="language-plaintext highlighter-rouge">git pack-objects</code></a> to create the pack of all objects in those revisions (in this example <code class="language-plaintext highlighter-rouge">1a2b3c4</code> is a placeholder for the common starting commit, <code class="language-plaintext highlighter-rouge">export</code> is simply a base-name for the pack files):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git rev-list <span class="nt">--objects</span> 1a2b3c4..HEAD | git pack-objects <span class="nb">export</span>
</code></pre></div></div>
<p>This will create two new files, <code class="language-plaintext highlighter-rouge">export-sha1.pack</code> and <code class="language-plaintext highlighter-rouge">export-sha1.idx</code> (where <code class="language-plaintext highlighter-rouge">sha1</code> is a hash based on the pack content).</p>
<p>On my repository, this generated two files total size of 128K, representing several months of changes that I have not yet copied over, on a repository with a total 768K of code and a total size of 3.5M (when including version history). While this is a small scale example (to test the principle), the advantages are clear when the changes deltas are small and the repository large - I work on one repository that is a number of gigabytes in size.</p>
<p>Now (and this is important), make a note of the current <code class="language-plaintext highlighter-rouge">HEAD</code> (or whatever your target was) commit it - for this example, I shall imagine it was <code class="language-plaintext highlighter-rouge">5d6e7f8</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git rev-parse <span class="nt">--short</span> HEAD
</code></pre></div></div>
<h2 id="applying-the-pack">Applying the pack</h2>
<p>To apply the pack to the existing codebase, first I unpacked the objects exported from the isolated network (the glob, <code class="language-plaintext highlighter-rouge">*</code>, is just for the example - I explicitly listed the actual filename from my export, just in case I left an old one behind9):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git unpack-objects <export-<span class="k">*</span>.pack
</code></pre></div></div>
<p>Now I created a new branch at the <code class="language-plaintext highlighter-rouge">HEAD</code> of the exported repository (which is why it was important to note its commit id):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout <span class="nt">-b</span> import 5d6e7f8
</code></pre></div></div>
<p>Alternatively <code class="language-plaintext highlighter-rouge">git fsck --lost-found</code> might show you the dangling commit id, however if there is other garbage in your current git clone it might be lost in the noise. <code class="language-plaintext highlighter-rouge">git gc</code> (to run the garbage collector) might help tidy things up and just leave the imported <code class="language-plaintext highlighter-rouge">HEAD</code> as the only dangling commit id or not, it really depends on how you have used the repository and when and you might permanently lose some other “lost” commits you really wanted.</p>
<p>The next thing to do is to merge with the current version of your main branch. In my case, this resulted in a number of conflicts to resolve as I had been doing parallel development on my desktop systems configuration management in the live network:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git pull origin main
</code></pre></div></div>
<p>Once this is all merged, the import branch can be pushed and (if applicable) a pull-request started to merge into the main branch:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git push <span class="nt">-u</span> origin import
</code></pre></div></div>
<p>You might need to repeat this process in reverse to copy the merged result back - in the case of my air-gapped network, it will get a read-only mirror of the merge repository to pull down from via my <a href="/notes/2022/11/16/adding-mirrors-to-laptop-a-k-a-improving-mirror-sync-2-0.html">existing mirror synchronisation</a> process.</p>LaurenceIn a previous post I described copy changes from my air-gapped home lab back upstream by copying the entire tar of the repository back and then pushing the changes. While this works and is fine for small repositories, it is highly inefficient for small changes to large repositories. I wrote the last post in full knowledge there would be a more efficient way, which this post documents. This is useful for other situations where there is some level of network isolation (but not necessarily a full air-gap), such as packing changes to move them via a jump host to another network, for pushing to a remote source.Quick-and-dirty Linux password generation2023-05-22T17:22:20+01:002023-05-22T17:22:20+01:00https://blog.entek.org.uk/notes/2023/05/22/quick-and-dirty-linux-password-generation<p>A quick and dirty way to generate a password on a Linux box: <code class="language-plaintext highlighter-rouge">tr -dc '[:print:]' < /dev/urandom | head -c32</code></p>
<p>32 is the number of characters for the resultant password - beware this will include any possible printable character, including space.</p>LaurenceA quick and dirty way to generate a password on a Linux box: tr -dc '[:print:]' < /dev/urandom | head -c32Setting up DrayTek Vigor 130 for Sky FTTC (VDSL) broadband2023-05-05T13:31:52+01:002023-05-05T13:31:52+01:00https://blog.entek.org.uk/notes/2023/05/05/setting-up-draytek-vigor-130-for-sky-fttc-vdsl-broadband<p>This post begins with a rant about <a href="https://www.virginmedia.com/">Virgin Media</a> ignoring their own contract and cutting us off 17 days before they told us our services would end (just 13 days after we gave the contractual “30 days notice” to leave). It follows with setting up a <a href="https://www.draytek.co.uk/products/business/vigor-130">DrayTek Vigor 130</a> VDSL2/ADSL modem with a Linux router for <a href="https://www.sky.com/">Sky</a>’s <a href="https://en.wikipedia.org/wiki/Fiber_to_the_x#FTTN_and_FTTC">fibre-to-the-cabinet (FTTC)</a> broadband service.</p>
<h2 id="background-and-a-rant-about-virgin-media">Background and a rant about Virgin Media</h2>
<p>After reviving notice of an over 37% price increase to our in-contract Virgin services, we cancelled on 17th April. In line with our contractual 30-day notice, we were told our services would end on 17th May (confirmed with Virgin’s support staff no less than 4 times, twice over the phone and twice in writing) so when they disconnected us at 0025hrs 1st May, 17 days early, I was not only annoyed but also left in the lurch as our new broadband service (with Sky) was not scheduled to be installed until 5th May.</p>
<p>It took 3 complaints to get them to offer to compensate me for the £10 is cost me to get sufficient mobile data to tide us over until the Sky service was installed, and the promise of £10 (which has not yet arrived) was all I got out of them - despite cutting us off early in breach of their contract with me and me having in writing from them our services would not end until 17th May. They initially tried to claim that a straight refund, issued before the complaints, for services between the 1st May to 2nd June (which we had already been billed and paid for) was compensation for the inconvenience of being cut off early!</p>
<p>Since Virgin Media seem to ignore the contract between us (consumer) and them (supplier), which they wrote, and cut us off 17 days before a date I had repeatedly re-confirmed with them; I do not currently feel we can ever enter into an agreement with them in the future.</p>
<h2 id="sky-settings">Sky settings</h2>
<p>DrayTek have a <a href="https://www.draytek.co.uk/support/guides/sky-fibre-setup-guide">page on configuring their Routers (but it also talks about the Vigor modems) for Sky</a>. Crucially Sky does not require authentication but does expect <a href="https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol">DHCP</a> with the client identifier (option <code class="language-plaintext highlighter-rouge">61</code>) is set to <code class="language-plaintext highlighter-rouge"><username>@skydsl|<password></code>. However <a href="https://helpforum.sky.com/t5/Broadband/Purchased-new-router-looking-for-Sky-compatible-modem/m-p/3987418">sources online</a> state that the username and password can be anything at all and most people use <code class="language-plaintext highlighter-rouge">Anyone</code> or <code class="language-plaintext highlighter-rouge">anything</code> for both. One key piece of information that is not immediately obvious is that Sky’s broadband, like most UK broadband services, requires tagging with <a href="https://en.wikipedia.org/wiki/VLAN">VLAN</a> <code class="language-plaintext highlighter-rouge">101</code> (it’s only mentioned for the Vigor 2750 router, the others say to disable VLANs - I presume as it’s the default usually for external modems).</p>
<p><del>Because the Sky broadband is VLAN tagged, by passing the VLAN through (instead of terminating it at the modem) it is possible to maintain administrative access to the modem. <a href="https://www.draytek.com/support/knowledge-base/5212">Details for configuring this are on the DrayTek website</a>.</del> I tried to do this but could not get it working - instead I ended up reverting to configuring the modem to terminate the VLAN, however I was still able to access it (even in bridge mode) by adding an IP address to the router’s public interface. The bits I reverted I have left in but <del>crossed out</del>.</p>
<p>It is possible to <a href="https://www.draytek.co.uk/support/guides/kb-local-certificate-management">install a custom certificate</a> for HTTPS, and I should explore adding this to <a href="/notes/2021/01/15/let's-encrypt-ssl-certificates-at-home.html">my Let’s Encrypt setup</a>.</p>
<h2 id="configuring-the-draytek-vigor">Configuring the DrayTek Vigor</h2>
<p>I began with a factor reset of the modem - it has been used with previous broadband supplies at my home and I was not sure what settings it had so I thought this was easiest way to start afresh.</p>
<p>The router’s <a href="https://www.draytek.co.uk/support/guides/login-vigor-router">default IP address</a> is <code class="language-plaintext highlighter-rouge">192.168.2.1</code>.</p>
<p>I plugged it directly into my laptop (without the DSL connected) to do the initial configuration, and it handed my laptop an IP via DHCP. I could then login via the web interface at <a href="http://http://192.168.2.1/">http://http://192.168.2.1/</a>.</p>
<p>As this connection is ethernet, <a href="https://networkmanager.dev/">NetworkManager</a> gave it a lower metric (i.e. higher precedence) than the WiFi connection - however this route (via the Vigor modem) has no connectivity to the outside world yet. To workaround this, I told NetworkManager to never use the ethernet connection as a default route and told it to re-raise the connection:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmcli connection modify <span class="s2">"Wired connection 1"</span> ipv4.never-default <span class="nb">yes </span>ipv6.never-default <span class="nb">yes
</span>nmcli connection up <span class="s2">"Wired connection 1"</span>
</code></pre></div></div>
<p><a href="https://www.draytek.co.uk/support/guides/kb-defaultpassword">The default username and password on a Vigor 130</a> are <code class="language-plaintext highlighter-rouge">admin</code> and <code class="language-plaintext highlighter-rouge">admin</code>. The firmware was a little out of date (<code class="language-plaintext highlighter-rouge">3.8.4.1_BT</code>), so I <a href="https://www.draytek.co.uk/support/downloads/vigor-130">downloaded and installed the latest (<code class="language-plaintext highlighter-rouge">3.8.5.1_BT</code>) firmware</a>.</p>
<p>Even with the latest firmware (dated ), I had to enable defunct key methods and cyphers to login via SSH:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-oKexAlgorithms</span><span class="o">=</span>+diffie-hellman-group14-sha1 <span class="nt">-oHostKeyAlgorithms</span><span class="o">=</span>+ssh-dss <span class="nt">-c</span> 3des-cbc <span class="nt">-l</span> admin 192.168.2.1
</code></pre></div></div>
<p>First thing I did was change the password (obviously not to <code class="language-plaintext highlighter-rouge">password1234</code> - that’s just a placeholder for your password!). The Vigor 130 claims the argument to <code class="language-plaintext highlighter-rouge">passwd</code> is <code class="language-plaintext highlighter-rouge"><ASCII string (max. 23 characters)></code> however it crashed with the first password I tried, so I presume not all ASCII characters are supported (I’m not sure which one caused it to die).:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sys passwd password1234
</code></pre></div></div>
<p>I then disabled all non-encrypted protocols (ftp, http, telnet, tr069) by selecting only https and ssh:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> mngt lanaccess -s HTTPS,SSH
</code></pre></div></div>
<p>And verified the new setting:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> mngt lanaccess -v
mngt lanaccess -v
Current LAN Access Control Setting:
* Enable:Yes
* Service:
- FTP:No
- HTTP:No
- HTTPS:Yes
- TELNET:No
- SSH:Yes
- TR069:No
* Subnet:
</code></pre></div></div>
<p><del>For passing through the VLAN encapsualtion to the router (and allowing managmeent access via untagged traffic), I enabled VLAN tagging and set the tag to <code class="language-plaintext highlighter-rouge">0</code> as <a href="https://www.draytek.com/support/knowledge-base/5212">per the DrayTek documentation</a>. This had to be done via the web UI, as the CLI does not support setting it to <code class="language-plaintext highlighter-rouge">0</code>:</del></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> wan phyvlan wan 1 tag 0
wan phyvlan wan 1 tag 0
% Tag value must be a 1~4095 number
</code></pre></div></div>
<p>It can, however, report it correctly once set to <code class="language-plaintext highlighter-rouge">0</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> wan phyvlan stat
wan phyvlan stat
% Interface Pri Tag Enabled
% ======================================
% WAN1 (ADSL) -- -- --
% WAN1 (VDSL) 0 0 v
% WAN2 -- -- --
</code></pre></div></div>
<p>I then <a href="https://www.draytek.co.uk/support/guides/kb-vigor-130-bridge">turned on bridging (<code class="language-plaintext highlighter-rouge">MPoA (RFC1483/2684)</code>)</a>, which I again had to do through the web interface as the CLI did not work. According to the documentation, this is sufficient to disable DHCP:</p>
<blockquote>
<p>Upon restarting in Bridge Mode, the modem will no longer provide an IP address through DHCP</p>
</blockquote>
<p>However, despite this also being repeated in the <a href="https://www.draytek.co.uk/support/guides/kb-vigor-130-setup#v130bridge">main setup guide</a>, the <a href="https://www.draytek.com/support/knowledge-base/5212">guide for configuring access in bridge mode</a> says to explicitly disable it. So I did:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> srv dhcp off
> sys reboot
</code></pre></div></div>
<p>I also set the MTU to <code class="language-plaintext highlighter-rouge">1500</code>, which was suggested as the correct setting for Sky broadband on some forums:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> wan mtu 1500
</code></pre></div></div>
<p>The Vigor is now configured and ready to be connected to the router, so unplugged it and restored the default behaviour of allowing the ethernet to be a default route on my laptop:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmcli connection modify <span class="s2">"Wired connection 1"</span> ipv4.never-default no ipv6.never-default no
</code></pre></div></div>
<h2 id="configuring-the-router">Configuring the router</h2>
<p>To get the DHCP client (<a href="">dhclient</a>) to send the correct identifier, I added the option to <code class="language-plaintext highlighter-rouge">/etc/dhcp/dhclient.conf</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>interface "enp4s0.101" {
supersede domain-name-servers localhost;
supersede domain-name "home.entek.org.uk";
#supersede domain-search "home.entek.org.uk", "wlan.home.entek.org.uk", "entek.org.uk";
supersede domain-search "home.entek.org.uk", "entek.org.uk";
# We don't want the dhcp client to override our dns settings, so don't request that information.
request subnet-mask, broadcast-address, time-offset, routers,
interface-mtu, rfc3442-classless-static-routes, ntp-servers;
# For Sky broadband
send dhcp-client-identifier "anything@skydsl|anything";
timeout 600;
}
</code></pre></div></div>
<p><del>Changed the existing external interface (<code class="language-plaintext highlighter-rouge">enp4s0</code>) from DHCP to static in <code class="language-plaintext highlighter-rouge">/etc/network/interfaces.d/enp4s0</code>:</del></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auto enp4s0
iface enp4s0 inet static
address 192.168.2.250/24
</code></pre></div></div>
<p><del>and created ``/etc/network/interfaces.d/enp4s0.101`:</del></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auto enp4s0.101
iface enp4s0.101 inet dhcp
</code></pre></div></div>
<p><del>I also updated my firewall script (a bash script that configures <a href="https://www.netfilter.org/projects/iptables/index.html">iptables</a>, which I really do need to replace…), setting the external interface to <code class="language-plaintext highlighter-rouge">enp4s0.101</code> instead of <code class="language-plaintext highlighter-rouge">enp4s0</code>.</del></p>
<p>As the VLAN method was not working, I set <code class="language-plaintext highlighter-rouge">enp4s0</code> bach to dhcp and added an alias of <code class="language-plaintext highlighter-rouge">enp4s0:0</code>, to <a href="https://wiki.debian.org/NetworkConfiguration#Multiple_IP_addresses_on_one_Interface">attach multiple IP addresses to the interface</a> - in <code class="language-plaintext highlighter-rouge">/etc/network/interfaces.d/enp4s0:0</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auto enp4s0:0
iface enp4s0:0 inet static
address 192.168.2.250/24
</code></pre></div></div>
<p>I would have preferred to pass the VLAN through completely and terminate VLAN <code class="language-plaintext highlighter-rouge">101</code> on the router - it feels more secure to have the modem’s management interface entirely outside the broadband VLAN however this works for now…</p>
<h2 id="update-19-may-2023">Update 19 May 2023</h2>
<p>I had to rip out the DrayTek and use Sky’s router, which won’t let me add static (internal) routes to make it aware of the multiple internal networks so I now have double-NAT. This is because we suddenly had the broadband connection become very unstable after 14 days of flawless working but Sky refused to get OpenReach to investigate unless we used their router and were told in no uncertain terms we cannot use out own equipment with Sky’s service.</p>LaurenceThis post begins with a rant about Virgin Media ignoring their own contract and cutting us off 17 days before they told us our services would end (just 13 days after we gave the contractual “30 days notice” to leave). It follows with setting up a DrayTek Vigor 130 VDSL2/ADSL modem with a Linux router for Sky’s fibre-to-the-cabinet (FTTC) broadband service.Deploying custom CA certificates on Linux from Windows share2023-04-20T16:18:29+01:002023-04-20T16:18:29+01:00https://blog.entek.org.uk/notes/2023/04/20/deploying-custom-ca-certificates-on-linux-from-windows-share<p>This post is about deploying custom certificate authority (CA) certificates onto Linux hosts, from an anonymous Windows share, then deploying them to be used by web-browsers (which seem to use their own CA stores these days). There are two scripts, one for each of these tasks, as installing to the system store usually requires super-user (i.e. <code class="language-plaintext highlighter-rouge">root</code>) access but installing to the browser stores is per-user (and should not be done using the super-user account).</p>
<h2 id="installing-ca-to-the-system-store">Installing CA to the system store</h2>
<p>This script does make several assumptions and, at present, needs to be configured using in-line variables <code class="language-plaintext highlighter-rouge">CERT_DIR</code>, <code class="language-plaintext highlighter-rouge">CERT_PREFIX</code> and <code class="language-plaintext highlighter-rouge">DEBUG</code> - these should be made command-line options.</p>
<p>Key assumptions are:</p>
<ul>
<li>Certificates are all in the same directory, directly specified as the download source.</li>
<li>There is no need to remove previously installed certificates that are no longer in the share.</li>
<li>The share allows anonymous access to fetch the certificates.</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># Standard bash safety - fail on error, do not allow unintended</span>
<span class="c"># globbing, do not allow uninitialised variables, errors in pipes</span>
<span class="c"># cause whole pipe to fail.</span>
<span class="nb">set</span> <span class="nt">-efuo</span> pipefail
<span class="c"># TODO: Make these command line options</span>
<span class="c"># UNC path to fetch the certificates from - should be a directory</span>
<span class="c"># containing one or more certifictes to be installed.</span>
<span class="nv">CERT_DIR</span><span class="o">=</span><span class="s2">"//isolinear/software/certs/my super ca"</span>
<span class="c"># This will be used as a directory (Debian-like) or filename prefix</span>
<span class="c"># (Red Hat-like) when installing the certificates.</span>
<span class="nv">CERT_PREFIX</span><span class="o">=</span><span class="s2">"LOCAL-SUPER-CA"</span>
<span class="nv">DEBUG</span><span class="o">=</span><span class="nb">true
</span>debug<span class="o">()</span> <span class="o">{</span>
<span class="o">[[</span> <span class="nv">$DEBUG</span> <span class="o">=</span> <span class="nb">true</span> <span class="o">]]</span> <span class="o">&&</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> <span class="o">></span>&2
<span class="o">}</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$UID</span> <span class="nt">-eq</span> 0 <span class="o">]]</span>
<span class="k">then
</span><span class="nv">USE_SUDO</span><span class="o">=</span><span class="nb">false
</span><span class="k">elif </span><span class="nb">sudo</span> <span class="nt">-n</span> <span class="nt">-l</span> &>/dev/null
<span class="k">then</span>
<span class="c"># N.B. being able to list sudo just means we can run sudo</span>
<span class="c"># passwordless (e.g. already authenticated or have nopasswd</span>
<span class="c"># commands available). It does not mean we can run the specific</span>
<span class="c"># commands required later - the script may still error.</span>
<span class="nv">USE_SUDO</span><span class="o">=</span><span class="nb">true
</span><span class="k">else
</span><span class="nb">echo</span> <span class="s2">"This script needs to be run as root, or be able to sudo,"</span> <span class="se">\</span>
<span class="s2">"to install the certificates."</span> <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">fi
</span>do_su<span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">USE_SUDO</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"true"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nb">sudo</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="k">else</span>
<span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="o">}</span>
<span class="nb">echo</span> <span class="s2">"Fetching certificates from </span><span class="k">${</span><span class="nv">CERT_DIR</span><span class="k">}</span><span class="s2">..."</span>
<span class="c"># Split into //host/share and directory</span>
<span class="nv">regex</span><span class="o">=</span><span class="s1">'^(//[^/]+/[^/]+)(/.*)?$'</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">CERT_DIR</span><span class="k">}</span> <span class="o">=</span>~ <span class="nv">$regex</span> <span class="o">]]</span>
<span class="k">then</span>
<span class="c"># First group</span>
<span class="nv">host_share</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">BASH_REMATCH</span><span class="p">[1]</span><span class="k">}</span><span class="s2">"</span>
<span class="c"># Is there a 2nd group, or is it the empty string (didn't match)?</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nt">-z</span> <span class="s2">"</span><span class="k">${</span><span class="nv">BASH_REMATCH</span><span class="p">[2]</span><span class="k">}</span><span class="s2">"</span> <span class="o">]]</span>
<span class="k">then</span>
<span class="c"># Presume root of share, if no directory</span>
<span class="nv">share_dir</span><span class="o">=</span><span class="s1">'/'</span>
<span class="k">else
</span><span class="nv">share_dir</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">BASH_REMATCH</span><span class="p">[2]</span><span class="k">}</span><span class="s2">"</span>
<span class="k">fi
</span>debug <span class="s2">"Calculated host and share name as: </span><span class="k">${</span><span class="nv">host_share</span><span class="k">}</span><span class="s2">"</span>
debug <span class="s2">"Calculated directory as: </span><span class="k">${</span><span class="nv">share_dir</span><span class="k">}</span><span class="s2">"</span>
<span class="c"># Work out how many slashes are in the directory name - we will</span>
<span class="c"># need to strip these off.</span>
<span class="nv">dir_components</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">share_dir</span><span class="p">//[^\/]</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">dir_component_count</span><span class="o">=</span><span class="k">${#</span><span class="nv">dir_components</span><span class="k">}</span>
debug <span class="s2">"Will strip </span><span class="k">${</span><span class="nv">dir_component_count</span><span class="k">}</span><span class="s2">+1 components."</span>
<span class="c"># Somewhere to save the files. Make sure this gets tidied up</span>
<span class="c"># (deleted) later on.</span>
<span class="nv">download_dir</span><span class="o">=</span><span class="si">$(</span> <span class="nb">mktemp</span> <span class="nt">-d</span> <span class="si">)</span>
<span class="nv">download_tar</span><span class="o">=</span><span class="si">$(</span> <span class="nb">mktemp</span> <span class="si">)</span>
<span class="c"># Get smbclient to tar the directory</span>
<span class="c"># This was originally done without a temporary tar file, directly piping</span>
<span class="c"># smbclient to tar but tar doesn't handle SIGPIPE correctly so that caused</span>
<span class="c"># a failure with '-o pipefail'.</span>
<span class="c"># See https://stackoverflow.com/questions/769564/error-code-141-with-tar</span>
debug <span class="s2">"Downloading files to tar file at </span><span class="k">${</span><span class="nv">download_tar</span><span class="k">}</span><span class="s2">..."</span>
smbclient <span class="s2">"</span><span class="k">${</span><span class="nv">host_share</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-N</span> <span class="nt">-d0</span> <span class="nt">-D</span> <span class="s2">"</span><span class="k">${</span><span class="nv">share_dir</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-Tc</span> <span class="s2">"</span><span class="k">${</span><span class="nv">download_tar</span><span class="k">}</span><span class="s2">"</span>
debug <span class="s2">"Extracting downloaded files into </span><span class="k">${</span><span class="nv">download_dir</span><span class="k">}</span><span class="s2">..."</span>
<span class="c"># Extract into the download_dir, stripping off the directory name.</span>
<span class="nb">tar</span> <span class="nt">-xC</span> <span class="s2">"</span><span class="k">${</span><span class="nv">download_dir</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-f</span> <span class="s2">"</span><span class="k">${</span><span class="nv">download_tar</span><span class="k">}</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--strip-components</span><span class="o">=</span><span class="k">$((</span> <span class="k">${</span><span class="nv">dir_component_count</span><span class="k">}</span> <span class="o">+</span> <span class="m">1</span> <span class="k">))</span>
<span class="c"># Actually install the certificates - where they go and what command to</span>
<span class="c"># use to update the system store depends on if this is a Debain or Red</span>
<span class="c"># Hat like distribution (others are available, only support those two</span>
<span class="c"># currently).</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> /etc/debian_version <span class="o">]</span>
<span class="k">then
</span>debug <span class="s2">"Debian-like distribution, installing to"</span> <span class="se">\</span>
<span class="s2">"/usr/local/share/ca-certificates and using update-ca-certificates"</span>
<span class="nv">target_dir</span><span class="o">=</span><span class="s2">"/usr/local/share/ca-certificates/</span><span class="k">${</span><span class="nv">CERT_PREFIX</span><span class="k">}</span><span class="s2">"</span>
<span class="c"># Check the install directory exists</span>
<span class="k">if</span> <span class="o">!</span> <span class="o">[</span> <span class="nt">-e</span> <span class="s2">"</span><span class="k">${</span><span class="nv">target_dir</span><span class="k">}</span><span class="s2">"</span> <span class="o">]</span>
<span class="k">then
</span>debug <span class="s2">"Creating target </span><span class="k">${</span><span class="nv">target_dir</span><span class="k">}</span><span class="s2">..."</span>
<span class="c"># /usr/local/share/ca-certificates should already exist (and be</span>
<span class="c"># empty on a new install) so '-p' is unnecessary.</span>
do_su <span class="nb">mkdir</span> <span class="s2">"</span><span class="k">${</span><span class="nv">target_dir</span><span class="k">}</span><span class="s2">"</span>
<span class="c"># Certificate files should be public</span>
do_su <span class="nb">chmod </span>755 <span class="s2">"</span><span class="k">${</span><span class="nv">target_dir</span><span class="k">}</span><span class="s2">"</span>
<span class="k">else</span>
<span class="c"># Don't currently handle deleting old/defunct certificate files</span>
debug <span class="s2">"</span><span class="k">${</span><span class="nv">target_dir</span><span class="k">}</span><span class="s2"> already exists - not cleaning out any old files."</span>
<span class="k">fi</span>
<span class="c"># Install the certificates</span>
<span class="nb">set</span> +f <span class="c"># Need to glob</span>
<span class="k">for </span>file <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">download_dir</span><span class="k">}</span><span class="s2">"</span>/<span class="k">*</span>
<span class="k">do
</span><span class="nb">set</span> <span class="nt">-f</span> <span class="c"># Done globbing</span>
<span class="c"># Only copy plain files - this presumes all certificates are in</span>
<span class="c"># the top level directory but saves us from worrying about</span>
<span class="c"># clashing file names in different directories in the Windows share.</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="o">]</span>
<span class="k">then
</span>debug <span class="s2">"Installing </span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">..."</span>
<span class="c"># Does the source file end with '.crt' (required for</span>
<span class="c"># update-ca-certificates)? If not, add it.</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">"*.crt"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nv">target_file</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span>
<span class="k">else</span>
<span class="c"># Strip existing extension</span>
<span class="nv">target_file</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="p">%.*</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">.crt"</span>
<span class="k">fi
</span>debug <span class="s2">"Will copy to destination filename </span><span class="k">${</span><span class="nv">target_file</span><span class="k">}</span><span class="s2">."</span>
<span class="c"># Check permissions are what we want on source</span>
do_su <span class="nb">chmod </span>444 <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span>
<span class="c"># Copy, preserving permissions and timestamps (default also</span>
<span class="c"># preserves owner but I _want_ it to be root).</span>
do_su <span class="nb">cp</span> <span class="nt">--preserve</span><span class="o">=</span>mode,timestamps <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="se">\</span>
<span class="s2">"</span><span class="k">${</span><span class="nv">target_dir</span><span class="k">}</span><span class="s2">/</span><span class="k">${</span><span class="nv">target_file</span><span class="k">}</span><span class="s2">"</span>
<span class="k">fi
done</span>
<span class="c"># In case the loop didn't get entered</span>
<span class="nb">set</span> <span class="nt">-f</span> <span class="c"># Done globbing</span>
<span class="c"># Update the system certificates</span>
debug <span class="s2">"Running update-ca-certificates to add new certs"</span>
do_su update-ca-certificates
<span class="k">elif</span> <span class="o">[</span> <span class="nt">-f</span> /etc/redhat-release <span class="o">]</span>
<span class="k">then
</span>debug <span class="s2">"Red Hat-like distribution, installing to"</span> <span class="se">\</span>
<span class="s2">"/etc/pki/ca-trust/source/anchors and using update-ca-trust"</span>
<span class="c"># Install the certificates</span>
<span class="nb">set</span> +f <span class="c"># Need to glob</span>
<span class="k">for </span>file <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">download_dir</span><span class="k">}</span><span class="s2">"</span>/<span class="k">*</span>
<span class="k">do
</span><span class="nb">set</span> <span class="nt">-f</span> <span class="c"># Done globbing</span>
<span class="c"># Only copy plain files - this presumes all certificates are in</span>
<span class="c"># the top level directory but saves us from worrying about</span>
<span class="c"># clashing file names in different directories in the Windows share.</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="o">]</span>
<span class="k">then
</span>debug <span class="s2">"Installing </span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">..."</span>
<span class="c"># Check permissions are what we want on source</span>
do_su <span class="nb">chmod </span>444 <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">target</span><span class="o">=</span><span class="s2">"/etc/pki/ca-trust/source/anchors/</span><span class="k">${</span><span class="nv">CERT_PREFIX</span><span class="k">}</span><span class="s2">-</span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span>
<span class="o">[</span> <span class="nt">-e</span> <span class="s2">"</span><span class="k">${</span><span class="nv">target</span><span class="k">}</span><span class="s2">"</span> <span class="o">]</span> <span class="o">&&</span> <span class="nb">echo</span> <span class="s2">"WARNING: Overwriting </span><span class="k">${</span><span class="nv">target</span><span class="k">}</span><span class="s2">"</span> <span class="o">></span>&2
<span class="c"># Copy, preserving permissions and timestamps (default also</span>
<span class="c"># preserves owner but I _want_ it to be root).</span>
do_su <span class="nb">cp</span> <span class="nt">-f</span> <span class="nt">--preserve</span><span class="o">=</span>mode,timestamps <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span> <span class="s2">"</span><span class="k">${</span><span class="nv">target</span><span class="k">}</span><span class="s2">"</span>
<span class="k">fi
done</span>
<span class="c"># In case the loop didn't get entered</span>
<span class="nb">set</span> <span class="nt">-f</span> <span class="c"># Done globbing</span>
<span class="c"># Update the system certificates</span>
debug <span class="s2">"Running update-ca-trust to add new certs"</span>
do_su update-ca-trust
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Unknown distribution - unable to install certificates."</span> <span class="o">></span>&2
<span class="nv">exit_state</span><span class="o">=</span>1
<span class="k">fi</span>
<span class="c"># Tidy up</span>
debug <span class="s2">"Deleting </span><span class="k">${</span><span class="nv">download_dir</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">rm</span> <span class="nt">-rf</span> <span class="s2">"</span><span class="k">${</span><span class="nv">download_dir</span><span class="k">}</span><span class="s2">"</span>
debug <span class="s2">"Deleting </span><span class="k">${</span><span class="nv">download_tar</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">rm</span> <span class="nt">-f</span> <span class="s2">"</span><span class="k">${</span><span class="nv">download_tar</span><span class="k">}</span><span class="s2">"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Unable to separate path into //host/share and directory parts."</span> <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">fi</span>
<span class="c"># Exit with exit_state or zero (0) if none given.</span>
<span class="nb">exit</span> <span class="k">${</span><span class="nv">exit_state</span><span class="k">:-</span><span class="nv">0</span><span class="k">}</span>
</code></pre></div></div>
<h2 id="adding-custom-certificates-to-a-users-web-browser-store">Adding custom certificates to a user’s web-browser store.</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># Standard bash safety - fail on error, do not allow unintended</span>
<span class="c"># globbing, do not allow uninitialised variables, errors in pipes</span>
<span class="c"># cause whole pipe to fail.</span>
<span class="nb">set</span> <span class="nt">-efuo</span> pipefail
<span class="c"># This will be used to identify the certificates to add, from the</span>
<span class="c"># system store.</span>
<span class="nv">CERT_PREFIX</span><span class="o">=</span><span class="s2">"LOCAL-SUPER-CA"</span>
<span class="nv">DEBUG</span><span class="o">=</span><span class="nb">true
</span>debug<span class="o">()</span> <span class="o">{</span>
<span class="o">[[</span> <span class="nv">$DEBUG</span> <span class="o">=</span> <span class="nb">true</span> <span class="o">]]</span> <span class="o">&&</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> <span class="o">></span>&2
<span class="o">}</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$UID</span> <span class="nt">-eq</span> 0 <span class="o">]]</span>
<span class="k">then</span>
<span class="c"># Not to mention the question "why are you running a web-browser as root?"</span>
<span class="nb">echo</span> <span class="s2">"This script should not be run as root - adding certificates"</span> <span class="se">\</span>
<span class="s2">"to root's web browser is a bad idea."</span> <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">fi
if</span> <span class="o">!</span> <span class="nb">command</span> <span class="nt">-v</span> certutil &>/dev/null
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"certutil command not found - perhaps it needs installing"</span> <span class="se">\</span>
<span class="s2">"(libnss3-tools on Debian)?"</span> <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">fi</span>
<span class="c"># Based on https://superuser.com/a/1717924</span>
add_certificate<span class="o">()</span> <span class="o">{</span>
debug <span class="s2">"Adding </span><span class="nv">$1</span><span class="s2"> to older browsers..."</span>
<span class="k">for </span>cert_db <span class="k">in</span> <span class="si">$(</span>find ~ <span class="nt">-name</span> cert8.db<span class="si">)</span>
<span class="k">do
</span>debug <span class="s2">"Adding to </span><span class="k">${</span><span class="nv">cert_db</span><span class="k">}</span><span class="s2">..."</span>
<span class="c"># Use base filename sans extension as name</span>
certutil <span class="nt">-A</span> <span class="nt">-n</span> <span class="s2">"</span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">1</span><span class="p">%.*</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span> <span class="nt">-t</span> <span class="s2">"TCu,Cu,Tc"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">-d</span> <span class="s2">"dbm:</span><span class="si">$(</span> <span class="nb">dirname</span> <span class="s2">"</span><span class="k">${</span><span class="nv">cert_db</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span>
<span class="k">done
</span>debug <span class="s2">"Adding </span><span class="nv">$1</span><span class="s2"> to newer browsers..."</span>
<span class="k">for </span>cert_db <span class="k">in</span> <span class="si">$(</span>find ~ <span class="nt">-name</span> cert9.db<span class="si">)</span>
<span class="k">do
</span>debug <span class="s2">"Adding to </span><span class="k">${</span><span class="nv">cert_db</span><span class="k">}</span><span class="s2">..."</span>
<span class="c"># Use base filename sans extension as name</span>
certutil <span class="nt">-A</span> <span class="nt">-n</span> <span class="s2">"</span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">1</span><span class="p">%.*</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span> <span class="nt">-t</span> <span class="s2">"TCu,Cu,Tc"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">-d</span> <span class="s2">"sql:</span><span class="si">$(</span> <span class="nb">dirname</span> <span class="s2">"</span><span class="k">${</span><span class="nv">cert_db</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span>
<span class="k">done</span>
<span class="o">}</span>
<span class="c"># Like installing the certificates, where they are depends on the OS</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> /etc/debian_version <span class="o">]</span>
<span class="k">then
</span><span class="nb">set</span> +f <span class="c"># Enable globbing</span>
<span class="nb">shopt</span> <span class="nt">-s</span> nullglob <span class="c"># No matching files is fine</span>
<span class="k">for </span>file <span class="k">in</span> <span class="s2">"/usr/local/share/ca-certificates/</span><span class="k">${</span><span class="nv">CERT_PREFIX</span><span class="k">}</span><span class="s2">"</span>/<span class="k">*</span>
<span class="k">do
</span><span class="nb">set</span> <span class="nt">-f</span> <span class="c"># Done globbing</span>
<span class="nb">shopt</span> <span class="nt">-u</span> nullglob <span class="c"># Restore default (no null globbing)</span>
debug <span class="s2">"Adding file </span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">..."</span>
add_certificate <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span>
<span class="k">done</span>
<span class="c"># These are in case the loop didn't get entered</span>
<span class="nb">set</span> <span class="nt">-f</span> <span class="c"># Done globbing</span>
<span class="nb">shopt</span> <span class="nt">-u</span> nullglob <span class="c"># Restore default (no null globbing)</span>
<span class="k">elif</span> <span class="o">[</span> <span class="nt">-f</span> /etc/redhat-release <span class="o">]</span>
<span class="k">then
</span><span class="nb">set</span> +f <span class="c"># Enable globbing</span>
<span class="nb">shopt</span> <span class="nt">-s</span> nullglob <span class="c"># No matching files is fine</span>
<span class="k">for </span>file <span class="k">in</span> <span class="s2">"/etc/pki/ca-trust/source/anchors/</span><span class="k">${</span><span class="nv">CERT_PREFIX</span><span class="k">}</span><span class="s2">-"</span><span class="k">*</span>
<span class="k">do
</span><span class="nb">set</span> <span class="nt">-f</span> <span class="c"># Done globbing</span>
<span class="nb">shopt</span> <span class="nt">-u</span> nullglob <span class="c"># Restore default (no null globbing)</span>
debug <span class="s2">"Adding file </span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">..."</span>
add_certificate <span class="s2">"</span><span class="k">${</span><span class="nv">file</span><span class="k">}</span><span class="s2">"</span>
<span class="k">done</span>
<span class="c"># These are in case the loop didn't get entered</span>
<span class="nb">set</span> <span class="nt">-f</span> <span class="c"># Done globbing</span>
<span class="nb">shopt</span> <span class="nt">-u</span> nullglob <span class="c"># Restore default (no null globbing)</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Unknown distribution - unable to install certificates."</span> <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">fi</span>
</code></pre></div></div>LaurenceThis post is about deploying custom certificate authority (CA) certificates onto Linux hosts, from an anonymous Windows share, then deploying them to be used by web-browsers (which seem to use their own CA stores these days). There are two scripts, one for each of these tasks, as installing to the system store usually requires super-user (i.e. root) access but installing to the browser stores is per-user (and should not be done using the super-user account).Linux healthcheck script2023-04-16T09:20:12+01:002023-04-16T09:20:12+01:00https://blog.entek.org.uk/notes/2023/04/16/linux-healthcheck-script<p>On my systems at home I use <a href="https://icinga.com/">Icinga2</a> to monitor health, adding new checks as and when I identify something I think needs checking or if a failure occurs that was not detected. Sometimes it is necessary to do some checks via other means, such as <a href="https://slurm.schedmd.com/slurm.conf.html#OPT_HealthCheckProgram">SLURM’s healthcheck program</a> so it can be useful to have checks in script form. On previous systems, we have used the <a href="https://www.nagios.org/downloads/nagios-plugins/">Nagios plugins</a> that Icinga uses to minimise the maintenance overhead of have duplicated tests. The script will be written in bash and minimise dependencies on non-<a href="https://www.gnu.org/software/coreutils/coreutils.html">Coreutils</a> files to try and keep it portable to different distributions.</p>
<h2 id="skeleton">Skeleton</h2>
<p>The basic layout of the test script’s directory is going to be:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">run</code> - Main script that runs the tests, reports success/failure of each script, provides a failure summary and exits status <code class="language-plaintext highlighter-rouge">2</code> if the check failed (to distinguish test failure from other types of failure, which will use the generic <code class="language-plaintext highlighter-rouge">1</code> exist status).</li>
<li><code class="language-plaintext highlighter-rouge">build-single-script</code> - Creates a self-extracting and running version of the script (including tests) as a single file, to make it easier to copy to a remote system and run.</li>
<li><code class="language-plaintext highlighter-rouge">lib/</code> - Used to modularise the code, contains files sourced by the main script.</li>
<li><code class="language-plaintext highlighter-rouge">test.d/</code> - Contains the tests.</li>
</ul>
<h2 id="finding-itself">Finding itself</h2>
<p>In order to include files from <code class="language-plaintext highlighter-rouge">lib</code> and run tests in <code class="language-plaintext highlighter-rouge">test.d</code>, the script needs to work out where it is. This is easiest done using <code class="language-plaintext highlighter-rouge">realpath</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Work out where the script is located</span>
<span class="nv">my_path</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">dirname</span> <span class="s2">"</span><span class="si">$(</span> <span class="nb">realpath</span> <span class="s2">"</span><span class="nv">$0</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
<h2 id="process-the-command-line-options">Process the command-line options</h2>
<p>The first thing the script does is to process the command line options in <code class="language-plaintext highlighter-rouge">lib/command-line.bash</code> (after including the <a href="#functions">functions</a>, which provides the <code class="language-plaintext highlighter-rouge">usage</code> function). Note that the configuration variable, <code class="language-plaintext highlighter-rouge">CONFIG</code> is not exported as it is only to be consumed in the healthcheck script (which sources this file), not by test scripts launched within it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Relies on GNU enhanced getopt (not posix compliant)</span>
<span class="nv">pre_processing_count</span><span class="o">=</span><span class="nv">$#</span>
<span class="nb">eval set</span> <span class="nt">--</span> <span class="se">\</span>
<span class="si">$(</span> <span class="se">\</span>
getopt <span class="se">\</span>
<span class="nt">-l</span> unicode,no-unicode,colour,no-colour,fatal-warnings,no-fatalwarnings,help <span class="se">\</span>
<span class="nt">-o</span> uUcCfFh <span class="se">\</span>
<span class="nt">--</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> <span class="se">\</span>
<span class="si">)</span>
<span class="c"># If getopt processed all arguments (no errors) then the post-processing</span>
<span class="c"># argument count should be the pre-processing count plus one (for the</span>
<span class="c"># '--' end marker).</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">$((</span> <span class="nv">$pre_processing_count</span> <span class="o">+</span> <span class="m">1</span> <span class="k">))</span> <span class="nt">-ne</span> <span class="nv">$# </span><span class="o">]]</span>
<span class="k">then</span>
<span class="c"># Presumes getopt has output an error message (but could have been</span>
<span class="c"># given positional argument instead of parameter).</span>
<span class="nb">echo</span> <span class="s2">"Invalid usage."</span> <span class="o">></span>&2
usage <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">fi</span>
<span class="c"># Get rid of temporary variable to avoid polluting script environment.</span>
<span class="nb">unset </span>pre_processing_count
<span class="c"># Defaults - also consider this the authoritative list of options set</span>
<span class="c"># by this scriptlet.</span>
<span class="nb">declare</span> <span class="nt">-A</span> <span class="nv">CONFIG</span><span class="o">=([</span>unicode]<span class="o">=</span><span class="nb">true</span> <span class="o">[</span>colour]<span class="o">=</span>auto <span class="o">[</span>fatal_warnings]<span class="o">=</span><span class="nb">false</span><span class="o">)</span>
<span class="k">while</span> <span class="o">[</span> <span class="nv">$# </span><span class="nt">-gt</span> 0 <span class="o">]</span>
<span class="k">do
case</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="k">in</span>
<span class="nt">-u</span> <span class="p">|</span> <span class="nt">--unicode</span><span class="p">)</span>
CONFIG[unicode]<span class="o">=</span><span class="nb">true</span>
<span class="p">;;</span>
<span class="nt">-U</span> <span class="p">|</span> <span class="nt">--no-unicode</span><span class="p">)</span>
CONFIG[unicode]<span class="o">=</span><span class="nb">false</span>
<span class="p">;;</span>
<span class="nt">-c</span> <span class="p">|</span> <span class="nt">--colour</span><span class="p">)</span>
CONFIG[colour]<span class="o">=</span><span class="nb">true</span>
<span class="p">;;</span>
<span class="nt">-C</span> <span class="p">|</span> <span class="nt">--no-colour</span><span class="p">)</span>
CONFIG[colour]<span class="o">=</span><span class="nb">false</span>
<span class="p">;;</span>
<span class="nt">-f</span> <span class="p">|</span> <span class="nt">--fatal-warnings</span><span class="p">)</span>
CONFIG[fatal_warnings]<span class="o">=</span><span class="nb">true</span>
<span class="p">;;</span>
<span class="nt">-F</span> <span class="p">|</span> <span class="nt">--no-fatal-warnings</span><span class="p">)</span>
CONFIG[fatal_warnings]<span class="o">=</span><span class="nb">false</span>
<span class="p">;;</span>
<span class="nt">-h</span> <span class="p">|</span> <span class="nt">--help</span><span class="p">)</span>
usage
<span class="nb">exit </span>0
<span class="p">;;</span>
<span class="nt">--</span><span class="p">)</span>
<span class="c"># End of arguments marker</span>
<span class="nb">shift
break</span>
<span class="p">;;</span>
<span class="k">esac</span>
<span class="nb">shift
</span><span class="k">done</span>
</code></pre></div></div>
<h2 id="useful-variables">Useful variables</h2>
<p>To avoid unnecessary duplication of the calculation of generally useful information, some variables are exported by the top-level script for tests to use. Booleans are set to the lowercase strings <code class="language-plaintext highlighter-rouge">true</code> or <code class="language-plaintext highlighter-rouge">false</code>. The variables are:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">IS_ROOT</code> - is the script being run as the <code class="language-plaintext highlighter-rouge">root</code> user.</li>
<li><code class="language-plaintext highlighter-rouge">CAN_SUDO</code> - can the current user run <code class="language-plaintext highlighter-rouge">sudo</code> (n.b. this does <em>not</em> determine what commands or as who are permitted).</li>
<li><code class="language-plaintext highlighter-rouge">COLOUR_SUPPORT</code> - does the current terminal claim to support colour?</li>
<li><code class="language-plaintext highlighter-rouge">COLOURS</code> - an associative array of convenience with named variables for each of the properties and 8 basic colours:
<ul>
<li><code class="language-plaintext highlighter-rouge">COLOURS[reset]</code> - the code to clear the currently set colours/properties back to default.</li>
<li><code class="language-plaintext highlighter-rouge">COLOURS[bold]</code> - the code for bold (bright)</li>
<li><code class="language-plaintext highlighter-rouge">COLOURS[dim]</code> - the code for faint (decreased brightness)</li>
<li><code class="language-plaintext highlighter-rouge">COLOURS[blink]</code> - the code for blinking</li>
<li><code class="language-plaintext highlighter-rouge">COLOURS[underline]</code> - the code for underlined</li>
<li><code class="language-plaintext highlighter-rouge">COLOURS[fg_*]</code> - the code for foreground colours (for each of <code class="language-plaintext highlighter-rouge">black</code>, <code class="language-plaintext highlighter-rouge">red</code>, <code class="language-plaintext highlighter-rouge">green</code>, <code class="language-plaintext highlighter-rouge">yellow</code>, <code class="language-plaintext highlighter-rouge">blue</code>, <code class="language-plaintext highlighter-rouge">magenta</code>, <code class="language-plaintext highlighter-rouge">cyan</code> and <code class="language-plaintext highlighter-rouge">white</code>)</li>
<li><code class="language-plaintext highlighter-rouge">COLOURS[bg_*]</code> - the code for background colours (for each of <code class="language-plaintext highlighter-rouge">black</code>, <code class="language-plaintext highlighter-rouge">red</code>, <code class="language-plaintext highlighter-rouge">green</code>, <code class="language-plaintext highlighter-rouge">yellow</code>, <code class="language-plaintext highlighter-rouge">blue</code>, <code class="language-plaintext highlighter-rouge">magenta</code>, <code class="language-plaintext highlighter-rouge">cyan</code> and <code class="language-plaintext highlighter-rouge">white</code>)</li>
</ul>
</li>
<li><code class="language-plaintext highlighter-rouge">DIST_FAMILY</code> - distribution family (in lowercase)</li>
<li><code class="language-plaintext highlighter-rouge">DIST_DISTRIBUTION</code> - exact distribution name (in lowercase)</li>
<li><code class="language-plaintext highlighter-rouge">DIST_VERSION</code> - exact version number of distribution</li>
<li><code class="language-plaintext highlighter-rouge">DIST_VERSION_MAJOR</code> - just the major part of the version number</li>
</ul>
<p>If <code class="language-plaintext highlighter-rouge">COLOUR_SUPPORT</code> is <code class="language-plaintext highlighter-rouge">false</code> then <code class="language-plaintext highlighter-rouge">COLOURS</code> will be populated with empty strings, so that tests can blindly use them without needed to worry about supporting non-terminal or non-colour output. I.e. <code class="language-plaintext highlighter-rouge">echo "${COLOURS[red]}Hello world!${COLOURS[reset]}"</code> will work regardless of if colour is supported, the variables will simply be empty if it is not.</p>
<h3 id="distribution">Distribution</h3>
<p>The <code class="language-plaintext highlighter-rouge">DIST_</code>… variables are populated by taking my <a href="/notes/2023/04/14/distribution-detection-in-bash.html">existing detection script</a> and using it to create <code class="language-plaintext highlighter-rouge">lib/distribution-detection.bash</code>. This is then sourced by the main script:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># DIST_ variables</span>
<span class="nb">source</span> <span class="s2">"</span><span class="k">${</span><span class="nv">my_path</span><span class="k">}</span><span class="s2">/lib/distribution-detection.bash"</span>
</code></pre></div></div>
<h3 id="privileges">Privileges</h3>
<p><code class="language-plaintext highlighter-rouge">IS_ROOT</code> and <code class="language-plaintext highlighter-rouge">CAN_SUDO</code> are populated by <code class="language-plaintext highlighter-rouge">lib/privilege-escalation.bash</code>, which is two simple tests:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Is the current UID the superuser (UID zero)?</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$UID</span> <span class="nt">-eq</span> 0 <span class="o">]]</span>
<span class="k">then
</span><span class="nv">IS_ROOT</span><span class="o">=</span><span class="nb">true
</span><span class="k">else
</span><span class="nv">IS_ROOT</span><span class="o">=</span><span class="nb">false
</span><span class="k">fi</span>
<span class="c"># Can the current user in the current environment sudo with no password?</span>
<span class="c"># Will succeed if either:</span>
<span class="c"># * configured for no password.</span>
<span class="c"># * user has already entered password and sudo has not timed out the</span>
<span class="c"># authentication yet.</span>
<span class="k">if </span><span class="nb">sudo</span> <span class="nt">-n</span> <span class="nt">-l</span> &>/dev/null
<span class="k">then
</span><span class="nv">CAN_SUDO</span><span class="o">=</span><span class="nb">true
</span><span class="k">else
</span><span class="nv">CAN_SUDO</span><span class="o">=</span><span class="nb">false
</span><span class="k">fi</span>
<span class="c"># Keep variable exports collated so it is easy to refer to, in</span>
<span class="c"># order to see what variables are exported.</span>
<span class="nb">export </span>IS_ROOT CAN_SUDO
</code></pre></div></div>
<h3 id="colours">Colours</h3>
<p>Colour detection is done simply (possibly naïvely?) by checking if the terminal reports colour capability and testing if STDOUT (file descriptor <code class="language-plaintext highlighter-rouge">1</code>) is connected to a terminal in <code class="language-plaintext highlighter-rouge">lib/colour-terminal.bash</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">declare</span> <span class="nt">-A</span> <span class="nv">colour_map</span><span class="o">=([</span>black]<span class="o">=</span>0 <span class="o">[</span>red]<span class="o">=</span>1 <span class="o">[</span>green]<span class="o">=</span>2 <span class="o">[</span>yellow]<span class="o">=</span>3 <span class="o">[</span>blue]<span class="o">=</span>4 <span class="se">\</span>
<span class="o">[</span>magenta]<span class="o">=</span>5 <span class="o">[</span>cyan]<span class="o">=</span>6 <span class="o">[</span>white]<span class="o">=</span>7<span class="o">)</span>
<span class="c"># Non-colour values</span>
<span class="nb">declare</span> <span class="nt">-A</span> <span class="nv">COLOURS</span><span class="o">=([</span>reset]<span class="o">=</span><span class="s1">$'</span><span class="se">\x</span><span class="s1">1b[0m'</span> <span class="o">[</span>bold]<span class="o">=</span><span class="s1">$'</span><span class="se">\x</span><span class="s1">1b[1m'</span> <span class="o">[</span>dim]<span class="o">=</span><span class="s1">$'</span><span class="se">\x</span><span class="s1">1b[2m'</span> <span class="se">\</span>
<span class="o">[</span>underlined]<span class="o">=</span><span class="s1">$'</span><span class="se">\x</span><span class="s1">1b[4m'</span> <span class="o">[</span>blink]<span class="o">=</span><span class="s1">$'</span><span class="se">\x</span><span class="s1">1b[5m'</span><span class="o">)</span>
<span class="c"># Foreground and background colours</span>
<span class="k">for </span>key <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="p">!colour_map[@]</span><span class="k">}</span><span class="s2">"</span>
<span class="k">do
</span>COLOURS[<span class="s2">"fg_</span><span class="k">${</span><span class="nv">key</span><span class="k">}</span><span class="s2">"</span><span class="o">]=</span><span class="s2">"</span><span class="k">${</span><span class="nv">v</span><span class="k">:-</span><span class="s1">$'</span><span class="se">\x</span><span class="s1">1b'</span><span class="k">}</span><span class="s2">[3</span><span class="k">${</span><span class="nv">colour_map</span><span class="p">[</span><span class="nv">$key</span><span class="p">]</span><span class="k">}</span><span class="s2">m"</span>
COLOURS[<span class="s2">"bg_</span><span class="k">${</span><span class="nv">key</span><span class="k">}</span><span class="s2">"</span><span class="o">]=</span><span class="s2">"</span><span class="k">${</span><span class="nv">v</span><span class="k">:-</span><span class="s1">$'</span><span class="se">\x</span><span class="s1">1b'</span><span class="k">}</span><span class="s2">[4</span><span class="k">${</span><span class="nv">colour_map</span><span class="p">[</span><span class="nv">$key</span><span class="p">]</span><span class="k">}</span><span class="s2">m"</span>
<span class="k">done</span>
<span class="c"># If colour mode is forced, or automatic and the terminal reports colour</span>
<span class="c"># support and STDOUT(fd 1) is connected to a terminal (e.g. as opposed</span>
<span class="c"># to a pipe).</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">CONFIG</span><span class="p">[colour]</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"true"</span> <span class="o">]]</span> <span class="o">||</span> <span class="se">\</span>
<span class="o">(</span> <span class="se">\</span>
<span class="o">[[</span> <span class="k">${</span><span class="nv">CONFIG</span><span class="p">[colour]</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"auto"</span> <span class="o">]]</span> <span class="o">&&</span> <span class="se">\</span>
<span class="o">[[</span> <span class="s2">"</span><span class="si">$(</span>tput colors<span class="si">)</span><span class="s2">"</span> <span class="nt">-gt</span> 0 <span class="o">]]</span> <span class="o">&&</span> <span class="se">\</span>
<span class="o">[</span> <span class="nt">-t</span> 1 <span class="o">]</span> <span class="se">\</span>
<span class="o">)</span>
<span class="k">then
</span><span class="nv">COLOUR_SUPPORT</span><span class="o">=</span><span class="nb">true
</span><span class="k">else
</span><span class="nv">COLOUR_SUPPORT</span><span class="o">=</span><span class="nb">false</span>
<span class="c"># Set the colour array values to empty string, so they can be used</span>
<span class="c"># without the author worrying about whether colour works.</span>
<span class="k">for </span>key <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="p">!COLOURS[@]</span><span class="k">}</span><span class="s2">"</span>
<span class="k">do
</span>COLOURS[<span class="nv">$key</span><span class="o">]=</span><span class="s2">""</span>
<span class="k">done
fi</span>
<span class="c"># unset temporary variables to avoid environment pollution when sourced.</span>
<span class="nb">unset </span>colour_map
<span class="c"># Keep exports together to easily see what this script intentionally exports.</span>
<span class="nb">export </span>COLOUR_SUPPORT COLOURS
</code></pre></div></div>
<h2 id="functions">Functions</h2>
<p>Two main functions are in <code class="language-plaintext highlighter-rouge">lib/functions.bash</code> - <code class="language-plaintext highlighter-rouge">usage()</code>, which displays a help message, and <code class="language-plaintext highlighter-rouge">run_test()</code>, which will run a single test and return a pass (<code class="language-plaintext highlighter-rouge">0</code>) or fail (test exit status) status code.</p>
<h3 id="usage">usage</h3>
<p>This deliberately prints unicode characters in the help message, to aid users in seeing if the unicode characters are supported by their present font.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>usage<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">unicode_tick</span><span class="o">=</span><span class="s1">$'</span><span class="se">\u</span><span class="s1">2713'</span>
<span class="nb">local </span><span class="nv">unicode_cross</span><span class="o">=</span><span class="s1">$'</span><span class="se">\u</span><span class="s1">2717'</span>
<span class="nb">cat</span> - <span class="o"><<</span><span class="no">EOF</span><span class="sh">
Usage:
</span><span class="nv">$0</span><span class="sh"> [-u|-U|--unicode|--no-unicode] [-c|-C|--colour|--no-colour] [-f|-F|--fatal-warnings|--no-fatal-warnings] [-hZ--help]
-u|--unicode: use characters </span><span class="k">${</span><span class="nv">unicode_tick</span><span class="k">}</span><span class="sh">, </span><span class="k">${</span><span class="nv">unicode_cross</span><span class="k">}</span><span class="sh"> and ! to
report pass/fail/warnings (default)
-U|--no-unicode: use words [ ok ], [FAIL] and ([WARN])[warn] to report
pass/fail/(fatal)warnings
-c|--colour: force colour output
-C|--no-colour: force non-colour output
(default is to use colour if output is a terminal and reports colour support,
not otherwise)
-f|--fatal-warnings: warnings (e.g. test is not executable) are considered
failures
-F|--no-fatal-warnings: warnings (e.g. test is not executable) are printed
but not considered failures (default)
-h|--help: display this message and exit
</span><span class="no">
EOF
</span><span class="o">}</span>
</code></pre></div></div>
<h3 id="run_test">run_test</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>run_test<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">script</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">test_name</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">script</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span>
<span class="c"># Support bash/python and c/php style comments</span>
<span class="nb">local </span><span class="nv">comment_regex</span><span class="o">=</span><span class="s2">"</span><span class="se">\(</span><span class="s2">#</span><span class="se">\|</span><span class="s2">//</span><span class="se">\)</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">comment_marker</span><span class="o">=</span><span class="s2">"TEST_DESCRIPTION:"</span>
<span class="nb">local </span><span class="nv">test_description</span><span class="o">=</span><span class="s2">""</span>
<span class="k">if </span><span class="nb">grep</span> <span class="nt">-q</span> <span class="s2">"</span><span class="k">${</span><span class="nv">comment_regex</span><span class="k">}${</span><span class="nv">comment_marker</span><span class="k">}</span><span class="s2">"</span> <span class="s2">"</span><span class="k">${</span><span class="nv">script</span><span class="k">}</span><span class="s2">"</span>
<span class="k">then</span>
<span class="c"># Challenge here is finding a delimiter for sed's substitution that</span>
<span class="c"># is not likely to be used in the comment regex or marker (ruling</span>
<span class="c"># out the usual candidates of /, # and ^. % seemed the best choice</span>
<span class="c"># (it rules out using MATLAB and LaTeX comment markers in the regex)</span>
<span class="nv">test_description</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="se">\</span>
<span class="nb">grep</span> <span class="nt">-o</span> <span class="s2">"</span><span class="k">${</span><span class="nv">comment_regex</span><span class="k">}${</span><span class="nv">comment_marker</span><span class="k">}</span><span class="s2">.*"</span> <span class="s2">"</span><span class="k">${</span><span class="nv">script</span><span class="k">}</span><span class="s2">"</span> | <span class="se">\</span>
<span class="nb">sed</span> <span class="s2">"s%</span><span class="k">${</span><span class="nv">comment_regex</span><span class="k">}${</span><span class="nv">comment_marker</span><span class="k">}</span><span class="se">\\</span><span class="s2">s*%%"</span> <span class="se">\</span>
<span class="si">)</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="c"># Make sure to update the list with any new output options, to keep</span>
<span class="c"># this declaration the authoritative list of all that need setting.</span>
<span class="nb">local</span> <span class="nt">-A</span> <span class="nv">output</span><span class="o">=([</span>good]<span class="o">=</span><span class="s2">""</span> <span class="o">[</span>bad]<span class="o">=</span><span class="s2">""</span> <span class="o">[</span>warn]<span class="o">=</span><span class="s2">""</span> <span class="o">[</span>warn_fatal]<span class="o">=</span><span class="s2">""</span><span class="o">)</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">CONFIG</span><span class="p">[unicode]</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"true"</span> <span class="o">]]</span>
<span class="k">then
</span>output[good]<span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[fg_green]</span><span class="k">}${</span><span class="nv">v</span><span class="k">:-</span><span class="s1">$'</span><span class="se">\u</span><span class="s1">2713'</span><span class="k">}${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">"</span>
output[bad]<span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[bold]</span><span class="k">}${</span><span class="nv">COLOURS</span><span class="p">[fg_red]</span><span class="k">}${</span><span class="nv">v</span><span class="k">:-</span><span class="s1">$'</span><span class="se">\u</span><span class="s1">2717'</span><span class="k">}${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">"</span>
output[warn]<span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[fg_yellow]</span><span class="k">}</span><span class="s2">!</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">"</span>
output[warn_fatal]<span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[bold]</span><span class="k">}${</span><span class="nv">COLOURS</span><span class="p">[fg_red]</span><span class="k">}</span><span class="s2">!</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">"</span>
<span class="k">else</span>
<span class="c"># Non-unicode feedback messages</span>
output[good]<span class="o">=</span><span class="s2">"[ </span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[fg_green]</span><span class="k">}</span><span class="s2">ok</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2"> ]"</span>
output[bad]<span class="o">=</span><span class="s2">"[</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[bold]</span><span class="k">}${</span><span class="nv">COLOURS</span><span class="p">[fg_red]</span><span class="k">}</span><span class="s2">FAIL</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">]"</span>
output[warn]<span class="o">=</span><span class="s2">"[</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[fg_yellow]</span><span class="k">}</span><span class="s2">warn</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">]"</span>
output[warn_fatal]<span class="o">=</span><span class="s2">"[</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[bold]</span><span class="k">}${</span><span class="nv">COLOURS</span><span class="p">[fg_red]</span><span class="k">}</span><span class="s2">WARN</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">]"</span>
<span class="k">fi</span>
<span class="c"># Only displays description (in brackets) after the test name if it is</span>
<span class="c"># set.</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[fg_cyan]</span><span class="k">}</span><span class="s2">>></span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2"> Running test"</span> <span class="se">\</span>
<span class="s2">"</span><span class="k">${</span><span class="nv">test_name</span><span class="k">}${</span><span class="nv">test_description</span>:+<span class="s2">" (</span><span class="k">${</span><span class="nv">test_description</span><span class="k">}</span><span class="s2">)"</span><span class="k">}</span><span class="s2">..."</span> <span class="o">></span>&2
<span class="k">if</span> <span class="o">[</span> <span class="nt">-x</span> <span class="s2">"</span><span class="k">${</span><span class="nv">script</span><span class="k">}</span><span class="s2">"</span> <span class="o">]</span>
<span class="k">then</span>
<span class="c"># Run the test script</span>
<span class="k">${</span><span class="nv">script</span><span class="k">}</span>
<span class="nb">local </span><span class="nv">test_result</span><span class="o">=</span><span class="nv">$?</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$test_result</span> <span class="nt">-eq</span> 0 <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">output</span><span class="p">[good]</span><span class="k">}</span><span class="s2"> test passed."</span> <span class="o">></span>&2
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">output</span><span class="p">[bad]</span><span class="k">}</span><span class="s2"> test failed with status </span><span class="k">${</span><span class="nv">test_result</span><span class="k">}</span><span class="s2">."</span> <span class="o">></span>&2
<span class="k">fi
else
</span><span class="nb">local </span><span class="nv">warn_msg</span><span class="o">=</span><span class="s2">"test is not executable. Unable to run."</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">CONFIG</span><span class="p">[fatal_warnings]</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"true"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">output</span><span class="p">[warn_fatal]</span><span class="k">}</span><span class="s2"> </span><span class="k">${</span><span class="nv">warn_msg</span><span class="k">}</span><span class="s2">"</span> <span class="o">></span>&2
<span class="nb">local </span><span class="nv">test_result</span><span class="o">=</span>1
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">output</span><span class="p">[warn]</span><span class="k">}</span><span class="s2"> </span><span class="k">${</span><span class="nv">warn_msg</span><span class="k">}</span><span class="s2">"</span> <span class="o">></span>&2
<span class="nb">local </span><span class="nv">test_result</span><span class="o">=</span>0
<span class="k">fi
fi
return</span> <span class="k">${</span><span class="nv">test_result</span><span class="k">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="tests">Tests</h2>
<p>The tests in <code class="language-plaintext highlighter-rouge">test.d</code> can be any executable file (e.g. script, binary program, etc.) that exits with a status of zero (<code class="language-plaintext highlighter-rouge">0</code>) if the test passes and non-zero if it fails.</p>
<p>They can contain a comment (<code class="language-plaintext highlighter-rouge">#</code> or <code class="language-plaintext highlighter-rouge">//</code> style comments are supported) beginning with the text <code class="language-plaintext highlighter-rouge">TEST_DESCRIPTION:</code> that provides a brief (to be printed alongside the test name, which is always the filename (so the can be no confusion about which test in <code class="language-plaintext highlighter-rouge">test.d</code> it is), in the output). There must be no space between the comment marker and <code class="language-plaintext highlighter-rouge">TEST_DESCRIPTION:</code> but can be optional spaces, which will be removed, between that and the description. For example, a bash or Python script might contain <code class="language-plaintext highlighter-rouge">#TEST_DESCRIPTION: this test's description</code>.</p>
<p>For commands known to require elevated privileges, or run as a different user, the tests should use the variables <code class="language-plaintext highlighter-rouge">IS_ROOT</code> and <code class="language-plaintext highlighter-rouge">CAN_SUDO</code> to determine the appropriate mechanism for running those commands (i.e. can run directly or wrap the command in <code class="language-plaintext highlighter-rouge">sudo</code>, respectively and whether to use <code class="language-plaintext highlighter-rouge">su</code> or <code class="language-plaintext highlighter-rouge">sudo</code> to become another user). It should not be assumed <code class="language-plaintext highlighter-rouge">sudo</code> is even installed on a system. Tests should bypass anything requiring elevated privileges (printing a warning message to STDERR) if no elevation route is available - that is all tests should be runnable (even in limited form) by an unprivileged user (and the test pass in the absence of any failures).</p>
<h3 id="distribution-1">distribution</h3>
<p>Example test which performs no checks but prints the detected distribution:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c">#TEST_DESCRIPTION: display detected distribution information</span>
<span class="c"># Just display the distribution - never fails (we could check if it's recognised?)</span>
<span class="nb">echo</span> <span class="s2">"Detected </span><span class="k">${</span><span class="nv">DIST_DISTRIBUTION</span><span class="k">}</span><span class="s2"> (</span><span class="k">${</span><span class="nv">DIST_FAMILY</span><span class="k">}</span><span class="s2"> family) version"</span> <span class="se">\</span>
<span class="s2">"</span><span class="k">${</span><span class="nv">DIST_VERSION</span><span class="k">}</span><span class="s2"> (major version number </span><span class="k">${</span><span class="nv">DIST_VERSION_MAJOR</span><span class="k">}</span><span class="s2">)."</span>
</code></pre></div></div>
<h3 id="privileges-1">privileges</h3>
<p>Example test which performs no checks but prints the detected capabilities of the current user:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c">#TEST_DESCRIPTION: summary of the current user's detected privileges</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">IS_ROOT</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"true"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nv">am_root</span><span class="o">=</span><span class="s2">"is"</span>
<span class="k">else
</span><span class="nv">am_root</span><span class="o">=</span><span class="s2">"is not"</span>
<span class="k">fi</span>
<span class="c"># XXX bad practice - reusing variable name in different case.</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">CAN_SUDO</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"true"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nv">can_sudo</span><span class="o">=</span><span class="s2">"can"</span>
<span class="k">else
</span><span class="nv">can_sudo</span><span class="o">=</span><span class="s2">"cannot"</span>
<span class="k">fi
</span><span class="nb">echo</span> <span class="s2">"Current user </span><span class="k">${</span><span class="nv">am_root</span><span class="k">}</span><span class="s2"> root and </span><span class="k">${</span><span class="nv">can_sudo</span><span class="k">}</span><span class="s2"> sudo."</span>
</code></pre></div></div>
<h3 id="fail">fail</h3>
<p>Example test that always fails:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c">#TEST_DESCRIPTION: always fail - for testing</span>
<span class="nb">echo</span> <span class="s2">"This test deliberately fails!"</span> <span class="o">></span>&2
<span class="nb">exit </span>2
</code></pre></div></div>
<h2 id="main-script">Main script</h2>
<p>In full, the main script that runs each test and summarises the failures:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># Standard bash safety - disable accidental globbing, no uninitialised</span>
<span class="c"># variables, errors are fatal, errors in pipes cause pipe to error</span>
<span class="nb">set</span> <span class="nt">-fueo</span> pipefail
<span class="c"># Work out where the script is located</span>
<span class="nv">my_path</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">dirname</span> <span class="s2">"</span><span class="si">$(</span> <span class="nb">realpath</span> <span class="s2">"</span><span class="nv">$0</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span>
<span class="c"># run_test() and usage() functions</span>
<span class="nb">source</span> <span class="s2">"</span><span class="k">${</span><span class="nv">my_path</span><span class="k">}</span><span class="s2">/lib/functions.bash"</span>
<span class="c"># Process command line options and populates `config` variable</span>
<span class="nb">source</span> <span class="s2">"</span><span class="k">${</span><span class="nv">my_path</span><span class="k">}</span><span class="s2">/lib/command-line.bash"</span>
<span class="c"># Colour support detection</span>
<span class="nb">source</span> <span class="s2">"</span><span class="k">${</span><span class="nv">my_path</span><span class="k">}</span><span class="s2">/lib/colour-support.bash"</span>
<span class="c"># Priviledge escalation detection</span>
<span class="nb">source</span> <span class="s2">"</span><span class="k">${</span><span class="nv">my_path</span><span class="k">}</span><span class="s2">/lib/privilege-escalation.bash"</span>
<span class="c"># DIST_ variables</span>
<span class="nb">source</span> <span class="s2">"</span><span class="k">${</span><span class="nv">my_path</span><span class="k">}</span><span class="s2">/lib/distribution-detection.bash"</span>
<span class="c"># Do tests</span>
<span class="nb">declare</span> <span class="nt">-a</span> failed_tests <span class="c"># Array to keep a list of failing tests</span>
<span class="c">#enable globbing</span>
<span class="nb">set</span> +f
<span class="c"># Allow the '*' to match nothing (in the case of no tests exist)</span>
<span class="nb">shopt</span> <span class="nt">-s</span> nullglob
<span class="c"># Globs are expanded alphabetically (see Bash manual), so no need to</span>
<span class="c"># do anything special to run them in sequence.</span>
<span class="k">for </span>test_ <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">my_path</span><span class="k">}</span><span class="s2">"</span>/test.d/<span class="k">*</span>
<span class="k">do</span>
<span class="c"># Disable null-globbing (default) and turn off accidental globbing</span>
<span class="c"># again.</span>
<span class="nb">shopt</span> <span class="nt">-u</span> nullglob <span class="p">;</span> <span class="nb">set</span> <span class="nt">-f</span>
<span class="k">if</span> <span class="o">!</span> run_test <span class="s2">"</span><span class="k">${</span><span class="nv">test_</span><span class="k">}</span><span class="s2">"</span>
<span class="k">then
</span>failed_tests+<span class="o">=(</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">basename</span> <span class="s2">"</span><span class="k">${</span><span class="nv">test_</span><span class="k">}</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span><span class="o">)</span>
<span class="k">fi
done</span>
<span class="c"># Just in case the look didn't get entered - reinforce disabling</span>
<span class="c"># accidental globbing.</span>
<span class="nb">shopt</span> <span class="nt">-u</span> nullglob <span class="p">;</span> <span class="nb">set</span> <span class="nt">-f</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="k">${#</span><span class="nv">failed_tests</span><span class="p">[@]</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-eq</span> 0 <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[underlined]</span><span class="k">}</span><span class="s2">All tests passed.</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">exit </span>0
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[underlined]</span><span class="k">}</span><span class="s2">Some tests failed.</span><span class="k">${</span><span class="nv">COLOURS</span><span class="p">[reset]</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"List of failed tests:"</span>
<span class="k">for </span>failure <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">failed_tests</span><span class="p">[@]</span><span class="k">}</span><span class="s2">"</span>
<span class="k">do
</span><span class="nb">echo</span> <span class="s2">" * </span><span class="k">${</span><span class="nv">failure</span><span class="k">}</span><span class="s2">"</span>
<span class="k">done</span>
<span class="c"># Use 2 to distinguish "some test failed" from "some unintended error</span>
<span class="c"># occurred"</span>
<span class="nb">exit </span>2
<span class="k">fi</span>
</code></pre></div></div>
<h2 id="build-single-script">build-single-script</h2>
<p>This script takes the main script, lib and all tests and bundles them into an archive that is prepended by a bash script that makes it a self-extracting and running script. This script takes a non-optional argument, which is the name of the script to create.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># Standard bash safety - disable accidental globbing, no uninitialised</span>
<span class="c"># variables, errors are fatal, errors in pipes cause pipe to error</span>
<span class="nb">set</span> <span class="nt">-fueo</span> pipefail
<span class="c"># Work out where the script is located</span>
<span class="nv">my_path</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">dirname</span> <span class="s2">"</span><span class="si">$(</span> <span class="nb">realpath</span> <span class="s2">"</span><span class="nv">$0</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span> <span class="si">)</span><span class="s2">"</span>
<span class="c"># Helper to print usage message</span>
usage<span class="o">()</span> <span class="o">{</span>
<span class="nb">cat</span> - <span class="o"><<</span><span class="no">EOF</span><span class="sh">
Usage:
</span><span class="nv">$0</span><span class="sh"> [-h|--help|script_name_to_create]
script_name_to_create: name of the script that will be created (must not
exist)
-h|--help: display this message and exit
</span><span class="no">
EOF
</span><span class="o">}</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$# </span><span class="nt">-ne</span> 1 <span class="o">]]</span>
<span class="k">then</span>
<span class="c"># No enough arguments</span>
<span class="nb">echo</span> <span class="s2">"Incorrect number of arguments: $#"</span> <span class="o">></span>&2
usage <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">elif</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">=</span> <span class="s1">'-h'</span> <span class="o">]]</span> <span class="o">||</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">=</span> <span class="s1">'--help'</span> <span class="o">]]</span>
<span class="k">then</span>
<span class="c"># Explicit help request</span>
usage
<span class="nb">exit </span>0
<span class="k">fi
</span><span class="nv">script_name</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nt">-e</span> <span class="k">${</span><span class="nv">script_name</span><span class="k">}</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Error: </span><span class="k">${</span><span class="nv">script_name</span><span class="k">}</span><span class="s2"> already exists."</span> <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">fi</span>
<span class="c"># Script preamble - quoting the heredoc tag disables interpolations</span>
<span class="nb">cat</span> - <span class="o">></span><span class="s2">"</span><span class="k">${</span><span class="nv">script_name</span><span class="k">}</span><span class="s2">"</span> <span class="o"><<</span><span class="sh">'</span><span class="no">END</span><span class="sh">'
#!/bin/bash
# Standard bash safety - disable accidental globbing, no uninitialised
# variables, errors are fatal, errors in pipes cause pipe to error
set -fueo pipefail
# Helper to print usage message
usage() {
cat - <<EOF
Usage:
</span><span class="nv">$0</span><span class="sh"> [-h|--help] [-ttempdir|--tmpdir=tempdir] [--] [run_arguments]
-t|--tmpdir: Specify (as an argument to this option) the temporary
directory to extract to - must allow execution (i.e.
not be mounted "noexec"). Defaults to TMPDIR
environment variable, if set, or /tmp if not.
--: anything after this marker will be passed to the extracted
healthcheck script's command line options.
-h|--help: display this message and exit
EOF
}
# Process commandline
eval set -- </span><span class="si">$(</span>getopt <span class="nt">-l</span> tmpdir:,help <span class="nt">-o</span> t:h <span class="nt">--</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="si">)</span><span class="sh">
temp_dir="</span><span class="k">${</span><span class="nv">TMPDIR</span><span class="k">:-</span><span class="p">/tmp</span><span class="k">}</span><span class="sh">"
while [ </span><span class="nv">$# </span><span class="sh">-gt 0 ]
do
case "</span><span class="nv">$1</span><span class="sh">" in
-t | --tempdir)
temp_dir="</span><span class="nv">$2</span><span class="sh">"
shift # need an extra script - processing 2 arguments here
;;
-h | --help)
usage
exit 0
;;
--)
# End of arguments marker
shift
break
;;
esac
shift
done
# Processed our arguments and shifted the end of arguments marker -
# everything left is for the 'run' script
# Make a temporary directory for the files
out_dir="</span><span class="si">$(</span> <span class="nb">mktemp</span> <span class="nt">-d</span> <span class="nt">-p</span> <span class="s2">"</span><span class="k">${</span><span class="nv">temp_dir</span><span class="k">}</span><span class="s2">"</span><span class="si">)</span><span class="sh">"
# Extract the archive at the end of this script
echo "Extracting healthcheck to </span><span class="k">${</span><span class="nv">out_dir</span><span class="k">}</span><span class="sh">..."
sed -e '1,/^__END__</span><span class="nv">$/</span><span class="sh">d' "</span><span class="nv">$0</span><span class="sh">" | tar -C "</span><span class="k">${</span><span class="nv">out_dir</span><span class="k">}</span><span class="sh">" -zx
# Run the run script
set +e # Want to tidy up, even if run fails
# Run the script with all remaining (not consumed by our options
# processing) arguments.
echo "Running healthcheck..."
"</span><span class="k">${</span><span class="nv">out_dir</span><span class="k">}</span><span class="sh">/run" "</span><span class="nv">$@</span><span class="sh">"
set -e # Die if this script errors, though.
# Tidy up the temporary files
echo "Deleting </span><span class="k">${</span><span class="nv">out_dir</span><span class="k">}</span><span class="sh">..."
rm -rf "</span><span class="k">${</span><span class="nv">out_dir</span><span class="k">}</span><span class="sh">"
# End of script - archive follows
echo "All done."
exit
__END__
</span><span class="no">END
</span><span class="c"># Create the archive at the end of the file</span>
<span class="nb">tar</span> <span class="nt">-C</span> <span class="s2">"</span><span class="k">${</span><span class="nv">my_path</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-zc</span> run lib/ test.d/ <span class="o">>></span> <span class="s2">"</span><span class="k">${</span><span class="nv">script_name</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Created </span><span class="k">${</span><span class="nv">script_name</span><span class="k">}</span><span class="s2">."</span>
</code></pre></div></div>
<p>Here is what asking for the healthcheck script’s help with the self-extracting script looks like:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>bash <span class="nb">test</span> <span class="nt">--</span> <span class="nt">-h</span>
<span class="go">Extracting healthcheck to /tmp/tmp.UWWlnK7f8Y...
Running healthcheck...
Usage:
/tmp/tmp.UWWlnK7f8Y/run [-u|-U|--unicode|--no-unicode] [-c|-C|--colour|--no-colour] [-f|-F|--fatal-warnings|--no-fatal-warnings] [-h|--help]
-u|--unicode: use characters ✓, ✗ and ! to
report pass/fail/warnings (default)
-U|--no-unicode: use words [ ok ], [FAIL] and ([WARN])[warn] to report
pass/fail/(fatal)warnings
-c|--colour: force colour output
-C|--no-colour: force non-colour output
(default is to use colour if output is a terminal and reports colour support,
not otherwise)
-f|--fatal-warnings: warnings (e.g. test is not executable) are considered
failures
-F|--no-fatal-warnings: warnings (e.g. test is not executable) are printed
but not considered failures (default)
-h|--help: display this message and exit
Deleting /tmp/tmp.UWWlnK7f8Y...
All done.
</span></code></pre></div></div>LaurenceOn my systems at home I use Icinga2 to monitor health, adding new checks as and when I identify something I think needs checking or if a failure occurs that was not detected. Sometimes it is necessary to do some checks via other means, such as SLURM’s healthcheck program so it can be useful to have checks in script form. On previous systems, we have used the Nagios plugins that Icinga uses to minimise the maintenance overhead of have duplicated tests. The script will be written in bash and minimise dependencies on non-Coreutils files to try and keep it portable to different distributions.Distribution detection in Bash2023-04-14T03:59:48+01:002023-04-14T03:59:48+01:00https://blog.entek.org.uk/notes/2023/04/14/distribution-detection-in-bash<p>In order to write some portable health-checking scripts, I needed to reliably detect both distribution (or at least distribution family) and version (e.g. to cope with <code class="language-plaintext highlighter-rouge">rpm</code> moving from <code class="language-plaintext highlighter-rouge">/bin</code> to <code class="language-plaintext highlighter-rouge">/usr/bin</code> in Red Hat 7). I based this on my previous <a href="https://www.lua.org/">Lua</a> <a href="/notes/2021/07/27/platform-detection-with-lmod.html">distribution detection script</a> for <a href="https://lmod.readthedocs.io/en/latest/index.html">Lmod</a>. In contrast to the Lmod script, I am not interested in any CPU/architecture detection (at present). I only need this for <a href="https://www.redhat.com/technologies/linux-platforms/enterprise-linux">Red Hat Enterprise Linux</a> family (including <a href="https://www.centos.org/">CentOS</a>, <a href="https://scientificlinux.org/">Scientific Linux</a> and <a href="https://rockylinux.org/">Rocky</a>), <a href="https://ubuntu.com/">Ubuntu</a> and <a href="https://www.debian.org/">Debian</a> distributions. Adding others would be trivial, it is just a case of finding their <a href="https://en.wikipedia.org/wiki/Linux_Standard_Base"><code class="language-plaintext highlighter-rouge">lsb_release</code></a> return values and an appropriate fallback (file) method.</p>
<p>Aside: <a href="https://listserv.fnal.gov/scripts/wa.exe?A2=SCIENTIFIC-LINUX-ANNOUNCE;11d6001.1904">Announcement on the end of Scientific Linux with version 7</a></p>
<p>These are the variables I am trying to populate:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">DIST_FAMILY</code> - in my case, either “RedHat” or “Debian”</li>
<li><code class="language-plaintext highlighter-rouge">DIST_DISTRIBUTION</code> - the actual distribution, e.g. “RedHatEnterpriseLinux”, “Rocky”, “Ubuntu”, “Debian” etc.</li>
<li><code class="language-plaintext highlighter-rouge">DIST_VERSION</code> - the full version</li>
<li><code class="language-plaintext highlighter-rouge">DIST_VERSION_MAJOR</code> - just the major version number</li>
</ul>
<p>To complicate things slightly, unlike the systems I wrote the Lmod version for, some systems I need this to work on are missing the <code class="language-plaintext highlighter-rouge">lsb_release</code> command so I do need the fall-back “old” detection methods.</p>
<h2 id="testing-with-docker">Testing with Docker</h2>
<p><a href="https://www.docker.com/">Docker</a> has fallen out of favour with many due to the <a href="https://www.docker.com/blog/updating-product-subscriptions/">changes to the Docker Desktop licence</a> that suddenly made it very expensive (previously free) in anything other than very small organisations that do not sell many or high-value goods (as the limit is based on employees and annual revenue, i.e. sales). Docker Engine (which is still free everywhere) is, however, still resting unused on my laptop so is convenient for this “hobby project” work until I get around to moving to something more open, like <a href="https://podman.io/">Podman</a>.</p>
<h3 id="fetching-containers">Fetching containers</h3>
<p>To get containers for all distributions and versions under test:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># redhat/ubi[89]-micro have no package manager, which I need to create -lsb variants</span>
<span class="c"># hjd48/redhat - claimed to be Red Hat 6,2 (but seems to be 6.3)</span>
<span class="c"># ubuntu:22.04 == ubuntu:latest (current LTS)</span>
<span class="k">for </span>image <span class="k">in </span>sl:7 <span class="se">\</span>
redhat/ubi9 redhat/ubi8 yjjy0921/redhat7.2 hjd48/redhat <span class="se">\</span>
centos:7 centos:6 <span class="se">\</span>
rockylinux:9 rockylinux:8 <span class="se">\</span>
debian:11-slim debian:10-slim debian:9-slim <span class="se">\</span>
ubuntu:22.10 ubuntu:22.04 ubuntu:20.04 ubuntu:18.04
<span class="k">do
</span>docker pull <span class="k">${</span><span class="nv">image</span><span class="k">}</span>
<span class="k">done</span>
</code></pre></div></div>
<p>Neither of the Red Hat 6 based (Red Hat 6 and CentOS 6) containers would run <code class="language-plaintext highlighter-rouge">bash</code> (or <code class="language-plaintext highlighter-rouge">yum</code>), they gave a segmentation fault error (exit status <code class="language-plaintext highlighter-rouge">139</code>) when I tried. However <code class="language-plaintext highlighter-rouge">cat</code> worked, so I was able to see their respective <code class="language-plaintext highlighter-rouge">/etc/redhat-release</code> files contained <code class="language-plaintext highlighter-rouge">Red Hat Enterprise Linux Server release 6.3 (Santiago)</code> and <code class="language-plaintext highlighter-rouge">CentOS release 6.10 (Final)</code>. For comparison, Red Hat 7 and CentOS 7 were <code class="language-plaintext highlighter-rouge">Red Hat Enterprise Linux Server release 7.2 (Maipo)</code> and <code class="language-plaintext highlighter-rouge">CentOS Linux release 7.9.2009 (Core)</code>.</p>
<h3 id="creating-lsb-release-variants">Creating lsb-release variants</h3>
<p>Out of the box, none of these minimal images provided <code class="language-plaintext highlighter-rouge">lsb_release</code> so I created variants (tagged with existing tag plus suffix <code class="language-plaintext highlighter-rouge">-lsb</code>) with them installed. The Red Hat 7 image I used has no functioning repositories configured, so I did not do that one, <code class="language-plaintext highlighter-rouge">lsb_release</code> has been <a href="https://access.redhat.com/solutions/6960807">removed from Red Hat 9</a> (and by extension Rocky 9).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for </span>image <span class="k">in </span>sl:7 <span class="se">\</span>
redhat/ubi8 <span class="se">\</span>
centos:7 <span class="se">\</span>
rockylinux:8 <span class="se">\</span>
debian:11-slim debian:10-slim debian:9-slim <span class="se">\</span>
ubuntu:22.10 ubuntu:22.04 ubuntu:20.04 ubuntu:18.04
<span class="k">do
</span>docker image build <span class="nt">-t</span> <span class="si">$(</span> <span class="k">if</span> <span class="o">[[</span> <span class="nv">$image</span> <span class="o">=</span> <span class="k">*</span>:<span class="k">*</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then </span><span class="nb">echo</span> <span class="k">${</span><span class="nv">image</span><span class="k">}</span><span class="nt">-lsb</span><span class="p">;</span> <span class="k">else </span><span class="nb">echo</span> <span class="k">${</span><span class="nv">image</span><span class="k">}</span>:lsb<span class="p">;</span> <span class="k">fi</span> <span class="si">)</span> - <span class="o"><<</span><span class="no">EOF</span><span class="sh">
FROM </span><span class="k">${</span><span class="nv">image</span><span class="k">}</span><span class="sh">
RUN /bin/bash -c 'set -e; if command -v apt-get &>/dev/null; </span><span class="se">\</span><span class="sh">
then </span><span class="se">\</span><span class="sh">
echo "Installing lsb_release with apt-get..."; </span><span class="se">\</span><span class="sh">
apt-get update &>/dev/null; </span><span class="se">\</span><span class="sh">
apt-get -y install lsb-release &>/dev/null; </span><span class="se">\</span><span class="sh">
elif command -v dnf &>/dev/null; </span><span class="se">\</span><span class="sh">
then </span><span class="se">\</span><span class="sh">
echo "Installing lsb_release with dnf..."; </span><span class="se">\</span><span class="sh">
dnf -q -y install redhat-lsb-core; </span><span class="se">\</span><span class="sh">
elif command -v yum &>/dev/null; </span><span class="se">\</span><span class="sh">
then </span><span class="se">\</span><span class="sh">
echo "Installing lsb_release with yum..."; </span><span class="se">\</span><span class="sh">
yum -q -y install redhat-lsb-core; </span><span class="se">\</span><span class="sh">
else </span><span class="se">\</span><span class="sh">
echo "Unable to install lsb_release - could not locate package manager" >&2; </span><span class="se">\</span><span class="sh">
exit 1; </span><span class="se">\</span><span class="sh">
fi; </span><span class="se">\</span><span class="sh">
command -v lsb_release'
</span><span class="no">EOF
</span><span class="k">done</span>
</code></pre></div></div>
<h3 id="testing-with-all-working-containers">Testing with all (working) containers</h3>
<p>Presuming the script to test is in the current directory, called <code class="language-plaintext highlighter-rouge">detection_test</code> and is executable; this will test with each image in turn:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for </span>image <span class="k">in </span>sl:7 <span class="se">\</span>
redhat/ubi9 redhat/ubi8 yjjy0921/redhat7.2 <span class="se">\</span>
centos:7 <span class="se">\</span>
rockylinux:9 rockylinux:8 <span class="se">\</span>
debian:11-slim debian:10-slim debian:9-slim <span class="se">\</span>
ubuntu:22.10 ubuntu:22.04 ubuntu:20.04 ubuntu:18.04
<span class="k">do
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\e</span><span class="s2">[1;36m></span><span class="se">\e</span><span class="s2">[1;0m Running test script in </span><span class="k">${</span><span class="nv">image</span><span class="k">}</span><span class="s2">..."</span>
<span class="k">if </span>docker run <span class="nt">--rm</span> <span class="nt">--mount</span> <span class="nb">type</span><span class="o">=</span><span class="nb">bind</span>,source<span class="o">=</span><span class="k">${</span><span class="nv">PWD</span><span class="k">}</span>,destination<span class="o">=</span>/script,readonly <span class="k">${</span><span class="nv">image</span><span class="k">}</span> /script/detection_test
<span class="k">then
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\e</span><span class="s2">[1;32m</span><span class="se">\u</span><span class="s2">2713</span><span class="se">\e</span><span class="s2">[1;0m </span><span class="k">${</span><span class="nv">image</span><span class="k">}</span><span class="s2"> </span><span class="se">\e</span><span class="s2">[1;32mpassed</span><span class="se">\e</span><span class="s2">[1;0m"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\e</span><span class="s2">[1;31m</span><span class="se">\u</span><span class="s2">2717</span><span class="se">\e</span><span class="s2">[1;0m </span><span class="k">${</span><span class="nv">image</span><span class="k">}</span><span class="s2"> </span><span class="se">\e</span><span class="s2">[1;31mFAILED</span><span class="se">\e</span><span class="s2">[1;0m"</span>
<span class="k">fi
</span><span class="nv">lsb_variant</span><span class="o">=</span><span class="si">$(</span> <span class="k">if</span> <span class="o">[[</span> <span class="nv">$image</span> <span class="o">=</span> <span class="k">*</span>:<span class="k">*</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then </span><span class="nb">echo</span> <span class="k">${</span><span class="nv">image</span><span class="k">}</span><span class="nt">-lsb</span><span class="p">;</span> <span class="k">else </span><span class="nb">echo</span> <span class="k">${</span><span class="nv">image</span><span class="k">}</span>:lsb<span class="p">;</span> <span class="k">fi</span> <span class="si">)</span>
<span class="c"># Is there an lsb variant to test?</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nt">-n</span> <span class="s2">"</span><span class="si">$(</span> docker images <span class="nt">-q</span> <span class="k">${</span><span class="nv">lsb_variant</span><span class="k">}</span> <span class="si">)</span><span class="s2">"</span> <span class="o">]]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\e</span><span class="s2">[1;36m>></span><span class="se">\e</span><span class="s2">[1;0m Running test script in </span><span class="k">${</span><span class="nv">lsb_variant</span><span class="k">}</span><span class="s2">..."</span>
<span class="k">if </span>docker run <span class="nt">--rm</span> <span class="nt">--mount</span> <span class="nb">type</span><span class="o">=</span><span class="nb">bind</span>,source<span class="o">=</span><span class="k">${</span><span class="nv">PWD</span><span class="k">}</span>,destination<span class="o">=</span>/script,readonly <span class="k">${</span><span class="nv">lsb_variant</span><span class="k">}</span> /script/detection_test
<span class="k">then
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\e</span><span class="s2">[1;32m</span><span class="se">\u</span><span class="s2">2713</span><span class="se">\e</span><span class="s2">[1;0m </span><span class="k">${</span><span class="nv">lsb_variant</span><span class="k">}</span><span class="s2"> </span><span class="se">\e</span><span class="s2">[1;32mpassed</span><span class="se">\e</span><span class="s2">[1;0m"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\e</span><span class="s2">[1;31m</span><span class="se">\u</span><span class="s2">2717</span><span class="se">\e</span><span class="s2">[1;0m </span><span class="k">${</span><span class="nv">lsb_variant</span><span class="k">}</span><span class="s2"> </span><span class="se">\e</span><span class="s2">[1;31mFAILED</span><span class="se">\e</span><span class="s2">[1;0m"</span>
<span class="k">fi
else
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\e</span><span class="s2">[1;33m?</span><span class="se">\e</span><span class="s2">[1;0m No LSB variant for </span><span class="k">${</span><span class="nv">image</span><span class="k">}</span><span class="s2">."</span> <span class="o">></span>&2
<span class="k">fi
done</span>
</code></pre></div></div>
<p>To help distinguish each test, and success from failure, I’ve used colour (and some unicode characters). For running interactively by copy & pasting into a shell, this is fine but good practice would be to test if the output is a terminal and supports colour if this is to be turned into a script.</p>
<p>I use <code class="language-plaintext highlighter-rouge">--mount</code> based on what is described as <a href="https://docs.docker.com/storage/bind-mounts/#choose-the--v-or---mount-flag">good practice in Docker’s documentation</a>:</p>
<blockquote>
<p>New users should use the –mount syntax. Experienced users may be more familiar with the -v or –volume syntax, but are encouraged to use –mount, because research has shown it to be easier to use.</p>
</blockquote>
<h2 id="doing-the-detection">Doing the detection</h2>
<h3 id="mapping-distribution-to-families">Mapping distribution to families</h3>
<p>Using Bash’s <a href="https://www.gnu.org/software/bash/manual/html_node/Arrays.html">associative array</a>, a direct replica of the lookup table in my Lmod version can be made.</p>
<p>Unlike the older <a href="#detecting-distribution-with-lsb_release"><code class="language-plaintext highlighter-rouge">lsb_release</code></a> and <a href="#detecting-distribution-with-release-files">distribution-specific release file</a> methods, <a href="#detecting-distribution-with-os-release-files"><code class="language-plaintext highlighter-rouge">os-release</code></a> mandates that distribution ID values are lowercase. I thought long and hard about whether to follow this pattern or map it to the older mixed-case form favoured by <code class="language-plaintext highlighter-rouge">lsb_release</code> and decided to go with lowercase for these reasons:</p>
<ol>
<li>Harder to read for a human (this is the major ‘con’ - i.e. it is less pretty)</li>
<li>“All values will be lowercase” is easier for script-writers (e.g. do not need to know if it’s “Centos” or “CentOS”, “RedHatEnterprise” or “RedhatEnterprise”)</li>
<li>As much as I hate aspects of it, <code class="language-plaintext highlighter-rouge">systemd</code> is here to stay and this is therefore the current standard. Complying with standards is usually good for portability and future maintenance.</li>
</ol>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Map of distributions to their family</span>
<span class="nb">declare</span> <span class="nt">-A</span> <span class="nv">distribution_family</span><span class="o">=([</span>debian]<span class="o">=</span>debian <span class="o">[</span>ubuntu]<span class="o">=</span>debian <span class="se">\</span>
<span class="o">[</span>scientific]<span class="o">=</span>redhat <span class="o">[</span>redhatenterpriseserver]<span class="o">=</span>redhat <span class="o">[</span>centos]<span class="o">=</span>redhat <span class="se">\</span>
<span class="o">[</span>redhatenterprise]<span class="o">=</span>redhat <span class="o">[</span>rocky]<span class="o">=</span>redhat<span class="o">)</span>
</code></pre></div></div>
<h3 id="detecting-distribution-with-os-release-files">Detecting distribution with <code class="language-plaintext highlighter-rouge">os-release</code> files</h3>
<p>These files were <a href="https://www.freedesktop.org/software/systemd/man/os-release.html">introduced by systemd</a> as an alternative to distribution-specific release files and <a href="http://0pointer.de/blog/projects/os-release">are mandatory for systemd support</a>. Most systems now run <a href="https://www.freedesktop.org/wiki/Software/systemd/">systemd</a>, making this file mandatory.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /etc/os-release has precedence over /usr/lib/os-release, so source second</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">source</span> /usr/lib/os-release<span class="p">;</span> <span class="nb">source</span> /etc/os-release <span class="p">;</span> <span class="nb">echo</span> <span class="nv">$ID</span> <span class="si">)</span><span class="s2">"</span>
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">source</span> /usr/lib/os-release<span class="p">;</span> <span class="nb">source</span> /etc/os-release <span class="p">;</span> <span class="nb">echo</span> <span class="nv">$VERSION_ID</span> <span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
<h3 id="detecting-distribution-with-lsb_release">Detecting distribution with lsb_release</h3>
<p>This is an easy case, since <code class="language-plaintext highlighter-rouge">lsb_release</code> provides a consistent interface to looking this up across distributions.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">lsb_dist</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> lsb_release <span class="nt">-i</span> <span class="nt">-s</span> <span class="si">)</span><span class="s2">"</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">lsb_dist</span><span class="p">,,</span><span class="k">}</span><span class="s2">"</span> <span class="c"># Convert to lowercase</span>
<span class="c"># Assuming version is numeric therefore doesn't need lowercasing</span>
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> lsb_release <span class="nt">-r</span> <span class="nt">-s</span> <span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
<h3 id="detecting-distribution-with-release-files">Detecting distribution with release files</h3>
<p>Red Hat family distributions (at least all the ones I care about) all have <code class="language-plaintext highlighter-rouge">/etc/redhat-release</code>, which means there’s one recipe for all of them. This is inspired slightly by <a href="https://access.redhat.com/discussions/1312334">a discussion on Red Hat’s website</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># RedHat family distributions all have this file</span>
<span class="c"># sed to make it conform to what `lsb_release -d` would produce if it were available</span>
<span class="nv">file_dist</span><span class="o">=</span><span class="si">$(</span> <span class="nb">sed</span> <span class="s1">'s/ release [0-9].*$//; s/ linux$//i; s/ linux //ig; s/ //g'</span> /etc/redhat-release <span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"Got dist </span><span class="k">${</span><span class="nv">file_dist</span><span class="p">,,</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">file_dist</span><span class="p">,,</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="si">$(</span> <span class="nb">sed</span> <span class="s1">'s/^.* release \([0-9][0-9.]*\).*$/\1/'</span> /etc/redhat-release <span class="si">)</span>
</code></pre></div></div>
<p>On Ubuntu, <code class="language-plaintext highlighter-rouge">/etc/lsb-release</code> exists even if <code class="language-plaintext highlighter-rouge">lsb_release</code> (<code class="language-plaintext highlighter-rouge">lsb-release</code> package) is not installed. So this can be used on that distribution:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Convert to lowercase</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">source</span> /etc/lsb-release <span class="p">;</span> <span class="nb">echo</span> <span class="k">${</span><span class="nv">DISTRIB_ID</span><span class="p">,,</span><span class="k">}</span> <span class="si">)</span><span class="s2">"</span>
<span class="c"># Assuming version is numeric therefore doesn't need lowercasing</span>
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">source</span> /etc/lsb-release <span class="p">;</span> <span class="nb">echo</span> <span class="k">${</span><span class="nv">DISTRIB_RELEASE</span><span class="k">}</span> <span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
<p>Finally, on “vanilla” Debian, <code class="language-plaintext highlighter-rouge">/etc/debian_version</code> holds the version number (although not on <code class="language-plaintext highlighter-rouge">sid</code>, a.k.a. <code class="language-plaintext highlighter-rouge">unstable</code>):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Can only be Debian (Ubuntu doesn't have this file)</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span>debian
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">cat</span> /etc/debian_version<span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
<h3 id="mapping-distribution-to-family-and-major-version-number">Mapping distribution to family and major version number</h3>
<p>The last piece, once the <code class="language-plaintext highlighter-rouge">DIST_DISTRIBUTION</code> and <code class="language-plaintext highlighter-rouge">DIST_VERSION</code> variables have been discovered, is to use <a href="#mapping-distribution-to-families">the lookup table defined at the start</a> to find the family and extract the major part from the version:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">DIST_FAMILY</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">distribution_family</span><span class="p">[</span><span class="nv">$DIST_DISTRIBUTION</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">DIST_DISTRIBUTION</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"ubuntu"</span> <span class="o">]]</span>
<span class="k">then</span>
<span class="c"># Every release of Ubuntu is a major release</span>
<span class="nv">DIST_VERSION_MAJOR</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">DIST_VERSION</span><span class="k">}</span><span class="s2">"</span>
<span class="k">else</span>
<span class="c"># Everything up to first '.' is the major version</span>
<span class="nv">DIST_VERSION_MAJOR</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">DIST_VERSION</span><span class="p">%%.*</span><span class="k">}</span><span class="s2">"</span>
<span class="k">fi</span>
</code></pre></div></div>
<h3 id="exporting-the-variables">Exporting the variables</h3>
<p>Finally, to make the variables available in sub-shells (including other scripts run from this shell session), the variables need to be exported. This is optional, if they are to be consumed in the script that does the detection. I explicitly export them, as a list, at the end as it provides a single place to see all of the variables this script exports (as opposed to exporting them as they are defined).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span>DIST_FAMILY DIST_DISTRIBUTION DIST_VERSION DIST_VERSION_MAJOR
</code></pre></div></div>
<h2 id="putting-it-together">Putting it together</h2>
<p>I started by using <a href="#detecting-distribution-with-os-release-files"><code class="language-plaintext highlighter-rouge">os-release</code></a> as the first thing used to attempt detection, due to it being required (and therefore available) on all systems with systemd. However, I quickly discovered that this has less detailed information (particularly for version) on many distributions (Red Hat 7 and other RHEL 7 clones, Debian), so moved it to being the last resort.</p>
<p>In the end, the detections in my script follow this sequence:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">lsb_release</code></li>
<li>os-specific release files:
<ol>
<li><code class="language-plaintext highlighter-rouge">/etc/redhat-release</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/lsb-release</code> (for Ubuntu without <code class="language-plaintext highlighter-rouge">lsb-release</code> package installed, to avoid mis-detection as Debian by next check)</li>
<li><code class="language-plaintext highlighter-rouge">/etc/debian_version</code></li>
</ol>
</li>
<li><code class="language-plaintext highlighter-rouge">os-release</code> (systemd)</li>
</ol>
<p>The full script looks like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># Standard bash safety - errors are fatal, no accidental globbing,</span>
<span class="c"># unset variables are errors, errors in pipes cause whole pipe to</span>
<span class="c"># error.</span>
<span class="nb">set</span> <span class="nt">-efuo</span> pipefail
<span class="c"># distribution -> family lookup</span>
<span class="nb">declare</span> <span class="nt">-A</span> <span class="nv">distribution_family</span><span class="o">=([</span>debian]<span class="o">=</span>debian <span class="o">[</span>ubuntu]<span class="o">=</span>debian <span class="se">\</span>
<span class="o">[</span>scientific]<span class="o">=</span>redhat <span class="o">[</span>redhatenterpriseserver]<span class="o">=</span>redhat <span class="o">[</span>centos]<span class="o">=</span>redhat <span class="se">\</span>
<span class="o">[</span>redhatenterprise]<span class="o">=</span>redhat <span class="o">[</span>rocky]<span class="o">=</span>redhat<span class="o">)</span>
<span class="c"># lsb_release for easiest, complete, cross-distro discovery</span>
<span class="k">if </span><span class="nb">command</span> <span class="nt">-v</span> lsb_release &>/dev/null
<span class="k">then
</span><span class="nv">lsb_dist</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> lsb_release <span class="nt">-i</span> <span class="nt">-s</span> <span class="si">)</span><span class="s2">"</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">lsb_dist</span><span class="p">,,</span><span class="k">}</span><span class="s2">"</span> <span class="c"># Convert to lowercase</span>
<span class="c"># Assuming version is numeric therefore doesn't need lowercasing</span>
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> lsb_release <span class="nt">-r</span> <span class="nt">-s</span> <span class="si">)</span><span class="s2">"</span>
<span class="c"># release file methods</span>
<span class="k">elif</span> <span class="o">[</span> <span class="nt">-e</span> /etc/redhat-release <span class="o">]</span>
<span class="k">then</span>
<span class="c"># RedHat family distributions all have this file</span>
<span class="c"># sed to make it conform to what `lsb_release -d` would produce if it were available</span>
<span class="nv">file_dist</span><span class="o">=</span><span class="si">$(</span> <span class="nb">sed</span> <span class="s1">'s/ release [0-9].*$//; s/ linux$//i; s/ linux //ig; s/ //g'</span> /etc/redhat-release <span class="si">)</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">file_dist</span><span class="p">,,</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="si">$(</span> <span class="nb">sed</span> <span class="s1">'s/^.* release \([0-9][0-9.]*\).*$/\1/'</span> /etc/redhat-release <span class="si">)</span>
<span class="k">elif</span> <span class="o">[</span> <span class="nt">-e</span> /etc/lsb-release <span class="o">]</span>
<span class="k">then</span>
<span class="c"># Convert to lowercase</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">source</span> /etc/lsb-release <span class="p">;</span> <span class="nb">echo</span> <span class="k">${</span><span class="nv">DISTRIB_ID</span><span class="p">,,</span><span class="k">}</span> <span class="si">)</span><span class="s2">"</span>
<span class="c"># Assuming version is numeric therefore doesn't need lowercasing</span>
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">source</span> /etc/lsb-release <span class="p">;</span> <span class="nb">echo</span> <span class="k">${</span><span class="nv">DISTRIB_RELEASE</span><span class="k">}</span> <span class="si">)</span><span class="s2">"</span>
<span class="k">elif</span> <span class="o">[</span> <span class="nt">-e</span> /etc/debian_version <span class="o">]</span>
<span class="k">then</span>
<span class="c"># Can only be Debian (Ubuntu caught by /etc/lsb-release, above)</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span>debian
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">cat</span> /etc/debian_version<span class="si">)</span><span class="s2">"</span>
<span class="c"># Last ditch attempt - systemd's os-release (might not have full version number0</span>
<span class="k">elif</span> <span class="o">[</span> <span class="nt">-e</span> /etc/os-release <span class="o">]</span> <span class="o">||</span> <span class="o">[</span> <span class="nt">-e</span> /usr/lib/os-release <span class="o">]</span>
<span class="k">then</span>
<span class="c"># /etc/os-release has precedence over /usr/lib/os-release, so source second</span>
<span class="nv">DIST_DISTRIBUTION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">source</span> /usr/lib/os-release<span class="p">;</span> <span class="nb">source</span> /etc/os-release <span class="p">;</span> <span class="nb">echo</span> <span class="nv">$ID</span> <span class="si">)</span><span class="s2">"</span>
<span class="nv">DIST_VERSION</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span> <span class="nb">source</span> /usr/lib/os-release<span class="p">;</span> <span class="nb">source</span> /etc/os-release <span class="p">;</span> <span class="nb">echo</span> <span class="nv">$VERSION_ID</span> <span class="si">)</span><span class="s2">"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Distribution detection failed!"</span> <span class="o">></span>&2
<span class="nb">exit </span>1
<span class="k">fi</span>
<span class="c"># Determine family and major version from distribution and full version</span>
<span class="nv">DIST_FAMILY</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">distribution_family</span><span class="p">[</span><span class="nv">$DIST_DISTRIBUTION</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[[</span> <span class="k">${</span><span class="nv">DIST_DISTRIBUTION</span><span class="k">}</span> <span class="o">=</span> <span class="s2">"ubuntu"</span> <span class="o">]]</span>
<span class="k">then</span>
<span class="c"># Every release of Ubuntu is a major release</span>
<span class="nv">DIST_VERSION_MAJOR</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">DIST_VERSION</span><span class="k">}</span><span class="s2">"</span>
<span class="k">else</span>
<span class="c"># Everything up to first '.' is major version</span>
<span class="nv">DIST_VERSION_MAJOR</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">DIST_VERSION</span><span class="p">%%.*</span><span class="k">}</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="o">[[</span> <span class="nt">-n</span> <span class="k">${</span><span class="nv">DIST_FAMILY</span><span class="k">}</span> <span class="o">]]</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"Missing DIST_FAMILY"</span> <span class="o">></span>&2
<span class="o">[[</span> <span class="nt">-n</span> <span class="k">${</span><span class="nv">DIST_DISTRIBUTION</span><span class="k">}</span> <span class="o">]]</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"Missing DIST_DISTRIBUTION"</span> <span class="o">></span>&2
<span class="o">[[</span> <span class="nt">-n</span> <span class="k">${</span><span class="nv">DIST_VERSION</span><span class="k">}</span> <span class="o">]]</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"Missing DIST_VERSION"</span> <span class="o">></span>&2
<span class="o">[[</span> <span class="nt">-n</span> <span class="k">${</span><span class="nv">DIST_VERSION_MAJOR</span><span class="k">}</span> <span class="o">]]</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"Missing DIST_VERSION_MAJOR"</span> <span class="o">></span>&2
<span class="c"># unset temporary variables to avoid environment pollution when sourced.</span>
<span class="nb">unset </span>distribution_family lsb_dist file_dist
<span class="c"># Keep exports together to easily see what this script intentionally exports.</span>
<span class="nb">export </span>DIST_FAMILY DIST_DISTRIBUTION DIST_VERSION DIST_VERSION_MAJOR
<span class="c">#echo "Detected ${DIST_DISTRIBUTION} (${DIST_FAMILY} family) version ${DIST_VERSION} (major version number ${DIST_VERSION_MAJOR})."</span>
</code></pre></div></div>LaurenceIn order to write some portable health-checking scripts, I needed to reliably detect both distribution (or at least distribution family) and version (e.g. to cope with rpm moving from /bin to /usr/bin in Red Hat 7). I based this on my previous Lua distribution detection script for Lmod. In contrast to the Lmod script, I am not interested in any CPU/architecture detection (at present). I only need this for Red Hat Enterprise Linux family (including CentOS, Scientific Linux and Rocky), Ubuntu and Debian distributions. Adding others would be trivial, it is just a case of finding their lsb_release return values and an appropriate fallback (file) method.