Tags

hardware network technology tplink linux saltstack server bash certificate debian openssl openvpn security ssl vpn web work backup iscsi backuppc automation luks cryptsetup icinga monitoring jekyll wordpress git webhook apt packages development docker singularity video laptop thinkpad lenovo graphics firefox containers ad osx apple authenticator sudo internet plusnet pppoe vigor javascript react react_native testing jest time_machine windows dell xps_9370 dvd broadband plex sky tplink_omada rawlplugs diy gpg playstation ps3 usb ppp bonding lacp nas netgear powershell scripting ssh microsoft teams hp hpe ilo kvm virtualisation vmware munin python dns dnsmasq centos kerberos redhat apc bx1400ui nut readynas rn214 ups github github_actions linting slurm ansible azure azure_devops devops vnc ldap life musing non-technical random raspberry-pi troubleshooting firewall iptables virgin hpc excel cloud docker-registry environment-modules lmod modules easybuild vs-code nextcloud terraform systemd television

Hardware

• 05 Dec 2021 One For All Universal Remote with Virgin Media TV360

  On Thursday we received our new Virgin Media equipment, including a TV360 box. Despite being physically the same box as their old V6 version (existing customers are getting TV360 pushed out as a software update) the TV360 software no longer works with existing Virgin TV remotes.

• 19 Sep 2021 Migrating Raspberry Pi to new SSD

  Since setting up my Raspberry Pi to boot from a USB SSD, I need to repurpose the SSD I used to set this up. I bought a smaller 240GB SSD to replace it and wanted to copy the existing filesystem over to it rather than reinstall.

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 11 Jul 2021 USB booting Raspberry PI 4

  This post describes the process of enabling boot-from-USB and then using a USB disk for booting my Raspberry PI 4. It was made from notes I jotted down when I did this, over a week ago, so I apologise if it seems a bit disjointed or bulleted.

• 08 Apr 2021 Finding SCSI generic device names

  Linux has a number of SCSI drivers, many devices are managed by their own driver as well as the sg generic driver. Some tools need access to both devices provided by the specific driver (e.g. sr for CD/DVD drives) and the generic one - requiring knowledge of which device files correspond to the same physical device to pass through to, e.g., Docker containers.

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

• 23 May 2020 Adding bonded connection to Netgear ReadyNAS 214

  My Netgear ReadyNAS 214 has two Ethernet ports. It is used as a safe place to store my data as well has hosting an iSCSI target that all my backups are done to (with regular off-site copies made to mitigate against failure or physical damage/loss of the NAS) so some extra bandwidth would not hurt and the web user-interface provides the option to bond them. Since my managed switch also supports bonding, I have decided to do this.

• 09 May 2020 Upgrading PS3 hard disk

  As I mentioned yesterday, I was planning to replacing the hard disk drive in my PS3™ with a larger SSD I had laying around my desk.

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

• 08 May 2020 Formatting FAT32 on Windows 10

  In order to backup my PS3™, in preparation for replacing the hard disk drive with a larger SSD I have laying around my desk, I needed a FAT32 formatted USB disk. Windows 10 will only allow disks larger than 32GB to be formatted as exFAT (or NTFS), which the PS3™ does not support.

• 20 Apr 2020 Rawlplug UNO sizes technical data

  Rawlplug’s UNO range are currently my favourite wall plug, reliable and I’m yet to have any problems with the installation of one. I struggle to find this information, particularly supported screw sizes, when I want to refer to it to here it is:

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 18 Apr 2020 The VLANs Strike Back

  Continuing from yesterday’s work on my new network kit, today I’m starting with configuring the new switch.

• 17 Apr 2020 A New Switch

  This post documents the re-introduction of VLANs to our home network, removed in 2018 to solve problems with 4-year old WAP and power-line adapters their replacement means we can move back to more secure and flexible networking set-up.

• 26 Dec 2019 Dell XPS 13 SSD upgrade

  In July I replaced my Lenovo ThinkPad x240 with a Dell XPS 13 9370 I impulse bought from the reduced section of John Lewis. Last week I finally got around to installing the new SSD I bought in a Black Friday deal on-line.

• 18 Aug 2019 PlusNet Fibre (FTTC)

  After much suffering from the incompitence of Virgin Media we have switched to PlusNet for our broadband provider. Thankfully FTTC has arrived in our neighbourhood so we can, at long last, get more than 300kbps estimated speed down the phone line (although we had to have a new phone line installed, as we did not already have one).

• 16 Jun 2019 Graphical Glitches on Lenovo ThinkPad X240 on resume

  For a while (possiblyprobably since I installed it) I've had weird graphical glitches on resuming my laptop from deep sleep. Until now I'd settled for rebooting it to fix it but I went on the hunt for a better solution.

• 26 Aug 2015 TP-Link TL-SG3210 JetStream 8-Port Gigabit L2 Managed Switch

  I have one TP-LINK TL-SG3210 JetStream 8-Port Gigabit L2 Managed Switch with 2 SFP Slots. Initially I had to connect via serial cable as it defaults to a static ip address of 192.168.0.1 which conflicts.

Network

• 04 Jan 2022 Reconfiguring network for satellite switch

  Due to port exhaustion I am moving my “desktop” (quoted as it includes some laptops) systems to a secondary switch. This post describes the switch and new VLAN arrangement for these systems.

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 11 Jul 2021 Finding number of tracked connections by host

  We had a problem with our network a few months ago, in which the router seemed to be running out of capacity to track connections. In order to try and identify the cause, I came up with this snippet that will display the number of currently tracked connections per-host on the Linux router. The first grep limits it to hosts that start ‘192.168.’, which is specific to the part of my network I was interested in:

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 22 Jul 2020 Windows telnet client

  Windows 10 does not come with a telnet client (one of the commands I still have to drop into a WSL session for). Today I rectified that by enabling the optional feature.

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

• 23 May 2020 Adding bonded connection to Netgear ReadyNAS 214

  My Netgear ReadyNAS 214 has two Ethernet ports. It is used as a safe place to store my data as well has hosting an iSCSI target that all my backups are done to (with regular off-site copies made to mitigate against failure or physical damage/loss of the NAS) so some extra bandwidth would not hurt and the web user-interface provides the option to bond them. Since my managed switch also supports bonding, I have decided to do this.

• 22 May 2020 updating broadband PPPoE configuration

  For a very long time I have been running my own Linux-based routers as gateways to the internet. The configuration was setup a very long time ago and it has not been revisited, other than to update logins etc., since. I discovered today that there is now a kernel-mode PPPoE module (since 2.4, from what I can gather) that I have missed the arrival of and a more performant driver for it. As I was monitoring the route due to line-speed problems, I noted that I could see the process pppoe appear in top hovering around the 12% CPU mark and wondered if that was expected or not. Googling this lead to these changes.

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 18 Apr 2020 The VLANs Strike Back

  Continuing from yesterday’s work on my new network kit, today I’m starting with configuring the new switch.

• 17 Apr 2020 A New Switch

  This post documents the re-introduction of VLANs to our home network, removed in 2018 to solve problems with 4-year old WAP and power-line adapters their replacement means we can move back to more secure and flexible networking set-up.

• 21 Nov 2019 Time Machine backup to network share

  I had been using my own external hard disk for backing up my work computer, however (despite being 256GB) Time Machine kept complaining it didn't have enough space. Fortunately I also have access to a 3TB network share at work, so here's how I changed my Mac to backup to there instead.

• 03 Jan 2019 Icinga2

  These notes are a bit rough because they have been copied more-or-less directly from my old wiki that was just for my own consumption. They are from when I finally set-up icinga at home. Note that the configuration has moved on substantially since these notes were written, however as it is in a git repository I have not made any more notes or blog posts about it, to date.

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

• 26 Aug 2015 TP-Link TL-SG3210 JetStream 8-Port Gigabit L2 Managed Switch

  I have one TP-LINK TL-SG3210 JetStream 8-Port Gigabit L2 Managed Switch with 2 SFP Slots. Initially I had to connect via serial cable as it defaults to a static ip address of 192.168.0.1 which conflicts.

Technology

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

• 11 Jul 2021 Finding number of tracked connections by host

  We had a problem with our network a few months ago, in which the router seemed to be running out of capacity to track connections. In order to try and identify the cause, I came up with this snippet that will display the number of currently tracked connections per-host on the Linux router. The first grep limits it to hosts that start ‘192.168.’, which is specific to the part of my network I was interested in:

• 11 Jul 2021 Getting an interactive job with Slurm

  I keep getting this wrong (forgetting the --pty option), so all (Slurm users) recite together now:

• 06 Jul 2021 Troubleshooting full filesystems where df and du disagree

  We have a recurring problem with disk space being exhausted on the root filesystem of a system, the root cause of which is gnome-terminal holding open file-handles to very large deleted temporary files in /tmp. I suspect there is a bug in gnome-terminal not closing the handles to its scrollback buffer (possibly only when set to unlimited scrollback, as some users have).

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 22 Jul 2020 Windows telnet client

  Windows 10 does not come with a telnet client (one of the commands I still have to drop into a WSL session for). Today I rectified that by enabling the optional feature.

• 22 May 2020 updating broadband PPPoE configuration

  For a very long time I have been running my own Linux-based routers as gateways to the internet. The configuration was setup a very long time ago and it has not been revisited, other than to update logins etc., since. I discovered today that there is now a kernel-mode PPPoE module (since 2.4, from what I can gather) that I have missed the arrival of and a more performant driver for it. As I was monitoring the route due to line-speed problems, I noted that I could see the process pppoe appear in top hovering around the 12% CPU mark and wondered if that was expected or not. Googling this lead to these changes.

• 09 May 2020 Upgrading PS3 hard disk

  As I mentioned yesterday, I was planning to replacing the hard disk drive in my PS3™ with a larger SSD I had laying around my desk.

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

• 08 May 2020 Formatting FAT32 on Windows 10

  In order to backup my PS3™, in preparation for replacing the hard disk drive with a larger SSD I have laying around my desk, I needed a FAT32 formatted USB disk. Windows 10 will only allow disks larger than 32GB to be formatted as exFAT (or NTFS), which the PS3™ does not support.

• 22 Apr 2020 Keysigning party post-party signing

  On 1st & 2nd February I was fortunate to attend FOSDEM 2020 in at Université Libre de Bruxelles, Brussels. While I was there I took part in the event’s keysigning party and now need to sign the keys I verified (only 8 days before the deadline of 30th April!).

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 18 Apr 2020 The VLANs Strike Back

  Continuing from yesterday’s work on my new network kit, today I’m starting with configuring the new switch.

• 17 Apr 2020 A New Switch

  This post documents the re-introduction of VLANs to our home network, removed in 2018 to solve problems with 4-year old WAP and power-line adapters their replacement means we can move back to more secure and flexible networking set-up.

• 14 Apr 2020 More forced subtitles in DVDs and BluRays

  Following my last post I’ve been going through my DVD collection and fixing the forced subtitles for the films that have them.

• 07 Apr 2020 (sort-of-but-not-really-)forced subtitles in DVDs and BluRays

  As part of an on-going project to convert my DVD and BluRay collection to a hard-disk based media collection (mainly to save shelf-space and having to keep swapping disks) I’ve encountered a disk that has so-called “forced subtitles” to subtitle some foreign language speech into the viewers native language.

• 16 Sep 2019 Testing React Native with Jest

  I've started a Mobile App at work using React Native. Obviously this needs testing. This post is about getting started on that process.

• 08 Aug 2019 HTML linter

  Trying to debug a missing close paragraph tag somewhere in a 300 line web page, I came across Tidy which helped with it's -e flag to report errors and warnings.

• 04 Jul 2019 Docker images and introspecting them

  I've been playing around with different ways to get graphical applications working in Docker and in the process creates some large images.

• 23 Jun 2019 Docker part 2

  After my previous post, on installing Docker, I needed to setup my first container. I choose to set-up a container for Firefox, first, as that required me to get a graphical application that also needed sound working.

• 14 Jun 2019 Making video indexes with FFmpeg

  I'm working on digitising some of my DVD collection at the moment, which means creating some pretty large files. Trying to view these over the network to discover what they are is pretty tedious (lots of buffering!), so I've been working on dumping an "index" image file that will helpfully give me enough information to determine what it is without actually loading/playing the file.

• 14 Jun 2019 Installing Docker on Debian

  I've been using virtual machines (via VirtualBox) for Linux-based testing and development for sometime but I've been persuaded that it's high-time I joined everyone else and started using containerisation.

• 12 Jun 2019 More blog automation - introducing Jekyll environments

  Further to my earlier post on automating deployment of my blog I found a small flaw with my method and have decided to fix it.

• 04 Jun 2019 Automating deployment of Blog with Gitea and Webhook

  Using webhook to automate deployment of this Blog via Gitea on push.

• 03 Jun 2019 New Blog (...new me?)

  So, I've started a new blog. It think this is my 3^rd or 4^th Blog, although it's the first new one in over 13 years, since I switched to Wordpress in April 2006.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

• 26 Aug 2015 TP-Link TL-SG3210 JetStream 8-Port Gigabit L2 Managed Switch

  I have one TP-LINK TL-SG3210 JetStream 8-Port Gigabit L2 Managed Switch with 2 SFP Slots. Initially I had to connect via serial cable as it defaults to a static ip address of 192.168.0.1 which conflicts.

Tplink

• 04 Jan 2022 Reconfiguring network for satellite switch

  Due to port exhaustion I am moving my “desktop” (quoted as it includes some laptops) systems to a secondary switch. This post describes the switch and new VLAN arrangement for these systems.

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 18 Apr 2020 The VLANs Strike Back

  Continuing from yesterday’s work on my new network kit, today I’m starting with configuring the new switch.

• 17 Apr 2020 A New Switch

  This post documents the re-introduction of VLANs to our home network, removed in 2018 to solve problems with 4-year old WAP and power-line adapters their replacement means we can move back to more secure and flexible networking set-up.

• 26 Aug 2015 TP-Link TL-SG3210 JetStream 8-Port Gigabit L2 Managed Switch

  I have one TP-LINK TL-SG3210 JetStream 8-Port Gigabit L2 Managed Switch with 2 SFP Slots. Initially I had to connect via serial cable as it defaults to a static ip address of 192.168.0.1 which conflicts.

Linux

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 19 Sep 2021 Migrating Raspberry Pi to new SSD

  Since setting up my Raspberry Pi to boot from a USB SSD, I need to repurpose the SSD I used to set this up. I bought a smaller 240GB SSD to replace it and wanted to copy the existing filesystem over to it rather than reinstall.

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

• 21 Jul 2021 Docker on CentOS 7

  Essentially a “quickstart” for Docker on CentOS 7.

• 19 Jul 2021 Searching Windows AD by username and email with ldapsearch

  Further to my post on querying AD with LDAP, to search for users by Windows username or email address one can use these commands:

• 11 Jul 2021 Finding number of tracked connections by host

  We had a problem with our network a few months ago, in which the router seemed to be running out of capacity to track connections. In order to try and identify the cause, I came up with this snippet that will display the number of currently tracked connections per-host on the Linux router. The first grep limits it to hosts that start ‘192.168.’, which is specific to the part of my network I was interested in:

• 11 Jul 2021 USB booting Raspberry PI 4

  This post describes the process of enabling boot-from-USB and then using a USB disk for booting my Raspberry PI 4. It was made from notes I jotted down when I did this, over a week ago, so I apologise if it seems a bit disjointed or bulleted.

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

• 06 Jul 2021 Troubleshooting full filesystems where df and du disagree

  We have a recurring problem with disk space being exhausted on the root filesystem of a system, the root cause of which is gnome-terminal holding open file-handles to very large deleted temporary files in /tmp. I suspect there is a bug in gnome-terminal not closing the handles to its scrollback buffer (possibly only when set to unlimited scrollback, as some users have).

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 10 May 2021 Query AD with LDAP

  Because I keep ending up looking at my old scripts to refresh myself on how to do this by hand, here’s the recipe for querying AD with ldapsearch.

• 28 Apr 2021 Decrypt VNC password

  Decrypt a VNC password file in one-line with nothing but openssl:

• 12 Apr 2021 Quick and dirty password generator

  Quick and dirty password generator

• 08 Apr 2021 Finding SCSI generic device names

  Linux has a number of SCSI drivers, many devices are managed by their own driver as well as the sg generic driver. Some tools need access to both devices provided by the specific driver (e.g. sr for CD/DVD drives) and the generic one - requiring knowledge of which device files correspond to the same physical device to pass through to, e.g., Docker containers.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

• 29 Oct 2020 Transfer output files with SCP

  Related to my previous post on monitoring output, I also needed to provide a script to transfer the output files from the scripts to a remote server. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Monitoring script outputs for updates

  Related to my previous post on launching 16 scripts in tmux, I also needed to provide a script to monitor the output files from some scripts and alert if they are not updated. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Launching scripts in tmux

  I was asked to help write a script to automate launching 16 scripts in a tmux session.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 22 Jul 2020 Windows telnet client

  Windows 10 does not come with a telnet client (one of the commands I still have to drop into a WSL session for). Today I rectified that by enabling the optional feature.

• 19 Jun 2020 First Python pull request

  Yesterday I opened my first issue against Python, and enhancement for mailbox, an old part of the standard library, to support Path-like objects, a very new part of the standard library.

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 06 Jun 2020 Building Debian VMs with debootstrap

  Following the awful time I had setting up a working preseed configuration for Debian installer I am exploring the alternative method of pre-building the disk image using debootstrap.

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

• 28 May 2020 Debian system updates

  I have automated monitoring (via Icinga2) of the update status of my various servers, all of which currently run Debian. To date I’ve been using cron to run a daily apt-get update, to my mind the obvious solution to “I need it to update daily”, which updates the local package cache and then the monitoring picks up whether there’s anything to update. While investigating a problem with another cron-job yesterday, I noticed messages from apt-daily and apt-daily-upgrade services and a quick Google revealed that there is a Debian way to do this.

• 22 May 2020 updating broadband PPPoE configuration

  For a very long time I have been running my own Linux-based routers as gateways to the internet. The configuration was setup a very long time ago and it has not been revisited, other than to update logins etc., since. I discovered today that there is now a kernel-mode PPPoE module (since 2.4, from what I can gather) that I have missed the arrival of and a more performant driver for it. As I was monitoring the route due to line-speed problems, I noted that I could see the process pppoe appear in top hovering around the 12% CPU mark and wondered if that was expected or not. Googling this lead to these changes.

• 21 May 2020 Dockerised salt-ssh

  For some systems that I have user accounts on but do not administer, I use Salt SSH (agent-less salt minion) to manage the common user-local files that are managed on the systems I do administer on these systems. I have previously made notes about it.

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

• 22 Apr 2020 Keysigning party post-party signing

  On 1st & 2nd February I was fortunate to attend FOSDEM 2020 in at Université Libre de Bruxelles, Brussels. While I was there I took part in the event’s keysigning party and now need to sign the keys I verified (only 8 days before the deadline of 30th April!).

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 18 Apr 2020 The VLANs Strike Back

  Continuing from yesterday’s work on my new network kit, today I’m starting with configuring the new switch.

• 24 Dec 2019 Backups with BackupPC and offsite DR backup - automation

  Shortly before this time last year I revolutionised my backup infrastructure. This year I finally got around to scripting updating the off-site version.

• 13 Sep 2019 Upgrading Debian systems

  Based on notes from my old wiki, refreshed as I remotely upgraded our home router from Debian 9 (stretch) to Debian 10 (buster).

• 18 Aug 2019 PlusNet Fibre (FTTC)

  After much suffering from the incompitence of Virgin Media we have switched to PlusNet for our broadband provider. Thankfully FTTC has arrived in our neighbourhood so we can, at long last, get more than 300kbps estimated speed down the phone line (although we had to have a new phone line installed, as we did not already have one).

• 08 Jul 2019 Secure sudo with Google Authenticator

  A little while ago, I "secured" root access on my GNU/Linux¹ systems by stopping using the root password and migrating to sudo. I also setup Google Authenticator for it.

• 04 Jul 2019 Docker images and introspecting them

  I've been playing around with different ways to get graphical applications working in Docker and in the process creates some large images.

• 23 Jun 2019 Docker part 2

  After my previous post, on installing Docker, I needed to setup my first container. I choose to set-up a container for Firefox, first, as that required me to get a graphical application that also needed sound working.

• 14 Jun 2019 Making video indexes with FFmpeg

  I'm working on digitising some of my DVD collection at the moment, which means creating some pretty large files. Trying to view these over the network to discover what they are is pretty tedious (lots of buffering!), so I've been working on dumping an "index" image file that will helpfully give me enough information to determine what it is without actually loading/playing the file.

• 14 Jun 2019 Installing Docker on Debian

  I've been using virtual machines (via VirtualBox) for Linux-based testing and development for sometime but I've been persuaded that it's high-time I joined everyone else and started using containerisation.

• 12 Jun 2019 More blog automation - introducing Jekyll environments

  Further to my earlier post on automating deployment of my blog I found a small flaw with my method and have decided to fix it.

• 12 Jun 2019 Unmarking Debian packages as manually installed

  I sometimes manually install a Debian package, whilst trying to figure out dependencies for something else, and then discover I should have installed a different "parent" package that will pull it in.

• 04 Jun 2019 Automating deployment of Blog with Gitea and Webhook

  Using webhook to automate deployment of this Blog via Gitea on push.

• 03 Jan 2019 Icinga2

  These notes are a bit rough because they have been copied more-or-less directly from my old wiki that was just for my own consumption. They are from when I finally set-up icinga at home. Note that the configuration has moved on substantially since these notes were written, however as it is in a git repository I have not made any more notes or blog posts about it, to date.

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

• 27 Jul 2018 Salt SSH

  salt-ssh provides a way to run salt remotely without it being installed on the destination system. ​ This means, for example, that I can use it to manage my user’s dotfiles on the BlueBEAR HPC cluster.

• 25 Jun 2018 Masterless Salt Minion

  A master-less Salt minion can be used to manage a standalone machine (e.g. my work Linux desktop) or bootstrap any master-controlled minion (or even the master itself) as the salt states will take over management of the minion’s configuration and reconfigure it appropriately on first run.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

• 27 Aug 2015 Salt

  Salt is a remote execution and configuration management tool that I have been using to manage the many Linux servers and desktops I have. Its state system also replaces some of my notes as a self-documenting machine-readable description of how each item is configured.

Saltstack

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 21 May 2020 Dockerised salt-ssh

  For some systems that I have user accounts on but do not administer, I use Salt SSH (agent-less salt minion) to manage the common user-local files that are managed on the systems I do administer on these systems. I have previously made notes about it.

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 18 Apr 2020 The VLANs Strike Back

  Continuing from yesterday’s work on my new network kit, today I’m starting with configuring the new switch.

• 27 Jul 2018 Salt SSH

  salt-ssh provides a way to run salt remotely without it being installed on the destination system. ​ This means, for example, that I can use it to manage my user’s dotfiles on the BlueBEAR HPC cluster.

• 25 Jun 2018 Masterless Salt Minion

  A master-less Salt minion can be used to manage a standalone machine (e.g. my work Linux desktop) or bootstrap any master-controlled minion (or even the master itself) as the salt states will take over management of the minion’s configuration and reconfigure it appropriately on first run.

• 27 Aug 2015 Salt

  Salt is a remote execution and configuration management tool that I have been using to manage the many Linux servers and desktops I have. Its state system also replaces some of my notes as a self-documenting machine-readable description of how each item is configured.

Server

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 19 Sep 2021 Migrating Raspberry Pi to new SSD

  Since setting up my Raspberry Pi to boot from a USB SSD, I need to repurpose the SSD I used to set this up. I bought a smaller 240GB SSD to replace it and wanted to copy the existing filesystem over to it rather than reinstall.

• 25 Aug 2021 Checking NextCloud version

  One of my current issues is that I do not notice my NextCloud install requires updating until it is no longer supported and the clients stops working with it. I already have icinga setup as my monitoring solution, so it has been on my to-do list for a while to create a plugin to check the version of my server against the latest release.

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

• 19 Jul 2021 Working with virtual machines using the Azure command line client

  Over the past 8 months I have gethered quite a few useful snippets for interacting with Azure VMs from the az command-line client. This is a summary of them.

• 11 Jul 2021 Finding number of tracked connections by host

  We had a problem with our network a few months ago, in which the router seemed to be running out of capacity to track connections. In order to try and identify the cause, I came up with this snippet that will display the number of currently tracked connections per-host on the Linux router. The first grep limits it to hosts that start ‘192.168.’, which is specific to the part of my network I was interested in:

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

• 06 Jul 2021 Troubleshooting full filesystems where df and du disagree

  We have a recurring problem with disk space being exhausted on the root filesystem of a system, the root cause of which is gnome-terminal holding open file-handles to very large deleted temporary files in /tmp. I suspect there is a bug in gnome-terminal not closing the handles to its scrollback buffer (possibly only when set to unlimited scrollback, as some users have).

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 28 Apr 2021 Decrypt VNC password

  Decrypt a VNC password file in one-line with nothing but openssl:

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 22 Jul 2020 Windows telnet client

  Windows 10 does not come with a telnet client (one of the commands I still have to drop into a WSL session for). Today I rectified that by enabling the optional feature.

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 06 Jun 2020 Building Debian VMs with debootstrap

  Following the awful time I had setting up a working preseed configuration for Debian installer I am exploring the alternative method of pre-building the disk image using debootstrap.

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

• 24 Dec 2019 Backups with BackupPC and offsite DR backup - automation

  Shortly before this time last year I revolutionised my backup infrastructure. This year I finally got around to scripting updating the off-site version.

• 03 Jan 2019 Icinga2

  These notes are a bit rough because they have been copied more-or-less directly from my old wiki that was just for my own consumption. They are from when I finally set-up icinga at home. Note that the configuration has moved on substantially since these notes were written, however as it is in a git repository I have not made any more notes or blog posts about it, to date.

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

• 27 Aug 2015 Salt

  Salt is a remote execution and configuration management tool that I have been using to manage the many Linux servers and desktops I have. Its state system also replaces some of my notes as a self-documenting machine-readable description of how each item is configured.

Bash

• 25 Aug 2021 Checking NextCloud version

  One of my current issues is that I do not notice my NextCloud install requires updating until it is no longer supported and the clients stops working with it. I already have icinga setup as my monitoring solution, so it has been on my to-do list for a while to create a plugin to check the version of my server against the latest release.

• 19 Jul 2021 Searching Windows AD by username and email with ldapsearch

  Further to my post on querying AD with LDAP, to search for users by Windows username or email address one can use these commands:

• 11 Jul 2021 Finding number of tracked connections by host

  We had a problem with our network a few months ago, in which the router seemed to be running out of capacity to track connections. In order to try and identify the cause, I came up with this snippet that will display the number of currently tracked connections per-host on the Linux router. The first grep limits it to hosts that start ‘192.168.’, which is specific to the part of my network I was interested in:

• 11 Jul 2021 Getting an interactive job with Slurm

  I keep getting this wrong (forgetting the --pty option), so all (Slurm users) recite together now:

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 10 May 2021 Query AD with LDAP

  Because I keep ending up looking at my old scripts to refresh myself on how to do this by hand, here’s the recipe for querying AD with ldapsearch.

• 28 Apr 2021 Decrypt VNC password

  Decrypt a VNC password file in one-line with nothing but openssl:

• 12 Apr 2021 Quick and dirty password generator

  Quick and dirty password generator

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

• 29 Oct 2020 Launching scripts in tmux

  I was asked to help write a script to automate launching 16 scripts in a tmux session.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

Certificate

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

Debian

• 19 Sep 2021 Migrating Raspberry Pi to new SSD

  Since setting up my Raspberry Pi to boot from a USB SSD, I need to repurpose the SSD I used to set this up. I bought a smaller 240GB SSD to replace it and wanted to copy the existing filesystem over to it rather than reinstall.

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

• 11 Jul 2021 USB booting Raspberry PI 4

  This post describes the process of enabling boot-from-USB and then using a USB disk for booting my Raspberry PI 4. It was made from notes I jotted down when I did this, over a week ago, so I apologise if it seems a bit disjointed or bulleted.

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 06 Jun 2020 Building Debian VMs with debootstrap

  Following the awful time I had setting up a working preseed configuration for Debian installer I am exploring the alternative method of pre-building the disk image using debootstrap.

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

• 28 May 2020 Debian system updates

  I have automated monitoring (via Icinga2) of the update status of my various servers, all of which currently run Debian. To date I’ve been using cron to run a daily apt-get update, to my mind the obvious solution to “I need it to update daily”, which updates the local package cache and then the monitoring picks up whether there’s anything to update. While investigating a problem with another cron-job yesterday, I noticed messages from apt-daily and apt-daily-upgrade services and a quick Google revealed that there is a Debian way to do this.

• 22 Apr 2020 Keysigning party post-party signing

  On 1st & 2nd February I was fortunate to attend FOSDEM 2020 in at Université Libre de Bruxelles, Brussels. While I was there I took part in the event’s keysigning party and now need to sign the keys I verified (only 8 days before the deadline of 30th April!).

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 18 Apr 2020 The VLANs Strike Back

  Continuing from yesterday’s work on my new network kit, today I’m starting with configuring the new switch.

• 24 Dec 2019 Backups with BackupPC and offsite DR backup - automation

  Shortly before this time last year I revolutionised my backup infrastructure. This year I finally got around to scripting updating the off-site version.

• 13 Sep 2019 Upgrading Debian systems

  Based on notes from my old wiki, refreshed as I remotely upgraded our home router from Debian 9 (stretch) to Debian 10 (buster).

• 18 Aug 2019 PlusNet Fibre (FTTC)

  After much suffering from the incompitence of Virgin Media we have switched to PlusNet for our broadband provider. Thankfully FTTC has arrived in our neighbourhood so we can, at long last, get more than 300kbps estimated speed down the phone line (although we had to have a new phone line installed, as we did not already have one).

• 04 Jul 2019 Docker images and introspecting them

  I've been playing around with different ways to get graphical applications working in Docker and in the process creates some large images.

• 23 Jun 2019 Docker part 2

  After my previous post, on installing Docker, I needed to setup my first container. I choose to set-up a container for Firefox, first, as that required me to get a graphical application that also needed sound working.

• 14 Jun 2019 Installing Docker on Debian

  I've been using virtual machines (via VirtualBox) for Linux-based testing and development for sometime but I've been persuaded that it's high-time I joined everyone else and started using containerisation.

• 12 Jun 2019 More blog automation - introducing Jekyll environments

  Further to my earlier post on automating deployment of my blog I found a small flaw with my method and have decided to fix it.

• 12 Jun 2019 Unmarking Debian packages as manually installed

  I sometimes manually install a Debian package, whilst trying to figure out dependencies for something else, and then discover I should have installed a different "parent" package that will pull it in.

• 04 Jun 2019 Automating deployment of Blog with Gitea and Webhook

  Using webhook to automate deployment of this Blog via Gitea on push.

• 03 Jan 2019 Icinga2

  These notes are a bit rough because they have been copied more-or-less directly from my old wiki that was just for my own consumption. They are from when I finally set-up icinga at home. Note that the configuration has moved on substantially since these notes were written, however as it is in a git repository I have not made any more notes or blog posts about it, to date.

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

Openssl

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 28 Apr 2021 Decrypt VNC password

  Decrypt a VNC password file in one-line with nothing but openssl:

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

Openvpn

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

Security

• 04 Jan 2022 Reconfiguring network for satellite switch

  Due to port exhaustion I am moving my “desktop” (quoted as it includes some laptops) systems to a secondary switch. This post describes the switch and new VLAN arrangement for these systems.

• 19 Jul 2021 Searching Windows AD by username and email with ldapsearch

  Further to my post on querying AD with LDAP, to search for users by Windows username or email address one can use these commands:

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 10 May 2021 Query AD with LDAP

  Because I keep ending up looking at my old scripts to refresh myself on how to do this by hand, here’s the recipe for querying AD with ldapsearch.

• 28 Apr 2021 Decrypt VNC password

  Decrypt a VNC password file in one-line with nothing but openssl:

• 12 Apr 2021 Quick and dirty password generator

  Quick and dirty password generator

• 12 Apr 2021 Change Windows user password from the command line

  Quick and dirty, how to change the windows password from the command line in an insecure fashion (exposes password on the command-line).

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 22 Apr 2020 Keysigning party post-party signing

  On 1st & 2nd February I was fortunate to attend FOSDEM 2020 in at Université Libre de Bruxelles, Brussels. While I was there I took part in the event’s keysigning party and now need to sign the keys I verified (only 8 days before the deadline of 30th April!).

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 18 Apr 2020 The VLANs Strike Back

  Continuing from yesterday’s work on my new network kit, today I’m starting with configuring the new switch.

• 17 Apr 2020 A New Switch

  This post documents the re-introduction of VLANs to our home network, removed in 2018 to solve problems with 4-year old WAP and power-line adapters their replacement means we can move back to more secure and flexible networking set-up.

• 08 Jul 2019 Secure sudo with Google Authenticator

  A little while ago, I "secured" root access on my GNU/Linux¹ systems by stopping using the root password and migrating to sudo. I also setup Google Authenticator for it.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

Ssl

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

Vpn

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

• 05 Jul 2021 Migrate OpenVPN endpoint to cloud

  For many years I have run an OpenVPN server on my home router, which provides access to my home network remotely when I am away from home and a private tunnel to my cloud server through which most monitoring and all command-and-control (via SaltStack at the time of writing) take place.

• 16 Nov 2015 OpenSSL CA setup

  This content is dumped, almost directly, from my old wiki and includes notes about 3 different iterations of orchestrating my own CA. From newest to oldest. It is a sister page to my notes on using an OpenSSL Certificate Authority.

• 13 Nov 2015 OpenSSL certificate authority

  Using OpenSSL as a Certificate Authority - ported from my old wiki. See also my OpenSSL certficiates in a nutshell post for client-level certficiate handling.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

• 28 Aug 2015 OpenVPN

  OpenVPN setup notes - ported from old wiki (unedited).

Web

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 25 Aug 2021 Checking NextCloud version

  One of my current issues is that I do not notice my NextCloud install requires updating until it is no longer supported and the clients stops working with it. I already have icinga setup as my monitoring solution, so it has been on my to-do list for a while to create a plugin to check the version of my server against the latest release.

• 08 Aug 2019 HTML linter

  Trying to debug a missing close paragraph tag somewhere in a 300 line web page, I came across Tidy which helped with it's -e flag to report errors and warnings.

• 03 Jan 2019 Icinga2

  These notes are a bit rough because they have been copied more-or-less directly from my old wiki that was just for my own consumption. They are from when I finally set-up icinga at home. Note that the configuration has moved on substantially since these notes were written, however as it is in a git repository I have not made any more notes or blog posts about it, to date.

• 13 Nov 2015 OpenSSL certificates in a nutshell

  These notes are a little rough-and-ready, copied more or less directly from my old wiki which was only intended for my consumption.

Work

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

• 28 Jul 2021 Multi-stage Python builds with Docker on CentOS

  Following on from my getting started with Docker on CentOS post, I found that the version of Docker is very old (1.13) and, amongst other things, does not support multi-stage builds which will allow use to cut down our image size (useful, as the Azure registry charges for data transfer).

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

• 21 Jul 2021 Docker on CentOS 7

  Essentially a “quickstart” for Docker on CentOS 7.

• 20 Jul 2021 Updating text file on Azure Windows VM from local text file with PowerShell

  In order for NFS identity mapping to work accross both platforms (Linux and Windows), pending either AD integration or fixing Windows NFS client integration with OpenLDAP server (neither of which is currently in place), we are currently maintaining a passwd file for Windows that maps the Windows identity to the corresponding UID/GID.

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

• 19 Jul 2021 Working with virtual machines using the Azure command line client

  Over the past 8 months I have gethered quite a few useful snippets for interacting with Azure VMs from the az command-line client. This is a summary of them.

• 19 Jul 2021 Searching Windows AD by username and email with ldapsearch

  Further to my post on querying AD with LDAP, to search for users by Windows username or email address one can use these commands:

• 14 Jul 2021 Creating Excel spreadsheets with Python

  Every month I produce usage spreadsheets for the HPC service I run. This involved much copying and pasting of data, which I already have a script to export in CSV format, into a bunch of spreadsheets for different people. In total, it takes about 7 hours (essentially a full working day of doing nothing else) to do - this post is taking some of that 7 hours to get the existing script to generate the required spreadsheets directly. This is definitely going to be a case of getting back more time than I invest.

• 11 Jul 2021 Getting an interactive job with Slurm

  I keep getting this wrong (forgetting the --pty option), so all (Slurm users) recite together now:

• 06 Jul 2021 Troubleshooting full filesystems where df and du disagree

  We have a recurring problem with disk space being exhausted on the root filesystem of a system, the root cause of which is gnome-terminal holding open file-handles to very large deleted temporary files in /tmp. I suspect there is a bug in gnome-terminal not closing the handles to its scrollback buffer (possibly only when set to unlimited scrollback, as some users have).

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

• 29 Oct 2020 Transfer output files with SCP

  Related to my previous post on monitoring output, I also needed to provide a script to transfer the output files from the scripts to a remote server. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Monitoring script outputs for updates

  Related to my previous post on launching 16 scripts in tmux, I also needed to provide a script to monitor the output files from some scripts and alert if they are not updated. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Launching scripts in tmux

  I was asked to help write a script to automate launching 16 scripts in a tmux session.

• 28 May 2020 Teams backgrounds

  To use your own custom backgrounds, drop the background file(s) in $Env:APPDATA\Microsoft\Teams\Backgrounds\Uploads (PowerShell, %APPDATA%\Microsoft\Teams\Backgrounds\Uploads in cmd). They must be PNG files and will just show up in the list of backgrounds to choose from in the user interface.

• 21 May 2020 Dockerised salt-ssh

  For some systems that I have user accounts on but do not administer, I use Salt SSH (agent-less salt minion) to manage the common user-local files that are managed on the systems I do administer on these systems. I have previously made notes about it.

• 22 Apr 2020 Keysigning party post-party signing

  On 1st & 2nd February I was fortunate to attend FOSDEM 2020 in at Université Libre de Bruxelles, Brussels. While I was there I took part in the event’s keysigning party and now need to sign the keys I verified (only 8 days before the deadline of 30th April!).

• 21 Nov 2019 Time Machine backup to network share

  I had been using my own external hard disk for backing up my work computer, however (despite being 256GB) Time Machine kept complaining it didn't have enough space. Fortunately I also have access to a 3TB network share at work, so here's how I changed my Mac to backup to there instead.

• 16 Sep 2019 Testing React Native with Jest

  I've started a Mobile App at work using React Native. Obviously this needs testing. This post is about getting started on that process.

• 08 Aug 2019 HTML linter

  Trying to debug a missing close paragraph tag somewhere in a 300 line web page, I came across Tidy which helped with it's -e flag to report errors and warnings.

• 08 Jul 2019 Changing Active Directory password on non-bound Mac

  To change an Microsoft Active Directory password on your (non-bound) Mac:

  $ kinit working_username@AD.DOMAIN
  $ kpasswd username_to_change@AD.DOMAIN

• 27 Jul 2018 Salt SSH

  salt-ssh provides a way to run salt remotely without it being installed on the destination system. ​ This means, for example, that I can use it to manage my user’s dotfiles on the BlueBEAR HPC cluster.

• 25 Jun 2018 Masterless Salt Minion

  A master-less Salt minion can be used to manage a standalone machine (e.g. my work Linux desktop) or bootstrap any master-controlled minion (or even the master itself) as the salt states will take over management of the minion’s configuration and reconfigure it appropriately on first run.

Backup

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 08 May 2020 Formatting FAT32 on Windows 10

  In order to backup my PS3™, in preparation for replacing the hard disk drive with a larger SSD I have laying around my desk, I needed a FAT32 formatted USB disk. Windows 10 will only allow disks larger than 32GB to be formatted as exFAT (or NTFS), which the PS3™ does not support.

• 24 Dec 2019 Backups with BackupPC and offsite DR backup - automation

  Shortly before this time last year I revolutionised my backup infrastructure. This year I finally got around to scripting updating the off-site version.

• 21 Nov 2019 Time Machine backup to network share

  I had been using my own external hard disk for backing up my work computer, however (despite being 256GB) Time Machine kept complaining it didn't have enough space. Fortunately I also have access to a 3TB network share at work, so here's how I changed my Mac to backup to there instead.

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

Iscsi

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

Backuppc

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 24 Dec 2019 Backups with BackupPC and offsite DR backup - automation

  Shortly before this time last year I revolutionised my backup infrastructure. This year I finally got around to scripting updating the off-site version.

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

Automation

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 25 Aug 2021 Checking NextCloud version

  One of my current issues is that I do not notice my NextCloud install requires updating until it is no longer supported and the clients stops working with it. I already have icinga setup as my monitoring solution, so it has been on my to-do list for a while to create a plugin to check the version of my server against the latest release.

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

• 28 Jul 2021 Multi-stage Python builds with Docker on CentOS

  Following on from my getting started with Docker on CentOS post, I found that the version of Docker is very old (1.13) and, amongst other things, does not support multi-stage builds which will allow use to cut down our image size (useful, as the Azure registry charges for data transfer).

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

• 19 Jul 2021 Working with virtual machines using the Azure command line client

  Over the past 8 months I have gethered quite a few useful snippets for interacting with Azure VMs from the az command-line client. This is a summary of them.

• 14 Jul 2021 Creating Excel spreadsheets with Python

  Every month I produce usage spreadsheets for the HPC service I run. This involved much copying and pasting of data, which I already have a script to export in CSV format, into a bunch of spreadsheets for different people. In total, it takes about 7 hours (essentially a full working day of doing nothing else) to do - this post is taking some of that 7 hours to get the existing script to generate the required spreadsheets directly. This is definitely going to be a case of getting back more time than I invest.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 29 Oct 2020 Transfer output files with SCP

  Related to my previous post on monitoring output, I also needed to provide a script to transfer the output files from the scripts to a remote server. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Monitoring script outputs for updates

  Related to my previous post on launching 16 scripts in tmux, I also needed to provide a script to monitor the output files from some scripts and alert if they are not updated. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Launching scripts in tmux

  I was asked to help write a script to automate launching 16 scripts in a tmux session.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 28 May 2020 Debian system updates

  I have automated monitoring (via Icinga2) of the update status of my various servers, all of which currently run Debian. To date I’ve been using cron to run a daily apt-get update, to my mind the obvious solution to “I need it to update daily”, which updates the local package cache and then the monitoring picks up whether there’s anything to update. While investigating a problem with another cron-job yesterday, I noticed messages from apt-daily and apt-daily-upgrade services and a quick Google revealed that there is a Debian way to do this.

• 24 Dec 2019 Backups with BackupPC and offsite DR backup - automation

  Shortly before this time last year I revolutionised my backup infrastructure. This year I finally got around to scripting updating the off-site version.

• 12 Jun 2019 More blog automation - introducing Jekyll environments

  Further to my earlier post on automating deployment of my blog I found a small flaw with my method and have decided to fix it.

• 04 Jun 2019 Automating deployment of Blog with Gitea and Webhook

  Using webhook to automate deployment of this Blog via Gitea on push.

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

Luks

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

Cryptsetup

• 16 Dec 2018 Backups with at-rest encryption, BackupPC, iSCSI and offsite DR backup

  For many, many years (at least since 2008) I’ve been using BackupPC to provide backups of my machines, at home and in the cloud. I recently replaces my NAS with one that has a larger capacity (as part of a project to turn my DVD collection into something more convenient to browse and watch) and that NAS has iSCSI support so I moved my backup solution from a USB attached disk to an iSCSI target and introduced off-site replication of the backup pool for DR purposes.

Icinga

• 19 Sep 2021 Migrating Raspberry Pi to new SSD

  Since setting up my Raspberry Pi to boot from a USB SSD, I need to repurpose the SSD I used to set this up. I bought a smaller 240GB SSD to replace it and wanted to copy the existing filesystem over to it rather than reinstall.

• 25 Aug 2021 Checking NextCloud version

  One of my current issues is that I do not notice my NextCloud install requires updating until it is no longer supported and the clients stops working with it. I already have icinga setup as my monitoring solution, so it has been on my to-do list for a while to create a plugin to check the version of my server against the latest release.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

• 03 Jan 2019 Icinga2

  These notes are a bit rough because they have been copied more-or-less directly from my old wiki that was just for my own consumption. They are from when I finally set-up icinga at home. Note that the configuration has moved on substantially since these notes were written, however as it is in a git repository I have not made any more notes or blog posts about it, to date.

Monitoring

• 19 Sep 2021 Migrating Raspberry Pi to new SSD

  Since setting up my Raspberry Pi to boot from a USB SSD, I need to repurpose the SSD I used to set this up. I bought a smaller 240GB SSD to replace it and wanted to copy the existing filesystem over to it rather than reinstall.

• 25 Aug 2021 Checking NextCloud version

  One of my current issues is that I do not notice my NextCloud install requires updating until it is no longer supported and the clients stops working with it. I already have icinga setup as my monitoring solution, so it has been on my to-do list for a while to create a plugin to check the version of my server against the latest release.

• 11 Jul 2021 USB booting Raspberry PI 4

  This post describes the process of enabling boot-from-USB and then using a USB disk for booting my Raspberry PI 4. It was made from notes I jotted down when I did this, over a week ago, so I apologise if it seems a bit disjointed or bulleted.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

• 03 Jan 2019 Icinga2

  These notes are a bit rough because they have been copied more-or-less directly from my old wiki that was just for my own consumption. They are from when I finally set-up icinga at home. Note that the configuration has moved on substantially since these notes were written, however as it is in a git repository I have not made any more notes or blog posts about it, to date.

Jekyll

• 12 Jun 2019 More blog automation - introducing Jekyll environments

  Further to my earlier post on automating deployment of my blog I found a small flaw with my method and have decided to fix it.

• 04 Jun 2019 Automating deployment of Blog with Gitea and Webhook

  Using webhook to automate deployment of this Blog via Gitea on push.

• 03 Jun 2019 New Blog (...new me?)

  So, I've started a new blog. It think this is my 3^rd or 4^th Blog, although it's the first new one in over 13 years, since I switched to Wordpress in April 2006.

Wordpress

• 03 Jun 2019 New Blog (...new me?)

  So, I've started a new blog. It think this is my 3^rd or 4^th Blog, although it's the first new one in over 13 years, since I switched to Wordpress in April 2006.

Git

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

• 19 Jun 2020 First Python pull request

  Yesterday I opened my first issue against Python, and enhancement for mailbox, an old part of the standard library, to support Path-like objects, a very new part of the standard library.

• 04 Jun 2019 Automating deployment of Blog with Gitea and Webhook

  Using webhook to automate deployment of this Blog via Gitea on push.

Webhook

• 04 Jun 2019 Automating deployment of Blog with Gitea and Webhook

  Using webhook to automate deployment of this Blog via Gitea on push.

Apt

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 28 May 2020 Debian system updates

  I have automated monitoring (via Icinga2) of the update status of my various servers, all of which currently run Debian. To date I’ve been using cron to run a daily apt-get update, to my mind the obvious solution to “I need it to update daily”, which updates the local package cache and then the monitoring picks up whether there’s anything to update. While investigating a problem with another cron-job yesterday, I noticed messages from apt-daily and apt-daily-upgrade services and a quick Google revealed that there is a Debian way to do this.

• 13 Sep 2019 Upgrading Debian systems

  Based on notes from my old wiki, refreshed as I remotely upgraded our home router from Debian 9 (stretch) to Debian 10 (buster).

• 12 Jun 2019 Unmarking Debian packages as manually installed

  I sometimes manually install a Debian package, whilst trying to figure out dependencies for something else, and then discover I should have installed a different "parent" package that will pull it in.

Packages

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 13 Sep 2019 Upgrading Debian systems

  Based on notes from my old wiki, refreshed as I remotely upgraded our home router from Debian 9 (stretch) to Debian 10 (buster).

• 12 Jun 2019 Unmarking Debian packages as manually installed

  I sometimes manually install a Debian package, whilst trying to figure out dependencies for something else, and then discover I should have installed a different "parent" package that will pull it in.

Development

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 29 Jul 2021 VS Code spell checking

  I switched from Sublime Text, which I have used for many years, to Visual Studio Code (or ‘VS Code’ for short) mainly due to being unhappy with Sublime Text’s new licencing model (which has gone from licences for a particular major version to a subscription model, with a 3 year subscription period). One thing I have missed is Sublime Text’s built-in spell checker but there is a VS Code marketplace extension for that.

• 28 Jul 2021 Multi-stage Python builds with Docker on CentOS

  Following on from my getting started with Docker on CentOS post, I found that the version of Docker is very old (1.13) and, amongst other things, does not support multi-stage builds which will allow use to cut down our image size (useful, as the Azure registry charges for data transfer).

• 14 Jul 2021 Creating Excel spreadsheets with Python

  Every month I produce usage spreadsheets for the HPC service I run. This involved much copying and pasting of data, which I already have a script to export in CSV format, into a bunch of spreadsheets for different people. In total, it takes about 7 hours (essentially a full working day of doing nothing else) to do - this post is taking some of that 7 hours to get the existing script to generate the required spreadsheets directly. This is definitely going to be a case of getting back more time than I invest.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

• 29 Oct 2020 Transfer output files with SCP

  Related to my previous post on monitoring output, I also needed to provide a script to transfer the output files from the scripts to a remote server. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Monitoring script outputs for updates

  Related to my previous post on launching 16 scripts in tmux, I also needed to provide a script to monitor the output files from some scripts and alert if they are not updated. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Launching scripts in tmux

  I was asked to help write a script to automate launching 16 scripts in a tmux session.

• 19 Jun 2020 First Python pull request

  Yesterday I opened my first issue against Python, and enhancement for mailbox, an old part of the standard library, to support Path-like objects, a very new part of the standard library.

• 08 Aug 2019 HTML linter

  Trying to debug a missing close paragraph tag somewhere in a 300 line web page, I came across Tidy which helped with it's -e flag to report errors and warnings.

• 23 Jun 2019 Docker part 2

  After my previous post, on installing Docker, I needed to setup my first container. I choose to set-up a container for Firefox, first, as that required me to get a graphical application that also needed sound working.

• 14 Jun 2019 Installing Docker on Debian

  I've been using virtual machines (via VirtualBox) for Linux-based testing and development for sometime but I've been persuaded that it's high-time I joined everyone else and started using containerisation.

Docker

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 28 Jul 2021 Multi-stage Python builds with Docker on CentOS

  Following on from my getting started with Docker on CentOS post, I found that the version of Docker is very old (1.13) and, amongst other things, does not support multi-stage builds which will allow use to cut down our image size (useful, as the Azure registry charges for data transfer).

• 21 Jul 2021 Docker on CentOS 7

  Essentially a “quickstart” for Docker on CentOS 7.

• 08 Apr 2021 Finding SCSI generic device names

  Linux has a number of SCSI drivers, many devices are managed by their own driver as well as the sg generic driver. Some tools need access to both devices provided by the specific driver (e.g. sr for CD/DVD drives) and the generic one - requiring knowledge of which device files correspond to the same physical device to pass through to, e.g., Docker containers.

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

• 21 May 2020 Dockerised salt-ssh

  For some systems that I have user accounts on but do not administer, I use Salt SSH (agent-less salt minion) to manage the common user-local files that are managed on the systems I do administer on these systems. I have previously made notes about it.

• 04 Jul 2019 Docker images and introspecting them

  I've been playing around with different ways to get graphical applications working in Docker and in the process creates some large images.

• 23 Jun 2019 Docker part 2

  After my previous post, on installing Docker, I needed to setup my first container. I choose to set-up a container for Firefox, first, as that required me to get a graphical application that also needed sound working.

• 14 Jun 2019 Installing Docker on Debian

  I've been using virtual machines (via VirtualBox) for Linux-based testing and development for sometime but I've been persuaded that it's high-time I joined everyone else and started using containerisation.

Singularity

• 14 Jun 2019 Installing Docker on Debian

  I've been using virtual machines (via VirtualBox) for Linux-based testing and development for sometime but I've been persuaded that it's high-time I joined everyone else and started using containerisation.

Video

• 14 Jun 2019 Making video indexes with FFmpeg

  I'm working on digitising some of my DVD collection at the moment, which means creating some pretty large files. Trying to view these over the network to discover what they are is pretty tedious (lots of buffering!), so I've been working on dumping an "index" image file that will helpfully give me enough information to determine what it is without actually loading/playing the file.

Laptop

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

• 26 Dec 2019 Dell XPS 13 SSD upgrade

  In July I replaced my Lenovo ThinkPad x240 with a Dell XPS 13 9370 I impulse bought from the reduced section of John Lewis. Last week I finally got around to installing the new SSD I bought in a Black Friday deal on-line.

• 16 Jun 2019 Graphical Glitches on Lenovo ThinkPad X240 on resume

  For a while (possiblyprobably since I installed it) I've had weird graphical glitches on resuming my laptop from deep sleep. Until now I'd settled for rebooting it to fix it but I went on the hunt for a better solution.

Thinkpad

• 16 Jun 2019 Graphical Glitches on Lenovo ThinkPad X240 on resume

  For a while (possiblyprobably since I installed it) I've had weird graphical glitches on resuming my laptop from deep sleep. Until now I'd settled for rebooting it to fix it but I went on the hunt for a better solution.

Lenovo

• 16 Jun 2019 Graphical Glitches on Lenovo ThinkPad X240 on resume

  For a while (possiblyprobably since I installed it) I've had weird graphical glitches on resuming my laptop from deep sleep. Until now I'd settled for rebooting it to fix it but I went on the hunt for a better solution.

Graphics

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

• 16 Jun 2019 Graphical Glitches on Lenovo ThinkPad X240 on resume

  For a while (possiblyprobably since I installed it) I've had weird graphical glitches on resuming my laptop from deep sleep. Until now I'd settled for rebooting it to fix it but I went on the hunt for a better solution.

Firefox

• 23 Jun 2019 Docker part 2

  After my previous post, on installing Docker, I needed to setup my first container. I choose to set-up a container for Firefox, first, as that required me to get a graphical application that also needed sound working.

Containers

• 28 Jul 2021 Multi-stage Python builds with Docker on CentOS

  Following on from my getting started with Docker on CentOS post, I found that the version of Docker is very old (1.13) and, amongst other things, does not support multi-stage builds which will allow use to cut down our image size (useful, as the Azure registry charges for data transfer).

• 21 Jul 2021 Docker on CentOS 7

  Essentially a “quickstart” for Docker on CentOS 7.

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

• 21 May 2020 Dockerised salt-ssh

  For some systems that I have user accounts on but do not administer, I use Salt SSH (agent-less salt minion) to manage the common user-local files that are managed on the systems I do administer on these systems. I have previously made notes about it.

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

• 04 Jul 2019 Docker images and introspecting them

  I've been playing around with different ways to get graphical applications working in Docker and in the process creates some large images.

Ad

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 08 Jul 2019 Changing Active Directory password on non-bound Mac

  To change an Microsoft Active Directory password on your (non-bound) Mac:

  $ kinit working_username@AD.DOMAIN
  $ kpasswd username_to_change@AD.DOMAIN

Osx

• 21 Nov 2019 Time Machine backup to network share

  I had been using my own external hard disk for backing up my work computer, however (despite being 256GB) Time Machine kept complaining it didn't have enough space. Fortunately I also have access to a 3TB network share at work, so here's how I changed my Mac to backup to there instead.

• 08 Jul 2019 Changing Active Directory password on non-bound Mac

  To change an Microsoft Active Directory password on your (non-bound) Mac:

  $ kinit working_username@AD.DOMAIN
  $ kpasswd username_to_change@AD.DOMAIN

Apple

• 08 Jul 2019 Changing Active Directory password on non-bound Mac

  To change an Microsoft Active Directory password on your (non-bound) Mac:

  $ kinit working_username@AD.DOMAIN
  $ kpasswd username_to_change@AD.DOMAIN

Authenticator

• 08 Jul 2019 Secure sudo with Google Authenticator

  A little while ago, I "secured" root access on my GNU/Linux¹ systems by stopping using the root password and migrating to sudo. I also setup Google Authenticator for it.

Sudo

• 08 Jul 2019 Secure sudo with Google Authenticator

  A little while ago, I "secured" root access on my GNU/Linux¹ systems by stopping using the root password and migrating to sudo. I also setup Google Authenticator for it.

Internet

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

• 18 Aug 2019 PlusNet Fibre (FTTC)

  After much suffering from the incompitence of Virgin Media we have switched to PlusNet for our broadband provider. Thankfully FTTC has arrived in our neighbourhood so we can, at long last, get more than 300kbps estimated speed down the phone line (although we had to have a new phone line installed, as we did not already have one).

Plusnet

• 22 May 2020 updating broadband PPPoE configuration

  For a very long time I have been running my own Linux-based routers as gateways to the internet. The configuration was setup a very long time ago and it has not been revisited, other than to update logins etc., since. I discovered today that there is now a kernel-mode PPPoE module (since 2.4, from what I can gather) that I have missed the arrival of and a more performant driver for it. As I was monitoring the route due to line-speed problems, I noted that I could see the process pppoe appear in top hovering around the 12% CPU mark and wondered if that was expected or not. Googling this lead to these changes.

• 18 Aug 2019 PlusNet Fibre (FTTC)

  After much suffering from the incompitence of Virgin Media we have switched to PlusNet for our broadband provider. Thankfully FTTC has arrived in our neighbourhood so we can, at long last, get more than 300kbps estimated speed down the phone line (although we had to have a new phone line installed, as we did not already have one).

Pppoe

• 22 May 2020 updating broadband PPPoE configuration

  For a very long time I have been running my own Linux-based routers as gateways to the internet. The configuration was setup a very long time ago and it has not been revisited, other than to update logins etc., since. I discovered today that there is now a kernel-mode PPPoE module (since 2.4, from what I can gather) that I have missed the arrival of and a more performant driver for it. As I was monitoring the route due to line-speed problems, I noted that I could see the process pppoe appear in top hovering around the 12% CPU mark and wondered if that was expected or not. Googling this lead to these changes.

• 18 Aug 2019 PlusNet Fibre (FTTC)

  After much suffering from the incompitence of Virgin Media we have switched to PlusNet for our broadband provider. Thankfully FTTC has arrived in our neighbourhood so we can, at long last, get more than 300kbps estimated speed down the phone line (although we had to have a new phone line installed, as we did not already have one).

Vigor

• 18 Aug 2019 PlusNet Fibre (FTTC)

  After much suffering from the incompitence of Virgin Media we have switched to PlusNet for our broadband provider. Thankfully FTTC has arrived in our neighbourhood so we can, at long last, get more than 300kbps estimated speed down the phone line (although we had to have a new phone line installed, as we did not already have one).

Javascript

• 16 Sep 2019 Testing React Native with Jest

  I've started a Mobile App at work using React Native. Obviously this needs testing. This post is about getting started on that process.

React

• 16 Sep 2019 Testing React Native with Jest

  I've started a Mobile App at work using React Native. Obviously this needs testing. This post is about getting started on that process.

React native

• 16 Sep 2019 Testing React Native with Jest

  I've started a Mobile App at work using React Native. Obviously this needs testing. This post is about getting started on that process.

Testing

• 16 Sep 2019 Testing React Native with Jest

  I've started a Mobile App at work using React Native. Obviously this needs testing. This post is about getting started on that process.

Jest

• 16 Sep 2019 Testing React Native with Jest

  I've started a Mobile App at work using React Native. Obviously this needs testing. This post is about getting started on that process.

Time machine

• 21 Nov 2019 Time Machine backup to network share

  I had been using my own external hard disk for backing up my work computer, however (despite being 256GB) Time Machine kept complaining it didn't have enough space. Fortunately I also have access to a 3TB network share at work, so here's how I changed my Mac to backup to there instead.

Windows

• 20 Jul 2021 Updating text file on Azure Windows VM from local text file with PowerShell

  In order for NFS identity mapping to work accross both platforms (Linux and Windows), pending either AD integration or fixing Windows NFS client integration with OpenLDAP server (neither of which is currently in place), we are currently maintaining a passwd file for Windows that maps the Windows identity to the corresponding UID/GID.

• 19 Jul 2021 Searching Windows AD by username and email with ldapsearch

  Further to my post on querying AD with LDAP, to search for users by Windows username or email address one can use these commands:

• 10 May 2021 Query AD with LDAP

  Because I keep ending up looking at my old scripts to refresh myself on how to do this by hand, here’s the recipe for querying AD with ldapsearch.

• 12 Apr 2021 Change Windows user password from the command line

  Quick and dirty, how to change the windows password from the command line in an insecure fashion (exposes password on the command-line).

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 22 Jul 2020 Windows telnet client

  Windows 10 does not come with a telnet client (one of the commands I still have to drop into a WSL session for). Today I rectified that by enabling the optional feature.

• 25 Jun 2020 Finding files with PowerShell

  From my May post, in which I started using PowerShell, I have been using it to do a search I would previously have used a Linux environment to use find for.

• 28 May 2020 Teams backgrounds

  To use your own custom backgrounds, drop the background file(s) in $Env:APPDATA\Microsoft\Teams\Backgrounds\Uploads (PowerShell, %APPDATA%\Microsoft\Teams\Backgrounds\Uploads in cmd). They must be PNG files and will just show up in the list of backgrounds to choose from in the user interface.

• 26 May 2020 PowerShell startup script

  Yesterday I wrote about getting started with PowerShell but in order to make certain things persist, such as aliases, they need to be created each time the shell starts in the “profile” start-up script.

• 25 May 2020 Getting to grips with PowerShell

  Over the past 6-12 months I have started using a more Microsoft-y environment on Windows, both on my own laptop and work’s. It started with using the bundled Windows OpenSSH client, removing the need for me to launch a Windows Subsystem for Linux (WSL) environment just to ssh to another system. I also been trying out Microsoft Terminal(yes, it is open source), which was released on Tuesday, and it is a really slick tool - a vast improvement on the old command-line windows and/or PuTTY. Yesterday I switched my default session from (WSL) Debian to PowerShell and have started using it as my main shell environment on Windows.

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

• 08 May 2020 Formatting FAT32 on Windows 10

  In order to backup my PS3™, in preparation for replacing the hard disk drive with a larger SSD I have laying around my desk, I needed a FAT32 formatted USB disk. Windows 10 will only allow disks larger than 32GB to be formatted as exFAT (or NTFS), which the PS3™ does not support.

• 26 Dec 2019 Dell XPS 13 SSD upgrade

  In July I replaced my Lenovo ThinkPad x240 with a Dell XPS 13 9370 I impulse bought from the reduced section of John Lewis. Last week I finally got around to installing the new SSD I bought in a Black Friday deal on-line.

Dell

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

• 26 Dec 2019 Dell XPS 13 SSD upgrade

  In July I replaced my Lenovo ThinkPad x240 with a Dell XPS 13 9370 I impulse bought from the reduced section of John Lewis. Last week I finally got around to installing the new SSD I bought in a Black Friday deal on-line.

Xps 9370

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

• 26 Dec 2019 Dell XPS 13 SSD upgrade

  In July I replaced my Lenovo ThinkPad x240 with a Dell XPS 13 9370 I impulse bought from the reduced section of John Lewis. Last week I finally got around to installing the new SSD I bought in a Black Friday deal on-line.

Dvd

• 14 Apr 2020 More forced subtitles in DVDs and BluRays

  Following my last post I’ve been going through my DVD collection and fixing the forced subtitles for the films that have them.

• 07 Apr 2020 (sort-of-but-not-really-)forced subtitles in DVDs and BluRays

  As part of an on-going project to convert my DVD and BluRay collection to a hard-disk based media collection (mainly to save shelf-space and having to keep swapping disks) I’ve encountered a disk that has so-called “forced subtitles” to subtitle some foreign language speech into the viewers native language.

Broadband

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 22 May 2020 updating broadband PPPoE configuration

  For a very long time I have been running my own Linux-based routers as gateways to the internet. The configuration was setup a very long time ago and it has not been revisited, other than to update logins etc., since. I discovered today that there is now a kernel-mode PPPoE module (since 2.4, from what I can gather) that I have missed the arrival of and a more performant driver for it. As I was monitoring the route due to line-speed problems, I noted that I could see the process pppoe appear in top hovering around the 12% CPU mark and wondered if that was expected or not. Googling this lead to these changes.

• 17 Apr 2020 A New Switch

  This post documents the re-introduction of VLANs to our home network, removed in 2018 to solve problems with 4-year old WAP and power-line adapters their replacement means we can move back to more secure and flexible networking set-up.

Plex

• 17 Apr 2020 A New Switch

  This post documents the re-introduction of VLANs to our home network, removed in 2018 to solve problems with 4-year old WAP and power-line adapters their replacement means we can move back to more secure and flexible networking set-up.

Sky

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

• 17 Apr 2020 A New Switch

  This post documents the re-introduction of VLANs to our home network, removed in 2018 to solve problems with 4-year old WAP and power-line adapters their replacement means we can move back to more secure and flexible networking set-up.

Tplink omada

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 19 Apr 2020 Return of the WAPs

  Setting up the final piece of the new network puzzle, the wireless access points…

Rawlplugs

• 20 Apr 2020 Rawlplug UNO sizes technical data

  Rawlplug’s UNO range are currently my favourite wall plug, reliable and I’m yet to have any problems with the installation of one. I struggle to find this information, particularly supported screw sizes, when I want to refer to it to here it is:

Diy

• 20 Apr 2020 Rawlplug UNO sizes technical data

  Rawlplug’s UNO range are currently my favourite wall plug, reliable and I’m yet to have any problems with the installation of one. I struggle to find this information, particularly supported screw sizes, when I want to refer to it to here it is:

Gpg

• 22 Apr 2020 Keysigning party post-party signing

  On 1st & 2nd February I was fortunate to attend FOSDEM 2020 in at Université Libre de Bruxelles, Brussels. While I was there I took part in the event’s keysigning party and now need to sign the keys I verified (only 8 days before the deadline of 30th April!).

Playstation

• 09 May 2020 Upgrading PS3 hard disk

  As I mentioned yesterday, I was planning to replacing the hard disk drive in my PS3™ with a larger SSD I had laying around my desk.

• 08 May 2020 Formatting FAT32 on Windows 10

  In order to backup my PS3™, in preparation for replacing the hard disk drive with a larger SSD I have laying around my desk, I needed a FAT32 formatted USB disk. Windows 10 will only allow disks larger than 32GB to be formatted as exFAT (or NTFS), which the PS3™ does not support.

Ps3

• 09 May 2020 Upgrading PS3 hard disk

  As I mentioned yesterday, I was planning to replacing the hard disk drive in my PS3™ with a larger SSD I had laying around my desk.

• 08 May 2020 Formatting FAT32 on Windows 10

  In order to backup my PS3™, in preparation for replacing the hard disk drive with a larger SSD I have laying around my desk, I needed a FAT32 formatted USB disk. Windows 10 will only allow disks larger than 32GB to be formatted as exFAT (or NTFS), which the PS3™ does not support.

Usb

• 11 Jul 2021 USB booting Raspberry PI 4

  This post describes the process of enabling boot-from-USB and then using a USB disk for booting my Raspberry PI 4. It was made from notes I jotted down when I did this, over a week ago, so I apologise if it seems a bit disjointed or bulleted.

• 08 May 2020 Dell XPS 13 (9370) Thunderbolt & eGPU

  This post is just pulling together notes I have made whilst researching eGPUs with this laptop. I currently use a Razer Core X Chroma™ with it, which works very well except for issues with USB-disconnects but this seems to be a general problem with USB devices and Windows 7 onwards.

Ppp

• 22 May 2020 updating broadband PPPoE configuration

  For a very long time I have been running my own Linux-based routers as gateways to the internet. The configuration was setup a very long time ago and it has not been revisited, other than to update logins etc., since. I discovered today that there is now a kernel-mode PPPoE module (since 2.4, from what I can gather) that I have missed the arrival of and a more performant driver for it. As I was monitoring the route due to line-speed problems, I noted that I could see the process pppoe appear in top hovering around the 12% CPU mark and wondered if that was expected or not. Googling this lead to these changes.

Bonding

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 23 May 2020 Adding bonded connection to Netgear ReadyNAS 214

  My Netgear ReadyNAS 214 has two Ethernet ports. It is used as a safe place to store my data as well has hosting an iSCSI target that all my backups are done to (with regular off-site copies made to mitigate against failure or physical damage/loss of the NAS) so some extra bandwidth would not hurt and the web user-interface provides the option to bond them. Since my managed switch also supports bonding, I have decided to do this.

Lacp

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 23 May 2020 Adding bonded connection to Netgear ReadyNAS 214

  My Netgear ReadyNAS 214 has two Ethernet ports. It is used as a safe place to store my data as well has hosting an iSCSI target that all my backups are done to (with regular off-site copies made to mitigate against failure or physical damage/loss of the NAS) so some extra bandwidth would not hurt and the web user-interface provides the option to bond them. Since my managed switch also supports bonding, I have decided to do this.

Nas

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 23 May 2020 Adding bonded connection to Netgear ReadyNAS 214

  My Netgear ReadyNAS 214 has two Ethernet ports. It is used as a safe place to store my data as well has hosting an iSCSI target that all my backups are done to (with regular off-site copies made to mitigate against failure or physical damage/loss of the NAS) so some extra bandwidth would not hurt and the web user-interface provides the option to bond them. Since my managed switch also supports bonding, I have decided to do this.

Netgear

• 04 Jan 2022 Reconfiguring network for satellite switch

  Due to port exhaustion I am moving my “desktop” (quoted as it includes some laptops) systems to a secondary switch. This post describes the switch and new VLAN arrangement for these systems.

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 23 May 2020 Adding bonded connection to Netgear ReadyNAS 214

  My Netgear ReadyNAS 214 has two Ethernet ports. It is used as a safe place to store my data as well has hosting an iSCSI target that all my backups are done to (with regular off-site copies made to mitigate against failure or physical damage/loss of the NAS) so some extra bandwidth would not hurt and the web user-interface provides the option to bond them. Since my managed switch also supports bonding, I have decided to do this.

Powershell

• 20 Jul 2021 Updating text file on Azure Windows VM from local text file with PowerShell

  In order for NFS identity mapping to work accross both platforms (Linux and Windows), pending either AD integration or fixing Windows NFS client integration with OpenLDAP server (neither of which is currently in place), we are currently maintaining a passwd file for Windows that maps the Windows identity to the corresponding UID/GID.

• 12 Apr 2021 Change Windows user password from the command line

  Quick and dirty, how to change the windows password from the command line in an insecure fashion (exposes password on the command-line).

• 22 Jul 2020 Windows telnet client

  Windows 10 does not come with a telnet client (one of the commands I still have to drop into a WSL session for). Today I rectified that by enabling the optional feature.

• 25 Jun 2020 Finding files with PowerShell

  From my May post, in which I started using PowerShell, I have been using it to do a search I would previously have used a Linux environment to use find for.

• 26 May 2020 PowerShell startup script

  Yesterday I wrote about getting started with PowerShell but in order to make certain things persist, such as aliases, they need to be created each time the shell starts in the “profile” start-up script.

• 25 May 2020 Getting to grips with PowerShell

  Over the past 6-12 months I have started using a more Microsoft-y environment on Windows, both on my own laptop and work’s. It started with using the bundled Windows OpenSSH client, removing the need for me to launch a Windows Subsystem for Linux (WSL) environment just to ssh to another system. I also been trying out Microsoft Terminal(yes, it is open source), which was released on Tuesday, and it is a really slick tool - a vast improvement on the old command-line windows and/or PuTTY. Yesterday I switched my default session from (WSL) Debian to PowerShell and have started using it as my main shell environment on Windows.

Scripting

• 25 Aug 2021 Checking NextCloud version

  One of my current issues is that I do not notice my NextCloud install requires updating until it is no longer supported and the clients stops working with it. I already have icinga setup as my monitoring solution, so it has been on my to-do list for a while to create a plugin to check the version of my server against the latest release.

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

• 14 Jul 2021 Creating Excel spreadsheets with Python

  Every month I produce usage spreadsheets for the HPC service I run. This involved much copying and pasting of data, which I already have a script to export in CSV format, into a bunch of spreadsheets for different people. In total, it takes about 7 hours (essentially a full working day of doing nothing else) to do - this post is taking some of that 7 hours to get the existing script to generate the required spreadsheets directly. This is definitely going to be a case of getting back more time than I invest.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 29 Oct 2020 Transfer output files with SCP

  Related to my previous post on monitoring output, I also needed to provide a script to transfer the output files from the scripts to a remote server. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Monitoring script outputs for updates

  Related to my previous post on launching 16 scripts in tmux, I also needed to provide a script to monitor the output files from some scripts and alert if they are not updated. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Launching scripts in tmux

  I was asked to help write a script to automate launching 16 scripts in a tmux session.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 25 Jun 2020 Finding files with PowerShell

  From my May post, in which I started using PowerShell, I have been using it to do a search I would previously have used a Linux environment to use find for.

• 19 Jun 2020 First Python pull request

  Yesterday I opened my first issue against Python, and enhancement for mailbox, an old part of the standard library, to support Path-like objects, a very new part of the standard library.

• 26 May 2020 PowerShell startup script

  Yesterday I wrote about getting started with PowerShell but in order to make certain things persist, such as aliases, they need to be created each time the shell starts in the “profile” start-up script.

• 25 May 2020 Getting to grips with PowerShell

  Over the past 6-12 months I have started using a more Microsoft-y environment on Windows, both on my own laptop and work’s. It started with using the bundled Windows OpenSSH client, removing the need for me to launch a Windows Subsystem for Linux (WSL) environment just to ssh to another system. I also been trying out Microsoft Terminal(yes, it is open source), which was released on Tuesday, and it is a really slick tool - a vast improvement on the old command-line windows and/or PuTTY. Yesterday I switched my default session from (WSL) Debian to PowerShell and have started using it as my main shell environment on Windows.

Ssh

• 25 May 2020 Getting to grips with PowerShell

  Over the past 6-12 months I have started using a more Microsoft-y environment on Windows, both on my own laptop and work’s. It started with using the bundled Windows OpenSSH client, removing the need for me to launch a Windows Subsystem for Linux (WSL) environment just to ssh to another system. I also been trying out Microsoft Terminal(yes, it is open source), which was released on Tuesday, and it is a really slick tool - a vast improvement on the old command-line windows and/or PuTTY. Yesterday I switched my default session from (WSL) Debian to PowerShell and have started using it as my main shell environment on Windows.

Microsoft

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 19 Jul 2021 Working with virtual machines using the Azure command line client

  Over the past 8 months I have gethered quite a few useful snippets for interacting with Azure VMs from the az command-line client. This is a summary of them.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

• 22 Jul 2020 Windows telnet client

  Windows 10 does not come with a telnet client (one of the commands I still have to drop into a WSL session for). Today I rectified that by enabling the optional feature.

• 28 May 2020 Teams backgrounds

  To use your own custom backgrounds, drop the background file(s) in $Env:APPDATA\Microsoft\Teams\Backgrounds\Uploads (PowerShell, %APPDATA%\Microsoft\Teams\Backgrounds\Uploads in cmd). They must be PNG files and will just show up in the list of backgrounds to choose from in the user interface.

Teams

• 28 May 2020 Teams backgrounds

  To use your own custom backgrounds, drop the background file(s) in $Env:APPDATA\Microsoft\Teams\Backgrounds\Uploads (PowerShell, %APPDATA%\Microsoft\Teams\Backgrounds\Uploads in cmd). They must be PNG files and will just show up in the list of backgrounds to choose from in the user interface.

Hp

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

Hpe

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

Ilo

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

Kvm

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 06 Jun 2020 Building Debian VMs with debootstrap

  Following the awful time I had setting up a working preseed configuration for Debian installer I am exploring the alternative method of pre-building the disk image using debootstrap.

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

Virtualisation

• 19 Jul 2021 Working with virtual machines using the Azure command line client

  Over the past 8 months I have gethered quite a few useful snippets for interacting with Azure VMs from the az command-line client. This is a summary of them.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 23 Jul 2020 Offsite DR backup with VM

  Since migrating BackupPC to a VM I have not been doing off-site backups (since I have been working from home full-time). Today I had to visit the office to retrieve some essential adaptors for my work laptop, and while I was there I grabbed my off-site backups disks. Now I have done this, I need to figure out how to pass the device through to my VM in order to update the oldest backup (which has not been updated since December 2019!).

• 22 Jul 2020 Migrating Omada controller to VM

  As part of my migration of core services off the router to VMs I am moving the Omada controller. I have already moved the SaltStack controller and Debian pre-seed web site, but as these were straight-forward migration of daemons and data I did not write any notes about the process.

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

• 08 Jun 2020 Migrate BackupPC to VM

  Now that I have deploying VMs sorted I am migrating the first service from my router onto a new VM, my BackupP server. This should be relatively straight forward as the server set-up is managed in SaltStack and the data resides on an iSCSI volume from my NAS, do configuration and “transfer” should be simple. Should be.

• 06 Jun 2020 Building Debian VMs with debootstrap

  Following the awful time I had setting up a working preseed configuration for Debian installer I am exploring the alternative method of pre-building the disk image using debootstrap.

• 04 Jun 2020 KVM setup

  Following on from tuesday’s post on setting up my first microserver, I am starting to set-up KVM and automate building virtual-machines in order to start migrating services off my router.

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

Vmware

• 02 Jun 2020 HP Microservers

  Over the weekend I bought and collected 2 8th generation HPE Microservers. The two I have bought have been upgraded to 16GB of ECC memory and have had their stock processors replaced with E3-1240 v2 processors. They also came with 10GbE cards fitted, although currently I have no infrastructure to make use of these.

Munin

• 19 Sep 2021 Migrating Raspberry Pi to new SSD

  Since setting up my Raspberry Pi to boot from a USB SSD, I need to repurpose the SSD I used to set this up. I bought a smaller 240GB SSD to replace it and wanted to copy the existing filesystem over to it rather than reinstall.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

• 14 Jun 2020 Migrate monitoring to VM

  Hot on the heals of migrating BackupPC from a bare-metal (router) system to a VM, the next set of services to migrate are my monitoring services. For this I run two seperate systems, Icinga2 and Munin. Icinga excels as monitoring and alerting to problems and faults as they happen, Munin provides resource monitoring and graphing that gives a better view of “what just happened to kill our performance?”(quoted from their website) and historic views of the same.

Python

• 28 Jul 2021 Multi-stage Python builds with Docker on CentOS

  Following on from my getting started with Docker on CentOS post, I found that the version of Docker is very old (1.13) and, amongst other things, does not support multi-stage builds which will allow use to cut down our image size (useful, as the Azure registry charges for data transfer).

• 14 Jul 2021 Creating Excel spreadsheets with Python

  Every month I produce usage spreadsheets for the HPC service I run. This involved much copying and pasting of data, which I already have a script to export in CSV format, into a bunch of spreadsheets for different people. In total, it takes about 7 hours (essentially a full working day of doing nothing else) to do - this post is taking some of that 7 hours to get the existing script to generate the required spreadsheets directly. This is definitely going to be a case of getting back more time than I invest.

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

• 29 Oct 2020 Transfer output files with SCP

  Related to my previous post on monitoring output, I also needed to provide a script to transfer the output files from the scripts to a remote server. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Monitoring script outputs for updates

  Related to my previous post on launching 16 scripts in tmux, I also needed to provide a script to monitor the output files from some scripts and alert if they are not updated. For the same reasons, so the researcher can maintain his own scripts, this was written in Python rather than Bash.

• 29 Oct 2020 Launching scripts in tmux

  I was asked to help write a script to automate launching 16 scripts in a tmux session.

• 19 Jun 2020 First Python pull request

  Yesterday I opened my first issue against Python, and enhancement for mailbox, an old part of the standard library, to support Path-like objects, a very new part of the standard library.

Dns

• 10 May 2021 Query AD with LDAP

  Because I keep ending up looking at my old scripts to refresh myself on how to do this by hand, here’s the recipe for querying AD with ldapsearch.

• 15 Jan 2021 Let's Encrypt SSL certificates at home

  For nearly 2 years I have been using Let’s Encrypt (like half the tech world) for SSL certificates on my public-facing projects and services. I have decided to try an extend their use to my internal sites too, and do-away with running my own certificate authority except for a few niche cases (OpenVPN, for example).

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

Dnsmasq

• 05 Jan 2021 Split DNS with Dnsmasq

  I had a need to split DNS resolution on a series of hosts between two DNS servers, one local to the environment and one for global address resolution. In the past I have always used BIND 9 for my go-to DNS server, however for this trivial forwarding tasks it seemed overkill and a good opportunity to give something lighter-weight a go. So I opted to try Dnsmasq instead.

Centos

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

• 28 Jul 2021 Multi-stage Python builds with Docker on CentOS

  Following on from my getting started with Docker on CentOS post, I found that the version of Docker is very old (1.13) and, amongst other things, does not support multi-stage builds which will allow use to cut down our image size (useful, as the Azure registry charges for data transfer).

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

• 21 Jul 2021 Docker on CentOS 7

  Essentially a “quickstart” for Docker on CentOS 7.

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

Kerberos

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

Redhat

• 06 Jan 2021 Migrate Linux system to AD authentication

  As a stepping stone in migration a CentOS Linux system from local user management to Active Directory(AD) integration, I setup Kerberos authentication to the AD but retaining local identity information.

Apc

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

Bx1400ui

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

Nut

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

Readynas

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

Rn214

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

Ups

• 22 Aug 2021 Automatically shutting down server and NAS

  At home we have a NAS (An APC Back-UPS BX1400UI) which protects our internet connectivity equipment, core network switch, one server and network-attached (NAS) storage device. The server runs the backup tool (BackupPC) that backs up all of my local and cloud systems, with the backups stored on an iSCSI volume on the NAS (with monthly “off-site” copies).

• 10 Jan 2021 UPS monitoring and auto shut-down with NUT

  Last night we had a power-cut that lasted approximately 20 minutes, about 5 minutes into which I started my usual routine of manually shutting down systems to shed load (and hence prolong runtime for our broadband infrastructure) from my UPS. At around 15 minutes into the power-cut I started to get a bit twitchy about how much runtime was left on the UPS and finally go around to doing something about it (not the ideal conditions to be setting it up, but a good motivator).

Github

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

Github actions

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

Linting

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

Slurm

• 14 Jul 2021 Creating Excel spreadsheets with Python

  Every month I produce usage spreadsheets for the HPC service I run. This involved much copying and pasting of data, which I already have a script to export in CSV format, into a bunch of spreadsheets for different people. In total, it takes about 7 hours (essentially a full working day of doing nothing else) to do - this post is taking some of that 7 hours to get the existing script to generate the required spreadsheets directly. This is definitely going to be a case of getting back more time than I invest.

• 11 Jul 2021 Getting an interactive job with Slurm

  I keep getting this wrong (forgetting the --pty option), so all (Slurm users) recite together now:

• 13 Feb 2021 Getting started with GitHub Actions

  One of the things we need more of in my new day-job is automation, something we were good at in my previous role. To get started with GitHub Actions I have decided to start with linting in one of my repositories, Slurm Helpers, and this post documents the journey to getting that working.

Ansible

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

Azure

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

• 20 Jul 2021 Updating text file on Azure Windows VM from local text file with PowerShell

  In order for NFS identity mapping to work accross both platforms (Linux and Windows), pending either AD integration or fixing Windows NFS client integration with OpenLDAP server (neither of which is currently in place), we are currently maintaining a passwd file for Windows that maps the Windows identity to the corresponding UID/GID.

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

• 19 Jul 2021 Working with virtual machines using the Azure command line client

  Over the past 8 months I have gethered quite a few useful snippets for interacting with Azure VMs from the az command-line client. This is a summary of them.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

Azure devops

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

Devops

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

• 13 Feb 2021 Ansible with Azure DevOps

  Since at least July 2013 (the initial commit in my current Salt configuration repository at home) I have been using SaltStack to orchestrate and configure my systems, including VMs, at home. In the last few years Ansible has grown in popularity and I have recently been looking closely at it due to its integration with Azure and Microsoft include Ansible in their Azure documentation and Cloud Shell Platform.

Vnc

• 28 Apr 2021 Decrypt VNC password

  Decrypt a VNC password file in one-line with nothing but openssl:

Ldap

• 19 Jul 2021 Searching Windows AD by username and email with ldapsearch

  Further to my post on querying AD with LDAP, to search for users by Windows username or email address one can use these commands:

• 10 May 2021 Query AD with LDAP

  Because I keep ending up looking at my old scripts to refresh myself on how to do this by hand, here’s the recipe for querying AD with ldapsearch.

Life

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

• 15 Jun 2021 Hay-fever

  For the first time in quite a few years I have been really suffering with hay-fever. The glorious sunshine during peak grass season (which is what causes mine) season has sent grass-pollen counts through the roof. This is coupled with me walking our two dogs during peak pollen times, in the morning and evenings (when the weather is cooler for the dogs) but I have been thinking about whether other factors are at play in my unusually high level of suffering this year. These are my entirely unscientific, layman, musings.

Musing

• 15 Jun 2021 Hay-fever

  For the first time in quite a few years I have been really suffering with hay-fever. The glorious sunshine during peak grass season (which is what causes mine) season has sent grass-pollen counts through the roof. This is coupled with me walking our two dogs during peak pollen times, in the morning and evenings (when the weather is cooler for the dogs) but I have been thinking about whether other factors are at play in my unusually high level of suffering this year. These are my entirely unscientific, layman, musings.

Non technical

• 15 Jun 2021 Hay-fever

  For the first time in quite a few years I have been really suffering with hay-fever. The glorious sunshine during peak grass season (which is what causes mine) season has sent grass-pollen counts through the roof. This is coupled with me walking our two dogs during peak pollen times, in the morning and evenings (when the weather is cooler for the dogs) but I have been thinking about whether other factors are at play in my unusually high level of suffering this year. These are my entirely unscientific, layman, musings.

Random

• 15 Jun 2021 Hay-fever

  For the first time in quite a few years I have been really suffering with hay-fever. The glorious sunshine during peak grass season (which is what causes mine) season has sent grass-pollen counts through the roof. This is coupled with me walking our two dogs during peak pollen times, in the morning and evenings (when the weather is cooler for the dogs) but I have been thinking about whether other factors are at play in my unusually high level of suffering this year. These are my entirely unscientific, layman, musings.

Raspberry pi

• 19 Sep 2021 Migrating Raspberry Pi to new SSD

  Since setting up my Raspberry Pi to boot from a USB SSD, I need to repurpose the SSD I used to set this up. I bought a smaller 240GB SSD to replace it and wanted to copy the existing filesystem over to it rather than reinstall.

• 11 Jul 2021 USB booting Raspberry PI 4

  This post describes the process of enabling boot-from-USB and then using a USB disk for booting my Raspberry PI 4. It was made from notes I jotted down when I did this, over a week ago, so I apologise if it seems a bit disjointed or bulleted.

• 17 Jun 2021 Installing Debian on Raspberry PI 4

  I have been planning to migrate my monitoring to my Raspberry PI 4, from its current location on a VM which is fine unless there is a fault with the VM or the host but this has happened (today in fact). It monitors these hosts and it is helpful to know what is going on if there is a problem. My intention is to also shutdown the VM host in the event of a power outage (to extend UPS battery runtime) and it would be nice for monitoring to continue during the outage.

Troubleshooting

• 06 Jul 2021 Troubleshooting full filesystems where df and du disagree

  We have a recurring problem with disk space being exhausted on the root filesystem of a system, the root cause of which is gnome-terminal holding open file-handles to very large deleted temporary files in /tmp. I suspect there is a bug in gnome-terminal not closing the handles to its scrollback buffer (possibly only when set to unlimited scrollback, as some users have).

Firewall

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

Iptables

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

Virgin

• 05 Dec 2021 One For All Universal Remote with Virgin Media TV360

  On Thursday we received our new Virgin Media equipment, including a TV360 box. Despite being physically the same box as their old V6 version (existing customers are getting TV360 pushed out as a software update) the TV360 software no longer works with existing Virgin TV remotes.

• 10 Jul 2021 Fixing Sky downloading on new broadband

  This post documents my attempt to send traffic from specific local hosts (Sky TV boxes) via my existing VPN connection, whilst retaining direct access to my broadband connection for all other hosts as a workaround to the annoying problems Sky services have when the client is behind a carrier-level network address translation (carrier-grade NAT or CGNAT).

Hpc

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

• 14 Jul 2021 Creating Excel spreadsheets with Python

  Every month I produce usage spreadsheets for the HPC service I run. This involved much copying and pasting of data, which I already have a script to export in CSV format, into a bunch of spreadsheets for different people. In total, it takes about 7 hours (essentially a full working day of doing nothing else) to do - this post is taking some of that 7 hours to get the existing script to generate the required spreadsheets directly. This is definitely going to be a case of getting back more time than I invest.

• 11 Jul 2021 Getting an interactive job with Slurm

  I keep getting this wrong (forgetting the --pty option), so all (Slurm users) recite together now:

Excel

• 14 Jul 2021 Creating Excel spreadsheets with Python

  Every month I produce usage spreadsheets for the HPC service I run. This involved much copying and pasting of data, which I already have a script to export in CSV format, into a bunch of spreadsheets for different people. In total, it takes about 7 hours (essentially a full working day of doing nothing else) to do - this post is taking some of that 7 hours to get the existing script to generate the required spreadsheets directly. This is definitely going to be a case of getting back more time than I invest.

Cloud

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

• 20 Jul 2021 Getting started with Ansible and Azure

  This post stated, like most of my posts, with me making notes as I went along trying to accomplish a technical task. I quickly became a bit of a rant as I encountered numerous issues with Ansible’s Azure integration.

Docker registry

• 21 Jul 2021 Docker on CentOS 7

  Essentially a “quickstart” for Docker on CentOS 7.

Environment modules

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

Lmod

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

Modules

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

• 27 Jul 2021 Platform detection with Lmod

  Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

Easybuild

• 27 Sep 2021 Custom EasyBlocks for easybuild

  Since my [post on creating separate EasyBuild development environments]/notes/2021/07/29/setting-up-easybuild-environments.html I have encountered a piece of software I need to develop some custom EasyBlocks for. My scripts allow this, however my magic module does not initialise the platform-specific installs using a custom EasyBlock source. In order to retain the convenience of architecture-local initialisation on each architecture I modified part of my module.

• 29 Jul 2021 Setting up EasyBuild environments

  This follows on from my post on platform detection with Lmod, using that as a tool from which to create platform-specific builds of software with EasyBuild. This is a long one, as it contains the full setup for getting this running.

Vs code

• 29 Jul 2021 VS Code spell checking

  I switched from Sublime Text, which I have used for many years, to Visual Studio Code (or ‘VS Code’ for short) mainly due to being unhappy with Sublime Text’s new licencing model (which has gone from licences for a particular major version to a subscription model, with a 3 year subscription period). One thing I have missed is Sublime Text’s built-in spell checker but there is a VS Code marketplace extension for that.

Nextcloud

• 25 Aug 2021 Checking NextCloud version

  One of my current issues is that I do not notice my NextCloud install requires updating until it is no longer supported and the clients stops working with it. I already have icinga setup as my monitoring solution, so it has been on my to-do list for a while to create a plugin to check the version of my server against the latest release.

Terraform

• 25 Nov 2021 Splitting up TerraForm configuration with a module

  Continuing from my last TerraForm post, I have split my TerraForm configuration into a number of files. I am now taking this one step further and creating a module that I can use to deploy a number of identical (or near-identical) resources following a pattern.

• 29 Sep 2021 TerraForm, cloud-init and Ansible

  As I mentioned at the end of yesterday’s TerraForm post, one of the two outstanding tasks with my test setup is to kick-off an initial configuration of the VM.

• 28 Sep 2021 TerraForm VM management

  Following on from my last TerraForm post, my next task is to deploy a VM with the tool. This is more complex than it sounds, as a ‘VM’ requires a number of components (network interface, disk) in addition to the VM as well as me wanting to configure it to do some initial boot-strapping on first start.

• 24 Sep 2021 TerraForm part 2

  Following on from my first steps with TerraForm post, this post covers the next steps. This includes external state storage and splitting up the terraform configuration.

• 23 Sep 2021 Getting started with TerraForm

  These notes document my first steps in using TerraForm to manage Azure infrastructure. This first post is how I brought some existing resources under TerraForm’s management.

Systemd

• 30 Sep 2021 docker-compose with systemd

  This post is about launching a Docker Compose orchestrated collection of containers as a system service using systemd. I provisioned a VM running Docker (it’s a loooong story but short version: Azure Kubernetes and Azure container services cannot be deployed without public IP addresses, which goes against some of our secure-by-design decisions) using Terraform and Ansible to deploy and configure it. The service it is running is a web application made up of two Docker containers and I have written a Docker Compose file that builds and runs the infrastructure.

Television

• 05 Dec 2021 One For All Universal Remote with Virgin Media TV360

  On Thursday we received our new Virgin Media equipment, including a TV360 box. Despite being physically the same box as their old V6 version (existing customers are getting TV360 pushed out as a software update) the TV360 software no longer works with existing Virgin TV remotes.