After finally breaking through the barrier of migrating sufficient of my SaltStack managed configuration to Ansible, I can build my first bastion host. My thinking has moved on slightly from my original design but, for now, I am going to try and focus this post on just building the bastion and leave discussing the network design for another post.

Following the recent SSH logging issue, dubbed regreSSHion (in light of it being a reintroduced vulnerability) I decided to use OpenBSD for the bastion because of:

OpenBSD systems are unaffected by this bug, as OpenBSD developed a secure mechanism in 2001 that prevents this vulnerability.

This also adds an element of defence-in-depth, since most of my systems are running Linux forcing traversal of a BSD system between security zones means that a Linux specific vulnerability would be confined to being exploited within one zone - likewise, a BSD specific vulnerability could allow compromise of the bastion but would not allow compromise of the non-BSD systems behind that host.

I decided (this is part of my rethinking) to create a new “demilitarized zone (DMZ)” VLAN on my switch and use switch ACLs to control access in and out of the DMZ. The OpenBSD host will exist inside this DMZ, as well as a front-end proxy to present some services out to the wider network.

Switch VLAN and routing configuration

Adding the DMZ vlan:

enable
configure
vlan 11
name Management-DMZ
exit

Add DMZ vlan trunk to Proxmox development port (as bastion will be a VM on the cluster - it’s already a trunk so just need to allow it):

interface gigabitEthernet 1/0/47
switchport general allowed vlan 11 tagged

Set the switch up to route between the networks:

interface vlan 11
ip address 192.168.11.250 255.255.255.0
exit
interface vlan 10
ip address 192.168.10.251 255.255.255.0
exit
interface vlan 20
ip address 192.168.20.251 255.255.255.0
exit

VM setup

I added the VM to the inventory:

proxmox_vms:
  hosts:
    # ...
    k7-mgmt:
      proxmox_vm:
        vlan: 11

Run the create-vms.yaml playbook:

ansible-playbook -i inventory.yaml create-vms.yaml

I downloaded an OpenBSD ISO and installed it on the VM manually. Key configurations:

  • manually configuring an IP address of 192.168.11.100
  • default route none
  • none DNS server
  • Setup a user for me
  • no to allow root ssh login
  • sd0 root disk
  • Encrypt root disk with passphrase
  • Installed filesets: bsd, base75.tgz and man75.tgz
  • timezone: Europe/London

Switch ACL configuration

I added a (temporary) route to my current central router, so existing systems will be able to go via that route, using the main network interface on the switch:

ip route add 192.168.11.0/24 via 192.168.20.251

I then attempted to ping it from the router which did not work, the switch replied “Destination Host Unreachable”. To troubleshoot, I logged into the switch and checked the routing table (show ip route), which appeared correct, and tried to ping from the switch, which failed. I then tried pinging the switch from the VM, which also failed.

I found two things I had overlooked:

  1. Adding the VLAN (tagged) to the interfaces of the (temporary) switch my cluster is plugged into.

  2. Adding routes to the OpenBSD box:

     route add 192.168.10/24 192.168.11.250
 route add 192.168.20/24 192.168.11.250

Once I corrected these, I was able to SSH to the OpenBSD host.

Bastion configuration

Making routing configuration persistent

To make the manual routes persistent, I needed to add them to /etc/hostname.<interface> - in my VM’s case /etc/hostname.vio0:

!route add 192.168.10/24 192.168.11.250
!route add 192.168.20/24 192.168.11.250

User configuration

I added a new user, ansible, for automation - this user will need doas access to allow Ansible to configure the system but my user will not need to escalate to jump through the host:

doas adduser
# [Add ansible user with password manually stored in vault]
# Add ansible user to doas:
echo "permit ansible as root" >> /etc/doas.conf
# Remove my existing user from wheel
usermod -S '' my_user

Installing Python

The next thing I wanted to do was to install Python, which proved harder than I thought. Initially, but Googling for “install python on openbsd”(other search engines are available) I found a Reddit post comment which suggested installing the package pkglocatedb to use pkg_locate to search for a binary.

Setting up repository access

The first hurdle was that my Basion could not access the internet (by design), so I needed to add access to the OpenBSD CDN via my mirror proxy - this was just a case of adding it to the list in the ansible variables:

mirror_proxies:
  #...
  - name: openbsd
    upstream: https://cdn.openbsd.org/pub/OpenBSD/
    description: OpenBSD main repositories

and re-running my site.yaml playbook against the mirror.

I then needed to update the /etc/installurl on the OpenBSD box to use the proxy:

http://mirror/openbsd

Getting the right version of Python

pkg_locate bin/python3 showed me multiple different versions, and being unsure which was the “correct” or “default” version I looked online for more information. I found the OpenBSD FAQ on Package Management which suggested adding -- to install default version, which I also tried (pkg_add python--) but was still prompted for which version to install from the possible matches. After much more Googling, I found another Reddit comment which told me to use pkg_add -I python3. Which worked.

Ansibleising the configuration

I turned the above steps into a bootstrap-openbsd.yaml playbook (with the hope of one day merging it with my existing, Linux, playbook), mostly using ansible.builtin.raw module and some carefully crafted commands to make it idempotent, as Python will not be available to use other modules yet (although the routes are a notable exception to the idempotence in terms of being “changed” to the same value even if already correct):

---
# Temporary file to configure OpenBSD bastion, until merged 
# with main bootstrap.yaml and/or site.yaml config.
# (existing host lookup and domain grouping config goes here - XXX needs to be separated to included playbook)
- name: Network configuration
  # XXX specific to this bastion
  hosts: k7-mgmt
  gather_facts: false
  tasks:
    # Not detectably (for `changed_when`) idempotent (substitution happens even if already correct)...
    - name: Routes exist
      become: true
      ansible.builtin.raw: "grep -q ^{{ match | ansible.builtin.quote }} /etc/hostname.{{ interface | ansible.builtin.quote }} && sed -i s_^{{ match | ansible.builtin.quote }}.\\*\\$_{{ value | ansible.builtin.quote }}_ /etc/hostname.{{ interface | ansible.builtin.quote }} || echo {{ value | ansible.builtin.quote }} >> /etc/hostname.{{ interface | ansible.builtin.quote }}"
      vars:
        target: '{{ item }}'
        via: 192.168.11.250  # Switch
        interface: vio0
        match: '!route add {{ target }}'
        value: '!route add {{ target }} {{ via }}'
      loop:
        - 192.168.10/24  # Management network
        - 192.168.20/24  # Desktop/main network

- name: Installation source is configured
  hosts: openbsd
  gather_facts: false
  tasks:
    - name: Package source is correct
      become: true
      ansible.builtin.raw: '[ "$( cat {{ url_file | ansible.builtin.quote }} )" == {{ url | ansible.builtin.quote }} ] && echo correct || echo {{ url | ansible.builtin.quote }} > {{ url_file | ansible.builtin.quote }}'
      register: installurl_output
      changed_when: installurl_output.stdout | trim != "correct"
      vars:
        url_file: /etc/installurl
        url: "{{ local_mirror.openbsd.uri | default('https://cdn.openbsd.org/pub/OpenBSD') }}"

- name: Python is installed
  hosts: openbsd
  gather_facts: false
  tasks:
    - name: Python is installed
      become: true
      ansible.builtin.raw: "which python3 && echo already-installed || pkg_add -I python3"
      register: pkg_add_output
      changed_when: pkg_add_output | trim != 'already-installed'

- name: Useful core utils are installed
  hosts: openbsd
  tasks:
    - name: Packages are installed
      become: true
      ansible.builtin.package:
        name: pkglocatedb
...

I added the host to my inventory.yaml file:

internal:
  hosts:
    #...
    k7-mgmt:

proxmox_vms:
  hosts:
    k7-mgmt:
      proxmox_vm:
        vlan: 11

openbsd:
  hosts:
    k7-mgmt:
  # When all bastions are openbsd
  #children:
  #  bastions:
  vars:
    ansible_become_method: doas

and the local mirror to group_vars/domain_home_entek_org_uk.yaml:

local_mirror:
  #...
  openbsd:
    uri: http://mirror/openbsd

Because Python wasn’t installed, I had to change the domain to be hardcoded for internal group hosts in group_vars/internal.yaml rather than rely on a fact (which I left set to '{{ ansible_facts.domain }}' in group_vars/all.yaml):

domain: home.entek.org.uk

Using the Bastion

ProxyJump ssh option:

Host k7-mgmt
  HostName 192.168.11.100
  User laurence

Host pve01
  User ansible
  HostName 192.168.10.51
  ProxyJump k7-mgmt

To be continued…

There is still some work to do to secure the new Bastion and enforce that access to the management VLAN only goes via it, but this will be the subject of a subsequent post.