Ansible to check update states
Post about adding tasks to Ansible to print information about the current state of updates across a (Debian) landscape. This is Debian specific as I do not have anything else in my production environment. On some level, this belongs in the monitoring tools (currently Icinga2 in my infrastructure) however when orchestrating infrastructure it can be handy to refer to from, and as a check that everything can be orchestrated by, the orchestration tool (Ansible, in this case). It was done to help work out the landscape during upgrading to Debian 11 (Bullseye)
The upgrade process calls for the systems to be in a “clean” state, and to be returned that way after the upgrade. That is to say (in no particular order):
- No packages in a broken (half installed or failed configuration) state - checked with
dpkg --audit - No unmerged configuration files from previous package updates/upgrades - checked by ensuring no files matching the globs
*.dpkg-*,*.ucf-*and*.merge-error - No non-Debian packages - checked with
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'(I ignored this, in relation to Docker, without a problem - I think the bottom line is “be sensible” and if you do ignore the advice “your mileage may vary”/”here be dragons”) - No “obsolete” packages (essentially, packages no longer available from an installation source) - checked with
aptitude search '~o' - No uninstalled packages with configuration files left behind - checked with
dpkg -l | awk '/^rc/ { print $2 }oraptitude search '~c'
To help do this across a small estate of systems, I created an Ansible playbook to do these checks and output a collated (debug) message with the results. As suggested in the introduction, I should integrate these with my Icinga monitoring at some-point to pro-actively monitor but for now this helps do the pressing task (upgrading). There is a group, called dummy, of systems that do not really exist (typically aliases for alternate routes to, or operating systems on, a system) which are excluded. The search for unmerged configurations required permissions to search the whole of /etc, so is the only command using become: yes - everything else requires no special privileges. This is the first task in order to do the search before my TOTP sudo token expires - I am working on replacing TOTP sudo with remote access via a bastion host to make working with Ansible easier (c.f. SaltStack which uses its own agent, salt-minion, running as root).
The playbook also displays each Debian version and a list of hosts at that version, and list of non-Debian package sources and a list of hosts with each of them, to make it easy to see where each host is at during the upgrades.
The playbook looks like this:
---
- hosts: all:!dummy
tasks:
# XXX TODO - add 'check packages in Half-Installed or Failed-Config':
# In order to do so, I need to know what 'bad' output from
# `dpkg --audit` looks like - all of my systems were clean. :)
# Check for old/unmerged configuration files
- name: Find unmerged configurations
become: yes
ansible.builtin.command: >
/usr/bin/find /etc
-name '*.dpkg-*'
-o
-name '*.ucf-*'
-o
-name '*.merge-error'
# Read-only command, never makes changes
changed_when: false
register: aptitude_unmerged_output
# Check for non-Debian packages
- name: Find non-Debian packages
ansible.builtin.command: >
/usr/bin/aptitude search
'?narrow(?installed, ?not(?origin(Debian)))'
# 0 (results found) and 1 (no results found) are expected exit
# statuses
failed_when: aptitude_non_debian_output.rc > 1
# Read-only command, never makes changes
changed_when: false
register: aptitude_non_debian_output
# Find non-Debian package sources
- name: Find non-Debian sources
ansible.builtin.shell: >
/usr/bin/apt-cache policy
|
awk '/^[ \t]+origin/ { if ($0 !~ /\.debian\.org$/) { print $2 } }'
# Read-only command, never makes changes
changed_when: false
register: non_debian_sources_output
- name: Store non-Debian sources as fact
ansible.builtin.set_fact:
non_debian_sources: "{{ non_debian_sources_output.stdout_lines }}"
# Check for obsolete packages
- name: Find obsolete packages
ansible.builtin.command: /usr/bin/aptitude search '~o'
# 0 (results found) and 1 (no results found) are expected exit
# statuses
failed_when: aptitude_obsolete_output.rc > 1
# Read-only command, never makes changes
changed_when: false
register: aptitude_obsolete_output
# Check for removed packages with configuration files left behind
- name: Find unpurged packages
ansible.builtin.command: /usr/bin/aptitude search '~c'
# 0 (results found) and 1 (no results found) are expected exit
# statuses
failed_when: aptitude_unpurged_output.rc > 1
# Read-only command, never makes changes
changed_when: false
register: aptitude_unpurged_output
- block:
# Build list of all Debian releases deployed and hosts with each
- ansible.builtin.set_fact:
hosts_debian_versions: >
{{
hosts_debian_versions |
combine(
{
hostvars[item].ansible_facts.distribution_release
+
' ('
+
hostvars[item].ansible_facts.distribution_version
+
')'
:
hosts_debian_versions[
hostvars[item].ansible_facts.distribution_release
+
' ('
+
hostvars[item].ansible_facts.distribution_version
+
')'
] | default([])
+
[item,]
}
)
}}
vars:
hosts_debian_versions: {}
loop: "{{ query('inventory_hostnames', 'all:!dummy') }}"
when: >
hostvars[item].get(
'ansible_facts',
{}
).get(
'distribution_release',
None
)
# Build list of all hosts with unmerged configuration files
- ansible.builtin.set_fact:
hosts_unmerged_configs: >
{{
hosts_unmerged_configs
+
[
item
+
' ('
+
hostvars[item].aptitude_unmerged_output.stdout_lines
|
length
|
string
+
')',
]
}}
vars:
hosts_unmerged_configs: []
loop: "{{ query('inventory_hostnames', 'all:!dummy') }}"
when: >
hostvars[item].get(
'aptitude_unmerged_output',
{}
).get(
'stdout_lines',
[]
)
|
length
> 0
# Build list of all hosts with non-Debian packages
- ansible.builtin.set_fact:
hosts_non_debian_packages: >
{{
hosts_non_debian_packages
+
[
item
+
' ('
+
hostvars[item].aptitude_non_debian_output.stdout_lines
| length | string
+
')',
]
}}
vars:
hosts_non_debian_packages: []
loop: "{{ query('inventory_hostnames', 'all:!dummy') }}"
# Aptitude returns an error status when there are no
# results, so use that to tell if there are obsolete
# packages or not.
when: >
hostvars[item].get(
'aptitude_non_debian_output',
{}
).get(
'rc',
None
)
== 0
# Build a list of all non-Debian sources
- ansible.builtin.set_fact:
all_non_debian_sources: >
{{
all_non_debian_sources
+
hostvars[item].non_debian_sources
|
unique
}}
vars:
all_non_debian_sources: []
loop: "{{ query('inventory_hostnames', 'all:!dummy') }}"
when: >
hostvars[item].get('non_debian_sources', [])
|
length
> 0
# Build a list of which hosts have which non-Debian sources
- ansible.builtin.set_fact:
hosts_non_debian_sources: >
{{
hosts_non_debian_sources
|
combine({
item[1]
:
(
hosts_non_debian_sources[item[1]] | default([])
+
[item[0],]
)
|
unique
})
}}
vars:
hosts_non_debian_sources: {}
loop: >
{{
query('inventory_hostnames', 'all:!dummy')
|
product(all_non_debian_sources)
}}
when: >
item[1] in hostvars[item[0]].get('non_debian_sources', [])
# Build list of all hosts with obsolete packages
- ansible.builtin.set_fact:
hosts_obsolete_packages: >
{{
hosts_obsolete_packages
+
[
item
+
' ('
+
hostvars[item].aptitude_obsolete_output.stdout_lines
| length | string
+
')',
]
}}
vars:
hosts_obsolete_packages: []
loop: "{{ query('inventory_hostnames', 'all:!dummy') }}"
# Aptitude returns an error status when there are no
# results, so use that to tell if there are obsolete
# packages or not.
when: >
hostvars[item].get(
'aptitude_obsolete_output',
{}
).get(
'rc',
None
)
== 0
# Build list of all hosts with configured but removed packages
- ansible.builtin.set_fact:
hosts_unpurged_packages: >
{{
hosts_unpurged_packages
+
[
item
+
' ('
+
hostvars[item].aptitude_unpurged_output.stdout_lines
| length | string
+
')',
]
}}
vars:
hosts_unpurged_packages: []
loop: "{{ query('inventory_hostnames', 'all:!dummy') }}"
# Aptitude returns an error status when there are no
# results, so use that to tell if there are obsolete
# packages or not.
when: >
hostvars[item].get(
'aptitude_unpurged_output',
{}
).get(
'rc',
None
)
== 0
- ansible.builtin.debug:
msg: "{{ msg.split('\n') }}"
vars:
msg: |
Debian versions and their hosts:
{%
for
version, hosts
in
(
hosts_debian_versions
|
default({})
).items()
%}
- {{ version }}: {{ hosts }}
{% endfor %}
{% if hosts_non_debian_sources | default({}) | length > 0 %}
Non-Debian sources and their hosts:
Non-debian.org origins can be seen with:
/usr/bin/apt-cache policy | awk '/^[ \t]+origin/ { if ($0 !~ /\.debian\.org$/) { print $2 } }'
{% for source, hosts in hosts_non_debian_sources.items() %}
- {{ source }}: {{ hosts }}
{% endfor %}
{% else %}
:-) No hosts have non-Debian package sources.
{% endif %}
{% if hosts_unmerged_configs | default([]) | length > 0 %}
Hosts with unmerged configuration files:
Files can be found with:
/usr/bin/find /etc -name '*.dpkg-*' -o -name '*.ucf-*' -o -name '*.merge-error'
{% for host in hosts_unmerged_configs %}
- {{ host }}
{% endfor %}
{% else %}
:-) No hosts have unmerged configuration files.
{% endif %}
{% if hosts_non_debian_packages | default([]) | length > 0 %}
Hosts with non-Debian packages:
Packages can be found with:
/usr/bin/aptitude search '?narrow(?installed, ?not(?origin(Debian)))
{% for host in hosts_non_debian_packages %}
- {{ host }}
{% endfor %}
{% else %}
:-) No hosts have non-Debian packages.
{% endif %}
{% if hosts_obsolete_packages | default([]) | length > 0 %}
Hosts with obsolete packages:
Packages can be found with:
/usr/bin/aptitude search '~o'
{% for host in hosts_obsolete_packages %}
- {{ host }}
{% endfor %}
{% else %}
:-) No hosts have obsolete packages.
{% endif %}
{% if hosts_unpurged_packages | default([]) | length > 0 %}
Hosts with redundant configuration files:
Removed packages with configurations left behind can be found with:
dpkg -l | awk '/^rc/ { print $2 }'
{% for host in hosts_unpurged_packages %}
- {{ host }}
{% endfor %}
{% else %}
:-) No hosts have redundant configuration files.
{% endif %}
run_once: true
...