This post is about something I created and am genuine quite proud of. It creates a graphical progress bar to feedback on the progress of the %post section(s) in a kickstart installation script. I have used it with Red Hat and Rocky Linux distributions. It is an alternative to forcing a text install just to be able to programmatically chvt to the log in order to display custom script progress. It might seem simple but it took a lot of work and testing to get to a reliable solution.

To see what it looks like, skip to the demo and screenshots section.

Slight aside, the company name is “Red Hat” (with a space) - I was writing “RedHat” (although it looks like it should have been “redhat” lower case, from the logo but as far as I can tell outside the logo it has always been “Red Hat”). This animation explains the logo changes:

redhat to Red Hat

It works by using zenity to create a progress dialogue in the installer’s environment that can then be updated from within the chrooted post-install script via a named socket.

“Why bother?”, you may ask. This has two major utilities in my world:

  1. Feedback for long running post scripts.

    My current modus operandi is to use kickstart or preseed to perform a minimal operating system [OS] install then use configuration management ( such as SaltStack, Puppet or Ansible - others are available, these are the 3 currently in my mix) to configure the installed OS, including installing and configuring additional software. This takes a long time and feeding back where in the process it is helps reassure it has not become stuck.

  2. Prompting a person to take action.

    For SaltStack and Puppet, the new machine’s key or certificate must be accepted or signed (respectively) on the master server. The feedback message can be used to tell the person doing the install when this is ready to be done (although, as I describe in another blog post, accepting the certificate or key when it appears can be automated). There are other approaches, such as preseeding the keys & pre-signing certificates externally to the tool and dropping the already accepted/signed files in the right place but I have not yet explored this (or thought through if I think it is a good idea).

The preamble

In order for this to work, the zenity program needs to be started in the installer environment with the DISPLAY environment correctly set and a named socket, pre-created correctly, within the chroot environment’s filesystem for it to be able to send feedback back to the progress bar. This is done as a systemd unit, which is then started, to make it persist beyond the end of the %post block that runs zenity (otherwise Anaconda will kill the process at the end of the block) - this is one of the things that took a lot of effort to get working smoothly. zenity is configured to exit when the progress bar reaches 100%, so only send this value at the end of the script and assume any message sent with it will not be seen (but it may end up in the log, e.g. if command echoing it turned on, so can still be helpful to specify a message).

The copying of files into the chroot (cp /tmp/kickstart_pre_config.bash /mnt/sysroot/tmp/) is not part of the progress bar, but I left it in to illustrate this piece as well. In the full kickstart, some environment detection is done in %pre which is used to create dynamic parts of the kickstart that are later included as well as populate the file you see copied with the results of the detection as Bash variables so they can be used (avoiding needed to re-do the detection) by the %post script within the chroot.

The %post --nochroot script is itself run in a subshell (via ( ... )) piped into a separate zenity process to show feedback of this stage too. The output of the subshell (i.e. what is printed) needs to be in zenity’s input format: percentage\n# message.

Without further ado, here is the %post --nochroot script:

# Copy files into chroot and start graphical feedback window in install
# environment (i.e. outside chroot)
%post --nochroot
# This should possibly have more error checking/handling
set -e # Abort on any error

# For zenity, which provides feedback and a progress bar.
export DISPLAY=:1

# The echo statements within the subshell are of the form:
# percentage\n# message
# See zenity manual for more details
(
  echo -e "20\n# Copying pre-variables to new install's /tmp..."
  cp /tmp/kickstart_pre_config.bash /mnt/sysroot/tmp/

  echo -e "40\n# Creating post script message socket daemon..."
  cat - >/etc/systemd/system/post-graphical-feedback.service <<EOF
[Unit]
Description=graphical feedback dialog with named pipe inside chroot

[Service]
Type=exec
User=root
Group=root
Environment=SOCKET=/mnt/sysroot/tmp/graphical-feedback
Environment=DISPLAY=${DISPLAY}
# Mush use bash for '||' (or) to work
ExecStartPre=/bin/bash -c '[ -e \${SOCKET} ] || /bin/mknod \${SOCKET} p'
# Must use bash for streams (pipe) to work.
# --auto-close will cause zenity to exit when a progress of 100% is
# send to it
ExecStart=/bin/bash -c 'cat \${SOCKET} | /bin/zenity --width=600 --progress --auto-close --no-cancel --title="Post install script progress" --text="Waiting for chrooted post script to start..."'
ExecStop=/bin/rm \${SOCKET}
Restart=on-failure
EOF

  echo -e "60\n# Reloading systemd..."
  systemctl daemon-reload

  echo -e "80\n# Starting new feedback service..."
  systemctl start post-graphical-feedback

  echo -e "100\n#End of non-chrooted post."
# Without auto-close, waits for user to press 'ok'
) | zenity --progress --no-cancel --auto-close
%end

Updating the progress feedback

To update the progress bar from within the chrooted %post, one just need to write messages in the correct format (percent\n# Message) to the named pipe. The easiest (and most reliable) way I found to do this was to create a file descriptor [fd], fd 3 (the standard streams are 0 (stdin), 1 (stdout) and 2 (stderr)), that redirects to the pipe with exec 3>socket where socket is the named pipe.

After this, standard echo commands can be used to send messages to the pipe of the form echo -e "percent\n# Message" >&3 (-e tells echo to expand the \n newline character, rather than print a literal “backslash n”). Although strictly unnecessary, as Anadonda will destroy the %post shell and any background processes etc. when the script end, for completeness I close the file descriptor with exec 3>&- at the end.

%post --erroronfail --log=/var/log/kickstart-post.log
# Standard bash safety - abort on error, use of unset variables is an
# error, disable accidental globbing, enable printing of each command
# being run (very useful for debugging this from kickstart-post.log file) and
# erroring commands in a pipe treated as a failure.
set -eufxo pipefail

# Setup file descriptor '3' to pipe to the graphical feedback socket
exec 3>/tmp/graphical-feedback
echo -e "0\n# Starting post-install..." >&3

# ... rest of post script goes here, printing percentages and progress
# messages as it goes using the same `echo -e` recipe ...

echo -e "100\n# All post script tasks completed." >&3
# Close file descriptor 3
exec 3>&-
%end

I did not bother to explicitly stop the service or delete the socket for two reasons:

  1. When the install is finished, the installer’s systemd should stop the service which will remove the socket.
  2. /tmp is cleared after a reboot, so it will be removed during the systems first boot anyway.

Prompting for Puppet’s certificate to be signed

This is another bit that became more complicated than it first looked. The idea is that, once the Puppet agent is installed and configured, run puppet agent and tell it to wait for its certificate to be accepted on the master. At this point, update the message to say that it is ready to be signed. Once the certificate is signed, automatically continue with the install. A similar recipe can be used to wait for Salt key signing to occur, it requires some tweaking of the minion configuration to get it to wait indefinitely for the signing to take place.

My first attempt at this simply assumed the certificate would be pending signing, so it displayed a message saying “Starting Puppet” then, when if saw the “waiting for certificate” message in Puppet’s output it changed the message to “Please go sign the certificate” then, when it saw the certificate had been retrieved updated the message to “Running puppet for the first time, please be patient”. The problem with this was that my original script assumed this progression would always happen but if the certificate was signed too quickly (e.g. by automation), Puppet would never output the waiting message and so my script did not change the message from “Starting Puppet” until Puppet finished (making it look stuck, precisely what the graphical feedback was intended to avoid).

In the end, I had to read every line of output and force an update regardless of the script’s view of the signing status when the certificate is received (to clear the initial “Starting” message, which is intended to be displayed until the certificate is ready).

Originally I set the --waitforcert value, which is how frequently the agent polls the master to see if its certificate is signed, to 120 (2 minutes) but this cause a bit of a wait once the certificate was signed in most cases. Later, I set it to 5 seconds and at this value the agent run appears to continue almost instantly once the certificate is signed. The load polling every 5 seconds would add to the puppet master was something I considered but, on reflection, I imagine even hundreds of servers all being installed simultaneously and polling at 5 second intervals should not cause problems for a mature product like Puppet.

Lastly, at the end of the script, I have left in the block of code that determines if Puppet made any changes and re-runs it (unless any errors also occurred) to verify the system is now in the correct state (no changes were made). In the end, I removed this as states fighting each one (one changes it one way, another changes it back in a cycle) were too common so this was causing the install to fail (failure of the %post causes Anaconda to declare the install has failed) but I like the idea of checking the systems state is stable. What I might propose is that we introduce continuous integration tests, to include verifying a 2nd run of Puppet results in no changes (i.e. the configuration is stable), on the Puppet code and then seek to reintroduce this functionality.

echo -e "60\n# Starting Puppet..." >&3
# Run puppet agent once, waiting for certificate (checking every 5
# seconds to see if it's been signed), to perform initial configuration.

# Disable abort-on-error - puppet will exit with error code on success.
set +e
/opt/puppetlabs/puppet/bin/puppet agent --test --waitforcert 5 | (
  cert_pending=0
  # This will keep "Starting Puppet..." message up until it is truly
  # waiting for its certificate (i.e. after CSR has been generated and
  # accepted by server).
  last_cert_state=$cert_pending
  while read input
  do
    # We still want to log all of puppet's output
    echo $input
    # Puppet annoyingly prints colour codes even when its output isn't
    # a colour supporting terminal. Use sed to strip them so we can
    # compare the message.
    if [[ $( echo $input | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mK]//g" ) == "Notice: Did not receive certificate" ]]
    then
      cert_pending=1
    # This happens after the certificate is signed - force a message
    # update in case we missed the signing (if it's automatically
    # signed by Ansible, for example).
    elif [[ $( echo $input | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mK]//g" ) == "Info: Caching certificate for $( hostname -f | tr '[:upper:]' '[:lower:]' )" ]]
    then
      cert_pending=3
    else
      cert_pending=0
    fi

    # Only update message if state has changed
    if [[ $cert_pending -ne $last_cert_state ]]
    then
      if [[ $cert_pending -eq 1 ]]
      then
        echo -e "61\n# Puppet is waiting for its certificate to be approved on the Puppet master. Please do this now..." >&3
      else
        echo -e "62\n# Puppet has received its certificate and is applying state for the first time (this will take a while)..." >&3
      fi
      # This is now the last state the message was updated.
      $last_cert_state=$cert_pending
    fi
  done
)
puppet_status=$?  # Save exit code from agent
set -e  # Restore abort-on-error behaviour

echo -e "65\n# Checking puppet output..." >&3
# From the Puppet docs (paraphrased slightly):
# > --test includes --detailed-exitcodes by default and exists with one
# > of these statuses:
# > 0: Success, no changes or failures. System already in desired state.
# > 1: Run failed or wasn't attempted.
# > 2: Run succeeded and changes made.
# > 4: Run succeeded but some resources failed.
# > 6: Run succeeded but there were both changes and failures.
# For us:
# - status 0 or 2 is fine (system should now be in correct state)
# - any other status means some sort of error occurred and we want to
#   indicate a failure of the %post script to Anaconda
if [[ $puppet_status -ne 0 ]] && [[ $puppet_status -ne 2 ]]
then
    echo -e "70\n# Puppet failed - check /var/log/install_post.log for why..." >&3
    echo "Puppet agent run failed - check above and puppet master's report for error messages." >&2
    exit $puppet_status  # Use Puppet Agent's error status to terminate
elif [[ $puppet_status -eq 2 ]]
then
    echo -e "80\n# Puppet made changed, running again to check state is now consistent..." >&3
    echo "Puppet agent made changes - re-running to check state is now correct." >&2
    # No need to disable abort-on-error this time - anything other than
    # no changes (0) should cause a failure here (as this is to confirm
    # system is now in correct state).
    /opt/puppetlabs/puppet/bin/puppet agent --test
fi

Demonstration and screenshots

It’s all very well talking about a graphical thing but it’s nice to show of just how well this integrates with Anaconda. Here’s some screenshots, and below the kickstart used to generate them.

Screenshots

Demo screenshot 1 - creating the daemon

Demo screenshot 2 - installing configuration management tool

Demo screenshot 3 - prompt to sign certificate

Demo screenshot 4 - applying configuration

Demo screenshot 5 - checking result of configuration

Demo screenshot 6 - tidying up at the end

Kickstart to generate screenshots

This is a small kickstart to do a fully automated minimal install, demonstrating the graphical feedback in the process:

#version=RHEL8
# Use graphical install - some options are unavailable in text mode
graphical

%packages
# minimal-environment and kexec-tools are what get installed by
# selecting "minimal install" in the interactive installer.
@^minimal-environment
kexec-tools
# Remove unnecessary extra packages
# ...wireless firmwares
-iwl*-firmware*
%end

# Keyboard layouts
keyboard --vckeymap=gp --xlayouts='gb'
# System language
lang en_GB.UTF-8

# EULA accepted at an organisational level - no need to accept on every
# install.
eula --agreed

# Network information
network --device=link --hostname=rocky-8-kickstart-test.vm.home.entek.org.uk --bootproto=dhcp --mtu=8000 --noipv6 --activate
url --url=https://lon.mirror.rackspace.com/rocky/8.7/BaseOS/x86_64/os/

zerombr  # Initialise any unitialised disks
clearpart --none --initlabel --disklabel=gpt
# Anaconda complains there is no EFI partition if /boot/EFI is used, although other distributions used the capitalised version.  *sigh*
part /boot/efi --fstype="efi" --size=512 --ondisk=sda
# For a ephermal VM, don't bother with LVM. Do want it on anything "real" for snapshotty goodness etc.
part / --fstype="xfs" --size=3072 --grow --ondisk=sda

# System timezone, use NTP - it is important for generating puppet
# certificates and authentication that the time is correct during
# install. Unqualified name should pick up correct host for
# network via DHCP search domain.
timezone Europe/London --isUtc --ntpservers=ntp

# Root password is "Root123" (without quotes) - do not use this in production!
# Encrypted password hashes can be generated with `openssl passwd -6`
rootpw --iscrypted $6$g1JIvz5A8RItJOE3$BY1Sz5Ql0ywmW42mMkP.saJZgW2Os9gqclfrI//Bjr21CqPai32291T7SYu.dYhM.qMSdk9WB45.kVuxUlzY3.

# Start graphical feedback window in install environment (i.e. outside chroot)
%post --nochroot
# This should possibly have more error checking/handling
set -e # Abort on any error

# For zenity, which provides feedback and a progress bar.
export DISPLAY=:1

# The echo statements within the subshell are of the form:
# percentage\n# message
# See zenity manual for more details
(
  echo -e "25\n# Creating post script messsage socket daemon..."
  cat - >/etc/systemd/system/post-graphical-feedback.service <<EOF
[Unit]
Description=graphical feedback dialog with named pipe inside chroot

[Service]
Type=exec
User=root
Group=root
Environment=SOCKET=/mnt/sysroot/tmp/graphical-feedback
Environment=DISPLAY=${DISPLAY}
# Mush use bash for '||' (or) to work
ExecStartPre=/bin/bash -c '[ -e \${SOCKET} ] || /bin/mknod \${SOCKET} p'
# Must use bash for streams (pipe) to work.
# --auto-close will cause zenity to exit when a progress of 100% is
# send to it
ExecStart=/bin/bash -c 'cat \${SOCKET} | /bin/zenity --width=600 --progress --auto-close --no-cancel --title="Post install script progress" --text="Waiting for chrooted post script to start..."'
ExecStop=/bin/rm \${SOCKET}
Restart=on-failure
EOF

  echo -e "50\n# Reloading systemd..."
  systemctl daemon-reload

  echo -e "75\n# Starting new feedback service..."
  systemctl start post-graphical-feedback

  echo -e "100\n#End of non-chrooted post."
# Without auto-close, waits for user to press 'ok'
) | zenity --progress --no-cancel --auto-close
%end

# Demo of graphical feedback
%post --erroronfail --log=/var/log/kickstart-post.log
# Standard bash safety - abort on error, use of unset variables is an
# error, disable accidental globbing, enable printing of each command
# being run (very useful for debugging this from ks-post.log file) and
# erroring commands in a pipe treated as a failure.
set -eufxo pipefail

# Setup stream '3' to pipe to the graphical feedback socket
exec 3>/tmp/graphical-feedback
echo -e "0\n# Starting post-install..." >&3

echo -e "10\n# This is a demo of the post-script progress feedback..." >&3
sleep 3

echo -e "20\n# If this weren't a demo, I might be installing a configuration management tool..." >&3
sleep 10

echo -e "30\n# Now I might be configuring the tool..." >&3
sleep 7

echo -e "40\n# This might be where I run the tool for the first time..." >&3
sleep 5

echo -e "50\n# And I might pause here for you to sign the certificate/accept the key on the master..." >&3
sleep 10

echo -e "60\n# Before continuing and doing the configuration (which might take a while)..." >&3
sleep 20

echo -e "70\n# Now I might be checking the output of the tool to see if it worked..." >&3
sleep 10

echo -e "80\n# Maybe I am now doing some final tidying up, registering the system or doing more checks..." >&3
sleep 7

echo -e "80\n# Finally I could be deleting some temporary files, or something..." >&3
sleep 3

echo -e "100\n# All post script tasks completed." >&3
# Close file descriptor 3
exec 3>&-
%end

# Automatically reboot on successful completion
reboot --eject

Kickstart log file

This is what the contents of the log file looks like, as you can see with set -x the progress messages are helpfully shown which significantly aids troubleshooting where the post problems happen, when debugging:

+ exec
+ echo -e '0\n# Starting post-install...'
+ echo -e '10\n# This is a demo of the post-script progress feedback...'
+ sleep 3
+ echo -e '20\n# If this weren'\''t a demo, I might be installing a configuration management tool...'
+ sleep 10
+ echo -e '30\n# Now I might be configuring the tool...'
+ sleep 7
+ echo -e '40\n# This might be where I run the tool for the first time...'
+ sleep 5
+ echo -e '50\n# And I might pause here for you to sign the certificate/accept the key on the master...'
+ sleep 10
+ echo -e '60\n# Before continuing and doing the configuration (which might take a while)...'
+ sleep 20
+ echo -e '70\n# Now I might be checking the output of the tool to see if it worked...'
+ sleep 10
+ echo -e '80\n# Maybe I am now doing some final tidying up, registering the system or doing more checks...'
+ sleep 7
+ echo -e '80\n# Finally I could be deleting some temporary files, or something...'
+ sleep 3
+ echo -e '100\n# All post script tasks completed.'
+ exec