Managing software start getting interesting in High-Performance Computing (HPC) when clusters become heterogeneous. One way to manage this, using a common shared filesystem, is to allow the software management tool to detect the current platform and make the appropriate software available. This post shows how to do this with the Lmod tool.

A homogeneous cluster contains the same (or at least the the same generation of) processor in every node, although the amount of memory may vary, and therefore the same software, optimised for the common processor platform, will generally work just as well on any of the nodes. When this stops being true, one enters a world that can lead to considerable pain trying to manage many builds of the same software optimised for each of the platforms. Another way is to carve up the filesystem and effectively mirror the software, with each mirror having the build optimised for that platform but this potentially loses benefits of sharing software where optimisation is not possible (e.g. pre-compiled, othen commercial, applications) and is really just shifting the complexity from being a file-management problem to an infrastructure one.

Solving this as a file-management problem on a shared filesystem, we simply want to create a per-platform directory each of which contains a software tree for that plaftorm and can have a sibling directory that contains software for all platforms. On my cluster, the all-platforms directory is called ‘handbuilt’, as the software in there invariably is where as the platform-specific software is all managed with another tool called EasyBuild(but that’s a seperate blog post…).

Get on with it!

Without futher ado, this is how to detect the plaform and stuff that information into environment variables with Lmod - this is a rework, from memory, of doing the same thing in TCL with the venerable Envrionemnt Modules system but updated for the more feature-rich Lmod in its native Lua.

So, in a new module file (I called mine hpc_environment, so these went in hpc_environment/1.lua somewhere in the module path for version 1 of the module):

1. Define the Lua variables

As we will be exporting these later, it is important to put in sensible failure values in case detection fails:

-- OS variables
local os_fullname = "UNKNOWN"
local os_shortname = "UNKNOWN"
local os_version = "UNKNOWN"
local os_version_major = "UNKNOWN"
local os_distribution = "UNKNOWN"
-- Architecture variables
local arch_platform = "UNKNOWN"
local arch_cpu_fullname = "UNKNOWN"
local arch_cpu_shortname = "UNKNOWN"
local arch_cpu_compat = ""

A quick run-down of the variables purpose:

  • os_fullname - the name and version of the opeating system (e.g. “Debian-10”)
  • os_shortname - a shorthand version of the OS (e.g. “Deb-10” for Debian)
  • os_version - the full version number of the OS (e.g. “7.8.2003” for that CentOS release)
  • os_version_major - just the major part of the OS version (e.g. “7” for the above CentOS release)
  • os_distribution - the full name of the distribution (e.g. “Ubuntu”, “RedHatEnterpriseLinux`)
  • arch_platform - The CPU platform (e.g. “intel”, “amd”, “power” etc.)
  • arch_cpu_fullname - The full (human-friendly) name for the CPU generation (e.g. “AMD EPYC Zen”)
  • arch_cpu_shortname - A shorthand version of the CPU generation (e.g. “zen”)
  • arch_cpu_compat - A space-seperated list of (typically older) compatible (i.e. subset of the same instruction set) CPUs

2. Some utility functions

These simple utility functions will be useful later on:

function file_exists(file_name)
    local file_found = io.open(file_name, "r")
    if file_found == nil then
        return false
    else
        return true
    end
end

function get_command_output(command)
    -- Run a command and return the output with whitespace stripped from the end
    return string.gsub(capture(command), '%s+$', '')
end

3. OS detection

This function will detect the OS and populate the variables we defined above. Anything ‘Red Hat compatible’ is treated as being the same as the version of Red Hat it is compatible with. Ubuntu and Debian are sufficiently different that there is no similar compatible/comparable versions between then. My original TCL version did fall-back to reading files (e.g. redhat-release and debian_version) from /etc if lsb_release was not installed but I believe this was to support older distributions - nothing I have to hand is missing the lsb_release utility so this fails if it is not there instead:

function detect_os()
    -- Detect the operating system
    local short_version_table = {
        Scientific = "EL",
        RedHatEnterpriseServer = "EL",
        CentOS = "EL",
        RedHatEnterprise = "EL",
        Debian = "Deb",
        Ubuntu = "Ubt"
    }

    if file_exists("/usr/bin/lsb_release") then
        os_version = get_command_output("lsb_release -r -s")
        os_distribution = get_command_output("lsb_release -i -s")

        os_fullname = os_distribution .. "-" .. os_version

        if os_distribution == "Ubuntu" then
            -- Ubuntu is unique in that the version after the dot is not a minor release number
            os_version_major = os_version
        else
            os_version_major = string.sub(os_version, string.find(os_version, '^%d+'))
        end
        os_shortname = short_version_table[os_distribution] .. "-" .. os_version_major
    else
        LmodError("No lsb_release command in /usr/bin - this version of the module has no fallback detection methods.")
    end
end

4. Architecture detection

This is the big, complicated one. Basically from the inforamtion in /proc/cpuinfo we need to infer the family and generation of processor, which as you will see is less than straightforward in some cases. For simplicity, this version only detects those architectures I need to detect in Azure virtual machines.

For the sake of keeping things simple, I strongly advise removing defunct entries as you add new ones to avoid this growing to be too complex.

function detect_arch()
    -- Detect architecture information
    local cpu_family = get_command_output("grep -m1 '^cpu family[[:space:]:]\\+' /proc/cpuinfo | sed 's/^cpu family[[:space:]:]\\+\\([0-9]\\+\\)$/\\1/'")
    local cpu_model = get_command_output("grep -m1 '^model[[:space:]:]\\+' /proc/cpuinfo | sed 's/^model[[:space:]:]\\+\\([0-9]\\+\\)$/\\1/'")
    local cpu_flags = get_command_output("grep -m1 '^flags[[:space:]:]\\+' /proc/cpuinfo | sed 's/^flags[[:space:]:]\\+\\(.\\+\\)$/\\1/'")

    -- We need to detect for Azure:
    --   Dv3: Haswell, Broadwell, Skylake or Cascade Lake
    --   Fsv2: Skylake or Cascade Lake
    --   NCv2: Broadwell
    --   NCv3: Broadwell
    --   HB: AMD Zen 1
    --   HBv2: AMD Zen 2
    --   HC: Skylake
    -- I also have an IvyBridge system as my home lab, so detect that

    -- cpu_family and cpu_model are integer codes, so we need some lookup-tables (made by examining /proc/cpuinfo on available systems)
    -- Treat Broadwell as being Haswell due to compatible instructon sets
    local cpu_table = {
        ["6"] = {
            ["58"] = "san", -- IvyBridge
            ["63"] = "has", -- Haswell
            ["71"] = "has", -- Broadwell
            ["79"] = "has", -- Broadwell
            ["86"] = "has", -- Broadwell
            ["85"] = "sky", -- Skylake or Cascade Lake
        },
        ["23"] = {
            ["1"] = "zen", -- AMD Zen 1
            ["49"] = "zen2", -- AMD Zen 2
        },
    }

    -- Only really care about the family to detect Intel vs AMD (grouped the cpu_table by it for my benefit)
    local cpu_plat_table = {
        ["6"] = "intel",
        ["23"] = "amd",
    }

    -- Human friendly CPU names
    local cpu_names = {
        san = 'SandyBridge of IvyBridge',
        has = 'Haswell or Broadwell',
        sky = 'Skylake',
        cas = 'Cascade Lake',
        zen = 'AMD EPYC Zen',
        zen2 = 'AMD EPYC Zen 2',
    }

    -- List of compatible architectures (i.e. subset of same instruction set)
    local backward_compat = {
        sky = {'has'},
        cas = {'sky', 'has'},
        zen2 = {'zen'},
    }

    local cpu_family_name = cpu_table[cpu_family][cpu_model]

    if cpu_family_name == "sky" then
        -- Skylake with avx512 VNNI is Cascade Lake
        -- see: https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512
        if string.find(cpu_flags, 'avx512_vnni') then
            cpu_family_name = 'cas'
        end
    end

    arch_platform = cpu_plat_table[cpu_family]
    arch_cpu_shortname = cpu_family_name
    arch_cpu_fullname = cpu_names[arch_cpu_shortname]
    if backward_compat[arch_cpu_shortname] ~= nil then
        arch_cpu_compat = table.concat(backward_compat[arch_cpu_shortname], ' ')
    end
end

5. The glue

Gluing these methods together, at the end of our module, we can create our useful environment variables. I prefixed them all with HPC_ (to help avoid clashes and make them easy to locate), followed by OS_ for the operating system information and ARCH_ for the platform information:


-- Detection is (relatively) expensive to do, so only do it if needed
-- (Lmod will quite happily unset these variables on unload even if the values don't match)
if mode() == "load" then
    detect_os()
    detect_arch()
end

-- Export the OS variables
setenv("HPC_OS_FULL", os_fullname)
setenv("HPC_OS_SHORT", os_shortname)
setenv("HPC_OS_VERSION", os_version)
setenv("HPC_OS_VERSION_MAJOR", os_version_major)
setenv("HPC_OS_DIST", os_distribution)
-- Export the architecture variables
setenv("HPC_ARCH_CPU_FULLNAME", arch_cpu_fullname)
setenv("HPC_ARCH_CPU_SHORTNAME", arch_cpu_shortname)
setenv("HPC_ARCH_CPU_COMPAT", arch_cpu_compat)
setenv("HPC_ARCH_PLATFORM", arch_platform)

Putting it to good use

With this module loaded, you will have a load of useful variables at your disposal:

$ module load hpc_environment
$ env | grep HPC_
HPC_OS_DIST=CentOS
HPC_OS_SHORT=EL-7
HPC_OS_FULL=CentOS-7.8.2003
HPC_ARCH_PLATFORM=intel
HPC_ARCH_CPU_SHORTNAME=sky
HPC_OS_VERSION_MAJOR=7
HPC_ARCH_CPU_FULLNAME=Skylake
HPC_OS_VERSION=7.8.2003
HPC_ARCH_CPU_COMPAT=has

If we use these same names for the platform-specific folders, for example for Red Hat 7 compatible OSs on Skylake we call the folder EL-7-sky and within it create a modules directory for the modules for that environment’s optimised software then Lmod can be easily told to use this directly from the set environment variables:

module use /path/to/applications/$HPC_OS_SHORT-$HPC_ARCH_CPU_SHORTNAME/modules

This same command, provided the folder convention is followed, will work for all platforms you support.

The same can be done in a module, which if part of your system modules will result in the correct platform-specific modules being loaded when you run module reset or when modules is initially loaded (usually via /etc/profile.d):

-- Because we use the variables from this within the paths, we need this dependency processed early on load and late on unload
if mode() == "load" then
    depends_on('hpc_environment')
end

append_path('MODULE_PATH', '/path/to/applications/' .. os.getenv('HPC_OS_SHORT') .. '-' .. os.getenv('HPC_ARCH_CPU_SHORTNAME') .. '/modules')

for compatible_arch in string.gmatch(os.getenv('HPC_ARCH_CPU_COMPAT'), "[^%s]+") do
    append_path('MODULE_PATH', '/path/to/applications/' .. os.getenv('HPC_OS_SHORT') .. '-' .. compatible_arch .. '/modules')
end

if mode() == "unload" then
    depends_on('hpc_environment')
end