Singularity: User Defined Software Environments (Version 3.8+)

Preliminaries

Singularity is a software environment which allows you to downlod, create, modify, and run containers from various sources (including Dockerhub) on a multiuser Linux system. With Singularity you can package an entire container (image of the userspace portion of a Linux root file system including your software and all necessary prerequisites: programs, libraries etc.) in a single, portable so-called SIF file, e.g. my-container.sif. This file can then be transferred and run by a non-privileged user on a HPC cluster almost like a normal program.

Compared to other container environments, Singularity's focus is on integration, not isolation. In Singularity, by default the only element of isolation is that a container has its own root file system, allowing you to use whatever Linux distribution and software packages best fit your needs. All other resources (your files, devices, networking, ...) can be used by a container in the same way as with a normal program running on the host, including arbitrary command line + arguments every time you start your container.

Since a container contains all software components needed to run your application, you are, provided a few requirements...

Kernel should be >= 3.8 - recommended >= 3.18 for full functionality;
Singularity must be installed;
compatible MPI installation if you use MPI;
compatible GPU drivers if you use GPUs

... are met on the host, completely independent from its OS and software setup.

As for performance, programs running in a container are isolated just by manipulations of name spaces (as opposed to a virtual machine's OS and emulation layers). Thus running a program in a container provably causes zero overhead compared to running it directly in the host environment.

Singularity Workflow - Basics

The following description gives a brief summary of the necessary steps. Please refer to the Singularity User Guide and the Release Notes for details.

Note: You need root privileges to create and modify containers, but not to run containers.

Preparing Your Workstation

To create and modify Singularity containers, you need a PC or virtual machine (we recommend Virtualbox) with a recent Linux installation (we recommend Ubuntu).

Install Singularity and its prerequisites (including the required version of the Golang compiler) on your Linux machine as described in the Install Guide.

We suggest to install Singularity to a nonstandard location (--prefix option of mconfig; default: /usr/local), so it will be easier to cleanly remove the current version before installing a new one in the future. The make uninstall of Singularity 2 is no longer available.

Example

./mconfig --prefix=/usr/local/singularity

will allow you to purge the installation by issuing

sudo rm -rf /usr/local/singularity

before installing another version.

Do not forget to add the PREFIX/bin directory to your PATH variable in $HOME/.bashrc

export PATH="/usr/local/singularity/bin:$PATH"

You will also need to update the secure_path attribute in /etc/sudoers (edit with sudo visudo) to include the Singularity binaries.

Defaults secure_path="/usr/local/singularity/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

Building Your Container

Containers may be downloaded from a variety of sources (including Dockerhub) or built from scratch. If a container satisfying all your software needs for a particular application exists, you can simply pull and start using this container.

Else, you start by downloading a suitable basic Linux OS container, e.g. from Dockerhub, into a sandbox directory, install missing OS packages and your own software, finally convert the sandbox to a SIF file.

Once you are satisfied with your build, you should automate the build by adding all necessary steps to a Singularity definition file to make sure that your build procedure can be reproduced.

Note: there is no guarantee that the base container you download from external sources such as Dockerhub will be available indefinitely, so if reproducibility is a concern, make sure you keep the SIF-files which you use for production in a safe place.

For details, see the Build Instructions in the User Guide.

Running Your Container

You can run your container on your local workstation for testing purposes as well as on any Linux multiuser system that has a working Singularity installation. Containers will run with your user credentials and privileges; you do not need root privileges to run containers.

Transfer your container to a directory on target system that has enough space available (UIBK: /scratch/user) using scp, e.g.

scp my-container.sif user@leoX.uibk.ac.at:/scratch/user

Log on to the HPC system and add the Singularity installation to your PATH (UIBK: module load singularity/3.x).

For an interactive shell, issue

singularity shell my-container.sif

To run a command installed in the container, issue

singularity exec my-container.sif my-program arg1 arg2 ...

Singularity Workflow - Example

The following example should serve as a rudimentary starting point for building your own container, assuming that you have installed Singularity 3.x on your workstation or virtual machine.

On your PC, download and create a writable copy of a container in a sandbox directory hierarchy. Example: get latest version of Ubuntu.
```
sudo singularity build --sandbox myubuntu docker://ubuntu:latest
```
This will create a directory myubuntu in your working directory and install a complete copy of the docker image. For available Docker images, see the Official Docker Repositories.
Customize your container: first start a shell within the container
```
sudo singularity shell --writable myubuntu
```
and then modify your installation, recording all steps for later automation:
1. Install software in the container using the guest's packet manager, download and install software from sources, copy necessary static data into the guest.
  Typically your first actions in a new Ubuntu container should be:
```
apt-get update
apt-get -y upgrade
apt-get -y install vim less gcc make
apt-get -y install mypackage-a mypackage-b ...
apt-get -y autoremove
```
  Use apt-get instead of apt because you will later want to automate these steps.
2. For installation of user data and software, use a directory which is unlikely to collide with other directories, e.g. /data.
```
mkdir /data
```
  Note that there currently seem to be no established structure and naming conventions for user defined contents in containers.
3. For optimal integration with the UIBK installation (and many other HPC sites), create a directory /scratch which will serve as a mount point of the HPC $SCRATCH directory.
```
mkdir /scratch
```
4. Note that the host's $HOME directory is mounted as $HOME in the guest, so any modifications to $HOME will happen outside the guest. This may introduce undesirable cross-contamination of configuration files, so some care is advised.
Repeat and test until you are satisfied. Convert the sandbox into an immutabe container image
```
sudo singularity build myubuntu.sif myubuntu
```

Copy this image to an HPC cluster

scp myubuntu.sif user@host.domain:/scratch/user

ssh user@host.domain
cd /scratch/user
module load singularity/3.x
singularity exec myubuntu.sif command [arg ...]

After testing, use your notes to prepare a definition file and completely automate the build process. For above example, a minimalistic definition file myubuntu.def might look like
```
Bootstrap: docker
From: ubuntu:latest

%setup
  mkdir $SINGULARITY_ROOTFS/scratch
  mkdir $SINGULARITY_ROOTFS/data

%files
  myubuntu.def /data/myubuntu.def

%post
  export LC_ALL=C.UTF-8
  export DEBIAN_FRONTEND=noninteractive
  ln -fs /usr/share/zoneinfo/Europe/Vienna /etc/localtime
  apt-get update
  apt-get -y upgrade
  apt-get -y install tzdata

  apt-get -y install apt-utils apt-file vim less wget openssh-client
  apt-get -y install build-essential

  apt-get -y install mypackage-a mypackage-b ...
  apt-get -y autoremove
  mkdir -p /data/bin

  cd /data
  ## include commands to download and install your
  ## software using --prefix=/data
  ## e.g.
  software=mypackage-x.y
  wget https://some-software.org/dist/${software}.tar.gz
  tar xf ${software}.tar.gz
  cd ${software}
  ./configure --prefix=/data
  make
  make install

%environment
  export LANG=C.UTF-8
  export PATH=/data/bin:$PATH
```
To directly build your SIF-File from this definition file, issue:
```
sudo singularity build myubuntu.sif myubuntu.def
```
The %setup and %post sections contain scriptlets that are run to build your container. Commands are echoed, and when a command fails, the build is aborted.

Commands in the %setup stage are executed as root in the host environment after the container has been bootstrapped. Keep this section as brief as possible.

In Singularity 3.x, the %files section is processed immediately after the %setup section (previous versions had this too late to be useful). This is a safe way to copy needed files into the container, e.g. a tarball of your software if it is not available from an external source. In above example, the definition file is added to the container as a means of self-documentation of the build process.

The %post section is run in the container environment after your container has been bootstrapped. This is the main part of container setup (updating the container OS, installing more packages, installing your software). Modify this section according to your needs. The language and time zone settings in above example serve to avoid errors while installing certain software components using Ubuntu's apt-get utility..

The %environment section contains commands to set up environment variables whenever your container is executed.

Note that software distributions may change, so keeping a buildfile is a good starting point for new, similar containers. Conversely, the immutable container image provides the vehicle for portability and reproducibility.

For more information, please refer to the Singularity Definition File Documentation.

Singularity Usage Notes for UIBK LEO Clusters

The following information is specific to the installation on our UIBK HPC clusters.

Singularity 3.x is installed on the LEO3e, LEO4, and LCC2 clusters. LEO3's legacy CentOS 6 is no longer supported.
Before using Singularity, issue
module load singularity/3.x
For security reasons, we aim at keeping Singularity current and will disable older versions as soon as the newest version is installed.
Our setup includes automatic mounting of $HOME (which is a default) and $SCRATCH (local setup). For the latter to work, directory /scratch to be used as the mount point must exist in your container.
Since your HOME directory is mapped into the container's $HOME, your container will use settings which you made in $HOME (such as .bashrc running commands meant to be run on the host and may not even exist in the container's context) and might inadvertently modify files in $HOME. If you do not like this, run your container with the -H option set to use a different directory for the container's HOME.
Likewise, make sure that software which you want to install into your container will not leave artefacts in your $HOME, or the container will not be portable to other systems.
You can run MPI code using Singularity, provided that the MPI versions inside the container and on the host match exactly.
Our setup of Singularity allows concurrent execution of multiple invocations of the same container using shared mount points. For MPI jobs that start many processes on the same host, this may save huge amounts of memory compared to the default non-shared mount points. Previously described workarounds using Singularity Instances to achieve this effect are no longer necessary.
Native OS distributions (e.g. ubuntu:latest) from Docker Hub are intentionally kept very lightweight, omitting many features of fully configured desktop OS installations. You will typically need to install more prerequisites than described in software installation instructions.
Warning: Make sure that you do not overwrite an existing SIF-file on the HPC system while there are still active jobs accessing such image. Else, your jobs will malfunction, and our sysadmins will be flooded with error messages.
Warning: Carefully inspect definition files from external sources for unintended effects. In particular, the %pre and %setup sections of the build file are run as root on your host system with no protection.

Case Study: Using MPI with Infiniband on the HPC clusters LEO3e, LEO4 (LeoX)

Version info: The following procedure has been adapted for Singularity 3.8 and Ubuntu 20.04LTS (focal fossa).

Singularity containers have access to /dev files, so MPI programs running in containers can use the Infiniband transport layer of OpenMPI. To run MPI programs in Singularity containers, we need the following prerequisites:

OpenMPI versions used in the host and the container must match exactly
Infiniband utilities and drivers must be installed in the container

What follows are - as an example - steps to get the Ohio State University MPI benchmark to run on LeoX in an Ubuntu container. Modify this workflow according to your needs. This is also an example for automating the setup of a slightly nontrivial container using the bootstrap method.

On your Linux workstation

Download and install Singularity. See above.

Prepare the following definition file mpitest-focal.def

Bootstrap: docker
From: ubuntu:focal

%setup
  mkdir $SINGULARITY_ROOTFS/scratch
  mkdir $SINGULARITY_ROOTFS/data

%files
  mpitest-focal.def /data/mpitest-focal.def

%post
  export LC_ALL=C.UTF-8
  export DEBIAN_FRONTEND=noninteractive
  ln -fs /usr/share/zoneinfo/Europe/Vienna /etc/localtime
  apt-get update
  apt-get -y upgrade
  apt-get -y install tzdata
  apt-get -y install apt-utils apt-file vim less wget openssh-client

  apt-get -y install build-essential openmpi-bin

  cd /data
  wget http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.3.2.tar.gz
  tar xf osu-micro-benchmarks-5.3.2.tar.gz
  cd osu-micro-benchmarks-5.3.2
  ./configure --prefix=/data CC=$(which mpicc) CXX=$(which mpicxx)
  make
  make install

  mkdir -p /data/bin
  find /data/libexec/osu-micro-benchmarks/mpi -type f | xargs ln -s -t /data/bin

%environment
  export LANG=C.UTF-8
  export PATH=/data/bin:$PATH

This definition file will pull a basic Ubuntu 20.04 (Focal) image from the Docker hub and will install some utilities, the development utilities, and MPI support. It will download (from the OSU web server) and install the OSU Benchmark in the container. Note that the OS running on LeoX is CentOS, but the container OS used in this example is Ubuntu.

Note: singularity help build will output a template to stdout which can be used to create your own definition file. Note that the percent signs of the section delimiters must be in column one and - as of version 3.9.1 and before - the sections are not in the order in which they are processed.

Still on your Linux workstation, build the container into a sandbox subdirectory named mpitest-focal using above definition file.
```
buildlog=build-$(date '+%Y%m%d-%H%M%S').log 
sudo true   # avoid password prompt in next background command
sudo singularity build --sandbox mpitest-focal mpitest-focal.def > $buildlog 2>&1 &
less $buildlog
```
Depending on your network connection, the build process will take several minutes. Watch the output. If errors occur, start an interactive shell in the container
```
sudo singularity shell -w mpitest-focal
```
and try to correct the problem, recording necessary changes to the definition file. When done, exit the shell, remove or rename the directory, correct the definition file, and start again.
After successful build, convert the sandbox directory into an immutable SIF-file:
```
sudo singularity build mpitest-focal.sif mpitest-focal
```

Copy the container image to your LeoX scratch directory

scp mpitest-focal.sif cXXXyyyy@leoX:/scratch/cXXXyyyy/mpitest-focal.sif

Note: Singularity provides full flexibility for building and converting containers between various formats. E.g., after all changes have been integrated into the build file and you are satisfied with the results, you may build directly into a SIF-file:

sudo singularity build mpitest-focal.sif mpitest-focal.def

You can also extract a sandbox directory from a SIF-file:

sudo singularity build --sandbox mpitest-focal mpitest-focal.sif

On LeoX

Log on to LeoX and cd to your $SCRATCH directory.
Issue module load singularity/3.x
Check for your container image ls -l mpitest-focal.sif
First functional test: Test local shared memory execution of the MPI benchmark. As of November 2021, Ubuntu 20.04 LTS (focal) comes with OpenMPI 4.0.3, so this is what we need to load in this test.
```
module load gcc/8.2.0 openmpi/4.0.3 singularity/3.x
mpirun -np 2 singularity exec mpitest-focal.sif osu_bw
```
You should get bandwidths up to approx. 4 GB/s

Second functional test: Test multihost execution under the batch system. Prepare an SGE batch job

#!/bin/bash
#$ -N singtest
#$ -q short.q
#$ -pe openmpi-1perhost 2
#$ -cwd
#$ -j yes

module load gcc/8.2.0 openmpi/4.0.3 singularity/3.x
set -ex
date
mpirun -np 2 singularity exec -B $UIBK_OPENMPI_ETC/openmpi-mca-params.conf:/etc/openmpi/openmpi-mca-params.conf mpitest-focal.sif osu_bw
date
echo finished

Note that the singularity commands are placed on the execution nodes using the host's mpirun command. The -B parameter maps the host's OpenMPI configuration file into the one seen by the processes running in the container. On Leo4, you should see bandwidths up to approx. 12 GB/s.

Sample output for OSU MPI benchmark

 $ cat singtest.o592416
# OSU MPI Bandwidth Test v5.3.2
# Size      Bandwidth (MB/s)
1                       3.28
2                       7.06
4                      14.01
8                      28.20
[...]
524288              11831.02
1048576             11670.49
2097152             11608.68
4194304             11901.30

The present document should provide you with a useful starting point for implementing your own workflow.

Subject to driver compatibility (unfortunately, CentOS 7.4 broke backwards compatibility with some older OpenMPI implementations), additional OpenMPI versions can be installed upon request.

Zentraler Informatikdienst der Universität Innsbruck