Building Containers#
Overview
Teaching: 20 min | Exercises: 10 min
Questions
How to build containers with my requirements?
Objectives
Download and assemble containers from available images in the repositories.
Running containers from the available public images is not the only option. In many cases, it is required to modify
an image or even to create a new one from scratch. For such purposes, Apptainer provides the command build,
defined in the documentation as the Swiss army knife of container creation.
The usual workflow is to prepare and test a container in a build environment (like your laptop), either with an interactive session or from a definition file, and then to deploy the container into a production environment for execution (such as your institutional cluster). Interactive sessions are great to experiment and test your new container. If you want to distribute the container or use it in production, then we recommend building it from a definition file, as described in the next episode. This ensures the greatest possibility of reproducibility and transparency.
Build a container in an interactive session#
While images contained in the .sif files are more compact and immutable objects, ideal for reproducibility, for building and testing images
it is more convenient to use a sandbox, which can be easily modified.
The command build provides a flag --sandbox that will create a writable directory, myAlma10, in your work directory:
apptainer build --sandbox myAlma10 docker://almalinux:9
Notes on shared file systems like AFS
Avoid using the AFS (Andrew File System) and possibly other shared file systems
as sandbox directory, as these systems can lead to permission issues.
Symptoms could be warnings like harmless EPERM on setxattr "security.capability" when building the sandbox and
IO errors during commands execution, e.g. failure to install RPM packages.
In particular, this applies to your home directory on lxplus
and to the home and nocache directories on cmslpc.
Instead, make sure to use the local file system by creating a folder in /tmp/: mkdir /tmp/$USER.
Then, replace myAlma10 in the previous (and next) command with /tmp/$USER/myAlma10.
The container name is myAlma10, and it has been initialized from the official Docker image
of AlmaLinux 9.
To initialize an interactive session use the shell command. And to write files within the sandbox directory use the --writable option.
Finally, the installation of new components will require superuser access inside the container, so use also the --fakeroot option, unless you are already root also outside.
apptainer shell --writable --cleanenv --fakeroot myAlma10
Apptainer> whoami
root
--cleanenv clears the environment. It has been added to make sure that the eventual setting of
variables on the host is not affecting the container.
Variables like PYTHONPATH or PYTHONHOME are affecting the Python execution inside the container.
A corrupted Python environment could cause errors like “ModuleNotFoundError: No module named ‘encodings’”.
Apptainer environment
Environment variables in Linux
are dynamic values that can affect programs and containers.
You can use the environment to pass variables to a container.
Apptainer by default preserves most of the outside environment inside the container
but has many options to control that.
You can clear the environment with the --cleanenv option and you can set variables with --env.
See the Apptainer manual
for more options and details.
PYTHONPATH and PYTHONHOME affect the Python execution, but other variables could affect other programs so,
if you don’t care about the outside environment, you can add --cleanenv
every time you start a container (apptainer shell, exec and instance start commands).
Depending on the Apptainer/Singularity installation (privileged or unprivileged) and the version,
you may have some requirements, like the fakeroot utility or newuidmap and newgidmap.
If you get an error when using --fakeroot have a look at the fakeroot documentation.
--fakeroot is not root
ATTENTION! --fakeroot allows you to be root inside a container that you own but is not changing who you are outside.
All the outside actions and the writing on bound files and directories will happen as your outside user, even if inside the container it is done by root.
As an example, let’s create a container with Pythia8 available using the myAlma10 sandbox.
First, we need to enable the EPEL (Extra Packages for Enterprise Linux) repositories
and to install the development tools (remember that in this interactive session we are superuser):
Apptainer> dnf install epel-release
Apptainer> dnf groupinstall 'Development Tools'
Apptainer> dnf install python3-devel
Where dnf is the package manager used in RHEL distributions
(like AlmaLinux).
Pythia is now distributed as RPM in EPEL, so you can use this to install it:
Apptainer> dnf install pythia8
Apptainer> dnf install python3-pythia8
Apptainer> dnf install pythia8-devel # optional, if you need also the development libraries to compile
Now, open an interactive session with your user (no --fakeroot). You can use now the container with Pythia8 ready in a
few steps. Let’s use the Python interface:
apptainer shell myAlma10
Apptainer> python3
>>> import pythia8
>>> pythia = pythia8.Pythia()
Build a container with Uproot available
Build a container to use Uproot, a library for reading and writing ROOT files in pure Python and NumPy, in Python 3.12.
Solution
Start from the Python 3.12 Docker image and create the myPython sandbox:
apptainer build --sandbox myPython docker://python:3.12
apptainer shell myPython
Once inside the container, you can install Uproot.
Apptainer> python3 -m pip install --upgrade pip
Apptainer> python3 -m pip install uproot awkward
Exit the container and use it as you like:
apptainer exec myPython python -c "import uproot; print(uproot.__doc__)"
Uproot: ROOT I/O in pure Python and NumPy.
...
Notice how we needed neither --writable nor --fakeroot for the installation, but everything worked fine since pip installs user packages in the user $HOME directory.
You will see new files under $HOME/.local/.
In addition, Apptainer/Singularity by default mounts the user home directory as read+write, even if the container is read-only.
This is why a sandbox is great to test and experiment locally, but should not be used for containers that will be shared or deployed. Manual changes and local directories are difficult to reproduce and control. Once you are happy with the content, you should use definition files, described in the next episode.
Key Points
The command
buildis the basic tool for the creation of containers.A sandbox is a writable directory where containers can be built interactively.
Superuser permissions are required to build containers if you need to install packages or manipulate the operating system.
Use interactive builds only for development and tests, use definition files for production or publicly distributed containers.