Container frameworks such as Docker allow an application, its dependencies and operating system to be encapsulated in a single file that can then be run in isolation from the host system. This allows your software to be more portable and consistent if it needs to be shared with others or other environments.
Containers should be considered if a software stack:
- Has complex or out-of-date dependencies that aren't easily installed on Spartan.
- Assumes a particular operating system, e.g. requires Ubuntu, but Spartan runs on Red Hat.
- Is highly-modified or a legacy version that isn't likely to be of use to other researchers.
- Is being run across heterogeneous infrastructure (e.g. Spartan, your laptop, other HPC systems and/or cloud), and containers make it easier to maintain consistency.
It should be noted that:
- The images themselves can be very large, consuming your storage quota, and being slow to transfer to/from Spartan.
- The container might be optimized for a particular processor architecture, running slowly (or not at all) on systems that differ.
- Their makeup and integrity can be opaque (although many common software packages will have officially supported container images).
It general Docker isn't appropriate for HPC environments like Spartan in which regular users don't have administrator (root) access. A good alternative is Singularity, a container framework targeted for research use, which many HPC centres support (including Spartan). Check out Singularity's documentation to learn more, in conjunction with below which is specific to Spartan.
Usually one will fetch an existing container (from Docker or Singularity Hub), build one on your own computer, or on a computer in the cloud. For the latter, the Melbourne Research Cloud is suitable, and often quite convenient as file transfer between Spartan and cloud instances are fast since they are in the same data centre.
An example job script for Singularity is available at
/usr/local/common/Singularity, which is also mirrored on GitLab.
Short for 'Virtual Environment', a virtualenv allows the creation of an isolated working copy of Python. In most cases it should not be necessary to use a virtual environment. However if very specific Python extensions are required (for example) this might be an alternative process to consider that does not use the system-wide installed versions.
To enable virtualenv, load a python module (e.g.,
module load Python/3.7.1-GCC-6.2.0), set up a virtualenv directory and environmenr (e.g.,
virtualenv ~/virtualenv/python3.7.1), activate it (
source ~/virtualenv/python3.7.1/bin/activate), and install the packages desired in that environment. One completed, one should exit the environment (
deactivate) and return to the normal shell prompt.
Detailed instructions are available at
/usr/local/common/Python/virtualenv.md on Spartan.