Python

Versions of Python

There are multiple versions of Python installed on Spartan for each toolchain, which you can see by using module spider Python.

e.g.

[scrosby@spartan-login3 ~]$ module load foss/2022a
[scrosby@spartan-login3 ~]$ module spider Python

--------------------------------------------------------------------------------------------------------------------------------------------------
  Python:
--------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Python is a programming language that lets you work more quickly and integrate your systems more effectively.

     Versions:
        Python/2.7.18-bare
        Python/2.7.18
        Python/3.10.4-bare
        Python/3.10.4
        Python/3.11.3
     Other possible modules matches:
        Biopython  Boost.Python  GitPython  IPython  IntelPython  Python-Packages  Python-bundle-PyPI  flatbuffers-python  intervaltree-python  ...

--------------------------------------------------------------------------------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*Python.*'

--------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "Python" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider Python/3.11.3
--------------------------------------------------------------------------------------------------------------------------------------------------

The default/main Python is 3.10.4 for the foss/2022a toolchain.

For applications built with another version of Python, we have added a suffix such as

mpi4py/3.1.4-Python-3.11.3

Please do not mix the Python versions.

Common packages like numpy are already installed with some versions, but may be missing from others.

Installing additional Python packages

If a Python package is missing, we recommend that you install additional packages locally using pip install into a virtualenv on the login node.

Example

1. Create a virtualenv and install packages2. Activate the virtualenv in your batch jobs

Creating the virtualenv only needs to be done once, and most often it can be done on the login node.

The basic workflow is:

Load the Python module from the module system
Create the virtualenv
Activate the virtualenv

Install the required Python packages into the virtualenv

[scrosby@spartan-login1 ~]$ module load GCCcore/11.3.0; module load Python/3.10.4

[scrosby@spartan-login1 ~]$ virtualenv ~/venvs/venv-3.10.4
created virtual environment CPython3.10.4.final.0-64 in 2307ms
creator CPython3Posix(dest=/home/scrosby/venvs/venv-3.10.4, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/scrosby/.local/share/virtualenv)
added seed packages: pip==24.3.1, setuptools==75.3.0, wheel==0.44.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

[scrosby@spartan-login1 ~]$ source ~/venvs/venv-3.10.4/bin/activate

(venv-3.10.4) [scrosby@spartan-login1 ~]$ pip install matplotlib
Collecting matplotlib
  Downloading matplotlib-3.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
Collecting contourpy>=1.0.1 (from matplotlib)
  Downloading contourpy-1.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.4 kB)
Collecting cycler>=0.10 (from matplotlib)
  Using cached cycler-0.12.1-py3-none-any.whl.metadata (3.8 kB)
Collecting fonttools>=4.22.0 (from matplotlib)
  Downloading fonttools-4.55.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (164 kB)
Collecting kiwisolver>=1.3.1 (from matplotlib)
  Downloading kiwisolver-1.4.7-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (6.3 kB)
Collecting numpy>=1.23 (from matplotlib)
  Downloading numpy-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)
Collecting packaging>=20.0 (from matplotlib)
  Downloading packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pillow>=8 (from matplotlib)
  Downloading pillow-11.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (9.1 kB)
Collecting pyparsing>=2.3.1 (from matplotlib)
  Using cached pyparsing-3.2.0-py3-none-any.whl.metadata (5.0 kB)
Collecting python-dateutil>=2.7 (from matplotlib)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting six>=1.5 (from python-dateutil>=2.7->matplotlib)
Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Downloading matplotlib-3.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB)
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/8.3 MB 29.1 MB/s eta 0:00:00
Downloading contourpy-1.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (324 kB)
  Using cached cycler-0.12.1-py3-none-any.whl (8.3 kB)
Downloading fonttools-4.55.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 167.9 MB/s eta 0:00:00
Downloading kiwisolver-1.4.7-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 97.7 MB/s eta 0:00:00
Downloading numpy-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.4 MB)
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.4/16.4 MB 179.3 MB/s eta 0:00:00
Downloading packaging-24.2-py3-none-any.whl (65 kB)
Downloading pillow-11.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.4 MB)
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.4/4.4 MB 177.4 MB/s eta 0:00:00
Using cached pyparsing-3.2.0-py3-none-any.whl (106 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pyparsing, pillow, packaging, numpy, kiwisolver, fonttools, cycler, python-dateutil, contourpy, matplotlib
Successfully installed contourpy-1.3.1 cycler-0.12.1 fonttools-4.55.2 kiwisolver-1.4.7 matplotlib-3.9.3 numpy-2.2.0 packaging-24.2 pillow-11.0.0 pyparsing-3.2.0 python-dateutil-2.9.0.post0 six-1.17.0

Once the virtualenv has been created, you can use it in your batch jobs.

Basic workflow is:

Load the Python module from the module system
Activate the virtualenv

Add the following to your batch script

module purge
module load GCCcore/11.3.0
module load Python/3.10.4
source ~/venvs/venv-3.10.4/bin/activate

You DO NOT need to pip install Python packages every time in your batch jobs. Once a package has been installed into your virtualenv, it will always be in that virtualenv.

This works well for pure-python packages, but you may encounter errors for those that link to other binary packages. Alternatively, let us know, we can install it for you.

Detailed instructions are available at /apps/examples/Python/2022virtualenv.md on Spartan.