Anaconda

Anaconda provides Python and a long list of packages as well as Jupyter and environment and package manager conda and pip.

Many bioinformatics software is distributed via Anaconda.

To use anaconda on Spartan, load the required anaconda module. You can then set the conda environment path, create a conda environment, install packages, and use it in a job.

Note

Load moduleSet conda environment pathCreate a conda environmentActivate conda environmentInstall packageUsing environment in a job

[scrosby@spartan-login1 ~]$ module load Anaconda3/2024.02-1

By default, conda creates environments in your home directory, which only has a 50GB limit. With many conda environments, this space can be easily filled.
 
You can change this path using the environment variable CONDA_ENVS_PATH.

To change the conda environments location to a project directory (e.g. /data/gpfs/projects/punim0001/anaconda3), add this to your ~/.bash_profile file

export CONDA_ENVS_PATH=/data/gpfs/projects/punim0001/anaconda3/envs
export CONDA_PKGS_DIR=/data/gpfs/projects/punim0001/anaconda3/pkgs

CONDA_PKGS_DIR will change the location of the Conda shared package cache directory (normally in ~/.conda/pkgs)

To create a conda env called myenv using Python 3.10.4, load the anaconda module, and then do

[scrosby@spartan-login1 ~]$ conda create --name myenv python==3.10.4
Collecting package metadata (current_repodata.json): done
Solving environment: done

If you don't specify a Python version during environment creation, Python won't be installed, and you'll use the Python version in the system Anaconda directory.

You can install Python later using conda install into your Conda environment.

To activate a created environment myenv, load the anaconda module, and then do

[scrosby@spartan-login1 ~]$ eval "$(conda shell.bash hook)"
[scrosby@spartan-login1 ~]$ conda activate myenv
(myenv) [scrosby@spartan-login1 ~]$ 

Warning

Do not use conda init as this adds unwanted and problematic code to your .bashrc file

If you have run conda init, you can stop conda from activating the (base) environment by running

conda config --set auto_activate_base false

This is highly recommended as it will then not override system libraries/binaries

To install a package in your conda env, activate your conda environment, and then do

(myenv) [scrosby@spartan-login1 ~]$ conda install numpy
Collecting package metadata (current_repodata.json): done
Solving environment: done

To use your conda environment in a job, add the following to your submit script

module load Anaconda3/2022.10
eval "$(conda shell.bash hook)"
conda activate myenv

Warning

Do not try to use modules from our module system as well as packages installed in a conda environment in the same job.
If you use a conda environment, install everything you require for that job inside of your conda environment (e.g. don't use Tensorflow from our module system inside a job where you are using a conda environment). Mixing modules from our software system with packages installed in conda environments will fail or cause major issues.