Status

Batch Submission and Schedulers

A batch system will track the resources available on a system and determine when jobs can run on compute nodes. This is often conducted through separate applications of a resource manager (which tracks what resources are available on each compute node) and a job scheduler (which determines when jobs can run). On the Spartan HPC system we use the Slurm Workload Manager, which combines both tasks into a single application.

To submit jobs to the cluster one needs to provide a job submission script. The script consists of two sets of directives. The first set are the resource requests that one is making to the scheduler. This includes how many nodes are needed, how many cores per node, what partition the job will run on, and how long these resources are required (walltime). These scheduler directives must come first. The second set is the batch of commands that are understood by the computer's operating system environment, the shell. This includes any modules that are being loaded, and the commands, including invoking other scripts, that will be run.

When a job is submitted to Slurm, it will go the scheduler which will receive information from the resource manager daemons that run on the compute nodes. The resource requests of the job are compared with the resources available and evaluated by a policy-based "Fair Share" system. When the jobs are available on the partition requested and the job has priority, it will run for as long as the time that the resources have been requested for. When the job completes (or aborts) the scheduler will write an output file, and the application may as well.

Spartan's Partitions

HPC systems often are built around queues or partitions representing homogenous hardware or administrative restrictions. With Slurm on Spartan, one can view the list of partitions with the sinfo -s command, like the following:

$ sinfo -s
PARTITION    AVAIL  TIMELIMIT   NODES(A/I/O/T) NODELIST 
physical*       up 30-00:00:0        46/0/7/53 spartan-bm[055-066,085-125]
long            up 90-00:00:0          2/0/0/2 spartan-snowy[030-031]
msps2           up 30-00:00:0          0/2/0/2 spartan-bm[047-048]
punim0396       up 30-00:00:0          0/2/0/2 spartan-bm[051-052]
shortgpgpu      up    1:00:00          0/2/0/2 spartan-gpgpu[001-002]
gpgpu           up 7-00:00:00       31/33/0/64 spartan-gpgpu[003-015,020-069,076]
longgpgpu       up 30-00:00:0          1/1/0/2 spartan-gpgpu[070-071]
deeplearn       up 30-00:00:0         5/7/0/12 spartan-gpgpu[072-075,078-082,086-088]
interactive     up 2-00:00:00          2/0/0/2 spartan-bm[083-084]
snowy           up 30-00:00:0       11/18/0/29 spartan-snowy[001-029]
mig             up 30-00:00:0         5/5/0/10 spartan-bm[067-076]
turbsim         up 30-00:00:0          0/6/0/6 spartan-bm[077-082]
mig-gpu         up 30-00:00:0          0/2/0/2 spartan-gpgpu[084-085]
gpgputest       up 30-00:00:0          0/6/0/6 spartan-gpgpu[016-019,089-090]
physicaltest    up 30-00:00:0          0/2/0/2 spartan-bm[053-054]
adhoc           up 7-00:00:00          0/2/0/2 spartan-bm[126-127]
debug           up 30-00:00:0     103/88/7/198 spartan-bm[047-048,051-127],spartan-gpgpu[001-076,078-082,084-090],spartan-snowy[001-031]

This provides the partition name (e.g., physical, snowy, gpgpu, etc). Some of the partitions are restricted to particular projects (e.g., punim0396, deeplearn, etc) and will require membership to a relevant group and an additional scheduler directive to access. Others (e.g., physical, snowy) have general access. In addition to the name, the output lists the Availability of the partition (up, down), and the maximum walltime that is available for job submission. Normally, this is 30-00:00:0, or 30 days for most partitions. If it is less than 30 days this indicates that a planned outage is pending, with the maximum walltime decrementing as the day of the outage approaches.

Following this is a summary status of nodes in the partition. These are either Allocated, Idle, Out, and Total. Finally, the list of nodes in the partition with hostnames and ranges. Note that individual nodes can belong to multiple partitions.

Partition Utilisation

Before submitting a job it may be worthwhile to check the utilisation of a partition. Whilst the sinfo -s command gives a high-level overview of the status of all the partitions, the showq command can be used to specify the status of a partitular partition, e.g., showq -p physical, showq -p gpgpu etc). For example

$ showq -p physical

SUMMARY OF JOBS FOR QUEUE: <physical>

ACTIVE JOBS--------------------
JOBID     JOBNAME    USERNAME      STATE     CORE   REMAINING  STARTTIME
==================================================================================
17325890  run-experi vivekkatial   Running   1       263:35:45  Wed Jul 22 15:23:00
17325891  run-experi vivekkatial   Running   1       263:35:46  Wed Jul 22 15:23:01
17325892  run-experi vivekkatial   Running   1       263:35:46  Wed Jul 22 15:23:01
17325893  run-experi vivekkatial   Running   1       263:35:46  Wed Jul 22 15:23:01
17325901  run-experi vivekkatial   Running   1       263:35:46  Wed Jul 22 15:23:01
...
     850 active jobs : 2335 of 3912 cores ( 59.69 %):   43 of   56 nodes ( 76.79 %)
...

As can be seen from the example, whilst all nodes are busy there are still some cores available. If a full-node job was launched at this point it would certainly go into the queue.

Monitoring job memory, CPU and GPU utilisation

As a result of the feedback obtained by the 2020 Spartan HPC user survey, a job monitoring system was developed.

This allows users to monitor the memory, CPU and GPU usage of their jobs via a simple command line script.

For more details, please see Job Monitoring

Job Priority and Limits

Spartan is a very busy system, sometimes with 100% worker node allocation on most days. Because no system has an infinite number of cores there needs to be some sort of method which establishes an order when a job can run. Demand for HPC resources typically outweights supply. By default, the scheduler allocates on a simple "first-in, first-out" (FIFO) approach. However the applications of rules and policies can change the priority of a job, which will be expressed as a number to the scheduler. Some of the main factors are:

  • Job size : The number of nodes, cores, or memory that a job is requesting. A higher priority is given to larger jobs.
  • Wait time : The priority of a job increases the longer it has been in the queue.
  • Fairshare - the difference between the portion of the computing resource that has been promised to a group or user and the amount of resources that has been consumed. It takes into account the resources used by a user's jobs in the last 14 days. The more resources used by a user's or group's jobs in the last 14 days, the lower the priority of your new jobs.
  • Backfilling: Where there is a gap in the resources and a job size, the scheduler will fill that gap to ensure maximum resource allocation.
  • Partition and QoS: A factor associated with each node partition.

On Spartan, the calculated priority is dominated by the fairshare component (aside from QoS restrictions), so the most common reason for a job taking a long time to start is because of the amount of resources consumed in the last 14 days

You can see your job priority, and what makes up the priority, by using the sprio command

# sprio -j 12409951
          JOBID PARTITION   PRIORITY        AGE  FAIRSHARE    JOBSIZE  PARTITION        QOS
       12409951 physical        4240       3000       1233          6          1          0

To ensure fair use of the GPGPU partition on Spartan, quotas are implemented to ensure that no one participant on the GPGPU partition (LaTrobe, Deakin, St Vincents, UoM, UoM Engineering, UoM MDHS) can use all of the available GPUs. If the GPGPU sponsor is using over their quota of GPUs, a job will be held with the message "QOSGrpGRES". Similarly if the GPGPU sponsor is using over their quota of CPUs, the job will be held with the message "QOSGrpCpuLimit". The job will run once enough current running jobs end.

Likewise on public partitions of Spartan (physical, snowy, interactive) CPU and memory quotas have been implemented. This ensures no one project can use all the resources in these partitions. The limits are currently set at 19.5% of the resources in each partition. If a job is not running due to "MaxCpuPerAccount", it means that the project's running jobs exceed the current CPU quota for that partition. If a job is not running due to "MaxMemoryPerAccount", it means that the project's running jobs exceed the current memory quota for that partition.

Partition CPU Quota (CPU cores) Memory Quota (MB RAM)
physical 750 9585888
snowy 200 1493750
interactive 8 98304
long 32 239000

Note: Users with COVID-19 projects may gain an additional priority with the directive --qos=covid19 at job submission

CPU and Memory Quotas

"MaxCpuPerAccount" and "MaxMemoryPerAccount"

To ensure fair use of the public partitions on Spartan (physical, snowy), we have implemented CPU and memory quotas. This ensures no one project can use all the resources in these partitions. The limits are currently set at 15% of the resources in each partition.

If your job is not running due to "MaxCpuPerAccount", it means that your project's running jobs exceed the current CPU quota for that partition.
If your job is not running due to "MaxMemoryPerAccount", it means that your project's running jobs exceed the current memory quota for that partition.

See the table above for the current limits per project.

GPU Partitions

Spartan hosts a GPGPU service, developed in conjunction with Research Platform Services, the Melbourne School of Engineering, Melbourne Bioinformatics, RMIT, La Trobe University, St Vincent's Institute of Medical Research and Deakin University. It was funded through ARC LIEF grant LE170100200. It consists of 72 nodes, each with four NVIDIA P100 graphics cards, which can provide a theoretical maximum of around 900 teraflops.

The GPGPU cluster is available to University researchers, as well as external institutions that partnered through the ARC LIEF grant.

Jobs submitted to the GPU partition require the partition information in the script (e.g., #SBATCH --partition=gpgpu), along with generic resource request in the job script, for example #SBATCH --gres=gpu:2 which will request two GPUs for the job. A range of GPU-accelerated software such as TensorFlow is available on Spartan, as well as CUDA for developing your own GPU applications. These are available on Spartan at /usr/local/common. Finally, these jobs will also require a #SBATCH --qos= setting to denote which of the project members have granted their project authority to use the partition. A table of valid qos entries and their corresponding groups is as follows

Group QOS setting Notes
General UoM gpgpuresplat Users can apply for this access on a case by case basis
MSE gpgpumse MSE users can apply for access
MDHS gpgpumdhs MDHS users can apply for access
St Vincent's gpgpusvi
RMIT gpgpurmit
Deakin gpgpudeakin
La Trobe gpgpultu

Note that the deeplearn parition also requires QOS (#SBATCH --qos=gpgpudeeplearn) and gres settings (#SBATCH --gres=gpu:n). This partition is limited to specific projects from the CIS department.

To ensure fair use of the GPGPU partition on Spartan, quotas are implemented to ensure that no one participant in the GPGPU project (LaTrobe, Deakin, St Vincents, UoM, UoM Engineering, UoM MDHS) can use all of the available GPUs. If a GPGPU sponsor is using over their quota of GPUs, their jobs will be held with the message "QOSGrpGRES". Similarly if a GPGPU sponsor is using over their quota of CPUs, their jobs will be held with the message "QOSGrpCpuLimit". The job will run once enough current running jobs end.

If you have a GPU project and the you're at the University of Melbourne you can access the CPU partitions as well by adding: #SBATCH -q normal to your job submission script, followed by the CPU partition name, e.g., #SBATCH -p physical

Use Cases

Tiffany Walsh - Deakin

DeepLearn

Spartan's deeplearn partition consists of hardware purchased by the Computing and Information Systems department; 13 nodes, each with four NVIDIA V100 graphics cards. The deeplearn partition is available to specific Engineering projects, especially those in CS and SE. You can request access to it at the time of creating a Spartan account.

To access the deeplearn partition:

#SBATCH --partition deeplearn
#SBATCH --qos gpgpudeeplearn
#SBATCH -A projectID
#SBATCH --gres=gpu:v100:4

This will request 4 v100 GPUs using the project projectID

You can see the specifications of the deeplearn nodes on Status and Specifications