Slurm

Slurm Workload Manager is a free and open-source job scheduler for Linux and Unix-like systems, widely used by supercomputers and clusters.

Official slurm documentation: https://slurm.schedmd.com/documentation.html

Slurm on Euler¶

On Euler, Slurm version 25.05 is used since 13. Jan 2026.

Euler runs jobs non-exclusively on compute nodes. This means multiple jobs can run on the same node at the same time. We do this to maximize resource utilization and efficiency.

Resource Requests

Specify your resource requirements (CPUs, memory, time, etc.) as accurately as possible; as specific as required, as loosely as possible. The smaller a job's footprint, the quicker it will start. Don't specify CPU types, GPU types, partitions, or other specific resources unless your job really depends on them.

Ignore Partitions

Don't specify partitions. Slurm internally uses partitions (queues) to manage resources, but as a user you best ignore them.

GPU Nodes Access

GPU nodes are only available to shareholders, which invest in GPU hardware.

Usage¶

Main client commands:

srun: Get a job allocation and execute an application
sbatch: Submit a batch script to Slurm
squeue: View the job queue
scancel: Remove a job from the queue

Additional commands:

salloc: Get a job allocation
sacct: View accounting data
sbcast: Broadcast file to a job's compute nodes
sinfo: View nodes and partitions
sstat: Display status information of a running job/step

Custom extensions on Euler:

myjobs: Human-readable job info (pretty print of squeue)
my_share_info: Shows your cluster share(s)
get_inefficient_jobs: Displays inefficient jobs

Web interfaces:

Common options for sbatch and srun:

Option	Description
`-t`, `--time=<time>`	Maximum runtime (default: 1 hour)
`-n`, `--ntasks=<number>`	Maximum number of tasks (default: 1)
`-c`, `--cpus-per-task=<ncpus>`	CPUs per task (default: 1)
`--mem-per-cpu=<size>[units]`	Memory per CPU (default: MB)
`-N`, `--nodes=<minnodes>`	Minimum nodes (default: 1)
`-G, --gpus=[type:]<number>`	Number of GPUs (default: 0)
`--gres=gpumem:<size>[units]`	GPU memory (default: 0)
`--tmp=<size>[units]`	Temporary disk space (default: 0)
`-o`, `--output=<filename>`	Output file for job stdout (default: `slurm-%j.out`)
`-e`, `--error=<filename>`	Output file for job stderr (default: `slurm-%j.err`)
`-d`, `--dependency=<dependency>`	Specify job dependencies (e.g., `afterok:<job_id>`)
`-A`, `--account=<share_name>`	Specify the account to charge the job to (default: your default account)
`--x11`	Enable X11 forwarding
`--wrap=<command_string>`	Wrap command string in a shell script

Jobs inherit the environment variables from the shell that submits them.

Interactive sessions¶

To start an interactive session, use

srun --pty bash

--pty allocates a pseudo-terminal for the session, allowing you to interact with the shell as if it were a local terminal. You can specify additional options like --ntasks, --cpus-per-task, etc. to customize the resources.

This is useful for debugging or for running commands interactively on a compute node.

Running parallel jobs (`srun`)¶

To run a parallel job, use srun with the --ntasks option:

srun --ntasks=5 --cpus-per-task=8 nproc

This command runs nproc in 5 tasks, each with 8 CPUs. The nproc command prints the number of available CPUs in the current task.

Submitting batch jobs (`sbatch`)¶

To submit a batch job, create a script file (e.g., batch_script.txt) with the following content:

#!/bin/bash
#SBATCH --ntasks=5
#SBATCH --cpus-per-task=8
srun nproc

Then submit the script using sbatch:

sbatch batch_script.txt

sbatch will put the job in the queue and execute it when resources are available. The output will be written to a file named slurm-<job_id>.out in the current directory. sbatch does not wait for the job to finish; it returns immediately after submitting the job; use --wait to make it wait. (You can check the job's status with squeue.)

#SBATCH directives must be placed before the first executable command in the script.

Without srun, nproc will not run in parallel across all tasks. It will only run the command once, using a single task.

Submitting job arrays (`sbatch --array`)¶

A job array allows you to submit multiple similar jobs with a single command. This is useful for executing the same thing with different parameters or datasets.

sbatch --array=1-10 batch_script.txt

This command submits 10 jobs, each with a different index (1 to 10). You can access the index in the script using the environment variable $SLURM_ARRAY_TASK_ID.

You can find the possible options for job arrays in the Slurm documentation.

Submitting MPI jobs¶

Slurm supports MPI (Message Passing Interface) jobs. For more information see https://slurm.schedmd.com/mpi_guide.html#open_mpi.

Submitting GPU jobs¶

GPU Nodes Access

GPU nodes are only available to shareholders, which invest in GPU hardware.

To submit a job that requires GPUs, use the --gpus option:

srun --gpus=1 nvidia-smi

This command requests 1 GPU and runs nvidia-smi to display GPU information.

You can also specify the amount of GPU memory a job needs with --gres=gpumem:<size>[units].

If your job needs a particular type of GPU (or a set of types) you can use -G, --gpus=[type:]<number>. For example --gpus=rtx_3090,rtx_4090:1 requests one RTX 3090 GPU or one RTX 4090 GPU.

GPU Model	Slurm Identifier
Nvidia Quadro RTX 6000	`rtx_6000`
Nvidia TITAN RTX	`nvidia_titan_rtx`
Nvidia Tesla A100 (40 GB)	`a100`
Nvidia Tesla A100 (80 GB)	`a100_80gb`
Nvidia RTX 2080 Ti	`rtx_2080`
Nvidia RTX 3090	`rtx_3090`
Nvidia RTX 4090	`rtx_4090`
Nvidia RTX Pro 6000	`pro_6000`

Nvidia RTX Pro 6000¶

Programs compiled with CUDA 12 libraries will not run on Blackwell GPUs such as the Nvidia RTX Pro 6000. Since CUDA 13 is relatively new and not used yet by many programs, we decided to not open the new RTX Pro 6000 GPUs to all jobs since we expect most would fail.

Only batch jobs explicitly requesting the RTX Pro 6000 GPU can therefore run on these nodes, e.g.,

srun --gpus:pro_6000:1 my_cuda13_program

expectedly until mid-2026.

Accounting and Shares¶

If you are a member of multiple shareholder groups, you can select which group a job should be scheduled and accounted to, by passing slurm the option -A <share_name>. To configure a default group, specify the following in your ~/.slurm/defaults file (create it if it does not exist and replace <share_name> with your preferred default share):

account=<share_name>

The public share, which is open to all members of ETH, is named "public".

Custom extensions on Euler¶

`myjobs`¶

Use myjobs -j <job_id> to view detailed information of a job. It provides a human-friendly summary of Slurm jobs, improving on squeue's output.

Here's an example of a pending job

$ myjobs -j 6038307
Job information
 Job ID                          : 6038307
 Status                          : PENDING
 Running on node                 : None assigned
 User                            : nmarounina
 Shareholder group               : es_cdss
 Slurm partition (queue)         : gpu.24h
 Command                         : script.sbatch
 Working directory               : /cluster/home/nmarounina
Requested resources
 Requested runtime               : 08:00:00
 Requested cores (total)         : 12
 Requested nodes                 : 1
 Requested memory (total)        : 120000 MiB
Job history
 Submitted at                    : 2023-01-09T15:56:09
 Started at                      : Job did not start yet
 Queue waiting time              : 8 s
 Resource usage
 Wall-clock                      : 
 Total CPU time                  : -
 CPU utilization                 : - %
 Total resident memory           : - MiB
 Resident memory utilization     : - %

And this is an example of a running job

$ myjobs -j 6038307
Job information
 Job ID                          : 6038307
 Status                          : RUNNING
 Running on node                 : eu-g3-022
 User                            : nmarounina
 Shareholder group               : es_cdss
 Slurm partition (queue)         : gpu.24h
 Command                         : script.sbatch
 Working directory               : /cluster/home/nmarounina
Requested resources
 Requested runtime               : 08:00:00
 Requested cores (total)         : 12
 Requested nodes                 : 1
 Requested memory (total)        : 120000 MiB
Job history
 Submitted at                    : 2023-01-09T15:56:09
 Started at                      : 2023-01-09T15:56:38
 Queue waiting time              : 29 s
 Resource usage
 Wall-clock                      : 00:00:36
 Total CPU time                  : 00:00:00
 CPU utilization                 : 0%
 Total resident memory           : 2.94 MiB
 Resident memory utilization     : 0%

`my_share_info`¶

Use my_share_info to list the shareholder groups you belong to.

Here's an example output

$ my_share_info
You are a member of the following shareholder groups on Euler:
es_example
es_euler

`get_inefficient_jobs`¶

Use get_inefficient_jobs to find jobs that are running inefficiently, i.e. using more resources than requested.

Here's an example output

$ get_inefficient_jobs
Looking for user dohofer
37279906
37301756
37301756
37301756
37301756
37305670
37305759
37306347
37306347
37306347
37306347
37306347
37306347
37306347
37306347
37498300
37602385

Average CPU used (higher better): 12.1% (equivalent to wasting 4.0 cores)
Average system RAM used (higher better): 25.2% (equivalent to wasting 14.0 GB)
Average GPU used (higher better): 100.0% (equivalent to wasting 0.0 GPUs)
Average GPU RAM used (higher better): 100.0% (equivalent to wasting 0.0 GB)
CPU to GPU ratio (lower better): 0.0% (equivalent to 0.0 GPUs blocked)
RAM to GPU ratio (lower better): 0.0% (equivalent to 0.0 GPUs blocked)
Number of short jobs (lower better): 1

Troubleshooting¶

ReqNodeNotAvail¶

Sometimes Slurm reports that requested nodes are not available and lists some nodes. It looks something like this:

JOBID PARTITION  NAME    USER   ST       TIME  NODES NODELIST(REASON)
12345 normal.12  my_job  myname PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:eu-a2p-[001,103],eu-g9-091-2)

If you have not requested these nodes explicitly, this message by Slurm is misleading. The job is actually waiting in the queue. No user action is required; just patience.

step creation temporarily disabled, retrying (Requested nodes are busy)¶

If you try to execute srun inside an other srun, you will see an error message like this:

srun: Job 65030366 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 65030366 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 65030366 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 65030366 step creation still disabled, retrying (Requested nodes are busy)

This is usually caused by nesting srun in an other srun (e.g. srun srun ls). srun sbatch --wrap="ls" and sbatch --wrap="srun ls" on the other hand will work just fine. To fix this issue, avoid nesting sruns.