Skip to content

Job Submission, Parameter, Variables

This guide provides a short tutorial on how to submit computing jobs to the workload scheduler Slurm.

Creating Batch Scripts

To submit jobs, you need to create a batch script containing both your computing job parameters and the executable code. You can create and edit these scripts in two ways:

  • Graphically: Connect to the cluster using FastX, where you can utilize graphical editing tools.
  • Command-Line: Connect to the cluster on the terminal and use a command line editor such as vim or nano.

The following example shows a batch script for a simple MPI job:

batch_script.sh
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#!/usr/bin/zsh

### Job Parameters 
#SBATCH --ntasks=8              # Ask for 8 MPI tasks
#SBATCH --time=00:15:00         # Run time of 15 minutes
#SBATCH --job-name=example_job  # Sets the job name
#SBATCH --output=stdout.txt     # redirects stdout and stderr to stdout.txt
#SBATCH --account=<project-id>  # Set your project-id or delete it

### Program Code
srun hostname

The batch script is comprised of three parts:

  1. The first line is always a shebang that defines the interpreter. The cluster only supports the ZSH shell.
  2. The next lines (4 - 8) specify the job parameters. Each line starts with #SBATCH followed by the option that you want to set.
  3. As soon as a non-comment line is encountered, job parameters are no longer read in and the execution of the job is started (line 11). In this job, each MPI rank prints their hostname.

Tip

In our Example Collection we provide a comprehensive list of example batch scripts for various pruposes such as:

  • OpenMP / multi-threading jobs
  • MPI / multi-processing jobs (single and multi-node)
  • Hybrid MPI + OpenMP
  • GPU jobs (single GPU, multi-GPU, multi-node-multi-GPU)
  • BeeOND

Submitting Batch Jobs

After specifying your batch script, you need to submit your job. Navigate to the directory where you created the batch script and use the following command to submit it to the job to the Slurm queue:

Submit your Batch Script
1
sbatch batch_script.sh

After successfully submitting your job script, you will obtain a job id from Slurm, which can be used to manage your job.

Submitted batch job 12345678

To check the current state of your jobs use squeue --me which will print the following:

   JOBID PARTITION        NAME      USER ST       TIME  NODES NODELIST(REASON)
12345678     c23ms example_job  AB123456  R       0:02      1 n23m0023

Most importantly, ST shows the current state of your job. PD stands for pending and R for running. When your job is no longer visible in the queue list, then it has completed.

Note

The directory from which sbatch is called, will be set as working directory.

Batch Job Parameters

There are several Slurm parameters that can be used in job scripts to specify resource requirements or control other apsects of jobs submission and execution. A detailed documentation can be found in the official Slurm documentation for sbatch.

The following table is a short summary of the most relevant flags:

Flag (long/short) Example Description / Remarks Default Value
--job-name / -J -J my_job Avoid using special characters. Only alphanumeric are recommended. -
--time / -t --time=1-05:10:15 (1 day, 5 hours, 10 minutes, 15 seconds) Maximum job run time. 15 min
--nodes / -N -N 10 Number of compute nodes, usually used for MPI parallelization. If not set, will computed based on other resource requirements
--ntasks / -n -n 10 Number of processes, usually used for MPI parallelization. 1
--ntasks-per-node --ntasks-per-node 2 Number of tasks per node If not set, automatically assigned
--cpus-per-task / -c -c 10 CPUs per Task, usually is used for OpenMP or Hybrid parallelisation. 1
--exclusive --exclusive Using nodes exclusively. For larger jobs, this is almost always what you want to use. not used
--account / -A -A rwthXXXX You must be a member of the computing time project to charge computing time to it. -
--gres=gpu:<N> --gres=gpu:2 (requests two GPUs) <N> is the number of GPUs per node. Note that your project needs to be allowed to use GPUs. A single GPU is limited to a proportional amount of CPUs and Memory within the node. More CPU/Memory per GPU requires requesting more GPUs. -
--partition / -p -p c23ms Sets the desired partition. Depending on your computational needs (e.g. memory) you may want to specify a specific partition on the cluster. on CLAIX: c23ms
--beeond --beeond Request BeeOND on demand file system utilizing local SSDs of compute nodes. Details on how to use BeeOND (BeeGFS On-Demand) can be found on the page about the available filesystems. Please mind that your job will become exclusive if you use BeeOND. -

Slurm Environment Variables

There are a number of environment variables that can accessed within a running job, which reflect information about the current job (e.g., SLURM_JOB_ID, SLURM_JOB_NAME, SLURM_JOB_PARTITION, etc.) and resource requirements (e.g., SLURM_NNODES, SLURM_NTASKS, SLURM_CPUS_PER_TASK, etc.).

A comprehensive list of available environment variables can be found in the official Slurm documentation.

Best Practices

  • Running multiple similar jobs which only differ in terms of the input often is required in the context of parameter study / sensitivity analysis. Array jobs are recommended for this scenario.
  • Long computations should be divided into shorter steps, creating a chain job.

In order to avoid pitfalls and finding errors after waiting hours for a job to start, the following procedure for setting up batch jobs has been judged as useful:

  • Avoid loading modules in your .zshrc or .bashrc.
  • Test your application whether it starts as expected on a login node.
  • Put all commands necessary for the program to start into a script file.
  • Don't forget the shebang at the first line of the script: #!/usr/bin/zsh
  • Use relative paths and ensure that you know the directory in which the job starts.
  • Make the script executable (chmod 775 myscript.sh), run it (./myscript.sh) and make sure that everything works as expected.
  • Ensure that you have configured your program's memory / stack limits when neccessary.
  • Then: Put all job requests in the script file using the #SBATCH directive. Pay attention to the limiting factors such as: memory, number of slots and job run time.
  • Submit your script with a run time of 5 minutes so that it has greater chance of failing early. If everything starts as expected.
  • Submit your final script with the correct time limit.