Skip to content

Job Script Template

Choose a template, replace YOUR_SLURM_ACCOUNT with your project account, and set your executable path. Follow the resource limits.

Compile and run with the same software stack. These templates use nvhpc-hpcx-cuda13-openmpi5/26.5; your application may require another environment.

1. GPU Jobs

One GPU on One Node

  1. Compile your code on a login node.
  2. Save this as gpu-job.sh. Replace the account and executable path, and adjust resources to your application.
  3. Submit the script to run on a compute node.
#!/bin/bash -l
# Replace YOUR_SLURM_ACCOUNT with SLURM account from your project's HPC resource.
#SBATCH --account=YOUR_SLURM_ACCOUNT
#SBATCH --job-name=gpu-job
#SBATCH --partition=gh200
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:nvidia_gh200_144g_hbm3e:1
#SBATCH --mem=64G
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

set -e
module purge
module load nvhpc-hpcx-cuda13-openmpi5/26.5

cd "$HOME/my-project"
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
srun --mpi=none ./your_gpu_executable

The module selects the compiler and CUDA libraries; --gres requests the GPU. This single-process example does not use MPI.

Two GPUs on One Node

Runs one process with two GPUs and eight CPU cores. Your application must support multiple GPUs.

Save as gpu-2-devices.sh:

#!/bin/bash -l
#SBATCH --account=YOUR_SLURM_ACCOUNT
#SBATCH --job-name=gpu-2-devices
#SBATCH --partition=gh200
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:nvidia_gh200_144g_hbm3e:2
#SBATCH --mem=64G
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

set -e
module purge
module load nvhpc-hpcx-cuda13-openmpi5/26.5

cd "$HOME/my-project"
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
srun --mpi=none ./your_multi_gpu_executable

Two GPUs with MPI on One Node

Runs two MPI ranks, each with one GPU and eight CPU cores. Build with a compatible MPI stack.

Save as gpu-2-mpi.sh:

#!/bin/bash -l
#SBATCH --account=YOUR_SLURM_ACCOUNT
#SBATCH --job-name=gpu-2-mpi
#SBATCH --partition=gh200
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:nvidia_gh200_144g_hbm3e:2
#SBATCH --mem=64G
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

set -e
module purge
module load nvhpc-hpcx-cuda13-openmpi5/26.5

cd "$HOME/my-project"
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
srun --gpu-bind=single:1 ./your_mpi_gpu_executable

--gres requests GPUs per node, and --gpu-bind=single:1 binds one task to each GPU. Each rank should use its assigned visible GPU. The memory request is 64 GiB per node.

Slurm sets GPU visibility; do not override CUDA_VISIBLE_DEVICES. See GPU resources.

2. OpenMP Job

OpenMP uses threads within one process on one node. This requests 16 cores for one process, with 32 GiB of host memory. Build your program with OpenMP support.

Save as openmp-job.sh:

#!/bin/bash -l
#SBATCH --account=YOUR_SLURM_ACCOUNT
#SBATCH --job-name=openmp-job
#SBATCH --partition=gh200
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

set -e
module purge
module load nvhpc-hpcx-cuda13-openmpi5/26.5

cd "$HOME/my-project"
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export OMP_PLACES=cores
export OMP_PROC_BIND=close
srun --mpi=none --cpu-bind=cores ./your_openmp_executable

Match OMP_NUM_THREADS to --cpus-per-task. Increasing the task count starts more processes rather than adding threads to this process.

3. MPI Job

MPI runs processes (ranks) across one or more nodes. This Basic tier example requests one node with 144 ranks in total, with one core per rank. Adjust these counts to the application and your allocation.

Save as mpi-job.sh:

#!/bin/bash -l
#SBATCH --account=YOUR_SLURM_ACCOUNT
#SBATCH --job-name=mpi-job
#SBATCH --partition=gh200
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=144
#SBATCH --cpus-per-task=1
#SBATCH --exclusive
#SBATCH --mem=64G
#SBATCH --time=00:30:00
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

set -e
module purge
module load nvhpc-hpcx-cuda13-openmpi5/26.5

cd "$HOME/my-project"
export OMP_NUM_THREADS=1
srun ./your_mpi_executable

srun is the default launcher in these templates. The MPI examples use the cluster’s default MPI plugin; use a compatible MPI build and verify it with an application run. See the Slurm MPI guide.

--mem=64G requests memory per node. Exclusive node access does not automatically request all node memory. Choose the memory request for your workload.

Optional: Explicitly Choose mpirun

Only if you explicitly choose mpirun for your application, replace the MPI template's srun line with:

mpirun -np "$SLURM_NTASKS" ./your_mpi_executable

mpirun discovers the Slurm allocation without a hostfile. Use one launcher per execution; do not wrap mpirun in srun. See Open MPI with Slurm.

MPI Environment

Modules set the compiler and library paths. Keep executables and inputs in shared $HOME; /tmp is local to each node.

Use runtime overrides only for diagnosed issues. UCX_TLS=sm,self restricts communication to a single node.

Submit and Monitor

With your account set in the script, submit from the directory where you want the logs:

sbatch gpu-job.sh
squeue -u "$USER"

Choose the corresponding filename for the other templates. See submission options and logs, monitoring, and cancellation.

These templates have not been executed on ICARUS; check a small run before scaling up.