Skip to content

Job Accounting

Use sacct to review your jobs from today, including completed work:

sacct -u "$USER" -S today --format=JobID,JobName,Partition,AllocCPUS,State,ExitCode

Example output:

JobID       JobName  Partition AllocCPUS State      ExitCode
12344       cpu-test gh200     16        COMPLETED  0:0
12344.batch batch              16        COMPLETED  0:0
12345       gpu-test gh200     72        CANCELLED  0:15

Customize the fields or select a date range:

sacct -u "$USER" -S today --format=JobID,JobName,Partition,Account,AllocCPUS,NNodes,Elapsed,CPUTime,State,ExitCode
sacct -u "$USER" -S 2026-09-01 -E 2026-09-10 --format=JobID,JobName,State,Elapsed,ExitCode
sacct --helpformat

Replace the dates with your reporting period. ExitCode records exit status and terminating signal; 0:0 indicates successful termination. Job steps can have separate records. CPUTime is allocated CPUs multiplied by elapsed time, not measured CPU utilization or the final GPU-weighted charge. For example, 72 allocated CPUs for 30 minutes represent 36 CPU core-hours. Check allocation usage in the User Dashboard for the project's reported balance. See the sacct reference.

Inspect Running Job Steps

sstat --allsteps -j 12345 --format=JobID,AveCPU,AveRSS,MaxRSS

Use your own running job ID. AveCPU reports average task CPU time; AveRSS and MaxRSS summarize task memory, not total node memory. Metrics depend on the accounting plugin and may be absent or delayed. If needed, inspect a specific step such as 12345.batch. See the sstat reference.

Usage Reports: CPU, Memory, and GPU Metrics

For a finished job, inspect recorded resource metrics with:

sacct -j 12345 --format=JobID,TRESUsageInAve%110

For all running steps of your job, use:

sstat --allsteps -j 12345 --format=JobID,TRESUsageInAve%110

Replace 12345 with your job ID. %110 sets the display width of the metrics column. Expand the reporting period with sacct -S YYYY-MM-DD when needed.

Illustrative output only, assuming GPU accounting is enabled:

JobID       TRESUsageInAve
12345.0     cpu=00:05:26,gres/gpumem=40960M,gres/gpuutil=87,mem=6291456K
Metric Interpretation
cpu Average task CPU time
mem Average task resident host memory; use the displayed unit
gres/gpumem GPU memory accounting metric, when collected; use the displayed unit
gres/gpuutil GPU utilization high-water mark averaged across tasks, when collected; not a time-averaged efficiency score

GPU metrics depend on the accounting configuration and are not yet confirmed for ICARUS. Slurm reports GPU memory and utilization high-water marks across tasks, not time-averaged efficiency or a core-hour bill. Missing fields do not prove zero usage. See GPU accounting. Accounting retention is not yet established.