Job Submission and Scheduling

Access and Scheduling

Partitions

Beehive has several types of partitions with different access levels and time limits:

Standard Partitions

• dev-cpu: Development partition for CPU jobs (1 hour limit)
• dev-gpu: Development partition for GPU jobs (1 hour limit)
• cpu: Standard CPU partition (8 hour limit)
• gpu: Standard GPU partition (8 hour limit)
• low-gpu: Lower priority standard GPU partition (8 hour limit)

dev-cpu is the default partition: a job submitted without --partition runs there (one node, test-c0, 1 hour limit). dev-gpu is the single node test-g0. Always set --partition explicitly for real work.

Every partition whose name contains gpu (other than the *-own-* private partitions) rejects jobs that do not request a GPU. Use --gpus/-G, --gres=gpu:<n>, --gpus-per-node, --gpus-per-socket or --gpus-per-task; submit CPU-only work to cpu.

Fair-share Scheduling

Beehive uses Slurm's fair-share scheduling with multifactor priority, which means:

• Job priority is decided by two factors: your own recent usage (fair-share, tracked per user - every user has their own Slurm account; there are no group accounts) and the partition you submit to (partition weight)
• Fair-share half-life is 8 hours - usage impact on priority decays by 50% every 8 hours
• The fair-share component has a very high weight (500000) compared to other factors
• The partition factor has weight 200000; job age, job size and QOS have weight 0

Research Group Partitions

Several research groups have partitions with increased priority:

These five partitions run on the same nodes as gpu with the same 8 hour limit; the numbers are Slurm partition priorities (gpu is 500, low-gpu is 5). Partition priority is a soft weight blended with fair-share, not an absolute override, so heavy recent usage can still queue a group-partition job behind a light user's gpu job.

Guaranteed Access Nodes

Some nodes are reserved for specific research groups. greg-gpu and tri-gpu keep the 8 hour limit; zhiyuanli-gpu, shiry-gpu and richard1xur-own-gpu have no time limit:

• greg-gpu: Nodes g0, g1, g4, g6, g8, g9, g10, g22 (RTX 2080 Ti through RTX Pro 6000; 8 hour limit; group greg-cluster)
• zhiyuanli-gpu: Nodes g20 and priv-g14 (RTX 6000 Ada and L40S)
• richard1xur-own-gpu: Node g16 (RTX 6000 Ada)
• tri-gpu: Node g11 (RTX Pro 6000 Blackwell), shared by the greg-cluster, speech-cluster, and ripl groups
• shiry-gpu: Node g2 (RTX Pro 6000 Blackwell Server Edition, 8 GPUs; no time limit), group shiry-cluster

Private Nodes

Many researchers have private nodes that are only accessible to their group:

Private partitions have no time limit, with one exception: speech-own-gpu has an 8 hour limit. Access is restricted to the owning group.

• greg-own-gpu: RTX A5000 GPUs
• kaylene-own-gpu: RTX Pro 5000 GPU
• jjery2243542-own-gpu: RTX 4000 Ada GPU
• marcelo-own-gpu: RTX Pro 5000 GPU
• mberamendi-own-gpu: RTX 2080 Ti GPU
• rcheng-own-gpu: RTX 2080 Ti GPU
• josecruzado-own-gpu: RTX 2080 Ti GPU
• shesterg-own-gpu: RTX Pro 5000 GPU
• speech-own-gpu: RTX Pro 5000 GPUs (8 hour limit)
• txu-own-gpu: Quadro RTX 6000 GPUs
• uriel-own-gpu: RTX 2080 Ti GPU
• takuma-own-gpu: RTX 2080 Ti GPU
• whc-own-gpu: Node priv-g4, 2 RTX A5000 GPUs (g22 returned to the shared/Greg pools on October 15)
• xdu-own-gpu: Quadro RTX 6000 GPUs
• zsun-own-gpu: RTX 6000 Ada GPU

Job Submission

Basic Job Submission

To submit a job to the cluster, use the sbatch command:

sbatch my_job_script.sh

Requesting Specific GPUs

Beehive supports specifying GPU types, which is important for jobs requiring particular architectures or memory sizes:

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=output_%j.log
#SBATCH --partition=gpu
#SBATCH --gpus=nvidia_rtx_a6000:1  # Request a specific GPU type
#SBATCH --cpus-per-task=4

python train.py

If you omit --cpus-per-task, a GPU job gets 2 CPUs per GPU by default (DefCpuPerGPU=2). Jobs are limited to a single node unless you set --nodes explicitly.

Available GPU Types

You can request specific GPU models using the following identifiers:

nvidia_geforce_rtx_2080_ti                                 # 11GB VRAM (Turing)
nvidia_rtx_a4000                                           # 16GB VRAM (Ampere)
nvidia_rtx_a5000                                           # 24GB VRAM (Ampere)
nvidia_rtx_a5500                                           # 24GB VRAM (Ampere)
nvidia_rtx_a6000                                           # 48GB VRAM (Ampere)
nvidia_rtx_4000_ada_generation                             # 20GB VRAM (Ada)
nvidia_rtx_5000_ada_generation                             # 32GB VRAM (Ada)
nvidia_rtx_6000_ada_generation                             # 48GB VRAM (Ada)
nvidia_rtx_pro_5000_blackwell                              # 48GB VRAM (Blackwell)
nvidia_rtx_pro_6000_blackwell_max-q_workstation_edition    # 96GB VRAM (Blackwell)
nvidia_rtx_pro_6000_blackwell_server_edition               # 96GB VRAM (Blackwell)
nvidia_l40s                                                # 48GB VRAM (Ada)
quadro_rtx_6000                                            # 24GB VRAM (Turing)

nvidia_rtx_a5000, nvidia_rtx_a5500, nvidia_rtx_4000_ada_generation, nvidia_rtx_pro_5000_blackwell and quadro_rtx_6000 exist only on private nodes and cannot be requested from gpu, low-gpu or the research-group partitions. nvidia_rtx_pro_6000_blackwell_server_edition is on g11 (shared pool) and g2.

GPU Architecture Features

You can also request GPUs based on architecture features:

#SBATCH --constraint=ada         # Ada architecture
#SBATCH --constraint=ampere      # Ampere architecture
#SBATCH --constraint=blackwell   # Blackwell architecture (newest)
#SBATCH --constraint=turing      # Turing architecture

GPU Memory Constraints

Request GPUs with specific minimum memory sizes:

#SBATCH --constraint=96g        # GPUs with at least 96GB VRAM
#SBATCH --constraint=48g        # GPUs with at least 48GB VRAM (includes the 96GB nodes)
#SBATCH --constraint=32g        # GPUs with at least 32GB VRAM
#SBATCH --constraint=24g        # GPUs with at least 24GB VRAM
#SBATCH --constraint=20g        # GPUs with at least 20GB VRAM
#SBATCH --constraint=16g        # GPUs with at least 16GB VRAM
#SBATCH --constraint=12g        # GPUs with at least 12GB VRAM (no 12GB card exists; same nodes as 16g)
#SBATCH --constraint=11g        # GPUs with at least 11GB VRAM (every GPU node)

Sample Job Scripts

Basic GPU Job

#!/bin/bash
#SBATCH --job-name=basic_gpu
#SBATCH --output=basic_%j.log
#SBATCH --partition=gpu
#SBATCH --gpus=1
#SBATCH --cpus-per-task=4

nvidia-smi
python my_script.py

High-Memory GPU Job (48 GB or more: A6000 / RTX 6000 Ada / L40S / RTX Pro 6000)

#!/bin/bash
#SBATCH --job-name=high_mem_gpu
#SBATCH --output=high_mem_%j.log
#SBATCH --partition=gpu
#SBATCH --constraint=48g        # node feature: GPUs with at least 48 GB VRAM
#SBATCH --gpus=1
#SBATCH --cpus-per-task=8

python train_large_model.py

Multi-GPU Training

#!/bin/bash
#SBATCH --job-name=multi_gpu
#SBATCH --output=multi_gpu_%j.log
#SBATCH --partition=gpu
#SBATCH --gpus=4
#SBATCH --cpus-per-task=16

python distributed_training.py

Development/Testing Job

#!/bin/bash
#SBATCH --job-name=dev_test
#SBATCH --output=dev_%j.log
#SBATCH --partition=dev-gpu
#SBATCH --gpus=1
#SBATCH --cpus-per-task=4

python debug_script.py

Singleton Jobs (Preventing Duplicate Runs)

The singleton dependency is specifically designed for managing a series of jobs that should run one after another. It ensures only one instance of a job with a specific name can run at a time:

#!/bin/bash
#SBATCH --job-name=my_job_series
#SBATCH --output=series_%j.log
#SBATCH --partition=gpu
#SBATCH --gpus=1
#SBATCH --cpus-per-task=4
#SBATCH --dependency=singleton

# Process one chunk of data
python process_data_chunk.py --chunk=$1   # chunk number passed by the submit loop below

To submit a series of jobs that will execute sequentially:

# Submit 10 jobs that will run one after another
for i in {1..10}; do
  sbatch --job-name=data_processing_series job_script.sh $i   # save the script above as job_script.sh; $i arrives as $1
done

Note: The singleton dependency is primarily useful for job series and should not be used for other purposes. All jobs must share the same name for the singleton dependency to work correctly. This is an efficient way to process large datasets in manageable chunks while preventing resource overload.

You can also combine singleton with array jobs:

#!/bin/bash
#SBATCH --job-name=sequential_array
#SBATCH --output=array_%A_%a.log
#SBATCH --array=1-100
#SBATCH --dependency=singleton
#SBATCH --partition=gpu
#SBATCH --gpus=1

# Process array tasks sequentially
python process.py --task=${SLURM_ARRAY_TASK_ID}

Checkpointing for Long Jobs

The standard partitions have an 8-hour time limit. For jobs that need to run longer, implement checkpointing in your code. Here are examples of how to save and load checkpoints with popular ML frameworks:

When a job reaches its limit Slurm sends it SIGTERM and, 30 seconds later, SIGKILL - anything still running is lost, and no further lines of the batch script execute. Save checkpoints periodically (every N epochs, as below) and, if you want a warning before the kill, request one with #SBATCH --signal=B:USR1@600 (USR1 to the batch shell 10 minutes before the limit) and trap it in your script.

Tip - set a time limit. None of the examples above sets --time; the cluster defines no default, so such a job is given the partition's full limit (8 hours on cpu/gpu/low-gpu, 1 hour on dev-cpu/dev-gpu). Adding a realistic #SBATCH --time=HH:MM:SS lets the backfill scheduler start short jobs sooner. A value above the partition limit (e.g. --time=10:00:00 on gpu) is rejected at submission.

PyTorch Checkpointing Example

Here's a complete example showing how to implement checkpointing in PyTorch:

import os
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
import time

# Define your model
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(784, 512),
            nn.ReLU(),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Linear(256, 10)
        )

    def forward(self, x):
        return self.layers(x)

# Initialize model, optimizer, loss function
model = MyModel()
optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

# Checkpoint directory
checkpoint_dir = './checkpoints'
os.makedirs(checkpoint_dir, exist_ok=True)
checkpoint_path = os.path.join(checkpoint_dir, 'model_checkpoint.pt')

# Training configuration
start_epoch = 0
total_epochs = 100
best_accuracy = 0.0

# Check if a checkpoint exists
if os.path.isfile(checkpoint_path):
    print(f"Loading checkpoint from {checkpoint_path}")
    checkpoint = torch.load(checkpoint_path)
    model.load_state_dict(checkpoint['model_state_dict'])
    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    start_epoch = checkpoint['epoch'] + 1
    best_accuracy = checkpoint['best_accuracy']
    print(f"Resuming from epoch {start_epoch} with best accuracy {best_accuracy:.4f}")
else:
    print("No checkpoint found, starting from scratch")

# Training loop with periodic checkpointing
for epoch in range(start_epoch, total_epochs):
    # Training code here...
    # ...

    # Calculate accuracy on validation set
    accuracy = 0.93  # This would be your actual validation accuracy

    # Save checkpoint periodically (every 5 epochs and when best accuracy improves)
    if (epoch + 1) % 5 == 0 or accuracy > best_accuracy:
        if accuracy > best_accuracy:
            best_accuracy = accuracy

        checkpoint = {
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'best_accuracy': best_accuracy,
            'training_time': time.time()  # Store current time for reference
        }

        # Save checkpoint
        torch.save(checkpoint, checkpoint_path)
        print(f"Checkpoint saved at epoch {epoch+1} with accuracy {accuracy:.4f}")

Job Script with Checkpointing

#!/bin/bash
#SBATCH --job-name=pytorch_training
#SBATCH --output=training_%j.log
#SBATCH --partition=gpu
#SBATCH --gpus=1
#SBATCH --cpus-per-task=4
#SBATCH --time=08:00:00

# Activate your environment if needed
source ~/venv/bin/activate

# Save this file as train_job.sh and submit it once: sbatch train_job.sh
DONE_FLAG=./checkpoints/DONE
LEG=${LEG:-1}
MAX_LEGS=10          # safety cap: never chain more than 10 x 8 h

if [ -f "$DONE_FLAG" ] || [ "$LEG" -ge "$MAX_LEGS" ]; then
  echo "training finished (or leg cap reached) - not resubmitting"
  exit 0
fi

# Queue the next leg BEFORE doing any work. A job that hits its time limit is
# killed (SIGTERM, then SIGKILL 30 s later) and never executes lines after the
# training command, so a resubmit placed at the end would never run. The next
# leg starts only after this job ends (afterany) and exits at once if training
# already finished.
sbatch --dependency=afterany:$SLURM_JOB_ID --export=ALL,LEG=$((LEG+1)) \
       "$SLURM_SUBMIT_DIR/train_job.sh"

# Run the training script; it resumes from the latest checkpoint and the DONE
# flag is created only when the final epoch completes
python train_with_checkpoints.py && touch "$DONE_FLAG"

The script above resubmits itself before each leg, so one sbatch train_job.sh is enough; cancel the whole chain with scancel --name=pytorch_training.

Getting Help

For cluster-specific questions or issues, contact the TTIC system administrators at admin@ttic.edu.

For general Slurm usage questions, refer to the Slurm documentation or use the man pages:

man sbatch
man squeue
man sinfo