Software Setup

⚠️ Important: Never install software on the login node (cluster.ttic.edu). Always request an interactive job on a compute node before installing packages or setting up environments.

---

Python Virtual Environments

Python virtual environments provide isolated Python installations for each project, preventing dependency conflicts and ensuring reproducible environments. This is the recommended approach for Python development on Beehive.

The system Python on every node is Debian 12's python3 (3.11). There is no module system and no site-wide conda on Beehive; python3 -m venv plus pip is the supported workflow.

Project-Based Setup (Recommended)

First, get an interactive job on a compute node:

srun -p cpu --pty bash        # cpu partition, up to 8 hours; without -p you get dev-cpu (one node, 1-hour limit)

For better organization and isolation, create a separate virtual environment for each project. You have two location options:

Option 1: Home directory (persistent across nodes)

export PROJECT_DIR=$HOME/my_project
mkdir -p $PROJECT_DIR && cd $PROJECT_DIR

Option 2: Scratch space (faster I/O, node-specific)

export PROJECT_DIR=/scratch/$USER/my_project
mkdir -p $PROJECT_DIR && cd $PROJECT_DIR

Note: /scratch is local to each compute node and provides faster I/O, but data is not shared between nodes. Your home directory (/home-nfs/$USER) and project space (/share/project/<name>) are on NFS and visible from every node - see the Storage section of Getting Started.

1. Create a virtual environment for this project:

python3 -m venv $PROJECT_DIR/venv

2. Create a project environment file:

cat > my_project.env << EOF
export PROJECT_DIR=$PROJECT_DIR
source $PROJECT_DIR/venv/bin/activate
export PYTHONPATH=$PROJECT_DIR\${PYTHONPATH:+:\$PYTHONPATH}
echo "Activated virtual environment for my_project"
EOF

Using Your Project Environment

1. Activate the environment:

source my_project.env

2. Update pip to the latest version:

pip install --upgrade pip --no-cache-dir

3. Install packages using pip (using --no-cache-dir to save home directory space):

# Install individual packages
pip install numpy pandas matplotlib --no-cache-dir

# Or install from requirements file (if you have an existing project)
pip install -r requirements.txt --no-cache-dir

4. Save your environment:

# Generate requirements file
pip freeze > requirements.txt

This approach has several advantages:

• Each project has its own isolated Python environment
• Uses Python's built-in virtual environment system
• Dependencies are tracked in requirements.txt
• Easy to share environment specifications with collaborators
• No conflicts between different projects' dependencies

---

PyTorch Installation

PyTorch is a popular deep learning framework with excellent GPU support. This guide will help you install PyTorch with CUDA support on the Beehive cluster using the Python virtual environment approach described above.

Prerequisites

First, set up a Python virtual environment following the Python Virtual Environments section above. For this example we will be using a project named pytorch_project.

What the nodes provide. GPU nodes run NVIDIA driver 595.104.02 with CUDA 13.0.2 and cuDNN 9.14 installed under /usr/local/cuda. PyTorch wheels ship their own CUDA runtime, so you do not need to match the toolkit version - but the gpu partition includes Blackwell cards (g0, g7, g11) that require CUDA 12.8-or-newer wheels (cu128 or later). Older cu124/cu126 wheels import fine but fail at the first kernel on those nodes.

Installation Steps

1. Get an interactive job on a GPU node:

srun -p gpu -G1 --pty bash

2. Set up your PyTorch project using the virtual environment approach:

# For persistent storage (recommended for most users)
export PROJECT_DIR=$HOME/pytorch_project
# Or for faster I/O during training (node-specific)
# export PROJECT_DIR=/scratch/$USER/pytorch_project

mkdir -p $PROJECT_DIR && cd $PROJECT_DIR
python3 -m venv $PROJECT_DIR/venv

# Create project environment file
cat > pytorch_project.env << EOF
export PROJECT_DIR=$PROJECT_DIR
source $PROJECT_DIR/venv/bin/activate
export PYTHONPATH=$PROJECT_DIR\${PYTHONPATH:+:\$PYTHONPATH}
echo "Activated PyTorch environment"
EOF

3. Activate the environment and install PyTorch:

source pytorch_project.env
pip install --upgrade pip --no-cache-dir
pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 --index-url https://download.pytorch.org/whl/cu128 --no-cache-dir   # cu128 or newer: the gpu partition includes Blackwell nodes (g0, g7, g11) that cu124/cu126 wheels cannot run on

4. Test your installation:

python -c "import torch; print(torch.cuda.get_device_name(0)); print(torch.cuda.get_arch_list()); print((torch.ones(4, device='cuda') * 2).sum().item())"

This should print the GPU model, the list of compiled architectures (it must include sm_120 for the Blackwell nodes g0, g7, g11) and 8.0. If the last line fails with 'no kernel image is available', your wheels do not support this GPU - reinstall from the cu128 (or newer) index.

Using PyTorch in Future Sessions

In future sessions, you'll just need to activate your virtual environment:

source pytorch_project.env

Running PyTorch Jobs

When submitting batch jobs that use PyTorch, make sure to include the activation command in your job script:

#!/bin/bash
#SBATCH --job-name=pytorch_job
#SBATCH --output=pytorch_%j.log
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4

# Activate the virtual environment with PyTorch (use the correct path to your project)
source $HOME/pytorch_project/pytorch_project.env

# Run your PyTorch script
python train_model.py

---

Jupyter Notebook Usage

Jupyter Notebooks provide an interactive environment for code development, data exploration, and visualization. This guide shows how to run Jupyter on the Beehive cluster using the Python virtual environment approach described above.

Prerequisites

First, set up a Python virtual environment following the Python Virtual Environments section above. For this example we will be using a project named jupyter_project.

Setup and Installation

1. Get an interactive job on a node:

srun -p dev-cpu --pty bash

2. Set up your Jupyter project using the virtual environment approach:

# For persistent storage (recommended for most users)
export PROJECT_DIR=$HOME/jupyter_project
# Or for faster I/O (node-specific)
# export PROJECT_DIR=/scratch/$USER/jupyter_project

mkdir -p $PROJECT_DIR && cd $PROJECT_DIR
python3 -m venv $PROJECT_DIR/venv

# Create project environment file
cat > jupyter_project.env << EOF
export PROJECT_DIR=$PROJECT_DIR
source $PROJECT_DIR/venv/bin/activate
export PYTHONPATH=$PROJECT_DIR\${PYTHONPATH:+:\$PYTHONPATH}
echo "Activated Jupyter environment"
EOF

3. Activate the environment and install Jupyter:

source jupyter_project.env
pip install --upgrade pip --no-cache-dir
pip install notebook --no-cache-dir

Running Jupyter Notebook

You can run Jupyter Notebook as either an interactive or batch job.

Interactive Method

1. Start an interactive job:

srun -p cpu --cpus-per-task=2 --pty bash     # up to 8 hours on the cpu partition; use -p gpu -G1 for a GPU notebook

2. Set up your environment:

source jupyter_project.env       # Activate virtual env
export NODEIP=$(hostname -I | awk '{print $1}')   # Get the IP address of your node
export NODEPORT=$(( $RANDOM + 1024 ))  # Get a random port above 1024
echo $NODEIP:$NODEPORT           # Note these values for the SSH tunnel
jupyter-notebook --ip=$NODEIP --port=$NODEPORT --no-browser

3. In a new terminal on your local machine, create an SSH tunnel:

ssh -N -L 8888:$NODEIP:$NODEPORT username@cluster.ttic.edu

(Replace $NODEIP, $NODEPORT, and username with the actual values)

4. Open your local browser and visit: http://localhost:8888 The login token is displayed in the output of the jupyter-notebook command.

Batch Method

1. Create a batch job script named jupyter-notebook.sbatch:

#!/bin/bash
#SBATCH --job-name=jupyter
#SBATCH --output=jupyter_%j.log
#SBATCH --partition=cpu
#SBATCH --cpus-per-task=2

NODEIP=$(hostname -i)
NODEPORT=$(( $RANDOM + 1024))
echo "ssh command: ssh -N -L 8888:$NODEIP:$NODEPORT $(whoami)@cluster.ttic.edu"

source $HOME/jupyter_project/jupyter_project.env
jupyter-notebook --ip=$NODEIP --port=$NODEPORT --no-browser

2. Submit the batch job:

sbatch jupyter-notebook.sbatch

3. Check the job output file to find the SSH command to use when accessing your notebook.

4. Create an SSH tunnel as instructed in the output:

ssh -N -L 8888:###.###.###.###:#### username@cluster.ttic.edu

5. Open your local browser and visit: http://localhost:8888

Troubleshooting

If you're having problems with the token or password:

1. Stop the notebook server
2. Remove the runtime files:

rm -rf ~/.local/share/jupyter/runtime

3. Restart the server