Department of Computing GPU Cluster Guide

Open a Terminal window from a lab PC  (Ubuntu/macOS, Windows 10/11 use Powershell built-in ssh or WSL/2), and type the following commands*:

ssh gpucluster2.doc.ic.ac.uk
# or ssh gpucluster3.doc.ic.ac.uk
sbatch /vol/bitbucket/shared/slurmseg.sh

In this example, a user first logs into a Slurm head node (gpucluster2.doc.ic.ac.uk via ssh) and then submits a pre-existing script using the sbatch command. The output will be stored, by default, in the user's ~/ home directory, with the filename slurm20-{xyz}.out (or from whichever directory the user happens to run an sbatch command from)

If you have a bash script ready, replace /vol/bitbucket/shared/slurmseg.sh

Follow the next steps to learn how to prepare your own scripts for submission.

If connecting  from your own computer or device, make sure you specify your College Username, an example:

gpucluster2.doc.ic.ac.uk and gpucluster3.doc.ic.ac.uk are now accessible from outside the college network, but if for some reason they are not accessible, use shell[1-5].doc.ic.ac.uk as a JumpHost:

ssh YourCollegeUserName@shell5.doc.ic.ac.uk YourCollegeUserName@gpucluster2.doc.ic.ac.uk

This 'interactive' method allows you to work as if you were using a terminal prompt on a Lab PC with GPU (for a maximum of four days)

Connect to gpucluster2 or gpucluster3 from a Lab PC or externally from your own device, use 'salloc' to queue your CPU, RAM and GPU resources.

Salloc will drop you straight into your allocated node, as indicated by the shell prompt eg. myaccount@gpuvm01

Run 'nvidia-smi' to show your allocated GPU. You can now commence writing your scripts and debugging with an Nvidia GPU

In addition you can ssh directly to the node hosting your GPU, as long as your job is running in the queue. Type the following command either on the headnode (gpucluster2 or gpucluster3) or in your running session, eg gpuvm01, etc:

There is a department-wide network share /vol/bitbucket for data and virtual environment storage. Create your personal folder as follows:

mkdir -p /vol/bitbucket/${USER}

Read the detailed Python Virtual Environments guide for best practice in using /vol/bitbucket and creating virtual environments.

Tip: shared folders such as /vol/bitbucket or your home directory /homes/username are vital to get your scripts running on remote GPU cluster nodes. On your own laptop or computer, you would store files on local storage but for the GPU cluster, make sure you copy all necessary files to shared storage, so your scripts can access files regardless of which server they are running from.

Here are some examples how you might use /vol/bitbucket in the course of a GPU cluster project.

Installation of Python Virtual Environment:

# connect to a random lab PC - remember to use a lab PC to create envs, use pip and git
ssh shell1.doc.ic.ac.uk
/vol/linux/bin/sshtolab
cd /vol/bitbucket/${USER}
python3 -m virtualenv /vol/bitbucket/${USER}/myvenv

Again, consult the Python Virtual Environment guide for more about managing virtual environments in your account.

There exists a 'base' read-only environment, located at /vol/bitbucket/starter with Pytorch and tensorflow pre-installed using 'pip' and may suffice when first submitting jobs. Enable this in scripts using 'source /vol/bitbucket/starter/bin/activate'

Follow the previous steps when you need to create an environment using your specific required pip/conda packages.

Most GPU jobs will make use of the Nvidia CUDA tool-kit. Multiple versions of this tool-kit are available under /vol/cuda (network share). Inside those directories are numbered sub-directories for different versions of the CUDA tool-kit. If you need to use CUDA, please consult the README under any one of those directories.

Suppose that you want to use CUDA tool-kit verson 12.0.0, add the following line/s to your submission script:

If your shell is bash; note the initial dot-space (.␣)

. /vol/cuda/12.0.0/setup.sh

OR if your shell is (t)csh

source /vol/cuda/12.0.0/setup.csh

The script will set up your unix path to access commands such as nvcc.

If you are using frameworks such as TensorFlow, PyTorch and Caffe, make sure you have chosen a compatible version of the Nvidia CUDA tool-kit. For example, Pytorch comes in CPU and GPU flavours, but also different versions of CUDA - sourcing the matching CUDA distribution from /vol/cuda will help reduce errors in your output.

Here is a template you can copy to a shell script to get started. Please adjust any paths that may point to folders you have created.

IMPORTANT: This example assumes you have followed the previous steps and installed a python environment (using virtualenv, extra lines may  be needed using minconda, check the example script furthe below) as directed. Please adjust paths if you have an existing python environment, or if you already load your environment in ~/.bashrc (note: sbatch does not load ~/.bashrc, source it as per example script) . Do not uncomment #SBATCH lines, keep them as below, make sure the #SBATCH directives are directly after #!/bin/bash

Remember to make your script executable (run this command in a shell, do not include it in your script):

chmod +x <script_name>.sh

Please note,  environment variables from ~/.bashrc or ~/.cshrc are not loaded by sbatch-submitted scripts, you should source them as in the preceding script. Your script can access your own home directory, your /vol/bitbucket folder or shared volumes such as /vol/cuda

gpucluster2.doc.ic.ac.uk and gpucluster3.doc.ic.ac.uk are submission hosts for the GPU cluster, from where you run the sbatch command to send your scripts to the remote GPU host servers.

Here is an example of the steps involved in submitting your script as a Slurm job:

1. Connect to a slurm submission host (see step 2 for connecting from your own laptop):
   
   

   ssh gpucluster2.doc.ic.ac.uk # or ssh gpucluster3.doc.ic.ac.uk

   
   
2. Change to an appropriate directory on the host:
   
   

   # this directory may already exist after Step 3 mkdir -p /vol/bitbucket/${USER} cd /vol/bitbucket/${USER}

   
   

3. Now try running an example job. A simple shell-script has been created for this purpose. You can view the file with less, more or view. You can use the sbatch command to submit that shell-script to run it as a Slurm job on a GPU host:

   sbatch /vol/bitbucket/shared/slurmseg.sh

   If you have composed your own script, in your bitbucket folder, for example, enter:
   

   cd /vol/bitbucket/${USER} sbatch /path_to_script/my_script.sh

   Substitute '/path_to_script/my_script.sh' for your actual script and path name.
   

4. You can invoke the squeue command to see information on running jobs:
   
   squeue
   
   

5. The results of sbatch will output to the directory where the command was invoked, eg /vol/bitbucket/${USER}. The filenames will be derived from the invoked command or script – for example:

   less slurm-XYZ.out
   
   where XYZ is a unique Slurm job number. Visit the FAQ below to find out how to customise the job output name

The GPU hosts (or nodes) each contain :

For example, to target a T4 GPU (taught students):
sbatch --partition gpgpuC /path/to/script.sh

*Research/PhD users are automatically assigned 'resgpu' versions of the above, but a smaller pool, please make use of the college HPC cluster for more resources

1. What GPU cards are installed on the GPU hosts?
   Answer: Nvidia Tesla A30 (24GB RAM split into 12GB instances), Tesla T4 (16GB RAM), Tesla A40 (48GB RAM) and Tesla A100 (80GB split into 10GB instances)
   
   
2. What are the general platform characteristics of the GPU hosts?
   Answer: 24-core/48 thread Intel Xeon CPUs with 256GB RAM and AMD EPYC 7702P 64-Core CPUs
   
   
3. How do I see what Slurm jobs are running?
   Answer: invoke any one of the following commands on gpucluster:
   
   # List all your current Slurm jobs in brief format
   squeue
   # List all your current Slurm jobs in extended format.
   squeue -l
   
   Please run man squeue on gpucluster for additional information.
   
   
4. How do I delete a Slurm job?
   Answer: First, run squeue to get the Slurm job ID from the JOBID column, then run:
   
   scancel <job ID>
   
   You can only delete your own Slurm jobs.
   
   
5. How many GPU hosts are there?
   Answer: As of July 2023, there are nine host GPU servers, with eight running DoC Cloud GPU nodes.
   
   
6. How do I analyse a specific error in the Slurm output file/e-mail after running a Slurm job?
   Answer: If the reason for the error is not apparent from your job’s output, make a post on the Edstem CSG board , including all relevant information – for example:
   
   • the context of the Slurm command that you are running. That is, what are you trying to achieve and how have you gone about achieving it? Have you, created a Python virtual environment? Are you using a particular server or deep learning framework?
   • the Slurm script/command that you have used to submit the job. Please include the full paths to the scripts if they live under /vol/bitbucket
   • what you believe should be the expected output.
   • the details of any error message displayed. You would be surprised at how many forget to include this.
     
     
7. I receive no output from a Slurm job. How do I go about debugging that?
   Answer: This is an open-ended question. Please first confirm that your Slurm job does indeed generate output when run interactively. You may be able to use one of the 'gpu01-36' interactive lab computers to perform an interactive test. If you still need assistance, please follow the advice in the preceding FAQ entry (Number vi).
   
   
8. How do I customise my job submission options?
   Answer: Add a Slurm comment directive to your job script – for example:
   
   # To request 1 or more GPUs (default is 1):
   #SBATCH --gres=gpu:1
   
   # To request a 48GB Tesla A40 GPU:
   #SBATCH --partition gpgpu
   # or 80GB A100 GPU
   #SBATCH --partition AMD7-A100-T
   # Please note, there are only a few 48GB/80GB GPUs available, interactive jobs are not permitted
   # For other GPUs, refer to 6b. GPU types, including the research equivalents of the above
   
   # To receive email notifications
   #SBATCH --mail-type=ALL
   #SBATCH --mail-user=<your_username>
   
   #Customise job output name
   #SBATCH --output=<your_job_name>%j.out
   
   
9. How do I run a job interactively?
   Answer: Use srun and specify a gpu, and other resources. eg. for a bash shell:
   
   srun --pty --gres=gpu:1 bash
   
   Update: use 'salloc' as detailed in Step 2c
   
   
10. I need a particular software package to be installed on a GPU host.
    Answer: Have you first tried installing the package in a Python virtual environment or in your own home directory with the command:
    
    pip install --user <packagename>
    
    If the above options do not work then make a post on the Edstem CSG board with details of the package that you would like to be installed on the GPU server(s). Please note: CSG are only able to install standard Ubuntu packages if doing so does not conflict with any exisiting package or functionality on all the GPU servers.
    
    
11. My job is stuck in queued status, what does this mean?
    Answer: This could be because all GPUs are in use. PD status occurs if you are already running two jobs, and will run (R) when one of your previous tasks is complete. (QOSMaxGRESPerUser) means you are using your maximum of two GPUs at any one time.
    
    
12. What are the CUDA compute capabilities for each GPU?
    Please consult the NVIDIA Compatiiblity Index for more information.
    The cluster GPUs support the following levels:
    sm75 (T4), sm80 (A30), sm86 (A40)
    These should be considered when, for example, using older versions of Pytorch and receiving 'not supported' errors

The following policies are in effect on the GPU Cluster:

• User can have two running jobs only (taught students), all other jobs will be queued until one of the two jobs completes running
• A job that runs for more than four days will be automatically terminated - this is a walltime restriction for taught students - configure checkpoints with your python framework to resume training.
• As with all departmental resources, any non-academic use of the GPU cluster is strictly prohibited.
• Any users who violate this policy will be banned from further usage of the cluster and will be reported to the appropriate departmental and college authorities.

ICT, the central college IT services provider, has approximately one-hundred CX1 cluster nodes which have GPUs installed. It is possible to select and use these computational resources through PBS Pro job specifications.

Students cannot request access to this resource but project supervisors can apply – on behalf of their students - for access to ICT GPU resources run by the Research Computing Service team:

Research Computing Service

Other resources

If you do not need a GPU for your computation then please do not use the GPU Cluster. You could end up inconveniencing users who do need a GPU. Please instead consider:

• The departmental DoC Condor service
  
  
• The departmental batch servers:
  Long Running Processes guide
  Long Running Processes PDF link