Walk Me Through

2.1 MPI (CPU) job

The Message Passing Interface (MPI) is a standardized and portable system for communication between the various tasks of parallized jobs in HPC environments. A number of different implementations of MPI libraries are available at our cluster. Although the MPI interface itself is somewhat standardized, the different versions are not binary compatible. It is important that you match the MPI implementation you use and with which your code was compiled. The recommended MPI library on BA-HPC cluster is Intel MPI libraries.

Let's start by compiling a sample MPI program written in C. The program initialize a defined number of processes that print the 'Hello World' line to a file along with process rank. The source code for this program can be found at this github gist. To start using MPI environment, load the intel impi module using: [username@login01 ~]$ module load impi Let's use our newly loaded module to compile our C program, using MPI C compiler wrapper mpicc. Execute [username@login01 ~]$ mpicc hello-mpi.c -o hello-mpi.bin Now that we got our binary file, let's create a job script to submit it to OGS.

Here's an example of a simple script that will specify the necessary job parameters, we'll call it hello-mpi.sh:

                    
#! /bin/bash
#SBATCH --job-name=mpi_job
#SBATCH --ntasks=24
#SBATCH --cpus-per-task=1
#SBATCH --time=00:15:00

mpirun -np 24 ./hello-mpi.bin
                    
                

1. #SBATCH --job-name=mpi_job specify the job name.
2. #SBATCH --ntasks=24 restart the job in the case the system has a crash or is rebooted.
3. #SBATCH --cpus-per-task=1 specifies the number of cores per task, we will need only one core per process.
4. mpirun -np 24 ./hello-mpi.bin run the MPI executable and specifies number of processes.

Now that you have a job script, you need to submit the job to the cluster with the sbatch command. Make sure the Intel MPI libraries are loaded first then use 'sbatch' to submit the job to the scheduler

                        
[username@login01 ~]$ module list
Currently Loaded Modulefiles:
  1) GCCcore/5.4.0
  2) binutils/2.26-GCCcore-5.4.0
  3) icc/2016.3.210-GCC-5.4.0-2.26
  4) ifort/2016.3.210-GCC-5.4.0-2.26
  5) iccifort/2016.3.210-GCC-5.4.0-2.26
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26

[username@login01 ~]$ sbatch hello-mpi.sh
Submitted batch job 156
                        
                        

At this point, your job has been placed in the queue, and will wait its turn for resources to be available. Depending on how heavily used the cluster is at that time, and how many resources you are requesting, your job might start within minutes or it might wait for hours.

Once resources become available, our scheduler will assign resources to your job, including one or more nodes.

The standard output and standard error streams will be directed to a file, by default slurm-.output in the directory where you started the job, where the job-id is the job number as described above.

Output from your job can be viewed in the above specified file shortly after it starts running (assuming it has output something). This can be used to check the status of your job, although it is recommended make your code generates a lot of output to redirect it to another file.

For our trivial example from the last section, when the job completes we should see something like

                                
[username@login01]$ cat slurm-156.output
Hello world: rank 12 of 24 running on comp085.local
Hello world: rank 1 of 24 running on comp085.local
Hello world: rank 2 of 24 running on comp085.local
Hello world: rank 4 of 24 running on comp085.local
Hello world: rank 7 of 24 running on comp085.local
Hello world: rank 8 of 24 running on comp085.local
Hello world: rank 9 of 24 running on comp085.local
Hello world: rank 14 of 24 running on comp085.local
Hello world: rank 15 of 24 running on comp085.local
Hello world: rank 16 of 24 running on comp085.local
Hello world: rank 17 of 24 running on comp085.local
Hello world: rank 18 of 24 running on comp085.local
Hello world: rank 20 of 24 running on comp085.local
Hello world: rank 21 of 24 running on comp085.local
Hello world: rank 0 of 24 running on comp085.local
Hello world: rank 3 of 24 running on comp085.local
Hello world: rank 5 of 24 running on comp085.local
Hello world: rank 6 of 24 running on comp085.local
Hello world: rank 10 of 24 running on comp085.local
Hello world: rank 11 of 24 running on comp085.local
Hello world: rank 13 of 24 running on comp085.local
Hello world: rank 19 of 24 running on comp085.local
Hello world: rank 22 of 24 running on comp085.local
Hello world: rank 23 of 24 running on comp085.local
                            
                        

As you can see in the output files above, the MPI program executed and each process was assigned a unique rank, which was printed off along with the hostname.

2.2 CUDA (GPU) job

CUDA is a parallel computing platform and API model created by Nvidia. It allows you to use a CUDA-enabled GPU for general purpose processing – an approach termed GPGPU (General-Purpose computing on Graphics Processing Units). The CUDA platform is a software layer that gives direct access to the GPU's virtual instruction set and parallel computational elements, for the execution of compute kernels.

Again let's start by compiling a sample CUDA program written in C. The program use GPU to add two 2 vectors of integers in parallel. It starts by generating 2 vectors of size n then pass these vectors to the GPU memory, after that we make each core simply sums a single element from each of the two input vectors and writes the result into the output vector. Finally, we print only m number of elements into the output file. The source code for this program can be found at this github gist . To start using the CUDA library, let's load Intel C compiler and CUDA [username@login01 ~]$ module load icc CUDA Then let's use the loaded Nvidia CUDA Compiler (NVCC) to compile our source code [username@login01 ~]$ nvcc vector-add.cu -o vector-add.bin

An example of a simple script that will specify the necessary job parameters for a GPU based job, we'll call it cuda-vec_add.sh:

                    
#!/bin/bash
#SBATCH --job-name=first-cuda-job
#SBATCH --account=g.projectname
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=00:15:00

./vector-add.bin 100000 10
                    
                

There's some additional options we've put in our job script:

1. #SBATCH --account=g.account_name Specify your GPU slurm account.
2. #SBATCH --partition=gpu Submit the job to the gpu partition.
3. #SBATCH --gres=gpu:1 Specify the number of GPUs. In this example we only need one GPU card.
4. #SBATCH --nodes=1 Specify the number of required nodes. In this example we only need one node.
5. #SBATCH --ntasks=1 Specify the number of CPU cores/process to be used. In this example we only need one process to initiate our program execution on the GPU. Maximum number of CPU cores/process to be used is 16.
6. ./vector-add.bin 100000 10 Generate two vectors of length 100000, and print only the first 10 elements in the output file.

Now that you have a job script, let's submit the job to the cluster. Make sure the CUDA library are loaded first then use 'sbatch' to submit the job to the scheduler

                    
[username@login01 ~]$ module list
Currently Loaded Modulefiles:
1) CUDA/8.0.44

[username@login01 ~]$ sbatch cuda-vec_add.sh
Submitted batch job 157
                    
                

After the job completes, we should see something like

                        
[username@login01]$ cat slurm-157.out
h_x = 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0

h_y = 100000.0 99999.0 99998.0 99997.0 99996.0 99995.0 99994.0 99993.0 99992.0 99991.0

The sum is:
100001.0 100001.0 100001.0 100001.0 100001.0 100001.0 100001.0 100001.0 100001.0 100001.0

                    
                

2.3 Gromacs job

In this section, you will see to run a gmx mdrun example, which is the "main computational chemistry engine within GROMACS. This sbatch script example uses GPU resources.

You will need to load the To start using the libraries you need through loading the module.

You will need to get the gromacs sbatch script from BA-HPC repository on github.

Open the gromacs sbatch script to edit the account name and the command to run your code or may be the change the GPU specs.

 
#!/bin/sh
#SBATCH --job-name=Apo
#SBATCH --partition=gpu
#SBATCH --account=<g.account_name>
#SBATCH --cpus-per-task=16
#SBATCH --gres=gpu:2
#SBATCH --time=120:00:00

gmx mdrun -nt 16 -s my_run.tpr -deffnm my_run

After editing and closing the file through ctlr + X then choose yes, you can submit the sbatch to slurm to run your code.

2.4 Quantum Espresso job

Quantum ESPRESSO is an open-source plane-wave periodic density functional theory code and considered one of the products of (SCM) Software for Chemistry and Materials.

You can check an example for input file scf.in.

For QE 6.3, you will need to load the required modules for running QE 6.3 [username@login01 ~]$ PATH="/share/apps/q-e-qe-6.3/bin:$PATH [username@login01 ~]$ module load intel/2018

For QE 6.7, you can easily use recent QE from the environment modules

Here's an example of a submission script called pw.x.sh for Quantum Espresso job.

 
#!/bin/sh
#SBATCH --job-name=qe-job
#SBATCH --account=<account_name>
#SBATCH --ntasks=24
#SBATCH --cpus-per-task=1
#SBATCH --time=03:00:00

mpirun -np "$SLURM_NTASKS" pw.x scf.in > opt.out

2.5 Python job

Using Anaconda3 module

Anaconda is a distribution of the Python and R programming languages for scientific computing that wil provide you with the packages you need by excuting the following

[username@login01 ~]$ module load Anaconda3
[username@login01 ~]$ source activate /cluster/envs/ba-hpc.bak
[username@login01 ~]$ conda list 

conda list will show you the available libraries we have, that most researchers use on the BA-HPC. If the anaconda library you need does not exist, you can mail us at hpc@bibalex.org.

Create your special conda environment

[username@login01 ~]$ module load Anaconda3
[username@login01 ~]$ conda create -n <env_name> python=3.8
[username@login01 ~]$ conda activate  <env_name>
 

Using Python pip

You can use pip after activating the ba-hpc env. Besides, --user must be specified in all pip commands.

Actions below, are quite helpful when cache files crowdens your home more than 100 MB. So, you will move bin and lib directories to data directory and create symbolic links under .local which refrence the actual moved files. The same idea goes for .cache directory too.

[username@login01 ~]$ cd .local 
[username@login01 ~]$ mv bin lib ../data/ 
[username@login01 ~]$ ln -s ../data/bin . 
[username@login01 ~]$ ln -s ../data/lib .
[username@login01 ~]$ cd ..
[username@login01 ~]$ cd .cache
[username@login01 ~]$ mv pip ../data/pip_cache 
[username@login01 ~]$ ln -s ../data/pip_cache pip 

Working with Python script

We have a code repository which is an accumulator for scripts intended as executable examples for users to carry out tasks related to the TensorFlow machine learning framework on a High-Performance Computing (HPC) system.

Listing GPU devices

The source code for this program and the submission script can be found at BA-HPC code repository.

Dynamic Recurrent Neural Network

A simple Tensorflow implementation of a Recurrent Neural Network (LSTM) that performs dynamic computation over sequences with variable length on a toy dataset. The source code for this program can be found at this link. The scripts to fetch and run this program can be found at BA-HPC code repository.

Convolutional Neural Network

A tensorflow tutorial demonstrates training a simple Convolutional Neural Network (CNN) to classify CIFAR images, which is a frequently used benchmark for image classification tasks. The source code for this program can be found at this link The scripts to fetch and run this program can be found at BA-HPC code repository.