CS 388 - HPC Lab
Scenario: A faculty member has came to us to build a system that can run their program "NAMD". Through initial research we have determined that NAMD is a program that takes advantage of high-performance of computing to simulate molecules.
The goal of this activity is to share a little what it is like to be an HPC administrator by building software that takes advantage of high-performance computing infrastructure.
While Monday's lecture was an overview of the process to generally build a computing cluster, it's hard to actually build a cluster as a class activity, so you'll be installing software instead.
NAMD
Website: https://www.ks.uiuc.edu/Research/namd/
Built by a group at the University of Illinois at Urbana-Champaign, NAMD is a software used in the chemistry field to simulate molecules in a 3D space and predict how they'd move over time based on various calculations.
For our lab, we are going to run the 'minimize' operation on our final program to take a protein structure and determine the minimum amount of energy required to retain the bonds for a molecular structure.
NAMD was built to be highly scalable and work across not only several CPU cores, but potentially run on upwards to thousands of computational servers at one time.
Log Into BOSE
For today's lab we are going to remotely log into the BOSE supercomputing cluster using terminal / command prompt.
If you are off campus, you will be required to use UWEC's VPN before you can connect: https://uwec.ly/vpn
Move to your "my_cs388" directory
Almost all work will be done under the my_cs388 directory
Copy Lab Files
Before we begin, we'll want to copy over all of the demo files for this lab.
Copy Files:
Verify files:
The above command should list several files, such as ionized.pdb
Run Interactive Session
Do this each time you log into the cluster and before you run any of the following steps. This will place you on one of our computational nodes and give you 16 CPU cores for building.
You can confirm this is working by looking at your prompt and seeing the host become "cn##".
Load Modules
Compiling our version of NAMD has two dependencies that we want to use: GCC 11.1.0, and OpenMPI 4.1.6. By default, these are not 'activated' in your environment, so you will need to load them using our module system. All of our computational software is ran though this system.
Confirm successfull activate of modules
GCC should be version 11.1.0, and mpirun should be 4.1.6.
If you want to see any of the other modules available on our cluster, you can use:
Obtaining the NAMD Source
Operating System: Linux (We specifically use the Rocky Linux and CentOS distributions)
Source Code: https://www.ks.uiuc.edu/Development/Download/download.cgi?PackageName=NAMD
Already Provided
The source is already provided for you, but it is free to 'register' for an account if you want to do this on your own.
Make a folder in your my_cs388 directory to house source files
Extract the source code to that new folder
The above command extracts the source code from the .tar.gz file and places it inside the namdsrc folder rather than its default "NAMD_3.0b4_source".
Why Source Code?
Generic binaries are significantly easier to use because they are pre-built for you, however they may not be optimized for your exact setup or contain the exact features you may want. By compiling the source code yourself, you have full control on what the program contains (such as multi-node or GPU support) as well as the matching instruction sets for your CPU (see AVX2 or AVX-512) for improved performance.
Building Charm++
Charm++ is a parallel programming framework used by NAMD (and several others) to build in support for running across multiple CPU cores or multiple nodes. This software was also built at the University of Illinois at Urbana-Champaign.
Charm++ is distributed with the NAMD source code, so we will not need to directly download it.
Website: https://charmplusplus.org/
Enter your NAMD source files directory
Extract the source code for Charm++ and enter the new folder
Build Charm++
Hooking into the OpenMPI module we loaded earlier, this builds the MPI version of Charm++ to support running across multiple nodes.
You've successfully completed this step if it says "Build successful."
$SLURM_NTASKS
$SLURM_NTASKS is an environment variable provided to us by the cluster's Slurm scheduling system. It automatically stores the number of CPU cores assigned to this session/job, which you specified when you did sinteract --ntasks=16.
You can see this in action by running:
Should show 16 as you requested 16 cores.
Building Support Libraries
While not required, NAMD uses the TCL and FFTW libraries to build in support for additional features. We will also install these.
Head back to your base NAMD source code directory
Download the TCL and FFTW source files
wget http://www.ks.uiuc.edu/Research/namd/libraries/fftw-linux-x86_64.tar.gz
wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl8.5.9-linux-x86_64.tar.gz
wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl8.5.9-linux-x86_64-threaded.tar.gz
Starting with FFTW - extract the files and rename the folder
Do the same with TCL
tar xvzf tcl8.5.9-linux-x86_64.tar.gz
tar xvzf tcl8.5.9-linux-x86_64-threaded.tar.gz
mv tcl8.5.9-linux-x86_64 tcl
mv tcl8.5.9-linux-x86_64-threaded tcl-threaded
When NAMD is being built, it relies on those exact folder names to find the files: "fftw", "tcl", and "tcl-threaded", which is why they were renamed.
Compiling + Installing NAMD
Time to finally start compiling NAMD!
Configure NAMD
Make sure you are in your namdsrc folder, and configure NAMD.
Build NAMD
Move your NAMD installation to your home directory with the name 'namd'
Look inside your namd folder with 'ls' and you should see the "namd3" binary. If so, you were successful!
Module File
Just like you used module load to load GCC and OpenMPI previously, you'll need to create your own module for NAMD.
What we're going to do is create a custom directory to hold all of your custom modulefiles, create one for namd called "my-namd", and then copy a rebuilt version over.
Note: This will "not" be in your my_cs388 directory.
Create the modulefiles directory under your HOME directory
Create the "my-namd" module
Verify the module works by loading the module and seeing if it finds the "namd3" binary
Should say "~/my_cs388/my-namd/namd3"
You can now close out of your interactive session by just typing "exit".
Should say [@bose ~] (or different path)
Slurm
To run your script, you are going to modify a pre-built Slurm script.
Modify the file using 'vim' and add the three required modules at the bottom
Add the following lines to the bottom of the file:
module use ~/modulefiles
module load gcc/11.1.0
module load openmpi/4.1.6
module load my-namd/3.0b4
Add the namd3 command to actually run the program
Your final version should look something like this:
# Put your commands here
module use ~/modulefiles
module load gcc/11.1.0
module load openmpi/4.1.6
module load my-namd/3.0b4
mpirun namd3 minimize.namd > minimize.out
Running Your Program
To submit a Slurm job to our scheduling system, you use the "sbatch" command.
Verify the job is running by typing and looking for "NAMD Job"
The NAMD script takes about 20 minutes to run. Once finished, you'll see an email about your job and how much resources it used. On the cluster, you should now see several new files in your ~/my_cs388/namd folder.:
If that exists, you have successfully ran NAMD, and our faculty can continue their research by using a high-perforamnce computing cluster!