GNU Parallel

Overview

GNU parallel is a command-line program that allows repeating the same script in parallel on multiple CPU cores and machines by changing the input through variable replacement or piping its output from another command. This can be done through a single line of code, and is a great way to submit one Slurm script that runs the same command when only a single value (such as an integer) needs to be changed.

Parallel can:

Repeat the same script by passing it a range of numbers, such as 1-10000
Repeat the same script by passing multiple variable arguments, which wil create an input combination for each script (A,B,C D,E,F = AD,AE,AF,BD,BE…)
Process multiple items in a list (such as 'ls -l') when the output is piped to parallel
Automate and speed up a task involving loops to use multiple CPU cores for each iteration

Another beneficial feature of parallel is that it keeps track of what it has already finished within a log file, so you can submit the same job and it'll continue where it left off.

Availability

Cluster	Module/Version
BOSE	parallel/20220622
BGSC	parallel/20220622

Note: You can simply use module load parallel to activate the most recently installed version of this software.

Arguments / Options

This is a list of arguments for the parallel command that we wanted to highlight. Use man parallel or https://www.gnu.org/software/parallel/man.html for a full list.

Option	Description
--dry-run	Send executed commands to run log, but don't actually execute then. (Used for debugging / testing)
--jobs # -j #	Number of jobs to run at one time. This should match the number of CPU cores your processing script requires.
--resume	Resume parallel from the last unfinished job / variation when rerunning the full command. Use with --joblog to keep track.
--resume-failed	Retry failed jobs, then resume parallel from last unfinished job (like --resume). Use with --joblog to keep track.
--joblog filename	File used to keep track of the jobs that parallel has already executed. If using with --resume, and would like to start over, you'll have to delete this file.

Converting Existing Script

When converting an existing script to start utilizing GNU parallel, you'll want to ask yourself these questions:

Am I running the same calculation multiple times by only changing an file, number, or variable?
Can I modify my script to pass in a file, number, or variable as an argument (Ex: my-script.sh 1, my-script.sh 200, my-script.sh file.jpg)?
How many resources does each calculation require on its own?
How many calculations do I want to do at one time? (More calculations = requires more resources from Slurm)

Once you answered those questions, you can use the following sample script as a starting point.

Sample Slurm Script

The following script gives you a basis for starting to incorporate GNU parallel into your work. You'll want to identify how many resources each individual run (indicated as 'my-script.sh' at the end) requires by modifying the "Parallel Settings" section, then determine how many runs you want to do at once by changing your overall Slurm resource request.

submit.sh

#!/bin/bash
# -- SLURM SETTINGS -- #
# [..] other settings here [..]

# The following settings are for the overall request to Slurm
# - Each parallel run uses a portion of this
#SBATCH --ntasks-per-node=32     # How many CPU cores do you want to request
#SBATCH --nodes=1                # How many nodes do you want to request

# -- SCRIPT COMMANDS -- #

# Load the needed modules
module load parallel    # Load GNU Parallel

# Parallel Settings
job_num_nodes=1        # Number of nodes to use per individual run
job_num_cores=8        # Number of cores to use per individual run
job_log="runtask.log"     # Location of job execution log (delete file if starting over)

# Identify how many jobs to run at one time within the overall Slurm allocation
num_jobs=$((job_num_nodes * SLURM_NTASKS / num_job_cores))

# Build the 'srun' command to isolate resources per run (similar to using sbatch)
cmd_srun="srun --exclusive --nodes=${job_num_nodes} --ntasks=1 --cpus-per-task=${job_num_cores}"

# Build the initial 'parallel' command
cmd_parallel="parallel --jobs ${num_jobs} --joblog=${job_log} --resume"

# Build the final script command
# -- Replace 'my-script.sh' with your own script
# -- Replace {1..1000} with your own range of numbers
# ---- {} after my-script.sh is replaced with each number on execution
$cmd_parallel "$cmd_srun my-script.sh {}" ::: {1..1000}

Real Example

The Bhattacharyay Lab used ADFR to study the binding of a small molecule onto the surface pockets of proteins through the process known as docking. The computational docking process provides a quantitative estimate (usually in a free energy term) of how tightly the small molecule is bound to the pocket of the protein. By using GNU parallel, the lab was able to process a large number of small molecules all at once through the use of a single Slurm script.

By requesting two full nodes with 64 cores each, and allocating eight cores to each docking run in their Slurm script (submit.sh), they were able to process up to 16 molecules at the same time. By passing in a range of numbers (1-261), GNU parallel automatically passed along each number as an argument to their ADFR script (docking_part2.sh) until all targeted molecules were finished.

submit.sh

#!/bin/bash

#SBATCH --nodes=2        # Requested Two Nodes
#SBATCH --ntasks-per-node=64    # Requested 64 Cores (128 Overall)

# -- SCRIPT COMMANDS -- #
module load adfrsuite/1.0
module load parallel

# Parallel Settings
srun="srun --exclusive --nodes=1 --ntasks=1 --cpus-per-task=8"        # Requested 8 cores per each adfr run
parallel="parallel --delay 0.2 -j 16 --joblog runtask.log --resume"   # Supports running up to 16 adfr runs at one time, delaying each run by .2 seconds before starting
$parallel "$srun ./docking_part2.sh {}> docking_part2.log" ::: {1..261}  # Passed incremented number to docking_part2.sh as an argument "{}" == "{1..261}"

docking_part2.sh

#!/bin/bash

adfr -c 8 -t ../target.trg -l $1.pdbqt -J $1.dock    # Takes the number from parallel as an argument to read input files.

Other Use Cases

Sequential Numbers

This script passes in a range of numbers (1-1000) to another command as an argument while running only 20 numbers / processes at a time.

parallel --jobs 20 my-program.sh {} ::: {1..1000}

my-program.sh

# File: my-program.sh
echo "I am number $1"

Image Thumbnail Creation

This script lists all jpgs in a folder and creates smaller thumbnails by using one CPU core per image for a max of four at one time. The "{}" is replaced with the listed file name into the convert command.

ls *.jpg | parallel --jobs 4 convert -geometry 120 {} thumb_{}

Citation

Please include the following citation in your papers to support continued development of GNU Parallel. Further details are available when you use the command parallel --citation.

Tange, O. (2022). GNU Parallel 20220622 ('Bongbong’). doi:10.5281/zenodo.6682930