#!/bin/bash
#SBATCH --job-name=mac_explore_read_histo
#SBATCH --partition=curie-best
#SBATCH --exclude=curie201,curie209
#SBATCH --ntasks=5
#SBATCH --mem=32G
# This script renders a Quarto file to obtain html output
# Assumes that user has the directory structure we have adopted for the watershed course
# The directory for the project should include a code subdirectory where the qmd file is located
# project path should be path the the project directory
# Output file will have the same name but different exension (.html instead of .qmd)
# I have been running this from within .../code/slurm in the project directory
umask 000
sif_file_for_R=${SIF_FILES}/r.sif
projectpath=~/Projects/watershedMacatawa
QMDfile=Mac_Data_Explore_Read_Histo.qmd
startsec=`date +%s` #%s gives 'epoch time' which is num secs since 1970-01-01
echo Starting job at `date`
echo Running on host `hostname`
echo Shell is $SHELL
echo Group is `id -gn`
# Execute R script
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo "Running script"
cd $projectpath
echo The working directory is `pwd`
inQMD=$projectpath/code/$QMDfile
singularity run --bind /home/byurk/water_share/raw_data $sif_file_for_R quarto render $inQMD --to html
echo Done
echo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
endsec=`date +%s`
echo Ending job at `date`
let "wallsec = $endsec - $startsec "
wallhrs=`echo "scale=2; $wallsec / 60 / 60 " | bc -l `
echo Total wallclock time was $wallhrs hours.R on HPC
Introduction
When working with large data sets computations can take a long time and can often require more resources than are available on a personal computer. In this case it is useful (sometimes essential) to run some of your computations on a shared computer with more resources. In this tutorial you will learn how to run your R code on a high performance computing cluster (HPC) that is located on Hope’s campus.
We will take a hybrid approach that takes advantage of the different strengths of the HPC and your local computer (e.g., your laptop or desktop computer). It is usually more efficient to write code, generate plots, and view numerical and graphical summaries of data on your local computer. However, sometimes we will need to offload heavier computations to the HPC. Our workflow will require us to move data and code back and forth between your local computer and the cluster to take advantage of both resources.
Prerequisites
In this tutorial we will assume that you have R and RStudio installed on your local computer, and that you are familiar with the RStudio interface. We also assume that you are familiar with RStudio projects, the standard directory structure we use for projects in the Watershed course, and Quarto documents. These are all introduced in this tutorial. Throughout this tutorial we will work with a project called watershedMacatawa. If you are working on a different project, you will simply need to replace watershedMacatawa with the name of your project.
There are various ways to interact with HPC from your local computer and to move files back and forth between your local computer and HPC. We will assume that you are using ssh to log in to HPC using your terminal application (Mac) or MobaXterm (Windows). If you are using Windows you can access the HPC filesystem within MobaXterm. If you are using a Mac you can mount your HPC filesystem using MacFuse and SSHFS. Instructions for doing these things from either on or off campus are in the documents for Mac users and Windows users. These approaches make it easy to upload files to HPC from your local computer, download files from HPC to your local computer, and edit files that are on HPC using your local computer. In particular, we will be able to use RStudio on your local computer to edit Quarto files located on HPC. We will upload Quarto files that you have created on your local computer to HPC.
Finally, we will assume that you are familiar with basic Unix commands as described in this Software Carpentry Unix shell introduction. (You may also want to keep this Software Carpentry Unix shell cheatsheet handy).
Setting up your watershedMacatawa project on HPC (do this the first time you set up each project on HPC)
We will walk through the process of creating the directory structure for an R project on HPC. You will need to do something similar every time you want to work on a new project.
We will also copy some code and data from an existing folder on HPC into your project so that you can run your first job on HPC. Some of this is specifically for your first time using running R code on HPC in the watershed course. If you are working on a different project in the future you will need to copy the code and data you need into the corresponding directories in your project before you run your code. You may be copying these (or linking them) from another directory on HPC or you may be uploading them from your local machine.
Project directory structure on HPC
Log into HPC. We will create the watershedMacatawa project directory structure on HPC. (You should already have a watershedMacatawa project with this directory structure on your local computer).
We will assume that this is the first R project you have created on HPC. We will start by creating a Projects directory in your home directory. This will host all of your R projects. Start be making sure you are in your home directory by entering the following command:
cdCreate the Projects directory and enter it. (If you already have a Projects directory in your home directory do not include the mkdir Projects command).
mkdir Projects
cd ProjectsCreate your watershedMacatawa directory and enter it
mkdir watershedMacatawa
cd watershedMacatawaCreate your code, raw_data, clean_data, and outputs and check your work.
mkdir code raw_data clean_data outputs
ls -FIf you misspelled something, remove it (e.g., rm -r ootputs) and create it again. We will also create a slurm subdirectory within the code directory that will host slurm scipts that we will use to submit jobs to HPC. Move into the code directory and create the slurm directory.
cd code
mkdir slurmLet’s view the entire directory structure that we have created so far. Is everything where it is supposed to be?
ls -RF ~/ProjectsNow move back into the watershedMacatawa directory.
cd ~/Projects/watershedMacatawaFinally, check the directory structure in the left pane of the MobaXterm window (Windows) or in your mounted HPC home directory using Finder (Mac).
Copying code and linking data into your HPC project (for your first time setting up your watershedMacatawa project on HPC)
The first time we submit a job on HPC in the watershed class, we want everyone to have the same code and to be working with the same data files. Rather than uploading the code and data to HPC from your local computer, you will copy existing code from a directory that is already on HPC. You will also create a link to the raw data files, which are already stored in another directory on HPC. This will reduce the amount of storage space we are using on the cluster, since we will all be using the same copy of the data instead of each of us making our own copy. However, by linking to the data it will appear to R as if the files are actually in your raw_data directory instead of in some other directory on the cluster.
First you will copy a project file and yaml file into your watershedMacatawa directory.
cd ~/Projects/watershedMacatawa
cp /home/byurk/water_share/_quarto.yml .
cp /home/byurk/water_share/watershedMacatawa.Rproj .Next you will copy a Quarto file into your code directory. Start by moving into your directory, then copy the file.
cd ~/Projects/watershedMacatawa/code
cp /home/byurk/water_share/code/Mac_Data_Explore_Read_Histo.qmd .Next, move into your slurm directory and copy the slurm script you will need to run your code on HPC.
cd slurm
cp /home/byurk/water_share/code/slurm/mac_explore_read_histo_qmd.slurm .Check to make sure you can open these files from your mounted home directory (Mac) or from MobaXterm (Windows) on your local computer using RStudio. Note that if you edit the files and save the changes in RStudio using your local computer, the changes will be saved on HPC! Make a small change to the Quarto file (not to the R code, though) and save the file to make sure it works. If you do the following on HPC, the output should reflect the changes you made using your local computer. Check to make sure this is the case.
cd ~/Projects/watershedMacatawa/code
cat Mac_Data_Explore_Read_Histo.qmdNext we will create links to the raw data files we will need for this project. The links we are using are referred to as symbolic links. Instead of copying the data we are just creating a file that tells the computer where to look for it. Start by moving into your raw_data directory then creating a link to the .shared file.
cd ~/Projects/watershedMacatawa/raw_data
ln -s /home/byurk/water_share/raw_data/mac2024_RUNS1-2-3-5-6-7-8b-9-10-11-12-14-15_2024-01-18.shared .Next we create a link to the .taxonomy file
ln -s /home/byurk/water_share/raw_data/mac2024_RUNS1-2-3-5-6-7-8b-9-10-11-12-14-15_2024-01-18.taxonomy .and finally the metadata file
ln -s /home/byurk/water_share/raw_data/watershedMetadataCourseSpring2024.csv .Let’s check our work. Links should be indicated by an @ symbol.
ls -FWe can also see the files that are links are pointing to.
ls -lCopying code and data from your local computer
In many cases you will have code and data on your local computer that you want to use for computions on HPC. In this case, instead of copying (or linking) data and code from another directory on the cluster you will simply need to copy the files you need from your local project directories to the corresponding directories on HPC. You can use MobaXterm (Windows) or Finder (Mac) to do this.
Running your code on HPC (do this every time)
Once you have the Quarto file that you want to run in your code directory on HPC and your raw data in the raw_data directory you are ready to run your code. (In some cases your code may use data in the clean_data directory, which includes results from previous computations. Of course if this is the case you will need to make sure these files are in the clean_data directory on HPC.)
HPC uses work load management software called Slurm to determine how jobs that are submitted by users will be run on the cluster. In particular, Slurm determines when a job will be run and which resources will be available for the job.
HPC also uses a container system called Singularity for computations. Singularity containers contain programs that can be used to perform certain tasks. R is not installed on HPC. Instead, there are several containers on HPC that include R. Running a container is like running a separate computer that has its own software installed but that shares some of the resources of the computer system that is running the container. In particular, your home directory is available when working in a container. One advantage of using containers is that your project becomes more portable. If you share both the code and the container that you used, then someone can run your code using exactly the same software versions as you did. This also allows you to run code for your new projects using new versions of R while running the code for your old projects using the versions of R that you originally used.
To run a job on HPC we need to submit it to Slurm. We do this using a script that has a .slurm extension. The following is a simple example of a slurm script. If you followed the instructions in the section Copying code and linking data into your HPC project (for your first time setting up your watershedMacatawa project on HPC) then this is the slurm script that you copied into your slurm directory. At the top of the script is some information telling Slurm how we would like to set up the job (commands starting with #SBATCH). Other lines starting with # are comments that will not be run.
We will go through the details of the Slurm script in class, but at this point we will point out that the line QMDfile=Mac_Data_Explore_Read_Histo.qmd is really the only thing you need to change to execute a different Quarto file in your code directory. It is also a good idea to change the job-name to something that better describes your job.
If we run this Slurm file, Slurm will open a container called r.sif that includes R, and it will run a command that will render the Mac_Data_Explore_Read_Histo.qmd file into .html output (similar to rendering a document in RStudio). This is all done in a single line in the Slurm script (the line starting with the command singularity).
To run the Slurm job we first move into the slurm directory, then we use the sbatch command.
cd ~/Projects/watershedMacatawa/code/slurm
sbatch mac_explore_read_histo_qmd.slurmAfter you submit the job to Slurm it will print out a message like Submitted batch job 17101 (the number for your job will be different).
You can check on the status of your Slurm jobs by entering
squeueIf your job is still running you should see it listed in the resulting table. You can identify your job using the JOBID or the NAME (if you gave it a unique name in the Slurm script). If your job is done running already it will not show up in the table.
Any output from your code along with output from your slurm script will be written to a file called slurm-17101.out (with your job ID instead of 17101). This file will be updated whenever there is new output, so you can view it’s content to check on the status of your job. For job 17101 this can be achieved by entering the following command while you are in the slurm directory:
cat slurm-17101.outWhen your code has finished running, the rendered .html file should appear in your code directory. To check this we can list the files in the directory and look for the .html file. To do this from within the slurm directory we can use the following commands:
cd ..
ls -FDowloading output to your local computer
Once the computation is done you can download the resulting output to your local computer. For example, running the Mac_Data_Explore_Read_Histo.qmd file will produce a rendered .html file called Mac_Data_Explore_Read_Histo.html, three clean data files: macatawa.rds, macatawa_tax.rds, and macatawa_meta.rds, and a plot file mac2024_seq_depth-fwdb.pdf. You should download these files to watershedMacatawa project directory on your local computer, placing the .html file in your local code directory, the clean data files in your clean_data directory, and the plot file in your outputs directory. You should also leave copies of the files on the cluster. This way you can do further computations using either your local computer or the cluster while using the same clean date files for the computation. View the .html file using a web browser (like Chrome) on your local computer.
You should also download the Quarto file and save it in your local code directory so that you can have a local copy of it.
Troubleshooting
If your code fails to run or if it does not produce the expected output (e.g., there is an error message or missing output in your rendered report), you may need to edit the Quarto file and before submitting another Slurm job. You should do this by opening the file in RStudio on your local computer (as you did earlier), editing it, and saving it so that the changes are reflected on the cluster. Then just use the same approach you used earlier to submit a new Slurm job and monitor it.
Variations in workflow
We have already discussed the fact that in some cases you will be uploading data to the cluster instead of copying it from other locations on the cluster.
In some cases you will write code (a Quarto) on your local computer while you are not connected to HPC, then you will copy that code to HPC so that you can run it.
Since you may have different versions of the same file in two different places with this approach is very important to keep track of which version you have locally and which version you have on the cluster. It is also important to know which version you are editing on your local machine (is it a version that is on HPC or on your local file system?). In particular, you should keep track of your most recent version and make an effort to have a copy of that version in both places.