Job Scheduling System (PBS Pro) at HPCF2 cluster

All jobs must be scheduled via the batch job system (PBS Pro). Jobs are submitted to PBS specifying requested resources, including the queue, number of CPUs, the amount of memory, and the length of time. PBS will then run job(s) on available compute nodes when the resources are available, subject to constraints on maximum resource usage.

General Job Queues

Special Queues (On Request)

Sometimes, you may have some jobs that would need more computing resources than those available in the general job queues. In this case, please send us the details of your job execution plan and resources needed. Depending on the current cluster resource usages and trends, we might setup a customized job queue with specific computing resources to enable your job execution on a short term basis.

Job Scripting

PBS Job Directives

Below are some commonly used PBS options in a job command file, which can also be used on the command line with qsub.

Some useful batch job directives to try are the following:

PBS Job Output/Error Files

You can specify the full file names of the PBS job output/error files:

Or you can specify them with directories:

Or if these two directives -e and -o are not specified, the working directory and default filenames as below would be used:

PBS Pro uses the directory that the job was submitted from to define the working directory for a job – no matter the location of the job submission script. Please note that by default, the PBS scheduler would store data relative to your home-directory. Yet it is recommended to specify a full path to the filename.

Tips:

The PBS job variables (e.g. $PBS_JOBID and $PBS_JOBNAME commonly used) would NOT be resolved in the #PBS job directives at the Omics cluster with new PBS scheduling system (that is different from the statgenpro cluster).

Interactive PBS Jobs

Use of PBS is not limited to batch jobs only. It also allows users to use the compute nodes interactively, when needed. For example, users can work with the developer environments provided by R on compute nodes, and run their jobs (until the walltime is expired).

Instead of preparing a submission script, users pass the job requirements directly to the qsub command. For example, the following PBS script:

 #PBS -l nodes=1:ppn=4 
 #PBS -l mem=2gb 
 #PBS -l walltime=15:00:00 
 #PBS -q small 

would correspond to the qsub command with parameters as below:

 qsub -I -q small -l nodes=1:ppn=4,walltime=15:00:00,mem=2gb  

Hence, the PBS scheduler will allocate 4 cores to the user as soon as nodes with given specifications become available, then automatically log the user into one of the compute nodes. Any Interactive PBS jobs (i.e. qsub –I ) will be logged out after a 30-minutes idle time to free up allocated but unused resources.

Job Management

Submit a batch job

Batch jobs allow users to submit a bunch of jobs that can be queued up and then executed across the cluster by the job scheduler, when the requested resources become available. Users submit jobs as scripts, which include instructions on how to run the job. The output of the job is written to a file for review later on. You can write a batch job that does anything that can be typed on the command-line.

Here is an example of batch script (with filename simple.sh) :

 #!/bin/bash
 #PBS -l nodes=1:ppn=1
 #PBS -l mem=2g
 #PBS -l walltime=00:01:00
 #PBS -m ae
 #PBS -N omics-simple
 #PBS -q small

 module load bamtools/2.5.1
 bamtools -v

Then you can submit this job script by:

 $ qsub simple.sh

See “PBS Job Directives” for more about the PBS job parameters that could be specified in the script file.

Check job status

After your job is submitted, you can use the qstat command to view the status of your job and the job queues. If there are resources available on the compute nodes, your job should turn into the R (running) state; If the compute nodes are busy, your job may be shown in the Q (queued) state until your requested resources are then available to run it. Jobs shown in C state means it has been completed.

Show running jobs

$ qstat -rn1

Example:

 [itsupport@omics ~]$qstat -rn1
 
 omics: 
                                                              Req'd  Req'd   Elap
 Job ID          Username  Queue    Jobname    SessID NDS TSK Memory Time  S Time
 --------------- --------  -------- ---------- ------ --- --- ------ ----- - -----
 1649.omics      itsupport large    BWA_19NT   121502  1  12   90gb  48:00 R hpch01/0*12

Show queuing/held jobs

$ qstat -i

Example:

 [itsupport@omics ~]$qstat -i
 
 omics: 
                                                              Req'd  Req'd   Elap
 Job ID          Username  Queue    Jobname    SessID NDS TSK Memory Time  S Time
 --------------- --------  -------- ---------- ------ --- --- ------ ----- - -----
 1651.omics      itsupport large    BWA_19T    ---     1  12   90gb  48:00 Q

Show all jobs (including completed jobs)

$ qstat -xan1

Example:

 [itsupport@omics Exome]$qstat -xan1
 
 omics: 
                                                            Req'd  Req'd   Elap
 Job ID          Username  Queue    Jobname    SessID NDS TSK Memory Time  S Time
 --------------- --------  -------- ---------- ------ --- --- ------ ----- - -----
 1648.omics      itsupport  large  BWA_23T     127476   1  12   45gb 48:00 F 00:03 hpch08/0*12
 1649.omics      itsupport  large  BWA_23NT    121502   1  12   90gb 48:00 R 01:12 hpch08/0*12
 1650.omics      itsupport  large  BWA_43T     5940     1  12   45gb 48:00 F 00:01 hpch08/0*12
 1651.omics      itsupport  large  BWA_43NT     --      1  12   90gb 48:00 Q   --   --

Show details of a job

$ qstat -xf JobID

Example:

 [itsupport@omics]$qstat -f 1649
 Job Id: 1649.omics
   Job_Name = BWA_34NT
   Job_Owner = itsupportn@omics
   resources_used.cpupercent = 916
   resources_used.cput = 05:56:56
   resources_used.mem = 59554972kb
   resources_used.ncpus = 12
   resources_used.vmem = 92494740kb
   resources_used.walltime = 01:28:38
   job_state = R
   ...

Check resource usage of a completed job

Function: Show job information and resource usage percentages of a particular job

Usage: myjob

Examples:

   $ myjob 440356
   
   Job information and usage summary of your HPCF job 440356 :
   +-------------------+----------+-------+-------------+------+---------------------+-----------+------+-------+
   | jobid             | username | queue | jobname     | E S  | End Time            | walltime% | mem% | cpu%  |
   +-------------------+----------+-------+-------------+------+---------------------+-----------+------+-------+
   | 440356.statgenpro | kelvin   | large | Large       |    0 | 2016-08-30 04:31:30 |     25.36 | 8.28 | 14.51 |
   +-------------------+----------+-------+-------------+------+---------------------+-----------+------+-------+
   +---------------------+---------------------+----------+----------+------+------+-------+----------+--------+
   | Submit Time         | Start Time          | wtime@   | wtime#   | mem@ | mem# | vmem@ | CPUTime@ | nproc# |
   +---------------------+---------------------+----------+----------+------+------+-------+----------+--------+
   | 2016-08-29 22:26:16 | 2016-08-29 22:26:18 | 06:05:13 | 24:00:00 | 3.31 | 40gb |  4.75 | 10:35:53 |     12 |
   +---------------------+---------------------+----------+----------+------+------+-------+----------+--------+
   E S = Exit Status ; % = usage percentage; # = requested ; @ = used ; mem@/vmem@ in GB ; nproc = number of processors

Only the owner of the job may check the usage

   $ myjob 123456
   ERROR: You (kelvin) not the owner of the job 123456.

Check resource usage summary of completed job(s)

Function: Display job summary and resource usage percentages of your recent jobs\\

Usage: myjobs [-v] [-j] []

Description: Display job summary and resource usage percentages of your recent jobs\\

Optional Parameters:

-v verbose mode with resource usage data of walltime/memory/cpu requested and used

-j Last jobs by JobID numbers (instead of the default by End Time). Note that the jobs in the list by JobID may be different from that by End Time.

specifying the number of jobs (default:20) to display

Examples

$ myjobs 5
 
Your last 5 HPCF completed jobs (by End Time):
+-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+
| jobid             | username | queue | jobname                | E S  | End Time            | cpu%   | mem%  | wtime% |
+-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+
| 458952.statgenpro | kelvin   | large | GATK_CK02              |    0 | 2016-10-28 16:45:05 |  69.82 | 10.80 |   0.42 |
| 458860.statgenpro | kelvin   | large | GATK_NG07              |    0 | 2016-10-28 11:19:56 |  53.44 | 10.70 |   0.40 |
| 458853.statgenpro | kelvin   | large | GATK_CK03              |    0 | 2016-10-28 11:13:53 |  69.20 | 10.80 |   0.42 |
| 458862.statgenpro | kelvin   | large | GATK_YJ01              |    0 | 2016-10-28 11:09:05 |  49.90 | 10.30 |   0.24 |
| 458858.statgenpro | kelvin   | large | GATK_KC04              |    0 | 2016-10-28 10:57:39 |  55.96 | 10.30 |   0.11 |
+-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+
E S = Exit Status ; % = usage percentage; wtime = walltime

Verbose mode

$ myjobs -v 5

Your last 5 HPCF completed jobs (by End Time): 
+-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+----------+--------+-------+-------+----------+-----------+
| jobid             | username | queue | jobname                | E S  | End Time            | cpu%   | mem%  | wtime% | CPUTime@ | CPUno# | mem@  | mem#  | wtime@   | wtime#    |
 +-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+----------+--------+-------+-------+----------+-----------+
| 458952.statgenpro | kelvin   | large | GATK_CK02              |    0 | 2016-10-28 16:45:05 |  69.82 | 10.80 |   0.42 | 00:41:55 |      2 |  1.08 | 10.00 | 00:30:01 | 120:00:00 |
| 458860.statgenpro | kelvin   | large | GATK_NG07              |    0 | 2016-10-28 11:19:56 |  53.44 | 10.70 |   0.40 | 00:31:03 |      2 |  1.07 | 10.00 | 00:29:03 | 120:00:00 |
| 458853.statgenpro | kelvin   | large | GATK_KC03              |    0 | 2016-10-28 11:13:53 |  69.20 | 10.80 |   0.42 | 00:42:17 |      2 |  1.08 | 10.00 | 00:30:33 | 120:00:00 |
| 458862.statgenpro | kelvin   | large | GATK_YJ01              |    0 | 2016-10-28 11:09:05 |  49.90 | 10.30 |   0.24 | 00:16:57 |      2 |  1.03 | 10.00 | 00:16:59 | 120:00:00 |
| 458858.statgenpro | kelvin   | large | GATK_KC04              |    0 | 2016-10-28 10:57:39 |  55.96 | 10.30 |   0.11 | 00:08:55 |      2 |  1.03 | 10.00 | 00:07:58 | 120:00:00 |
+-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+----------+--------+-------+-------+----------+-----------+
E S = Exit Status ; % = usage percentage ; wtime = walltime ; @ = used ; # = requested ; CPUno = number of processors ; mem@/vmem@/mem# in GB

Display last jobs by JobID numbers

$ myjobs -j 5

Your last 5 HPCF completed jobs (by JobID):
+-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+
| jobid             | username | queue | jobname                | E S  | End Time            | cpu%   | mem%  | wtime% |
+-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+
| 458952.statgenpro | kelvin   | large | GATK_CK02              |    0 | 2016-10-28 16:45:05 |  69.82 | 10.80 |   0.42 |
| 458862.statgenpro | kelvin   | large | GATK_YJ01              |    0 | 2016-10-28 11:09:05 |  49.90 | 10.30 |   0.24 |
| 458860.statgenpro | kelvin   | large | GATK_NG07              |    0 | 2016-10-28 11:19:56 |  53.44 | 10.70 |   0.40 |
| 458858.statgenpro | kelvin   | large | GATK_KC04              |    0 | 2016-10-28 10:57:39 |  55.96 | 10.30 |   0.11 |
| 458857.statgenpro | kelvin   | large | GATK_LR06              |    0 | 2016-10-28 10:54:35 |  55.46 | 10.30 |   0.11 |
+-------------------+----------+-------+------------------------+------+---------------------+--------+-------+--------+
E S = Exit Status ; % = usage percentage; wtime = walltime

Delete a job

$ qdel JobID

Example:

 [itsupport@omics]$qdel 1651