PBS tutorial
Submitting jobs (qsub)
This document describes how jobs are submitted on University of
Innsbruck HPC systems using the Portable Batch System (PBS
Professional). As a user you might want to be
familiar with the following commands: qsub,
qstat and qdel,
which are briefly described in the following sections.
For more information see the respective man pages of these
commands, refer to the PBS User Guide or consult the
PBS
Professional homepage.
The command qsub allows to submit jobs to the batch-system. qsub uses the following syntax:
qsub [options] job_script [ argument ... ]
where job_script represents the path to a script containing the commands to be run on the system using a shell. Note that we are currently only supporting Bash (/bin/bash). Please contact system administration, if you should depend on using another shell.
There are two ways to submit a job.
Method 1:
You may add the options directly to the qsub command, like:
qsub -o pbs_out.dat -e pbs_err.dat job_script [ argument ... ]
Method 2: (recommended)
The qsub options can also be written to the job description file job_script, which is then passed on to qsub on the command line:
qsub job_script [ argument ... ]
The contents of job_script may look like the following:
|
#!/bin/bash #PBS -o pbs_out.dat #PBS -e pbs_err.dat ./your_commands |
Please note that any PBS directives after the first shell command are
ignored by the system. So keep PBS directives at the beginning of your
job file.
Description of the most commonly used options to qsub:
| Name, Input/Output | |
|---|---|
| -N name | Optional, recommended. name of the job. Default: File name of script. The job name is also reflected in the default file names for standard output and standard error (see below). |
| -o opath | Optional.
Standard output will go to file opath.
If the file exists, it will be overwritten. If you plan to run multiple instances of a job time, using this option is not recommended (possible loss out output data). Default: name.ojob_id. Here, name is the job's name (see above), and the unique job_id is automatically created by the system for each job. |
| -e epath | Standard error goes to file epath. Default: name.ejob_id. name and job_id are the same as above. |
| -j oe | Join standard error to standard output. |
| Notification | |
| -M email address | Notifications will be sent to this email address. Please use only if you submit very few jobs. |
| -m b|e|a|n | notifications on the following events: begin, end, abort, no mail (default) Do not forget to specify an email address (with -M) if you want to get these notifications. |
| Resources | |
| -l walltime=[hours:minutes:]seconds | requested real time; the default (maximum) depends on the system and, if applicable, the chosen queue |
| -l select=N:ncpus=NCPU | request N times NCPU slots (=CPU cores) for the job (default for NCPU: 1) |
| -l select=N:mem=size[m|g] | request N times size bytes / megabytes / gigabytes of memory for the job. This should be slightly less than an integer multiple of 64GB. |
| -l select=N:ncpus=NCPU:mem=size[m|g] | request N times NCPU CPUs
and size bytes / megabytes / gigabytes of memory
for the job. Please Note: Multiple -l select statements are ignored by PBS, only the first statement will be used. If you specify memory and CPU requirements in the same statement, PBS will create a CPU-set that is big enough to satisfy both of your requirements. |
| -W depend=afterok:job-id | start job only if the job with job id job-id has finished successfully |
Note that it is often advisable to have the actual setup of the
machine's hardware in mind, when choosing the appropriate number of
processes/threads or the right amount of memory for your program and
specifying your job resources accordingly. This will result in a more
efficient use of the system and as a consequence in better system
response times for your jobs.
There are differences to consider between the various supported (parallel) programming models. The following examples illustrate the different procedures:
Sequential jobs
qsub job_script
where the contents of job_script may look like this:
(if you just copy&paste this example please be
aware of line breaks and special characters)
|
#!/bin/bash # Redirect output stream to this file. #PBS -o pbs_out.dat # Redirect error stream to this file. #PBS -e pbs_err.dat # Send status information to this email address. #PBS -M Karl.Mustermann@xxx.com # Send an e-mail when the job is done. #PBS -m e # Change to current working directory (directory where qsub was executed) # within PBS job (workaround for SGE option "-cwd") cd $PBS_O_WORKDIR # For example an additional script file to be executed in the current # working directory. In such a case assure that script.sh has # execution rights (chmod +x script.sh). ./script.sh |
Parallel MPI jobs
qsub job_script
where the contents of job_script may look like this:
(if you just copy&paste this example please be
aware of line breaks and special characters)
|
#!/bin/bash # Redirect output stream to this file. #PBS -o pbs_out.dat # Join the error stream to the output stream. #PBS -j oe # Send status information to this email address. #PBS -M Karl.Mustermann@xxx.com # Send me an e-mail when the job has finished. #PBS -m e # Reserve resources for 16 parallel processes # (Note: the default ncpus=1 could also be omitted) #PBS -l select=16:ncpus=1 # Change to current working directory (directory where qsub was executed) # within PBS job (workaround for SGE option "-cwd") cd $PBS_O_WORKDIR NSLOTS=`cat $PBS_NODEFILE | wc -l` mpirun -np $NSLOTS ./your_mpi_executable [extra params] |
Parallel OpenMP jobs
qsub job_script
where the contents of job_script may look like this:
(if you just copy&paste this example please be
aware of line breaks and special characters)
|
#!/bin/bash # Redirect output stream to this file. #PBS -o pbs_out.dat # Join the error stream to the output stream. #PBS -j oe # Send status information to this email address. #PBS -M Karl.Mustermann@xxx.com # Send me an e-mail when the job has finished. #PBS -m e # Reserve resources for 16 threads #PBS -l select=1:ncpus=16 # Change to current working directory (directory where qsub was executed) # within PBS job (workaround for SGE option "-cwd") cd $PBS_O_WORKDIR # OMP_NUM_THREADS is automatically set to 16 with the above select statement ./your_openmp_executable |
Submitting interactive jobs
The submission of interactive jobs is useful in situations where a job requires some sort of direct intervention. This is usually the case for X-Windows applications or in situations in which further processing depends on your interpretation of immediate results. A typical example for both of these cases is a graphical debugging session.
Note: Interactive sessions are particularly helpful for getting acquainted with the system or when building and testing new programs.
Starting interactive sessions within PBS only requires specifying the option -I to the qsub command, so in the simplest case
qsub -I
This will bring up a further Bash session on the system. To ensure X forwarding to the X-server display indicated by your actual DISPLAY environment variable, add the option -v DISPLAY to your qsub command. All qsub options can be specified as usual on the command line, or they can be provided within an optionfile with
qsub -I optionfile
A valid optionfile might contain the following lines (only #PBS directives are parsed within this file):
|
# Name your job #PBS -N myname # Export some of your current environment variables, #PBS -v var1[=val1],var2[=val2],... # e.g. your current DISPLAY variable for graphical applications #PBS -v DISPLAY # Reserve resources for 8 MPI processes #PBS -l select=8 # Specify one hour runtime #PBS -l walltime=1:0:0 |
When your session was started prepare your environment as needed, e.g. by loading all
necessary modules and then start your programs as usual. For your parallel applications refer to the script.sh files for parallel MPI, respectively parallel OpenMP batch jobs above.
Note: Make sure to end your interactive sessions as soon as they are no longer needed!
Observing jobs (qstat)
To get information about running or waiting jobs use
qstat [options]
Useful qstat options :
| -u user | prints all jobs of a given user |
| -f job-id | prints full information of the job with the given job-id. Note that information such as resources_used.ncpus are on resources allocated to the job by PBSPro, not actual resource consumption. Use top -u $USER to observe your program's behaviour in real time. |
| -aw | prints more information in a wider display |
| -help | prints all possible qstat options |
In case of pending jobs, you might get a hint on when your job will be started, by executing
qstat -awT
In this case the runtime of waiting jobs is replaced with their estimated start time. Don't be alarmed, though, this only provides a worst case estimation.
Deleting a job (qdel)
To delete a job with the job identifier job-id, execute
qdel job-id
Special notes on PBS
PBS has - in comparison to the SGE batch scheduler, which is usually deployed on our HPC systems - some specialities to consider:
Input/output staging
The two streams stdout and stderr are directed to a special directory (naming: pbs.$PBS_JOBID.x8z) subordinate to your home directory and copied to the job's working directory only at the end of the job.
Note: If your application writes large amounts of data to stdout and/or stderr, please redirect this output directly to a location within your $SCRATCH space. Otherwise your jobs might be terminated for exceeding your $HOME quota. You can redirect both the output to stdout and to stderr of your application my_executable to a file job_output.dat within your current working directory with:
./my_executable >job_output.dat 2>&1
WARNING: Do not specify the staged output of PBS (options -o or -e) with the same name as your redirected job output destination. Otherwise your job's output might be overwritten as soon as the PBS output is "staged out", i.e. copied to the final output destination - also named job_output.dat - at the end of the job.
PBS umask setting
Per default PBS jobs run under the uncommon umask setting 0077 (i.e. all read/write/execute permissions for group members or others are removed). If you need another umask setting for your job runs (e.g., if you want to share information with group members) please specify your desired setting explicitly within your job scripts. For example
umask 0022
sets the usual Linux bit mask, removing write permissions for group and others.
X forwarding in interactive sessions
In contrast to SGE your actual DISPLAY variable is not automatically set in interactive PBS sessions. In order to ensure X forwarding, you need to explicitly export your current DISPLAY to the job's environment by adding the option -v DISPLAY to your qsub command.