Harvard Cluster Batch System

User information

The batch system installed in the Harvard Computer cluster is the Portable Batch System, PBS Pro 5.1.2 . It was developed by the Lawrence Livermore Labs, NASA and Veridian Corp with the intent of working on very large computer systems. It is capable of managing several computers simultaneously without explicity intervention of the user. PBS is composed of three main components: a job server (pbs_server), a job executor (pbs_mom) and a job scheduler (pbs_sched).

Due to its modularity, PBS is highly configurable, with some work it can be made to do almost anything! Most of the "job selection" work is done by the scheduler. This can be written in many laguages including C. So, we can write our own job scheduler! The system currently implemented in the Harvard cluster uses a semi-FIFO scheduler which is the package default. Besides the FIFO job scheduling it also allows for some load balancing between nodes. Other schedulers are also compatible with PBS. The most interesting one is MAUI which uses detailed load information on each node to maximize the usage of the cluster. We might want to upgrade to this some day....

System summary

The old system information is still available here .

The computers in the Harvard cluster that are included in the batch system are: harv11, harv3, harv4, harv5, harv7, harv8, harv9.

There are 2 queues defined according to the CPU time requested for the job: short and long.
A default routing queue will route jobs to the correct executing queue based on the CPU time information.
Jobs submitted without a specific CPU time request are sent to the long queue.
The long queue is further divided into two executing queues: harv11 and slow. The slow queue runs jobs in the slower machines: harv3 to harv9.
Jobs sent to the long queue will run by default on the harv11 queue. If there are more than 6 jobs in the harv11 queue, the jobs will run on the slow queue. If both queues are full jobs will wait on the long queue for the next available slot.
The slow queue submits the job to the fastest machine available (starting with harv3). Memory limits are also taken in consideration as long as the user provides the memory usage of the job.
Alternatively jobs can be sent directly to an executing queue: short, harv11 or slow.
The maximum number of jobs running at anytime in the cluster is 12.
Harv11 can run up to 5 jobs simultaneously. The slower machines will only run 1 at a time.
The choice of the executing node is partially based on a load balancing algorithm. However, this feature is mostly turn off because it can produce unwanted results (like sending several jobs to Harv3).
The executing node can be selected by the user at the submission time, e.g. the user can select if the job runs on harv3 or harv4. To do this submit the job with the "-l" switch, e.g. -l nodes=1:harv3.
There is a fair_share algorithm that should decide which jobs run first. Jobs sent by a low-volume user will run first than the ones sent by someone that attempts to take over the machine.
Jobs on the "short" queue have priority. Jobs on the long queue will be preempted so that short jobs can run.

Queue layout

route ----> short (harv11 only)
      |
      |
      +---> long (routing queue) ------> harv11 (long queue on harv11)
                                                   |
                                                   |
                                                  +-----> slow (harv3,4,5,7,8,9)

Available Queues	short	long	harv11	slow
Type	Executing	Routing	Executing	Executing
Minimum CPU time per job (hh:mm:ss)	00:00	05:01	05:01	00:00
Maximum CPU time per job (hh:mm:ss)	05:00	2006	2006	2006
Nice value	4	19	19	19
Maximum number of jobs in the queue	-	-	6	6
Maximum number of jobs running	5	-	4	6
Maximum number of jobs running per user	2	-	2	6

Documentation

There are three main sources of information:

The PBS web site has some documentation although little of pratical use.
The PBS administrator guide - mostly administration stuff.
The PBS reference guide - lots of detailed information.
The man pages of the PBS commands - These are your most important resource. A good starting point is man pbs , with general information, and man qsub , with infomation on how to submit a job. The man pages are installed on harv11 only.

Quick start

For now, to submit a job you will still have to login at harv11. (The job server and job scheduler are running on harv11. The other machines are just running a job executer (pbs_mom) and using the same job server and scheduler from harv11). In the future, it might be possible to submit the job from your own workstation. Currently, I am having problems which I suspect are due to security restrictions.

Before you start you need to added the following 2 lines to your .cshrc file.

# --- Include PBS batch system executables in the path
set path = ( $path /usr/pbs/bin /usr/pbs/sbin )

Once you save the file don't forget to do "source .cshrc". Now, you should have access to the PBS binaries.

Below is a sample script file to run cdf code within this batch system:

#!/bin/tcsh
# Set CPU time limit
#PBS -l cput=10:00
# Work from the directory where the job was started. Important because of
# the way LD_LIBRARY_PATH is setup.
cd $PBS_O_WORKDIR
# Need to set CDF environmental variables
source ~cdfsoft/cdf2.cshrc
setenv USESHLIBS 1
setup cdfsoft2 3.18.0int3
SZTree_build ./SZObjects/test/runSZTM_119128.tcl

Important issues:

Make sure that you define the shell to be used otherwise sh wil be used.
PBS directives are given by lines starting with #PBS. The directives are the same as the ones at the command line
PBS always starts the job from you home directory. So, jump to $PBS_O_WORKDIR, the directory from where the job was submitted.
Need to include setup statements in the script.
To submit the job do: qsub script_name
The log files go the working directory: standard output is <script_name>.o## and the standard error is <script_name>.e##. Where ## stands for the job id number. There is an optional PBS directive that will merge both log files into one.
While the job is running, the log files are kept in the directory: /var/spool/PBS/spool. The log files are identified by their job id, e.g. 286.harv11..OU and 286.harv11..ER. You can look at them using the Unix "tail" command.

In the table below, we assume that the job script is called: test.pbs

PBS command	Command Description
qsub test.pbs	Submits job to default routing queue. Job will be forward to correct queue according to CPU time requested.
qsub test.pbs -q long	Job send directly to queue "long". It will be accepted with default CPU time as long as queue CPU maximum time.
qsub test.pbs -l cput=10:00	Request of CPU time in the command line.
qsub test.pbs -l nodes=1:harv3	Request explicitly to be executed in the harv3 node.
qdel job.id	Deletes a job
qstat	Provides information about the queue status including jobs running
qmove	Moves a queued job to a different queue
pbsnodes -a	Provides information about the status of all the executing nodes

Another sample script file:

#!/bin/tcsh -f
# Set CPU time limit (advisable)
#PBS -l cput=40:00
#
# Join the standard output and error together as standard output (optional)
#PBS -j oe
#
# Mail is sent when job is aborted or terminates (optional)
#PBS -m ae
#
# Name for this job (optional)
#PBS -N StnMakerZbb
#
# Queue where to run the job (optional) - by default it runs according to
# the cput requested or in the short queue
#PBS -q long
#
# Request a specific node. In this case harv3.
#PBS -l nodes=1:harv3
#
# Work from the directory where the job was started. Important because of
# the way LD_LIBRARY_PATH is setup.
cd $PBS_O_WORKDIR
#
# Need to set CDF environmental variables
#
source ~cdfsoft/cdf2.cshrc
setenv USESHLIBS 1
setup cdfsoft2 3.18.0int7
# Get local release bin directory in the path
srt_setup -a
echo $CDFSOFT2_DIR

Helpful scripts

The PBS keeps the log files in a system location (/usr/spool/PBS/spool) until the jobs is finished. When the job ends both the std out and std error log files are sent to the user directory.
The two tcsh scripts below allow to look at the logfiles before the jobs are finished.
peeklog -- Looks at the std out logfile while job is running
peekloge -- Looks at the std err logfile while job is running

Note that using the switch "-j oe" when submitting the job joins the std output and std error together as std output.

Administration Stuff

Server configuration as of August 28.

Nodes are setup as cluster nodes. Harv11 with 5 virtual processors and Harv3 with 1. This forces Harv11 to be used more often. Maximum number of jobs running in the cluster is 6.

PBS command	Command Description
qmgr -c "c n harv11 np=5"	Create a node (harv11) with 5 virtual processors.
qmgr -c "d n harv11"	Delete node harv11

pbsnodes -a	Information about the cluster nodes