CoEPP RC
 

Torque Tricks

This page contains a collection of small tricks that you might find useful when running many similar jobs at the same time.

Unique output files

If you submit the same batch file to the cloud you will find that any output files produced are corrupted since all of the running jobs wrote to the output file(s) at the same time.

How can we ensure that each job writes to its own private output file (or files)?

A running batch job inherits many torque-sourced environment variables. Most are of little use to you, the user. One, however, is very useful as it contains the batch job number which is unique to your batch job. This is the $PBS_JOBID environment variable. If you run a simple batch job which just does:

echo "$PBS_JOBID"

then you will see something like this in the output file:

390505.c3torque.cloud.coepp.org.au

You can use the contents of $PBS_JOBID if you need some unique string to name directories or files. For instance, if you want to create many files, do it in a directory, as this keeps everything in one place and allows easy deletion when required:

mkdir /tmp/$PBS_JOBID
cd /tmp/$PBS_JOBID
    <create many files>
    <store results somewhere>
cd
rm -Rf /tmp/$PBS_JOBID

If the full value of $PBS_JOBID is too long for your purposes you can shorten it by extracting just the job number:

JOBNUM=$(echo $PBS_JOBID | cut -d. -f1)

Here JOBNUM would contain just “390505”.

cloud/advanced_torque.txt · Last modified: 2013/12/13 14:50 by rwilson
 
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki