Example: 3/2 directories

This is a mostly useless example of a complicated job script involving staging a few directories. Create a directory complex under your home directory ($HOME/complex). In here we will create three directories, one, two and program. Under complex/program we will create the executable script

# -- --
mkdir ${output}
for fname1 in $(ls $one) ; do
    for fname2 in $(ls $two) ; do
    cat ${one}/${fname1} ${two}/${fname2} > ${output}/${fname1}-${fname2}

This script could be described as calculating the cartesian product of two directories, producing a “set” [directory] containing each pair [concatenation] of an element from the first set [directory] and an element from the second set [directory]. Next, fill one and two with a few text files, then create the following job script:

# -- --
# Set the name of this batch job
#PBS -N complex
# Specify the batch queue to use
#PBS -q batch
# Join standard and error ouputs into one file
#PBS -j oe
# Set the batch job parameters.
#PBS -l nodes=1
#PBS -l mem=512MB
#PBS -l vmem=512MB
#PBS -l walltime=0:05:00
#PBS -W stagein=$TMPDIR/$HOME/complex/one
#PBS -W stagein=$TMPDIR/$HOME/complex/two
#PBS -W stagein=$TMPDIR/$HOME/complex/program
#PBS -W stageout=$TMPDIR/$HOME/complex/one-two
#PBS -W stageout=$TMPDIR/$HOME/complex/two-one
./program/ one two one-two
./program/ two one two-one

In a nutshell, we stage in our source directories, one and two, as well as our program (which is really just one file, but we'll leave it as a directory to simulate a more realistic case of a complex program). produces as output directories $TMPDIR/one-two, and then $TMPDIR/two-one. These two output directories are staged back out to $HOME/complex on the interactive node.

cloud/example-3-2.txt · Last modified: 2013/02/28 18:17 by neilds
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki