CoEPP RC
 

Example: 3/2 directories

This is a mostly useless example of a complicated job script involving staging a few directories. Create a directory complex under your home directory ($HOME/complex). In here we will create three directories, one, two and program. Under complex/program we will create the executable script product.sh:

#!/bin/bash
# -- product.sh --
 
one=$1
two=$2
output=$3
 
mkdir ${output}
 
for fname1 in $(ls $one) ; do
    for fname2 in $(ls $two) ; do
    cat ${one}/${fname1} ${two}/${fname2} > ${output}/${fname1}-${fname2}
    done
done

This script could be described as calculating the cartesian product of two directories, producing a “set” [directory] containing each pair [concatenation] of an element from the first set [directory] and an element from the second set [directory]. Next, fill one and two with a few text files, then create the following job script:

#!/bin/bash
# -- job.sh --
 
# Set the name of this batch job
#PBS -N complex
 
# Specify the batch queue to use
#PBS -q batch
 
# Join standard and error ouputs into one file
#PBS -j oe
 
# Set the batch job parameters.
#PBS -l nodes=1
#PBS -l mem=512MB
#PBS -l vmem=512MB
#PBS -l walltime=0:05:00
 
#PBS -W stagein=$TMPDIR/one@cxin01.cloud.coepp.org.au:$HOME/complex/one
#PBS -W stagein=$TMPDIR/two@cxin01.cloud.coepp.org.au:$HOME/complex/two
#PBS -W stagein=$TMPDIR/program@cxin01.cloud.coepp.org.au:$HOME/complex/program
 
#PBS -W stageout=$TMPDIR/one-two@cxin01.cloud.coepp.org.au:$HOME/complex/one-two
#PBS -W stageout=$TMPDIR/two-one@cxin01.cloud.coepp.org.au:$HOME/complex/two-one
 
cd $TMPDIR
./program/product.sh one two one-two
./program/product.sh two one two-one

In a nutshell, we stage in our source directories, one and two, as well as our program (which is really just one file, but we'll leave it as a directory to simulate a more realistic case of a complex program). product.sh produces as output directories $TMPDIR/one-two, and then $TMPDIR/two-one. These two output directories are staged back out to $HOME/complex on the interactive node.

cloud/example-3-2.txt · Last modified: 2013/02/28 18:17 by neilds
 
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki