CoEPP RC
 

BELLE / BELLE2 User Guide

Resources

  • Local BELLE / BELLE2 users can take profit from different kinds of resources simultaneously:
    1. Local resources at their local institution (Melbourne, Sydney and Adelaide)
    2. Cloud resources
  • Researchers can currently execute interactive and batch submissions in both local and cloud resources. However, the model should evolve in the near future to the situation where local resources are only used for interactive applications, and cloud resources are only used for batch submissions. This new model is still not in place, and it will be properly announce when ready.

Cloud resources

Access

  • Access to the cloud is available through the following machines:
ssh -Y <user_name>@cxin01.cloud.coepp.org.au
ssh -Y <user_name>@cxin02.cloud.coepp.org.au
  • The interactive nodes are used to submit jobs to the cloud and also for interactive use.
  • Each interactive node has 16 cores and 64 GB memory. However, the types of resources may vary since they are being provided has virtual machines.

Filesystems

  • Interactive cloud machines provide the following relevant filesystems:
/imports/home  (~61 TiB)              ---> Non Shared Home
/data (~33 TiB)                       ---> Shared User Space
/scratch (~ 500 GiB)                  ---> Scratch Area
  • Your own home directory /home/<user_name> should be used to develop and prepare executables. Nevertheless, it is not shared with all cloud machines.
  • To run jobs on the cloud, you will need to copy your data to a different directory. A directory '/data/<user_name>' is available to you on both the interactive nodes and all the batch worker nodes. You must place your executable files and any input data required under your '/data/<user_name>' directory before submitting a batch job. Similarly, any output files written by your batch job will be stored under your '/data/<user_name>' directory.

Batch System

  • The interactive nodes are also used to submit jobs to the cloud.
  • Using the Cloud Batch System, you can profit up to 240 processors.
  • A brief introduction on how to execute jobs in the cloud is available here: A Walk in the Cloud. Frequently Asked Questions for the Cloud are available here: Cloud Frequently Asked questions. Please note that the above information is generic and not BELLE specific. You may have to adjust it to your particular case.

Melbourne local resources

General

  • If you are interested in the technical details of Melbourne servers (processors, memory, models, etc…), they are available here: Melbourne Servers and Hardware

Access

  • Users should preferencially login-in through mui.coepp.org.au (an alias to melui1.mel.coepp.org.au). This machine is only accessible from an academic network.
ssh -Y <user_name>@mui.coepp.org.au
  • melui1.mel.coepp.org.au should be used to execute interactive applications. However, if this specific machine is overloaded, users can also directly login to melui[2,3,4].mel.coepp.org.au.
  • melui1.mel.coepp.org.au should also be used to submit batch jobs to the Melbourne batch system.

Filesystems

  • All machines provide the following filesystems:
/imports/home (~ 61 TiB)        ---> Shared Homes
/imports/rcs5_data ( ~ 33 TiB)  ---> Shared User Space
  • If you need to write on any of the 'Shared User Spaces', please contact the Research Computing Team which will then create a directory for you with the correct permissions.

Batch system

  • mui.coepp.org.au (an alias to melui1.mel.coepp.org.au) should also be used to submit batch jobs to the Melbourne batch system. However, if this specific machine is overloaded, users can also directly login to melui[2,3,4].mel.coepp.org.au.
  • In the Melbourne Cluster, users can take profit from up to 40 processors, shared with all Melbourne users under a fairshare mechanism.

Sydney local resources

General

  • If you are interested in the technical details of Sydney servers (processors, memory, models, etc…), they are available here: Sydney Servers and Hardware

Access

  • Users should preferencially login-in through sui.coepp.org.au (an alias to sydui1.syd.coepp.org.au)
ssh -Y <user_name>@sui.coepp.org.au
  • sydui1.syd.coepp.org.au should be used to execute interactive applications. However, if this specific machine is overloaded, users can also directly login to sydui[2,3,4].syd.coepp.org.au.
  • sydui1.syd.coepp.org.au should also be used to submit batch jobs to the Sydney batch system.

Filesystems

  • All machines provide the following filesystems:
/jobfs ( ~800 GiB)             ---> Scratch Area
/sydpp ( ~800 GiB)             ---> Shared Homes
/export/sydpp1 ( ~19 TiB)      ---> Shared User Space
/export/sydpp2 ( ~28 TiB)      ---> Shared User Space
  • If you need to write on any of the 'Shared User Spaces', please contact the Research Computing Team which will then create a directory for you with the correct permissions.

Batch system

  • sui.coepp.org.au (an alias to sydui1.syd.coepp.org.au) should also be used to submit batch jobs to the Sydney batch system. However, if this specific machine is overloaded, users can also directly login to sydui[2,3,4].syd.coepp.org.au.
  • In the Sydney Cluster, users can take profit from up to 40 processors, shared with all Sydney users under a fairshare mechanism.
  • Specific information about Sydney Batch System configuration is available here: Sydney Batch System

Adelaide local resources

General

  • If you are interested in the technical details of Adelaide servers (processors, memory, models, etc…), they are available here: Adelaide Servers and Hardware

Access

  • Users should preferencially login-in through aui.coepp.org.au (an alias to adlui1.adl.coepp.org.au). This machine is only accessible from an academic network.
ssh -Y <user_name>@aui.coepp.org.au
  • adlui1.adl.coepp.org.au should be used to execute interactive applications. However, if this specific machine is overloaded, users can also directly login to adlui[2,3].adl.coepp.org.au.
  • adlui1.adl.coepp.org.au should also be used to submit batch jobs to the Adelaide batch system.

Filesystems

  • All machines provide the following filesystems:
/imports/home (~ 13 TiB)        ---> Shared Homes

Batch system

  • aui.coepp.org.au (an alias to adlui1.adl.coepp.org.au) should also be used to submit batch jobs to the Adelaide batch system. However, if this specific machine is overloaded, users can also directly login to adlui[2,3].adl.coepp.org.au.
  • In the Adelaide Cluster, users can take profit from up to 24 processors, shared with all Adelaide users under a fairshare mechanism.

Software

  • BELLE and BELLE2 software is provided through CVMFS, both in local resources (Sydney or Melbourne) and in the Cloud. CVMFS is a hierarchical caching mechanism which allows CVMFS clients to cache data available in the server. Users can then access it in a fast and reliable way.
  • The CVMFS BELLE and BELLE2 directories are automounted on the fly:
  1. /cvmfs/belle.cern.ch for BELLE2 software
  2. /cvmfs/experimental.cloud.coepp.org.au/belle/ for BELLE software
$ df | grep cvmfs
$

$ ls /cvmfs/belle.cern.ch
belle2  externals  gbasf2  releases  sl5  sl6  SteveDeleteMe  tools

$ ls /cvmfs/experimental.cloud.coepp.org.au/belle/
belle  cern  local

$ df | grep cvmfs
cvmfs2                                     12288000     6429794     5858207  53% /cvmfs/belle.cern.ch
cvmfs2                                     12288000     6429794     5858207  53% /cvmfs/experimental.cloud.coepp.org.au

Database

  • The BELLE software needs to contact a POSTGRESQL DB with the experiments data calibration. A dedicated host (rcbelledb.mel.coepp.org.au) contains all BELLE DBs which can be accessible through the 'belle' user.
export BELLE_POSTGRES_SERVER=rcbelledb.mel.coepp.org.au
export PGUSER=belle

Access to Data

  • The following information was kindly provided by Martin Sevior and James Kahn

Data @ QCIF

  • QCIF has a POSIX filesystem where the BELLE data is stored.
  • Some time ago, the transfer from QCIF to Melbourne was very slow, which was explained by bottlenecks at QCIF. These are apparently fixed in the new high-bandwidth connections there.
  • Therefore, there is now the option to move that data (at QCIF) to a server with a higher network bandwidth. I haven’t asked for that since there wasn’t interested in using it a few months ago. Case the interest arrises, please alert Martin.
  • In addition we could look to set up cluster at QCIF with the Belle Data NFS mounted and available. Case the interest arrises, please alert Martin.

Data @ Adelaide

  • Users with a valid grid certificate, and subscribed to the belle VO, should be able to access it.

BELLE examples

BASF example: interactive execution

  • Login at your preferred resource center (Sydney or Melbourne or Cloud)
$ ssh -l goncalo <hostname>
  • In your local resources, you will execute your code in a subdirectory inside your '/home/<user_home>'.
$ cd /home/goncalo/BELLE/interactive-example/
$ ls -l
total 30968
-rw-r--r-- 1 goncalo people      717 Dec  8 00:27 belle_env.sh
-rw-r--r-- 1 goncalo people 31687972 Dec  8 00:51 lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst
-rw-r----- 1 goncalo people     4235 Dec  8 00:59 Makefile
-rw-r--r-- 1 goncalo people     3804 Dec  8 00:49 myparticle.cc
-rwxr-xr-x 1 goncalo people      515 Dec  8 01:56 runbasf.sh
  • In the cloud, you can also execute your code from a subdirectory inside '/data/<user_login>/'
$ cd /data/goncalo/BELLE/interactive-example/
$ ls -l
total 30968
-rw-r--r-- 1 goncalo people      717 Dec  8 00:27 belle_env.sh
-rw-r--r-- 1 goncalo people 31687972 Dec  8 00:51 lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst
-rw-r----- 1 goncalo people     4235 Dec  8 00:59 Makefile
-rw-r--r-- 1 goncalo people     3804 Dec  8 00:49 myparticle.cc
-rwxr-xr-x 1 goncalo people      515 Dec  8 01:56 runbasf.sh
  • The source code for this example will be the following:
    1. belle_env.sh: An belle environment initialization script
    2. myparticle.cc: A specific BELLE module for BASF BELLE software
    3. Makefile: Makefile to compile the BELLE module
    4. lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst: Input data for the application
    5. runbasf.sh: The script that invokes the BASF execution
  • The environment script is as follow:
$ cat belle_env.sh
#!/bin/bash

unset DYLD_LIBRARY_PATH
unset LIBPATH
unset SHLIB_PATH

# --- start BELLE setup ---

export BELLE_SOFT=/cvmfs/experimental.cloud.coepp.org.au/
export BHOME=$BELLE_SOFT/belle
export BELLE_LEVEL=b20090127_0910
export BELLE_DEBUG=opt
export USE_GRAND_REPROCESS_DATA=1
export BELLE_MSG_MAX_SHLVL=1                              # prevent bashrc_general output

source ${BHOME}/local/etc/bashrc_general                  # source belle init script

export BELLE_POSTGRES_SERVER=rcbelledb.mel.coepp.org.au   # redefine BELLE_POSTGRES_SERVER
export PGUSER=belle                                       # redefine PGUSER
export PANTHER_VERBOSE_INIT=0                             # redefine PANTHER_VERBOSE_INIT
export FPDA_TIMEOUT=0                                     # redefine FPDA_TIMEOUT
  • Source the environment script
$ source belle_env.sh
** Database setup is for the analysis with the caseB data
**   caseB = data with NEW (old) tracking for e31- (e7-27).
** Unset USE_GRAND_REPROCESS_DATA for the analysis of the caseA data
  • Compile the module. As a result of a successfull compilation, the following files should be created
    1. myparticle.d
    2. myparticle.o
    3. myparticle.so
$ make
(...)
  • Check the script that invokes the BASF execution
$ cat runbasf.sh
#!/bin/sh

# --- Application Definitions ---

MODULE=myparticle
HBKFILE=test.hbk
# OUTFILE=test.dat
INFILE=lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst
NEVENT=25

# --- Application Execution ---

basf <<EOF
path create main
path create analysis
path add_module main fix_mdst
path add_condition main >:0:analysis
path add_condition main <=:0:KILL
path add_module analysis ${MODULE}
initialize
#histogram define ${HBKFILE}
process_event ${INFILE} ${NEVENT}
#output close
terminate
EOF
  • Run the script, and check the output
$ ./runbasf.sh > runbasf.log 2>&1
$
$ cat runbasf.log
path create main
path create analysis
path add_module main fix_mdst
path add_condition main >:0:analysis
path add_condition main <=:0:KILL
path add_module analysis myparticle
***** [myparticle] Module loaded successfully.
initialize

[myparticle] parameters
#histogram define test.hbk
7_0910.mdst 25lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b2009012
0 1 7 1 0
3 0 8 4 1
0 1 8 1 0
0 1 6 1 2
2 0 10 0 1
3 2 15 0 0
0 1 8 1 0
4 1 10 0 0
0 2 7 2 1
0 1 6 2 0
3 1 5 0 0
2 1 13 0 1
2 0 7 2 0
0 1 6 0 1
1 0 13 1 2
0 2 5 0 0
1 0 13 0 0
2 1 7 1 0
1 1 4 2 0
2 0 8 3 0
2 1 14 1 0
2 0 8 0 1
2 0 8 2 0
RESULT : Basf_io:md5::d41d8cd98f00b204e9800998ecf8427e  lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst
-------------------------- BASF Execution Statistics ------------------------------------
      Module         |       CPU time(sec)         |   # of calls     | CPU time/call
---------------------+-----------------------------+------------------+------------------
    fix_mdst         |             0(       0)     |          23      |       0.000
    myparticle       |             0(       0)     |          23      |       0.000
---------------------+-----------------------------+------------------+------------------
 Total CPU time(sec) |             0(       0)                        |       0.001
---------------------+-----------------------------+------------------+------------------
#output close
terminate
psql: could not translate host name "can52" to address: Name or service not known
   Exp   Run    #evts        Lum   #evts/lum            4S     Continuum            3S      above 4S         other
Total On4S lum. =             0 Continuum Lum. =             0 3S Lum. =             0 Above4S Lum. =             0 Other Lum. =             0

 MZSTOR.  ZEBRA table base TAB(0) in /MZCC/ at adr          -1    FFFFFFFF HEX

 MZSTOR.  Initialize Store  0  in /GCBANK/
          with Store/Table at absolute adrs     1023773          -1
                                        HEX       F9F1D    FFFFFFFF
                                        HEX       F9D4A           0
                              relative adrs     1023306           0
          with     1 Str. in     2 Links in   5300 Low words in 5999970 words.
          This store has a fence of   16 words.

 MZLOGL.  Set Log Level 0 for store  0

BASF example: batch execution

  • Login at your preferred resource center (Sydney or Melbourne or Cloud)
$ ssh -l goncalo <hostname>
  • In your local resources, you will execute your code in a subdirectory inside your '/home/<user_home>'.
$ cd /home/goncalo/BELLE/batch-example/
$ ls -l
total 30968
-rw-r--r-- 1 goncalo people      717 Dec  8 00:27 belle_env.sh
-rw-r--r-- 1 goncalo people 31687972 Dec  8 00:51 lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst
-rw-r----- 1 goncalo people     4235 Dec  8 00:59 Makefile
-rw-r--r-- 1 goncalo people     3804 Dec  8 00:49 myparticle.cc
-rwxr-xr-x 1 goncalo people      515 Dec  8 01:56 runbasf_batch.sh
  • In the cloud, you must execute your code from a subdirectory inside '/data/<user_login>/'
$ cd /data/goncalo/BELLE/batch-example/
$ ls -l
total 30968
-rw-r--r-- 1 goncalo people      717 Dec  8 00:27 belle_env.sh
-rw-r--r-- 1 goncalo people 31687972 Dec  8 00:51 lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst
-rw-r----- 1 goncalo people     4235 Dec  8 00:59 Makefile
-rw-r--r-- 1 goncalo people     3804 Dec  8 00:49 myparticle.cc
-rwxr-xr-x 1 goncalo people      515 Dec  8 01:56 runbasf_batch.sh
  • The source code for this example will be the following:
    1. belle_env.sh: An belle environment initialization script
    2. myparticle.cc: A specific BELLE module for BASF BELLE software
    3. Makefile: Makefile to compile the BELLE module
    4. lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst: Input data for the application
    5. runbasf_batch.sh: The batch submission script that invokes the BASF execution
  • The environment script is as follow:
$ cat belle_env.sh
#!/bin/bash

unset DYLD_LIBRARY_PATH
unset LIBPATH
unset SHLIB_PATH

# --- start BELLE setup ---

export BELLE_SOFT=/cvmfs/experimental.cloud.coepp.org.au/
export BHOME=$BELLE_SOFT/belle
export BELLE_LEVEL=b20090127_0910
export BELLE_DEBUG=opt
export USE_GRAND_REPROCESS_DATA=1
export BELLE_MSG_MAX_SHLVL=1                              # prevent bashrc_general output

source ${BHOME}/local/etc/bashrc_general                  # source belle init script

export BELLE_POSTGRES_SERVER=rcbelledb.mel.coepp.org.au   # redefine BELLE_POSTGRES_SERVER
export PGUSER=belle                                       # redefine PGUSER
export PANTHER_VERBOSE_INIT=0                             # redefine PANTHER_VERBOSE_INIT
export FPDA_TIMEOUT=0                                     # redefine FPDA_TIMEOUT
  • Set up your environment. In batch submissions, you can use one of two alternative methods:
    1. Either source the environment script inside your submission script;
    2. Or source the environment script before submission, and submit your job with the '-V' option. This option ensures that the submmited job inherits the environment in the submission shell . This is the preferred use case for this example.
$ source belle_env.sh
** Database setup is for the analysis with the caseB data
**   caseB = data with NEW (old) tracking for e31- (e7-27).
** Unset USE_GRAND_REPROCESS_DATA for the analysis of the caseA data
  • Compile the module. You can also compile the module inside your submission script, or compile it before the submission. If you are submiting several jobs using the same source code and different inputs, the best solution is to compile the module before submission, and invoke the produced binary in the many different jobs. As a result of a successfull compilation, the following files should be created
    1. myparticle.d
    2. myparticle.o
    3. myparticle.so
$ make
(...)
  • Check your submission script. Please note that there are a couple of diferences with respect to the interactive script:
    • There is a section for batch system directives (start with '#PBS')
    • You need to ensure that you are executing your code in the same directory where it was submitted ('cd $PBS_O_WORKDIR')
$ cat runbasf_batch.sh
#!/bin/sh

# --- Start PBS directives ---

# Job Name
#PBS -N myparticle

# The batch job inherits the submission environment
#PBS -V

# Combining standard output and error
#PBS -j oe

# Send an email on abort or successfull ending
#PBS -m ae
#PBS -M goncalo@physics.usyd.edu.au


# --- Application Current Working Directory ---

cd $PBS_O_WORKDIR

# --- Application Definition ---

MODULE=myparticle
HBKFILE=test.hbk
INFILE=lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst
NEVENT=25

# --- Aplication Execution ---

basf <<EOF
path create main
path create analysis
path add_module main fix_mdst
path add_condition main >:0:analysis
path add_condition main <=:0:KILL
path add_module analysis ${MODULE}
initialize
#histogram define ${HBKFILE}
process_event ${INFILE} ${NEVENT}
#output close
terminate
EOF
  • Submit the application to the batch system and check its status
$ qsub runbasf_batch.sh
155623.sydtorque.syd.coepp.org.au

$ qstat
Job ID                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
155623.sydtorque          myparticle       goncalo                0 R syd_medium
  • If the job aborts or if it ends successfully, you should receive an alert email. The following is for a successfull execution:
BS Job Id: 155623.sydtorque.syd.coepp.org.au
Job Name:   myparticle
Exec host:  sydui1.syd.coepp.org.au/0
Execution terminated
Exit_status=0
resources_used.cput=00:00:00
resources_used.mem=30772kb
resources_used.vmem=813228kb
resources_used.walltime=00:00:16
Error_Path: sydui4.syd.coepp.org.au:/imports/sydpp/goncalo/BELLE/batch-example/myparticle.o155623
Output_Path: sydui4.syd.coepp.org.au:/imports/sydpp/goncalo/BELLE/batch-example/myparticle.o155623
  • The standard output and error of this example jobs will be available at the 'Output / Error' Paths (normally the submission directory) with the format <name_of_job>.o<jobid> / <name_of_job>.e<jobid>
$ cat myparticle.o155623
path create main
path create analysis
path add_module main fix_mdst
path add_condition main >:0:analysis
path add_condition main <=:0:KILL
path add_module analysis myparticle
***** [myparticle] Module loaded successfully.
initialize

[myparticle] parameters
#histogram define test.hbk
7_0910.mdst 25lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b2009012
0 1 7 1 0
3 0 8 4 1
0 1 8 1 0
0 1 6 1 2
2 0 10 0 1
3 2 15 0 0
0 1 8 1 0
4 1 10 0 0
0 2 7 2 1
0 1 6 2 0
3 1 5 0 0
2 1 13 0 1
2 0 7 2 0
0 1 6 0 1
1 0 13 1 2
0 2 5 0 0
1 0 13 0 0
2 1 7 1 0
1 1 4 2 0
2 0 8 3 0
2 1 14 1 0
2 0 8 0 1
2 0 8 2 0
RESULT : Basf_io:md5::d41d8cd98f00b204e9800998ecf8427e  lnu-e000055r000657r000669-s04-evtgen-charged-on_resonance-b20090127_0910.mdst
-------------------------- BASF Execution Statistics ------------------------------------
      Module         |       CPU time(sec)         |   # of calls     | CPU time/call
---------------------+-----------------------------+------------------+------------------
    fix_mdst         |             0(       0)     |          23      |       0.000
    myparticle       |             0(       0)     |          23      |       0.001
---------------------+-----------------------------+------------------+------------------
 Total CPU time(sec) |             0(       0)                        |       0.002
---------------------+-----------------------------+------------------+------------------
#output close
terminate
psql: could not translate host name "can52" to address: Name or service not known
   Exp   Run    #evts        Lum   #evts/lum            4S     Continuum            3S      above 4S         other
Total On4S lum. =             0 Continuum Lum. =             0 3S Lum. =             0 Above4S Lum. =             0 Other Lum. =             0

 MZSTOR.  ZEBRA table base TAB(0) in /MZCC/ at adr          -1    FFFFFFFF HEX

 MZSTOR.  Initialize Store  0  in /GCBANK/
          with Store/Table at absolute adrs     1023773          -1
                                        HEX       F9F1D    FFFFFFFF
                                        HEX       F9D4A           0
                              relative adrs     1023306           0
          with     1 Str. in     2 Links in   5300 Low words in 5999970 words.
          This store has a fence of   16 words.

 MZLOGL.  Set Log Level 0 for store  0

BELLE 2 examples

  • The previous twiki is protected by useraname and password. Please request access to your nearest BELLE 2 coordinator.

Using Ganga for cloud job submission

Ganga is a job submission and monitoring program written for the interactive python shell IPython. It is/was used at ATLAS/LHCb to make submitting user analysis jobs to the grid/LSF/local machine as easy as possible https://ganga.web.cern.ch/ganga/ Although Ganga isn't officially supported for use with BELLE 2 software, it's possible to run a local install yourself to submit basf2 jobs to the RC cloud and keep a record of what you've done.

  • These instructions are written for running on the cloud, so ssh into cxin01.cloud.coepp.org.au (or any of the nodes).
  • The cloud software currently doesn't include python 2.7 by default, which the current version of Ganga requires. So we'll install a local version of python 2.7 which will make things easier later anyway. Execute the commands below with your username to install python 2.7.11 and the rest of the Ganga install.

Python Install

Here the commands will install python to your /data/<username> area. It's useful to have a python 2.7 version accessible to submitted jobs rather than on the /home area

& cd /data/<username>/
& mkdir python-versions && cd python-versions
& wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tar.xz
& tar -xf Python-2.7.11.tar.xz
& cd Python-2.7.11

And after that

$ mkdir Install
$ ./configure --prefix=/data/<username>/python-versions/Python-2.7.11/Install/
$ make install

Now you should have python installed locally. You can check that the executable 'python2.7' exists in the '/data/<username>/python-versions/Python-2.7.11/Install/bin/' directory and that you can use it to run a simple python file.

Installing Ganga to your area

You can go to https://github.com/ganga-devs/ganga/releases to download the latest release of Ganga. Simply copy the URL of the tar file and download it into a directory that you are happy having the Ganga source code in e.g. For release 6.1.18

$ cd /data/<username>
$ mkdir ganga-versions && cd ganga-versions
$ wget https://github.com/ganga-devs/ganga/archive/6.1.18.tar.gz
$ tar -xf 6.1.18.tar.gz

This will create a directory similar to 'ganga-6.1.18'. Since Ganga is a python program we don't need to compile it. However it seems that Ganga does need to be in your /data/username area to work correctly. It may also be possible to use the 'pip install' procedure below to install Ganga directly, but that hasn't been tried yet

Installing required python packages and setting up virtual environments

Because Ganga requires an ipython version that works for python 2.7 we have to install it ourselves as well. At this point a good way to make sure we can set up the correct python version and correct ipython version for ganga is to use virtual environments. The cloud has the package 'virtualenv' already installed which makes this easy. First lets make a directory to install everything to and setup a python 2.7 virtual environment.

$ cd $HOME
$ mkdir python-environments && cd python-environments
$ virtualenv -p /data/<username>/python-versions/Python-2.7.11/Install/bin/python2.7 ganga_env

This creates a new directory called 'ganga_env' in which the correct python executable and the python package manager 'pip' have been installed (we didn't have to put it in the /home directory, you can put it wherever you want). Now you can activate the environment to make python 2.7 the default version, and then deactivate to go back to the default shell environment.

$ source ganga_env/bin/activate

(ganga_env)$ python --version
Python 2.7.11
(ganga_env)$ deactivate

$ python --version
Python 2.6.6

Note that the weird (ganga_env) prefix in the above commands is just something that your terminal will probably display to let you know that you're inside an virtual environment. It's not a command to be run.

Now we can install ipython in the ganga_env environment using pip (I noticed some issues with the latest ipython so we install 4.1.1 version)

$ source ganga_env/bin/activate
(ganga_env)$ pip install ipython==4.1.1

Check that ipython works by doing

(ganga_env)$ ipython --version

If you get errors it may be that ipython is missing packages. Check the error and install any missing packages e.g. pathlib2

(ganga_env)$ pip install pathlib2
(ganga_env)$ ipython --version
4.1.1

Finally we can run Ganga! First let's make the setup much easier for the future. Add the following lines to your ~/.bash_profile file to be able to automatically enter the virtual environment and have ganga available as a command (you will have to log out and back in again).

SETUPGANGA_ENV=/home/<username>/python-environments/ganga_env/bin/activate
alias setup_ganga_belle2="source $SETUPGANGA_ENV"
alias ganga="/data/<username>/ganga-versions/ganga-6.1.18/bin/ganga"

Now all you have to do to run Ganga is:

$ setup_ganga_belle2
(ganga_env)$ ganga

Remember, to exit from the 'ganga_env' virtual environment, just type 'deactivate'.

Configuring Ganga

You will probably get some warnings when running Ganga (it is expecting you to be on ATLAS). If you have a grid certificate in your .globus directory it might also ask you for your grid passphrase to create a proxy even though we won't use it to submit to the grid (At the moment there's no way round this but I will keep checking). This is not a problem, just enter it and allow Ganga to continue.

The first time running Ganga it will create a .gangarc file and a gangadir directory in your $HOME directory. The .gangarc is a configuration file that we need to change.

The gangadir is the location where Ganga will store all of the output files from your cloud jobs. Since the cloud nodes can't write to your $HOME directory we need to remove this gangadir and tell Ganga to put a new one in the /data/<username> area.

To make Ganga use the /data partition Go into your .gangarc and uncomment the relevant line and change it to the following

gangadir = /data/<username>/gangadir

Great! Now you can remove the $HOME/gangadir and when you next run Ganga it will create a directory on your /data/<username> area.

Using Ganga

Google is your friend (try searching for ATLAS/LHCb Ganga tutorials). Here is the main Github tutorial https://github.com/ganga-devs/ganga/wiki/Full-Tutorial

Ganga is designed to let you construct job submissions in ipython interactively, but you may prefer to create simple scripts to automate this. Here are two simple files that I've used before to submit multiple jobs to the cloud PBS backend running a basf2 steering file.

create_job.py

j = Job(name="test_basf2")
j.application = Executable()
j.application.exe = File('test_ganga.sh')
j.backend = PBS()
j.splitter = GenericSplitter()
j.splitter.attribute = 'application.args'
j.splitter.values = ['1', '2']
j.outputfiles = [LocalFile('DST_VXDAlignment.root'), LocalFile('Belle2FileCatalog.xml')]

test_ganga.sh

#!/bin/bash
# Start by setting up basf2 environment
source /cvmfs/belle.cern.ch/sl6/tools/setup_belle2
WORKDIR=$PWD
# current_release is just a basf2 installation I have in my /data area that is up to date, rather than an official build or release version on cvmfs.
cd /data/ddossett/software/BelleII/current_release
setuprel
cd $WORKDIR

# Now set the arguments to this particular basf2 steering file
# Which experiment
EXP_NUM=1
# How many events per run
N_EVENTS=10
# Cosmic? 1=True 0=False
COSMIC=0
# $1 is the run. We are using the splitter above to do one run per subjob and change this argument.

basf2 /data/ddossett/1_generate_VXDAlignment.py $EXP_NUM $1 $N_EVENTS $COSMIC

Then just call these commands to submit the job. The first just runs the python commands in create_job.py and then leaves you in the Ganga ipython shell. You can always inspect the Ganga Job object before doing j.submit() and change it's parameters. Note that if we had chosen Local() as the backend, instead of PBS(), the job would run locally on the node. This is makes it easy to test small jobs locally and then submit larger runs once you're happy it works.

$ ganga -i create_job.py
Ganga In [1]: j.submit()

When running Ganga you can check the past and present jobs, check the status of subjobs, resubmit failed jobs, check the job output and much more (check the documentation). The basic command to see all of your previous jobs is 'jobs'.

Ganga In [1]: jobs
Registry Slice: jobs (1 objects)
--------------
    fqid |    status |      name | subjobs |    application |        backend |                             backend.actualCE |                       comment
-------------------------------------------------------------------------------------------------------------------------------------------------------------
       0 | submitted |test_basf2 |       2 |     Executable |      PBS       | vm-115-146-84-190.melbourne.rc.nectar.org.au |

Moving from the old to the new KEKCC

Setup

Begin by copying your old home directory to your new home directory:

cd ~/../
cp -a /gpfs/home/old/belle/<user_name> ./

Note replace 'belle' with 'belle2' depending on your group.

Then you will need to overwrite your .bashrc (or .cshrc) to properly set up the environment variables. Copy over the contents of ~nishida6/public/Env/bashrc:

cd ~
rm .bashrc
cp ~nishida6/public/Env/bashrc ./
mv bashrc .bashrc

Note that this will completely remove your old shell script. If you want to keep some of the contents please make a copy of your .bashrc first, and then copy the desired parts over to your new .bashrc. You may also want to copy the contents of '/home/belle2/harat/OriginalSetting/.bashrc'. Exchange 'bashrc' with 'cshrc' if required.

Similarly, to copy your old group folder to the new working group area:

cd /group/belle/users
cp -a /group/belle/users_old/<user_name> ./

As before, replace 'belle' with 'belle2' if required.

tutorial/belle.txt · Last modified: 2016/09/15 11:51 by ahawthorne
 
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki