Tier-3 Software Management

• ATLAS and non ATLAS users can profit from a diversity of software available (through CVMFS mechanism) in the cluster nodes.

• There are 3 kinds of commands than can be executed to configure software
  • setupATLAS: For ATLAS users
  • setupSW: For non-ATLAS users
  • module <avail|list|load|unload>: for ALL users

• setupATLAS is a bash function which enables localSetup<TARGET> commands for specific <TARGET> configurations

$ setupATLAS
(...)

$ printMenu
...Type localSetupAGIS to setup AGIS
...Type localSetupAtlantis to setup Atlantis
...Type localSetupDQ2Client to use DQ2 Client
...Type localSetupEIClient to setup EIClient
...Type localSetupEmi to use emi
...Type localSetupFAX to use FAX
...Type localSetupGanga to use Ganga
...Type localSetupGcc to use alternate gcc
...Type localSetupPacman to use Pacman
...Type localSetupPandaClient to use Panda Client
...Type localSetupPyAMI to setup pyAMI
...Type localSetupPoD to setup Proof-on-Demand
...Type localSetupROOT to setup (standalone) ROOT
...Type localSetupRucioClients to setup rucio-clients
...Type localSetupSFT to setup SFT packages
...Type localSetupXRootD to setup XRootD
...Type showVersions to show versions of installed software
...Type asetup to setup a release (changeASetup to change asetup version)
...Type rcSetup to setup an ASG release (changeRCSetup to change rcSetup ver.)
...Type diagnostics for diagnostic tools
...Type helpMe for more help
...Type printMenu to show this menu

• For example, to known which ROOT versions are available, run localSetupROOT –help

$ localSetupROOT --help

Usage: localSetupROOT [options] [version]

    This sets up the ROOT environment.  It is meant to be used outside
    Athena so do not source this file if you intend to run Athena jobs.

    You need to set the environment variable ATLAS_LOCAL_ROOT_BASE first.

    Options (to override defaults) are:
     -h  --help             Print this help message
     --skipConfirm          Skip the confirmation dialogue
     --quiet                Print no output (also sets skipConfirm)
     --xrdVer=STRING        xrootd version to setup
                             it will setup the approproate OS/platform version
     --davixVer=STRING      davix version to setup
                             This is dev so default is current

    Note: if you run h2root, you may see an error about a missing shared
          library libshift.so (used by Castor).  If you see this, do
    export LD_LIBRARY_PATH=$LD_LIBARY_PATH:<path to an ATLAS kit>/external/slc3compat/2.0.0/i686-slc3-gcc323/lib

    Note: root 5.32 and newer will setup external xrootd package.
          To disable this, do prior to setup
            export ALRB_SKIP_XDR=1


ROOT versions:
 ---> 5.28.00g-i686-slc5-gcc43-opt
 ---> 5.28.00g-x86_64-slc5-gcc43-opt
(...)

• To configure a specific ROOT version, execute the following command (please note that the gcc and python versions are set accordingly)

$ localSetupROOT 5.34.25-x86_64-slc6-gcc48-opt
************************************************************************
Setting up ROOT ...
************************************************************************
Setting up ROOT version 5.34.25-x86_64-slc6-gcc48-opt
  Current GCC version: gcc (GCC) 4.8.1
  Python version in your path is 2.7.4
  Python version used to build ROOT is 2.7
Setting up gccxml version 0.9.0_20120309p2-x86_64-slc6-gcc48-opt
Setting up gsl version 1.10-x86_64-slc6-gcc48-opt
Setting up fftw version 3.1.2-x86_64-slc6-gcc48-opt
Setting up XRootD version 3.3.6.p1-x86_64-slc6-gcc48-opt
Setting up davix version 0.4.0-x86_64-slc6
 DAVIX root is $ALRB_DAVIX_ROOT
************************************************************************
 Tip for _this_ standalone ROOT and grid (ie prun) submission:
  avoid --athenaTag if you do not need athena
  use --rootVer=5.34/25 --cmtConfig=x86_64-slc6-gcc48-opt
************************************************************************

$ which root
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/root/5.34.25-x86_64-slc6-gcc48-opt/bin/root

$ which python
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/2.7.4-x86_64-slc6-gcc48/sw/lcg/external/Python/2.7.4/x86_64-slc6-gcc48-opt/bin/python

$ which gcc
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/Gcc/gcc481_x86_64_slc6/slc6/gcc48/bin/gcc

• setupSW executes the same function as setupATLAS. It was setup so that non-ATLAS users do not get confused, and understand that they can also profit from the software suites deployed by ATLAS. They do not need to be ATLAS members to profit from ATLAS software stacks.

$ setupSW
(...)

$ printMenu
...Type localSetupAGIS to setup AGIS
...Type localSetupAtlantis to setup Atlantis
...Type localSetupDQ2Client to use DQ2 Client
...Type localSetupEIClient to setup EIClient
...Type localSetupEmi to use emi
...Type localSetupFAX to use FAX
...Type localSetupGanga to use Ganga
...Type localSetupGcc to use alternate gcc
...Type localSetupPacman to use Pacman
...Type localSetupPandaClient to use Panda Client
...Type localSetupPyAMI to setup pyAMI
...Type localSetupPoD to setup Proof-on-Demand
...Type localSetupROOT to setup (standalone) ROOT
...Type localSetupRucioClients to setup rucio-clients
...Type localSetupSFT to setup SFT packages
...Type localSetupXRootD to setup XRootD
...Type showVersions to show versions of installed software
...Type asetup to setup a release (changeASetup to change asetup version)
...Type rcSetup to setup an ASG release (changeRCSetup to change rcSetup ver.)
...Type diagnostics for diagnostic tools
...Type helpMe for more help
...Type printMenu to show this menu

• People from theory may be specially interested in the localSetupSFT command since it makes available a wide set of MC generators and 3rd party pieces of software like blas, boost, cernlib, clhep, and much more.

$ localSetupSFT --help

Usage: localSetupSFT [options] [version]

    This sets up the ATLAS environment for LCG externals packages

    You need to set the environment variable ATLAS_LOCAL_ROOT_BASE first.

    Options (to override defaults) are:
     -h  --help             Print this help message
     --skipConfirm          Skip the confirmation dialogue
     --quiet                Print no output
     --lcgExtSW=STRING      LCG External software to setup
                             This is from the list below.
                             Separate multiple values with commas.
     --cmtConfig=STRING     CMTCONFIG version to use
                             Default is Athena or Standalone ROOT's cmtconfig

     For more details, please see
       https://twiki.atlas-canada.ca/bin/view/AtlasCanada/LocalSetupSFT

sft versions:
 bjam/3.1.13
 blas/20070405
 blas/20110419
 Boost/1.42.0_python2.5
(...)

• A 3rd party software can be configured running localSetupSFT <TARGET>

$ localSetupSFT MCGenerators_lcgcmt67b/herwig++/2.7.1
************************************************************************
  Current GCC version: gcc (GCC) 4.8.1
  Python version in your path is 2.7
Setting up herwig++ 2.7.1 ...
 The env $SFT_HOME_herwigPP is the home dir
************************************************************************

• See the setupATLAS explanations to further understand how to configure other target sofwares

• This is a mechanism to make available (different versions of) software which:
  • are not available via setupSW / setupATLAS
  • are incompatible with the OS default software
  • are needed by a considerable set of users

• Users can issue the following commands:
  • module avail: show the modules currently installed in the system
  • module list: show the modules currently loaded by the user
  • module load <TARGET>: load a module
  • module unload <TARGET>: unload a module

• Modules names are currently following the convention bellow because:
  1. The name is self-explanatory about how the software was built
  2. The name is self-explanatory about software requirements (specially important for specific gcc and python versions)
  3. This structure allows to provide the same software compiled under different architecture, wih different compilers and python versions, if necessary

    <OS VERSION>/<COMPILER VERSION>/<PYTHON VERSION>/<NAME>-<VERSION>.<ARCH>

• If the module you want to use depends on a particular ROOT, gcc or python version, you may need to have those enabled before you load the module.
  • In the majority of the cases, that can be achieved running setupATLAS/SW and choosing the particular ROOT version you want to use, before loading the module.

• The purpose of this module is to provide a more recent version of texlive than the one provided, by default, with the SL6/EPEL6 repositories.

1) Check the current latex version

# which latex
/usr/bin/latex

# rpm -qf /usr/bin/latex
texlive-latex-2007-57.el6_2.x86_64

--- * ---

2) Check which software modules are available

# module avail
----------- /usr/share/Modules/modulefiles ------------
sl6/gcc447/bbcp-15.02.03.00.1.x86_64
sl6/gcc447/texlive-2014.x86_64
sl6/gcc480/python274/coepp/pytools-1.0.0.x86_64
sl6/gcc480/python274/madanalysis5-v1.1.11_patch1b.x86_64
----------- /etc/modulefiles --------------------------
openmpi-x86_64

--- * ---

3) Load the **sl6/gcc447/texlive-2014.x86_64** module

# module load sl6/gcc447/texlive-2014.x86_64

# module list
Currently Loaded Modulefiles:
  1) sl6/gcc447/texlive-2014.x86_64

# which latex
/cvmfs/experimental.cloud.coepp.org.au/sw/sl6/gcc447/x86_64/texlive/2014/bin/x86_64-linux/latex

--- * ---

4) Go back to the default latex version

# module unload sl6/gcc447/texlive-2014.x86_64

$ module list
No Modulefiles Currently Loaded.

# which latex
/usr/bin/latex

• This module provides bbcp, a multistream transfer tool developed at BABAR.

1) Check which software modules are available

# module avail
----------- /usr/share/Modules/modulefiles ------------
sl6/gcc447/bbcp-15.02.03.00.1.x86_64
sl6/gcc447/texlive-2014.x86_64
sl6/gcc480/python274/coepp/pytools-1.0.0.x86_64
sl6/gcc480/python274/madanalysis5-v1.1.11_patch1b.x86_64
----------- /etc/modulefiles --------------------------
openmpi-x86_64

--- * ---

2) Load the sl6/gcc447/bbcp-15.02.03.00.1.x86_64 module so that the bbcp command becomes available

$ module load sl6/gcc447/bbcp-15.02.03.00.1.x86_64

$ which bbcp
/cvmfs/experimental.cloud.coepp.org.au/sw/sl6/gcc447/x86_64/bbcp/15.02.03.00.1/bin/bbcp

--- * ---

3) List the current loaded modules

$ module list
Currently Loaded Modulefiles:
  1) sl6/gcc447/bbcp-15.02.03.00.1.x86_64

--- * ---

4) Unload the module

$ module unload sl6/gcc447/bbcp-15.02.03.00.1.x86_64

$ module list
No Modulefiles Currently Loaded.

• 'bbcp -P <sec> -s <nstreams> -D -e -E <chk>-r <dir> <user>@<remote_host>:/path/dir/'
  1. 'P <sec>': produces progress messages every sec seconds.
  2. '-s <nstreams>': Number of streams
  3. '-D': Debug
  4. '-e -E <chk>': Options to compute checksums during transfers, at source and target, using algorithm <chk> (md5 is normally used)
  5. '-r <dir>': Recursive copy

• The –N option enables copying via a combination of named pipes or programs.
• The arguments to –N indicate whether the source (i) is a named pipe or program or the target (o) is a named pipe or program.
• When the source is a named pipe or program, only one source may be specified. Determination of whether the data source or target is a pipe or program is done at copy-time on each node. If a pipe is detected, it is treated as a regular file with no specified size.

bbcp -P 10 -s 8 -e -E md5 -N io 'tar cvf - <source directory>' root@<remote_host>:'tar -C <destiny dir> -xvf -'

# Does not unpacks the tar at the destiny (good for lustre)
bbcp -P 10 -s 8 -e -E md5 -N i "tar cvf - <source directory>" root@<remote_host>:<destiny dir>

• http://www.slac.stanford.edu/~abh/bbcp/

• This module deploys several python packages under an independent virtualenv, namely:
  1. cython-0.22
  2. ipython-3.1.0
  3. numpy-1.9.2
  4. matplotlib-1.4.3
  5. pandas-0.16.1
  6. root_numpy-4.2.0
  7. rootplot-2.2.1
  8. rootpy-0.7.1
  9. scipy-0.15.1
  10. sympy-0.7.6

• Some of the python packages dependent on ROOT. This module was compiled with the default ROOT version for ATLAS (presently 5.34.25-x86_64-slc6-gcc48-opt) available through CVMFS. By choosing this ROOT version, automatically the compilation is set to use gcc480 and pytyhon 2.7.4

• To use this module, ROOT has to be available in advance

1) Load ROOT

# setupATLAS (or setupSW)
# localSetupROOT
$ which root
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/root/5.34.25-x86_64-slc6-gcc48-opt/bin/root
$ which gcc
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/Gcc/gcc481_x86_64_slc6/slc6/gcc48/bin/gcc
$ which python
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/2.7.4-x86_64-slc6-gcc48/sw/lcg/external/Python/2.7.4/x86_64-slc6-gcc48-opt/bin/python

--- * ---

2) Check that numpy and scipy are not available

$ python
Python 2.7.4 (default, May  5 2013, 17:33:10)
[GCC 4.8.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy
>>> import scipy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named scipy
>>>

--- * ---

3) Load the 'sl6/gcc480/python274/coepp/pytools-1.0.0.x86_64' module, and retry to import numpy and scipy

$ module load sl6/gcc480/python274/coepp/pytools-1.0.0.x86_64

$ module list
Currently Loaded Modulefiles:
  1) sl6/gcc480/python274/coepp/pytools-1.0.0.x86_64

$ python
python 2.7.4 (default, May  5 2013, 17:33:10)
[GCC 4.8.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import scipy
>>>

--- * ---

4) Unload the module

$ module unload sl6/gcc480/python274/coepp/pytools-1.0.0.x86_64

• Coepp pytools is currently deployed as a virtualenv. All python software that is thought to be useful and used by multiple persons should be deployed through this mechanism.

• However, users have the possibility to create their own virtual envs in some personnel / private areas for specific things. The problem is that, by default, there is no inheritance of multiple virtualenv, and by enabling a personnel virtualenv, the user looses access to the coepp pytools. VirtualEnvs are there to isolate environments, and there is no option at the command line level, and at VirtualEnv creation time, to allow inheritance.

• To enable private virtualenvs without loosing access to the central coepp pytools virtualenv, users will have to activate their private environments but do not use the python binary deployed there. To achieve this, users have to:
  • not activate their personnel virtualenv
  • Run the 'activate_this.py' script which is deployed with their personnel virtualenv

• In the following example, I demonstrate how I can use django, deployed under my private virtual env, and still use the python software deployed through coepp pytools virtualenv:

-bash-4.1$ python
Python 2.7.4 (default, May  5 2013, 17:33:10)
[GCC 4.8.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> activate_this = '/home/goncalo/coepp/mypersonnelvirtualenv/bin/activate_this.py'
>>> execfile(activate_this, dict(__file__=activate_this))
>>> import django
>>> import rootpy

• Madanalysis5 depends on ROOT and on several python software:
  • This module was compiled with the default ROOT version for ATLAS (presently 5.34.25-x86_64-slc6-gcc48-opt) available through CVMFS. By choosing this ROOT version, automatically the compilation is set to use gcc480 and pytyhon 2.7.4
  • The python software is provided by coepp pytools, which is loaded automatically.

1) Load ROOT

# setupATLAS (or setupSW)
# localSetupROOT
$ which root
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/root/5.34.25-x86_64-slc6-gcc48-opt/bin/root
$ which gcc
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/Gcc/gcc481_x86_64_slc6/slc6/gcc48/bin/gcc
$ which python
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/2.7.4-x86_64-slc6-gcc48/sw/lcg/external/Python/2.7.4/x86_64-slc6-gcc48-opt/bin/python

--- * ---

2) Load 'sl6/gcc480/python274/madanalysis5-v1.1.11_patch1b.x86_64' module. Check that 'sl6/gcc480/python274/coepp/pytools-1.0.0.x86_64' is loaded automatically

$ module load sl6/gcc480/python274/madanalysis5-v1.1.11_patch1b.x86_64

$ module list
Currently Loaded Modulefiles:
  1) sl6/gcc480/python274/coepp/pytools-1.0.0.x86_64            2) sl6/gcc480/python274/madanalysis5-v1.1.11_patch1b.x86_64

$ which ma5
/cvmfs/experimental.cloud.coepp.org.au/sw/sl6/gcc480/python274/x86_64/madanalysis5/v1.1.11_patch1b/bin/ma5

--- * ---

3) Run madanalysis

$ ma5

*************************************************************
*                                                           *
*        W E L C O M E  to  M A D A N A L Y S I S  5        *
*                         ______  ______                    *
*                 /'\_/`\/\  __ \/\  ___\                   *
*                /\      \ \ \_\ \ \ \__/                   *
*                \ \ \__\ \ \  __ \ \___``\                 *
*                 \ \ \_/\ \ \ \/\ \/\ \_\ \                *
*                  \ \_\\ \_\ \_\ \_\ \____/                *
*                   \/_/ \/_/\/_/\/_/\/___/                 *
*                                                           *
*   MA5 release : 1.1.11                       2014/09/15   *
*                                                           *
*         Comput. Phys. Commun. 184 (2013) 222-256          *
*           J. Phys. Conf. Ser. 123 (2014) 012032           *
*                  arXiv:1405.3982 [hep-ph]                 *
*                                                           *
*   The MadAnalysis Development Team - Please visit us at   *
*            https://launchpad.net/madanalysis5             *
*                                                           *
*              Type 'help' for in-line help.                *
*                                                           *
*************************************************************
Platform: Linux 2.6.32-504.3.3.el6.x86_64 [Linux mode]
Reading user settings ...
Checking mandatory packages:
     - python                   [OK]
     - python library: numpy    [OK]
     - g++                      [OK]
     - GNU Make                 [OK]
     - Root                     [OK]
     - PyRoot libraries         [OK]
Checking optional packages:
     - pdflatex                 [OK]
     - latex                    [OK]
     - dvipdf                   [OK]
     - zlib                     [OK]
     - FastJet                  [OK]
     - Delphes                  [OK]
     - Delphes-MA5tune          [OK]
     - Recasting tools          [OK]
Checking the MadAnalysis library:
  => MadAnalysis libraries found.
  => MadAnalysis test program works.
*************************************************************
MadGraph 5 NOT found:
  => Particle labels from input/particles_name_default.txt
  => 87 particles successfully exported.
  => Multiparticle labels from madanalysis/input/multiparticles_default.txt
  => Creation of the label 'invisible' (-> missing energy).
  => Creation of the label 'hadronic' (-> jet energy).
  => 8 multiparticles successfully exported.
ma5>exit

--- * ---

4) Unload the module

$ module unload sl6/gcc480/python274/madanalysis5-v1.1.11_patch1b.x86_64

$ module list
No Modulefiles Currently Loaded.