Grid Basics

ON-NET vs OFF-NET Network Traffic

Large data transfers can lead to large network costs if they go through commercial networks instead of research networks. Before doing large dataset transfers it is recommended that you check that the remote sites are reachable via ON-NET routes. AARNET provide a useful website that you can use to determine whether remote sites are On-net or not:

Note: The Melbourne UI and batch servers (agu1,agu2,agu3,agu4) and the Australian ATLAS TIER 2 servers are all on networks which are restricted to On-net traffic only - if you are using any of these you do not need to check the On-net status of your transfers since they can only reach research networks.

Creating a voms proxy

A VOMS-enabled Grid proxy is a certificate that uniquely identifies you on the grid, allowing you access resources. Create a VOMS-enabled Grid proxy as below:

[fifieldt@ui ~]$ au-voms-proxy-init

Enter GRID pass phrase:
Your identity: /C=AU/O=APACGrid/OU=The University of Melbourne/CN=Tom Fifield
Creating temporary proxy ........................................ Done
Contacting [/DC=ch/DC=cern/OU=computers/] "atlas" Done
Creating proxy ................................. Done
Your proxy is valid until Wed Jul 30 02:27:59 2008

You may encounter the following error:

Error: Self signed certificate in certificate chain.
This is cause by some rubbish left behind in the .globus/certificates directory when creating a cert with Grix. You can fix this error by simply deleting the .globus/certificates directory and running voms-proxy-init -voms atlas again.
$ rm -r ~/.globus/certificates

Working with grid data

The basics of grid data

The absolute basics of the grid and data…

  • The TIER2 at Melbourne is a grid site called, AUSTRALIA-ATLAS
  • The grid is broken up into sites with data storage at each site
  • The storage at each site is broken up into chunks for the different types of data from the experiment (AOD, ESD, User created datasets etc.)
  • There are two basic ways to add or remove data from our site (AUSTRALIA-ATLAS)
    • The Distributed Data Management (DDM) system and Panda data requests
      • DDM is used to move datasets to the site automatically and very quickly
      • The different chunks of storage at the site are called space tokens in DDM
    • DQ2
      • DQ2 is used to create new datasets and upload data into the grid, and to download small datasets for prototyping
      • The different chunks of storage at the site are called sources or destinations (interchangeably) in DQ2

Space Tokens/Sources/Destinations and what data goes in them

You can see the sources/space tokens of our TIER2 by running:

$ dq2-sources | grep AUS

Each of these “sources” are just chunks of our total storage, each of a limited size and with a specific purpose:

Space Token/Source/Destination name Usage Size Read Write via Panda Write via DQ2
AUSTRALIA-ATLAS_DATADISK Production datasets as distributed by the Distributed Data Management (DDM) team, of interest to us; AOD & ESD are stored here and automatically transfered via DDM 65T Yes No No
AUSTRALIA-ATLAS_SCRATCHDISK The temporary input or output data for your grid jobs, this is a scratch area for your grid jobs knock yourself out 5T Yes Yes Yes
AUSTRALIA-ATLAS_LOCALGROUPDISK An area for any unique datasets that are shared by the group. This should be unique data *NOT* duplicates of existing ATLAS grid data sets. 10T Yes Yes Yes
AUSTRALIA-ATLAS_PRODDISK Used by the production job system (PANDA) to stage job data 10T Yes No No
AUSTRALIA-ATLAS_MCDISK Used for MC data, some users will have access to subscribe data to this area via Panda if it is appropriate to thier work 21T Yes Yes, if appropriate No

How much space do we have available in the Space Tokens/Sources/Destinations

The space free in each of these sources/destinations (A.K.A. space tokens) is here: DDM Prediction This shows the space available in each destination/source (A.K.A. space token) for our site. For the MCDISK the crosses indicate the predicted levels for data that is currently being transfered by DDM, both our normal fraction of AOD and any additional requested data sets (by users whom have been granted the right to do so).

Requesting data from DDM via Panda

The DDM system automatically transfers a fraction of the AOD from CERN to our site, users who are registered with Panda can also subscribe additional datasets to the site via DDM and they will be transfered. This system transfers data extremely quickly (>60MBs), and if you need a dataset at Melbourne this is the way to get it. Note: Our site cannot take all the data from the experiment, if you need a large dataset which is not on the site you will need to run the job another site where the data already exists!

Registering for access to Panda

Depending on what data and activity you are working on you will be granted access to transfer data to different sources/destinations (space tokens) in the site. The Panda monitor is used to request data sets, visit and fill in the form: Note: If you have access to transfer data to MCDISK, please we careful of two things:

  • We receive an automatic fraction of AOD into MCDISK as part of DDM, so please check we have not already got a data set before transferring2. .
  • The automatic AOD subscriptions combined with any you make may FILL THE MCDISK and PREVENT THE SITE FROM WORKING. Please keep an eye on the free space in MCDISK using the link the free space section above

Which datasets are subscribed to our site via DDM?

You can use dq2 tools to search for replicas of datasets in the grid, or to list all datasets at our site. You can see what data is currently being transfered into our site by taking a look at the DDM Section of the ATLAS Dashboard.


DQ2 allows you to download and upload datasets into the site, this is great for small datasets you want to copy lcoally for testing and for uploading and creating your own datasets i.e. at the end of a grid job or in order to share your data with other people who are at another site. The DQ2 Howto provides an excellent intro to get you started. Once you are running jobs you may need some more in-depth info, check the DQ2 User Guide for this. Note: The DQ2_LOCAL_SITE_ID is very important! On our site it is set to AUSTRALIA-ATLAS_LOCALGROUPDISK, you will want to change this for many grid operations. More about this below

Adding a dataset to the AUSTRALIA-ATLAS_LOCALGROUPDISK via DQ2

Taken from the DQ2 User Guide

$ dq2-put -L AUSTRALIA-ATLAS_LOCALGROUPDISK -s *directory-with-dataset* *dataset-name*
Dataset Pattern Put the search string you used in DQ2 here. If it's a user-created dataset add user* in front of this.
Software Version says you don't care about the version, or there is no version in the dataset name. Otherwise, choose the correct version.
Data Format Select the data format
Destination AUSTRALIA-ATLAS_LOCALGROUPDISK so it goes to the Australia-ATLAS local user area
Req Type Select the type of dataset
Data Management Mode DataTransfer always (otherwise you will get the PhysicsCoordinator on to you!)
Validity (transfer only) OneTimeCopy will copy the existing data set and leave it there for eternity. Periodic will update the dataset if there is changes to it
Priority Be considerate of other users. Be aware that the priority can be changed by the ROC
Transfer Volume 100% if you want the full dataset, smaller if you're just doing a test run
Comments A concise explanation of the reason for transfer.
tutorial/grid_basics.txt · Last modified: 2013/02/19 10:01 by lucien
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki