Atlas Distributed Computing Operations Shifts

• ADCoS team shifts help to ensure that ATLAS computing resources working well and data is delivered to physicists.
• Monitor data transfers between T1s and T2s; MC production, data reprocessing, group production
• It is a class 2 shift with 24/7 coverage in 3 time zones EU(8-16)/US(16-24)/AP(0-8) CERN time. Shift can be taken from Home institute.
• Information about ATLAS Distributed Computing Operations Shifts is available at https://twiki.cern.ch/twiki/bin/viewauth/Atlas/ADCoS

Shifters start as Trainees to gather experience

1. Make sure you are registered with ATLAS
2. Have a valid grid certificate
3. Member of the ATLAS VO.

1. Send an email to Shift Cordinator Alexey Sedov asedov@pic.es or Shift Captain Hiroshi Sakamoto with the following text.

   Hi Alexey, 
   
   I'd like to sign up to be a trainee ADCoS shifter for "Asia Pacific/EU/US".
   My CERN username is <username> and my DN is DN: </C=XX/O=YYYYY/OU=ZZZZZ/CN=Your Name>
   Can you add me to shift bookings and put in a GGUS ticket for TEAM membership for me?
   
   Regards,
   <Your Name>

   • Note: Your DN can be found in the following link, search for your name and mark to show “Member DN”.
   • https://lcg-voms.cern.ch:8443/vo/atlas/vomrs?path=/RootNode/MemberAction/SetRgstrStatus&action=execute
2. Register in elog https://atlas-logbook.cern.ch/elog/ATLAS+Computer+Operations+Logbook/?cmd=New+user You will be required to fill the following
3. Create a CERN Savannah account at https://savannah.cern.ch/account/register.php. You can also use your CERN account details
4. Create a GGUS supporter account at https://gus.fzk.de/admin/get_account.php?accounttype=support

1. Go to OTP Tool http://atlas-otp.cern.ch/
2. Sign in using your normal CERN username and account
3. You should see two parts to your homepage: “Book My Shifts” and “Find My Shifts”
4. Expand the “Book My Shifts” by pressing the + symbol, if this section is not already expanded
5. Under “Book My Shifts” is a list for all shifts you are allowed to book. The number “Id” is the task id for the shift. Please click on the Task Id for the shift you wish to book.
6. If your shift (or shadow shift) does not appear here you have either not yet been listed as a member or the booking period is not (yet) open. Ask the responsible person for that shift to add you as a member or to open the shift booking period.
7. You should now be able to see a page titled “Shift Booking”, referencing this single shift. Next to the name of the shift is a + symbol. Click this + symbol in order to expand the section.
8. You should now be able to see a calendar. Please note that the you may have to edit the drop down menus which specify the start month, and how many months the calendar shows. You can only book shifts in the future, not in the past!
9. Any shift that is “pink” is unallocated. Select any “pink” shifts that you wish to book. They should now turn green.
10. To save these shifts: please click “Save Shift Booking”

1. Download and install a Jabber Client such as pidgin from http://www.pidgin.im/
2. Go to Accounts –> Manage Account –> Add. Fill in the details as shown below. Use you CERN username and password
3. Once the account is added and you are successfully connected to the server. Join the adcvcr room by going to Buddies –> Join a chat and fill the details as shown below.

1. Open the url http://panda.cern.ch/server/pandamon/query?dash=prod in your browser. You get a quick snapshot of the overall status for the production by cloud
2. Look at cloud efficiency (right hand side of the cloud table) and spot most problematic clouds.
3. First look for sites with lot of failing jobs as shown in figure 1
4. Then look for tasks with lot of failing jobs by going to url http://panda.cern.ch/server/pandamon/query?dash=task&show=active shown in figure 2
5. Compare them to know if the problem is site related or task related.
6. if several sites for one task are found , it seems to be the task , open a savannah bug report
7. if all jobs are at one site, it seems to be the site, open GGUS team ticket to the site

1. Open the DDM dashboard URL http://dashb-atlas-data.cern.ch/dashboard/request.py/site
2. Look for problematic cloud
3. Check the problematic site
4. Check the error pattern
5. Check the Endpoints
6. Check the downtime status
7. Check the downtime status in GOCDB
8. We have a site which fails FTS transfer. There is no ongoing downtime for this site. Now we have to check if there is any bug report on this in Adcos elog.
9. Checking the elog.
10. List of elogs fullfilling your search criteria appears.
11. Go back to the DDM Dashboard.
12. Get more details
13. After Clicking on Transfer Placement time you will be able to get the SRC SURL, DEST SURL and attempt counter.
14. We have collected the error details and link to dashboard page with error details. Now its time to check GGUS tickets.
15. Search the GGUS team ticket database.
16. If there is no GGUS team ticket regarding this error, submit a new one.
17. Click on Open Team Ticket.
18. Fill in the details such as error example, link to the dashboard, SRC SURL, DEST SURL, number of attempts and click on submit.
19. Make an entry in Adcos Elog.
20. Fill in the details, reference the GGUS ticket and click on submit.