Pipeline- EGEE GRID interface

This page compiles the development approach for the implementation of an interface between the Pipeline2 system and european EGEE (Enabling Grids for E-sciencE) grid.  Pipeline2 system is now able to launch jobs either in SLAC or CC IN2P3 batch farms.   This project is then aimed to extend this distributed system to EGEE grid ressources. 

According to Pisa Team (Michael Kuss, Johan Bregeon, Francesco Longo) work, the use of EGEE workers is a good solution to produce huge amounts of MonteCarlo simulation data.   On this purpose, they have created EGEE GLAST Virtual Organization which is now operational and through which GLAST software has been installed in some EGEE sites (see VO status document by M.Kuss in attachment).  An important amount of data has been produced using this system, independently of Pipeline2 interfaces and data management procedures.  

The development of the link between Pipeline2 interfaces and EGEE GLAST Virtual Organization infrastructure should then allow to use EGEE ressources to answer to MonteCarlo simulations Pipeline2 requests. 

Some preliminary ideas have been given in order to organize this work:

  • Keep taskConfig.xml file as parameterization basis, new job type: EGEE (in addition to SLAC and IN2P3)
  • Keep the Structured file System @ CC IN2P3
  • Keep JobControlService Deamon at CC IN2P3
  • use EGEE UI environement @ Lyon  (generic certificate & proxy manual or automatic renewal, if possible) to launch Job.
  • Completely separate job launching and data retrieval

Some implementation steps have been suggested as well:

  • The files attached to the Job instance should be different for EGEE type tasks.   Then org-glast-pipeline-server package should be able to distinguish between an EGEE job and SLAC/IN2P3 farms ones, according to  taskConfig.xml file "type" variable value.
  • A new package contained in org-glast-jobcontrol jar file needs to be developped:  EGEEJobControlService (equivalent to LSFJobControlService and BQSJobControlService),  including appropriate submit(), getStatus() and cancel() methods.
  • Job outputs can remain in EGEE Storage Elements, provided that they are referenced in catalog databases, and that they can be easily downloaded by users 
  • No labels