Data Catalog

  • New release 0.7.1
  • Typed meta-data now available for setting and querying
    • Jython scripts works (mostly)
    • Linemode Client coming soon
  • New ability to tag datasets with multiple locations (ie SLAC NFS and XROOT)
  • Documentation coming soon!

Pipeline

  • New release 0.7.1 supporting new data catalog features
  • Can create multiple streams in one call to line mode client
  • Stream status now set properly during rollback
  • Submission to Lyon working
  • New job throttling mechanism (by task for now, run-time-modifiable)
  • Proper handling of processes that die when batch machines suddenly crash

Web Front End

  • New release of Pipeline front end
    • Complete view of processing by streams
    • Improved display of processes and subtasks (almost by execution order)
  • New release of Data Catalog front end
    • Displays new meta data, xrootd file locatoins

Data Server

  • Connection to data catalog is working. User can download file list from data catalog
  • Jobs run normally for both nfs and xrootd
  • Tony and Igor are investigating misterious crashes in some batch job runs on the same file lists when the number of parallel streams increases. (like 50 parallel streams) Could be a problem with the backend skimmer
  • Tony is working on a DataPortal interface that will be used for SC2, in which DataCatalog, Skimmer, Astro server have common History and user interface. 
  • No labels