Page History

The data handling group met last on Friday May 16 23 (Data Handling Meeting Agenda Page.)

Here are the agenda items from the May 16th 23rd meeting: 

May 23, 2008

• Nagios Overview/Status
• xrootd update
• Run Quality Update
  • runQuality object to be added to scriptlet environment
    • runQuality.setL1RunStatus(long run, String status)
      • One of Running, Complete, InComplete, InProgress, Running, Unset
    • runQuality.setReviewStatus(long run, String status)
      • One of Not ready, Waiting review, (Reviewed, Unset)
• Datacat versions update
• AOB

Updates Since the meeting

• Data Catalog Versions
  • DEV and TEST databases have been converted to the new "Versioned Dataset" tables and code.
  • Tony has updated the Dataset Crawler to use the new tables and it is running in DEV
  • Warren is making progress converting L1 to use versions.
  • Dataset registration and querying works with the new code, at least in example tasks
    • Jim Chiang is having problems with registration in his task
      • I'm looking into this now
    • Jim Chiang & Co were having trouble with Dataset querying yesterday, but Tony made a change that should have fixed this
      • Jim has confirmed that he is able to query datasets successfully
    • The new code is meant to be backward compatible to existing tasks
      • This seems to be generally true except for Jim's task
  • We would love to have more people exercising their tasks in DEV to confirm these claims
  • Migrating the data in DEV (550k Datasets) took 22 minutes
    • If this scales linearly, it will take 7.5 hrs to migrate the 11M Datasets in PROD
    • Would be really nice to clean up datasets that no longer exist, in order to speed the migration
      • Tony ran a special crawl on NFS files last night to identify those that no longer exist on disk.
        • 91k files were identified.
          • Not enough to make a big difference
        • Would be nice to do this in XROOT, but an unresponsive node could give us false positives (or would that be 'positive falses'?)
• Run Quality
  • there is now a runQuality object available to scriptlets running in the DEV pipeline, to allow L1 to set the run status. See:  http://glast-ground.slac.stanford.edu/Pipeline-II/xml.jsp?task=825965&dataSourceMode=Dev  for an example of its use.