You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 8
Next »
The data handling group met last on Friday May 23 (Data Handling Meeting Agenda Page.)
Here are the agenda items from the May 23rd meeting:
May 23, 2008
- Nagios Overview/Status
- xrootd update
- Run Quality Update
- runQuality object to be added to scriptlet environment
- runQuality.setL1RunStatus(long run, String status)
- One of Running, Complete, InComplete, InProgress, Running, Unset
- runQuality.setReviewStatus(long run, String status)
- One of Not ready, Waiting review, (Reviewed, Unset)
- Datacat versions update
- AOB
Updates Since the meeting
- Data Catalog Versions
- DEV and TEST databases have been converted to the new "Versioned Dataset" tables and code.
- Tony has updated the Dataset Crawler to use the new tables and it is running in DEV
- Warren is making progress converting L1 to use versions.
- Dataset registration and querying works with the new code, at least in example tasks
- Jim Chiang is having problems with registration in his task
- I'm looking into this now
- Jim Chiang & Co were having trouble with Dataset querying yesterday, but Tony made a change that should have fixed this
- The new code is meant to be backward compatible to existing tasks
- This seems to be generally true except for Jim's task
- We would love to have more people exercising their tasks in DEV to confirm these claims
- Migrating the data in DEV (550k Datasets) took 22 minutes
- If this scales linearly, it will take 7.5 hrs to migrate the 11M Datasets in PROD
- Would be really nice to clean up datasets that no longer exist, in order to speed the migration
- Tony ran a special crawl on NFS files last night to identify those that no longer exist on disk.
- 91k files were identified.
- Not enough to make a big difference
- Would be nice to do this in XROOT, but an unresponsive node could give us false positives (or would that be 'positive falses'?)
- Run Quality