Page History

Incoming: On rh6 incoming raw data are in /nfs/farm/g/glast/u23/ISOC-flight/Archive/level0 (fastcopy in ../fcopy). End of April 2023 this dir was copied to /sdf/group/fermi/n/u23/ISOC-flight/Archive/level0. That's why tests on dev are done with runs of April1, 2023. Additionally we need RetDef files, which are in rhel6-64:/nfs/farm/g/glast/u42/ISOC-flight/Downlinks/<dl>/. These files typically have time stamps of some dozens of seconds before the submission of the HalfPipe stream.  Which process creates these files?

• launchChunks: is just an embedded jython scriptlet launching a stream for every chunk: for i in range(0, numChunks): pipeline.createSubstream("doChunk", i+1)
• doChunk.makeEvt.sh
  • each makeEvt creates identical lock files ${HALFPIPE_OUTPUTBASE}/lock/halfpipe-${HALFPIPE_DOWNLINKID} .  Have to check when this file is being removed.
  • RetDef files $HALFPIPE_OUTPUTBASE/$HALFPIPE_DOWNLINKID/RetDef-$HALFPIPE_DOWNLINKID-$HALFPIPE_CHUNKID.xml are missing.  u42 isn't on s3df.  We copied these files from rhel6-64:/nfs/farm/g/glast/u42/ISOC-flight/Downlinks/<dl>/ .
• mergeIndices.sh
  • tokendir is read dinamically: tokendir=`cat ${taskBase}/config/stagedir`/chunktokens . For testing we changed stagedir from /nfs/farm/g/glast/u28/stage to /sdf/group/fermi/a/isoc/flightOps/volumes/vol5/offline/halfPipe/v7r0p0_devel/u28/stage .
  • at the end a mergeEvt stream for each run is created.  This won't work in the container.  This task is being taken by launchMerge , an embedded scriptlet in the xml.
• launchMerge: embedded scriptlet to launch mergeEvt
• mergeEvt.sh
  • should receive ${HALFPIPE_RUNSTART} from launchMerge.  Doesn't work right now.  Workaround: read the decimal runid from PIPELINE_STREAM and convert to hex.  Michael thinks this should be safe.
  • not fault of mergeEvt, I guess: mergeEvt runs always on sdfmilan11[56].  There is no /lscratch but /scratch , and /sdf/group/fermi/a/isoc/flightOps/volumes/vol5/offline/halfPipe/v7r0p0_devel/outputBase is mapped to uid 7234 , while glastraw is 9503 .   How to report this? Ticket open.
  • Michael did something else yesterday to make mergeEvt finish.  But probably the above should be understood and fixed first.
• launchL1 should work (I (Nicola) haven’t added the "submit" script yet, but it gets to the point where it call the “submit stream” command successfully created.
• launchOnline: there is still the question about the “evt” directory… we need to check on the rhel6 who is making this dir and how it is used? We also have multiple directories called “stage” (one in /sdf/group/fermi/a/isoc/flightOps/volumes/vol5/offline/halfPipe/v7r0p0_devel/u28/stage/  one in /sdf/group/fermi/a/isoc/flightOps/volumes/vol5/offline/halfPipe/v7r0p0_devel/outputBase/stage/  we probably need a bit of cleaning.