Reason for change
This is the L1 that we will use for the public data release.
Because of an issue - still not understood - with CalCfpEnergy and Root 5.20 in Glast Release, we have decided to stick with Root 5.18 for the moment. This means that we had to take Science Tools v9r15p3 - which we need and which is compiled against Root 5.20 because of important bug fixes to TMinuit2 - and compile it against Root 5.18.00c-gl1. This has become ST v9r15p3gl2. The reason Science Tools and Glast Release are connected in such a way for L1 is because of ft2Util that links against both ST and GR (it reads in digi and merit files and writes out FT2 files). We have gone through the dependencies to make sure that nothing L1 uses from ST uses TMinuit2 (i.e. would need the bug fixes in Root 5.20) so ST v9r15p3gl2 should be safe to use for L1.
Test Procedure
We have processed run 271088335 in the DEV pipeline with this version of L1Proc and compared it to the original L1 processing. The files are:
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/1.75/ft1/gll_ph_r0271088335_v003.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/1.75/ft2/gll_pt_r0271088335_v003.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/1.75/ls1/gll_ev_r0271088335_v003.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/1.75/ls3/gll_lt_r0271088335_v003.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/1.75/merit/r0271088335_v003_merit.root
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/1.75/filteredMerit/r0271088335_v002_filteredMerit.root
We have validated the data in the following ways:
- CalCfpEnergy, beyond the Root 5.20 issue, also seems to produce slightly different results depending on the computer architecture. The SLAC batch farm contains several different ones, in particular rhel3 and rhel4-64, in addition to Intel and AMD. A run is always split into hundreds of pieces that randomly run on all of these different machines. This makes an event-by-event comparison more difficult. It also occasionally changes the number of events in the FT1 file. In this case the number of events in the FT1 file decreased from 36685 to 36682 from the original processing to the new one.
- I have verified for the test run that all FT1 columns except for ENERGY are identical between the two processings. For the ENERGY column, I see 49 events with a difference. All but two are tiny, <<1%. While two changes are at the 10% and 20% level. Neither of the two were Diffuse photons. This is consistent with changes previously seen with different computer architectures.
- Note that the CalCfpEnergy issue that has just been discovered is not understood at all and all comparisons should be seen in that light. It's clear that all data processed with the current GR (v15r47p12) - which we are not updating in the L1 release - suffers from this feature. And it's not clear when it was first introduced. Also note that this CalCfpEnergy issue seems to predominantly affect background events. When there is a change for Diffuse photons it's at the level of 10^-6 relative energy variation. This probably indicates that the problem is related to unstable/sensitive fits and background events. But that is just a guess.
- We have verified that FT1 EVENT_CLASS in the new processing is identical to FT1 CTBCLASSLEVEL in the original processing.
- We have verified through spot checks that the new FT1 Diffuse response columns are the same as the ones in Benoit's 11 months sample. Columns DIFRSP0 and DIFRSP1 in our FT1 files are the same as DIFRSP0 and DIFRSP2 in Benoit's file (there are tiny numerical differences between DIFRSP1 and DIFRPS2 - presumably round off errors). For this comparison we needed an older run and used run 263204954. The reprocessed FT1 file is here: root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/1.75/ft1/gll_ph_r0263204954_v001.fit. Benoit's file was /nfs/farm/g/glast/u33/lott/11month_dataset/ft1_may_rock43.fits.
- We have verified that the following columns are identical in the two FT2 files: START,STOP,LAT_GEO,LON_GEO,RAD_GEO,RA_ZENITH,DEC_ZENITH,B_MCILWAIN,L_MCILWAIN, GEOMAG_LAT,RA_SCZ,DEC_SCZ,RA_SCX,DEC_SCX,LAT_MODE,LIVETIME,QSJ_1,QSJ_2,QSJ_3,QSJ_4. The SC_POSITION column has to be expanded and is not covered by the diff script. I made a few spot checks. No changes.
- We have verified that the following new FT2 columns are present:: RA_NPOLE, DEC_NPOLE, ROCK_ANGLE, LAT_CONFIG, DATA_QUAL
- We have done spot checks to verify the consistency between LS1 and the original Merit ntuple.
Rollback procedure
We could switch back to the previous version of L1Proc. In that case we would have to turn off exporting files to the FSSC and switch to a mode where we do continous reprocessing for the FSSC files. Note that the current L1 pipeline uses the same Glast Release as this new one.
CCB Jira
Details
L1Pipeline v1r75:
- Make ft1 on chunks.
- Add diffuse response to ft1.
- Make our own gap file.
- Add quality and config to ft2.
- Creating filtered Merit files (photons only, for now).
- Move a bunch of stuff from setSuccessful to setRunning.
- Allow specifying the version to fileNames.fileName.
- Add version to chunk file names (but it's always 0 at the mo).
- Tweak hsplit to work around ROOT 5.20 issue.
- Allow data catalog type to differ from group.
- Allow attaching extra metadata when registering files - use to specify cut for filtered merit.
- fileNames.fileName can get file name from a variable, if so it overrides the normal rules.
scienceTools: v9r15p3gl2
- This was a major upgrade for Science Tools - the old L1 was still at v9r8p2.
- L1 is only using makeFT1, gtltcube, gtmktime and gtdiffrsp from the ScienceTools.
- In addition, ft2Utils and Verify are linking to Science Tools..
- This special version of Science Tools is compiled against Root 5.18.00c-gl1. See the top of the page for additional details. Note that the 'gl2' in the ST version number is not related to the Root version it is based on, but is purely a cvs branch tag number.
evtClassDef: v0r14p00
- v0r6p1: added updated LS1 dictionary file from Julie
- v0r7: add cuts and classifier script for Pass 7
- v0r8: add (pass-through) P7_electron_Classifier.py
- v0r9: add classifier script to map CTBClassLevel to EVENT_CLASS for Pass 6 reprocessing
- v0r10: new LS1variables
- v0r11: removed 5 new LS1variables (per Julie)
- v0r12: removed 'return' chars from LS1variables
- v0r13: revert to LS1variables of v0r9 (per Julie) to be used for first public data release
- v0r14: update Pass6_Reprocessing_Classifier
ft2Util: v1r2p31
- new version handling both old and new ft2 tpl file
- force DigiStart and DigiStop from command line
- added LiveTimeTolerance parameter to set to zero values below the tolerance
- fixed bug in mergeft2.cxx concerning ra_sc* values
- zero-length entries at the end of the 1-second ft2file are checked and skipped
- Fixes merging bug for IN_SAA (was previously not defined).
Complete set of tags for L1Proc 1.75
Code Versions
GlastRelease (sim/recon): v15r47p12
ScienceTools (Level 2) : v9r15p3gl2*
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r75*
calibTkrUtil v2r7p3
calibGenTKR v4r5
dataMonitoring/AlarmsCfg: v5r17p4
dataMonitoring/FastMonCfg: v2r0p0
dataMonitoring/DigiReconCalMeritCfg: v1r4p1
dataMonitoring/Common: v6r0p0
dataMonitoring/FastMon: v5r0p0
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r2p37
svac/EngineeringModelRoot: v4r4
svac/TestReport: v9r2
users/richard/pipelineDatasets: v0r6
ft2Util: v1r2p31*
evtClassDefs v0r14*
GPLtools: fileOps1
Reason for change
The database tables have been modified to allow users to associate:
- LAT catalog sources to ASP point sources (in order to fetch light curve data)
- ASP point sources with other ASP point sources (the same object has been named differently from different groups)
- LAT catalog sources with other LAT catalog sources (when a new catalog is released the names change)
The changes to the tables AND the web applications are backward compatible and transparent to the user.
Test Procedure
This version has been tested on the DEV server.
Rollback procedure
Version 3.3 could be reinstalled
CCB Jira
Reason for change
The new version (v5r18p3, as opposed to v5r18p2) includes some minor limit changes to protect a few alarms when the statistics is not enough.
Test Procedure
We have processed monitoring products from real on-orbit data (LPA) locally with this version of AlarmsCfg.
Rollback procedure
The package can be rolled back to the previous version by flipping a soft link. Also note that the package is completely independent from any other package running in the pipeline and will not cause a version change of L1Proc.
CCB Jira
Details (release notes for dataMonitoring/AlarmCfg for v5r18p3)
v5r18p3
A few more alarms protected against low statistics.
- Recon eor:
- ReconAcdPhaMipsCorrectedAngle_PMTA_TH1_AcdTile_* and ReconAcdPhaMipsCorrectedAngle_PMTB_TH1_AcdTile_*: condition on min entries changed from 10 to 35.
- Digi eor:
- CondArrCNOTrigger_TH1, algorithms x_average, x_rms, low_high_ratio, y_values and reference_histogram: condition on min entries changed from 10000 to 15000.
- FastMon eor:
- CondArrCNOTrigger_TH1, algorithms x_average, x_rms, low_high_ratio, y_values and reference_histogram: condition on min entries changed from 10000 to 15000.
Reason for change
The new version (v5r18p2, as opposed to v5r17p4) includes some minor limit changes to protect a few alarms when the statistics is not enough.
Test Procedure
We have processed monitoring products from real on-orbit data (LPA) locally with this version of AlarmsCfg.
Rollback procedure
The package can be rolled back to the previous version by flipping a soft link. Also note that the package is completely independent from any other package running in the pipeline and will not cause a version change of L1Proc.
CCB Jira
Details (release notes for dataMonitoring/AlarmCfg for v5r18p2)
v5r18p2
A few more alarms protected against low statistics.
- Digi eor:
- CondArrCNO_CalLoOpensWindow_TH1: added condition on min entries (1000) for algorithms x_average, x_rms and low_high_ratio, reference_histogram.
- CondArrTKR_CalLoOpensWindow_TH1: added condition on min entries (20000) for reference histogram.
- FastMon eor:
- CondArrCNO_CalLoOpensWindow_TH1: added condition on min entries (1000) for algorithms x_average, x_rms and low_high_ratio, reference_histogram.
- CondArrTKR_CalLoOpensWindow_TH1: added condition on min entries (20000) forreference histogram.
v5r18p1
A few more alarms protected against low statistics.
- Digi eor:
- CalX_NHit_TH1_Tower_*: min entries 100000 -> 500000 for algorithm x_rms.
- CalX_Total_NHit_TH1: min entries 100000 -> 500000 for algorithm low_high_ratio.
- CondArrCalHiTrigger_TH1: added condition on min entries (60000) for algorithms x_average, x_rms and low_high_ratio.
- CondArrTKR_CalLoOpensWindow_TH1: added/changed condition on min entries (20000) for algorithms x_average, x_rms and low_high_ratio.
- TkrHits_TH1_Tower_*: added condition on min entries (500000) for algorithm x_average.
- FastMon eor:
- CalX_NHit_TH1_Tower_*: min entries 100000 -> 500000 for algorithm x_rms.
- CalX_Total_NHit_TH1: min entries 100000 -> 500000 for algorithm low_high_ratio.
- CondArrCalHiTrigger_TH1: added condition on min entries (60000)for algorithms x_average, x_rms and low_high_ratio.
- CondArrTKR_CalLoOpensWindow_TH1: added/changed condition on min entries (20000) for algorithms x_average, x_rms and low_high_ratio.
- TkrHits_TH1_Tower_*: added condition on min entries (500000) for algorithm x_average.
v5r18p0
- Upper warning limit changed from 3.0 to 3.5 for CAL channel 2784, calpeds_eor alarms CalXAdcPedPedMeanDeviation_LEX1_TH1 and CalXAdcPedPedMeanDeviation_HEX1_TH1.
- Separate alarm limits for CAL channel 2785 on calpeds_eor alarm CalXAdcPedRMS_HEX1_TH1 (0.6, 0.8 -> 0.9, 1.2)
- Relevant jira(s): GDQMQ-315
Reason for change
This new GR, v15r47p12, contains an improved error message in case of event errors and also removes an incorrect warning, repeated 160 times per event, in case we have data with TEM diagnostics. Most importantly, it fixes a bug with the ldfReader which prevented the TEM diagnostic from being interpreted correctly. We anticipate that taking data with the TEM diagnostics will be the new default running mode when the new FSW B2-1-0 is uploaded and activated. There is no change to any functionality in this GlastRelease.
We have also added a few new monitoring quantities related to event sizes at Eric Siskind's request, plus a new variable that monitors the distance from the SAA.
Finally, we are getting rid of the chunk-based host list (it didn't obviously help but produced a lot of trouble when some configuration changes were applied to LFS).
Test Procedure
We have processed data runs in the DEV pipeline with this version of L1Proc.
Rollback procedure
We can switch back to the previous version of L1Proc.
CCB Jira
Details
L1Pipeline v1r74:
- Getting rid of the chunk-based hostlist.
GlastRelease v15r47p12:
- There are a few changes in this GR wrt the version in use in L1, GR v15r47p7:
- ldfReader v7r4p1: windows patch to use const_iterator versus iterator; patch to set diagnostic and error bits properly by taking OR across contributions and making sure all diagnostic data is stored, even those without associated CAL or TKR data; patch check on diagnostic size
- RootIo v21r12gr1: patch to return kTRUE in GleamMessageHandler::Notify
- LdfConverter: patch to set various levels for EbfDebug
- EventIntegrity v0r8p2: Print out the error flag (JIRA GRINF-51) and event id when errors are found.
- There are no changes in the system tests wrt GR v15r47p7.
svac/TestReport: v9r2
- Completeness job option modified (was inconsistent with L1 config)
dataMonitoring/DigiReconCalMeritCfg: v1r4p1
- Added quantities to address JIRA GDQM-309.
dataMonitoring/FastMonCfg: v2r0p0
- Added a new variable (distance to the SAA) in the baseConfig.xml file (JIRA GDQMQ-311).
dataMonitoring/Common: v6r0p0
- Added -s option to pOptionParser to pass to the xml file with the SAA definition to dataMonitoring/FastMon/pDataProcessor.py (necessary for the plot of the distance to the SAA, as in JIRA GDQMQ-311).
dataMonitoring/FastMon: v5r0p0
- Quite a few files modified to accomodate the plot of the distance to the SAA (JIRA GDQMQ-311). The detailed description of the changes is:
- pDataProcessor.py: one more command line option added (-s) to pass the xml file with the definition of the SAA.
- pGeomagProcessor.py: added filling of the output tree with the distance to the SAA variable.
- pM7Parser.py: new class member (the SAA polygon) added.
- pSAAPolygon.py: EARTH_RADIUS not imported anymore from pSCPosition due to an import clash.
- pSCPosition.py: added a new class member (the SAA polygon) and a new method returning the distance of a generic point from the polygon itself.
Complete set of tags for L1Proc 1.74
Code Versions
GlastRelease (sim/recon): v15r47p12*
ScienceTools (Level 2) : v9r8p2
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r74*
calibTkrUtil v2r7p3
calibGenTKR v4r5
dataMonitoring/AlarmsCfg: v5r17p4
dataMonitoring/FastMonCfg: v2r0p0*
dataMonitoring/DigiReconCalMeritCfg: v1r4p1*
dataMonitoring/Common: v6r0p0*
dataMonitoring/FastMon: v5r0p0*
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r2p37
svac/EngineeringModelRoot: v4r4
svac/TestReport: v9r2*
users/richard/pipelineDatasets: v0r6
ft2Util: v1r2p23
evtClassDefs v0r6
GPLtools: fileOps1
Reason for change
The new version (v5r17p4, as opposed to v5r16p2) introduces a few minor limit changes and protections against low statistics detailed below.
Test Procedure
We have processed monitoring products from real on-orbit data (LPA) locally with this version of AlarmsCfg.
Rollback procedure
The package can be rolled back to the previous version by flipping a soft link. Also note that the package is completely independent from any other package running in the pipeline and will not cause a version change of L1Proc.
CCB Jira
Details (release notes for dataMonitoring/AlarmCfg for v5r17p4)
v5r17p4
Quite a few alarms protected against low statistics.
- Digi eor:
- AcdVetoMap_TH1 (x_rms): added condition on minimum number of entries (100000).
- Acd_GemVeto_AcdTile_TH1 (x_rms): added condition on minimum number of entries (100000).
- CondArrCNO_CalLoOpensWindow_TH1 (x_rms): added condition on minimum number of entries (100).
- CondArrCNO_TKROpensWindow_TH1 (x_rms): added condition on minimum number of entries (200).
- TkrHits_TH1_Tower_* (x_average): added condition on minimum number of entries (10000).
- CondSummaryWord_TH1 (reference_histogram): added condition on minimum number of entries (10000).
- CondArrCalLo_TKROpensWindow_TH1 (reference_histogram): added condition on minimum number of entries (5000).
- CondArrTKR_CalLoOpensWindow_TH1 (reference_histogram): added condition on minimum number of entries (1000).
- FastMon eor:
- CondArrCalLo_TKROpensWindow_TH1 (reference_histogram): added condition on minimum number of entries (5000).
- CondArrTKR_CalLoOpensWindow_TH1 (reference_histogram): added condition on minimum number of entries (1000).
- CondSummaryWord_TH1 (reference_histogram): added condition on minimum number of entries (10000).
- CondArrCNO_CalLoOpensWindow_TH1 (x_rms): added condition on minimum number of entries (100).
- CondArrCNO_TKROpensWindow_TH1 (x_rms): added condition on minimum number of entries (200).
v5r17p3
- Alarm limits changed for CondArrCNO_CalLoOpensWindow_TH1 in both digi and fastmon eor.
- warning max: 0.0006 -> 0.0008
- error max : 0.0010 -> 0.0015
v5r17p2
- Condition on the minimum number of entries changed from 10 to 15 in CondArrCalHi_CNOOpensWindow_TH1 for the alarms on the average and the rms of the distribution.
v5r17p1
- Warning high limit on CondSummaryWord_TH1 (reference histogram) for both digi end of run and fastmon end of run changed from 100 to 110 (see http://www-glast.stanford.edu/protected/mail/datamon/2892.html)
v5r17p0
- Warning upper limit for OutF_Ratio_EvtSize_CompressedEvtSize in digi_trend changed from 3.2 to 3.25.