The astroserver relies on FILTEREDMERIT files and EXTENDEDFT1/EXTENDEDLS1 files to have an equal amount of events in order to match events between these two files if people request merit output (and people do).
This all gets a little confusing, so here are the definitions of the software used to process data:
1. Level 1 "P120" active from 1 Aug 2011 to 19 Apr 2012
2. Level 1 "P130" active from 19 Apr 2012 to present
3. P120 reprocessing - data from 4 Aug 2008 to 1 Aug 2011
4. P130 reprocessing - data from 4 Aug 2008 to 19 Apr 2012
Level 1 P120 [1] produced EXTENDEDFT1/EXTENDEDLS1 files which had the exact same number of events as the FILTEREDMERIT files.
P130 Reprocessing [4] produced EXTENDEDFT1/EXTENDEDLS1 files which had the exact same number of events as the FILTEREDMERIT up until the August 1st run, 333886531.
After run 333886531, P130 Reprocessing [4] produced EXTENDEDFT1/EXTENDEDLS1 which have more of events than FILTEREDMERIT up until the end of reprocessing, run 356429251.
Level 1 P130 [2] currently produces EXTENDEDFT1/EXTENDEDLS1 files which have the exact same number of events as the FILTEREDMERIT files, starting at run 356434975.
Level 1 P120 [3] EXTENDEDFT1/EXTENDEDLS1 files, between runs 333886531 up to 356429251, has less events than P130 Reprocessing [4] EXTENDEDFT1/EXTENDEDLS1 files in that period.
The extra events in question have the LLE and LLESDVETO bits set, but no other bits as far as I can tell.
There are 5 runs that are nadIrOps, they had their events added to the database in P120, but not P130. EXTENDEDFT1/EXTENDEDLS1 P130 versions of these files do exist:
335815703: nadirOps
336319408: nadirOps
336314402: nadirOps
335811842: nadirOps
335816822: nadirOps
In addition to this, 90 runs were not reprocessed, and subsequently also not loaded into P130 and EXTENDEDFT1/EXTENDEDLS1 P130 version do not exist, but were loaded into P120, and most are nadirOps runs. They are also marked bad.
337765922
338088786
338089442
338853242
338868584 (nomSciOps_diagEna)
339195062
340861622
340519982
340513802
340863902
340850917
340851662
340847286
339200382
339201242
339191042
340857782
340521602
337757702
337756818
337753682
337759802
338103782
339189846
339532046
339541142
339544304
340515422
336888182
336890102
338433422
339197222
339206448
335800806
336325549
336310866
336326402
336312602
336316202
336320402
336880502
338852046
339194217
339538233
339539042
339545222
335802602
335804402
336892022
338100827
338097662
338094663
338095562
338101682
335806202
335809468
337751582
340523762
339203282
339535022
339528906
339532922
336322202
336886262
336894122
336878942
336882422
336884342
340517702
337750446
337762970
338427786
338432507
338429402
338865482
340853282
340849382
340855502
340511582
340513012
340509306
340859462
351707102
351695467
351700922
351703142
351710075
351705122
351696962
351699002
2 Comments
Maria Elena Monzani
1. EXTENDEDFT1/EXTENDEDLS1 files MUST HAVE the exact same number of events as the FILTEREDMERIT files at all times.
In particular, events with the LLE and LLESDVETO bits set, but no other bit set, SHOULD NEVER BE INCLUDED in either EXTENDEDFT1/EXTENDEDLS1 files or FILTEREDMERIT files.
From the look of it, all EXTENDEDFT1/EXTENDEDLS1 files produced by P130 are wrong, because they contain LLE events.
These files need to be remade ASAP and the correct ones need to be reimported in the astroserver.
This should not be a problem for the FSSC, because we never exported EXTENDEDLS1 files from P130 to them.
2. For nadirOps runs, they should be excluded from the astroserver and from the reprocessing.
The reason why they disappeared from P120 is because sometimes they were marked as "bad" after the fact. We need to come up with a better way to handle this.
As of now, we don't mark as "bad" and remake the FT2 for these files. This may not be that big a deal because all their FT1 files are empty (or at least they should be... if that's not the case, then we have a problem).
Tom Glanzman
A few more details.
The P120 reprocessing introduced a new set of (Pass7) event classifications and produced these data products: MERIT, FILTEREDMERIT, ELECTRONMERIT in task P120-MERIT, and FT1, LS1, ELECTRONFT1, EXTENDEDFT1, EXTENDEDLS1 in task P120-FT1.
The P130 reprocessing was commissioned to make but two substantive changes to the P120-FT1 task: update the ScienceTools version (09-26-02) to include the new and improved diffuse response calculation; update the list of runs to include the period 1 Aug 2011 through 18 Apr 2012. This resulted in the P130-FT1 task which produced: FT1, LS1, ELECTRONFT1, EXTENDEDFT1, EXTENDEDLS1.
The discrepancy in the astroserver is due to a couple reasons:
Regarding NadirOps runs, at the time of the reprocessing configuration, there were differing opinions as to whether or not they should be included. This should be clarified.