Why this matters:

The astroserver relies on FILTEREDMERIT files and EXTENDEDFT1/EXTENDEDLS1 files to have an equal amount of events in order to match events between these two files if people request merit output (and people do).

Runs with "extra" events, where the EXTENDEDFT1 and EXTENDEDLS1 files are supersets of FILTEREDMERIT:

This all gets a little confusing, so here are the definitions of the software used to process data:

1.  Level 1 "P120" active from 1 Aug 2011 to 19 Apr 2012

2.  Level 1 "P130" active from 19 Apr 2012 to present

3.  P120 reprocessing - data from 4 Aug 2008 to 1 Aug 2011

4.  P130 reprocessing - data from 4 Aug 2008 to 19 Apr 2012

So:

Level 1 P120 [1] produced EXTENDEDFT1/EXTENDEDLS1 files which had the exact same number of events as the FILTEREDMERIT files. 

P130 Reprocessing [4] produced EXTENDEDFT1/EXTENDEDLS1 files which had the exact same number of events as the FILTEREDMERIT up until the August 1st run, 333886531.

After run 333886531, P130 Reprocessing [4] produced EXTENDEDFT1/EXTENDEDLS1 which have more of events than FILTEREDMERIT up until the end of reprocessing, run 356429251.

Level 1 P130 [2] currently produces EXTENDEDFT1/EXTENDEDLS1 files which have the exact same number of events as the FILTEREDMERIT files, starting at run 356434975.

Level 1 P120 [3] EXTENDEDFT1/EXTENDEDLS1 files, between runs 333886531 up to 356429251, has less events than P130 Reprocessing [4] EXTENDEDFT1/EXTENDEDLS1 files in that period.

The extra events in question have the LLE and LLESDVETO bits set, but no other bits as far as I can tell.

Summary table of the selections criteria for the different time periods involved:

All other changes:

There are 5 runs that are nadIrOps, they had their events added to the database in P120, but not P130.  EXTENDEDFT1/EXTENDEDLS1 P130 versions of these files do exist:

335815703: nadirOps

336319408: nadirOps

336314402: nadirOps

335811842: nadirOps

335816822: nadirOps

In addition to this, 90 runs were not reprocessed, and subsequently also not loaded into P130 and EXTENDEDFT1/EXTENDEDLS1 P130 version do not exist, but were loaded into P120, and most are nadirOps runs.  They are also marked bad.

337765922

338088786

338089442

338853242

338868584 (nomSciOps_diagEna)

339195062

340861622

340519982

340513802

340863902

340850917

340851662

340847286

339200382

339201242

339191042

340857782

340521602

337757702

337756818

337753682

337759802

338103782

339189846

339532046

339541142

339544304

340515422

336888182

336890102

338433422

339197222

339206448

335800806

336325549

336310866

336326402

336312602

336316202

336320402

336880502

338852046

339194217

339538233

339539042

339545222

335802602

335804402

336892022

338100827

338097662

338094663

338095562

338101682

335806202

335809468

337751582

340523762

339203282

339535022

339528906

339532922

336322202

336886262

336894122

336878942

336882422

336884342

340517702

337750446

337762970

338427786

338432507

338429402

338865482

340853282

340849382

340855502

340511582

340513012

340509306

340859462

351707102

351695467

351700922

351703142

351710075

351705122

351696962

351699002

  • No labels

2 Comments

  1. Maria Elena Monzani

    1. EXTENDEDFT1/EXTENDEDLS1 files MUST HAVE the exact same number of events as the FILTEREDMERIT files at all times.
    In particular, events with the LLE and LLESDVETO bits set, but no other bit set, SHOULD NEVER BE INCLUDED in either EXTENDEDFT1/EXTENDEDLS1 files or FILTEREDMERIT files.

    From the look of it, all EXTENDEDFT1/EXTENDEDLS1 files produced by P130 are wrong, because they contain LLE events.
    These files need to be remade ASAP and the correct ones need to be reimported in the astroserver.
    This should not be a problem for the FSSC, because we never exported EXTENDEDLS1 files from P130 to them.

    2. For nadirOps runs, they should be excluded from the astroserver and from the reprocessing.
    The reason why they disappeared from P120 is because sometimes they were marked as "bad" after the fact. We need to come up with a better way to handle this.

    As of now, we don't mark as "bad" and remake the FT2 for these files. This may not be that big a deal because all their FT1 files are empty (or at least they should be... if that's not the case, then we have a problem).

  2. Tom Glanzman

    A few more details.

    The P120 reprocessing introduced a new set of (Pass7) event classifications and produced these data products: MERIT, FILTEREDMERIT, ELECTRONMERIT in task P120-MERIT, and FT1, LS1, ELECTRONFT1, EXTENDEDFT1, EXTENDEDLS1 in task P120-FT1.

    The P130 reprocessing was commissioned to make but two substantive changes to the P120-FT1 task: update the ScienceTools version (09-26-02) to include the new and improved diffuse response calculation; update the list of runs to include the period 1 Aug 2011 through 18 Apr 2012. This resulted in the P130-FT1 task which produced: FT1, LS1, ELECTRONFT1, EXTENDEDFT1, EXTENDEDLS1.

    The discrepancy in the astroserver is due to a couple reasons:

    1. The creation of new event classifications means that "Filtered" and "Extended" data products must now choose between the traditional "all event classes" (FT1EventClass!=0) or a subset of all events (((FT1EventClass & 0x00003EFF)!=0). P120 used "all events" and P130 inherited that config.
    2. The astroserver has a (arbitrary) requirement that requires event indexes in EXTENDED* and FILTERED* file pairs be in synch. An inconsistency arose because P130 did not remake the FILTEREDMERIT data product and for the period 1 Aug 2011 through 18 Apr 2012. Rather, the FILTEREDMERIT came from L1 which used a more restrictive evt selection.

    Regarding NadirOps runs, at the time of the reprocessing configuration, there were differing opinions as to whether or not they should be included. This should be clarified.