...
- Define the name of the columns
- Update File Format Document & Interface Control Document.
- Implement the new columns in fitsGen, which means adding them to the template FT2 file: https://github.com/fermi-lat/fitsGen/blob/master/data/ft2.tpl
- Change the ft2 code. to get the velocities from the magic7 and save them in the new columns
- Create pipeline tasks to reprocess the FT2 data
- Create pipeline tasks to properly apply the Bad Time interval data.
- Test everything end-to-end with a sample of the new ft2 files. (by this I mean the pulsar software)
- Use the code to reprocess 1-2 weeks of data for testing.
- we reprocessed a two-week chunk of data from runs 545208791 to 546407418. This included several files with BTI data in the database
- more files were reprocessed with the P310 task(s). This includes a variety of cases for BTIs and a selection of different DATA_QUALs.
- Check to make sure the velocities are being copied correctly from the magic7 packets and that they make sense based on the change in coordinates - Don/Tom.
- Ingest files at FSSC - Don/Alex
- Verify that the BTI data is correct - ME/Tom/Simone
- Test with Pulsar Tools - Dave Smith
- Non-pulsar tests - Joe Eggen.
- Check with the C& A group. Don brought it up at the 6/24 C&A group meeting. No one raised any issues (or volunteered to do any testing).
- Check with other mission elements. Don checked with Michelle Hui at the GIOC that the change won't affect them.
- Paul Ray checked the files against the velocities computed by PINT and found good agreement.
- Use the code to reprocess 1-2 weeks of data for testing.
- Fix all the runs that need repiping (Runs to be rePiped and reprocessed): still in progress.
- Obtain a new GlastRelease:
- JIRA issue for the new GR:
Jira server SLAC National Accelerator Laboratory serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546 key LPATE-198 - We also need to update the spacecraft's geoposition (
). A new version of the astro package is included in the new GR.Jira server SLAC National Accelerator Laboratory serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546 key GRINF-76
- JIRA issue for the new GR:
- Remake the P310 tasks
- Re-reprocess a selection of files
- Verify that the geo lattitude and longitude are properly created (Toby will do this).
- Repeat the checks in item 5. above (spot-checks are probably fine).
- Verify that FT1 files produced with the new GR are correct/unchanged from the previous version
- Reprocess the FT2 files (Usually done in two steps. Initial reprocess & backfill):
- Pick an end date/time for the reprocessing window, once we all agree that we are ready to go.
- Create a list of runs for reprocessing and BTI flagging from the datacatalog. Validate against the list at the FSSC server.
- Install the reprocessing tasks in the PROD pipeline.
- The reprocessing step will be putting them into /Data/Flight/Reprocess/P310.
- FSSC will pick them up from xrootd for ingestion directly.
- The reprocessing will take ~1 month. Started 2020-03-04.
- Transfer the files to the FSSC and ingest them into the FSSC's data server.
- Have Wilko setup a proxy server so Don can copy the files directly from xroot.
- Don copies files and does basic checks and archives them.
- Alex starts the process to ingest them into the data server database. He estimates the ingest may take ~1 week.
- Create a new version of the L1Proc, rePipeL1 and Flag FT2 tasks.
- Update the International Geomagnetic Reference Field (IGRF). See Updating the IGRF-13 implementation in the astro package.
- Test them meaningfully on DEV.
- including check that the FT1 files haven't changed.(see Checking FT1 files in the context of the FT2 reprocessing and Bad interpolation of FT2 information during processing).
- Get CCB approval to make the change operational.
- Coordinate a switchover date between ISOC and FSSC.
- Create the backfill
- Reprocess 2020 runs with the new IGRF model.
- Switch to the new L1 task(s). Note: backfill and switch must be coordinated and timed together.
- Switch the data server to use the new files. The FSSC has an internal wiki page on the steps for this.
- Create the backfill
...