Overview
This page contains some notes on where to look when you think you are missing SSR dump data.
The SSR dump data travels from the S/C down to the GD PTP and then several stops before runs are populated with analysis in the weblog.
GD PTP --> GD Data Server --> lat-sskiw02 --> MCR --> SLAC --> web eLog
There are many places along this path that copy processes can get stalled. By starting at the web eLog and working backwards many times the stall point can be found.
Where to look at SLAC
The web eLog
The first stop is the web elog, from there you can determine the expected runs and their start and and stop times from the operator entries. http://www.slac.stanford.edu/cgi-wrap/eLog.pl/index
How to see the latest science packets at SLAC
To see a list of the latest packets at SLAC connect to a machine at SLAC which can see the SLAC AFS file system, like glastlnx06.
$ source /u/gl/glastops/isoc_config_prod.csh $ PktLatest.py -b '-5 days' --sci
Here is an example of the output.
glastlnx06 $ PktLatest.py -b '-5 days' --sci
TFIRST | TLAST | GAP | RECEIVED | PACKETS | L0FILE
---------------------------+----------------------------+-----+---------------------+---------+--------------------------------------
2007-02-20 05:31:10.713677 | 2007-02-20 07:23:05.962753 | * | 2007-02-20 16:50:13 | 53479 | GLAST_2007051_164604_VC08_SCI.0.00.gz
2007-02-16 01:52:37.163874 | 2007-02-16 03:46:42.563931 | | 2007-02-16 22:50:13 | 46196 | GLAST_2007047_224614_VC08_SCI.0.00.gz
2007-02-16 01:17:05.424026 | 2007-02-16 01:52:36.803662 | | 2007-02-16 22:50:13 | 16161 | GLAST_2007047_224622_VC08_SCI.0.00.gz
The ISOC utility PktLatest.py will show the timespans of data in level-0 data files received for a user-specified interval. The output will be a table of timespans, packet counts, and filenames sorted in reverse chronological order by packet-data start time. There is a "gap" indication in the table which shows where the starting time of data in one file does not overlap with the ending time of data in the previous file.
How to see the dispatcher log at SLAC
To see the dispatcher log at SLAC connect to a machine which can see the SLAC AFS file system like glastlnx06.
tail /nfs/farm/g/glast/u23/ISOC/glastops/dispatchRunData.log
Here is an example of the output.
----------------------------- Wed 21 Feb 2007 00:40:03 UTC ------------------------------ MakeRetDef: reading SVAC dispatch configuration from /afs/slac/g/glast/ground/PipelineConfig/EM-tasks/configDispatch MakeRetDef: searching elogbook for new runs MakeRetDef: no new runs found ----------------------------- Wed 21 Feb 2007 00:45:01 UTC ------------------------------ MakeRetDef: reading SVAC dispatch configuration from /afs/slac/g/glast/ground/PipelineConfig/EM-tasks/configDispatch MakeRetDef: searching elogbook for new runs MakeRetDef: no new runs found
How to see the pipeline eventfile log
To see the pipline eventfile log check this webpage: http://glast-ground.slac.stanford.edu/Pipeline/process.jsp?d-49653-o=2&maxDate=None&submit=Filter&d-49653-p=1&min=&d-49653-s=0&process=2808&status=5&task=2395&max=&minDate=None
Look for the runid you are missing and see if the status reports failed
Here is an example of the information on that page:
Run Status Submitted Memory (MB) CPU (secs) Job Id Links (?) 77014191 Failure 2007-02-22 09:17 32 131 815744 Log : Files : Out : Err 77014192 Failure 2007-02-22 09:18 34 881 815746 Log : Files : Out : Err 77014194 Failure 2007-02-22 09:19 34 2794 815762 Log : Files : Out : Err
Where to look on the MCR
How to see the log of SSR dump files copied from GD (MCR SIDE)
To see the log of SSR dump files copied from the GD network, connect to a machine on the MCR like lat-ws01 or lat-dmz01:
cd /nfs/data/LICOS/log tail fcopy_vcdu_VCID8.log
Here is an example of the output.
2007-02-16 22:43:58.584922 found 28777 packets (2007-02-14 20:59:24.701242-->2007-02-14 22:04:56.363883) in /mnt/sass/VCID8/VCDU_Archive8_2007_0214_2229.bin 2007-02-16 22:44:04.283995 found 28777 packets (2007-02-14 20:59:24.701242-->2007-02-14 22:04:56.363883) in /mnt/sass/VCID8/VCDU_Archive8_LAT_POWER_UP_2007_0214_2229.bin 2007-02-16 22:44:08.927584 found 16075 packets (2007-02-14 12:22:53.163827-->2007-02-14 13:19:33.203626) in /mnt/sass/VCID8/VCDU_Archive8_SCD2_02142007_1336.bin 2007-02-16 22:44:11.852710 found 15980 packets (2007-02-14 11:47:49.327889-->2007-02-14 12:22:52.803678) in /mnt/sass/VCID8/VCDU_Archive8_SC_02142007_1316.bin 2007-02-16 22:45:14.798719 found 46196 packets (2007-02-16 01:52:37.163874-->2007-02-16 03:46:42.563931) in /mnt/sass/VCID8/VCDU_Archive8_CONFIG3_RESET_2007_0215_0355.bin 2007-02-16 22:45:18.745226 found 16161 packets (2007-02-16 01:17:05.424026-->2007-02-16 01:52:36.803662) in /mnt/sass/VCID8/VCDU_Archive8_LAT_CHAR_2007_0216_0212.bin 2007-02-16 22:45:25.981386 found 67717 packets (2007-02-15 12:24:53.079412-->2007-02-15 15:17:20.802713) in /mnt/sass/VCID8/VCDU_Archive8_SC_02152007_1751.bin 2007-02-16 23:34:06.750862 found 28772 packets (2007-01-19 05:28:43.462892-->2007-01-19 06:22:35.162939) in /mnt/sass/VCID8/VCDU_Archive8_L-OBS-005_01182007_0633_old.bin 2007-02-16 23:41:08.589552 found 42242 packets (2007-01-12 06:34:33.043911-->2007-01-12 08:02:52.562902) in /mnt/sass/VCID8/VCDU_Archive8_L_OBS-001_01122007.bin 2007-02-20 07:29:08.161653 found 53479 packets (2007-02-20 05:31:10.713677-->2007-02-20 07:23:05.962753) in /mnt/sass/VCID8/VCDU_Archive8_L-OBS-001_02202007_0726.bin
Where to look on lat-sskiw02
How to see the log of SSR dump files copied from GD
To see the log of SSR dump files copied from the GD network, connect to machine lat-sskiw02 on the GD I&T network.
cd /gnfs/data/LICOS/log tail fcopy_vcdu_VCID8.log
Here is an example of the output.
2007-02-16 22:43:58.584922 found 28777 packets (2007-02-14 20:59:24.701242-->2007-02-14 22:04:56.363883) in /mnt/sass/VCID8/VCDU_Archive8_2007_0214_2229.bin 2007-02-16 22:44:04.283995 found 28777 packets (2007-02-14 20:59:24.701242-->2007-02-14 22:04:56.363883) in /mnt/sass/VCID8/VCDU_Archive8_LAT_POWER_UP_2007_0214_2229.bin 2007-02-16 22:44:08.927584 found 16075 packets (2007-02-14 12:22:53.163827-->2007-02-14 13:19:33.203626) in /mnt/sass/VCID8/VCDU_Archive8_SCD2_02142007_1336.bin 2007-02-16 22:44:11.852710 found 15980 packets (2007-02-14 11:47:49.327889-->2007-02-14 12:22:52.803678) in /mnt/sass/VCID8/VCDU_Archive8_SC_02142007_1316.bin 2007-02-16 22:45:14.798719 found 46196 packets (2007-02-16 01:52:37.163874-->2007-02-16 03:46:42.563931) in /mnt/sass/VCID8/VCDU_Archive8_CONFIG3_RESET_2007_0215_0355.bin 2007-02-16 22:45:18.745226 found 16161 packets (2007-02-16 01:17:05.424026-->2007-02-16 01:52:36.803662) in /mnt/sass/VCID8/VCDU_Archive8_LAT_CHAR_2007_0216_0212.bin 2007-02-16 22:45:25.981386 found 67717 packets (2007-02-15 12:24:53.079412-->2007-02-15 15:17:20.802713) in /mnt/sass/VCID8/VCDU_Archive8_SC_02152007_1751.bin 2007-02-16 23:34:06.750862 found 28772 packets (2007-01-19 05:28:43.462892-->2007-01-19 06:22:35.162939) in /mnt/sass/VCID8/VCDU_Archive8_L-OBS-005_01182007_0633_old.bin 2007-02-16 23:41:08.589552 found 42242 packets (2007-01-12 06:34:33.043911-->2007-01-12 08:02:52.562902) in /mnt/sass/VCID8/VCDU_Archive8_L_OBS-001_01122007.bin 2007-02-20 07:29:08.161653 found 53479 packets (2007-02-20 05:31:10.713677-->2007-02-20 07:23:05.962753) in /mnt/sass/VCID8/VCDU_Archive8_L-OBS-001_02202007_0726.bin
To see the SSR dump files in the GD network folder
To see the log of SSR dump files on the GD network, connect to machine lat-sskiw02 on the GD I&T network.
cd /mnt/sass/VCID8 ls -alrt
To check if the SSR data files were placed in the GD data server
Talk to the GD Operator and have them check if the file from the time you are missing has been copied to the "K" drive on the GD data server. The folder should be called: \data1\VCID8