This document describes the current state of automated PV archiving procedures and scripts for LCLS.
Unless otherwise noted all scripts operate on and connect to the LCLS Archiver Appliance instance (managed directly via http://lcls-archapp.slac.stanford.edu/mgmt/ui/index.html).
lclsResumePausedPVs
Cronjob:
lcls-prod01 00 * * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/lclsResumePausedPVs.sh
Log:
/nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsResumePausedPVs/lclsResumePausedPVs.log
Schedule: Every hour
Description:
Gets a list of PVs that are paused and then resumes the archive request for each active PV.
Optionally, we can pass in a folder with a list of .archive files.
Only those paused PV's which are present in archive files that have been modified recently will be checked.
Example report:
Sent to: controls-cd, ppascual, bhill
2020-07-08 13:00:06,309 - Starting resumePausedPVs...
2020-07-08 13:00:06,309 - 2020-07-08 13:00:06.309367
2020-07-08 13:00:06,309 - Logfile: /nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsResumePausedPVs/lclsResumePausedPVs.log
2020-07-08 13:00:06,313 - Starting new HTTP connection (1): lcls-archapp:80
2020-07-08 13:00:56,262 - http://lcls-archapp:80 "GET /mgmt/bpl/getPausedPVsReport HTTP/1.1" 200 None
2020-07-08 13:00:56,487 - 89727 total PVs are paused
2020-07-08 13:00:56,643 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-bsy0-mp01/archive/MPS_LinkProcessor.archive modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-feh1-pp01/archive/ioc-feh1-pp01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-ev01/archive/beam_code_rates.archive modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-mc01/archive/ioc-in20-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-pp01/archive/ioc-in20-pp01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-li21-mc01/archive/ioc-li21-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li24-mc01/archive/ioc-li24-mc01.archive modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-li29-mc01/archive/ioc-li29-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-li30-mc01/archive/ioc-li30-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mc01/archive/ioc-ltuh-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mc02/archive/ioc-ltuh-mc02.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mc03/archive/ioc-ltuh-mc03.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mg01/archive/ioc-ltuh-mg01.archive modified within 60 days
2020-07-08 13:00:56,645 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mg02/archive/ioc-ltuh-mg02.archive modified within 60 days
[...]
2020-07-08 13:01:14,937 - 397 total PVs have been paused in the past 60 days:
IOC:SYS0:FB02:TODSUM
SIOC:GUNB:PM02:UPTIME
SIOC:LI26:BP01:TODSUM
ACSW:UNDH:NW01:6POWERSTATE
SIOC:LTUH:MG01:TODSUM
CRAT:UNDH:UC19:CHV5_V
VIOC:LI21:MG01:TODDIFF
VIOC:LI21:MG03:FD_FREE
SIOC:LTUH:MG01:HEARTBEATOK
CRAT:LI24:BC01:FANTRAY
SIOC:GUNB:TM02:START_CNT
IOC:SYS0:FB01:CA_CONN_CNT
IOC:SYS0:FB04:TODDIFF
IOC:LI21:MG02:UPTIME
SIOC:LTUH:WA01:TODDIFF
[...]
2020-07-08 13:01:35,018 - No live PVs in this batch.
2020-07-08 13:01:35,018 - Done.
2020-07-08 13:01:35,018 - Total run time: 88.71944975852966
Notes:
lclsProcessArchiveFiles
Cronjob:
lcls-prod01 00 1 * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/lclsProcessArchiveFiles.sh
Log:
/nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsProcessArchiveFiles/lclsProcessArchiveFiles.log
Schedule: Every hour
Description:
This is one of the steps used in automation of archiver configuration.
This assumes the requests for the archiver are somehow stored in the IOC (perhaps in INFO fields) and then submitted as archive files.
Archive files are consolidated together into a IOC_DATA like folder and can be identified using a bash glob expression.
Archive files are space/tab separated text files with these rules
1) They can have blank lines
2) They can have comment lines beginning with a #
3) Space/tab separated lines are interpreted as PVName, sampling period (in secs), sampling method (one of scan/monitor).
4) The PVName is mandatory; the other two are optional in which case defaults (optional arguments to this script) are applied.
Example report:
Sent to: controls-cd, ppascual, bhill
2020-07-08 13:00:06,230 - Starting processArchiveFiles...
2020-07-08 13:00:06,230 - 2020-07-08 13:00:06.230638
2020-07-08 13:00:06,230 - Logfile: /nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsProcessArchiveFiles/lclsProcessArchiveFiles.log
2020-07-08 13:00:06,230 - Searching for archive files: /mccfs2/u1/lcls/epics/ioc/data*/archive/*.archive
2020-07-08 13:00:06,944 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-bsy0-mp01/archive/MPS_LinkProcessor.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-feh1-pp01/archive/ioc-feh1-pp01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-ev01/archive/beam_code_rates.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-mc01/archive/ioc-in20-mc01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-pp01/archive/ioc-in20-pp01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li21-mc01/archive/ioc-li21-mc01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li24-mc01/archive/ioc-li24-mc01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li29-mc01/archive/ioc-li29-mc01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li30-mc01/archive/ioc-li30-mc01.archive modified within 999 days
[...]
2020-07-08 13:03:18,451 - Processing archive file /mccfs2/u1/lcls/epics/ioc/data/sioc-gunb-va02/archive/sioc-gunb-va02.archive
2020-07-08 13:03:18,452 - All 8 PVs from sioc-gunb-va02/archive/sioc-gunb-va02.archive are in the archiver
2020-07-08 13:03:18,452 - Processing archive file /mccfs2/u1/lcls/epics/ioc/data/sioc-gunb-va03/archive/sioc-gunb-va03.archive
2020-07-08 13:03:18,455 - All 42 PVs from sioc-gunb-va03/archive/sioc-gunb-va03.archive are in the archiver
2020-07-08 13:03:18,456 - Processing archive file /mccfs2/u1/lcls/epics/ioc/data/sioc-in20-mg01/archive/sioc-in20-mg01.archive
2020-07-08 13:03:18,532 - Processing archive file /mccfs2/u1/lcls/epics/ioc/data/sioc-in20-mg02/archive/sioc-in20-mg02.archive
2020-07-08 13:03:18,621 - Checking for liveness of 1013 PVs
2020-07-08 13:03:23,717 - Submitting 2 new PVs to the archiver
2020-07-08 13:03:23,717 - Submitting these PVs to the archiver:
SIOC:GUNB:MC01:APP_DIR
SIOC:GUNB:MC01:ST_SCRIPT
2020-07-08 13:03:23,721 - Starting new HTTP connection (1): lcls-archapp:80
2020-07-08 13:03:23,831 - http://lcls-archapp:80 "POST /mgmt/bpl/archivePV HTTP/1.1" 200 None
2020-07-08 13:03:23,832 - archivePVs returned with status 200
[...]
2020-07-08 13:03:32,410 - Checking for liveness of 1335 PVs
2020-07-08 13:03:37,523 - Skipped 1335 potentially stale PVs:
PPS:SYSE:1:CLTS:HRTBT_OK_LTH_A
PPS:SYSE:1:CLTS:PAUSEHB_A
PLC:BSY0:PPxx:FAULTLED
PPS:SYSE:1:CLTS:HRTBT_OUTP_A
PPS:SYSE:1:CLTS:HRTBT_INP_B
PLC:BSY0:PPxx:IFAULTLED
PLC:BSY0:PPxx:PWLED
PLC:BSY0:PPxx:RUNLED
PLC:BSY0:PPxx:DIAGLED
PPS:SYSE:1:CLTS:HRTBT_OK_LTH_B
PLC:BSY0:PPxx:CHECKSUM
PPS:SYSE:1:CLTS:HRTBT_INP_A
PLC:BSY0:PPxx:OFAULTLED
PPS:SYSE:1:CLTS:HRTBT_OUTP_B
[...]
2020-07-08 13:04:40,949 - Total run time: 274.7326340675354
2020-07-08 13:04:40,950 - Done.
Notes:
lclsPauseDisconnectedPVs
Cronjob:
lcls-prod01 00 2 * * 3,4 /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/lclsPauseDisconnectPVs.sh
Log:
/tmp/lclsPauseDisconnectedPVs.log
Schedule: Every Wednesday and Thursday at 02:00
Description:
Gets a list of PVs that are disconnected and then pauses those that have not connected for a specified amount of time.
Example report:
Sent to: controls-cd, ppascual, bhill
2020-02-12 02:00:06,347 - _main_ - INFO - 9149 PVs are disconnected
2020-02-12 02:00:06,371 - _main_ - INFO - 303 PVs have been disconnected for more than 86400 minutes
2020-02-12 02:00:11,410 - _main_ - INFO - Detected 303 disconnected PVs
Notes:
lclsResumeAllPausedPVs
Cronjob:
lcls-prod01 00 3 * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/lclsResumeAllPausedPVs.sh
Log:
/tmp/lclsResumeAllPausedPVs.log
Schedule: Daily at 03:00
Description:
This script gets a list of PVS that are paused and then resumes the archive request for each active PV.
Optionally, we can pass in a folder with a list of .archive files.
Only those paused PV's which are present in archive files that have been modified recently will be checked
Example report:
Sent to: controls-cd, ppascual, bhill
2020-03-11 03:00:48,430 - _main_ - INFO - 98958 PVs are paused
2020-03-11 06:27:12,587 - _main_ - INFO - Resuming 1 live PVs
2020-03-11 06:27:14,869 - _main_ - INFO - Resuming 3 live PVs
2020-03-11 06:34:17,535 - _main_ - INFO - Resuming 2 live PVs
2020-03-11 06:34:20,094 - _main_ - INFO - Resuming 4 live PVs
2020-03-11 06:34:22,429 - _main_ - INFO - Resuming 4 live PVs
2020-03-11 06:34:24,508 - _main_ - INFO - Resuming 4 live PVs
2020-03-11 06:34:27,125 - _main_ - INFO - Resuming 2 live PVs
2020-03-11 08:31:11,755 - _main_ - INFO - Done.
Notes:
lclsSetupEnv
Notes: