This document describes the current state of automated PV archiving procedures and scripts for LCLS.
Unless otherwise noted all scripts operate on and connect to the LCLS Archiver Appliance instance (managed directly via http://lcls-archapp.slac.stanford.edu/mgmt/ui/index.html).
lclsResumePausedPVs
Cronjob:
lcls-prod01 00 * * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/lclsResumePausedPVs.sh
Log:
/nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsResumePausedPVs/lclsResumePausedPVs.log
Schedule: Every hour
Description:
Gets a list of PVs that are paused and then resumes the archive request for each active PV.
Optionally, we can pass in a folder with a list of .archive files.
Only those paused PV's which are present in archive files that have been modified recently will be checked.
Example report:
Sent to: controls-software-reports@slac.stanford.edu, ppascual@slac.stanford.edu, bhill@slac.stanford.edu
2020-07-08 13:00:06,309 - Starting resumePausedPVs...
2020-07-08 13:00:06,309 - 2020-07-08 13:00:06.309367
2020-07-08 13:00:06,309 - Logfile: /nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsResumePausedPVs/lclsResumePausedPVs.log
2020-07-08 13:00:06,313 - Starting new HTTP connection (1): lcls-archapp:80
2020-07-08 13:00:56,262 - http://lcls-archapp:80 "GET /mgmt/bpl/getPausedPVsReport HTTP/1.1" 200 None
2020-07-08 13:00:56,487 - 89727 total PVs are paused
2020-07-08 13:00:56,643 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-bsy0-mp01/archive/MPS_LinkProcessor.archive modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-feh1-pp01/archive/ioc-feh1-pp01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-ev01/archive/beam_code_rates.archive modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-mc01/archive/ioc-in20-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-pp01/archive/ioc-in20-pp01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-li21-mc01/archive/ioc-li21-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li24-mc01/archive/ioc-li24-mc01.archive modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-li29-mc01/archive/ioc-li29-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-li30-mc01/archive/ioc-li30-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mc01/archive/ioc-ltuh-mc01.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mc02/archive/ioc-ltuh-mc02.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Ignoring file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mc03/archive/ioc-ltuh-mc03.archive that has not been modified within 60 days
2020-07-08 13:00:56,644 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mg01/archive/ioc-ltuh-mg01.archive modified within 60 days
2020-07-08 13:00:56,645 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-ltuh-mg02/archive/ioc-ltuh-mg02.archive modified within 60 days
[...]
2020-07-08 13:01:14,937 - 397 total PVs have been paused in the past 60 days:
IOC:SYS0:FB02:TODSUM
SIOC:GUNB:PM02:UPTIME
SIOC:LI26:BP01:TODSUM
ACSW:UNDH:NW01:6POWERSTATE
SIOC:LTUH:MG01:TODSUM
CRAT:UNDH:UC19:CHV5_V
VIOC:LI21:MG01:TODDIFF
VIOC:LI21:MG03:FD_FREE
SIOC:LTUH:MG01:HEARTBEATOK
CRAT:LI24:BC01:FANTRAY
SIOC:GUNB:TM02:START_CNT
IOC:SYS0:FB01:CA_CONN_CNT
IOC:SYS0:FB04:TODDIFF
IOC:LI21:MG02:UPTIME
SIOC:LTUH:WA01:TODDIFF
[...]
2020-07-08 13:01:35,018 - No live PVs in this batch.
2020-07-08 13:01:35,018 - Done.
2020-07-08 13:01:35,018 - Total run time: 88.71944975852966
Notes:
Script set to check PVs in $IOC_DATA/<IOC>/archive/*.archive files modified within the past 60 days
Processes 100 PVs at a time
lclsProcessArchiveFiles
Cronjob:
lcls-prod01 00 * * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/lclsProcessArchiveFiles.sh
Log:
/nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsProcessArchiveFiles/lclsProcessArchiveFiles.log
Schedule: Every hour
Description:
This is one of the steps used in automation of archiver configuration.
This assumes the requests for the archiver are somehow stored in the IOC (perhaps in INFO fields) and then submitted as archive files.
Archive files are consolidated together into a IOC_DATA like folder and can be identified using a bash glob expression.
Archive files are space/tab separated text files with these rules
1) They can have blank lines
2) They can have comment lines beginning with a #
3) Space/tab separated lines are interpreted as PVName, sampling period (in secs), sampling method (one of scan/monitor).
4) The PVName is mandatory; the other two are optional in which case defaults (optional arguments to this script) are applied.
Example report:
Sent to: controls-software-reports@slac.stanford.edu, ppascual@slac.stanford.edu, bhill@slac.stanford.edu
2020-07-08 13:00:06,230 - Starting processArchiveFiles...
2020-07-08 13:00:06,230 - 2020-07-08 13:00:06.230638
2020-07-08 13:00:06,230 - Logfile: /nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsProcessArchiveFiles/lclsProcessArchiveFiles.log
2020-07-08 13:00:06,230 - Searching for archive files: /mccfs2/u1/lcls/epics/ioc/data*/archive/*.archive
2020-07-08 13:00:06,944 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-bsy0-mp01/archive/MPS_LinkProcessor.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-feh1-pp01/archive/ioc-feh1-pp01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-ev01/archive/beam_code_rates.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-mc01/archive/ioc-in20-mc01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-in20-pp01/archive/ioc-in20-pp01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li21-mc01/archive/ioc-li21-mc01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li24-mc01/archive/ioc-li24-mc01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li29-mc01/archive/ioc-li29-mc01.archive modified within 999 days
2020-07-08 13:00:06,945 - Processing file /mccfs2/u1/lcls/epics/ioc/data/ioc-li30-mc01/archive/ioc-li30-mc01.archive modified within 999 days
[...]
2020-07-08 13:03:18,451 - Processing archive file /mccfs2/u1/lcls/epics/ioc/data/sioc-gunb-va02/archive/sioc-gunb-va02.archive
2020-07-08 13:03:18,452 - All 8 PVs from sioc-gunb-va02/archive/sioc-gunb-va02.archive are in the archiver
2020-07-08 13:03:18,452 - Processing archive file /mccfs2/u1/lcls/epics/ioc/data/sioc-gunb-va03/archive/sioc-gunb-va03.archive
2020-07-08 13:03:18,455 - All 42 PVs from sioc-gunb-va03/archive/sioc-gunb-va03.archive are in the archiver
2020-07-08 13:03:18,456 - Processing archive file /mccfs2/u1/lcls/epics/ioc/data/sioc-in20-mg01/archive/sioc-in20-mg01.archive
2020-07-08 13:03:18,532 - Processing archive file /mccfs2/u1/lcls/epics/ioc/data/sioc-in20-mg02/archive/sioc-in20-mg02.archive
2020-07-08 13:03:18,621 - Checking for liveness of 1013 PVs
2020-07-08 13:03:23,717 - Submitting 2 new PVs to the archiver
2020-07-08 13:03:23,717 - Submitting these PVs to the archiver:
SIOC:GUNB:MC01:APP_DIR
SIOC:GUNB:MC01:ST_SCRIPT
2020-07-08 13:03:23,721 - Starting new HTTP connection (1): lcls-archapp:80
2020-07-08 13:03:23,831 - http://lcls-archapp:80 "POST /mgmt/bpl/archivePV HTTP/1.1" 200 None
2020-07-08 13:03:23,832 - archivePVs returned with status 200
[...]
2020-07-08 13:03:32,410 - Checking for liveness of 1335 PVs
2020-07-08 13:03:37,523 - Skipped 1335 potentially stale PVs:
PPS:SYSE:1:CLTS:HRTBT_OK_LTH_A
PPS:SYSE:1:CLTS:PAUSEHB_A
PLC:BSY0:PPxx:FAULTLED
PPS:SYSE:1:CLTS:HRTBT_OUTP_A
PPS:SYSE:1:CLTS:HRTBT_INP_B
PLC:BSY0:PPxx:IFAULTLED
PLC:BSY0:PPxx:PWLED
PLC:BSY0:PPxx:RUNLED
PLC:BSY0:PPxx:DIAGLED
PPS:SYSE:1:CLTS:HRTBT_OK_LTH_B
PLC:BSY0:PPxx:CHECKSUM
PPS:SYSE:1:CLTS:HRTBT_INP_A
PLC:BSY0:PPxx:OFAULTLED
PPS:SYSE:1:CLTS:HRTBT_OUTP_B
[...]
2020-07-08 13:04:40,949 - Total run time: 274.7326340675354
2020-07-08 13:04:40,950 - Done.
Notes:
Archives PVs in $IOC_DATA/<IOC>/archive/*.archive files modified within the past 999 days
Defaults archive parameters for PVs to 1.0s, MONITOR
lclsPauseDisconnectedPVs
Cronjob:
lcls-prod01 00 00 * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/lclsPauseDisconnectedPVs.sh
Log:
/nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsPauseDisconnectedPVs/lclsPauseDisconnectedPVs.log
Schedule: Daily, at 00:00
Description:
Gets a list of PVs that are disconnected and then pauses those that have not connected for a specified amount of time.
Example report:
Sent to: controls-software-reports@slac.stanford.edu, ppascual@slac.stanford.edu, bhill@slac.stanford.edu
2020-07-08 00:00:08,032 - Starting pauseDisconnectedPVs...
2020-07-08 00:00:08,033 - 2020-07-08 00:00:08.033155
2020-07-08 00:00:08,033 - Logfile: /nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsPauseDisconnectedPVs/lclsPauseDisconnectedPVs.log
2020-07-08 00:00:08,044 - Starting new HTTP connection (1): lcls-archapp:80
2020-07-08 00:00:08,541 - http://lcls-archapp:80 "GET /mgmt/bpl/getCurrentlyDisconnectedPVs HTTP/1.1" 200 None
2020-07-08 00:00:08,587 - 5486 disconnected PVs found
2020-07-08 00:00:08,600 - No PVs detected that have been disconnected for more than 86400 minute(s)
Notes:
Pauses PVs that have not connected for the past 24 hours
lclsResumeAllPausedPVs
Cronjob:
lcls-prod01 00 03 * * 06 /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/lclsResumeAllPausedPVs.sh
Log:
/nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsResumeAllPausedPVs/lclsResumeAllPausedPVs.log
Schedule: Sundays at 03:00
Description:
This script gets a list of PVS that are paused and then resumes the archive request for each active PV.
Optionally, we can pass in a folder with a list of .archive files.
Only those paused PV's which are present in archive files that have been modified recently will be checked
Example report:
Sent to: controls-software-reports@slac.stanford.edu, ppascual@slac.stanford.edu, bhill@slac.stanford.edu
2020-07-04 03:00:10,074 - Starting resumePausedPVs...
2020-07-04 03:00:10,074 - 2020-07-04 03:00:10.074341
2020-07-04 03:00:10,074 - Logfile: /nfs/slac/g/lcls/tools/archiver_appliance_automation/lclsResumeAllPausedPVs/lclsResumeAllPausedPVs.log
2020-07-04 03:00:10,079 - Starting new HTTP connection (1): lcls-archapp:80
2020-07-04 03:00:48,310 - http://lcls-archapp:80 "GET /mgmt/bpl/getPausedPVsReport HTTP/1.1" 200 None
2020-07-04 03:00:48,568 - 89736 total PVs are paused
2020-07-04 03:00:48,576 - Checking for liveness of 10 PVs:
ACSW:DMP1:NW01:3POWERSTATE
ADS:UND1:100:RFBPM_LTC_ROLL
ADS:UND1:100:RFBPM_LTC_YPOS
ADS:UND1:100:RFBPM_LTQ_YPOS
ADS:UND1:100:RFBPM_REF_ROLL
ADS:UND1:100:RFBPM_REF_XPOS
ADS:UND1:100:RFBPM_RFQ_XPOS
ADS:UND1:100:RFBPM_RFQ_YPOS
ADS:UND1:100:RFBPM_STC_ROLL
ADS:UND1:100:RFBPM_STC_YPOS
2020-07-04 03:00:50,590 - No live PVs in this batch.
2020-07-04 03:00:50,590 - Checking for liveness of 10 PVs:
ADS:UND1:100:RFBPM_STQ_YPOS
ADS:UND1:1010:BFW_LTC_XPOS
ADS:UND1:1010:BFW_LTQ_YPOS
[...]
2020-07-04 03:53:21,694 - Resuming 9 live PVs:
OTRS:DMPH:695:DROI:Dim4SA
OTRS:DMPH:695:DROI:Dimensions_RBV
OTRS:DMPH:695:DROI:PluginType_RBV
OTRS:DMPH:695:DROI:QueueSize
OTRS:DMPH:695:DROI:SetYHOPR
OTRS:DMPH:695:DROI:SigmaXY
OTRS:DMPH:695:DROI:SortFreeLow
OTRS:DMPH:695:Dim1SA
OTRS:DMPH:695:Dim3SA
2020-07-04 03:53:21,698 - Starting new HTTP connection (1): lcls-archapp:80
2020-07-04 03:53:21,870 - http://lcls-archapp:80 "POST /mgmt/bpl/resumeArchivingPV HTTP/1.1" 200 1862
Notes:
Attempts to resume all paused PVs in the LCLS Archiver Appliance instance
Waits 2 seconds for PVs to connect
Processes 10 PVs at a time
lclsSetupEnv
Notes:
Runs from custom Python3 conda environment
Uses EPICS 3.15.5-1.0 environment
Sets IOC_DATA to LCLS production directory (/u1/lcls/epics/ioc/data)