FACET Auto Archiving

This document describes the current state of automated PV archiving procedures and scripts for FACET.

Unless otherwise noted all scripts operate on and connect to the FACET Archiver Appliance instance (managed directly via http://facet-archapp.slac.stanford.edu/mgmt/ui/index.html).

facetResumePausedPVs

Cronjob:

lcls-prod01 00 * * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/facetResumePausedPVs.sh

Log:

/nfs/slac/g/facet/tools/archiver_appliance_automation/facetResumePausedPVs/facetResumePausedPVs.log

Schedule: Every hour

Description:
Gets a list of PVs that are paused and then resumes the archive request for each active PV.
Optionally, we can pass in a folder with a list of .archive files.
Only those paused PV's which are present in archive files that have been modified recently will be checked.

Example report:

Sent to: controls-software-reports@slac.stanford.edu, ppascual@slac.stanford.edu, bhill@slac.stanford.edu

2020-09-28 14:00:06,630 - Starting resumePausedPVs...
2020-09-28 14:00:06,630 - 2020-09-28 14:00:06.630968
2020-09-28 14:00:06,631 - Logfile: /nfs/slac/g/facet/tools/archiver_appliance_automation/facetResumePausedPVs/facetResumePausedPVs.log
2020-09-28 14:00:06,638 - Starting new HTTP connection (1): facet-archapp:17665
2020-09-28 14:00:07,409 - http://facet-archapp:17665 "GET /mgmt/bpl/getPausedPVsReport HTTP/1.1" 200 None
2020-09-28 14:00:07,439 - 6381 total PVs are paused
2020-09-28 14:00:07,475 - Ignoring file /mccfs2/u1/facet/epics/ioc/data/ArchiveBrowser/archive/toBeArchivedList.archive that has not been modified within 60 days
2020-09-28 14:00:07,475 - Ignoring file /mccfs2/u1/facet/epics/ioc/data/eioc-in10-mp01/archive/eioc-in10-mp01.arConfig that has not been modified within 60 days
2020-09-28 14:00:07,475 - Ignoring file /mccfs2/u1/facet/epics/ioc/data/eioc-in10-mp02/archive/eioc-in10-mp02.arConfig that has not been modified within 60 days
2020-09-28 14:00:07,475 - Ignoring file /mccfs2/u1/facet/epics/ioc/data/ioc-in10-mp01/archive/ioc-in10-mp01.arConfig that has not been modified within 60 days
2020-09-28 14:00:07,475 - Ignoring file /mccfs2/u1/facet/epics/ioc/data/ioc-li11-mc01/archive/ioc-in10-mc01.arConfig that has not been modified within 60 days
2020-09-28 14:00:07,475 - Processing file /mccfs2/u1/facet/epics/ioc/data/ioc-lr10-ls02/archive/ioc-lr10-ls02.archive modified within 60 days
2020-09-28 14:00:07,476 - Processing file /mccfs2/u1/facet/epics/ioc/data/ioc-lr10-ls02/archive/ioc-lr10-ls02.archive~ modified within 60 days
2020-09-28 14:00:07,476 - Ignoring file /mccfs2/u1/facet/epics/ioc/data/ioc-sys1-mp01/archive/ioc-sys1-mp01.arConfig that has not been modified within 60 days
[...]
2020-09-28 14:00:07,479 - Ignored 19 total files older than 60 days
2020-09-28 14:00:07,479 - Processing file ioc-lr10-ls02/archive/ioc-lr10-ls02.archive last modified on 2020-09-24 16:09:32.768593
2020-09-28 14:00:07,479 - 1 PVs in recently modified file ioc-lr10-ls02/archive/ioc-lr10-ls02.archive
2020-09-28 14:00:07,479 - Processing file ioc-lr10-ls02/archive/ioc-lr10-ls02.archive~ last modified on 2020-09-24 16:08:10.490690
2020-09-28 14:00:07,480 - 1 PVs in recently modified file ioc-lr10-ls02/archive/ioc-lr10-ls02.archive~
2020-09-28 14:00:07,480 - Processing file sioc-in10-bp01/archive/sioc-in10-bp01.archive last modified on 2020-09-22 10:56:57.022831
2020-09-28 14:00:07,517 - 599 PVs in recently modified file sioc-in10-bp01/archive/sioc-in10-bp01.archive
2020-09-28 14:00:07,517 - Processing file sioc-in10-bp02/archive/sioc-in10-bp02.archive last modified on 2020-09-22 13:17:54.720241
2020-09-28 14:00:07,548 - 771 PVs in recently modified file sioc-in10-bp02/archive/sioc-in10-bp02.archive
[...]
2020-09-28 14:00:08,512 - Processing file sioc-sys1-mg02/archive/sioc-sys1-mg02.archive last modified on 2020-09-18 16:53:27.812340
2020-09-28 14:00:08,599 - 15343 PVs in recently modified file sioc-sys1-mg02/archive/sioc-sys1-mg02.archive
2020-09-28 14:00:08,599 - Processing file sioc-sys1-nw01/archive/sioc-sys1-nw01.archive last modified on 2020-09-22 10:15:54.633856
2020-09-28 14:00:08,713 - 17951 PVs in recently modified file sioc-sys1-nw01/archive/sioc-sys1-nw01.archive
2020-09-28 14:00:08,714 - 221 total PVs have been paused in the past 60 days:
EIOC:IN10:MP01:CA_CLNT_CNT
SIOC:LI20:PM20:UPTIME
EIOC:LI20:RF12:FD_FREE
CRAT:LI10:CV01:FAN3SPEED
IOC:LI20:PM02:CA_CONN_CNT
IOC:LI20:PM12:HEARTBEATECT
SIOC:LI20:PM21:CA_CONN_CNT
CRAT:LI10:CV01:POWERSTATE
[...]
2020-09-28 14:00:18,749 - No live PVs in this batch.
2020-09-28 14:00:18,749 - Checking for liveness of 21 PVs:
IOC:LI20:PM11:HEARTBEATSUM
SIOC:LI20:PM21:UPTIME
CRAT:LI10:CV01:FAN1SPEED
IOC:LI20:MC01:FD_FREE
SIOC:IN10:PM02:UPTIME
EIOC:IN10:MP01:CA_CONN_CNT
IOC:LI20:PM11:CA_CLNT_CNT
IOC:LI20:PM05:FD_FREE
SIOC:LI20:PM23:CA_CLNT_CNT
IOC:LI10:CV01:TODDIFF
IOC:LI20:PM12:TODDIFF
IOC:LI20:PM06:FD_FREE
SIOC:LI10:CV01:CA_CONN_CNT
IOC:LI20:PM02:CA_CLNT_CNT
IOC:LI20:PM12:HEARTBEATSUM
PPS:IN10:1:ACR_IGN_KEYSWITCH_B
SIOC:LI10:KY00:CA_CONN_CNT
EIOC:IN10:MP01:SUSP_TASK_CNT
EIOC:LI20:RF12:CA_CLNT_CNT
CRAT:LI10:CV01:FAN4SPEED
IOC:LI10:CV01:HEARTBEATOK
2020-09-28 14:00:23,756 - No live PVs in this batch.
2020-09-28 14:00:23,756 - Done.
2020-09-28 14:00:23,756 - Total run time: 17.141407012939453

Notes:

facetProcessArchiveFiles

Cronjob:

lcls-prod01 00 * * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/facetProcessArchiveFiles.sh

Log:

/nfs/slac/g/facet/tools/archiver_appliance_automation/facetProcessArchiveFiles/facetProcessArchiveFiles.log

Schedule: Every hour

Description:
This is one of the steps used in automation of archiver configuration.
This assumes the requests for the archiver are somehow stored in the IOC (perhaps in INFO fields) and then submitted as archive files.
Archive files are consolidated together into a IOC_DATA like folder and can be identified using a bash glob expression.
Archive files are space/tab separated text files with these rules
1) They can have blank lines
2) They can have comment lines beginning with a #
3) Space/tab separated lines are interpreted as PVName, sampling period (in secs), sampling method (one of scan/monitor).
4) The PVName is mandatory; the other two are optional in which case defaults (optional arguments to this script)  are applied.

Example report:

Sent to: controls-software-reports@slac.stanford.edu, ppascual@slac.stanford.edu, bhill@slac.stanford.edu

2020-09-28 14:00:06,639 - Starting processArchiveFiles...
2020-09-28 14:00:06,639 - 2020-09-28 14:00:06.639676
2020-09-28 14:00:06,639 - Logfile: /nfs/slac/g/facet/tools/archiver_appliance_automation/facetProcessArchiveFiles/ProcessArchiveFiles.log
2020-09-28 14:00:06,639 - Searching for archive files: /mccfs2/u1/facet/epics/ioc/data*/archive/*.ar*
2020-09-28 14:00:06,814 - Processing file /mccfs2/u1/facet/epics/ioc/data/ArchiveBrowser/archive/toBeArchivedList.archive modified within 999 days
2020-09-28 14:00:06,814 - Processing file /mccfs2/u1/facet/epics/ioc/data/eioc-in10-mp01/archive/eioc-in10-mp01.arConfig modified within 999 days
2020-09-28 14:00:06,814 - Processing file /mccfs2/u1/facet/epics/ioc/data/eioc-in10-mp02/archive/eioc-in10-mp02.arConfig modified within 999 days
2020-09-28 14:00:06,814 - Processing file /mccfs2/u1/facet/epics/ioc/data/ioc-in10-mp01/archive/ioc-in10-mp01.arConfig modified within 999 days
2020-09-28 14:00:06,814 - Processing file /mccfs2/u1/facet/epics/ioc/data/ioc-li11-mc01/archive/ioc-in10-mc01.arConfig modified within 999 days
2020-09-28 14:00:06,814 - Processing file /mccfs2/u1/facet/epics/ioc/data/ioc-lr10-ls02/archive/ioc-lr10-ls02.archive modified within 999 days
2020-09-28 14:00:06,814 - Processing file /mccfs2/u1/facet/epics/ioc/data/ioc-lr10-ls02/archive/ioc-lr10-ls02.archive~ modified within 999 days
[...]
2020-09-28 14:00:06,818 - Ignored 0 total files older than 999 days
2020-09-28 14:00:06,824 - Starting new HTTP connection (1): facet-archapp:17665
2020-09-28 14:00:07,629 - http://facet-archapp:17665 "GET /mgmt/bpl/getAllExpandedPVNames HTTP/1.1" 200 None
2020-09-28 14:00:13,432 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/ArchiveBrowser/archive/toBeArchivedList.archive
2020-09-28 14:00:13,683 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/eioc-in10-mp01/archive/eioc-in10-mp01.arConfig
2020-09-28 14:00:13,685 - All 21 PVs from eioc-in10-mp01/archive/eioc-in10-mp01.arConfig are in the archiver
2020-09-28 14:00:13,685 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/eioc-in10-mp02/archive/eioc-in10-mp02.arConfig
2020-09-28 14:00:13,687 - All 21 PVs from eioc-in10-mp02/archive/eioc-in10-mp02.arConfig are in the archiver
2020-09-28 14:00:13,687 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/ioc-in10-mp01/archive/ioc-in10-mp01.arConfig
2020-09-28 14:00:13,689 - All 23 PVs from ioc-in10-mp01/archive/ioc-in10-mp01.arConfig are in the archiver
2020-09-28 14:00:13,689 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/ioc-li11-mc01/archive/ioc-in10-mc01.arConfig
2020-09-28 14:00:13,695 - All 161 PVs from ioc-li11-mc01/archive/ioc-in10-mc01.arConfig are in the archiver
2020-09-28 14:00:13,695 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/ioc-lr10-ls02/archive/ioc-lr10-ls02.archive
2020-09-28 14:00:13,696 - All 1 PVs from ioc-lr10-ls02/archive/ioc-lr10-ls02.archive are in the archiver
2020-09-28 14:00:13,696 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/ioc-lr10-ls02/archive/ioc-lr10-ls02.archive~
2020-09-28 14:00:13,696 - All 0 PVs from ioc-lr10-ls02/archive/ioc-lr10-ls02.archive~ are in the archiver
2020-09-28 14:00:13,696 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/ioc-sys1-mp01/archive/ioc-sys1-mp01.arConfig
2020-09-28 14:00:13,699 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-in10-bp01/archive/sioc-in10-bp01.archive
2020-09-28 14:00:13,727 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-in10-bp02/archive/sioc-in10-bp02.archive
2020-09-28 14:00:13,755 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-in10-bp03/archive/sioc-in10-bp03.archive
2020-09-28 14:00:13,782 - Checking for liveness of 1002 PVs
2020-09-28 14:00:18,877 - Submitting 780 new PVs to the archiver
2020-09-28 14:00:18,877 - Submitting these PVs to the archiver:
XCOR:GUNB:293:IGND.HIHI
QUAD:GUNB:823:2:R.LOLO
SOLN:GUNB:100:R.LOW
SOLN:GUNB:100:IGND.LOW
QUAD:GUNB:212:2:IMON.HIHI
YCOR:GUNB:513:BACT.LOLO
XCOR:GUNB:513:BMON.LOLO
LGPS:GUNB:1:IGND.LOLO
[...]
MCOR:GUNB:116:ISETPT
SOLN:GUNB:212:CALBSLOPE
QUAD:GUNB:823:1:EDESSAVE
QUAD:GUNB:823:1:IMON.HIGH
SOLN:GUNB:100:BCON.LOLO
QUAD:GUNB:212:1:IGND.LOW
XCOR:GUNB:927:BACT.HIGH
XCOR:GUNB:293:R.HIHI
YCOR:GUNB:388:IRIPL.LOW
QUAD:GUNB:823:1:IRIPL.HIGH
XCOR:GUNB:927:BCON.LOLO
SOLN:GUNB:212:BMON.LOLO
XCOR:GUNB:927:BCON.HIHI
XCOR:GUNB:927:R.LOLO
XCOR:GUNB:927:R.HIHI
QUAD:GUNB:212:1:BACT.LOLO
QUAD:GUNB:823:1:BACT.HIGH
XCOR:GUNB:293:CALBGOODFIT
YCOR:GUNB:927:BMON.LOLO
QUAD:GUNB:823:2:BACT.LOLO
QUAD:GUNB:212:1:BMON.HIGH
XCOR:GUNB:513:IGND.HIHI
QUAD:GUNB:823:2:BACT.LOW
XCOR:GUNB:713:IACT.HIHI
XCOR:GUNB:927:R.LOW
LGPS:GUNB:1:BACT.HIHI
YCOR:GUNB:388:IGND.LOLO
SOLN:GUNB:100:BACT.HIGH
XCOR:GUNB:388:R.HIHI
QUAD:GUNB:212:1:CALBOFFSET
MCOR:GUNB:200:CRATE_I3_3VCC
SOLN:GUNB:100:EDES
MCOR:GUNB:100:LATCHED_MAGNET5_STATE
XCOR:GUNB:293:TRIMDISABLE
YCOR:GUNB:388:NOANLGCTRL
QUAD:GUNB:212:2:R.LOW
YCOR:GUNB:513:IMON.LOW
QUAD:GUNB:212:1:IRIPL.LOLO
LGPS:GUNB:1:IMON.HIGH
LGPS:GUNB:1:BMON.LOLO
YCOR:GUNB:293:BACT.HIHI
SOLN:GUNB:823:IRIPL.HIHI
YCOR:GUNB:513:BCON.HIHI
QUAD:GUNB:212:1:ECON
YCOR:GUNB:713:BMAGX
IOC:SYS1:MP01:PATTERNDIAG.I
IOC:SYS1:MP01:PATTERNDIAG.M
IOC:SYS1:MP01:PATTERNDIAG.K
IOC:SYS1:MP01:PATTERNDIAG.O
MCOR:GUNB:100:INHIBITSTATE
YCOR:GUNB:927:STATE
SOLN:GUNB:100:MPS
2020-09-28 14:00:18,883 - Starting new HTTP connection (1): facet-archapp:17665
2020-09-28 14:00:21,815 - http://facet-archapp:17665 "POST /mgmt/bpl/archivePV HTTP/1.1" 200 None
2020-09-28 14:00:30,180 - archivePVs returned with status 200
[...]
2020-09-28 14:00:36,636 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-lr10-ls06/archive/sioc-lr10-ls06.archive
2020-09-28 14:00:36,637 - All 5 PVs from sioc-lr10-ls06/archive/sioc-lr10-ls06.archive are in the archiver
2020-09-28 14:00:36,637 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-lr10-ls06/archive/sioc-lr10-ls06.archive~
2020-09-28 14:00:36,637 - All 0 PVs from sioc-lr10-ls06/archive/sioc-lr10-ls06.archive~ are in the archiver
2020-09-28 14:00:36,637 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-lr10-tm01/archive/sioc-lr10-tm01.archive
2020-09-28 14:00:36,644 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-sys1-mg01/archive/sioc-sys1-mg01.archive
2020-09-28 14:00:36,665 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-sys1-mg02/archive/sioc-sys1-mg02.archive
2020-09-28 14:00:36,786 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-sys1-ml00/archive/sioc-sys1-ml00.archive
2020-09-28 14:00:36,943 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-sys1-ml01/archive/sioc-sys1-ml01.archive
2020-09-28 14:00:37,099 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-sys1-ml02/archive/sioc-sys1-ml02.archive
2020-09-28 14:00:37,256 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-sys1-ml03/archive/sioc-sys1-ml03.archive
2020-09-28 14:00:37,415 - Processing archive file /mccfs2/u1/facet/epics/ioc/data/sioc-sys1-nw01/archive/sioc-sys1-nw01.archive
2020-09-28 14:00:37,575 - Checking for liveness of 1190 PVs
2020-09-28 14:00:42,673 - Skipped 1190 potentially stale PVs:
KLY:LI10:51:PHASE_W1CH0
KLY:LI10:51:ADES05
KLY:LI10:51:AACT_JITT02
KLY:LI10:51:FBWGT1
KLY:LI10:51:PREF02
KLY:LI10:51:PREF05
KLY:LI10:51:AMPLSCL2
KLY:LI10:51:AACT05
KLY:LI10:51:AREF04
[...]
IOC:LI20:MC01:UPTIME
SIOC:LI20:TZ02:SUSP_TASK_CNT
SIOC:MCC1:CUD1:START_CNT
SIOC:LI20:SP01:START_TOD
SIOC:SYS1:OP00:FD_FREE
IOC:LI20:PM03:SUSP_TASK_CNT
CRAT:LI20:MP01:POWERSTATE
IOC:LI20:RF01:START_CNT
IOC:IN10:BP01:HEARTBEATOK
SIOC:LI20:PM01:FD_FREE
2020-09-28 14:00:42,674 - Total run time: 36.04696464538574
2020-09-28 14:00:42,674 - Done.

Notes:

lclsPauseDisconnectedPVs

Cronjob:

lcls-prod01 00 00 * * * /afs/slac/g/lcls/package/ArchiverAppliance/tools/script/facetPauseDisconnectedPVs.sh

Log:

/nfs/slac/g/lcls/tools/archiver_appliance_automation/facetPauseDisconnectedPVs/facetPauseDisconnectedPVs.log

Schedule: Daily, at 00:00

Description:
Gets a list of PVs that are disconnected and then pauses those that have not connected for a specified amount of time.

Example report:

Sent to: controls-software-reports@slac.stanford.edu, ppascual@slac.stanford.edu, bhill@slac.stanford.edu

2020-09-26 03:00:09,906 - Starting pauseDisconnectedPVs...
2020-09-26 03:00:09,906 - 2020-09-26 03:00:09.906773
2020-09-26 03:00:09,906 - Logfile: /nfs/slac/g/facet/tools/archiver_appliance_automation/facetPauseDisconnectedPVs/facetPauseDisconnectedPVs.log
2020-09-26 03:00:09,913 - Starting new HTTP connection (1): facet-archapp:17665
2020-09-26 03:00:09,962 - http://facet-archapp:17665 "GET /mgmt/bpl/getCurrentlyDisconnectedPVs HTTP/1.1" 200 9785
2020-09-26 03:00:09,963 - 26 disconnected PVs found
2020-09-26 03:00:09,964 - 26 PVs have been disconnected for more than 2880 minutes:
PMTR:LA20:10:PWR
SIOC:LA20:SC01:TODSUM
SIOC:LI20:SC01:TODDIFF
SIOC:LI20:SC10:HEARTBEATSUM
SIOC:LA20:SC01:TODDIFF
SIOC:LI20:SC10:HEARTBEATOK
SIOC:LI20:SC11:HEARTBEATSUM
SIOC:LA20:SC01:HEARTBEATOK
SIOC:MCC0:SC02:HEARTBEATECT
SIOC:LI20:SC11:HEARTBEATOK
SIOC:MCC0:SC02:TODSUM
SIOC:LI20:SC10:HEARTBEATECT
SIOC:LI20:SC11:TODSUM
SIOC:LI20:SC01:HEARTBEATSUM
SIOC:MCC0:SC02:TODDIFF
SIOC:LA20:SC01:HEARTBEATSUM
SIOC:LI20:SC01:TODSUM
SIOC:MCC0:SC02:HEARTBEATOK
SIOC:LI20:SC01:HEARTBEATECT
SIOC:LA20:SC01:HEARTBEATECT
SIOC:LI20:SC01:HEARTBEATOK
SIOC:LI20:SC10:TODSUM
SIOC:LI20:SC11:HEARTBEATECT
SIOC:LI20:SC10:TODDIFF
SIOC:LI20:SC11:TODDIFF
SIOC:MCC0:SC02:HEARTBEATSUM
2020-09-26 03:00:09,964 - Checking for liveness of 26 PVs
2020-09-26 03:00:14,980 - Found 26 PVs disconnected for at least 5 minutes
2020-09-26 03:00:14,984 - Starting new HTTP connection (1): facet-archapp:17665
2020-09-26 03:00:15,080 - http://facet-archapp:17665 "POST /mgmt/bpl/pauseArchivingPV HTTP/1.1" 200 8510
2020-09-26 03:00:15,081 - Done.
2020-09-26 03:00:15,081 - Total run time: 5.188454866409302

Notes:

facetSetupEnv

Notes: