...
The SLAC PV crawler (runSLACPVCrawler.csh) runs the crawler a couple of times to accommodate the various CD IOC directory structures.
As of mid-January 2014, the LCLS PV crawler now has a function to crawler $IOC_DATA/<ioc>/IOC.dbDumpRecord files, which contain an IOC's record names and a subset of configured fields. This file is waaaaaay simpler to crawler than the ioc's st.cmd syntax. If IOC.dbDumpRecord is found, it will be crawled. If the file is not found, the old crawler code will operate on the ioc's st.cmd file.
At the same time, the FACET PV crawler scripts were modified to use the LCLS crawler code, so IOC.dbDumpRecord files are also crawled for FACET.
As IOCs are rebooted during maintenance and upgrades, IOC.dbDumpRecord will eventually become the only files crawled in the system, resulting in easier maintenance.
cron jobs
- LCLS side: laci on lcls-daemon2: runLCLSPVcrawlerLx.bash: crawls LCLS PVs and creates lcls-specific tables (bsa_root_names, devices_and_attributes), copies LCLS client config files to dir where CD client crawlers can see them.
- LCLS side: laci on lcls-daemon2: caget4curr_ioc_device.bash: does cagets to populate curr_ioc_devices for the IOC Info APEX app. Run separately from the crawlers because cagets can hang unexpectedly – they are best done in an isolated script!
- CD side: laci on lcls-prod01: runAllCDCrawlers.csh: runs CD PV crawler and all client crawlers, data validation, and sync to MCCO.
- FACET side: flaci on facet-daemon1: runFACETPVcrawlerLx.bash: crawls FACET PVs, copies FACET client config files to dir where CD client crawlers can see them.
...
| Wiki Markup |
|---|
{table:border=1}
{tr}
{th}script name{th}
{th}descr{th}
{tr}
{td}{color:#0000ff}runAllCDCrawlers.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td}
{td}runs
* SLAC PV crawler
* all client crawlers
* rec client cleanup
* data validation for all current crawl data (LCLS and CD)
* sync to MCCO
\\
Logfile: /nfs/slac/g/cd/log/irmis/pv/ CDCrawlerAll.log
{td}
{tr}
{td}{color:#0000ff}runSLACPVCrawler.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td}
{td}run by runAllCDCrawlers.csh: crawls NLCTA IOCs (previously handled PEPII IOCs)
\\
The PV crawler is run 4 times within this script to accommodate the various boot directory structures:
|| system || IOCs crawled || log files ||
| NLCTA | IOCs booted from $CD_IOC on gtw00. \\
Boot dirs mirrored to /nfs/mccfs0/u1/pepii/mirror/cd/ioc for crawling | /nfs/slac/g/cd/log/irmis/pv/pv_crawlerLOG_ALL.\* \\
/tmp/pvCrawler.log |
| TR01 | TR01 only (old boot structure) | /nfs/slac/g/cd/log/irmis/pv/pv_crawlerLOG_TR01.\* \\
/tmp/pvCrawler.log |
{td}
{tr}
{tr}
{td}{color:#008000}runLCLSPVCrawlerLx.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td}
{td}crawls LCLS IOCs
|| system || IOCs crawled || log files ||
| LCLS | LCLS PRODUCTION IOCs. all IOC dirs in \\
$IOC /usr/local/lcls/epics/iocCommon | /u1/lcls/tools/crawler/pv/pv_crawlerLOG_LCLS.timestamp \\
/tmp/ LCLSPVCrawl.log |
{td}
{tr}
{tr}
{td}{color:#800080}runFACETPVCrawlerLx.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color} runs LCLS crawler code{td}
{td}crawls FACET IOCs
|| system || IOCs crawled || log files ||
| FACET | FACET PRODUCTION IOCs. all IOC dirs in facet $IOC /usr/local/facet/epics/iocCommon | /u1/facettools/crawler/pv/pv_crawlerLOG_LCLS.timestamp \\
/tmp/FACETPVCrawl.log |
{td}
{tr}
{tr}
{td}{color:#0000ff}runClientCrawlers.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_utils{_}{color}\\
\\ {color:#0000ff}individual client crawler scripts are here{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td}
{td}
Runs PV client crawlers in sequence. *LCLS client config files are all scp-ed to /nfs/slac/g/cd/crawler/lcls \*Configs for crawling*
|| Client crawler || config file with list of directories/files crawled || log files ||
| ALH | /afs/slac/g/cd/tools/irmis/cd_config/SLACALHDirs | /nfs/slac/g/cd/log/irmis/alh/alh_crawlerLOG.\* \\
/tmp/pvCrawler.log |
| Channel Watcher (CW) | /afs/slac/g/cd/tools/irmis/cd_config/SLACCWdirs.lst | /nfs/slac/g/cd/log/irmis/cw/cw_crawlerLOG.\* \\
/tmp/pvCrawler.log |
| OBSOLETE: Channel Archiver (CAR) | /afs/slac/g/cd/tools/irmis/cd_config/SLACCARfiles.lst \\
note: all CAR files for LCLS are scp-ed over from lcls-archsrv \\
to /nfs/slac/g/cd/crawler/lclsCARConfigs for crawling | /nfs/slac/g/cd/log/irmis/car/car_crawlerLOG.\* \\
/tmp/pvCrawler.log |
*Also runs* load_vuri_rec_client_type.pl for clients that don't handle vuri_rec_client_type records (sequence crawler only, at the moment)
{td}
{tr}
{tr}
{td}{color:#0000ff}runRecClientCleanup.csh{color}\\ {color:#0000ff}{_}in in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td}
{td}deletes all non-current rec client, rec client flag and vuri rows. Logs to /nfs/slac/g/cd/log/irmis/client_cleanupLOG.\*{td}
{tr}
{tr}
{td}{color:#008000}run_find_devices.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td}
{td}for LCLS and FACET only, populate the devices_and_attributes table, a list of device names and attributes based on the LCLS PV naming convention. For PV DEV:AREA:UNIT:ATTRIBUTE, DEV:AREA:UNIT is the device, ATTRIBUTE is the attribute.{td}
{tr}
{tr}\\
{td}{color:#008000}run_load_bsa_root_names.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td}{td}
loads bsa_root_names table by running stored procedure LOAD_BSA_ROOT_NAMES. LCLS and FACET names.{td}
{tr}
{tr}
{td}{color:#008000}ioc_report.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}
{color:#800080}ioc_report-facet.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td}
{td}runs at the end of the LCLS PV crawl, which is last, creates the web ioc report:
\\
[http://www.slac.stanford.edu/grp/cd/soft/database/reports/ioc_report.html]
{td}
{tr}
{tr}
{td}{color:#0000ff}updateMaterializedViews.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td}
{td}refreshes materialized view from curr_pvs{td}
{tr}
{tr}
{td}{color:#008000}findDupePVs.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}
{color:#800080}findDupePVs-all.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td}
{td}finds duplicate PVs for reporting to e-mail
\\
(the \--all version takes system as a parameter){td}
{tr}
{tr}
{td}{color:#008000}copyClientConfigs.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}
{color:#800080}copyClientConfigs-facet.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td}
{td}copies alh, cw and car config files to /nfs for crawling by the client crawler job{td}
{tr}
{tr}
{td}{color:#008000}copyClientConfigs.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}
{color:#800080}copyClientConfigs-facet.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td}
{td}copies alh, cw and car config files to /nfs for crawling by the client crawler job{td}
{tr}
{tr}
{td}{color:#008000}refresh_curr_ioc_device.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td}
{td}refreshes the curr_ioc_device table with currently booted info from ioc_device (speeds up query in the Archiver PV APEX app).
\\
Also see caget4curr_ioc_device.bash below.{td}
{tr}
{tr}
{td}{color:#008000}find_LCLSpv_count_changes.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}
{color:#800080}find_pv_count_changes-all.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td}
{td}finds the IOCs with PV counts that changed in the last LCLS crawler run. This is reported in the logfile, and in the daily e-mail.
\\
(the \--all version takes system as a parameter){td}
{tr}
{tr}
{td}{color:#008000}populate_dtyp_io_tab.bash{color}\\ {color:#008000}populate_io_curr_pvs_and_fields.bash{color}\\ {color:#008000}run_load_hw_dev_pvs.bash{color}\\ {color:#008000}run_parse_camac_io.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td}
{td}set of script to populate various tables for the EPICS camdmp APEX application.{td}
{tr}
{tr}
{td}{color:#008000}write_data_validation_row.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}
{color:#0000ff}write_data_validation_row.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}
{color:#800080}write_data_validation_row-facet.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td}
{td}writes a row to the controls_global.data_validation_audit table (all environments){td}
{tr}
{tr}
{td}runDataValidation.csh
\\
runDataValidation-facet.csh
\\
_in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{td}
{td}runs IRMISDataValidation.pl -- CD and LCLS PV data validation{td}
{tr}
{tr}
{td}{color:#b00e00}runSync.csh{color}\\ {color:#b00e00}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td}
{td}runs data replication to MCCO, currently these objects ONLY:
* curr_pvs
* bsa_root_names
* devices_and_attributes{td}
{tr}
{tr}
{td}{color:#008000}caget4curr_ioc_device.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td}
{td}does cagets to obtain IOC parameters for the curr_ioc_device table. Run as a separate cron job from the crawlers because cagets
\\
can hang unexpectedly -- they are best done in an isolated script\!{td}
{tr}
{table} |
...
To bypass crawling specific IOC(s): update the IOC table and for the IOC(s) in question, set SYSTEM to something other than LCLS or NLCTA (e.g. LCLS-TEMP-NOCRAWL) so it/they will not be crawled next time. NOTE: if you change the SYSTEM column for an IOC, it will no longer appear in the curr_pvs view.
In the event of an error that stops crawling, here are the affected end-users of IRMIS PVs:
...