...
Wiki Markup |
---|
{table:border=1} {tr} {th}script name{th} {th}descr{th} {tr} {td}{color:#0000ff}runAllCDCrawlers.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td} {td}runs * SLAC PV crawler * all client crawlers * rec client cleanup * data validation for all current crawl data (LCLS and CD) * sync to MCCO \\ Logfile: /nfs/slac/g/cd/log/irmis/pv/ CDCrawlerAll.log {td} {tr} {td}{color:#0000ff}runSLACPVCrawler.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td} {td}run by runAllCDCrawlers.csh: crawls NLCTA IOCs (previously handled PEPII IOCs) \\ The PV crawler is run 4 times within this script to accommodate the various boot directory structures: || system || IOCs crawled || log files || | NLCTA | IOCs booted from $CD_IOC on gtw00. \\ Boot dirs mirrored to /nfs/mccfs0/u1/pepii/mirror/cd/ioc for crawling | /nfs/slac/g/cd/log/irmis/pv/pv_crawlerLOG_ALL.\* \\ /tmp/pvCrawler.log | | TR01 | TR01 only (old boot structure) | /nfs/slac/g/cd/log/irmis/pv/pv_crawlerLOG_TR01.\* \\ /tmp/pvCrawler.log | {td} {tr} {tr} {td}{color:#008000}runLCLSPVCrawlerLx.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td} {td}crawls LCLS IOCs || system || IOCs crawled || log files || | LCLS | LCLS PRODUCTION IOCs. all IOC dirs in \\ $IOC /usr/local/lcls/epics/iocCommon | /u1/lcls/tools/crawler/pv/pv_crawlerLOG_LCLS.timestamp \\ /tmp/ LCLSPVCrawl.log | {td} {tr} {tr} {td}{color:#800080}runFACETPVCrawlerLx.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td} {td}crawls FACET IOCs || system || IOCs crawled || log files || | FACET | FACET PRODUCTION IOCs. all IOC dirs in facet $IOC /usr/local/facet/epics/iocCommon | /u1/facettools/crawler/pv/pv_crawlerLOG_LCLS.timestamp \\ /tmp/FACETPVCrawl.log | {td} {tr} {tr} {td}{color:#0000ff}runClientCrawlers.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_utils{_}{color}\\ \\ {color:#0000ff}individual client crawler scripts are here{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td} {td} Runs PV client crawlers in sequence. *LCLS client config files are all scp-ed to /nfs/slac/g/cd/crawler/lcls \*Configs for crawling* || Client crawler || config file with list of directories/files crawled || log files || | ALH | /afs/slac/g/cd/tools/irmis/cd_config/SLACALHDirs | /nfs/slac/g/cd/log/irmis/alh/alh_crawlerLOG.\* \\ /tmp/pvCrawler.log | | Channel Watcher (CW) | /afs/slac/g/cd/tools/irmis/cd_config/SLACCWdirs.lst | /nfs/slac/g/cd/log/irmis/cw/cw_crawlerLOG.\* \\ /tmp/pvCrawler.log | | Channel Archiver (CAR) | /afs/slac/g/cd/tools/irmis/cd_config/SLACCARfiles.lst \\ note: all CAR files for LCLS are scp-ed over from lcls-archsrv \\ to /nfs/slac/g/cd/crawler/lclsCARConfigs for crawling | /nfs/slac/g/cd/log/irmis/car/car_crawlerLOG.\* \\ /tmp/pvCrawler.log | *Also runs* load_vuri_rec_client_type.pl for clients that don't handle vuri_rec_client_type records (sequence crawler only, at the moment) {td} {tr} {tr} {td}{color:#0000ff}runRecClientCleanup.csh{color}\\ {color:#0000ff}{_}in in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td} {td}deletes all non-current rec client, rec client flag and vuri rows. Logs to /nfs/slac/g/cd/log/irmis/client_cleanupLOG.\*{td} {tr} {tr} {td}{color:#008000}run_find_devices.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td} {td}for LCLS and FACET only, populate the devices_and_attributes table, a list of device names and attributes based on the LCLS PV naming convention. For PV DEV:AREA:UNIT:ATTRIBUTE, DEV:AREA:UNIT is the device, ATTRIBUTE is the attribute.{td} {tr} {tr}\\ {td}{color:#008000}run_load_bsa_root_names.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td}{td} loads bsa_root_names table by running stored procedure LOAD_BSA_ROOT_NAMES. LCLS and FACET names.{td} {tr} {tr} {td}{color:#008000}ioc_report.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color} {color:#800080}ioc_report-facet.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td} {td}runs at the end of the LCLS PV crawl, which is last, creates the web ioc report: \\ [http://www.slac.stanford.edu/grp/cd/soft/database/reports/ioc_report.html] {td} {tr} {tr} {td}{color:#0000ff}updateMaterializedViews.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td} {td}refreshes materialized view from curr_pvs{td} {tr} {tr} {td}{color:#008000}findDupePVs.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color} {color:#800080}findDupePVs-all.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td} {td}finds duplicate PVs for reporting to e-mail \\ (the \--all version takes system as a parameter){td} {tr} {tr} {td}{color:#008000}copyClientConfigs.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color} {color:#800080}copyClientConfigs-facet.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td} {td}copies alh, cw and car config files to /nfs for crawling by the client crawler job{td} {tr} {tr} {td}{color:#008000}copyClientConfigs.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color} {color:#800080}copyClientConfigs-facet.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td} {td}copies alh, cw and car config files to /nfs for crawling by the client crawler job{td} {tr} {tr} {td}{color:#008000}refresh_curr_ioc_device.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td} {td}refreshes the curr_ioc_device table with currently booted info from ioc_device (speeds up query in the Archiver PV APEX app). \\ Also see caget4curr_ioc_device.bash below.{td} {tr} {tr} {td}{color:#008000}find_LCLSpv_count_changes.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color} {color:#800080}find_pv_count_changes-all.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td} {td}finds the IOCs with PV counts that changed in the last LCLS crawler run. This is reported in the logfile, and in the daily e-mail. \\ (the \--all version takes system as a parameter){td} {tr} {tr} {td}{color:#008000}populate_dtyp_io_tab.bash{color}\\ {color:#008000}populate_io_curr_pvs_and_fields.bash{color}\\ {color:#008000}run_load_hw_dev_pvs.bash{color}\\ {color:#008000}run_parse_camac_io.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td} {td}set of script to populate various tables for the EPICS camdmp APEX application.{td} {tr} {tr} {td}{color:#008000}write_data_validation_row.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color} {color:#0000ff}write_data_validation_row.csh{color}\\ {color:#0000ff}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color} {color:#800080}write_data_validation_row-facet.bash{color}\\ {color:#800080}{_}in /usr/local/facet/tools/irmis/script/_{color}{td} {td}writes a row to the controls_global.data_validation_audit table (all environments){td} {tr} {tr} {td}runDataValidation.csh \\ runDataValidation-facet.csh \\ _in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{td} {td}runs IRMISDataValidation.pl -- CD and LCLS PV data validation{td} {tr} {tr} {td}{color:#b00e00}runSync.csh{color}\\ {color:#b00e00}{_}in /afs/slac/g/cd/soft/tools/irmis/cd_script{_}{color}{td} {td}runs data replication to MCCO, currently these objects ONLY: * curr_pvs * bsa_root_names * devices_and_attributes{td} {tr} {tr} {td}{color:#008000}caget4curr_ioc_device.bash{color}\\ {color:#008000}{_}in /usr/local/lcls/tools/irmis/script/_{color}{td} {td}does cagets to obtain IOC parameters for the curr_ioc_device table. Run as a separate cron job from the crawlers because cagets \\ can hang unexpectedly -- they are best done in an isolated script\!{td} {tr} {table} |
...
- bsa_root_names row count checks: check for problem accessing the table, or 0 or otherwise unexpected row counts for expected bsa types
- devices_and_attributes row count checks: problem accessing the table, or row count 0, of devices_and_attributes row count differs from distinct rows in the curr_pvs view.
- check for successful completion of all crawler steps: check to make sure all crawler steps enter a start and a completion row in controls_global.data_validation_audit.
some circumstances where steps are missing:- step launched but didn’t finish: check the status of processes launched by the cron job using ps --ef | grep. An example: when perl dbi was hanging due to the 199-day-Linux-server-uptime bug. Several LCLS PV crawler jobs had launched, but had hung in the db_connect statement, and had to be killed from the Linux command line.
- step launched and finished, but the completed step was never written: the getPwd problems cause this symptom. See entries starting 9/23 9 pm for an illustration.
- step never launched: is the script available? Is the server up? Is crontab/trscrontab configured correctly? Are there permission problems? etc.
- other mysteries: figure out where the job in question stopped, using ps --ef, logfiles, etc…
- check for successful completion of all crawler steps: check to make sure all crawler steps enter a start and a completion row in controls_global.data_validation_audit.
...