...
** Only LCLS data validation will affect the sync to MCCO. FACET, and CD e-mails are fyi only.
Daily monitoring of e-mails
- Check the "IRMIS data validation" e-mail title:SUCCESS or FAILED? If it's SUCCESSful, and there is a subsequent "IRMIS data sync to MCCO completed", then all is well.
- If the data validation FAILED:
- What does the message for the DATA_VALIDATION step say? The errors found will be listed. (e.g. "PV Count drop > 5000", and/or "LCLS Crawler did not complete", etc.)
- Refer to the <acc> PV Crawler errors and warnings e-mails. These show the changes in PV counts by IOC (useful in locating which IOC's PVs count dropped for example)
- If needed, go to the logfiles: see "IRMIS data validation" email for the log directory for each crawler. Also, see table below for logfiles to refer to.
...
At SLAC, what the PV crawlers update, in a nutshell
...
description | cvs root | production directory tree root | details |
|---|---|---|---|
IRMIS software | SLAC code has diverged from the original collaboration version, and LCLS IRMIS code has diverged from the main SLAC code (i.e. we have 2 different version of the IRMIS PV crawler.) | CD: |
|
CD scripts | For ease and clarity, the CD scripts are also in the LCLS CVS repository under | /afs/slac/g/cd/tools/irmis |
|
LCLS scripts | /afs/slac/g/lcls/cvs | /usr/local/lcls/tools/irmis |
|
FACET scripts | These scripts share the LCLS repository (different names so they don’t collide with LCLS scripts) | /usr/local/facet/tools/irmis |
|
...
- runSLACPVCrawler.csh, runLCLSPVCrawlerLx.bash and runFACETPVCrawlerLx.bash set up for and run the IRMIS pv crawler multiple times to hit all the boot structures and crawl groups. Environment variables set in pvCrawlerSetup.bash, pvCrawlerSetup-facet.bash and pvCrawlerSetup.csh, point the crawler to IOC boot directories, log directories, etc. Throughout operation, errors and warnings are written to log files.
- The IOC table in the IRMIS schema contains the master list of IOCs. The SYSTEM column designates which crawler group the IOC belongs to.
- An IOC will be hit by the pv crawler if its ACTIVE column is 1 in the IOC table: ACTIVE is set to 0 and 1 by the crawler scripts, depending on SYSTEM, to control what is crawled. i.e. LCLS IOCs are set to active by the LCLS crawler job. NLCTA IOCs are set to active by the CD crawler job.
- An IOC will be crawled if it has been rebooted since the last crawer run: if its LCLS/FACET STARTTOD or CD TIMEOF BOOT (boot time) PV has changed since the last crawl. Otherwise it will be skipped.
- Crawler results will be saved to the DB if at least boot file has changed, as determined by mod date and file size size.
- Crawling specific files can be triggered by changing the mod date (e.g. touch).
- In the pv_crawler.pl script, there’s a mechanism for forcing the crawling of subsets of IOCs (see the pv_crawler.pl code)
- IOCs without a STARTTOD/TIMEOFBOOT PV will be crawled every time. (altho PV data is only written when IOC .db files have changed)
...
- . (altho PV data is only written when IOC .db files have changed)
Removing an IOC from the active crawler list
Sometimes an IOC becomes obsolete. Although IOCs are automatically added to the IRMISDB.IOC table when the crawler encounters a new $IOC/<ioc> directory, IOCs are not automatically removed from the system. Typically, an IOC engineer will request that her IOC be removed from the crawler (and hence all current PV and IOC lists, and reports).
To remove an IOC from the active crawler list:
Update the IRMISDB.IOC table in MCCQA to set the SYSTEM value to something that is not understood by the crawler:
update ioc
set system = 'LCLS-NOCRWL'
where ioc_nm = 'iocname';
or
update ioc
set system = FACET-NOCRWL'
where ioc_nm = 'iocname';
TROUBLESHOOTING: post-crawl emails and data_validation_audit entries: what to worry about, how to bypass or fix for a few…
...
- log into mcclogin
- cvs co irmis gui code:
export CVSROOT=/afs/slac/package/epics/slaconly/cvs
cd to a working directory
cvs co -d irmis tools/irmisV2_SLAC - cd irmis
- edit site.build.properties and change this line:
db.trust-read-write.password=newpw - set java environment to version 1.4 (this may not be needed):
export JAVA_HOME=/afs/slac/package/java/@sys/jdk1.4
export JAVA_VER=1.4
export PATH=${JAVA_HOME}/bin:${PATH}
- cd db
- ant clean
- ant deploy
- cd ..
- cp db/build/irmisDb.jar apps/deploy
- cd apps/deploy
- test the gui:
java -jar irmis.jar
The gui should connect to the db. - deploy new jars to production:
scp irmisirmisDb.jar over to /usr/local/lcls/package/irmis/irmisV2_SLAC/apps/deploy/. (this one is used by LCLS and FACET)
cp irmisirmisDb.jar /afs/slac/package/epics/tools/irmisV2_SLAC/apps/deploy/.
*** Please note, the app directory (parallel to the db directory) is not build-able in ant anymore, because of changes in java version syntax and method calls deep in the gui code. Therefore, please follow the directions above to BUILD AND DEPLOY THE DB JAR ONLY!
...
IRMISDB history data cleanup
$TOOLS/irmis/data_cleanup has scripts and a README for cleaning historical IRMIS data. The procedure has been done once in February, 2012.
Description and plans and click IRMIS_database_cleanup
...
SCHEMA DIAGRAM
| Anchor | ||||
|---|---|---|---|---|
|
...