Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

OPERATIONAL and SUPPORT DETAILS

Each job and process sends a status e-mail to a selected list of recipients.

Emails, in chronological order

  1. FACET PV Crawler errors and warnings: output from the FACET crawler  (to: Shashi, Jingchen, Poonam, Judy)
  2. LCLS PV Crawler errors and warnings: output from the LCLS crawler  (to: controls-software-reports, Shashi, Jingchen, Poonam, Judy)
  3. MCCQA SLAC CD PV Crawler errors and warnings: output from the CD (NLCTA) crawler  (to: Shashi, Jingchen, Poonam, Judy)
  4. <timestamp> IRMIS data validation <SUCCESSFUL or FAILED>: shows whether the data validation succeeded or not, and the status of each job in the sequence in reverse cron order, and logfiles to refer to in case of issues  (to: controls-software-reports, Shashi, Jingchen, Poonam, Judy)
  5. IRMIS data sync to MCCO completed: Only received if the data validation was successful, and the sync to MCCO completed.  (to: Shashi, Jingchen, Poonam, Judy)
  6. DUPLICATE PVs FOUND in FACET and LCLS: Only received if duplicate PVs were found.  (to: controls-software-reports, Ernest, Shashi, Jingchen, Mike Z., Murali)

Daily monitoring of e-mails

  1. Check the "IRMIS data validation" e-mail title:SUCCESS or FAILED?  If it's SUCCESSful, and there is a subsequent "IRMIS data sync to MCCO completed", then all is well.
  2. If the data validation FAILED:
    1. What does the message for the DATA_VALIDATION step say?  The errors found will be listed.  (e.g. "PV Count drop > 5000", and/or "LCLS Crawler did not complete", etc.)
    2. Refer to the <acc> PV Crawler errors and warnings e-mails.  These show the changes in PV counts by IOC (useful in locating which IOC's PVs count dropped for example)
    3. If needed, go to the logfiles: see "IRMIS data validation" email for the log directory for each crawler.  Also, see table below for logfiles to refer to.

_____________________________

At SLAC, in a nutshell the PV crawlers update:

...