Date | Time | Equipment * | Action |
---|---|---|---|
Fri 20 Dec 2019 | |||
Mon 6 Jan 2020 |
* Equipment categories
Category | Machine status |
---|---|
NC | non-critical for entire 16-day shutdown period |
NC5 | non-critical if down only during the 5 days of the outage and, possibly, the three days preceding the outage |
XC | experiment critical but not in H.A. rack, only a few, short outages acceptable |
HA | high-availability (continuous operation) |
Table of LISOC Tasks and Services
Function/Service | Sub-Functions | Needed Servers | Needed Databases | Needed File Systems | Other Needs | Needed During Shutdown? | Available During Shutdown? |
---|---|---|---|---|---|---|---|
Mission Planning, LAT Configurations | FastCopy | fermilnx01 or fermilnx02 | TCDB | AFS | Fermi LAT Portal: Timeline Webview; Confluence, JIRA, Mission Planning s/w | yes | |
Real Time Telemetry Monitoring | fermilnx01 or fermilnx02 | spread | during anomalies | ||||
Logging | fermilnx01 or fermilnx02 | TCDB | Fermi LAT Portal: Log Watcher | yes | |||
Trending | TCDB | Fermi LAT Portal: Telemetry Trending | yes | ||||
L0 File Ingest and Archive | FastCopy | L0 Archive | yes | ||||
Data Gap Checking and Reporting | FastCopy | fermilnx01 or fermilnx02 | L0 Archive | yes, continuously | |||
L1 processing | pipeline | SLAC Farm | Data Catalog | Fermi LAT Portal: Pipeline, Data Processing | yes | ||
L1 Data Quality Monitoring | Fermi LAT Portal, Telemetry Trending | ||||||
L1 delivery | FastCopy | fermilnx01 or fermilnx02 | Data Catalog | yes | |||
L2 processing (ASP) and Delivery | FastCopy | fermilnx01 or fermilnx02 | Data Catalog | Fermi LAT Portal: Pipeline, Data Processing | daily, weekly | ||
RSP | ? | ||||||
The following table of servers must remain powered up and operational for Fermi Level 1 to succeed.
Category | server | VM/service | function |
---|---|---|---|
XC | fermi-gpfs02 | xrootd server | |
XC | fermilnx05-vmm | fermilnx-v02 | xrootd redirector |
XC | fermilnx07-vmm | fermilnx-v12 | xrootd redirector |
HA | wain031 (or equivalent) | NFS storage | |
HA | fermilnx01 | LAT config, fastcopy and real-time telemetry | |
HA | fermilnx02 | LAT config, fastcopy and real-time telemetry | |
XC | fermilnx05-vmm | fermilnx-v03 | archiver |
HA | fermi-oracle01 | oracle primary | |
HA | fermi-oracle02 | oracle secondary | |
HA | mysql05/06 | mysql-node03 | calibration, etc. DB |
HA | hequ0001-0030 | batch hosts. queues={express,short,long,glastdataq} users={glast,lsstsim,lsstprod,glastmc,glastraw} | |
XC | hequ0031-0050 | batch hosts | |
HA | fermilnx03-vmm | fermilnx-v07/tomcat01 | Commons, Group manager |
XC | fermilnx09-vmm | fermilnx-v16/tomcat06 | rm2 |
XC | fermilnx07-vmm | fermilnx-v05/tomcat08 | dataCatalog |
XC | fermilnx09-vmm | fermilnx-v17/tomcat09 | Pipeline-II |
XC | fermilnx08-vmm | fermilnx-v15/pipeline-mail01 | Pipeline-II email server |
XC | fermilnx09-vmm | fermilnx-v18/tomcat10 | FCWebView, ISOCLogging, MPWebView TelemetryMonitor, TelemetryTableWebUI |
XC | fermilnx07-vmm | fermilnx-v10/tomcat11 | DataProcessing |
XC | fermilnx07-vmm | fermilnx-v11/tomcat12 | TelemetryTrending |
NC5 | (non-Fermi server) | astore | data archive |
HA | (non-Fermi server) | trscron | tokenized cron |
HA | (non-Fermi server) | lnxcron | cron |
XC | (non-Fermi server) | (farm manager, etc.) | LSF management |
HA | yfs01 (non-Fermi) | AFS | |
HA | yfs02 (non-Fermi) | AFS | |
HA | (non-Fermi server) | JIRA | issue tracking (HA as of 10/20/2017) |
XC | fermi-gpfs03/04 | GPFS | Fermi user disk |
XC | fermi-cnfs01/02 | GPFS/NFS bridge | Fermi user disk |
For general information about the High-availability racks, Shirley provided this pointer to the latest list:
"Service Now, Knowledge Base, search for "High Availability" , following link for current servers"
And here is the current statement about high-availability functionality:
Current Services in HA Racks •CATER application •Confluence application •Data center management tool •Drupal web •Email lists •Email transport infrastructure •ERP application •Exchange email •EXO application •Facilities monitoring •Fermi application •IT Ticketing system •Network infrastructure •Site Security infrastructure •Unix authentication infrastructure •Unix AFS infrastructure •Unix mailboxes •Unix monitoring •VPN •Windows authentication infrastructure •Windows file servers and SAN •Windows monitoring •Windows web
The services for L1:
oracle
mysql
tomcats
isoc servers
xroot
nfs
LSF
Here's what ISOC tasks need:
FASTCopy chain
--------------
wain031
fermilnx01
fermilnx02
trscron
fermilnx-v03 (Archiver)
Whatever the pipeline server runs on.
xroot servers
astore system
Web servers
-----------
tomcat01 Commons
tomcat06 rm2
tomcat09 Pipeline-II
tomcat10 FCWebView, ISOCLogging, MPWebView
TelemetryMonitor, TelemetryTableWebUI
tomcat11 DataProcessing
tomcat12 TelemetryTrending
Notes:
9/29/2017 - (TG and WK) added list of xrootd servers needed to bridge the gap in December; Started table of server names