Figuring out what we need to get through. None of the pages mapping services to servers are up to date.
Version 1.2 (5 Oct 2017)
Power outage time-line (from Shirley). Note that the times of "10am" and "5pm" are notional and current guesses, but could change.
| Date | Time | Equipment | Action |
|---|---|---|---|
| Fri 22 Dec 2017 | EOB? | non-critical, general use | power off |
| Tue 26 Dec 2017 | before 10am | non-critical, special request exp-critical | power off power off |
| 10am | Bldg 50 power off | ||
| after 10am | exp-critical | power ON | |
| High-avail | continuously powered ON | ||
| Sat 30 Dec 2017 | before 5pm | exp-critical | power off |
| 5pm | Bldg 50 power ON | ||
| after 5pm | exp-critical non-critical, special request | power ON power ON | |
| Mon 8 Jan 2018 | starting 8am | non-critical, general use | power ON |
The following table of servers must remain powered up and operational for Fermi Level 1 to succeed.
- Confirm current H.A. rack occupants. See spreadsheet here (thanks Shirley!) https://portal.slac.stanford.edu/info/ITHelp/KB%20Assets/HA-Servers.xlsx
- Confirm the VM-master for a given VM. Use the 'node' command, e.g., $ node -whereis fermilnx-v12
- Confirm the tomcat <-> service associations. Table here.
- Confirm the tomcat-VM associations in this table. Use the 'node' command, e.g., $ node -whereis glast-tomcat01
| server | VM/service | function | HA rack |
|---|---|---|---|
| fermi-gpfs02 | xrootd server | ||
| fermilnx05-vmm | fermilnx-v02 | xrootd redirector | |
| fermilnx07-vmm | fermilnx-v12 | xrootd redirector | |
| wain031 (or equivalent) | NFS storage | yes | |
| fermilnx01 | LAT config, fastcopy and real-time telemetry | yes | |
| fermilnx02 | LAT config, fastcopy and real-time telemetry | yes | |
| fermilnx05-vmm | fermilnx-v03 | archiver | |
| fermi-oracle01 | oracle primary | yes | |
| fermi-oracle02 | oracle secondary | yes | |
| mysql05/06 | mysql-node03 | calibration, etc. DB | yes |
| hequNNN - hequNNN+24 | batch hosts | ||
| fermilnx03-vmm | fermilnx-v07/tomcat01 | Commons, Group manager | yes |
| fermilnx09-vmm | fermilnx-v16/tomcat06 | rm2 | |
| fermilnx07-vmm | fermilnx-v05/tomcat08 | dataCatalog | |
| fermilnx09-vmm | fermilnx-v17/tomcat09 | Pipeline-II | |
| fermilnx09-vmm | fermilnx-v18/tomcat10 | FCWebView, ISOCLogging, MPWebView TelemetryMonitor, TelemetryTableWebUI | |
| fermilnx07-vmm | fermilnx-v10/tomcat11 | DataProcessing | |
| fermilnx07-vmm | fermilnx-v11/tomcat12 | TelemetryTrending | |
| (non-Fermi server) | astore | data archive | |
| (non-Fermi server) | trscron | tokenized cron | |
| (non-Fermi server) | lnxcron | cron | |
| (non-Fermi server) | (farm manager, etc.) | LSF | |
| yfs01 (non-Fermi) | AFS | yes | |
| yfs02 (non-Fermi) | AFS | yes |
High availabilty racks
For general information about the High-availability racks, Shirley provided this pointer to the latest list:
"Service Now, Knowledge Base, search for "High Availability" , following link for current servers"
And here is the current statement about high-availability functionality:
The services for L1:
oracle
- pipeline
- data catalog
- group manager
mysql
- calibrations
tomcats
- pipeline
- data catalog
- data processing
isoc servers
xroot
- fermi-gpfs02 (xrootd server)
- fermilnx-v02 (redirector)
- fermilnx-v12 (redirector)
nfs
- Pretty much everything that's currently on wain031
LSF
- ~25 hosts should let us keep up
Here's what ISOC tasks need:
FASTCopy chain
--------------
wain031
fermilnx01
fermilnx02
trscron
fermilnx-v03 (Archiver)
Whatever the pipeline server runs on.
xroot servers
astore system
Web servers
-----------
tomcat01 Commons
tomcat06 rm2
tomcat09 Pipeline-II
tomcat10 FCWebView, ISOCLogging, MPWebView
TelemetryMonitor, TelemetryTableWebUI
tomcat11 DataProcessing
tomcat12 TelemetryTrending
Notes:
9/29/2017 - (TG and WK) added list of xrootd servers needed to bridge the gap in December; Started table of server names