December 2019 Central Computing Outage

Power outage time-line.  

[Tentative proposal]  This power outage will affect substations #7 (next to bldg 50) and #8 (located on the 4th floor of bldg 50).  All of bldg 50 will be without normal power.  The facilities (F&O) group plan to do their maintenance during the 4-day period starting 26 Dec 2019.  However, the outage will start earlier due to lack of staff during the holiday shutdown.  Minimally, it is expected that all H.A. (High Availability) and experiment-critical equipment will be powered throughout the 16+ days of the holiday shutdown.  This page captures what Fermi will need to maintain a minimal data processing effort running during the outage.

* Equipment categories

BELOW THIS LINE HAS NOT YET BEEN FULLY UPDATED FOR DEC 2019 OUTAGE

LISOC Operations Functions

Table of LISOC Tasks and Services

The following table of servers must remain powered up and operational for Fermi Level 1 to succeed.

• Confirm current H.A. rack occupants.  See spreadsheet here (thanks Shirley!) https://portal.slac.stanford.edu/info/ITHelp/KB%20Assets/HA-Servers.xlsx
• Confirm the VM-master for a given VM.  Use the 'node' command, e.g., $ node -whereis fermilnx-v12
• Confirm the tomcat <-> service associations.  Table here.
• Confirm the tomcat-VM associations in this table. Use the 'node' command, e.g., $ node -whereis glast-tomcat01
  
  

High availability racks

For general information about the High-availability racks, Shirley provided this pointer to the latest list:

"Service Now, Knowledge Base,  search for "High Availability" , following link for current servers"

And here is the current statement about high-availability functionality:

Current Services in HA Racks
•CATER application
•Confluence application
•Data center management tool
•Drupal web
•Email lists
•Email transport infrastructure
•ERP application
•Exchange email
•EXO application
•Facilities monitoring
•Fermi application
•IT Ticketing system
•Network infrastructure
•Site Security infrastructure
•Unix authentication infrastructure
•Unix AFS infrastructure
•Unix mailboxes
•Unix monitoring
•VPN 
•Windows authentication infrastructure
•Windows file servers and SAN
•Windows monitoring
•Windows web

The services for L1:

oracle

• pipeline
• data catalog
• group manager

mysql

• calibrations

tomcats

• pipeline
• data catalog
• data processing

isoc servers
xroot

     The following sewrvers are needed to allow processing of new data (older data on fermi-xrd or HPSS will not be available):

• fermi-gpfs01/02, fermi-gpfs05/06 and fermi-gpfs07/08
  These are the servers that make up the fermi xrootd gpfs space.  Each pair servers a part of the total gpfs space.
  • fermi-gpfs05 runs the xrootd server for the gpfs space
• fermilnx-v02 (redirector)
• fermilnx-v12 (redirector)

nfs

• Pretty much everything that's currently on wain031

LSF

• ~25 hosts should let us keep up

Here's what ISOC tasks need:

FASTCopy chain
--------------
wain031
fermilnx01
fermilnx02
trscron
fermilnx-v03 (Archiver)
Whatever the pipeline server runs on.
xroot servers
astore system

Web servers
-----------
tomcat01 Commons
tomcat06 rm2
tomcat09 Pipeline-II
tomcat10 FCWebView, ISOCLogging, MPWebView
TelemetryMonitor, TelemetryTableWebUI
tomcat11 DataProcessing
tomcat12 TelemetryTrending

Notes:

9/29/2017 - (TG and WK) added list of xrootd servers needed to bridge the gap in December; Started table of server names