Page History
...
| Function/Service | Sub-Functions | Needed Servers | Needed Databases | Needed File Systems | Other Needs | Needed During Shutdown? | Available During Shutdown? |
|---|---|---|---|---|---|---|---|
| Mission Planning, LAT Configurations | FastCopy | fermilnx01 and | TCDB | AFS | Fermi LAT Portal: Timeline Webview; Confluence, JIRA, Mission Planning s/w, FastCopy Monitoring Sharepoint (reference for PROCs and Narrative Procedures for commanding in case of anomalies) | yes | |
| Real Time Telemetry Monitoring | fermilnx01 and fermilnx02 | spread Fermi LAT Portal: Real Time Telemetry, Telemetry Monitor | during anomalies | ||||
| Logging | fermilnx01 and fermilnx02 | TCDB | Fermi LAT Portal: Log Watcher | yes | |||
| Trending | TCDB | Fermi LAT Portal: Telemetry Trending | yes | ||||
| L0 File Ingest and Archive | FastCopy | L0 Archive | yes | ||||
| Data Gap Checking and Reporting | FastCopy | fermilnx01 and fermilnx02 | L0 Archive | yes, continuously | |||
| L1 processing | pipeline | SLAC Farm | Data Catalog | Fermi LAT Portal: Pipeline, Data Processing | yes | ||
| L1 Data Quality Monitoring | Fermi LAT Portal, Telemetry Trending | ||||||
| L1 delivery | FastCopy | fermilnx01 and fermilnx02 | Data Catalog | yes | |||
| L2 processing (ASP) and Delivery | FastCopy | fermilnx01 and fermilnx02 | Data Catalog | Fermi LAT Portal: Pipeline, Data Processing | daily, weekly |
The following table of servers must remain powered up and operational for Fermi Level 1
...
and critical Science Pipelines to function.
Fermi has requested that all VMs be relocated (at least temporarily) to the two H.A. hypervisor machines, thus some of the tasks listed below are no longer relevant.
- Confirm current H.A. rack occupants. spreadsheet from Christian Pama
Old (2017) spreadsheet here - (thanks Shirley!) Confirm the VM-master for a given VM. Use the 'node' command, e.g., $ node -whereis fermilnx-v12 (obsolete)
- Confirm the tomcat <-> service associations. Table here.
- Confirm the tomcat-VM associations in this table. Use the 'node' command, e.g., $ node -whereis glast-tomcat01
...
| Category† | server | VM/service | function |
|---|---|---|---|
| XC | fermi-gpfs01 fermi-gpfs02 fermi-gpfs05 fermi-gpfs06 fermi-gpfs07 fermi-gpfs08 | xrootd server and storage | |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v02 | xrootd redirector |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v12 | xrootd redirector |
| XC | fermi-gpfs03 fermi-gpfs04 | GPFS | Fermi NFS/GPFS storage |
| XC | fermi-cnfs01 fermi-cnfs02 | GPFS/NFS bridge | Fermi NFS storage access |
| HA | staas-gpfs50 staas-gpfs51 | Critical ISOC NFS storage | |
| HA | fermilnx01 | LAT config, fastcopy and real-time telemetry | |
| HA | fermilnx02 | LAT config, fastcopy and real-time telemetry | |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v03 | archiver |
| HA | fermi-oracle03 | oracle primary | |
| XC | fermi-oracle04 | oracle secondary | |
| HA | mysql05 mysql06 | mysql-node03 | calibration, etc. DB |
| XC | 400 cores | (25 "hequ" equivalents) batch hosts for LISOC queues={express,short,medium,long,glastdataq} users={glast,lsstsim,lsstprod,glastmc,glastraw} | |
| XC | 200 cores | (12.5 "hequ" equivalents) batch hosts for Science Pipelines | |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v07/tomcat01 | Commons, Group manager |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v16/tomcat06 | rm2 |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v05/tomcat08 | dataCatalog |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v17/tomcat09 | Pipeline-II |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v15/pipeline-mail01 | Pipeline-II email server |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v18/tomcat10 | FCWebView, ISOCLogging, MPWebView TelemetryMonitor, TelemetryTableWebUI |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v10/tomcat11 | DataProcessing |
| XC/HA | fermi-vmclust01/02/03/04 | fermilnx-v11/tomcat12 | TelemetryTrending |
| NC | (non-Fermi server) | astore-new (HPSS) | FastCopy data archive **We have arranged a temporary quota increase of 1 TB on /nfs/farm/g/glast/u23, which has allowed this item to become "NC"** |
| HA | (non-Fermi server) | trscron | tokenized cron |
| HA | (non-Fermi server) | lnxcron | cron |
| XC | (non-Fermi server) | (farm manager, etc.) | LSF management |
| HA | yfs01/NN (non-Fermi) | basically all of AFS | |
| HA | (non-Fermi server) | JIRA | issue tracking (HA as of 10/20/2017) |
...
Category | Machine status |
|---|---|
| NC | non-critical for entire 16-day shutdown period |
| XC | experiment critical but not in H.A. rack, only a few, short outages acceptable |
| HA | high-availability (continuous operation) |
Total non-HA machines to receive emergency power:
The services for L1:
oracle
...
Overview
Content Tools