Notes on VMWare Cluster
The VMWare cluster, noted as fermi-vmclust, consists of 4 hypervisors. Two of the machines are on HA, and two are not. A VM may migrate between these two hypervisors on demand- the machines have 128GB of memory and two hypervisors are enough to run all the VMs without oversubscription of memory.
The VMWare is cluster managed by OCIO Platforms team, and they can migrate VMs on demand.
Note: OCIO should be contacted about setting up occasional VM snapshots, so that a VM may always be brought up even if a hypervisor dies.
Table of LISOC Tasks and Services
| Function/Service | Sub-Functions | Needed Servers | Needed Databases | Needed File Systems | Other Needs | Needed During Shutdown? | Available During Shutdown? |
|---|---|---|---|---|---|---|---|
| Mission Planning, LAT Configurations | FastCopy | fermilnx01 and | TCDB | AFS | Fermi LAT Portal: Timeline Webview; Confluence, JIRA, Mission Planning s/w, FastCopy Monitoring Sharepoint (reference for PROCs and Narrative Procedures for commanding in case of anomalies) | yes | |
| Real Time Telemetry Monitoring | fermilnx01 and fermilnx02 | spread Fermi LAT Portal: Real Time Telemetry, Telemetry Monitor | during anomalies | ||||
| Logging | fermilnx01 and fermilnx02 | TCDB | Fermi LAT Portal: Log Watcher | yes | |||
| Trending | TCDB | Fermi LAT Portal: Telemetry Trending | yes | ||||
| L0 File Ingest and Archive | FastCopy | L0 Archive | yes | ||||
| Data Gap Checking and Reporting | FastCopy | fermilnx01 and fermilnx02 | L0 Archive | yes, continuously | |||
| L1 processing | pipeline | SLAC Farm | Data Catalog | Fermi LAT Portal: Pipeline, Data Processing | yes | ||
| L1 Data Quality Monitoring | Fermi LAT Portal, Telemetry Trending | ||||||
| L1 delivery | FastCopy | fermilnx01 and fermilnx02 | Data Catalog | yes | |||
| L2 processing (ASP) and Delivery | FastCopy | fermilnx01 and fermilnx02 | Data Catalog | Fermi LAT Portal: Pipeline, Data Processing | daily, weekly |
| VM or Service | Category† | server | function |
|---|---|---|---|
| xrootd | XC | fermi-gpfs01 fermi-gpfs02 fermi-gpfs05 fermi-gpfs06 fermi-gpfs07 fermi-gpfs08 | xrootd server and storage |
| fermilnx-v12 | XC/HA | fermi-vmclust | xrootd redirector |
| GPFS | XC | fermi-gpfs03 fermi-gpfs04 | Fermi NFS/GPFS storage |
| GPFS/NFS bridge | XC | fermi-cnfs01 fermi-cnfs02 | Fermi NFS storage access |
| HA | staas-gpfs50 staas-gpfs51 | Critical ISOC NFS storage | |
| HA | fermilnx01 | LAT config, fastcopy and real-time telemetry | |
| HA | fermilnx02 | LAT config, fastcopy and real-time telemetry | |
| fermilnx-v03 | XC/HA | fermi-vmclust | archiver |
| HA | fermi-oracle03 | oracle primary | |
| XC | fermi-oracle04 | oracle secondary | |
| mysql-node03 | HA | mysql05 mysql06 | calibration, etc. DB |
| XC | 400 cores | (50 "hequ" equivalents) batch hosts for LISOC queues={express,short,medium,long,glastdataq} users={glast,lsstsim,lsstprod,glastmc,glastraw} | |
| XC | 200 cores | (25 "hequ" equivalents) batch hosts for Science Pipelines | |
| fermilnx-v02 | XC/HA | fermi-vmclust[1] | xrootd redirector |
| fermilnx-v07/tomcat01 | XC/HA | fermi-vmclust | Commons, Group manager |
| fermilnx-v16/tomcat06 | XC/HA | fermi-vmclust | rm2 |
| fermilnx-v05/tomcat08 | XC/HA | fermi-vmclust | dataCatalog |
| fermilnx-v17/tomcat09 | XC/HA | fermi-vmclust | Pipeline-II |
| fermilnx-v15/pipeline-mail01 | XC/HA | fermi-vmclust | Pipeline-II email server |
| fermilnx-v18/tomcat10 | XC/HA | fermi-vmclust | FCWebView, ISOCLogging, MPWebView TelemetryMonitor, TelemetryTableWebUI |
| fermilnx-v10/tomcat11 | XC/HA | fermi-vmclust | DataProcessing |
| fermilnx-v11/tomcat12 | XC/HA | fermi-vmclust | TelemetryTrending |
| astore-new (HPSS) | NC | (non-Fermi server) | FastCopy data archive **We have been granted a temporary quota increase of 1 TB on /nfs/farm/g/glast/u23, which has allowed this item to become "NC"** |
| trscron | HA | (non-Fermi server) | tokenized cron |
| lnxcron | HA | (non-Fermi server) | cron |
| (farm manager, etc.) | XC | (non-Fermi server) | LSF management |
| HA | yfs01/NN (non-Fermi) | basically all of AFS | |
| JIRA | HA | (non-Fermi server) | issue tracking (HA as of 10/20/2017) |
| XC | rhel6-64 | public login nodes (a small number is needed for interactive access) | |
[1] fermi-vmclust is the VMWare cluster.
† Equipment categories
Category | Machine status |
|---|---|
| NC | non-critical for entire 16-day shutdown period |
| XC | experiment critical but not in H.A. rack, only a few, short outages acceptable |
| HA | high-availability (continuous operation) |