Page History

...

• we are trying to collect information about upcoming Fermi computing outages (disks, oracle, network) to improve planning
• when planning an outage, please send an email to datalist and write the description here (including requested duration and preferred timeframe)
• we will try to combine outages as much as possible, in order to maximize uptime for time-critical services (FASTCopy, pipeline, etc.)
• once the plan is finalized, don't forget to send a message to glast-outage and the collaboration (if applicable)

Dec 2019 Power Outage (Fermi)

Dec 2017 Power Outrage (Fermi)

Upcoming outage requests

• Outage of mysql-node03 to move to HA rack.

Feb 03, 2014 - Oracle and OS patching (ghost vulnerability patches)

• Outage of FASTCopy starting at 9:00am, reboot of FASTCopy machines
• Oracle OS reboot and patching starting at 10:00am
• Reboot Fermi linux xrootd servers and fermilnx machines

Feb 11, 2014 - Oracle and OS patching; also retirement of various glastlnx machines

• 10am  - duration is likely several hours
• This outage affects all NFS servers (wains), including user disk as well as xroot servers.
• Expect interruptions in all Fermi services as they are moved from old glastlnx -> new fermilnx machines

Dec 11, 2013 - Oracle server battery replacement

• 10am - glast-oracle03 to have battery replaced in storage array.  Expected outage duration: 30m

Dec 4, 2013 - OS Patching and re-IP'ing

• 10am - all Fermi wain-class servers will be rebooted for OS patching.  
• 10am - glast-oracle03/04 will be rebooted for OS patching.
• At the same time, 16 wains will have new IP addresses assigned in anticipation of retiring old network switches and reconfiguring the network in January 2014.
• Three wains will be physically relocated to consolidate rack space

Oct 2, 2013 - ISOC logging gateways to be shut down

...

Feb 13, 2013 - Oracle password change

• 2 PM. Semi-annual password change for Fermi accounts:

  No Format
  Oracle Instance Oracle Account Password Expires --------------- ----------------------- ----------------- GLASTDEV GLAST_ISOC 14-FEB-2013 GLASTDEV ISOC_NIGHTLY 14-FEB-2013 GLASTDEV ISOC_TEST 14-FEB-2013 GLASTP GLAST_CAL 14-FEB-2013 GLASTP GLAST_ISOC 14-FEB-2013 GLASTP ISOC_FLIGHT 14-FEB-2013

...

• 10am - 11:30am: migrating calib* and mood* databases from glastlnx01/02 to mysql-node03

May 10 2012

...

• \[10am-12:30pm\] Oracle quarterly update. This will affect pipeline, data catalog, flight operations and any other databases on the main Fermi Oracle server.

• Wiki Markup\[10am-12:30pm\] xroot server reboot for OS upgrade. This will affect all 36 of the wain (Solaris) xroot servers.unmigrated-wiki-markup

• \[10am-12:30pm\] Fermi USER DISK (wain006) reboot for OS upgrade.

• Wiki Markup\[9am-3pm\] xroot file server move. This will affect only two xroot servers: wain070 and wain071.unmigrated-wiki-markup

• \[9am-3pm\] NFS file server move. This will affect the following servers which will be unplugged and physically moved to new rack space
  in building 50: sulky33, sulky34, sulky35, sulky36