Versions Compared

Old Version 16

changes.mady.by.user Maria Elena Monzani

Saved on

compared with

New Version 17

changes.mady.by.user Maria Elena Monzani

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Aug 16, 2012 - site wide power outage.

  • From John: everything except the servers on the generator will go down. Building 50 is supposed to be the first (or one of the first) buildings brought back up. Power goes off at 5:30 am 8/16. We could have power restrored by 6:30am. Bring up would begin after that, most services back in 2-4 hours. NOTE, however, we tentatively plan to start taking machines down at 17:30 the night before (Aug 15). So we are talking about a ~16 hour outage, if things go well.
  • Update: some of the Power Distribution Units (PDUs) are old and need inspection. Apparently this needs to be done after the outage. Each inspection is estimated to take ~45 minutes, and Boris (et al.) propose inspecting four: PDU 40, 41, 42, 44. Each of these powers multiple servers (mostly file servers), and it turns out FGST has servers on all four. We've agreed to wait until the inspection is over before restarting the pipeline. A few extra hours don't make any difference for this.
  • We have asked FOT and FSSC to buffer the regular FastCopy data deliveries to the ISOC starting at 4pm on Aug 15, to have time to clear our buffers. We will inform them when we are back online and ready to accept data.

July 11, 2012

  • 11:00am - 1:00pm: Replacing a bad fan on sulky34. Since that server holds the LAT raw data, FASTCopy ingestion will be stopped about an hour beforehand to let the pipeline drain.
    Also, the remaining databases will be migrated off of glastlnx01/02 onto mysql-node01.

...