Versions Compared

Old Version 7

changes.mady.by.user Brian Van Klaveren

Saved on

compared with

New Version Current

changes.mady.by.user Warren Focke

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

There are back-out procedures I’ve written just in case there is a need to back out of the pipeline upgrade to 1.5 and use 1.4.2 instead. Those back out procedures would be fine for the case where we turn the pipeline back on and something isn’t immediately fixable, but those back out procedures are not written in the case that we decide to drop back to 1.4.2 after a week. Basically, I keep the snapshot of the tables I’m changing and I can reinsert them for use, but I don’t have a way of updating the tables with any changes.

...

Migration

The remaining stages can be boiled down to 3 parts:

  1.  Migration of GLASTTREND
    1. Transfer of files
    2. Conversion of files
    3. Starting of the database
  2. Migration of GLAST_DP_TEST, GLASTGEN, etc...
    1. Transfer of files
    2. Conversion of files
    3. Starting of the database

Databases to be transferred, ordered by priority

  1. GLASTTREND
  2. GLASTGEN
  3. GLAST_DP_TEST
  4. LAT (this was just a user, not a tablespace)
  5. GLAST_ASP
  6. GLAST_RSP

Any databases on glast-oracle machines that are not on this list should still be transferred, but they are at the lowest priority.

Timeline

DateChecklistsEvents
5/5, 11am Readiness meeting
5/5, 4pm
  •  Pipeline shutdown
  •  L1 shutdown
  •  FastCopy shut down
Shutdown of L1, pipeline. Disable pipeline job submission from FASTCopy. Some web applications may stay up.
5/5, 5pm
  •  Ian cleared to proceed
Remaining jobs drained out of batch farm. GLASTTREND in read-only.
5/5, 6pm
  •  GLASTTREND transfer started
GLASTTREND transfer (1a) should be be started by now
5/5, 10pm 

OPTIONAL:

Once at least 600GB has been transferred, we might be able to start conversion of some GLASTTREND files.

Not too likely to impact transfer performance, as it’s not likely to be hitting the same discs.

5/6, 6am
  •  GLASTTREND transfer done
  •  GLASTTREND conversion started
  •  Everything on glast-oracle machines in read-only
  •  GLASTTREND transfer done
  •  GLASTTREND conversion started
  •  GLASTGEN transfer started
  •  GLAST_DP_TEST transfer started

Transfer of GLASTTREND should be completed.

Notification that all applications will be down, including any that may have still been up after 6pm.

Transfer of remaining databases (step 2a) on glast-oracle machines should commence.

5/6, 10am
  •  GLASTTREND more than halfway converted

CONTINGENCY POINT:

Should the transfer (1a), but NOT conversion (1b), of GLASTTREND exceed this time limit, then we shall ABORT step 2 entirely, and proceed ONLY with step 1.

5/6, 2pm
  •  Transfer of any remaining databases started
  •  tnsnames.ora updated

The conversion of GLASTTREND (1b) must be more than half way finished, and the transfer of the remaining files (2a) must be more than halfway finished.

If not, we need to decide on which to proceed with based on how far along they are.

5/6, 3:30pm
  •  Transfer of all databases finished.
    Including: GLASTGEN, GLAST_DP_TEST, GLAST_ASP,
    GLAST_RSP, LAT, GLAST_CAL, etc...

The transfer of remaining files (2a) must be completed.

The conversion of the files (2b) should be started.

5/6, 5pm
  •  Conversion of GLASTTREND completed
  •  GLASTTREND is up
  •  Conversion of GLASTGEN completed
  •  Conversion of GLAST_DP_TEST completed
  •  Conversion of the rest of databases completed
  •  Pipeline Schema migrations started

The conversion GLASTTREND (1a) must be completed.

The conversion of the files (2b) must be completed. GLAST_DP_TEST database must be up, and the schema migrations for the pipeline will be started.

5/6, 7pm
  •  Pipeline Schema migrations finished
  •  Pipeline back up
  •  L1 back up
  •  FastCopy back up
  •  Web Applications restarted

The schema migrations should be finished.

GLASTTREND, GLASTGEN, and other databases should be up.

Applications should be started.

...

  1. Take a full outage. Everything which was still connecting to glast-oracle03/04 will be shut down.
  2. Ian will start transferring the rest of the database files for all other accounts. This includes:
    1. GLAST_DP_TEST
    2. CONFIG (GLASTGEN)
    3. GLAST_ASP
    4. GLAST_RSPGLAST_ISOC
    5. LAT
    6. ... and various other accounts.
  3. Connections to the databases in tnsnames.ora will be updated
  4. GLAST_DP_TEST will be prioritized to be transferred first.
  5. Headers will be modified for little-endianness on files which have finished transferring while other files are transferring.
  6. Upon successful transfer, which will take an estimated 4 hours:
    1. Ian will create 4 new database files to add to available space for a schema change
    2. Ian verifies database configuration (file location, permissions), the database is brought up
    3. The schema change described on the Pipeline Migration page will be performed
  7. Services will be started back up
    1. All tomcat servers should be restarted
    2. Any applications that were left running which rely on database connectivity should be restarted

...