Intent of this document

This document is intended to be a summary of the changes planned in migrating from glast-oracle03 and glast-oracle04 to their new homes.

The new servers are identical Dell R620s, and consist of:

  • 2x Intel Xeon E5-2690 2.90GHz 20M Cache 8 Cores
  • 256GB RAM
  • 10x 300GB 15k internal drives for software, configuration and logging.
  • 1x PERC 710 RAID controller
  • 2x  MD1220 Disk arrays, configured with 24x600GB 10k drives in RAID 10 

This gives us an effective configuration of 16 3Ghz cores, 256GB RAM, and 14TB+ disk space.

Migration Procedures

In conjunction with the physical database migration from glast-oracle03 and 04, we have been planning concurrently performing a reversible pipeline migration. Because of constraints on storage, it is not possible to perform a reversible migration on Oracle 10g, and Oracle 10g doesn't support SQL features we are using in the upgraded pipeline software. More information about the pipeline migration can be found here:    Pipeline Migration and Pipeline II, v1.5  

Pipeline Software  compatibility  information

The current pipeline is 1.4.2. The new pipeline will be 1.5.

  • 1.4.2 is incomatible with 1.5 schema, and 1.5 is incompatible with 1.4.2 schema.
  • 1.4.2 is compatible with 10g and 11g
  • 1.5 is compatible ONLY with 11g 

There are back-out procedures I’ve written just in case there is a need to back out of the pipeline upgrade to 1.5 and use 1.4.2 instead. Those back out procedures would be fine for the case where we turn the pipeline back on and something isn’t immediately fixable, but those back out procedures are not written in the case that we decide to drop back to 1.4.2 after a week. Basically, I keep the snapshot of the tables I’m changing and I can reinsert them for use, but I don’t have a way of updating the tables with any changes.

Migration

The remaining stages can be boiled down to 3 parts:

  1.  Migration of GLASTTREND
    1. Transfer of files
    2. Conversion of files
    3. Starting of the database
  2. Migration of GLAST_DP_TEST, GLASTGEN, etc...
    1. Transfer of files
    2. Conversion of files
    3. Starting of the database

Databases to be transferred, ordered by priority

  1. GLASTTREND
  2. GLASTGEN
  3. GLAST_DP_TEST
  4. LAT (this was just a user, not a tablespace)
  5. GLAST_ASP
  6. GLAST_RSP

Any databases on glast-oracle machines that are not on this list should still be transferred, but they are at the lowest priority.

Timeline

DateChecklistsEvents
5/5, 11am Readiness meeting
5/5, 4pm
  • Pipeline shutdown
  • L1 shutdown
  • FastCopy shut down
Shutdown of L1, pipeline. Disable pipeline job submission from FASTCopy. Some web applications may stay up.
5/5, 5pm
  • Ian cleared to proceed
Remaining jobs drained out of batch farm. GLASTTREND in read-only.
5/5, 6pm
  • GLASTTREND transfer started
GLASTTREND transfer (1a) should be be started by now
5/5, 10pm 

OPTIONAL:

Once at least 600GB has been transferred, we might be able to start conversion of some GLASTTREND files.

Not too likely to impact transfer performance, as it’s not likely to be hitting the same discs.

5/6, 6am
  • Everything on glast-oracle machines in read-only
  • GLASTTREND transfer done
  • GLASTTREND conversion started
  • GLASTGEN transfer started
  • GLAST_DP_TEST transfer started

Transfer of GLASTTREND should be completed.

Notification that all applications will be down, including any that may have still been up after 6pm.

Transfer of remaining databases (step 2a) on glast-oracle machines should commence.

5/6, 10am
  • GLASTTREND more than halfway converted

CONTINGENCY POINT:

Should the transfer (1a), but NOT conversion (1b), of GLASTTREND exceed this time limit, then we shall ABORT step 2 entirely, and proceed ONLY with step 1.

5/6, 2pm
  • Transfer of any remaining databases started
  • tnsnames.ora updated

The conversion of GLASTTREND (1b) must be more than half way finished, and the transfer of the remaining files (2a) must be more than halfway finished.

If not, we need to decide on which to proceed with based on how far along they are.

5/6, 3:30pm
  • Transfer of all databases finished.
    Including: GLASTGEN, GLAST_DP_TEST, GLAST_ASP,
    GLAST_RSP, LAT, GLAST_CAL, etc...

The transfer of remaining files (2a) must be completed.

The conversion of the files (2b) should be started.

5/6, 5pm
  • Conversion of GLASTTREND completed
  • GLASTTREND is up
  • Conversion of GLASTGEN completed
  • Conversion of GLAST_DP_TEST completed
  • Conversion of the rest of databases completed
  • Pipeline Schema migrations started

The conversion GLASTTREND (1a) must be completed.

The conversion of the files (2b) must be completed. GLAST_DP_TEST database must be up, and the schema migrations for the pipeline will be started.

5/6, 7pm
  • Pipeline Schema migrations finished
  • Pipeline back up
  • L1 back up
  • FastCopy back up
  • Web Applications restarted

The schema migrations should be finished.

GLASTTREND, GLASTGEN, and other databases should be up.

Applications should be started.

 

Stages with detailed steps

There was actually 3 stages, the first stage was completed Wednesday, April 30. That stage was to migrate ISOC_FLIGHT tablespace.

Since the GLASTTREND step will take an absolute minimum of 8 hours, and likely closer to 10-12, the first part will be completed overnight.

Stage 1

In Stage 1, we will migrate the Trending data (GLASTTREND account). This involves the following:

  1. Shutdown of L1, pipeline. Disabling of pipeline job submission from FASTCopy automation.

  2. Let remaining jobs drained out of batch farm. GLASTTREND in read-only.

  3. Ian will Start transfer of GLASTTREND data, which is close to 2.3TB
  4. Endianness conversion will be completed.
  5. Ian verifies database configuration (file location, permissions)

Stage 2

Stage 2 will be the rest of the database migration, and may start once the transfer steps of stage 1 are completed.

  1. Take a full outage. Everything which was still connecting to glast-oracle03/04 will be shut down.
  2. Ian will start transferring the rest of the database files for all other accounts. This includes:
    1. GLAST_DP_TEST
    2. GLASTGEN
    3. GLAST_ASP
    4. GLAST_RSP
    5. LAT
    6. ... and various other accounts.
  3. Connections to the databases in tnsnames.ora will be updated
  4. GLAST_DP_TEST will be prioritized to be transferred first.
  5. Headers will be modified for little-endianness on files which have finished transferring while other files are transferring.
  6. Upon successful transfer, which will take an estimated 4 hours:
    1. Ian will create 4 new database files to add to available space for a schema change
    2. Ian verifies database configuration (file location, permissions), the database is brought up
    3. The schema change described on the Pipeline Migration page will be performed
  7. Services will be started back up
    1. All tomcat servers should be restarted
    2. Any applications that were left running which rely on database connectivity should be restarted

 

  • No labels