You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Intent of this document

This document is intended to be a summary of the changes planned in migrating from glast-oracle03 and glast-oracle04 to their new homes.

 

The new servers are identical Dell R620s, and consist of:

  • 2x Intel Xeon E5-2690 2.90GHz 20M Cache 8 Cores
  • 256GB RAM
  • 10x 300GB 15k internal drives for software, configuration and logging.
  • 1x PERC 710 RAID controller
  • 2x  MD1220 Disk arrays, configured with 24x600GB 10k drives in RAID 10 

This gives us an effective configuration of 16 3Ghz cores, 256GB RAM, and 14TB+ disk space.

Migration Procedures

In conjunction with the physical database migration from glast-oracle03 and 04, we have been planning concurrently performing a reversible pipeline migration. Because of constraints on storage, it is not possible to perform a reversible migration on Oracle 10g, and Oracle 10g doesn't support SQL features we are using in the upgraded pipeline software. More information about the pipeline migration can be found here:    Pipeline Migration and Pipeline II, v1.5  

Pipeline Software  compatibility  information

The current pipeline is 1.4.2. The new pipeline will be 1.5.

  • 1.4.2 is incomatible with 1.5 schema, and 1.5 is incompatible with 1.4.2 schema.
  • 1.4.2 is compatible with 10g and 11g
  • 1.5 is compatible ONLY with 11g 

There are back-out procedures I’ve written just in case there is a need to back out of the pipeline upgrade to 1.5 and use 1.4.2 instead. Those back out procedures would be fine for the case where we turn the pipeline back on and something isn’t immediately fixable, but those back out procedures are not written in the case that we decide to drop back to 1.4.2 after a week. Basically, I keep the snapshot of the tables I’m changing and I can reinsert them for use, but I don’t have a way of updating the tables with any changes.

 

Stages

The remaining stages can be boiled down to 3 parts:

  1.  Migration of GLASTTREND
    1. Transfer of files
    2. Conversion of files
    3. Starting of the database
  2. Migration of GLAST_DP_TEST, GLASTGEN, etc...
    1. Transfer of files
    2. Conversion of files
    3. Starting of the database

Timeline

DateEvents
5/5, 11amReadiness meeting
5/5, 4pmShutdown of fastcopy, L1, pipeline. Some web applications may stay up.
5/5, 5pmRemaining jobs drained out of batch farm. GLASTTREND in read-only.
5/5, 6pmGLASTTREND transfer (1a) should be be started by now
5/5, 10pm

OPTIONAL:

Once at least 600GB has been transferred, we might be able to start conversion of some GLASTTREND files.

Not too likely to impact transfer performance, as it’s not likely to be hitting the same discs.

5/6, 6am

Transfer of GLASTTREND should be completed.

Notification that all applications will be down, including any that may have still been up after 6pm.

Transfer of remaining databases (step 2a) on glast-oracle machines should commence.

5/6, 10am

CONTINGENCY POINT:

Should the transfer (1a), but NOT conversion (1b), of GLASTTREND exceed this time limit, then we shall ABORT step 2 entirely, and proceed ONLY with step 1.

5/6, 2pm

The conversion of GLASTTREND (1b) must be more than half way finished, and the transfer of the remaining files (2a) must be more than halfway finished.

If not, we need to decide on which to proceed with based on how far along they are.

5/6, 3pm

The transfer of remaining files (2a) must be completed.

The conversion of the files (2b) should be started.

5/6, 5pm

The conversion GLASTTREND (1a) must be completed.

The conversion of the files (2b) must be completed. GLAST_DP_TEST database must be up, and the schema migrations for the pipeline will be started.

5/6, 7pm

The schema migrations should be finished.

GLASTTREND, GLASTGEN, and other databases should be up.

Applications should be started.

 

Stages with detailed steps

There was actually 3 stages, the first stage was completed Wednesday, April 30. That stage was to migrate ISOC_FLIGHT tablespace.

Since the GLASTTREND step will take an absolute minimum of 8 hours, and likely closer to 10-12, the first part will be completed overnight.

Stage 1

In Stage 1, we will migrate the Trending data (GLASTTREND account). This involves the following:

  1. Shutdown of fastcopy, L1, pipeline

  2. Let remaining jobs drained out of batch farm. GLASTTREND in read-only.

  3. Ian will Start transfer of GLASTTREND data, which is close to 2.3TB
  4. Endianness conversion will be completed.
  5. Ian verifies database configuration (file location, permissions)

Stage 2

Stage 2 will be the rest of the database migration, and may start once the transfer steps of stage 1 are completed.

  1. Take a full outage. Everything which was still connecting to glast-oracle03/04 will be shut down.
  2. Ian will start transferring the rest of the database files for all other accounts. This includes:
    1. GLAST_DP_TEST
    2. CONFIG (GLASTGEN)
    3. GLAST_ASP
    4. GLAST_RSP
    5. GLAST_ISOC
    6. ... and various other accounts.
  3. Connections to the databases in tnsnames.ora will be updated
  4. GLAST_DP_TEST will be prioritized to be transferred first.
  5. Headers will be modified for little-endianness on files which have finished transferring while other files are transferring.
  6. Upon successful transfer, which will take an estimated 4 hours:
    1. Ian will create 4 new database files to add to available space for a schema change
    2. Ian verifies database configuration (file location, permissions), the database is brought up
    3. The schema change described on the Pipeline Migration page will be performed
  7. Services will be started back up
    1. All tomcat servers should be restarted
    2. Any applications that were left running which rely on database connectivity should be restarted

 

  • No labels