You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Intent of this document

This document is intended to be a summary of the changes planned in migrating from glast-oracle03 and glast-oracle04 to their new homes.

 

The new servers are identical Dell R620s, and consist of:

  • 2x Intel Xeon E5-2690 2.90GHz 20M Cache 8 Cores
  • 256GB RAM
  • 10x 300GB 15k internal drives for software, configuration and logging.
  • 1x PERC 710 RAID controller
  • 2x MD1220 Disk arrays, configured with 24x600GB 10k drives in RAID 10 

This gives us an effective configuration of 16 3Ghz cores, 256GB RAM, and 14TB+ disk space.

Migration Procedures

In conjunction with the physical database migration from glast-oracle03 and 04, we have been planning concurrently performing a reversible pipeline migration. Because of constraints on storage, it is not possible to perform a reversible migration on Oracle 10g, and Oracle 10g doesn't support SQL features we are using in the upgraded pipeline software. More information about the pipeline migration can be found here:  Pipeline Migration and Pipeline II, v1.5 

Pipeline Software compatibility information

The current pipeline is 1.4.2. The new pipeline will be 1.5.

  • 1.4.2 is incomatible with 1.5 schema, and 1.5 is incompatible with 1.4.2 schema.
  • 1.4.2 is compatible with 10g and 11g
  • 1.5 is compatible ONLY with 11g 

There are back-out procedures I’ve written just in case there is a need to back out of the pipeline upgrade to 1.5 and use 1.4.2 instead. Those back out procedures would be fine for the case where we turn the pipeline back on and something isn’t immediately fixable, but those back out procedures are not written in the case that we decide to drop back to 1.4.2 after a week. Basically, I keep the snapshot of the tables I’m changing and I can reinsert them for use, but I don’t have a way of updating the tables with any changes. 

Stages

Due to the size of the GLASTTREND database, we plan to perform the migration in two stages over two days time.

The actual timing of the stages is somewhat arbitrary, but I’ve suggested that we actually do GLASTTREND first, although it doesn’t matter much to me or Max. Since the GLASTTREND stage will take an absolute minimum of 8 hours, and likely closer to 10-12, I think Ian and Arash might need to coordinate their schedules a bit, so I’d like to see if Ian actually has a preference on which to do first.

Stage 1

In Stage 1, we will migrate the Trending data (GLASTTREND account). This involves the following:

  1. Put all connections to GLASTTREND to read-only
    1. This will mean L1Proc ingest* scripts, and any other scripts which report to GLASTTREND, will fail.
  2. Ian will Start transfer of GLASTTREND data, which is close to 2.3TB
  3. Over an 8-10 hour period, the data will be transferred and Ian will modify headers for little-endianness.
  4. Once the data has transferred, connections in the tnsnames.ora file will be modified again, and accounts will no longer be in read-only
  5. We will roll back the ingest* process instances. From this point on, any L1 tasks will communicate directly with the new database for Trending data.

It's estimated that up-to-date trending data may be unavailable for a 12 hour period.

Stage 2

Stage 2 will be the rest of the database migration, including the pipeline migration

  1. Take a full outage. Everything which connects to glast-oracle03 will be shut down.
  2. Ian will start transferring the rest of the database files for all other accounts. This includes:
    1. GLAST_DP_TEST
    2. CONFIG (GLASTGEN)
    3. GLAST_ASP
    4. GLAST_RSP
    5. ISOC_FLIGHT
    6. GLAST_ISOC
    7. ... and various other accounts.
  3. GLAST_DP_TEST will be prioritized to be transferred first.
  4. Headers will be modified for little-endianness on files which have finished transferring while other files are transferring.
  5. Upon successful transfer, which will take an estimated 4 hours:
    1. Ian will create 4 new database files to add to available space for a schema change
    2. The schema change described on the Pipeline Migration page will be performed
  6. Connections to the databases in tnsnames.ora will be updated
  7. Services will be started back up
    1. All tomcat servers should be restarted
    2. Any applications that were left running which rely on database connectivity should be restarted

 

  • No labels