Versions Compared

Old Version 2

changes.mady.by.user Brian Van Klaveren

Saved on

compared with

New Version 3

changes.mady.by.user Brian Van Klaveren

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

There are back-out procedures I’ve written just in case there is a need to back out of the pipeline upgrade to 1.5 and use 1.4.2 instead. Those back out procedures would be fine for the case where we turn the pipeline back on and something isn’t immediately fixable, but those back out procedures are not written in the case that we decide to drop back to 1.4.2 after a week. Basically, I keep the snapshot of the tables I’m changing and I can reinsert them for use, but I don’t have a way of updating the tables with any changes.

 

Stages

Due to the size of the GLASTTREND database, we plan to perform the migration in two stages over two days time.

The remaining stages can be boiled down to 3 parts:

  1.  Migration of GLASTTREND
    1. Transfer of files
    2. Conversion of files
    3. Starting of the database
  2. Migration of GLAST_DP_TEST, GLASTGEN, etc...
    1. Transfer of files
    2. Conversion of files
    3. Starting of the database

Timeline

5/5, 11am

  • Readiness  meeting

5/5, 4pm            

  •  Shutdown of fastcopy, L1, pipeline

5/5, 5pm

  • Remaining jobs drained out of batch farm. GLASTTREND in read-only.

5/5, 6pm

  • GLASTTREND transfer (1a) should be be started by now

5/5, 10pm, OPTIONAL:

  • Once at least 600GB has been transferred, we might be able to start conversion of some GLASTTREND files. Not too likely to impact transfer performance, as it’s not likely to be hitting the same discs.

5/6, 6am

  • Transfer of GLASTTREND should be completed. Notification that all applications will be down, including any that may have still been up after 6pm. Transfer of remaining databases (step 2a) on glast-oracle machines should commence.

5/6, 10am, CONTINGENCY POINT

  • Should the transfer (1a), but NOT conversion (1b), of GLASTTREND exceed this time limit, then we shall ABORT step 2 entirely, and proceed ONLY with step 1.

5/6, 2pm

  • The conversion of GLASTTREND (1b) must be more than half way finished, and should the transfer of the remaining files (2a) must be more than halfway finished. If not, we need to decide on which to proceed with based on how far along they are.

5/6, 3pm

  • The transfer of remaining files (2a) must be completed. The conversion of the files (2b) should be started.

5/6, 5pm

  • The conversion GLASTTREND (1a) must be completed. The conversion of the files (2b) must be completed. GLAST_DP_TEST database must be up, and the schema migrations for the pipeline will be started.

5/6, 7pm

  • The schema migrations should be finished. GLASTTREND, GLASTGEN, and other databases should be up. Applications should be started.

Stages with detailed steps

There was actually 3 stages, the first stage was completed Wednesday, April 30. That stage was to migrate ISOC_FLIGHT tablespace.

Since the GLASTTREND step The actual timing of the stages is somewhat arbitrary, but I’ve suggested that we actually do GLASTTREND first, although it doesn’t matter much to me or Max. Since the GLASTTREND stage will take an absolute minimum of 8 hours, and likely closer to 10-12, I think Ian and Arash might need to coordinate their schedules a bit, so I’d like to see if Ian actually has a preference on which to do firstthe first part will be completed overnight.

Stage 1

In Stage 1, we will migrate the Trending data (GLASTTREND account). This involves the following:

  1. Shutdown of fastcopy, L1, pipeline

  2. Let remaining jobs drained out of batch farm. GLASTTREND in read-only

    Put all connections to GLASTTREND to read-onlyThis will mean L1Proc ingest* scripts, and any other scripts which report to GLASTTREND, will fail

    .

  3. Ian will Start transfer of GLASTTREND data, which is close to 2.3TB
  4. Over an 8-10 hour period, the data will be transferred and Ian will modify headers for little-endianness.
  5. Once the data has transferred, connections in the tnsnames.ora file will be modified again, and accounts will no longer be in read-only
  6. We will roll back the ingest* process instances. From this point on, any L1 tasks will communicate directly with the new database for Trending data.

It's estimated that up-to-date trending data may be unavailable for a 12 hour period.

  1. Endianness conversion will be completed.
  2. Ian verifies database configuration (file location, permissions)

Stage 2

Stage 2 will be the rest of the database migration, including the pipeline migrationand may start once the transfer steps of stage 1 are completed.

  1. Take a full outage. Everything which connects was still connecting to glast-oracle03/04 will be shut down.
  2. Ian will start transferring the rest of the database files for all other accounts. This includes:
    1. GLAST_DP_TEST
    2. CONFIG (GLASTGEN)
    3. GLAST_ASP
    4. GLAST_RSPISOC_FLIGHT
    5. GLAST_ISOC
    6. ... and various other accounts.
  3. Connections to the databases in tnsnames.ora will be updated
  4. GLAST_DP_TEST will be prioritized to be transferred first.
  5. Headers will be modified for little-endianness on files which have finished transferring while other files are transferring.
  6. Upon successful transfer, which will take an estimated 4 hours:
    1. Ian will create 4 new database files to add to available space for a schema change
    2. Ian verifies database configuration (file location, permissions), the database is brought up
    3. The schema change described on the Pipeline Migration page will be performed
    Connections to the databases in tnsnames.ora will be updated
  7. Services will be started back up
    1. All tomcat servers should be restarted
    2. Any applications that were left running which rely on database connectivity should be restarted

...