Would you like to help test Confluence and Jira updates? Please fill out the form here: https://forms.gle/5TbSqvYYCnTyyxSN7
This document is intended to be a summary of the changes planned in migrating from glast-oracle03 and glast-oracle04 to their new homes.
The new servers are identical Dell R620s, and consist of:
This gives us an effective configuration of 16 3Ghz cores, 256GB RAM, and 14TB+ disk space.
In conjunction with the physical database migration from glast-oracle03 and 04, we have been planning concurrently performing a reversible pipeline migration. Because of constraints on storage, it is not possible to perform a reversible migration on Oracle 10g, and Oracle 10g doesn't support SQL features we are using in the upgraded pipeline software. More information about the pipeline migration can be found here: Pipeline Migration and Pipeline II, v1.5
Pipeline Software compatibility information
The current pipeline is 1.4.2. The new pipeline will be 1.5.
There are back-out procedures I’ve written just in case there is a need to back out of the pipeline upgrade to 1.5 and use 1.4.2 instead. Those back out procedures would be fine for the case where we turn the pipeline back on and something isn’t immediately fixable, but those back out procedures are not written in the case that we decide to drop back to 1.4.2 after a week. Basically, I keep the snapshot of the tables I’m changing and I can reinsert them for use, but I don’t have a way of updating the tables with any changes.
The remaining stages can be boiled down to 3 parts:
Any databases on glast-oracle machines that are not on this list should still be transferred, but they are at the lowest priority.
|5/5, 11am||Readiness meeting|
|5/5, 4pm||Shutdown of L1, pipeline. Disable pipeline job submission from FASTCopy. Some web applications may stay up.|
|5/5, 5pm||Remaining jobs drained out of batch farm. GLASTTREND in read-only.|
|5/5, 6pm||GLASTTREND transfer (1a) should be be started by now|
Transfer of GLASTTREND should be completed.
Notification that all applications will be down, including any that may have still been up after 6pm.
Transfer of remaining databases (step 2a) on glast-oracle machines should commence.
Should the transfer (1a), but NOT conversion (1b), of GLASTTREND exceed this time limit, then we shall ABORT step 2 entirely, and proceed ONLY with step 1.
The conversion of GLASTTREND (1b) must be more than half way finished, and the transfer of the remaining files (2a) must be more than halfway finished.
If not, we need to decide on which to proceed with based on how far along they are.
The transfer of remaining files (2a) must be completed.
The conversion of the files (2b) should be started.
The conversion GLASTTREND (1a) must be completed.
The conversion of the files (2b) must be completed. GLAST_DP_TEST database must be up, and the schema migrations for the pipeline will be started.
The schema migrations should be finished.
GLASTTREND, GLASTGEN, and other databases should be up.
Applications should be started.
Stages with detailed steps
There was actually 3 stages, the first stage was completed Wednesday, April 30. That stage was to migrate ISOC_FLIGHT tablespace.
Since the GLASTTREND step will take an absolute minimum of 8 hours, and likely closer to 10-12, the first part will be completed overnight.
In Stage 1, we will migrate the Trending data (GLASTTREND account). This involves the following:
Shutdown of L1, pipeline. Disabling of pipeline job submission from FASTCopy automation.
Let remaining jobs drained out of batch farm. GLASTTREND in read-only.
Stage 2 will be the rest of the database migration, and may start once the transfer steps of stage 1 are completed.