Migration from the old server to new server, 10g to 11g
Distilled Steps (Estimated 30 work days to completion, 6 calendar weeks)
- Take outage, Migrate pipeline and datacat dev database to new server
- Test out new server on Pipeline-II 1.4
- Test changes on new Pipeline-II 1.5 branch
- Take outage, perform pipeline and datacat prod TEST migration
- Test out temporary trending database, L1Proc at scale
- After Pipeline/datacat prod migration TEST, schedule actual migration
- Run migrated production pipeline on temporary trending table space for a few days
- After the trending database is migrated to the new server, update it with temporary trending values
- Swap to permanent trending database
Detailed migration of GLAST_DP_TEST tablespace (pipeline and datacatalog tables)
- Requires transfer of .dbf (database files) and endian change. (2 days prep, 1 day exec)
- Targeting Pipeline/datacat dev database
- Requires the database files to be locked, so no updates are allowed to the database
- For pipeline/datacat-dev, shouldn't be more than an hour or so
- For pipeline/dacatacat-prod, size is somewhere around 200GB I believe
- Assuming 20MB/sec transfer off of glast-oracle03, this will require a 3 hour outage
- Test current pipeline on new database at scale (3 days)
- Verify that this will work with a new pipeline dev-test pipeline instance
- Test that it works at scale with 10,000 simulated parallel jobs
- Try to run an L1Proc dev job, may be a bit difficult
- Test performance changes to database structure for pipeline (2 days, 3 days, 3 days, 5 days)
- Make 1.5 branch of Pipeline-II
- Enable new partitioning features available in 11g for Stream table
- Attempt to enable those for ProcessInstance table
- Normalize ProcessInstance table to BatchProcessInstance, ProcessInstance
- If normalization takes too long for production tables (more than an hour), we will perform initial migration targeting pipeline 1.4 branch, and then perform migration to 1.5 online at a later date
- Verify changes
- Make 1.5 release of Pipeline II
- Make migration scripts
- Migrate SRS pipeline with scripts, verify again 1.5 release of pipeline
- SRS and pipeline-dev should be migrated to 1.5, new tables at the end of this
- Create Temporary Trending database tables on new server (3 days)
- Attempt to verify L1Proc on the dev/1.5 server works with this
- Plan outage of pipeline/datacat prod for production migration TEST
- Again, estimated 3 hours for transfer of roughly 200GB.
- If the database is much larger than 200GB, the outage should scale at an estimated rate of 20MB/s transfer
- Repeat steps 1-3 for pipeline prod database migration test (3 days)
- Verify L1Proc production will write to new temporary trending database
- Once pipeline production migration test is verified, plan outage for actual production migration (2 days)
- Verify TNS names update/migration
- GLAST_DP_TEST
- GLAST_TREND
- others?
- Plan migration of glast_trend
- Execute pipeline (and datacat) production migration (2 days prep, 1 day exec)
- Outage time should be equivalent to outage time from production migration test
- Perform TNS names migration (glast_dp_test and @pipeline-ii, glast_trend)
- Pipeline production is migrated by now. Should be using glast_trend temporary tablespace
- While the pipeline is running with glast_trend temporary database, we will notify glast collab that trending info will be wonky for a few days
- Perform migration of GLAST_TREND to a new tablespace. As it is several terabytes, this should take a day or so.
- Once migration of GLAST_TREND is completed, update the tablespace with the new information from the temporary GLAST_TREND database.
- Should be able to be done online, with very small if no outage time.
- Migrate TNS name for GLAST_TREND
- Drop temporary GLAST_TREND info
- If Pipeline-II 1.5 Migration wasn't completed for Production pipeline, perform that.