January 29-30 Oracle meltdown

Version 3 by Tony Johnson
on Jan 31, 2008 17:12.

<< Changes from 2 to 3

compared with

Current by Tony Johnson
on Feb 01, 2008 09:56.

Key

These lines were removed. This word was removed.

These lines were added. This word was added.

There are 1 changes. View first change.

		We observed that the time taken to register datasets rose dramatically.

		!opssim2.png!
		!plot1.png!

		Possible causes:

		# Tom's was attempting to delete an old task with 1.5million datasets, starting at about 8am January 29. This request never completed and was canceled at about 5pm on January 29
		# Karen attempted to set up materialized views on the dataset table, at around 11am on January 29.
		# Glastlnx11 was importing large amounts of housekeeping data (exact time?)
		# There were problems with the nightly backup on January 29/30 night, it was restarted and ran for much of the day on January 30.