Versions Compared

Old Version 13

changes.mady.by.user Tony Johnson

Saved on

compared with

New Version 14

changes.mady.by.user Tony Johnson

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Overview

  • Overall we were pleased with the behavior of the tools
  • Test was very useful for seeing how tools performed under semi-realistic load
    • Revealed some issues we need to address between now and launch

Oracle crash

Due to "out of shared pool space". Shared pool is used, among other things, to cache recent queries so that they can be reused. Non optimal queries in the DataQualityMonitoring application seem to have been responsible for this. The queries were fixed immediately.

...

  • Learn to use GRIDControl, an insanely complicated tool to monitor Oracle. We can use it to find out what queries are heavier on the system and ways to optimize them.
  • RAC system: Real Application Clusters. For load balancing and fail over.

Web Server Crashes

There were a few (5?) crashes of the web server on which the DataQualityMonitoring application is deployed. All due to the server running out of memory.

...

Overall heavily used by shifters.

Digi Trending Ingestion Failure (2)

Due to a deadlock when digi ingestion overlapped with recon ingestion of the previous processing run.
This is not really a deadlock in the sense that Digi and Recon data is non overlapping. But they share the same tables.

  • indexing might fix the problem
  • catching exceptions and waiting some time will fix it for sure

Trending problems

  • Counter Normal Evts quantity jumping by a factor of two. This problem was due to a duplicate variable in Digi and Recon that should have not been ingested in the database. The jump happened after Recon ingestion.
    • This was the only duplicated variable

...

We need experts to sit down and compare trending plots from the web to the ones produced with desktop tools opening root files (there are only 30K of them!).

Pipeline

  • Does not seem to like oracle crashing. Will try to make it more robust. Difficult to test without actually taking oracle down.
  • SubStream rollback did not work as expected.

Safari Compatibility

We might have to spend some time making sure that the javascript we use is compatible with Safari.

Things we need to fix afterwards

In no particular order...

...