...
The DataQualityMonitoring application needs to be fined tuned:
When multiple runs are selected, histograms are added and trending quantities are fetched from the database for the resulting time period.
The first crash happened 2.5 hours into the test on Tue morning. Applications were rebalanced (i.e. DataQualityMonitoring application was isolated) and the memory was increased to 1.5 Gb. Three crashes happened in the last shift in an hour (guess that shifters liked to see the trended data over the 16 orbits (at 15 seconds intervals?) ).
The first crash
...