Page History

...

• Pass 8 Recon & Analysis Upgrades Weekly Meeting Agenda
• LPATE-102@JIRA

Summary

• Pretty stable in the past months, we've improved some things but also added more
• As of GR 20-08-10: Tkr, CAL2 and Other are each 1/3 of total CPU
• CRFinder is a 15% of total CPU, must be speed up
• Work to be done on outliers
• Look Have to look into the merit tuple Auditor variables
• Work to be done on outliers
  • Outliers in the plots below are jobs running on slow machines (fell)
  Try
  • Outliers seem to be related to swapping
  • I'll try to ask for more 2GB of RAM instead of just 1GB when submitting jobs, it may help speeding up things

Plots

• I gather data from log files of the Pass8 Solar Flare reprocessing task:

  Code Block
  /nfs/farm/g/glast/u44/CalibSets-Tasks/Pass8_SFRs_Repro/output/Pass8_SFRs_Repro/"version"/run-setupCrumbs/digi-skimNrepro-crumb/runPass8Repro/"run"/"chunk"/"crumb"/logFile.txt with "version" in ['2.6', '2.7', '2.8', '2.9'] corresponding to GR 20-08-06 to 20-08-10

• All CPU times are normalized to jobs running on the hequ workers
• CPU Time since 20-08-00
  

• Plot for the past 4 releases: 20-08-06 to 20-08-10 (GRScan_p6Top10.root)

  CPU Time	RAM	Swap	Cal2 (Energy)
  Image Added	Image Added	Image Added	Image Added
  Tkr First Pass	Tkr Tree Links	Tkr Tree	Cal2 (Energy)
  Image Removed	Image Removed	Image Removed	Image Removed	Image Removed	Image Removed	Image Removed

  Image Added | Image Added | Image Added |

Breaking up 20-08-11

• Global performances
  Image Added

• Outliers at 3 sigma

  Code Block

  Min/Average/Max CPU time 548/2521/6401 CPU outliers host hequ run 1 chunk 15 crumb 45 normcpu 2751.960000 host kiso run 2 chunk 22 crumb 10 normcpu 2953.872000 host kiso run 2 chunk 22 crumb 46 normcpu 2685.288000 host kiso run 2 chunk 22 crumb 42 normcpu 2820.112000 host hequ run 2 chunk 22 crumb 38 normcpu 3027.380000 host kiso run 2 chunk 22 crumb 124 normcpu 2654.792000 host kiso run 2 chunk 22 crumb 123 normcpu 2795.560000 host fell run 2 chunk 21 crumb 52 normcpu 2883.887500 host kiso run 2 chunk 5 crumb 92 normcpu 2854.632000 host kiso run 2 chunk 5 crumb 76 normcpu 2829.296000 host kiso run 2 chunk 5 crumb 24 normcpu 2767.400000 host fell run 2 chunk 23 crumb 18 normcpu 4000.725000 SWAP outliers host boer run 1 chunk 15 crumb 39 swap 2545.000000 host boer run 1 chunk 15 crumb 25 swap 2584.000000 host boer run 1 chunk 24 crumb 13 swap 2553.000000 host fell run 2 chunk 22 crumb 132 swap 2551.000000 host dole run 2 chunk 5 crumb 63 swap 2591.000000 CAL2 outliers host kiso run 2 chunk 22 crumb 10 normcal2 931.200000 host kiso run 2 chunk 22 crumb 42 normcal2 936.000000 host fell run 2 chunk 21 crumb 52 normcal2 1245.000000 host kiso run 2 chunk 5 crumb 76 normcal2 960.000000 host fell run 2 chunk 23 crumb 18 normcal2 1102.500000 TREE outliers host hequ run 1 chunk 15 crumb 45 normtree 1320.000000 host hequ run 2 chunk 22 crumb 38 normtree 1590.000000 host boer run 2 chunk 5 crumb 42 normtree 920.000000 host fell run 2 chunk 23 crumb 18 normtree 2073.750000 LINK outliers host kiso run 2 chunk 22 crumb 10 normlink 389.600000 host kiso run 2 chunk 22 crumb 46 normlink 380.000000 host kiso run 2 chunk 22 crumb 42 normlink 385.600000 host kiso run 2 chunk 22 crumb 123 normlink 387.200000 host kiso run 2 chunk 5 crumb 92 normlink 360.000000 host kiso run 2 chunk 5 crumb 76 normlink 402.720000 FIRST outliers host hequ run 1 chunk 15 crumb 45 normfirst 1812.000000 host kiso run 2 chunk 22 crumb 10 normfirst 1627.200000 host kiso run 2 chunk 22 crumb 42 normfirst 1502.400000 host hequ run 2 chunk 22 crumb 38 normfirst 2070.000000 host kiso run 2 chunk 22 crumb 123 normfirst 1502.400000 host kiso run 2 chunk 5 crumb 92 normfirst 1526.400000 host fell run 2 chunk 23 crumb 18 normfirst 2576.250000

Breaking up 20-08-10

• Global performances
  

• Outliers at 3 sigma

  Code Block

  CPU outliers host fell run 2 chunk 22 crumb 73 normcpu 3006.050000 host fell run 2 chunk 22 crumb 38 normcpu 3609.431250 host fell run 1 chunk 25 crumb 5 normcpu 4184.412500 SWAP outliers host fell run 2 chunk 5 crumb 5 swap 2590.000000 host boer run 2 chunk 5 crumb 108 swap 2586.000000 host hequ run 2 chunk 5 crumb 6 swap 2589.000000 host bali run 2 chunk 5 crumb 27 swap 2586.000000 host yili run 2 chunk 5 crumb 84 swap 2549.000000 host boer run 2 chunk 21 crumb 32 swap 2579.000000 host boer run 2 chunk 22 crumb 71 swap 2585.000000 host fell run 2 chunk 22 crumb 50 swap 2542.000000 host fell run 2 chunk 22 crumb 73 swap 2581.000000 host dole run 1 chunk 24 crumb 28 swap 2560.000000 host dole run 1 chunk 24 crumb 19 swap 2583.000000 host fell run 1 chunk 24 crumb 23 swap 2631.000000 host hequ run 1 chunk 15 crumb 5 swap 2585.000000 host fell run 1 chunk 25 crumb 46 swap 2625.000000 host dole run 1 chunk 25 crumb 0 swap 2580.000000 host dole run 1 chunk 17 crumb 18 swap 2557.000000 host fell run 1 chunk 17 crumb 57 swap 2591.000000 host dole run 1 chunk 17 crumb 91 swap 2584.000000 host dole run 0 chunk 3 crumb 4 swap 2541.000000 CAL2 outliers host fell run 2 chunk 22 crumb 73 normcal2 1301.250000 host fell run 1 chunk 25 crumb 5 normcal2 1691.250000 Tkr TREE outliers host dole run 2 chunk 23 crumb 18 normtree 939.622642 host fell run 2 chunk 22 crumb 14 normtree 795.000000 host fell run 2 chunk 22 crumb 38 normtree 2006.250000 host kiso run 1 chunk 15 crumb 45 normtree 1012.800000 host fell run 1 chunk 25 crumb 5 normtree 990.000000 host fell run 1 chunk 25 crumb 10 normtree 862.500000 Tkr Tree LINK outliers host fell run 2 chunk 5 crumb 99 normlink 386.250000 host fell run 2 chunk 22 crumb 73 normlink 390.000000 host fell run 1 chunk 25 crumb 5 normlink 540.000000 Tkr FIRST outliers host dole run 2 chunk 23 crumb 18 normfirst 1386.792453 host fell run 2 chunk 22 crumb 38 normfirst 2445.000000 host kiso run 1 chunk 15 crumb 45 normfirst 1483.200000 host fell run 1 chunk 25 crumb 5 normfirst 1882.500000