This page gathers a bit of information about CPU time used by the Pass8 GRs.
Here is the associate Jira item: LPATE-102@JIRA

Summary

  • Pretty stable in the past months, we've improved some things but also added more
  • As of GR 20-08-10: Tkr, CAL2 and Other are each 1/3 of total CPU
  • CRFinder is 15% of total CPU, must be speed up
  • Have to look into the merit tuple Auditor variables
  • Work to be done on outliers
    • Outliers in the plots below are jobs running on slow machines (fell)
    • Outliers seem to be related to swapping
    • I'll try to ask for more 2GB of RAM instead of just 1GB when submitting jobs, it may help speeding up things

Plots

  • I gather data from log files of the Pass8 Solar Flare reprocessing task: 
    /nfs/farm/g/glast/u44/CalibSets-Tasks/Pass8_SFRs_Repro/output/Pass8_SFRs_Repro/"version"/run-setupCrumbs/digi-skimNrepro-crumb/runPass8Repro/"run"/"chunk"/"crumb"/logFile.txt
with "version" in ['2.6', '2.7', '2.8', '2.9']
corresponding to GR 20-08-06 to 20-08-10
  • All CPU times are normalized to jobs running on the hequ workers
  • CPU Time since 20-08-00
  • Plot for the past 4 releases: 20-08-06 to 20-08-10 (GRScan_p6Top10.root)

    CPU Time

    RAM

    Swap

    Cal2 (Energy)

    Tkr First Pass

    Tkr Tree Links

    Tkr Tree
    | | |

Breaking up 20-08-11

  • Global performances
  • Outliers at 3 sigma 
    Min/Average/Max CPU time 548/2521/6401
CPU outliers
host hequ run 1 chunk 15 crumb 45 normcpu 2751.960000
host kiso run 2 chunk 22 crumb 10 normcpu 2953.872000
host kiso run 2 chunk 22 crumb 46 normcpu 2685.288000
host kiso run 2 chunk 22 crumb 42 normcpu 2820.112000
host hequ run 2 chunk 22 crumb 38 normcpu 3027.380000
host kiso run 2 chunk 22 crumb 124 normcpu 2654.792000
host kiso run 2 chunk 22 crumb 123 normcpu 2795.560000
host fell run 2 chunk 21 crumb 52 normcpu 2883.887500
host kiso run 2 chunk 5 crumb 92 normcpu 2854.632000
host kiso run 2 chunk 5 crumb 76 normcpu 2829.296000
host kiso run 2 chunk 5 crumb 24 normcpu 2767.400000
host fell run 2 chunk 23 crumb 18 normcpu 4000.725000

SWAP outliers
host boer run 1 chunk 15 crumb 39 swap 2545.000000
host boer run 1 chunk 15 crumb 25 swap 2584.000000
host boer run 1 chunk 24 crumb 13 swap 2553.000000
host fell run 2 chunk 22 crumb 132 swap 2551.000000
host dole run 2 chunk 5 crumb 63 swap 2591.000000

CAL2 outliers
host kiso run 2 chunk 22 crumb 10 normcal2 931.200000
host kiso run 2 chunk 22 crumb 42 normcal2 936.000000
host fell run 2 chunk 21 crumb 52 normcal2 1245.000000
host kiso run 2 chunk 5 crumb 76 normcal2 960.000000
host fell run 2 chunk 23 crumb 18 normcal2 1102.500000

TREE outliers
host hequ run 1 chunk 15 crumb 45 normtree 1320.000000
host hequ run 2 chunk 22 crumb 38 normtree 1590.000000
host boer run 2 chunk 5 crumb 42 normtree 920.000000
host fell run 2 chunk 23 crumb 18 normtree 2073.750000

LINK outliers
host kiso run 2 chunk 22 crumb 10 normlink 389.600000
host kiso run 2 chunk 22 crumb 46 normlink 380.000000
host kiso run 2 chunk 22 crumb 42 normlink 385.600000
host kiso run 2 chunk 22 crumb 123 normlink 387.200000
host kiso run 2 chunk 5 crumb 92 normlink 360.000000
host kiso run 2 chunk 5 crumb 76 normlink 402.720000

FIRST outliers
host hequ run 1 chunk 15 crumb 45 normfirst 1812.000000
host kiso run 2 chunk 22 crumb 10 normfirst 1627.200000
host kiso run 2 chunk 22 crumb 42 normfirst 1502.400000
host hequ run 2 chunk 22 crumb 38 normfirst 2070.000000
host kiso run 2 chunk 22 crumb 123 normfirst 1502.400000
host kiso run 2 chunk 5 crumb 92 normfirst 1526.400000
host fell run 2 chunk 23 crumb 18 normfirst 2576.250000

Breaking up 20-08-10

  • Global performances
  • Outliers at 3 sigma 
    CPU outliers
host fell run 2 chunk 22 crumb 73 normcpu 3006.050000
host fell run 2 chunk 22 crumb 38 normcpu 3609.431250
host fell run 1 chunk 25 crumb 5 normcpu 4184.412500

SWAP outliers
host fell run 2 chunk 5 crumb 5 swap 2590.000000
host boer run 2 chunk 5 crumb 108 swap 2586.000000
host hequ run 2 chunk 5 crumb 6 swap 2589.000000
host bali run 2 chunk 5 crumb 27 swap 2586.000000
host yili run 2 chunk 5 crumb 84 swap 2549.000000
host boer run 2 chunk 21 crumb 32 swap 2579.000000
host boer run 2 chunk 22 crumb 71 swap 2585.000000
host fell run 2 chunk 22 crumb 50 swap 2542.000000
host fell run 2 chunk 22 crumb 73 swap 2581.000000
host dole run 1 chunk 24 crumb 28 swap 2560.000000
host dole run 1 chunk 24 crumb 19 swap 2583.000000
host fell run 1 chunk 24 crumb 23 swap 2631.000000
host hequ run 1 chunk 15 crumb 5 swap 2585.000000
host fell run 1 chunk 25 crumb 46 swap 2625.000000
host dole run 1 chunk 25 crumb 0 swap 2580.000000
host dole run 1 chunk 17 crumb 18 swap 2557.000000
host fell run 1 chunk 17 crumb 57 swap 2591.000000
host dole run 1 chunk 17 crumb 91 swap 2584.000000
host dole run 0 chunk 3 crumb 4 swap 2541.000000

CAL2 outliers
host fell run 2 chunk 22 crumb 73 normcal2 1301.250000
host fell run 1 chunk 25 crumb 5 normcal2 1691.250000

Tkr TREE outliers
host dole run 2 chunk 23 crumb 18 normtree 939.622642
host fell run 2 chunk 22 crumb 14 normtree 795.000000
host fell run 2 chunk 22 crumb 38 normtree 2006.250000
host kiso run 1 chunk 15 crumb 45 normtree 1012.800000
host fell run 1 chunk 25 crumb 5 normtree 990.000000
host fell run 1 chunk 25 crumb 10 normtree 862.500000

Tkr Tree LINK outliers
host fell run 2 chunk 5 crumb 99 normlink 386.250000
host fell run 2 chunk 22 crumb 73 normlink 390.000000
host fell run 1 chunk 25 crumb 5 normlink 540.000000

Tkr FIRST outliers
host dole run 2 chunk 23 crumb 18 normfirst 1386.792453
host fell run 2 chunk 22 crumb 38 normfirst 2445.000000
host kiso run 1 chunk 15 crumb 45 normfirst 1483.200000
host fell run 1 chunk 25 crumb 5 normfirst 1882.500000