Introduction

To facilitate the analysis and presentation of the PingER data by others we made the historical hourly data available via anonymous FTP.

  • The initial audiences were:
  • The hourly data was chosen rather than the raw data since it is cleaner having been through several filters.
    • The hourly data is in: /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the file name <metric><size><by><yyyy><mm>-<dd>.txt.gz
  • The raw (daily gathered data from all the monitoring hosts) is close to 400GBytes. This includes:

    • /nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY>-<MM>.txt  (15GB) SLAC measurements
    • /nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt (8GB) SLAC measurements
    • /nfs/slac/g/net/pinger/pingerdata/ (375GB) data gathered at SLAC
    • We ignore these data in the rest of this web page.

Volume of data

There are roughly 100,000 files. The volumes of the files are shown below.

Uncompressed Volume of files per Year for all metricsUncompressed Volume for 3 metricsVolume of uncompressed data by metric

Compression

If  I multiply the file size times the frequency to get the bytes in each bin, and then sum I get 11,566,219,714 Bytes from compressed and  58,434,764,384 from uncompressed. This is ~ a compression ratio of 5:1.  The graph below also shows that Uncompressed files are larger than compressed.

Volume of compressed data by metricFrequency of files by size from compressed & uncompressed dataCumulative & yearly compressed data volumes

Missing data

There are two types of missing data:

  • Given the number of days, number of metrics we expect 6209 files in a year. However typically some files are missing (e.g. data may not have been gathered that day,  the gathering host may have been down, the data may have been corrupted etc.)
  • The monitoring host may not have been working or the host being monitored may not have responded to the pings. In this case since there is no value for the metric for the monitoring-remote host pair, then a dot is recorded.
Missing files/year by metricTotal number of dots per yearDots per metric

Spreadsheets

Hourly file data analysis from Renan and Christiane, see also 

https://docs.google.com/spreadsheets/d/1357xGkpYFeW0DcnDB-i7ZER2RBhjdQHEbmMDOvdc7bA/edit?usp=sharing

Pinger Data Volume from Les

 

  • No labels