...
- 2.1 MB flat file per site * 65 monitoring sites = ~137 MB per day.** -bash-4.1$ ls -lh /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2012-06-08.txt.gz
-rw-rw-r-- 1 pinger iepm 2.1M Jun 9 01:04 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2012-06-08.txt.gz - 137 per day * 30 days = 4.1 GB per month
- This is a rough estimation for size of data table. Others such as host and meta-data tables were not yet considered.
- A solution is to divide MySQL tables in terms of months, regions or weeks (to make it slightly more scalable in case monitoring sites in increase in the future).
- To shard is also better for performance in future. As the data increases, queries will take longer. More tables mean things can be loaded in parallel.
...
Ghulam/Zafar> Making "packet_size" a joint Primary Key should work. Latest schema here (just a minor change in meta data table).
What is left to be done
1.Getdata.pl is currently saving raw pings after every hour rather than after every half an hour. Modification is required to store data for every half an hour means every ping(total 48 for one pair per day)
...