The PingER metrics motion charts are primarily used to visualize the trends in the Internet end-to-end performance statistics measured to over 160 countries from the 40+ PingER monitoring nodes spread worldwide. Having gathered data since 1998, the charts enable the users to study the trends, step changes, significant improvements/degradations with the help of these 4-dimensional charts. The charts are based on Google Motion Charts.
The different sets of charts (w.r.t. regions) enables the users to study the progress made by countries in comparison to their neighbors as well as the world in general.
Below are some examples:
Observations made from Stanford Linear Accelerator Center (SLAC), USA in 2008.
View of the world | View of the world | View of the world |
|---|---|---|
|
|
|
Observations made from SLAC 2019
| View of world from SLAC June 2019 (Norm throughput against min RTT, bubble size=population) | View of World from SLAC June 2019 (IDI index vs Packet loss) | View of world from SLAC June 2019 (Norm throughput vs IDI, color=minimum RTT) | |
|---|---|---|---|
 |  |  |
Observations made from CERN, Geneva, Switzerland:
View of South Asia | View of the Balkans | View of Africa |
|---|---|---|
|
|
|
By default, PingER metrics motion chart shows a motion scatter/bubble plot of the ping (100 Byte) average RTT in msec. ( x ) versus the normalized derived TCP throughput in kbits/sec ( y ). Each bubble represents a country. By default the bubbles are colored by region and their sizes are proportional to the Internet users. The motion is the time. The time granularity is by year. Click on the play button to initiate the motion.
To create a motion chart, the data must be populated in a data structure (as stated by Google Visualization API and copied below):
<html>
<head>
<script type="text/javascript" src="http://www.google.com/jsapi"></script>
<script type="text/javascript">
google.load("visualization", "1", {packages:["motionchart"]});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = new google.visualization.DataTable();
data.addColumn('string', 'Country');
data.addColumn('date', 'Month');
data.addColumn('string', 'Region');
data.addColumn('number', 'Average RTT (ms)');
data.addColumn('number', 'Normalized Throughput (Kbps)');
data.addColumn('number', 'Internet Users (#)');
data.addColumn('number', 'Internet Penetration (%)');
data.addColumn('number', 'Population (#)');
data.addColumn('number', 'Minimum RTT (ms)');
data.addColumn('number', 'Packet Loss (%)');
data.addColumn('number', 'Unreachability (%)');
data.addColumn('number', 'Jitter (ms)');
data.addColumn('number', 'DOI Index');
data.addColumn('number', 'HDI Index');
data.addRows(1);
data.setValue(0, 0, "Pakistan");
data.setValue(0, 1, new Date(2007,02,01));
data.setValue(0, 2, 'South Asia');
data.setValue(0, 3, 421.346);
data.setValue(0, 4, 1568.95315647059);
data.setValue(0, 5, 7500000);
data.setValue(0, 6, 4.7);
data.setValue(0, 7, 172800051);
data.setValue(0, 8, 311.482);
data.setValue(0, 9, 1.331);
data.setValue(0, 10, 15.591);
data.setValue(0, 11, 19.906);
data.setValue(0, 12, 0.26);
data.setValue(0, 13, 0.551);
data.addRows(1);
data.setValue(1, 0, "Sri Lanka");
data.setValue(1, 1, new Date(2007,02,01));
data.setValue(1, 2, 'South Asia');
data.setValue(1, 3, 418.664);
data.setValue(1, 4, 2760.35944717647);
data.setValue(1, 5, 280000);
data.setValue(1, 6, 1.4);
data.setValue(1, 7, 21128773);
data.setValue(1, 8, 270.494);
data.setValue(1, 9, 0.304);
data.setValue(1, 10, 8.871);
data.setValue(1, 11, 42.093);
data.setValue(1, 12, 0.33);
data.setValue(1, 13, 0.743);
data.addRows(1);
data.setValue(2, 0, "Pakistan");
data.setValue(2, 1, new Date(2007,03,01));
data.setValue(2, 2, 'South Asia');
data.setValue(2, 3, 397.897);
data.setValue(2, 4, 1394.21896911765);
data.setValue(2, 5, 7500000);
data.setValue(2, 6, 4.7);
data.setValue(2, 7, 172800051);
data.setValue(2, 8, 305.533);
data.setValue(2, 9, 1.248);
data.setValue(2, 10, 12.917);
data.setValue(2, 11, 31.064);
data.setValue(2, 12, 0.26);
data.setValue(2, 13, 0.551);
data.addRows(1);
data.setValue(3, 0, "Sri Lanka");
data.setValue(3, 1, new Date(2007,03,01));
data.setValue(3, 2, 'South Asia');
data.setValue(3, 3, 357.202);
data.setValue(3, 4, 4634.87959694118);
data.setValue(3, 5, 280000);
data.setValue(3, 6, 1.4);
data.setValue(3, 7, 21128773);
data.setValue(3, 8, 274.092);
data.setValue(3, 9, 0.144);
data.setValue(3, 10, 36.319);
data.setValue(3, 11, 11.889);
data.setValue(3, 12, 0.33);
data.setValue(3, 13, 0.743);
data.addRows(1);
data.setValue(4, 0, "Pakistan");
data.setValue(4, 1, new Date(2007,04,01));
data.setValue(4, 2, 'South Asia');
data.setValue(4, 3, 372.313);
data.setValue(4, 4, 1918.28198552422);
data.setValue(4, 5, 7500000);
data.setValue(4, 6, 4.7);
data.setValue(4, 7, 172800051);
data.setValue(4, 8, 310.145);
data.setValue(4, 9, 0.969);
data.setValue(4, 10, 13.844);
data.setValue(4, 11, 24.146);
data.setValue(4, 12, 0.26);
data.setValue(4, 13, 0.551);
data.addRows(1);
data.setValue(5, 0, "Sri Lanka");
data.setValue(5, 1, new Date(2007,04,01));
data.setValue(5, 2, 'South Asia');
data.setValue(5, 3, 397.434);
data.setValue(5, 4, 2450.70159022405);
data.setValue(5, 5, 280000);
data.setValue(5, 6, 1.4);
data.setValue(5, 7, 21128773);
data.setValue(5, 8, 298.214);
data.setValue(5, 9, 0.410);
data.setValue(5, 10, 7.527);
data.setValue(5, 11, 6.979);
data.setValue(5, 12, 0.33);
data.setValue(5, 13, 0.743);
var chart = new google.visualization.MotionChart(document.getElementById('chart_div'));
chart.draw(data, {width: 850, height:500});
}
</script>
</head>
<body>
<div id="chart_div" style="width: 850px; height: 500px;"></div>
</body>
</html>
|
The data must comply with the formatting requirements (as stated by Google Visualization API and copied below):
Note that the data must be loaded in a chronologically ascending order. See the code above as an example. Also note that the order in which the columns are organized, defines the default metrics with which the axes are labeled. The first number column defines the label for the x-axis where as the second number column defines the default label for the y-axis. By changing the order of the number columns, the default labels for the axes can be changed.
Following are the files involved in the process of generating the HTML page:
Application:
File path:/afs/slac/package/pinger/motion-chart/create-motion-chart-page.pl |
The script that defines the input arguments to generate the relevant .html files:
File path: /afs/slac/package/pinger/motion-chart/create-motion-chart-pages.pl |
Demographics:
File path: /afs/slac/package/pinger/motion-chart/demographics.csv File format: #Country, Year, Population, Internet Users, Internet Penetration, DOI, HDI United States, 2008, 303824646, 213364918, 70.7, 0.62, 0.951 |
PingER data (obtained from prmout) Average RTT, Jitter (IPDV), Minimum RTT, Normalized Throughput, Packet Loss and Unreachability as seen from CERN, SLAC and ICTP:
The following files are automatically updated on a daily basis by the pinger@pinger.slac.stanford.edu trscrontab. File formats: allyearly: #allyearly,?,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008 monthly: #monthly,?,Oct2008,Sep2008,Aug2008, ... Jan2007,Dec2006,Nov2006 last60days: #last60days,?,08Oct27,08Oct26,08Oct25,08Oct24, ... 08Aug31,08Aug30,08Aug29 File paths: /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-CH.CERN.N2-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-CH.CERN.N2-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-CH.CERN.N2-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-EDU.SLAC.STANFORD.N3-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-EDU.SLAC.STANFORD.N3-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-EDU.SLAC.STANFORD.N3-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-IT.ICTP.N1-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-IT.ICTP.N1-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-IT.ICTP.N1-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-CH.CERN.N2-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-CH.CERN.N2-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-CH.CERN.N2-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-EDU.SLAC.STANFORD.PINGER-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-EDU.SLAC.STANFORD.PINGER-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-EDU.SLAC.STANFORD.PINGER-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-IT.ICTP.N1-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-IT.ICTP.N1-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/ipdv-IT.ICTP.N1-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-CH.CERN.N2-country-allyearly.cvs /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-CH.CERN.N2-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-CH.CERN.N2-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-EDU.SLAC.STANFORD.PINGER-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-EDU.SLAC.STANFORD.PINGER-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-EDU.SLAC.STANFORD.PINGER-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-IT.ICTP.N1-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-IT.ICTP.N1-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-IT.ICTP.N1-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-CH.CERN.N2-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-CH.CERN.N2-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-CH.CERN.N2-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-EDU.SLAC.STANFORD.PINGER-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-EDU.SLAC.STANFORD.PINGER-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-EDU.SLAC.STANFORD.PINGER-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-IT.ICTP.N1-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-IT.ICTP.N1-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-IT.ICTP.N1-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-CH.CERN.N2-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-CH.CERN.N2-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-CH.CERN.N2-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-EDU.SLAC.STANFORD.PINGER-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-EDU.SLAC.STANFORD.PINGER-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-EDU.SLAC.STANFORD.PINGER-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-IT.ICTP.N1-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-IT.ICTP.N1-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-IT.ICTP.N1-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-CH.CERN.N2-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-CH.CERN.N2-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-CH.CERN.N2-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-EDU.SLAC.STANFORD.PINGER-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-EDU.SLAC.STANFORD.PINGER-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-EDU.SLAC.STANFORD.PINGER-country-monthly.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-IT.ICTP.N1-country-allyearly.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-IT.ICTP.N1-country-last60days.csv /afs/slac/g/www/www-iepm/pinger/prmout/unreachability-IT.ICTP.N1-country-monthly.csv |
Country to region mapping:
The following file is automatically updated on a daily basis by the pinger@pinger.slac.stanford.edu trscrontab. File path: /afs/slac.stanford.edu/www/comp/net/mon/countries.tsv File format: Country Name Region Name Note that the country name and region name are separated by a tab. #================================================================================================== #This is file /afs/slac.stanford.edu/www/comp/net/mon/countries.tsv #Current as of: Sat Jun 29 00:15:05 2019 #Written by Script: /afs/slac/package/pinger/write_country_list. #Written on: lnxcron.slac.stanford.edu #Executing Directory: /afs/slac.stanford.edu/u/sf/pinger #User: pinger #Usually run via trscron from pinger@pinger #Data extracted from the Guthrie NODEDETAILS Oracle database. #================================================================================================== Afghanistan South Asia Albania Balkans Algeria Africa Andorra Europe Angola Africa Argentina Latin America Armenia Central Asia Australia Oceania Austria Europe Azerbaijan Central Asia Bahamas Latin America Bahrain Middle East Bangladesh South Asia Belarus Europe Belgium Europe Benin Africa Bhutan South Asia Bolivia Latin America Bosnia Herzegovina Balkans ... |
Following is the pseudo code for implementation:
1. Load the country to region mapping to the hash-table "%region" which uses the country
names (as defined by PingER) as the index.
e.g. $region{'Kenya'} = Africa and
$region{'Korea, Rep'} = East Asia.
2. Load the PingER data to the data structure "%data" which uses PingER metrics as the index.
Here each entry stores a reference to a unique hash-table defined to store the PingER data
related to a country. Understandably the second hash-table uses country names as the index.
Each entry of the hash-table stores a reference to another hash-table used for storing
performance measurements for the country. Here timestamps are used as indices.
e.g. $data{'minrtt'}->{'Kenya'}->{'1998'},
$data{'packetloss'}->{'United States'}->{'Oct2008'} and
$data{'ipdv'}->{'Pakistan'}->{'08Oct27'}
The metrics used are: minrtt, avgrtt, nthroughput, ipdv, packetloss and unreachability.
An important factor to consider here is the file format. The yearly data from prmout is listed
in an ascending order from left to right, where as for monthly and daily data, the
measurements are listed in a descending order. Thus while loading, the iterators need to
be defined accordingly.
3. Load the demographics to the data structure "%data" which uses labels as the index.
Here each entry stores a reference to a unique hash-table defined to store the demographics
of a country. The second hash-table uses country names as the index. Each entry of
the hash-table stores a reference to another hash-table used for storing statistics
for the country. Here timestamps are used as indices.
e.g. $data{'population'}->{'Kenya'}->{'1998'},
$data{'doi'}->{'United States'}->{'Oct2008'} and
$data{'inetpenetration'}->{'Pakistan'}->{'08Oct27'}
The metrics used are: population, inetusers, inetpenetration, doi and hdi.
4. Generate the .html file in compliance with the Google Visualization API constraints.
Here the data is loaded in the following format (code in Javascript):
data.addColumn('string', 'Country');
data.addColumn('date', 'Month');
data.addColumn('string', 'Region');
data.addColumn('number', 'Average RTT (ms)');
data.addColumn('number', 'Normalized Throughput (Kbps)');
data.addColumn('number', 'Internet Users (#)');
data.addColumn('number', 'Internet Penetration (%)');
data.addColumn('number', 'Population (#)');
data.addColumn('number', 'Minimum RTT (ms)');
data.addColumn('number', 'Packet Loss (%)');
data.addColumn('number', 'Unreachability (%)');
data.addColumn('number', 'Jitter (ms)');
data.addColumn('number', 'DOI Index');
data.addColumn('number', 'HDI Index');
N.B. the data must be uploaded in a chronologically ascending order.
5. Create .html files for each vantage point: CERN Switzerland (CH.CERN.N2),
SLAC USA (EDU.SLAC.STANFORD.N3), ICTP Italy (IT.ICTP.N1)
for each region: Africa, Balkans, Central Asia, East Asia, Europe, Latin America,
Middle East, North America, Oceania, S.E. Asia, South Asia, World, for each
granularity: Yearly (since 1998), Monthly (last 20 months) and Daily (last 3 weeks).
|
The PingER data files are generated using the prm scripts. There are some trscronjobs running under pinger@pinger.slac.stanford.edu. They appear as:
################################################################################
#prm.pl runs monthly to create the executive plots to be accessesd via:
#http://www-iepm.slac.stanford.edu/pinger/prmout/
pinger;600 30 02 1 * * /afs/slac.stanford.edu/package/pinger/prm/prm.pl \
-c /afs/slac.stanford.edu/package/pinger/prm/prm-allmonthly.conf
#And yearly (Jan 5th at 4:30am) to get the yearly data.
pinger;600 30 04 05 1 * /afs/slac.stanford.edu/package/pinger/prm/prm.pl \
-c /afs/slac.stanford.edu/package/pinger/prm/prm-allyearly.conf
#It also runs each morning to update the last 21 days information.
pinger;600 30 00 * * * /afs/slac.stanford.edu/package/pinger/prm/prm.pl \
-c /afs/slac.stanford.edu/package/pinger/prm/prm-last60days.conf
#Create the motion bubble plots:
pinger;10 00 08 * * * /afs/slac/package/pinger/motion-chart/create-motion-chart-pages.pl
|
On the other hand, the demographics need to be updated manually. The file containing the coalesced statistics is demographics.csv. However to update the records, the steps listed below must be followed. Files relevant to the process are:
Flat file listing the statistics: /afs/slac/package/pinger/motion-chart/demographics.csv The file looks like: #Country, Year, population, inetusers, inetpenetration, doi, hdi, cpi, idi Afghanistan, 1998, 22912814, 0, 0.1, 0, , 0, 0 ... Afghanistan, 2019, 37209007, 0, 11.45, 0, 0.498, 0, 1.95 The script to generate csv files (each reflecting records for a single statistic): /afs/slac/package/pinger/motion-chart/generate-csv-from-demographics.pl* The script to generate demographics.csv from the independent csv files: /afs/slac/package/pinger/motion-chart/generate-demographics-from-csv.pl* |
The intermediate flat files listing each statistic separately are:
CPI: /afs/slac/package/pinger/motion-chart/cpi.csv #Corruptions Perceptions Index from Transparency International
Appears as:
Country, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019
Afghanistan, , , , , 0.373, 0.383, 0.398, 0.408, 0.417, 0.429, 0.437, 0.453, 0.463, 0.471, 0.482, 0.487, 0.491, 0.493, 0.494, 0.498, 0, 0.498
DOI: /afs/slac/package/pinger/motion-chart/doi.csv #Digital Opportunity Index from ITU
Appears as:
Country, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019
Algeria, 0.39, 0.39, 0.39, 0.39, 0.39, 0.39, 0.39, 0.39, 0.42, 0.42, 0.42, 0.42, 0.42, 0, 0, 0, 0, 0, 0, 0, 0, 0
HDI: /afs/slac/package/pinger/motion-chart/hdi.csv #Human Development Index from UNDP
Appears as:
Country, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019
Afghanistan, , , , , 0.373, 0.383, 0.398, 0.408, 0.417, 0.429, 0.437, 0.453, 0.463, 0.471, 0.482, 0.487, 0.491, 0.493, 0.494, 0.498, 0, 0.498
IDI: /afs/slac/package/pinger/motion-chart/idi.csv #ICT Development Index from ITU
Internet Penetration: /afs/slac/package/pinger/motion-chart/inetpenetration.csv
Appears as:
Country, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019
Afghanistan, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.71, 1.71, 1.71, 1.71, 1.71, 1.71, 1.71, 1.95, 1.95, 1.95
Internet Users: /afs/slac/package/pinger/motion-chart/inetusers.csv
Appears as:
Country, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019
Afghanistan, 0, 0, 0, 1000, 1000, 20000, 25000, 300000, 535000, 500000, 500000, 1000000, 1000000, 0, 0, 0, 0, 0, 0, 0, 0, 0
Population: /afs/slac/package/pinger/motion-chart/pop.csv
Appears as:
Country, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019
Afghanistan, 22912814, 23646128, 23898198, 23997412, 25268405, 27060359, 28513677, 29928987, 31056997, 31889923, 32738376, 28396000, 29121286, 30
019922, 30918557, 31817193, 32715829, 33614464, 34513100, 35411736, 36310371, 37209007
Note that the format of these files is such that each row represents statistics of a
country and the columns list the annual measurements. e.g. the hdi.csv has the
following format:
#Country, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
Albania, 0.801, 0.801, 0.801, 0.801, 0.801, 0.801, 0.801, 0.801, 0.801, 0.801, 0.801, 0.801
Algeria, 0.733, 0.733, 0.733, 0.733, 0.733, 0.733, 0.733, 0.733, 0.733, 0.733, 0.733, 0.733
AmericanSamoa, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
Note that the header of the intermediate csv files must be of the form:
Country, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
|
Updating the records
:
1. Create a backup of the demographics.csv. This is just to avoid loss of information
(in case of any error).
(command: cp demographics.csv demographics.csv.back)
2. Execute the following command to generate the separate .csv files:
./generate-csv-from-demographics.pl
This will generate the required files (cpi.csv, doi.csv, hdi.csv, inetpenetration.csv,
inetusers.csv and pop.csv).
3. These separate csv files maintain records for a single statistic. Any update required
to the demographics data should be made to these csv files. For example to add records
for the year 2010, open each csv (for the separate statistic) in MS Excel and add
another column. The header must state the correct year (note the file format). Similarly,
to add countries to the records, add a row with the annual measurements separated by
commas.
4. Once all the csv files have been updated, execute the script to generate the coalesced
demographics statistics.
./generate-demographics-from-csv.pl
This command would read the separate .csv files and generate an updated demographics.csv
file.
|
There are certain conditions that must be met:
Both the scripts generate-csv-from-demographics.pl and generate-demographics-from-csv.pl take options as input parameters for customization.
Adding a new statistic would require simple changes(/additions) to the scripts below. The code is documented to facilitate this process.
/afs/slac/package/pinger/motion-chart/generate-csv-from-demographics.pl* /afs/slac/package/pinger/motion-chart/generate-demographics-from-csv.pl* |
To generate up to date .html files of the PingER metric motion charts execute the following script:
/afs/slac/package/pinger/motion-chart/create-motion-chart-pages.pl |
The charts generated (listed below) may then viewed publicly via SLAC's web server at https://www-iepm.slac.stanford.edu/pinger/prmout/
/afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-IT.ICTP.N1-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-CH.CERN.N2-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-CH.CERN.N2-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-CH.CERN.N2-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.N3-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.N3-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.N3-monthly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-IT.ICTP.N1-allyearly.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-IT.ICTP.N1-last60days.html /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-IT.ICTP.N1-monthly.html |
Sometimes instead of balloons, stars or asterisks are displayed. Try a complete refresh (CTRL-Refresh).
Due to unavailability of data for population and internet users for 2011, the static version of demographics.csv was generated. Instead of generating the demographics.csv from the script generate-demographics-from-csv.pl use the static version of demographics.csv.
Internet users and population have to be integer values. If these are decimal values, the HTML page chart fails to generate even though create-motion-chart-pages.pl runs fine.
The text appears but no chart. You probably need to enable flash.
[cottrell@pinger ~]$ tar -cvf umar.tar /afs/slac/package/pinger/motion-chart tar: Removing leading `/' from member names /afs/slac/package/pinger/motion-chart/ /afs/slac/package/pinger/motion-chart/generate-demographics-from-csv.pl /afs/slac/package/pinger/motion-chart/generate-csv-from-demographics.pl /afs/slac/package/pinger/motion-chart/inetusers.csv /afs/slac/package/pinger/motion-chart/population.csv /afs/slac/package/pinger/motion-chart/demographics.csv /afs/slac/package/pinger/motion-chart/log.stdout /afs/slac/package/pinger/motion-chart/backup-2009/ /afs/slac/package/pinger/motion-chart/backup-2009/generate-csv-from-demographics.pl /afs/slac/package/pinger/motion-chart/backup-2009/generate-demographics-from-csv.pl /afs/slac/package/pinger/motion-chart/backup-2009/inetusers.csv /afs/slac/package/pinger/motion-chart/backup-2009/demographics.csv /afs/slac/package/pinger/motion-chart/backup-2009/inetpenetration.csv /afs/slac/package/pinger/motion-chart/backup-2009/idi.csv /afs/slac/package/pinger/motion-chart/backup-2009/hdi.csv /afs/slac/package/pinger/motion-chart/backup-2009/cpi.csv /afs/slac/package/pinger/motion-chart/backup-2009/doi.csv /afs/slac/package/pinger/motion-chart/backup-2009/create-motion-chart-pages.pl /afs/slac/package/pinger/motion-chart/backup-2009/pop.csv /afs/slac/package/pinger/motion-chart/backup-2009/create-motion-chart-page.pl /afs/slac/package/pinger/motion-chart/backup-2009/log.conf /afs/slac/package/pinger/motion-chart/backup-2008/ /afs/slac/package/pinger/motion-chart/backup-2008/population.csv /afs/slac/package/pinger/motion-chart/backup-2008/population.csv.old2 /afs/slac/package/pinger/motion-chart/backup-2008/demographics.csv /afs/slac/package/pinger/motion-chart/backup-2008/population.csv.old3 /afs/slac/package/pinger/motion-chart/backup-2008/population.csv.old4 /afs/slac/package/pinger/motion-chart/backup-2008/create-motion-chart-page.pl.bak /afs/slac/package/pinger/motion-chart/backup-2008/create-motion-chart-page.pl.old2 /afs/slac/package/pinger/motion-chart/backup-2008/create-motion-chart-pages.pl /afs/slac/package/pinger/motion-chart/backup-2008/population-old.csv /afs/slac/package/pinger/motion-chart/backup-2008/population.csv.old /afs/slac/package/pinger/motion-chart/backup-2008/create-motion-chart-page.pl /afs/slac/package/pinger/motion-chart/backup-2008/generate-pinger-metrics-chart.sh /afs/slac/package/pinger/motion-chart/backup-2008/create-motion-chart-page.pl.old /afs/slac/package/pinger/motion-chart/backup-2008/create-motion-chart-pages.pl.bak /afs/slac/package/pinger/motion-chart/generate-csv-from-demographics.v1.pl.bak /afs/slac/package/pinger/motion-chart/generate-demographics-from-csv.v1.pl.bak /afs/slac/package/pinger/motion-chart/inetpenetration.csv /afs/slac/package/pinger/motion-chart/.demographics.csv.swp /afs/slac/package/pinger/motion-chart/idi.csv /afs/slac/package/pinger/motion-chart/hdi.csv /afs/slac/package/pinger/motion-chart/cpi.csv /afs/slac/package/pinger/motion-chart/doi.csv /afs/slac/package/pinger/motion-chart/log.conf.bk /afs/slac/package/pinger/motion-chart/.create-motion-chart-page.pl.swp /afs/slac/package/pinger/motion-chart/create-motion-chart-pages.pl /afs/slac/package/pinger/motion-chart/.population.csv.swp /afs/slac/package/pinger/motion-chart/demographics.csv.bk /afs/slac/package/pinger/motion-chart/pop.csv /afs/slac/package/pinger/motion-chart/create-motion-chart-page.pl /afs/slac/package/pinger/motion-chart/log.conf /afs/slac/package/pinger/motion-chart/logs/ /afs/slac/package/pinger/motion-chart/logs/log.stdout /afs/slac/package/pinger/motion-chart/logs/archives/ /afs/slac/package/pinger/motion-chart/logs/archives/motion_chart.log.2009-10-09.tar.gz /afs/slac/package/pinger/motion-chart/logs/archives/motion_chart.log.2009-10-11.tar.gz /afs/slac/package/pinger/motion-chart/logs/archives/motion_chart.log.2009-10-13.tar.gz /afs/slac/package/pinger/motion-chart/logs/archives/motion_chart.log.2009-10-12.tar.gz /afs/slac/package/pinger/motion-chart/logs/archives/motion_chart.log.2009-10-10.tar.gz /afs/slac/package/pinger/motion-chart/logs/manage-logs.pl /afs/slac/package/pinger/motion-chart/backup-2010/ /afs/slac/package/pinger/motion-chart/backup-2010/inetusers.csv /afs/slac/package/pinger/motion-chart/backup-2010/demographics.csv /afs/slac/package/pinger/motion-chart/backup-2010/inetpenetration.csv /afs/slac/package/pinger/motion-chart/backup-2010/idi.csv /afs/slac/package/pinger/motion-chart/backup-2010/hdi.csv /afs/slac/package/pinger/motion-chart/backup-2010/cpi.csv /afs/slac/package/pinger/motion-chart/backup-2010/doi.csv /afs/slac/package/pinger/motion-chart/backup-2010/create-motion-chart-pages.pl /afs/slac/package/pinger/motion-chart/backup-2010/pop.csv /afs/slac/package/pinger/motion-chart/backup-2010/create-motion-chart-page.pl /afs/slac/package/pinger/motion-chart/.create-motion-chart-page.pl.swo /afs/slac/package/pinger/motion-chart/idi-20100809.csv /afs/slac/package/pinger/motion-chart/pop-20190629.csv /afs/slac/package/pinger/motion-chart/demographics-20190630.csv /afs/slac/package/pinger/motion-chart/junk /afs/slac/package/pinger/motion-chart/ir.csv /afs/slac/package/pinger/motion-chart/ye.csv /afs/slac/package/pinger/motion-chart/.generate-demographics-from-csv.pl.swp /afs/slac/package/pinger/motion-chart/demographics-20190713.csv /afs/slac/package/pinger/motion-chart/demographics-20190714.csv /afs/slac/package/pinger/motion-chart/demographics-20190714-new.csv [cottrell@pinger ~]$ ls -l umar.tar -rw-r--r-- 1 cottrell sf 7884800 Sep 23 14:59 umar.tar |
Or with compression which reduces the tarball from 7,884,800 Bytes to 1,509,685 Bytes
[cottrell@pinger ~]$ tar -czvf umar.tar.gz /afs/slac/package/pinger/motion-chart tar: Removing leading `/' from member names /afs/slac/package/pinger/motion-chart/ /afs/slac/package/pinger/motion-chart/generate-demographics-from-csv.pl ... [cottrell@pinger ~]$ ls -l umar.tar.gz -rw-r--r-- 1 cottrell sf 1509685 Sep 23 15:07 umar.tar.gz |
In addition one will also need to access:
The anonymous ftp space for user cottrell is at /afs/slac/public/users/cottrell, thus all we need to do is cp umar.tar.gz /afs/slac/public/users/cottrell
[cottrell@pinger ~]$ cp umar.tar.gz /afs/slac/public/users/cottrell/
[cottrell@pinger ~]$ ls -l /afs/slac/public/users/cottrell/umar.tar.gz
-rw-r--r-- 1 cottrell sf 1509685 Sep 23 15:51 /afs/slac/public/users/cottrell/umar.tar.gz
One can access the file from the web via ftp://ftp.slac.stanford.edu/users/cottrell
Log4perl has been used to create logs and alert the administrators (currently kalim & cottrell) of any anomaly.
The logging process maintains the logs by using two appenders (sinks):
The configuration file for the logging process is located at:
/afs/slac/package/pinger/motion-chart/log.conf |
The log itself is stored at:
/afs/slac/package/pinger/motion-chart/logs/motion_chart.log |
The configuration file is described below:
# Define the root logger and the filters. Set the trigger level to INFO
log4perl.logger = INFO, AppInfo, AppError
# Filter to match level ERROR. Use this filter for the appender
# which sends emails about critical errors
log4perl.filter.MatchError = Log::Log4perl::Filter::LevelMatch
log4perl.filter.MatchError.LevelToMatch = ERROR
log4perl.filter.MatchError.AcceptOnMatch = true
# Filter to match level INFO. Use this filter for the appender
# maintaining detailed logs.
log4perl.filter.MatchInfo = Log::Log4perl::Filter::LevelMatch
log4perl.filter.MatchInfo.LevelToMatch = INFO
log4perl.filter.MatchInfo.AcceptOnMatch = true
# Define the Error appender
log4perl.appender.AppError = Log::Dispatch::Email::MailSend
log4perl.appender.AppError.to = kalim@slac.stanford.edu, cottrell@slac.stanford.edu
log4perl.appender.AppError.subject = [Missing data in Motion Charts] see /afs/slac/package/pinger/motion-chart/logs/motion_chart.log
log4perl.appender.AppError.layout = \
Log::Log4perl::Layout::PatternLayout
log4perl.appender.AppError.layout.ConversionPattern = %d %p (%L) %m %n
# Inform about critical errors only
log4perl.appender.AppError.Threshold = ERROR
# log4perl.appender.AppError.Filter = MatchError
# Information appender
log4perl.appender.AppInfo = Log::Log4perl::Appender::File
log4perl.appender.AppInfo.filename = logs/motion_chart.log
log4perl.appender.AppInfo.mode = write
log4perl.appender.AppInfo.layout = \
Log::Log4perl::Layout::PatternLayout
log4perl.appender.AppInfo.layout.ConversionPattern = %d %p %L %m %n
# log4perl.appender.AppInfo.Filter = MatchInfo
|
The log maintaining detailed information may grow to a large size. To avoid this problem I wrote a script which rotates the logs on daily basis while maintaining archives of the last five days.
The script is located at:
/afs/slac/package/pinger/motion-chart/logs/maintain-logs.pl |
The archives at stored at:
/afs/slac/package/pinger/motion-chart/logs/archives/ |
The script maintain-logs is executed daily as a cron-job
Updated several of the metrics, in particular the PingER metrics (minimum and average RTT, jitter, loss, unreachability, throughput), plus IDI, HDI, Penetration and Population. the following were not updated and hence terminate in 2010: DOI, CPI, Users.
The frame for the chart was being displayed but with no contents.
Umar pointed out
Sat Oct 12 10:48:28 2019 create-motion-chart-pages.pl: finished, took 239 secs.
I followed the instructions for Enabling Flash in Chrome at https://support.digication.com/hc/en-us/articles/115003963468-Enabling-Flash-for-Google-Chrome-Windows-Macintosh-2.0.0 is installed in this browser
Chrome No change.
Firefox No change
Umar pointed the problem was related to having no data for Normalized throughput. If one chooses another metric for the y axis default then the content is displayed.
Normalized throughput is automatically updated by prm.pl via trscrontab on pinger@pinger.slac.stanford.edu
pinger;1300 30 07 04 1 * /afs/slac.stanford.edu/package/pinger/prm/prm.pl -lp true -c /afs/slac.stanford.edu/package/pinger/prm/prm-allyearly-continent.conf #takes 11 mins on PingER 11/24/2012 |
allyearly,?,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019
EDU.SLAC.STANFORD.PINGER-to-Afghanistan,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,
EDU.SLAC.STANFORD.PINGER-to-Albania,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,
Re-ran /afs/slac.stanford.edu/package/pinger/prm/prm.pl -lp true -c /afs/slac.stanford.edu/package/pinger/prm/prm-allyearly-slac.conf to yield https://www-iepm.slac.stanford.edu/pinger/prmout/nthroughput-EDU.SLAC.STANFORD.PINGER-country-allyearly.csv AKA /afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-EDU.SLAC.STANFORD.PINGER-country-allyearly.csv which has data in it.
Normalized throughput is metric_index 4 in data.setValue(datapoint_index,metric_index,region). It is added around line 444 in /afs/slac/package/pinger/motion-chart/create-motion-chart-page.pl. Using perl -d /afs/slac/package/pinger/motion-chart/create-motion-chart-page.pl -r Europe -f EDU.SLAC.STANFORD.PINGER -t and halting at
442: unless (!defined($data{"nthroughput"}->{$country}->{$interval})) {
443: if ( (($data{"nthroughput"}->{$country}->{$interval}) ne ".")
444 && (($data{"nthroughput"}->{$country}->{$interval}) ne "")
445 && (($data{"nthroughput"}->{$country}->{$interval}) != 0)){
446: print OUTFILE " data.setValue($j, 4, "
447 . $data{"nthroughput"}->{$country}->{$interval} . ");\n";
448 }
449 }
DB<15> p "data.setValue($j, 4, $data{'nthroughput'}->{$country}->{$interval});\n"
data.setValue(693, 4, 12240.9);
And it is written to OUTFILE /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.PINGER-allyearly.html
Looking in /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.PINGER-allyearly.html we see lines of the form:
data.addRows(1);
data.setValue(5, 0, 'Denmark');
data.setValue(5, 1, 1998);
data.setValue(5, 2, 'Europe');
data.setValue(5, 3, 210.241);
data.setValue(5, 4, 2919.71); #Note the metric_index=4, which is for nthroughput
data.setValue(5, 16, 360.566)
The metric_index of 4 is deduced from:
var data = new google.visualization.DataTable();
data.addColumn('string', 'Country');
data.addColumn('number', 'Year');
data.addColumn('string', 'Region');
data.addColumn('number', 'Average RTT (ms)');
data.addColumn('number', 'Normalized Throughput (Kbps)');
data.addColumn('number', 'Population (#)');
Then we ran
time /afs/slac/package/pinger/motion-chart/create-motion-chart-pages.pl
It took 225 seconds.
However when we look at
/afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart.html
we see:
data.addRows(1);
data.setValue(39, 0, 'Denmark');
data.setValue(39, 1, 1998);
data.setValue(39, 2, 'Europe');
data.setValue(39, 3, 210.241);
data.setValue(39, 16, 360.566)
We also verified that /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.PINGER-allyearly.html had not changed So the new data is not getting between create-motion-chart-page.pl to create-motion-chart-pages.pl
Also:
-bash-4.1$ ls -l /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 pinger sf 465357 Oct 16 12:44 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.PINGER-allyearly.html
Where as the file being displayed by
bash-4.1$ ls -l /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart.html
lrwxr-xr-x 1 pinger sf 103 Jun 29 21:54 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart.html -> /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.PINGER-allyearly.html
-bash-4.1$ ls -l /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 pinger sf 2152879 Oct 14 10:22 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.PINGER-allyearly.html
Which is unexpected since I thought re-running create-motion-chart-pages.pl would have updated the above file to today (Oct 16). I am unclear what
updated the file on Oct 14.
The possible files are:
bash-4.1$ ls -l /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart*EDU*allyearly*
-rw-rw-r-- 1 kalim sg 420174 Jun 29 18:17 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 588037 Oct 16 12:43 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Africa-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 97918 Jun 29 18:17 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 137858 Oct 16 12:43 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Balkans-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 101016 Jun 29 18:17 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 135565 Oct 16 12:44 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Central_Asia-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 65743 Jun 29 18:18 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 73439 Oct 16 12:44 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-East_Asia-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 334304 Jun 29 18:18 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 465357 Oct 16 12:44 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Europe-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 224179 Jun 29 18:18 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 310831 Oct 16 12:45 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Latin_America-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 141526 Jun 29 18:18 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 209985 Oct 16 12:45 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Middle_East-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 50387 Jun 29 18:18 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 66114 Oct 16 12:45 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-North_America-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 54170 Jun 29 18:19 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 75277 Oct 16 12:46 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-Oceania-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 103053 Jun 29 18:19 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 149525 Oct 16 12:46 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-S.E._Asia-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 87208 Jun 29 18:19 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 123603 Oct 16 12:46 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-South_Asia-EDU.SLAC.STANFORD.PINGER-allyearly.html
-rw-rw-r-- 1 kalim sg 1588320 Jun 29 18:19 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.N3-allyearly.html
-rw-rw-r-- 1 pinger sf 2152879 Oct 14 10:22 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.PINGER-allyearly.html
lrwxr-xr-x 1 pinger sf 64 Jun 29 21:51 /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart-all-EDU.SLAC.STANFORD.SLAC-allyearly.html -> /afs/slac/g/www/www-iepm/pinger/pinger-metrics-motion-chart.html
Also below are the crontab entries:
#Create the motion bubble plots, create-motion-chart-pages.pl takes ~ 4 mins:
pinger;30 00 18 * * * /afs/slac/package/pinger/motion-chart/create-motion-chart-pages.pl # Takes ~5 minutes
pinger;02 05 11 * * * /afs/slac/package/pinger/motion-chart/logs/manage-logs.pl -n 5 > /afs/slac/package/pinger/motion-chart/logs/log.stdout
|
