Work in Progress

Motivation

The PingER metrics motion charts are primarily used to visualize the trends in the Internet end-to-end performance statistics measured from the 100+ PingER monitoring nodes spread worldwide. Having gathered data since 1998, the charts enable the users to study the trends, step changes, significant improvements/degradations with the help of these 4-dimensional charts.

The different sets of charts (w.r.t. regions) enables the users to study the progress made by countries in comparison to their neighbours as well as the world in general.

Below are some examples:

Observations made from Stanford Linear Accelerator Center (SLAC), USA:

View of the world from SLAC in Aug 2008 (min RTT vs norm. throughput & Internet penetration)	View of the world from SLAC in Sep 2007 (min RTT vs jitter, HDI & Internet users)	View of the world from SLAC in 2008 (packet loss vs min RTT, region & throughput)

Observations made from CERN, Geneva, Switzerland:

View of South Asia from CERN in Aug 2008 (avg RTT vs norm. throughput, packet loss & Internet penetration)	View of the Balkans from CERN in Sep 2008 (min RTT vs jitter, norm. throughput & Internet users)	View of Africa from CERN in Sep 2008 (min RTT vs norm. throughput, Internet penetration & unreachability)

Usage

By default, PingER metrics motion chart shows a motion scatter/bubble plot of the ping (100 Byte) average RTT in msec. ( x ) versus the normalized derived TCP throughput in kbits/sec ( y ). Each bubble represents a country. By default the bubbles are colored by region and their sizes are proportional to the Internet users. The motion is the time. The time granularity is by year. Click on the play button to initiate the motion.

To change the metric represented by the size of the bubble use the 'size' combo box to the right of the chart.
Click on the pull down box of an axis' label to change the metric that is displayed on an axis or in a table. Other PingER metrics that are available include minimum RTT, jitter and unreachability.
Move the mouse over a bubble to identify the values for the bubble.
Linear or logarithmic scales may be selected using the combo box along the axis.
Move the mouse over a coloured legend symbol to blink the balloons associated with the symbol's color.
To follow the changes in detail simply move the slider bar with the mouse.
One can select a bubble by clicking on it to provide a label for the bubble that persists with the motion. The trails check box leaves a trail to follow the motion of selected bubbles. N.B. selected bubbles determine how far the data to be displayed goes back in time.
By clicking the links at the right top of the page, one may select the vantage point, the observed region and the granularity of the measurements.

Implementation Details

HTML output

To create a motion chart, the data must be populated in a data structure (as stated by Google Visualization APIand copied below):

<html>
  <head>
    <script type="text/javascript" src="http://www.google.com/jsapi"></script>
    <script type="text/javascript">
      google.load("visualization", "1", {packages:["motionchart"]});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();

              data.addColumn('string', 'Country');
              data.addColumn('date', 'Month');
              data.addColumn('string', 'Region');
              data.addColumn('number', 'Average RTT (ms)');
              data.addColumn('number', 'Normalized Throughput (Kbps)');
              data.addColumn('number', 'Internet Users (#)');
              data.addColumn('number', 'Internet Penetration (%)');
              data.addColumn('number', 'Population (#)');
              data.addColumn('number', 'Minimum RTT (ms)');
              data.addColumn('number', 'Packet Loss (%)');
              data.addColumn('number', 'Unreachability (%)');
              data.addColumn('number', 'Jitter (ms)');
              data.addColumn('number', 'DOI Index');
              data.addColumn('number', 'HDI Index');
 
              data.addRows(1);
              data.setValue(0, 0, "Pakistan");
              data.setValue(0, 1, new Date(2007,02,01));
              data.setValue(0, 2, 'South Asia');
              data.setValue(0, 3, 421.346);
              data.setValue(0, 4, 1568.95315647059);
              data.setValue(0, 5, 7500000);
              data.setValue(0, 6, 4.7);
              data.setValue(0, 7, 172800051);
              data.setValue(0, 8, 311.482);
              data.setValue(0, 9, 1.331);
              data.setValue(0, 10, 15.591);
              data.setValue(0, 11, 19.906);
              data.setValue(0, 12, 0.26);
              data.setValue(0, 13, 0.551);

              data.addRows(1);
              data.setValue(1, 0, "Sri Lanka");
              data.setValue(1, 1, new Date(2007,02,01));
              data.setValue(1, 2, 'South Asia');
              data.setValue(1, 3, 418.664);
              data.setValue(1, 4, 2760.35944717647);
              data.setValue(1, 5, 280000);
              data.setValue(1, 6, 1.4);
              data.setValue(1, 7, 21128773);
              data.setValue(1, 8, 270.494);
              data.setValue(1, 9, 0.304);
              data.setValue(1, 10, 8.871);
              data.setValue(1, 11, 42.093);
              data.setValue(1, 12, 0.33);
              data.setValue(1, 13, 0.743); 

 
              data.addRows(1);
              data.setValue(2, 0, "Pakistan");
              data.setValue(2, 1, new Date(2007,03,01));
              data.setValue(2, 2, 'South Asia');
              data.setValue(2, 3, 397.897);
              data.setValue(2, 4, 1394.21896911765);
              data.setValue(2, 5, 7500000);
              data.setValue(2, 6, 4.7);
              data.setValue(2, 7, 172800051);
              data.setValue(2, 8, 305.533);
              data.setValue(2, 9, 1.248);
              data.setValue(2, 10, 12.917);
              data.setValue(2, 11, 31.064);
              data.setValue(2, 12, 0.26);
              data.setValue(2, 13, 0.551);

              data.addRows(1);
              data.setValue(3, 0, "Sri Lanka");
              data.setValue(3, 1, new Date(2007,03,01));
              data.setValue(3, 2, 'South Asia');
              data.setValue(3, 3, 357.202);
              data.setValue(3, 4, 4634.87959694118);
              data.setValue(3, 5, 280000);
              data.setValue(3, 6, 1.4);
              data.setValue(3, 7, 21128773);
              data.setValue(3, 8, 274.092);
              data.setValue(3, 9, 0.144);
              data.setValue(3, 10, 36.319);
              data.setValue(3, 11, 11.889);
              data.setValue(3, 12, 0.33);
              data.setValue(3, 13, 0.743);

 
              data.addRows(1);
              data.setValue(4, 0, "Pakistan");
              data.setValue(4, 1, new Date(2007,04,01));
              data.setValue(4, 2, 'South Asia');
              data.setValue(4, 3, 372.313);
              data.setValue(4, 4, 1918.28198552422);
              data.setValue(4, 5, 7500000);
              data.setValue(4, 6, 4.7);
              data.setValue(4, 7, 172800051);
              data.setValue(4, 8, 310.145);
              data.setValue(4, 9, 0.969);
              data.setValue(4, 10, 13.844);
              data.setValue(4, 11, 24.146);
              data.setValue(4, 12, 0.26);
              data.setValue(4, 13, 0.551);

              data.addRows(1);
              data.setValue(5, 0, "Sri Lanka");
              data.setValue(5, 1, new Date(2007,04,01));
              data.setValue(5, 2, 'South Asia');
              data.setValue(5, 3, 397.434);
              data.setValue(5, 4, 2450.70159022405);
              data.setValue(5, 5, 280000);
              data.setValue(5, 6, 1.4);
              data.setValue(5, 7, 21128773);
              data.setValue(5, 8, 298.214);
              data.setValue(5, 9, 0.410);
              data.setValue(5, 10, 7.527);
              data.setValue(5, 11, 6.979);
              data.setValue(5, 12, 0.33);
              data.setValue(5, 13, 0.743);

        var chart = new google.visualization.MotionChart(document.getElementById('chart_div'));
        chart.draw(data, {width: 850, height:500});
      }
    </script>
  </head>

  <body>
    <div id="chart_div" style="width: 850px; height: 500px;"></div>
  </body>
</html>

The data must comply with the formatting requirements (as stated by Google Visualization APIand copied below):

The first column must be of type 'string' and contain the entity names (e.g., "Pakistan", "Sri Lanka" in the example above).
The second column must contain time values. Time can be expressed in a few different ways:
- Years. Column type: 'number'. Example: 2008.
- Month, day and year. Column type should be 'date' and the values should be javascript Date instances.
- Week numbers. Column type should be 'string' and the values should have the pattern YYYYWww, which conforms to ISO 8601. Example: '2008W03'.
- Quarters. Column type should be 'string' and the values should have the pattern YYYYQq, which conforms to ISO 8601. Example: '2008Q3'.
Subsequent columns can be of type 'number' or 'string'. Number columns will show up in the drop down menus for X, Y, Color and Size axes. String columns will only appear in the drop down menu for Color.

Note that the data must be loaded in a chronologically ascending order. See the code above as an example.

Relevant files

Following are the files involved in the process of generating the HTML page:

Application:

File path: /afs/slac/package/pinger/motion-chart/create-motion-chart-page.pl

The script that defines the input arguments to generate the relevant .html files:

File path: /afs/slac/package/pinger/motion-chart/create-motion-chart-pages.pl

Demographics:

File path: /afs/slac/package/pinger/motion-chart/population.csv

 
File format:
#Country, Year, Population, Internet Users, Internet Penetration, DOI, HDI
United States, 2008, 303824646, 213364918, 70.7, 0.62, 0.951

PingER data (obtained from prmout) Average RTT, Jitter (IPDV), Minimum RTT, Normalized Throughput, Packet Loss and Unreachability as seen from CERN, SLAC and ICTP:

File paths:
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-CH.CERN.N2-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-CH.CERN.N2-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-CH.CERN.N2-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-EDU.SLAC.STANFORD.N3-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-EDU.SLAC.STANFORD.N3-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-EDU.SLAC.STANFORD.N3-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-IT.ICTP.N1-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-IT.ICTP.N1-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/average_rtt-IT.ICTP.N1-country-monthly.csv

/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-CH.CERN.N2-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-CH.CERN.N2-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-CH.CERN.N2-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-EDU.SLAC.STANFORD.N3-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-EDU.SLAC.STANFORD.N3-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-EDU.SLAC.STANFORD.N3-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-IT.ICTP.N1-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-IT.ICTP.N1-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/ipdv-IT.ICTP.N1-country-monthly.csv

/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-CH.CERN.N2-country-allyearly.cvs
/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-CH.CERN.N2-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-CH.CERN.N2-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-EDU.SLAC.STANFORD.N3-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-EDU.SLAC.STANFORD.N3-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-EDU.SLAC.STANFORD.N3-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-IT.ICTP.N1-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-IT.ICTP.N1-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/minimum_rtt-IT.ICTP.N1-country-monthly.csv

/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-CH.CERN.N2-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-CH.CERN.N2-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-CH.CERN.N2-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-EDU.SLAC.STANFORD.N3-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-EDU.SLAC.STANFORD.N3-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-EDU.SLAC.STANFORD.N3-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-IT.ICTP.N1-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-IT.ICTP.N1-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/nthroughput-IT.ICTP.N1-country-monthly.csv

/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-CH.CERN.N2-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-CH.CERN.N2-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-CH.CERN.N2-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-EDU.SLAC.STANFORD.N3-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-EDU.SLAC.STANFORD.N3-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-EDU.SLAC.STANFORD.N3-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-IT.ICTP.N1-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-IT.ICTP.N1-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/packet_loss-IT.ICTP.N1-country-monthly.csv

/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-CH.CERN.N2-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-CH.CERN.N2-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-CH.CERN.N2-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-EDU.SLAC.STANFORD.N3-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-EDU.SLAC.STANFORD.N3-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-EDU.SLAC.STANFORD.N3-country-monthly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-IT.ICTP.N1-country-allyearly.csv
/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-IT.ICTP.N1-country-last60days.csv
/afs/slac/g/www/www-iepm/pinger/prmout/unreachability-IT.ICTP.N1-country-monthly.csv

Country to region mapping:

File path: /afs/slac.stanford.edu/www/comp/net/mon/countries.tsv

Implementation

Following is the pseudo code for implementation:

1. Load the country to region mapping to the hash-table "%region" which uses the country
    names (as defined by PingER) as the index.
    e.g. $region{'Kenya'} = Africa and
           $region{'Korea, Rep'} = East Asia.

 
2. Load the PingER data to the data structure "%data" which uses PingER metrics as the index.

    Here each entry stores a reference to a unique hash-table defined to store the PingER data
    related to a country. Understandably the second hash-table uses country names as the index.
    Each entry of the hash-table stores a reference to another hash-table used for storing
    performance measurements for the country. Here timestamps are used as indices.
    e.g. $data{'minrtt'}->{'Kenya'}->{'1998'},
           $data{'packetloss'}->{'United States'}->{'Oct2008'} and
           $data{'ipdv'}->{'Pakistan'}->{'08Oct27'}
    The metrics used are: minrtt, avgrtt, nthroughput, ipdv, packetloss and unreachability.
 
3. Load the demographics to the data structure "%data" which uses labels as the index.
    Here each entry stores a reference to a unique hash-table defined to store the demographics
    of a country. The second hash-table uses country names as the index. Each entry of
    the hash-table stores a reference to another hash-table used for storing statistics
    for the country. Here timestamps are used as indices.
    e.g. $data{'population'}->{'Kenya'}->{'1998'},
           $data{'doi'}->{'United States'}->{'Oct2008'} and
           $data{'inetpenetration'}->{'Pakistan'}->{'08Oct27'}

    The metrics used are: population, inetusers, inetpenetration, doi and hdi.

 
4. Generate the .html file in compliance with the Google Visualization API constraints.
    Here the data is loaded in the following format:
              data.addColumn('string', 'Country');
              data.addColumn('date', 'Month');
              data.addColumn('string', 'Region');
              data.addColumn('number', 'Average RTT (ms)');
              data.addColumn('number', 'Normalized Throughput (Kbps)');
              data.addColumn('number', 'Internet Users (#)');
              data.addColumn('number', 'Internet Penetration (%)');
              data.addColumn('number', 'Population (#)');
              data.addColumn('number', 'Minimum RTT (ms)');
              data.addColumn('number', 'Packet Loss (%)');
              data.addColumn('number', 'Unreachability (%)');
              data.addColumn('number', 'Jitter (ms)');
              data.addColumn('number', 'DOI Index');
              data.addColumn('number', 'HDI Index');
 
    N.B. the data must be uploaded in a chronologically ascending order.
 
5. Create .html files for each vantage point (SLAC, CERN and ICTP)

Miscellaneous Details

Tool: The interactive graph was generated using the Google Motion Chart Widget.
Data: The data presented here was collected by the PingER project, processed by prmout. The same data is available in tabular format. The statistics of Population were acquired from the U.S. Census Bureau, the country to region mapping was obtained by the geographical database maintained by the PingER project, where as the Internet usage statistics were taken from Internet World Stats. Please note that while all the statistics were acquired over several years (i.e. since 1998), the Internet usage statistics were documented in Nov. 2007.
Loading: The time to load and render the web page is largely determined speed of the link, the file length (about a MByte that is determined by the number of metrics and frequency of data points and the number of metrics) and the speed of the client rendering.
Metrics: Average RTT (ms), Normalized Throughput (Kbps), Internet Users (#), Internet Penetration (%), Population (#), Minimum RTT (ms), Packet Loss (%), Unreachability (%), Jitter (ms), DOI Index, HDI Index,
Authors: Umar Kalim & Les Cottrell 10/10/2008, version 1.3. Idea champion: Shahryar Khan.