ePinger Project at SLAC

Table of Contents

Introduction

This is a project suggested by Bebo White to build and validate a PingER Measurement Agent (MA) based on an inexpensive piece of hardware called a Raspberry Pi (see more about Raspberry Pi) using a linux distribution as the Operating System (see more about Raspbian). If successful one could consider using these in production: reducing the costs, power drain (they draw about 2W of 5V DC power compared to typically over 100W for a deskside computer or 20W for a laptop) and space (credit card size). This is the same type of power required for a smartphone so appropriate off the shelf products including a battery and solar cells are becoming readily available. Thus the Raspberry could be very valuable for sites in developing countries where cost, power utilization and to a lesser extent space may be crucial. 

PingER

PingER (Ping End-to-end Reporting) is the name given to the Internet End-to-end Performance Measurement (IEPM) project to monitor end-to-end performance of Internet links. It is led by SLAC and development has included NUST/SEECS (formerly NIIT), FNAL, and ICTP/Trieste, together with UM,UNIMAS and UTM in Malaysia. Originally, in 1995 it was for the High Energy Physics community, however, this century it has been more focused on measuring the Digital Divide from an Internet Performance viewpoint. The project now involves measurements to over 700 sites in over 160 countries, and we are actively seeking new sites to monitor and MA sites for this project, as well as people interested in our data. It uses the ubiquitous ping facility so no special software has to be installed on the targets.

Measurements are made by ~60 measurement Agents (MAs) in 23 countries. They make measurements to over 700 targets in ~ 160 countries containing more than 99% of the world's connected population. The measurement cycle is scheduled at roughly 30 minute intervals. At each measurement cycle, each MA issues a set of pings to each target, stopping when it receives 10 ping responses or it has issued 30 ping requests. From each  set of pings one can derive various metrics such as minimum (Min) ping Round Trip Time (RTT) response, average (Avg) RTT, maximum (Max)  RTT, standard deviation (Stdev) of RTTs, 25% probability of RTT, 75% probability of RTT, Inter Quartile Range (IQR, ), loss, and reachability (host is unreachable if get 100 % loss).

The data is publicly available and since the online data goes back to January 1998, it provides 19 years of historical data on Internet performance.

Raspberry Pi Model and Specifications

The Raspberry belongs to Bebo White and it is the version 1 of Raspberry Pi, model B. The cost is about $25/each + costs of the SD card. The Raspberry purchased  has 512MB RAM, on a 700Mhz ARM CPU and a 32GB SD Card ($18)  was used. It has 2 USB ports, 1 100Mb/s Ethernet interface and 1 HDMI port. For reasons of economy it does not have a Real Time Clock (RTC).  Instead, the Pi is intended to be connected to the Internet via Ethernet or WiFi, updating the time automatically from the global ntp (nework time protocol) servers (see https://learn.adafruit.com/adding-a-real-time-clock-to-raspberry-pi/overview).

The Voltage requirement for the power is 5V+-5%. Keep in mind that it is necessary to have a keyboard, a mouse and a HDMI monitor to do the installation process, but once PingER is working they are not necessary anymore.  We measured the power (Wattage) during normal use and it is 2.7 Watts. When using the Dell mouse, with an LED, powered from the Raspberry Pi it crept up to 3.2Watts. If we have a 5000mA-hour fully charged solar cell battery, then the Raspberry Pi might be expected to run for ~ 25 hours. However we need to be able to recharge from the sun. If we had a 10W 5V solar panel then with an efficiency factor of 2.5 we may estimate the recharge time of the battery to be ~6.25 hours.

Operating System

The installed system is called Raspbian a Debian Linux variant. The OS had Perl, Make, dig, ping and mail installed. We accessed it through the graphic interface of Raspbian. We just had to install: Apache and XML::Simple.

Installation of PingER2 MA on Raspberry Pi

The first step, before start the installation process we had to change the hostname in Raspbian.

sudo nano /etc/hostname 

sudo /etc/init.d/hostname.sh

Notice that the hostname here must include the domain. So, our hostname was pinger.raspberry.slac.stanford.edu. 

Then, we followed the instructions in PingER End-to-end Reporting version 2. After installing the PingER2 monitoring code, we installed the ping_data gathering agent,  the traceroute server and the pinger_trimmer following the instructions.

When we tested everything, we got a error message on the pingerCronStat.stderr file telling us that the ParserDetails.ini file was missing. We used this approach to fix this.

We entered the machines as monitors in the PingER meta data base of hosts.

Obs: Make sure to change the default password for Raspbian.

Requirement

Two major points need to be addressed before we can comfortably deploy Raspberry PI MAs.

1. The Raspberry Pi PingER MA must be robust and reliable. It needs to run for months to years with no need for intervention. This still needs to be verified, so far the Raspberry Pi has successfully run without intervention for over a month. This has included automatic recovery after two test power outages. 
2. The important metrics derived from the measurements made by the Raspberry Pi should not be significantly different from those made by a bare metal PingER MA, or if they are then this needs to be understood.

We define the important metrics measured by PingER as being the minimum, average, median and jitter of the RTTs, the packet loss, together with the reachability (i.e. a target host is unreachable when no ping requests are responded by the target host). These are the main metrics that impact applications such as throughput, voice over IP, streaming video, haptics, and estimating the geolocation of a host by pinging it from well know landmarks.  Such differences might result in significant discontinuities in the metric measurements if we were to change the monitoring host from a bare metal server to a Raspberry Pi.

In this report the jitter is represented by the Inter Packet Delay (IPD), the absolute values of the IPF (Abs(IPD)) and the Stdev(IPD).

Methodology

Since we believed, both a priori and from observations, that the major impact on the measurements was the network and not the servers' hardware or OS, we chose to make detailed analysis of PingER measurements between two representative MAs at SLAC and  two sites well separated from SLAC and hence with very different RTTs. The MAs at SLAC were:

1. The Dell Poweredge 2650 bare metal pinger.slac.stanford.edu server running Red Hat Linux  2.6.32-504.8.1.el6.i686 with specs.

   103cottrell@pinger:~$uname -a
   Linux pinger 2.6.32-504.8.1.el6.i686 #1 SMP Fri Dec 19 12:14:17 EST 2014 i686 i686 i386 GNU/Linux

2. The Raspberry Pi pinger-raspberry.slac.stanford.edu an armv61 running Gnu Linux (see above) with specs.

   pi@pinger-raspberry ~ $ uname -a
   Linux pinger-raspberry.slac.stanford.edu 3.18.11+ #781 PREEMPT Tue Apr 21 18:02:18 BST 2015 armv6l GNU/Linux

Both were in the same building at SLAC, i.e. roughly at latitude 37.4190 N, longitude 122.2085 W, but on different floors. The machines are about 30 metres apart or about 0.0003 msec based on the speed of light in a direct fibre. 

The measurements used were:

• between pinger.slac.stanford.edu and pinger-raspberry.slac.stanford.edu, and
• from both pinger.slac.stanford.edu and  pinger-raspberry.slac.stanford.edu to targets at varying distances and hence varying minimum RTTs from SLAC.
• from two representative MAs at sitka.triumf.ca and ping.cern.ch to pinger.slac.stanford.edu and pinger-raspberry.edu

Information on the various hosts involved is given in the table below. 

The Directivity in the table below provides a measure of how direct the route is between the MA and target. The Directivity is given as:
 Directivity = great circle distance between MA & target [in km] / (RTT [ms] * 100 [km/ms]
The Directivity is <=  1, and a value of 1 means the RTT is the same as given by the speed of light in a fibre. 

49.2475 N 

123.2308 W

46.23 N

6.07 E

The following plots are the raw metric data from the MA measurements from which we have to craft our text (with some plot examples).

Example target = pinger.unimas.my (~220 msec.)

For both 100Byte and 1000 Byte pings (not shown above) the round trip time series for RTTs have similar behaviour and there are similar losses 7:10 (pinger : pinger-raspberry for 100 Byte pings), note the different Y scales for losses. The losses are about double for 1000Byte pings.

Time Series	Frequency Distributions

Example target sitka.triumf.ca (~22msec.)

For 100Byte the round trip time series for RTTs did not have similar behaviour. We noticed a great change mainly in the maximum round trip time. The average minimum RTT did not change that much. Another point about pinger-raspberry is that it increases significantly the  RTT for near nodes (about ~1ms). The difference is greater than if we compare a node which is in a long distance.

Time series	Frequency distributions

pinger.slac.stanford.edu and pinger-raspberry.slac.stanford.edu

Now, we compared the RTT between pinger and pinger-raspberry. They are located in the same network and the RTT should be very small. However, as noticed before pinger-raspberry has a greater maximum RTT than pinger. The average RTT also has some difference, but now as much as the maximum time has. Note that the second graph represents the third graph using the same scale as the first (pinger graph).

pinger to Pinger-raspberry	pinger-raspberry to pinger

Using full set of pings for RTT frequency distributions

The frequency plots above are for the frequencies of the minimum, average and maximum RTTs.  Below we show the frequencies when we take the individual pings (usually 10 assuming little loss) for all the ping RTTs in each measurement set.

pinger to pinger-raspberry	pinger-raspberry to pinger

Frequency distribution for absolute interpacket delays.

The magnitude of the RTT is very dependent on the distance of the path between the source and destination. Many applications such as voice over IP, video streaming, or haptics are very dependent on the variability or jitter of the RTT. The jitter is often more dependent on the network edges compared to the RTT. There are many ways to calculate the jitter (see for example http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html#variable). We calculate the inter packet delay (IPD)and the absolute IPD and display the frequency distributions and statistics. 

	pinger to pinger-raspberry	pinger-raspberry to pinger
IPD
Abs(IPD)

To sitka.triumf.ca from SLAC

	pinger.slac.stanford.edu to sitka.triumf.ca	pinger-raspberry.slac.stanford.edu to sitka.triumf.ca
Time Series
Frequency distribution RTT
Frequency distribution Abs(IPD)
Frequency distribution IPD

From sitka.triumf.ca to SLAC

	sitka.triumf.ca to pinger.slac.stanford.edu	sitka.triumf.ca to pinger-raspberry.slac.stanford.edu
Time series
Frequency Distribution of RTT
Frequency distribution of Abs(IPD)
Frequency distribution of IPD

To sitka.triumf.ca from SLAC

	To ping.cern.ch from pinger	To ping.cern.ch from pinger-raspberry
Time series
Frequency Distribution RTT
Frequency Distribution Abs(IPD)
Frequency Distribution IPD

Results

The table below shows the various aggregate metrics measured from MA to target comparing pinger.slac.stanford with pinger-raspberry.slac.stanford.edu. The columns are arranged in pairs. The first of each pair is for pingerr.slac.stanford.edu, the second for pinger-raspberry.slac.stanford.edu. Each pair is measured over the same time period. Different pairs are measured over different time periods.

The errors (+-) are estimated as follows: 

• The Min RTT error is the IQR of the minimum RTTs from each set of pings for the time period.
• The Avg RTT error is the standard deviation of all the average RTTs from each set of pings for the time period
• The Median RTT error is the IQR of all the individual RTTs from all sets of measurements for the time period
• The IQR IPD error is the IQR of the hourly IQRs for the time period  (see Spreadsheet)

To find the probability of the distributions overlapping use a nomogram of mean differences versus error ratios given in Overlapping Normal Distributions. John M. Linacre for normal distributions.

We also use the Z Tests

Spreadsheet

Kolmogorov-Smirnov Test

The Kolmogorov-Smirnov test (KS-test) tries to determine if two datasets differ significantly. The KS-test has the advantage of making no assumption about the distribution of data. In other words it is non-parametric and distribution free. The method is explained here and makes use of an Excel tool called "Real Statiscs". The tests were made using the raw data and distributions, both methods had similar results except for the 100Bytes Packet that  had a great difference in the results. The results using raw data says both samples does not come from the same distribution with a significant difference, however if we use distributions the result says that only the 1000Bytes packet does not come from the same distribution. Bellow you will find the graphs for the distributions that were created and the cumulative frequency in both cases plotted one above other (in order to see the difference between the distributions).

If D-stat is greater than D-crit the samples are not considerated from the same distribution with a (1-Alpha) of accuracy. Remember that D-stat is the maximum difference between the two cumulative frequency curves.

Source: http://www.real-statistics.com/non-parametric-tests/two-sample-kolmogorov-smirnov-test/

Futures

Extension to Android systems 

Topher: Here is where we would like you to say something about the potential Android opportunity.

Installation process

The installation procedures for a PingER MA are relatively simple, but do require a Unix knowledgeable person to do the install and it typically takes a couple hours and may require a few corrections pointed out by the central PingER admin. It is possible to pre-configure the Raspberry Pi at the central site and ship it pre-configured to the MA site. However that requires funding the central site Raspberry Pi acquisitions, may raise issues of on-going commitment, and may not be acceptable for the Cyber security folks at the MA site. We are looking at simplifying the install process, possibly by creating an ISO Image

Robustness and Reliability

This still needs to be demonstrated in the field. We also need to more fully understand the solar power requirements.