By Les Cottrell and Shahryar Khan
In March and April 2007 we worked with people from Palestine and the University at Buffalo to install PingER Measurement Points (MPs) in Al Quds University (AQU) in Abu Dis on the West Bank and the Islamic University of Gaza (IUG) in Gaza City. They were both connected to the Mediterranean fibre optic cable though at first the host used at AQU was routed through Barak an Israeli commercial Internet porovider. There is a probability the Palestinian hosts will lose their connection to the Mediterranean fibre optic cable in Summer 2007.
Warren Matthews of Georgia Tech stepped in and assist IUGaza in getting PingER2 running. There were problems with the cronjob and possibly ntpd. IUG has been monitoring successfully since April 7th, 2007. They did not monitor data all day on April 22. There have been problems on 2 days gathering the data where I have either contacted the IUGaza site contact (Apr 28th to fix the web server) or had to re-run the gathering script. It was having problems again on May 3-7, 2007 and has been reported to the contact.
On April 3rd the Al-Quds Open University (OAU) in Ramallah said they would prepare a new machine and install software. I responded on April 4th and April 17th, but have heard nothing since. I prompted them again 5/4/07 and 5/9/07.
The locations of the sites are shown below.
Pinger CoverageThe sites from which there is PingER monitoring are shown in red below, sites which are monitored by all monitoring hosts (these are referred to as beacon sites and also include monitor sites) are shown in blue, and other monitored remote sites are shown in green. From SLAC PingER monitors about 460 sites, CERN used to monitor about 132 sites, however after re-installing in December 2006) they now only monitor beacon sites (56). ICTP Trieste monitors 105 sites but only has data going back to September 2006. To the right of the PingER map we also show a map which defines how we are assigning countries to regions.
PingER Coverage Worldwide
Worldwide ComparisonBefore comparing regions of the world it is useful to look at a World Map of Internet Users. This shows that for most the developed world (US and Canada, W. Europe, Japan, Taiwan, S. Korea) typically 40% or more of the people have Internet connectivity while for Africa according to ITU figures for 2004 it is less than 3% and for Sub-Saharan Africa closer to 1.8% and without South Africa 2% closer to 1.3%.
Packet Loss as seen from N. America
min RTT as seen from N. America
The left hand figure shows the packet loss to various regions of the world as seen from N. America. Since losses are fairly distance independent no attempt has been made to normalize the data. It is seen that the world divides up into two major super-regions: N. America, Europe, E. Asia and Oceania (mainly Australia and New Zealand) with losses below 0.1%, and Latin America, C. Asia, Russia, S.E. Asia, S. Asia and Africa with losses > 0.1% and as high as as a few per-cent. All countries are improving exponentially, but Africa is falling further behind most regions.
The second figure from the left shows the drop in the Minimum RTT from 2002-2006. The large step for S. Asia in 2003 was due to the change over from satellite to fibre. Central Asia (also Afghanistan) has hardly moved in its minimum RTT since it continues to use geostationary satellites.
The third figure from the left shows the unreachability of world regions seen from the US. A host is deemed unreachable if all pings of a set fail to respond. It shows the fragility of the link and is mainly distance independent (the reasons for fragility are usually in the last mile, the end site or host). Again the developed regions - US and Canada, E. Asia, and Oceania - have the lowest unreachability (< 0.3%) while the other regions vary from 0.7% to 2%. Africa shows poor statistics and is not improving, where as S. Asia has the second worst results.
The last graph shows the jitter (or variability of RTT) of world regions seen from the US. The jitter is defined as the Inter Quartile Range (IQR) of the Inter Packet Delay Variability (IPDV i = RTT i - RTT i-1). Jitter is relatively distance independent, it measures congestion, and has little impact on the Web and email. It decides the length of VoIP codec buffers and impacts streaming. We see the usual division into developed versus developing regions.
MOS for various RegionsThe telecommunications industry uses the Mean Opinion Score (MOS) as a voice quality metric. The values of the MOS are: 1= bad; 2=poor; 3=fair; 4=good; 5=excellent. A typical range for Voice over IP is 3.5 to 4.2 (see VoIPTroubleShooter). In reality, even a perfect connection is impacted by the compression algorithms of the codec, so the highest score most codecs can achieve is in the 4.2 to 4.4 range.
There are three factors that significantly impact call quality: latency, packet loss, and jitter. We calculate the jitter using the Inter Packet Delay Variability (IPDV) , see the tutorial for further details.
Most tool-based solutions calculate what is called an "R" value and then apply a formula to convert that to an MOS score. Then the R to MOS calculation is relatively standard. The R value score is from 0 to 100, where a higher number is better. To convert latency, loss, and jitter to MOS we follow Nessoft's method. The graphs below shows the Exponentially Weighted Moving Average (using EWMI i = alpha * EWMI i-1 + (1 - alpha) * Obs i where alpha = 0.7 and EWMI 1 = Obs 1) for the MOS as seen from the W. Coast of America (SLAC) and Switzerland (CERN). N.B. MOS values of one are reported for heavy loss (loss > 40 %).
Mean Opinion Score as seen from the US
Mean Opinion Score as seen from CERN
Mean Opinion Score as seen from Europe
CERN monitors fewer remote hosts than SLAC (56 versus over 400) so the data is not as complete in the CERN case. Comparing the two graphs, it can be seen that, as expected, the MOS is better for the shorter distances (i.e. CERN to Europe is better than SLAC to Europe, SLAC to N. America is better than CERN to N. America.).
It is also seen in both graphs that the Balkans, Russia and Latin America improved dramatically in 2000-2002. Much of Latin America and Russia moved from satellite to land lines in this period. It can be seen from the above plot that VoIP ought to be successful between SLAC and the US, Europe, E. Asia, Russia, Latin America and the Mid East (all above MOS = 3.5). S. E. Asia is marginal, S. Asia people will have to be very tolerant of one another, and C. Asia and Africa are pretty much out of the question in general. In general the CERN graphs looks similar to the SLAC graph to the various regions, except that S. E. Asia is worse for CERN than SLAC as is S. Asia
The third graph shows the Mean Opinion Score (MOS) from Europe two various regions. We have five monitoring sites in Europe (one at CERN, one at ICTP, one in Germany, and two at UK. The improvement in Latin America and Russia in 2002 is the result of shift from satellite to fiber. The drop for Russia in Sep, 2006 is because we installed a new version of PingER and it started monitoring 9 hosts in Russia whereas previously it was monitoring 20 hosts. For Central Asia the number of sites went up from 3 to 15 in Sep 2006, so the latter results are a better indication of the overall performance of Central Asia. For Sub Sahara Africa the coverage improved in Sep, 2006 (increased from 8 to 39 sites). So for sub Sahara Africa the results after Sep 2006 presents a better picture. South Asia as seen from Europe is performing better than as seen from US because MOS is derived from average RTT which is distance dependent.
TCP throughput from CERN & SLAC to World Regions
The graphs above show the derived TCP throughput using the Mathis formula. The macroscopic behavior of the TCP congestion avoidance algorithm by Mathis, Semke, Mahdavi & Ott in Computer Communication Review, 27(3), July 1997, provides a short and useful formula for the upper bound on the transfer rate:
Rate <= (MSS/RTT)*(1 / sqrt(p))
Rate: is the TCP transfer rate or throughputd
MSS: is the maximum segment size (fixed for each Internet path, typically 1460 bytes)
RTT: is the round trip time (as measured by TCP)
p: is the packet loss rate.
In the left hand two graphs the data points (average throughput per month) are fitted to exponential functions and for simplicity the trend lines only are shown. These lines enable us to see that Russia and Latin America are 6 years behind Europe, the Mid-East and SE Asia are 7 years behind, and S. Asia, C. Asia and Africa are 10 years or more behind. What is even more concerning is that Africa (in particular), South and Central Asia are not catching up.
The right hand most figure shows the yearly average derived TCP throughput normalized by the minimum RTT for the region (to reduce the proximity effects). It is seen that the throughputs are not simply exponential straigh lines, but typically change in steps as major changes are made in the routing and circuits.
The main Autonomous System (AS) domains in the routes from SLAC to the Middle East countries for the sites we monitor are shown in the figure to the left below. It can be seen that many providers are used. Al Quds connects to PADI2. The Palestinian sites (Gaza and the West Bank) use different providers from each other and from the neighboring countries. The figure on the right shows the ASs used to route from AQU to various countries of the world.
Routes Al Quds University to Middle East Countries
All routes from Al-Quds University (Palestine, West Bank) except to OAU go via Israel. Al-Quds is connected to OAU via PADI2 with an E1 link from Al-Quds. There are three hops all in the PADI2 ASN and the distance is a << 100 miles. The link is heavily congested during work hours. However, even when 25% congested the traceroute measured from Al-Quds University is close to 90 ms. This maybe due asymmetric routing
Routes to other than OAU and the Islamic University of Gaza also usually go via Europe to other Middle Eastern Countries.
The route from Al-Quds (Palestine West Bank) to Cyprus goes via Israel, Europe, United Kingdom.
The route to Iran goes via Israel, Europe, Turkey and then reaches Iran.
From Al-Quds to Jordan goes via Israel, US (Kansas), France and then reaches Jordan.
From West Bank to Gaza goes via Israel.
To Turkey goes via Israel, Europe and then reaches Turkey.
From Palestine to UAE goes via Israel, Europe, UK and then goes to Dubai.
From Palestine to Oman goes via Israel, Germany, Italy and then reaches Oman.
Here is a map of connectivity from Al Quds to other Middle East Countries.
Routes Al Quds University to Other Countries
Palestine to China: Palestine->Israel->Germany->Japan->China;
Palestine to US*:* Palestine-> Israel -> Europe-> US (California);
Palestine to New Zealand: Palestine -> Israel -> US (Florida) -> US (Kansas) -> US (New Mexico) -> New Zeland;
Palestine to Australia: Palestine-> Israel ->Europe -> US(Kansas) -> Australia;
Palestine to Russia: Palestine -> Israel -> Germany -> Russia(Moscow);
Palestine to Pakistan: Palestine -> Israel -> Europe -> Italy -> Pakistan;
Palestine to Greece: Palestine -> Israel -> Europe -> United Kingdom -> Greece;
Palestine to Taiwan: Palestine -> Israel -> US (Kansas) -> Tiawan;
Palestine to India: Palestine -> Israel -> Germany -> Netherlands -> United Kingdom -> India;
Palestine to Chile: Palestine -> Israel ->Germany -> US -> Mexico -> Chile;
Palestine to Argentina: Palestine -> Israel -> Europe -> US (Kansas) -> US (Texas) -> Argentina;
Palestine to Switzerland: Palestine -> Israel -> Europe -> United Kingdom -> Switzerland;
Palestine to Egypt: Palestine -> Israel -> Germany -> Netherlands -> United Kingdom -> Egypt.
Round Trip Times, Losses and Througputs for Palestine with Various Regions
The hourly losses and RTTs seen from SLAC for Israel, Al Quds and IUGaza are shown below. It is seen that israel has almost no variability so the link is un-congested. Both the Al Quds and IUGaza time series show strong diurnal changes indication congestion/queuing during work hours. Comparing Al Quds and IUGaza it appears they have similar minimum RTTs of about 250ms. IUGaza has improved with less congestion since April 17, 2007. Al Quds had two periods of unreachability (100% loss on the green line).
The fallowing graph shows the daily TCP throughput seen from SLAC to Al Quds University, IUGaza and the Palestine Standards Institution (www.psi.gov.ps). It can be seen that the performance is very variable, probably due to diurnal congestion. In addition IUGaza and the Palestinian Standards Institution have similar derived throughput, whil Al-Quds University is lower.
The monthly average Round Trip Times (RTTs) from Al Quds and IUGaza are shown below. The individual data points daily avergae(average_RTT) are shown to show the variability.
The minimum RTTs are shown below with the individual points (since minimum RTTs are more stable).
The packets losses are shown below: