Effects of Mediterranean Fibre Cuts December 2008

By Les Cottrell, SLAC

The purpose of this study is to examine the effccts of fibre cuts in the Mediterranean in December 2008. The fibre cuts were reported by many sources including the BBC, Blomberg News, Al Jazeera, Orange and Wired on December 19th 2008, also see the article in Wikipedia. We decided to have a look at the impact on  ping performance using the PingER data.

For performance issues following the previous Mediterranean fibre cut in January 2008, see:
https://confluence.slac.stanford.edu/display/IEPM/Effects+of+Fibre+Outage+through+Mediterranean;

Affected Countries

From  Orange at http://www.orange.com/en_EN/press/press_releases/cp081219en.html

France Telecom observed that 3 major underwater cables were cut: "Sea Me We 4" at 7:28am, "Sea Me We3" at 7:33am and FLAG at 8:06am.
The causes of the cut, which is located in the Mediterranean between Sicily and Tunisia, on sections linking Sicily to Egypt, remain unclear.

Most of the B to B traffic between Europe and Asia is rerouted through the USA.
Traffic from Europe to Algeria and Tunisia is not affected, but traffic from Europe to the Near East and Asia is interrupted to a greater or lesser extent (see country list below).
Part of the internet traffic towards Réunion is affected as well as 50% towards Jordan.
A first appraisal at 7:44 am UTC gave an estimate of the following impact on the voice traffic (in percentage of out of service capacity):
-    Saudi Arabia: 55% out of service
-    Djibouti: 71% out of service
-    Egypt: 52% out of service
-    United Arab Emirates: 68% out of service
-    India: 82% out of service
-    Lebanon: 16% out of service
-    Malaysia: 42% out of service
-    Maldives: 100% out of service
-    Pakistan: 51% out of service
-    Qatar: 73% out of service
-    Syria: 36% out of service
-    Taiwan: 39% out of service
-    Yemen: 38% out of service
-    Zambia: 62% out of service 

Maps of the cables and the affected countries can be found at SEAMEWE3 and SEAMEWE4. A Telegeography map of the cable routes in the Mediterranean is seen below and there is a report of progress on January 22nd:

Metric Motion Charts

Probably the most dynamic way to show the effects is to use the PingER Metrics Motion Charts for the regions most badly affected: Middle East, South Asia and Africa (Egypt and Libya). These show the variation of multiple metrics as a function of time. To assist in this we have captured the charts for 21 days starting December 14th 2008. The charts for the three regions are shown below. To view them, for example, click on the Africa thumbnail, then click on the right arrow to play the data. After viewing this, select the size metric (by default to be Population (#)) to be Packet Loss (%). Under Select choose the country of interest to be trailed (e.g. Egypt), then click on the arrow to playback the performance of the throughput, the average RTT and packet loss all the while trailing the bubble for Egypt.  Note the dramatic change in increase of loss (bubble size), and average RTT starting on December 19th, and the recovery after December 22nd.  The effect is more dramatic if one chooses a log scale for the x axis (Packet Loss (%)).

Losses

The measured effects on the losses seen from N. America (SLAC in California) to hosts in N. Africa (Egypt, Sudan); Middle East (Bahrein, Palestine, UAE, Oman, Jordan, Lebanon, Saudia Arabia); S. Asia (Sri Lanka, Maldives, Pakistan) are huge. As seen from PingER daily loss data measured from SLAC, comparing the losses on Friday Dec 19 with those on Mon-Thu (15-18 Dec) the losses have increased by a factor of 5 to 30 times for most of the hosts monitored in the above countries.

Unreachability

Unreachability is defined in PingER terms as when none of the 10 pings sent each 30 mins gets a response. Looking at the PingER data despite the magnitude of these cuts and the large impact on losses there is no indication  of unrteachability (100%) losses to any of the hosts measured in the affected regions. This does not mean that applications that require higher bandwidth than pings (1000bits/sec for a short period (~10secs) ), low loss or jitter or Round Trip Time (RTT) will not be badly impacgfed causing them to fail or to be effectively useless. On the other hand it does show that Internet connectivity was successfully maintained due to the use of redundant paths etc.

Round Trip Time (RTT)

Examples of the increase in RTT can be seen below in the  plot of RTT seen from SLAC to two hosts in Egypt, a host at the Lahore School of Economics Pakistan and a host at the NCP provider in Pakistan, a host in Bahrein, one in Oman, one in Bangladesh (www.brauniversity.net), one in Jordan and one in Saudi Arabia. All show an increase in RTT  around 9-10am on Friday Dec 19th 2009 (UDT). In the case of the www.tanta.edu.eg host in Egypt no loss of connectivity was observed at the time of the cut. However for the frcu.edun.eg besides the increase in RTT frequent losses of connectivity are observed and the link appears to be restored on January 23rd (www.frcu.edu.eg recovered at about the same time). In the case of Lahore there was a short period of no connectivity and over the days of the outage the RTT varied dramatically dropping back to normal and full recovery did not appear to be complete by January 27th. NCP had a small increase in RTT and a long period of no connectivity later on Friday stretching into Saturday. The Bahrein host experienced large changes in RTT between daytime (busy times, more congested, more queing and longer RTTs) and night-times. The increases in RTT during the daytime for Oman were extremely large (~ 5 seconds), however the problem appears by-passed/fixed by December 23rd. The Bangladesh host follows a very reproducible pattern switching between 250ms at night and 300ms RTT for the rest of the time and the effect continues to at least the end of December 30th. The Jordan host is less regular in its behavior and there is little effect on the weekend (December 20th & 21st) following the start of the outage. The Saudi Arabai host RTT increases and even at nights and wekends the miniumum RTT is greater than before the cuts. These graphs illustrate the variability of the impact even within a country.

 
 Another view of the effect of the RTT increases is seen in the plot below of the median RTTs measured (thed error bars are the 25% and 75%-tiles) from 3 monitoring hosts in Pakistan (NUST, COMSATS and NCP) to FNAL and SLAC (i.e. 6 pairs of monitor/remote hosts) for the first 19 days of December 2008.

Jitter

We measure the jitter as the Inter Packet Delay Variability (IPDV) (see Tutorial). The plots below show the RTT and Loss plus the IPDV and maximum and minimum (of 10 pings) measured for a host (Sudan University of Science and Technology (SUSTECH)) in the Sudan. The dramatic increases in RTT, jiiter and losses are clearly seen foillowing the December 19th cable cuts. The graph on the right shows the drop in throughput derived from the average RTTs and losses as:

throughput - 1460*8/(RTT(in ms)*sqrt(loss)) kbits/sec

Minimum RTT

The minimum (of 10 pings) seen from SLAC to selected hosts in various affected countries is seen below. The Top Level Domain of each host appears at the start of the host name (EG = Egypt,  IN = India, LK = Sri Lanka, OM = Oman, PK = Pakistan, SD = Sudan).  Not all hosts in each country was similarly affected, due to the use of different carriers. The sudden changes in miniumum RTT are presumably due the choice of different routes to carry the traffic. It is seen that steps in minimum RTT occur after the initial cut as carriers changed the routing. Be aware that just because the minimum ERTT may return to its previous fibre cut values does not mean the performance has been restored, the route though being short may have insufficinent capacity and still be heavily congested.

Traceroutes from SLAC to Egypt

Following the outage the routes changed dramatically and pretty often as the carriers found alternate routes.  Before the cut the routes went Eastwards across the US via ESnet, then via GEANT (including routers at Vienna and Italy) to Egypt. Immediately following the cut starting at 23:43 Dec 18th (PST), traffic still went Eastwards but via NTT and London and FLAG across Europe to Alexandria. At 15:32 on Dec 19th the route switched to going Westwards via Singapore using TATA as the carrier beyond the US. Later it went Westwards from Palo Alto in California to Tokyo and Hong Kong using FLAG (until ~13:35 Dec 20th). At 14:52 Dec 20th, it switched to TeleGlobe and TATA going westwards via Singapore. Then on Dec 21st at 03:15 it switched to using Cogent and went eastwards via Boston, London and Paris.

TCP Throughput

 We derive the throughput from the loss and RTT measurements as described elsewhere.  We then calculate the Residual_Bandwidth_Fraction

This results were sorted by Residual_Bandwidth_Fraction to create the following table of the worst affected countries:

The derived throughputs as a for tghe above countries  are shown below in a contour plot. The drastic reduction occuring December 19-20 2008 is quickly seen to the left.

Another contour plot of the RBF but for a longedr period and using:

is shown in the table below together with the RBF per country. It is seen that hosts in some countries such as UAE and Egypt seem to have recovered by December 23rd, others such as Sudan, Cape Verde, Bangladesh (one of two hosts) and Libya are still badly affected. Others such as Saudi Arabia and Jordan are partially recovered. Also note the discrepancies in the recovery of the Pakistani hosts. Tthe grpah to trhe right shows the impact in terms of the Reduced Bandwidth Fraction for the various hosts. The inclusion of Russia is probably a statistical anomaly given the variability of the Dervided TCP throughputs (Inter Quartile Range ~ 990 kbits/sec) measured to the Russian host, the same can be said for Zimbabwe.

More Information

• Information on Submarine Communications
• 2008 Submarine Cable Disruption from Wikipedia.
• Mediterranean Fibre Cable Cut: A RIPE NCC Analysis, February 2008
• Mother Earth Mother Board, Neal Stephenson, Wired, all about the laying of the longest underwater telephony cable in history. Stephenson goes into a lot of details as to how the cable is laid, what happens to the cable when it reaches shore, what is the cable made of, how does it work, how it is repaired etc.
• From Harvey Newman, Caltech:;

  Interestingly, however, the current (Nov. 2008) issue of the newsletter [Submarine Telecoms Forum], in addition to an article on the use of subsea cables for monitoring compliance to the International Nuclear Test Ban Treaty,  features a story on Navy Undersea Cable Systems on Page 34. The entire issue can be found here:http://www.subtelforum.com/issues/WFN_41.pdf.Also give a look at the cable ship sighting section on Page 54 where you'll find A Global Guide to the Latest Known Locations of the World's Cable Ships as of May 2008, as provided by Lloyds List.And for those interested in how subsea cables are upgraded to higher throughput rates, see Same Pipes, Faster Speed, by Andy Lumsden on p. 38.

• The following quote from Telegeography on April 11, 2011 is also interesting concerning by-passing the effects of such outages in future.

  Carriers face a different challenge in Egypt. Egypt serves a pivotal role in international connectivity, because all undersea cables between Europe and Asia transit the country. Carriers have been seeking to introduce five undersea cable systems connected to and across Egypt to meet burgeoning capacity requirements in the Middle East, East Africa, and India, but have been delayed for over a year by regulatory problems in Egypt. This has left carriers scrambling to identify alternative routes.

  A variety of new projects have emerged that seek to provide terrestrial connectivity directly to Middle Eastern countries. In 2010 a consortium of carriers launched the Jeddah-Amman-Damascus-Istanbul (JADI) network, which links Saudi Arabia and Turkey via Jordan and Syria. Another group has plans for the Regional Cable Network (RCN), which will link the UAE to Turkey via Saudi Arabia, Jordan and Syria. The recently-unveiled Europe Persia Express Gateway (EPEG) project hopes to connect Germany to Oman via Russia and Iran by 2012 through a mix of terrestrial and undersea segments. While the planned undersea cables in Egypt will ultimately be completed, the development of new terrestrial options will further enhance the reliability of the global network.