South Africa damage to the SAT-3 and WACS undersea cable Jan 16 2020

Report

See https://spectrum.ieee.org/tech-talk/telecom/internet/undersea-cable-repairs-south-africa for a very full discussion,  also early reports from https://mybroadband.co.za/news/internet/336416-how-south-africas-internet-survived-cable-breaks.html and the aftermath at https://mybroadband.co.za/news/internet/336426-slow-internet-speeds-in-south-africa-at-least-12-more-days.html.

Both the SAT-3 and WACS undersea cable were damaged Jan 16, 2020. The cable systems are deployed in the Atlantic Ocean and connect various African countries to Europe – including South Afric. According to reports, the issue with the SAT-3 is believed to have been located near Libreville, Gabon. WACs meanwhile is believed to have suffered two faults one of the coast of Congo and another near the UK. This affected more than a dozen countries besides South Africa. This left many South Africans with slow Internet connections to international websites. a. That leaves Africa Coast to Coast (ACE) and Equiano providing access to Europe and beyond and SAFE providing access via India and Malaysia and the South Africa Express (SAEx2) which will provide access via India and Malaysia in 2021. 

From https://mybroadband.co.za/news/internet/336426-slow-internet-speeds-in-south-africa-at-least-12-more-days.html

“[The cable ship] Leon Thevenin on its way to the WACS S1i cable grounds,” TENET said. “We have a provisional repair date of the 8th February for the WACS cable system.”

“SAT-3 will be attended to thereafter. This information may be subject to change.”

Cable Maps

Maps of the cables (from https://www.submarinecablemap.com/)  to Europe are shown below:

SAT3 (RFS 2002 April)	WACS (RFS 2012 May)	ACE (RFS 2012 December)	Equiano (RFS 2021)

Cables going East to Europe

EASSY RFS 2010 July	SEACOM/Tata TGN-Eurasia RFS 2009 July

Cables going East to India and Malaysia are shown below.

SAFE (RFS 2002 April)	South Asia Express (SAEx2) (RFS 2021 Q1)

RFS = Ready For Service so the current backups to Europe and probably the US are  ACE and SAFECOM both going East.

Result as observed by PingER

Looking at the minimum RTT seen from SLAC to 20 targets in Southern Africa we see a step function going from a 25%-75% of ~ 300-320 ms to 325-355ms. See the statistics below.

Minimum RTT measured every 30 minutes  for Sites in Southern AFRICA seen from EDU.SLAC.STANFORD.PINGER 

For the 7 targets in South Africa the 25%/median/75% before 2016 and after are ~294/300/306ms and 327/345/356ms.

Looking in more detail at the ping responses each hour from SLAC to TENET South Africa we see  a couple of sudden steps.

and in more detail

The first step occurred between 08:00am and 09:00am Jan 14th, the second step between 07:00am and 08:00am Jan 16th (both times are GMT). I can find no evidence of a failure on January 14th, however it sure looks like there was one. On further inspection, it appears that this step only appears for the TENET target site (see below for the minimum RTTs in msec measured each hour (0 1 ...) from SLAC to the target (Remote-Site)  on Jan 14th, 2020), and thus is not related to the cable problems. This can be seen more clearly in the plot below. The spreadsheets are available here.  

N.b. two sites (www.bos.gov.ls and www.dut.ac.za) do not appear to have been affected on January 16th or in the daily plots. Maybe they were not using SAT3 or WACS, for example maybe they were using ACE. According to ^ieee "TENET restored its overseas connections by switching to an alternative 80-gigabit-per-second service on the SEACOM and EASSy cables on Africa's east coast. "The latency ... increased due to this failover, but there are currently no reported issues of this being detrimental to connectivity," a representative of TENET said on 30 January." The impact on the PingER measurements from SLAC was that the minimum RTT went from ~280ms to 355ms.

One can also look at the various statistics for both Average RTT, losses and minimum RTTs, see below (the spreadsheet is seen here). The Average and Losses are heavily influenced by the congestion. The minimum is mainly influenced by the route change or change in the distance of the cable route. Also note that the impact on the minimum RTT is not seen until Jan 17th since some of the PingER January 16 measurements were made before the change of cables used. It is also worth noting as reported in ^ieee "TENET restored its overseas connections by switching to an alternative 80-gigabit-per-second service on the SEACOM and EASSy cables on Africa's east coast. '"The latency ... increased due to this failover, but there are currently no reported issues of this being detrimental to connectivity," a representative of TENET said on 30 January.'

Traceroute

The route measured 19:00 hrs 1/28/2020 PST from SLAC to South Africa goes East across the US via Chicago, Washington to London and thence S. Africa:

Tue Jan 28 18:59:32 2020: executing exec(traceroute -m 30 -q 1 -w 1 -A 196.24.110.67 140)=traceroute from 134.79.197.214(www.slac.stanford.edu) to  196.24.110.67(brunsvigia.tenet.ac.za) for 134.79.138.4
traceroute to 196.24.110.67 (196.24.110.67), 30 hops max, 140 byte packets
 1  rtr-serv01-02-serv01-dmz-webserv.slac.stanford.edu (134.79.197.131) [AS3671]  0.788 ms
 2  rtr-core2-p2p-serv01-01.slac.stanford.edu (134.79.254.65) [AS3671]  0.591 ms
 3  rtr-fwcore1-trust-p2p-core1.slac.stanford.edu (134.79.254.134) [AS3671]  1.123 ms
 4  rtr-core1-p2p-fwcore1-untrust.slac.stanford.edu (134.79.254.137) [AS3671]  1.538 ms
 5  rtr-border1-7k-p2p-core1.slac.stanford.edu (134.79.252.173) [AS3671]  1.482 ms
 6  rtr-border2-7k-p2p-border1-7k.slac.stanford.edu (192.68.191.214) [AS3671]  1.614 ms
 7  sunn-cr5-ip-p2p-border2-7k.slac.stanford.edu (192.68.191.233) [AS3671]  1.858 ms
 8  134.55.40.145 (134.55.40.145) [AS293]  4.548 ms
 9  134.55.40.150 (134.55.40.150) [AS293]  46.417 ms
10  washcr5-ip-a-chiccr5.es.net (134.55.36.46) [AS293]  63.692 ms
11  *
12  londcr5-ip-c-aofacr5.es.net (134.55.37.22) [AS293]  138.575 ms
13  *
14  lt-0-0-0-2-ldn1-ir1.net.tenet.ac.za (155.232.208.28) [AS2018]  331.160 ms
15  londcr5-sdn-a-newycr5.es.net (134.55.220.21) [AS293]  137.494 ms
16  xe-11-3-0-201-ldn1-ir1.net.tenet.ac.za (155.232.212.106) [AS2018]  137.352 ms
17  lt-0-3-0-1-mtz1-pe1.net.tenet.ac.za (155.232.152.20) [AS2018]  329.393 ms
18  et-1-0-0-0-ams1-ir1.net.tenet.ac.za (155.232.1.80) [AS2018]  328.407 ms
19  xe-1-3-0-0-isd1-pe2.net.tenet.ac.za (155.232.1.60) [AS2018]  339.301 ms
20  eth3-39-cpt3-p1.net.tenet.ac.za (155.232.1.37) [AS2018]  354.698 ms
21  lt-0-3-0-0-mtz1-pe1.net.tenet.ac.za (155.232.152.21) [AS2018]  330.381 ms
22  et-0-0-1-0-cpt2-p2.net.tenet.ac.za (155.232.64.64) [AS2018]  354.586 ms
23  eth3-39-cpt3-p1.net.tenet.ac.za (155.232.1.37) [AS2018]  353.473 ms
24  et-0-0-1-0-cpt3-p2.net.tenet.ac.za (155.232.64.61) [AS2018]  353.617 ms
25  196.24.111.221 (196.24.111.221) [AS2018]  357.517 ms
26  pinger-cpt-01.tenet.ac.za (196.24.110.67) [AS2018]  354.664 ms
traceroute -m 30 -q 1 -w 1 -A 196.24.110.67 140 took 5secs. Total script traceroute.pl time=5secs,

AS3671 is ESnet and  AS2018 is TENET

^ieee Submarine Cable Repairs Underway in South Africa. IEEESpectrum, 3 February 2020, see https://spectrum.ieee.org/tech-talk/telecom/internet/undersea-cable-repairs-south-africa