Report
See https://spectrum.ieee.org/tech-talk/telecom/internet/undersea-cable-repairs-south-africa for a very full discussion, also early reports from https://mybroadband.co.za/news/internet/336416-how-south-africas-internet-survived-cable-breaks.html and the aftermath at https://mybroadband.co.za/news/internet/336426-slow-internet-speeds-in-south-africa-at-least-12-more-days.html.
Both the SAT-3 and WACS undersea cable were damaged Jan 16, 2020. The cable systems are deployed in the Atlantic Ocean and connect various African countries to Europe – including South Afric. According to reports, the issue with the SAT-3 is believed to have been located near Libreville, Gabon. WACs meanwhile is believed to have suffered two faults one of the coast of Congo and another near the UK. This affected more than a dozen countries besides South Africa. This left many South Africans with slow Internet connections to international websites. a. That leaves Africa Coast to Coast (ACE) and Equiano providing access to Europe and beyond and SAFE providing access via India and Malaysia and the South Africa Express (SAEx2) which will provide access via India and Malaysia in 2021.
“[The cable ship] Leon Thevenin on its way to the WACS S1i cable grounds,” TENET said. “We have a provisional repair date of the 8th February for the WACS cable system.”
“SAT-3 will be attended to thereafter. This information may be subject to change.”
Cable Maps
Maps of the cables (from https://www.submarinecablemap.com/) to Europe are shown below:
SAT3 (RFS 2002 April) | WACS (RFS 2012 May) | ACE (RFS 2012 December) | Equiano (RFS 2021) |
---|---|---|---|
Cables going East to Europe
EASSY RFS 2010 July | SEACOM/Tata TGN-Eurasia RFS 2009 July |
---|---|
Cables going East to India and Malaysia are shown below.
SAFE (RFS 2002 April) | South Asia Express (SAEx2) (RFS 2021 Q1) |
---|---|
RFS = Ready For Service so the current backups to Europe and probably the US are ACE and SAFECOM both going East.
Result as observed by PingER
Looking at the minimum RTT seen from SLAC to 20 targets in Southern Africa we see a step function going from a 25%-75% of ~ 300-320 ms to 325-355ms. See the statistics below.
Minimum RTT measured every 30 minutes for Sites in Southern AFRICA seen from EDU.SLAC.STANFORD.PINGER
Tick | min | 25th% | avg | median | 75th% | 90th% | 95th% | max | iqr | std dev | # pairs |
---|---|---|---|---|---|---|---|---|---|---|---|
20Jan01 | 280.444 | 303.117 | 312.614 | 316.141 | 321.182 | 325.130 | 325.531 | 329.477 | 18.065 | 12.637 | 20 |
20Jan02 | 280.498 | 297.056 | 309.626 | 314.285 | 317.884 | 324.135 | 324.826 | 325.112 | 20.828 | 12.793 | 20 |
20Jan03 | 280.517 | 303.163 | 312.347 | 316.178 | 321.177 | 324.280 | 324.960 | 329.499 | 18.014 | 12.433 | 20 |
20Jan04 | 280.461 | 303.154 | 312.365 | 316.158 | 321.146 | 324.649 | 324.978 | 329.489 | 17.992 | 12.457 | 20 |
20Jan05 | 280.447 | 303.144 | 312.051 | 316.130 | 321.151 | 324.690 | 324.943 | 329.539 | 18.007 | 12.395 | 20 |
20Jan06 | 280.451 | 296.392 | 310.530 | 315.753 | 320.903 | 324.763 | 324.981 | 329.496 | 24.511 | 13.628 | 20 |
20Jan07 | 280.468 | 296.436 | 310.834 | 315.841 | 321.240 | 324.988 | 325.506 | 329.450 | 24.804 | 13.775 | 20 |
20Jan08 | 280.476 | 303.144 | 311.708 | 314.254 | 321.011 | 325.028 | 325.524 | 329.554 | 17.867 | 12.457 | 20 |
20Jan09 | 280.518 | 303.151 | 311.782 | 314.255 | 319.939 | 323.939 | 325.036 | 328.314 | 16.788 | 11.658 | 20 |
20Jan10 | 280.465 | 300.798 | 309.671 | 311.542 | 315.925 | 324.952 | 325.496 | 328.318 | 15.127 | 12.842 | 20 |
20Jan11 | 280.474 | 301.814 | 309.915 | 311.603 | 315.171 | 325.012 | 325.430 | 329.738 | 13.357 | 12.748 | 20 |
20Jan12 | 280.515 | 296.730 | 311.775 | 315.344 | 322.416 | 324.938 | 329.779 | 330.877 | 25.686 | 14.228 | 20 |
20Jan13 | 280.429 | 305.275 | 311.683 | 313.730 | 316.397 | 325.007 | 325.491 | 329.706 | 11.122 | 12.046 | 20 |
20Jan14 | 280.531 | 303.975 | 309.400 | 311.155 | 315.672 | 318.695 | 324.584 | 329.770 | 11.697 | 11.031 | 20 |
20Jan15 | 296.380 | 305.315 | 313.079 | 312.597 | 318.192 | 324.055 | 325.876 | 336.248 | 12.877 | 9.760 | 20 |
20Jan16 | 296.726 | 305.008 | 313.498 | 312.889 | 319.100 | 325.828 | 328.229 | 336.273 | 14.092 | 10.343 | 20 |
20Jan17 | 301.328 | 330.032 | 348.988 | 348.759 | 355.767 | 387.082 | 392.328 | 408.823 | 25.735 | 25.731 | 20 |
20Jan18 | 301.306 | 330.101 | 345.969 | 348.990 | 355.285 | 366.246 | 385.621 | 389.538 | 25.184 | 21.546 | 20 |
20Jan19 | 301.268 | 329.970 | 346.692 | 346.227 | 355.276 | 385.389 | 389.286 | 396.487 | 25.306 | 24.191 | 20 |
20Jan20 | 301.307 | 327.376 | 342.864 | 340.419 | 352.677 | 369.400 | 386.111 | 389.233 | 25.301 | 22.337 | 20 |
20Jan21 | 301.146 | 327.248 | 346.865 | 342.833 | 352.699 | 396.258 | 412.843 | 415.990 | 25.451 | 30.435 | 20 |
20Jan22 | 301.350 | 323.368 | 341.581 | 339.179 | 352.644 | 369.499 | 386.202 | 389.188 | 29.276 | 23.344 | 20 |
20Jan23 | 305.829 | 324.277 | 342.455 | 339.436 | 352.019 | 369.405 | 386.094 | 389.259 | 27.742 | 22.234 | 20 |
20Jan24 | 305.722 | 323.285 | 341.580 | 338.587 | 352.055 | 366.430 | 386.448 | 389.562 | 28.770 | 22.446 | 20 |
20Jan25 | 305.680 | 324.671 | 342.747 | 340.071 | 351.452 | 369.217 | 385.966 | 389.601 | 26.781 | 22.061 | 20 |
20Jan26 | 305.761 | 323.724 | 343.401 | 349.319 | 356.299 | 386.491 | 389.553 | 389.553 | 32.575 | 22.892 | 19 |
20Jan27 | 305.794 | 321.125 | 341.578 | 338.668 | 352.731 | 369.613 | 386.166 | 389.261 | 31.606 | 23.321 | 20 |
For the 7 targets in South Africa the 25%/median/75% before 2016 and after are ~294/300/306ms and 327/345/356ms.
Looking in more detail at the ping responses each hour from SLAC to TENET South Africa we see a couple of sudden steps.
and in more detail
The first step occurred between 08:00am and 09:00am Jan 14th, the second step between 07:00am and 08:00am Jan 16th (both times are GMT). I can find no evidence of a failure on January 14th, however it sure looks like there was one. On further inspection, it appears that this step only appears for the TENET target site (see below for the minimum RTTs in msec measured each hour (0 1 ...) from SLAC to the target (Remote-Site) on Jan 14th, 2020), and thus is not related to the cable problems. This can be seen more clearly in the plot below. The spreadsheets are available here.
N.b. two sites (www.bos.gov.ls and www.dut.ac.za) do not appear to have been affected on January 16th or in the daily plots. Maybe they were not using SAT3 or WACS, for example maybe they were using ACE. According to ieee "TENET restored its overseas connections by switching to an alternative 80-gigabit-per-second service on the SEACOM and EASSy cables on Africa's east coast. "The latency ... increased due to this failover, but there are currently no reported issues of this being detrimental to connectivity," a representative of TENET said on 30 January." The impact on the PingER measurements from SLAC was that the minimum RTT went from ~280ms to 355ms.
One can also look at the various statistics for both Average RTT, losses and minimum RTTs, see below (the spreadsheet is seen here). The Average and Losses are heavily influenced by the congestion. The minimum is mainly influenced by the route change or change in the distance of the cable route. Also note that the impact on the minimum RTT is not seen until Jan 17th since some of the PingER January 16 measurements were made before the change of cables used. It is also worth noting as reported in ieee "TENET restored its overseas connections by switching to an alternative 80-gigabit-per-second service on the SEACOM and EASSy cables on Africa's east coast. '"The latency ... increased due to this failover, but there are currently no reported issues of this being detrimental to connectivity," a representative of TENET said on 30 January.'
Traceroute
The route measured 19:00 hrs 1/28/2020 PST from SLAC to South Africa goes East across the US via Chicago, Washington to London and thence S. Africa:
Tue Jan 28 18:59:32 2020: executing exec(traceroute -m 30 -q 1 -w 1 -A 196.24.110.67 140)=traceroute from 134.79.197.214(www.slac.stanford.edu) to 196.24.110.67(brunsvigia.tenet.ac.za) for 134.79.138.4 traceroute to 196.24.110.67 (196.24.110.67), 30 hops max, 140 byte packets 1 rtr-serv01-02-serv01-dmz-webserv.slac.stanford.edu (134.79.197.131) [AS3671] 0.788 ms 2 rtr-core2-p2p-serv01-01.slac.stanford.edu (134.79.254.65) [AS3671] 0.591 ms 3 rtr-fwcore1-trust-p2p-core1.slac.stanford.edu (134.79.254.134) [AS3671] 1.123 ms 4 rtr-core1-p2p-fwcore1-untrust.slac.stanford.edu (134.79.254.137) [AS3671] 1.538 ms 5 rtr-border1-7k-p2p-core1.slac.stanford.edu (134.79.252.173) [AS3671] 1.482 ms 6 rtr-border2-7k-p2p-border1-7k.slac.stanford.edu (192.68.191.214) [AS3671] 1.614 ms 7 sunn-cr5-ip-p2p-border2-7k.slac.stanford.edu (192.68.191.233) [AS3671] 1.858 ms 8 134.55.40.145 (134.55.40.145) [AS293] 4.548 ms 9 134.55.40.150 (134.55.40.150) [AS293] 46.417 ms 10 washcr5-ip-a-chiccr5.es.net (134.55.36.46) [AS293] 63.692 ms 11 * 12 londcr5-ip-c-aofacr5.es.net (134.55.37.22) [AS293] 138.575 ms 13 * 14 lt-0-0-0-2-ldn1-ir1.net.tenet.ac.za (155.232.208.28) [AS2018] 331.160 ms 15 londcr5-sdn-a-newycr5.es.net (134.55.220.21) [AS293] 137.494 ms 16 xe-11-3-0-201-ldn1-ir1.net.tenet.ac.za (155.232.212.106) [AS2018] 137.352 ms 17 lt-0-3-0-1-mtz1-pe1.net.tenet.ac.za (155.232.152.20) [AS2018] 329.393 ms 18 et-1-0-0-0-ams1-ir1.net.tenet.ac.za (155.232.1.80) [AS2018] 328.407 ms 19 xe-1-3-0-0-isd1-pe2.net.tenet.ac.za (155.232.1.60) [AS2018] 339.301 ms 20 eth3-39-cpt3-p1.net.tenet.ac.za (155.232.1.37) [AS2018] 354.698 ms 21 lt-0-3-0-0-mtz1-pe1.net.tenet.ac.za (155.232.152.21) [AS2018] 330.381 ms 22 et-0-0-1-0-cpt2-p2.net.tenet.ac.za (155.232.64.64) [AS2018] 354.586 ms 23 eth3-39-cpt3-p1.net.tenet.ac.za (155.232.1.37) [AS2018] 353.473 ms 24 et-0-0-1-0-cpt3-p2.net.tenet.ac.za (155.232.64.61) [AS2018] 353.617 ms 25 196.24.111.221 (196.24.111.221) [AS2018] 357.517 ms 26 pinger-cpt-01.tenet.ac.za (196.24.110.67) [AS2018] 354.664 ms traceroute -m 30 -q 1 -w 1 -A 196.24.110.67 140 took 5secs. Total script traceroute.pl time=5secs,
AS3671 is ESnet and AS2018 is TENET
ieee Submarine Cable Repairs Underway in South Africa. IEEESpectrum, 3 February 2020, see https://spectrum.ieee.org/tech-talk/telecom/internet/undersea-cable-repairs-south-africa