Report
See https://spectrum.ieee.org/tech-talk/telecom/internet/undersea-cable-repairs-south-africa for a very full discussion, also early reports from https://mybroadband.co.za/news/internet/336416-how-south-africas-internet-survived-cable-breaks.html and the aftermath at https://mybroadband.co.za/news/internet/336426-slow-internet-speeds-in-south-africa-at-least-12-more-days.html.
Both the SAT-3 and WACS undersea cable were damaged Jan 16, 2020. This left many South Africans with slow Internet connections to international websites. The cable systems are deployed in the Atlantic Ocean and connect various African countries to Europe – including South Africa. That leaves Africa Coast to Coast (ACE) and Afric. According to reports, the issue with the SAT-3 is believed to have been located near Libreville, Gabon. WACs meanwhile is believed to have suffered two faults one of the coast of Congo and another near the UK. This affected more than a dozen countries besides South Africa. This left many South Africans with slow Internet connections to international websites. a. That leaves Africa Coast to Coast (ACE) and Equiano providing access to Europe and beyond and SAFE providing access via India and Malaysia and the South Africa Express (SAEx2) which will provide access via India and Malaysia in 2021.
“Leon “[The cable ship] Leon Thevenin on its way to the WACS S1i cable grounds,” TENET said. “We have a provisional repair date of the 8th February for the WACS cable system.”
“SAT-3 will be attended to thereafter. This information may be subject to change.”
Cable Maps
Maps of the cables (from https://www.submarinecablemap.com/) to Europe are shown below:
| SAT3 (RFS 2002 April) | WACS (RFS 2012 May) | ACE (RFS 2012 December) | Equiano (RFS 2021) |
|---|---|---|---|
 |  |  |  |
Cables going East to Europe
| EASSY RFS 2010 July | SEACOM/Tata TGN-Eurasia RFS 2009 July |
|---|---|
 |  |
Cables going East to India and Malaysia are shown below.
| SAFE (RFS 2002 April) | South Asia Express (SAEx2) (RFS 2021 Q1) |
|---|---|
 |  |
RfS RFS = Ready For Service so the current backups to Europe and probably the US are ACE going West and SAFE SAFECOM both going East.
Result as observed by PingER
Looking at the minimum RTT seen from SLAC to 20 targets in Southern Africa we see a step function going from a 25%-75% of ~ 300-320 ms to 325-355ms. See the statistics below.
Minimum RTT
...
measured every 30 minutes for Sites in Southern AFRICA seen from EDU.SLAC.STANFORD.PINGER
| Tick | min | 25th% | avg | median | 75th% | 90th% | 95th% | max | iqr | std dev | # pairs |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 20Jan01 | 280.444 | 303.117 | 312.614 | 316.141 | 321.182 | 325.130 | 325.531 | 329.477 | 18.065 | 12.637 | 20 |
| 20Jan02 | 280.498 | 297.056 | 309.626 | 314.285 | 317.884 | 324.135 | 324.826 | 325.112 | 20.828 | 12.793 | 20 |
| 20Jan03 | 280.517 | 303.163 | 312.347 | 316.178 | 321.177 | 324.280 | 324.960 | 329.499 | 18.014 | 12.433 | 20 |
| 20Jan04 | 280.461 | 303.154 | 312.365 | 316.158 | 321.146 | 324.649 | 324.978 | 329.489 | 17.992 | 12.457 | 20 |
| 20Jan05 | 280.447 | 303.144 | 312.051 | 316.130 | 321.151 | 324.690 | 324.943 | 329.539 | 18.007 | 12.395 | 20 |
| 20Jan06 | 280.451 | 296.392 | 310.530 | 315.753 | 320.903 | 324.763 | 324.981 | 329.496 | 24.511 | 13.628 | 20 |
| 20Jan07 | 280.468 | 296.436 | 310.834 | 315.841 | 321.240 | 324.988 | 325.506 | 329.450 | 24.804 | 13.775 | 20 |
| 20Jan08 | 280.476 | 303.144 | 311.708 | 314.254 | 321.011 | 325.028 | 325.524 | 329.554 | 17.867 | 12.457 | 20 |
| 20Jan09 | 280.518 | 303.151 | 311.782 | 314.255 | 319.939 | 323.939 | 325.036 | 328.314 | 16.788 | 11.658 | 20 |
| 20Jan10 | 280.465 | 300.798 | 309.671 | 311.542 | 315.925 | 324.952 | 325.496 | 328.318 | 15.127 | 12.842 | 20 |
| 20Jan11 | 280.474 | 301.814 | 309.915 | 311.603 | 315.171 | 325.012 | 325.430 | 329.738 | 13.357 | 12.748 | 20 |
| 20Jan12 | 280.515 | 296.730 | 311.775 | 315.344 | 322.416 | 324.938 | 329.779 | 330.877 | 25.686 | 14.228 | 20 |
| 20Jan13 | 280.429 | 305.275 | 311.683 | 313.730 | 316.397 | 325.007 | 325.491 | 329.706 | 11.122 | 12.046 | 20 |
| 20Jan14 | 280.531 | 303.975 | 309.400 | 311.155 | 315.672 | 318.695 | 324.584 | 329.770 | 11.697 | 11.031 | 20 |
| 20Jan15 | 296.380 | 305.315 | 313.079 | 312.597 | 318.192 | 324.055 | 325.876 | 336.248 | 12.877 | 9.760 | 20 |
| 20Jan16 | 296.726 | 305.008 | 313.498 | 312.889 | 319.100 | 325.828 | 328.229 | 336.273 | 14.092 | 10.343 | 20 |
| 20Jan17 | 301.328 | 330.032 | 348.988 | 348.759 | 355.767 | 387.082 | 392.328 | 408.823 | 25.735 | 25.731 | 20 |
| 20Jan18 | 301.306 | 330.101 | 345.969 | 348.990 | 355.285 | 366.246 | 385.621 | 389.538 | 25.184 | 21.546 | 20 |
| 20Jan19 | 301.268 | 329.970 | 346.692 | 346.227 | 355.276 | 385.389 | 389.286 | 396.487 | 25.306 | 24.191 | 20 |
| 20Jan20 | 301.307 | 327.376 | 342.864 | 340.419 | 352.677 | 369.400 | 386.111 | 389.233 | 25.301 | 22.337 | 20 |
| 20Jan21 | 301.146 | 327.248 | 346.865 | 342.833 | 352.699 | 396.258 | 412.843 | 415.990 | 25.451 | 30.435 | 20 |
| 20Jan22 | 301.350 | 323.368 | 341.581 | 339.179 | 352.644 | 369.499 | 386.202 | 389.188 | 29.276 | 23.344 | 20 |
| 20Jan23 | 305.829 | 324.277 | 342.455 | 339.436 | 352.019 | 369.405 | 386.094 | 389.259 | 27.742 | 22.234 | 20 |
| 20Jan24 | 305.722 | 323.285 | 341.580 | 338.587 | 352.055 | 366.430 | 386.448 | 389.562 | 28.770 | 22.446 | 20 |
| 20Jan25 | 305.680 | 324.671 | 342.747 | 340.071 | 351.452 | 369.217 | 385.966 | 389.601 | 26.781 | 22.061 | 20 |
| 20Jan26 | 305.761 | 323.724 | 343.401 | 349.319 | 356.299 | 386.491 | 389.553 | 389.553 | 32.575 | 22.892 | 19 |
| 20Jan27 | 305.794 | 321.125 | 341.578 | 338.668 | 352.731 | 369.613 | 386.166 | 389.261 | 31.606 | 23.321 | 20 |
The targets are
www.bac.ac.bw BW Africa EDU.SLAC.STANFORD.PINGER BW.BANKOFBOTSWANA.N1 ?r 318.350 315.112 318.421 318.471 318.347 316.128 316.113 313.872 313.840 310.265 319.300 322.674 315.800 315.672 312.367 312.494 345.173 333.960 333.886 327.376 327.248 324.411 324.277 324.183 324.671 324.704 321.125 . . . . pinger.slac.stanford.edu EDU North_America www.bankofbotswana.bw BW Africa EDU.SLAC.STANFORD.PINGER BW.BTC.NS1 ?B 318.105 310.232 318.059 318.033 317.995 315.570 315.655 313.392 313.424 309.400 308.416 321.490 315.272 315.280 312.064 312.033 342.591 335.493 335.089 328.815 328.344 317.865 321.564 321.678 322.408 322.357 320.194 . . . . pinger.slac.stanford.edu EDU North_America ns1.btc.bw BW Africa EDU.SLAC.STANFORD.PINGER LS.BOS.GOV.N1 ?r 320.642 320.646 320.649 320.677 320.617 320.672 320.684 320.641 318.165 312.819 312.769 311.337 312.826 312.796 312.827 319.100 319.119 319.101 319.085 319.139 319.119 319.180 324.089 320.882 321.582 321.854 318.699 . . . . pinger.slac.stanford.edu EDU North_America www.bos.gov.ls LS Africa EDU.SLAC.STANFORD.PINGER LS.LCE.AC.WWW ?r 321.182 321.169 321.177 321.146 321.151 321.208 321.240 321.211 319.939 313.324 313.304 313.291 312.696 313.268 313.279 313.284 323.389 323.391 323.414 323.429 322.093 322.055 326.344 326.385 329.391 . 325.995 . . . . pinger.slac.stanford.edu EDU North_America www.lce.ac.ls LS Africa EDU.SLAC.STANFORD.PINGER LS.LEO.CO.N1 ?B 329.477 313.719 329.499 329.489 329.539 329.496 329.450 329.554 328.314 328.318 329.738 329.779 329.706 329.770 321.155 321.234 332.555 332.555 332.472 328.800 328.813 329.410 329.425 320.615 331.170 328.207 327.932 . . . . pinger.slac.stanford.edu EDU North_America www.leo.co.ls LS Africa EDU.SLAC.STANFORD.PINGER NA.AGRINAMIBIA.COM.N1 ?r 315.936 315.946 315.953 315.984 315.937 315.936 316.027 316.014 315.989 315.925 314.801 316.014 315.944 315.959 315.901 316.053 392.328 389.538 389.286 389.233 415.990 389.188 389.259 389.562 389.601 389.553 389.261 . . . . pinger.slac.stanford.edu EDU North_America www.agrinamibia.com.na NA Africa EDU.SLAC.STANFORD.PINGER NA.MME.GOV.N1 ?B 316.347 316.362 316.402 316.332 316.322 316.364 316.384 316.482 316.415 315.327 315.171 316.427 316.397 316.419 316.430 316.317 387.082 385.621 385.389 386.111 412.843 386.202 386.094 386.448 385.966 386.491 386.166 . . . . pinger.slac.stanford.edu EDU North_America www.mme.gov.na NA Africa EDU.SLAC.STANFORD.PINGER SZ.ACAT.ORG.N1 ?B 324.809 324.826 324.280 324.649 324.690 324.763 325.506 325.524 322.184 325.496 325.430 320.066 325.491 304.647 325.876 325.828 348.544 352.605 348.279 352.677 352.699 352.644 350.042 348.319 350.738 352.707 352.731 . . . . pinger.slac.stanford.edu EDU North_America www.acat.org.sz SZ Africa EDU.SLAC.STANFORD.PINGER SZ.REALBROADBAND.CO ?r 325.130 325.112 324.960 324.978 324.943 324.981 324.988 325.028 325.036 324.952 325.012 324.938 325.007 303.975 324.055 323.997 352.055 350.766 346.496 350.842 350.888 350.876 350.843 346.518 350.261 350.920 350.916 . . . . pinger.slac.stanford.edu EDU North_America www.realbroadband.co.sz SZ Africa EDU.SLAC.STANFORD.PINGER SZ.UNISWA.ORG.WWW ?r 296.820 297.056 296.440 296.561 296.496 296.392 296.436 296.710 301.597 300.798 301.814 296.730 296.578 296.395 296.380 296.726 322.065 327.683 328.114 327.660 327.768 323.368 323.314 323.285 323.404 323.724 323.525 . . . . pinger.slac.stanford.edu EDU North_America www.uniswafoundation.org.sz SZ Africa EDU.SLAC.STANFORD.PINGER ZA.AC.NMMU.N1 ?r 303.117 290.335 303.163 303.154 303.144 290.439 290.395 303.144 303.151 290.444 290.414 290.340 303.145 303.139 303.150 303.204 349.023 349.006 348.992 349.006 348.695 348.981 349.006 348.978 349.005 349.595 349.569 . . . . pinger.slac.stanford.edu EDU North_America www.nmmu.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.BONTANY.UWC.AC.N1 ?r 314.822 314.852 314.703 314.684 314.675 314.589 314.686 314.635 314.669 314.630 314.688 314.674 314.634 315.537 336.248 336.273 357.306 356.769 356.722 357.327 357.296 357.309 357.595 357.630 357.645 357.923 357.904 . . . . pinger.slac.stanford.edu EDU North_America 196.21.45.139 ZA Africa EDU.SLAC.STANFORD.PINGER ZA.DUT.AC.N1 ?r 301.836 301.877 301.715 301.587 301.628 301.695 301.629 301.676 301.660 305.970 305.726 305.837 305.880 306.191 306.307 301.725 301.328 301.306 301.268 301.307 301.146 301.350 305.829 305.722 305.680 305.761 305.794 . . . . pinger.slac.stanford.edu EDU North_America www.dut.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.MUSEUMSNC.CO ?r 305.233 305.164 305.241 305.224 305.208 305.180 305.272 303.456 305.281 302.788 308.788 305.306 305.275 304.774 305.315 305.609 408.823 365.016 396.487 327.329 325.966 325.853 327.682 330.656 330.728 330.769 327.637 . . . . pinger.slac.stanford.edu EDU North_America www.museumsnc.co.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.RU.AC.N1 ?r 306.108 293.330 306.076 306.099 306.146 293.397 293.400 306.076 306.035 293.377 293.372 293.288 306.045 306.057 306.053 306.113 348.974 348.974 345.958 348.962 348.741 348.948 349.004 348.956 348.971 349.319 349.217 . . . . pinger.slac.stanford.edu EDU North_America www.ru.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.SAGRID.AC.N1 ?r 294.768 294.750 294.756 294.807 294.785 294.770 294.738 294.732 294.776 294.768 294.765 294.776 294.765 294.805 306.533 306.669 352.051 352.033 352.011 352.057 351.874 352.028 352.019 352.055 351.452 352.698 352.653 . . . . pinger.slac.stanford.edu EDU North_America sagrid.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.TENET.AC.N1 ?M 280.444 280.498 280.517 280.461 280.447 280.451 280.468 280.476 280.518 280.465 280.474 280.515 280.429 280.531 301.541 301.659 355.767 355.285 355.276 355.842 355.843 355.845 356.170 356.180 356.201 356.299 356.167 . . . . pinger.slac.stanford.edu EDU North_America brunsvigia.tenet.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZW.HELLENIC.AC.N1 ?B 309.551 309.511 309.506 309.483 306.234 306.159 308.309 306.323 309.553 309.566 309.584 324.719 309.626 309.514 304.837 305.008 366.140 366.246 369.388 369.400 396.258 369.499 369.405 366.430 369.217 369.612 369.613 . . . . pinger.slac.stanford.edu EDU North_America www.hellenic.ac.zw ZW Africa EDU.SLAC.STANFORD.PINGER ZW.PAYNET.CO.N1 ?r 324.077 324.135 324.157 324.265 321.244 320.903 323.835 324.198 323.939 324.464 324.302 330.877 324.711 324.584 323.073 328.229 330.032 330.101 329.970 330.101 328.696 328.553 329.868 329.913 329.615 329.708 328.118 . . . . pinger.slac.stanford.edu EDU North_America www.paynet.co.zw
For the 7 targets in South Africa the 25%/median/75% before 2016 and after are ~294/300/306ms and 327/345/356ms.
Looking in more detail at the ping responses each hour from SLAC to TENET South Africa we see a couple of sudden steps.
and in more detail
The first step occurred between 08:00am and 09:00am Jan 14th, the second step between 07:00am and 08:00am Jan 16th (both times are GMT). I can find no evidence of a failure on January 14th, however it sure looks like there was one. On further inspection, it appears that this step only appears for the TENET target site (see below for the minimum RTTs in msec measured each hour (0 1 ...) from SLAC to the target (Remote-Site) on Jan 14th, 2020), and thus is not related to the cable problems. This can be seen more clearly in the plot below. The spreadsheets are available here.
 |  |
N.b. two sites (www.bos.gov.ls and www.dut.ac.za) do not appear to have been affected on January 16th or in the daily plots. Maybe they were not using SAT3 or WACS, for example maybe they were using ACE. According to ieee "TENET restored its overseas connections by switching to an alternative 80-gigabit-per-second service on the SEACOM and EASSy cables on Africa's east coast. "The latency ... increased due to this failover, but there are currently no reported issues of this being detrimental to connectivity," a representative of TENET said on 30 January." The impact on the PingER measurements from SLAC was that the minimum RTT went from ~280ms to 355ms.
One can also look at the various statistics for both Average RTT, losses and minimum RTTs, see below (the spreadsheet is seen here). The Average and Losses are heavily influenced by the congestion. The minimum is mainly influenced by the route change or change in the distance of the cable route. Also note that the impact on the minimum RTT is not seen until Jan 17th since some of the PingER January 16 measurements were made before the change of cables used. It is also worth noting as reported in ieee "TENET restored its overseas connections by switching to an alternative 80-gigabit-per-second service on the SEACOM and EASSy cables on Africa's east coast. '"The latency ... increased due to this failover, but there are currently no reported issues of this being detrimental to connectivity," a representative of TENET said on 30 January.'
 |  |  |
Traceroute
The route measured 19:00 hrs 1/28/2020 PST from SLAC to South Africa goes East across the US via Chicago, Washington to London and thence S. Africa:
Tue Jan 28 18:59:32 2020: executing exec(traceroute -m 30 -q 1 -w 1 -A 196.24.110.67 140)=traceroute from 134.79.197.214(www.slac.stanford.edu) to 196.24.110.67(brunsvigia.tenet.ac.za) for 134.79.138.4
traceroute to 196.24.110.67 (196.24.110.67), 30 hops max, 140 byte packets
1 rtr-serv01-02-serv01-dmz-webserv.slac.stanford.edu (134.79.197.131) [AS3671] 0.788 ms
2 rtr-core2-p2p-serv01-01.slac.stanford.edu (134.79.254.65) [AS3671] 0.591 ms
3 rtr-fwcore1-trust-p2p-core1.slac.stanford.edu (134.79.254.134) [AS3671] 1.123 ms
4 rtr-core1-p2p-fwcore1-untrust.slac.stanford.edu (134.79.254.137) [AS3671] 1.538 ms
5 rtr-border1-7k-p2p-core1.slac.stanford.edu (134.79.252.173) [AS3671] 1.482 ms
6 rtr-border2-7k-p2p-border1-7k.slac.stanford.edu (192.68.191.214) [AS3671] 1.614 ms
7 sunn-cr5-ip-p2p-border2-7k.slac.stanford.edu (192.68.191.233) [AS3671] 1.858 ms
8 134.55.40.145 (134.55.40.145) [AS293] 4.548 ms
9 134.55.40.150 (134.55.40.150) [AS293] 46.417 ms
10 washcr5-ip-a-chiccr5.es.net (134.55.36.46) [AS293] 63.692 ms
11 *
12 londcr5-ip-c-aofacr5.es.net (134.55.37.22) [AS293] 138.575 ms
13 *
14 lt-0-0-0-2-ldn1-ir1.net.tenet.ac.za (155.232.208.28) [AS2018] 331.160 ms
15 londcr5-sdn-a-newycr5.es.net (134.55.220.21) [AS293] 137.494 ms
16 xe-11-3-0-201-ldn1-ir1.net.tenet.ac.za (155.232.212.106) [AS2018] 137.352 ms
17 lt-0-3-0-1-mtz1-pe1.net.tenet.ac.za (155.232.152.20) [AS2018] 329.393 ms
18 et-1-0-0-0-ams1-ir1.net.tenet.ac.za (155.232.1.80) [AS2018] 328.407 ms
19 xe-1-3-0-0-isd1-pe2.net.tenet.ac.za (155.232.1.60) [AS2018] 339.301 ms
20 eth3-39-cpt3-p1.net.tenet.ac.za (155.232.1.37) [AS2018] 354.698 ms
21 lt-0-3-0-0-mtz1-pe1.net.tenet.ac.za (155.232.152.21) [AS2018] 330.381 ms
22 et-0-0-1-0-cpt2-p2.net.tenet.ac.za (155.232.64.64) [AS2018] 354.586 ms
23 eth3-39-cpt3-p1.net.tenet.ac.za (155.232.1.37) [AS2018] 353.473 ms
24 et-0-0-1-0-cpt3-p2.net.tenet.ac.za (155.232.64.61) [AS2018] 353.617 ms
25 196.24.111.221 (196.24.111.221) [AS2018] 357.517 ms
26 pinger-cpt-01.tenet.ac.za (196.24.110.67) [AS2018] 354.664 ms
traceroute -m 30 -q 1 -w 1 -A 196.24.110.67 140 took 5secs. Total script traceroute.pl time=5secs,AS3671 is ESnet and AS2018 is TENET...
ieee Submarine Cable Repairs Underway in South Africa. IEEESpectrum, 3 February 2020, see https://spectrum.ieee.org/tech-talk/telecom/internet/undersea-cable-repairs-south-africa