Report
See https://spectrum.ieee.org/tech-talk/telecom/internet/undersea-cable-repairs-south-africa for a very full discussion, also early reports from https://mybroadband.co.za/news/internet/336416-how-south-africas-internet-survived-cable-breaks.html and the aftermath at https://mybroadband.co.za/news/internet/336426-slow-internet-speeds-in-south-africa-at-least-12-more-days.html.
Both the SAT-3 and WACS undersea cable were damaged Jan 16, 2020. The cable systems are deployed in the Atlantic Ocean and connect various African countries to Europe – including South Afric. According to reports, the issue with the SAT-3 is believed to have been located near Libreville, Gabon. WACs meanwhile is believed to have suffered two faults one of the coast of congo Congo and another near the UK. This affected more than a dozen countries besides South Africa. This left many South Africans with slow Internet connections to international websites. a. That leaves Africa Coast to Coast (ACE) and Equiano providing access to Europe and beyond and SAFE providing access via India and Malaysia and the South Africa Express (SAEx2) which will provide access via India and Malaysia in 2021.
...
SAT3 (RFS 2002 April) | WACS (RFS 2012 May) | ACE (RFS 2012 December) | Equiano (RFS 2021) |
---|---|---|---|
Cables going East to Europe
EASSY RFS 2010 July | SEACOM/Tata TGN-Eurasia RFS 2009 July |
---|---|
Cables going East to India and Malaysia are shown below.
...
RFS = Ready For Service so the current backups to Europe and probably the US are ACE going West and possibly SAFE SAFECOM both going East.
Result as observed by PingER
...
Tick | min | 25th% | avg | median | 75th% | 90th% | 95th% | max | iqr | std dev | # pairs |
---|---|---|---|---|---|---|---|---|---|---|---|
20Jan01 | 280.444 | 303.117 | 312.614 | 316.141 | 321.182 | 325.130 | 325.531 | 329.477 | 18.065 | 12.637 | 20 |
20Jan02 | 280.498 | 297.056 | 309.626 | 314.285 | 317.884 | 324.135 | 324.826 | 325.112 | 20.828 | 12.793 | 20 |
20Jan03 | 280.517 | 303.163 | 312.347 | 316.178 | 321.177 | 324.280 | 324.960 | 329.499 | 18.014 | 12.433 | 20 |
20Jan04 | 280.461 | 303.154 | 312.365 | 316.158 | 321.146 | 324.649 | 324.978 | 329.489 | 17.992 | 12.457 | 20 |
20Jan05 | 280.447 | 303.144 | 312.051 | 316.130 | 321.151 | 324.690 | 324.943 | 329.539 | 18.007 | 12.395 | 20 |
20Jan06 | 280.451 | 296.392 | 310.530 | 315.753 | 320.903 | 324.763 | 324.981 | 329.496 | 24.511 | 13.628 | 20 |
20Jan07 | 280.468 | 296.436 | 310.834 | 315.841 | 321.240 | 324.988 | 325.506 | 329.450 | 24.804 | 13.775 | 20 |
20Jan08 | 280.476 | 303.144 | 311.708 | 314.254 | 321.011 | 325.028 | 325.524 | 329.554 | 17.867 | 12.457 | 20 |
20Jan09 | 280.518 | 303.151 | 311.782 | 314.255 | 319.939 | 323.939 | 325.036 | 328.314 | 16.788 | 11.658 | 20 |
20Jan10 | 280.465 | 300.798 | 309.671 | 311.542 | 315.925 | 324.952 | 325.496 | 328.318 | 15.127 | 12.842 | 20 |
20Jan11 | 280.474 | 301.814 | 309.915 | 311.603 | 315.171 | 325.012 | 325.430 | 329.738 | 13.357 | 12.748 | 20 |
20Jan12 | 280.515 | 296.730 | 311.775 | 315.344 | 322.416 | 324.938 | 329.779 | 330.877 | 25.686 | 14.228 | 20 |
20Jan13 | 280.429 | 305.275 | 311.683 | 313.730 | 316.397 | 325.007 | 325.491 | 329.706 | 11.122 | 12.046 | 20 |
20Jan14 | 280.531 | 303.975 | 309.400 | 311.155 | 315.672 | 318.695 | 324.584 | 329.770 | 11.697 | 11.031 | 20 |
20Jan15 | 296.380 | 305.315 | 313.079 | 312.597 | 318.192 | 324.055 | 325.876 | 336.248 | 12.877 | 9.760 | 20 |
20Jan16 | 296.726 | 305.008 | 313.498 | 312.889 | 319.100 | 325.828 | 328.229 | 336.273 | 14.092 | 10.343 | 20 |
20Jan17 | 301.328 | 330.032 | 348.988 | 348.759 | 355.767 | 387.082 | 392.328 | 408.823 | 25.735 | 25.731 | 20 |
20Jan18 | 301.306 | 330.101 | 345.969 | 348.990 | 355.285 | 366.246 | 385.621 | 389.538 | 25.184 | 21.546 | 20 |
20Jan19 | 301.268 | 329.970 | 346.692 | 346.227 | 355.276 | 385.389 | 389.286 | 396.487 | 25.306 | 24.191 | 20 |
20Jan20 | 301.307 | 327.376 | 342.864 | 340.419 | 352.677 | 369.400 | 386.111 | 389.233 | 25.301 | 22.337 | 20 |
20Jan21 | 301.146 | 327.248 | 346.865 | 342.833 | 352.699 | 396.258 | 412.843 | 415.990 | 25.451 | 30.435 | 20 |
20Jan22 | 301.350 | 323.368 | 341.581 | 339.179 | 352.644 | 369.499 | 386.202 | 389.188 | 29.276 | 23.344 | 20 |
20Jan23 | 305.829 | 324.277 | 342.455 | 339.436 | 352.019 | 369.405 | 386.094 | 389.259 | 27.742 | 22.234 | 20 |
20Jan24 | 305.722 | 323.285 | 341.580 | 338.587 | 352.055 | 366.430 | 386.448 | 389.562 | 28.770 | 22.446 | 20 |
20Jan25 | 305.680 | 324.671 | 342.747 | 340.071 | 351.452 | 369.217 | 385.966 | 389.601 | 26.781 | 22.061 | 20 |
20Jan26 | 305.761 | 323.724 | 343.401 | 349.319 | 356.299 | 386.491 | 389.553 | 389.553 | 32.575 | 22.892 | 19 |
20Jan27 | 305.794 | 321.125 | 341.578 | 338.668 | 352.731 | 369.613 | 386.166 | 389.261 | 31.606 | 23.321 | 20 |
The targets are
For the 7 targets in South Africa the 25%/median/75% before 2016 and after are ~294/300/306ms and 327/345/356ms.
Looking in more detail at the ping responses each hour we see a sudden step
and in more detail
The first step occurred between 08:00am and 09:00am Jan 14th, the second step between 07:00am and 08:00am Jan 16th (both times are GMT). I can find no evidence of a failure on January 14th, however it sure looks like there was one. On further inspection, it appears that this step only appears for the TENET target site (see below for the minimum RTTs measured each hour (0 1 ...) from SLAC to the target (Remote-Site) on Jan 14th, 2020), and thus is not related to the cable problems.
Monitoring-Site Remote-Site ? 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Monitoring-Node Mon-TLD Mon-Region Remote-Node Rem-TLD Rem-Region EDU.SLAC.STANFORD.PINGER BW.AC.BAC.WWW ?r 323.430 323.483 323.440 323.527 323.338 323.437 318.915 318.695 320.438 343.658 345.564 331.147 343.837 326.039 331.680 321.791 321.934 321.922 322.115 321.810 321.662 321.599 321.957 321.701 pinger.slac.stanford.edu EDU North_America www.bac.ac.bw BW Africa EDU.SLAC.STANFORD.PINGER BW.BANKOFBOTSWANA.N1 ?r 315.727 315.882 315.738 315.672 315.758 315.877 316.134 315.879 316.063 333.509 345.746 317.119 330.161 316.179 324.060 316.010 315.978 315.854 315.866 315.804 315.837 315.713 315.724 315.842 pinger.slac.stanford.edu EDU North_America www.bankofbotswana.bw BW Africa EDU.SLAC.STANFORD.PINGER BW.BTC.NS1 ?B 315.399 315.622 315.758 315.422 315.327 315.343 315.536 315.452 315.409 315.444 316.058 315.952 315.530 315.280 315.599 315.400 315.880 315.394 315.349 315.332 315.397 315.312 315.698 315.403 pinger.slac.stanford.edu EDU North_America ns1.btc.bw BW Africa EDU.SLAC.STANFORD.PINGER LS.BOS.GOV.N1 ?r 312.842 312.871 312.830 312.884 312.876 312.887 312.891 312.929 312.953 312.976 312.990 312.809 312.942 312.840 312.924 312.910 312.873 312.885 312.893 312.796 312.820 312.840 312.877 312.846 pinger.slac.stanford.edu EDU North_America www.bos.gov.ls LS Africa EDU.SLAC.STANFORD.PINGER LS.LCE.AC.WWW ?r 313.298 313.367 313.335 313.400 313.375 313.374 313.433 313.519 313.521 313.796 313.998 313.553 313.403 313.544 313.423 313.402 313.361 313.358 313.372 313.268 313.329 313.371 313.393 313.427 pinger.slac.stanford.edu EDU North_America www.lce.ac.ls LS Africa EDU.SLAC.STANFORD.PINGER LS.LEO.CO.N1 ?B 329.891 329.871 329.937 329.809 329.847 329.973 329.877 329.885 329.944 329.805 329.898 329.871 329.834 329.770 329.867 329.901 329.900 329.913 329.776 329.809 329.860 329.886 329.876 329.913 pinger.slac.stanford.edu EDU North_America www.leo.co.ls LS Africa EDU.SLAC.STANFORD.PINGER LS.LNDC.ORG.WWW ?r . . . . . . . . . . . . . . . . . . . . . . . . pinger.slac.stanford.edu EDU North_America www.lndc.org.ls LS Africa EDU.SLAC.STANFORD.PINGER NA.AGRINAMIBIA.COM.N1 ?r 316.059 316.112 315.991 316.023 316.063 315.959 324.203 324.279 324.300 316.118 316.152 315.962 316.082 315.996 316.180 316.112 316.085 316.160 316.060 316.051 316.061 316.100 316.017 316.082 pinger.slac.stanford.edu EDU North_America www.agrinamibia.com.na NA Africa EDU.SLAC.STANFORD.PINGER NA.BRAHMAN.IWAY ?r . . . . . . . . . . . . . . . . . . . . . . . . pinger.slac.stanford.edu EDU North_America www.brahman.iway.na NA Africa EDU.SLAC.STANFORD.PINGER NA.MME.GOV.N1 ?B 316.476 316.546 316.501 316.480 316.587 316.452 324.635 324.729 316.452 316.492 316.617 316.492 316.496 316.422 316.489 316.580 316.644 316.568 316.499 316.504 316.450 316.471 316.522 316.419 pinger.slac.stanford.edu EDU North_America www.mme.gov.na NA Africa EDU.SLAC.STANFORD.PINGER SZ.ACAT.ORG.N1 ?B 325.502 304.647 304.720 304.756 304.689 304.832 326.869 344.121 351.111 350.593 350.049 340.553 328.354 337.106 330.511 325.965 325.950 326.011 329.815 329.824 330.056 329.972 330.004 329.891 pinger.slac.stanford.edu EDU North_America www.acat.org.sz SZ Africa EDU.SLAC.STANFORD.PINGER SZ.REALBROADBAND.CO ?r 325.069 303.975 304.023 304.026 304.019 304.121 304.406 355.842 346.205 348.319 346.952 328.566 324.738 326.837 325.829 324.329 324.219 324.102 328.097 328.116 328.134 328.286 328.066 328.144 pinger.slac.stanford.edu EDU North_America www.realbroadband.co.sz SZ Africa EDU.SLAC.STANFORD.PINGER SZ.UNISWA.ORG.WWW ?r 298.450 298.153 298.117 297.186 297.131 296.841 298.139 296.903 296.395 296.558 296.601 296.959 296.640 296.524 296.668 296.753 296.702 296.896 297.137 296.894 296.587 296.615 297.133 296.753 pinger.slac.stanford.edu EDU North_America www.uniswafoundation.org.sz SZ Africa EDU.SLAC.STANFORD.PINGER ZA.AC.NMMU.N1 ?r 303.254 303.259 303.252 303.242 303.222 303.224 303.260 303.255 303.253 303.287 303.260 303.199 303.190 303.171 303.267 303.228 303.282 303.280 303.265 303.179 303.139 303.188 303.173 303.249 pinger.slac.stanford.edu EDU North_America www.nmmu.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.AC.NWU.N1 ?r . . . . . . . . . . . . . . . . . . . . . . . . pinger.slac.stanford.edu EDU North_America www.nwu.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.BONTANY.UWC.AC.N1 ?r 315.577 315.549 315.543 315.623 315.598 315.537 315.603 315.560 315.556 315.569 336.325 336.267 336.290 336.224 336.356 336.402 336.354 336.385 336.349 336.416 336.351 336.299 336.341 336.421 pinger.slac.stanford.edu EDU North_America 196.21.45.139 ZA Africa EDU.SLAC.STANFORD.PINGER ZA.DUT.AC.N1 ?r 306.191 306.324 306.287 306.328 306.377 306.398 306.441 306.591 306.860 306.605 306.572 306.618 306.578 306.474 306.455 306.428 306.346 306.378 306.491 306.318 306.284 306.395 306.262 306.248 pinger.slac.stanford.edu EDU North_America www.dut.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.MUSEUMSNC.CO ?r 305.339 305.407 305.384 305.513 305.397 305.504 305.418 305.434 305.378 305.498 304.907 304.774 304.900 305.073 305.491 305.404 305.437 305.478 305.808 305.546 311.584 305.420 305.553 305.436 pinger.slac.stanford.edu EDU North_America www.museumsnc.co.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.RU.AC.N1 ?r 306.200 306.242 306.202 306.153 306.186 306.186 306.246 306.178 306.193 306.206 306.217 306.114 306.129 306.057 306.227 306.228 306.166 306.149 306.203 306.129 306.131 306.161 306.118 306.243 pinger.slac.stanford.edu EDU North_America www.ru.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.SAGRID.AC.N1 ?r 294.901 294.915 294.805 294.903 294.892 294.889 294.856 294.905 294.873 294.897 306.606 306.592 306.546 306.530 306.675 306.643 306.603 306.616 306.597 306.578 306.555 306.573 306.613 306.609 pinger.slac.stanford.edu EDU North_America sagrid.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.TECHCONCEPTS.CO.N1 ?r . . . . . . . . . . . . . . . . . . . . . . . . pinger.slac.stanford.edu EDU North_America speedtest.techconcepts.co.za ZA Africa EDU.SLAC.STANFORD.PINGER ZA.TENET.AC.N1 ?M 280.581 280.644 280.561 280.593 280.590 280.626 280.597 280.531 280.614 301.662 301.652 301.571 301.545 301.668 301.701 301.725 301.650 301.724 301.577 301.644 301.540 301.622 301.704 301.707 pinger.slac.stanford.edu EDU North_America brunsvigia.tenet.ac.za ZA Africa EDU.SLAC.STANFORD.PINGER ZW.ACROSSAFRICATOURS.CO.N2 ?r . . . . . . . . . . . . . . . . . . . . . . . . pinger.slac.stanford.edu EDU North_America www.acrossafricatours.co.zw ZW Africa EDU.SLAC.STANFORD.PINGER ZW.HELLENIC.AC.N1 ?B 309.769 309.585 309.702 309.621 309.666 309.611 309.852 309.847 309.936 309.951 309.736 309.811 310.127 309.780 309.897 309.704 309.912 309.779 309.759 309.742 309.720 309.514 309.610 309.729 pinger.slac.stanford.edu EDU North_America www.hellenic.ac.zw ZW Africa EDU.SLAC.STANFORD.PINGER ZW.PAYNET.CO.N1 ?r 324.584 324.776 324.691 325.179 324.917 324.998 325.087 325.175 324.887 325.771 325.117 325.435 325.500 324.762 325.113 325.411 325.560 325.380 325.045 324.755 324.942 325.346 324.855 325.750 pinger.slac.stanford.edu EDU North_America www.paynet.co.zw ZW Africa Monitoring-Site Remote-Site ? 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Monitoring-Node Mon-TLD Mon-Region Remote-Node Rem-TLD Rem-Region
...
For the 7 targets in South Africa the 25%/median/75% before 2016 and after are ~294/300/306ms and 327/345/356ms.
Looking in more detail at the ping responses each hour from SLAC to TENET South Africa we see a couple of sudden steps.
and in more detail
The first step occurred between 08:00am and 09:00am Jan 14th, the second step between 07:00am and 08:00am Jan 16th (both times are GMT). I can find no evidence of a failure on January 14th, however it sure looks like there was one. On further inspection, it appears that this step only appears for the TENET target site (see below for the minimum RTTs in msec measured each hour (0 1 ...) from SLAC to the target (Remote-Site) on Jan 14th, 2020), and thus is not related to the cable problems. This can be seen more clearly in the plot below. The spreadsheets are available here.
N.b. two sites (www.bos.gov.ls and www.dut.ac.za) do not appear to have been affected on January 16th or in the daily plots. Maybe they were not using SAT3 or WACS, for example maybe they were using ACE. According to ieee "TENET restored its overseas connections by switching to an alternative 80-gigabit-per-second service on the SEACOM and EASSy cables on Africa's east coast. "The latency ... increased due to this failover, but there are currently no reported issues of this being detrimental to connectivity," a representative of TENET said on 30 January." The impact on the PingER measurements from SLAC was that the minimum RTT went from ~280ms to 355ms.
One can also look at the various statistics for both Average RTT, losses and minimum RTTs, see below (the spreadsheet is seen here). The Average and Losses are heavily influenced by the congestion. The minimum is mainly influenced by the route change or change in the distance of the cable route. Also note that the impact on the minimum RTT is not seen until Jan 17th since some of the PingER January 16 measurements were made before the change of cables used. It is also worth noting as reported in ieee "TENET restored its overseas connections by switching to an alternative 80-gigabit-per-second service on the SEACOM and EASSy cables on Africa's east coast. '"The latency ... increased due to this failover, but there are currently no reported issues of this being detrimental to connectivity," a representative of TENET said on 30 January.'
Traceroute
The route measured 19:00 hrs 1/28/2020 PST from SLAC to South Africa goes East across the US via Chicago, Washinton Washington to London and thence S. Africa:
...
AS3671 is ESnet and AS2018 is TENET
...
ieee Submarine Cable Repairs Underway in South Africa. IEEESpectrum, 3 February 2020, see https://spectrum.ieee.org/tech-talk/telecom/internet/undersea-cable-repairs-south-africa