Page History

Introduction

A massive route leak initiated by Telekom Malaysia (AS4788), starting at 8:43 June 12, 2015 UTC, caused significant network problems for the global routing system. Primarily affected was Level3 (AS3549 – formerly known as Global Crossing) and their customers. See http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/ for details.

Correlation for Level3 AS

We looked at the daily traceroutes measured from SLAC to ~ 800 PingER monitored hosts to see which one were using AS3539:

Code Block

[cottrell@pinger ~]$ grep 3549 /afs/slac/package/pinger/chktrace/2015_06_12/*

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_pinger.uum.edu.my_2015_06_12:21  210.19.33.130 (210.19.33.130) [AS3549]  268.581 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_speedtest.bitcom.com.br_2015_06_12:11  208.178.58.177 (208.178.58.177) [AS3549]  4.213 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.ait.ac.th_2015_06_12:14  61.19.14.46 (61.19.14.46) [AS4651/AS3549]  216.855 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.ait.asia_2015_06_12:14  61.19.14.38 (61.19.14.38) [AS4651/AS3549]  225.769 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.ait.asia_2015_06_12:15  61.19.14.38 (61.19.14.38) [AS4651/AS3549]  225.796 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.mju.ac.th_2015_06_12:13  61.19.9.109 (61.19.9.109) [AS4651/AS3549]  205.348 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.mjurmutsv.ac.th_2015_06_12:1411  61.19.79.110181 (61.19.7.110) [AS4651/AS3549]  211.300 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.rmutsvusim.acedu.thmy_2015_06_12:1120  61210.19.9163.18126 (61210.19.9163.18126) [AS4651/AS3549]  210202.115149 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.rmutsv.ac.th_2015_06_12:12  61.19.9.181 (61.19.9.181) [AS4651/AS3549]  210.167 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.usim.edu.my_2015_06_12:20  210.19.163.26 (210.19.163.26) [AS3549]  202.149 ms

/afs/slac/package/pinger/chktrace/2015_06_12/

Looking at those in S. E. Asia:

pinger.uum.edu.my has not been accessible for several months.
there is no obvious correlation (increase in losses or RTT) in degraded ping performance and the time span of the injected route problems for www.ait.ac.th, www.ait.asia, www.rmutsv.ac.my, www.usim.ac.my
www.mju.ac.th has not been accessible for several days

Correlations for Malaysian hosts seen from SLAC

For the 26 Malaysian hosts monitored from SLAC on 6/12/2015 looking at the 95 percentile frequency losses as a function of hour of day (see http://www-wanmon.slac.stanford.edu

...

/cgi-wrap/pingtable.pl?file=packet_loss&by=by-node&size=100&tick=hourly&year=2015&month=06&day=12&from=EDU.SLAC.STANFORD.N3&to=Malaysia&ex=none&only=all&dataset=hep&percentage=any) then apart from the hours of 9:00 and 10:00am UTC the losses are zero. The two majn contributors are: www.mahsa.edu.my and www.mmu.esu.my. Below are shown the RTTs and losses for these hosts on 6/12/2015.

SLAC to www.mahsa.edu.my

Looking at those in S. E. Asia:

pinger.uum.edu.my has not been accessible for several months.
there is no obvious correlation in degraded ping performance for

It is seen in the graph below that we observed losses of > 80 80% at 9:09, 9:39 and 10:09 and zero loss at 8:38am and again at 10:38. Also the Round Trip times increased from ~ 200 msec to over 450 msec. The route from SLAC to www.mahsa.edu.my passes through AS4788.

Image RemovedImage Added