Massive route leak slows down Internet June 12 2015

Introduction

A massive route leak initiated by Telekom Malaysia (AS4788), starting at 8:43 June 12, 2015 UTC, caused significant network problems for the global routing system until about 10:40 June 12, 2015 UTC. Primarily affected was Level3 (AS3549 – formerly known as Global Crossing) and their customers. See http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/ for details. This study shows the effects as seen by PingER monitoring hosts at SLAC and at the University of Malaysia.

Correlation for Level3 AS

We looked at the daily traceroutes measured from SLAC to ~ 800 PingER monitored hosts to see which one were using AS3539:

[cottrell@pinger ~]$ grep 3549 /afs/slac/package/pinger/chktrace/2015_06_12/*

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_pinger.uum.edu.my_2015_06_12:21  210.19.33.130 (210.19.33.130) [AS3549]  268.581 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_speedtest.bitcom.com.br_2015_06_12:11  208.178.58.177 (208.178.58.177) [AS3549]  4.213 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.ait.ac.th_2015_06_12:14  61.19.14.46 (61.19.14.46) [AS4651/AS3549]  216.855 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.mju.ac.th_2015_06_12:13  61.19.9.109 (61.19.9.109) [AS4651/AS3549]  205.348 ms

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.rmutsv.ac.th_2015_06_12:11  61.19.9.181 (

/afs/slac/package/pinger/chktrace/2015_06_12/www-wanmon.slac.stanford.edu_to_www.usim.edu.my_2015_06_12:20  210.19.163.26 (210.19.163.26) [AS3549]  202.149 ms

Looking at those in S. E. Asia:

pinger.uum.edu.my has not been accessible for several months.
there is no obvious correlation (increase in losses or RTT) in degraded ping performance and the time span of the injected route problems for www.ait.ac.th, www.ait.asia, www.rmutsv.ac.my, www.usim.ac.my
www.mju.ac.th has not been accessible for several days

Correlations for Malaysian hosts seen from SLAC

or the 26 Malaysian hosts monitored from SLAC on 6/12/2015 looking at the 95 percentile frequency losses as a function of hour of day (see http://www-wanmon.slac.stanford.edu/cgi-wrap/pingtable.pl?file=packet_loss&by=by-node&size=100&tick=hourly&year=2015&month=06&day=12&from=EDU.SLAC.STANFORD.N3&to=Malaysia&ex=none&only=all&dataset=hep&percentage=any) then apart from the hours of 9:00 and 10:00am UTC the losses are zero. The two majn contributors are: www.mahsa.edu.my and www.mmu.esu.my. Below are shown the RTTs and losses for these hosts on 6/12/2015.

SLAC to www.mahsa.edu.my

It is seen in the graph below that we observed losses of > 80% at 9:09, 9:39 and 10:09 and zero loss at 8:38am and again at 10:38. Also the Round Trip times increased from ~ 200 msec to over 450 msec. The route from SLAC to www.mahsa.edu.my passes through AS4788.