Japanese Earthquake March 11th, 2011

There are some interesting results from the Japanese earthquake of March 11th 2011. According to http://earthquake.usgs.gov/earthquakes/recenteqsww/Quakes/usc0001xgp.phpl  the big (8.9) one was Friday, March 11, 2011 at 05:46:23 UTC.

Overview:

Internet connectivity to the hosts PingER monitors in Japan was maintained. Round Trip Times (RTT) to some hosts increased significantly as seen from SLAC. However as seen from RIKEN in Japan they did not increase. It appears the increase in RTT depends on the route from the monitoring host to Japan. This suggests a possible cable disruption. For more on cables see here.

The Japanese Internet appears to have kept running remarkably well though one of the 6 end hosts monitored did go down for some time after the earthquake. This compares well with the impact of the Chilean earthquake on the hosts monitored in Chile in February 2010 in which both hosts monitored in Chile were not reachable following the earthquake. One was again reachable a few hours later, the other was not reachable until March 3rd 2010. The Japanese phone network did not fare so well as its Internet.

PingER hosts in Japan that were monitored

The hosts monitored are seen in the table below.

The map below shows the location of the hosts.

Immediate impacts of earthquake on Japanese hosts seen from SLAC

All of the 6 hosts that PingER monitors in Japan stayed up  at the time of the earthquake.Looking from SLAC there were big increases in the average RTTs and minimum RTTs for some Japanese sites (but not all), see below. The [spreadsheet|^japan-earthquake.xlsx] gives more details.

Further pinpointing of causes for increased RTTs

RIKEN seen from the world

Looking at RIKEN (a monitoring mode and so easy to select on and also one of the most affected as seen from SLAC) seen from the world looking at avg RTT and min RTT we saw:

• No effect seen from Africa, E. Asia, Europe, L. America, M. East 
• Big effect from N. America (Canada 163ms=>264ms, US 120ms=>280ms) 
• India CDAC Mumbia no effect, Pune 380ms=> 460ms, VSNL Mumbia 360ms=>400ms 
• Sri Lanka no effect 
• Pakistan (we have lots of monitors so should be interesting).  
  • NIIT sees no effect (nb not on PERN) 
  • The PERN (Pakistan Education and Research Network) nodes starting with 111. (apart from UAAR see later, this needs more investigation) see 420ms=>500ms 
  • The PERN nodes starting with 121. See no effect         

Conclusion It is not the site RIKEN that has gone bad, rather it is some of the routes

RIKEN Looking at Japan

Japanese hosts seen from JP.RIKEN.N3 (RIKEN) see no impact on RTT

It looks the problem is in the route to Japan not within Japan itself. I wonder if the undersea earthquake has disrupted some  cables? This appears to be in line with the information from http://www.renesys.com/blog/2011/03/japan-quake.shtml and http://www.networkworld.com/news/2011/031411-quake-damage-to-japan-cables.html?source=NWWNLE_nlt_network_architecture_2011-03-15.

Undersea Cables

A map of the cables from Telegeography is seen below:

The following quote from Telegeography on 4/11/2011 confirms our early suspicions that the initial impact was probably due to rerouting of traffic as some undersea route were affected.

The massive earthquake off the coast of Japan damaged several undersea cables, some of which are still awaiting repair. Despite these outages, communications between Japan and the rest of the world were largely unaffected, due to the large array of undersea cables linked to Japan. ‘The earthquake temporarily knocked out approximately 30% of Japan’s international capacity,’ according to TeleGeography Research Director Alan Mauldin. ‘The deployment of multiple new trans-Pacific cables and intra-Asian cables over the past three years proved instrumental in preventing this disaster from also disrupting communications.’

Longer term impacts

However, we were not monitoring a Japanese host near the epicenter.

On 3/12/2011 we therefore added Tohoku University (www.tohoku.ac.jp), which we were not monitoring previously. It is on the outskirts of Sendai and close to the earthquake. At first  (on 3/12/2011 12:46pm PST) it was not responding. It did respond on 3/14/2011 at 12:45pm PDT.

53cottrell@pinger:~>ping www.tohoku.ac.jp
ping: unknown host www.tohoku.ac.jpExit 2

Also www.jp.kek (for more on the earthquake's impact on KEK see here) although responding on 3/10/2011 it was no longer responding at noon 3/11/2011.

60cottrell@pinger:~>ping www.jp.kek
ping: unknown host www.jp.kek
Exit 2
64cottrell@pinger:~>ping 130.87.104.107
PING 130.87.104.107 (130.87.104.107) 56(84) bytes of data.
From 134.79.252.133 icmp_seq=31 Destination Host Unreachable
From 134.79.252.133 icmp_seq=58 Destination Host Unreachable

It was responding again at 1:10pm 3/14/2011 PDT.

54cottrell@pinger:~>ping www.kek.jp
PING wlb00dn1.kek.jp (130.87.104.107) 56(84) bytes of data.
64 bytes from wlb00dn1.kek.jp (130.87.104.107): icmp_seq=0 ttl=237 time=276 ms
64 bytes from wlb00dn1.kek.jp (130.87.104.107): icmp_seq=1 ttl=237 time=276 ms
64 bytes from wlb00dn1.kek.jp (130.87.104.107): icmp_seq=2 ttl=237 time=276 ms

--- wlb00dn1.kek.jp ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 276.357/276.388/276.437/0.608 ms, pipe 2

Looking at the PingER data it started responding again at Mon, 14 Mar 2011 07:38:32 GMT.

Looking at the plots below we see all hosts except KEK during and after the earthquake maintained reachability. The increase in RTT is also seen to be a spike of several hours. KEK it is seen to lose reachability around 6:00am March 11th and was again reachable on March14th at about 7:00am:

Looking again on April 5th 2011, the RTTs from SLAC to the University of Osaka, RIKEN, University of Tokyo and KEK all increased with a step function from around 135ms to around 200ms around midday on the 24th March. Below is shown the RTTs for RIKEN as an example.

Routes

Comparing the routes from SLAC to RIKEN (ping.riken.jp) and from SLAC to NET.APAN.N2 (ns.jp.apan.net) on 3/11/2011, we see RIKEN has more hops and goes via the Avenue of the Americas in NY, while APAN goes directly via Sunnyvale near SLAC and then via Pacific Wave directly to Japan. The traceroutes from SLAC to the  University of OSAKA and the University of TOKYO  are similar to traceroute from SLAC to  RIKEN.

However on 3/22/2011 the route to RIKEN went westwards across the Pacific and the RTT was very stable with Min/Avg/Max/Stdev 131.985/132.173/132.607/0.443 ms for 148 pings. The routes to the University of OSAKA and the University of TOKYO were similar.

Theroute from SLAC to RIKEN on April  6th (i.e. after the step change from 135ms to 200ms on March 24th) went from SLAC eastwards through Sunnyvale, Denver ,Kansas, Clevland, Boston to the Avenue of the Americas in New York and then to Japan. This trip eastward across the US added an extra 60-70ms to the RTT.