There are some interesting results from the Japanese earthquake of March 11th 2011. Accordingto According to http://wwwearthquake.renesysusgs.comgov/blogearthquakes/2011recenteqsww/03Quakes/japan-quake.shtml the usc0001xgp.phpl the big (8.9) one was Friday, March 11, 2011 at 05:46:23 UTC.
h3
None of the 6 hosts that PingER monitors in Japan went down for an extended period of time (although right now I cannot ping www.kek.jp).
60cottrell@pinger:~>ping www.jp.kek
ping: unknown host www.jp.kek
Exit 2
64cottrell@pinger:~>ping 130.87.104.107
PING 130.87.104.107 (130.87.104.107) 56(84) bytes of data.
From 134.79.252.133 icmp_seq=31 Destination Host Unreachable
From 134.79.252.133 icmp_seq=58 Destination Host Unreachable
From 134.79.252.133 icmp_seq=85 Destination Host Unreachable
...
Overview:
Internet connectivity to the hosts PingER monitors in Japan was maintained. Round Trip Times (RTT) to some hosts increased significantly as seen from SLAC. However as seen from RIKEN in Japan they did not increase. It appears the increase in RTT depends on the route from the monitoring host to Japan. This suggests a possible cable disruption. For more on cables see here.
The Japanese Internet appears to have kept running remarkably well though one of the 6 end hosts monitored did go down for some time after the earthquake. This compares well with the impact of the Chilean earthquake on the hosts monitored in Chile in February 2010 in which both hosts monitored in Chile were not reachable following the earthquake. One was again reachable a few hours later, the other was not reachable until March 3rd 2010. The Japanese phone network did not fare so well as its Internet.
PingER hosts in Japan that were monitored
The hosts monitored are seen in the table below.
IP name | Alias | City | Institution |
|
|---|---|---|---|---|
glbb.jp | JP.GLBB | Okinawa | Speedtest |
|
www.kek.jp | JP.KEK | Tsukuba |
| |
ns.osaka-u.ac.jp | JP.U-OSAKA | Osaka |
| |
ping.riken.jp | JP.RIKEN | Wako-Shi |
| |
www.u-tokyo.ac.jp | JP.U-Tokyo | Tokyo |
| |
ns.jp.apan.net | NET.APAN | Tokyo |
|
The map below shows the location of the hosts.
Immediate impacts of earthquake on Japanese hosts seen from SLAC
All of the 6 hosts that PingER monitors in Japan stayed up at the time of the earthquake.Looking from SLAC there were big increases in the average RTTs and minimum RTTs
-- 130.87.104.107 ping statistics --
121 packets transmitted, 0 received, +4 errors, 100% packet loss, time 119996ms
, pipe 2
Exit 1
1. Looking from SLAC there are big increases in the avg RTT and min RTT for some Japanese sites (but not all)
, see below. The [spreadsheet|^japan-earthquake.xlsx] gives more details.
Further pinpointing of causes for increased RTTs
RIKEN seen from the world
According to http://earthquake.usgs.gov/earthquakes/eqinthenews/2011/usc0001xgp/ the big (8.9) one was Friday, March 11, 2011 at 05:46:23 UTC
2. Looking at RIKEN (a monitoring mode and so easy to select on and also one of the most affected as seen from SLAC) seen from the world looking at avg RTT and min RTTwe seeRTT we saw: a.
- No effect seen from Africa, E. Asia, Europe, L. America, M.
...
- East
...
- Big effect from N. America (Canada 163ms=>264ms, US 120ms=>280ms)
...
- India CDAC Mumbia no effect, Pune 380ms=> 460ms, VSNL Mumbia 360ms=
...
- >400ms
...
- Sri Lanka no
...
- effect
...
- Pakistan (we have lots of monitors so should be interesting).
...
- NIIT sees no effect (nb not on PERN)
...
- The PERN (Pakistan Education and Research Network) nodes starting with 111. (apart from UAAR see later, this needs more investigation) see 420ms=
...
- >500ms
...
- The PERN nodes starting with 121. See no effect
effect
iv. I am suspicious of UAAR since every other Pakistan node has 400+ms to RIKEN whereas UAAR has 190ms. Someone should look at this.
Conclusion It is not the site RIKEN that has gone bad, rather it is some of the routes
RIKEN Looking at Japan
...
Japanese hosts seen from JP.RIKEN.N3 (RIKEN) see no impact on RTT
It looks the problem is in the route to Japan not within Japan itself. I wonder if the undersea earthquake has disrupted some cables?
Comparing the routes (see below) from SLAC of RIKEN (ping.riken.jp) and NET.APAN.N2 (ns.jp.apan.net) we see RIKEN has more hops and goes Eastwards via the Avenue of the Americas in NY, while APAN goes directly via Sunnyvale near SLAC and then via Pacific Wave directly to Japan. OSAKA and U-TOKYO are similar to RIKEN. See below
It would be interesting to see the difference in routes from the two (three) types of Pakistani hosts. Are they different does that explain anything? Did the routes change around the time of the earthquake? Unfortunately we do not have such data. It may be useful to monitor all beacons from say SLAC once daily just like we do Pakistani hosts from Pakistan. Alternatively find someone else who has traceroute histories.
Does someone want to put together a case study? I have to do my taxes!
Traceroutes
===========
53cottrell@pinger:~>traceroute ping.riken.jp 200
traceroute to snoopy.riken.go.jp (134.160.4.17), 30 hops max, 200 byte packets
1 rtr-servcore1-serv01-iepm (134.79.104.66) 0.326 ms 0.260 ms 0.235 ms
2 rtr-core2-p2p-servcore1 (134.79.252.162) 0.355 ms 0.272 ms 0.261 ms
3 rtr-border2-p2p-core2 (134.79.252.145) 0.536 ms 0.359 ms 0.330 ms
4 slac-mr2-p2p-rtr-border2 (192.68.191.249) 0.250 ms 0.229 ms 0.237 ms
5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 0.629 ms 0.649 ms 0.632 ms
6 sunncr1-sunnsdn2.es.net (134.55.209.98) 0.778 ms 0.739 ms 0.712 ms
7 denvcr2-sunncr1.es.net (134.55.220.49) 28.295 ms 27.861 ms 27.889 ms
MPLS Label=321392 CoS=6 TTL=1 S=0
8 kanscr1-ip-denvcr2.es.net (134.55.209.46) 41.048 ms 40.982 ms 41.033 ms
MPLS Label=357088 CoS=6 TTL=1 S=0
9 chiccr1-ip-kanscr1.es.net (134.55.221.58) 51.719 ms 51.627 ms 51.637 ms
MPLS Label=302528 CoS=6 TTL=1 S=0
10 clevcr1-ip-chiccr1.es.net (134.55.217.53) 60.673 ms 60.652 ms 60.673 ms
MPLS Label=366704 CoS=6 TTL=1 S=0
11 bostcr1-ip-clevcr1.es.net (134.55.41.146) 73.838 ms 73.778 ms 73.818 ms
MPLS Label=363424 CoS=6 TTL=1 S=0
12 aofacr2-ip-bostcr1.es.net (134.55.41.122) 78.534 ms 78.530 ms 78.535 ms
MPLS Label=300768 CoS=6 TTL=1 S=0
13 aofasdn1-ip-aofacr2.es.net (134.55.38.50) 83.867 ms 78.413 ms 78.403 ms
14 198.124.216.154 (198.124.216.154) 79.404 ms 78.533 ms 78.537 ms
15 tokyo1-dc-RM-P-2-3-0-11.sinet.ad.jp (150.99.203.57) 270.323 ms 270.283 ms 270.250 ms
16 * * *
17 riken-LAN.gw.sinet.ad.jp (150.99.192.170) 277.472 ms 277.412 ms 277.447 ms
18 134.160.6.2 (134.160.6.2) 277.686 ms 277.544 ms 277.833 ms
...
55cottrell@pinger:~>traceroute ns.jp.apan.net
traceroute to ns.jp.apan.net (203.181.248.3), 30 hops max, 38 byte packets
1 rtr-servcore1-serv01-iepm (134.79.104.66) 0.388 ms 0.238 ms 0.245 ms
2 rtr-core1-p2p-servcore1 (134.79.252.166) 0.318 ms 0.262 ms 0.244 ms
3 rtr-border2-p2p-core1 (134.79.252.141) 0.376 ms 0.332 ms 0.313 ms
4 slac-mr2-p2p-rtr-border2 (192.68.191.249) 0.232 ms 0.212 ms 0.218 ms
5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 0.667 ms 0.640 ms 0.612 ms
6 sunncr1-sunnsdn2.es.net (134.55.209.98) 0.769 ms 0.728 ms 0.721 ms
7 transpac-1-is-jmb-780.lsanca.pacificwave.net (207.231.246.136) 8.888 ms 8.897 ms 8.877 ms
8 tokyo-losa-tp2.transpac2.net (192.203.116.146) 124.537 ms 124.355 ms 124.403 ms
...
This appears to be in line with the information from http://www.renesys.com/blog/2011/03/japan-quake.shtml and http://www.networkworld.com/news/2011/031411-quake-damage-to-japan-cables.html?source=NWWNLE_nlt_network_architecture_2011-03-15.
Undersea Cables
A map of the cables from Telegeography is seen below:
The following quote from Telegeography on 4/11/2011 confirms our early suspicions that the initial impact was probably due to rerouting of traffic as some undersea route were affected.
The massive earthquake off the coast of Japan damaged several undersea cables, some of which are still awaiting repair. Despite these outages, communications between Japan and the rest of the world were largely unaffected, due to the large array of undersea cables linked to Japan. ‘The earthquake temporarily knocked out approximately 30% of Japan’s international capacity,’ according to TeleGeography Research Director Alan Mauldin. ‘The deployment of multiple new trans-Pacific cables and intra-Asian cables over the past three years proved instrumental in preventing this disaster from also disrupting communications.’
Longer term impacts
However, we were not monitoring a Japanese host near the epicenter.
On 3/12/2011 we therefore added Tohoku University (www.tohoku.ac.jp), which we were not monitoring previously. It is on the outskirts of Sendai and close to the earthquake. At first (on 3/12/2011 12:46pm PST) it was not responding. It did respond on 3/14/2011 at 12:45pm PDT.
| Code Block |
|---|
53cottrell@pinger:~>ping www.tohoku.ac.jp
ping: unknown host www.tohoku.ac.jpExit 2
|
Also www.jp.kek (for more on the earthquake's impact on KEK see here) although responding on 3/10/2011 it was no longer responding at noon 3/11/2011.
| Code Block |
|---|
60cottrell@pinger:~>ping www.jp.kek
ping: unknown host www.jp.kek
Exit 2
64cottrell@pinger:~>ping 130.87.104.107
PING 130.87.104.107 (130.87.104.107) 56(84) bytes of data.
From 134.79.252.133 icmp_seq=31 Destination Host Unreachable
From 134.79.252.133 icmp_seq=58 Destination Host Unreachable
|
It was responding again at 1:10pm 3/14/2011 PDT.
| Code Block |
|---|
54cottrell@pinger:~>ping www.kek.jp
PING wlb00dn1.kek.jp (130.87.104.107) 56(84) bytes of data.
64 bytes from wlb00dn1.kek.jp (130.87.104.107): icmp_seq=0 ttl=237 time=276 ms
64 bytes from wlb00dn1.kek.jp (130.87.104.107): icmp_seq=1 ttl=237 time=276 ms
64 bytes from wlb00dn1.kek.jp (130.87.104.107): icmp_seq=2 ttl=237 time=276 ms
--- wlb00dn1.kek.jp ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 276.357/276.388/276.437/0.608 ms, pipe 2
|
Looking at the PingER data it started responding again at Mon, 14 Mar 2011 07:38:32 GMT.
Looking at the plots below we see all hosts except KEK during and after the earthquake maintained reachability. The increase in RTT is also seen to be a spike of several hours. KEK it is seen to lose reachability around 6:00am March 11th and was again reachable on March14th at about 7:00am:
glbb.jp | www.kek.jp | ns.osaka-u.ac.jp |
|---|---|---|
| | |
ping.riken.jp | www.u-tokyo.ac.jp | ns.jp.apan.net |
| | |
Looking again on April 5th 2011, the RTTs from SLAC to the University of Osaka, RIKEN, University of Tokyo and KEK all increased with a step function from around 135ms to around 200ms around midday on the 24th March. Below is shown the RTTs for RIKEN as an example.
Routes
Comparing the routes from SLAC to RIKEN (ping.riken.jp) and from SLAC to NET.APAN.N2 (ns.jp.apan.net) on 3/11/2011, we see RIKEN has more hops and goes via the Avenue of the Americas in NY, while APAN goes directly via Sunnyvale near SLAC and then via Pacific Wave directly to Japan. The traceroutes from SLAC to the University of OSAKA and the University of TOKYO are similar to traceroute from SLAC to RIKEN.
However on 3/22/2011 the route to RIKEN went westwards across the Pacific and the RTT was very stable with Min/Avg/Max/Stdev 131.985/132.173/132.607/0.443 ms for 148 pings. The routes to the University of OSAKA and the University of TOKYO were similar.
Theroute from SLAC to RIKEN on April 6th (i.e. after the step change from 135ms to 200ms on March 24th) went from SLAC eastwards through Sunnyvale, Denver ,Kansas, Clevland, Boston to the Avenue of the Americas in New York and then to Japan. This trip eastward across the US added an extra 60-70ms to the RTT.
traceroute to ns.osaka-u.ac.jp (133.1.192.4), 30 hops max, 38 byte packets
1 rtr-servcore1-serv01-iepm (134.79.104.66) 1.515 ms 0.330 ms 0.618 ms
2 rtr-core1-p2p-servcore1 (134.79.252.166) 0.857 ms 0.508 ms 0.539 ms
3 rtr-border1-p2p-core1 (134.79.252.133) 0.870 ms 0.328 ms 0.813 ms
4 slac-mr2-p2p-rtr-border1 (192.68.191.245) 0.284 ms 0.240 ms 0.201 ms
5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 0.627 ms 0.632 ms 0.611 ms
6 sunncr1-sunnsdn2.es.net (134.55.209.98) 0.790 ms 0.710 ms 1.163 ms
7 denvcr2-sunncr1.es.net (134.55.220.49) 56.055 ms 27.906 ms 27.910 ms
MPLS Label=321392 CoS=6 TTL=1 S=0
8 kanscr1-ip-denvcr2.es.net (134.55.209.46) 41.150 ms 101.222 ms 41.102 ms
MPLS Label=357088 CoS=6 TTL=1 S=0
9 chiccr1-ip-kanscr1.es.net (134.55.221.58) 51.712 ms 51.668 ms 51.685 ms
MPLS Label=302528 CoS=6 TTL=1 S=0
10 clevcr1-ip-chiccr1.es.net (134.55.217.53) 60.720 ms 60.746 ms 60.731 ms
MPLS Label=366704 CoS=6 TTL=1 S=0
11 bostcr1-ip-clevcr1.es.net (134.55.41.146) 73.842 ms 73.808 ms 73.777 ms
MPLS Label=363424 CoS=6 TTL=1 S=0
12 aofacr2-ip-bostcr1.es.net (134.55.41.122) 78.715 ms 79.035 ms 78.606 ms
MPLS Label=300768 CoS=6 TTL=1 S=0
13 aofasdn1-ip-aofacr2.es.net (134.55.38.50) 78.453 ms 78.478 ms 78.556 ms
14 198.124.216.154 (198.124.216.154) 78.589 ms 78.693 ms 78.636 ms
15 tokyo1-dc-RM-P-2-3-0-11.sinet.ad.jp (150.99.203.57) 270.322 ms 270.290 ms 270.370 ms
16 nagoya-dc-RM-AE-0-11.sinet.ad.jp (150.99.203.26) 278.341 ms 278.488 ms 278.573 ms
17 osaka-dc-RM-AE-0-11.sinet.ad.jp (150.99.203.30) 282.247 ms 300.091 ms 281.682 ms
18 osaka-u.gw.sinet.ad.jp (150.99.188.62) 282.419 ms 282.440 ms 282.385 ms
19 133.1.0.10 (133.1.0.10) 282.380 ms 282.477 ms 282.408 ms
20 * * *
21 * * *
...
57cottrell@pinger:~>traceroute www.u-tokyo.ac.jp
traceroute to www.u-tokyo.ac.jp (133.11.114.194), 30 hops max, 38 byte packets
1 rtr-servcore1-serv01-iepm (134.79.104.66) 1.042 ms 0.857 ms 1.103 ms
2 rtr-core1-p2p-servcore1 (134.79.252.166) 0.944 ms 0.896 ms 0.900 ms
3 rtr-border2-p2p-core1 (134.79.252.141) 0.941 ms 0.993 ms 0.974 ms
4 slac-mr2-p2p-rtr-border2 (192.68.191.249) 0.858 ms 0.225 ms 0.542 ms
5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 1.108 ms 0.639 ms 1.201 ms
6 sunncr1-sunnsdn2.es.net (134.55.209.98) 1.399 ms 1.282 ms 1.395 ms
7 denvcr2-sunncr1.es.net (134.55.220.49) 28.594 ms 28.434 ms 27.904 ms
MPLS Label=321392 CoS=6 TTL=1 S=0
8 kanscr1-ip-denvcr2.es.net (134.55.209.46) 41.506 ms 41.070 ms 41.107 ms
MPLS Label=357088 CoS=6 TTL=1 S=0
9 chiccr1-ip-kanscr1.es.net (134.55.221.58) 51.696 ms 51.688 ms 51.724 ms
MPLS Label=302528 CoS=6 TTL=1 S=0
10 clevcr1-ip-chiccr1.es.net (134.55.217.53) 60.689 ms 60.753 ms 66.794 ms
MPLS Label=366704 CoS=6 TTL=1 S=0
11 bostcr1-ip-clevcr1.es.net (134.55.41.146) 112.087 ms 73.904 ms 73.876 ms
MPLS Label=363424 CoS=6 TTL=1 S=0
12 aofacr2-ip-bostcr1.es.net (134.55.41.122) 78.626 ms 78.702 ms 78.628 ms
MPLS Label=300768 CoS=6 TTL=1 S=0
13 aofasdn1-ip-aofacr2.es.net (134.55.38.50) 78.539 ms 78.489 ms 78.507 ms
14 198.124.216.154 (198.124.216.154) 78.619 ms 82.098 ms 78.698 ms
15 tokyo1-dc-RM-P-2-3-0-11.sinet.ad.jp (150.99.203.57) 270.472 ms 270.375 ms 270.376 ms
16 UTnet-1.gw.sinet.ad.jp (150.99.190.102) 198.587 ms 197.493 ms 197.495 ms
17 ra36-vlan3.nc.u-tokyo.ac.jp (133.11.127.66) 198.251 ms 197.398 ms 197.402 ms
18 ra35.nc.u-tokyo.ac.jp (133.11.127.41) 197.550 ms 197.259 ms 197.242 ms
19 www.u-tokyo.ac.jp (133.11.114.194) 197.492 ms !<10> 197.565 ms !<10> 197.472 ms !<10>