There are some interesting results from the Japanese earthquake of March 11th 2011. Accordingto http://www.renesys.com/blog/2011/03/japan-quake.shtml the big (8.9) one was Friday, March 11, 2011 at 05:46:23 UTC.
53cottrell@pinger:~>ping www.tohoku.ac.jp
ping: unknown host www.tohoku.ac.jp
Exit 2
53cottrell@pinger:~>ping www.tohoku.ac.jp ping: unknown host www.tohoku.ac.jpExit 2
Also though www.
PingER results
None of the 6 hosts that PingER monitors in Japan went down for an extended period of time (although right now I cannot ping www.kek.jp). Also we do not monitor a Japanese host near the epicenter. Tohoku University (www.tohoku.ac.jp), which were not monitoring previously, on the outskirts of Sendai was not responding on 3/12/2011 12:46pm PST.
60cottrell@pinger:~>ping www.jp.kek
ping: unknown host www.jp.kek
Exit 2
64cottrell@pinger:~>ping 130.87.104.107
PING 130.87.104.107 (130.87.104.107) 56(84) bytes of data.
From 134.79.252.133 icmp_seq=31 Destination Host Unreachable
From 134.79.252.133 icmp_seq=58 Destination Host Unreachable
From 134.79.252.133 icmp_seq=85 Destination Host Unreachable
From 134.79.252.133 icmp_seq=114 Destination Host Unreachable
-- 130.87.104.107 ping statistics --
121 packets transmitted, 0 received, +4 errors, 100% packet loss, time 119996ms
, pipe 2
Exit 1
Japanese hosts seen From SLAC
# Looking from SLAC there are big increases in the avg RTT and min RTT for some Japanese sites (but not all)
RIKEN seen from the world
Looking at RIKEN (a monitoring mode and so easy to select on and also one of the most affected as seen from SLAC) seen from the world looking at avg RTT and min RTT we see:
- No effect seen from Africa, E. Asia, Europe, L. America, M. East
- Big effect from N. America (Canada 163ms=>264ms, US 120ms=>280ms)
- India CDAC Mumbia no effect, Pune 380ms=> 460ms, VSNL Mumbia 360ms=>400ms
- Sri Lanka no effect
- Pakistan (we have lots of monitors so should be interesting).
- NIIT sees no effect (nb not on PERN)
- The PERN (Pakistan Education and Research Network) nodes starting with 111. (apart from UAAR see later, this needs more investigation) see 420ms=>500ms
- The PERN nodes starting with 121. See no effect
Conclusion It is not the site RIKEN that has gone bad, rather it is some of the routes
RIKEN Looking at Japan
Japanese hosts seen from JP.RIKEN.N3 (RIKEN) see no impact on RTT
It looks the problem is in the route to Japan not within Japan itself. I wonder if the undersea earthquake has disrupted some cables?
Routes
Comparing the routes from SLAC to RIKEN (ping.riken.jp) and from SLAC to NET.APAN.N2 (ns.jp.apan.net) we see RIKEN has more hops and goes Eastwards via the Avenue of the Americas in NY, while APAN goes directly via Sunnyvale near SLAC and then via Pacific Wave directly to Japan. OSAKA and U-TOKYO are similar to RIKEN. See below
It would be interesting to see the difference in routes from the two (three) types of Pakistani hosts. Are they different does that explain anything? Did the routes change around the time of the earthquake? Unfortunately we do not have such data. It may be useful to monitor all beacons from say SLAC once daily just like we do Pakistani hosts from Pakistan. Alternatively find someone else who has traceroute histories.
Does someone want to put together a case study? I have to do my taxes!
Traceroutes
===========
55cottrell@pinger:~>traceroute ns.jp.apan.net
traceroute to ns.jp.apan.net (203.181.248.3), 30 hops max, 38 byte packets
1 rtr-servcore1-serv01-iepm (134.79.104.66) 0.388 ms 0.238 ms 0.245 ms
2 rtr-core1-p2p-servcore1 (134.79.252.166) 0.318 ms 0.262 ms 0.244 ms
3 rtr-border2-p2p-core1 (134.79.252.141) 0.376 ms 0.332 ms 0.313 ms
4 slac-mr2-p2p-rtr-border2 (192.68.191.249) 0.232 ms 0.212 ms 0.218 ms
5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 0.667 ms 0.640 ms 0.612 ms
6 sunncr1-sunnsdn2.es.net (134.55.209.98) 0.769 ms 0.728 ms 0.721 ms
7 transpac-1-is-jmb-780.lsanca.pacificwave.net (207.231.246.136) 8.888 ms 8.897 ms 8.877 ms
8 tokyo-losa-tp2.transpac2.net (192.203.116.146) 124.537 ms 124.355 ms 124.403 ms
9 ns.jp.apan.net (203.181.248.3) 124.285 ms 124.245 ms 124.249 ms
traceroute to ns.osaka-u.ac.jp (133.1.192.4), 30 hops max, 38 byte packets
1 rtr-servcore1-serv01-iepm (134.79.104.66) 1.515 ms 0.330 ms 0.618 ms
2 rtr-core1-p2p-servcore1 (134.79.252.166) 0.857 ms 0.508 ms 0.539 ms
3 rtr-border1-p2p-core1 (134.79.252.133) 0.870 ms 0.328 ms 0.813 ms
4 slac-mr2-p2p-rtr-border1 (192.68.191.245) 0.284 ms 0.240 ms 0.201 ms
5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 0.627 ms 0.632 ms 0.611 ms
6 sunncr1-sunnsdn2.es.net (134.55.209.98) 0.790 ms 0.710 ms 1.163 ms
7 denvcr2-sunncr1.es.net (134.55.220.49) 56.055 ms 27.906 ms 27.910 ms
MPLS Label=321392 CoS=6 TTL=1 S=0
8 kanscr1-ip-denvcr2.es.net (134.55.209.46) 41.150 ms 101.222 ms 41.102 ms
MPLS Label=357088 CoS=6 TTL=1 S=0
9 chiccr1-ip-kanscr1.es.net (134.55.221.58) 51.712 ms 51.668 ms 51.685 ms
MPLS Label=302528 CoS=6 TTL=1 S=0
10 clevcr1-ip-chiccr1.es.net (134.55.217.53) 60.720 ms 60.746 ms 60.731 ms
MPLS Label=366704 CoS=6 TTL=1 S=0
11 bostcr1-ip-clevcr1.es.net (134.55.41.146) 73.842 ms 73.808 ms 73.777 ms
MPLS Label=363424 CoS=6 TTL=1 S=0
12 aofacr2-ip-bostcr1.es.net (134.55.41.122) 78.715 ms 79.035 ms 78.606 ms
MPLS Label=300768 CoS=6 TTL=1 S=0
13 aofasdn1-ip-aofacr2.es.net (134.55.38.50) 78.453 ms 78.478 ms 78.556 ms
14 198.124.216.154 (198.124.216.154) 78.589 ms 78.693 ms 78.636 ms
15 tokyo1-dc-RM-P-2-3-0-11.sinet.ad.jp (150.99.203.57) 270.322 ms 270.290 ms 270.370 ms
16 nagoya-dc-RM-AE-0-11.sinet.ad.jp (150.99.203.26) 278.341 ms 278.488 ms 278.573 ms
17 osaka-dc-RM-AE-0-11.sinet.ad.jp (150.99.203.30) 282.247 ms 300.091 ms 281.682 ms
18 osaka-u.gw.sinet.ad.jp (150.99.188.62) 282.419 ms 282.440 ms 282.385 ms
19 133.1.0.10 (133.1.0.10) 282.380 ms 282.477 ms 282.408 ms
20 * * *
21 * * *
22 * *
57cottrell@pinger:~>traceroute www.u-tokyo.ac.jp
traceroute to www.u-tokyo.ac.jp (133.11.114.194), 30 hops max, 38 byte packets
1 rtr-servcore1-serv01-iepm (134.79.104.66) 1.042 ms 0.857 ms 1.103 ms
2 rtr-core1-p2p-servcore1 (134.79.252.166) 0.944 ms 0.896 ms 0.900 ms
3 rtr-border2-p2p-core1 (134.79.252.141) 0.941 ms 0.993 ms 0.974 ms
4 slac-mr2-p2p-rtr-border2 (192.68.191.249) 0.858 ms 0.225 ms 0.542 ms
5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 1.108 ms 0.639 ms 1.201 ms
6 sunncr1-sunnsdn2.es.net (134.55.209.98) 1.399 ms 1.282 ms 1.395 ms
7 denvcr2-sunncr1.es.net (134.55.220.49) 28.594 ms 28.434 ms 27.904 ms
MPLS Label=321392 CoS=6 TTL=1 S=0
8 kanscr1-ip-denvcr2.es.net (134.55.209.46) 41.506 ms 41.070 ms 41.107 ms
MPLS Label=357088 CoS=6 TTL=1 S=0
9 chiccr1-ip-kanscr1.es.net (134.55.221.58) 51.696 ms 51.688 ms 51.724 ms
MPLS Label=302528 CoS=6 TTL=1 S=0
10 clevcr1-ip-chiccr1.es.net (134.55.217.53) 60.689 ms 60.753 ms 66.794 ms
MPLS Label=366704 CoS=6 TTL=1 S=0
11 bostcr1-ip-clevcr1.es.net (134.55.41.146) 112.087 ms 73.904 ms 73.876 ms
MPLS Label=363424 CoS=6 TTL=1 S=0
12 aofacr2-ip-bostcr1.es.net (134.55.41.122) 78.626 ms 78.702 ms 78.628 ms
MPLS Label=300768 CoS=6 TTL=1 S=0
13 aofasdn1-ip-aofacr2.es.net (134.55.38.50) 78.539 ms 78.489 ms 78.507 ms
14 198.124.216.154 (198.124.216.154) 78.619 ms 82.098 ms 78.698 ms
15 tokyo1-dc-RM-P-2-3-0-11.sinet.ad.jp (150.99.203.57) 270.472 ms 270.375 ms 270.376 ms
16 UTnet-1.gw.sinet.ad.jp (150.99.190.102) 198.587 ms 197.493 ms 197.495 ms
17 ra36-vlan3.nc.u-tokyo.ac.jp (133.11.127.66) 198.251 ms 197.398 ms 197.402 ms
18 ra35.nc.u-tokyo.ac.jp (133.11.127.41) 197.550 ms 197.259 ms 197.242 ms
19 www.u-tokyo.ac.jp (133.11.114.194) 197.492 ms !<10> 197.565 ms !<10> 197.472 ms !<10>