Traceroute.pl is configured to send an email warning to selected email addresses if greater than a threshold number of traceroute.pl processes are in progress. In this case it aslo does not launch more traceroute.pl processes, and responds to the requester.

The emails are of the form:

Sat Jan  7 04:49:44
2017 HTTP_REFERER =  found 12 running
traceroute processes in SERVER_NAME = www.slac.stanford.edu
= nobody@www1.slac.stanford.edu(134.79.197.200)
when doing traceroute to address=216.34.181.45 host=slashdot.org. Request was
received from REMOTE_ADDR = 79.47.72.33(REMOTE_HOST =
)(name=host33-72-dynamic.47-79-r.retail.telecomitalia.it).

The currently being accumulated log can be found by
logging onto the SERVER_NAME (see above) and looking in
/var/apache/log/access_log.*

The archived logs are contained in:
/afs/slac/u/sf/webmastr/logs/apache/log.*

These are large zipped files so you will need to use
gunzip to unzip them, e.g.
>cp
/afs/slac/u/sf/webmastr/logs/apache/log.20110929.gz /tmp/junk.gz 
>gunzip /tmp/junk.gz
There can be a thousand or more traceroute.pl requests in
a day so usually it is best to do a:
>grep traceroute.pl /tmp/junk >! /tmp/tr

and then look at /tmp/tr

You can do a further clean up of the log by using grep
"GET /cgi" /tmp/tr >! /tmp/tr1

To see what traceroute processes are currently running on www, do a:
ps -ef | grep traceroute

An example of determining what is the cause is given below:

325cottrell@www1:~>ls -l /var/apache/log/access_log.*

-rw-r--r--   1 nobody   nobody   3959116 Jun  3  2015 /var/apache/log/access_log.1433289600.gz

-rw-r--r--   1 root     other    28024541 Jan  7 12:31 /var/apache/log/access_log.1483747200

330cottrell@www1:~>grep traceroute /var/apache/log/access_log.1483747200 >! traceroute.log

332cottrell@www1:~>head traceroute.log

200.94.128.220 - - [06/Jan/2017:16:00:57 -0800] "GET /cgi-bin/nph-traceroute.pl?target=67.21.6.70 HTTP/1.1" 200 - "-" "-" www.slac.stanford.edu TLSv1 -

…

333cottrell@www1:~>tail traceroute.log

…

77.78.14.216 - - [07/Jan/2017:12:40:04 -0800] "GET /cgi-bin/nph-traceroute.pl HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" www.slac.stanford.edu - -

 

335cottrell@www1:~>grep '^200' traceroute.log | grep target | wc #Look for big hitter

     662    9930  103254

Looking at this we see 662 hits in 20:40 hours from 200.94.128.220, they are typically a minute or so apart though they can come in bursts of  up to 5 more times to the same target every 4 seconds.

We used cymru to get an idea of where the host was located.  

[Querying v4.whois.cymru.com]

[v4.whois.cymru.com]

AS      | IP               | AS Name

11172   | 200.94.128.220   | Alestra, S. de R.L. de C.V., MX

This is confirmed by the traceroute
339cottrell@www1:~>traceroute 200.94.128.220
traceroute to 200.94.128.220 (200.94.128.220), 30 hops max, 40 byte packets
 1  134.79.197.131 (134.79.197.131)  0.913 ms  1.547 ms  1.647 ms
 2  rtr-core2-p2p-serv01-02.slac.stanford.edu (134.79.254.61)  0.473 ms  0.316 ms  0.379 ms
 3  rtr-core2-p2p-core2-new.slac.stanford.edu (134.79.252.50)  0.481 ms  0.308 ms  0.443 ms
 4  rtr-fwcore2-trust-p2p-core2.slac.stanford.edu (134.79.254.146)  0.997 ms  0.890 ms  0.800 ms
 5  rtr-core2-p2p-fwcore2-untrust.slac.stanford.edu (134.79.254.149)  1.042 ms  1.003 ms rtr-border1-p2p-core1.slac.stanford.edu (134.79.252.133)  1.067 ms
 6  rtr-border2-p2p-border1.slac.stanford.edu (192.68.191.253)  1.473 ms  1.362 ms  1.326 ms
 7  sunncr5-ip-c-slac.slac.stanford.edu (192.68.191.233)  1.660 ms  1.566 ms  1.505 ms
 8  sacrcr5-ip-a-sunncr5.es.net (134.55.40.5)  4.139 ms  4.063 ms  4.030 ms
 9  pnwgcr5-ip-a-sacrcr5.es.net (134.55.43.21)  18.518 ms  18.496 ms  18.381 ms
10  ge3-0.cr02.sea01.pccwbtn.net (206.81.80.13)  18.645 ms  18.643 ms  18.634 ms
11  TenGE0-0-0-11.br01.lar02.pccwbtn.net (63.223.21.81)  91.233 ms  91.861 ms  91.123 ms
12  alestra.te0-0-0-8.br01.mca02.pccwbtn.net (63.222.46.6)  95.188 ms  95.346 ms  95.243 ms
13  static-201-163-102-210.alestra.net.mx (201.163.102.210)  98.899 ms static-201-163-102-206.alestra.net.mx (201.163.102.206)  98.503 ms static-201-163-102-186.alestra.net.mx (201.163.102.186)  97.230 ms
14  rdmty2.alestra.net.mx (201.151.27.82)  95.213 ms  95.151 ms static-201-151-27-254.alestra.net.mx (201.151.27.254)  98.907 ms
15  * * *
16  * * *

and:

343cottrell@www1:~>nslookup 200.94.128.220
Server:         127.0.0.1
Address:        127.0.0.1#53
Non-authoritative answer:
220.128.94.200.in-addr.arpa     name = static-200-94-128-220.alestra.net.mx.

  • No labels