psonar_auto.pl

This script automates the whole process of finding new PerfSonar landmarks. It works as follows:

  1. Fetch the list of all PerfSONAR sites from the perfSONAR lookup servers;

    1. Go to http://ps1.es.net:8096/lookup/activehosts.json to get the list of perfSONAR lookup services. 

      1. A locator record looks like:

        {"hosts":[{"priority":100,"locator":"http://ps-west.es.net:8090/lookup/records","status":"alive"},{"priority":100,"locator":"http://ps-east.es.net:8090/lookup/records","status":"unreachable"},{"priority":100,"locator":"http://monipe-ls.rnp.br:8090/lookup/records","status":"alive"},{"priority":100,"locator":"http://ps-sls.sanren.ac.za:8090/lookup/records","status":"unreachable"},{"priority":100,"locator":"http://nsw-brwy-sls1.aarnet.net.au:8090/lookup/records/","status":"alive"}]}



  2. Query each URL in the .locator. field of JSON object returned back and append ?service-type=traceroute. to get the host objects, e.g. the URL might look like

    $url='http://ps-west.es.net:8090/lookup/records?service-type=traceroute'

  3. Get the list of perfSONAR traceroute servers from all the JSON records:

    #Each record is delimited within matching {},  e.g.
{"location-state":["CA"],"location-city":["Pasadena"],"expires":"2017-04-08T22:57:03.775Z","location-code":["91125"],"psservice-eventtypes":["http://ggf.org/ns/nmwg/tools/traceroute/1.0"],"service-type":["traceroute"],"group-communities":["Caltech_IPAC"],"type":["service"],"uri":"lookup/service/3d2e4f8f-4c00-4287-99b6-0e3b9ca7589f","service-host":["lookup/host/7c846099-5376-4b16-9cc5-73236fca4e9f"],"service-name":["IPAC-PRP @ Caltech Traceroute Responder"],"location-country":["US"],"location-longitude":["-118.127629"],"location-latitude":["34.133406"],"client-uuid":["ff78aac6-fc7b-460b-ab77-8c6ff1e5c76a"],"location-sitename":["IPAC-PRP @ Caltech"],"group-domains":[],"state":"renewed","service-administrators":[""],"service-locator":["2607:f380:a4f:71::5","192.84.86.210"],"ttl":[]},{"location-city":["Lodz"]

    For each record extract the IP address , the latitude, longitude, cite, state and country and save it

  4. Compare the final list of working nodes to the ones in TULIP database to list the new sites.

The script is located at : /afs/slac.stanford.edu/package/pinger/tulip/psonar_auto.pl

Multiple text files are generated in the /tmp directory

perfsonar_list.txt -> complete list of PerfSONAR sites

perfsonar_list_working.txt -> sites with working traceroute server

perfsonar_list_working_wl.txt -> sites with working traceroute server and location information

The following file is generated in /afs/slac.stanford.edu/package/pinger/tulip/ .

perfsonar_list_new.txt -> working Perfsonar sites that are not in TULIP DB. The format is: ipaddr | city | state | country | lat | long

Since it requires access to TULIP DB, it must be run from pinger@pinger.

History

34 landmarks added on 6-13-14 (Asad)

92 landmarks added on 4-3-14 (Asad)

7 landmarks added on 1-10-14 (Asad)

56 landmarks added on 1-9-14 (Asad)

94 landmarks added on 1-8-14 (Asad)

trscrontab

This script runs at the start of each month from trscrontab.

perfdb.pl

This script is written to facilitate the process of adding newly found perfSONAR landmarks to TULIP MySQL database. Before running perfdb.pl one must load perfsonar_list_new.txt (generated by psonar_auto.pl) in MS Excel. When loading the text file choose both 'tab' and 'comma' as delimiting characters. Now most of rows would have this format:

IP Addr | City | State | Country | Latitude | Longitude

However the format required by perfdb.pl is:

IP Addr | City | Country | Region | Latitude | Longitude

Where the region could be Europe, North America, Latin America, S.E. Asia, South Asia, East Asia, Russia, Africa, Australasia e.t.c. New regions could be defined provided we have sufficient landmarks.

The excel file should be manually checked to ensure that all the landmarks entries are correct. It is not uncommon for the coordinates to be incorrect. In most of the cases the error would be due to a typo and can be fixed easily. Check for the following:

Latitude and Longitude swapped

Missing - negative sign

single digit typo

It is best to delete landmark entries which have their coordinates as whole integers since this location might be inaccurate to over 100 kilometers.

The final excel file should be saved in tab separated format .txt (tsv). The path of this file should be provided in perfdb.pl (line 63) and finally the script can be executed.

script location: /nfs/slac/g/net/pinger/tulip/perfdb.pl

Disclaimer: This script can cause serious damage if improperly used. You have been warned !! 

perf-hn.pl

This script uses nslookup to find the hostname of a landmark using it's IP address. After adding new PerfSONAR landmarks, one should run this script to find hostnames of such nodes that have been been entered with IP addresses only. This will automatically update the TULIP database with the found hostnames.

script location: /afs/slac/package/pinger/tulip/perf-hn.pl

  • No labels