Page History

 We use the "tulip " database to generate our sites.xml which is further used in probing the landmarks. We added pingER nodes from the Nodedetails database to the tulip database but with some defined rules. We only added those nodes which contain a traceroute server. To implement this and make it working , we developed three packages. Many of the scripts are executed automatically from trscrontab running under pinger@pinger.slac.stanford.edu  (see section below).

•  TULIP/ANALYSIS/NODEDETAILNODES.pm
•  insert_sites-xml.pl
•  create_sites-xml.pl

We also add nodes from PlanetLab.

TULIP/ANALYSIS/NODEDETAILNODES.pm

It is found at /afs/slac/package/pinger/tulip/TULIP/ANALYSIS/NODEDETAILNODES.pm. In order to build this module, we used some predefined  predefined perl modules and scripts (require '/afs/slac/package/pinger/nodes.cf' and require Text::CSV_XS). To get data from node details the PingER Nodedetails database we used require '/afs/slac/package/netmon/pinger/nodes.cf' and we used standard package Text::CSV_XS to convert our data in comma-separated values. In order to define our node to be a candidate for a tulip landmark we tested it for a few conditions which include that the node must have traceroute server, it should not be set to NOT-SET and it's its project type should not be set to "D" which means deleted disabled as per nodedetails Nodedetails database semantics. The nodes Nodes which qualified these conditions were put into a separate array. This array is used by insert_sites-xml.pl to insert these sites to into the Tulip database

 insert insert_sites-xml.pl

This perl script is used to create the insert query from the data of the above nodes. Again using the perl package theperlpackage Text::CSV_XS we XSwe divide our data into separate chunks. The data is then fed to the structure which contains parameters for the query. This script resolves each hosts host with host names hostnames taken from nodedetailnodesNODEDETAILS.pm this helps in eliminating bad hosts if they exist. Before inserting new nodes into the database it checks weather whether the node is in nodedetail Nodedetail or not. We use ipv4Addr as our unique key. We traverse through the tulip database and check if there exists some node with the same ipv4addr if . If it exists we ignore the entry and if not we go ahead with inserting it in the database.  It has a use  TULIP::ANALYSIS::NODEDETAILNODES;

create_sites-xml.pl pl

 This This perl script is used to create the sites.xml file which is further used in our TULIP project as a source for node information and landmarks. This perl module uses the template library in order to generate the required xml. It traverses through the tulip database and gets each node, checks the service type and generates the file with all the available parameters in the database.  It has a require '/afs/slac/package/pinger/tulip/insert_sites-xml.pl'.

Tulip  Tulip Transition to Sites.XML

Our next step in the process is to transform TULIP so that it can get data from created sites.xml. TULIP version 1 was having two different data sources one to get data from nodedetail data base database and the other was to fetch data from a list containing the Planet lab sites.

Inorder In order to perfom perform this step we used JAVA Xerces  API Xerces API, which is described as the best option in terms of resources used and efficiency in parsing XML data. We used the SAX model because of the following reasons.

1) The list of landmarks would increase as more pinger monitoring sites are added

2) We donot do not require to rewrite or add xml to the document from TULIP

Tutorial posted by SUN is helpful in understanding the parsing of XML. The tutorial can be found over found  here

 TULIP Client

  TULIP has a Java based client which have has the important functions and classes for sending the query over the web to reflector.cgi. In addition to sending query and getting/parsing responses the TULIP client uses the algorithm(Trilateration or Multilateration) to find the location of queried host. This section explains the technical details about the tulip client and the classes used and important function. The re-implementation of TULIP has led to proper packages and the use of ANT package management tool by Apache for code compilation.

...

Dependencies

Current implementation of the TULIP Java client depend depends on two files for its execution

• Sites.xml
• /afs/slac/package/netmonpinger/tulip/Initial.txt

The former is dicussed discussed above in this article. The latter later is used to Automate test. Its a list of nodes/sites which TULIP tries to locate when it is run runs from the command linline. 

Code Deployment at SLAC

TULIP is deployed at SLAC in /afs/slac/package/netmonpinger/tulip/src

The class hierarchy is defined in three folders which are defined in library and tulip client hierarchy.

• Library Folder
  • The lib directory contains third-party packages used for implementation of TULIP. E. For eg g we are using  Java Matrix Class which provides the fundamental operations of numerical
       linear algebra.
• TULIP Folder
  • This folder contains the actual implementation of the TULIP client. It has two branches core and utill. The core package contains core classes or classes which are used in directly invloved in  implementation of Tulip. The utill package contains all the utility classes for the core classes i.e they help in implementation. For instance to get the data we need to parse sites.xml. So, getData is core class where as whereas the xml parsing class is helping it to achieve its goal so its it's in the utill package. 

TULIP Core Classes

TULIP core classes are divided into three sections.

• Automate.java
  • This class contains the control of the program. This class have has the "main" function which initiates the test and decides the flow of the program.
• GetPingDataPL.java
  • This class handles the all the issues relating related to parsing sites.xm quering reflector xml, querying reflector.cgi and parsing the results. It uses Sites.java as its child class which is in the utill functionpackage. Sites.java contains all the parameters like location information, node IP address and rtt values. To populate the geo-location information it parses sites.xml and for the rest of the parameters it parses response responses from the reflector.
• Locate.java and PhysicalDistance.java
  • These classes contains the algorithm it self. Once GetPingDataPL has fiished its functionality Automate.java handles the control to these two classes and they use Triangulation/Multilateration Algorithm to identify the location of the host.

...

The code has been reformated in a way that it should work with a well know package management tool (ANT). The main benifit benefit of ant is that you do not need not to compile a class and set the path once you have edited the class. The package management tool handes handles it all. The build.xml file has been written to set the paths of classes once the code gets compiled. It would require the following steps to compile and run.

cd /afs/slac/package/pinger/tulip

ant -f build.xml

This command would compile the code and create a directory build in the same folder with all the .class files. To run the code you need to run the following command from build directory is needed.

 Running code for List of hosts

There are different command line  line options available for running the code. Running the code without any options would yield basic test for a list of host in Initial.txt available at tulip directory. The output would be entries in Result.txt which is in $tulip /afs/slac/package/pinger/tulip with best three nodes and RTT's.  The format of Result.txt is Target node followed by first best node its RTT then second and third best in the same manner.

Run following command from /afs/slac/package/netmonpinger/tulip/build if Class path is not set
 java -cp commons-httpclient-3.1.jar:commons-logging-1.1.1.jar:commons-codec-1.3.jar:. tulip.core.AutomateTest

Running Code for specific Target

The second option available for TULIP command line is running the code for specified target we are trying to locate. The result would be the last line in Result.txt file with the above-mentioned format. Following command is required to run the code to locate your that specified target.

java -cp commons-httpclient-3.1.jar:commons-logging-1.1.1.jar:commons-codec-1.3.jar:. tulip.core.AutomateTest target 128.2.115.21 startNewTest

Running Code for List of Targets and updated Log File

If you have conducted new tests and want to run the code on an updated log file, you need to download the log file. This implementation is in place to save the network traffic as code as to go back and forth on the web page and then search for the required targets. In this implementation, we download the log file after every update of log i.e. every time we conduct new tests. The web front end of the log file is at

http://www-wanmon/cgi-wrap/reflector.cgi?function=log

The command for running the code for a list of targets and required tiers is:

java -cp commons-httpclient-3.1.jar:commons-logging-1.1.1.jar:commons-codec-1.3.jar:.
tulip.core.AutomateTest tierFile /afs/slac/package/pinger/tulip/Tiers.txt
targetFile /afs/slac/package/netmon/tulip/Initial.txt downloadLog analyzeLog

The output of this command would be a file rtt.txt at $tulip directory. This file would contain rtts of our selected nodes.

Note that the log file by default is on wanmon.slac.stanford.edu in /tmp/tulip_log, e.g.

cottrell@wanmon ~]$ ls -l /tmp/tulip_log
s-rw-rw-rw- 1 apache apache 2072902 Nov  1 02:30 /tmp/tulip_log

 Selection Selection of Planet Lab Sites

To select the Planet Lab sites we need to go through a process. In this section, we would highlight the scripts used in the process and their functionalities.

 generatePL generatePL.pl

...

...

The

...

process

...

starts with

...

a

...

script

...

/afs/slac/package/

...

 This script is deployed in crontab which generate a all the available PlanetLab sites. The list is gather through the following link

...

pinger/tulip/generatePL.pl

...

This script generate Sites-yyyyyyy-mm-ddd dd files in /afs/slac/package/netmonpinger/tulip/sitesxml with the nodes which are marked up by the scriptroute adminstratorsadministrators.analyzePL.pl. The contents appear as:

Code Block

#Sun Nov 4 03:35:03 2018 /afs/slac/package/pinger/tulip/generatePL.pl: created 360 lines (excluding this line) on pinger.slac.stanford.edu for pinger from wget --tries=5 --timeout=30 --wait=120 --no-verbose --quiet 'http://www.scriptroute.org:3967/' -O - Cleveland,"United States",129.22.150.78,41.5074,-81.6053,,NorthAmerica Vancouver,Canada,142.103.2.2,49.25,-123.1333,,NorthAmerica Warsaw,Poland,194.29.178.14,52.25,21,,Europe ,"United States",198.133.224.147,38,-97,,NorthAmerica ...

analyzePL.pl

This script opens up each file generated by generatePL.pl and counts the number of times each node appears in the files. This way we can have the idea for the uptime of every node. And based on the results we can choose the node to be added to our repository(tulip database) as the primilanary preliminary passed node.

After primarily primary evaluation, we run tulip from command the command line and test the added nodes for their ping servers. The complete test contains 107 sites to be traced by TULIP client.  This generates the log file with all the successful and failing landmarks.

tulip-log-analyze.pl

 The purpose of this script is to parse the log file and provide us a hawk bird's eye view of the landmarks which are failing or are successful. We then disable all those nodes which do not reply to requests by reflector.cgi. This script also generates the total delay/time taken by each landmark to respond. We will use this parameter for the final selection of our tier 0 landmarks.

Cronjobs