Added by Umar Kalim, last edited by Umar Kalim on Oct 29, 2009  (view change)

Labels:

Enter labels to add to this page:
Wait Image 
Looking for a label? Just start typing.

Structure

The archive node is: pinger.seecs.edu.pk

The executable scripts are located in the home directory of the user "pinger" 

/home/pinger/bin
The meta data is placed at: 
/home/pinger/metadata
The output files are available at: 
/var/www/html/
The libraries are available at: 
/home/pinger/lib
The log files are available at:

/home/pinger/logs

The output from all cronjobs are emailed to umar.kalim@seecs.edu.pk.

1. List of offsitenodes for timeping

This script reads nodeinformation from nodedetails database and formats the output into an automated version of offsite.nodes wich is used to execute timeping. This script eliminates the need to edit the .txt file 'offsite.nodes'. This script is to be executed at least once at ~7pm to collect any new nodes added during the day.

Cronjob at SLAC:

0 0,6,12,18 * * * /afs/slac/g/scs/net/pinger/bin/write_offsitenodes

Cronjob at NUST:

# Create the offsite.nodes list of nodes to ping from NODEDETAILS database
# writes out the offsite.nodes file four times a day (midnight, 6am, noon and 6pm)
0 0,6,12,18 * * * /home/pinger/bin/write_offsitenodes.pl

The output is accessible at: http://pinger.seecs.edu.pk/offsites.nodes

2. List of countries and TLDs

This script selects all the countries and their TLDs from the PingER database and compiles a txt file. This text file is then used by various scripts.

The list of countries are kept up-to-date by executing this script as a cronjob on daily basis by the pinger user.

Cronjob at SLAC:

15 0 * * * /afs/slac/g/scs/net/pinger/bin/write_country_list

Cronjob at NUST:

15 0 * * * /home/pinger/bin/write_country_list.pl

The output is accessible at: http://pinger.seecs.edu.pk/countries.tsv

3. Dump of the NODEDETAILS data as a perl configuration file

This script s used to create a dump of the database in the form of a perl configuration file. This file is then used by several scripts used for PingER's management

Cronjob at SLAC:

30 00 * * * /afs/slac/package/pinger/guthrie/node.pl -o > /afs/slac/package/pinger/nodes.cf

Cronjob at NUST:

# Update nodes.cf (Perl dump) from the NODEDETAIL database
# Though node.pl is called from runall.pl, runall.pl is not called until 5am,
# so it is updated here first to be sure.
30 00 * * * /home/pinger/bin/node.pl -o > /var/www/html/nodes.cf

The output is accessible at: http://pinger.seecs.edu.pk/nodes.cf

4. List of participants

This script identifies the project participants -- sites with monitoring nodes -- and lists the services available and contact persons. The output is accessible online.

Cronjob at NUST:

 4 0 * * * /home/pinger/bin/participants.pl --list
              --output-file /var/www/html/participants.html

5. Downloading data from Monitoring Sites

The scripts loads the list of nodes from nodes.cf and then fetches data from the monitoring nodes. The downloaded data is then stored locally at: 

/pinger_data/
The crontab command is as shown below. It is important to note though that monitoring nodes located west of Pakistan are in different timezones. Thus the archive process must wait untill the node in the last time zone has collected its data. In our case it would be SLAC which is in the time zone GMT-8. Since Pakistan is at GMT+5, any time after 1.00 pm (Pak. Std Time) would be appropriate.

6 14 * * * /home/pinger/bin/getdata.pl > /home/pinger/logs/getdata.err
The script getdata_all.pl is used to fetch data from monitoring sites whose web-server may have gone offline.

/home/pinger/bin/getdata_all.pl

6. Checking availability of data from Monitoring Sites

The crontab command is as follows: 

 1 15 * * * /home/pinger/bin/checkdata_gif.pl
This script uses the following modules/scripts: 
/home/pinger/bin/PingIt.pm
/home/pinger/bin/checkdata.pl
HTML files are generated to facilitate maintenance. The latest file is available as index.html, where as monthly archives are stored with abbreviated names e.g. Sep.html

/var/www/html/checkdata/index.html
/var/www/html/checkdata/Sep.html
The associated files are:

  /var/www/html/checkdata/checkdata_image_dir

7. Data Analysis

The analysis and aggregation scripts are classified in terms of epochs (hourly, daily, monthly and yearly).

The data is stored at: 

/pinger_data
Where as the reports are stored at:

 /pinger_reports -> /pinger_data/reports

Analyze Hourly

05 15 * * * /home/pinger/bin/analysis/analyze-hourly.pl --usemetric --dataset hep >
   /home/pinger/logs/analyze-hourly.log 2>&1
10 15 * * * /home/pinger/bin/analysis/analyze-hourly.pl --usemetric --dataset hep --by by-site >
   /home/pinger/logs/analyze-hourly-by-site.log 2>&1
15 15 * * * /home/pinger/bin/analysis/analyze-hourly.pl --usemetric --dataset hep --size 1000 >
   /home/pinger/logs/analyze-hourly-1000.log 2>&1
20 15 * * * /home/pinger/bin/analysis/analyze-hourly.pl --usemetric --dataset hep --size 1000 --by by-site >
   /home/pinger/logs/analyze-hourly-by-site-1000.log 2>&1

Analyze Daily

Current Month

25 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep >
              /home/pinger/logs/analyze-daily.log 2>&1
32 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --by by-site >
              /home/pinger/logs/analyze-daily-by-site.log 2>&1
39 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --size 1000 >
              /home/pinger/logs/analyze-daily-1000.log 2>&1
46 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --by by-site --size 1000 >
              /home/pinger/logs/analyze-daily-by-site-1000.log 2>&1

Last 60 Days

30 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --date 60days >
              /home/pinger/logs/analyze-daily-60d.log   #  2>&1
31 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --by by-site --date 60days >
              /home/pinger/logs/analyze-daily-by-site-60d.log   #  2>&1
32 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --size 1000 --date 60days >
              /home/pinger/logs/analyze-daily-1000-60d.log   #  2>&1
33 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --by by-site --size 1000 --date 60days >
              /home/pinger/logs/analyze-daily-by-site-1000-60d.log   #  2>&1

Last 120 Days

50 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --date 120days >
                    /home/pinger/logs/analyze-daily-120d.log   #  2>&1
51 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --by by-site --date 120days >
                    /home/pinger/logs/analyze-daily-by-site-120d.log  #   2>&1
52 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --size 1000 --date 120days >
                    /home/pinger/logs/analyze-daily-1000-120d.log   #  2>&1
53 15 * * * /home/pinger/bin/analysis/analyze-daily.pl  --usemetric --dataset hep --by by-site --size 1000 --date 120days >
                    /home/pinger/logs/analyze-daily-by-stide-1000-120d.log   # 2>&1

Analyze Monthly

30 16 * * * /home/pinger/bin/analysis/analyze-monthly.pl --usemetric --dataset hep >
               /home/pinger/logs/analyze-monthly.log #  2>&1
31 16 * * * /home/pinger/bin/analysis/analyze-monthly.pl --usemetric --dataset hep --by by-site >
               /home/pinger/logs/analyze-monthly-by-site.log # 2>&1
32 16 * * * /home/pinger/bin/analysis/analyze-monthly.pl --usemetric --dataset hep --size 1000 >
               /home/pinger/logs/analyze-monthly-1000.log # 2>&1
33 16 * * * /home/pinger/bin/analysis/analyze-monthly.pl --usemetric --dataset hep --size 1000 --by by-site >
               /home/pinger/logs/analyze-monthly-by-site-1000.log # 2>&1

Analyze AllMonths

50 16 1 * * /home/pinger/bin/analysis/analyze-allmonths.pl --usemetric --dataset hep >
              /home/pinger/logs/analyze-allmonths.log 2>&1
51 16 1 * * /home/pinger/bin/analysis/analyze-allmonths.pl --usemetric --dataset hep --by by-site >
              /home/pinger/logs/analyze-allmonths-by-site.log 2>&1
52 16 1 * * /home/pinger/bin/analysis/analyze-allmonths.pl --usemetric --dataset hep --size 1000 >
              /home/pinger/logs/analyze-allmonths-1000.log 2>&1
53 16 1 * * /home/pinger/bin/analysis/analyze-allmonths.pl --usemetric --dataset hep --size 1000 --by by-site >
              /home/pinger/logs/analyze-allmonths-by-site-1000.log 2>&1

Analyze AllYears

01 17 1 1 * /home/pinger/bin/analysis/analyze-allyears.pl --usemetric --dataset hep >
               /home/pinger/logs/analyze-allyears.log 2>&1
05 17 1 1 * /home/pinger/bin/analysis/analyze-allyears.pl --usemetric --dataset hep --by by-site >
               /home/pinger/logs/analyze-allyears-by-site.log 2>&1
10 17 1 1 * /home/pinger/bin/analysis/analyze-allyears.pl --usemetric --dataset hep --size 1000 >
               /home/pinger/logs/analyze-allyears-1000.log 2>&1
15 17 1 1 * /home/pinger/bin/analysis/analyze-allyears.pl --usemetric --dataset hep --size 1000 --by by-site >
               /home/pinger/logs/analyze-allyears-by-site-1000.log 2>&1

8. Reports

pingtable.pl

Pinger raw data is stored at: 

/pinger_data/hep/data
and the analysis reports are stored at: 
/pinger_reports/hep/ which translates to /pinger_data/reports/hep/
The pingtable.pl is stored in the cgi folder and is accessible here. Note that pingtable.pl presents the hep data only.

In addition to the analysis reports, the associated files are:

 /home/pinger/metadata/pingtable_filter.cfg

prmout

The prm.pl script is available at: 

/home/pinger/bin/prm/prm.pl
The associated configuration files (prm.conf, prm-monthly.conf, prm-allyearly.conf and prm-last60days.conf) are located in the same folder. Note that the deployment at SEECS uses MySQL instead of Oracle and therefore the SQL queries where changed to use the keyword DISTINCT instead of UNIQUE.

The reports generated are available online. This page was generated by the script createpage.pl which is available at:

 /home/pinger/bin/prm/createpage.pl

prm.pl is executed via the crontab as shown below: 

prm.pl runs monthly to create the executive plots to be accessesd via:
#http://pinger.seecs.edu.pk/prmout/
30 02 1 * * /home/pinger/bin/prm/prm.pl -c /home/pinger/bin/prm/prm-monthly.conf
#And yearly (Jan 5th at 4:30am) to get the yearly data.
30 04 05 1 * /home/pinger/bin/prm/prm.pl -c /home/pinger/bin/prm/prm-allyearly.conf
#It also runs each morning to update the last 21 days information.
30 00 * * * /home/pinger/bin/prm/prm.pl -c /home/pinger/bin/prm/prm-last60days.conf
The prmout files are used by the motion charts.

9. Pinger Coverage Google Map

The pinger coverage google map is available here. 

/var/www/html/gmap/pinger-coverage-gmap.html
A new maps API key was required and obtained from here. The associated files are placed in the same folder.

The RSS files listed below are critical to the google-maps layout. 

/var/www/html/gmap/rss-googlemaps.xml
/var/www/html/gmap/rss-ihy-only-googlemaps.xml
/var/www/html/gmap/rss-asan-googlemaps.xml
/var/www/html/gmap/rss-whi-ihy.xml
rss-googlemaps.xml is created with the help of the generateRSS.pl script, where as the others were created manually.

/home/pinger/bin/gmap/generateRSS.pl -g

Usage:	 generateRSS.pl [opts]
        Opts: -d, -s, -c, -v, -g

   -d generates an RSS file including Disabled hosts.
   -s generates the slaconly RSS file with the names and IP addresses
      default is generate the world version of the RSS file without
      IP addresses. If -s is specified the output
      is written to /home/pinger/bin/gmap//rss-slaconly.xml else it is written
      to /home/pinger/bin/gmap//rss.xml
   -c include contact info and IP addresses in the file
      output is written to /home/pinger/bin/gmap//rss-with-contacts.xml
   -v generates this output
   -g generate the rss.xml file for the Google Maps Visualization
 
Note that generateRSS.pl was customized to use MySQL instead of Oracle used at SLAC.

A crontab command has been setup to keep the rss-googlemaps.xml up to date:

 #Generate XML for Google PingER maps
09 0 * * * /home/pinger/bin/gmap/generateRSS.pl -g;
     cp /home/pinger/bin/gmap/rss-googlemaps.xml /var/www/html/gmap/rss-googlemaps.xml

10. Shockwave Map

To transfer the shockwave map, the relevant files were simply copied to the server. These files are available under the path listed below and can be viewed here. 

 /var/www/html/pingerworld/
A crontab command has been setup to keep the rss.xml up to date:

#Generate XML for Shockwave PingER map
08 0 * * * /home/pinger/bin/gmap/generateRSS.pl;
     cp /home/pinger/bin/gmap/rss.xml /var/www/html/pingerworld/rss.xml

11. Update.pl

Update.pl lists all the remote sites monitored by the pinger infrastructure. The script is available here: 

/home/pinger/bin/update.pl
It executes as a cronjob daily: 
#Monitoring sites HTML file.also all the remote nodes etc.
10 0 * * * /home/pinger/bin/update.pl
The relevant files are:

/var/www/html/nodes.cf
/var/www/html/pinger.new.cf

12. sites-per-country.pl

The script is setup as a cronjob: 

# Create an HTML table of PingER nodes per country and region, add by Kalim 10/27/09
# I will have to comment this out later because runall.pl also calls it
6 0 * * * /home/pinger/bin/sites-per-country.pl
The listing is available online.

13. Nodes per country Maps

The Hosts per country maps are generated by the script 

 /home/pinger/bin/geo-map/create-node-count-geomap.pl
The output for remote nodes and monitoring nodes are available online.

The maps are updated regularly with the help of a cronjob: 

#Update the node count geo-map of the hosts per country and metrics/country
7 0 * * * /home/pinger/bin/geo-map/create-pinger-node-count-geomap.pl -r M
      -o /var/www/html/geo-maps/pinger-mon-node-count-geomap.html
      > /home/pinger/logs/pinger-mon-node-count-geomap.stdout
8 0 * * * /home/pinger/bin/geo-map/create-pinger-node-count-geomap.pl
      -o /var/www/html/geo-maps/pinger-node-count-geomap.html
      > /home/pinger/logs/pinger-node-count-geomap.stdout
The documentation is available here.

14. Metric Motion Charts

The metric motion charts are available here. The detailed documentation is available here.

The motion charts are generated by the scripts: 

 /home/pinger/bin/motion-chart/create=motion-chart-pages.pl
They are regularly updated with the help of a cronjob:

#Create the motion bubble plots:
9 0 * * * /home/pinger/bin/motion-chart/create-motion-chart-pages.pl
         > /home/pinger/logs/motion-charts-log.stdout

15. Miscellaneous Documentation

Download and Installation

I copied the installation package to NUST as well as the setup procedure for a) pinger and b) ping_data.pl and traceroute.plscript.

These files are stored at:

/var/www/html/install1.html
/var/www/html/install2.html

/var/www/html/code/pinger-2.0.2.tar.gz

Reverse Traceroute Servers

The list of reverse traceroute servers is also maintained here.

/var/www/html/traceroute-srv.html

Monitoring and Remote Node Requirements pages

The monitoring node requirements as well as the remote node requirements are published online. The files are placed here:

/var/www/html/wan-req.html
/var/www/html/mon-req.html

List of countries, their regions and their TLDs

The list of countries with their regions and TLDs are maintained online. These files are stored at: 

/var/www/html/countries.tsv
/var/www/html/tlds_country_list.txt
_END_