Agenda for SEECS/SLAC meeting June 28, 2011.

Bold face topics are to be addressed in the meeting.

Upcoming

Zafar Gilani left for Pakistan on July 1st, 2011. He should have recovered by now.

Amber appears to have food poisoning and is resting.

Les will be away next week.

Future & Publicity

• Faisal and Les have put together a motion charts video. It is at: http://www-iepm.slac.stanford.edu/pinger/motionchart/ please look at and advertise to network folks.
  Amber is putting together an email list of PingER contacts to send it to. We will also send to the SEECS list, ICFA/SCIC, Harvey Newman. Next is a YouTube video more for the general public.
• Public Data Explorer Next Gen Motioncharts. Faisal is creating DSPL dataset
  • Concepts are done, tested few slices and a table.
  • Incrementally building tables based on concepts and slices (no debugging or error tracking)
  • Amber is getting the spreadsheet data from various sources.

Pakistani case study - Amber, Anjum

1. 36 hosts are deployed so far and are aiming to deploy 60 hosts in total in Pakistan. As I understand it SEECS will be responsible for the hosts and their PingER app
2. Another 10 nodes are in progress and they would be active in about a month. In about two months, 17 new nodes are expected to be functional.

Latest PERN network map

Anjum got latest PERN map. Notes from Anjum: The provided map is better than the existing one but its still not complete in information and not immediately useable for our purposes. HEC PERN topology maps have yet not been received.

Status of Pakistani PingER hosts

No update, Amber sick.

Responsible people:

• Muhammad Talal Hussain
• Joun Muhammad

PingER managment

• Sadia has almost finished the Connectivity. pl modifications. They add more access to the data, monitor node selection by means of a pull down list. She just has to update the help.
• Sadia is working on modifying pcm.html to give the current monitor hosts by generating the html file from a script.

PingER traceroute archive site

• Sadia will work next on Extending to traceroutes from SLAC to the Beacon sites..

PingER archive site - FYP (Ghulam, Farhan, Zafar)

• Changed getdata.pl script to collect nodes data from Nodes table instead of nodes.cf file because it will not be used in new architecture. For fields where sequence numbers or rtts are not present, NULL is used.
• Main analysis script is analyze-hourly.pl which executes daily on archive site and does analysis. This script has also been changed to get input for analysis from ping_data table instead of files. Tested and optimized.
• Other scripts analyze-daily.pl, analyze-monthly.pl, analyze-allmonths.pl, analyze-allyears.pl will be changed to get nodes information from nodes table instead of nodes.cf file. Remember these scripts are using the same data that analyze-hourly.pl is inserting into the analysis table. They should be very simple given the new database access to the data.
• Analysis of new schema is done. Its working almost 3 times faster in terms of time. See https://confluence.slac.stanford.edu/download/attachments/17162/analysis-db-cf.docx?version=1&modificationDate=1309974754811. Need to correct axis labeling, it also looks way faster than a factor of 3 what am I missing?
• A next step is to port to SLAC. This will be done by Sadia, once she is done with pcm.pl. Les is looking at the SEECS getdata.pl. There are a few mods made at SLAC since 6/4/2011 which will need to be back-ported.
• Completed analyze hourly, completed get data.pl, apparently getdata previous version was taken and there would be a few changes required to make sure that it works with the new getdata.pl. interface will be the same as now, however, at the back end DB would be used instead of flat files. For monthly and yearly we will need some optimization as welll so that browser does not get stuck  or turns off.
• They made a mistake by misunderstanding Zafar. The graph that they showed in the last meeting was between number of files and time. However it should be between number of monitoring hosts and time.  Now they are making the corrections.

Adding MOS and Alpha to pingtable.pl

• Analysis scripts to add Mean Opinion Score and Alpha, some things need to be correctly configured. It has been deployed at [[[http://pinger.seecs.edu.pk/cgi-bin/pingtable.pl|http://pinger.seecs.edu.pk/cgi-bin/pingtable.pl]|http://pinger.seecs.edu.pk/cgi-bin/pingtable.pl]|http://pinger.seecs.edu.pk/cgi-bin/pingtable.pl] for testing.
• Alpha and MOS to be implemented at SLAC site. Sadia will be doing this with the help of Zafar. Currently Ghulam and Farhan are working on synchronizing the SLAC and SEECS scripts. 

TULIP

• TULIP is still slow. The tiering into 0 and 1 done by reflex.cgi reduced times by a factor of 2.
• Enabling/disabling of hosts appears to be working. However, the disabling of hosts has not been turned back on in the daily cronjob. 
• Reflex.cgi is ready for production use. Bilal has a copy. Update

CBG TULIP Integration -- FYP (Bilal)

• TULIP setup on maggie2 server and CBG is running on PERN machine.
• CBG is modified to talk to TULIP. TULIP is modified for integration.
• CBG TULIP integration is almost done. user request will be forwarded to the server(clients java code not server), and then it goes back to CBG code and displays the result in CBG browser.
• Bestline requires much larger data and the data remains same for one week, so to speed up tulip using bestline we are trying to save the data separately (not picking the data runtime). Once this is done, it will speed up the TULIP.
• Sadia is still waiting for the MatLab license.

Best Line Approach CBG-TULIP (Zafar and Bilal):

First thing is the need of landmark sets and their RTT that can be fetched from a DB (They will be designing this DB).

Designing Database:

• need rules for defining sets . (landmark will have sets with all landmarks or will have selected ones  i.e. within a region or max RTT based )
• update frequency( daily/weekly)

Schema: one Table having RTT with pair of LandMark in each row.

create table pairs (
  hostName1 varchar(80) default null,
  hostName2 varchar(80) default null,
  continentOf1 varchar(100) default null,
  continentOf2 varchar(100) default null,
  rttFrom1To2 float default null,
  rttFrom2To1 float default null,
  lastUpdateDate timestamp not null
);

Procedure: Pick landmark fromTULIP db. For each landmark

• Pick another landmark 
• Get RTT from first to second landmark
• Get RTT from second to first landmark
• Take average of both landmarks
• Apply set defining rule. 
• Insert (land1,land2,RTT) in DB table if not present.
• Update RTT if land1 and land2 are present in DB table

Made a database named bestline with a landmarks table in it. TULIP Code is reformed such that it collects data from reflector.cgi and sites.xml and fills results in landmarks table. The data of this table is updated every time TULIP code is executed for finding a target. Currently working on bestline approach. With bestline TULIP has 70% efficiency. Bestline will run before tiering and then reflex.cgi will be used.

Next Task is to enhance bestline.m so as to read data from table and use it for calculating geographical locations.

 Zafar will take a look at parallel scripting of reflector,  currently it is sequential. Tulip will become fast after parallel scripting.

PerfSONAR (Pakistan)

• PerfSONAR at SEECS: Problems were fixed. NTP servers were causing considerable clock delay. Added close-by Stratum 1 NTP servers to solve the problem. Nodes were updated to PerfSONAR version 3.2 (Fedora distro). Nodes however are offline since they were disrupting normal traffic. We are waiting for 10 Mbps dedicated connection to switch the nodes back on. We have a 1 Mbps link for PerfSONAR (on temporary purposes). NUST is purchasing a 2 Mbps dedicated link from WorldCall. No progress yet .. routing issues are showing live IPs as inaccessible.
• PERN will deploy perfSONAR at HEC/Quetta. Someone is working on this. The university is close by HEC/Quetta. Hope in 4 weeks to have PingER monitoring node in 4 universities in the Quetta region.
•  Dr. Anjum is trying to get some live IPs for deploying PerfSONAR, however, no progress yet.

PerfSONAR (USA)

1. Zafar: Yee will meet Jason in person (at Joint Techs in Alaska in July) to tell him about SNMP-MA extensions. He thinks emailing is not a good idea. :P
2. Zafar: Currently working on extending FTMA (which is a new PerfSONAR service and currently under developement) to add in support for MySQL.
3. Faisal is fixing bugs with the new perfSonar mashup that Yee put up.

Possible projects

• See [https://confluence.slac.stanford.edu/display/IEPM/Future+Projects].
• Extend the NODEDETAILS data base to allow entry support for whether the host is currenty pingable. 
• Extend Checkdata to provide emails automatically, see [https://confluence.slac.stanford.edu/display/IEPM/Extend+checkdata+to+make+it+more+useful]. Many of the ideas in the script node-contacts.pl are a step in this direction.
• Improve the PingER2 installation procedures to make it more robust. This might be something for the person(s) in Pakistan who are responsible for installing PingER2 at the Pakistani monitoring sites. They probably have found where the failures occurs. Also look at the FAQ, and ping_data.pl which has been improved to assist in debugging, could it be further improved (e.g. provide access to the httpd.conf file so one can see if it properly configured)? There are 2 students working on the PingER archive. Is this something they could work on?
•  [Fix PingER archiving/analysis package to be IPv6 conformant|IEPM:Make PingER IPV6 compliant]. Will build a proposal for an IPv6 testbed. They will try various transition techniques. A proposal has been prepared and that has been submitted to PTA. Adnan is a co PI. It is being evaluated today.  A small testbed has been established in SEECS and the plan to shift some of the network to IPv6. Bilal is part of 3 students involved with PingER and they will be involved with IPv6. They are porting the PingER archive site site to using a database. They have redeveloped the archive site using Umar's documentation. They have set up a small test archive site. They have gathering, archiving, analysis. They will design a new database. They will also try a port of PingER to IPv6. 
• Look at RRD event detection based on thresholds and how to extend, maybe adding plateau algorithm. Umar's algorithm did  not work in a predictable manner. 
• Provide near realtime plots of current pinger data using getdata_all.pl/wget. It will work as a CGI script with a form to select the host, the ping size, and the time frame to plot. It will use wget or getdata_all.pl to get the relevant data and possibly RRD/smokeping to display the data. 

Future meeting time - Les

The next meeting is Wednesday, July 20th, 2011 (8 pm) for people in US and Thursday July 21st, 2011 (8 am) for people in Pakistan.