SLAC SEECS Status report  May 22nd, 2012

General

The SEAMEW cable cut off Alexandria at about 12noon Pak time May 11th was observed in the monitoring stats of Micronet. As a result I have added two Nayatel/Micronet hosts to the PingER monitoring at SLAC? One connects via TransWorld the other by PTCL.

Asad will take over TULIP and include the conversion of CBG from MatLab to Mathematica as their MS thesis. He will be joing us mid June, just after his exams.

We have the signed MOU between SLAC and the University of Malaysia in Sarawak (near the capital Kuching). Les will put together a presentation for the new SLAC CIO

Anjum has applied for an HEC grant, also trying NUST to partially fund. Do not see a student at SLAC for 6 months and this may be optimistic. No progress yet.

Maggie is not working due to no disk space. Les does not recognize any of the files. Anjum gave the go ahead to clean /tmp/

We are about to order new hardware for the PingER monitoring host at SLAC and the www-wanmon web server.

HEC Report - Anjum, Amber and Imdad

Amber is working on this report. The basic structure is the same as was of the last 6 monthly report. She will be sharing the report in a few days.  

Pakistani Hosts

There are some Pakistani nodes that are recorded as working from SEECS while they are not working from SLAC. This mismatch was recorded by Amber and Joun. Amber and Joun will get together and see if the problem still exists or not.

• Les looked at which Pakistani monitoring hosts SLAC is unable to gather data from since the start of May. They are as follows:
  • hu.seecs.edu.pk aka PK.HU.EDU.N2 does not appear as a monitoring host in SEECS pingtable.pl. Also SEECS pingtable.pl has no data this month from AIOU to hu.seecs
  • lse.seecs.edu.pk aka PK.LSE.EDU.N3 does not appear as a monitoring host in SEECS pingtable.pl. Also SEECS pingtable.pl has no data this month from AIOU to lse
  • pinger-itc.pu.edu.pk aka PK.PU.EDU.N2 does not appear in the SEECS pingtable.pl as a monitoring host
  • pinger.giki.edu.pk aka PK.GIKI.EDU.N1 does not appear as a monitoring host in SEECS pingtable.pl
  • pinger.uaar.edu.pk aka PK.UAAR.EDU.N1 does not appear as a monitoring host in SEECS pingtable.pl
  • pinger.ustb.edu.pk aka PK.USTB.EDU.N2 does not appear as a monitoring host in SEECS pingtable.pl
  • sbkwu.seecs.edu.pk aka PK.SBKWU.SEECS.EDU does not appear as a monitoring host in SEECS pingtable.pl
• Should they be designated as Monitoring hosts?
• Amber reports: Looking at this weeks status report on wiki, the list of the nodes which are not monitoring nodes in SEECS pingtable but are monitoring nodes at SLAC can be due to the fact that these nodes are the usual problematic nodes since months. It might be possible that SEECS team (who manage SEECS pingtable) have made these nodes as remote nodes to avoid hassle. However, I shall confirm this from Kashif tomorrow and will update you on this. If this turns out to be true, I shall change these nodes to remote nodes in SLAC pingtable as well.

FSBD  POP have high unreachability values, which is not acceptable. They are looking into it. Backhaul network is currently leased from PTCL however in 3-4 months they will replace it with their own network. There would be no commercial traffic on it. As a result it is expected that RTT and losses will improve drastically. So next 6 months are important for observing the network performance.

PingER Archive Site - Sadia

Sadia had a meeting with Ghulam and Bilal . They gave her some relative material which she will document. It includes query regarding schema,  and how to run Tulip test from Seecs machine. This would help new student to generate the reports for regions etc. As far as pinger database schema is concerned we need to discuss it with someone like Zafar who has worked on Perfsonar to see is it feasible or not . The main concern is that PerfSonar has single table for raw and analyzed data and it keeps data for not long enough like pinger. What Ghulam suggested is that there should be two main tables raw and data separately. Raw will have node id, ping sequence no, rrts , packet sent, packet received similar to what is currently in raw flat files. It will then have multiple tables to store either yearly or 6months data in each table. Similarly for analyzed data there would multiple hourly tables ,multiple monthly table and so on.Each table will be similar to analyzed data as in pinger a flat files. In such a case it will be different from PerfSonar where everything is in one table and if we place everything what is in raw table into analyzed table, there will replicate date(like  4 columns min/max/avg rtts and  seqno) however it will be similar to PerfSonar.

The other query has to do with the  timestamp. The timestamp is unique key in pingerDB assigned by Ghulam. So e.g

pinger.slac.stanford.edu 134.79.240.30 www.eldjazair.net.dz 193.194.64.71 100 1208217601 10 10 217.063 218.753 223.276 0 1 2 3 4 5 6 7 8 9 219 219 217 218 218 218 217 223 217 218 
pinger.slac.stanford.edu 134.79.240.30 www.eldjazair.net.dz 193.194.64.71 1000 1208217611 10 10 219.238 221.357 227.909 0 1 2 3 4 5 6 7 8 9 220 220 221 221 220 227 219 222 219 219

For above two record, method would be 1  and timestamp  as 1208217601 for both .Database wont allow to add two duplicate records. However its not duplicate, its for two different packet sizes. What Ghulam suggested is to add another column as data_id  with auto increment property. It will give each record a unique number. What Sadia believes could be done is to  make packet size also unique key.

The last work done by Ghulam was to save raw pings like old pinger schema meaning concatenating pings for one hour. However we had decided , it would be storing all 48 pings half hour pings/day. So this is also left to be done.Sadia will prepare documentation and upload all information provided by Ghulam.  

Future concerns:(Will be considered once  the performance of above monthly aggregated data is observed) We await a working version before this can start.

1. How to store raw data for one year
2. How should it be sharded
3. For how long data should be in database
4. Ghulam thinks it will speed up our work if we remove the unused columns (metrics) of raw data from the pingtable. Only those fields will be left that are in perfsonar.

Ghulam will be submitting a document on what he did and what needs to be done in his work. Sadia will then review it.
Sadia proposed to hire another student tro complete the work of Bilal and Ghulam. Anjum says, after June he can choose an undergraduate student who can take this as his FYP project. 

TULIP - Bilal

BS. Asia as a whole looks bad, Pakistan looks good (i.e. CBG is way better than GeoIP). 

For South Asia, quest.seecs.edu.pk has high RTT and a distance of 816km for Islamabad nodes which is because the node is in Nawabshah Karachi but has the DNS entry of SEECS. Similarly, sbkwu.seecs.edu.pk has high error and large distance because the node is in Quetta but DNS entry is of SEECS. As Anjum can see, all the nodes that are showing bad results are the ones that have unstable behavior (i.e either they are unreachable most of the time or they have high RTTs).

Bilal needs to add the number of landmarks in the region for his Excel Spreadsheest. This will be useful for his paper, i.e. reporting typical accuracy as a function of landmarks in region.

Another interesting study would be whether using different alphas (in distance[km]=alpha*min_RTT[ms]*100[km/ms]) based on the alphas found in PingER for the various regions (see for examplehttp://www-wanmon.slac.stanford.edu/cgi-wrap/pingtable.pl?file=alpha&by=by-node&size=100&tick=daily&year=2012&month=03&from=United+States&to=United+States&ex=none&only=all&dataset=new&percentage=any) provides much benefit compared to the single current value of alpha. To facilitate this we have added PingER groups for N.AMERICA, EUROPE, AUSTRALASIA, S.ASIA, S.AMERICA.

Bilal will be sending the tulip draft paper before he leaves the job. 

Bilal will be submitting a document on what he did and what needs to be done in his work. Sadia will then review it.

Bilal will rerun the stress testing of North America using new reflex. He has submitted it*. Did this happen or is it dead?*

Possible projects

• There can be a paper about Pinger if we could just find the right conference. MCN, ICC and Globecomm do provide network monitoring topics. It could talk of the various metrics and their importance (in particular; MOS, Alpha, max RTT, min RTT), the lessons learnt from running such a worldwide infrastructure, the uses of the data etc.
• We can talk of GEO-Location experiences. For example within Pakistan it works fine, however as we go within regions or continents this gets worse. We can publish some stats on that for example. We can add the impact of changing alpha. We can also indicate the importance of landmark proximity. 
• See [https://confluence.slac.stanford.edu/display/IEPM/Future+Projects].
• Extend the NODEDETAILS data base to allow entry support for whether the host is currenty pingable. 
• Improve the PingER2 installation procedures to make it more robust. This might be something for the person(s) in Pakistan who are responsible for installing PingER2 at the Pakistani monitoring sites. They probably have found where the failures occurs. Also look at the FAQ, and ping_data.pl which has been improved to assist in debugging, could it be further improved (e.g. provide access to the httpd.conf file so one can see if it properly configured)? There are 2 students working on the PingER archive. Is this something they could work on?
•  [Fix PingER archiving/analysis package to be IPv6 conformant|IEPM:Make PingER IPV6 compliant]. Will build a proposal for an IPv6 testbed. They will try various transition techniques. A proposal has been prepared and that has been submitted to PTA. Adnan is a co PI. It is being evaluated today.  A small testbed has been established in SEECS and the plan to shift some of the network to IPv6. Bilal is part of 3 students involved with PingER and they will be involved with IPv6. They are porting the PingER archive site site to using a database. They have redeveloped the archive site using Umar's documentation. They have set up a small test archive site. They have gathering, archiving, analysis. They will design a new database. They will also try a port of PingER to IPv6.
• Look at RRD event detection based on thresholds and how to extend, maybe adding plateau algorithm. Umar's algorithm did  not work in a predictable manner. 
• Provide near realtime plots of current pinger data using getdata_all.pl/wget. It will work as a CGI script with a form to select the host, the ping size, and the time frame to plot. It will use wget or getdata_all.pl to get the relevant data and possibly RRD/smokeping to display the data.

Future meeting time - Les

Tentatively next meeting is on Tuesday 22nd May, 2012 at 8:00 pm in US and Wednesday 23rd May, 2012 at 8:00am in Pakistan. However, if we don't hear anything / don't get any document from Bilal and Ghulam then next meeting will be on Tuesday 12th June, 2012 at 8pm in US and Wednesday 13th June, 2012 at 8am in Pakistan.