Time & date 

Wednesday Mar 4th 2015 8:00pm Pacific Standard Time, Thursday Mar 5th  2015  9:00am Pakistan time, Thursday Mar  5th 2015 noon Malaysian time, Thursday  Mar 5th, 2015 02:00am Rio Standard Time.  

Attendees

Invitees:

Anjum+ (may be couple on mins late), Hassaan Khaliq, Kashif, Raja,  Samad Riaz+, Johari+, Nara, Adnan Khan+, Abdullah, Badrul, Ridzuan, Ibrahim+, Hanan, Saqib+, Adib + (15 mins late), Fatima+, Les+, Renan, Bebo+ (few minutes late)

+ Confirmed attendance

- Responded but  Unable to attend: 

Actual attendees:

Anjum, Samad, Johari, Adnan, Ibrahim. Saqib, Les.

Les was late (35mins) in starting the meeting and apologized, this may have caused some people to give up. Adib and Fatima sent emails after the meeting explaining their absence.  

Administration

  • Membership of pinger-my in https://groups.google.com
  • Now that Saqib has left UTM he will continue to take care of the project at UTM.
  • Fatima Binta Adamu of UUM has been added to the list. She is putting together a master's proposal to be submitted in March.

  • Starting next week, Anjum will complete this paper on Geolocation by measuring round trip tiems. He has contacted Raja Asad requesting Raja to  share the code with Anjum and spare some time to explain the functionality and how Raja  takes results. Raja has provided the scripts and data etc.

  • Johari has got the OK from the conference organizing committee to hold a colocated PingER/BigData workshop on August 3rd the day before the  CITA 2015 (see http://www.cita.my/ an International Conference 4th - 6th August 2015, on transforming Big Data into Knowledge. Johari will provide relevant information to Bebo. Bebo will be able to make a presentation. Les has sent Bebo some relevant slide decks. Johari is awaiting an abstract from Bebo. Also Ridzuan or Ibrahim or Renan have interest in submitting a full paper by April 2nd 2015. 

UFRJ

Maria Luiza reports:

' I had a meeting with the students last week, trying to get some feedback just after they had returned from the Carnival holidays. We have still not solved the infrastructure problem (it does not depend on us, as we have the money but not the autonomy to hire someone to work on the network and electrical installations), but we are now trying to move to a different location.

I asked them for a specific report on some of the issues you mentioned and we should get back with this to you till Tuesday.

I thought we could make an offer to students that are already in the US with a Science without Border grant, for a period at SLAC working on one of the current solutions, just like Renan did. I asked Renan if there is a list where we could try to make such announcement, but I am also trying to contact the person in charge for UFRJ Computing students abroad. Of course, we cannot guarantee it will be another dedicated "Renan", but we would try to select a student with some background on the subject.'

Les has requested (in January) Renan to provide an estimate of how DF bloats the data. Renan/Christiane are looking at this. Renan's pointed out "Finding RDF data size in bytes is not simple because it depends on which Triple Store will be used and how each triple is physically stored. One may store triples as plain texts, other may do as compressed data in specific formats, which would be much smaller."Once we have the number of PingER triples and how much the used Triple Store needs (in bytes) to store a known number of general triples, we may estimate PingER RDF data size.". Requested an update by email to Cristiane & Renan 2/4/2015.

UUM

pinger.uum.edu.my is down Jan 31st, Feb 1st. Adib reports it is currently down disconnected due to a technical problem at UUM computer centre. Adib is trying to get more reliable power (UPS). It is still down March 6th. Adib was not at the meeting so no update.

Adib has been discussing with Anjum looking at potential PingER  projects. Adib has a master student. In particular they are interested in providing more flexible access to PingER data rather than the limited time windows pingtable.pl provides. This may be by providing database access rather than using flat files. Les will provide:

  • Several emails exchanged between Les and the student at UUM (Fatima Binta Adamu) to clarify access and the format of the PingER data. I think it is clear now. Documentation has been updated.

UM

Ibrahim is conducting experiments on using Big data with PingER data and writing a conference paper. he plans to complete the middle of this month. He will send a copy for review before submitting. He now has a cloud of 4 computers at UM running Hadoop and 15 at MYREN. The installs are done, he has downloaded the PingER data from the SLAC ftp site and is about to start on the mining.

No word from Ridzuan.

UNIMAS

Johari had no updates 2/4/2015, 3/4/2015. His top priorities are:

  1. Reviving the Raspbery Pi. Johari went to the computer center but could not figure out what was wrong with the raspberry Pi. It may be a firewall issue or he may need to replace the SD card.
  2. Getting the research student going on anomalous behaviour detection methods. He is working on a Conference paper.

Johari still has to uncover the problem of the traceroute from UNIMAS. UDP has been unblocked. The MYREN  host works fine and share most of the hops. Thus the problem must be in the first few hops.

From previous meetings

The two major issues with the Raspberry Pi would be:

  • are the results statistically the same as for the other monitor at UNIMAS (e.g. use the Kolmogorov-Smirnov test); There is Advanced Project (Master by coursework student) working on the statistics of the data from the raspberry Pi and the production PingER monitor at UNIMAS to see how much they differ.
  • is it reliable/robust is it clear what to do to debug problems remotely (e.g. if it is at Bario).  Looking at the monitoring data I have been unable to collect any from it (it is pingable, and port 80 responds, however the remote traceroute and ping_data.pl are not working) since Oct 20th which does not sound promising. Will need to evaluate the robustness of the unit by doing simulated scenario of various events such as power failure, hard and cold reboot, etc. Johari will need access to computer center to verify it comes up correctly after reboot etc.
  • Johari will go to the computer center the coming weekend and look at improving the auto re-start.

If/when it works it would be instructive to look at the data from pinger and raspberry pi to Malaysia since the distances are shorter and the differences may show up better. For Sep-Oct 2014 when there was data measured from both Oct-Nov the averages for 20 paths was 52+-21ms (from pinger.unimas.my to 20 other Malaysian hosts) and 56+-21ms for raspberry pi to 20 other Malaysian hosts.

Custom iso: He can get as far as the boot screen, but is unable to get to the desktop. It is on hold as of 1/7/2015 awaiting  student with the appropriate skills/background.

They are also looking at anomaly detection:  http://slac.stanford.edu/pubs/slacpubs/13250/slac-pub-13399.pdf or http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.363.1087 for comparisons of some techniques and http://people.cs.missouri.edu/~calyamp/publications/ontimedetect_mascots10.pdf. Next they will look at performance among correlated routes. There are quite a lot of papers in this are so a literature search is highly recommended.

UTM

Saqib has updated the case study and is available in Google drive as a "Shared-PingER" document for review at https://drive.google.com/folderview?id=0B-NEKleLll79ZFNmUnhiVGJ0Nmc&usp=sharing_eid 

The traceroute problem regarding maximum reachable hops ( i.e. 11 hopes ) may be since the Unix/Linux/OSX  traceroute uses UDP to send the requests. The first request is sent to a particular port (33434), with a ttl  to tell it how many hops to go to.  The ttl starts at 1 is incremented as it tries the next hop, also the port is incremented (up to 33465).  It looks like the first few UDP ports are enabled and then they are blocked. The Windows traceroute uses ICMP to send the probes so does not see the problem..

MYREN

Fizi Jalil was not at the meeting.

Fizi Jalil has set up 3 new Pinger node at Universiti Sains Malaysia (USM), Universiti Malaya (UM) and Universiti Teknologi Malaysia (UTM). There is one left to do at Universiti Malaysia Pahang (UMP) because the server is inaccessible now. MYREN has servers  at only 5 universities which and these are theit our point-of-presence (PoP) in each region. There are now 5 PingER nodesi n MYREN. Traceroute/ping and ping_data.pl are working in all of them.

  • A goal would be to compare and contrast the benefits of perfSONAR and PingER.

NUST

pinger.uob.edu.pk appears to be partially working according to http://www-iepm.slac.stanford.edu/monitoring/checkdata/, however it appears to be unreachable at regular intervals. Smada reports this is due to power outages:

The numl host is also very unreliable which could be power related. It also loses its DNS name which may be something else.

The following is from Samad 2/24/2015.

  • The IP of "pingerqta.pern.edu.pk" has been changed, please update the database with the following new IP.Les has updated the database at SLAC.

  • buitms.seecs.edu.pk #We have to disable gathering data from this host because the person still don't want to continue with us as i have tried once again to convince him but the answer is same. Les had deleted.

  • pinger.isra.edu.pk # This node is also working fine now.

  • pinger.uet.edu.pk # this was also not working from so many days. and now its working fine and collecting data as well. 

  • pingerjms.pern.edu.pk #This node is working now.

  • sau.seecs.edu.pk. #This Node is working fine now.

  • pinger.uettaxila.edu.pk #The node is working fine from last two weeks.

  • pingerlhr-pu.pern.edu.pk # This node is working as of  2/24/2015

  • nukhimain.seecs.edu.pk # We were unable to gather data since 20th November, 2014 and now the Node is working fine and collecting data as well.

  • pinger.kohat.edu.pk #Fixed 2/25/2015.

He is working on:

At the meeting Samad says he has 5 hosts that are not working. 3 are nearby so he can visit. 2 are far away and so more challenging. He will try and fix the 3 in the coming week.

Anjum proposed to Samad that he contact Dr Adnan Iqbal at the naml node in Bolochistan to get PingER installed there. After the meeting email was sent to Adnan who is happy to do this, but will need to get upper management approval.

For the latest see: http://www-iepm.slac.stanford.edu/monitoring/checkdata/Jan.htm

Pinger at SLAC

Working on the following hosts to be able to gather data:

HostStatelast seentraceroute.pl
brunsvigia.tenet.ac.zaFixed 2/26/2015 needed powering upJan 27, 2015Was down
hunnas.learn.ac.lkemailed 2/26/2015Nov 13, 2014Works
multivac.sdsc.eduemailed 2/26/2015, 3/6/2015.Jan 20, 2015Works
web.hepgrid.uerj.eduemails 12/2/2014, 12/8/2014, 2/26/2015Oct 23, 2014Works
www.umss.edu.boemails 8/30,2014, 9/12/2014, 11/27/2014, 2/27/2015Jul 6, 2014No response

The virtual machine pinger2.pl measurement agent at SLAC is now running successfully and recovers correctkly after a power outage. See http://www-wanmon.slac.stanford.edu/cgi-wrap/pingtable.pl?file=average_rtt&by=by-node&size=100&tick=daily&from=EDU.SLAC.STANFORD.N3&to=EDU.SLAC.PINGERVM&ex=none&dataset=hep&percentage=any for example.

The analysis of 6 days of pinging by PibgER shows there is little statistical significant difference (the distributions are within a standard deviation assuming a normal distribution). However for short RTTs (between a physical and a virtual machine in the same room) there is a difference e.g. 2.0ms from virtual to physical and 2.06ms from physical to virtual. See PingER VM Comparative analysis of significant statistical difference with non VM. We are investigating with VM developer (Nebular). The next step is to move from a floating IP address to a fixed one for the VM.

Completed putting together annual report for PingER. It is available at: http://www-iepm.slac.stanford.edu/pinger under reports pull down or more directly at: http://www.slac.stanford.edu/xorg/icfa/icfa-net-paper-jan15/report-jan15.docx. It  has been presented to the International Committee on Future Accelerators (ICFA).

Updated report on Duplicate pings as seen by mining the PingER data, also fixed the analysis (hence the data in pingtable.pl)  to detect duplicate pings (it was missing most of them).

Bebo arranged a meeting with the Colombia RENATA NREN folks and the minister of IT to discuss the use of PingER in Colombia. There is a web page at: Colombia. Les has sent an email asking them to install pinger2.pl at at least one site in Columbia. Sent a reminder email 2/27/2015.

Next meeting

Next meeting:  Wednesday Apr 8th 2015 8:00pm Pacific Standard Time, Thursday Apr 9th  2015  9:00am Pakistan time, Thursday Apr 9th 2015 noon Malaysian time, Thursday  Apr 9th, 2015 02:00am Rio Standard Time.  

Old Items

Linked Open Data

Feb 2015

The plan is still the one seen before (see project proposal), experimenting those alternatives. Right now, they managed to triplify the data according to a new ontology that takes advantage of a combination of a current standard for multidimensional data (called data cube vocabulary) and a revised version of Renan's Moment ontology adaptation. With this we expect to have a better data organization than the previous solution.

They are now preparing a test plan (like a small benchmark) to be used on all alternatives so that we can compare the results accordingly. 

Aug 2014

Renan  finished the new pingerlod web site. The new thing is that it should be much easier now to modify the info texts. What Renan did was to put the texts into a separate file. The new version has been loaded on the server and some text added to describe how to use the map. However there is a bug that prevents it from executing the map. Renan reports that the bugs should be easy to fix. He has talked to his professor who suggested trying RDF Owlink, it should have faster responses to queries. Renan will research this.  It will probably mean reloading the PingER data so is a lot of work, hopefully this will improve performance. Before the rebuild he will make the fixes and provide a new WAR for us to load on pingerlod.slac.stanford.edu. He is also working on documentation (he has finished the ontology and has a nice interactive tool for visualizing it, since the ontology is the core of the data model of our semantic solution, this will be very helpful for anyone who uses our system, both a developer of the system and a possible user) and his thesis. Bebo pointed out that to get publicity and for people to know about the data, we will need to add pingerlod to lod.org.

Things he will soon do regarding documentation:

  1. A task/process flow writing all java classes involved on all those batch jobs;
  2. A Javadoc <http://www.oracle.com/technetwork/java/javase/documentation/index-jsp-135444.html> which will explain all classes and how they are used.

For the Linked Open Data / RDF which is in pre-alpha days, you can go to http://pingerlod.slac.stanford.edu. As can be seen this page is not ready for prime time. However the demos work as long as one carefully elects what to look at:

  • Click on Visualizations, there are two choices:
    • Multiple Network Metrics: Click on the image: gives a form, choose from Node pinger.slac.stanford.edu pinging to www.ihep.ac.cn, time parameters yearly, 2006 2012, metrics throughput, Average RTT Packet loss and display format Plot graph, then click on submit. In a few seconds time series graph should come up. Mouse over to see details of values at each x value (year).
    • A mashup of network metrics x university metrics Click on image: gives another form, pinging from pinger.slac.stanford.edu, School metric number of students, time metric years 2006 2012, display format plot graph, click on submit. Longer wait, after about 35 seconds a google map should show up. Click on "Click for help." Area of dots = number of students, darkness of dots = throughput (lighter is better), inscribing circle color gives university type (public, private etc.) Click on circle for information on university etc.
  • Renan will be working on providing documentation on the programs, in particular the install guide for the repository and web site etc. This will assist the person who takes this over. 

Renan is using OWLIM as RDF Repository. He is using an evaluation version right now. Renan looked into the price for OWLIM (that excellent RDF Database Management System he told us about). It would cost 1200EUR minimum  (~ 1620 USD, according to Google's rate for today) for a one time eternal license. It seems too expensive. No wonder it is so good. Anyhow, he heard about a different free alternative. Just not sure how good it would be for our PingER data. He will try it out and evaluate. He will also get a new evaluation of the free OWLIM lite.  

He has also made some modifications on the ontology of the project (under supervision of his professor in Rio) hence he  will have to modify the code to load the data accordingly.

Maria and Renan are advancing in some approaches to deal with PingER data, making it easier to be analyzed and integrated. In particular they have been busy studying and evaluating alternatives, analyzing results from the latest benchmarks on NoSQL (including RDF and graph based storages) database management, distributed processing and mediated  solutions over relational databases, and also other experiments with multidimensional analyses on Linked Data.  The new students involved are now understanding better the scenario and they have been interacting with Renan regularly. 

UM

Moved here 3/4/2015:

Ibrahim has setup distributed hadoop clusters. He has 2TB of disk space. Les has provided information on getting a subset of PingER data by anonymous ftp via ftp://ftp.slac.stanford.edu/users/cottrell.  It was put there last September. Information on how the data was put together is at https://confluence.slac.stanford.edu/display/IEPM/Archiving+PingER+data+by+tar+for+retrieval+by+anonymous+ftp. There is information on formatting etc at http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html and some on the dataflows at https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC. Renan at UFRJ has successfully used this data, he has also characterized the data in terms of bytes/metric per year etc.

Ibrahim has started downloading all zip files in the local machines. 6 weeks ago he downloaded 2 GB of Weather data to test his nodes cluster, he  wrote a simple Java program (Map, Reduce) to find the Average and it was working fine. 

NUST

The following is from Samad 2/24/2015.

Tulip
Follow up from workshop
  • Hossein Javedani of UTM is interested in anomalous event detection with PingER data. Information on this is available at https://confluence.slac.stanford.edu/display/IEPM/Event+Detection. We have sent him a couple of papers and how to access the PingER data. Hossein and Badrul have been put in contact. Is there an update Badrul?

The Next step in funding is to go for bigger research funding, such as LRGS or eScience. Such proposals must lead to publications in high quality journals. They will need an infrastructure such as the one we are building. We can use the upcoming workshop (1 specific session) to brainstorm and come up with such proposal. We need to do some groundwork before that as well. Johari will take the lead in putting together 1/2 page descriptions of the potential research projects. 

  1. Need to identify a few key areas of research related to PingER Malaysia Initiative and this can be shared/publicized through the website. These might include using the infrastructure and data for: anomaly detection; correlation of performance across multiple routes; and for GeoLocation. Future projects as Les listed in Confluence herehttps://confluence.slac.stanford.edu/display/IEPM/Future+Projects can also be a good start and also Bebo's suggestion. 
  2. Need to synchronize and share research proposals so as not to duplicate research works. how to share? Maybe not through the website, or maybe can create a member only section of the website to share sensitive data such as research proposal?

Anjum suggested Saqib,  Badrul and Johari put together a paper on user experiences with using the Internet in Malaysia as seen from Malaysian universities. In particular round trip time, losses, jitter, reliability, routing/peering, in particular anomalies, and the impact on VoIP, throughput etc.  It would be good to engage someone from MYREN.

Ibrahim

Ibrahim Abaker  is planning to work on a topic initially entitled " leveraging pingER big data with a modified pingtable for event-correlation and clustering".  Ibrahim has a proposal, see https://confluence.slac.stanford.edu/download/attachments/17162/leveraging+pingER+big+data+with+a+modified+pingtable+for+event-correlation+and+clustering.docx. Ibrahim reports 7/15/2014 "I have spent the last few months trying to understand the concept of big data storage and its retrieval as well as the traditional approach of storing RDF data. I have integrated a single hadoop cluster in our cloud. but for this project we need multiple clusters, which I have already discussed with Dr. Badrul and he will provide me with big storage for the experiment." No Update 8/20/2014.

"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"

Potential projects

See list of Projects

Coordinates of team members:

See: http://pinger.unimas.my/pinger/contact.php

  • No labels