Versions Compared

Old Version 5

changes.mady.by.user Les Cottrell

Saved on

compared with

New Version Current

changes.mady.by.user Les Cottrell

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Time & date 

Wednesday Dec 9th 16th 2015 7:30pm Pacific Standard Time, Thursday Dec 10th 17th 2015  8:30am Pakistan time, Thursday Dec 10th 17th 2015 11:30am  Malaysian time, Thursday Dec10th Dec17th  2015 01:30 am Rio Standard Time.  

Coordinates of team members:

See: http://pinger.unimas.my/pinger/contact.php

Attendees

Invitees:

Hassaan Khaliq?,  Kashif, Raja,  Samad Riaz (SEECS); Johari+, Nara, Adnan Khan (UNIMAS); Abdullah, Badrul, Anjum+,  Ridzuan, Ibrahim+ ? (UM); Hanan,  Saqib + (UTM); Adib, Fatima ? (UUM); Fizi Jalil (MYREN);  Thiago, Les+, Bebo+ - (SLAC).

Note change of time since US no longer on daylight saving (summer) time, and need to accommodate Johari.

...

- Responded but  Unable to attend: 

? Individual emails sent

Actual attendees:

 Anjum, Les

Administration

  • Membership of pinger-my in https://groups.google.com

  • Hassaan reports "I have received revisions on my proposal and these days I am revising my proposal. 

    • IN In the meantime he has also added another student (Anas Abrar) on this project. He is in learning phase and will follow the nodes which are not working. He shall give us an update very soon. 

    • He is  very hopeful that if the proposal is accepted then we can easily have a full time RA for the project.

    • However, Anjum points out it may take a while to get acceptance and funding (e.g. 6 months)

  • NETAPPS2015 1-3 December 2015 was very successful, five papers were presented in the PingER session 

  • 9 students from Amity University near New Delhi in India interested in coming to SLAC 

    • They are busy with exams right now, we are waiting to hear from them about visa etc.

  • Joao a Brazillian student identified by Renan is interested in a 3 month internship at SLAC similar to those completed  by Renan and Thiago.
    • He is gathering the information requested by SLAC
Geolocation Anjum, 

  • This is a proposed project for one of the Indian students.

Android - Bebo

  • Bebo said Topher would be interested in getting a student to port PingER to an Android. Possibly one of the Indian students. Les put out a version of the PingER MA on Github at https://github.com/iepm/ in 2013 as part of coordinating with Google( which did not get very far). It has been dormant since it was put out there. Les is not a github expert or user so is  unclear how complete it is or who can access it.   Can somebody test and if not available let Les know what to do. 

  • Topher will be back in a couple of days. He will set up a shareable Github for PingER files. Bebo and Topher are working on it. Les has provided the source code location. Once we have it then we can share with Amity to check it works.

  • There are some students at UNIMAS who have expressed interest in the illegal logging audio project. This might be tied into Microsoft's Imagine. 

Saqib

Saqib has a student Aqsa Hameed who has been working with Anjum to look at a project to create a Hadoop/Cloudera PingER warehouse to enable easier, more powerful access to PingER historical data.

Saqib has submitted a project with title "A Fundamental Active Internet Performance Monitoring Framework for Pakistan Education & Research Network (PERN) in University of Agriculture, Faisalabad"

    • Anjum reports there was a huge crowd from 15 countries, it provided good publicity for PingER. In particular there was interest from Jordan and Sri Lanka and a person in UUM
      • After NETAPPS, Adib received several requests from undergraduate students to do their internship (practicum) on PingER related projects.  Hopefully, at least two of them will get the school approval and they will be placed in UUM InterNetWorks lab under my supervision. The internship will start in Feb and the proposed project is PingER IPv6. In addition, one network engineer from HP was very interested in PingER. After a short discussion with him, he has decided to pursue his master study and work on project related to PingER. I have shared PingER future projects link with him and currently he is waiting for an official letter of admission as a master student. The topic is not yet decided and open to our group to suggest.
    • Unfortunately Anjum and Saqib were unable to link up since Saqib was only able to be present on the last day.
    • Currently, NetApps organizers are  in the process of sending NETAPPS2015 proceedings to be indexed in the major indexing services.

  • 9 students from Amity University near New Delhi in India interested in coming to SLAC. Bebo sent reminder recently, response below:

    • Sorry for the delay.  Students are giving end semester exams.  They are busy till dec 20th. They will get back to work as soon as the exams  are done. Once the exam is over we will be setting up  "PingER measurement agent at Amity"

    • Three students have written a short paper for confluence. 

      • Analysis and clustering PingER network data

      • Application for the emulation of pinger on Android devices

      • Correlation analysis  on Real Time  Tab Delimited  Network monitoring data

    • Btech computer science students   would be graduating in May 2016.  Their visas can be arranged after May 2016.

    • Jahin Majumdar ( Btech computer science student) already has an 5 year F1 student Visa for University of Florida since he will be completing his 8th semester in that university. He will be completing his semester at Florida in May as well

  • Joao a Brazillian student identified by Renan is interested in a 3 month internship at SLAC similar to those completed  by Renan and Thiago.
    • "IIE took some time to send me the information I asked, but now I think I have all I need. They sent me an e-mail two days ago with the informations about academic training and they will send me another e-mail as soon as my Letter of Support is avaiable. The academic training verification form is already avaiable, however they asked the students to wait for the upcoming webinar where they will give instructions about the academic training and how to fill the academic training form. As soon as my form is complete, I will need your signature in it. Also, I´m sending you my health insurance document attached in this e-mail." 
Geolocation Anjum, 

  • This is a proposed project for one of the Indian students.

Android - Bebo 

Bebo has set up a Github codebase  as a new project. It contains the PingER MA (pinger2.pl and the traceroute/ping server).Anyone  needs to sign up for a Github account (if you don't already have one), so you can be added as a project member.

Saqib

  • Saqib has a  5  MS students from the Database team

    • Farah Massoud

    • Sabah Massoumil working on Linked Open Data (please excuse spelling)

    • Aqsa Hameed working on big data/analysis of PingER data

    • Aqsa Hameed has been working with Anjum to look at a project to create a Hadoop/Cloudera PingER warehouse to enable easier, more powerful access to PingER historical data. 

      • Thinking a bit on Aqsa her work appears to be related to the work done by Thiago on PingER warehouse using a cluster/Cloudera/HDFS/Impala earlier this year. In particular see the presentation at NETAPPS2015 (see https://confluence.slac.stanford.edu/download/attachments/123309267/NETAPPS_PRESENTATIONrevLuiza.pptx. Once the paper is published we can also provide her with that (Adib will let us know when this is OK). 

        Further we are working on repeating the interns at SLAC of Renan and Thiago with another Brazilian student Joao this coming year.  Thus maybe we can provide some continuity of effort.

      •  Les will email the relevant people to put them in contact.

  • Funding source will accept the projects if it is designed for the development of Agriculture in the University.
  • Objective: The active internet performance monitoring framework for the University of   Agriculture, Faisalabad helps in forecasting and detecting the events that cause bottleneck links in the network affecting the overall performance of the University of   Agriculture, Faisalabad.
  • Initially the project is accepted. However, I received the following comment
  • The PI was advised to include in the project the justification how it will have an impact on agriculture development in the University of Agriculture Faisalabad Pakistan. 
  • For justification I have added the following paragraph:
  • The aim of this project is to provide the performance analysis and anomaly detection framework for the University of Agriculture, Faisalabad  in order to detect the events that cause bottleneck links in the network affecting the overall performance of the University. Further, It will help to improve the performance of the web applications running in the university.
  • Les added: 

    • Internet connectivity is critical to agriculture, especially since agriculture is usually conducted in rural and possibly remote areas. In particular it enables farmers to ascertain the optimum prices for seeds, fertilizers at, the most promising markets etc for their goods, upcoming weather conditions, forecasting yields, providing information for addressing challenges etc. It also supports timely sharing of information between farmers, their customers, and learning of new relevant developments from research and industrial centers such as the universities. This is very important since the adoption of new technologies has always been an important factor in agricultural productivity. See http://pnwsteep.wsu.edu/directseed/conf2k5/pdf/gillis.pdf

       

      Understanding the Internet performance between the University of Agriculture and other areas of Pakistan will enable setting up: information centers at Faisalabad, outreach to agricultural communities (e.g. by social tools etc.),  by setting realistic expectations for connectivity and hence capabilities to various regions of Pakistan for email, web, chat rooms, data sharing etc.

    • Bebo provided a coupe of links to IOT that may assist:

      http://bits.blogs.nytimes.com/2015/08/03/the-internet-of-things-and-the-future-of-farming/?_r=1

      http://www.liveworx.com/Portals/0/2014%20Presentations/OnFarm_Smart%20Agriculture%20and%20the%20Internet%20of%20Things.pdf

  • Saqib has submitted a project with title "A Fundamental Active Internet Performance Monitoring Framework for Pakistan Education & Research Network (PERN) in University of Agriculture, Faisalabad"

    • Project is accepted.

    • working on a project to develop a node using Raspberry Pi 2 and IoT to measure the air and soil quality.

    • Currently, students are analyzing the project to develop a problem statement for their research project.

    • Saqib has 5 students interested in PingER. In particular in Big Data, Data Mining and Geolocation.

  • For the GeoLocation Saqib should contact Anjum

  • For Big Data and Data Mining I suspect they will need a cloud/cluster. Given the problem at other sites it is important to be assured they will have the appropriate resources to pursue their work.

UUM

UUM

Fatima  presented her work at NetApps. Fatima has passed her master oral examination (Viva) with minor correction and she is submitting her final dissertation coming Sunday to UUM school of computing. Indeed, we are very grateful to PingER team members support and help. Currently, Fatima is planning to pursue her PhD and possibly on a topic related to her master dissertation. We need PingER team suggestion and advice before we can decide on the topic. I will attend next meeting to further discuss on this.Fatima  implemented her work (involving HDFS and MapReduce with Pinger Data) on 50MB to 200MB data sets, using a 2-node cluster, and then a 3-node cluster. The work is not completed, but she has obtained some good results and is putting together the presentation for her final viva. She and Ibrahim are using the same cloud/cluster at MYREN. No update 11/3/2015

UM

Ibrahim has extracted the  PingER Zip manually. He is reconstructing 11Gb of data, heas has 15GBytes of data there. He is trying to use SPAR to classify the data. He alsop was looking at RDF. the next step si to use MapReduce to organize and reduce the output of dataso can vizualize it. He will be using a the same techniques he used for looking at 1996-2006 weather data.  However at the moment he cannot access his  VMs from myren for the last two weeks, even for the myren site, He has emailed  them but till now they have not fixed the issue.  He has updated all his data in their cloud. 

Anjum is working on a paper to submit to IEEE Internet Computing on the topic of 'Measuring the Internet’. Due date extended to Nov 13, 2015

UNIMAS

CUstom ISO for both Ubuntu and Raspbian Wheezy is in progress, student is familiarizing himself with the tools and steps to install Pinger on both platform before creating the custom iso. It is in progress as of 11/3/2015.

UTM

UNIMAS

Custom ISO still in progress, unable to release it due to bug that causes kernel panic error when booting from customized iso

UTM

Johari has had no response from UTM so far. Not sure whether they still want to be in the project. Last straw, need to contact someone from the network group so that they understand and appreciate the Pinger projectJohari will contact Hanan to request someone to support PingER at UTM, now Saqib has left. Johari will try dean to get a replacement for Saqib. However the Dean is new so Johari wants to gather some information first.

Saqib reports:

"Faculty management in the Faculty of Computing-UTM is changed. They ask me to re-locate the PingER server from the faculty server room as students are not allowed for physical access of the server room. Only faculty members are allowed to physically manage the servers in the server room.  Therefore, I asked my supervisor (Dr Md Asri Ngadi/ Prof Hanan) from UTM to take  the responsibility of the server. He is agreed, however, I have to manage the server remotely. Further, he needs an official invitation from your side to fulfill the official requirements. Further, he wants to make this project as a source of an international collaboration and linkages between UTM and SLAC-Stanford."

...

The students are expected to comply with all policies and procedures, including safety guidelines.  Your mentor will provide them with any specific policies, procedures or safety guidelines for their assigned area. As a program participant, the student does not  enter an employee/employer relationship with DOE, SLAC or any other federal agency.

MYREN

No update 8/12/2015, no Update 9/2/2015. 

...

We have been unable to gather data from pingersonar-utm.myren.net.my  since September 11th, 2015 pingersonar-utm.myren.net.my. It is pingable  however there is no response from http://pingersonar-utm.myren.net.my/cgi-bin/traceroute.pl?function=ping&target=www-wanmon.slac.stanford.edu&options=-i%200.2

NUST

Hassaan reports "I have received revisions on my proposal and these days I am revising my proposal. In the meanwhile, I have also added another student (Anas Abrar) on this project. He is in learning phase and will follow the nodes which are not working. I shall give you an update very soon. "

  •  I am very hopeful that if the proposal is accepted then we can easily have a full time RA for the projectaccepted then we can easily have a full time RA for the project.

Hassaan has re-submitted the proposal after revisions. He would like to get Anas Abrar more trained on monitoring operation and then will inform us to add him to the mailing list at http://pinger.unimas.my/pinger/contact.php.

PingER at SLAC

Bebo is looking at setting up a Github with the PingER monitoring code and traceoute.pl.

...

HostStatelast seenStatus
web.hepgrid.uerj.eduemails 12/2/2014, 12/8/2014, 2/26/2015, 4/30/2015, 6/1/2015Oct 23, 2014traceroute.pl works but no response from ping_data.pl
pinger.stanford.eduemail 3/14/2015Feb 18, 2015Works
pinger.unesp.bremail 11/28/2014, 5/22/2015, 6/1/2015.Nov 3, 2014Host is pingable from SLAC.

Next Meeting

Next meeting:  Wednesday Dec  2nd 7:30pm Jan 6th 8:00pm Pacific Standard Time, Thursday Dec 3rd 2015  8:30am Jan 7th, 2016  9:00am Pakistan time, Thursday Dec 3rd  2015 11:30am  Malaysian Jan 7th 2016 12:00noon Malaysian time, Thursday Dec 3rd 2015 01:30 am Jan 7th 2016 02:00am Rio Standard Time.  

Old Items

  • Oct 2015. Following the last meeting, Anjum, Hassaan and Les met to discuss the way forward. 

    • "Adnan currently is unable to find resources for handling the project. Similarly, there is no progress on hiring of a full time RA by NUST HQ. 
    • However, I (Hassaan) checked from HEC about the proposal that I submitted last year. They have informed me that 2 reviewers have asked for revisions while they are waiting for the third review. I am very hopeful hopefull about it. If the proposal is accepted then we can easily have a full time RA for the project. I have plans to talk to Dr. Zaidi about hiring an RA on assuming that our proposal will be accepted by HEC. We can then get his salary deducted later from the HEC project. I shall update you very soon in this regard." 

    • Hassaan is  waiting to hear from HEC about the comments on the proposal. 

    • Moreover, he has asked a student to work on the project for the time being. His name is Mian Anas however he will need few weeks to understand the project. 

  • Thiago completed setting up the  PingER data SQL Impala warehouse running on a Nebula/Cloudera cluster using the Hadoop File System (HDFS). Unfortunately it is not currently accessible from outside SLAC. There have been several attempts to provide outside access, but no success yet, we need to engage the subject matter experts. Thiago is now a SLAC associate so he still has an account at SLAC. There was a cyber security alert on the version of java installed with Cloudera. Les has replaced the cloudera version of java which should fix the vulnerability. However the new version has not been tested.

Geolocation

Anjum believes the TULIP Geolocation application  can be improved significantly. At least there are few ideas that we can try. For this, either a group of undergraduate students or an active masters student is required. The resultant work can easily be the thesis of masters level. Who is interested? 

  • Saqib at Faisalabad has an MS student interested to work on Geolocation project. He requests an initial  paper on the project.  Les has responded to Saqib. He also has some other students. Anjum will contact him. Potential projects/asks include: take over management of PingER monitoring in Pakistan (say 5 monitors/student; case study of how Pakistan's network performance/connectivity has improved over thea years especially as function of funding etc;  geolocation with variable alpha; indoor geolocation

  • Johari will contact Anjum to learn more of the requirements. Update Johari/Adnan

  • See http://www.slac.stanford.edu/comp/net/tulip/. Basically TULIP uses pings to a target from landmarks at known locations and converts the minimum RTTs to estimate the distances. Then uses the distances with mulitlateration to estimate the location of the target

  • To improve TULIP one needs the right selection of landmarks, i.e. good (working landmarks) at the right locations (not too far from the target), straddling the target, and with a a reasonable estimate of the indirectness (directivity or alpha) of the path from the landmark to the target (so we can reasonably accurately estimate the distance). One also needs a reasonable density of landmarks (e.g. number of targets/100,000sq km)

  • The landmarks come from PingER and perfSONAR sites.  We have a reasonable density in the US, Pakistan and Europe. Currently Anjum is getting better than 20km accuracy for Pakistani targets

  • As the number of landmarks goes up so does the accuracy, but so does the time to make the measurements (pings). 

  • One needs to find the optimal density

  • Anjum proposes to speed up the measurements using a cluster for parallelization and also proposes to improve the adaptation of alpha based region. He regards the adaptive geolocation and parallelization as  MS projects. 

  • He is also interested in geolocation in small proximity (e.g.indoors), e.g. using cell tower signals. This is a new area of research. It is possible that the port of PingER to an Android could  be related to this. This is a PhD project

  • Anjum reports he can supervise the students on Geolocation. He will need to know when the students are ready. We can start with a joint meeting involving Les and the students. Later on, Anjum can have the meeting with students every week while Les can join if he has time.

NUST/SEECS Pakistani PingER nodes status

Pink Background indicates host was bad last month, strike through says it is fixed, yellow is an new bad host.

...

Is it time to start paring down the list of PingER monitor hosts in Pakistan, starting with those that have been down for a while and despite your efforts they are not cooperating.  One might also look at the coverage by region in Pakistan and try and keep good coverage for all regions.

Traceroute at UTM 5/9/2015

The traceroute problem regarding maximum reachable hops ( i.e. 11 hopes ) may be since the Unix/Linux/OSX  traceroute uses UDP to send the requests. The first request is sent to a particular port (33434), with a ttl  to tell it how many hops to go to.  The ttl starts at 1 is incremented as it tries the next hop, also the port is incremented (up to 33465).  It looks like the first few UDP ports are enabled and then they are blocked. The Windows traceroute uses ICMP to send the probes so does not see the problem.

Linked Open Data

Cristiane reports (7/1/2015): "I am trying to automatize the triplification of PingER data on Kettle. For now, part of the transformation is made on Kettle and another is made by a Java code. Although this solution works for a data sample, is important to have the entire process on Kettle because it facilitates to understand, modify and control the triplification process."

...

Christiane's report is at: Size Inflation of PingER Data for use in PingER LOD

UM

Moved here 3/4/2015:

Ibrahim has setup distributed hadoop clusters. He has 2TB of disk space. Les has provided information on getting a subset of PingER data by anonymous ftp via ftp://ftp.slac.stanford.edu/users/cottrell.  It was put there last September. Information on how the data was put together is at https://confluence.slac.stanford.edu/display/IEPM/Archiving+PingER+data+by+tar+for+retrieval+by+anonymous+ftp. There is information on formatting etc at http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html and some on the dataflows at https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC. Renan at UFRJ has successfully used this data, he has also characterized the data in terms of bytes/metric per year etc.

...

Anjum reported that UM had experienced a TCP syn DOS attack prior to Mar 12th (when an IDS was put in place). It occurred mainly for several days before between the hours on noon- 2pm and 7-7 in the evening (Malaysia time). He suggested looking to see if PingER could spit the effect.  Ibrahim, Les and Anjum will look at. Les analyzed the data and sent it to Anjum

NUST

The following is from Samad 2/24/2015.

Follow up from workshop
  • Hossein Javedani of UTM is interested in anomalous event detection with PingER data. Information on this is available at https://confluence.slac.stanford.edu/display/IEPM/Event+Detection. We have sent him a couple of papers and how to access the PingER data. Hossein and Badrul have been put in contact. Is there an update Badrul?

...

Anjum suggested Saqib,  Badrul and Johari put together a paper on user experiences with using the Internet in Malaysia as seen from Malaysian universities. In particular round trip time, losses, jitter, reliability, routing/peering, in particular anomalies, and the impact on VoIP, throughput etc.  It would be good to engage someone from MYREN.

Ibrahim

Ibrahim Abaker  is planning to work on a topic initially entitled " leveraging pingER big data with a modified pingtable for event-correlation and clustering".  Ibrahim has a proposal, see https://confluence.slac.stanford.edu/download/attachments/17162/leveraging+pingER+big+data+with+a+modified+pingtable+for+event-correlation+and+clustering.docx. Ibrahim reports 7/15/2014 "I have spent the last few months trying to understand the concept of big data storage and its retrieval as well as the traditional approach of storing RDF data. I have integrated a single hadoop cluster in our cloud. but for this project we need multiple clusters, which I have already discussed with Dr. Badrul and he will provide me with big storage for the experiment." No Update 8/20/2014.

"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"

Potential projects

See list of Projects