Versions Compared

Old Version 1

changes.mady.by.user Les Cottrell

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (jlrulff)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

 

Time & date 

Wednesday April 6  9May 18th 9:00pm Pacific Standard Time, Thursday Apr 7May 19th, 2016  9:00am Pakistan time, Thursday April 7 2016 May 19th  2016 12:00 noon 00noon Malaysian time, Thursday  Apr 7 May 19th 2016 02:00am Rio Standard Time.  

Coordinates of team members:

See: http://pinger.unimas.my/pinger/contact.php

Attendees

Invitees:

Hassaan Khaliq  Khaliq?  (SEECS)?; Saqib+, Aqsa (UAF); Johari-, Adnan Khan (UNIMAS);  Badrul,  Ridzuan, Ibrahim (UM); ?????? (UTM); Adib-?, Fatima (UUM); Fizi Jalil (MYREN);   Les+, Bebo-, Joao+ (SLAC).

+ Confirmed attendance

- Responded but  Unable to attend: 

? Individual emails sent

Actual attendees:

Saqib, Les

Administration

Geolocation Anjum, 

  • This is a proposed project for one of the Indian students.

Android - Bebo 

  • arrived at SLAC 5/16/2016.  Here until end August 2016.

Android - Bebo 

No update 5/18/2016

Bebo has set up a Github codebase  as a new project. It contains the PingER MA (pinger2.pl and the traceroute/ping server). Anyone  needs to sign up for a Github account (if you don't already have one), so you can be added as a project member.

  • slac-pinger/pinger created by topherwhite.  PingER project https://github.com/slac-pinger/pinger
  • Now we have it we can share with Amity to check it works.
    • Bebo reports that someone from Amity requested to be a member of the Android group.

UUM

No update 5/18/2016

From Adib:

  • "Since we're still waiting on an account, at SLAC, I asked the student to explore another idea (which is quite relevant) proposed by Prof. Bebo “Creative visualization of PingER data, including rich interaction;”. He has already started looking at the possible attractive/interactive design, at the same time he is working to improve his programming skills."

  • "On the other hand, I am in the final stage of writing a case study on the Internet performance in ASEAN countries and its implication on the Socio-Economic Indexes. Will try to share with Dr.Les and Prof. Bebo soon to comment and get your advice on the publication possibility of the paper."

UNIMAS

No update 5/18/2016

Student started last month on the ISO.
pinger.unimas.my does not ping. Reported to JoharoJohari.
Adnan has not been assigned any tasks. He is interested in writing papers and will be discussing with Johari on the way forward.

...

Get back to RPi2 at Datacenter no progress 3/12/216

UAF (Saqib)

Name: Aqsa Hameed

Title: visualization of PingER historical data using warehouse

...

There are 4 students:  Aqsa Hameed, Saba Muzamil, Tahseen and Sara Masood. They are all busy this week and last week with exams. We can expect more updates in a couple of weeks time when the exams are done.

  • Aqsa and Saba are working together. Their goal is focusing on visualization of PingER historical data using warehouse.  The idea is to develop a warehouse in

...

  • UAF university and make it publicly available.

...

  •  They are 50-60% done with setting up a Hadoop cluster with 3 nodes, 1 master, 2 slaves. She is currently working on importing the PingER data into hdfs on the  cluster. they have run some Impala queries on the data and are working on visualization
  • Tahseen is working on Handling Missing Data in PingER by using Artificial Intelligence. He has executed some Java queries and looking at parallel scientific workflow management using SciCumulus.
  • Sara Masood is working on a PingER Internet Performance Monitoring Agent On Android Device. There has been a discussion to pin down more closely what is needed, i.e. a PingER Measurement Agent running on an Android making measurements to the Beacons, and the ability to upload the data nightly. We held a  skype group meeting with Topher, Sara, Saqib, Bebo and Les to move forward. See 20160331 Meeting on PingER/Android, also see Discussion on porting PingER to Android. She has what she needs for now.

Project on Internet performance has been accepted, but no funding yet.

UTM 

Saqib's old supervisor  agreed to appoint a master student to take of PingER in UTM. Saqib has emailed  3/9/2016, no progress 4/6/2016, no progress 5/18/2016..

UM

MYREN

pingersonar-utm.myren.net.my is down. Emailed Fitzi 5/12/2016.

NUST

No update 5/18/2016

Hassaan's student (Mian Anas Abrar <anas.abrar@gmail.com>) reports: "Sir haven't disabled any of them which i told earlier and actually can't even get back to them because of their wrong contact numbers or their numbers not responding since then. I have tried contacting Dr.Les earlier  few time as per your instructions but don't know why i wasn't able to email him. rest the status is actually same nodes being down". We believe we have solved the mail problem.

We are unable to gather data from the following Pakistani PingER monitors:

* This hosts was noted at the 3/9/2016 meeting as being down for the last 30 days.

Dr Hassaan Khaliq contacted and said "We shall look into the issues with them" 5/13/2016. 

PingER at SLAC

 Working on the following hosts to be able to gather data

HostStatelast seenStatus
web.hepgrid.uerj.eduWorked from April 26th to May 3rd, then not pingable. Sent email 5/12/2016. Fixed 5/16/2016Apr 26, 2016

does not ping

pinger.arn.dzemail 1/30/2016, 2/22/2016, 4/11/2016 no response. Giving up.Nov 2015Does not ping
pinger.unimas.my email 5/12/2016March 2016Does not ping
pingersonar-utm.myren.net.myemail to Fitzi 5/12/2016 Does not ping

Next Meeting

Next meeting:  Wednesday Jun 22nd 9:00pm Pacific Standard Time, Thursday June 23rd, 2016  9:00am Pakistan time, Thursday Jun 23rd  2016 12:00noon Malaysian time, Thursday June 23rd 2016 02:00am Rio Standard Time.  

Old Items

Visualization ideas for PingER moved here 5/20/2016

 

  • One way of drilling down is via a map of all sites. Faisal Zahid put something together a couple of years ago. It is at http://www.slac.stanford.edu/comp/net/wan-mon/viper/pinger-coverage-gmap.html. It uses Google maps, allows pan and zoom, plus selection of type (measurement agent, beacon, target) of host to display. It also allows one to select a host (Search location),  mouse click on the balloon to find out more about the host, from there click on the sites web site, click on the country or region to get the relevant pingtable.pl information. The Visualize allows one to choose a measurement agent and a country and it draws coloured lines between the sites  to represent the metric choosen. This was very useful but parts of it are broken.  It would be great to fix or replace. It has some documentation.
  •  For plotting time series, Grafana looks very promising
  •  Updating the motion charts, adding newer data etc. would be most useful.  See http://www-iepm.slac.stanford.edu/pinger/pinger-metrics-motion-chart.html and http://www-iepm.slac.stanford.edu/pinger/explorer.html
  •  Enhance the selection Metrics, Ticks (see pingtable.pl)  to allow multiple metrics (min, avg, max RTT, loss, thrpughput, , MOS ..) to be plotted as a time series. 
  •  Enhanced selection remote hosts for selected tick, selected metric selected mutiple targets (pingtable.pl currently only allows one target), e.g. Once you have plot enable fitting the data to timelines (linear, exponential, power series)
  •  Map pf countries the world colored by metric allow selection of metric and for a particular year. with chosen MA,

 

For plotting time series, Grafana looks very promising

Updating the motion charts, adding newer data etc. would be most useful.  See http://www-iepm.slac.stanford.edu/pinger/pinger-metrics-motion-chart.html and http://www-iepm.slac.stanford.edu/pinger/explorer.html

Enhance the selection Metrics, Ticks (see pingtable.pl)  to allow multiple metrics (min, avg, max RTT, loss, thrpughput, , MOS ..) to be plotted as a time series. 

Enhanced selection remote hosts for selected tick, selected metric selected mutiple targets (pingtable.pl currently only allows one target), e.g.

Once you have plot enable fitting the data to timelines (linear, exponential, power series)

Map pf countries the world colored by metric allow selection of metric and for a particular year. with chosen MA, e.g.

 

Name: Sara Masood

Title: PingER Internet Performance Monitoring Agent On Android Device

Status: There has been a discussion to pin down more closely what is needed, i.e. a PingER Measurement Agent running on an Android making measurements to the Beacons, and the ability to upload the data nightly. We held a  skype group meeting with Topher, Sara, Saqib, Bebo and Les to move forward. See 20160331 Meeting on PingER/Android, also see Discussion on porting PingER to Android. She has what she needs for now.

 

Name: Saba Muzamil

Title: She is interested to work on Environment that supports the execution of analytical distributed queries in Hadoop Cluster.

 

Name: Tahseen

Title: Handling Missing Data in PingER by using Artificial Intelligence

Status: He is doing the literature review to find out the best algorithm to handle missing data in PingER.

 

About IoT:

Title: Air Quality measuring network in University of Agriculture Faisalabad.

Status: Now we have two Raspberry Pi-2 and few air quality measuring sensor i.e.; NO, CO, O3. Currently, working on the configuration and programming of the Pi-2 and the sensors.

Project on Internet performance has been accepted, but no funding yet.

UTM 

Saqib's old supervisor  agreed to appoint a master student to take of PingER in UTM. Saqib has emailed  3/9/2016, no progress 4/6/2016.

UM

 

MYREN

perfsonar-unimas.myren.net.my is working again. 

However pingersonar-um.myren.net.my and pingersonar-usm.myren.net.my were not responding to pings. Emali sent to Fitzi 4/5/2016. pingersonar-um.myren.net.my is now working again. pingersonar-usm.myren.net.my is still not. Tomorrow Fitzi will try abd to contact person at USM tomorrow because he noticed that on the MYREN switch to the server the connection port is down.

NUST

Hassaan's student (Mian Anas Abrar <anas.abrar@gmail.com>) reports: "Sir haven't disabled any of them which i told earlier and actually can't even get back to them because of their wrong contact numbers or their numbers not responding since then. I have tried contacting Dr.Les earlier  few time as per your instructions but don't know why i wasn't able to email him. rest the status is actually same nodes being down". We believe we have solved the mail problem.

No responses for over 30 days from

PingER at SLAC

Preparing a presentation for an NREN conference (BDREN) in Dhaka Bangladesh on 18 April. Will present by video.

 Working on the following hosts to be able to gather data

HostStatelast seenStatus
web.hepgrid.uerj.eduemails 5/1/2016, still no response 1/30/2016, email 1/30/2016, 2/23/2016: They decided to move server to a new network. 

They will start to move PingER installation to a new machine and let us know when it's done.. The IP address of this new server will change. No longer use IP 200.143.196.20.

No progress 4/6/2016

Dec, 2014does not ping electrical problems
pinger.arn.dzemail 1/30/2016, 2/22/2016Nov 2015Does not ping
DESYFixed 4/18/2016  
Pinger.stanford.eduFixed 4/19/2016May 2015does not ping

Gathering data from the following Pakistani monitoring hosts that we (pinger.slac.stanford.edu) have been unable to gather data from for 6 months or more has been disabled:

Are there any of these that are likely to return to service?

Next Meeting

Next meeting:  Wednesday May 4th 9:00pm Pacific Standard Time, Thursday May 5th, 2016  9:00am Pakistan time, Thursday May 5th  2016 12:00noon Malaysian time, Thursday may 5th 2016 02:00am Rio Standard Time.  

Old Items

Ibrahim has extracted the  PingER Zip manually. He is reconstructing 11Gb of data, has 15GBytes of data there. He is trying to use SPAR to classify the data. He also was looking at RDF. the next step is to use MapReduce to organize and reduce the output of data so can visualize it. He will be using a the same techniques he used for looking at 1996-2006 weather data.  However at the moment he cannot access his  VMs from myren for the last two weeks, even for the myren site, He has emailed  them but till now they have not fixed the issue.  He has updated all his data in their cloud. No update 1/6/2016. Moved here 4/6/2016.

Huffington Post moved here 3/8/2016

UOA (Saqib) placed here 2/3/2016.

Saqib has a  5  MS students from the Database team

...

  • Jan 5, 2015 Hassaan reports "I have received revisions on my proposal and these days I am revising my proposal. In the meanwhile, I have also added another student (Anas Abrar) on this project. He is in learning phase and will follow the nodes which are not working. I shall give you an update very soon. "

    •  Hassaan is  very hopeful that if the proposal is accepted then we can easily have a full time RA for the project.

    Hassaan has re-submitted the proposal after revisions. He would like to get Anas Abrar more trained on monitoring operation and then will inform us to add him to the mailing list at http://pinger.unimas.my/pinger/contact.php.

     

  • Oct 2015. Following the last meeting, Anjum, Hassaan and Les met to discuss the way forward. 

    • "Adnan currently is unable to find resources for handling the project. Similarly, there is no progress on hiring of a full time RA by NUST HQ. 
    • However, I (Hassaan) checked from HEC about the proposal that I submitted last year. They have informed me that 2 reviewers have asked for revisions while they are waiting for the third review. I am very hopefull about it. If the proposal is accepted then we can easily have a full time RA for the project. I have plans to talk to Dr. Zaidi about hiring an RA on assuming that our proposal will be accepted by HEC. We can then get his salary deducted later from the HEC project. I shall update you very soon in this regard." 

    • Hassaan is  waiting to hear from HEC about the comments on the proposal. 

    • Moreover, he has asked a student to work on the project for the time being. His name is Mian Anas however he will need few weeks to understand the project. 

  • Thiago completed setting up the  PingER data SQL Impala warehouse running on a Nebula/Cloudera cluster using the Hadoop File System (HDFS). Unfortunately it is not currently accessible from outside SLAC. There have been several attempts to provide outside access, but no success yet, we need to engage the subject matter experts. Thiago is now a SLAC associate so he still has an account at SLAC. There was a cyber security alert on the version of java installed with Cloudera. Les has replaced the cloudera version of java which should fix the vulnerability. However the new version has not been tested.

Geolocation

Anjum believes the TULIP Geolocation application  can be improved significantly. At least there are few ideas that we can try. For this, either a group of undergraduate students or an active masters student is required. The resultant work can easily be the thesis of masters level. Who is interested? 

  • Saqib at Faisalabad has an MS student interested to work on Geolocation project. He requests an initial  paper on the project.  Les has responded to Saqib. He also has some other students. Anjum will contact him. Potential projects/asks include: take over management of PingER monitoring in Pakistan (say 5 monitors/student; case study of how Pakistan's network performance/connectivity has improved over thea years especially as function of funding etc;  geolocation with variable alpha; indoor geolocation

  • Johari will contact Anjum to learn more of the requirements. Update Johari/Adnan

  • See http://www.slac.stanford.edu/comp/net/tulip/. Basically TULIP uses pings to a target from landmarks at known locations and converts the minimum RTTs to estimate the distances. Then uses the distances with mulitlateration to estimate the location of the target

  • To improve TULIP one needs the right selection of landmarks, i.e. good (working landmarks) at the right locations (not too far from the target), straddling the target, and with a a reasonable estimate of the indirectness (directivity or alpha) of the path from the landmark to the target (so we can reasonably accurately estimate the distance). One also needs a reasonable density of landmarks (e.g. number of targets/100,000sq km)

  • The landmarks come from PingER and perfSONAR sites.  We have a reasonable density in the US, Pakistan and Europe. Currently Anjum is getting better than 20km accuracy for Pakistani targets

  • As the number of landmarks goes up so does the accuracy, but so does the time to make the measurements (pings). 

  • One needs to find the optimal density

  • Anjum proposes to speed up the measurements using a cluster for parallelization and also proposes to improve the adaptation of alpha based region. He regards the adaptive geolocation and parallelization as  MS projects. 

  • He is also interested in geolocation in small proximity (e.g.indoors), e.g. using cell tower signals. This is a new area of research. It is possible that the port of PingER to an Android could  be related to this. This is a PhD project

  • Anjum reports he can supervise the students on Geolocation. He will need to know when the students are ready. We can start with a joint meeting involving Les and the students. Later on, Anjum can have the meeting with students every week while Les can join if he has time.

NUST/SEECS Pakistani PingER nodes status

Pink Background indicates host was bad last month, strike through says it is fixed, yellow is an new bad host.

...

Is it time to start paring down the list of PingER monitor hosts in Pakistan, starting with those that have been down for a while and despite your efforts they are not cooperating.  One might also look at the coverage by region in Pakistan and try and keep good coverage for all regions.

Traceroute at UTM 5/9/2015

The traceroute problem regarding maximum reachable hops ( i.e. 11 hopes ) may be since the Unix/Linux/OSX  traceroute uses UDP to send the requests. The first request is sent to a particular port (33434), with a ttl  to tell it how many hops to go to.  The ttl starts at 1 is incremented as it tries the next hop, also the port is incremented (up to 33465).  It looks like the first few UDP ports are enabled and then they are blocked. The Windows traceroute uses ICMP to send the probes so does not see the problem.

Linked Open Data

Cristiane reports (7/1/2015): "I am trying to automatize the triplification of PingER data on Kettle. For now, part of the transformation is made on Kettle and another is made by a Java code. Although this solution works for a data sample, is important to have the entire process on Kettle because it facilitates to understand, modify and control the triplification process."

...

Christiane's report is at: Size Inflation of PingER Data for use in PingER LOD

UM

Moved here 3/4/2015:

Ibrahim has setup distributed hadoop clusters. He has 2TB of disk space. Les has provided information on getting a subset of PingER data by anonymous ftp via ftp://ftp.slac.stanford.edu/users/cottrell.  It was put there last September. Information on how the data was put together is at https://confluence.slac.stanford.edu/display/IEPM/Archiving+PingER+data+by+tar+for+retrieval+by+anonymous+ftp. There is information on formatting etc at http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html and some on the dataflows at https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC. Renan at UFRJ has successfully used this data, he has also characterized the data in terms of bytes/metric per year etc.

...

Anjum reported that UM had experienced a TCP syn DOS attack prior to Mar 12th (when an IDS was put in place). It occurred mainly for several days before between the hours on noon- 2pm and 7-7 in the evening (Malaysia time). He suggested looking to see if PingER could spit the effect.  Ibrahim, Les and Anjum will look at. Les analyzed the data and sent it to Anjum

NUST

The following is from Samad 2/24/2015.

Follow up from workshop
  • Hossein Javedani of UTM is interested in anomalous event detection with PingER data. Information on this is available at https://confluence.slac.stanford.edu/display/IEPM/Event+Detection. We have sent him a couple of papers and how to access the PingER data. Hossein and Badrul have been put in contact. Is there an update Badrul?

...

Anjum suggested Saqib,  Badrul and Johari put together a paper on user experiences with using the Internet in Malaysia as seen from Malaysian universities. In particular round trip time, losses, jitter, reliability, routing/peering, in particular anomalies, and the impact on VoIP, throughput etc.  It would be good to engage someone from MYREN.

Ibrahim

Ibrahim Abaker  is planning to work on a topic initially entitled " leveraging pingER big data with a modified pingtable for event-correlation and clustering".  Ibrahim has a proposal, see https://confluence.slac.stanford.edu/download/attachments/17162/leveraging+pingER+big+data+with+a+modified+pingtable+for+event-correlation+and+clustering.docx. Ibrahim reports 7/15/2014 "I have spent the last few months trying to understand the concept of big data storage and its retrieval as well as the traditional approach of storing RDF data. I have integrated a single hadoop cluster in our cloud. but for this project we need multiple clusters, which I have already discussed with Dr. Badrul and he will provide me with big storage for the experiment." No Update 8/20/2014.

"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"

Potential projects

See list of Projects