...
Attendees
Invitees:
Hassaan Khaliq?, Kashif, Raja, Samad Riaz (SEECS); Johari?+, Nara, Adnan Khan (UNIMAS); Abdullah, Badrul, Anjum+, Ridzuan, Ibrahim- (UM); Hanan, Saqib Saqib? (UTM); Adib-, Fatima? (UUM); Fizi Jalil (MYREN); Thiago, Les+, Bebo? + (SLAC).
Note change of time since US no longer on daylight saving (summer) time, and need to accommodate Johari.
...
? Individual emails sent
Actual attendees:
Anjum, Johari, Saqib, Bebo, Les
Administration
- Membership of pinger-my in https://groups.google.com
We are working with Dr Zaidi the new head of SEECS to investigate resources need to continue support of PingER in Pakistan. We communicated first week in August. No word since. Sent a reminder 8/1/2015. We had a rump meeting of Anjum, Hassaan and Les after the previous meeting to discuss the way forward. Hassaan is waiting to hear from HEC about the comments on the proposal. Any Update?
Hassaan reports "I have received revisions on my proposal and these days I am revising my proposal.
IN the meantime he has also added another student (Anas Abrar) on this project. He is in learning phase and will follow the nodes which are not working. He shall give us an update very soon.
He is very hopeful that if the proposal is accepted then we can easily have a full time RA for the project.
However, Anjum points out it may take a while to get acceptance and funding (e.g. 6 months)
- NETAPPS2015 1-3 December 2015NETAPPS2015:
All four papers were accepted.
Draft conference program is ready, just waiting for paper registration before we can publish it officially in our website. Pinger session is going to be in the first day morning (Malaysian time).
Thiago has updated the Data Warehouse submission:
- Working out details of funding the registration, whether Thiago can be
Thiago has updated the Data Warehouse submission:
- Working out details of funding the registration, whether Thiago can be online to answer questions and who will present it.
9 students from Amity University near New Delhi in India interested in coming to SLAC for 3 months starting Feb until May. All expenses paid by them.
SLAC putting together document on proposed projects, skills needs, setting up chat group
- Joao a Brazillian student identified by Renan is interested in a 3 month internship at SLAC similar to those completed by Renan and Thiago. We are currently working on:
- Obtaining authorization from his J-1 sponsor to participate in research/work at SLAC.
- Letter of Support for Academic Training from his organization
- Academic Training Verification Form (this has his mentor's signature it and a brief explanation of the work you'll be learning, etc.)
- Copy of his health insurance
Needed to move this meeting 30 minutes earlier in future to accommodate Johari's schedule. Is this ongoing?
Geolocation Anjum,
This is a proposed project for one of the Indian students.
Android - Bebo
Bebo said Topher would be interested in getting a student to port PingER to an Android. Possibly one of the Indian students. Les put out a version of the PingER MA on Github at https://github.com/iepm/ in 2013 as part of coordinating with Google( which did not get very far). It has been dormant since it was put out there. Les is not a github expert or user so is unclear how complete it is or who can access it. Can somebody test and if not available let Les know what to do.
Topher is at home for 2 weeks, he will set up a shareable Github for PingER files. Any progress
Saqib
Saqib has submitted a project with title "A Fundamental Active Internet Performance Monitoring Framework for Pakistan Education & Research Network (PERN) in University of Agriculture, Faisalabad"
- Funding source will accept the projects if it is designed for the development of Agriculture in the University.
- Objective: The active internet performance monitoring framework for the University of Agriculture, Faisalabad helps in forecasting and detecting the events that cause bottleneck links in the network affecting the overall performance of the University of Agriculture, Faisalabad.
- Initially the project is accepted. However, I received the following comment
- The PI was advised to include in the project the justification how it will have an impact on agriculture development in the University of Agriculture Faisalabad Pakistan.
- For justification I have added the following paragraph:
- The aim of this project is to provide the performance analysis and anomaly detection framework for the University of Agriculture, Faisalabad in order to detect the events that cause bottleneck links in the network affecting the overall performance of the University. Further, It will help to improve the performance of the web applications running in the university.
- Les added:
Internet connectivity is critical to agriculture, especially since agriculture is usually conducted in rural and possibly remote areas. In particular it enables farmers to ascertain the optimum prices for seeds, fertilizers at, the most promising markets etc for their goods, upcoming weather conditions, forecasting yields, providing information for addressing challenges etc. It also supports timely sharing of information between farmers, their customers, and learning of new relevant developments from research and industrial centers such as the universities. This is very important since the adoption of new technologies has always been an important factor in agricultural productivity. See http://pnwsteep.wsu.edu/directseed/conf2k5/pdf/gillis.pdf
Understanding the Internet performance between the University of Agriculture and other areas of Pakistan will enable setting up: information centers at Faisalabad, outreach to agricultural communities (e.g. by social tools etc.), by setting realistic expectations for connectivity and hence capabilities to various regions of Pakistan for email, web, chat rooms, data sharing etc.
UUM
Fatima implemented her work (involving HDFS and MapReduce with Pinger Data) on 50MB to 200MB data sets, using a 2-node cluster, and then a 3-node cluster. The work is not completed, but she has obtained some good results and is putting together the presentation for her final viva. She and Ibrahim are using the same cloud/cluster at MYREN.
UM
Ibrahim has extracted the PingER Zip manually. He is reconstructing 11Gb of data, heas 15GBytes of data there. He is trying to use SPAR to classify the data. He alsop was looking at RDF. the next step si to use MapReduce to organize and reduce the output of dataso can vizualize it. He will be using a teh same techniques he used for looking at 1996-2006 weather data.
UNIMAS
UTM
Johari will contact Hanan to request someone to support PingER at UTM, now Saqib has left. Johari will try dean to get a replacement for Saqib. However the Dean is new so Johari wants to gather some information first.
Saqib
Has a student interested to work on Hadoop and pinger data. Les has sent him information on the SLAC Nebula cluster Cloudera/impala/HDFS Pinger Data warehouse dedveloped by Thiago and Co.
Further he has a few more MS students searching for their research topics. We can also utilize them in our pinger project. They are from software engineering, database and network background.
MYREN
No update 8/12/2015, no Update 9/2/2015.
perfsonar-unimas.myren.net.my is back up and running
We have been unable to gather data from pingersonar-utm.myren.net.my since September 11th, 2015 pingersonar-utm.myren.net.my. It is pingable however there is no response from http://pingersonar-utm.myren.net.my/cgi-bin/traceroute.pl?function=ping&target=www-wanmon.slac.stanford.edu&options=-i%200.2
NUST
Following the last meeting, Anjum, Hassaan and Les met to discuss the way forward.
- "Adnan currently is unable to find resources for handling the project. Similarly, there is no progress on hiring of a full time RA by NUST HQ.
- However, I (Hassaan) checked from HEC about the proposal that I submitted last year. They have informed me that 2 reviewers have asked for revisions while they are waiting for the third review. I am very hopeful about it. If the proposal is accepted then we can easily have a full time RA for the project. I have plans to talk to Dr. Zaidi about hiring an RA on assuming that our proposal will be accepted by HEC. We can then get his salary deducted later from the HEC project. I shall update you very soon in this regard."
Hassaan is waiting to hear from HEC about the comments on the proposal.
- Moreover, he has asked a student to work on the project for the time being. His name is Mian Anas however he will need few weeks to understand the project.
PingER at SLAC
Thiago completed setting up the PingER data SQL Impala warehouse running on a Nebula/Cloudera cluster using the Hadoop File System (HDFS). Unfortunately it is not currently accessible from outside SLAC. There have been several attempts to provide outside access, but no success yet, we need to engage the subject matter experts. Thiago is now a SLAC associate so he still has an account at SLAC. There was a cyber security alert on the version of java installed with Cloudera. Les has replaced the cloudera version of java which should fix the vulnerability. However the new version has not been tested.
Working on the following hosts to be able to gather data
Host | State | last seen | Status |
---|---|---|---|
web.hepgrid.uerj.edu | emails 12/2/2014, 12/8/2014, 2/26/2015, 4/30/2015, 6/1/2015 | Oct 23, 2014 | traceroute.pl works but no response from ping_data.pl |
pinger.stanford.edu | email 3/14/2015 | Feb 18, 2015 | Works |
pinger.unesp.br | email 11/28/2014, 5/22/2015, 6/1/2015. | Nov 3, 2014 | Host is pingable from SLAC. |
Next Meeting
Next meeting: Wednesday Dec 2nd 7:30pm Pacific Standard Time, Thursday Dec 3rd 2015 8:30am Pakistan time, Thursday Dec 3rd 2015 11:30am Malaysian time, Thursday Dec 3rd 2015 01:30 am Rio Standard Time.
...
- Maria will use resources from their project on Big Data for the registration..
Johari has forwarded application to our Research Division for the payment of the conference fee. This will cover the paper he is presenting on PingER on a Raspberry Pi
Need to put together a presentation slide deck - Johari, Les (put slides together)
Anjum is registered. He is reducing the size of the paper to 6 pages to avoid the extra cost.
The due date for camera ready is 15th November.
9 students from Amity University near New Delhi in India interested in coming to SLAC for 3 months starting Feb until May. All expenses paid by them.
SLAC putting together document on proposed projects, skills needs, setting up chat group
There are 5 students working remotely on various aspect of PinGER:
- Jahin Majumdar- Correlation and Association of Data.
- Anwesha Mal-Clustering and Dealing with how to correlate a new node’s IP address with its old one rather than storing it as two separate unrelated entities.
- Damini Singh- Data mining using Hadoop and Outlier Analysis
Aneesh Devasthale- XML schema for the data and Uniform Data Storage.
Will ask them to set up a PingER Measurement Agent and ping responder
Could use Github to provide code.
Asked if one or more may be interested in Geolocation.
- Joao a Brazillian student identified by Renan is interested in a 3 month internship at SLAC similar to those completed by Renan and Thiago. We are currently working on:
- Obtaining authorization from his J-1 sponsor to participate in research/work at SLAC.
- Letter of Support for Academic Training from his organization
- Academic Training Verification Form (this has his mentor's signature it and a brief explanation of the work you'll be learning, etc.)
- Copy of his health insurance
Needed to move this meeting 30 minutes earlier in future to accommodate Johari's schedule. This is ongoing until the new year.
Geolocation Anjum,
This is a proposed project for one of the Indian students.
Android - Bebo
Bebo said Topher would be interested in getting a student to port PingER to an Android. Possibly one of the Indian students. Les put out a version of the PingER MA on Github at https://github.com/iepm/ in 2013 as part of coordinating with Google( which did not get very far). It has been dormant since it was put out there. Les is not a github expert or user so is unclear how complete it is or who can access it. Can somebody test and if not available let Les know what to do.
Topher will be back in a couple of days. He will set up a shareable Github for PingER files. Bebo and Topher are working on it. Les has provided the source code location. Once we have it then we can share with Amity to check it works.
There are some students at UNIMAS who have expressed interest in the illegal logging audio project. This might be tied into Microsoft's Imagine.
Saqib
Saqib has submitted a project with title "A Fundamental Active Internet Performance Monitoring Framework for Pakistan Education & Research Network (PERN) in University of Agriculture, Faisalabad"
- Funding source will accept the projects if it is designed for the development of Agriculture in the University.
- Objective: The active internet performance monitoring framework for the University of Agriculture, Faisalabad helps in forecasting and detecting the events that cause bottleneck links in the network affecting the overall performance of the University of Agriculture, Faisalabad.
- Initially the project is accepted. However, I received the following comment
- The PI was advised to include in the project the justification how it will have an impact on agriculture development in the University of Agriculture Faisalabad Pakistan.
- For justification I have added the following paragraph:
- The aim of this project is to provide the performance analysis and anomaly detection framework for the University of Agriculture, Faisalabad in order to detect the events that cause bottleneck links in the network affecting the overall performance of the University. Further, It will help to improve the performance of the web applications running in the university.
- Les added:
Internet connectivity is critical to agriculture, especially since agriculture is usually conducted in rural and possibly remote areas. In particular it enables farmers to ascertain the optimum prices for seeds, fertilizers at, the most promising markets etc for their goods, upcoming weather conditions, forecasting yields, providing information for addressing challenges etc. It also supports timely sharing of information between farmers, their customers, and learning of new relevant developments from research and industrial centers such as the universities. This is very important since the adoption of new technologies has always been an important factor in agricultural productivity. See http://pnwsteep.wsu.edu/directseed/conf2k5/pdf/gillis.pdf
Understanding the Internet performance between the University of Agriculture and other areas of Pakistan will enable setting up: information centers at Faisalabad, outreach to agricultural communities (e.g. by social tools etc.), by setting realistic expectations for connectivity and hence capabilities to various regions of Pakistan for email, web, chat rooms, data sharing etc.
Bebo provided a coupe of links to IOT that may assist:
http://bits.blogs.nytimes.com/2015/08/03/the-internet-of-things-and-the-future-of-farming/?_r=1
Saqib has 5 students interested in PingER. In particular in Big Data, Data Mining and Geolocation.
For the GeoLocation Saqib should contact Anjum
For Big Data and Data Mining I suspect they will need a cloud/cluster. Given the problem at other sites it is important to be assured they will have the appropriate resources to pursue their work.
UUM
Fatima implemented her work (involving HDFS and MapReduce with Pinger Data) on 50MB to 200MB data sets, using a 2-node cluster, and then a 3-node cluster. The work is not completed, but she has obtained some good results and is putting together the presentation for her final viva. She and Ibrahim are using the same cloud/cluster at MYREN. No update 11/3/2015
UM
Ibrahim has extracted the PingER Zip manually. He is reconstructing 11Gb of data, heas 15GBytes of data there. He is trying to use SPAR to classify the data. He alsop was looking at RDF. the next step si to use MapReduce to organize and reduce the output of dataso can vizualize it. He will be using a the same techniques he used for looking at 1996-2006 weather data. However at the moment he cannot access his VMs from myren for the last two weeks, even for the myren site, He has emailed them but till now they have not fixed the issue. He has updated all his data in their cloud.
Anjum is working on a paper to submit to IEEE Internet Computing on the topic of 'Measuring the Internet’. Due date extended to Nov 13, 2015
UNIMAS
UTM
Johari will contact Hanan to request someone to support PingER at UTM, now Saqib has left. Johari will try dean to get a replacement for Saqib. However the Dean is new so Johari wants to gather some information first.
Saqib reports:
"Faculty management in the Faculty of Computing-UTM is changed. They ask me to re-locate the PingER server from the faculty server room as students are not allowed for physical access of the server room. Only faculty members are allowed to physically manage the servers in the server room. Therefore, I asked my supervisor (Dr Md Asri Ngadi/ Prof Hanan) from UTM to take the responsibility of the server. He is agreed, however, I have to manage the server remotely. Further, he needs an official invitation from your side to fulfill the official requirements. Further, he wants to make this project as a source of an international collaboration and linkages between UTM and SLAC-Stanford."
Les sent back:
The UTM PingER server is part of a worldwide collaboration led by SLAC that includes about 70 sites in over 20 countries. These sites are monitoring on-going Internet end-to-end performance , both currently and in the past with a history of 18 years of collected data.
SLAC is happy to host postgraduate students from UTM for a period of up to a year to contribute to this project as visiting scientists.
The students participate in collaborative research and development on the Internet End-to-end Performance Measurement (IEPM) project to monitor, analyze and present information on the end-to-end performance of Internet links. This project includes researching and developing techniques to optimize the measurements for the purposes of characterizing the “Digital Divide”. Dr. Les. Cottrell of SLAC’s Office of the Chief Information Officer is the host. All support for the visit is provided by sources other than SLAC.
The students are expected to comply with all policies and procedures, including safety guidelines. Your mentor will provide them with any specific policies, procedures or safety guidelines for their assigned area. As a program participant, the student does not enter an employee/employer relationship with DOE, SLAC or any other federal agency.
MYREN
No update 8/12/2015, no Update 9/2/2015.
perfsonar-unimas.myren.net.my is back up and running
We have been unable to gather data from pingersonar-utm.myren.net.my since September 11th, 2015 pingersonar-utm.myren.net.my. It is pingable however there is no response from http://pingersonar-utm.myren.net.my/cgi-bin/traceroute.pl?function=ping&target=www-wanmon.slac.stanford.edu&options=-i%200.2
NUST
Hassaan reports "I have received revisions on my proposal and these days I am revising my proposal. In the meanwhile, I have also added another student (Anas Abrar) on this project. He is in learning phase and will follow the nodes which are not working. I shall give you an update very soon. "
- I am very hopeful that if the proposal is accepted then we can easily have a full time RA for the project.
PingER at SLAC
Bebo is looking at setting up a Github with the PingER monitoring code and traceoute.pl.
Working on the following hosts to be able to gather data
Host | State | last seen | Status |
---|---|---|---|
web.hepgrid.uerj.edu | emails 12/2/2014, 12/8/2014, 2/26/2015, 4/30/2015, 6/1/2015 | Oct 23, 2014 | traceroute.pl works but no response from ping_data.pl |
pinger.stanford.edu | email 3/14/2015 | Feb 18, 2015 | Works |
pinger.unesp.br | email 11/28/2014, 5/22/2015, 6/1/2015. | Nov 3, 2014 | Host is pingable from SLAC. |
Next Meeting
Next meeting: Wednesday Dec 2nd 7:30pm Pacific Standard Time, Thursday Dec 3rd 2015 8:30am Pakistan time, Thursday Dec 3rd 2015 11:30am Malaysian time, Thursday Dec 3rd 2015 01:30 am Rio Standard Time.
Old Items
Oct 2015. Following the last meeting, Anjum, Hassaan and Les met to discuss the way forward.
- "Adnan currently is unable to find resources for handling the project. Similarly, there is no progress on hiring of a full time RA by NUST HQ.
- However, I (Hassaan) checked from HEC about the proposal that I submitted last year. They have informed me that 2 reviewers have asked for revisions while they are waiting for the third review. I am very hopeful about it. If the proposal is accepted then we can easily have a full time RA for the project. I have plans to talk to Dr. Zaidi about hiring an RA on assuming that our proposal will be accepted by HEC. We can then get his salary deducted later from the HEC project. I shall update you very soon in this regard."
Hassaan is waiting to hear from HEC about the comments on the proposal.
- Moreover, he has asked a student to work on the project for the time being. His name is Mian Anas however he will need few weeks to understand the project.
Thiago completed setting up the PingER data SQL Impala warehouse running on a Nebula/Cloudera cluster using the Hadoop File System (HDFS). Unfortunately it is not currently accessible from outside SLAC. There have been several attempts to provide outside access, but no success yet, we need to engage the subject matter experts. Thiago is now a SLAC associate so he still has an account at SLAC. There was a cyber security alert on the version of java installed with Cloudera. Les has replaced the cloudera version of java which should fix the vulnerability. However the new version has not been tested.
Geolocation
Anjum believes the TULIP Geolocation application can be improved significantly. At least there are few ideas that we can try. For this, either a group of undergraduate students or an active masters student is required. The resultant work can easily be the thesis of masters level. Who is interested?
...