...
Attendees
Invitees:
Hassaan Khaliq+, Kashif, Raja, Samad Riaz (SEECS); Saqib+, Aqsa (UOA); Johari+, Nara, Adnan Khan + (UNIMAS); Abdullah, Badrul, Anjum+, Ridzuan, Ibrahim (UM); Hanan (UTM); Adib+, Fatima + (UUM); Fizi Jalil (MYREN); Thiago, Les+, Bebo + (SLAC).
Note returned to original time on the hour since no longer need to accomodate Johari.
...
? Individual emails sent
Actual attendees:
Hassaan, Saqib, Aqsa, Johari, Adnan, Anjum, Fatima, Les, Bebo
Administration
- Membership of pinger-my in https://groups.google.comAqsa Hameed has been added as of Dec 20, 2015. Johari has added her to
Huffington Post article on PingER and African Internet performance, see http://
pingerunimasmypinger/contact.php.This will most likely be Anjum's last meeting. He has already resigned from the University of Malaya. He will be leaving Malaysia on 31st January or 1 February. He will stay for 15 days in Pakistan then travel to Canada. We will miss him, and wish him all the best in his new startup.
Huffington Post article on PingER and African Internet performance, see http://www.huffingtonpost.com/david-tereshchuk/a-giant-leap-in-2016-africa_b_8901556.html
Anjum said it would be good for Hassaan to attach this to his proposal.
Les will contact SLAC communications to get it in SLAC Today.
david-tereshchuk/a-giant-leap-in-2016-africa_b_8901556.html
Anjum said it would be good for Hassaan to attach this to his proposal. Any action?
Les is working with the SLAC CIO to try and put something together for publicity concerning nthe life of PingER.
- NETAPPS2015 1-3 December 2015 was very successful, five papers were presented in the PingER session
- Currently, NetApps organizers are in the process of sending NETAPPS2015 proceedings to be indexed in the major indexing services.
9 students from Amity University near New Delhi in India interested in coming to SLAC. Bebo sent reminder recently, response below:
Sorry for the delay. Students are giving end semester exams. They are busy till dec 20th. They will get back to work as soon as the exams are done. Once the exam is over we will be setting up "PingER measurement agent at Amity"
Three students have each written a short paper for the 6th International Conference on Cloud System and Big Data Engineering at Amity Jan 14-15, 2016. I have reviewed and made some corrections
443: Analysis and clustering PingER network data
463: Application for the emulation of pinger on Android devices
432: Correlation analysis on Real Time Tab Delimited Network monitoring data
Btech computer science students would be graduating in May 2016. Their visas can be arranged after May 2016.
- Joao a Brazillian student identified by Renan is interested in a 3 month internship at SLAC similar to those completed by Renan and Thiago.
- Have letter of Support for Academic Training from his organization. Les filled out form for Joao.
Geolocation Anjum,
This is a proposed project for one of the Indian students.
Android - Bebo
Bebo has set up a Github codebase as a new project. It contains the PingER MA (pinger2.pl and the traceroute/ping server). Anyone needs to sign up for a Github account (if you don't already have one), so you can be added as a project member.
- slac-pinger/pinger created by topherwhite. PingER project https://github.com/slac-pinger/pinger
- Now we have it we can share with Amity to check it works.
- Bebo reports that someone from Amity requested to be a member of the Android group.
Saqib
- Saqib has a 5 MS students from the Database team
Sara Massoud came up with ideas to improve the info put together by Amity
Sabah Massoumil working on Linked Open Data (please excuse spelling)
Aqsa Hameed working on big data/analysis of PingER data. The initial idea was to set up a big data/big analysis
Aqsa Hameed has been working with Anjum to look at a project to create a Hadoop/Cloudera PingER warehouse to enable easier, more powerful access to PingER historical data.
Thinking a bit on Aqsa her work appears to be related to the work done by Thiago on PingER warehouse using a cluster/Cloudera/HDFS/Impala earlier this year. In particular see the presentation at NETAPPS2015 (see https://confluence.slac.stanford.edu/download/attachments/123309267/NETAPPS_PRESENTATIONrevLuiza.pptx. Once the paper is published we can also provide her with that (Adib will let us know when this is OK).
Les emailed the relevant people to put them in contact with one another. It appears there is a lot of overlap between what Aqsa proposed as a Masters project and what Thiago has already done. However Thiago's system was mainly a proof of concept and not in production. We need to look at the next steps: Internet access, auto-updating of information in near realtime, production service, maintainable etc. With this in place one could really mine the data looking for all kinds of interesting correlations, clustering, event impacts, comparisons etc.
Given that such a warehouse is available, then the next step would be to automatically create queries that would produce in near real-time the plots we produce manually for the PingER annual reports.
Following this publish the data in RDF to tie in with the Southampton RDF web observatory repository (similar to what Renan did as a proof of concept).
Since Thiago's warehouse is only available at SLAC, it may assist to get Aqsa an account at SLAC, alternatively Aqsa will need to set up a repository at here home base.
There is some documentation written by Les on the usage of the warehouse at SLAC. It is at: PingER Data Warehouse using Big Data with Cloudera on Nebula.
Les will contact Thiago to see if there is other documentation or where to find the programs etc.
We need to put together all we know:
Paper from Thiago on PingER warehouse presented in Malaysia at NETAPP2015 Thiago M. Da S. Barbosa, Renan F. Souza, Sergio M. S. Da Cruz, Maria L. Campos and R. Les Cottrell. Applying Data Warehousing and Big Data Techniques to Analyze Internet Performance
Document on Warehouse
Documents on PingERLOD
Linked Open Data Publication Strategies: Application in Networking PPerformance Measurement Data, Renan Souza, Les Cottrell, Bebo White, Maria Campos, Marta Mattoso, poster presented at the BigDataScience - Stanford conference, CA, USA May 27-31, 2014.
Reviews/proposals/ from Aqsa/Fatima
Survey on Big Data Indexing strategies, Fatima Bintu Adama, Adib Habbal, Suhaidi Hassan, R. Les Cottrell. Bebo White, Ibrahim Abdullahi.
Saqib will provide Aqsa's
Tehseen is working on missing PingER data.
- Saqib has submitted a project with title "A Fundamental Active Internet Performance Monitoring Framework for Pakistan Education & Research Network (PERN) in University of Agriculture, Faisalabad"
Project is accepted.
working on a project to develop a node using Raspberry Pi 2 and IoT to measure the air and soil quality.
Have 2 RPis and setting up air quality with sensors from market, idea is to distribute on campus.
Currently, students are analyzing the project to develop a problem statement for their research project.
For the GeoLocation Saqib should contact Anjum
- Project on Internet performance has been accepted, but no funding yet.
UUM
Fatima presented her work at NetApps. Fatima has passed her master oral examination (Viva) with minor correction and she is submitting her final dissertation coming Sunday to UUM school of computing. Indeed, we are very grateful to PingER team members support and help. Currently, Fatima is planning to pursue her PhD and possibly on a topic related to her master dissertation. We need PingER team suggestions and advice before we can decide on the topic. Adib will attend next meeting to further discuss on this.
UM
Ibrahim has extracted the PingER Zip manually. He is reconstructing 11Gb of data, has 15GBytes of data there. He is trying to use SPAR to classify the data. He alsop was looking at RDF. the next step is to use MapReduce to organize and reduce the output of data so can visualize it. He will be using a the same techniques he used for looking at 1996-2006 weather data. However at the moment he cannot access his VMs from myren for the last two weeks, even for the myren site, He has emailed them but till now they have not fixed the issue. He has updated all his data in their cloud. No update 1/6/2016
UNIMAS
UTM
Johari has had no response from UTM so far. Not sure whether they still want to be in the project. Last straw, need to contact someone from the network group so that they understand and appreciate the Pinger project. Update 1/6/2016.
Saqib please contact Dr Md Asri Ngadi. Will provide update after talk.
Johari will try and contact Hafizi
Saqib reports:
"Faculty management in the Faculty of Computing-UTM is changed. They ask me to re-locate the PingER server from the faculty server room as students are not allowed for physical access of the server room. Only faculty members are allowed to physically manage the servers in the server room. Therefore, I asked my supervisor (Dr Md Asri Ngadi/ Prof Hanan) from UTM to take the responsibility of the server. He is agreed, however, I have to manage the server remotely. Further, he needs an official invitation from your side to fulfill the official requirements. Further, he wants to make this project as a source of an international collaboration and linkages between UTM and SLAC-Stanford."
Les sent back:
The UTM PingER server is part of a worldwide collaboration led by SLAC that includes about 70 sites in over 20 countries. These sites are monitoring on-going Internet end-to-end performance , both currently and in the past with a history of 18 years of collected data.
SLAC is happy to host postgraduate students from UTM for a period of up to a year to contribute to this project as visiting scientists.
The students participate in collaborative research and development on the Internet End-to-end Performance Measurement (IEPM) project to monitor, analyze and present information on the end-to-end performance of Internet links. This project includes researching and developing techniques to optimize the measurements for the purposes of characterizing the “Digital Divide”. Dr. Les. Cottrell of SLAC’s Office of the Chief Information Officer is the host. All support for the visit is provided by sources other than SLAC.
The students are expected to comply with all policies and procedures, including safety guidelines. Your mentor will provide them with any specific policies, procedures or safety guidelines for their assigned area. As a program participant, the student does not enter an employee/employer relationship with DOE, SLAC or any other federal agency.
MYREN
No update 8/12/2015, no Update 9/2/2015.
perfsonar-unimas.myren.net.my is back up and running
We have been unable to gather data from pingersonar-utm.myren.net.my since September 11th, 2015 pingersonar-utm.myren.net.my. It is pingable however there is no response from http://pingersonar-utm.myren.net.my/cgi-bin/traceroute.pl?function=ping&target=www-wanmon.slac.stanford.edu&options=-i%200.2
NUST
Hassaan reports: "There are total of 48 active nodes for Pakistan out of which 18 nodes are working properly whereas 4 nodes have configuration issues. Lahore college for Woman has changed its data center and they want to know what do have to do with the new one. Faisalabad PERN had some critical issues and burnt their power supply so they also wanted to know what can be done to overcome that damage. FAST University Karachi and Air University, Islamabad are not in a mood to help as they say that they will put the node up, but haven’t done that in past 2 months [Hassaan would like to discuss this with Dr. Anjum for excluding them from the list]. Last but not the least Quetta HEC regional center wanted to change their static IP to 121.52.157.148."
Les has Disabled the following Pakistani monitors since they have no DNS name and are not responding by IP address. pinger.numl.edu.pk, pinger.uet.edu.pk, pinger.uob.edu.pk, pinger.usindh.edu.pk. We are still monitoring hosts at the site.
...
- - Adib any update
- Papers are now SLAC-PUBS
Survey on Big Data Indexing strategies, also SLAC-PUB-16460, Fatima Bintu Adama, Adib Habbal, Suhaidi Hassan, R. Les Cottrell. Bebo White, Ibrahim Abdullahi, , submitted to NETAPPS 2015.
Worldwide Internet Performance Measurements Using Lightweight Measurement Platforms, alsoSLAC-PUB-16461,R. Les Cottrell, Thiago Barbosa, Bebo White, Johari Abdullah and Topher White, submitted to NETAPPS 2015.
- Applying Data Warehousing and Big Data Techniques to Analyze Internet Performance, also SLAC-PUB-16464, T. M. S. Barbosa, R. F. Souza, S. M. S. Cruz, M. L. Campos and R. L. Cottrell, submitted to NETAPPS 2015.
PingER Malaysia-Internet Performance Measuring Project: A Case Study, also see SLAC-PUB-16462, Saqib Ali, R. Les Cottrell, Anjum Naveed, submitted to NETAPPS 2015.
Adaptive Geolocation of Internet Hosts, also SLAC-PUB-16463, Raja A.A. Khan, Roger L.A. Cottrell, Anjum Naveed, SLAC-PUB-16360, submitted to NETAPPS 2015.
9 students from Amity University near New Delhi in India interested in coming to SLAC. Bebo sent reminder recently, response below:
They have gone through the installation instructions for traceroute.pl, ping_data.pl and pinger2.pl. However there is a problem that the server appears to be local, see Setting up PingER servers at Amity
- Joao a Brazillian student identified by Renan is interested in a 3 month internship at SLAC similar to those completed by Renan and Thiago.
- We just need one more signature then I believe we can send the invitation
Geolocation Anjum,
This is a proposed project for one of the Indian students.
Android - Bebo
Bebo has set up a Github codebase as a new project. It contains the PingER MA (pinger2.pl and the traceroute/ping server). Anyone needs to sign up for a Github account (if you don't already have one), so you can be added as a project member.
- slac-pinger/pinger created by topherwhite. PingER project https://github.com/slac-pinger/pinger
- Now we have it we can share with Amity to check it works.
- Bebo reports that someone from Amity requested to be a member of the Android group.
Saqib
- Saqib has a 5 MS students from the Database team
Sara Massoud came up with ideas to improve the info put together by Amity
Sabah Massoumil working on Linked Open Data (please excuse spelling)
Aqsa Hameed working on big data/analysis of PingER data. The initial idea was to set up a big data/big analysis
Aqsa Hameed has been working with Anjum to look at a project to create a Hadoop/Cloudera PingER warehouse to enable easier, more powerful access to PingER historical data.
Thinking a bit on Aqsa her work appears to be related to the work done by Thiago on PingER warehouse using a cluster/Cloudera/HDFS/Impala earlier this year. In particular see the presentation at NETAPPS2015 (see https://confluence.slac.stanford.edu/download/attachments/123309267/NETAPPS_PRESENTATIONrevLuiza.pptx. Once the paper is published we can also provide her with that (Adib will let us know when this is OK).
Les emailed the relevant people to put them in contact with one another. It appears there is a lot of overlap between what Aqsa proposed as a Masters project and what Thiago has already done. However Thiago's system was mainly a proof of concept and not in production. We need to look at the next steps: Internet access, auto-updating of information in near realtime, production service, maintainable etc. With this in place one could really mine the data looking for all kinds of interesting correlations, clustering, event impacts, comparisons etc.
Given that such a warehouse is available, then the next step would be to automatically create queries that would produce in near real-time the plots we produce manually for the PingER annual reports.
Following this publish the data in RDF to tie in with the Southampton RDF web observatory repository (similar to what Renan did as a proof of concept).
Since Thiago's warehouse is only available at SLAC, it may assist to get Aqsa an account at SLAC, alternatively Aqsa will need to set up a repository at here home base.
There is some documentation written by Les on the usage of the warehouse at SLAC. It is at: PingER Data Warehouse using Big Data with Cloudera on Nebula.
Les will contact Thiago to see if there is other documentation or where to find the programs etc.
We need to put together all we know:
Paper from Thiago on PingER warehouse presented in Malaysia at NETAPP2015 Thiago M. Da S. Barbosa, Renan F. Souza, Sergio M. S. Da Cruz, Maria L. Campos and R. Les Cottrell. Applying Data Warehousing and Big Data Techniques to Analyze Internet Performance
Document on Warehouse
Documents on PingERLOD
Linked Open Data Publication Strategies: Application in Networking PPerformance Measurement Data, Renan Souza, Les Cottrell, Bebo White, Maria Campos, Marta Mattoso, poster presented at the BigDataScience - Stanford conference, CA, USA May 27-31, 2014.
Reviews/proposals/ from Aqsa/Fatima
Survey on Big Data Indexing strategies, Fatima Bintu Adama, Adib Habbal, Suhaidi Hassan, R. Les Cottrell. Bebo White, Ibrahim Abdullahi.
Saqib will provide Aqsa's
Tehseen is working on missing PingER data.
- Saqib has submitted a project with title "A Fundamental Active Internet Performance Monitoring Framework for Pakistan Education & Research Network (PERN) in University of Agriculture, Faisalabad"
Project is accepted.
working on a project to develop a node using Raspberry Pi 2 and IoT to measure the air and soil quality.
Have 2 RPis and setting up air quality with sensors from market, idea is to distribute on campus.
Currently, students are analyzing the project to develop a problem statement for their research project.
For the GeoLocation Saqib should contact Anjum
- Project on Internet performance has been accepted, but no funding yet.
UUM
Fatima presented her work at NetApps. Fatima has passed her master oral examination (Viva) with minor correction and she is submitting her final dissertation coming Sunday to UUM school of computing. Indeed, we are very grateful to PingER team members support and help. Currently, Fatima is planning to pursue her PhD and possibly on a topic related to her master dissertation. We need PingER team suggestions and advice before we can decide on the topic. Adib will attend next meeting to further discuss on this.
UM
Ibrahim has extracted the PingER Zip manually. He is reconstructing 11Gb of data, has 15GBytes of data there. He is trying to use SPAR to classify the data. He also was looking at RDF. the next step is to use MapReduce to organize and reduce the output of data so can visualize it. He will be using a the same techniques he used for looking at 1996-2006 weather data. However at the moment he cannot access his VMs from myren for the last two weeks, even for the myren site, He has emailed them but till now they have not fixed the issue. He has updated all his data in their cloud. No update 1/6/2016
UNIMAS
UTM
Johari has had no response from UTM so far. Not sure whether they still want to be in the project. Last straw, need to contact someone from the network group so that they understand and appreciate the Pinger project. Update 1/6/2016.
Saqib please contact Dr Md Asri Ngadi. Will provide update after talk.
Johari will try and contact Hafizi
Saqib reports:
"Faculty management in the Faculty of Computing-UTM is changed. They ask me to re-locate the PingER server from the faculty server room as students are not allowed for physical access of the server room. Only faculty members are allowed to physically manage the servers in the server room. Therefore, I asked my supervisor (Dr Md Asri Ngadi/ Prof Hanan) from UTM to take the responsibility of the server. He is agreed, however, I have to manage the server remotely. Further, he needs an official invitation from your side to fulfill the official requirements. Further, he wants to make this project as a source of an international collaboration and linkages between UTM and SLAC-Stanford."
Les sent back:
The UTM PingER server is part of a worldwide collaboration led by SLAC that includes about 70 sites in over 20 countries. These sites are monitoring on-going Internet end-to-end performance , both currently and in the past with a history of 18 years of collected data.
SLAC is happy to host postgraduate students from UTM for a period of up to a year to contribute to this project as visiting scientists.
The students participate in collaborative research and development on the Internet End-to-end Performance Measurement (IEPM) project to monitor, analyze and present information on the end-to-end performance of Internet links. This project includes researching and developing techniques to optimize the measurements for the purposes of characterizing the “Digital Divide”. Dr. Les. Cottrell of SLAC’s Office of the Chief Information Officer is the host. All support for the visit is provided by sources other than SLAC.
The students are expected to comply with all policies and procedures, including safety guidelines. Your mentor will provide them with any specific policies, procedures or safety guidelines for their assigned area. As a program participant, the student does not enter an employee/employer relationship with DOE, SLAC or any other federal agency.
MYREN
No update 8/12/2015, no Update 9/2/2015.
perfsonar-unimas.myren.net.my is back up and running
We have been unable to gather data from pingersonar-utm.myren.net.my since September 11th, 2015 pingersonar-utm.myren.net.my. It is pingable however there is no response from http://pingersonar-utm.myren.net.my/cgi-bin/traceroute.pl?function=ping&target=www-wanmon.slac.stanford.edu&options=-i%200.2
NUST
Hassaan reports: "There are total of 48 active nodes for Pakistan out of which 18 nodes are working properly whereas 4 nodes have configuration issues. Lahore college for Woman has changed its data center and they want to know what do have to do with the new one. Faisalabad PERN had some critical issues and burnt their power supply so they also wanted to know what can be done to overcome that damage. FAST University Karachi and Air University, Islamabad are not in a mood to help as they say that they will put the node up, but haven’t done that in past 2 months [Hassaan would like to discuss this with Dr. Anjum for excluding them from the list]. Last but not the least Quetta HEC regional center wanted to change their static IP to 121.52.157.148."
namal.seecs.edu.pk is working once again
No responses for over 30 days from
- npm.neduet.edu.pk
- pinger.uettaxila.edu.pk
- pingerisl-qau.pern.edu.pk
- pingerlhr-pu.pern.edu.pk
- pingerpwr.pern.edu.pk
- pingerqta.pern.edu.pk
- quest.seecs.edu.pk
PingER at SLAC
Put together a web page on the Fertility Rate and PingER measured Internet performance. The concern is the high Fertility rate in Sub-Saharan which is dominating the global growth in population and could result in over 6B people in Africa and 11B people on the planet by 2100. This puts a huge strain on resources. To reduce the Fertility Rate one needs education and information which is greatly enabled by the Internet. Unfortunately Internet performance is generally at its worst in West, Middle and East Africa where the Fertility Rates are highest.
Also put together a web page on ways of providing Internet access in hard to reach places, e.g. weather balloons, drones, low earth satellites.
The 2016 annual report on network monitoring is almost ready, it includes a section on PingER.
Working on the following hosts to be able to gather data
Host | State | last seen | Status | last seen | Status | ||
---|---|---|---|---|---|---|---|
web.hepgrid.uerj.edu | emails 5/1/2016, still no response 1/30/2016, email 1/30/2016 | Dec, 2014 | does not ping electrical problems | ||||
pinger.unesp.br | email 1/30/2015 | over a year ago | Does not ping | ||||
pinger.arn.dz | email 1/30/2015 | Nov 2015 | Does not ping | web.hepgrid.uerj.edu | emails 5/1/2015 | Dec, 2014 | does not ping electrical problems|
Pinger.stanford.edu | email 1/6/2016 | May 2015 | does not ping |
I propose Disabling Gathering data from the following Pakistani monitoring hosts that we (pinger.slac.stanford.edu) have been unable to gather data from for 6 months or more . This includeshas been disabled:
- Airuniversity: airuniversity.seecs.edu.pk
- GIKI: pinger.giki.edu.pk
- LCWU: pinger.lcwu.edu.pk
- NU: pinger.lhr.nu.edu.pk
- NUISB: nuisb.seecs.edu.pk
- PERNFSBDPOP: pingerfsbd.pern.edu.pk
- PINGERISL-QAU: pingerisl-qau.pern.edu.pk
- PINGERKHI-CPSP.PERN.EDU: pingerkhi-cpsp.pern.edu.pk
- PINGERQTA.PERN: pingerqta.pern.edu.pk
- SAU.SEECS: sau.seecs.edu.pk
Are there any of these that are likely to return to service?
Last month we Disabled the following (not responding to ping and no data for 12 months)
...
Potential projects