Time & date
Wednesday July 1st 2015 9:00pm Pacific Standard Time, Thursday July 2nd 2015 9:00am Pakistan time, Thursday July 2nd 2015 noon Malaysian time, Thursday July 2nd, 2015 02:00am Rio Standard Time.
Coordinates of team members:
See: http://pinger.unimas.my/pinger/contact.php
Attendees
Invitees:
Hassaan Khaliq?, Kashif, Raja, Samad Riaz? (SEECS); Johari+, Nara, Adnan Khan (UNIMAS); Abdullah, Badrul, Anjum, Ridzuan?, Ibrahim? (UM); Hanan, Saqib (UTM); Adib+, Fatima+ (UUM); Fizi Jalil (MYREN)?; Thiago+, Les+, Bebo- (SLAC)
...
- Responded but Unable to attend:
? Individual emails sent
Actual attendees:
The meeting failed to transpire, Les did not make the call (his bad)!
Administration
- Membership of pinger-my in https://groups.google.com
Arshad has left SEECS/NUST to become the Rector of the National Textile University in Faisalabad. We have contacted the Rector at NUST to discuss continued support for PingER at SEECS. He has been very supportive and responded as follows:
We will surely continue with PingER project. Dr Zaidi who has succeeded Arshad has spent as many years in NIIT and later SEECS as Arshad. He will take care of the project.
I have copied this email to Director Research and Director Academics too. They will fully support Zaidi. Hope your worries are removed. If you find any issues, please give an email to me and I will ensure that we achieve all the targets of this project. Thanks and regards.
I have emailed Dr Zaidi.Acting principal (Engineer Habeel) asked for details about project that Anjum provided. They have also contacted Hassaan and asked him to be faculty in-charge of the project. Dr. Zaidi and Habeel Ahmed are not clear of the status and direction of the project so Dr. Arshad told them he can discuss it during his next visit of SEECS. That is what I last know. I think Dr. Arshad visits SEECS every 15 days or so and is helping with smooth transition.
Dr Zaidi writes:
Thank you for your email and for posing confidence in NUST-SEECS for the collaboration that has been on-going for a number of years and has only grown stronger with time. It is indeed a pleasure for us here at NUST-SEECS to be working in collaboration with your team at SLAC. We certainly value this collaborative effort and look forward to take it to newer heights.
I have recently conducted a special review meeting to ensure that things are back on track. Through this email, I would like to reiterate our commitment to this collaboration. We assure you of our full support for taking this a step further. As for your requirement of a System Administrator and a Faculty member in-charge of this project, you will be happy to know that Dr Hassan Khaliq -- a faculty member at SEECS who is already involved in the project, has been made responsible to manage the affairs from our end. In addition, we have deployed an MS research student to assist him in this effort. We have also raised the requirement of a full-time resource for this project and we are hoping to have this resource with us soon
Please do not hesitate to contact me directly should you face any problem from this day forward.
Anjum How are the measurements, analysis and paper on GeoLocation coming along?
He has been looking at the alpha (directivity) behavior, there was an exponential behavior, but it was unclear how to take advantage of it. Anjum cut N. America into regions to facilitate improving the accuracy of the alpha prediction. He believes this will improve things.
- Anjum has not started on this yet, he hopes to get to it when he returns from Canada
Johari has got the OK from the conference organizing committee to hold a colocated PingER/BigData workshop on August 3rd the day before the CITA 2015 (see http://www.cita.my/ an International Conference 4th - 6th August 2015, on transforming Big Data into Knowledge. Johari will provide relevant information to Bebo. Bebo will be able to make a presentation. Les has sent Bebo some relevant slide decks. Johari has an abstract from Bebo. There are PingER related papers submitted from UM
UFRJ
Maria Luiza has requested Raphaela and Christiane to give an update on what they've been working on in the last weeks, and maybe talk with Thiago on how they've been using their local cloud of 4 nodes with Cloudera and a data cube version of the ontology.
Cristiane reports (7/1/2015): "I am trying to automatize the triplification of PingER data on Kettle. For now, part of the transformation is made on Kettle and another is made by a Java code. Although this solution works for a data sample, is important to have the entire process on Kettle because it facilitates to understand, modify and control the triplification process."
UUM
Adib reports 6/1/2015 UUM
PingER UUM is online again, problem has solved half way.is solved already, a new IP is assigned to UUM PingER server (http://103.5.183.4/ <http://103.5.183.4/> ). I am still testing the connectivity and stability of the new connection. Les updated the address in the meta database at SLAC, However thehost is not currently pingable
Fatima has installed hadoop on all three machines. One will stand as the host machine from which the remote machines will be controlled. She is currently trying out some MapReduce examples and Hive installation. MapReduce examples and Hive installation.
They have managed to get a copy of PingER data for Fatima research from UM friends, Thanks to Dr. Anjum and Mr. Ibrahim for their help and support.
Adib is involved in NETAPPS 2015. It is a forum for scientists, researchers, students, and practitioners from all over the world to present their latest research results, ideas, and developments in the area of Future Internet and discuss advancement of next generation networks. More details about NETAPPS2015 can be found in the following link: http://internetworks.my/netapps2015/v2/index.php. Included in the advisory committee are: Bebo, Anjum, Les. the conference is in December in KL. There was a discussion of how to engage PingER. There is a track on Internet protocols and service. This was agreed to be a good place to present some PingER papers on Internet measurements and a Malaysian Case Study. Saqib started a case study. It is at: https://drive.google.com/folderview?id=0B-NEKleLll79ZFNmUnhiVGJ0Nmc&usp=sharing_eid. It is incomplete and needs updating. Anjum will find a new PhD student to look at this.
NETAPPS2015 submission deadline is extended to 31/July (firm deadline), ample time to submit papers. So far, no paper has been received from PingER group. if we have enough papers, we can have one dedicated session for PingER papers and Internet performance.
UM
Badrul reports 6/28/2015: "pinger.fsktm.um.edu.my 103.18.2.152 (unable to gather data since June 26th 2015) currently has a problem with the ISP. We just changed to a new ISP last Friday and we are working on solving the problem. Also www.um.edu.my is not pingable. Anjum reports that it is old and needs replacing."
...
Renan reports: "I had a similar experience. HDFS works better with bigger files rather than many small files. What I did was to create a Map-Reduce job to reduce all those thousands of small files into only 17 big files, each of them containing all data for a given year [1998-2014].I didn't use Hadoop MapReduce for this, though. I used a different dataflow distributed engine that also implements map and reduce operators. I am working on providing Thiago these 17 big files. Once he gets the data, he can share them with you and explain how the data on each file are stored".
UNIMAS
Johari has a Raspberry Pi 2. It has double the RAM and a better processor. It will go to the Data Center in June.
...
UTM
Johari will contact Hanan to request someone to support PingER at UTM, now Saqib has left. (6/3/2015),
...
Saqib points out a number of Malaysian routes are IPv6 which could have problems for traceroute. Saqib is checking
MYREN
pingersonar-um.myren.net.my is down since may 31st.
Fizi had some problems with tracepath in https://perfsonar.myren.net.my/toolkit/gui/reverse_traceroute.cgi?function=tracepath. He has fixed it by changing the initial MTU.
NUST
Pink Background indicates host was bad last month, strike through says it is fixed, yellow is an new bad host.
...
Is it time to start paring down the list of PingER monitor hosts in Pakistan, starting with those that have been down for a while and despite your efforts they are not cooperating. One might also look at the coverage by region in Pakistan and try and keep good coverage for all regions.
PingER at SLAC
Thiago has modified ping_data_plot.pl and frequency.pl to provide much improved GUI for the plots. This has been very useful for plotting the differences between pinger.slac.stanford.edu and pinger-raspberry.slac.stanford.edu. See ePinger Project at SLAC which we are putting together. the data is available at http://www-wanmon.slac.stanford.edu/cgi-wrap/pingtable.pl?file=average_rtt&by=by-node&size=100&tick=daily&year=2015&month=06&from=EDU.SLAC.STANFORD.RASPBERRY&to=WORLD&ex=none&only=all&dataset=hep&percentage=any
...
Bebo arranged a meeting with the Colombia RENATA NREN folks and the minister of IT to discuss the use of PingER in Colombia. There is a web page at: Colombia. Les has sent an email asking them to install pinger2.pl at at least one site in Columbia. Sent a reminder email 2/27/2015. Bebo will send a gentle reminder to the RENATA people of Columbia to see whether they continue to be interested and need a meeting.
Next meeting
Next meeting: Wednesday August 5th 2015 12th 2015 9:00pm Pacific Standard Time, Thursday JAugust 6th 2015 August 13th 2015 9:00am Pakistan time, Thursday August 6th 13th 2015 noon Malaysian time, Thursday Thuirsday Thursday August 6th, 13th , 2015 02:00am Rio Standard Time.
Old Items
Traceroute at UTM 5/9/2015
The traceroute problem regarding maximum reachable hops ( i.e. 11 hopes ) may be since the Unix/Linux/OSX traceroute uses UDP to send the requests. The first request is sent to a particular port (33434), with a ttl to tell it how many hops to go to. The ttl starts at 1 is incremented as it tries the next hop, also the port is incremented (up to 33465). It looks like the first few UDP ports are enabled and then they are blocked. The Windows traceroute uses ICMP to send the probes so does not see the problem.
Raspberry Pi 5/9/2015
The two major issues with the Raspberry Pi would be:
...
If/when it works it would be instructive to look at the data from pinger and raspberry pi to Malaysia since the distances are shorter and the differences may show up better. For Sep-Oct 2014 when there was data measured from both Oct-Nov the averages for 20 paths was 52+-21ms (from pinger.unimas.my to 20 other Malaysian hosts) and 56+-21ms for raspberry pi to 20 other Malaysian hosts.
Linked Open Data
Feb 2015
The plan is still the one seen before (see project proposal), experimenting those alternatives. Right now, they managed to triplify the data according to a new ontology that takes advantage of a combination of a current standard for multidimensional data (called data cube vocabulary) and a revised version of Renan's Moment ontology adaptation. With this we expect to have a better data organization than the previous solution.
...
Christiane's report is at: Size Inflation of PingER Data for use in PingER LOD
UM
Moved here 3/4/2015:
Ibrahim has setup distributed hadoop clusters. He has 2TB of disk space. Les has provided information on getting a subset of PingER data by anonymous ftp via ftp://ftp.slac.stanford.edu/users/cottrell. It was put there last September. Information on how the data was put together is at https://confluence.slac.stanford.edu/display/IEPM/Archiving+PingER+data+by+tar+for+retrieval+by+anonymous+ftp. There is information on formatting etc at http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html and some on the dataflows at https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC. Renan at UFRJ has successfully used this data, he has also characterized the data in terms of bytes/metric per year etc.
...
Anjum reported that UM had experienced a TCP syn DOS attack prior to Mar 12th (when an IDS was put in place). It occurred mainly for several days before between the hours on noon- 2pm and 7-7 in the evening (Malaysia time). He suggested looking to see if PingER could spit the effect. Ibrahim, Les and Anjum will look at. Les analyzed the data and sent it to Anjum
NUST
The following is from Samad 2/24/2015.
- buitms.seecs.edu.pk #We have to disable gathering data from this host because the person still don't want to continue with us as i have tried once again to convince him but the answer is same. Les has disabled from SLAC.
- nukhimain.seecs.edu.pk # We were unable to gather data since 20th November, 2014 and now the Node is working fine and collecting data as well.
- pinger.uettaxila.edu.pk #The node is working fine from last two weeks.
- sau.seecs.edu.pk. #This Node is working fine now.
- pingerjms.pern.edu.pk #This node is working now.
- pinger.uet.edu.pk # this was also not working from so many days. and now its working fine and collecting data as well.
- pinger.isra.edu.pk # This node is also working fine now.
- pingerlhr-pu.pern.edu.pk # This is also working fine now.
- pinger.kohat.edu.pk # Collecting data now.
The IP of "pingerqta.pern.edu.pk" has been changed, Les has updated the databas at SLAC with the following
Old IP: 121.52.157.157
New IP: 121.52.157.148
Follow up from workshop
- Hossein Javedani of UTM is interested in anomalous event detection with PingER data. Information on this is available at https://confluence.slac.stanford.edu/display/IEPM/Event+Detection. We have sent him a couple of papers and how to access the PingER data. Hossein and Badrul have been put in contact. Is there an update Badrul?
...
Anjum suggested Saqib, Badrul and Johari put together a paper on user experiences with using the Internet in Malaysia as seen from Malaysian universities. In particular round trip time, losses, jitter, reliability, routing/peering, in particular anomalies, and the impact on VoIP, throughput etc. It would be good to engage someone from MYREN.
Ibrahim
Ibrahim Abaker is planning to work on a topic initially entitled " leveraging pingER big data with a modified pingtable for event-correlation and clustering". Ibrahim has a proposal, see https://confluence.slac.stanford.edu/download/attachments/17162/leveraging+pingER+big+data+with+a+modified+pingtable+for+event-correlation+and+clustering.docx. Ibrahim reports 7/15/2014 "I have spent the last few months trying to understand the concept of big data storage and its retrieval as well as the traditional approach of storing RDF data. I have integrated a single hadoop cluster in our cloud. but for this project we need multiple clusters, which I have already discussed with Dr. Badrul and he will provide me with big storage for the experiment." No Update 8/20/2014.
"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"
Potential projects