Page History

...

• Saqib has a  5  MS students from the Database team

  • Sara Massoud came up with ideas to improve the info put together by Amity

  • Sabah Massoumil working on Linked Open Data (please excuse spelling)

  • Aqsa Hameed working on big data/analysis of PingER data. The initial idea was to set up a big data/big analysis 

  • Aqsa Hameed has been working with Anjum to look at a project to create a Hadoop/Cloudera PingER warehouse to enable easier, more powerful access to PingER historical data. 

    • Thinking a bit on Aqsa her work appears to be related to the work done by Thiago on PingER warehouse using a cluster/Cloudera/HDFS/Impala earlier this year. In particular see the presentation at NETAPPS2015 (see https://confluence.slac.stanford.edu/download/attachments/123309267/NETAPPS_PRESENTATIONrevLuiza.pptx. Once the paper is published we can also provide her with that (Adib will let us know when this is OK). 

    •  Les emailed the relevant people to put them in contact with one another. It appears there is a lot of overlap between what Aqsa proposed as a Masters project and what Thiago has already done. However Thiago's system was mainly a proof of concept and not in production. We need to look at the next steps: Internet access, auto-updating of information in near realtime, production service, maintainable etc. With this in place one could really mine the data looking for all kinds of interesting correlations, clustering, event impacts, comparisons etc.

      • Given that such a warehouse is available, then the next step would be to automatically create queries that would produce in near real-time the plots we produce manually for the PingER annual reports.

      • Following this publish the data in RDF to tie in with the Southampton RDF web observatory repository (similar to what Renan did as a proof of concept).

      • Since Thiago's warehouse is only available at SLAC,  it may assist to get Aqsa an account at SLAC, alternatively Aqsa will need to set up a repository at here home base.

      • There is some documentation written by Les on the usage of the warehouse at SLAC. It is at: PingER Data Warehouse using Big Data with Cloudera on Nebula.

      • Les will contact Thiago to see if there is other documentation or where to find the programs etc.

      • We need to put together all we know:

        • Paper from Thiago on PingER warehouse presented in Malaysia at NETAPP2015NETAPP2015 Thiago M. Da S. Barbosa, Renan F. Souza, Sergio M. S. Da Cruz, Maria L. Campos and R. Les Cottrell. Applying Data Warehousing and Big Data Techniques to Analyze Internet Performance

        • Document on Warehouse

          • PingER Data Warehouse using Big Data with Cloudera on Nebula
            

        • Documents on PingERLOD

          • Possible follow on to PingERLOD project, PingER#PingERLOD
            

          • Linked Open Data Publication Strategies: Application in Networking PPerformance Measurement Data, Renan Souza, Les Cottrell, Bebo White, Maria Campos, Marta Mattoso, poster presented at the BigDataScience - Stanford conference, CA, USA  May 27-31, 2014.
            

        • Reviews/proposals/ from Aqsa/Fatima

          • Survey on Big Data Indexing strategies, Fatima Bintu Adama, Adib Habbal, Suhaidi Hassan, R. Les Cottrell. Bebo White, Ibrahim Abdullahi.
            

          • Saqib will provide Aqsa's

    • Tehseen is working on missing PingER data.

• Saqib has submitted a project with title "A Fundamental Active Internet Performance Monitoring Framework for Pakistan Education & Research Network (PERN) in University of Agriculture, Faisalabad"

  • Project is accepted.

  • working on a project to develop a node using Raspberry Pi 2 and IoT to measure the air and soil quality.

    • Have 2 RPis and setting up air quality with sensors from market, idea is to distribute on campus.

  • Currently, students are analyzing the project to develop a problem statement for their research project.

• For the GeoLocation Saqib should contact Anjum

• Project on Internet performance has been accepted, but no funding yet.

...

Ibrahim has extracted the  PingER Zip manually. He is reconstructing 11Gb of data, has 15GBytes of data there. He is trying to use SPAR to classify the data. He alsop was looking at RDF. the next step is to use MapReduce to organize and reduce the output of data so can visualize it. He will be using a the same techniques he used for looking at 1996-2006 weather data.  However at the moment he cannot access his  VMs from myren for the last two weeks, even for the myren site, He has emailed  them but till now they have not fixed the issue.  He has updated all his data in their cloud.   No update 1/6/2016

UNIMAS

...

• Working on the following hosts to be able to gather data 

• Pinger is now on a VM. 

Next Meeting

Next meeting:  Wednesday Feb 3 8:00pm Pacific Standard Time, Thursday Feb 4, 2016  9:00am Pakistan time, Thursday Feb 4 2016 12:00noon Malaysian time, Thursday Feb 4 2016 02:00am Rio Standard Time.  

...