Page History

...

Attendees

Invitees:

Hassaan Khaliq  ?, Muhammad Anas Abrar (SEECS); Saqib+, Aqsa (UAF); Johari-+, Adnan Khan (UNIMAS);  Badrul,  Ridzuan, Ibrahim (UM); (UTM); Adib, Fatima (UUM); Fizi Jalil (MYREN);   Les+, Bebo, Joao+ (SLAC).

...

? Individual emails sent

Actual attendees:

Joao, Adib, Les

Administration

• Membership of pinger-my in https://groups.google.com

...

• "Since we're still waiting on an account, at SLAC, I asked the student to explore another idea (which is quite relevant) proposed by Prof. Bebo “Creative visualization of PingER data, including rich interaction;”. He has already started looking at the possible attractive/interactive design, at the same time he is working to improve his programming skills."

• "On the other hand, I am in the final stage of writing a case study on the Internet performance in ASEAN countries and its implication on the Socio-Economic Indexes. Will try to share with Dr.Les and Prof. Bebo soon to comment and get your advice on the publication possibility of the paper."

UNIMAS

No update 5/18/2016

• Adib has created a case study SE Asia (ASEAN). Les to read the study provide feedback international connectivity for leaders of the country.
• Adamu expecting baby so very busy early next month. So no progress.
• There is another student, (Jaafrau) has a design.  Looking at visualization.
• Adib has a student from Thailand who is working with Adib at UUM who  may be interested in installaing a PIngER MA in Thailand. Adib will talk next week after exams to see if can get permissions.

UNIMAS

No update 5/18/2016

...

• Aqsa and Saba are working together. Their goal is focusing on visualization of PingER historical data using warehouse.  The idea is to develop a warehouse in UAF university and make it publicly available. They are 50-60% done with setting up a Hadoop cluster with 3 nodes, 1 master, 2 slaves. She is currently working on importing the PingER data into hdfs on the  cluster. they have run some Impala queries on the data and are working on visualization
• Topic: visualization on pingER data (email from Aqsa and Response from Renan)

  I have studied the google charts as visualization tools but here are some points need to be discussed.
  1. The idea of applying visualization on Data warehouse (Impala query results) does not seem to be so useful because Data warehouse contains static data and visualization charts will also remains static and need to be updated with the time.
  
  Yes, it needs to be updated with the time. My suggestion is to transform PingER data into data to be inserted into the data warehouse. Myself and some other Brazilian students have developed codes to do this. Such process should occur at least once a day to keep the data warehouse updated daily. This has never been done by any of us.
  
  2. Google charts API cannot integrate with Impala As Impala is hadoop distributed Big Data supported database Google can only integrate with flat files or flat databases like Mysql. 
  
  If Google charts API can only read flat files (e.g., CSV files), it is trivial to save a database query result as a CSV flat file that would be consumed by Google charts.  Can Google charts generate a plot dynamically after reading a just-created CSV file? 
  Is using a different data visualization library (e.g., D3 https://d3js.org/ ) an option? 
Tahseen is

• Aqsa and  team members are working on

Handling Missing Data in PingER by using Artificial Intelligence. He has executed some Java queries and looking at parallel scientific workflow management using SciCumulus.
• Sara Masood is working on a PingER Internet Performance Monitoring Agent On Android Device. There has been a discussion to pin down more closely what is needed, i.e. a PingER Measurement Agent running on an Android making measurements to the Beacons, and the ability to upload the data nightly. We held a  skype group meeting with Topher, Sara, Saqib, Bebo and Les to move forward. See 20160331 Meeting on PingER/Android, also see Discussion on porting PingER to Android. She has what she needs for now.

Project on Internet performance has been accepted, but no funding yet.

UTM 

Saqib's old supervisor  agreed to appoint a master student to take of PingER in UTM. Saqib has emailed  3/9/2016, no progress 4/6/2016, no progress 5/18/2016. Any progress 6/22/2016.

UM

MYREN

• Email 6/15/2016 to Fitzi, pinger.fsksm.utm.my is down.

NUST

Emailed Dr Hassaan 6/15/2016 asking for update and 6/18/2016.

Dr Hassaan Khaliq contacted and said "We shall look into the issues with them" 5/13/2016. 

 Last update 4/6/2016. 

Hassaan's student (Mian Anas Abrar <anas.abrar@gmail.com>) reports: "Sir haven't disabled any of them which i told earlier and actually can't even get back to them because of their wrong contact numbers or their numbers not responding since then. I have tried contacting Dr.Les earlier  few time as per your instructions but don't know why i wasn't able to email him. rest the status is actually same nodes being down". We believe we have solved the mail problem.

We are unable to gather data from the following Pakistani PingER monitors:

...

• creating Data warehouse and we are very close to complete this. Here are some updates.

  • Tehseen qureshi has transformed the pingER text files into binaries and soon he will be able to get CSV files.

  • Saba is working on defining a 4 node cluster.
  • Aqsa has uploaded some sample CSV files on HDFS and run Impala queries as i will get the actual CSV from tehseen these steps are also will be completed

• Visualization Status 

• Aqsa has 

  drawn a line chart and bar chart on the data of sample CSV file and i am exploring some more charts to be drawn by using Google API's.

• Sara Masood:

1. 1. pingER  monitoring host on android . I am using perl-android-apk Interpreter to run pearl on android.. it include execution of  3 perl scripts

      1. getdata.pl

      2. ping_data.pl

      3. pinger2.pl

   2. Resources Required

      1. i need ping_data.pl script file and getdata.pl script file.

      2. Joao will provide them via anonymous ftp.

   3. Research Status

   • • Perl-android-interpreter is not executing .pl file mainly because old Perl version (Perl 5.10) which is not comparable with android lollipop 5.0 version, i am compilingPerl .5.22.0 for android ,it may solve issue. 

Project on Internet performance has not received funding.

Saqib will proceed to install Pinger in UAF Pakistan

UTM 

Saqib's old supervisor  agreed to appoint a master student to take of PingER in UTM. Saqib has emailed  3/9/2016, no progress 4/6/2016, no progress 5/18/2016. 

1.     Currently, Pinger server is located in the faculty server room. Recently, faculty has closed the Server room. Now we have to move it to the University Central Server Room. For this we need support from Prof Johari and Prof Abdul Hanan.

2.     For the M.Sc. student in UTM we have to request again to Prof Hanan and Prof Asri and we need support from Prof Johari.

UM

MYREN

• Email 6/15/2016 to Fitzi, pinger.fsksm.utm.my is down.

NUST

• Currently it does not respond to ping pingerqta.pern.edu.pk 
• Ping 121.52.157.148, responds
• http://121.52.157.148/ Apache responds. http://121.52.157.148//cgi-bin/ping_data.pl? responds (albeit very slowly) with Get Ping Data from sharedw228.b2bhosting.net  
  • This is a funny hostname. It comes from the system response to the perl commands:

    • use Net::Domain qw(hostname hostfqdn hostdomain);
      my $hostname = hostfqdn();

• Also there is data. 
• The following needs to reflect the new IP address at 121.52.157.148

  • Output from: /bin/cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1		localhost.localdomain localhost
    121.52.157.157   pingerqta.pern.edu.pk

• It is very easy to change the IP address once the Measurement Agent is up and running and responding, in fact the PingER meta data at SLAC was changed in February 2015. Howeer the DNS still points to:
  

  [cottrell@pinger ~]$ nslookup pingerqta.pern.edu.pk

  Server:  134.79.111.111

  Address: 134.79.111.111#53

  Non-authoritative answer:

  Name: pingerqta.pern.edu.pk

  Address: 121.52.157.157

   

Following hosts are down and not responsive. Nodes were contacted many times for identifying their issues. i.e turn on their pinGER Computers but never responded well.

• aup.seecs.edu.pk #AUP said that they will look intothe issue in a while everytime I ask them or give them the reminder. Does not ping
• cae.seecs.edu.pk #Does not ping
• kinniard.seecs.edu.pk+ #Unknown host
• maggie1.seecs.edu.pk # Ping responds with Time to live exceeded
• maggie2.seecs.edu.pk #Does not ping
• npm.neduet.edu.pk* # Not pingable 
• pinger.uettaxila.edu.pk* #UET taxila are not receiving the provided Number, Unknown host.
• pingerisl-qau.pern.edu.pk*  #number is not responding for QUA. Not pingable
• quest.seecs.edu.pk* #Quest are not receiving my Call now. Not pingable

• pingerlhr-pu.pern.edu.pk* #Site pings via http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=pingerlhr-pu.pern.edu.pk&function=ping, however  http://pingerlhr-pu.pern.edu.pk/cgi-bin/ping_data.pl? gives site can't be reached.
• pingerpwr.pern.edu.pk* # Site pings via http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=pingerpwr.pern.edu.pk&function=ping however  the configuration file /usr/local/share/pinger/pinger.xml is truncated, see FAQ for how to go about fixing.

...

...

PingER at SLAC

Joao making the data on FTP to be up-to-date. 

Also has a 4 node cluster with Cloudera manager for Hadoop, Next step are to get impala working for queries.

...

+ Not seen in June 2016

Strikethrough working again in June

...

 Working on the following hosts to be able to gather data

Next Meeting

Next meeting:  Wednesday Aug 24 Jul  27 9:00pm Pacific Standard Time, Thursday Aug 25Jul 28, 2016  9:00am Pakistan time, Thursday Aug 25 Jul 28  2016 12:00noon Malaysian time, Thursday Aug 25 Jul 28 2016 02:00am Rio Standard Time.  

...

Ibrahim Abaker  is planning to work on a topic initially entitled " leveraging pingER big data with a modified pingtable for event-correlation and clustering".  Ibrahim has a proposal, see https://confluence.slac.stanford.edu/download/attachments/17162/leveraging+pingER+big+data+with+a+modified+pingtable+for+event-correlation+and+clustering.docx. Ibrahim reports 7/15/2014 "I have spent the last few months trying to understand the concept of big data storage and its retrieval as well as the traditional approach of storing RDF data. I have integrated a single hadoop cluster in our cloud. but for this project we need multiple clusters, which I have already discussed with Dr. Badrul and he will provide me with big storage for the experiment." No Update 8/20/2014.

"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"

Image Removed

Potential projects

See list of Projects

initially entitled " leveraging pingER big data with a modified pingtable for event-correlation and clustering".  Ibrahim has a proposal, see https://confluence.slac.stanford.edu/download/attachments/17162/leveraging+pingER+big+data+with+a+modified+pingtable+for+event-correlation+and+clustering.docx. Ibrahim reports 7/15/2014 "I have spent the last few months trying to understand the concept of big data storage and its retrieval as well as the traditional approach of storing RDF data. I have integrated a single hadoop cluster in our cloud. but for this project we need multiple clusters, which I have already discussed with Dr. Badrul and he will provide me with big storage for the experiment." No Update 8/20/2014.

"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"

Image Added

Potential projects

See list of Projects

1)  pingER  monitoring host on android .

 I am using perl-android-apk Interpreter to run pearl on android.

it include execution of  3 pearl scripts

*             PingER2.pl

*             ping_data.pl

*             getdata.pl

2) Resources Required

 i need ping_data.pl script file and get data.pl script file .

3) Research Status

   Perl-android-interpreter is not executing .pl file mainly because old Perl version (Perl 5.10) which is not comparable with android lollipop 5.0 version, i am compiling  Perl .5.22.0 for android ,it may solve issue.