Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Hassaan Khaliq?, Muhammad Anas Abrar? (SEECS); Saqib?+, Aqsa? + (UAF); Johari?+, Adnan Khan (UNIMAS);  Badrul,  Ridzuan, Ibrahim (UM); (UTM); Adib?+, Fatima (UUM); Fizi Jalil (MYREN);  Les+, Bebo+, Joao (SLAC).

+ Confirmed attendance

...

? Individual emails sent

Actual attendees:

Saqib, Johari. Aqsa (but we could not hear her), Saba, Les, Bebo, SLAC

Administration

...

Bebo

...

Topher is here for a few days. He and Bebo discussed whether it would be better to use native Android tools rather than loading the perl interpreter and using it.

  1. Pro: easier for user, no need to load perl interpreter which may require missing skills, especially for a non technical user
  2. Con: the android version has to mimic pinger2.pl with all its logs, careful checking and specific format

To a large extent it depend on how we plan to use this. If the phones are just MAs in a fixed location then 1 is easier and probably sufficient.  If this is intended to grow into a mobile application for general use then it needs to be 2.

We agreed that Sara Masood at UUM should continue pursuing using the standard pinger2.pl version running under the perl interpreter while Amity will try and put together a native android version.

We also briefly discussed how to get the data from the phone MA to the archive. The current method ping_data.pl requires a public IP address for the phone which mnay not exist if its is mobile. Getting the MA to put the data to the archive may raise some security issue for the archiver.  We did ot come to a conclusion

UUM

From Adib:

  • Adib asked a student (Jaafrau)  to explore  “Creative visualization of PingER data, including rich interaction;”.  He has come up with a good design, yet the development part needs a lot of work. Adib will ask him to upload his interface and share the link.

  • Adib has a student from

No update 5/18/2016

Bebo has set up a Github codebase  as a new project. It contains the PingER MA (pinger2.pl and the traceroute/ping server). Anyone  needs to sign up for a Github account (if you don't already have one), so you can be added as a project member.

  • slac-pinger/pinger created by topherwhite.  PingER project https://github.com/slac-pinger/pinger
  • Now we have it we can share with Amity to check it works.
    • Bebo reports that someone from Amity requested to be a member of the Android group.

UUM

From Adib:

  • "Since we're still waiting on an account, at SLAC, I asked the student to explore another idea (which is quite relevant) proposed by Prof. Bebo “Creative visualization of PingER data, including rich interaction;”. He has already started looking at the possible attractive/interactive design, at the same time he is working to improve his programming skills."

  • There is another student, (Jaafrau) has a design.  Looking at visualization.
  • Adib has a student from Thailand who is working with Adib at UUM who  may be interested in installaing a PIngER MA in Thailand. Adib will talk next week after exams to see if can get permissions. 8/3/2016: No update yet, as they have a long holiday in Thailand. Will try to contact him.

  • Adib has written a case study on the Internet performance in ASEAN countries and its implication on the Socio-Economic Indexes. It has been shared with Dr.Les and Prof. Bebo  who have commented. We are looking for where to publish.

...

pinger.unimas.my is working again as of ~ June 18th,

No update 5/18/2016

Student started last month on the ISO.
Adnan has not been assigned any tasks. He is interested in writing papers and will be discussing with Johari on the way forward.
Will revisit traceroute server. No progress 3/12/2016
Update PingER Malaysia site, hope to start in April.
Get back to RPi2 at Datacenter no progress 3/12/216

also the traceroute server is working.

Student will be back in september and will start the ISO then.
Johari has given up on the  RPi2. He now has an RPIv3 with WiFi and Bluetooth which he will be testing in the coming weeks. 
The postgrad who was working on anomaly detection quit. Johari will pick up what is left and submit a paper.

UAF (Saqib)

There are 4 students:  Aqsa Hameed, Saba Muzamil, Tahseen and Sara Masood. T

  • Aqsa has completed her research work on "visualization on pingER data" and now working on to publish a conference paper. Here are few details of the research work.

    • The query results can be exported as CSV file. I use the CSV file of Query results from Impala to draw Line and Bar charts by using Google API.

    • She has created a Data warehouse on pingER data. First we transform the pingER text files into CSV files. Then i upload these CSV files on HDFS and populate Impala Tables and queries.

    • Line and Bar charts are created on a webpage running or executed by localhost server. it can be updated as the query results varies.
  • Aqsa and Saba are working together. Their goal is focusing on visualization of PingER historical data using warehouse.  The idea is to develop a warehouse in UAF university and make it publicly available. 

  • Aqsa has put together an  abstract of a conference paper and has put together an  abstract of a conference paper and submitted to "The 3rd IEEE/ACM International Conference on Big Data Science, Engineering and Applications (BDSEA 2016)" (http://computing.derby.ac.uk/bdcat2016/).  "Applying Big Data Warehousing and Visualization Techniques on pingER Data",  Aqsa Hameed, Dr. Saqib Ali, Dr. Les Cottrell and Bebo White, submitted to BDSEA 2016. Authors can access it via:   https://easychair.org/conferences/?conf=bdsea2016  

  • Aqsa and Saba are working together. Their goal is focusing on visualization of PingER historical data using warehouse.  The idea is to develop a warehouse in UAF university and make it publicly available. They are 50-60% done with setting up a Hadoop cluster with 3 nodes, 1 master, 2 slaves. She is currently working on importing the PingER data into hdfs on the  cluster. they have run some Impala queries on the data and are working on visualization
  • Topic: visualization on pingER data (email from Aqsa and Response from Renan)
    I have studied the google charts as visualization tools but here are some points need to be discussed.
    1. The idea of applying visualization on Data warehouse (Impala query results) does not seem to be so useful because Data warehouse contains static data and visualization charts will also remains static and need to be updated with the time.
    Yes, it needs to be updated with the time. My suggestion is to transform PingER data into data to be inserted into the data warehouse. Myself and some other Brazilian students have developed codes to do this. Such process should occur at least once a day to keep the data warehouse updated daily. This has never been done by any of us.
    2. Google charts API cannot integrate with Impala As Impala is hadoop distributed Big Data supported database Google can only integrate with flat files or flat databases like Mysql. If Google charts API can only read flat files (e.g., CSV files), it is trivial to save a database query result as a CSV flat file that would be consumed by Google charts.  Can Google charts generate a plot dynamically after reading a just-created CSV file? 
    Is using a different data visualization library (e.g., D3 https://d3js.org/ ) an option? 
  • Aqsa and  team members are working on creating Data warehouse and we are very close to complete this. Here are some updates.

    • Tehseen qureshi has transformed the pingER text files into binaries and soon he will be able to get CSV files.

    • Saba is working on defining a 4 node cluster.
    • Aqsa has uploaded some sample CSV files on HDFS and run Impala queries as i will get the actual CSV from tehseen these steps are also will be completed
  • Visualization Status 

  • Aqsa has 

    drawn a line chart and bar chart on the data of sample CSV file and i am exploring some more charts to be drawn by using Google API's.

  • Sara Masood - pingER  monitoring host on android . 
    • She is using perl-android-apk Interpreter to run perl on android. It includes execution of 3 perl scripts:
      • getdata.pl, ping_data.pl, pinger2.pl
    • First step is to install pre-requisite i.e. lynx, perl, XML::Simple, make etc. However, she is having permission denied issue when she executes "make install” command at path usr/local/bin because it’s a read only file system. She is  searching for the solution but maybe needs to change the path to /data/local/tmp.

    • Step 2 Is to execute ping _data and cgi-bin, traceroute.pl <http://traceroute.pl/

    • Step 3: To execute pinger2.pl <http://pinger2.pl/>  that will start gathering data

Project on Internet performance has not received funding.

  • Sara Masood - pingER  monitoring host on android . 
    • She is using perl-android-apk Interpreter to run perl on android. It includes execution of 3 perl scripts:
      • getdata.pl, ping_data.pl, pinger2.pl
    • First step is to install pre-requisite i.e. lynx, perl, XML::Simple, make etc. However, she was having permission denied issue when she executes "make install” command at path usr/local/bin because it’s a read only file system.

      • Perl-android-interpreter application was depreciated so install perl package using GNURoot Debian Application

      •  "Make install" issue is resolved too it create directorys and files required by pinger2 in usr/local/share/pinger

      •  But now when i  execute pinger2.pl <http://pinger2.pl/>  (that will ping host and start getting data) using IP address its generating two errors image is attached. IP address is also updated in pinger.xml

        • First error is," cannot exec DnsV4Cmd.pl, file not found", but DnsV4Cmd.pl  is in PingER directory with execute permission.

        • Second error, "SrcIP could not be determined for DNS 203.124.57.137. Without the results produced by PINGER2 is not usable.".
        • Les suspects the second error is result of the 1st error. maybe using the perl -d debugger would assist.
    • Step 2 Is to execute ping _data and cgi-bin, traceroute.pl <http://traceroute.pl/

    • Step 3: To execute pinger2.pl <http://pinger2.pl/>  that will start gathering data

  • Saqib will proceed to install Pinger in UAF Pakistan 6/22/2016.There used to be an MA at UAF Les will send the information to Saqib. it is below:

    Code Block
     NODENAME:        uaf.seecs.edu.pk 
     IPADDRESS:       121.52.152.22 
     SITENAME:        seecs.edu.pk 
     NICKNAME:        PK.UAF.SEECS.EDU.N1 
     FULLNAME:        University of Agriculture, Faisalabad 
     LOCATION:        Faisalabad 
     COUNTRY:         Pakistan 
     CONTINENT:       South Asia 
     LATANDLONG:      31.433741 73.070476 
     PROJECTTYPE:     D 
     PINGSERVER:      http://uaf.seecs.edu.pk/cgi-bin/traceroute.pl?function=ping 
     TRACESERVER:     http://uaf.seecs.edu.pk/cgi-bin/traceroute.pl? 
     DATASERVER:      http://uaf.seecs.edu.pk/cgi-bin/ping_data.pl? 
     URL:             http://www.uaf.edu.pk 
     CONTACTS:        Ahsan Raza Sattar ,   
     COMMENTS:        Add by Cottrell 11/5/2012.
    Host not working as a monitor and not pingable. There is another Monitor in faisalabad, so Disabled by Cottrell 6/26/2014. 

 Saqib will proceed to install Pinger in UAF Pakistan

UTM 

Saqib's old supervisor  agreed to appoint a master student to take of PingER in UTM. Saqib has emailed  3/9/2016, no progress 4/6/2016, no progress 5/18/2016. 

...

2.     For the M.Sc. student in UTM we have to request again to Prof Hanan and Prof Asri and we need support from Prof Johari.

Given there is a working MYREN host at UTM, we wll give up on this.

UM

MYREN

NUST

Asked for update on the following 7/30/2016:

...

  • Currently it does not respond to ping pingerqta.pern.edu.pk
  • 274cottrell@pinger:~$ping 121.52.157.157
    PING 121.52.157.157 (121.52.157.157) 56(84) bytes of data.
    ^C
    --- 121.52.157.157 ping statistics ---
    6 packets transmitted, 0 received, 100% packet loss, time 5625ms

  • http://121.52.157.148, responds with the default Apache web page
  • http://121.52.157.148//cgi-bin/ping_data.pl? responds (albeit very slowly) with Get Ping Data from sharedw228.b2bhosting.net  Also there is data. 
    • This is a funny hostname. It comes from the system response to the perl commands:
      • use Net::Domain qw(hostname hostfqdn hostdomain);
        my $hostname = hostfqdn();

  • The following needs to reflect the new IP address at 121.52.157.148. It is very easy to change the IP address once the Measurement Agent is up and running and responding, in fact the PingER meta data at SLAC was changed in February 2015. However the DNS still points to:
    • Output from: /bin/cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 121.52.157.157 pingerqta.pern.edu.pk
     

...

HostStatelast seenStatus
pinger.unimas.myemail 5/12/2016, fixed 6/16/2016, fixed June 16thMarch 2016Does not ping
pingersonar-utm.myren.net.myemail to Fitzi 5/12/2016, fixed May 18, 2016 Does not ping
pinger.unesp.bremail 6/15/2016, fixed 6/20/2016Feb 2016Looked like cronjob not running, cannot ping
pinger.fsksm.utm.myemail 6/15/2016, no responseMay 17Does not ping
www.univ-ouaga.bfUnreliable, No data since June 20th.June 2016Unreliable, No data since June 20th.June 2016Does not ping
www.ihep.ac.cnEmail 8/3/2016, ping_data.pl works but there is no dataJuly 28th, 2016ping_data.pl works but there is no dataDoes not ping

Next Meeting

Next meeting:  Wednesday Aug 24 9:00pm Pacific Standard Time, Thursday Aug 25, 2016  9:00am Pakistan time, Thursday Aug 25  2016 12:00noon Malaysian time, Thursday Aug 25 2016 02:00am Rio Standard Time.  

...

"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"

Image Modified

Potential projects

...

   Perl-android-interpreter is not executing .pl file mainly because old Perl version (Perl 5.10) which is not comparable with android lollipop 5.0 version, i am compiling  Perl .5.22.0 for android ,it may solve issue. 

 

 

| timechart span=1d count by application limit=300 
|  rename comment AS "The search is for subnet VPN-CLIENTS-GW2 and VPN-CLIENTS-GW1 inbound"