Versions Compared

Old Version 1

changes.mady.by.user Les Cottrell

Saved on

compared with

New Version 2

changes.mady.by.user Les Cottrell

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Time & date 

Meeting:  Tuesday, October 9th 9 pm Pacific time; Wednesday, October 10th, 2018 9:00 am Pakistan time; 12:00 noon Malaysian & Guangzhou time; and 11 am Thailand time.

Format

New items and updates are in boldface.

Coordinates of team members:

See: http://pinger.unimas.my/pinger/contact.php

  • Need to add Umar and Dr. Taha - Johari

Attendees

Invitees:

Wajahat Hussain (SEECS), Saqib (GZHU); Johari (UNIMAS);  Adib (UUM); Dr. Charnsak Srisawatsakul (Ubru), Les, Bebo, Umar

  • + Confirmed attendance; - Responded but  Unable to attend: ? Individual emails sent
Actual Attendees

Wajahat, Saqib, Les, Bebo, Taha (Skype ID nightwingduck, email taha.ali@seecs.edu.pk)

Others

Charnsak was in another meeting but listening in to our meeting.

...

Action Items

  • Future meetings are moved to Tuesdays Pacific time.
  • Bebo: will try and contact Dr. A. Sai Sabitha to see how to move forward with the Android PingER project, in particular, schedule a meeting. 
    • Les will forward recent emails to Amity, to Bebo. - Done
  • Wajahat: will provide a list of working or potentially working (e.g. being worked on) Pakistani MAs to Les.Saqib and Les: are working on being able to gather data from  2001:da8:270:2018:f816:3eff:fef3:bd3 IPv6 node in Beijing working again.
    • Les: if they are not enabled, Les will enter them into the database or re-enable the gathering of data from these MAs
  • Umar: update his section on the comparison IPv6 vs IPv4 ping RTTs and TCP vs ICMP/ping RTTs.
  • Johari: enter Umar and Dr. Taha into http://pinger.unimas.my/pinger/contact.php
  • Charnsack: We need to get the latest pinger2.pl measurement agent script installed at Ubru so we can get better logging and see why the other hosts are not being monitored.
  • Les: will invite Dr. Taha to join the pinger-my email list. - Done. 

Amity (Updated 8/6/2018, No update 9/6/2018)

  • Amity MA is unreliable so using it for a case study does not appear fruitful.
  • The Android version of the PingER MA, is described with  comments at  ePingER on Android Native - Amity project (this a proposal/description from Aayush Jain)
    • It describes a multipurpose, stand-alone device that can be widely distributed, something that we have brainstormed about for a long time. 

    • Bebo mentioned it to Topher and he feels that their app (when completed and vetted by his team) could easily be installed as a default service on his rainforest monitors (certainly future ones, not devices already in place). Merging the service data that he already collects with that unique to PingER has the potential to lead to some interesting results. 

    • Bebo asks: should we try to increase our communication with Amity and do we have faith that they would follow through?

    • How should we proceed? 

      • We agreed to request Amity to share the App and instructions with us; we will look at installing on a jailbroken Android phone at the San Francisco end and try it out.

      • Also encourage them to put together a paper.

      • Also August 16th and again September 3rd proposed a meeting between: Bebo, Umar and Les. and the Amity folks. No response. Bebo will try and contact 

Thailand (No update 8/9/2018, No update 9/6/2018)

For his IPv6 monitoring site pinger6.ubru.cs.ac.th Charnsak has a pinger.xml configuration file with over 160 IPv6 targets. However, there is a huge discrepancy since according to pingtable.pl there are only about 13 targets responding. We need to get the latest pinger2.pl measurement agent script installed at Ubru so we can get better logging and see why the other hosts are not being monitored.   Charnsak plans to remedy in August.

Charnsak is looking at a host in Champasak University, Chan Parsa province in Laos as a potential site for a PingER MA. Charnsak just got approved to make contact with the Champasak University. He expects to set up the MA in the next 4-5 months (say towards end 2018). It also depends on the partner university, and there may be a lot of paperwork.

UNIMAS (No update 8/9/2018, no update 9/6/2018)

Need to add Umar Kalim to http://pinger.unimas.my/pinger/contact.php. From the 7/5/2018 meeting: Johari can't ssh into the server so he will go to it on Monday.  He will also upload the new UNIMAS PingER website next week.

Sent reminder emails 8/6/2018. 9/3/2018.

UUM (No update 8/9/2018, No update 9/6/2018)

Les has sent Adib updates to Figs 3, 4, 5 to extend out to 2018. This is for the paper  Socio-economic Development Indices and Their Reflection on Internet Performance in ASEAN Countries

...

  • 121.52.146.180 (kohat.edu.pk) down since Nov 22/2017. Wajahat recommends continuing at least until the new student is up to speed (3/8/2018). No data available 3/24/2018.
  • cae.seecs.edu.pk last time we were able to gather any data was February 27th.
  • pinger-ncp.ncp.edu.pk pings but can't gather data 8/11/2017 and 9/16/2017. Contacted. Pings but can't gather data 10/24/2017. They are in the process of restoring 1/17/2018. Still down February 28, 2018, await new student.  (3/8/2018). No data 3/24/2018. Still down 8/6/2018.  It has been Disabled so data is no longer being gathered. 
  • pinger.isra.edu.pk unable to gather data since 3/6/2018, also does not ping. It is working as of September 2nd, 2018.
  • Wajahat says they will get these nodes up. These have been good nodes. They just need the weekly push. NUST will push them soon.  No update 8/9/2018.

UAF/GHZU (Updated 8/9/2018)

Saqib's future at GZHU will be much clearer after 3 months (i.e. November). His current contract expires February 2019. 

...

HostStatelast seenStatus
pinger.cern.chPingable, web server does not respond.9/11/2018.Mail sent
pingersonar-um.myren.net.myNo response6/26/2018Pings
121.56.146.180 (pinger.kohat.edu.pk) DownNov 22nd, 2017 
cae.seecs.edu.pkDownFeb 27, 2018 
pinger.isra.edu.pkDownMarch 6, 2018 
pingeramity.inIt has been working since 28th July. It is unclear how stable it is. It is down again 9/5/2018)April 27, 2018 

Next Meeting

Next meeting:  Tuesday October 9th 9 pm Pacific time; Wednesday October 10th, 2018 9:00 am Pakistan time; 12:00 noon Malaysian & Guangzhou time; and 11 am Thailand time.

...

Old information

Umar looking at extending the comparison IPv6 vs IPv4 ping RTTs and TCP vs ICMP/ping RTTs. Last update  6/7/2018

  • See Towards Analysis of ICMP vs TCP Ping Latencies - Umar
    • Looked into Traffic Differentiation - Rate Limiting vs. Traffic Prioritization (QoS)

    • IPv6 results gathered using ping-vs-tcp.pl script. About 56 nodes with IPv6 addresses, 14 of which responded with Npings

    • IPv4 results gathered from SLAC and Virginia Tech
      • SLAC's batch may be downloaded here (approx. 24 MB)
      • Skimmed results; findings are pretty much the same as before
  • Pending
    • Identified relevant events in the network stack that highlight timing (_RECVFROM, _RECVMSG, _IP_RECV, _NETIF_RX etc.). Looking for instrumentation that enables us to measure timestamps. We also need to figure out how to determine whether ICMP & TCP traffic are treated differently? and then how to measure the difference?
      • perf-tools allows us to measure transport events
      • If we could assume that the path for ICMP & TCP through the network is the same, then the only difference between two (controlled) tests would be the time spent in the transport layers. This can be measured using perftools. 
      • However, such measurements must be made in a controlled environment where ICMP and TCP are treated the same. (I say so because some results — e.g., in East Asia and South Asia — clearly show that ICMP performs much worse than TCP.)
    • We would also need to cater for cross traffic and queuing delays. Given how small the differences are, one may argue that the variations in measurements are due to cross traffic. Perhaps we should start with controlled tests and then see if real world measurements reflect similar behavior.
    • We need to setup a test environment. We can either setup a bare-metal box or use a VM. 
      • I will see if I can arrange for a bare-metal box.

Discussion item (7/5/2018)

Saqib sent an email to the team:

...

Bebo's impression is that Saqib will lead in putting the ideas in his paper into practice. Saqib will need some students.  Saqib's boss is going to the NY City meeting.

GZHU China - Saqib (moved here 7/2/2018)

Saqib submitted the Camera ready paper on  “A Blockchain-based Decentralized Data Storage and Access Framework for PingER” and it has been accepted in Trustcom2018.

...

NUST (moved here 6/29/2018)

There is an upcoming grant call for projects between Pakistan and the US. Topics may be focused on cybersecurity, health, and education. It has not been announced yet. Wajahat will get the details and share them with the team as soon as they are available. It is interesting since getting a US partner appears to be a roadblock for many potential Pakistani responders. However, the topics may not be very related to PingER. NUST is looking at applying to set up a cyber lab. Getting the funding will be in competition with other Pakistani Universities. For cyber the main things we could think of from PingER were: quantifying what fraction of hosts block pings, punching holes in firewalls to allow pings, how to misuse ping (e.g. ping-of-death, or using anomalous ping packets to deduce the OS etc. flood pings for DOS), the host can respond to ping but applications do not work.  Fear of misuse of pings can result in the system administrator, network administrator or cybersecurity blocking pings. A possibility might be a study of what fraction of say working www/dns etc. apps (i.e. checking if a host responds to the relevant port) do not respond to pings.  This could be by application, by country or by region etc.  Also how to protect a remote pinger traceroute or server from being used in DOS attacks. As of 3/27/2018 there is no call so far. There was one last year, so Wajahat is expecting one. Emailed Wajahar 6/3/2018 asking for update. He responded "There is no call yet. There was one last year. May be change in the US-Pak policy. Just a guess."

UUM (moved here 6/29/2018)

Adib, Bebo, Les met with Southampton Web observatory person. There seemed to be enthusiasm. Adib was going to send some materials to Southampton. The person at Southampton gave us some links. Adib is in the early stages of exploring what web observatory data to link with such as business context indicators, social media and government sites. There was no update 3/29/2018, or 5/3/2018.

GZHU (moved here 3/8/2018)

Saqib submitted a project in CERNET to monitor the performance of  IPv6 network using PingERv6. He received the news that the project is accepted with 100K RMB. Now he has 2 accepted projects regarding PingER and total amount he has is near about 40K USD. Further, in his lab, three U1 servers have already arrived through another grant for research purpose. We can also use them for our PingER project.  

...

Currently, no data is available on PingER on Android due to unavailability of the live IP address. No update 4/19/2017, 7/6/2017. Email sent to Sara Masood. No update 9/24/2017. Any update 10/24/2017. No progress 1/18/2018.

GZHU (moved here 1/15/2018)

PingER has valuable historical data for the last 20 years.Many analysis and case studies have been carried using this data. A lot of information is available on the website. Saqib's idea is to publish the brief summary all these analysis through a survey paper covering the history and utilization of PingER data starting from 1998 to 2017. Saqib started on it, Les is providing assistance. Need your feedback on the idea of Measuring the Digital Development of the Countries using PingER data.          Is there something you want me to review some, e.g. some draft document on Measuring the Digital Development of the Countries using PingER data , or are you asking if it is a good idea to review and create such a document. If the latter I think this is a fascinating subject. Part of the challenge is the chicken and egg problems: i.e. is it network performance influencing advancement of the country, or is it the reverse that advanced countries can afford good networks.  My belief is it goes both ways. Also one needs to extend the analysis beyond just Africa else it’s kind of a repeat of Pinging Africa , R. Les Cottrell, IEEE Spectrum February 2013. Also see A Simple Tool for Measuring Digital Development , by R. Les Cottrell, IEEE Spectrum February 2013. This is derived from SLAC-PUB-15333. 

UUM (moved here 10/24/2017)

"BIND: An Indexing Strategy for Big Data Processing" that uses PingER data. Submitted and accepted by the 2017 IEEE Region 10 Conference (TENCON) that takes place in November. In Penang Malaysia

GZHU

The paper title: "Detecting Anomalies from End-to-end Internet Performance Measurements (PingER) using Cluster Based Local Outlier Factor" is submitted in ISPA 2017 (http://trust.gzhu.edu.cn/conference/ISPA2017/). It has been accepted as of 9/17/2017.

The thesis of Aqsa Hameed title “Applying Data Mining and Visualization Techniques on Pinger Data” is published in ODBMs.org and is accessible through http://www.odbms.org/2017/07/applying-data-mining-and-visualization-techniques-on-pinger-data/

SEECS (moved here 9/19/2017.)

  • Aqsa who was working with Saqib submitted  "Applying Big Data Warehousing and Visualization Techniques on pingER Data",  Aqsa Hameed, Dr. Saqib Ali, Dr. Les Cottrell and Bebo White, to BDSEA 2016.
  • I see it is available from ACM online on the following link: http://dl.acm.org/citation.cfm?doid=3006299.3006337 for $15.
  • This might be useful to Wajahat's student.

Amity (moved here 9/16/2017)

Preparing a paper on the impact of the cyclone Verdha that hit the Indian coast along with many countries like Thailand, Sri Lanka, Malaysia, Maldives on December 6th. They use K-Means clustering (see https://en.wikipedia.org/wiki/K-means_clustering) to identify anomalies in packets received (inverse of loss) and maximum RTT. Note that for December 7th the reduction in packets received.

Amity (moved here 5/18/2017)

From: Aayush Jain <aayush.2896@gmail.com>

...

Subject: PingER Android Team

Abstract for PingER on Android

Progress Made So Far

So far Shivnarayan Rajappa and Rohan Sampson’s team have succeeded in making a bare-bones Android Application that can ping beacons, parse data, and generate a text file in a format specified by SLAC ready for uploading. The proposed model involved the application pulling the beacon list from SLAC’s servers for pinging. However, the present application has a small percentage of the beacon list hard-coded into the application. As of now, the link between the application and proxy server has not been established.

...

By the end of the project, the server will be able to handle multiple hosts which would all forward it data, and it would in turn reorganise it again for SLAC’s servers to pull.

Amity (moved here 4/13/2017)

  • The paper on Implementation of PingER on Android has been accepted by IEEE Section. The paper to be online will take  5 months. 
  • Students are very interested in working with different projects. They have divided the students into three batches (each batch has min of 4 students).  The  projects currently  they are working are:
    • android, 
    • data analysis(vardha cyclone) 
    • and bigdata

Amity (moved here 3/12/2017)

The students successfully presented the paper on the PingER implementation on Android.at the confluence 2017 conference.. The paper is submitted to IEEE section. 

...

  • During the next 6 months their research will  study the impact Vardha cyclone that hit the Indian coast(South India/Chennai) and a few other neighboring countries in December 2016 as seen bu PingER. 

    • The idea is to study and  analyze the PingER data during the corresponding time frame  and deduce significant trends and patterns from the data using

    •  1. Clustering techniques
       2. Time series 
       3. Correlation and Regression concepts

Amity - Java approach (A. Sai Sabitha and Shivnarayan Rajappa)
  1. They are using the native java tools, they are not running the pinger2.pl <http://pinger2.pl>  script on android since the native java tools have the following advantages
    1. easier for user, 
    2. no need for prior installation of any software, e.g. load perl interpreter which may require missing skills, especially for a non technical user
    3. doesn't need a rooted phone
    4. only the apk needs to be installed to run

  2.  They have fixed the final sequence number change by using regex, and  pushed these changes to github repository.

  3. They have installed apache tomcat in the server and plan to use a java file on the server which would connect to the phones that send the request. This java file will then take the input stream received from the phone and write the output stream to a file that would be stored on the server. We are facing some problems regarding a blocked port that is not allowing the phone to connect to the server we are currently working on resolving the issue.

  4.  SLAC can then regularly pull these files which would be stored based on the month they are received. 

  5. The Android students have started writing a paper on " implementation  of pinger  on android " .

  6. Next steps:

    1. Extend the target list by getting the Beacon list from SLAC. It is at http://www-iepm.slac.stanford.edu/pinger/pinger.xml on a regular basis and updating the <BeaconList> section at their site. This was part of pinger2.pl. 

    2. Also they will need a utility to clean out old recorded data (say older than 3 months), since it will be gathered from SLAC (via the proxy) and eventually they may run out memory on the Android.
Discussion

To a large extent it depends on how we plan to use this.

...