Versions Compared

Old Version 1

changes.mady.by.user Les Cottrell

Saved on

compared with

New Version Current

changes.mady.by.user Les Cottrell

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Invitees:

Wajahat Hussain (SEECS)-+, Saqib+ (GZHU); Johari+ (UNIMAS);  Adib? (UUM)+; Fizi Jalil (MYREN);  Dr. Charnsak Srisawatsakul? (Ubru), Les+, Bebo+, Umar+

+ Confirmed attendance

- Responded but  Unable to attend: 

? Individual emails sent

Actual Attendees

Wajahat, Saqib, Johari, Charnsak, Les, Bebo, Umar

Saqib had problems with his connection and was able to hear but not be heard.

Others

Administration

Amity (Updated 5/28/2018)

  • Their PingER MA is up and running and we are successfully gathering some data.  
  • However, there is a problem with not being able to find data in the requested time window.
  • There is also a problem with a missing beacons.txt.
    • Les is working with Amity, emails We are unable to gather data from the Amity MA, or even ping it, emails sent May 28th and June 3rd, 2018, June 30.
  • Need to load the most recent version of gathering application (ping_data.pl) to assist in diagnosing problem. 
  • Check if Amity can upload beacons.txt table from SLAC.  They say they can.
  • The Android version of the PingER MA 
    • We are looking at how to store the data and set up a proxy 

Bebo (No updates: 3/8/2018, 5/3/2018)

Looking into moving PingER to a "blockchain" database good for decentralizing distribution of data. Monitoring sites would then be able to write to a distributed ledger. This would change the architecture to a more peer to peer architecture. It helps with continuity of PingER since reduces dependence on a single site (SLAC). See Block Chain in Future PingER Projects. Bebo sent several references to Saqib who has looked at them. We could start with real-time data without including the whole archive, i.e. in parallel to the continued centrally managed archive. It would be a private Blockchain and hence not be as compute intensive as a public blockchain.  Bebo's impression is that Saqib will lead in putting the ides in his paper into practice. Saqib will need some students.  Saqib's boss is going to the NY City meeting.

Thailand (Updated 6/3/2018)

For his IPv6 monitoring Charnsak is using ~ 100 IPv6 targets from Saqib. They are in the <HostList> of the Ubra PingER MA. It can be seen from http://www-wanmon.slac.stanford.edu/cgi-wrap/pingtable.pl?file=average_rtt&by=by-node&size=100&tick=monthly&from=TH.UBRU.CS.AC.PINGER6&to=WORLD&ex=none&only=all&ipv=all&dataset=hep&percentage=any. It is still running smoothly.

Charnsak is looking at a host in Chan Parsa province in Laos as a potential site for a PingER MA. Charnsak just got approved to make contact with the Champasak University. He expects to set up the MA in the next 4-5 months. It also depends on the partner university. 

Charnsak would like to have write access to parts of the PingER Wiki site. Les investigated and it appears this can be done, even if Charsack does not have a SLAC account. Les sent Charnsak the relevant information after the previous meeting.  Charnsak filled out the form and submitted. However, it went to the wrong person for approval.  This has now been fixed and he should be able to move forward. Charnsak has tested and it is working.

UUM (Updated 6/7/2018)

Regarding the paper " Socioeconomic Development indices and their Reflection on Internet Performance in the ASEAN Countries ", Adib tried the Elsevier Telecommunications journal. They responded and Adib commented:

  1. insufficient understanding of the complexities in the causality argumentation and related findings. 

    Adib responds: we have addressed IEEE access reviewers' comments carefully. complexities and even validation can be further enhanced, but our data is a bit limited, we have only ten values for each index. We have no data available for ALL indices in 2017 (the year this study was conducted), not even 2016. Values of all social economic indices are only available in 2015. I would suggest considering this point in our next paper, where we can update our internet performance data to 2018, and hopefully other indices data are available as well.

  2. the relevant previous literature is not well addressed 

     Adib responds: Actually, we reduced LR session to minimize the paper size so it can meet the journal requirements. Adding more LR can be done, but then the size will be increased again. In fact, not many close and recent papers were published in this particular area. so it is not necessary to add more LR, especially we are not comparing our final model with them. 

  3. As a minor but still disturbing deficiency, references to the cited papers are incomplete. 
    Adib agrees and it is already amended

Adib will submit the paper to World Development: https://www.journals.elsevier.com/world-development

We were unable to gather data from pinger.uum.edu.my. It appears to have been fixed.

NUST: (Updated 5/3/2018))

The web page http://maggie.seecs.nust.edu.pk was not working, email sent to Wajahat 6/3/2018, it has been re-discovered and is working

No intern has joined Wajahat's Lab. So there is not much progress.

Wajahat proposes to get a list of the new Universities in Pakistan and contact them encouraging them to participate in PingER and set up MA. They have made a list of new university sites, communications networks, Labs in different regions of Pakistan (especially the remote regions) and will make contact.

  • The list of new universities is ready. Just need resources to engage them. 
  • Unsure how this is affected by lack of interns.

Discussion items

Wajahat is hoping to get few students to work on Master thesis related to pinger data. If possible kindly share ideas related to data analytics which he can share with students

  • Suggestions anyone?

 Wajahat also asked:

  •  “Has any work gone into to predicting the cause of failures(blackout, flood, coup (Turkey)) using pinger data?”
  • Les> there have been several case studies looking at the impacts of failures, however nothing on predicting failures. I am not sure how one might use PinGER to predict the cause of a failure. For some cases (e.g. earthquakes, tsunamis, flood, coup), it is unclear to me how Network performance monitoring could add to other means of predicting the cause.
  • Wajahat's thinking is along the lines of "Regarding other sources, I was thinking, internet is the pulse of digital world. Other sources require additional setup that might not be a possibility in developing countries. Internet being a necessity and having other uses is still prevalent in these developing countries. Can it be used as a real time sensor."
  • Again I am forwarding the question in case others in the team may have some suggestions?  Below are some responses:
    • Umar mentioned the use of dark fibre testing and monitoring for earthquakes. There was a recent paper.
    • The idea of raising alerts from anomalies gives rise to the need for 3 types of alerts: Informational; need action but not critical; severe, wake person up etc. 
    • Things to look at include changes in a metric (e.g. minimum RTT), route changes, loss of connectivity. Correlation with multiple metrics in particular significant route changes and changes in RTT. A requirement is to get few false positives, and false negatives and then triage. Also apply some hysteresis so do not get multiple alerts for the same event.
 
SLAC was unable to gather data from:
  • The Android version of the PingER MA 
    • email sent June 5th and again June 30th:
      • Now that you have support for regularly updating the Beacons, one of the next things is to think about is how to get the data to the repository/archive (SLAC).
        A possibility is outlined in https://confluence.slac.stanford.edu/display/IEPM/Proxy+support+for+PingER.  Perhaps your team might be interested in pursuing step 1.  If necessary we could also think about providing an account at SLAC to enable a student to work on step 2.
      • The students have gone on the summer break, they will be back by the 2nd week in July.

Thailand (Updated 7/5/2018)

For his IPv6 monitoring site pinger6.ubru.cs.ac.th Charnsak has a pinger.xml configuration file with over 160 IPv6 targets. However, there is a huge discrepancy since according to pingtable.pl there are only about 13 targets responding. We need to get the latest pinger2.pl measurement agent script installed at Ubru so we can get better logging and see why the other hosts are not being monitored. Les and  Charnsak are working on this.

Charnsak is looking at a host in Champasak University, Chan Parsa province in Laos as a potential site for a PingER MA. Charnsak just got approved to make contact with the Champasak University. He expects to set up the MA in the next 4-5 months (say towards end 2018). It also depends on the partner university, and there may be a lot of paperwork.

UNIMAS (updated 7/5/2018)

Need to add Umar Kalim to http://pinger.unimas.my/pinger/contact.php. Johari can't ssh into the server so he will go to it on Monday.  He will also upload the new UNIMAS PingER website next week.

UUM (Updated 7/5/2018)

Les has sent Adib updates to Figs 3, 4, 5 to extend out to 2018. This is for the paper  Socio-economic Development Indices and Their Reflection on Internet Performance in ASEAN Countries

Adib will submit the paper to World Development: https://www.journals.elsevier.com/world-development

NUST: (Updated 7/5/2018))

No intern has joined Wajahat's Lab. So there is not much progress. The students are away on summer vacation so no progress.

Wajahat proposes to get a list of the new Universities in Pakistan and contact them encouraging them to participate in PingER and set up MA. They have made a list of new university sites, communications networks, Labs in different regions of Pakistan (especially the remote regions) and will make contact.

  • The list of new universities is ready. Just need resources to engage them. 
  • Unsure how this is affected by lack of interns.

Discussion item:

Wajahat is hoping to get few students to work on Master thesis related to pinger data. If possible kindly share ideas related to data analytics which he can share with students. 

 Wajahat also asked last time:

  •  “Has any work gone into to predicting the cause of failures(blackout, flood, coup (Turkey)) using pinger data?”
  • Les> there have been several case studies looking at the impacts of failures, however nothing on predicting failures. I am not sure how one might use PinGER to predict the cause of a failure. For some cases (e.g. earthquakes, tsunamis, flood, coup), it is unclear to me how Network performance monitoring could add to other means of predicting the cause.
  • Wajahat's thinking is along the lines of "Regarding other sources, I was thinking, internet is the pulse of digital world. Other sources require additional setup that might not be a possibility in developing countries. Internet being a necessity and having other uses is still prevalent in these developing countries. Can it be used as a real time sensor."
  • Again I am forwarding the question in case others in the team may have some suggestions?  Below are some responses:
    • Umar mentioned the use of dark fibre testing (e.g strain) and monitoring for earthquakes. There was a recent paper.
  • Umar was thinking that using the PingER data to detect and log anomalies and create alerts might be interesting.  This would include triaging the logs into 3 tiers of alerts:
    • Low = Information; 
    • Medium = needs action but non-critical, create a ticket to be looked at later; 
    • Severe: e.g. wake someone up.
  •  It requires hysteresis so one does not get continuous alerts for say something that has gone down. 
  • This makes a lot of sense for say routers and switches which are usually very critical. 
  • The PingER measurement script (pinger2.pl) has a lot of logs that are prioritized by severity (e.g. target not responding for various reasons for the last n runs of pinger2.pl.) 
  • Tiering of events into alert levels would be the focus of the research
  • PingER might be an interesting field to try out the marshaling of events and categorizing into alert levels/tiering. 
  • Possibly mine the data with say Splunk. 
    • Umar looked at this: Splunk would be useful for rudimentary for simple scenarios, but not for detailed work.
  • Might be an interesting project for an MS or PhD student.
  • For PingER it might include correlation with feeds from elsewhere (e.g. earthquakes, tsunamis, social unrest)
  • Prediction/tracking:
    • For unexpected events such as earthquakes/tsunamis, it is doubtful PingER data would be useful for predictions
    • For situations that are developing such as hurricanes, civil unrest: knowing roughly where it is developing might make it possible to temporarily increase the density of targets in the region. 
      • To do this would mean having a lot more targets available worldwide ready to be monitored.
      • One would need to be careful to ensure that the extra monitoring does not interfere with Internet access in the area.
    • This tracking might be very valuable for less developed areas where there may be few other (non-PingER) measurements available. However, it can be hard to find the coordinates of ping responding targets is such areas. This would need to be one of the tools developed.
 
SLAC was unable to gather data from:
  • 121.52.146.180 (kohat.edu.pk) down since Nov 22/2017. Wajahat recommends continuing at least until the new student is up to speed (3/8/2018). No data available 3/24/2018.
  • cae.seecs.edu.pk last time we were able to gather any data was February 27th.
  • pinger-ncp.ncp.edu.pk pings but can't gather data 8/11/2017 and 9/16/2017. Contacted. Pings but can't gather data 10/24/2017. They are in the process of restoring 1/17/2018. Still down February 28, 2018, await new student. 121.52.146.180 (kohat.edu.pk) down since Nov 22/2017. Wajahat recommends continuing at least until the new student is up to speed (3/8/2018). No data available 3/24/2018.
  • caepinger.seecsisra.edu.pk last time we were able  unable to gather any data was February 27th.
  • pinger-ncp.ncp.edu.pk pings but can't gather data 8/11/2017 and 9/16/2017. Contacted. Pings but can't gather data 10/24/2017. They are in the process of restoring 1/17/2018. Still down February 28, 2018, await new student. (3/8/2018). No data 3/24/2018.
  • pinger.isra.edu.pk unable to gather data since 3/6/2018, also does not ping.
  • Wajahat says they will get these nodes up. These have been good nodes. They just need the weekly push. NUST will push them soon.

UNIMAS (No update 3/8/2018, 5/3/2018)

UAF/GHZU (Updated 5/5/2018))

Saqib submitted the Camera ready paper on  “A Blockchain-based Decentralized Data Storage and Access Framework for PingER” and it has been accepted in Trustcom2018.

Two previously accepted papers are now online on the following links.

IPv6 node in Beijing is up and running for the last 7 weeks but not available from outside China. It does not appear possible to make the host accessible outside China. We are looking at alternates (e.g. using the anonymous FTP server at SLAC as a proxy) that may also be relevant for PingER on Android gathering of data.

Discussion item

Saqib sent an email to the team:

In early years of PingER, the framework was designed to check the latency and other Internet performance metrics between CERN and SLAC to facilitate the data transfer between the two sites.

I am thinking, is there any possibility to use PingER to monitor the health of the Bitcoin blockchain network? Since the latency is critical in Bitcoin blockchain network as all the incentives depend on the propagation of transactions and mined blocks. Thus, I am only interested in measuring the latency to check its effect on the propagation of the transactions and mined block among different mining pools. I think if we can do such thing on a historical basis as PingER already does for the Internet, it will increase the worth of the framework and its usability.

Maybe a few test experiments can guide us to a good research paper. I am not sure about the feasibility idea, therefore, need your kind feedback

Umar responded:

"This is an interesting idea. I would like to think about it a bit more thought before I respond at length.
From what I gather, nodes in the network may appear and disappear without notice. Therefore, it is safe to assume that the network state is changing at all times. I wonder, what is it that we would be measuring, for it to be meaningful. Would it be the latency between full nodes? Would it be the latency from a PingER monitoring node to the full nodes? Would the monitoring nodes be representative of typical clients? Is latency the metric to measure? What are all the projects  that measure latency or other metrics? (The bitcoin nodes project is interesting, which shows the size of the network. Similarly, the project that measures average-transaction-completion time is relevant. As you pointed out, the bitcoin stats about data exchange are relevant too.)
  • data since 3/6/2018, also does not ping.
  • Wajahat says they will get these nodes up. These have been good nodes. They just need the weekly push. NUST will push them soon.  Any update?

UAF/GHZU (Updated 7/5/2018))

Les has implemented, and we are now successfully using the anonymous FTP server at SLAC as a proxy for gathering data from GZHU MA in Beijing.  This gets around not being able to 'get' the data via the web from the Beijing MA, due to blocks. This may also be relevant for PingER on Android gathering of data, email was sent to Amity.

Looking into moving PingER to a "blockchain" database good for decentralizing distribution of data. Monitoring sites would then be able to write to a distributed ledger. This would change the architecture to a more peer to peer architecture. It helps with continuity of PingER since reduces dependence on a single site (SLAC). See BlockChain in Future PingER Projects. Bebo sent several references to Saqib who has looked at them. We could start with real-time data without including the whole archive, i.e. in parallel to the continued centrally managed archive. It would be a private Blockchain and hence not be as compute intensive as a public blockchain.  Bebo's impression is that Saqib will lead in putting the ides in his paper into practice. Saqib will need some students.  Saqib's boss is going to the NY City meeting.

Discussion item

Saqib sent an email to the team:

In early years of PingER, the framework was designed to check the latency and other Internet performance metrics between CERN and SLAC to facilitate the data transfer between the two sites.

I am thinking, is there any possibility to use PingER to monitor the health of the Bitcoin blockchain network? Since the latency is critical in Bitcoin blockchain network as all the incentives depend on the propagation of transactions and mined blocks. Thus, I am only interested in measuring the latency to check its effect on the propagation of the transactions and mined block among different mining pools. I think if we can do such thing on a historical basis as PingER already does for the Internet, it will increase the worth of the framework and its usability.

Maybe a few test experiments can guide us to a good research paper. I am not sure about the feasibility idea, therefore, need your kind feedback

Umar responded:

"This is an interesting idea. I would like to think about it a bit more thought before I respond at length.
From what I gather, nodes in the network may appear and disappear without notice. Therefore, it is safe to assume that the network state is changing at all times. I wonder, what is it that we would be measuring, for it to be meaningful. Would it be the latency between full nodes? Would it be the latency from a PingER monitoring node to the full nodes? Would the monitoring nodes be representative of typical clients? Is latency the metric to measure? What are all the projects that measure latency or other metrics? (The bitcoin nodes project is interesting, which shows the size of the network. Similarly, the project that measures average-transaction-completion time is relevant. As you pointed out, the bitcoin stats about data exchange are relevant too.)
Thank you for sharing the URLs. I will think about it a bit more and get back to you."
Bebo emailed:

  • More and more I am convinced that we could identify an interesting and relevant project taking advantage of our PingER experience and involving Blockchain technology.I also think that it may be possible for us to get funding/grant money for such a project which might clearly draw the attention of Blockchain entrepreneurs, conferences, and publications. It might also allow us to attract the collaboration of other institutions and/or universities that might otherwise have not been interested in the basic PingER goals and technology.

Though the transactional latency is important especially for achieving consensus, most of the latency is computational that is much greater than the communications latency. Possibly PingER RTT could be included in a measure of BlockChain health and status.  Bebo sent a reference (https://mastanbtc.github.io/blockchainnotes/consensustypes/) pointing out milliseconds may make a difference to miners to get rewards which is important for BlockChain cash
Saqib sent email of an analysis of the impact of various RTTs, see https://ethereum.stackexchange.com/questions/18201/does-network-latency-significantly-affect-mining-rewards.
A possible problem is finding the IP addresses of the bitcoin miners. Bebo dug up a URL showing how it can be done, it is at https://bitcoin.stackexchange.com/questions/66260/finding-ip-addresses-of-all-bitcoin-miners.
Bebo will get together with Umar and Les at SLAC on Monday the 9th around noon to discuss Bitcoin/Blockchain to do some education/brain-storming. Unfortunately noon PST is 3:00am China time. We will try and send Saqib an executive overview of the discussionThank you for sharing the URLs. I will think about it a bit more and get back to you."

PingER at SLAC (Updated 6/7/2018)

...

  • See Towards Analysis of ICMP vs TCP Ping Latencies - Umar
    • Looked into Traffic Differentiation - Rate Limiting vs. Traffic Prioritization (QoS)

    • IPv6 results gathered using ping-vs-tcp.pl script. About 56 nodes with IPv6 addresses, 14 of which responded with Npings

    • IPv4 results gathered from SLAC and Virginia Tech
      • SLAC's batch may be downloaded here (approx. 24 MB)
      • Skimmed results; findings are pretty much the same as before
  • Pending
    • Identified relevant events in the network stack that highlight timing (_RECVFROM, _RECVMSG, _IP_RECV, _NETIF_RX etc.). Looking for instrumentation that enables us to measure timestamps. We also need to figure out how to determine whether ICMP & TCP traffic are treated differently? and then how to measure the difference?
      • perf-tools allows us to measure transport events
      • If we could assume that the path for ICMP & TCP through the network is the same, then the only difference between two (controlled) tests would be the time spent in the transport layers. This can be measured using perftools. 
      • However, such measurements must be made in a controlled environment where ICMP and TCP are treated the same. (I say so because some results — e.g., in East Asia and South Asia — clearly show that ICMP performs much worse than TCP.)
    • We would also need to cater for cross traffic and queuing delays. Given how small the differences are, one may argue that the variations in measurements are due to cross traffic. Perhaps we should start with controlled tests and then see if real world measurements reflect similar behavior.
    • We need to setup a test environment. We can either setup a bare-metal box or use a VM. 
      • I will see if I can arrange for a bare-metal box.

PingER IPV6 support

  • Les added an item to the pingtable.pl form to enable the ability to select IPv4 or IPv6 measurements or both.

Meeting with Amazon Web Services

  • Dr Ali Khayam (ex NUST) is now at Amazon.  He has a colleague at Amazon (Awais Nemet) who heads the External Network Services group for AWS. Awais is kicking off many projects in the Internet monitoring space and Ali recommended that Awais talks to Les to learn about the experiences of operating Pinger and leveraging the data that Pinger generated.  Les updated the PingER front page at http://www-iepm.slac.stanford.edu/pinger/)  so he could walk Awais through PingER. 

     

HostStatelast seenStatus
pinger-ncp.ncp.edu.pkpinger-ncp.ncp.edu.pk downNov 29, 2017 
121.56.146.180 (pinger.kohat.edu.pk) DownNov 22nd, 2017 
cae.seecs.edu.pkDownFeb 27, 2018 
pinger.isra.edu.pkDownMarch 6, 2018 
pingeramity.inIt is partially working. Now working on missing beacons.txt file and missing data (i.e data disappears a few hours after it is measured and saved at MA). Also it is unable to access http://www-iepm.slac.stanford.edu/pinger/beacons.txtApril 27, 2018 

Next Meeting

Next meeting:  Thursday, July 5th 9pm Pacific time; Friday, July 6th, 2018  9:00am Pakistan time; 12:00noon Malaysian & Guangzhou time; and 11am Thailand time.

...

  • PingER now seems to be fully IPv6 capable and stable, the logs have been improved to make it easier to debug problems.
HostStatelast seenStatus
pingersonar-um.myren.net.myNo response6/26/2018Pings
pinger-ncp.ncp.edu.pkpinger-ncp.ncp.edu.pk downNov 29, 2017 
121.56.146.180 (pinger.kohat.edu.pk) DownNov 22nd, 2017 
cae.seecs.edu.pkDownFeb 27, 2018 
pinger.isra.edu.pkDownMarch 6, 2018 
pingeramity.inIt was partially working. Now working on missing beacons.txt file and missing data (i.e data disappears a few hours after it is measured and saved at MA). Also it is unable to access http://www-iepm.slac.stanford.edu/pinger/beacons.txt. Awaits Amity.April 27, 2018 

Bebo 

Discussion

Bebo sent email:

More and more I am convinced that we could identify an interesting and relevant project taking advantage of our PingER experience and involving Blockchain technology.

For example,

https://bitcoin.stackexchange.com/questions/66260/finding-ip-addresses-of-all-bitcoin-miners <https://bitcoin.stackexchange.com/questions/66260/finding-ip-addresses-of-all-bitcoin-miners>

I also think that it may be possible for us to get funding/grant money for such a project which might clearly draw the attention of Blockchain entrepreneurs, conferences, and publications. It might also allow us to attract the collaboration of other institutions and/or universities that might otherwise have not been interested in the basic PingER goals and technology.

Thoughts/responses?

Next Meeting

Next meeting:  Thursday, August 9th 9 pm Pacific time; Friday, August 10th, 2018 9:00 am Pakistan time; 12:00 noon Malaysian & Guangzhou time; and 11 am Thailand time.

...

Old information

GZHU China - Saqib (moved here 7/2/2018)

Saqib submitted the Camera ready paper on  “A Blockchain-based Decentralized Data Storage and Access Framework for PingER” and it has been accepted in Trustcom2018.

Two previously accepted papers are now online on the following links.

NUST (moved here 6/29/2018)

...

  1. They are using the native java tools, they are not running the pinger2.pl <http://pinger2.pl>  script on android since the native java tools have the following advantages
    1. easier for user, 
    2. no need for prior installation of any software, e.g. load perl interpreter which may require missing skills, especially for a non technical user
    3. doesn't need a rooted phone
    4. only the apk needs to be installed to run

  2.  They have fixed the final sequence number change by using regex, and  pushed these changes to github repository.

  3. They have installed apache tomcat in the server and plan to use a java file on the server which would connect to the phones that send the request. This java file will then take the input stream received from the phone and write the output stream to a file that would be stored on the server. We are facing some problems regarding a blocked port that is not allowing the phone to connect to the server we are currently working on resolving the issue.

  4.  SLAC can then regularly pull these files which would be stored based on the month they are received. 

  5. The Android students have started writing a paper on " implementation  of pinger  on android " .

  6. Next steps:

    1. Extend the target list by getting the Beacon list from SLAC. It is at http://www-iepm.slac.stanford.edu/pinger/pinger.xml on a regular basis and updating the <BeaconList> section at their site. This was part of pinger2.pl. 

    2. Also they will need a utility to clean out old recorded data (say older than 3 months), since it will be gathered from SLAC (via the proxy) and eventually they may run out memory on the Android.
Discussion

To a large extent it depends on how we plan to use this.

...