Page History

...

Actual Attendees

Wajahat, Saqib, Umar, Les, Bebo, Taha (Skype ID nightwingduck, email taha.ali@seecs.edu.pk)Taha 

Others

Administration

• Membership of pinger-my is in https://groups.google.com.
• Bebo pointed out that the International Web Conference will be in San Francisco in 2019 (see http://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=77195&copyownerid=85335).  Thereare a whole range of topics. He would love to see something submitted related to PingER.
  • Maybe something on Android PingER and potential uses.
• Bebo, possible interest in an MA in Jordan - 10/9/2018.
  • It is a university of Science and Technology.
  • He Bebo is looking to send the Home page: http://www-iepm.slac.stanford.edu/pinger/, plus  Remote Host site invitation letter template, Monitor Host site invitation letter template:(txt, doc), form to register a host to the contact. 
  • Bebo will look into inviting them to our next meeting on Nov 6th.

Action Items from previous meeting

• Future meetings are moved to Tuesdays Pacific time.
• Figure out how to move forward with the Android PingER project, in particular, schedule a meeting. 
  • Trying to set a date.
• Les: invited Dr. Taha to join the pinger-my email list.  Dr Taha joined 9/9/2018. Completed
• Wajahat: will provide a list of working or potentially working (e.g. being worked on) Pakistani MAs to Les. They are working on nodes as of email 10/6/2018.
• Saqib and Les: worked on being able to gather data from  2001:da8:270:2018:f816:3eff:fef3:bd3 IPv6 node in Beijing working again.
  • Les and Saqib added a lot of debugging capability, appears to be successfully working since 9/11/2018. - Completed
• Umar: update his section on the comparison IPv6 vs IPv4 ping RTTs and TCP vs ICMP/ping RTTs. Les sent reminder 10/3/2018., see below
• Johari: enter Umar and Dr. Taha into http://pinger.unimas.my/pinger/contact.php, Les sent reminder email 10/3/2018.
• Charnsak: We need to get the latest pinger2.pl measurement agent script installed at Ubru so we can get better logging and see why the other hosts are not being monitored. Les believes the current logging is sufficient. It would not hurt, in fact, provide a slight improvement, if it is updated. Completed.

...

• Amity MA is unreliable so using it for a case study does not appear fruitful. Les is working with Amity to try and understand this unreliability (emails 10/6/2018), they say "There is an internet problem we are taking care of it and will get back soon".
• The Android version of the PingER MA,is described with  comments at  ePingER on Android Native - Amity project (this a proposal/description from Aayush Jain)

  • It describes a multipurpose, stand-alone device that can be widely distributed, something that we have brainstormed about for a long time. 

    • We agreed to request Amity to share the App and instructions with us; we will look at installing on a jailbroken Android phone at the San Francisco end and try it out.

    •  They are working on a paper. They sent a  1st draft they have sent to Bebo and Les that we responded to 10/6/2018, a 2nd draft was responded to by Bebo and Les 10/13/2018. It has been submitted to EasyChair.

    • They will be attaching the android app for review after testing it again.

      • Topher  feels that the Android app (when completed and vetted by his team) could easily be installed as a default service on his rainforest monitors (certainly future ones, not devices already in place). Merging the service data that he already collects with that unique to PingER has the potential to lead to some interesting results.

    • Also August 16th and again September 3rd, then  October 6th and October 13th proposed a meeting between: Bebo, Umar and Les and Lesand the Amity folks.

      • 10/24/2018: Proposed: attend a regular one hour meeting of the pingER team: Next meeting:  Tuesday, November 6th 8 pm Pacific time (Nb now on winter time); Wednesday, November 7th, 2018 9:00 am Pakistan time; 12:00 noon Malaysian & Guangzhou time; and 11 am Thailand time, 9:30 am Indian time. The downside is that the focus would be on several PingER matters as well as Amity.  However, maybe that is a good way to get started. To do this we would need thier Skype ID. I found an ID Sai Sabitha in noida, India, and we appear to have a mutual contact.  I left a message.
      • The topics are: 
        • Amity's documents  describe a multipurpose, stand-alone device that can be widely distributed, something that we have brainstormed about for a long time.
        • Bebo mentioned it to Topher (the Princpal Investigator of the RainForest project) and he feels that the Amity PingER/Android  app (when completed and vetted by Topher’s team) could easily be installed as a default service on his rainforest monitors (certainly future ones, not devices already in place). Merging the service data that he already collects with that unique to PingER has the potential to lead to some interesting results.
          • How should we proceed? 
          • We (Topher and the PingER team) agreed to request Amity to share the App and instructions with us; we will look at installing on a jailbroken Android phone at the San Francisco end and try it out.
          • We also need to finish up the proxy acquisition of data from the Android MAs. A first step is outlined inhttps://confluence.slac.stanford.edu/display/IEPM/Proxy+support+for+PingER. 
          • A later step is to look at, evaluate and find resources to implement the ideas expounded in the Future Proposals section of the paper you are working on (see below)
          • Also encourage Amity to put together a paper.  This appears to have been completed.
        •  Google Firebase Application (see below).
    • The paper has an interesting suggestion for a way forward - TOPIC to discuss
      • Firebase Application: The need for a proxy server can be
      The paper has an interesting suggestion for a way forward - TOPIC to discuss
      • Firebase Application: The need for a proxy server can be completely eliminated by shifting to a cloud-based architecture for managing files. Instead of SLAC pulling in the generated txt files from MAs around the world, the MAs can themselves push these files to a centralized application server hosted on the cloud; which makes it easier for SLAC to access files as per their need.
        Considering the great degree of integration capabilities Google offers with its products, the flexibility arising from using the Google Cloud Platform would be pronounced. Google’s Firebase is a mobile and web application development platform that provides developers with a variety of tools and services to help develop high-quality apps, that can scale easily as per changing demands, and which delivers 99.99% uptime. The Firebase SDK allows mobile app developers to quickly add critical and reliable functionality to their applications in a short time. The recommended option here would be to leverage the Cloud Storage for Firebase that allows robust uploads and downloads onto the Google Cloud Storage buckets. Apart from the user authentication module that comes bundled with Firebase, developers can also declare file security parameters so as to allow only certain file types to be uploaded. Having all user uploaded files in one place will then enable SLAC to access and process files as and when needed. Server-side processing can be done on the Google Cloud Platform as well, thereby eliminating any need for SLAC to maintain its own physical infrastructure. This form of implementation can be highly beneficial in any kind of region of the world, also including remote places, like deep inside tropical rainforests, or places that have recently been hit by a natural calamity. As this implementation model is not dependent on any local measuring agent, the android mobile apps can directly deliver data to SLAC over any form on internet connection in minimal time with high reliability.. The Firebase SDK allows mobile app developers to quickly add critical and reliable functionality to their applications in a short time. The recommended option here would be to leverage the Cloud Storage for Firebase that allows robust uploads and downloads onto the Google Cloud Storage buckets. Apart from the user authentication module that comes bundled with Firebase, developers can also declare file security parameters so as to allow only certain file types to be uploaded. Having all user uploaded files in one place will then enable SLAC to access and process files as and when needed. Server-side processing can be done on the Google Cloud Platform as well, thereby eliminating any need for SLAC to maintain its own physical infrastructure. This form of implementation can be highly beneficial in any kind of region of the world, also including remote places, like deep inside tropical rainforests, or places that have recently been hit by a natural calamity. As this implementation model is not dependent on any local measuring agent, the android mobile apps can directly deliver data to SLAC over any form on internet connection in minimal time with high reliability.
        • Bebo agrees it is another reduction on independence from SLAC. 
        • It is unclear whether it would work for China since Google services are not available there.
        • Is it only free for low utilization. 
          • Might work for a cache for the latest data, but not for all archived PingER data (hundreds of GB). 
        • It was developed for Android applications, might be good for Android/PingER
        • There may be traffic limits.
        • Umar suggests try for one node and see how it works out in practice

Thailand (No update 8/9/2018, No update 9/6/2018)

...

We need to think about how to handle these hosts with both IPv4 and IPv6 addresses. I am leaning . We could do nothing, the simplest solution. Or we could move towards moving to the future and simply changing change the IPv4 addresses to IPv6 addresses. In a few cases we few cases, we may be able to find two hosts at a site or two names for the same host (i.e. an A and a CNAME record), using an IPv4 address on one and an IPv6 on the other. This is the next simplest solution. Doing this loses having a historical record of a target with both an IPv4 and IPv6 address. Umar and Les have some scripts that enable comparisons to be made between access to a host via its IPV4 and its IPv6 address without using the PingER data. An alternative would be to also have a pseudo name for such ambiguous hosts, another would be to modify the database schema. Both of the latter two would require changes to the code in several places.

Charnsak is looking at a host in Champasak University, Chan Parsa province in Laos as a potential site for a PingER MA. Charnsak just got approved to make contact with the Champasak University. He expects to set up the MA in the next 4-5 months (say towards end 2018). It also depends on the partner university, and there may be a lot of paperwork.

UNIMAS (No update 8/9/2018, no update 9/6/2018, no update 10/7/2018)

Need to add Umar Kalim to http://pinger.unimas.my/pinger/contact.php. From the 7/5/2018 meeting: Johari can't ssh into the server so he will go to it on Monday.  He will also upload the new UNIMAS PingER website next week.

Sent reminder emails 8/6/2018. 9/3/2018.

...

Another alternative might be to create a second record using either the IPv4 or IPv6 address as the name. This requires no changes to the code but is pretty unpleasant. If one could find a CNAME as well as the regular name (A reg) then one could use the CNAME for one of the addresses and the regular name for the other address. 

• Looking at a spreadsheet of  155 perfSONAR hosts from Saqib, Les found 3 that have a CNAME.  From these Les has added perfSONAR nodes  planchet.heanet.ie and perfsonar.ndsu.NoDak.edu  to NODEDETAILS the PingER meta database using the A record with the IP address and the CNAME with the IPv6 address.

Charnsak is looking at a host in Champasak University, Chan Parsa province in Laos as a potential site for a PingER MA. Charnsak just got approved to make contact with the Champasak University. He expects to set up the MA in the next 4-5 months (say towards end 2018). It also depends on the partner university, and there may be a lot of paperwork.

UNIMAS (No update 8/9/2018,

...

no update 9/6/2018

...

Les has sent Adib updates to Figs 3, 4, 5 to extend out to 2018. This is for the paper  Socio-economic Development Indices and Their Reflection on Internet Performance in ASEAN Countries

Adib will submit the paper to World Development: https://www.journals.elsevier.com/world-development

NUST:

Wajahat proposes to get a list of the new Universities in Pakistan and contact them encouraging them to participate in PingER and set up MA. They have made a list of new university sites, communications networks, Labs in different regions of Pakistan (especially the remote regions) and will make contact. 

...

• Wajahat will provide a list of working or potentially working Pakistani MAs to Les, and if they are not enabled, Les will enter or re-enable the gathering of data from these MAs.

, no update 10/7/2018)

Need to add Umar Kalim to http://pinger.unimas.my/pinger/contact.php. From the 7/5/2018 meeting: Johari can't ssh into the server so he will go to it on Monday.  He will also upload the new UNIMAS PingER website next week.

Sent reminder emails 8/6/2018. 9/3/2018.

UUM (No update 8/9/2018, No update 9/6/2018)

Les has sent Adib updates to Figs 3, 4, 5 to extend out to 2018. This is for the paper  Socio-economic Development Indices and Their Reflection on Internet Performance in ASEAN Countries

Adib will submit the paper to World Development: https://www.journals.elsevier.com/world-development

NUST:

• 121.52.146.180 (kohat.edu.pk) down since Nov 22/2017. Wajahat recommends continuing at least until the new student is up to speed (3/8/2018). No data available 3/24/2018. Disabled 10/3/2018. Wajahat emailed 10/12/2018.
  • pinger.kohat.edu.pk status is that the concerned person is on leave and the other person doesn't know any status of the system at site. 
• cae.seecs.edu.pk last time we were able to gather any data was February 27th. Disabled 8/19/2018. Wajahat emailed 10/12/2018.
  • cae.seecs.edu.pk status is that the system at node is totally out of order. The system isn't in working condition. The concerned person there requires full technical support on how the linux can be installed and used. He knows windows only. He also requires all the support on making the system run the pinger scripts etc.
• For several months we were unable to gather data from pinger.isra.edu.pk starting 3/6/2018, also it did not ping. Then it started working as of September 2nd, 2018 and we gathered data for 3 days. Failed again starting on Sept 5th. At this time (Sept 6^th) it was pinging, however, all pings failed (i.e. no response after 31 pings). Now (10/14/2018) it is not pingable.
• Wajahat now has a permanent staff engineer  so that person can be trained on how to manage MAs and possibly improve the installation process

UAF/GHZU (Updated 8/9/2018)

Saqib's future at GZHU will be much clearer after November, current contract expires February 2019. 

Gathering data is working since 9/11/2018.

Blockchain

• Looking into moving PingER to a "blockchain" database good for decentralizing distribution of data. Monitoring sites would then be able to write to a distributed ledger. This would change the architecture to a more peer to peer architecture. It helps with continuity of PingER since reduces dependence on a single site (SLAC). See BlockChain in Future PingER Projects. Bebo sent several references to Saqib who has looked at them. We could start with real-time data without including the whole archive, i.e. in parallel to the continued centrally managed archive. It would be a private Blockchain and hence not be as compute intensive as a public blockchain. 
• There was a meeting to discuss blockchain possibilities, see 20180709 PingER Meeting on Blockchains. 
• Bebo's impression is that Saqib will lead in putting the ides in his paper into practice. Saqib will need some students.  Saqib is OK with this. He has 2 masters students but they are working in different areas.  Maybe NUST can assist with this. Saqib's partner gave a talk/paper on work so far at the New York meeting on July 31st. The talk went fine but there were not many comments/questions.
• Saqib is pursuing PingER and Blockchain. He is looking at different references shared by Prof. Bebo and the implementation details using Hyperledger Fabric. Saqib is looking at making a test implementation. The blocksize will be 2MB-10MB. It does not appear to be computationally expensive. He will start testing with Internet of Things measurements such as humidity and temperature
• There was a discussion on the use of DataBases and whether they could be avoided by caches. Hyperlogic keys are not in an SQL DB, basically, it appears like a cache.  there was a question whether a 10MB block would be adequate for PingER. For example, PingER from SLAC has about 700+ targets, the measurements are each 30 mins (48/day) and for pings of 100B and 1000 Bytes i.e. 2*48*700 measurements and each measurement is ~ 140Bytes, so a day's sets of measurements from just SLAC is ~ 10MBytes. We could choose to ignore the 1000Byte pings which would reduce it to ~ 5MBytes/day. The latency of retrieving a block is proportional to the block size. Things will be clearer after the test set up is in use.

IPv6 measurements

• There are now several months of IPv6 PingER measurements from GZHU/BJ, UBRU and SLAC. It is time to think about in-depth analysis of the data.
• Umar has not made much progress on the comparisons of IPv6 vs IPv4 or TCP vs ICMP.  
• The PingER measurements would provide longer time spans where one might look for changes with time such as diurnal changes, impact of holidays, anomalies etc. 

PingER at SLAC 

PingER IPV6 support

• Les identified and sanitized a Cross Site Scripting (XSS) exposure in ping-data.pl. He sent out a notice to update. Saqib has updated. Unfortunately cannot see from outside the Great Wall of China.
• Les cleaned up the Beacons. Now have 1 or more Beacons in  > 158 different countries, all are working except one inSyria.
• There was a problem gathering data from  2001:da8:270:2018:f816:3eff:fef3:bd3 the Guangzhou IPv6 host in Beijing. Since it does not use the standard way of gathering data (ping_data.pl), but instead uses anonymous FTP, the standard way of debugging the problem does not work.  Les and Saqib worked on to provide SLAC access to various log files etc. via the anonymous FTP proxy. Added extra debugging to proxy gatherer, it is in production.

Raspberry Pi version 1 PingER MA at SLAC

• This was installed in mid 2015 as part of the ePingER project (see ePinger Project at SLAC). It has run flawlessly without manual intervention for over 3 years, automatically recovering from power outages etc.  The OS failed a security scan. Rather than upgrade etc., since it has served its time, we decided to decommission it and focus more on the PingER/Android project. Bebo I have the Raspberry Pi and SD card, if you want it.

Context:

• Is there any statistical difference between ICMP and TCP Ping? The context here is the Internet (not data center). This is important because the network stack is different (e.g., MPI over infiniband) and latencies are significantly less.

Questions:

• Why should we focus on minimum RTT instead of average RTT
  • Min RTT essentially reflects fixed delay, while average RTT subsumes variations and path load
• Are the R plots generated using minRTT?
  • Averages and computed. Min RTT is available. Scripts need to be updated to use minRTT.
• What is the breakdown of latency between endpoints?If there is a difference, is it because of the type or location of the source? What if the source of traffic was not SLAC?
  • Latency for an echo packet to travel up the stack and back down is about 3.75 micro seconds (see StackMap https://www.usenix.org/conference/atc16/technical-sessions/presentation/yasukata). As expected, this is negligible when considered with milli second latencies.The remaining components would be propagation and queuing delay. As we cannotbreakdown the two in a public network without using an active look like
    • To replicate use system tap. See: https://unix.stackexchange.com/questions/419449/how-can-i-determine-if-a-latency-is-due-to-a-driver-or-the-scheduler
  • pathchirp etc, we'll continue to consider these as a single component.
• Is there a correlation with the distance between the endpoints?
• Are the differences limited to a particular region? How do we determine/understand if traffic prioritization is implemented?
  • It may be that end hosts which are farther away have larger variances and thus the pronounced differences.
• Test in a controlled environment to avoid variables such as traffic prioritization, queuing delay due to cross traffic.
• Review the time series of latencies for both ICMP and TCP ping, instead of averages?
• Is there a difference between IPv4 measurements vs IPv6.

...

Wajahat says (7/10/2018): "I was discussing the blockchain related email that you sent earlier with Dr. Taha (security researcher at SEECS NUST). He was excited about it. He wanted to try a few things and had a few queries which I was not able to answer.  He might join us in our future meetings." DrTaha joined us for the previous meeting and the discussion follows:

• DrTaha joined this meeting. He has been working with BlockChain (BC) for 3-4 years and recently heard of Pinger
• He wondered about our interest in BC since there does not appear to be a strong security interest.  It appears to be mainly a storage issue. BC may be a hinderance since the data will be replicated.
• Bebo said the interest is more in the way of a decentralized system to reduce the dependence on SLAC.
• The proof of work is nothing like as complex as it is for the cryptocurrency.
• There are file storage technologies that are distributed but all the data is not at all the sites. In particular, the InterPlanetary File System (IPFS see https://en.wikipedia.org/wiki/InterPlanetary_File_System) was mentioned. There is a draft paper.
• Saqib recently presented a paper (to be published) that used the IPFS and BC to handle issues in the current PingER architecture. Saqib's mobile connection was very poor so he provided the overview below:
  • 1. In this paper, we proposed nearly the same idea as you mentioned in our last skype meeting. We proposed to store the metadata of PingER measurements on the Blockchain whereas actual data is stored off-chain using IPFS file system. 
    2. For redundancy, we stored the data of each MA at 3 different locations (links of them are stored in BC)  so that it can be successfully retrieved in case the target MA is offline.
    3. We already have a set of 50+ MAs, we are thinking of utilizing them as a peer in BC network.
    4. We are more inclined towards the use of permissioned blockchain (currently our focus is Hyperledger Fabric) where consensus is not computational expensive as peers are pre-authenticated.

  •  This is just an initial idea. I am looking forward to having your comments and feedback.

• MIT has been working on a system using IPFS together with a revocation capability (e.g. for student degrees/diplomas).

• PingER does not have a need for revocation or editing of data.

• 121.52.146.180 (kohat.edu.pk) down since Nov 22/2017. Wajahat recommends continuing at least until the new student is up to speed (3/8/2018). No data available 3/24/2018. Disabled 10/3/2018. Wajahat emailed 10/12/2018.
  • pinger.kohat.edu.pk status is that the concerned person is on leave and the other person doesn't know any status of the system at site. 
• cae.seecs.edu.pk last time we were able to gather any data was February 27th. Disabled 8/19/2018. Wajahat emailed 10/12/2018.
  • cae.seecs.edu.pk status is that the system at node is totally out of order. The system isn't in working condition. The concerned person there requires full technical support on how the linux can be installed and used. He knows windows only. He also requires all the support on making the system run the pinger scripts etc.
• For several months we were unable to gather data from pinger.isra.edu.pk starting 3/6/2018, also it did not ping. Then it started working as of September 2nd, 2018 and we gathered data for 3 days. Failed again starting on Sept 5th. At this time (Sept 6^th) it was pinging, however, all pings failed (i.e. no response after 31 pings). Now (10/14/2018) it is not pingable.

UAF/GHZU (Updated 8/9/2018)

Saqib's future at GZHU will be much clearer after November, current contract expires February 2019. 

Gathering data is working since 9/11/2018.

Blockchain

• Looking into moving PingER to a "blockchain" database good for decentralizing distribution of data. Monitoring sites would then be able to write to a distributed ledger. This would change the architecture to a more peer to peer architecture. It helps with continuity of PingER since reduces dependence on a single site (SLAC). See BlockChain in Future PingER Projects. Bebo sent several references to Saqib who has looked at them. We could start with real-time data without including the whole archive, i.e. in parallel to the continued centrally managed archive. It would be a private Blockchain and hence not be as compute intensive as a public blockchain. 
• There was a meeting to discuss blockchain possibilities, see 20180709 PingER Meeting on Blockchains. 
• Bebo's impression is that Saqib will lead in putting the ides in his paper into practice. Saqib will need some students.  Saqib is OK with this. He has 2 masters students but they are working in different areas.  Maybe NUST can assist with this. Saqib's partner gave a talk/paper on work so far at the New York meeting on July 31st. The talk went fine but there were not many comments/questions.
• Saqib is pursuing PingER and Blockchain. He is looking at different references shared by Prof. Bebo and the implementation details using Hyperledger Fabric. Saqib is looking at making a test implementation. The blocksize will be 2MB-10MB. It does not appear to be computationally expensive. He will start testing with Internet of Things measurements such as humidity and temperature
• There was a discussion on the use of DataBases and whether they could be avoided by caches. Hyperlogic keys are not in an SQL DB, basically, it appears like a cache.  there was a question whether a 10MB block would be adequate for PingER. For example, PingER from SLAC has about 700+ targets, the measurements are each 30 mins (48/day) and for pings of 100B and 1000 Bytes i.e. 2*48*700 measurements and each measurement is ~ 140Bytes, so a day's sets of measurements from just SLAC is ~ 10MBytes. We could choose to ignore the 1000Byte pings which would reduce it to ~ 5MBytes/day. The latency of retrieving a block is proportional to the block size. Things will be clearer after the test set up is in use.

IPv6 measurements

• There are now several months of IPv6 PingER measurements from GZHU/BJ, UBRU and SLAC. It is time to think about in-depth analysis of the data.
• Umar has not made much progress on the comparisons of IPv6 vs IPv4 or TCP vs ICMP.  
• The PingER measurements would provide longer time spans where one might look for changes with time such as diurnal changes, impact of holidays, anomalies etc. 

PingER at SLAC 

PingER IPV6 support

• Les identified and sanitized a Cross Site Scripting (XSS) exposure in ping-data.pl. He sent out a notice to update. Saqib has updated. Unfortunately cannot see from outside the Great Wall of China.
• Les cleaned up the Beacons. Now have 1 or more Beacons in  > 158 different countries, all are working except one inSyria.
• There was a problem gathering data from  2001:da8:270:2018:f816:3eff:fef3:bd3 the Guangzhou IPv6 host in Beijing. Since it does not use the standard way of gathering data (ping_data.pl), but instead uses anonymous FTP, the standard way of debugging the problem does not work.  Les and Saqib worked on to provide SLAC access to various log files etc. via the anonymous FTP proxy. Added extra debugging to proxy gatherer, it is in production.

Raspberry Pi version 1 PingER MA at SLAC

• This was installed in mid 2015 as part of the ePingER project (see ePinger Project at SLAC). It has run flawlessly without manual intervention for over 3 years, automatically recovering from power outages etc.  The OS failed a security scan. Rather than upgrade etc., since it has served its time, we decided to decommission it and focus more on the PingER/Android project. Bebo I have the Raspberry Pi and SD card.

...

Next Meeting

Next meeting:  Tuesday, November 6th 8 pm Pacific time (Nb now on winter time); Wednesday, November 7th, 2018 9:00 am Pakistan time; 9:30 am India time; 12:00 noon Malaysian & Guangzhou time; and 11 am Thailand time.

...