20201029 PingER Team Meeting

Time & date 

This meeting:  November 29 2020 9 pm Pacific time; a day later 9:00 am Pakistan time; 9:30 am India time; 12:00 pm Malaysian & Guangzhou time;  11:00 am Thailand time; 6:00 am Jordan and 5:00am Turkey.  

Coordinates of team members:

Mailing list: pinger-my@googlegroups.com for membership see  https://groups.google.com.

Invitees:

Wajahat Hussain?, Kiran Liaqat- (SEECS), Saqib+ (UAF), Bebo White+, Umar Kalim+, Les Cottrell+, Johari (UNIMAS);  Adib (Turkey), Dr. Charnsak Srisawatsakul (Ubru), Eyad Ayoubi (Turkey), Baraa Muslmani ( Jordan), Dr. Shadi Jawarneh (Jordan)

•  + Responded via Doodle can attend; - Responded, but unable to attend on this date; ? extra email sent asking if attending 
• Email addresses (all the following are in pinger-my). The link for the Doodle poll is https://doodle.com/poll/9wc4hxwakqk74wkh
  • cottrell@slac.stanford.edu; wajahat.hussain@seecs.edu.pk; saqibutm@outlook.com; bebo@slac.stanford.edu; umar.kalim@gmail.com; kliaqat.msee17seecs@seecs.edu.pk; johari.abdullah@gmail.com; adibhabbal@karabuk.edu.tr; charnsak.s@ubru.ac.th; eyadayoubi@gmail.com; b.muslmani@yahoo.com; shadi.jawarneh@yahoo.com; xsaifahmadx@gmail.com

Actual Attendees

PingER Project Update using Zoom

IMPORTANT NOTE: The meeting is set up to record automatically. By joining the meeting you are agreeing to be recorded (see details) 

Bebo White is inviting you to a scheduled Zoom meeting.

Join Zoom Meeting
https://us02web.zoom.us/j/82571424573?pwd=dTJWSmU2TEdPd056ZGRENUJqMityZz09

Meeting ID: 825 7142 4573
Passcode: 874566
One tap mobile
+16699006833,,82571424573#,,,,,,0#,,874566# US (San Jose)
+13462487799,,82571424573#,,,,,,0#,,874566# US (Houston)

Dial by your location
+1 669 900 6833 US (San Jose)
+1 346 248 7799 US (Houston)
+1 253 215 8782 US (Tacoma)
+1 301 715 8592 US (Germantown)
+1 312 626 6799 US (Chicago)
+1 929 205 6099 US (New York)
Meeting ID: 825 7142 4573
Passcode: 874566
Find your local number: https://us02web.zoom.us/u/kccv8XfFPn

When	Thu Oct 29, 2020 9pm – 10pm Pacific Time - Los Angeles
Where	https://us02web.zoom.us/j/82571424573?pwd=dTJWSmU2TEdPd056ZGRENUJqMityZz09 (map)
Calendar	cottrell@slac.stanford.edu
Who	Yes: 4  No: 0  Maybe: 0  Waiting: 3  Optional: 0 bebo.white@gmail.com - organizer cottrell@slac.stanford.edu   johari.abdullah@gmail.com   saqibutm@outlook.com  NO kliaqat.msee17seecs@seecs.edu.pk  ? wajahat.hussain@seecs.edu.pk umar.kalim@gmail.com

How to set up and use Zoom:

To use the software, you would have to download the Zoom client (and installed if prompted). The instructions to do so are listed here: https://zoom.us/download#client_4meeting

The instructions about setting up the software are listed at the URL below. You may want to create an account if you do not have one already. If you are invited to a meeting (i.e., you are not hosting the meeting), you are not required to create an account; you can simply join the meeting using the meeting ID.

We had a great meeting as Bebo put it "It was a good mixture of technical brainstorming, future planning, and personal exchange."

Amity 

Amity team photo:

Apart from invitations to present papers at upcoming conferences we have not heard from Amity since Sep 4, 2020 when they forwarded an email originating July 11, 2020

Earlier they had said

• "We are still working on the portability of the app from Android to Flutter, which will enable the app to be downloaded in iOS environment along with Android. Currently, our application is supported by Android only. Samaksh is looking forward to this issue. We are also discussing how the location feature will add functionalities to our application, and the PingER project will be more than just static servers and we can better monitor the internet with growing mobile communication. Shivang will mail you the list of all the proposed ideas we have on our mind, on what we can do with our mobile application. We will forward the list to Les Cottrell Sir and Bebo White Sir and get it discussed with them too."

  "One more thing I wanted to ask you is, can we have the ideas to be working on, or the subdomains, some particular topic we should pursue? Because there are a lot of facts we collected alone in the Coronavirus topic itself, and we need something more specific to work upon, like monitoring some particular countries, the government transparency on data, or measuring the impact the pandemic has on each country on a global scale. Please guide us into this. It would be a great help."

In response we sent them an analysis of the impact of Covid-19 on PingER measurements. See the full report so far (main focus is on Africa and to a lesser extent South America) is at PingER and Covid-19 by Region. At the time we suggested "A possible project for Amity might be to extend the detailed analysis to South Asia. We would be happy to assist as needed."

They responded: 

•  People's movement and the wave patterns can give us great insight into how the virus is spreading and its impact. With the advent of festivals in India in coming months, we are afraid we may be nearing a wave. We are trying to find its relation with various other factors, one being internet performance. The government has imposed various lockdowns and unlocking with different rules, we are also going to study its direct impact with COVID-19.

Concerning the Android PingER app, the July 11 2020 email said:

• We have successfully implemented the Ping through our mobile application, and the data is saved in the default PingER format, with 2 fields in addition, the latitude and longitude of the place.

  An android application was implemented before, but this time we made a Flutter app, which makes it possible to distribute the app to both Android and iOS platforms via a single source code.

  Regarding the format of data stored, we can only store it in JSON format, because it is the most convenient and best way to store. However, we made a python script which can parse the JSON data to Plain Text format and give the output.

  In addition to functionality of the previous app, we are also storing the data locally on the user's phone, which is plain text format in a file, under the folder //Internal Storage/SLAC_Pinger/<date> <time>.txt

Concerning the Android App, Umar pointed out that we need to understand/specify the end goal of the project. The project can support mobility since it records GPS coordinates. One needs to decide how to utilize the data gathered. It can be useful for evaluating Telco service providers. Unlike today's PingER which has continuous data between monitoring stations and targets, we would expect the mobile data to be much more sparse both in terms of time and monitors (cell-phones). Further, it is community-driven rather than centrally. It would appear to be similar to Thousand Eyes or Speedtest app that enables cell phone users to make on-demand measurements of throughput from their phones to targets around the world. The measurements would be archived and can then be summarized and reports presented, e.g. comparing connectivity from area to area on maps and or by a carrier. Telcos are also making such connectivity measurements for their network. An advantage of the current proposal is that it would be across service providers, and also have less network impact than say Speedtest. 

We agreed that Les would contact Amity to understand their current status, in particular with respect to the above project (e.g. who is leading the project and involved at the Amity end, the current status) and if relevant suggest setting up a Zoom meeting between the Amity folks and ourselves.

Saqib plans to continue work on PingER monitoring for remote areas, in particular in Pakistan. He will use Android PingERs. This may partially tie into the Amity project.

8/31/2020 there was an exchange of emails: Saqib wrote: I" am healthy and virus-free, however, things turned quite hectic on my end due to online lectures and assignments. Now things are getting better. I have sent a message to pinger.gcwuf.edu.pk to turn the server live again. Further, I am trying to figure out a student to look at the possible impact of Covid-19 interventions on Internet responsiveness as measured by PingER." 

At the meeting, Saqib clarified that the student(s) would be under-graduates, and they would be available in the next 1 to 2 months. This could be an opportunity for collaboration between the UAF team and Amity on the PingER cell phone App.

Saqib mentioned that when he installed PingER in Guangzhou there was pushback from some of the Chinese sites monitored concerning the continuous pinging. We are still monitoring about 100 targets in China.

Covid

Saqib

People do not appear to be cautious and not strictly following guidelines. Yet at the same time the impact in terms of number of confirmed cases for Pakistan compared to India is very different:

While the median age for Pakistan is quite young (late 20's to early 30's) and that the fact the population leans to B+- rather than A+- blood both should help reduce the incidence, this would not appear to be a big difference between India and Pakistan. Saqib mentioned that a difference might be in the way Pakistan imposes a hard 2 week lockdowns  (enforced by police with barricades) on small areas (e.g. a street in Islamabad) where there is an outbreak.  There may also be some under-reporting.

Saqib also reports that as of the middle of September graduate research students can make short visits to their labs and are resident on campus, while undergraduates are studying remotely.

Umar

Concerning the impact of Covid-19 on work, Umar said that compared to the old regime  of being physically at work working, or at home on personal business, compared to today's lack of boundaries where all work is at home, is quite exhausting. For example, the children are not used to parents being at home all the time, communications come during meals, there is a need to indicate when one is not available etc.

Les

Working on analyzing the data from Johns Hopkins University on confirmed cases. and deaths with a library of reports at https://www-iepm.slac.stanford.edu/pinger/excel, including:

• US Politics:
  • The impact of Trump's political rallies on spreading Covid-19 
    • Also see Covid-19 US cases by counties following political rallies
  • Cases in US states: in favor of, or  leaning to, or undecided  for Trump or for Biden
  • States with Republican governors
  • States with Democratic governors
• Cases in: Europe: Central,  Northern, Western, Eastern, South-West, 
• Cases in: East Asia, Middle East, South Asia, South West Asia
• Cases in: Central America
• I will be adding South America and Africa
• US:
  • San Francisco Bay
  • Compared to Western Europe
  • US most populated states

There are instructions at Covid-19 - Visualizing the data with Excel (https://confluence.slac.stanford.edu/display/IEPM/Covid-19+-+Visualizing+the+data+with+Excel)

Next Meeting

Next meeting:  There will be a Doodle poll, for  2020 at 9 pm Pacific time; a day later 9:00 am Pakistan time; 9:30 am India time; 12:00 pm Malaysian & Guangzhou time;  1:00 pm Thailand time; 6:00 am Jordan time. 5:00am Turkey time.  

Old information

SLAC moved here 28 Oct 2020 

• Bebo posed the question of whether PingER can identify changes in Internet responsiveness as measured by PingER as a result of the impact of the Covid-19 virus.
• The first attempts looking et PingER statistics and see if there were any noticeable changes in performance attributable to Covid-19 interventions (e.g. shutting down schools,  social distancing, gathering cancellations, lockdowns), were not very successful (see PingER and COVID-19). See https://covid19-interventions.com/ for interventions in European, and some Asian countries.
• Another attempt to correlate the impacts might be to use a tool such as Google Motion Charts to look for correlations between say Covid-19 cases (cases, hospital admissions, deaths etc) and PingER metrics as a function of time, by country, continent, plus population or density of population. The idea is that as Covid-19 cases pass say 100, the country will impose interventions such as school closing,  social distancing etc. which will change the usage patterns of the Internet which may have noticeable impacts on the PingER measurements.
• Unfortunately Google Motion metrics relies on Flash and is no longer supported .
  • A possible alternative is Charta.ca
  • I have therefore put together a Perl script (chart-covid.pl) to take daily PingER metrics for the last 120 days together with up-to-date raw data on the coronavirus from the Johns HopkinsUniversity GitHub site: https://github.com/CSSEGISandData/COVID-19, for various countries with their population, area, population density and cast the data in a format suitable for Charta.ca.
  • There is a web page I put together on how to use Charta.ca at Using Chart.ca.
  • There is an example at https://charteca.blob.core.windows.net/pubs/3Z3qcBXqCT9B3WHz
  • Instead of using Covid-19 confirmed cases, it might be good to use the Unacast cell phone location data to get a measure of how people are social distancing. I have sent them two emails but no response.
    
• Discussion:
  
  • Les pointed out that many of our targets (and of course the SLAC Measurement Agent) for the US (ESnet and Internet2), Europe (Dante, and national education/research networks etc.), Australia (AARNET) and parts of Asia (TEIN) are on well-provisioned, lightly loaded networks so it is the transit networks are not expected to see big changes.
  • Umar pointed out that the closing of schools, people staying at home should increase the traffic (e.g. between students and profs teaching remotely, people surfing the web watching videos etc) on CDNs (Content Deliverable Networks), and much of this traffic is on public networks. Thus one might see big changes for traffic transitting public networks. 
  • Since PingER is not monitoring homes and content deliverers, PingER would not expect to see big changes in responsiveness for most of our targets in the above regions.
  • On the other hand, African networks are less well developed and PingER may see more of an effect for such targets.

NUST moved here 4/14/2020

Pv6 host at NUST:

Wajahat has requested Hasan to install PingER on an IPv6 host at NUST, Saqib has contacted Hasan to tell him where to find and upload the code. There is not a host name for the MA yet. Hopefully, this will be available by next month's meeting.

•  7/24/2019 Les emailed Wajahat with a request for an update.
• At the meeting Wajahat had asked the student to contact Saqib and get the information on energizing IPv6. Saqib provided the information, but the student could not access the URL (apparently the URLwas down). Wajahat was on leave and when he returned the internship had expired, so Wajahat is unsure what happened. Wajahat will start a new student next week and will set him on the IPv6 project. Progress? 

Sent email on 2/4/2020 to Kiran at NUST on how to bring the vu host's pinger.xml up to date. As of 2/3/2020 pinger.xml at vu now looks good and we now appear to be able to gather good data.

SLAC moved here 4/13/2020.

The new version of Linux CentOS7, does not appear to support the following modules that are used by the PingER analysis.

• The Oracle database interface. PingER uses Oracle so there could be a problem. 
  
• The perl statistics module
• I will restrict the PingER analysis jobs to the earlier version of Linux (RHEL6) to avoid these problems. I am working with the systems folks at SLAC and looking into when the current version of Linux at SLAC (RHEL6) will no longer be supported and following SLAC's planning on how or whether to address the issues.
  • Rhel6 has full support until Nov 30, 2020. It then goes on "extended" support until June 2024. I expect SLAC will continue to use it during the extended support phase, at least for a couple of years.
  • Umar did some research and found:
    • I did a cursory search and it appears that in addition to having the Oracle database server, we would also need the items listed at the bottom. 
      If I were to test, I'd consider setting up a VirtualBox VM on my laptop and install CentOS7, Oracle database, instant client, Perl, and Perl driver for Oracle and test in the VM. I also searched to see if I could find some reference of "Oracle driver not available for CentOS7", but I couldn't find any. 
      I would recommend testing in an isolated VM so that we don't break the existing setup.
      1) Oracle instant client
      https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html
      2) Oracle DBD - last updated in 7/2019
      https://metacpan.org/pod/DBD::Oracle
      https://metacpan.org/source/MJEVANS/DBD-Oracle-1.80/README
      Here is a good breakdown that explains the relevant pieces (ignore the references to MySQL here). With testing we'll be sure what works and what doesn't.
      https://stackoverflow.com/questions/1958069/what-is-the-difference-between-dbi-and-dbd
      May be I am missing something, but I do not understand why there won't be a perl driver for Oracle on CentOS7.

Umar moved here 1/10/2020

Motion charts at https://www-iepm.slac.stanford.edu/pinger/pinger-metrics-motion-chart.html was failing it no longer displayed the contents of the charts. Umar looked at the problem and suggested a solution.  Les applied solution but did not fix in Chrome or Firefox.  See PingER Metrics Motion Chart#October2019 for details. Les finally discovered and fixed a few problems with prm.pl, create-motion-charts-page.pl, create-motion-charts-pages.pl. It is now working again but requires enabling Flash. Discussed with Umar, it would be possible to make the web page not use Flash, but it could be non-trivial. An alternative might be to use a similar tool called Explorer (see https://www-iepm.slac.stanford.edu/pinger/explorer.html). The web is very valuable for identifying anomalous behavior which may require filtering the data.

Unfortunately Umar has not had any cycles to look into IPv6 vs IPv4 study. I suspect we should give up on this.

Les moved here 1/10/2020

The talk on PingER at the Silicon Valley Samsung HQ 7/17/2019, went well, see link below. We can use this talk for the Internet Archive.

• Long term Monitoring of the World's Networks, Les Cottrell Invited talk at the Samsung HQ in Silicon Valley, Sept 17, 2019
• A video is available at https://www.samsung.com/us/ssic/session/monitoring-the-worlds-internet/ .

UNIMAS moved here 1/10/2020

MA is down, email sent asking for update 10/13/2019. No response by 11/3/2019,  MA has been Disabled.

Thailand, Charnsak 10/15/2019 mioved here 1/10/2020

MAs are down. Charnsak (by email 10/22/2019) said: "They have changed the Networking policy. The pinger may need to wait for 5-6 months for me to go back (from post-doc at University of Vienna) and change things myself." Charnsak is in Vienna for several months. Both the IPv4 and IPv6 MAs were disabled 11/5/2019. Gathering the data from the MAs has been disabled.

Bebo moved here 1/10/2020

Bebo noted the recent unrest in Iran and asked whether the impact could be seen in the PingER data. Les investigated.  The impact is very noticeable see PingER Iran Internet interrupted November 2019. 

Bebo wondered if we could detect such impacts automatically. Les pointed out it may be possible. A problem maybe avoiding a large number of false positives. Some work was been done on detecting similar anomalies in TCP throughput data.  See for example Anomalous Event Detection that included the Plateau project. This would be an interesting project for a student.

Bebo Moved here 11/20/2019

Is wondering about a meeting of PingER collaborators with possible funding from the Internet Archive or possibly others. Also possibly seeking other sources of funding for PingER activities.

It is time to re-raise and discuss the talk for the Internet Archive, and what we might expect from them and how much effort it is to prepare, document and make the data available to the Archive. So far:

• Les and Bebo visited the Internet Archive (IA, see https://archive.org/) and initiated a discussion on archiving PingER data (i.e. a mirror for SLAC) at the IA and also giving a talk on PingER at the IA. 

  We agreed to put together a disk with a snapshot of the appropriate archive files containing analyzed data back to 1998 plus some documentation and then approach them about archiving the data, and a possible talk showing how the data can be used, e.g. comparison with economic indices, identifying on the network on events such as earthquake, societal unrest, cutting of cables etc.  First part of the talk based on the presentation at Samsung. Depending on level of interest and support, we could . offer to update the data.
  • The IA responded with information on using an IA Python library to archive the data at the IA. Les responded with interest but lack of resources until mid-September at the earliest.  Also maybe after giving a talk at IA there might be interest in collaborating from some of the attendees.
  • The IA also responded pointing to a form to fill out for a request to give a talk at the IA.
    • Les filled out the form.
    • Les sent an email to the IA to make sure they had received the form and asking if anything else is required.
    • IA suggested September, given the China meeting, this was missed.

Adib  (Updated 7/22/2019) Moved here 11/7/2019.

•  Adib submitted to "Socio-economic Development Indices and Their Reflection on Internet Performance in ASEAN Countries" to Journal: Engineering Science and Technology, an International Journal. They have requested a re-submission since it is over the limit of 10 pages. Adib re-submitted 7/22/2019.
  • 8/26/2019 status of this submission has changed to Revision Requested.

Exit plan for PingER moved here 10/10/2019

• The general opinion at the July 2019 meeting was that we move to the Soft stop for PingER
  • Continue running monitors, gathering data, archiving and presenting  as long as SLAC provides storage (1TB) and pinger.slac.stanford.edu VM
    • Give up on monitors after email prompt and no resolution
    • Do not add any extra functions
  • Keep historical data publicly available as long as SLAC provides storage (1TB) and pinger.slac.stanford.edu VM

Turkey (No update 7/23/2019), moved here 10/10/2019

Jordan  moved here 10 Oct 2019 

• Baraa has just graduated from Princess Sumaya University with a Masters Degree (Major: Enterprise Systems Engineering) and is looking for Universities to continue his education (PhD.)
  • 7/24/2019 Les sent email to Baraa aksing whether and how might Baraa's future affect the MA in Jordan.
  • 7/24/2019 Baraa responded "I will have to check with my system's administrator regarding that, however I will do my best to keep the MA live.". However, Baraa responded 8/15/2019 "Since I will be leaving. The new system administrator turned it off." Les has disabled it.

UAF moved here 8/25/2019

The 4th  Call for Proposals for The Asi@Connect Project are now open. Saqib will submit a concept note in WP6 with Wajahat and Les as co-applicants

• From Les: WP6 mentions Pakistan, however, the examples are a bit of a stretch for PingER. Possibly deploying PingER MAs based on the Android in remote regions (NWFP, Gilgit, Balochistan, etc.) The difficulty will be getting collaborators in the remote regions, also probably a lot of travel to shake hands, give encouragement, assist, troubleshoot, etc. A challenge will be convincing the collaborators that they are getting something useful out of this. Maybe need to extend PingER to add some easy to use tools.  Another possibility is reviving the PingER MAs in Pakistan. Again, as Wajahat indicates, it is finding collaborators willing to install and manage the MAs. Some development may be needed to make the installation easier and maybe add remote management. Again an issue may be what do the collaborators stand to gain, and the longer-term continuation of the collaboration as people move on.
  

  WP4 talks about PerfSONAR which is monitoring. Providing training on PingER, the reverse traceroute servers might have some traction.  

   

  For both of these if one could install a mesh of MAs in Pakistan, including the reverse traceroute/ping servers (which are trivial to install on a web server, take minimal maintenance, and are much simpler than an MA) then one could address the problem of locating the lat/long of routers  by trilateration of ping response time. Note though getting the lat longs of end hosts is fairly well addressed, routers are usually identified as being located to the owner (e.g. all ESnet's routers on the US and Europe are defined as being located at Berkeley California (the HQ of ESnet)

Turkey moved here 8/9/2019

Amity moved here 7/24/2019

Android progress:

• Instructions from Aayush to load the application are at ePingER Functional prototype (for Androids only)
  • They sent a new copy of the app 4/6/2019, see ePingER Functional prototype
    • The information has been passed to Topher, who plans to install on an Android. The ball is now in our court
    • There is a discussion between Les and Aayush on how to format the raw data records, in particular with respect to the lat/long. Sent a reminder 4/18/2019.
    • We (SLAC) an email to Aayush asking how we gather the raw data from the cloud. Sent reminder 4/18/2019.
      • Aayish responded 4/18/2019: 
        Yes, we have modified the app to write the semi-colon to the file as a separator.
        However, at this point we suggest you to not make any changes to the pinger project yet. 
        
        
        Let us see and run in this parallel to what is already setup at Amity MA. This will allow us to compare the quality of results by both the systems – one at present which does not utilize the app, and the new one which requires an app for data collection.
        We send the data from the app to the firebase cloud service as JSON objects via a REST API. 
        On the cloud-server side, we intend to build a compute-service that automatically converts the json data to a txt format, for consumption by data researcher team at amity (because they are already used to running analysis on the txt files; but seem to be open to working with JSON data as well). [For the SLAC gatherer we would at least initially prefer the data to be retrieved/available in txt form. This way the changes to the gatherer are minimized and more consistent with current practice.]
        
        
        So once the team finds that the results are significant enough, then we can formally propose to include lat/long data in the Pinger gathering, archiving and analysis.
        For now, it would be best that we do not disturb the current Amity MA, and that the android project runs parallel to this."

• Topher is back, Bebo is contacting him. Bebo's thoughts were that:PingER will need a fixed name for each Android MA.  Since the Android may be mobile the IP address may be dynamic. There is a DNS name to dynamic IP address service that may be useful. Maybe there is another unique fixed identifier in an Android that could be used such as a serial number or SIM ID.
  •  If the Amity app is robust  (i.e. it does not noticeably impact the other services, power, networking, security etc) make it part of the standard Rainforest installation;
  • Topher's package knows the GPS location, so it should be available to PingER for recording.

NUST moved here 7/24/2019

• Wajahat said the letter has been received and a written response is being drafted. Is there an update?
• They (NUST) would also be interested in other collaborations with SLAC
  • Since Les is no longer an employee of SLAC but rather an emeritus this may complicate things.
  • A possibility might be in high-speed data transfer. It would need a champion from NUST, and a plan and funding to provide access to 100Gbps

Wajahat said that contacts at some sites are not very interested in PingER and wonder why there are so many pings. Thus he believes we need new sites representing the regions of Pakistan that have willing collaborators. Thus we should give up on sites such as CAE and ISRA that have not had any data to gather for a year or so.  Les has since disabled these sites.