Time & date
This meeting: ????, 2020 9 pm Pacific time; a day later 9:00 am Pakistan time; 10:30 am India time
Invitees:
- Email addresses
- abansal1@amity.edu; assabitha@amity.edu; sachanakshat@gmail.com; cottrell@slac.stanford.edu; saqibutm@outlook.com; bebo@slac.stanford.edu; umar.kalim@gmail.com;
PingER Project Update using Zoom
IMPORTANT NOTE: The meeting is set up to record automatically. By joining the meeting you are agreeing to be recorded (see details)
Agenda:
- Introductions
- US & Pakistani folks
- India folks
- Status of pingeramity.in
- Covid-19 & PingER
- Mobile PingER
Recording of the meeting can be found at:
Minutes:
Attendees:
Amity: Asshat Sachan, Dr. A. Sai Sabitha, Prof. Dr. Abhay Bansai, Samaksh Agarwal, Shivang Gupta, Sakshi Arora, Shwang
US: Bebo White, Umar Kalim
Pakistan: Saqib Ali.
Introductions:
Les Cottrell: Working on keeping PingER running, and on the impact of Covid-19.
Bebo White: Main interest is finding new ways to collect, analyze and visualize data. Possibly use data in correlating with other events (e.g. Covid-19). Also looking at alternative monitoring architectures such as using Android/iPhones and possibly looking at mobility.
Umar Kalim (no photo available): Mainly an observer.
Akshat: working on developing PingER for Android and IOS. Looking at extending to a SmartTV so have a fixed monitor.
Abhay Bansai: Wants data to be analyzed and visualized, come out with new ways of looking at and utilizing data, and making an impact on society, extend analysis of correlation with Covid-19.
Samaksh Agarwal: Working on the Android App. Moved from native to run on more platforms. Look at turning the app into a data service which sends the data every 15 minutes. Also thinking of making a speed map maybe to show how storms may affect the QOS.
Shivang Gupta: working on a machine learning model. modeling the data. He is collecting the data for different countries and comparing daily, active, deaths Covid-19 data for different countries.
Sakshi Arora: working on the machine learning with Shivang, also looking at impacts of Covid-19 lockdowns etc. on PingER data measured by the Android application.
A Sai Sabitha: as team leader every year she wants to come out with papers concerning applying machine learning to the PingER and Covid data.
Status of pingeramity monitoring host
According to Akshat email 11/9/2020m pingeramity.in is up and running 8:30pm PDT - 4:00am PDT. By means of a cronjob SLAC gathers the data from 204.12.103.71 (pingeramity.in does not resolve by name, hence we use the IP address) via https://pingeramity.in/cgi-bin.ping_data.pl at around 1:30am PDT +-30 mins each morning. However, in the last 100 days only on one day ( Dec 11 1:30am PDT) was the collection script able to gather any data. The data itself was strange in that it appeared as:
pingeramity.in 204.11.58.194 www.andi.dz 213.179.181.44 100 1607593552 10 10 189.987 190.653 193.137 1 2 3 4 5 6 7 8 9 10 190 193 190 190 190 190 191 189 190 190
N.b. Thu Dec 10 09:45:52 2020 GMT for 1607593552
It is strange that the host identifies itself with the IP address 204.11.58.19 rather than the address it was accessed with(204.12.103.71).
Further, the host is only occasionally pingable around the time of each day's measurement (see for example https://www-iepm.slac.stanford.edu/monitoring/checkdata/ where the host's box is white if the host is blue if there is data, green if there is only data for part of the day, magenta if there is no data but the host is pingable, and white if there is no data and the host is not pingable.
There has been an exchange of emails between Akshat and Les. Akshat reports the machine is up and running and accessible from India, although there have been power problems. Les, however, has been unable to access the web server https://pingeramity.in/cgi-bin.ping_data.pl or http://202.12.103.71/cgi-bin/ping_data.pl. Work is in progress to understand and remedy the situation.
Covid-19 and PingER
Les put together a study of the correlation of PingER and Covid-19 cases. It is at https:confluence.slac.stanford.edu/display/IEPM/PingER+and+COVID-19 and was last updated May 19, 2020.
Shivang and Sakshi have been collecting data from the MHRD (Ministry of Human Resource Development India) site. LIke Johns Hopkins University in the US, MHRD gathers and provides COVID-19 data on cases, deaths etc. for states and cities such as Karnataka, Bangalore, Delhi in India. Now that the pingeramity server is running they are gathering its (latency) data. Besides India they are also looking at USA and Brazil. They then want to compare the pingeramity data with the COVID-19 data looking for correlations.
Mobile PingER
Akshat proposed to extend the ideas of the new Android-Mobile-based PingER to SmartTVs. This will enable a new PingER structure again with static monitors, but new code for the measurements, recording, and distribution. In particular, they will push the data out every 10-20 minutes, as opposed to today's version where we gather the data from the monitors on a daily basis each morning). Also, the new version would be Android based rather than Linux based.
They are also working on porting the current Android PingER that only works in the foreground to one that works in background. Part of the idea is also that it will be able to run on an iPhone. A next step is thus to complete a working version of the new PingER (let's for the moment call it ePingER) that runs in background, supports SmartTV, Android and iOS devices. Since Amity's end of semester is coming up with semester exams finishing December 11th, a goal was set to shoot for reporting on progress and possible demonstrations on December 21st
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Meeting Preparation Notes
Amity
Apart from invitations to present papers at upcoming conferences we heard from Amity since Sep 4, 2020 when they forwarded an email originating July 11, 2020.
Earlier they had said
"We are still working on the portability of the app from Android to Flutter, which will enable the app to be downloaded in iOS environment along with Android. Currently, our application is supported by Android only. Samaksh is looking forward to this issue. We are also discussing how the location feature will add functionalities to our application, and the PingER project will be more than just static servers and we can better monitor the internet with growing mobile communication. Shivang will mail you the list of all the proposed ideas we have on our mind, on what we can do with our mobile application. We will forward the list to Les Cottrell Sir and Bebo White Sir and get it discussed with them too."
"One more thing I wanted to ask you is, can we have the ideas to be working on, or the subdomains, some particular topic we should pursue? Because there are a lot of facts we collected alone in the Coronavirus topic itself, and we need something more specific to work upon, like monitoring some particular countries, the government transparency on data, or measuring the impact the pandemic has on each country on a global scale. Please guide us into this. It would be a great help."
In response, we sent them an analysis of the impact of Covid-19 on PingER measurements. See the full report so far (main focus is on Africa and to a lesser extent South America) is at PingER and Covid-19 by Region. At the time we suggested "A possible project for Amity might be to extend the detailed analysis to South Asia. We would be happy to assist as needed."
They responded:
- People's movement and the wave patterns can give us great insight into how the virus is spreading and its impact. With the advent of festivals in India in the coming months, we are afraid we may be nearing a wave. We are trying to find its relation with various other factors, one being internet performance. The government has imposed various lockdowns and unlocking with different rules, we are also going to study its direct impact with COVID-19.
Concerning the Android PingER app, the July 11 2020 email said:
We have successfully implemented the Ping through our mobile application, and the data is saved in the default PingER format, with 2 fields, in addition, the latitude and the longitude of the place.
An android application was implemented before, but this time we made a Flutter app, which makes it possible to distribute the app to both Android and iOS platforms via a single source code.
Regarding the format of data stored, we can only store it in JSON format, because it is the most convenient and best way to store. However, we made a python script that can parse the JSON data to Plain Text format and give the output.
In addition to the functionality of the previous app, we are also storing the data locally on the user's phone, which is plain text format in a file, under the folder //Internal Storage/SLAC_Pinger/<date> <time>.txt
Concerning the Android App, Umar pointed out that we need to understand/specify the end goal of the project. The project can support mobility since it records GPS coordinates. One needs to decide how to utilize the data gathered. It can be useful for evaluating Telco service providers. Unlike today's PingER which has continuous data between monitoring stations and targets, we would expect the mobile data to be much more sparse both in terms of time and monitors (cell-phones). Further, it is community-driven rather than centrally. It would appear to be similar to Thousand Eyes or Speedtest app that enables cell phone users to make on-demand measurements of throughput from their phones to targets around the world. The measurements would be archived and can then be summarized and reports presented, e.g. comparing connectivity from area to area on maps and or by a carrier. Telcos are also making such connectivity measurements for their network. An advantage of the current proposal is that it would be across service providers, and also have less network impact than say Speedtest.
pingeramity.in
No data since Jan 6, 2020. Cannot ping gives unknown host. Email from Amity Sep 4, 2020: "Also, for our Amity SLAC server, physical access is not possible and therefore we are unable to configure the scripts manually there and hence it is running offline. It will be set as soon as the college reopens." Is there any idea when this may occur or what has to happen before the college is allowed to open?
Disabled by Cottrell 10/19/2020
SLAC Overview
PingER is still actively monitoring from 15 sites in 10 countries (US, Canada, Pakistan, Malaysia, Germany, Switzerland, China, Taiwan, Brazil, South Africa). Apart from SLAC, the monitoring sites each monitor 171 Beacons. SLAC monitors over 700 monitored hosts in 171 countries including all continents except Antarctica. See https://www-iepm.slac.stanford.edu/pinger/sites-per-country.html. Also, see the state of various problem monitoring hosts below.
Covid-19 in S. Asia and US
While the median age for Pakistan is quite young (late 20's to early 30's) and that the fact the population leans to B+- rather than A+- blood both should help reduce the incidence, this would not appear to be a big difference between India and Pakistan. Saqib mentioned that a difference might be in the way Pakistan imposes a hard 2 week lockdowns (enforced by police with barricades) on small areas (e.g. a street in Islamabad) where there is an outbreak. There may also be some under-reporting.
Covid-19 data
Les is working on analyzing the data from Johns Hopkins University on confirmed cases. and deaths with a library of reports at https://www-iepm.slac.stanford.edu/pinger/excel, including:
- US Politics:
- The impact of Trump's political rallies on spreading Covid-19
- Cases in US states: in favor of, or leaning to, or undecided for Trump or for Biden
- States with Republican governors
- States with Democratic governors
- Cases in: Europe: Central, Northern, Western, Eastern, South-West,
- Cases in: East Asia, Middle East, South Asia, South West Asia
- Cases in: Central America
- I will be adding South America and Africa
- US:
- San Francisco Bay
- Compared to Western Europe
- US most populated states
There are instructions at Covid-19 - Visualizing the data with Excel (https://confluence.slac.stanford.edu/display/IEPM/Covid-19+-+Visualizing+the+data+with+Excel)