History of growth of PingER hosts

1995

The PingER project has its origins in 1995 in the WAN Monitoring group at SLAC, which monitored links to many sites that SLAC collaborated with by using the IP Ping facility.

1996

In 2006, the ESnet Network Monitoring Task Force (NMTF) quickly become involved in the project and the work was extended to cover ESnet sites. Several DOE Labs installed the PingER monitoring software which greatly improved our knowledge of the network.

1997

In April 1997 the International Committee on Future Accelerators (ICFA) created the Network Task Force to cover the needs of the High Energy and Nuclear Physics (HENP) community.

1998

In 1998, the Cross Industry Working Team (XIWT) adopted the PingER tools developed out of the above efforts and used them to monitor their member sites.

2001

In July 2001, there were 32 monitoring sites around the world, and over 3500 links were monitored.Towards the end of 2001 the number of sites monitored started dropping as sites blocked pings due to security concerns. The rate of blocking was such that, for example, out of 214 hosts that were pingable in July 2003, 33 (~15%) were no longer pingable in December 2003 even though they were still up and running (as measured by responding to TCP probes).

2002

The increases in monitored sites towards the end of 2002 and early 2003 was due to help from the Abdus Salam Institute of Theoretical Physics (ICTP). The ICTP held a Round Table meeting on Developing Country Access to On-Line Scientific Publishing: Sustainable Alternatives in Trieste in November 2002 that included a Proposal for Real time monitoring in Africa. Following the meeting a formal declaration was made on Recommendations of the Round Table held in Trieste to help bridge the digital divide. The PingER project started collaborating closely with the ICTP to develop a monitoring project aimed at better understanding and quantifying the Digital Divide. On December 4th, 2002 the ICTP electronic Journal Distribution Service (eJDS) sent an email entitled Internet Monitoring of Universities and Research Centers in Developing Countries to their collaborators informing them of the launch of the monitoring project and requesting participation. By January 14th 2003, with the help of ICTP, we added about 23 hosts in about 17 countries including: Bangladesh, Brazil, China, Columbia, Ghana, Guatemala, India (Hyderabad and Kerala), Indonesia, Iran, Jordan, Korea, Mexico, Moldova, Nigeria, Pakistan, Slovakia and the Ukraine. The increase towards the end of 2003 was spurred by preparations for the second Open Round Table on Developing Countries Access to Scientific Knowledge: Quantifying the Digital Divide 23-24 November Trieste, Italy and the WSIS conference and associated activities in Geneva December 2003.

2004

The increases in remote nodes 2004 were due to adding new sites especially in Africa, S. America, Russia and several outlying islands.

2005

In 2005, the Pakistan Ministry Of Science and Technology (MOST) and the US State Department funded SLAC and the National University of Sciences and Technology’s (NUST), School of Electrical Engineering and Computer Sciences (SEECS, formerly known as NUST Institute of Information Technology (NIIT)) to collaborate on a project to improve and extend PingER. As part of this project and the increased interest from Internet2 in the “Hard to Reach Network Places” Special Interest Group, many new sites in the South Asia and Africa were added to increase the coverage in these regions and also to replace sites that were blocking pings. For instance we were unable to find pingable sites in Angola prior to December 2005. Also as part of this project we started to integrate PingER with the NLANR/AMP project and as a result a number of the AMP nodes were added as PingER remote hosts in the developing regions. With help of Duncan Martin and the South Africa Tertiary Education Network (TENET) (http://www.tenet.ac.za), we successfully set up a monitoring node in South Africa, which became a great help in viewing the Digital Divide from within the Divide. With the help of SEECS, NUST, a monitoring node was set up at NUST and in Nov. 2005, another node was added at NTC (National Telecommunication Corporation}, which is the service provider for the PERN (Pakistan Educational and Research Network.

2006

Again in 2006 in preparation for a conference on Sharing Knowledge across the Mediterranean at ICTP Trieste Nov 6-8, 2006, we added many new sites especially in Africa. Additionally, new monitoring nodes were setup in Pakistan (National Center for Physics (NCP)), Australia (University of New South Wales) and South Korea (Kyung Hee University).

2007

In 2007, an effort was made to find new monitored nodes in countries not previously being observed. This was:

• To improve comparisons with human and economic development indices from the ITU, the UNDP, the World Bank, the CIA and also measures of International bandwidth capacity/country.
• To better enable validation of PingER derived throughputs versus throughput measures from Ookla Speedtest.net and ZDnet speedtest.
• To prepare for case studies on South Asia and Sub-Saharan Africa.
• To prepare for invited talks given at the American Physical Society (APS) meeting in Jacksonville Florida, the IHY in Addis Ababa, Ethiopia, and theSharing Knowledge Foundation in Montpellier, France . In addition a talk was given at the Internet2 Spring Members meeting.
• To prepare for a visit to NUST in Pakistan and talks to be given there.
• Collaboration with James Whitlock of the Bethlehem Alliance resulted in two monitoring hosts in Palestine (Jerusalem and the Gaza Strip).

As a result, in 2007, the total number of hosts monitored from SLAC went up from 334 to 442, the main increases being in Africa which went from 58 to 95 hosts, South Asia from 20 to 37 hosts, Middle East 15 to 26 hosts, and South East Asia from 12 to 22 hosts. We added over a hundred new hosts from Ookla servers which cover over 50 countries.

2008

In 2008 due to US Science budget cuts in particular in HEP, there were layoffs at SLAC and a redirection of goals that led to a much reduced support for PingER. This is discussed in the section “Outlook: cloudy” in http://www.symmetrymagazine.org/cms/?pid=1000639. Despite this, with some remaining funding from past projects, and with three graduate students from SEECS Pakistan and donating time, the project has successfully continued running.

2009

In 2009 the support for PingER continued at a similar level to that in 2008. We were fortunate to have continued support from Pakistan, including 2-3 graduate students and a lecturer, at SLAC for a year. The increase in number of hosts in Africa was enabled by invited talks in Ethiopia and Zambia, a paper at a conference in Namibia, a series of four lectures to African computing and networking people at a meeting at the ICTP in Trieste, and a talk on African Internet performance at the European Geophysical Union in Vienna.

2010

In 2010 support for PingER continued especially in Pakistan, where about 17 new nodes were added. NUST SEECS also sent 2 students for one year each for work related to the IEPM project, there was an increase in the number of hosts in Africa, Latin America, East Asia and South East Asia as well.

2011

In 2011, NUST SEECS sent two more students for one year each for research related to the IEPM project. In 2011, there was a concentration on carefully locating the hosts to aid in geo-location and plotting on the PingER map. We also removed hosts not responsive for > 3 months and replaced those in developing countries with responsive hosts and where possible ensured that there were >=2 hosts in developing countries.

2012

The funding from the Pakistan Higher Education Commission (HEC) for the PingER project ended. As a result there were no students from Pakistan at SLAC from April though the end of the year. The program with Pakistan continued at a reduced level with fortnightly Skype meetings. The main focus was on GeoLocation (TULIP) and extending the PingER monitoring within Pakistan with bi-monthly reports from NUST to the HEC. There was a visit to SLAC in September, by the Director General of SEECS (Dr. Arshad Ali) and the Rector of NUST. As a result of this we hope for limited funding from NUST to support a NUST student at SLAC for a year starting in 2013.

In 2012, we also completed a Memorandum of Understanding (MoU) between SLAC and the University of Malaysia in Sarawak (UNIMAS). As a result of this program we added a monitoring host at UNIMAS and about 30 hosts in Malaysia and S. E Asia. In December, we also gave a workshop on PingER at UNIMAS, as well as a talk at the Malaysian National Regional and Education Network (MYREN) in Kuala Lumpur on the way to Sarawak.

More publicity for PingER was forthcoming following

• a talk to the opening session of the Internet2 Joint Techs meeting;
• a talk at the eGYAfrica workshop in Nairobi.
• PingER was nominated for the Silicon Valley Tech Awards 2012

2013

We extended the collaboration with the University of Malaysia in Sarawak (UNIMAS) to add the University of Malaya (UM) in Kuala Lumpur and Universiti Technologi Malaysia (UTM) in Johor Baru.  In June 2013 we held a two day workshop on PingER at UM. We also visited UTM talked to students and gave a talk there. We held fortnightly Skype meetings with NUST, UM, UTM and UNIMAS.

We more than doubled the number of hosts monitored in Malaysia and S. E. Asia, and added extra monitoring hosts in Pakistan and Malaysia.

Raja Asad from NUST arrived at SLAC in August 2013 to spend a year as a visiting scientist working on PingER. His main role was to develop Trilateration Utility for Location of IP hosts (TULIP) geolocation of Internet hosts based on distance estimates from ping response times.

We gave a SLAC-wide colloquium on The Emergence of the Internet and Africa.

Submitted paper on dynamic ping delay based Geolocation to ACM/IEEE, Started development of geolocation Visual traceroute.

Renan Sousa from UERJ in Brazil visited SLAC for 3 months and put up a prototype Linked Open Data access to PingER data for mashups.

2014

We extended the collaboration with the University of Malaysia in Sarawak (UNIMAS), the University of Malaya (UM) in Kuala Lumpur and Universiti Tehcnologi Malaysia (UTM) in Johor Baru to add Universiti Utara in Northern mainland Malaysia and the Malaysia Research and Education Network (MYREN) Internet Service Provider. We held 18 Skype meetings with NUST, UM, UTM and UNIMAS.

In November we held a half day workshop on PingER[2] in Kuala Lumpur.

We prepared 9 hours of  lectures on Internet and cell phone communications for a two week School on Space Weather at the University of Koudouga[3] in Burkina Faso in November 2014. Unfortunately due to civil disturbance that burned down the parliament, deposed the old government and president and resuledt in interim military rule, the school was postponed.

We more than doubled the number of hosts monitored in Malaysia and S. E. Asia, and added extra monitoring hosts in Pakistan and Malaysia

Raja Asad from NUST developed a Visual Traceroute web application that uses the TULIP ping based geolocation technique for locating routers along the route.

Submitted paper on dynamic ping delay based Geolocation to ACM/IEEE, Started development of geolocation Visual traceroute.

2015

We extended the collaboration with the University of Malaysia in Sarawak (UNIMAS), the University of Malaya (UM) in Kuala Lumpur and Universiti Tehcnologi Malaysia (UTM) in Johor Bahru to add Universiti Utara in Northern mainland Malaysia (UUM) and the Malaysia Research and Education Network (MYREN) Internet Service Provider. We held 12 (monthly) Skype meetings[1] with NUST, UM, UTM, UUM, UNIMAS and MTREN.

Worked with the Rector of NUST and the new Director of SEECS to ensure continuity of the PingER project at NUST, following Dr Arshad Ali’s move from the Director of SEECS to the executive director of the Pakistani Higher Education Commission (HEC). This post is the second highest in HEC, right after the chairman.

We met with Colombia RENATA NREN leaders and the Columbia minister of IT to discuss the use of PingER in Colombia. To assist we put together a short case study on Colombia[2]. However after several months this collaboration did not go anywhere.

To support multiple Big Data Analysis of PingER data projects  at UM, UUM, UFRJ, and Amity University in New Delhi we gathered both the raw (as measured) data and the hourly analyzed data from 1998 to the present and made it available[3] via anonymous FTP. We also provided information on retrieving the data[4].

To support the porting of PingER to an Android smartphone we made the PingER measurement Agent available via Github[5].

Utilizing a SLAC 4 host cloud with~16Gbytes each, plus access to 220Gbytes each, with Hadoop, Cloudera and Impala, Thiago Barbosa put together a warehouse of hourly PingER data going back to 1998. It successfully provided database queries to access the full set of PingER data with response times of several seconds. Unfortunately due to lack of time at SLAC, we were unable to make the access available outside of SLAC.

We put together a report on Duplicate ping responses[6].

Successfully ported PingER to a Raspberry Pi version 1[7].

We put together a case study on the impact of demonstrations against the government in Malaysia (August28-31^st 2015) on various Malaysian hosts[8].

We prepared and presented five papers at the 4^th International Conference on Internet Applications, Protocols and Services (NETAPP2015) Cyberjaya, Malaysia, 1-3 December 2015[9].

Bebo White made a keynote presentation at the CITA  9^th International Conference on IT in Asia[10] on Transforming Big Data into Knowledge. Bebo used PingER as a case study on what you can do with PingER and how to access the data.