...
Attendees
Invitees:
Hassaan Khaliq ?, Muhammad Anas Abrar (SEECS); Saqib+, Aqsa (UAF); Johari-+, Adnan Khan (UNIMAS); Badrul, Ridzuan, Ibrahim (UM); (UTM); Adib, Fatima (UUM); Fizi Jalil (MYREN); Les+, Bebo, Joao+ (SLAC).
...
? Individual emails sent
Actual attendees:
Joao, Adib, Les
Administration
...
"Since we're still waiting on an account, at SLAC, I asked the student to explore another idea (which is quite relevant) proposed by Prof. Bebo “Creative visualization of PingER data, including rich interaction;”. He has already started looking at the possible attractive/interactive design, at the same time he is working to improve his programming skills."
"On the other hand, I am in the final stage of writing a case study on the Internet performance in ASEAN countries and its implication on the Socio-Economic Indexes. Will try to share with Dr.Les and Prof. Bebo soon to comment and get your advice on the publication possibility of the paper."
UNIMAS
No update 5/18/2016
- Adib has created a case study SE Asia (ASEAN). Les to read the study provide feedback international connectivity for leaders of the country.
- Adamu expecting baby so very busy early next month. So no progress.
- There is another student, (Jaafrau) has a design. Looking at visualization.
- Adib has a student from Thailand who is working with Adib at UUM who may be interested in installaing a PIngER MA in Thailand. Adib will talk next week after exams to see if can get permissions.
UNIMAS
No update 5/18/2016
...
- Aqsa and Saba are working together. Their goal is focusing on visualization of PingER historical data using warehouse. The idea is to develop a warehouse in UAF university and make it publicly available. They are 50-60% done with setting up a Hadoop cluster with 3 nodes, 1 master, 2 slaves. She is currently working on importing the PingER data into hdfs on the cluster. they have run some Impala queries on the data and are working on visualization
- Topic: visualization on pingER data (email from Aqsa and Response from Renan)I have studied the google charts as visualization tools but here are some points need to be discussed.1. The idea of applying visualization on Data warehouse (Impala query results) does not seem to be so useful because Data warehouse contains static data and visualization charts will also remains static and need to be updated with the time.
Yes, it needs to be updated with the time. My suggestion is to transform PingER data into data to be inserted into the data warehouse. Myself and some other Brazilian students have developed codes to do this. Such process should occur at least once a day to keep the data warehouse updated daily. This has never been done by any of us.
2. Google charts API cannot integrate with Impala As Impala is hadoop distributed Big Data supported database Google can only integrate with flat files or flat databases like Mysql.
If Google charts API can only read flat files (e.g., CSV files), it is trivial to save a database query result as a CSV flat file that would be consumed by Google charts. Can Google charts generate a plot dynamically after reading a just-created CSV file? Aqsa and team members are working on
- Sara Masood is working on a PingER Internet Performance Monitoring Agent On Android Device. There has been a discussion to pin down more closely what is needed, i.e. a PingER Measurement Agent running on an Android making measurements to the Beacons, and the ability to upload the data nightly. We held a skype group meeting with Topher, Sara, Saqib, Bebo and Les to move forward. See 20160331 Meeting on PingER/Android, also see Discussion on porting PingER to Android. She has what she needs for now.
Project on Internet performance has been accepted, but no funding yet.
UTM
Saqib's old supervisor agreed to appoint a master student to take of PingER in UTM. Saqib has emailed 3/9/2016, no progress 4/6/2016, no progress 5/18/2016. Any progress 6/22/2016.
UM
MYREN
Email 6/15/2016 to Fitzi, pinger.fsksm.utm.my is down.
NUST
- Currently it does not respond to ping pingerqta.pern.edu.pk
- Ping 121.52.157.148, responds
- http://121.52.157.148/ Apache responds. http://121.52.157.148//cgi-bin/ping_data.pl? responds (albeit very slowly) with Get Ping Data from sharedw228.b2bhosting.net
- This is a funny hostname. It comes from the system response to the perl commands:
use Net::Domain qw(hostname hostfqdn hostdomain);
my $hostname = hostfqdn();
- This is a funny hostname. It comes from the system response to the perl commands:
- Also there is data.
- The following needs to reflect the new IP address at 121.52.157.148
Output from: /bin/cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 121.52.157.157 pingerqta.pern.edu.pk
- It is very easy to change the IP address once the Measurement Agent is up and running and responding, in fact the PingER meta data at SLAC was changed in February 2015. Howeer the DNS still points to:
[cottrell@pinger ~]$ nslookup pingerqta.pern.edu.pk
Server: 134.79.111.111
Address: 134.79.111.111#53
Non-authoritative answer:
Name: pingerqta.pern.edu.pk
Address: 121.52.157.157
Following hosts are down and not responsive. Nodes were contacted many times for identifying their issues. i.e turn on their pinGER Computers but never responded well.
- aup.seecs.edu.pk #AUP said that they will look intothe issue in a while everytime I ask them or give them the reminder. Does not ping
- cae.seecs.edu.pk #Does not ping
- kinniard.seecs.edu.pk+ #Unknown host
- maggie1.seecs.edu.pk # Ping responds with Time to live exceeded
- maggie2.seecs.edu.pk #Does not ping
- npm.neduet.edu.pk* # Not pingable
- pinger.uettaxila.edu.pk* #UET taxila are not receiving the provided Number, Unknown host.
- pingerisl-qau.pern.edu.pk* #number is not responding for QUA. Not pingable
- quest.seecs.edu.pk* #Quest are not receiving my Call now. Not pingable
- pingerlhr-pu.pern.edu.pk* #Site pings via http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=pingerlhr-pu.pern.edu.pk&function=ping, however http://pingerlhr-pu.pern.edu.pk/cgi-bin/ping_data.pl? gives site can't be reached.
- pingerpwr.pern.edu.pk* # Site pings via http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=pingerpwr.pern.edu.pk&function=ping however the configuration file /usr/local/share/pinger/pinger.xml is truncated, see FAQ for how to go about fixing.
...
creating Data warehouse and we are very close to complete this. Here are some updates.
Tehseen qureshi has transformed the pingER text files into binaries and soon he will be able to get CSV files.
- Saba is working on defining a 4 node cluster.
- Aqsa has uploaded some sample CSV files on HDFS and run Impala queries as i will get the actual CSV from tehseen these steps are also will be completed
Visualization Status
Aqsa has
drawn a line chart and bar chart on the data of sample CSV file and i am exploring some more charts to be drawn by using Google API's.
- Sara Masood:
- pingER monitoring host on android . I am using perl-android-apk Interpreter to run pearl on android.. it include execution of 3 perl scripts
getdata.pl
ping_data.pl
pinger2.pl
Resources Required
i need ping_data.pl script file and getdata.pl script file.
Joao will provide them via anonymous ftp.
Research Status
Perl-android-interpreter is not executing .pl file mainly because old Perl version (Perl 5.10) which is not comparable with android lollipop 5.0 version, i am compilingPerl .5.22.0 for android ,it may solve issue.
- pingER monitoring host on android . I am using perl-android-apk Interpreter to run pearl on android.. it include execution of 3 perl scripts
Project on Internet performance has not received funding.
Saqib will proceed to install Pinger in UAF Pakistan
UTM
Saqib's old supervisor agreed to appoint a master student to take of PingER in UTM. Saqib has emailed 3/9/2016, no progress 4/6/2016, no progress 5/18/2016.
1. Currently, Pinger server is located in the faculty server room. Recently, faculty has closed the Server room. Now we have to move it to the University Central Server Room. For this we need support from Prof Johari and Prof Abdul Hanan.
2. For the M.Sc. student in UTM we have to request again to Prof Hanan and Prof Asri and we need support from Prof Johari.
UM
MYREN
Email 6/15/2016 to Fitzi, pinger.fsksm.utm.my is down.
NUST
- Currently it does not respond to ping pingerqta.pern.edu.pk
- Ping 121.52.157.148, responds
- http://121.52.157.148/ Apache responds. http://121.52.157.148//cgi-bin/ping_data.pl? responds (albeit very slowly) with Get Ping Data from sharedw228.b2bhosting.net
- This is a funny hostname. It comes from the system response to the perl commands:
use Net::Domain qw(hostname hostfqdn hostdomain);
my $hostname = hostfqdn();
- This is a funny hostname. It comes from the system response to the perl commands:
- Also there is data.
- The following needs to reflect the new IP address at 121.52.157.148
Output from: /bin/cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 121.52.157.157 pingerqta.pern.edu.pk
- It is very easy to change the IP address once the Measurement Agent is up and running and responding, in fact the PingER meta data at SLAC was changed in February 2015. Howeer the DNS still points to:
[cottrell@pinger ~]$ nslookup pingerqta.pern.edu.pk
Server: 134.79.111.111
Address: 134.79.111.111#53
Non-authoritative answer:
Name: pingerqta.pern.edu.pk
Address: 121.52.157.157
Following hosts are down and not responsive. Nodes were contacted many times for identifying their issues. i.e turn on their pinGER Computers but never responded well.
- aup.seecs.edu.pk #AUP said that they will look intothe issue in a while everytime I ask them or give them the reminder. Does not ping
- cae.seecs.edu.pk #Does not ping
- kinniard.seecs.edu.pk+ #Unknown host
- maggie1.seecs.edu.pk # Ping responds with Time to live exceeded
- maggie2.seecs.edu.pk #Does not ping
- npm.neduet.edu.pk* # Not pingable
- pinger.uettaxila.edu.pk* #UET taxila are not receiving the provided Number, Unknown host.
- pingerisl-qau.pern.edu.pk* #number is not responding for QUA. Not pingable
- quest.seecs.edu.pk* #Quest are not receiving my Call now. Not pingable
- pingerlhr-pu.pern.edu.pk* #Site pings via http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=pingerlhr-pu.pern.edu.pk&function=ping, however http://pingerlhr-pu.pern
...
...
- gives site can't be reached.
- pingerpwr.pern
...
- .edu.pk* # Site
...
...
...
...
* This host was noted at the 3/9/2016 meeting as being down for the last 30 days.
+ Not seen in June 2016
Les proposes:
- Give up (i.e. remove from Monitoring node table by setting Projecttype to D (for Disabled)) on non-responsive hosts not working since 3/9/2016.
- I would not oppose extending thsi to all non-responsive, non-working hosts.
- Focus on the close to working hosts
- however the configuration file /usr/local/share/pinger/pinger.xml is truncated, see FAQ for how to go about fixing.
- pinger.iba-suk.edu.pk # Site pings via http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=pinger.iba-suk.edu.pk&function=ping however http://pinger.iba-suk.edu.pk:1313/cgi-bin/ping_data.pl? gives site can't be reached and http://pinger.iba-suk.edu.pk/cgi-bin/ping_data.pl? takes me to a generic site for iba-suk.
- pinger.isra.edu.pk # Site can't be pingerd via http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=pinger.isra.edu.pk&function=ping
* This host was noted at the 3/9/2016 meeting as being down for the last 30 days.
+ Not seen in June 2016
Les proposes:
- Give up (i.e. remove from Monitoring node table by setting Projecttype to D (for Disabled)) on non-responsive hosts not working since 3/9/2016.
- I would not oppose extending thsi to all non-responsive, non-working hosts.
- Focus on the close to working hosts
PingER at SLAC
Joao making the data on FTP to be up-to-date.
Also has a 4 node cluster with Cloudera manager for Hadoop, Next step are to get impala working for queries.
Working on the following hosts to be able to gather data
Host | State | last seen | Status |
---|---|---|---|
pinger.unimas.my | email 5/12/2016, fixed 6/16/2016 | March 2016 | Does not ping |
pingersonar-utm.myren.net.my | email to Fitzi 5/12/2016, fixed May 18, 2016 | Does not ping | |
pinger.unesp.br | email 6/15/2016, fixed 6/20/2016 | Feb 2016 | Looked like cronjob not running, cannot ping |
pinger.fsksm.utm.my | email 6/15/2016, no response | May 17 | Does not ping |
www.univ-ouaga.bf | Unreliable, but works about 50% time. | June 2016 | Does not ping |
Next Meeting
Next meeting: Wednesday Jul 27 9:00pm Pacific Standard Time, Thursday Jul 28, 2016 9:00am Pakistan time, Thursday Jul 28 2016 12:00noon Malaysian time, Thursday Jul 28 2016 02:00am Rio Standard Time.
Old Items
Visualization ideas for PingER moved here 5/20/2016
- One way of drilling down is via a map of all sites. Faisal Zahid put something together a couple of years ago. It is at http://www.slac.stanford.edu/comp/net/wan-mon/viper/pinger-coverage-gmap.html. It uses Google maps, allows pan and zoom, plus selection of type (measurement agent, beacon, target) of host to display. It also allows one to select a host (Search location), mouse click on the balloon to find out more about the host, from there click on the sites web site, click on the country or region to get the relevant pingtable.pl information. The Visualize allows one to choose a measurement agent and a country and it draws coloured lines between the sites to represent the metric choosen. This was very useful but parts of it are broken. It would be great to fix or replace. It has some documentation.
- For plotting time series, Grafana looks very promising
- Updating the motion charts, adding newer data etc. would be most useful. See http://www-iepm.slac.stanford.edu/pinger/pinger-metrics-motion-chart.html and http://www-iepm.slac.stanford.edu/pinger/explorer.html
- Enhance the selection Metrics, Ticks (see pingtable.pl) to allow multiple metrics (min, avg, max RTT, loss, thrpughput, , MOS ..) to be plotted as a time series.
- Enhanced selection remote hosts for selected tick, selected metric selected mutiple targets (pingtable.pl currently only allows one target), e.g. Once you have plot enable fitting the data to timelines (linear, exponential, power series)
- Map pf countries the world colored by metric allow selection of metric and for a particular year. with chosen MA,
Huffington Post moved here 3/8/2016
- article on PingER and African Internet performance, see http://www.huffingtonpost.com/david-tereshchuk/a-giant-leap-in-2016-africa_b_8901556.html
Anjum said it would be good for Hassaan to attach this to his proposal. Any action?
Les is working with the SLAC CIO to try and put something together for publicity concerning the life of PingER.
UOA (Saqib) placed here 2/3/2016.
Saqib has a 5 MS students from the Database team
Sara Massoud came up with ideas to improve the info put together by Amity
Sabah Massoumil working on Linked Open Data (please excuse spelling)
Aqsa Hameed working on big data/analysis of PingER data. The initial idea was to set up a big data/big analysis
Aqsa Hameed has been working with Anjum to look at a project to create a Hadoop/Cloudera PingER warehouse to enable easier, more powerful access to PingER historical data.
Thinking a bit on Aqsa her work appears to be related to the work done by Thiago on PingER warehouse using a cluster/Cloudera/HDFS/Impala earlier this year. In particular see the presentation at NETAPPS2015 (see https://confluence.slac.stanford.edu/download/attachments/123309267/NETAPPS_PRESENTATIONrevLuiza.pptx. Once the paper is published we can also provide her with that (Adib will let us know when this is OK).
Les emailed the relevant people to put them in contact with one another. It appears there is a lot of overlap between what Aqsa proposed as a Masters project and what Thiago has already done. However Thiago's system was mainly a proof of concept and not in production. We need to look at the next steps: Internet access, auto-updating of information in near realtime, production service, maintainable etc. With this in place one could really mine the data looking for all kinds of interesting correlations, clustering, event impacts, comparisons etc.
Given that such a warehouse is available, then the next step would be to automatically create queries that would produce in near real-time the plots we produce manually for the PingER annual reports.
Following this publish the data in RDF to tie in with the Southampton RDF web observatory repository (similar to what Renan did as a proof of concept).
Since Thiago's warehouse is only available at SLAC, it may assist to get Aqsa an account at SLAC, alternatively Aqsa will need to set up a repository at here home base.
PingER at SLAC
Working on the following hosts to be able to gather data
Host | State | last seen | Status |
---|---|---|---|
pinger.unimas.my | email 5/12/2016, fixed 6/16/2016 | March 2016 | Does not ping |
pingersonar-utm.myren.net.my | email to Fitzi 5/12/2016, fixed May 18, 2016 | Does not ping | |
pinger.unesp.br | email 6/15/2016 | Feb 2016 | Looked like cronjob not running, cannot ping |
pinger.fsksm.utm.my | email 6/15/2016 | May 17 | Does not ping |
www.univ-ouaga.bf | Unreliable, but works about 50% time. | June 2016 | Does not ping |
Next Meeting
Next meeting: Wednesday Aug 24 9:00pm Pacific Standard Time, Thursday Aug 25, 2016 9:00am Pakistan time, Thursday Aug 25 2016 12:00noon Malaysian time, Thursday Aug 25 2016 02:00am Rio Standard Time.
Old Items
Visualization ideas for PingER moved here 5/20/2016
- One way of drilling down is via a map of all sites. Faisal Zahid put something together a couple of years ago. It is at http://www.slac.stanford.edu/comp/net/wan-mon/viper/pinger-coverage-gmap.html. It uses Google maps, allows pan and zoom, plus selection of type (measurement agent, beacon, target) of host to display. It also allows one to select a host (Search location), mouse click on the balloon to find out more about the host, from there click on the sites web site, click on the country or region to get the relevant pingtable.pl information. The Visualize allows one to choose a measurement agent and a country and it draws coloured lines between the sites to represent the metric choosen. This was very useful but parts of it are broken. It would be great to fix or replace. It has some documentation.
- For plotting time series, Grafana looks very promising
- Updating the motion charts, adding newer data etc. would be most useful. See http://www-iepm.slac.stanford.edu/pinger/pinger-metrics-motion-chart.html and http://www-iepm.slac.stanford.edu/pinger/explorer.html
- Enhance the selection Metrics, Ticks (see pingtable.pl) to allow multiple metrics (min, avg, max RTT, loss, thrpughput, , MOS ..) to be plotted as a time series.
- Enhanced selection remote hosts for selected tick, selected metric selected mutiple targets (pingtable.pl currently only allows one target), e.g. Once you have plot enable fitting the data to timelines (linear, exponential, power series)
- Map pf countries the world colored by metric allow selection of metric and for a particular year. with chosen MA,
Huffington Post moved here 3/8/2016
- article on PingER and African Internet performance, see http://www.huffingtonpost.com/david-tereshchuk/a-giant-leap-in-2016-africa_b_8901556.html
Anjum said it would be good for Hassaan to attach this to his proposal. Any action?
Les is working with the SLAC CIO to try and put something together for publicity concerning the life of PingER.
UOA (Saqib) placed here 2/3/2016.
Saqib has a 5 MS students from the Database team
Sara Massoud came up with ideas to improve the info put together by Amity
Sabah Massoumil working on Linked Open Data (please excuse spelling)
Aqsa Hameed working on big data/analysis of PingER data. The initial idea was to set up a big data/big analysis
Aqsa Hameed has been working with Anjum to look at a project to create a Hadoop/Cloudera PingER warehouse to enable easier, more powerful access to PingER historical data.
Thinking a bit on Aqsa her work appears to be related to the work done by Thiago on PingER warehouse using a cluster/Cloudera/HDFS/Impala earlier this year. In particular see the presentation at NETAPPS2015 (see https://confluence.slac.stanford.edu/download/attachments/123309267/NETAPPS_PRESENTATIONrevLuiza.pptx. Once the paper is published we can also provide her with that (Adib will let us know when this is OK).
Les emailed the relevant people to put them in contact with one another. It appears there is a lot of overlap between what Aqsa proposed as a Masters project and what Thiago has already done. However Thiago's system was mainly a proof of concept and not in production. We need to look at the next steps: Internet access, auto-updating of information in near realtime, production service, maintainable etc. With this in place one could really mine the data looking for all kinds of interesting correlations, clustering, event impacts, comparisons etc.
Given that such a warehouse is available, then the next step would be to automatically create queries that would produce in near real-time the plots we produce manually for the PingER annual reports.
Following this publish the data in RDF to tie in with the Southampton RDF web observatory repository (similar to what Renan did as a proof of concept).
Since Thiago's warehouse is only available at SLAC, it may assist to get Aqsa an account at SLAC, alternatively Aqsa will need to set up a repository at here home base.
There is some documentation written by Les on the usage of the warehouse at SLAC. It is at: PingER Data Warehouse using Big Data with Cloudera on Nebula.
Les will contact Thiago to see if there is other documentation or where to find the programs etc.
We need to put together all we know:
Paper from Thiago on PingER warehouse presented in Malaysia at NETAPP2015 Thiago M. Da S. Barbosa, Renan F. Souza, Sergio M. S. Da Cruz, Maria L. Campos and R. Les Cottrell. Applying Data Warehousing and Big Data Techniques to Analyze Internet Performance
Document on Warehouse
There is some documentation written by Les on the usage of the warehouse at SLAC. It is at:
PingER Data Warehouse using Big Data with Cloudera on Nebula.
Les will contact Thiago to see if there is other documentation or where to find the programs etc.
We need to put together all we know:
Paper from Thiago on PingER warehouse presented in Malaysia at NETAPP2015 Thiago M. Da S. Barbosa, Renan F. Souza, Sergio M. S. Da Cruz, Maria L. Campos and R. Les Cottrell. Applying Data Warehousing and Big Data Techniques to Analyze Internet Performance
Document on Warehouse
Documents on PingERLOD
Documents on PingERLOD
Linked Open Data Publication Strategies: Application in Networking PPerformance Measurement Data, Renan Souza, Les Cottrell, Bebo White, Maria Campos, Marta Mattoso, poster presented at the BigDataScience - Stanford conference, CA, USA May 27-31, 2014.
Reviews/proposals/ from Aqsa/Fatima
Survey on Big Data Indexing strategies, Fatima Bintu Adama, Adib Habbal, Suhaidi Hassan, R. Les Cottrell. Bebo White, Ibrahim Abdullahi.
Saqib will provide Aqsa's
Tehseen is working on missing PingER data.
Project is accepted.
working on a project to develop a node using Raspberry Pi 2 and IoT to measure the air and soil quality.
Have 2 RPis and setting up air quality with sensors from market, idea is to distribute on campus.
Currently, students are analyzing the project to develop a problem statement for their research project.
For the GeoLocation Saqib should contact Anjum
...
Ibrahim has setup distributed hadoop clusters. He has 2TB of disk space. Les has provided information on getting a subset of PingER data by anonymous ftp via ftp://ftp.slac.stanford.edu/users/cottrell. It was put there last September. Information on how the data was put together is at https://confluence.slac.stanford.edu/display/IEPM/Archiving+PingER+data+by+tar+for+retrieval+by+anonymous+ftp. There is information on formatting etc at http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html and some on the dataflows at https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC. Renan at UFRJ has successfully used this data, he has also characterized the data in terms of bytes/metric per year etc.
Ibrahim has started downloading all zip files in the local machines. 6 weeks ago he downloaded 2 GB of Weather data to test his nodes cluster, he wrote a simple Java program (Map, Reduce) to find the Average and it was working fine.
Anjum reported that UM had experienced a TCP syn DOS attack prior to Mar 12th (when an IDS was put in place). It occurred mainly for several days before between the hours on noon- 2pm and 7-7 in the evening (Malaysia time). He suggested looking to see if PingER could spit the effect. Ibrahim, Les and Anjum will look at. Les analyzed the data and sent it to Anjum
NUST
The following is from Samad 2/24/2015.
- buitms.seecs.edu.pk #We have to disable gathering data from this host because the person still don't want to continue with us as i have tried once again to convince him but the answer is same. Les has disabled from SLAC.
- nukhimain.seecs.edu.pk # We were unable to gather data since 20th November, 2014 and now the Node is working fine and collecting data as well.
- pinger.uettaxila.edu.pk #The node is working fine from last two weeks.
- sau.seecs.edu.pk. #This Node is working fine now.
- pingerjms.pern.edu.pk #This node is working now.
- pinger.uet.edu.pk # this was also not working from so many days. and now its working fine and collecting data as well.
- pinger.isra.edu.pk # This node is also working fine now.
- pingerlhr-pu.pern.edu.pk # This is also working fine now.
- pinger.kohat.edu.pk # Collecting data now.
The IP of "pingerqta.pern.edu.pk" has been changed, Les has updated the databas at SLAC with the following
Old IP: 121.52.157.157
New IP: 121.52.157.148
Follow up from workshop
...
://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html and some on the dataflows at https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC. Renan at UFRJ has successfully used this data, he has also characterized the data in terms of bytes/metric per year etc.
Ibrahim has started downloading all zip files in the local machines. 6 weeks ago he downloaded 2 GB of Weather data to test his nodes cluster, he wrote a simple Java program (Map, Reduce) to find the Average and it was working fine.
Anjum reported that UM had experienced a TCP syn DOS attack prior to Mar 12th (when an IDS was put in place). It occurred mainly for several days before between the hours on noon- 2pm and 7-7 in the evening (Malaysia time). He suggested looking to see if PingER could spit the effect. Ibrahim, Les and Anjum will look at. Les analyzed the data and sent it to Anjum
NUST
The following is from Samad 2/24/2015.
- buitms.seecs.edu.pk #We have to disable gathering data from this host because the person still don't want to continue with us as i have tried once again to convince him but the answer is same. Les has disabled from SLAC.
- nukhimain.seecs.edu.pk # We were unable to gather data since 20th November, 2014 and now the Node is working fine and collecting data as well.
- pinger.uettaxila.edu.pk #The node is working fine from last two weeks.
- sau.seecs.edu.pk. #This Node is working fine now.
- pingerjms.pern.edu.pk #This node is working now.
- pinger.uet.edu.pk # this was also not working from so many days. and now its working fine and collecting data as well.
- pinger.isra.edu.pk # This node is also working fine now.
- pingerlhr-pu.pern.edu.pk # This is also working fine now.
- pinger.kohat.edu.pk # Collecting data now.
The IP of "pingerqta.pern.edu.pk" has been changed, Les has updated the databas at SLAC with the following
Old IP: 121.52.157.157
New IP: 121.52.157.148
Follow up from workshop
- Hossein Javedani of UTM is interested in anomalous event detection with PingER data. Information on this is available at https://confluence.slac.stanford.edu/display/IEPM/Event+Detection. We have sent him a couple of papers and how to access the PingER data. Hossein and Badrul have been put in contact. Is there an update Badrul?
The Next step in funding is to go for bigger research funding, such as LRGS or eScience. Such proposals must lead to publications in high quality journals. They will need an infrastructure such as the one we are building. We can use the upcoming workshop (1 specific session) to brainstorm and come up with such proposal. We need to do some groundwork before that as well. Johari will take the lead in putting together 1/2 page descriptions of the potential research projects.
...
- Need to identify a few key areas of research related to PingER Malaysia Initiative and this can be shared/publicized through the website. These might include using the infrastructure and data for: anomaly detection; correlation of performance across multiple routes; and for GeoLocation. Future projects as Les listed in Confluence here
...
The Next step in funding is to go for bigger research funding, such as LRGS or eScience. Such proposals must lead to publications in high quality journals. They will need an infrastructure such as the one we are building. We can use the upcoming workshop (1 specific session) to brainstorm and come up with such proposal. We need to do some groundwork before that as well. Johari will take the lead in putting together 1/2 page descriptions of the potential research projects.
...
- Future+Projects can also be a good start and also Bebo's suggestion.
- Need to synchronize and share research proposals so as not to duplicate research works. how to share? Maybe not through the website, or maybe can create a member only section of the website to share sensitive data such as research proposal?
Anjum suggested Saqib, Badrul and Johari put together a paper on user experiences with using the Internet in Malaysia as seen from Malaysian universities. In particular round trip time, losses, jitter, reliability, routing/peering, in particular anomalies, and the impact on VoIP, throughput etc. It would be good to engage someone from MYREN.
Ibrahim
Ibrahim Abaker is planning to work on a topic initially entitled " leveraging pingER big data with a modified pingtable for event-correlation and clustering". Ibrahim has a proposal, see https://confluence.slac.stanford.edu/download/
...
...
17162/leveraging+pingER+big+data+with+a+modified+pingtable+for+event-correlation+and+clustering.docx. Ibrahim reports 7/15/2014 "I have spent the last few months trying to understand the concept of big data storage and its retrieval as well as the traditional approach of storing RDF data. I have integrated a single hadoop cluster in our cloud. but for this project we need multiple clusters, which I have already discussed with Dr. Badrul and he will provide me with big storage for the experiment." No Update 8/20/2014.
"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"
Potential projects
1) pingER monitoring host on android .
I am using perl-android-apk Interpreter to run pearl on android.
it include execution of 3 pearl scripts
* PingER2.pl
* ping_data.pl
* getdata.pl
2) Resources Required
i need ping_data.pl script file and get data.pl script file .
3) Research Status
Perl-android-interpreter is not executing .pl file mainly because old Perl version (Perl 5.10) which is not comparable with android lollipop 5.0 version, i am compiling Perl .5.22.0 for android ,it may solve issue.
...
Anjum suggested Saqib, Badrul and Johari put together a paper on user experiences with using the Internet in Malaysia as seen from Malaysian universities. In particular round trip time, losses, jitter, reliability, routing/peering, in particular anomalies, and the impact on VoIP, throughput etc. It would be good to engage someone from MYREN.
Ibrahim
Ibrahim Abaker is planning to work on a topic initially entitled " leveraging pingER big data with a modified pingtable for event-correlation and clustering". Ibrahim has a proposal, see https://confluence.slac.stanford.edu/download/attachments/17162/leveraging+pingER+big+data+with+a+modified+pingtable+for+event-correlation+and+clustering.docx. Ibrahim reports 7/15/2014 "I have spent the last few months trying to understand the concept of big data storage and its retrieval as well as the traditional approach of storing RDF data. I have integrated a single hadoop cluster in our cloud. but for this project we need multiple clusters, which I have already discussed with Dr. Badrul and he will provide me with big storage for the experiment." No Update 8/20/2014.
"I have come up with initial proposed solution model. This model consists of several parts. The upper parts of the Figure below shows the data source, in which PingER data will be convert into RDF format. Then the data pre-processor will take care of converting RDF/XML into N-triples serialization formats using N-triples convertor module. This N-triple file of an RDF graph will be as an input and stores the triples in storage as a key value pair using MapReduce jobs"
Potential projects