Versions Compared

Old Version 16

changes.mady.by.user Les Cottrell

Saved on

compared with

New Version Current

changes.mady.by.user Les Cottrell

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Anjum+, Hassaan Khaliq?, Kashif, Raja,  Samad Riaz?, Johari+, Nara, Adnan Khan+, Abdullah, Badrul, Ridzuan, Ibrahim?, Hanan, Saqib+, Adib+, Les+, Renan, Bebo+

+ Confirmed attendance

? Emails sent just before the meeting as a reminder since we needed updates.

- Responded but  Unable to attend: 

Actual attendees:
  • Adib, Johari,

...

  • Adnan, Les, Bebo and Hafizi Jalil of MYREN.
  • Saqib

...

  • had network problems,

Administration

  • Anjum had a meeting that ran over.
  • Hassaan had a death in the family he will update the NUST hosts information.

Administration

  • Hafizi Jalil of MYREN  joined the meeting. He introduced himself. He has been at MYREN for 1.5 years. He maintains developments for the MYREN network. He has PingER installations at Cyberjaya and UNIMAS. He has more machines, currently running perfSONAR at UMP, UTM, UM and another site. This week he will install PingER and the traceroute servers on these machines. A goal would be to compare and contrast the benefits of perfSONAR and PingER.
  • Anjum and Raja have been working on a paper on
  • Following the workshop Johari contacted a  MYREN technical guy who seemed very interested and there has been an exchange of emails. PingER monitors have been installed and are working at MYREN hosts in Cyberjaya and at UNIMAS. The traceroute servers also work. They plan to add an extra 10 monitors.  With some good fortune (Insha'Allah) he may join us.  he has been added to the contacts. He has been invited to join Skype. He will also need adding to the google group pinger-my.
  • Anjum and Raja have been working on a paper on Geolocation as developed for TULIP. Using an exponential relation between the Directivity (Alpha) and RTT for Pakistan the accuracy is ~ 18Km. Now Raja needs to run for Europe and the US. Meanwhile Raja has got a job and has less time to work on this so it was stalled. Les contacted Raja and Raja agrees it is important to finish the measurements and the paper, and will endeavor to do so. Anjum is currently working on another paper. Once he finishes that, If Raja didn’t find time to complete the results, he will try to see if he can take over and run the code him-self.
  • Anjum's contract has been extended for 1 year
  • Saqib's contract expired last month, he will be returning to Pakistanis back in Pakistan.
  • The 2015 ICFA/SCIC reprt report is completed. It is available atBebo will be in Kuching for the CITA 2015 (see at http://www.slac.stanford.edu/xorg/icfa/icfa-net-paper-jan15/report-jan15.docx
  • Hafizi Jalil will send Les his email address.  Les will ask Badrul to add it to the pinger-my email list. Done 2/5/2015, Badrul has added Hafizi and added the ability for Les to manage the list. LA copy of the list is at: Membership of pinger-my
  • Johari will add Ibrahim to the PingER contacts list. Done 2/5/2015

  • Johari has got the OK from the conference organizing committee to hold a colocated PingER/BigData workshop on August 3rd the day before the  CITA 2015 (see http://www.cita.my/ an International Conference 4th - 6th August 2015, on transforming Big Data into Knowledge. Johari will provide relevant information to Bebo. Bebo will be able to make a presentation.  Also Ridzuan or Ibrahim or Renan have interest in submitting a full paper by March 2nd 2015. cita.my/ an International Conference 4th - 6th August 2015, on transforming Big Data into Knowledge, Sponsored by UNIMAS and including workshops) that precedes the RAIN FOREST MUSIC FESTIVAL. Is there interest in co-locating a PingER workshop?  Also Ridzuan or Ibrahim or Renan have interest in submitting a full paper by March 2nd 2015. Johari will suggest to the conference committee on having a workshop/tutorial session about PingER project. We need a specific topic for the workshop, and it should be inline with the conference theme which is about big data, 

UFRJ

They are having some difficulties trying to get a proper platform to run our tests. They have already received funding for a new infrastructure but to get everything installed and running took us longer than they  thought, as it is summer vacation in Rio and part of the staff at the university is not at work.  

...

They are now preparing a test plan (like a small benchmark) to be used on all alternatives so that we can compare the results accordingly. 

Les will contact Maria Luiza Campos of UFRJ (now on a one year leave of absence in Italy) to see if we can repeat the visit of Renan with a new student. Done 2/5/2015.

Les has requested (in January) Renan to provide an estimate of how DF bloats the data. Renan/Christiane are looking at this. Renan's pointed out "Finding RDF data size in bytes is not simple because it depends on which Triple Store will be used and how each triple is physically stored. One may store triples as plain texts, other may do as compressed data in specific formats, which would be much smaller."Once we have the number of PingER triples and how much the used Triple Store needs (in bytes) to store a known number of general triples, we may estimate PingER RDF data size.". Requested an update by email to Cristiane & Renan 2/4/2015.

...

pinger.uum.edu.my is down Jan 31st, Feb 1st. Adib reports it is currently down disconnected due to a technical problem at UUM computer centre. Adib is trying to get more reliable power (UPS).

UM

There has been no feedback to an email to Ibrahim (2/4/2015). Last we noted was:

...

Adib has been discussing with Anjum looking at potential PingER  projects. Adib has a master student. In particular they are interested in providing more flexible access to PingER data rather than the limited time windows pingtable.pl provides. This may be by providing database access rather than using flat files. Les will provide:

  • Access to Pinger Data in flat files via FTP.  Done 2/5/2015. Les has sent information on the archive available via anonymous ftp
  • Perl scripts for: making measurements, gathering the data, and analyzing the data. Done 2/5/2015: Les has sent getdata.pl (the gathering script), wrap-analyze-daily.pl (the initial analysis script that takes the raw gathered data and creates hourly data points for each metric),  the link to the measurement script pinger2.pl
  • Documentation on the data formats. Done see: http

...

UM

There has been no feedback to an email to Ibrahim (2/4/2015). Last we noted was:

Ibrahim has setup distributed hadoop clusters. He has 2TB of disk space. Les has provided information on getting a subset of PingER data by anonymous ftp via ftp://ftp and some on the dataflows at https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC. Renan at UFRJ has successfully used this datausers/cottrell.  It was put there last September. Information on how the data was put together is at https://confluence.slac.stanford.edu/display/IEPM/Archiving+PingER+data+by+tar+for+retrieval+by+anonymous+ftp. There is information on formatting etc at http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html and some on the dataflows at https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC. Renan at UFRJ has successfully used this data, he has also characterized the data in terms of bytes/metric per year etc.

Ibrahim has started downloading all zip files in the local machines. 6 weeks ago he downloaded 2 GB of Weather data to test his nodes cluster, he  wrote a simple Java program (Map, Reduce) to find the Average and it was working fine. 

UNIMAS

The two major issues with the Raspberry Pi would be:

Johari had no updates 2/4/2015. His top priorities are:

  1. Reviving the Raspbery Pi
  2. Getting the research student going on anomalous behaviour detection methods.

Johari still has to uncover the problem of the traceroute from UNIMAS. UDP has been unblocked. The MYREN  host works fine and share most of the hops. Thus the problem must be in the first few hops.

From previous meetings

The two major issues with the Raspberry Pi would be:

  • are the results statistically the same as for the other monitor at UNIMAS (e.g. use the Kolmogorov-are the results statistically the same as for the other monitor at UNIMAS (e.g. use the Kolmogorov-Smirnov test); There is Advanced Project (Master by coursework student) working on the statistics of the data from the raspberry Pi and the production PingER monitor at UNIMAS to see how much they differ.
  • is it reliable/robust is it clear what to do to debug problems remotely (e.g. if it is at Bario).  Looking at the monitoring data I have been unable to collect any from it (it is pingable, and port 80 responds, however the remote traceroute and ping_data.pl are not working) since Oct 20th which does not sound promising. Will need to evaluate the robustness of the unit by doing simulated scenario of various events such as power failure, hard and cold reboot, etc. Johari will need access to computer center to verify it comes up correctly after reboot etc.
  • Johari will go to the computer center the coming weekend and look at improving the auto re-start.

If/when it works it would be instructive to look at the data from pinger and raspberry pi to Malaysia since the distances are shorter and the differences may show up better. For Sep-Oct 2014 when there was data measured from both Oct-Nov the averages for 20 paths was 52+-21ms (from pinger.unimas.my to 20 other Malaysian hosts) and 56+-21ms for raspberry pi to 20 other Malaysian hosts.

The traceroute problem maybe the same as for UTM (see below). Johari will request unblocking of the appropriate UDP ports,

Custom iso: He can get as far as the boot Custom iso: He can get as far as the boot screen, but is unable to get to the desktop. It is on hold as of 1/7/2015 awaiting  student with the appropriate skills/background.

...

After revision the FRGS proposal was submitted to RMC. It was not accepted. We need to update it again in order to fulfill the requirements of the grant. Is there an update?

Saqib has updated the case study and is available in Google drive as a "Shared-PingER" document for review at https://drive.google.com/folderview?id=0B-NEKleLll79ZFNmUnhiVGJ0Nmc&usp=sharing_eid (thanks to Bebo who will notify all of how to access). Further it needs some updates from UNIMAS (on Raspberry  Pi),  UM (on big data) and UUM.

The traceroute problem regarding maximum reachable hops ( i.e. 11 hopes ) may be since the Unix/Linux/OSX  traceroute uses UDP to send the requests. The first request is sent to a particular port (33434), with a ttl  to tell it how many hops to go to.  The ttl starts at 1 is incremented as it tries the next hop, also the port is incremented (up to 33465).  It looks like the first few UDP ports are enabled and then they are blocked. The Windows traceroute uses ICMP to send the probes .

...

so does not see the problem..

NUST

Nobody on the meeting from NUST 2/4/2015.

We are unable to resolve the name of the hosts: pinger.uet.edu.pk, pinger.isra.edu.pk,  web.hepgrid.uerj.br

pinger.uob.edu.pk appears to be partially working according to http://www-iepm.slac.stanford.edu/monitoring/checkdata/, however it appears to be unreachable at regular intervals:

...

Added report on Duplicate pings as seen by mining the PingER data, still working on.

Next meeting

Bebo arranged a meeting with the Colombia RENATA NREN folks and the minister of IT to discuss the use of PingER in Colombia. There is a web page at: Colombia. Les has sent an email asking them to install pinger2.pl at at least one site in Columbia.

Next meeting

Next meeting:  Wednesday Mar 4th 2015 8Next meeting:  Wednesday Mar 4th 2015 8:00pm Pacific Standard Time, Thursday Mar 5th  2015  9:00am Pakistan time, Thursday Mar 5th 2015 noon Malaysian time, Thursday  Mar 5th, 2015 02:00am Rio Standard Time.  

...

  1. Quantitative analysis on PingER data
    1. They want to know how PingER has grown, since 1998 until today and how it might be in the next years. By doing this, we may focus on more suitable technologies that deal with scenarios that have a similar profile with PingER.
      1. Two students are working on this.
  2. Approaches to handle PingER current data
    1. Conventional approach – Utilization of Cassandra as back-end database to provide easy crossing of parameters to get PingER data.
      1. One student is working on this.
    2. Distributed and parallel approach – Utilization of a data warehouse on top of a distributed file system to provide low latency response to complex queries (like the ones we were not able to do on my previous work). Additionally, how Scientific Workflow Management Systems may help in the ETL process of transforming PingER so it can easily be stored on the data warehouse.
      1. Renan is working on this.
    3. Pure RDF approach – Good ways of modeling and natively storing RDF data.
      1. Maria-Luiza is working on this.
    4. NoSQL approaches – How other NoSQL DBMS may be adequate for PingER multidimensional data.
      1. Two students are evaluating existing NoSQL solutions for multidimensional scenarios (such as PingER)
    5. Key-Value storages for PingER data in RDF
      1. This is Ibrahim’s work.

In the end, they want to compare all these approaches.

NUST

...

    2. Key-Value storages for PingER data in RDF
      1. This is Ibrahim’s work.

In the end, they want to compare all these approaches.

NUST
Tulip
Follow up from workshop

...