...
- Aqsa and Saba are working together. Their goal is focusing on visualization of PingER historical data using warehouse. The idea is to develop a warehouse in UAF university and make it publicly available. They are 50-60% done with setting up a Hadoop cluster with 3 nodes, 1 master, 2 slaves. She is currently working on importing the PingER data into hdfs on the cluster. they have run some Impala queries on the data and are working on visualization
- Topic: visualization on pingER data (email from Aqsa and Response from Renan)I have studied the google charts as visualization tools but here are some points need to be discussed.1. The idea of applying visualization on Data warehouse (Impala query results) does not seem to be so useful because Data warehouse contains static data and visualization charts will also remains static and need to be updated with the time.
Yes, it needs to be updated with the time. My suggestion is to transform PingER data into data to be inserted into the data warehouse. Myself and some other Brazilian students have developed codes to do this. Such process should occur at least once a day to keep the data warehouse updated daily. This has never been done by any of us.
2. Google charts API cannot integrate with Impala As Impala is hadoop distributed Big Data supported database Google can only integrate with flat files or flat databases like Mysql.
If Google charts API can only read flat files (e.g., CSV files), it is trivial to save a database query result as a CSV flat file that would be consumed by Google charts. Can Google charts generate a plot dynamically after reading a just-created CSV file?Is using a different data visualization library (e.g., D3 https://d3js.org/ ) an option? Aqsa and team members are working on creating Data warehouse and we are very close to complete this. Here are some updates.
Tehseen qureshi has transformed the pingER text files into binaries and soon he will be able to get CSV files.
- Saba is working on defining a 4 node cluster.
- Aqsa has uploaded some sample CSV files on HDFS and run Impala queries as i will get the actual CSV from tehseen these steps are also will be completed
Visualization Status
Aqsa has
drawn a line chart and bar chart on the data of sample CSV file and i am exploring some more charts to be drawn by using Google API's.
- Sara Masood :
...
- - pingER monitoring host on android .
...
-
- She is using perl-android-apk Interpreter to run
...
- perl on android.
...
- It includes execution of 3 perl scripts:
- getdata.pl
- It includes execution of 3 perl scripts:
...
- , ping_data.pl
...
- , pinger2.pl
First step is to install pre-requisite i.e. lynx, perl, XML::Simple, make etc. However, I am having permission denied issue when i execute "make install” command at path usr/local/bin because it’s a read only file system. I am searching for the solution but maybe I need to change the path to /data/local/tmp.
Step 2 Is to execute ping _data and cgi-bin, traceroute.pl <http://traceroute.pl/>
Step 3: To execute pinger2.pl <http://pinger2.pl/> that will start gathering data
Resources Required
i need ping_data.pl script file and getdata.pl script file.
Joao will provide them via anonymous ftp.
Research Status
Perl-android-interpreter is not executing .pl file mainly because old Perl version (Perl 5.10) which is not comparable with android lollipop 5.0 version, i am compilingPerl .5.22.0 for android ,it may solve issue.
...