Maggie-NS Tasklist

NEXT MEETING: 9:00 AM California, USA Time, Thursday 24th April 2008, VRVS or Skype video (in Les' directory under Umar), call Arshad 9 011 92-51-928-0443

Team members (* means attended previous meeting): Arshad, Fahad*, Qasim Bilal Lone*, Shahryar*, Umar*, Jared*, Les*, Yee

Mailing lists: maggie-niit@ns.niit.edu.pk; backup: maggie-niit@googlegroups.com

Awaits something, also provides start of wait date
Done or drop, remove < 2 months are task done
: Person doing the work
Bold face: task in progress, or for discussion at meeting
Recent progress/discussion. Need student to be assigned. Critical.

Action Items

PingER: Shahryar, Les

  1. PingER web services: ?
    1. Extend ViPER to turn into front end for PingER, fix bugs and  extend to Real Time (see ViPER Extension Details and Mapping PingER Metrics): Nauman (has left), Shahryar
      1. Nauman has a real time version working with ping_data.pl. looks good.
      2. Fix bugs
        1. Still have points in to LH corner
      3. Make JNLP loadable.
      4. Add long term so replace old ViPER
    2. Ping-data.pl form only shows beacons.
      1. Need to point $sites_fn to pinger.xml instead of beacons.txt & check if the file is xml or flat (for backward compatibility): Shahryar
    3. Case Studies
      1. Case study for E. Asia: Umar, Farhan, Shahryrar
        1. Rough draft ready including comparison with other countries, needs more study to make presentable
    4. Monitoring Hosts:
      1. Contacts with Peru, Argentina, Mexico, Chile: Les
      2. Trying to re-energize African countries in association with IHY and Montpellier meetings
      3. Setup node in UNSW, AU: Umar

Triangulate position of an unknown host (see http://www.slac.stanford.edu/comp/net/wan-mon/tulip): Shahryar, Les

  1. Visualization 
    1. When zooming maintain size of dots (awaits TULIP2)
  2. Bugs/questions (await TULIP2):
    1. How do Sno correlate when there are landmarks from SLAC and PlanetLab. It appears Sno for PlanetLab in Ping Results is not same as Sno in PlanetLab landmarks? (currently it is by order of of appearance in Ping Results 10/2/07):
    2. The alpha optimization using PingER data seems to do nothing useful:
    3. Confidence Estimates table does not sort on table heading (what does it mean, what are units, will replace with IQR/Median 10/2/07):
    4. Location History needs headings or something to explain what the columns are:
    5. Change Ladnmarks to Landmarks on tab in Ping window (reported 9/21/07, will be done in future release 10/2/07):
    6. Sort of LG/SLAC ping results for target = www.cern.ch of sort of  Min RTT gives the top entry as 1 (SWITCH,Switzerland), the next is 0.820 (CERN, Geneva), i.e. it has mis-sorted (reported 10/8/07):
    7. PlanetLab ping results table does not sort by table heading (reported 10/8/07):
    8. The alpha test under Discovery Statistics needs to throw away useless values of RTT else plot scaling can be useless:
    9. The alpha optimization using PingER seems to do nothing useful:
    10. South Africa landmark only accepts IP addresses so need to accommodate:
    11. What is fixed vs Min RTT under DiscoveryStatistics (Faran will talk to Asif 10/2/07, not needed now):
    12. Sometimes fails, maybe because when tulip starts it downloads Site.txt from niit server. If that be the case the left side table on the main frame should be empty and status bar should say "Error downloading config. file."
  3. Improvements/TULIP2 with Centralized Reflector
    1. Modify reflector.cgi to use the new sites.xml file, install at SLAC (done 1/12/08): Shahryar, Les
    2. Modify client to use new sites.xml file: Shahryar
    3. Redo NODEDETAILS database to clean up data , improve schema and generate the sites.xml file (done for landamrks only 11/30/07, awaits analyze hourly and ICFA year end report): Jared
      1. Clean up and add rest of PingER data: Jared
      2. Provide user interface
      3. Migrate from old to new version
    4. Allow alpha to change (lot of work, many changes, could improve, need analysis & understanding)
    5. Look at using PingER historical data to optimize alpha (agreed 4/12/07):
    6. Evaluate whether using traceroute data helps (Suggested by Umar & Les 2/26/07, could add 3 more RTTs to minimize from):
    7. Add traceroute location:
    8. Add results from GeoIP:
  4. Landmarks
    1. Get more landmarks for TULIP, ideally need at least one landmark in all major developing countries that do not have direct connections to nearby countries.:
      1. Is IHEP in list?
  5. Documentation
    1. Add information on the interpretive script sent to PlanetLab hosts: Shahryar
    2. Update thesis documentation and make it available online: Umar
  6. Analysis
    1. Provide API (agreed 6/28/07, changes to TULIP mean current API is not working):
    2. Literature search on triangulation, multi-lateration:
    3. Optimize alpha by landmark (after have new location algorithm working, will try 1:2 for US, 1:3 for Europe, 2:15 Africa, done at NIIT, need to deploy at SLAC 4/10/07, Faran is worried about how to group nodes by region etc. and anomalies 6/13/07), hope to review results by 6/5/07): Faran
    4. Map TULIPs accuracy by region. This will help in identifying where to add new landmarks : Shahryar
    5. Analyze Speedtest sites with and without PlanetLabs landmarks. Also study results for North America using all 150 sites, will report results by 03/16/08: Shahryar
      1. Determine which sites have the highest frequency of being selected as one of the top 3 landmarks.
      2. Observe why some nodes are showing results with errors greater than 1000 Km.
      3. (Data used for analysis should be available to everyone (to allow them to conduct analysis)
      4. Why isn't SLAC listed in the list of sites?
      5. Why in the worst cases, the landmarks in the vicinity of the target failed to respond?
      6. Why do the results worsen (in some cases) when using both Planet Lab and PingER nodes?
      7. What values of alpha to use for each tier?
      8. Does the alpha value vary with increase in distance between the landmark and the target, how much and why?
      9. Document the findings
    6. Compare with GeoIP Tool & Hostip.info
      1. Find cases where TULIP beats GeoIP Tool [Done 4.68.116.16, probably backbone routers too]
      2. Compare with MaxMind (see http://www.maxmind.com/app/geolitecountry): Faran
      3. Document comparison in Wiki (is this done?): Shahryar
      4. Confidence estimates of accuracy
      5. Applied to PingER hosts
      6. Outside triangles, area coverage
      7. Effect/need to vary alpha: Faran
      8. Look at big discrepancies
    7. Provide bubble maps of accuracy: Shahryar
    8. What area/population is covered by TULIP
    9. Show accuracy by region and country: Shahryar
  7. Look at publication
    1. Find conference: all
    2. Can we get Cisco interested?: Arshad, Faran
    3. Update documentation: Les, Faran
    4. Read paper at http://research.microsoft.com/%7Epadmanab/papers/sigcomm2001.pdf:
    5. Put Faran's final year project documentation online: Faran, Umar

IEPM: Jared, Fahad

  1. Traceanal
    1. Compress traceanal table needs new version of Simile Exhibit (8/30/07): Qasim
    2. Make nodename clickable to viewgraphviz of traceroute [Done 8/30/07]: Asif
    3. Add comment or help on how to find the real hostname (do we need this)
    4. Prepare distributable version of traceanal (assign to an NIIT student) - Qasim
    5. Get CGI script approved by Security (add Tainting) - Asif
    6. Install in production at SLAC
  2. Deploy at NIIT (80% done 8/23/07): Fahad
    1. Pathload segmentation errors (maybe a gcc version problems) [Done 8/30/07]
    2. Setup node at NIIT: Umar
    3. Improve documentation and FAQs: Jaredg
    4. Problem with mtlaytcp (Jared will look at 8/30/07)
  3. Provide installation package: Tanzeel (QAU)
    1. Lot of co-dependencies
    2. Fahad working on  document how he installed, problems and how resolved, will upload to SLAC
  4. Provide ViPER like interface with realtime to IEPM data: Fahad, Shahryar
    1. Problem with node visibility with FireFox (OK with MSIE 6): Fahad
    2. Problem with Java time module
    3. Need to understand how to get updates from iepm in real-time (try accessing URL from web browser): Fahad, Shahryar
      1. Write his own script which will access the data from IEPM-BW database or use Web Service which Asif will develop. So that historic data can be shown on the visualization
  5. Provide data to perfSONAR
  6. Six-month plan for improvements/enhancements, have a draft and comments.
  7. Put together a proposal for an IEPM archive server for a student's final year project, see https://confluence.slac.stanford.edu/display/IEPM/Fahad%27s+proposal): Fahad
  8. DB Schema: Fahad
    1. Revision of schema to include information required by perfSONAR, PingER and TULIP: Jared
      1. Incorporate suggestions made by Jared: Fahad
    2. Stress testing of the data base and documentation of the findings: Fahad - deadline 06/03/08
      1. List all the tables with their schema for the Monitoring node as well as the Archive Server.
      2. List all possible SQL queries that application will use to respond to user queries as shown here 1
      3. List all possible SQL queries that will be run internally
      4. Run all the queries in 1, 2 & 3 to test for correctness
      5. Analyze the performance results
        1. Add the details of the result sets (# of rows effected etc)
      6. Document your findings
    3. Implementation of the script that aggregates the results
    4. Implementation of the script that transfers the summary & raw data to the archive servers
    5. Deploy the software on maggie2 (archive server) and a prototype monitoring node
    6. Implementation of the scripts triggering tools used by IEPM and consequently parse the results and enter them into the database
    7. Test and deploy the setup at different nodes. Explore the possible scenarios where the setup would include about 60 nodes at various locations in the PERN network.
    8. Devise and implement a scheduler to run tests at appropriate intervals
    9. User interface to view/edit information such as contact details, node description, etc

Netflow

  1. Frontend - Yee, Ahmad
    1. Offer suggestions on interface improvements (ahmad)
    2. Fix incompatibility with IE6,7 (Ahmad)
    3. Javascript documentation - non currently exists, inline with javascript
    4. spider & pie charts (Shahryar)
      1. Tidy up problem with legends
      2. implement PNG functionality
      3. Documentation on wiki
    5. Implement and test distribution mechanism for frontend (web) code into installation scripts (Ahmad)
    6. Implement web frontend as virtual host for apache (Ahmad)
    7. RRD probs
      1. Get Yee an account so he can look around on Ahmed's host to help understand problems.
  2. Backend (Akbar, Yee)
    1. Make scoreboarding more useful in terms of categorising under subnets, application and countries.
    2. RRD file organisation - refactor code to enable full paths in rrd files and also /source/dest/source-dest_<type>.rrd file location.
    3. Test Ahmed's new installation script (Akbar)

NETRAT, SLAC Contact: Adnan, Les, Umar interested

  1. Evaluate PCA multivariate method of Crovella: Waqar, Adnan,
    1. Compare PCA results with Holt-Winters/plateau: Adnan
      1. Fetch Data From SLAC, BNL, CALTECH (Pathchirp, Thrulay, Ping): Adnan
      2. Fetch data from CERN
      3. Apply PCA on same data and to get potential alerts: Adnan
      4. Comparison of Run-period results and PCA results: Adnan
      5. Improve event detector: Adnan
      6. Docment findings on PCA: Adnan
  2. Forecasting comparison of Active vs. passive data: Fareena, Adnan
    1. Look at eview implementation of ARIMA (QU has copy): Fareena
      1. Extend to more sites/metrics, see need for Integration & higher orders: Farena, Adnan
      2. Try with longer time base: Fareena
      3. Get time estimates : Fareena
    2. Detailed Graphs description: Fareena
    3. Regularization [Done]
    4. Look at moodss (http://jfontain.free.fr/statistics.htm): Fareena
    5. Send results/documentation to SLAC [Done 3/15/07]: Fareena
    6. Share documentation with COMSATS: Umar
    7. Review write up, and send changes by FAX to +92-51-9280782: Les
  3. Automatic event isolation: Adnan, Yee
    1. Review the results we are getting at the moment
    2. Provide email lists to get all reports from all monitors together; Jaredg
  4. Extend to closer to real time, especially for loss of connectivity or more frequent measurements

PerfSONAR

  1. Middleware
    1. Document User API for perfSONAR: Asif, Qasim
      1. how does this differ from perfsonarui?
    2. Study the perfSONAR NMWG request/response schema [Done 5/1/07]: Qasim
      1. Document NMWG request/response schema on wiki (append to existing docs): Qasim
    3. Output from some script (mtml.pm?) is unintelligible (e.g. control characters) need help from Asif, Nauman will send snapshot o Asif (5/10/07)
    4. Having Java socket error, maybe firewall problem: Asif
    5. Add updated code into SVN: Asif
  2. Make traceroute.pl (reverse traceroute and ping server) perfSONAR compliant.
  3. Visualisation
    1. Provide 3-dimensional spinning (interactive) globe interface like with mona-lisa in java 
      1. integrate into Asif's topoviz frontend: QasimB
        1. Will send screen shots (7/26/06): QasimB
        2. Consider GoogleEarth for 3D Topology visualization: QasimB
      2. integrate into perfsonarUI: TBA
        1. Map relevant links & paths perfSONAR node information (see https://svn.internet2.edu/svn/nmwg/trunk/nmwg/schema/rnc/topo for schema) .
        2. Integrate into perfSONAR libraries Asif was working on
          1. Get documentation on perfSONAR user libraries: Asif
      3. Make useful for PingER?
        1. Display mon & remote sites
        2. Mouse overr provides site details
        3. Mouse click on monitor provides sites-per-country.html like output for monitor node
        4. Select metric & ticks src dest submit pingtable request and get plot.
    2. Googlemaps 
      1. Download and understand code from internet2 svn
      2. install locally
      3. Investigate google maps API's and extensions to provide 'traffic' like visualisation (look at google maps at the Bay Area for example)
    3. Ping summary table: Nauman
      1. status? (was first thing ever assigned to nauman)

PerfSONAR-UI

  1. General: Qasim Bilal, Yee
    1. Study the link and documentation of PerfSONAR (available online): Qasim
    2. Application Requirements
      1. Get the data from MA for Pinger
      2. Parse it to show mesh
      3. Draw utilization graph for site(s) selected
      4. Check for schema deficiencies
  2. PerfSonar Java Libraries for Backend: Qasim
    1. PerfSONAR user libraries - i.e. libraries that will make it easy for others to gather and consume data from perfsonar services. the specific requirments are to integrate the Topology Service, Lookup Services, the RRD/SNMP MA's, and Status MA's
      1. Using XQuery or templating library query the schema.
      2. Study and understand XQuery
      3. Implement it using the schemas given here
  3. 3D User Interface for Pinger and Perfsonar: Qasim, Les, Yee
    1. Create nodes (with colour/icons) and links (with either arrows or a moving animation of direction, with colour)
    2. Automated spinning
    3. Zooming and panning
    4. Conversion from 3D Globe to 2D image
    5. Integration of this code with PerfsonarUI and also maintain separate copy.

Route Visualizer

  1. Correct traceroute number in raw file, currently it is always 1 [Done 8/30/07].
  2. Correct nodes (routers mainly) identified like n0x9435b80, tried deleting in dot file but more complex (9/15/07)
  3. Some routers identified by ip address but should be name
  4. Fix the legend or whatever it is [Done 8/30/07]
  5. Many end countries have no node [Done 8/30/07]
  6. Add unique traceroute number to end node
  7. Some end nodes are ellipses, should only be rectangles: Akbar
  8. For Africa there are some end countries such as CH, NO that are not in Africa [Done 8/30/07]
  9. Can we include the RTT in the end node [Drop probably not very useful and a lot of work]?
  10. Can we color the links by the RTT [Drop 8/30/07]?
  11. Some countries (e.g. SN for Africa) have no name
  12. Why for Kinshasa router server in DRC (CD) is it labelled CG (Congo) [Understood]?
  13. Add as a comment the landmark, userid, executing host, name & path of executing script, date  to raw and annotated traceroute data (Les is working on 9/1/07): Les
  14. Need to allow replacement with Hostip.info see for example http://www.maxmind.com/app/geolitecountry and http://www.maxmind.com/app/geolitecity (Akbar will do 8/9/07): Akbar.
  15. Look at how to disambiguate horizontal lines:
    1. Write script to remove the subgraph clusters while retaining the country names.
    2. Play with the ranksep (which increases the vertical separation)
    3. Randomly color edges: Akbar
  16. Need to update the documentation,
    1. Indicate how to interpret  the topology graph [Done Les 9/1/07]
    2. The information on debugging seems to need attention [Done Les 9/15/07].
    3. Reduce the width needed [Done Les 8/23/07]
  17. Does not correctly show end country, in the example below the route is identified from BR to UY but it should be BR to PE. Since the tracveroute terminates before Peru it does not find the end country [Done 8/30/07].
traceroute to 200.37.45.39 (200.37.45.39), 30 hops max, 38 byte packets
 1  200.143.197.129 (200.143.197.129)  0.892 ms  0.783 ms  0.759 ms
 2  ge-1-0-0-r1-rj.bkb.rnp.br (200.143.252.181)  3.058 ms  0.590 ms  0.706 ms
 3  so-0-1-0-r1-sp.bkb.rnp.br (200.143.252.21)  6.326 ms  6.250 ms  21.632 ms
 4  rnp-whren-stm.ampath.net (198.32.252.221)  146.245 ms  143.362 ms  143.321 ms
 5  nlr.ampath.net (198.32.252.161)  177.944 ms  178.762 ms  181.123 ms
 6  216.24.184.90 (216.24.184.90)  341.576 ms  341.484 ms  342.770 ms
 7  saopaulo-buenosaires.core.redclara.net (200.0.204.37)  369.670 ms  369.690 ms  370.923 ms
 8  buenosaires-santiago.core.redclara.net (200.0.204.29)  389.121 ms  389.505 ms  389.805 ms
 9  raap-pe-sant.peer.redclara.net (200.0.204.158)  480.197 ms  484.125 ms  479.740 ms 10  * * *
11  * * *
12  *
__END__

Communications etc.

  1. Projects (see http://www-iepm.slac.stanford.edu/pinger/projects/)
  2. Resolve next graduates/RAs/visitors: Arshad, Les
    1. Umar tentative arrival Feb 17, 2008
    2. Next student probably in July/Aug 2008
    3. Umar is hiring RAs (3 people for MAGGIE for 2 years). Shahryar 7 Asif will apply, could be 6 or 7 people.
  3. Paper for Nepal can't get travel funding, investigating whether will accept paper without attendance: Shahryar
    1. Funding for registration fee acquired. Paper has been registered. Waiting for title of the proceedings and access to an online copy: Shahryar
  4. Paper on event diagnosis: Adnan
    1. Future plans Needs improving: Adnan
    2. There are now (1/12/08) 2 students working on diagnosis, and one on Network Weather Forecasting. If they turn out well, they will be introduced to SLAC.
    3. Hope for something in a few months
  5. Proposals
    1. HEC funding approved
    2. USAID proposal due 6/30/07### Troubles have delayed review from November to January
    3. Proposal to Cisco on TULIP:  Shahryar
    4. Proposal to Cisco on perfSONAR (sent, await response): Les
  6. Progress Reports from NIIT
    1. Customized the net-weekly.pl script and deployed at NIIT (03/03/08): Umar
    2. Students to maintain rough drafts of their project descriptions at NIIT's wiki
  7. WIKI Account at SLAC
    1. Couldn't find details of resetting the password for shahryar's (confluence) wiki account. Applied for a new account.
  • No labels