Maggie-NS Tasklist
NEXT MEETING: 9:00 AM California, USA Time, May 10, VRVS or Skype video, call +1 92 321 9830406 or Arshad 92-51-928-0443 or Umar 9 011 92-333-535-3672 (cell)
Team members: Abdullah*, Adnan*, Ahmad*, Azher, Fahad*, Faran, Nouman*, Qasim*, Umar*
Akbar*, Arshad, Asif*, Les*, Shahryar*, Yee
Mailing lists: maggie-niit@ns.niit.edu.pk; backup: maggie-niit@googlegroups.com
Awaits something, also provides start of wait date
Done or drop, remove < 2 months are task done
: Person doing the work
Bold face: task in progress, or for discussion at meeting
Recent progress/discussion. Need student to be assigned. Critical.
Action Items
PingER: Shahryar, Umar Khan, Les
- PingER web services: ?
- PingER & AMP Integration (AMP/NLANR has run out of funding, AMP will shut down 8/31/06. Given the demise of AMP this may need to be changed to integration with PerfSONAR, on the other hand it may be useful for converting from the perfSONAT MP/MA format to PingER): Abdullah
- Document work with AMP integration (hope to complete 11/18/06): Abdullah
- Extend to provide perfSONAR data reading front end (awaits perfSONAR): Abdullah, Asif
- Install & integrate code at SLAC: Abdullah
- Working on password protection: Abdullah
- Improve code, make production quality, automate getting data (requested with suggestions 8/20/06, making progress hope to be done in 3-4 days 8/31/06, sent back to SLAC, SLAC responded further formatting etc. improvements needed 9/12/06, we need to either finish this up or say Abdullah is not going to do it 3/20/07, Abdullah agreed to complete and send 3/22/07, Abdullah sent code but it needs indenting and some example documentation mail sent to Abdullah 4/4/07, Abdullah will have the code ready by 5/3/07): Abdullah
- Needs to work with -w and strict
- Enable user to specify AMP host an option with a sensible default
- Comment what hosts work
- Deploy at SLAC
- Integrate with pingtable.pl: Abdullah
- Create AMP file directory:
- Set up daily crontab:
- Set up analyze-hourly etc: (Done)Jerrod
- Add src & dst amp nodes to Guthrie:
- AMP and Pinger data comparison: Abdullah
- Smokeping & PingER Integration: Shahryar (Asma is no longer working on this)
- Install at SLAC: Shahryar
- Installation on test www-dev test server : Shahryar, Les
Needs to set the cron job on a Solaris machine [Done will use tersk 5/7/07]: Shahryar, Les
- Carried out detail performance analysis of the code. Tried to make the code run faster, by eliminating unecessary instructions. The details of the performance analysis can be found at https://confluence.slac.stanford.edu/display/IEPM/PingER+Smokeping+Integration. We have decide it to run for SLAC only hosts in order to test it for the time being. The results of the analysis shows that it takes 2 hours on average to update all the (536 SLAC to rest of the world RRDs) and will consume approximately 250 MB of disk space.
- The generation and presentation of the Smokeping graphs works
- Make documentation point to Asma's detailed reference documentation too: Shahryar Done
- Look at adding visualization of uptime (Umar will check with Asma about uptime 4/30/07): Shahryar, Asma
- Will use as a top level access to PingER and also as links from Pingtable to display last 120 days using smokeping: Shahryar
- Can/should we monitor display uptime (communicating with Asma 5/1/07, Asma answered): Umar, Asma, Shahryar
- Need to deploy the code on www-wanmon
- Install at SLAC: Shahryar
- Provide real time access to cached data
- Do for SLAC
- Decide on how to extend to other monitoring sites, try with NIIT: Akbar, Umer
- Hassan is looking into the code sent by Fawad (next step is to get ML web services client to get data): Akbar, Hassan
- Do for SLAC
- Ping-data.pl form only shows beacons.
- Need to point $sites_fn to pinger.xml instead of beacons.txt & check if the file is xml or flat (for backward compatibility): Shahryar
- Monitoring Hosts:
- Pakistan hosts
- sfsmds2.nsnl.net not responding (sent email 4/12/07, started responding but hostname incorrect, tried to fix now not responding 4/29/07 sent email, following discussion with Warren sent suggestion 5/1/07 by email, probably will need rebuild): Les
- PingER & AMP Integration (AMP/NLANR has run out of funding, AMP will shut down 8/31/06. Given the demise of AMP this may need to be changed to integration with PerfSONAR, on the other hand it may be useful for converting from the perfSONAT MP/MA format to PingER): Abdullah
Triangulate position of an unknown host (see http://www.slac.stanford.edu/comp/net/wan-mon/tulip,Faran Faran expects to be able to devote more effort to this following 25th Jan 2007): Faran, Les
- Deployment: Faran
- Look into Mathematics of Singular Vector Decomposition technique. (agreed 11/16/06): Faran
- Visualization
- Provide documentation (in progress 2/27/07, documented maths of selection 3/20/07, needs updating, agreed to update 3/22/07, should it be a wiki? - Umar thinks good idea need Faran view 4/30/07): Farhan, Faran
- Can we add AS's to routes: Yee, Faran
- DiscoveryStatistics =>LandMark Analysis need to identify which Site number is displayed (i.e. from which window): Faran
- Sites.txt
- Add diagnostic to Sites.txt to identify bad formatting: Faran
- Allow alpha to change (lot of work, many changes, will improve greatly, changes design, Faran focussing on 4/12/07, need progress)
- Look at using PingER historical data to optimize alpha (agreed 4/12/07): Faran
- Need a way to display alpha used: Faran
- Make project to provide a user interface (e.g. edit) to Sites.txt (Adnan may have a student from his presentation, normalizing students to understand their applicability to task 4/30/07)
- When zooming maintain size of dots (Faran will propose to Umar as a student project 4/10/07, he is concerned about the download time, Asif suggests using NASA images for Java platform, will send link 4/12/07): Faran
- Bugs:
- Sort on min-RTT is alphabetic, should be numerical (reported 5/3/07): Faran
- Double click on line in Ping Results gives web page with wrong landmark.
- Evaluate whether using traceroute data helps (Suggested by Umar & Les 2/26/07, could add 3 more RTTs to minimize from): Faran
- Landmarks
- Get more landmarks for TULIP, ideally need at least one landmark in all major developing countries that do not have direct connections to nearby countries.:
- China (sent email to ctsnet 7/26/06, reminded 12/6/06, 1/27/07): Les
- Try and get sites in Africa (Les sent email 11/28/06): Les
- COMTEC, Japan: Faran
- Need to add site to Sites.txt (5/2/07): Faran
- Add traceroute/Looking Glass servers in critical areas, if they have a common format (asked for 11/15/06, Les did a manual look at traceroute.org and found about 20 potential landmarks 11/17/06, Faran not clear if one can automate 2/27/07, several are automateable, to prove it Les has added 8, Faran will work on, re-requested 3/26/07 with suggestions): Faran
- Get site in Korea at Ajou universitytrying to persuade 5/8/07): Umar
- Get site at KHU (server available, problems with permissions to open UDP sockets 5/8/07): Umar
- Add NetDiGi route servers (asked for 8/30/06, is able to parse them, will look at adding appropriate hosts 3/22/07): Faran
- Get more landmarks for TULIP, ideally need at least one landmark in all major developing countries that do not have direct connections to nearby countries.:
- Apply to PingER node database to check the lat/longs are roughly correct (tried with 300-400 sites but cannnot get results due to fragile Internet at NIIT, will interact with Akbar to make tests at SLAC 8/31/06, exchanged emails on list of PingER hosts to locate 9/3/06): Faran
- Analysis
- Literature search on triangulation, multi-lateration: Faran
Measure landmarks from landmarks [Done 2/15/07]: Faran
- Optimize alpha by landmark (after have new location algorithm working, will try 1:2 for US, 1:3 for Europe, 2:15 Africa, done at NIIT, need to deploy at SLAC 4/10/07) : Faran
- Document Faran's algorithm, try new algorithm that does not require destination to be inside triangle: Faran, Umar
- Analyse the effect of passing IP addresses to landmarks rather than names: Faran
- Look at impact of increasing number of pings for getting accurate min_RTT (Faran measured for a few sites, Les gave ideas on how to extend and analyze for multiple sites, could be a major project 4/4/07): Faran
- Compare with GeoIP Tools
Find a case where TULIP beats GeoIP Tool [Done 4.68.116.16, probably backbone routers too]:
- For US compare Faran's algorithm with GeoIP & simple triangulation for PingER hosts(in progress 2/1/07)
- Confidence estimates
- Applied to PingER hosts
- Outside triangles, area coverage
- Effect/need to vary alpha: Faran
- Look at big discrepancies
- Analysis of Accuracy and applicability
- Accuracy vs known locations (landmarks & PingER hosts): faran
- Provide bubble maps of accuracy: Faran
- What area/population is covered by TULIP
- Show accuracy by region and country: Faran
- Come up with a way to estimate the accuracy/confidence of estimates: Faran
- Document in the TULIP web page: Faran
- Look at publication (helps advance Faran's career and possibly "best project", will need more analysis)
- Find conference: all
- Can we get Cisco interested?: Arshad, Faran
- Update documentation: Les, Faran
- Read paper at http://research.microsoft.com/%7Epadmanab/papers/sigcomm2001.pdf: Faran
Terapaths Yee
- Netflow presentation (Yee sent project to Ahmad http://confluence.slac.stanford.edu/display/IEPM/Netflow+Presentation+Frontend): Ahmad
NETRAT, SLAC Contact: Adnan, Les, Umar interested
- Evaluate PCA multivariate method of Crovella: Waqar, Adnan,
- Compare PCA results with Holt-Winters/plateau: Adnan
- Fetch Data From SLAC, BNL, CALTECH (Pathchirp, Thrulay, Ping): Adnan
- Fetch data from CERN
- Apply PCA on same data and to get potential alerts: Adnan
- Comparison of Run-period results and PCA results: Adnan
- Improve event detector: Adnan
- Docment findings on PCA: Adnan
- Compare PCA results with Holt-Winters/plateau: Adnan
- Forecasting comparison of Active vs. passive data: Fareena, Adnan
- Look at eview implementation of ARIMA (QU has copy): Fareena
- Extend to more sites/metrics, see need for Integration & higher orders: Farena, Adnan
- Try with longer time base: Fareena
- Get time estimates : Fareena
- Detailed Graphs description: Fareena
Regularization [Done]
- Look at moodss (http://jfontain.free.fr/statistics.htm): Fareena
Send results/documentation to SLAC [Done 3/15/07]: Fareena
- Share documentation with COMSATS: Umar
- Review write up, and send changes by FAX to +92-51-9280782: Les
- Automate procedures
- Define project
- Put together presentation for workshop (for Sep 2006): Fareena
- Gather team
- Look at eview implementation of ARIMA (QU has copy): Fareena
- Automatic event isolation: Adnan, Yee
IEPM
- Traceanal
- Get desk-side working to SLAC: Nauman, Azher, Umar
- Compress traceanal table by using image instead of HTML table (hope compleye by May 7) - Nouman, Asif
- Add node name in RH column
- Make nodename clickable to view ping time-series
- Add comment or help on how to find the real hostname
- Prepare distributable version of traceanal (assign to an NIIT student) - Asif, Yee
- Re-install at NIIT to enable viz (he should have appropriate permissions and files from Les 4/10/07): Fahan
- Deploy at NIIT (done 2/1/07, needs cert): Fahad
Fahad's account has access to iepm-bw and to v3src/iepmtools [Done 4/30/07]: Les
- Need iepmtools (should be able to get now account has access 4/30/07): Fahad
- Enter nodes at NIIT (nodes script showed errors with missing perl modules): Fahad
- Need to enter node iepm-maggie.niit.edu.pk at SLAC (Fahad will send how to to Asif): Fahad
- Provide installation package: Tanzeel (QAU), Nauman
- Lot of co-dependencies
- Provide ViPER like interface to IEPM data: Fahad, Shahryar
- Problem with node visibility with FireFox (OK with MSIE 6): Fahad
- Need to understand how to get updates from iepm in real-time (try accessing URL from web browser): Fahad, Shahryar
- Formats for ping and traceroute are different (will have to work around): Fahad
- Write his own script which will access the data from IEPM-BW database or use Web Service which Asif will develop. So that historic data can be shown on the visualization
PerfSONAR
- Provide Google maps interface to PerfSONAR: Shahryar, Qasim (Qasim Bilal with us 6 months starting 3/22/07)
Study the perfSONAR NMWG request/response schema [Done 5/1/07]: Qasim
- Execute sample request queries for utilization and metadata from perfSONARUI: Qasim
- Needs trace route parsing code from Asif.
- Having Java socket error, maybe firewall problem
New project description [Done 5/1/07]: Asif
Pathneck (see http://maggie.niit.edu.pk/website/projects/Topological_Analysis.html) Yee is the lead at the SLAC end: Asif
- Try and get pathneck working from NIIT, study debug output, work with Ningning <hnn@cs.cmu.edu> (awaits site upgrade): Asif
- Update production version at SLAC (12/106): Asif, Yee
- Develop and install pathneck reverse server (would like to start at NIIT just to assure code works work with -y option): Les, Asif
- Modularize security of traceroute.pl: Asif, Yee
- Add in PerfSonar: Asif, Yee
- Evaluate and understand how pathneck needs to be configured, the failure modes, the areas of applicability etc. using iepm-resp.slac.stanford.edu: Asif
- Provide web site documenting failure modes: Asif
- Develop a vizualization
- Topology (currently with Touchgraph), color links, mouseover ID router, provide snapshot
- Click on link/router to get graph
- Incorporate location (e.g. via TULIP)
- Need a schema
- Investigate integration with traceanal and graphviz: Asif
- Integrate topology map with pathneck: Asif
- Look at using PerfSONAR router information (do not agree well, need to get more statistics, to enable multiple measures need API to get PerfSONAR data, look perfsonar UI): Asif
- Try on Ultralight (problems if < 4 hops): Asif
- Problem of 0 results (reported to author but not much happening 4/20/06): Asif
- Add position and AS information to router database: Asif
Communications etc.
- Projects (see http://www-iepm.slac.stanford.edu/pinger/projects/)
- Azher replacement for PERN, COMSATS, NIIT hosts (Ahmed, Siggiqui, Umar?)
- Resolve next graduates/visitors: Arshad, Les
- Get paperwork going
- Paper on event diagnosis (it has been accepted at San Jose, now need to prepare paper in camera ready form): Adnan
It has been withdrawn [Done 4/20/07]: Adnan
- Future plans Needs improving): Adnan
- Get student: Adnan, Umar
- Decide if and who will present (Adnan is trying to get travel approval)### Register paper (SLAC willing to pay fee)
- Reformat paper in IEEE format
- Address most pertinent comments (no student available, nobody working on)
- Do we have another author
- Get copyright exemption
- Get length of paper and figure out extra pages cost, if any
- Submit final paper (by April 20)
- Write & present talk (July 1-6)
- Proposals
- Develop new proposal with more a Grid Health viewpoint (Cisco raised issues, await resolution 3/22/07, talked to people who will supprt, now talking to Cisco. Hope for 100% from HEC, then Cisco chip in 4/12/07): Yee, Arshad, Les, Cisco
- USAID proposal due 6/30/07
- Proposal to Cisco on TULIP: Faran
- Proposal to Cisco on perfSONAR: Les
- World Bank may be able to fund travel
- Work with NSF to see if we submit to them: Les
- send NSF 1 page outline of proposal (Les & Yee are creating 4/12/07, done NSF cannot help)
- Proposal for perfSONAR for HEC (shown to Atta 3/15/07)
- Write up (working on Outline, we have draft, been thru several iterations 4/10/07, Les needs to review section 5, Yee needs to read and comment, Ready to go needs Arshad to review 4/30/07): Umar
Wiki
- Create space for NIIT student reports and populate with initial reports - Yee, Les
Umar has set one up at NIIT and it works OK [Done 4/4/07]: Umar