The following is a list of web applications that will be used in the October ISOC Operations Simulation, with a short description, their status and to-do list.
The purpose of this document is to collect comments to help us prioritize the work to get the ready for the October deadline.
Data Quality Monitoring
Scope
This application will provide output histograms and data trends resulting from the Fast Monitoring, Digi and Recon processing. The driving process is the L1Proc; root files from each of these processing steps will be registered with the data catalog and picked up by the application in order to display quality histograms. Separate processes will load Digi and Recon tuple files to ingest them into a database for the data trending.
Status
It is currently possible to display histograms from Fast Monitoring, Digi and Recon.
We have a first implementation of the code to ingest trending data into general purpose database tables and managed to produce plots from the web application. The result of this effort was that we need to create new database tables specific to this application.
To-Do List
- Alarm Handling
- The Fast Monitoring process should output an xml file with a list of the alarms/warnings/errors detected on the produced monitoring histograms
- This file should be registered with the data catalog (is it there already?)
- A notification should be added to the Logging application which should point to the Data Quality Monitoring application (is it possible to add the desired target link as part of the notification's metadata?)
- The file needs to be ingested producing summary and detailed information on the alarms/warnings/errors
- Data Trending
- (done) Design tables to ingest the trending data: some 20K quantities at a frequency between 10 seconds and 5 minutes
- (done) We might have several copies of the same tables to accumulate data at different frequencies
- Given the volume of data a database table only approach might be insufficient. We might have to consider a hybrid solution that involves reading data straight from tuple files (less efficient that reading from a db)
- (done) Write the code to ingest the data.
- Find somebody responsible for the ingestion code. Should be somebody that understands the data!
- (in progress Max) Write the code to produce trends
- Improve the application's UI
- User preferences
- Improvements based on user's feedback
Source Monitoring Jira
Scope
Display ASP data products for a pool of sources.
Status
This is the second implementation of this application. We have developed a set of databases to keep a list of sources and to ingest data to be trended at different frequencies. The sources database can be loaded from xml files used by ASP. Scripts are available to ingest the data from the pipeline at the end of ASP processing.
This is still a preliminary version of the application and needs feedback to better define its scope and use.
To-Do List
- (in progress Charlotte) Improve the application's UI
- User preferences
Pipeline Jira
To-Do List
- (in progress Karen) stream and process filtering
- (in progress Dan) Process rollback
- This is something which is really needed by the L1Proc folks. It would be good to look at the existing code and see if it is possible to implement this before rewriting all the stored procedures in Java.
Data Catalog Jira FE Jira BE
To-Do List
- (in progress Dan) Finish migration from dataportal-model to datacat-client
- (in progress Dan) Add the ability to make files from the data catalog search result in the pipeline
- (in progress Dan) Finish proposed changes to datacat line mode client
- easier registration of files
- ability to remove existing files
- (in progress Karen) add meta-data from the web interface
Data Processing
Scope
This application should provide a quick and intuitive look at the status of data processing from the moment it is being Fast Copied through the various processing steps that lead to the final data products.
Status
Only a mock-up version is ready. It was meant to be used to prompt a discussion.
To-Do List
- (in progress Bryson) Downlink-RunId database table
- We need a database table to extract which run numbers are contained in each downlink (Bryson will write and populate this database table?)
- Define the progress mechanism
- Each of the processing steps has to provide some feedback on its progress status. It might be possible to extract it from the Pipeline. Otherwise we have to define a mechanism for it.
- Meet with Tony and Warren to talk about this
Ops Log
Scope
Status
To-Do List
GCN/GRB Web front end
Scope
Tabular overview of most recent Noticies and possibility to browse GRB/Noticies.
Status
- (in progress Jim/Karen/Max) Databases are being designed.
Portal
The portal is the icing on the cake. It will be targeted and developed at the very end.
Scope
Provide a rich and highly customizable environment for viewing data from all the above (and below) applications.
Status
We have developed three portlets to prove that we can extract data from external applications. These portlets can provide tables (from Logging and Fast Copy) and pots (from TelemetryTrending).
To-Do List
Do the rest.
Cross Trending/Reports
Scope
This application should give users the possibility to fetch histograms and data trends from all the above applications and to create scatter plots, overlays or tables of data.
It should also be possible to load simple user-written jsp pages as reports. Users can create a list of favorite reports to be processed at different frequencies (say last 24 hours, last week, last month etc.)
Status
A toy version is available for the cross trending part. The Reports are still in the discussion phase.