Page History
| Wiki Markup |
|---|
{toc}
h1. Motivation
Development of this application was stimulated by the discussion with Marcin Sikorski (meeting on 2012-08-30), doing xcs experiments.
Users need in real-time algorithm for calculation of image vs time auto-correlation function
{code}
g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>),
{code}
where {{I(t)}} is an image intensity at time {{t}}, and {{tau}} is a delay between two measurements.
Typical experimental condition can be described as follows:
* Run duration is about one hour at frequency up to 120 Hz that gives up to 10^5-10^6 images.
* Currently typical imaging devise is a Princeton camera with 1300x1340 pixels.
* Need to calculate {{g2(tau)}} for each pixel, averaged over all possible image times {{t}} with time difference {{tau}} between images.
* A set of {{tau}} should have about 30-100 points in log scale uniformly covering the run duration.
* Use for example xcsi0112-r0015: 500 images with 8 sec delay between images.
Desired time for evaluation of the auto-correlation function should be comparable with run duration <1 hour. Currently this algorithm takes a few hours that can not be used for fast feedback in real time experiment.
In first approximation this problem was solved, as it is described in the [PCDS:Command Line Interface For Time Correlation Analysis]. However, in order to be useful, the command line algorithm needs to be integrated in the global analysis system, which _de bene esse_ called as *Integrated Analysis Environment for Time Correlation Experiments* and is discussed in [PCDS:IDPE for TCE in XCS - Problems and tentative solutions].
----
----
----
----
----
----
{HTMLcomment:hidden}
Here is my comment
{HTMLcomment}
h1. GUI Implementation
Updated on 2013-02-13
Three [versions of GUI|PCDS:Versions of GUI for Integrated Data Processing Environment for Time Correlation Experiments] were implemented for this application. It was decided to use the {{GUIMainTB}}
layout, where:
* All input windows are integrated in one with tab-bar for switching and custom tool bar on the top
* Logger and File Browser windows are isolated
* All plots are isolated, although similar plots appear in the same windows.
!cora-xcsi0112-r0015-gui-files-data.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-files-dark.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-files-blamish.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-files-flat.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-files-confpars.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-files-work.png|thumbnail,border=1!
!p-data-img.png|thumbnail,border=1!!p-data-peds.png|thumbnail,border=1!!p-data-time.png|thumbnail,border=1!
!GUILogger.png|thumbnail,border=1!!GUIFileBrowser.png|thumbnail,border=1!
!cora-xcsi0112-r0015-gui-setup-info.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-setupinfo-data.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-analysissettings-dynamic.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-system.png|thumbnail,border=1!
!p-mask-editor.png|thumbnail,border=1!
!cora-xcsi0112-r0015-gui-intensitymonitors.png|thumbnail,border=1!!p-intmon.png|thumbnail,border=1!
!cora-xcsi0112-r0015-gui-run-input.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-run-split.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-run-process.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-run-merge.png|thumbnail,border=1!!cora-xcsi0112-r0015-gui-run-auto.png|thumbnail,border=1!
!cora-xcsi0112-r0015-gui-view-results.png|thumbnail,border=1!
!p-X.png|thumbnail,border=1!!p-Y.png|thumbnail,border=1!!p-R.png|thumbnail,border=1!
!p-PHI.png|thumbnail,border=1!!p-Q.png|thumbnail,border=1!
!p-mask.png|thumbnail,border=1!
!p-phi-stat.png|thumbnail,border=1!!p-q-stat.png|thumbnail,border=1!!p-q-phi-stat.png|thumbnail,border=1!
!p-phi-dyna.png|thumbnail,border=1!!p-q-dyna.png|thumbnail,border=1!!p-q-phi-dyna.png|thumbnail,border=1!
!p-Ip.png|thumbnail,border=1!!p-IpxIf.png|thumbnail,border=1!!p-g2-map.png|thumbnail,border=1!
!p-g2-raw.png|thumbnail,border=1!!p-1oIp.png|thumbnail,border=1!!p-g2-dyna.png|thumbnail,border=1!
!p-g2-vs-tau.png|thumbnail,border=1!!p-g2-tau.png|thumbnail,border=1!
h1. Code location
Everything resides in {{CorAna}} package, which is included in LCLS offline releases beginning from {{ana-0.7.16}}.
h1. Modules
Updated on 2013-03-12
*Notations:*
(/) - is done (at least it is assumed for now...)
(+) - needs more work
(i) - test or deprecated modules
All module names beginning with letters {{GUI}} implements different Graphical User Interfaces.
|| Module || Description ||
| (/) AppDataPath.py | Local version with added path to data from src directory. |
| (/) BatchJob.py | Suprclass for other {{BatchJob*.py}} modules |
| (+) BatchJobCorAna.py | |
| (/) BatchJobData.py | |
| (/) BatchJobPedestals.py | Class contains methods for batch job submission and monitoring for pedestals. |
| (+) BatchLogParser.py | |
| (+) ConfigFileGenerator.py | Class has methods to generate psana configuration and other scripts from stubs located in {{CorAna/data/scripts/}}. |
| (/) ConfigParameters.py | Base class for configuration parameters. |
| (+) ConfigParametersCorAna.py | Sub-class for {{CorAna}} specific configuration parameters. |
| (/) CorAnaUtils.py | Junk for eamples. |
| (/) Drag.py | |
| (/) DragCenter.py | |
| (/) DragCircle.py | |
| (/) DragLine.py | |
| (/) DragObjectSet.py | |
| (/) DragPolygon.py | |
| (/) DragRectangle.py | |
| (/) DragWedge.py | |
| (/) EventTimeRecords.py | |
| (+) FileNameManager.py | Class dynamically generates all file names for current configuration parameters. |
| (/) GUIAnaPartitions.py | |
| (/) GUIAnaSettings.py | |
| (/) GUIAnaSettingsLeft.py | |
| (/) GUIAnaSettingsOptions.py | |
| (/) GUIAnaSettingsRight.py | |
| (/) GUIBlamish.py | Sub-GUI of GUIFiles.py |
| (/) GUICCDSettings.py | |
| (/) GUIConfigParameters.py | GUI for the configuration parameters file management. |
| (/) GUIDark.py | Sub-GUI of GUIFiles.py - manipulations with dark runs |
| (/) GUIData.py | |
| (/) GUIELogPostingDialog.py | Dialog window for submission of messages with attachments to ELog |
| (/) GUIELogPostingFields | Widget with necessary fields for GUIELogPostingDialog |
| (/) GUIFileBrowser.py | Text file browser for this project |
| (/) GUIFiles.py | Central GUI for file settings contains tab-bar for other widget selection |
| (/) GUIFlatField.py | Sub-GUI of GUIFiles.py |
| (/) GUIHelp.py | WIdget for messages, i.e. help |
| (/) GUIImgSizePosition.py | |
| (/) GUIInstrExpRun.py | depricated GUI |
| (/) GUIIntensityMonitors.py | Control GUI for intensity monitors |
| (/) GUIKineticMode.py | |
| (/) GUILogger.py | GUI for logger |
| (/) GUIMain.py | The first GUI at start of this application |
| (/) GUIMainSplit.py | The same as GUIMainTB.py with integrated GUILogger.py |
| (/) GUIMainTB.py | The first GUI at start of this application with tab bar |
| (/) GUINonKineticMode.py | |
| (/) GUIRun.py | |
| (/) GUIRunAuto.py | |
| (/) GUIRunInput.py | |
| (/) GUIRunMerge.py | |
| (/) GUIRunProc.py | |
| (/) GUIRunSplit.py | |
| (/) GUISetupBeamZero.py | |
| (/) GUISetupData.py | |
| (/) GUISetupEnergyAngle.py | |
| (/) GUISetupInfo.py | |
| (/) GUISetupInfoLeft.py | |
| (/) GUISetupInfoRight.py | |
| (/) GUISetupPars.py | |
| (/) GUISetupSpecular.py | |
| (/) GUISystemSettings.py | |
| (/) GUISystemSettingsLeft.py | |
| (/) GUISystemSettingsRight.py | |
| (/) GUIViewControl.py | |
| (/) GUIViewResults.py | |
| (/) GUIWorkResDirs.py | |
| (/) GlobalExternal.py | |
| (+) GlobalUtils.py | Module contains all possible global methods. |
| (i) ImgSpeNavToolBar.py | Re-implemented standard tool-bar (depricated) |
| (/) Logger.py | Core class for logger |
| (/) MaskEditor.py | |
| (/) MaskEditorButtons.py | |
| (i) Overlay.py | Demonstration of how to draw something on the top of GUI |
| (/) PlotArray.py | Main class for array presentation plot |
| (/) PlotArrayButtons.py | Widget for custom button-bar |
| (/) PlotArrayWidget.py | Widget for graphical window |
| (/) PlotG2.py | |
| (/) PlotG2Buttons.py | |
| (/) PlotG2Widget.py | |
| (/) PlotImgSpe.py | Main class for interactive plot with custom button bar |
| (/) PlotImgSpeButtons.py | Widget for custom button-bar |
| (/) PlotImgSpeWidget.py | Graphical image for image and spectral histogram |
| (/) PlotTime.py | Main class for time record presentation plot |
| (/) PlotTimeWidget.py | Widget for graphical window |
| (/) ThreadWorker.py | |
| (/) ViewResults.py | |
| (+) data/scripts/psana-*.cfg | Stub-scripts for ConfigFileGenerator.py |
h1. Status of the project
*Updated on 2013-02-07*
h2. PSANA modules
h3. For dark run pre-processing
*Scaner*
Runs in batch for the dark file to get preliminary information.
(/) {{ImgAlgos.ImgTimeStampList}}
* counts number of events in the file
* makes file with time stamps
* evaluate time intervals between frames (for dark run)
*Pedestals*
Runs in batch for the dark file to get averaged pedestals.
(/) {{ImgAlgos.PrincetonImageProducer}} gets image from event as an {{ndarray}} object
(/) {{ImgAlgos.Tahometer}} evaluates performance of the batch job
(/) {{ImgAlgos.ImgAverage}} produces file with averaged image for given range of events.
h3. For data pre-processing
*Scaner*
Runs in batch for the data file to get preliminary information.
(/) {{ImgAlgos.Tahometer}} evaluates performance of the batch job, counts number of events in the data file
(/) {{ImgAlgos.ImgTimeStampList}} makes file with time stamps and time record counters for tau
(/) {{ImgAlgos.IntensityMonitorsData}} makes file with intensity monitor records
*Average*
Runs in batch for the data file to get averaged image
(/) {{ImgAlgos.Tahometer}} evaluates performance of the batch job
(/) {{ImgAlgos.PrincetonImageProducer}} gets image from event as an {{ndarray}} object
(/) {{ImgAlgos.ImgAverage}} produces file with averaged image for given range of events
(/) {{ImgAlgos::ImgMaskEvaluation}} - module is configured to evaluate masks:
* saturated mask - pixel is considered as saturated if its amplitude exceeds the threshold at least once per run
* noise mask - pixel is considered as noisy if its amplitude exceeds the MEAN+5\*RMS in 5\% of events. The "noise" MEAN and RMS are evaluated for 8 (or less on the boarder) surrounding pixels.
h3. For data processing
*Split*
(/) {{ImgAlgos.Tahometer}}
(/) {{ImgAlgos.PrincetonImageProducer}}
(/) {{ImgAlgos.ImgVsTimeSplitInFiles}}
(/) {{ImgAlgos::ImgCalib}} module is configured to:
* subtract pedestals obtained for dark run
* account for threshold (LLD) constant or in number of RMS
(-) {{ImgAlgos::???IntensityMonitorsData}} and {{ImgAlgos::ImgIntMonCorr}} Intensity monitors
*Process*
Data processing is implemented in stand alone (non-psana) modules
(/) {{ImgAlgos.CorAna.cpp}}
(/) {{ImgAlgos.CorAnaInputParameters.cpp}}
(/) {{ImgAlgos.CorAnaData.cpp}}
*Merge*
(/) {{CorAnaMergeFiles.cpp}} - saves binary file for float(32) with shape (Ntau,3,rows,cols), where 3 stands for <Ip>, <If>, and <Ip*If>
Can be acessed in python as
{code}
sp.cor_arr = np.fromfile(sp.fname, dtype=np.float32)
<image-size> = rows * cols
nptau = <file-size>/<image-size>/3
sp.cor_arr.shape = (nptau, 3, rows, cols)
{code}
h2. GUI
The system of GUIs, consisting of dozens of {{CorAna.GUI...}} modules is implemented in the draft approximation. Roughly it reproduces all features of the old program.
h3. Input parameters GUIs
(/)(+) {{CorAna.GUI...}} most of them are available. Will be added or extended if necessary.
(/) Files - define input files and do pre-processing
(/) Setup Info
(/) Analysis Info
(/) System
(/) Intensity Monitor (-) needs to be used at processing stage
h3. Run GUI
(/) Input - short summary of input info for data processing
(/) Split - control and monitoring for the 1st stage of processing
(/) Process - ... 2nd stage ...
(/) Merge - ... 3d stage ...
(/) Auto - ... for all 3 stages ...
h3. View Results GUI
Contains a set of control fields for presentation of results
(/) Direct and reflected beam geometry is implemented in {{ViewResults.py}}. Currently the switch between the direct and reflected beam geometry is used from tab status variable: {{cp.exp_setup_geom.value()}}.
(+) If the PV variable will be used, then the switch should be changed in {{ViewResults.get_q_map(sp)}} module.
h2. Graphics
(/) {{PlotArray*.py}} - for Intensityintensity Monitorsmonitors
(/) {{PlotImgSpe*.py}} - for images, partition maps, masks etc.
(/) {{PlotTime*.py}} - for time stamp monitoring
(/) {{PlotG2*.py}} - G2 plot peresentation
(/) {{MaskEditor*.py}}, {{Drag*.py}} - [PCDS:Mask Editor] for
* region of interest (ROI-mask)
* blemish mask
h2. Infrastructural modules
Infrastructural modules provide basic infrastructure of the project.
(/)(+) {{ConfigParameters.py}}, {{ConfigParametersCorAna.py}}, and {{GUIConfigParameters.py}} provides convenient approach for maintenance of all configuration parameters.
(/) Infrastructural modules {{Logger.py}} and {{GUILogger.py}} provides a generic approach to logging system.
(/)(+) Module {{ConfigFileGenerator.py}} use current settings of configuration parameters and stub-file scripts from {{CorAna/data/scripts/}} and generates the psana configuration files.
(/)(+) Module {{FileNameManager.py}} is a single place which provides a dynamic file names for current version of the configuration parameters.
(/) {{GlobalUtils.py}} - global utilities for common operations
(/) {{BatchJob.py}} - superclass for batch job submission
(/) {{BatchJobPedestals.py}} - pre-processing for dark run files
(/) {{BatchJobData.py}} - re-processing for data files
(/) {{BatchJobCorAna.py}} - main data processing - calculation algorithm.
h1. Progress
See [PCDS:Weekly progress of the IDPE project for TCE]
h1. To-do list
Processing (at psana Split level)
(-) Intensity monitors: needs to be used at processing in psana
(-) Get {{cp.photon_energy}}, {{cp.nominal_angle}} and other PV variables from data scan
View Results
(-) more plots for results
(-) fits
ELog
(+) Add submission in ELog using response ID and Igor's new interface: [Python module for posting message into ELog]
(+) GlobalUtils: comment out/uncomment submission in ELog
|
Overview
Content Tools