Weekly progress of the Integrated Data Processing Environment (IDPE) for Time Correlation Experiments (TCE)
2012-11-05 – 2012-11-11 Week progress
A src/PsanaConfigFileGenerator.py
A src/FileNameManager.py
A src/BatchJobPedestals.py
A src/AppDataPath.py
A src/ConfigFileGenerator.py
A src/GUIDark.py
A src/ImgSpeButtons.py
A src/ImgSpeNavToolBar.py
A src/ImgSpeWidget.py
A src/ImgSpeWithGUI.py
A data/scripts/psana-pedestals.cfg
A data/scripts/psana-tahometer.cfg
2012-11-12 – 2012-11-18 Week progress
A src/GUIFileBrowser.py - file browser for text files of this project.
A src/GUIHelp.py - pop-up box with message as a widget.
A src/GUIWorkResDirs.py - separate widget for setting of work and result output directory and common prefix file name.
In src/GUILogger.py and src/Logger.py - now messages are accumulated in 2d tuple. Add filtration by the message type "debug", "info", "warning", etc.
src/GUIFiles.py is significantly re-organized; now it is GUI with tabs.
All significant fields are moved in GUIs like src/GUIDark.py (GUIData, GUIFlat).
src/GUIDark.py - is re-organized.
In the interactive plot:
- In src/ImgSpeWithGUI.py the src/ImgSpeNavToolBar.py - is depricated
- src/ImgSpeButtons.py - re-organized: add save, reset, and menu buttons, log check-box.
- In src/ImgSpeWidget.py - the log functionality needs to be implemented...
2012-11-19 – 2012-11-25 Week progress
Suggestions from Marcin:
- Time(sec) - should be the averaged time between frame time-stamps.
- Spectral plot: log-scale for X-axis only.
- Flat field and Blamish files will be provided from external apps. No need to evaluate them now. Needs to be selected by the path.
- Substitute the start and end events after the scan.
Progress:
- Restracturization:
tahometer
replaced bypeds_scan
. - update the start and end fields after scan.
- remove psana log files before job submission in batch.
- the start and end fields are set from cofig.pars, if the scan log-file is availabe, they are updated from this file.
- reset fields to default if the input xtc file is changed.
- A src/GUIFlatField.py - file selection, plot, browse file
- A src/GUIBlamish.py
- A src/GUIMainTB.py - try different view for composite window
- A src/GUIMainSplit.py - try different view for composite window
- Add psana module: ImgAlgos::ImgTimeStampList - creates the time-stamp list and evaluates the mean time between frames and its RMS.
- in GUIDark.py use variables produced by the ImgAlgos::ImgTimeStampList.
2012-11-26 – 2012-12-02 Week progress
Suggestions from Marcin:
- Marcin prefers layout with separated GUILogger window, like in GUIMainTB.py
- Need in a tool reporting problems in time-stamp sequence.
- In GUIDark.py - untill the total number of events is unknown, use end "-1" and generate a warning if it is submitted in batch for pedestals.
- Add check-box showing if the Blamish and Flat-field files will be applied
- Marcin returned the paper-copy of GUIs with comments about titles and parameters.
- The list of variables for intensity monitor:
IPM2 [CH0,...,CH3]
IPMMON0 [...]
IPM4 [...]
IPM5 [...]
GASDET [...]
- Needs in separate GUI for intensity monitors for normalization and veto.
Progress:
- In GUIBlamish.py and src/GUIFlatField.py add check-boxes.
- Apply Marcin's comments to GUIs with System, Setup etc. parameters, GUICCDCorrectionSettings.py is removed..
- In src/GUIMainTB.py optimized the GUI sizes.
- Changed names from GUIBatch... to GUISystem...
- New psana module Psana Module Catalog, which gets data for equivalent HDF5 variables:
in psana they can be accessed as:
/Configure:0000/Run:0000/CalibCycle:0000/Bld::BldDataFEEGasDetEnergy/FEEGasDetEnergy /Configure:0000/Run:0000/CalibCycle:0000/Bld::BldDataIpimbV1/XCS-IPM-02 /Configure:0000/Run:0000/CalibCycle:0000/Bld::BldDataIpimbV1/XCS-IPM-mono /Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV2/XcsBeamline.1:Ipimb.4 /Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV2/XcsBeamline.1:Ipimb.5
and share pointers:m_srcFEEGasDetE = configStr("feeSource", "BldInfo(FEEGasDetEnergy)"); m_srcIPM2 = configStr("ipm2", "BldInfo(XCS-IPM-02)"); m_srcIPMMono = configStr("ipmmono", "BldInfo(XCS-IPM-mono)"); m_srcIPM4 = configStr("ipm4", "DetInfo(XcsBeamline.1:Ipimb.4)"); m_srcIPM5 = configStr("ipm5", "DetInfo(XcsBeamline.1:Ipimb.5)");
shared_ptr<Psana::Bld::BldDataFEEGasDetEnergy> fee = evt.get(m_srcFEEGasDetE); shared_ptr<Psana::Ipimb::DataV2> data2 = evt.get(m_src); // for m_srcIPM4 and m_srcIPM5 shared_ptr<Psana::Bld::BldDataIpimbV1> ipimb1 = evt.get(m_src_ipimb); // for m_srcIPM2 and m_srcIPMMono
- In GUISystemPars: add total number of events and dt+-rms to GUISystemPars.
- In GUIDark: add check-box and file-browser button with complete functionality.
- In GUIDark: add protection for pedestal job submission in batch with default number of events.
- In BatchJobPedestals: add protection against duplicated job submission in batch using timeout (now 100 sec) or until the previous job is DONE.
- In ImgSpeWidget: implemented the log-scale plot with full functionality and protections...
2012-12-03 – 2012-12-09 Week progress
No meeting this week due to experiments in xcs hatch.
Progress:
- In GUIFileBrowser: add one more parameters after the list of files, the file to start browser
- GUIData is implemented with full functionality including
- check box to activate/protect buttons
- "File" button and info field - selects the xtc file
- "Scan" button - submits scanner job in batch which
- counts events,
- saves in file the time records ind evaluate an average interval between frames
- saves in file the intensity monitor data
- "Average" button - submits job in batch for averaging image for the specified range of events
- "Check status" button - checks the status of both batch jobs, check if the file with averaged image is available, update the info fields for number of events and averaged time between frames
- "Check files" button - prints the list of work files for data and check their availability
- "Browse" button - open the file browser with a list of work files for data
- "img-Plot" button - plots the averaged image from file
- "t-Plot" button - shows the plot for time records saved in file
- "Remove" button - removes all work files from the work directory
- GUIIntensityMonitors: GUI for Intensity monitors
- PlotArray.py, PlotArrayButtons.py, PlotArrayWidget.py - plot for array presentation
- PlotTime.py, PlotTimeWidget.py - plot for time records presentation
2012-12-10 – 2012-12-16 Week progress
Q&A Marcin:
- Q: XcsBeamline.2 - is it used simultaneously with XcsBeamline.1 ?
- A: XcsBeamline.1
- Q: What control parameters should be added for intensity monitors?
- A: should be added:
- radio button for normalization or no-normalization
- filter data check box
- low-upper amplitude range
- A: should be added:
- Q: Layout of GUIMainTB.py with variable height, is it ok?
- A: Looks fine.
- Q: Layout variation for check-box - disappearance of buttons, is it ok?
- A: Change color of button-titles.
- Q: What image buttons are used as a mask editor etc.
- A: GUIAnalysisSettings:
- "MaskPolygon" button - for polygon editor on data image
- "Show Mask & Partitions" button - shows the mask and partitions on data image
- A: GUIAnalysisSettings:
Suggestions from Marcin:
- Add button to all plots and gui to submit them in ELog.
- Change log presentation for image histogram - log for vertical scale only.
Progress
- GUIDark and GUIData - inappropriate change of style for "end" field is fixed
- GUIDark, GUIData, GUIBlamish, GUIFlatField - have changed behavior for non-active buttons:
- for now it is set to default - frame of buttons disappeared for non-active
- GUIFlatField has an example of how to set a dynamic style-sheet for non-default behavior styles
- PlotImgSpeWidget.py - change log scale from x to y
- GUIIntensityMonitors.py - add more fields:
- radio buttons for normalization
- check boxes for selection
- edit fields for Imin and Imax values with changing behavior and style depending on radio and check box status. Check them for equity to default and Imin >= Imax
- Check move window coordinates for plots... Some of them opens in unexpected places.
- GUIELogPostingDialog, GUIELogPostingFields, PlotArrayButtons, PlotImgSpeButtons, and GlobalUtils: submission of message with attached plots in ELog
- Add icons to some of frequently used buttons.
- Add button for some of GUIs amd plots to submit them in ELog.
2012-12-17 – 2012-12-23 Week progress
Q&A Marcin:
- Discuss layout of fields in
- GUIIntensityMonitors - Is it necessary to use Imax/Imin for normalization?
- GUIELogPostingDialog, GUIELogPostingFields - is it consistent with what you want?
- Layout of buttons
- GUIMainTB - it is possible to change the "buttons with icons and text" -> "buttons with icons" -> "icons". Which style is preferred?
- GUIDark.py, GUIData.py, etc. - style for active/non-active buttons - just default or add more colors, like in GUIFlatField ?
Progress
- GUIDark and GUIData - add check box for parameters cp.use_dark_xtc_all and cp.use_data_xtc_all with appropriate behavior for check box but_path and edi_path fields. Now, the name of the file for batch job is taken from fnm.path_dark_xtc_cond() and fnm.path_data_xtc_cond(), respectively.
- FileNameManager - fixed a few typos in the file name methods: "dark"->"data"
2012-12-24 – 2013-01-06 Week progress (winter break)
Progress
- GUIRun, GUIRunInfo, GUIRunSplit - new style GUI for the splitting procedure.
2013-01-07 – 2013-01-13 Week progress
Progress
- GUIRunProc, BatchJobCorAna, GlobalUtils
2013-01-14 – 2013-01-20 Week progress
Q&A Marcin:
Add lock for ELog submission dialog box
Add lock for nparts field in GUIRunInput
GUIIntensityMonitors:
- change response on radio button
- short intensity monitor names
Progress
- GUIRunInfo change name to GUIRunInput
- GUIRunProc - is completed
- GUIRunMerge
- GUIRunAuto
- GUIELogPostingDialog, GUIELogPostingFields - add lock, restructorization for editable fields.
- GUIIntensityMonitors - (1) remove the selection fields style change for radio button, (2) use a list of short intensity monitor names.
2013-01-21 – 2013-02-03 Week progress
Q&A Marcin:
Reorganize GUISetupInfoLeft.py for CCD position input in modes: Beam Zero, Specular, and Data
Progress
Fixed image partitioning, processing, and merging in ImgAlgos/src/ImgVsTimeSplitInFiles.cpp, CorAnaData.cpp, and CorAnaMergeFiles.cpp
Add CCD orientation in GUICCDSettings
- CCD is always displayed as a matrix of (row,col) with origin in the top left corner.
- CCD orientation is assumed as entire world is rotated w.r.t. CCD
Work on
- GUIViewControl.py
- GUIViewResults.py
- ViewResults.py
- g2 Normalization - need to use parameters from GUISetupInfo etc.
Map evaluation speed in ViewResults.py for 1300x1340 img - x,y maps = 0.02sec
- r map = 0.05sec
- theta map = 0.12sec
- q map = 0.4sec
2013-02-04 – 2013-02-10 Week progress
Progress
Work in progress on
- GUIViewControl.py
- ViewResults.py
- PlotG2*.py
Currently available plots for:
- Raw data maps (tau): <Ip>, <If>, <Ip*If>, g2-raw
- Geometry maps: X, Y, R, Phi, Q for direct beam mode
- Partitioning maps: Phi, Q, Q-Phi maps for
static
anddynamic
partitions - Masks: Image limits, blemish, hot pixels, good regions, total
- Calculated maps (tau): total mask as applied at normalization and for maps
- normalization factors 1/<Ip>, 1/<If> for static bins,
- g2 as a map
- g2 averaged over dynamic bins
- g2 for dynamic bins vs tau as a 2D histogram
- g2(tau) plot for dynamic bins as a set of graphic
Q to Marcin:
- How to calculate angle and Q in reflective geometry?
- Presentation of CCD image and detector orientation issues.
2013-02-11 – 2013-02-17 Week progress
Progress
Work on mask editor application:
- All Drag*.py
- MaskEditor.py, MaskEditorButtons.py
2013-02-18 – 2013-02-24 Week progress
Progress
Mask Editor is completed at 1st approximation. Can be executed as an embedded and standalone application.
- DragPolyline.py is added
- Status "button" is added
- finger-pointer for sensitive objects is added
- I/O features are extended to: Load image, Load forms, Save forms, Clear forms, Print forms, Save mask, save Inv-M(ask).
- standalone application
med
is added with a banch of optional parameters.
2013-02-25 – 2013-03-03 Week progress
Progress
Frame orientation in lab x-y coordinates
Use parameters: cp.y_is_flip.value() (True) and cp.ccd_orient.value()
- Image plot in PlotImgSpeWidget.on_draw(...): flip Y axis in image plot and add rotation
- Mask calculation for figures defined in lab x-y frame: MaskEditorButtons.get_mask_total()
- Pixel maps generation in ViewResults.ccd_pixel_coordinates(sp)
Q: Is it ok?
- Partition limits for q and phi? Use beam x0,y0 and mask total.
- Will use 0-th bin for everything outside the allowed binning area from q
- In Setup Info CCD Image Size: use rows cols, not x,y
- Hot pixel and saturated pixel masks are always on?
2013-03-04 – 2013-03-10 Week progress
Progress
- GUIViewControl.py - hide/show more result buttons
- ImgAlgos::ImgMaskEvaluation - evaluate hot and saturated pixel masks
- Integrate ImgMaskEvaluation module in configuration file
- ImgAlgos::ImgCalib, ConfigFileGenerator.py, ImgAlgos::ImgParametersV1
- subtraction of pedestals
- apply threshold in N*RMS or constant
2013-03-11 – 2013-03-17 Week progress
Q to Marcin:
map_image_refl()
inq_functions.txt
does not work for reasonable set of parameters...wavelength = (1.236)/energy
- should be: 1.23984 ?- Need in PV variables for
- photon beam energy (from electron beam energy Bld.ebeamL3Energy()?)
- (x,y) detector position for direct beam, specular, and data
- detector orientation
- selection of geometry - transition or specular
- Comparison for
/reg/data/ana12/xcs/xcsi0112/xtc/e167-r0015-s00-c00.xtc
– data,/reg/data/ana12/xcs/xcsi0112/xtc/e167-r0020-s00-c00.xtc
– dark
Progress
- Reflected beam geometry is implemented in
ViewResults.py
inget_q_map_for_rb(sp)
,get_theta_map_for_rb(sp)
,get_reflected_beam_geometry_pars(sp)
modules.
Currently the switch between the direct and reflected beam geometry is used from tab status variable:cp.exp_setup_geom.value()
If the PV variable will be used, then the switch should be changed inViewResults.get_q_map(sp)
module.