Rough Idea
Idea: take advantage of SAXS/WAXS images all looking the same (mpeg idea) but use a constant image to subtract (perhaps with a scale factor for shot-intensity). Use the raw 16-bit data. How to handle subtraction of gain-switched pixels?
example bits:
UGDDDD
14 bits stored in a 16 bit word
U=unused
G=gain
D=data
could reuse lowest bits of data: they might be noise?
if pixel is in the same range as reference image: D(compressed)=D(shot)-D(ref)
(have to worry about sign-bit: can become negative)
if pixel is not in the same range, store it as-is and mark it as "unsubtracted" with one of the extra bits
try using "gzip" and tell gzip that we're compressing 16-bit numbers
possible future feature? D(compressed)=D(shot)-scalefactor*D(ref) (scalefactor represents brightness of each shot)
could make it better-but-lossy by eliminating low-ADC bits?
Test Data
Looking at det.raw(evt) it appears inner panels for xcs1008621 run 340 are in fixed-low-gain mode and outer panels are in fixed-high-gain mode:
The first few events:
Test Script And Results
I believe the Shannon-entropy is the number of bits of information in each 16-bit word, so we only see a 10% improvement in the compressibility of the data with this approach. Feels like there is quite a bit of entropy in the "noise" of the image even if they appear visually similar.
from psana import * import numpy as np import matplotlib.pyplot as plt from skimage.measure import shannon_entropy import gzip ds = DataSource('exp=xcsl1008621:run=340') det = Detector('epix10k2M') ref=None for nevt,evt in enumerate(ds.events()): raw = (det.raw(evt).astype(np.int32))&0xbfff # eliminate gain bit img = det.image(evt) img_med = np.median(img[612:680,663:726]) # a "normalization area" in the water-ring if ref is None: ref = raw ref_med = img_med residuals = (raw-(img_med/ref_med)*ref).astype(np.int16) f = open('junk%d.dat'%nevt,'w') residuals.tofile(f) f.close() #plt.hist(residuals.flatten(),bins=1000) #plt.show() print(shannon_entropy(residuals),shannon_entropy(det.raw(evt)) if nevt==4: break
Output
(ana-4.0.55-py3) python junk.py 0.0 10.892801295795591 8.99940177591452 10.57374113419068 8.881920763769553 10.716387079966387 9.244652303773888 11.009271179948547 8.73624428989834 10.868081019635044 (ana-4.0.55-py3)