Rough Idea

Idea: take advantage of SAXS/WAXS images all looking the same (mpeg idea) but use a constant image to subtract (perhaps with a scale factor for shot-intensity).  Use the raw 16-bit data.  How to handle subtraction of gain-switched pixels?

example bits:
UGDDDD

14 bits stored in a 16 bit word

U=unused
G=gain
D=data

could reuse lowest bits of data: they might be noise?

if pixel is in the same range as reference image: D(compressed)=D(shot)-D(ref)
(have to worry about sign-bit: can become negative)

if pixel is not in the same range, store it as-is and mark it as "unsubtracted" with one of the extra bits

try using "gzip" and tell gzip that we're compressing 16-bit numbers

possible future feature? D(compressed)=D(shot)-scalefactor*D(ref) (scalefactor represents brightness of each shot)

could make it better-but-lossy by eliminating low-ADC bits?

Test Data

Looking at det.raw(evt) it appears inner panels for xcs1008621 run 340 are in fixed-low-gain mode and outer panels are in fixed-high-gain mode:

The first few events:

Test Script And Results

I believe the Shannon-entropy is the number of bits of information in each 16-bit word, so we only see a 10% improvement in the compressibility of the data with this approach.  Feels like there is quite a bit of entropy in the "noise" of the image even if they appear visually similar.

from psana import *
import numpy as np
import matplotlib.pyplot as plt
from skimage.measure import shannon_entropy
import gzip
ds = DataSource('exp=xcsl1008621:run=340')
det = Detector('epix10k2M')
ref=None
for nevt,evt in enumerate(ds.events()):
    raw = (det.raw(evt).astype(np.int32))&0xbfff # eliminate gain bit
    img = det.image(evt)
    img_med = np.median(img[612:680,663:726]) # a "normalization area" in the water-ring
    if ref is None:
        ref = raw
        ref_med = img_med
    residuals = (raw-(img_med/ref_med)*ref).astype(np.int16)
    f = open('junk%d.dat'%nevt,'w')
    residuals.tofile(f)
    f.close()
    #plt.hist(residuals.flatten(),bins=1000)
    #plt.show()
    print(shannon_entropy(residuals),shannon_entropy(det.raw(evt))
    if nevt==4: break

Output

(ana-4.0.55-py3) python junk.py
0.0 10.892801295795591
8.99940177591452 10.57374113419068
8.881920763769553 10.716387079966387
9.244652303773888 11.009271179948547
8.73624428989834 10.868081019635044
(ana-4.0.55-py3)