Our first large area detector (2Mpx, 5kHz). Total data volume 2Mpx*5kHz*2bytes/px=20GB/s. 5 tiles in each quad. Each tile is 4 asics. Each ASICs is 144*192*4 (4 is number of asics).
Schedule
(on Dec. 19, 2022) Sometime in February/March a prototype with 3 tiles will be released (could generate fake data for missing panels for 1 quad). Full detector: summer?
Calibration Time
Running this script on a psffb node (twice so we get all the data in cache so it behaves more like the daq):
(ana-4.0.48-py3) drp-srcf-eb004:~$ more junk.py
import time
from psana import *
ds = DataSource('exp=mfxx1005021:run=340')
det = Detector('epix10k2M')
for nevt,evt in enumerate(ds.events()):
img = det.image(evt)
#print(img.shape)
if nevt>=20: break
if nevt==0: tstart=time.time() # start the timer after first event (which is slow)
print('time per evt:',(time.time()-tstart)/nevt,'nevt:',nevt)
(ana-4.0.48-py3) drp-srcf-eb004:~$ python junk.py
time per evt: 0.17038806676864623 nevt: 20
(ana-4.0.48-py3) drp-srcf-eb004:~$ python junk.py
time per evt: 0.17048853635787964 nevt: 20
(ana-4.0.48-py3) drp-srcf-eb004:~$
So one core can do about 5Hz. That implies for 5kHz we need 16 nodes * 60 cores/node.
This code implies that we may be able to reduce this time significantly with a "single pass" calibration:
(ps-4.5.24) drp-neh-cmp015:lcls2$ more junk.cc
#include <stdint.h>
#include <stdio.h>
#define RAW_SIZE 2000000
#define NIMG 100
uint16_t raw[NIMG][RAW_SIZE];
float result[RAW_SIZE];
uint16_t peds[RAW_SIZE];
float gains[7][RAW_SIZE];
int main() {
for (unsigned count=0; count<NIMG; count++) {
for (unsigned i=0; i<RAW_SIZE; i++) {
unsigned val = raw[count][i];
unsigned range = val&0x7000;
result[i] = (val-peds[i])*gains[range][i];
}
}
}
With these results:
(ps-4.5.24) drp-neh-cmp015:lcls2$ g++ -o junk junk.cc (ps-4.5.24) drp-neh-cmp015:lcls2$ time ./junk real 0m0.744s user 0m0.739s sys 0m0.005s (ps-4.5.24) drp-neh-cmp015:lcls2$
Fiber Connections
A slack conversation with Dionisio and Matt:
Hi Dionisio, I believe TXI is getting an epixhr 2Mpx and we have to purchase daq equipment for it now.. Can you tell me how many fibers that detector will have? Thanks!
That detector will use the 12 lane transceiver
and there will be 4 fiber bundles per 2M camera
one fiber per quad similar to the analog 10k camera
I assume you mean “one fiber bundle per quad” above? And 12 fibers per bundle, so a total of 48?
exactly
Sorry, one more question: each fiber will be 6Gbps or something higher?
48 fiber pairs, I assume.
all lanes are bi-dir
so yes, 48 pairs
6gbps?
these are the fibers I am planning to use here for testing
We need single-mode.
Those look like multi-mode.
these are the fibers that can connect to the transceiver. MM to SM conversion will be needed for this camera and we need to plan of which option we will use for that (same as it was done for ePix HR? or something else)
Perhaps we discussed before and I forgot. Doing the MM to SM conversion for 96 fibers feels unsustainable.
We discussed before. The transceiver on the detector end only exists for mm now.
Do I understand correctly we would need 48/8=6 kcu cards to to convert all of them? That will require a larger node.
I’m also worried because I had significant problems with lane-locking with the converter-card for the epix100. I had to mix-and-match lanes to find one that locked. Optical powers looked good. If we really need to do this conversion we should probably use that converter card to understand the lane locking problems. Maybe I needed to clean all the fibers somehow, even though the optical powers looked good?
Fiber Lane Assignments
(Tentative)
Lane[0].VC[0] = Data[0] Lane[1].VC[0] = Data[1] Lane[2].VC[0] = Data[2] Lane[3].VC[0] = Data[3] Lane[4].VC[0] = Data[4] Lane[5].VC[0] = SRPv3 Lane[5].VC[1] = XVC Lane[6].VC[3:0] = slow monitoring[3:0] Lane[7].VC[3:0] = o-scope[3:0] Lane[10:8] = Reserved for edgeML Lane[11] = LCLS-II Timing
Can split the fibers onto different nodes.
Data is not currently in a natural order in the fiber, but plan on doing this in the tile electronics.
Overview
Content Tools