ESTB 2016/Sep-Oct T545 + T539

Participants: 

LBNL: Katie Dunne, Timon Heim, Peilian Liu, Ben Nachman, Maurice Garcia Sciveres
SLAC: Nicole Hartman, Peter Kim, Martin Kocian, Rob Mina, Francesco Rubbo, Su Dong
ANL: Sergei Chekanov
KEK: Koji Nakamura 

Shift Schedule (tentative) 

Event Log

Sep/19-26 prep      

Sep/19: Previous dissembled Caladium is reassembled by Test Facility in the ESTB tunnel but parked on the side as beam pipe was reinstalled for a down stream experiment (somewhat of a surprise for us) this week. Caladium mover control electronics is still missing and to be installed next week. We checked the re-connections and found a few errors and missing links which got corrected. Connected the double chip module 94-04-03 as DUT with upgraded DUT CAT6 data cables and tried pixel digital calibration which worked as expected. Reordered the power supplies on the remote power strip to match the original panel in EPICS.     

Sep/20: Checking with Carsten about what is really going on this week, it turned out T-542 (sFlash) is taking the entire period Sep/22-Sep/26 so that the originally agreed Sep/26 for us has evaporated and we were never informed of that change. 

Sep/23: Checked with sFlash at ESA control room. The intense scene confirmed the fear that they will probably need the entire period, including Monday.

Sep/26: sFlash is continuing with their runs so that we are unlikely to have any chance to run for Monday, considering that Caladium still needs to be brought onto beam line after beam pipe removal. So Monday is expected to mainly doing lab test of the first RTI module to get proper configurations prepared and check other preparation items.   

Sep/27

• 8AM MCC meeting mentioned ESA beam delivery problem for sFlash on Monday but eventually took very good beam data. sFlash cleanup.
• 10AM ESA became permitted access. 
• Caladium moved onto beamline after removing section of beampipe and mover electronics box reinstalled. Chiller level low, refilled. DUT stage also back on and hold current is nominally driven. Caladium powered up current anomaly traced to Clock-Jtag cables swapped. Missing TLU-NI USB cable located and reconnected. Caladium auto-trigger run and standard DUT (mounted 93-04-03 IBL double chip module back) both operating.        
• Noticed IBL double-chip DUT LV PSU setting sagged from 1.5V->1.2V when module current went up to 0.5A. Realized this was due to the current limit of 0.5A appropriate for single chip modules. Raised current limit to 0.8A stablized voltage through threshold tuning exercise.    
• FE65_P2 readout arrived from LBNL. Mike Dunning/Juan Cruz laid two DUT ethernet lines for the two readout PCs to connect to the ESA 104 private subnet. Mike granted fixed IP addresses: 172.27.104.52 esadutlbnl1 (riverhorse) and 172.27.104.53 esadutlbnl2 (inkfish), gateway is 172.27.104.1 and mask 255.255.252.0.   
• Various cameras setup - see main portal.

Sep/28

Plan for today:

• Priority is to debug FE65_P2 readout integration with telescope (Timon, Ben) and taking telescope whenever needed,
• Install the 2nd RTI SC module into the ESA DUT seat to do in situ threshold tuning (Sergei,SD).
• Threshold tuning for the first RTI SC module at group C lab (SD,Francesco,Rob,Nicole)
• HV remote control/mon with EPICS (Ben,SD with help from Mike)
• possible telescope geometry optimization (SD)
• Presentation [slides] of our plan and beam requirement to MCC (SD) at the 4:15pm accelerator weekly planning meeting next to the accelerator control room at the ground floor of B52. Everyone is welcome.      
• Discuss preparation status, run plan and shift assignment at ESA control room at 5pm. 

Activities:

• RTISC2B replaced the 93-04-03 as the RCE DUT to get threshold tuning going for it.
• When attempting to move the DUT closer to beamline we finally realized the oddity of the vertical position was due to the DUT stage mounted upside down. Juan corrected the mounting and now position close to be central.
• Sergei installed a script duplicate.sh in the moduleconfig directory on esadutdaq1 so that one clone an existing module's RCE DUT config directory into a new module's starting configuration with underlying sud-directory file name to follow the new module name.    

• The Research Yard construction people are unhappy about us rushing across the area in speed so that we are currently disallowed to go to ESA from the Research Yard ground floor through Gate 17/SSRL route. Everyone should go to ESA from Gate 30 to the top level counting room then down the stairs to ESA hall. If we have to drive in with equipment, we can give Keith Jobe a notice to arrange the passage via Research Yard. 

• Timon exercised first integration of YARR readout infrastructure with EUDAQ: YARR Producer correctly booted up in EUDAQ Run Control. Correctly receiving configure transition and acknowledge completion while actual configuration content and event loop handshake still in progress.     

• Re-orientated the RTI module by mounting inside the L to give some space for the HV LEMO bend not touching EUDET. The HV socket facing perpendicular to the PCB is not test beam friendly. TImon mounted the FE65_P2 nad looks like RTI and FE65_P2 are reasonably aligned in height and horizontal position to go together. Just some DUT stage fine shifts should bring the DUT to beam.

• Segei's threshold tuning for RTISC2B is giving high noise at 500e. Realized we didn't tape the sensor window. Proper blacktake hard to find. Kluged some thicker tape for now. Simple threshold scan didn't get sensible results. To be tried again tomorrow.   

Sep/29

Plan of the day:

• ESA search for returning to controlled access is either ~9am or ~11am. First beam by noon ? 
• Preparation the RTISC2B + FE65_P2 pair DUT suite for first beam: 
  • DUT stage fine adjustment to align DUT into beamline and horizontal move of telescope into beam. Visually locate FE65_P2 window wrt RTI. 
  • Adjust overall EUDET planes to pack closer and narrow the DUT region. Needs an LEMO-L for the RTI HV socket to avoid HV cable bowing out to EUDET. 
  • Proper black tape for RTI sensor window and recheck threshold tuning with taped window.
  • Further auto-trigger integration of YARR with EUDET.
  • EUDAQ config file for the DUT suite 
• Continued threshold tuning for RTISC4A in the lab to get ready to go in later, and cross check on noise effects seen on RTISC2B.   
• First beam activities
  • Time-in of FE65_P2 devices if DAQ event handshake fully functional. If no new device interpreter with EUDAQ online mon, offline dumps sufficient to verify ? Hopefully the known lab test delta of 3 clk ticks with FE-I4 will save us from a long chase. 
  • Beam window size, location and rate adjustment.  

Activities:

• 8am MCC meeting clarified that the ESA search is probably going to be 11am-noon, not the originally hoped 9:30am. We declared ready to run right after ESA search. It was brought up that a few weeks ago some ESA user from Japan brought in an irradiated sample without declaring to RP which caused quite some disturbance that even temporarily shutdown the ESA operations. Lia's question on this caused a major overhaul that required us to send updated participant list, and TestFac experiment review form and RP in person interview etc. before getting approval to start at ~2pm.
• 13:00 - ESA search has been completed and the cave is now set to controlled access.
• DUT stage fine adjustments brought DUT aligned with Caladium MIMOSAs, and whole telescope moved horizontally into the beam line. 
• RTISC2B appears have gone into very strange state and cannot be calibrated even after removing the rough taping. Brought in RTISC4A to replace it and sensor window properly taped. Threshold calibration runs but the noise rather high at ~400e, with some systematic structure also. Suspected bias voltage and seems to measure 0V on the HV ch-0 used. Switched over to ch-1 can at least drive the voltage up. Maurice concerned that the leakage current seemed a bit high >>few uA for very small starting bias voltage. However, cranking up to -53V (these Micron 300um sensors actually need -80V to deplete) didn't bring much more leakage current to settle at 4uA. Left at this setting to try to run. Sergei managed a standard 3000e tune with nice narrow dispersion. The noise is still high at ~400e. Also checked the various grounds and decided we can leave the HSIO2-PSU ground hook off since the HSIO-FE cable is now proper CAT6 with ground. This didn't make any difference. Decided to just take this tune as running config (RTISC2B__cern__25.cfg) - calib directory stayed with RTISC2B but this is really RTISC4A.    
• Tried EUDET ni_autotrig runs but somewhat confused by the behavior: without starting up the TLU producer, the autotrig setup actually runs with event build flowing and histograms in monitoring updating, although trigger/particle counts remain 0. When added the TLU producer, particle counts still go up but no trigger and no event build.  
• 18:05: Carsten called and informed us about the delay. The people before us wrapped Al foil around the emergency buttons which of course upset a lot of people. They stopped ESTA operation until it is resolved, which should be in the next hour.
• Beam finally arrived ! We don't seem to get any trigger. Decided to ask for access to look at the trigger and change RTI sensor HV control at the same time. One problem with trigger quickly found: the Caladium Date Reducer box data link to TLU was misplaced on  the Data Reducer box side to the port right next to the 6 MIMOSA data cables, but it really should be plugged into the channel at the far end. Fixing this resolved the TLU producer anomaly above.   
• The remote control to the HV in the NIM crate (the orange one in the middle) now works via the KB 9130 power supply.  There is a panel in the epics for T-542 for controlling the low voltage.  The granularity is pretty good - by setting 0.08 V (=80 V in HV), we get 0.086 measured (and displayed on the monitor).  The only problem is that the current is in A so we can't monitor it (need to change to uA).  Should ask Mike tomorrow during the day for help with the quick fix.
• After we moved HV control and set voltage -53 > -86V volt, Sergei rechecked the RTI sensor threshold scan noise and found noise reduced from ~400e to ~300e. This is consistent with the Micron sensor depletion voltage of 70 - 80V. 
• Upon restart with beam, we still don't get the beam triggers, although auto trigger runs with TLU now works correctly.  
• 21:19: Accelerator shift change, beam energy changed and was off for 20min, back now. Will take access to investigate trigger problem.
• ESA access again to examine the beam trigger. Traced the trigger cable from the DUT rack to the patch panel area and placed a yellow tag on it. It is connected to the row of BNC plugs saying "To B061-01-06" socket J5. Brought scope to the DUT rack and placed a BNC T on the trigger cable to check it on the scope by moving over the rack top camera.   Upon restart with beam, we saw from the camera that there are blips of pulses at ~5 Hz as expected and negative polarity as expected from NIM but the pulse height is much lower than the -1V NIM. We though the trigger signal might be available at the counting house but the "beam spill" plug had nothing on it. Toggling the trigger panel channel #10 SiTrack- Caladium Enable on/off one can see the 5 Hz blip goes on/off together with it which suggests the trigger is still hooked up to the right channel but too low an amplitude for the TLU box.  Couldn't make much progress without Mike. 
• Decided to quit for the night and reconvene tomorrow when Mike Dunning is around. 
• New calibration is RTISC2B/config/RTISC2B__cern__93.cfg for 89V shift volts.  The main features of this calibration: Threshold distribution has mean 2988e, with sigma 68e. The noise is 288 e.

Sep 30

• Some description and tutorials how to take the calibration scans is https://atlaswww.hep.anl.gov/asc/wikidoc/doku.php?id=itk:testbeam. All figures correspond to the previous scan on Sep 29 ( RTISC2B/config/RTISC2B__cern__93.cfg)
• 5.00AM: New ANALOG and THRESHOLD scans. It was found that, without calibration, we see large number of low-efficient analog channels. About 70% of channels have 40 counts (50 expected). This drastic difference compared to Sep 29, right after installation of this module, when about 1% of channels showed 40 counts (mostly for first 2 rows). The noise level from the threshold scan increased by a factor 2. Low voltage increased to 2.32A (from 2.08A)
• 5.15AM: New calibration for the RTI module is RTISC2B/config/RTISC2B__cern__158.cfg.  After the calibration,  it was noticed a large number of bad modules, large noise 400e, and the  analog scan has more loses on the left edge.
• Preliminary conclusion: Similar to the red module installed on Sep 28, the green module shown signs of the degraded performance for the analog part: lower efficiency and a larger noise after 6h inside the test beam setup. This cannot be related to the beam, since there was no beam for the red module on Sep 28. This also not related to the light, since the sensor of the green module was covered.
• New calibration RTISC2B/config/RTISC2B__cern__225.cfg
• 08:36: Doug and Juan sent by Carsten are helping us with the Trigger. We found where it is patched from crate B060-14 signal J10 (as it is #10 in the tirgger panel). The issues of the small signal seems to be caused by a loose cable.
• 8:58: beam on and we get triggers. Now looking into the actual data. Initial beam ~2 particles per pulse. 
• 09:02: beam off for energy change, roughly 20 min
• 09:29: beam back with higher intensity (~10 per pulse) using it to align the telescope. Needed to move whole telescope by a few mm in X and Y to center MIMOSAs to beam. Initial beam size (dX,dY) ~ (6mm,4mm). Asked Tonee to enlarge it but it appeared to be not simple. First attempt enlarged vertical size only at high X side and made it trapezoidal. Decided to pursue basic DUT test first and come back to this later. 
• 80lb dry ice for the 2nd period is delivered to ESA. 
• FE65_P2 took first data with Run 497 (not yet timed-in hits) by enlarging DUTMASK to 5 to include the first device, but without DUT device details so that online mon cannot yet process data. 
• RTI with EUDET run used the configuration SLAC_1dut_FEI4B_RTISC4A.conf. Synched beam hits appeared right way but sparser than expected. Tried again later  got higher rate. 
• 2:00: Brought the cooler for the dry ice from lab C and brought it into ESTB.  Matt is sick at the moment, so see Marco for help with lab things (for example, the blue gloves can go back to the table in the lab when we are done with them and he can put them back where they belong).  During the access, we setup the monitoring of the voltage and current from the HV power supply that is now remotely controlled.  The voltage seems roughly correct, but the current is too small to measure at the moment (below the pedestal).  These values are called Caladium HV-Imon or HV-Vmon in KL3404 in the Caladium Beckhoff epics panel. Imon seems to have a pedestal of 5mV while AVO measured 1.5mV.  
• Finally managed to fully integrate YARR into EUDAQ with the converter so we can see the DUT in the online monitoring.  Now, working on timing in the FE65P2.
• 5:00: Carsten stopped by to check on us.  There are only two operators this weekend, so we should be patient with them.  Some things we learned from this conversation:
  • If we need to show monitoring plots to MCC, might be easiest to just walk over there.
  • We can see the accelerator optics in the BSY (=Beam switchyard) on the ESTB epics panel.
    • SL10 (slit 10) is at 2mm and if closed, we can reduce the rate.
    • C24 (collimator) this can make the spot smaller (cuts into the edges)

• Requested to MCC to enlarge beam spot and up the particle rate to get ready for FE65_P2 time-in. Made only little impression and ended up with a fan shaped spot. First did RTI+EUDET runs to try to center beam spot on the RTI sensor using the DUT stage fine adjustments. Looks like DUT Y is really Y, but DUT X was -x. The runs typically got stuck after 1500 - 2500 events due to RCE DUT readout hang and run would end there. The list of runs with EUDET+RTI:

  Run	Comment
  534	Start position
  535	fineX +5mm, fineY -2mm, then in the middle if the run fineY +2mm
  536	fineX -2mm, fineY +7mm
  537	fineY +2mm (bi-modal Y distribution. Sharper peak is the real spot)
  538	fineY +2mm

• While having shift change, dropped the IBL irradiated sensors LUB1,LUB2 to the freezer inside ESA.
• 21:50: ACR called and said that apower supply of the A-band tripped, they are trying to send us as much as possible, but we basically only see noise. They called someone in who should arive in around ~20min.
• 23:05: Tech is on site, ACR is guessing 1h to fix if fixable.
• 23:30: Fixed! Beam back! ACR played around and cranked up the rate, we are now getting around 200 per pulse.
• 00:22: Have tested every possible latency setting assuming the trigger arrives after the particle without success. Now seeing if it might be possible that the trigger comes from the future.
• 00:56: Finally seeing particles in FE65-P2!! Confirmed trigger is from the future, to be precise particles come ~245 bc after trigger. Now taking some data.
• Putting trigger signal on Scope ESA:SC03 channel 4, orange seems to be particles as they fit the timing described above
• Run 596 probably first Okish run with FE65p2, TLU tag might be off.
• Getting 800e per pulse, rate in FE65p2 seems low, moving 2mm up to 23mm for 597
• 2mm more so 25mm for 599
• Run 600 at 30mm
• 603 22mm
• Now module is constantly noisy ....
• Sometimes noisy, sometimes not, no clue if i'm moving it in the right direction
• Asking ACR to reduce number of electrons, slowly working our way to ~10
• Run 615 adjusted to ~10e per pulse, letting it run for a while.
• At this point we are flying blind again, I would need to know if this data can actually be decoded and if we are in the beam.
• Starting 616, to take a closer look at 615.
• We are calling it a night, I need to reconstruct this data to understand what is going on. More importantly I have serious concerns that the data format might not work with reconstruction and I need to know that in time. Will go to sleep and take a look with Ben tomorrow. The module will stick around and YARR data taking (except the noise issue) is very stable, so should be able to run in parallel to the irrad studies.
• 02:26 : stopping and letting ACR know.

Oct/1

• 8am MCC meeting met incoming LCLS  primary user with ePix (500um silicon for X-rays) who are also battling with radiation damage and showed interests in what we are doing.
• Did a threshold scan (Run 178) on the RTI module and found threshold still narrowed distributed around 3000e but noise got substantially worse at ~500e, 
• Turned off RTI bias from the remote control console, but the LV read back still says 6mV, Caladium ADC Vmon=15mV, Imon=5mV  which are effectively pedestals. Noise up to 1100e when unbiased.
• Reestablished RTI bias for -80V with BK setV=0.080, readback 0.085 and ADC Vmon 0.088-0.093. Threshold scan Run 179: threshold still narrowly groups around 3000e, noise still ~500e, but he right edge bad pixels cleaned up. Bizarre looking noise RMS vs channel map with systematic alternating column pattern and multiple clusters of values within even the same column from 300e – 900e depending on pixel location.      
• 11:00: Used an access to remove Koji's FE65P2 module for shipping.  Now will run YARR parasitically for the remainder of the beam time.  Next move is to do a scan over the RTI module before switching to the cold box.
• Started the beam spot systematic scan over the RTI module by moving the DUT stage. Beam was ~20 particles per crossing. DUT LVL1 peak ta at 4 - moved the original value ~8. Run list see table below.  
• 3 FE65_P2 surveyed by RP and shipped to Koji.

• The almost predictable hang of RCE DUT readout was consistently happening after every 1500-3000 events, that needed HSIO reboot to recover. Later run with old irradiated IBL LUB2 at even much higher particle rate of ~150 were running OK for 10K events. The RTI module data had corruption ? 
• We also saw sudden big burst of noise hits causing temporary peak in LVL1 time distribution which gradually averaged out.
• Call for access to switch to cold box setup. Made new cut out cable slot on cooling box (but under a screw) to allow easy cable exit without running into the MIMOSA plane. Mounted LUB2 as vertically and perpendicular to beam as DUT. Remeasured component vertical position inside box and marked the module center on box. Tried to lower the box into beam but fine Y mover maxed out at the bottom. Lowered the sliding bracket of DUT stage done by ~15mm to keep it in range.
• We dismantled the TLU trigger scope monitoring and put the original cable back to TLU.
• One remaining FE65_P2 is mounted on the back plate of the last EUDET plane down stream. DAQ computer also moved to the back end. LV=5V. Xlinx dongle plugged into the left of the two USB slots of the computer. The YARR readout card connection has a bit of snag as the last mm of the conenction could not go in one one side as it hits the mounting hole shroud so that the HDMI connection is slightly tilted and not all the way in on one side,   
• We notice at some point during the access that the horizontal position of the whole telescope was several cm towards to +x wall according the ceiling alignment laser, but we don't understand how it could have been there.  
• Calibration failed initially when LV was set to 1.5V. It was then realized the for these irradiated old FE-I4A the correct LV is 1.2V. HV is now on remote control.
• 8:25pm restarted beam. Indeed confirmed the whole telescope was way off in X and had to move a long way back to beam. DUT also needed to move up by ~1cm to center to beam.
• HV of -800V was giving LUB2 decent hit efficiency already. The last run with collimator opened to 150 particles/crossing was recorded as Run 648 with config SLAC_1dut_rad_FEI4A. L1V1 timing peak between 5-6. The beam spot shape flip between EUDET and DUT clearly says the DUT x is EUDET -x.            
• 9pm: beam stopped to switch to low E primary. DUT HV off. Our shift ends here.
• Cold box block is <-50C.

Oct/2

• Temperature still -40 (air), -50 (block)!  Here is a picture of the strip at 9:00 am.

• 09:30 FE65-P 2: ran tune_globalthreshold and tune_pixel threshold and got okay results from an analogscan:
• 09:42 We have beam