Event log

Jul/06

• Caladium back to beam line, but chillier not yet on and g-2 platform post is blocking beam line to the dump.
• Cleaned up the remnants of g-2 from DUT rack. Mounted DUT rotation stage. Reconfigured the web cams and updated the confluence portal list.
• Queuing one BK Precision 9130 supply for survey to exit ESA for let Mike Dunning working on its RS232 interface for EPICS.  

Jul/07  

Operations team: Maurice Garcia-Sciveres, Katie Dunn, Su Dong

Maurice's CMOS sensor shipped to SLAC from Sensor Creator's Inc directly. A 'camera' like standalone readout unit connecting the sensor via a thin flex. Beamline support scheme worked out with Matt McCulloch with CMOS sensor taped behind the 94-04-03 double chip FE-I4 module. Vertical readout chain to allow easy rotation. Ethernet port address 192.168.2.1.

• Doug McCormick turned on chiller, moved aside g-2 stage and reordered web cams. 
• Alignment encountered several issues: coarse Y appeared to be not stopping and having strange pauses during moves. Coarse X is at hard limit and we are still ~2cm off. Fine XY panel scale factor didn't look right. Rotation goes scarily fast.  
• Caladium commissioning got stuck on NI crate not ping-able. Traced the network route for NI crate and DUT Sun server to table below. Early stage found NI crate power dropped - tracked to loose power cable at the back, but firm reconnect didn't get it back still. tried to swap NI crate's Belden channel to 22 but made no difference.

• Prep-prod HSIO-2 X1 moved into ESTB tunnel DUT rack. Network table needed update of HSIO-2 MAC address. 
• CMOS cam ethernet readout needs to find a host server. Sun server esadutdaq1 both ports occupied. 
   

Jul/08

Operations team: Maurice Garcia-Sciveres, Katie Dunn, Su Dong

• Doug McCormick and Juan Cruz moved the whole Caladium to the south (-x) by 25mm to center the telescope to beam and still in the middle X coarse range. Also adjusted the yaw.   
• Following recommendation from Matthias Wittgen and Martin Kocian to add a NetGear G105 router between Sun server and readout devices such as HSIO2 and the CMOS cam so that the single DUT readout ETH1 port of the Sun server can take in multiple DUT readout sources in a private network. To avoid the complication of changing the CMOS cam IP config, the RCE and Sunserver private network IPs are modified to create a common private network:

• Mike Dunning and Juan Cruz helped to debug the NI crate unreachable issue. Belden channel moved to 22 and again verified made no difference. Hooked up a monitor to NI crate discovered it got stuck in an unbooted state, presumably due to uncontrolled power downs. Followed its own long self-recovery procedure and eventually booted up normally and final became ping-able.    
• Matthias Wittgen offered a recipe to open a ssh tunnel from a counting room node to the CMOS cam via esadutdaq1 
  1. Window 1 on ar-esaux2
     ar-esaux2 > ssh -L8080:192.168.2.1:80 tbslac@esadutdaq1 
  2. Window 2 on ar-esaux2 open a web browser (firefox didn’t work for me, but konqueror did)
     ar-esaux2 > conqueror
     Navigate to  http://localhost:8080
  and installed ftp to esadutdaq1 to allow copying of data.
• Mike and Juan further debugged the XY stage scaling parameters to correct them to the right values. Movement steep on some on them also slowed down (especially the rotation). Mike showed how the advanced option panel can help control the movement rata and soft limits. However the problem of current position display lagging until next move is still there. Needs to use the advanced option panel to to manual refresh to check current position. 
• Upon retry of EUDAQ integration, RCE producer always aborts at start of run. Running calibGui reveals connection error and a suspicious of DUT private network IP address for the Sun server. Debugging with Martin Kocian uncovered that the environment variable ORBHOST for the host node esadutdaq1 used by RCE DAQ also needs update to follow the DUT private network change above. The default definition in ~/daq/rce/scripts/setup-gen3.sh is updated to point to 192.168.2.3. This brought whole EUDAQ to run with the SLAC_1dut_FE14B configuration (1 module both FEs). 
• Ready for beam !     
• Upon requesting to start beam MCC found out that the primary LCLS experiment at CXI was initially unwilling to share out the 5 Hz beam, partly because their overall beam delivery for the last day was rather rough. 1Hz setup appeared to be not available. However, CXI eventually agreed to release the lowest possible rate at ~3.5Hz so we had the first run. Whenever we are not running we should inform MCC to take away the beam to give back the 5%. 
• Run 269 is the first run with the LBNL CMOS taped at the back of the regular double-chip module 94-04-03, started out at perpendicular to beam. CMOS is untriggered. MCC didn't have dump signal to guide setup quality, but we could tell it was a rather wide spread spot and about 10 particles per MIMOSA per shot. For the last 1/4 of the run, the module is rotated by~45 degrees but not sure if still aligned within beam.
• The CMOS camera timing window can only open up to 1ms so that it was only 1-2% duty cycle. Got an 5V->3V BNC+resistor voltage divider from Mark Freytag to try to run triggered mode. Also got a 1M Ohm resistor to series a bias voltage LV drive. Installed both on the CMOS camera.
• Upon requesting to restart beam, MCC complained that some vacuum valve for ESA section didn't look right and wondered how we could operate like that. Indeed the beam spot looked very wide and nearly uniform over the whole MIMOSA area.However we were getting ~10 particles per beam crossing on each MIMOSA. This is actually OK for us so asked MCC not to bother tuning it.   
• NI producer repeatedly got stuck and no events were built while beam triggers were piping through at 5Hz. Eventually went back to start from the very beginning of NI setup to get everything going again.
• Run 277 with DUT at ~45 degrees and CMOS camera in triggered mode. Beam appeared to be hitting left X edge of the FE-I4 module. Moved the module to -X in 5mm steps several times during the run to confirm hits gradually moving to the center of DUT. Still not seeing hits on CMOS camera. Not clear if the TLU TRIG output was really getting into the camera trigger line.         
• Saved monitoring histograms to ~tfuser/eudaq/bin and archived at ~tfuser/Desktop/2016July_onlinemon/.
• Decided to retrench on the CMOS camera for now. Dismounted and packed up for survey exit ESA. Several things to prepare for retry next week:
  1. Activating another beam trigger line to allow independent trigger time delays while keeping pairing FE_I4 and Caladium data in time, or still using the same primary trigger line to make a duplicated copy at DUT rack and find a way to do 25ns step delays one of the derived triggers. The primary trigger must be before the actual beam crossing.   
  2. Verify triggering pulse electrical spec at LBNL bench tests
  3. Pulser and scope is ESA to examine trigger signals  
  4. Keithley to measure microA level current on bias supply ?

Jul/13

Operations team: Ben Nachman, Su Dong

• Lab cold tests obtained good tuning for both LUB1 and LUB2, even 1500e for LUB2, at HV=600V.
• Presented plan for 2nd period to MCC at 4:15pm meeting: ESTB_MCC_7_13_2016_SD.pdf
• MCC meeting had brief discussion of possible effect of ESA stealing 5Hz beam. Some effects of missing pulse causing irregular effects for pulse after that a Low E beam experiment, but high E beam experiments never cared. However, still needs to verify this for MFX/LM51 tomorrow, before 9am so that both dat and night shift beam to ESA would be possible hopefully.
• Lab testing pushed up HV. For LUB1, 1KV immediately trips the 1mA max current, while 900V still hanging there but current still slowly creeping up. For LUB2, 1KV started out at 700uA (-29C) but current slowly creeping up and after several hours it eventually tripped and temperature went up to -26C. 

Jul/14

Operations team: Ben Nachman, Su Dong

• Moved lab setup of cold box with LUB2 to ESA.
• The dry CO2 chamber is reversed from May to be on the corridor side while electrical connections from rack side.
• LV VDD=1.5V. I=0.33A after config.
• One temperature probe (Ts) taped to the Cu cooling tape tail from the DUT, the other temperature probe (Tg) in cold gas. 
• Leakage current high initially but temperature was just -10C. 2nd access raised HV to 800V leakage current 830uA  at Ts= -23C.
• After ensuring only ch=0 for RCE activated in readout, calibration ran, but threshold distribution is broad. Returned and got a quite decent RMS now. Saved to LUB2/config_rad_np_0_335: 1500e threshold, 8000e=10 for TOT.
• Made EUDAQ config SLAC_1dut_rad_FEI4A.conf
• Alignment was still tricky. Fine-Y adjustments not holding up the position. Any adjustment would trigger a long sag. Mike/Keith both thought the motor holding current may be too low. Will be looked at tomorrow before 9am by TestFac.
• Because DUT X/Y transposed, we cannot see correlations any more between MIMOSA and DUT as correlation only plots X vs 'X'.
• Horizontally we would like to move Fine-X towards +X (rack wall) by ~1cm to align with EUDET but that reached limit and cold box cable exit is touching Caladium plane frame.
• Requested access ~4:45pm. Put a ~2cm block spacer in the cold box to move DUT towards -x to recenter better in X wrt Caladium and space out the cable exit from Caladium. Also jacked up the DUT box by hand to be close to center in Y. Topped off dry ice.
• MCC notified us that LCLS MFX requested change of beam energy up to 14 GeV (but secondary always 11 GeV) which will affect our rate for a while. After reconfigure the beam spot shape changed completely with more definite edge and more uniform. Previous beam spot had a sharp hot spot of on the top right corner and more diffused spread below that.    
• 6:45pm first serious data run 294. HV=800V, leakage current 200uA. Ts=-34C, Tg=-29C. ~350 hits/MIMOSA, 150 hits/evt in APIX. DUT beam window: Local X=40-75 which is 35 pixel ~8.5mm; Local Y=30-330 which is 300 pixels ~15mm. MIMOSA beam window X=200-1000 also ~15mm; Y=20-460, also 8mm.  
• Noticed some 'scars' on the DUT which are straps of area with clearly lower hit density than neighboring areas.
• Ended a good run 294 to request lower rate. MCC had no clear recipe to reduce rate other than collimating to smaller beam size which doesn't really help us as it is the hit density that matters. They are trying to reach Toni Smith for advise. Ryan tried to tune this from upstream adjustments and succeeded to bring the rate down to 30 hits/shot at DUT with relatively small change in beam spot shape. We needed to make a few mm adjustments to keep both MIMOSA and DUT beam window fully contained.
• 8:34pm 2nd run 299 with DUT hit rate ~30/frame. Temperature climbing back up Ts=-32C, Tg=-27C. Leakage current 250uA. MIMOSA beam window X=150-700 (10mm) and Y=full span 0-570 and perhaps a bit more (11mm). DUT window 'X'=30-74 (11mm), 'Y'=40-260 (11mm) a bit trapezoidal.   
• 9:06pm started a new run 300 when last run reached 10K event limit. Exactly same condition. Ended 21:42.   
        
  

• Requested access to change DUT tilt. Had to pull out the block to tighten the rotation screw and retape the DUT Ts temperature probe. Cold box stage sagged again which needed manual repositioning. Topped off dry ice. 
• Upon restarting to find the location of the DUT from remote motors, the whole DUT stage sagged to Ymin again. Only saw a lot of out of time noise hits at local X min and max edges (this was probably due to temperature not yet low enough).
• Turned off beam to do a low TOT tune of 1500e threshold and 400e per TOT bit. Calib tuning kept crashing. 600e still crashing, while 800e existing tuning still worked. 
• Gaming with Y-fine stage to hold up the cold box at ~5mm below nominal (any further up push will cause full scale sag down). Cold box lid is opening a significant gap.
• Now a bit colder, returned to previous configuration and brought beam back. Noise lower. Moved DUT fine-X by -6mm, starting to see streaks. Data populating lower 'X' half of the DUT but did not dare to jack up the cold box.
• 11:39pm started run 303 for the first streak data. They looked a bit shorter than expected typically. DUT data rate similar to MIMOSA at ~80 pixel/frame, but there are many streak clusters. Pixel TOT peak at ~5 (5*800e=4000e). This is a bit larger than expected from Bichsel calculation that predicted peak at ~3300e for 50micron silicon even without radiation damage ( foldspec_elec_11GeV_50mu.png ). 
• 00:20am ended run. Let MCC took away beam. Remote power down of HV+LV.

Jul/15

Operations team: Ben Nachman, Rebecca Carney, Su Dong

• Temperatures at 8:30 am: Ts = -27 and Tg = -22 (the clear drop just before the linear rise is where we turned off the DUT power).
• Primary beam energy up a bit from last night (12.7 GeV) - needed to work with MCC to adjust beam params (was blasting at ~500 particles to begin with).  Amazingly, the DUT is still aligned (didn't move it up or down when filling the ice).
• 10:00 - Made entry to turn on LV. Checked HV whilst in cavern (first check since leaving last night), and had drifted to 870V. Was last set to 800V so we set it back to that value.
• Starting with Run 307 at 10:52 that has the same setup as Run 303 from last night (streak data).  Originally started with Run 305 but EUDAQ seg-faulted so we had to restart it.
• Run 308 starting at 11:31 is the same as 303.  Note that during this run, the rate drifted up again so there was a small intervention with the upstream collimators.
• The Caladium DUT Y-stage sagging issue was confirmed to be due to motor current responsible for holding position was not set so that weight can cause position drift. Some current setting enabled in the morning verified that one could hole Y-fine adjustment now but there is concern of motor overheating etc. so that Doug and Mike did further checking and tuning during noon access to settle it.  
• ~12:30 - Went in to increase the HV and caught Doug and Mike. We now have automatic monitoring of the HV current (KL 3404 ADC Chan01) and voltage (KL3132 ADC Chan 01, but raw->voltage conversion factor in EPICS look wrong: 1.01V display 0.75V ). We can't yet control the HV because the cable we have between the HV and steering LV has a male end when it actually needs to be female.
• ~14:00 - Currently the conversion factor seems to off by 0.746 consistently in the  ( we measured the ADC output and the value at the src for 600V, 800V, 900V, 1kV). On the next entry we will try switching the current and the voltage monitoring ADC channels to confirm it is just the conversion factor. The HV current monitoring seems representative of that measured near the source.
• ~15:30 - 1kV run,, with the beam set roughly between 'columns 30-60' ended after taking ~10,000 events. Then in run 311-314 adjusted the box position, moving it down by 6mm to take 1kV data over 'columns 10-40'. Run 315: taking data at 1kV for 10k events.
• ~16:30 - Tried switching the ADC channels to determine the other conversion factor: ADC channel 3132-4 is actually displaying 0.75 of the real value. ADC channel 3404 is close to the correct value: displaying 0.9*real value.
• We reentered the chamber at ~16:00 and refilled the dry ice as it was getting low. Tried increasing the HV bias to 1.1kV as suspected that the lack of a cluster size = 15 peak was due to the sensor being underdepleted, but the current quickly ran away. After adding the new dry ice: current seems stable so started new run.

• Doug and Craig gave us some commands to set the voltage (once we have the correct connector between the voltage supplies ofc). We must telnet into the machine and run edm from the ESTB gui. A few instructions (so far), not that none are case sensitive:
• Login to tfuser@ar-esaux2:~
• To login to voltage remote control: telnet ts-esa-01 2005
• To check if login successful, query machine ID: *idn?
• To query current voltage setting: volt?
• To set voltage: volt $VALUE
• To exit telnet: 'ctrl-'] , as in the control button and the right square brace button
• Then you'll be in some login window, to fully exit type: q
• More later.

• Start run 310 with 1.0KV HV (leakage current 440uA) and DUT pixel hit rate ~60. Beam window on the right hand half ('X'=30-60).
• Requested access at 6:30 to install the second ice chamber.  Now the temp is nearly -45 or so!  Changed the HV to 1200 V.

• Retroactive recap by SD: Y fine stage adjustment not functioning any more. Readout errors couldn't be cleared. Everything powered off to save dry ice overnight. Temperature reached  ~ -60C at some point (but temperature time history lost). Unfortunately, not enough notes were left in this Event log for the various problems at the end of the shift so that the next shift ended up spending many hours rediscover the same problems.          

Jul/16

Operations team: Veronica Wallengen, Su Dong

• Cold box temperature still at -45C after long overnight warm up.
• Started with swapping in LUB1, placed at the 73 degree tilt. Also noticed Caladium is placed rather high which appeared to be due to the stuck DUT stage at slightly low fine Y so that Caladium is used to lift the DUT up into the beamline. We dropped two piece of metal slabs of ~1cm thick to lift the DUT up within the box. This bear made it to a partial overlap of top of DUT and bottom of MIMOSA.     
• Readout errors. Replaced data ethernet cable from Lupe's to the black ESA commercial cable. No difference. Touched the LV cable inside box to reduce sharp bend with better loop. This recovered readout (or the LV power up sequence made the difference? Martin's theory was that one should hold off plugging in the actual LV to frontend after the PSU setting settled).    
• 19:06 started run 333 with LUB1 @ 1.1KV HV (leakage current 170uA) and DUT pixel rate ~70, but MIMOSA is only capturing the top half of the beam spot to allow the beam just catching the bottom part of the DUT. DUT beam window 'X'=5-30; 'Y' almost full span. MIMOSA beam window X=100-700; Y=0-160 (clipped at Y=0). 
• 19:44 another run 335 with the same conditions.
• Attempted another 1500e tune for LUB1 following Martin's suggestion to skip the Noise Scan at step 13 but the process stumbled at step 16 for Noise Scan again, exactly the same way. So have to continue with the 2000e config.
• Raise HV to 1.2KV and every still looked benign with leakage current ~180uA.
• Just as we started to config, some major glitch took place ~9:40pm and entire Caladium stage control/ADC/temperature etc in an error state that never self-recovered. All temperature, HV readings became unavailable and Caladium stage also became uncontrollable.
• We did not realize the seriousness of the Caladium control loss and went ahead lifted out the DUT and turned it to 90 degrees to beam after quite a bit if trouble.
• Reconnecting DUT also had trouble with readout error again. Upon playing with the LV power on sequence, it was noticed that the BK precision LV PSU went into a spontaneous chaos mode with displays flickering and numbers fluctuating and even partial screen blackout. This may well be the source the various readout flakiness.
• After waiting for 1-2 hours, the HV current went down enough to allow HV setting to go up to ~700V but current ~500 uA still fluctuating a lot and couldn't settle. Don't know what temperature this was. Dry ice is rather low and doesn't look like temperature can help this much from here on.
• Too many things telling us it's time to quit: Flaky LV PSU and readout errors, unsettling HV, broken Caladium drive and display, and we are out of dry ice. Dismounted the cold box from Caladium and placed the LUB1 in the ESA freezer together with LUB2.
• Concluded the day with just two good tilted runs with LUB1 @ 1.1KV.              
  
  
• The temperature time history below for July/16 showing initial blue=probe tied to DUT CU tape, green-cold box gas. The step at t=-5 hours is the big glitch that took out the Caladium drive/display.
  
• The newly installed Caladium ADC for the HV Vset and Imon: The green leakage current mon is roughly calibrated to the HV module's LEMO mon socket claimed of 1V=1mA. The blue Vset calibration was a bit strange which showed -0.75V for -1KV. Glitch at -5h made Caladium stage control into permanent error state, all values after that were invalid.