...

Schedule

Data taking run shift list

Data Run Summary List

Event Log

May/7  

Test period formally allocated for May/31-Jun/4, which is the very last period before LCLS beam summer shutdown (until Sep/2018) to give as much time as possible to prepare the setup for the RD53A module. 3 other experiments are scheduled before us and some of them are rather invasive so that the Caladium telescope will be removed from the tunnel which we have to reinstall just before our period.       

...

Jun/3

  6:00 Event data taking came to a halt during Run 105:

Run stopped early - noticed histograms stopped filling

ERROR IN DUT DATA PRODUCER

vent: tag(0) l1id(14 bcid(11398) hits (0)
Event: tag(0) l1id(15 bcid(11399) hits (0)
####################
### Stopping Run ###
####################
Got 1791events!
Caught interrupt, stopping data taking!
Abort will leave buffers full of data!
Caught interrupt, stopping data taking!
Abort will leave buffers full of data!
Waiting for run thread ...
-> Processor done, waiting for histogrammer ...
virtual void Fei4Histogrammer::process(): processorDone!
-> Processor done, waiting for analysis ...
virtual void Fei4Analysis::process(): histogrammerDone!
void Fei4Analysis::end()
-> All rogue threads joined!
-> Cleanup
Plotting : EnMask-0
Warning: empty cb range [1:1], adjusting to [0.99:1.01]
Plotting : OccupancyMap-0
Plotting : L1Dist-0
Empty data containers.
 -> send EORE
Command Thread exiting due to exception:
Error sending data:
  From /local/pixel/YarrEudaqProducer/EUDAQ/src/EUDAQ/main/lib/src/TransportTCP.cc:229
  In void eudaq::{anonymous}::do_send_data(SOCKET, const unsigned char*, size_t)

ERROR IN TLU PRODUCER:

Error sending data:
  From /local/pixel/eudaq/eudaq-1.7/main/lib/src/TransportTCP.cc:229
  In void eudaq::{anonymous}::do_send_data(SOCKET, const unsigned char*, size_t)

While copying this error message pressed control C in TLU Producer ------------------------------------

restarted ./STARTRUN

reran  ./TLUProducer.exe -r 172.27.100.8:44000

reconfigured EUDAQ in EUDAQ RD using ni_coins_YARR because it seems like it was the most recently updated

starting run 933

Histograms not popping up; killing all jobs and restarting ------------------------------------------------

went through steps 1-3

Called Su Dong: suggested we look at size of files

df -k

Filesystem           1K-blocks     Used Available Use% Mounted on

/dev/sda1             20030100 18989568     16392 100% /

moved data files to

 /home/tfuser/data_20180603/

 Now down to 81% full

Filesystem           1K-blocks     Used Available Use% Mounted on

/dev/sda1             20030100 15243540   3762420  81% /

Okay, so source of the problem was the disk full. --------------------------------------

Step 3 shows errors in finding libraries

[tfuser@ar-eudaq eudaq-1.7]$ ./OnlineMon.exe: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./OnlineMon.exe)

./OnlineMon.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./OnlineMon.exe)

NEEDED TO SOURCE SETUP_EUDAQ.SH. Edited twiki to reflect this. ---------------------------- 

./TLUProducer.exe -r 172.27.100.8:44000

bin/YarrProducer -h configs/controller/specCfgExtTrigger.json -c configs/connectivity/example_rd53a_setup.json -s configs/scans/rd53a/std_exttrigger.json -r 172.27.100.8

Step 4 Cannot connect to pixel@172.27.104.52

ssh_exchange_identification: Connection closed by remote host

 We can ping 172.27.104.52 but cannot log into it? Why? ------------------------------------------

From our end we are able to ssh into other places, so the issue appears to be with the 172.27.104.52 computer. We also tried just sshing with no username and this just hangs. Looked through ssh keys, no username associated with IP address although a key exists.

We think we might need to reset the computer, but the beam is on.

 8:00  Timon and Alexandra have arrived, Timon made an access and they are now resetting the computer. We stopped the HGTD run before the access request was made.17:00 Controlled access to rotate the RD53A module.

10:30 Offline analysis for Caladium X vs RD53A Y (due to transposed placement) showed clear correlation, but there are some puzzling extra spurious events.

12:00 RD53A data from last day had spurious extra events, which are tracked down as generated dummy events in the DUTproducer example template that was not removed after the actual DUT code implementation.  

17:00 Access to remount RD53A DUT. HGTD added another sensor in front of the 4th Caladium plane (now two sensors attached to both front and back).  

18:19 Access complete. RD53A rotated so display cables plugged in from the top.

18:21 Beam back on

21:00 Jason Nielsen & Murtaza Safdari on shift.  Aleksandra stays to monitor data-taking and supervise rotation of the RD53A module.

21:48 The most recent run, 148, shows nice correlation between the RD53A and Mimosa hit locations.

Image AddedImage Added

21:50 The Main Control Room called to say there will be no beam for the next 10 minutes.  We took the opportunity to perform a calibration scan.

22:00 Full shift program for remaining period signup complete. Thanks to everyone contributing. 

22:10 Beam is restarting in ESA. The Main Control Room mentioned that they will be programmed for 6 keV X-rays.  We do not think it will affect our beam much.

We confirmed that the cluster size is roughly 7-8 pixels in this last run.

Image AddedImage Added

00:01 Beam is off for calibration

00:10 Beam is back on.

After we saw a very weak correlation between RD53A and Mimosas in the previous run, we decided to pass through the configure step before starting the next run.  It may be a good idea to do this before each run, just to be safe.

00:58 Control Room called to let us know that they will be going up a couple of GeV in energy.  This may cause increased rate for us.

2:55 Early morning crew arrived (Vallary, Carolyn)

03:07 Control Room called to let us know that the energy will be increased again for the primary, starting in 10 minutes.

05:44 Control Room called to let us know that the energy will be increased again for the primary. Will start the new run once beam is stable.

10:05 Control Room called to let us know that the beam is down, they are changing the energy. Restart when the beam is back.

10:32 Beam is back, beam position changed a bit

12:13 Beam lost, should be back in 30 minutes

12:59 Beam is back, rate to high

15:15 Requested access to make reference for the angle of the chip. Please take a screenshot after changing position!

16:07 Beam is back on

17:34 Plan:

• take two runs for each position of the chip
  • 150
  • 140
  • 130
  • 120
  • 115
  • 110
  • 105
  • 100
  • 95
  • 90
• take a snapshot for each position, save the picture on the desktop, note the name of the picture in the table
• note the position of the motor in the table
• when reached perpendicular to the beam take the data for the end of the beam
• when the beam is off (and all angles are covered) notify Ben/Timon so we can tune the chip to a different threshold and wait for futher instructions
• before each start of the run, load the config files
• restart the DAQ system
• check the correlation plots Telescope - RD53a

18:00 Ke and SD on shift

18:20 Adjusted DUT rotation to RBV=151.24 (step -10). Rotation stage stuttered a bit but got there eventually.  

18:30 Runs 187-189 had some suspiciously low rate of DUT hits, online mon crash because hitting reset during start. Finally got Run 190 going with reconfig.

18:35 Run summary info now recorded in Google Doc Run List directly. 

19:30 Noticing the mean hit rate per event rather low seems to be due to beam going on and off. So the total number of events may not be a reliable number to use for the actual track statistics.

19:41 Run 191 reached 10000 events but very low entries in correlation plots so we just let the Run roll over to 192 without saving/reseting monitoring histograms.

20:15 Run 192 reached max events then rolled over to 193 which we stopped after small number of events to conclude this measurements for this angle, with combined mon histograms from Run 191-193 saves as run193.root. DUT rotation -10 RBV=141.24. 

20:30 We did a simple calculation from the DUT X vs MIMOSA26_2 X correlation plot from the previous angle's Run190.root histo which showed a slope of (50um/18.4um)*dX(DUT)/dx(MIMOSA)=1/sin(theta) which implied 180-theta=155.3 degrees, which is not far off from RBV=151.24 which is supposedly quite close to the actual angle in degrees. This method is also immune to small telescope tilt which cancels in the slope. The X correlation should be also mostly independent of the active Si depth knowledge. The cluster sized based estimate from  tan(theta)=150um/(Cluster-size) gave 150 degrees while the cluster size measurement is somewhat less precise and needs to be sure of the 150um Si depth.

21:22 Run 195 looks like has lost the DUT vs MIMOSA correlation. Stopped run, terminated Run Control and restarted everything for Run 196.

21:40 Run 196 still showing only very faint DUT correlation so we stopped it to reconfig, but looked like we lost beam upon the start of Run 197 so we stopped that short run also. 

21:45 Beam back. Started Run 198 and correlations looked much more healthy. 

21:50 Primary beam energy went up to 12.48 briefly and then back to 12.12 GeV. Starting a new run for HGTD anyway. 

22:18 Stopped Run 198 to change DUT rotation angle to 130 deg. DUT rotation -10 RBV=131.24. Reconfigured twice to be sure to start Run 199. Correlations look good.

22:23 Noticed that each time we rotate and correlated hit region of DUT keeps on marching to higher X and the top edge is getting closer to the right edge. Since the top tip of the correlation diagonal line is running behind the histogram stat box, we tried to move the display stat box but that crashed the online Mon. Had to restart everything for new Run 200. 

23:00 After the end of Run 200, we moved the DUT by -2.5mm (towards ESA gate - camera confirmed) to get away from the +x edge.

23:02 Started Run 201.  X correlation moved left by the desired amount as expected.

23:14 The DUT correlation looked rather faint/low stat. Ending Run 201 early to reconfigure.

23:15 Started new Run 215 after double reconfig. Correlations looked better.

23:42 MCC called to change primary beam energy to 12.40 GeV. Ending Run 202.

23:45 Rotated DUT -10 to RBV=121.24.

23:46 Started Run 203, but noticed no beam, so stopped it with 0 event. 

0:39 Moved DUT dx=-2mm to avoid +x edge

0:39 Run 207 started. Seemed to have more background hits and correlation faint. Ending Run to reconfig.

0:53 Started Run 208. Rate and correlation looked better.

1:30 Rotated DUT -5 RBV=116.24.

1:32 Started Run 209.  

2:00 Yubo and Jannicke are on shift

2:19 Stopped Run 210 because of faint correlation plots, reconfigured

2:20 Start Run 211 correlation looks better now

7:45 Run 222 Rate went funny in online monitor (down from 5.1 (4.97) Hz to 4.8 (inf) Hz. Stopped the run (also on HGTD). On restart Yarr producer dead. After restarting Yarr producer, Data collector dead. 

8:07 After terminating in eudaq Run Control and restarting, we are up and running again! Restarted HGTD as well. 

8:55 Run 227 at hard rotation limit (RBV from step: 98.71), not pushing further to 95, but instead taking data at 98.71.

10:04 Ended Run 228 to access so that we can get around the rotation hard limit to reach 0 degree setting. Simone replaced HGTD sensors with 3-pack in front of 4th plane while removed the one behind that plane. Timon doing calibrations to get a low threshold tune.

10:30 While one can turn the rotation stage knob to make it pass the 0 degree point. The behavior of the software control limit is mysterious. We settled to a position that is 0 degree with the best we can do by measuring distance from 3rd MIMOSA plane to the DUT support frame to be equal on both left and right sides with within 1mm. This doesn't set the sensor angle to beam exactly to zero but makes the DUT parallel to MIMOSA plane. Rotation (new scale) RBV=146.51 for this 0 deg setting. Unfortunately, we can only increase RBV by remote control from this point and it cannot go back. We also took measurements of the plane positions while there.    

11:40 moved -4mm in x to center diff FE more in beam. Stopped run and started new one. Only diff FE is enabled with new 800e threshold cal.

11:50 were not seeing correlations with Mimosa 26-2 either in x or y. So decided to stop run, configure twice, and start new run. Forgot to save monitoring plots (the ones showing no correlation). Now in new run we can see correlations.

12:24 stopped run just shy of 9800 events. Just over 1000 hits in diff FE recorded. Started new run, same angle (0)

12:37 for final run plan to rotate in such a way as to match the angle seen in file RD53_Rotation_131_25_dx2p5.png. Want to allow for a 45 minute run + 10 minute rotation and verification of hit rate. Therefore must stop this tun at 13:05 even if it has not reached 10k events yet.

12:50 looks like we lost the correlation at some point. Correlation plots looked bad. Stopped run, configured twice, and started new one. Still at 0 angle

13:05 stopped run. Rotated (file:RD53_Rotation_188_51_new-scale.png). Started a run to see hits. Then moved +4mm in x to get better illumination (saved new image). Noticed no correlation. Stopped, configured twice, and started new run.

13:16 no correlation seen. Stopped run (forgot to same mon. plots). configured. started new one.Looks good. But I don't like this angle (45 deg.) Clusters are too short (size 3). Will take stop at 13:30 and take one more angle (high).

13:29 stopped run. Rotated to very shallow angle (RD53_Rotation_213_51_new-scale-moved-in-x.png). Started new run. Cluster size is now 7.

13:41 correlation seems lost. Stopping, configuring, starting new run.

14:02 14:00 has come and gone and MCR has not called yet. Beam is still going. Keep running

14:08 the beam is gone. Stopped run. Going into access. The end.

Post Beam Session Upgrades

Two main goals: 

• Find a more permanent host machine for the TPUProducer (which was running on the YARR DUT machine temporarily during May-Jun beam session). The TLUProducer used to run on the NI Windows system, but the difficulties in building Windows applications during the upgrade process to EUDAQ 1.7 led to the conclusion that it is better to host TLUProducer on a Linux node with easier build framework.   
• Bring RCEproducer forward to be compatible with EUDAQ 1.7 for FE-I4 devices

The Sun server for DUT readout is retired and removed from ESA. A new Kingdel fanless PC box running Centos 7 is setup by Matthias and moved into the Caladium rack to replace the old Sun server and also assumed the role of the host for the TLUProducer which communicates with the TLU through a local USB link in the rack. The new Kingdel box retained esadutdaq1 node name and the IP address of 172.27.104.45 on the ESA_RESTRICTED network. 

The test associated with this was plagued by events not built errors when FE-I4 DUT were added to the DAQ. This was traced (Sep/11) to the missing ground connection between HSIO2 and TLU. The CAT6 cable cage between TLU DUT port and HSIO2 RTM was insufficient. HSIO2 RTM design consciously grounded the RJ45 cage to the cable cage contact, but the TLU DUT port probably didn't. So the kluge of a LEMO cable between any of the socket in the HSIO2 main board LEMO bank and the TLU DUT trigger LEMO port (e.g. the TTL trig0) is necessary to establish the DC reference level between HSIO2 and TLU.