...
Run Number | Start time | End time | Number of Events | Window coordinates (DUT) | Comments |
---|---|---|---|---|---|
90 | 02/06/18 11pm | 02/06/18 11:25 pm | 8655 | x=100~300 | |
91 | 02/06/18 11:27 pm | ||||
92 | 02/06/18 11:27 pm | 03/06/18 00:27 am | 19007 | We hadn't saved the histogram at the end of run 91, so the run histogram just has twice as much data. But the beam conditions were stable between these two runs. | |
93 | 03/06/18 00:30 am | 03/06/18 01:03 am | 9021 | ||
94 | 03/06/18 01:05 am | 03/06/18 01:36 am | 9284 | ||
95 | 03/06/18 01:39 am | 03/06/18 02:07 am | 9013 | ||
96 | moving the RD53: up 1cm (Motor 3: Y fine : > 1cm) | ||||
97 | 03/06/18 02:24 am | 03/06/18 02:54 am | 9113 | ||
98 | 03/06/18 02:57 am | 03/06/18 | 6845 | Changed shift to Ke and Jannicke Stopped early because we were looking at Sensor 0 for measuring the number of events, will be using number of events from Sensor 2 from now on - it does not have hot pixels! | |
99 | 03/06/18 3:14 am | 03/06/18 3:40am | 9210 | ||
100 | 03/06/18 3:40am | 03/06/18 4:06am | 9406 | ||
101 | 03/06/18 4:06am | 03/06/18 4:34am | 9425 | ||
102 | 03/06/18 4:34am | 03/06/18 5:04 | 9173 | ||
103 | 03/06/18 5:04am | 03/06/18 5:31am | 9150 | X: (0,~100) | Moved RD53 (Motor 3 Y:fine < 2 cm) RBV from step at 10.00mm now. Aiming for coverage of 0-100 in X |
104 | 03/06/18 5:31am | 03/06/18 | 9245 | Moved RD53 (Motor 3 Y:fine > 2 mm) RBV from step at 12.00mm now, adjustment was made to get slightly better coverage in the 50-100 range. Looks good now! | |
105 | 03/06/18 | 03/06/18 6:12am | 1197 | Run stopped early - noticed histograms stopped filling ERROR IN DUT DATA PRODUCER vent: tag(0) l1id(14 bcid(11398) hits (0) ERROR IN TLU PRODUCER: Error sending data: While copying this error message pressed control C in TLU Producer ------------------------------------ restarted ./STARTRUN reran ./TLUProducer.exe -r 172.27.100.8:44000 reconfigured EUDAQ in EUDAQ RD using ni_coins_YARR because it seems like it was the most recently updated starting run 933 Histograms not popping up; killing all jobs and restarting ------------------------------------------------ went through steps 1-3 Called Su Dong: suggested we look at size of files df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 20030100 18989568 16392 100% /
moved data files to /home/tfuser/data_20180603/ Now down to 81% full Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 20030100 15243540 3762420 81% / Okay, so source of the problem was the disk full. -------------------------------------- Step 3 shows errors in finding libraries [tfuser@ar-eudaq eudaq-1.7]$ ./OnlineMon.exe: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./OnlineMon.exe) ./OnlineMon.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./OnlineMon.exe) NEEDED TO SOURCE SETUP_EUDAQ.SH. Edited twiki to reflect this. ----------------------------
./TLUProducer.exe -r 172.27.100.8:44000 bin/YarrProducer -h configs/controller/specCfgExtTrigger.json -c configs/connectivity/example_rd53a_setup.json -s configs/scans/rd53a/std_exttrigger.json -r 172.27.100.8 Step 4 Cannot connect to pixel@172.27.104.52 ssh_exchange_identification: Connection closed by remote host We can ping 172.27.104.52 but cannot log into it? Why? ------------------------------------------ From our end we are able to ssh into other places, so the issue appears to be with the 172.27.104.52 computer. We also tried just sshing with no username and this just hangs. Looked through ssh keys, no username associated with IP address although a key exists. We think we might need to reset the computer, but the beam is on.
| |
03/06/18 | Timon and Alexandra have arrived, Timon made an access and they are now resetting the computer. We stopped the HGTD run before the access request was made. | ||||
106 | 03/06/18 | 03/06/18 9:05 | 9334 | Beam is back, restarted a new run. Also restarted a new run for HGTD | |
107 | 03/06/18 9:08 | 03/06/18 9:40 | 9384 | Transferred to Alexandra | |
108-119 | 03/06/18 09:41 | 03/06/18 | empty | The beam was turned off at 9:51. 108-119 are runs without beam used for testing the YarrProducer setup. No root files saved. RD53a config file changed. | |
120 | 03/06/18 10:50 | 03/06/18 10:53 | few events | To test the config files. | |
121 | 03/06/18 10:54 | 03/06/18 11:09 | 4439 | First run for analyzing with a new config. Restarted a new run for HGTD. The EUDAQ online monitor crashed; root file not saved. | |
122-126 | 03/06/18 11:13 | 03/06/18 11:40 | Testing Yarr and EUDAQ; no good data for analyzing. Root files not saved.
| ||
127 | 03/06/18 11:41 | 03/06/18 12:15 | 10000 | EUDAQ online histograms not reset or saved. | |
128 | 03/06/18 12:15 | 03/06/18 12:49 | 9812 | Automatic new run. Reset EUDAQ online histograms. | |
129 | 03/06/18 12:51 | 03/06/18 12:59 | 2300 | No root file saved. | |
130 | 03/06/18 12:59 | 03/06/18 13:08 | 2518 | No root file saved. | |
131 | 03/06/18 13:10 | 03/06/18 13:12 | 702 | No root file saved. FIXED in this run: Example producer was sending fake data every 20ms. | |
132 | 03/06/18 13:12 | 03/06/18 13:47 | 10000 | Automatic new run. EUDAQ online histograms were not reset. | |
133 | 03/06/18 13:47 | 03/06/18 14:20 | 9915 | EUDAQ online histograms with previous run. | |
134-137 | 03/06/18 | 03/06/18 | Path for data changed. Monitor didn't work after path changed, so took extra runs trying to fix it. Disregard files with 'Run 1' (0x0B57_000001.*); this was from eudaq not pointing to the run number .dat file. | ||
138 | 03/06/18 14:51 | 03/06/18 15:25 | 9861 | Monitor works now.
| |
139 | 03/06/18 15:37 | 03/06/18 15:47 | Used for moving RD53A | ||
140 | 03/06/18 15:48 | 03/06/18 16:22 | 9866 | X: (~150,~350) | Moved RD53 (Motor 3 Y:fine > 10 mm) RBV from step at 22.00mm now. (Previous RBV from step: 12.00mm.) Aiming for X~300. |
141 | 03/06/18 16:22 | 03/06/18 16:56 | 9875 |
Jun/3
Datataking came to a halt due to data disk /dev/sda1 full and YARR computer USB errors prevented login. More detail in Event log. Ended Run early. | |||||
106 | 03/06/18 | 03/06/18 9:05 | 9334 | Beam is back, restarted a new run. Also restarted a new run for HGTD | |
107 | 03/06/18 9:08 | 03/06/18 9:40 | 9384 | Shift Transferred to Alexandra & Charilou | |
108-119 | 03/06/18 09:41 | 03/06/18 | empty | The beam was turned off at 9:51. 108-119 are runs without beam used for testing the YarrProducer setup. No root files saved. RD53a config file changed. | |
120 | 03/06/18 10:50 | 03/06/18 10:53 | few events | To test the config files. | |
121 | 03/06/18 10:54 | 03/06/18 11:09 | 4439 | First run for analyzing with a new config. Restarted a new run for HGTD. The EUDAQ online monitor crashed; root file not saved. | |
122-126 | 03/06/18 11:13 | 03/06/18 11:40 | Testing Yarr and EUDAQ; no good data for analyzing. Root files not saved.
| ||
127 | 03/06/18 11:41 | 03/06/18 12:15 | 10000 | EUDAQ online histograms not reset or saved. | |
128 | 03/06/18 12:15 | 03/06/18 12:49 | 9812 | Automatic new run. Reset EUDAQ online histograms. | |
129 | 03/06/18 12:51 | 03/06/18 12:59 | 2300 | No root file saved. | |
130 | 03/06/18 12:59 | 03/06/18 13:08 | 2518 | No root file saved. | |
131 | 03/06/18 13:10 | 03/06/18 13:12 | 702 | No root file saved. FIXED in this run: Example producer was sending fake data every 20ms. | |
132 | 03/06/18 13:12 | 03/06/18 13:47 | 10000 | Automatic new run. EUDAQ online histograms were not reset. | |
133 | 03/06/18 13:47 | 03/06/18 14:20 | 9915 | EUDAQ online histograms with previous run. | |
134-137 | 03/06/18 | 03/06/18 | Path for data changed. Monitor didn't work after path changed, so took extra runs trying to fix it. Disregard files with 'Run 1' (0x0B57_000001.*); this was from eudaq not pointing to the run number .dat file. | ||
138 | 03/06/18 14:51 | 03/06/18 15:25 | 9861 | Monitor works now.
| |
139 | 03/06/18 15:37 | 03/06/18 15:47 | Used for moving RD53A | ||
140 | 03/06/18 15:48 | 03/06/18 16:22 | 9866 | X: (~150,~350) | Moved RD53 (Motor 3 Y:fine > 10 mm) RBV from step at 22.00mm now. (Previous RBV from step: 12.00mm.) Aiming for X~300. |
141 | 03/06/18 16:22 | 03/06/18 16:56 | 9875 |
Jun/3
6:00 Event data taking came to a halt during Run 105:
Run stopped early - noticed histograms stopped filling
ERROR IN DUT DATA PRODUCER
vent: tag(0) l1id(14 bcid(11398) hits (0)
Event: tag(0) l1id(15 bcid(11399) hits (0)
####################
### Stopping Run ###
####################
Got 1791events!
Caught interrupt, stopping data taking!
Abort will leave buffers full of data!
Caught interrupt, stopping data taking!
Abort will leave buffers full of data!
Waiting for run thread ...
-> Processor done, waiting for histogrammer ...
virtual void Fei4Histogrammer::process(): processorDone!
-> Processor done, waiting for analysis ...
virtual void Fei4Analysis::process(): histogrammerDone!
void Fei4Analysis::end()
-> All rogue threads joined!
-> Cleanup
Plotting : EnMask-0
Warning: empty cb range [1:1], adjusting to [0.99:1.01]
Plotting : OccupancyMap-0
Plotting : L1Dist-0
Empty data containers.
-> send EORE
Command Thread exiting due to exception:
Error sending data:
From /local/pixel/YarrEudaqProducer/EUDAQ/src/EUDAQ/main/lib/src/TransportTCP.cc:229
In void eudaq::{anonymous}::do_send_data(SOCKET, const unsigned char*, size_t)
ERROR IN TLU PRODUCER:
Error sending data:
From /local/pixel/eudaq/eudaq-1.7/main/lib/src/TransportTCP.cc:229
In void eudaq::{anonymous}::do_send_data(SOCKET, const unsigned char*, size_t)
While copying this error message pressed control C in TLU Producer ------------------------------------
restarted ./STARTRUN
reran ./TLUProducer.exe -r 172.27.100.8:44000
reconfigured EUDAQ in EUDAQ RD using ni_coins_YARR because it seems like it was the most recently updated
starting run 933
Histograms not popping up; killing all jobs and restarting ------------------------------------------------
went through steps 1-3
Called Su Dong: suggested we look at size of files
df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 20030100 18989568 16392 100% /
moved data files to
/home/tfuser/data_20180603/
Now down to 81% full
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 20030100 15243540 3762420 81% /
Okay, so source of the problem was the disk full. --------------------------------------
Step 3 shows errors in finding libraries
[tfuser@ar-eudaq eudaq-1.7]$ ./OnlineMon.exe: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./OnlineMon.exe)
./OnlineMon.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./OnlineMon.exe)
NEEDED TO SOURCE SETUP_EUDAQ.SH. Edited twiki to reflect this. ----------------------------
./TLUProducer.exe -r 172.27.100.8:44000
bin/YarrProducer -h configs/controller/specCfgExtTrigger.json -c configs/connectivity/example_rd53a_setup.json -s configs/scans/rd53a/std_exttrigger.json -r 172.27.100.8
Step 4 Cannot connect to pixel@172.27.104.52
ssh_exchange_identification: Connection closed by remote host
We can ping 172.27.104.52 but cannot log into it? Why? ------------------------------------------
From our end we are able to ssh into other places, so the issue appears to be with the 172.27.104.52 computer. We also tried just sshing with no username and this just hangs. Looked through ssh keys, no username associated with IP address although a key exists.
We think we might need to reset the computer, but the beam is on.
8:00 Timon and Alexandra have arrived, Timon made an access and they are now resetting the computer. We stopped the HGTD run before the access request was made.17:00 Controlled access to rotate the RD53A module.
10:30 Offline analysis for Caladium X vs RD53A Y (due to transposed placement) showed clear correlation, but there are some puzzling extra spurious events.
12:00 RD53A data from last day had spurious extra events, which are tracked down as generated dummy events in the DUTproducer example template that was not removed after the actual DUT code implementation.
17:00 Access to remount RD53A DUT. HGTD added another sensor in front of the 4th Caladium plane (now two sensors attached to both front and back). 17:00 Controlled access to rotate the RD53A module.
18:19 Access complete. RD53A rotated so display cables plugged in from the top.
...
21:50 The Main Control Room called to say there will be no beam for the next 10 minutes. We took the opportunity to perform a calibration scan.
22:00 Full shift program for remaining period signup complete. Thanks to everyone contributing.