In this page:
Prerequisites
To use the xtc indexing, users need to
- update the myana package to the latest one. See "A Simple Online Analysis Example" for more details
- make sure the *.xtc.idx files (xtc index files) are present with the *.xtc files
Introduction to index files
The index files are generated by DAQ automatically along with the xtc files. They are located in a subdirectory "index" of
the xtc files. For example, in daq-cxi-dss01:/u2/pcds/pds/cxi/e55, we can see the xtc file e55-r1055-s00-c00.xtc.tobedeleted,
and the file e55-r1055-s00-c00.xtc.idx in the index subdirectory. The *.xtc.idx files are the xtc index files.
Since the above xtc file (run 1055) has the corresponding index file, when we run myana program to analyze the xtc file, the myana program will automatically load the corresponding index (*.xtc.idx)
file to perform indexing functions.
Local copy of the xtc file
Sometimes we want to copy the xtc files to our own directory to do further myana analysis. To enable the xtc indexing functions,
we need to copy the corresponding index files to either:
- the same directory where xtc file is located
- a subdirectory called "index", under our own xtc directory
Then when we run the myana program on the private directory, it will automatically checks for the presence of the index
file.
For example:
Copy xtc files to local directory
[tomytsai@daq-cxi-dss01 e55]$ cp e55-r1055-s00-c00.xtc.tobedeleted ~/myana/ [tomytsai@daq-cxi-dss01 e55]$ cp index/e55-r1055-s00-c00.xtc.idx ~/myana/ or [tomytsai@daq-cxi-dss01 e55]$ cp index/e55-r1055-s00-c00.xtc.idx ~/myana/index/
Missing index files
If, for some xtc files, there is no corresponding index file, we can re-generate the index files, based on the existing xtc files. Please contact
Tomy (tomytsai at slac.stanford.edu) for mor details.
Random seeking inside an xtc file
With the presence of the index file, users can "jump" to any event in an xtc file.
We will use a simple myana program, called myana_index, to demonstrate this "jumping" effect. First, the ordinary myana_index
program output is as follows:
myana_index output
[tomytsai@daq-cxi-dss01 e55]$ myana_index -f e55-r1055-s00-c00.xtc.tobedeleted Opening ROOT output file e55-r1055-s00-c00.xtc.root <0> PDT Wed 2011-07-13 22:18:03.205827134 fid 0x1ffff vec 000 Configure extent 0x8d65c damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.559318129 fid 0x1ffff vec 000 BeginRun extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.680735389 fid 0x1ffff vec 000 BeginCalibCycle extent 0x1a0 damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.681734803 fid 0x1ffff vec 000 Enable extent 0xc8 damage 0x0 <0> 22:18:07.712267237 fid 0x0b01f vec 003 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 01 (global 01) <0> 22:18:07.745608886 fid 0x0b02b vec 007 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 02 (global 02) ... <0> 22:18:21.383067771 fid 0x0c357 vec 1643 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 411 (global 411) <0> 22:18:21.416410000 fid 0x0c363 vec 1647 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 412 (global 412) <0> PDT Wed 2011-07-13 22:18:21.429841617 fid 0x1ffff vec 000 Disable extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:21.656270293 fid 0x1ffff vec 000 EndCalibCycle extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:21.730197578 fid 0x1ffff vec 000 EndRun extent 0xc8 damage 0x0 User analysis endrun() routine called. Processed 412 events, 0 damaged, with damage mask 0x0. User analysis endjob() routine called.
As seen above, myana_index doesn't do any practical analysis, but it just process each event in the xtc file, and print out the
timestamp (in HH:MM:SS.NANO_SECOND format) for each event. Also it shows the event number (calib XX event XX) at the end of
each line. Some xtc file might have multiple BeginCalibCycles, and in this case the event number is a "local" number within
the same BeginCalibCycle.
Now, we can use myana_index -j XX to jump to any event in this xtc file:
myana_index output with event jumping
[tomytsai@daq-cxi-dss01 e55]$ myana_index -f e55-r1055-s00-c00.xtc.tobedeleted -j 400 Will jump to Event# 400 (After Calib Cycle) Opening ROOT output file e55-r1055-s00-c00.xtc.root <0> PDT Wed 2011-07-13 22:18:03.205827134 fid 0x1ffff vec 000 Configure extent 0x8d65c damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.559318129 fid 0x1ffff vec 000 BeginRun extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.680735389 fid 0x1ffff vec 000 BeginCalibCycle extent 0x1a0 damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.681734803 fid 0x1ffff vec 000 Enable extent 0xc8 damage 0x0 Jumped to Calib# 1 Event# 400 in e55-r1055-s00-c00.xtc.tobedeleted XtcRun::jump(): Jumped to Calib# 1 Event# 400 (Global# 400) <0> 22:18:21.016275563 fid 0x0c2d3 vec 1599 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 400 (global 400) <0> 22:18:21.049620046 fid 0x0c2df vec 1603 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 401 (global 401) ... <0> 22:18:21.383067771 fid 0x0c357 vec 1643 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 411 (global 411) <0> 22:18:21.416410000 fid 0x0c363 vec 1647 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 412 (global 412) <0> PDT Wed 2011-07-13 22:18:21.429841617 fid 0x1ffff vec 000 Disable extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:21.656270293 fid 0x1ffff vec 000 EndCalibCycle extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:21.730197578 fid 0x1ffff vec 000 EndRun extent 0xc8 damage 0x0 User analysis endrun() routine called. Processed 13 events, 0 damaged, with damage mask 0x0. User analysis endjob() routine called.
As seen above, the first event process by myana_index is event 400, and then all the following events will
be processed, until the end of xtc file. Since this xtc file has totally 412 events, we
are free to jump to any events between 1 and 412.
Note that the "-j" command option is supported thorugh myana package, so we can write our own
myana program to compile with the myana package. Then the custom myana program will also support the
"-j XXX" option do random event seeking.
Jumping to any BeginCalibCycle (scan)
In addition to "-j XXX" option, myana supports the "-y XXX" option to jump to any BeginCalibCycle (
provided that the xtc file has multiple BeginCalibCycles) inside the file.
Here we use an xtc file, e70-r0320-s00-c00.xtc from XPP as exmaple:
xtc file with multiple BeginCalibCycle
[tomytsai@psdev ana]$ myana_index -f e70-r0320-s00-c00.xtc Opening ROOT output file e70-r0320-s00-c00.root <0> PST Mon 2010-11-15 20:59:44.411399828 fid 0x1ffff vec 000 Configure extent 0x9cbc damage 0x0 <0> PST Mon 2010-11-15 21:00:05.226408131 fid 0x1ffff vec 000 BeginRun extent 0x8c damage 0x0 <0> PST Mon 2010-11-15 21:00:05.347084126 fid 0x1ffff vec 000 BeginCalibCycle extent 0x1b8 damage 0x0 <0> PST Mon 2010-11-15 21:00:05.348690968 fid 0x1ffff vec 000 Enable extent 0x8c damage 0x0 <0> 21:00:06.481951226 fid 0x17346 vec 000 L1Accept extent 0x355664 damage 0x0 calib 1 event 01 (global 01) <0> 21:00:07.090321303 fid 0x17421 vec 001 L1Accept extent 0x304 damage 0x2 calib 1 event 02 (global 02) <0> PST Mon 2010-11-15 21:00:07.091625522 fid 0x1ffff vec 000 Disable extent 0x8c damage 0x0 <0> PST Mon 2010-11-15 21:00:10.521935889 fid 0x1ffff vec 000 EndCalibCycle extent 0x8c damage 0x0 ... <0> PST Mon 2010-11-15 21:00:29.541235539 fid 0x1ffff vec 000 BeginCalibCycle extent 0x1b8 damage 0x0 <0> PST Mon 2010-11-15 21:00:29.542685937 fid 0x1ffff vec 000 Enable extent 0x8c damage 0x0 <0> 21:00:30.482200801 fid 0x19506 vec 008 L1Accept extent 0x355664 damage 0x0 calib 5 event 01 (global 09) <0> 21:00:31.257151924 fid 0x1961d vec 009 L1Accept extent 0x304 damage 0x2 calib 5 event 02 (global 10) <0> PST Mon 2010-11-15 21:00:31.258483312 fid 0x1ffff vec 000 Disable extent 0x8c damage 0x0 <0> PST Mon 2010-11-15 21:00:34.523275271 fid 0x1ffff vec 000 EndCalibCycle extent 0x8c damage 0x0 <0> PST Mon 2010-11-15 21:01:09.487026185 fid 0x1ffff vec 000 EndRun extent 0x8c damage 0x0 User analysis endrun() routine called. Processed 10 events, 5 damaged, with damage mask 0x2. User analysis endjob() routine called.
We see that this xtc file contains 5 BeginCalibCycles, and 10 events in total. Now we can freely jump
to any BeginCalibCycle, between 1 and 5, and any events within that cycle, between 1 and 2 (since
one cycle only contains two events):
jump to events within a BeginCalibCycle
[tomytsai@psdev ana]$ myana_index -f e70-r0320-s00-c00.xtc -y 4 -j 2 Will jump to Calib# 4 Will jump to Event# 2 (After Calib Cycle) Opening ROOT output file e70-r0320-s00-c00.root <0> PST Mon 2010-11-15 20:59:44.411399828 fid 0x1ffff vec 000 Configure extent 0x9cbc damage 0x0 <0> PST Mon 2010-11-15 21:00:05.226408131 fid 0x1ffff vec 000 BeginRun extent 0x8c damage 0x0 Jumped to Calib# 4 Event# 0 in e70-r0320-s00-c00.xtc <0> PST Mon 2010-11-15 21:00:22.636371775 fid 0x1ffff vec 000 BeginCalibCycle extent 0x1b8 damage 0x0 <0> PST Mon 2010-11-15 21:00:22.637707354 fid 0x1ffff vec 000 Enable extent 0x8c damage 0x0 Jumped to Calib# 4 Event# 2 in e70-r0320-s00-c00.xtc XtcRun::jump(): Jumped to Calib# 4 Event# 2 (Global# 8) <0> 21:00:24.674123011 fid 0x18cdb vec 007 L1Accept extent 0x304 damage 0x2 calib 4 event 02 (global 08) <0> PST Mon 2010-11-15 21:00:24.675114061 fid 0x1ffff vec 000 Disable extent 0x8c damage 0x0 <0> PST Mon 2010-11-15 21:00:29.422665753 fid 0x1ffff vec 000 EndCalibCycle extent 0x8c damage 0x0 <0> PST Mon 2010-11-15 21:00:29.541235539 fid 0x1ffff vec 000 BeginCalibCycle extent 0x1b8 damage 0x0 <0> PST Mon 2010-11-15 21:00:29.542685937 fid 0x1ffff vec 000 Enable extent 0x8c damage 0x0 <0> 21:00:30.482200801 fid 0x19506 vec 008 L1Accept extent 0x355664 damage 0x0 calib 5 event 01 (global 09) <0> 21:00:31.257151924 fid 0x1961d vec 009 L1Accept extent 0x304 damage 0x2 calib 5 event 02 (global 10) <0> PST Mon 2010-11-15 21:00:31.258483312 fid 0x1ffff vec 000 Disable extent 0x8c damage 0x0 <0> PST Mon 2010-11-15 21:00:34.523275271 fid 0x1ffff vec 000 EndCalibCycle extent 0x8c damage 0x0 <0> PST Mon 2010-11-15 21:01:09.487026185 fid 0x1ffff vec 000 EndRun extent 0x8c damage 0x0 User analysis endrun() routine called. Processed 3 events, 2 damaged, with damage mask 0x2. User analysis endjob() routine called.
After applying the command option "-y 4 -j 2" to myana program, we see that the first processed event becomes BeginCalibCycle 4, event 2. After the first
event, the following events will be processed until the end of file.
Time searching inside an xtc file
In addition to random seeking, index file allow us to do fast searching inside xtc file, by giving a timestamp. Before we demonstrate
the result, we have to discuss the corroct format for specifying the timestamp.
Format of a timestamp string
The format of timestamp is in the form of "YYYY-MM-DD HH:MM:SS.NANO_SECOND TIME_ZONE".
For example, "2011-07-13 22:18:10.123 PDT " is a valid timestamp. For TIME_ZONE, we could use either one of the following:
- Don't specify time zone. By default the system uses PST - Pacific Standard Time (GMT-08.00)
- Use PDT to denote Pacific Daylight Saving Time
- Use DST to denote Daylight Saving Time (same as above)
- Use GMT, UTC, or Z to denote Greenwich Mean Time
From the above myana_index output examples, we could see the file e55-r1055-s00-c00.xtc.tobedeleted was reported using
PDT time, and e70-r0320-s00-c00.xtc PST time, but it is just the program's behavior. The xtc file
always stores UTC times internally, so it is up to user's choice to use which timezone.
Time searching results
We can use -t "timestamp" in the command line to make myana search for the event with the exact timestamp. If there
is no event having the exact timestamp, myana will jump to the first event that is later than the specified time:
searching with timestamp
[tomytsai@daq-cxi-dss01 e55]$ myana_index -f e55-r1055-s00-c00.xtc.tobedeleted -t "2011-07-13 22:18:10.123 PDT" Opening ROOT output file e55-r1055-s00-c00.xtc.root Time PDT Wed 2011-07-13 22:18:10.123s (adjusted by DST) (seconds 0x4e1e7c12 nanosecs 0x754d4c0) Going to event with time 2011-07-13 22:18:10.123 PDT Calib# 1 Event# 74 <0> PDT Wed 2011-07-13 22:18:03.205827134 fid 0x1ffff vec 000 Configure extent 0x8d65c damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.559318129 fid 0x1ffff vec 000 BeginRun extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.680735389 fid 0x1ffff vec 000 BeginCalibCycle extent 0x1a0 damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.681734803 fid 0x1ffff vec 000 Enable extent 0xc8 damage 0x0 Jumped to Calib# 1 Event# 74 in e55-r1055-s00-c00.xtc.tobedeleted XtcRun::jump(): Jumped to Calib# 1 Event# 74 (Global# 74) <0> 22:18:10.146263325 fid 0x0b38b vec 295 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 74 (global 74) <0> 22:18:10.179607221 fid 0x0b397 vec 299 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 75 (global 75) <0> 22:18:10.212950903 fid 0x0b3a3 vec 303 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 76 (global 76) ... <0> 22:18:21.383067771 fid 0x0c357 vec 1643 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 411 (global 411) <0> 22:18:21.416410000 fid 0x0c363 vec 1647 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 412 (global 412) <0> PDT Wed 2011-07-13 22:18:21.429841617 fid 0x1ffff vec 000 Disable extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:21.656270293 fid 0x1ffff vec 000 EndCalibCycle extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:21.730197578 fid 0x1ffff vec 000 EndRun extent 0xc8 damage 0x0 User analysis endrun() routine called. Processed 339 events, 0 damaged, with damage mask 0x0. User analysis endjob() routine called.
We see that after we run myana with -t "2011-07-13 22:18:10.123 PDT", the first processed event has
timestamp 22:18:10.146263325, which is slightly later than our specified time.
We can verify if the earlier events really occured earlier than our specified time, by intentionally jumping
to an earlier event:
verify time searching result
[tomytsai@daq-cxi-dss01 e55]$ myana_index -f e55-r1055-s00-c00.xtc.tobedeleted -j 70 Opening ROOT output file e55-r1055-s00-c00.xtc.root <0> PDT Wed 2011-07-13 22:18:03.205827134 fid 0x1ffff vec 000 Configure extent 0x8d65c damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.559318129 fid 0x1ffff vec 000 BeginRun extent 0xc8 damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.680735389 fid 0x1ffff vec 000 BeginCalibCycle extent 0x1a0 damage 0x0 <0> PDT Wed 2011-07-13 22:18:07.681734803 fid 0x1ffff vec 000 Enable extent 0xc8 damage 0x0 Jumped to Calib# 1 Event# 70 in e55-r1055-s00-c00.xtc.tobedeleted XtcRun::jump(): Jumped to Calib# 1 Event# 70 (Global# 70) <0> 22:18:10.012894147 fid 0x0b35b vec 279 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 70 (global 70) <0> 22:18:10.046236048 fid 0x0b367 vec 283 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 71 (global 71) <0> 22:18:10.079577679 fid 0x0b373 vec 287 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 72 (global 72) <0> 22:18:10.112920806 fid 0x0b37f vec 291 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 73 (global 73) <0> 22:18:10.146263325 fid 0x0b38b vec 295 L1Accept extent 0x9809b0 damage 0x0 calib 1 event 74 (global 74) ...
Here we see the the events before event 74 occured earlier than our specified time 22:18:10.123. That means
event 74 is the indeed the first event that occured later than the specified time.
Run indexing functions for an xtc run
An xtc run is typically composed of multiple slices, and each slice may contain multiple chunks. We can do the
above indexing functions (event jump, and time search) in an xtc run. To demonstrate the usage, we use the
xtc files for e60 run329 as an example. This xtc run is composed of 20 files, and we need to list all filenames
in a text file for processing this run:
list of xtc files in an xtc run
[tomytsai@psana101 teste60]$ cat e60-r0329.lst e60-r0329-s00-c00.xtc e60-r0329-s00-c01.xtc e60-r0329-s00-c02.xtc e60-r0329-s00-c03.xtc e60-r0329-s01-c00.xtc e60-r0329-s01-c01.xtc e60-r0329-s01-c02.xtc e60-r0329-s01-c03.xtc e60-r0329-s02-c00.xtc e60-r0329-s02-c01.xtc e60-r0329-s02-c02.xtc e60-r0329-s02-c03.xtc e60-r0329-s03-c00.xtc e60-r0329-s03-c01.xtc e60-r0329-s03-c02.xtc e60-r0329-s03-c03.xtc
Then we use the myana -l <list file> to process all xtc files in this run:
process an xtc run
[tomytsai@psana101 teste60]$ myana_index -l e60-r0329.lst Opening ROOT output file e60-r0329.root Opening filelist e60-r0329.lst Analyzing files e60-r0329 [16] <0> PST Sun 2011-02-20 06:18:51.712917944 fid 0x1ffff vec 000 Configure extent 0x4cf28 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.810424731 fid 0x1ffff vec 000 BeginRun extent 0xa0 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.943730116 fid 0x1ffff vec 000 BeginCalibCycle extent 0x178 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.944620324 fid 0x1ffff vec 000 Enable extent 0xa0 damage 0x0 <0> 18:38:49.955460505 fid 0x0d1a3 vec 2184 L1Accept extent 0x526648 damage 0x4000 calib 1 event 01 (global 01) <2> 18:38:49.963796644 fid 0x0d1a6 vec 2185 L1Accept extent 0x526648 damage 0x4000 calib 1 event 02 (global 02) <1> 18:38:49.972127838 fid 0x0d1a9 vec 2186 L1Accept extent 0x526648 damage 0x4000 calib 1 event 03 (global 03) <3> 18:38:49.980462413 fid 0x0d1ac vec 2187 L1Accept extent 0x526648 damage 0x4000 calib 1 event 04 (global 04) ...
Now if we want to use the indexing functions for this run, first we make sure if the index files (*.xtc.idx) exist,
either in the same directory as the xtc files, or in the "index" subdirectory. If yes, myana program will
automatically use the index files.
We can do random event seeking insiden this run by using -j XXX command line option:
event jumping in an xtc run
[tomytsai@psana101 teste60]$ myana_index -l e60-r0329.lst -j 200000 Will jump to Event# 200000 (After Calib Cycle) Opening ROOT output file e60-r0329.root Opening filelist e60-r0329.lst Analyzing files e60-r0329 [16] <0> PST Sun 2011-02-20 06:18:51.712917944 fid 0x1ffff vec 000 Configure extent 0x4cf28 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.810424731 fid 0x1ffff vec 000 BeginRun extent 0xa0 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.943730116 fid 0x1ffff vec 000 BeginCalibCycle extent 0x178 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.944620324 fid 0x1ffff vec 000 Enable extent 0xa0 damage 0x0 Jumped to Calib# 1 Event# 49982 in e60-r0329-s00-c02.xtc Jumped to Calib# 1 Event# 50058 in e60-r0329-s01-c02.xtc Jumped to Calib# 1 Event# 49982 in e60-r0329-s02-c02.xtc Jumped to Calib# 1 Event# 49981 in e60-r0329-s03-c02.xtc XtcRun::jump(): Jumped to Calib# 1 Event# 200000 (Global# 200000) <3> 19:06:52.539431709 fid 0x01062 vec 5499 L1Accept extent 0x524afc damage 0x4000 calib 1 event 200000 (global 200000) <0> 19:06:52.547765176 fid 0x01065 vec 5500 L1Accept extent 0x524afc damage 0x4000 calib 1 event 200001 (global 200001) <2> 19:06:52.556099157 fid 0x01068 vec 5501 L1Accept extent 0x524afc damage 0x4000 calib 1 event 200002 (global 200002) <1> 19:06:52.564435184 fid 0x0106b vec 5502 L1Accept extent 0x524afc damage 0x4000 calib 1 event 200003 (global 200003) ...
We can see the first processed event is the 200,000th event in this xtc run, and then the following events will be
processed until the end of run.
xtc run with multiple BeginCalibCycles (scans)
In addition to -j XXX for event jumping, we can use -y XXX to seek to any BeginCalibCycle inside an xtc run. We will
use an xtc run with 20 BeginCalibCycles and 620 events as example. The xtc run is composed of 20 xtc files, so again we
will need to create a file list of all xtc filenames, and then use -y XXX -j XXX command line option to jump to any
BeginCalibCyle and any event in this xtc run.
jump to a BeginCalibCyle inside an xtc run
[tomytsai@psana101 teste0]$ myana_index -l e0-r0000A.lst -y 15 -j 10 Will jump to Calib# 15 Will jump to Event# 10 (After Calib Cycle) Opening ROOT output file e0-r0000A.root Opening filelist e0-r0000A.lst Analyzing files e0-r0000 [20] <0> PDT Thu 2011-05-12 18:15:43.867675517 fid 0x1ffff vec 000 Configure extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:15:48.252447044 fid 0x1ffff vec 000 BeginRun extent 0x28 damage 0x0 Jumped to Calib# 15 Event# 0 in e0-r0000-s00-c03.xtc Jumped to Calib# 15 Event# 0 in e0-r0000-s01-c03.xtc Jumped to Calib# 15 Event# 0 in e0-r0000-s02-c03.xtc Jumped to Calib# 15 Event# 0 in e0-r0000-s03-c03.xtc <0> PDT Thu 2011-05-12 18:16:00.271893141 fid 0x1ffff vec 000 BeginCalibCycle extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:16:00.272738903 fid 0x1ffff vec 000 Enable extent 0x28 damage 0x0 Jumped to Calib# 15 Event# 3 in e0-r0000-s00-c03.xtc Jumped to Calib# 15 Event# 3 in e0-r0000-s01-c03.xtc Jumped to Calib# 15 Event# 3 in e0-r0000-s02-c03.xtc Jumped to Calib# 15 Event# 4 in e0-r0000-s03-c03.xtc XtcRun::jump(): Jumped to Calib# 15 Event# 10 (Global# 444) <0> 18:16:00.358060639 fid 0x043cb vec 443 L1Accept extent 0x4c damage 0x0 calib 15 event 10 (global 444) <1> 18:16:00.366394478 fid 0x043ce vec 444 L1Accept extent 0x4c damage 0x0 calib 15 event 11 (global 445) <2> 18:16:00.374727133 fid 0x043d1 vec 445 L1Accept extent 0x4c damage 0x0 calib 15 event 12 (global 446) ...
Time searching inside an xtc run
As usual, we can use -t "timestamp" command option to go to the first event after the specified timestamp:
Time searching inside an xtc run
[tomytsai@psana101 teste60]$ myana_index -l e60-r0329.lst -t "2011-02-20 18:52:45.333" Opening ROOT output file e60-r0329.root Opening filelist e60-r0329.lst Analyzing files e60-r0329 [16] Time PST Sun 2011-02-20 18:52:45.333s (seconds 0x4d61d37d nanosecs 0x13d92d40) Going to event with time 2011-02-20 18:52:45.333 Calib# 1 Event# 100040 <0> PST Sun 2011-02-20 06:18:51.712917944 fid 0x1ffff vec 000 Configure extent 0x4cf28 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.810424731 fid 0x1ffff vec 000 BeginRun extent 0xa0 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.943730116 fid 0x1ffff vec 000 BeginCalibCycle extent 0x178 damage 0x0 <0> PST Sun 2011-02-20 18:38:49.944620324 fid 0x1ffff vec 000 Enable extent 0xa0 damage 0x0 Jumped to Calib# 1 Event# 24992 in e60-r0329-s00-c01.xtc Jumped to Calib# 1 Event# 25068 in e60-r0329-s01-c01.xtc Jumped to Calib# 1 Event# 24992 in e60-r0329-s02-c01.xtc Jumped to Calib# 1 Event# 24991 in e60-r0329-s03-c01.xtc XtcRun::jump(): Jumped to Calib# 1 Event# 100040 (Global# 100040) <3> 18:52:45.338877960 fid 0x168b4 vec 3843 L1Accept extent 0x524afc damage 0x4000 calib 1 event 100040 (global 100040) <0> 18:52:45.347213508 fid 0x168b7 vec 3844 L1Accept extent 0x524afc damage 0x4000 calib 1 event 100041 (global 100041) <2> 18:52:45.355543089 fid 0x168ba vec 3845 L1Accept extent 0x524afc damage 0x4000 calib 1 event 100042 (global 100042) ...
We see from above that the first processed event is event number 100,040 with timestamp "18:52:45.338877960", which is
just a bit later than our specified timestamp "2011-02-20 18:52:45.333".
Processing only selected events
The above indexing functions let us to jump to some specified event inside an xtc run, and then process
all of following events until the end of an xtc file or an xtc run. If we can filter out the events of interests,
and only want to process these interesting events, we can create a event list file that contains the list of event
numbers, and do the following steps:
In the example below, we want to process only events 2, 4, 6, 8 and 10, so we create a list of event numbers:
process selected events
[tomytsai@psana101 teste0]$ cat e0-r0000a.evts 2 4 6 8 10 [tomytsai@psana101 teste0]$ myana_index -l e0-r0000A.lst -e e0-r0000a.evts Opening ROOT output file e0-r0000A.root Read 5 event numbers from event list file e0-r0000a.evts Opening filelist e0-r0000A.lst Analyzing files e0-r0000 [20] <0> PDT Thu 2011-05-12 18:15:43.867675517 fid 0x1ffff vec 000 Configure extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:15:48.252447044 fid 0x1ffff vec 000 BeginRun extent 0x28 damage 0x0 <0> PDT Thu 2011-05-12 18:15:48.461648725 fid 0x1ffff vec 000 BeginCalibCycle extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:15:48.462616726 fid 0x1ffff vec 000 Enable extent 0x28 damage 0x0 ... XtcRun::jump(): Jumped to Calib# 1 Event# 2 (Global# 2) <2> 18:15:48.482970573 fid 0x03318 vec 001 L1Accept extent 0x4c damage 0x0 calib 1 event 02 (global 02) ... XtcRun::jump(): Jumped to Calib# 1 Event# 4 (Global# 4) <0> 18:15:48.499637351 fid 0x0331e vec 003 L1Accept extent 0x4c damage 0x0 calib 1 event 04 (global 04) ... XtcRun::jump(): Jumped to Calib# 1 Event# 6 (Global# 6) <2> 18:15:48.516304517 fid 0x03324 vec 005 L1Accept extent 0x4c damage 0x0 calib 1 event 06 (global 06) ... XtcRun::jump(): Jumped to Calib# 1 Event# 8 (Global# 8) <0> 18:15:48.532970785 fid 0x0332a vec 007 L1Accept extent 0x4c damage 0x0 calib 1 event 08 (global 08) ... XtcRun::jump(): Jumped to Calib# 1 Event# 10 (Global# 10) <2> 18:15:48.549637939 fid 0x03330 vec 009 L1Accept extent 0x4c damage 0x0 calib 1 event 10 (global 10) User analysis endrun() routine called. Processed 5 events, 0 damaged, with damage mask 0x0. User analysis endjob() routine called.
We see that after processing all specified events in our event list file, myana program stop the running
and call endrun() and endjob().
Alternative event list format
In the event list file, we can use commas (,) or spaces to separate two event numbers, instead using new lines:
alternative event number formats
[tomytsai@psana101 teste0]$ cat e0-r0000b.evts 2,4,6,8,10 [tomytsai@psana101 teste0]$ cat e0-r0000c.evts 2 4 6 8 10
The above two event list files have the same effect as the previous example, when we use the -e <event list file>
to feed them into a myana program. All three of them will make myana program processes only events 2, 4, 6, 8 and 10.
Processing events with different BeginCalibCyles
In the event list file, we can use C<XXX> to specify the BeginCalibCycle number. This is equivalent to the
-y XXX command line option.
process events in different BeginCalibCycles
[tomytsai@psana101 teste0]$ cat e0-r0000d.evts C10 2 4 C5 4 2 [tomytsai@psana101 teste0]$ myana_index -l e0-r0000A.lst -e e0-r0000d.evts ... <0> PDT Thu 2011-05-12 18:15:43.867675517 fid 0x1ffff vec 000 Configure extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:15:48.252447044 fid 0x1ffff vec 000 BeginRun extent 0x28 damage 0x0 ... <0> PDT Thu 2011-05-12 18:15:57.221828870 fid 0x1ffff vec 000 BeginCalibCycle extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:15:57.222654806 fid 0x1ffff vec 000 Enable extent 0x28 damage 0x0 ... <1> 18:15:57.241370668 fid 0x03f69 vec 280 L1Accept extent 0x4c damage 0x0 calib 10 event 02 (global 281) ... <3> 18:15:57.258037008 fid 0x03f6f vec 282 L1Accept extent 0x4c damage 0x0 calib 10 event 04 (global 283) ... <0> PDT Thu 2011-05-12 18:15:53.091743117 fid 0x1ffff vec 000 BeginCalibCycle extent 0xbc damage 0x0 ... <0> 18:15:53.683009788 fid 0x03a68 vec 127 L1Accept extent 0x4c damage 0x0 calib 5 event 04 (global 128) ... <2> 18:15:53.666343643 fid 0x03a62 vec 125 L1Accept extent 0x4c damage 0x0 calib 5 event 02 (global 126) User analysis endrun() routine called. Processed 4 events, 0 damaged, with damage mask 0x0. User analysis endjob() routine called.
In the above example, we see the myana program first processed BeginCalibCycle 10 event 2 and 4, and then
processed BeginCalibCycle 5 event 4 and 2.
Processing events with timestamps
In the event list file, we can use "timestamp" to specify an event, instead of giving the event number.
In the following example, we want to process only two events, one with timestamp "2011-05-12 18:16:07.05 PDT",
and the other with timestamp "2011-05-12 18:15:49:40 PDT". Note that myana will find the earliest event
that occured after the specified timestamp.
process only events with timestamps
[tomytsai@psana101 teste0]$ cat e0-r0000e.evts "2011-05-12 18:16:07.05 PDT" "2011-05-12 18:15:49:40 PDT" [tomytsai@psana101 teste0]$ myana_index -l e0-r0000A.lst -e e0-r0000e.evts ... Time PDT Thu 2011-05-12 18:16:07.050s (adjusted by DST) (seconds 0x4dcc8657 nanosecs 0x2faf080) Time 2011-05-12 18:16:07.05 PDT is mapped to Calib# 19 Event# 11 Time PDT Thu 2011-05-12 18:15:49.000s (adjusted by DST) (seconds 0x4dcc8645 nanosecs 0x0) Time 2011-05-12 18:15:49:40 PDT is mapped to Calib# 2 Event# 1 <0> PDT Thu 2011-05-12 18:15:43.867675517 fid 0x1ffff vec 000 Configure extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:15:48.252447044 fid 0x1ffff vec 000 BeginRun extent 0x28 damage 0x0 ... <0> PDT Thu 2011-05-12 18:16:06.932029980 fid 0x1ffff vec 000 BeginCalibCycle extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:16:06.962664648 fid 0x1ffff vec 000 Enable extent 0x28 damage 0x0 ... <1> 18:16:07.058111706 fid 0x04d37 vec 568 L1Accept extent 0x4c damage 0x0 calib 19 event 11 (global 569) ... <0> PDT Thu 2011-05-12 18:15:49.071660298 fid 0x1ffff vec 000 BeginCalibCycle extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:15:49.149732095 fid 0x1ffff vec 000 Enable extent 0x28 damage 0x0 <0> 18:15:49.241309463 fid 0x03429 vec 031 L1Accept extent 0x4c damage 0x0 calib 2 event 01 (global 32) User analysis endrun() routine called. Processed 2 events, 0 damaged, with damage mask 0x0. User analysis endjob() routine called.
Processing events within a time range
We can use "timestamp1"-"timestamp2" in the event list file to let myana process all events
occuring between timestamp1 and timestamp2.
In the following exmaple, we want to process all events between "2011-05-12 18:16:07.21 PDT" and
"2011-05-12 18:16:07.59 PDT". Note that the last event may be slightly later than the
specified timestamp.
process events within a time range
[tomytsai@psana101 teste0]$ cat e0-r0000f.evts "2011-05-12 18:16:07.21 PDT"-"2011-05-12 18:16:07.59 PDT" [tomytsai@psana101 teste0]$ myana_index -l e0-r0000A.lst -e e0-r0000f.evts ... <0> 18:16:07.216445776 fid 0x04d70 vec 587 L1Accept extent 0x4c damage 0x0 calib 19 event 30 (global 588) ... <1> 18:16:07.216683687 fid 0x04d73 vec 588 L1Accept extent 0x4c damage 0x4000 calib 19 event 31 (global 589) ... <0> PDT Thu 2011-05-12 18:16:07.572043109 fid 0x1ffff vec 000 BeginCalibCycle extent 0xbc damage 0x0 <0> PDT Thu 2011-05-12 18:16:07.572981253 fid 0x1ffff vec 000 Enable extent 0x28 damage 0x0 <2> 18:16:07.583115846 fid 0x04df4 vec 589 L1Accept extent 0x4c damage 0x0 calib 20 event 01 (global 590) ... <3> 18:16:07.591449254 fid 0x04df7 vec 590 L1Accept extent 0x4c damage 0x0 calib 20 event 02 (global 591) User analysis endrun() routine called. Processed 4 events, 1 damaged, with damage mask 0x4000. User analysis endjob() routine called.
Here we see the first processed event has the timestamp slightly later than our specified input timestamp
"2011-05-12 18:16:07.21 PDT", and the last processed event has timestamp slightly later than the second
input timestamp "2011-05-12 18:16:07.59 PDT".
Usage of myana program
The command line options for myana programs can be listed by running myana -h, as follows:
myana usage
[tomytsai@psana101 teste0]$ myana_index -h
Usage: myana_index -f <filename> | -l <filewithfilelist>
[-h]
[-c <caliblist>] [-s <skipevts>]
[-n <maxevts>]
[-O <reorder file>]
[-S] (split event recovery)
[-L] (live file read)
[-d <debug level>]
[-j <jumpToEvent>]
[-y <jumpToCalibCycle>]
[-t <jumpToTime>]
[-e <eventListFile>]
* The -l and -c arguments require files with a list of files in them.
* The -e argument requires a list of events. The file format is as follows:
<event1> <event2> # Events are separated by space, ',' or newlines
<event1>-<event2> # Include all events between <event1> and <event2>
C<calibCycle#> <event1> <event2> # Move to some CalibCycle, and read specified events
In this page, we have used -j XXX for random event seeking, -y XXX for BeginCalibCycle seeking,
-l <xtc list> for reading a xtc run, and -e <event list> to read a list interesting events.
Future extendibility
So far we have shown how to use xtc index files to do random seeking and time searching. The xtc index file
is acutally much more powerful then the above functions. Basically the xtc index stores the following information
for each shot:
- timestamp
- fiducial
- EVR event codes
- DAQ damage value (which is a summary of all detectors' damage values)
- Which detector has damages in this shot
- Which detector doesn't send out data in this shot (for exmaple, epics doesn't send out data for each shot)
Hence we can see there is some potential for using the above information to do some fast indexing and filtering.
Please contact Tomy (tomytsai at slac.stanford.edu) if you have new ideas about indexing functions.
Overview
Content Tools