You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Adding random access to LCIO files.

Basic idea.

Insert one or more special "index" records into the LCIO file. This contains pointers to the location of the events and other records within the file. The record can optionally also contain additional event tags, which allow events which satisfy certain criteria to be located.

Each index record contains:

#records in this index record
for each event
{
   record type
   run #
   event # (or 0 if this is not an event)  
   offset (4 or 8 bytes) 
} 
<possible additional tags>

There are two variations on this theme depending on how the records will be written.

Variation 1

Under this variation files can be written sequentially, with no random access required. This will enable files to be written over socket connections, or using other methods where random access is not available. Here a single "index location record" is written at the end of the file, with a pointer giving the offset from the end of the file of the index record. There are some disadvantages of this method:

  1. The information on the location of each record must be kept in memory and written to disk only when the file is finally closed. This could be a problem if the file contains a very large number of events.
  1. It will be difficult to append LCIO files, since the index records would have to be combined and moved to the end of the file.

Variation 2

In this variation the "index location record" is written at the beginning of the file, with a pointer to the first and last index record in the file. Each index record then contains a pointer to the next index record, except for the last index record which has 0 as its "next record pointer". When writing the file the space for each index record must be reserved in advance, writing continues with events beyond the index record, then when the index record is full, the next index record location is reserved, and the first index record is committed to the file. When the file is closed the pointers to the last index record is written into the file. Implementing this requires random access to the file, since we have to "backtrack" to commit the index files.

Disadvantages of this method are:

  1. The index record can not be compressed, since its size needs to be deterministic so the space can be reserved in advantage.
  2. Files cannot be written to non random access media, such as sockets (without some extra protocol in the socket stream).
  • No labels