Page History

This page provides description of the command-line tools that interact with HDF5 translation service. More user-friendly interface to translation service is implemented inside LCLS Web Portal, but its functionality is more limited.

Account Setup

To start using command-line tools one has to set the environment for standard (as opposed to analysis) offline releases. This is done by executing a single command from the shell:

(ba)sh: . /reg/g/psdm/etc/sit_env.sh
(t)csh: source /reg/g/psdm/etc/sit_env.csh

or adding one of these commands to the user login script (.bashrc or .login).

If user environment is already setup for analysis releases following the Analysis Workbook procedure then one can switch to using regular off-line release with command:

sit_setup current

The commands described below could be run from any machine which has access to off-line releases (directory /reg/g/psdm).

Checking status of translator system

One can check the sate of the various parts of the translation system using the command:

ic-ws-status -s

Which should produce output like:

    ID  Host                              Status    Time                    PID  Instruments
--------------------------------------------------------------------------------------------
   509  psexport02.slac.stanford.edu      Running   2011-12-08 16:36:21   29279  

Normally the hosts psexport02 should be in a running state. If no hosts are listed by this command then the translation will not happen.

Checking status of translation

To see the list of runs and their translation status one have to pass two parameters to ic-ws-status command – instrument name and experiment name. Here is example of the command and its output:

% ic-ws-status AMO amo14410
   Run  Status                 ReqID     JobID  Prio  Req Time             Files
  --------------------------------------------------------------------------------------
     1  FAIL                    8543         0     0  2010-06-07 08:04:23  /reg/d/psdm/AMO/amo14410/xtc/e23-r0001-s01-c00.xtc
                                                                           /reg/d/psdm/AMO/amo14410/xtc/e23-r0001-s00-c00.xtc
     2  DONE                    8545         0     0  2010-06-07 09:27:02  /reg/d/psdm/AMO/amo14410/xtc/e23-r0002-s00-c00.xtc
                                                                           /reg/d/psdm/AMO/amo14410/xtc/e23-r0002-s01-c00.xtc
     3  DONE                    8546         0     0  2010-06-07 09:27:02  /reg/d/psdm/AMO/amo14410/xtc/e23-r0003-s01-c00.xtc
                                                                           /reg/d/psdm/AMO/amo14410/xtc/e23-r0003-s00-c00.xtc

The output shows the run number, status of the translation process, request ID, request time and list of XTC files. Probably the most important here is the status column which can display one of the following values:

• WAIT_FILES – New request, waiting for translation, file list will be empty.
• WAIT – New request, waiting for translation, file list will be non-empty.
• PENDING – Translator job has been submitted to batch queue and is waiting for execution.
• RUN – Translator process is running.
• SUSPENDED – Translator job was suspended by batch system.
• DONE – Successfully translated.
• FAIL_NOINPUT – No XTC files have been found for the run.
• FAIL_MKDIR – Failure to create temporary directory for output files.
• FAIL_COPY – Translation finished, but HDF5 files could not be copied to destination directory.
• FAIL – Translation process failed, usually means input data is bad.

Translating new runs

Important: before submitting translation request for a new run one has to make sure that all XTC files from that run have been copied to off-line data space at /reg/d/psdm/...

One more command ic-ws-translate sends requests to the system to translate data for the new runs. It takes three parameters:

1. Instrument name
2. Experiment name
3. List of runs

List of runs is defined as a comma-separated list of run ranges. Run range is defined as a single run number or two run number separated by dash. To make it simple here are examples of the valid lists of runs:

• 1 – single run #1
• 1,2,3,4,5 – five runs from #1 to #5
• 1-5 – same as above
• 1-5,8,11-15,18 – twelve runs: #1 to #5, #8, #11 to #15, and #18

Here is an example of submitting request for translation of several runs:

% ic-ws-translate AMO amo14410 105-108
Run 105 OK (id=8919, prio=0)
Run 106 OK (id=8920, prio=0)
Run 107 OK (id=8921, prio=0)
Run 108 OK (id=8922, prio=0)

For every submitted request it will print one-line summary. If there is an error which happened during submission the error message will be printed for that run like in this example:

% ic-ws-translate AMO amo14410 108-110
Run 108 failed: fileset already exists: instr=AMO exper=amo14410 run=108
Run 109 OK (id=8927, prio=0)
Run 110 OK (id=8928, prio=0)

In this case the script also returns non-zero return code to the shell which can be checked.

Right after successful submission the status of the new request becomes WAIT_FILES and will have empty file list. It's only when translation process start the file list will be populated with the files from the corresponding run.

One can also specify request priority when submitting translation requests using -w option:

% ic-ws-translate -w 60 AMO amo14410 109
Run 109 OK (id=8932, prio=10)

Requests with higher priorities will be scheduled for translation sooner than lower priority requests. Default priority value is 50, allowed priority range is from 1 to 100.

Re-translating runs

By default ic-ws-translate command does not allow repeated translation requests for the same run. Sometimes one needs to correct an error in the previous translation attempt after fixing the origin of the problem (like waiting until files have been copied). In this case -r option allows to re-submit the request:

% ic-ws-translate -r AMO amo14410 108
Run 108 OK (id=8931, prio=1)

If the output HDF5 already exist when re-translation request is submitted then a backup copy will be created for existing file which has an additional suffix "-TIMESTAMP.bck" where TIMESTAMP is the string reflecting the time when backup was created (something like amo14410-r0108.h5-20101208T154326.bck). Backup will be made only when re-translation job finishes and moves new file to a final destination, if re-translation fails the original file will remain in its place.