Page History
This page provides description of the command-line tools that interact with HDF5 translation service. More user-friendly interface to translation service is implemented inside LCLS Web Portal, but its functionality is more limited.
Account Setup
To start using command-line tools
In the future there will be a Web UI interface for querying the status of the translator jobs and submitting new translation requests. Until future becomes a reality (and maybe after that) we provide a set of command-line tools for this tasks.
Account Setup
The tools are now part of the standard off-line releases (but not analysis releases) starting with release 0.2.9. To make the tools from release available one has to set the environment for standard (as opposed to analysis) offline releases. This is done by executing a single command from the shell:
...
No Format |
---|
ID Host Status Time PID Instruments -------------------------------------------------------------------------------------- 155 psdev.slac.stanford.edu Stopped 2010-06-28 10:37:29 6855 160 psanamvr101.slac.stanford.edu Stopped 2010-07-14 18:03:02 5271------ 163509 psanamvr102psexport02.slac.stanford.edu Running 20102011-0712-1508 1216:1536:0421 29279 27900 |
Normally the hosts psanamvr101 and psanamvr102 psexport02 should be in a running state. If all no hosts are stopped listed by this command then the translation will not happen.
...
No Format |
---|
% ic-ws-status AMO amo14410 Run Status ReqID JobID Prio Req Time Files -------------------------------------------------------------------------------------- 1 Translation_Error FAIL 8543 0 0 2010-06-07 08:04:23 /reg/d/psdm/AMO/amo14410/xtc/e23-r0001-s01-c00.xtc /reg/d/psdm/AMO/amo14410/xtc/e23-r0001-s00-c00.xtc 2 DONE Complete 8545 0 0 2010-06-07 09:27:02 /reg/d/psdm/AMO/amo14410/xtc/e23-r0002-s00-c00.xtc /reg/d/psdm/AMO/amo14410/xtc/e23-r0002-s01-c00.xtc 3 DONE Complete 8546 0 0 2010-06-07 09:27:02 /reg/d/psdm/AMO/amo14410/xtc/e23-r0003-s01-c00.xtc /reg/d/psdm/AMO/amo14410/xtc/e23-r0003-s00-c00.xtc |
The output shows the run number, status of the translation process, request ID, request time and list of XTC files. Probably the most important here is the status column which can display one of the following values:
InitialWAIT_EntryFILES
– New request, waiting for translation, file list will be empty.Waiting_Translation
WAIT
– New request, waiting for translation, file list will be non-empty..PENDING
– Translator job has been submitted to batch queue and is waiting for execution.RUN
– Translator process is running.SUSPENDED
– Translator job was suspended by batch system.DONE
– Successfully translated.FAIL_NOINPUT
Empty_Fileset
– No XTC files have been found for the run.H5DirFAIL_ErrorMKDIR
– Failure to create temporary directory for output files.BeingFAIL_Translated
– Translator process is running.Translation_Error
– Translation process failed, usually means input data is bad or repeated attempt to translate run which has been translated already.Archive_Error
–COPY
– Translation finished, but HDF5 files could not be copied to destination directory.Complete
– Successfully translatedFAIL
– Translation process failed, usually means input data is bad.
Translating new runs
Important: before submitting translation request for a new run one has to make sure that all XTC files from that run have been copied to off-line data space at /reg/d/psdm/...
...
Right after successful submission the status of the new request becomes InitialWAIT_EntryFILES
and will have empty file list. It's only when translation process start the file list will be populated with the files from the corresponding run.
One can also specify request priority when submitting translation requests using -w option:
No Format |
---|
% ic-ws-translate -w 60 AMO amo14410 109
Run 109 OK (id=8932, prio=10)
|
Requests with higher priorities will be scheduled for translation sooner than lower priority requests. Default priority value is 50, allowed priority range is from 1 to 100.
Re-translating runs
By default ic-ws-translate
command does not allow repeated translation requests for the same run. Sometimes one needs to correct an error in the previous translation attempt after fixing the origin of the problem (like waiting until files have been copied). In this case -r
option allows to re-submit the request:
No Format |
---|
% ic-ws-translate -r AMO amo14410 108 Run 108 OK (id=8931, prio=1) |
One can also specify request priority when submitting translation requests using -w option:
No Format |
---|
% ic-ws-translate -w 10 AMO amo14410 109
Run 109 OK (id=8932, prio=10)
|
Requests with higher priorities will be scheduled for translation sooner than lower priority requests. Default priority value is 0If the output HDF5 already exist when re-translation request is submitted then a backup copy will be created for existing file which has an additional suffix "-TIMESTAMP.bck" where TIMESTAMP is the string reflecting the time when backup was created (something like amo14410-r0108.h5-20101208T154326.bck
). Backup will be made only when re-translation job finishes and moves new file to a final destination, if re-translation fails the original file will remain in its place.