noric13:dflath> /afs/slac/g/glast/ground/bin/datacat -h find
Command-specific help for command find
...
options:
--recurse Recurse sub-folders
--search-folders Search for datasets inside folder(s)
--search-groups Search in groups. This option is superseded by the -G (--group) option if they are both supplied.
--group <group name> Dataset Group under which to search for datasets.
--site <site name> Name of Site to search. May be used multiple times to specify a list of sites in which case order is taken as preference. Defaults to the Master-location if not provided.
--filter <filter expression> Criteria by which to filter datasets. ie: 'DatasetDataType=="MERIT" && nMetStart>=257731220 && nMetStop <=257731580'
--display <meta name> Name of meta-data field to display in output. Default is to display only the file location. May be used multiple times to specify an ordered list of fields to display.
--sort <meta name> Name of meta-data field to sort on. May be used multiple times to specify a list of fields to sort on. Order determines precedence.
--show-unscanned-locations If no "OK" (ie: verified by file crawler) location exists, display first location (if any) which has not yet been scanned. If this option and '--show-non-ok-locations' are both specified, an unscanned location will be returned before a non-ok location regardless of their sequence in the ordered site list.
--show-non-ok-locations If no "OK" (ie: verified by file crawler) location exists, display first location (if any) which exists in the list of sites.
...
Argument | Explanation |
---|---|
<logical folder> | This parameter is required and comes after all options are specified. Replace it with the Data Catalog folder path where you want to begin your search. |
--recurse | If you specify this option, the find command will traverse the entire folder tree under <logical folder> searching for datasets that meet your criteria. |
--search-folders | Tells find to look inside the folder (or folders if --recurse specified) and consider datasets that live there. In general, datasets live in groups, and this option is not used. |
--search-groups | Tells find to look inside all dataset groups in the specified folder (or folder tree if using --recurse) for your files. May be combined with --search-folders and --recurse. |
--group <group name> | Tells find to look inside groups only if they have the name specified by <group name>. May be used with --recurse to search in groups of the given name in a folder tree. |
--site <site name> | Specifies a specific site you want to get datasets from. May be used multiple times to specify a list of sites to search where order indicates preference. |
--filter <filter expression> | Also known as "search criteria" is an expression using logical operators, meta-data fields and constant values on which to filter the output results. See below for details. |
--display <meta name> | Causes the meta-data value associated with 'meta name' to be displayed in the output. May be used multiple times. Columns are tab-separated. |
--sort <meta name> | Specifies the name of the specific meta-data field on which to sort the output results. May be used multiple times to specify a list of fields to sort on where order indicates preference. |
--show-unscanned-locations | If a verified disk location can not be found in the specified site-list, the first location (site-preference order) which has not been scanned yet will be returned. |
--show-non-ok-locations | Similar to --show-unscanned-locations, but will return the first location in the ordered site list that has a disk location regardless of the file scan-status. The file may be missing, 'bad', or otherwise. (Caveat emptor.) If you specify this option in addition to the --show-unscanned-locations option. An unscanned location will be returned before a non-ok location if both exist. |
*Important note: At least one of --search-folders, --search-groups, --group <group name> must be specified.
SLAC_XROOT | XROOT servers at SLAC. Almost everything lives here. |
SLAC | NFS (or AFS) at SLAC. Some FT1 and FT2 data are stored here until the ftools learn to read from XROOT. |
IN2P3 | Some Monte Carlo data is stored at Lyon |
IN2P3_HPSS | Lyon Monte Carlo backups |
UW | University of Washington. |
An expression composed of logical, arithmetic, and comparison operators along with meta-data fields used to select datasets that meet specific criteria.
Panel | ||
---|---|---|
| ||
Expr ::= Expr |
Name | Type | Description |
---|---|---|
Name | String | Dataset Name |
FileFormat | String | File encoding. ex: "root", "fits" |
DataType | String | Type of data in file. Always uppercase. ex: "RECON" |
VersionID | Integer | Version of the Dataset this file represents. |
CreateDate | Timestamp | Date this Version of the Dataset was created. |
Source | String | What created this Version of the Dataset. ex: "PIPELINE", "LINEMODE CLIENT" |
TaskName | String | If Source=="PIPELINE" this will contain the name of the Task which created this Version of the Dataset. |
RunMin | Long Integer | Smallest Run Identifier found in this file, if applicable. |
RunMax | Long Integer | Largest Run Identifier found in this file, if applicable. |
NumberEvents | Long Integer | Number of events in the file, if applicable. |
FileSizeBytes | Long Integer | Size of this file on disk, in bytes. |
RootVersion | String | If FileFormat=="root", the version of root which wrote this file. |
SOLibVersion | String | If FileFormat=="root", the version of the shared object library that the events correspond to, if applicable. |
TTreeName | String | If FileFormat=="root", the name of the first TTree in the file, if one exists. |
(Feel free to fill in the description field for those you are responsible for.)
Name | Type | Description |
---|---|---|
sDatasource | STRING |
|
nMetStop | NUMBER |
|
nMetStart | NUMBER |
|
sOrigFilename | STRING |
|
nOrigBytes | NUMBER |
|
nOrigCkSum | NUMBER |
|
sBTRversion | STRING |
|
sPhysList | STRING |
|
nBtRunId | NUMBER |
|
sDataSource | STRING |
|
nDownlink | NUMBER |
|
nRun | NUMBER |
|
sRunStatus | STRING |
|
sCreator | STRING |
|
sIntent | STRING |
|
nMootKey | NUMBER |
|
type | STRING |
|
packetTime | STRING |
|
packetApid | STRING |
|
startAddress | STRING |
|
functionCode | STRING |
|
stopAddress | STRING |
|
transactionId | STRING |
|
tstop | STRING |
|
tstart | STRING |
|
nMootKey | STRING |
|
startedAt | STRING |
|
firstTimeStamp | STRING |
|
counterType | STRING |
|
lastTimeStamp | STRING |
|
nDatasetId | NUMBER |
|
TCut | STRING |
|
noric13:dflath> /afs/slac/noric15:dflath> datacat find --site SLAC --group FT2 /Data/Flight/Level1/LPA/
/nfs/farm/g/glast/ground/bin/datacat find --filter 'RunMin>=236191699 && RunMax<=236211846' --sort nMetStart --group FT1 /Data/Flight/Level1/LPA/
root://glast-rdr.slac.stanford.edu/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5857/ft2ft1/gll_ptph_r0236511638r0236191699_v003v002.fit
/nfs/farm/g/glast/u20/FT1-2copies root://glast-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/prod/1.57/ft2ft1/gll_ptph_r0236339577r0236197643_v000v001.fit
root:/nfs/farm/g/glast/u20/FT1-2copies/glast-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/prod/1.57/ft2ft1/gll_ptph_r0236345681r0236198321_v000v001.fit
root:/nfs/farm/g/glast/u20/FT1-2copies/glast-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/prod/1.5756/ft2ft1/gll_ptph_r0236409925r0236209517_v001.fit
root:/nfs/farm/g/glast/u20/FT1-2copies/glast-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/prod/1.5856/ft2ft1/gll_ptph_r0236443723r0236211846_v002v001.fit /nfs/farm
noric13:dflath> /afs/slac/g/glast/u20/FT1-2copies/glast/ground/bin/datacat find --filter 'RunMin>=236191699 && RunMax<=236211846' --sort nMetStart --group FT1 --site SLAC /Data/Flight/Level1/LPA/
prod/1.57/ft2/gll_pt_r0236271733_v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5657/ft2ft1/gll_ptph_r0236090205r0236191699_v001v002.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.57/ft2ft1/gll_ptph_r0236351742r0236197643_v002v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.57/ft2ft1/gll_ptph_r0236387337r0236198321_v002v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5756/ft2ft1/gll_ptph_r0236437748r0236209517_v000v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5856/ft2ft1/gll_ptph_r0236505503r0236211846_v000v001.fit /nfs/farm
noric13:dflath> /afs/slac/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.58/ft2/gll_pt_r0236398466_v003.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.58/ft2/gll_pt_r0236523746_v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.57/ft2/gll_pt_r0236455683_v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.58/ft2/gll_pt_r0236501595_v003.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.58/ft2/gll_pt_r0236517724_v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glastground/bin/datacat find --filter '(DataType=="FT1" || DataType=="FT2") && RunMin>=236191699 && RunMax<=236211846' --sort Name --sort DataType --search-groups --site SLAC /Data/Flight/Level1/LPA/prod/1.56/ft2/gll_pt_r0236096298_v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.57/ft2ft1/gll_ptph_r0236375554r0236191699_v002.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.57/ft2/gll_pt_r0236431706r0236191699_v001v002.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5657/ft2ft1/gll_ptph_r0236121367r0236197643_v000v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5657/ft2/gll_pt_r0236126697r0236197643_v000v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.57/ft2ft1/gll_ptph_r0236357728r0236198321_v002v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5657/ft2/gll_pt_r0236108455r0236198321_v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.56/ft2ft1/gll_ptph_r0236295990r0236209517_v000v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.56/ft2/gll_pt_r0236227863r0236209517_v000v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5756/ft2ft1/gll_ptph_r0236425612r0236211846_v000v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.56/ft2/gll_pt_r0236232904r0236211846_v001.fit
noric15:dflath> datacat find --site SLAC --group FT2 v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast /Data/Flight/Level1/LPA/prod/1.56/ft2/gll_pt_r0236230384_v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.58/ft2/gll_pt_r0236484407r0236511638_v002v003.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5857/ft2/gll_pt_r0236570342r0236339577_v001v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5657/ft2/gll_pt_r0236223184r0236345681_v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5657/ft2/gll_pt_r0236233264r0236409925_v000v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5658/ft2/gll_pt_r0236229303r0236443723_v000v002.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.5657/ft2/gll_pt_r0236222464r0236271733_v000.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.56/ft2/gll_pt_r0236211846r0236090205_v001.fit
/nfs/farm/g/glast/u20/FT1-2copies/glast/Data/Flight/Level1/LPA/prod/1.57/ft2/gll_pt_r0236363676r0236351742_v001v002.fit
... etc ...