sdfdata is the analysis storage system that holds the LCLS experiment data. It is a a tier weka storage system that consists of a fast nvme  SSD layer (cache) and a Ceph Object-store base on hard disk. The  total usable size is about 10PB with 650 TB for the nvme layer and ~9PB object-store.

  • New data are always written to the SSD layer. 
  • After some time (~30min) new data are also copied to the Object-store. 
  • When the SSD layer is filling up old files (which are already in  the object store) will be purge from the SSD layer and only reside in the Object-store.
  • A user will always access the data from the SSD layer
  • If a user accesses data that are only in the object-store the data will be first copied to SSD layer first (if a user accesses only part of a file only this data will be copied but not the whole file. 
  • Access to data in the SSD layer is fast.
  • Copying data from Object-store to SSD is typically slow if user initiated.

Using the weka command for tiering information

The command weka fs tier allows to manage the tiering of a file:

Show tiering status: 

The command weka fs tier location shows how many bytes of a a file are in SSD and Object-store:

% cd /sdf/data/lcls/ds/rix/rixx1011723
% weka fs tier location  hdf5/smalldata/rixx1011723_Run0136*
PATH                                          FILE TYPE  FILE SIZE  CAPACITY IN SSD (WRITE-CACHE)  CAPACITY IN SSD (READ-CACHE)  CAPACITY IN OBJECT STORAGE  CAPACITY IN REMOTE STORAGE
hdf5/smalldata/rixx1011723_Run0136.h5         regular    3.53 MB    0 B                            3.53 MB                       3.53 MB                     0 B
hdf5/smalldata/rixx1011723_Run0136_part0.h5   regular    27.14 GB   0 B                            27.11 GB                      27.14 GB                    0 B
hdf5/smalldata/rixx1011723_Run0136_part1.h5   regular    27.07 GB   0 B                            25.88 GB                      27.07 GB                    0 B
hdf5/smalldata/rixx1011723_Run0136_part10.h5  regular    18.72 GB   0 B                            17.58 GB                      18.72 GB                    0 B

% weka fs tier location  hdf5/smalldata/rixx1011723_Run0137*
PATH                                          FILE TYPE  FILE SIZE  CAPACITY IN SSD (WRITE-CACHE)  CAPACITY IN SSD (READ-CACHE)  CAPACITY IN OBJECT STORAGE  CAPACITY IN REMOTE STORAGE
hdf5/smalldata/rixx1011723_Run0137.h5         regular    3.49 MB    0 B                            0 B                           3.49 MB                     0 B
hdf5/smalldata/rixx1011723_Run0137_part0.h5   regular    2.66 GB    0 B                            0 B                           2.66 GB                     0 B
hdf5/smalldata/rixx1011723_Run0137_part1.h5   regular    1.79 GB    0 B                            0 B                           1.79 GB

It shows the file size and how much data are in nvme (CAPACITY IN SSD) and object-store (CAPACITY IN OBJECT STORAGE). Notice that for run 136 most but not all data are in the nvme layer. For run 137 no data are in nvme. 

Copy from object-store to SSD

The weka fs tier fetch will copy files from the object store to the SSD cache. In general this operation is much faster compare to have the application (psana, jupyter reading hdf5) trigger the transfer. 


% cd /sdf/data/lcls/ds/cxi/
% weka fs tier location cxil1038423/xtc/cxil1038423-r0182-s03-c00.xtc
PATH                                           FILE TYPE  FILE SIZE  CAPACITY IN SSD (WRITE-CACHE)  CAPACITY IN SSD (READ-CACHE)  CAPACITY IN OBJECT STORAGE  CAPACITY IN REMOTE STORAGE
cxil1038423/xtc/cxil1038423-r0182-s03-c00.xtc  regular    38.61 GB   0 B                            0 B                           38.61 GB                    0 B

% weka fs tier location cxil1038423/xtc/cxil1038423-r0182-s03-c00.xtc

% weka fs tier location cxil1038423/xtc/cxil1038423-r0182-s03-c00.xtc
PATH                                           FILE TYPE  FILE SIZE  CAPACITY IN SSD (WRITE-CACHE)  CAPACITY IN SSD (READ-CACHE)  CAPACITY IN OBJECT STORAGE  CAPACITY IN REMOTE STORAGE
cxil1038423/xtc/cxil1038423-r0182-s03-c00.xtc  regular    38.61 GB   0 B                            38.61 GB                      38.61 GB                    0 B


Release file from SSD



  • No labels