It may be advantageous or necessary to do data analysis without a connection to SLAC. Below are the steps to install psana along with the data to be used in the analysis onto a local machine for later, offline use on a RHEL 5, 6 or 7 machine. In the simplest case, a single conda environment is maintained but multiple conda environments can be created and used if necessary. The steps for both cases are shown below.
Add the following script to it. This checks for the .bashrc and will run it on start up
#!/bin/bash if [ -f $HOME/.bashrc ]; then source $HOME/.bashrc fi |
Run conda update -y conda to update miniconda
(optional, since we try to have an up-to-date copy of this in conda) Copy the experiment database from /reg/g/psdm/data/ExpNameDb/experiment-db.dat
Make a directory in the home directory with mkdir -p psdm/data/ExpNameDb
Copy the database
With rsync: rsync -t psexport:/reg/g/psdm/data/ExpNameDb/experiment-db.dat ~psdm/data/ExpNameDb/
With SCP: scp -p psexport:/reg/g/psdm/data/ExpNameDb/experiment-db.dat ~psdm/data/ExpNameDb/
Copy the experiment data that will be used for analysis. This step requires patience if many runs will be copied
From /reg/d/psdm/xpp/xpptut15/xtc to ~/psdm/xpp/xpptut15/xtc:
-bash-4.2$ ls /reg/d/psdm/xpp/xpptut15/xtc/ | grep 54 e665-r0054-s00-c00.xtc e665-r0054-s01-c00.xtc e665-r0054-s02-c00.xtc e665-r0054-s03-c00.xtc e665-r0054-s04-c00.xtc e665-r0054-s05-c00.xtc |
From /reg/d/psdm/xpp/xpptut15/xtc/index to ~/psdm/xpp/xpptut15/xtc/index:
-bash-4.2$ ls /reg/d/psdm/xpp/xpptut15/xtc/index | grep 54 e665-r0054-s00-c00.xtc.idx e665-r0054-s01-c00.xtc.idx e665-r0054-s02-c00.xtc.idx e665-r0054-s03-c00.xtc.idx e665-r0054-s04-c00.xtc.idx e665-r0054-s05-c00.xtc.idx |
From /reg/d/psdm/xpp/xpptut15/xtc/smalldata to ~/psdm/xpp/xpptut15/xtc/smalldata:
-bash-4.2$ ls /reg/d/psdm/xpp/xpptut15/xtc/smalldata | grep 54 e665-r0054-s00-c00.smd.xtc e665-r0054-s01-c00.smd.xtc e665-r0054-s02-c00.smd.xtc e665-r0054-s03-c00.smd.xtc e665-r0054-s04-c00.smd.xtc e665-r0054-s05-c00.smd.xtc |
Now psana conda can be used. If new packages need to be added, removed, etc., the simple conda comands can be used. A link the specifics of these commands is given in the section below.
More information on managing Conda environments can be found here. Begin by following the steps above for the single environment but do not install psana conda yet since it will be installed when the new environment is created (step 8) or set the environment variables (step 10). It is recommended to have these variables set when the environment is activated and unset when deactivated. First create the environment with conda create --name <environment name> -c lcls-rhel<num> psana-conda where <environment name> is the name of the environment to be created and <num> is the RHEL version. Then follow the steps below, for a given environment examplenv, which were pulled from the link given before.
The complicated piece is that SIT_DATA must include the sub-directory 'data' to your Conda environment, as well as where the experiment-db.dat file is. To etc/conda/activate.d/env_vars.sh:
#!/bin/sh # Make sure to replace examplenv with the actual environment name export SIT_DATA=$HOME/miniconda2/envs/examplenv/data:$HOME/psdm/data export SIT_PSDM_DATA=$HOME/psdm |
To /etc/conda/deactivate.d/env_vars.sh:
#!/bin/sh unset SIT_DATA unset SIT_PSDM_DATA |
Now examplenv will have the correct environment variables when activated with the command source activate examplenv and deactivate with source deactivate. And packages may be added, removed, upgraded, downgraded, etc. from this conda environment like usual. For information on this, follow link given at the beginning of this section.
To copy files from one machine to another that aren't huge (i.e. hundreds of GBs), rsync and SCP are good choices. Their basic use is like so:
With SCP: scp -rp <machine copying from>:<file/directory name being copied> <directory copying to>. Like with rsync, the -r tag copies recursively and the -p tag has the same functionality as rsync's -t tag.
Below are several useful Conda commands that may have been stated above:
It's important to test whether or not the above steps actually worked. Using examples copied from the Building Blocks section will work. The peak finding algorithm specifically is a good example to use because it calls det.calib() and uses ImgAlgos. Below is the code copied from this example.
from ImgAlgos.PyAlgos import PyAlgos from psana import * ds = DataSource('exp=xpptut15:run=54:smd') det = Detector('cspad') alg = PyAlgos() alg.set_peak_selection_pars(npix_min=2, npix_max=50, amax_thr=10, atot_thr=20, son_min=5) hdr = '\nSeg Row Col Npix Amptot' fmt = '%3d %4d %4d %4d %8.1f' for nevent, evt in enumerate(ds.events()): if nevent >= 2: break nda = det.calib(evt) if nda is None: continue peaks = alg.peak_finder_v1(nda, thr_low=5, thr_high=21, radius=5, dr=0.05) print hdr for peak in peaks: seg, row, col, npix, amax, atot = peak[0:6] print fmt % (seg, row, col, npix, atot) |
The experiment name, run number and detector name should all be replace with whatever is being used. If this can run successfully, then psana conda has been installed completely and correctly.