Content
Introduction
This page covers documentation about the conda release and build system relevant for psana developers. This documentation is also for LCLS users that want to migrate their test releases based on psana C++ modules and the SConsTools scons build system to the conda environments. This page is up to date with our conversion to github, for the older deprecated svn page, see SVN based Psana Developer Documentation.
Creating Standard Python Package
Old to New Conda Command Table
Below we go over the steps to create, develop, build and manage test releases - comparing old commands from the RPM release system to the new conda based system.
Old commands like newrel, addpkg and relinfo are not available in the conda world. Most all steps are executed using the
condarel
program. Do condarel -h for the latest help on this script.
step | old | conda | notes |
---|---|---|---|
get started | source /reg/g/psdm/etc/ana_env.sh or source /reg/g/psdm/etc/ana_env.csh | source /reg/g/psdm/bin/conda_setup | Bash only for conda, no .csh After sourcing conda_setup, /reg/g/psdm/bin is removed from your PATH (if it was there) it is replaced with /reg/g/psdm/sw/conda/manage/bin |
create new test release directory | newrel ana-current myrel | condarel --newrel --name myrel | in conda, myrel is based on the current conda environment. The previous conda_setup command activated a conda If you are not in a conda environment with psana-conda The old myrel directory has the hidden file .sit_release The new myrel directory will also have the hidden file |
activate test release | cd myrel sit_setup | cd myrel source conda_setup | conda_setup looks in the current directory (like sit_setup) for conda_setup will add *tr* to your prompt to indicate that you are |
activate test | sit_setup /path/to/my/release | source conda_setup --reldir /path/to/my/release | As above, but activate a test release in another directory |
create new package | newpkg MyPkg | condarel --newpkg --name MyPkg | Creates the directory MyPkg with a minimal structure, include/src/app/data, and SConscript |
create new package in psdm svn repo | psvn newpkg MyPkg | ** use git ** | We won't make packages in the svn psdm repo anymore. We have removed psvn. Work directly with github/lcls-psana |
create new package in psdm users repo | psvn -u newpkg MyPkg | ** use git ** | With github, bitbucket, slaclab, etc, there is no need for users to |
checkout new package | addpkg MyPkg | condarel --addpkg --name MyPkg | for packages that are not part of psana-conda gets it from master in lcls-psana |
checkout existing package | addpkg XtcInput | condarel --addpkg --name XtcInput | Looks up the appropriate tag for the version of psana-conda. Checks out that tag *note this puts you in a HEADLESS state, if developing, checkout master* |
checkout exisiting using https | condarel --addpkg --name XtcInput --https | We default to ssh keys, but you can generate the https based git clone command with the --https flag | |
checkout existing package from psdm users repo | addpkg -u MyPkg | condarel --addpkg --user --name MyPkg | condarel takes --user flag to checkout from psdm users repo. You can also use --tag if you maintain tags in your repo. |
checkout from HEAD or master | addpkg XtcInput HEAD | condarel --addpkg --name XtcInput --tag HEAD | You can also specify specific tags with the --tag argument, HEAD means master for git |
build | scons | scons | same |
develop | very awkward | condarel --addpkg --name pdsdata condarel --addpkg --name pdsdata_ext | See also Building the psalg and pdsdata packages With conda, pdsdata and psalg are part of the psana-conda package. They get put in a subdirectory called extpkgs. You need the proxy packages to build/develop. |
develop ndarray | very awkward | condarel --addpkg --name ndarray --tag HEAD update version in ndarray recipe meta.yaml: | ndarray live in it's own conda package. After tagging your changes to ndarray, a new ndarray conda |
release info | relinfo | ** not implemented ** | this is an important missing feature we need for github should be condarel --relinfo |
upgrade release | relupgrade ana-19.0.20 sit_setup scons -c scons | source activate ana-1.0.8 | In conda, first activate, using standard conda commands, the environment you want to build against. Then use the --chenv command, it picks up the current conda environment. |
develop Translator | addpkg Translator scons | condarel --addpkg --name Translator | Since the Translator includes headers using package names, i.e, #include "hdf5/hdf5.h" You must first include the hdf5 and openmpi proxy packages |
work with
|
addpkg SConsTools
rm -r SConsTools |
condarel --addpkg --name SConsTools
rm -r SConsTools | If you remove SConsTools after checking it out, you have to |
Test | scons test | scons test | same |
work | addpkg MyPkg HEAD | condarel --addpkg --name MyPkg --tag HEAD | We have removed the psvn tool Please see Version control with git for details of using git |
track diffs | svn diff -r7810:HEAD file.h | ?? | Please see Version control with git for details of using git |
exit conda | -- | undo_conda | If you need to get out of the conda world, and go back to where you were before (rpm based psana, if you are sourcing /reg/g/psdm/etc/ana_env.sh) then the undo_conda command does this. NOTE: releases built in the conda world will not work in the old RPM world. |
list releases | ls $SIT_RELDIR | conda env list | Think of the old RPM based releases as conda environments - use standard conda commands to see them note - this lists your own environments (if you've made any) in addition to the ana environments maintained at LCLS. |
identify ana-current | ls -l $SIT_RELDIR/ana-current | more /reg/g/psdm/sw/conda/current/ana/ana-current | changed around April 1 2017, used to be conda/ana-current |
identify | ls -l $SIT_RELDIR/current (I think) | more /reg/g/psdm/sw/conda/current/dm/dm-current | |
look at source code | ls $SIT_RELDIR/ana-current/<pkg> ls $SIT_RELDIR/<release>/<pkg> | ls /reg/g/psdm/sw/conda/scratch/<release>/<pkg> ls $CONDA_PREFIX/lib/python2.7/site-packages/<pkg> | Package source code is available through the scratch directory. In the conda environment only python code is available. |
add pkg to psana | edit the file ana-tags in the | update the file psana-conda-svn-pkgs | We should document this more completely in the Admin Documentation |
Missing Functionality
Not all functionality of addpkg, relinfo, sit_setup etc have been implemented. If there is a feature that you need, let me know.
If you want to use the old commands, i.e, addpkg instead of condrel --addpkg, we can write new wrappers - but I think while we transition it is good to keep the interfaces distinct as psana developers will be working with both build systems.
Converting a Release
I recommend leaving an old release alone and starting new ones based on conda, however there are two commands in condarel to convert back and forth. This should work for users writing their own C++ psana modules, but won't work for psana developers that have checked out certain external proxy packages. These commands are
condarel --convert2conda
note the name of the old rpm release that you lost. If you want to switch back, use that name with the
condarel --convert2rpm
command. See condarel -h for details.
References
- Version control with git
- https://github.com/lcls-psana/
- SVN based Psana Developer Documentation
- SConsTools