With the new-style psana1 that supports both py2 and py3 use the following commands to create a test release:
mkdir ana-4.0.5 cd ana-4.0.5 source /reg/g/psdm/etc/psconda.sh (defaults to py2, optionally add a "-py3" flag for py3) # If a different environment needs to be activated, it needs to be activated here # before setup_testrel. source /cds/sw/ds/ana/conda1/manage/bin/setup_testrel git clone git@github.com:lcls-psana/Detector.git scons
cpo found that (for reasons I don't currently understand, but perhaps related to conda) LD_LIBRARY_PATH is not taking precedence with test releases and I was forced to use LD_PRELOAD like this (I also have the impression LD_PRELOAD_PATH didn't work):
export LD_PRELOAD=/cds/home/c/cpo/ana-4.0.12/arch/x86_64-rhel7-gcc48-opt/lib/libpsddl_pds2psana.so:/cds/home/c/cpo/ana-4.0.12/arch/x86_64-rhel7-gcc48-opt/lib/libPSXtcInput.so
To compile debug code one has to put "dbg" at end of SIT_ARCH: export SIT_ARCH=x86_64-rhel7-gcc48-dbg. Remember to set LD_PRELOAD above to point to the dbg area (instead of opt) as well.
This variable is used in https://github.com/lcls-psana/SConsTools/blob/master/src/tools/psdm_cplusplus.py
External Packages
If working on external packages (for example, ndarray, psalg, pdsdata, python), you need to also to check out the corresponding proxy package. For example: ndarray_ext, Python, psalg_ext, pdsdata_ext, etc. The Python proxy package allows the python header files to be visible to C++ code.
Some Notes From Dan D. On the LD_PRELOAD Problem
Hi,
The fundamental issue is that the shared libraries are being built in a way to effectively ignore LD_LIBRARY_PATH (more details below*). To get around this I made a modified branch of SConsTools to restore the old behavior of the scons build.
These are the instructions you need to follow:
- Run ‘scons -c’ to clean up your test release (or just nuke the ‘arch’ and ‘build’ directories)
- In your test release directory clone the SConsTools and checkout the testrel-fix branch (git clone -b testrel-fix git@github.com:lcls-psana/SConsTools.git)
- Now if you build all the shared libraries you built will use LD_LIBRARY_PATH.
- Unfortunately, the shared libraries from the base release still won’t use LD_LIBRARY_PATH. To get around this I had to add psana, PSXtcInput, and PSEvt to my test release and rebuild them. One alternative solution if we don’t want to affect the production releases is to make say a ana.X.Y.Z-dev version of all the base releases that is built with the old behavior and use it as the ‘base’ release for development.
* SConsTools changes:
At some point a line in src/tools/psdm_cplusplus.py was changed from “env.Append(LINKFLAGS = ' ' + _ld_opt.get(opt,'') + ' -Wl,--copy-dt-needed-entries -Wl,--enable-new-dtags')” to “env.Append(LINKFLAGS = ' ' + _ld_opt.get(opt,''))”
This changed the behavior of the shared libraries/executables that we build so that they set RPATH instead of RUNPATH. The generally priority on Linux for searching is RPATH -> LD_LIBRARY_PATH -> RUNPATH. This means if a matching library is in the paths defined in RPATH it will always be found there and LD_LIBRARY_PATH isn’t used. This means that you can’t override the libraries in the base release by putting them in arch/<arch>/lib directory of the test release and adding it to LD_LIBRARY_PATH.
----------------------------------
Mikhail tried the above and it didn't work: instead Dan suggested:
for i in ${PWD}/arch/x86_64-rhel7-gcc48-opt/lib/*.so; do export "LD_PRELOAD=$i:$LD_PRELOAD"; done
Conda LD_LIBRARY_PATH Incompatibility
- conda has it's own "sysroot" with /lib and /include
- these can conflict with OS default sys root, but conda avoids that by using RPATH in the executables
- setting LD_LIBRARY_PATH can couple these two incompatible worlds: we saw this with conda's libattr.so.1 (dependency of Qt) breaking "ls"
- but psana1 test releases need to load alternate libraries (currently we use LD_LIBRARY_PATH)
- proposed solution: use LD_PRELOAD which can be set on a library-by-library basis for psana1 test releases
- when we build a binary in a development release we don't use RPATH, so that's why we had been using LD_LIBRARY_PATH (we think)
- new test-release possible proposal:
- eliminate use of LD_LIBRARY_PATH
- use RPATH for building development binaries
- can be set to ONLY point to conda's sysroot (this would be just like conda)
- could also set to first the test-release library path then conda, but it might simpler to just use LD_PRELOAD everywhere
- use LD_PRELOAD (which can be set on per-library basis) to "jam" development .so files to run
- to test this: checkout "psana" (builds some binaries) and "Detector" (this creates a .so)
- order of library path searching: https://en.wikipedia.org/wiki/Rpath#GNU_ld.so. we think conda uses RPATH not RUNPATH.