Page History

...

mkdir ana-4.0.5
cd ana-4.0.5
source /reg/g/psdm/etc/psconda.sh (defaults to py2, optionally add a "-py3" flag for py3)
# If a different environment needs to be activated, it needs to be activated here
# before setup_testrel.
source /cds/sw/ds/ana/conda1/manage/bin/setup_testrel
git clone git@github.com:lcls-psana/Detector.git
scons

cpo found that (for reasons I don't currently understand, but perhaps related to conda) LD_LIBRARY_PATH is not taking precedence with test releases and I was forced to use LD_PRELOAD like this (I also have the impression LD_PRELOAD_PATH didn't work): (found in Dec. 2022 that we can use rpath and don't need LD_PRELOAD: sometimes have to check out the right external packages (see section below).

export LD_PRELOAD=/cds/home/c/cpo/ana-4.0.12/arch/x86_64-rhel7-gcc48-opt/lib/libpsddl_pds2psana.so:/cds/home/c/cpo/ana-4.0.12/arch/x86_64-rhel7-gcc48-opt/lib/libPSXtcInput.so

To compile debug code one has to put "dbg" at end of SIT_ARCH: export SIT_ARCH=x86_64-rhel7-gcc48-dbg.  Remember to set LD_PRELOAD above to point to the dbg area (instead of opt) as well.

This "dbg" variable is used in https://github.com/lcls-psana/SConsTools/blob/master/src/tools/psdm_cplusplus.py

...

If working on external packages (for example, ndarray, psalg, pdsdata, python), you need to also to check out the corresponding proxy package. For example: ndarray_ext, Python, psalg_ext, pdsdata_ext, etc. The Python proxy package allows the python header files to be visible to C++ code.  Sometimes you will also get missing symbols for packages like boost and then that external package must also be checked out.

Some Notes From Dan D. On the LD_PRELOAD Problem

NOTE: as of Dec. 2022 we don't think LD_PRELOAD is needed.  A combination of the existing rpath approach plus checking out external packages (see above section) seems to be sufficient.

Hi,

The fundamental issue is that the shared libraries are being built in a way to effectively ignore LD_LIBRARY_PATH (more details below*). To get around this I made a modified branch of SConsTools to restore the old behavior of the scons build.

These are the instructions you need to follow:

1. Run ‘scons -c’ to clean up your test release (or just nuke the ‘arch’ and ‘build’ directories)
2. In your test release directory clone the SConsTools and checkout the  testrel-fix branch (git clone -b testrel-fix git@github.com:lcls-psana/SConsTools.git)
3. Now if you build all the shared libraries you built will use LD_LIBRARY_PATH.
4. Unfortunately, the shared libraries from the base release still won’t use LD_LIBRARY_PATH. To get around this I had to add  psana, PSXtcInput, and PSEvt to my test release and rebuild them. One alternative solution if we don’t want to affect the production releases is to make say a ana.X.Y.Z-dev version of all the base releases that is built with the old behavior and use it as the ‘base’ release for development.

* SConsTools changes:

At some point a line in src/tools/psdm_cplusplus.py was changed from “env.Append(LINKFLAGS = ' ' + _ld_opt.get(opt,'') + ' -Wl,--copy-dt-needed-entries -Wl,--enable-new-dtags')” to “env.Append(LINKFLAGS = ' ' + _ld_opt.get(opt,''))”

This changed the behavior of the shared libraries/executables that we build so that they set RPATH instead of RUNPATH. The generally priority on Linux for searching is RPATH -> LD_LIBRARY_PATH -> RUNPATH. This means if a matching library is in the paths defined in RPATH it will always be found there and LD_LIBRARY_PATH isn’t used. This means that you can’t override the libraries in the base release by putting them in arch/<arch>/lib directory of the test release and adding it to LD_LIBRARY_PATH.

----------------------------------

Mikhail tried the above and it didn't work:  instead Dan suggested: 

for i in ${PWD}/arch/x86_64-rhel7-gcc48-opt/lib/*.so; do export "LD_PRELOAD=$i:$LD_PRELOAD"; done

...