You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

With the new-style psana1 that supports both py2 and py3 use the following commands to create a test release:

mkdir ana-4.0.5
cd ana-4.0.5
source /reg/g/psdm/etc/psconda.sh (defaults to py2, optionally add a "-py3" flag for py3)
# If a different environment needs to be activated, it needs to be activated here
# before setup_testrel.
source /cds/sw/ds/ana/conda1/manage/bin/setup_testrel
git clone git@github.com:lcls-psana/Detector.git
scons

cpo found that (for reasons I don't currently understand, but perhaps related to conda) LD_LIBRARY_PATH is not taking precedence with test releases and I was forced to use LD_PRELOAD like this (I also have the impression LD_PRELOAD_PATH didn't work):

export LD_PRELOAD=/cds/home/c/cpo/ana-4.0.12/arch/x86_64-rhel7-gcc48-opt/lib/libpsddl_pds2psana.so:/cds/home/c/cpo/ana-4.0.12/arch/x86_64-rhel7-gcc48-opt/lib/libPSXtcInput.so

To compile debug code one has to put "dbg" at end of SIT_ARCH: export SIT_ARCH=x86_64-rhel7-gcc48-dbg.  Remember to set LD_PRELOAD above to point to the dbg area (instead of opt) as well.

This variable is used in https://github.com/lcls-psana/SConsTools/blob/master/src/tools/psdm_cplusplus.py

External Packages

If working on external packages (for example, ndarray, psalg, pdsdata, python), you need to also to check out the corresponding proxy package. For example: ndarray_ext, Python, psalg_ext, pdsdata_ext, etc. The Python proxy package allows the python header files to be visible to C++ code.

Some Notes From Dan D. On the LD_PRELOAD Problem

Hi,

The fundamental issue is that the shared libraries are being built in a way to effectively ignore LD_LIBRARY_PATH (more details below*). To get around this I made a modified branch of SConsTools to restore the old behavior of the scons build.

These are the instructions you need to follow:

  1. Run ‘scons -c’ to clean up your test release (or just nuke the ‘arch’ and ‘build’ directories)
  2. In your test release directory clone the SConsTools and checkout the  testrel-fix branch (git clone -b testrel-fix git@github.com:lcls-psana/SConsTools.git)
  3. Now if you build all the shared libraries you built will use LD_LIBRARY_PATH.
  4. Unfortunately, the shared libraries from the base release still won’t use LD_LIBRARY_PATH. To get around this I had to add  psana, PSXtcInput, and PSEvt to my test release and rebuild them. One alternative solution if we don’t want to affect the production releases is to make say a ana.X.Y.Z-dev version of all the base releases that is built with the old behavior and use it as the ‘base’ release for development.

* SConsTools changes:

At some point a line in src/tools/psdm_cplusplus.py was changed from “env.Append(LINKFLAGS = ' ' + _ld_opt.get(opt,'') + ' -Wl,--copy-dt-needed-entries -Wl,--enable-new-dtags')” to “env.Append(LINKFLAGS = ' ' + _ld_opt.get(opt,''))”

This changed the behavior of the shared libraries/executables that we build so that they set RPATH instead of RUNPATH. The generally priority on Linux for searching is RPATH -> LD_LIBRARY_PATH -> RUNPATH. This means if a matching library is in the paths defined in RPATH it will always be found there and LD_LIBRARY_PATH isn’t used. This means that you can’t override the libraries in the base release by putting them in arch/<arch>/lib directory of the test release and adding it to LD_LIBRARY_PATH.

----------------------------------

Mikhail tried the above and it didn't work:  instead Dan suggested: 

for i in ${PWD}/arch/x86_64-rhel7-gcc48-opt/lib/*.so; do export "LD_PRELOAD=$i:$LD_PRELOAD"; done

Conda LD_LIBRARY_PATH Incompatibility

  • conda has it's own "sysroot" with /lib and /include
  • these can conflict with OS default sys root, but conda avoids that by using RPATH in the executables
  • setting LD_LIBRARY_PATH can couple these two incompatible worlds: we saw this with conda's libattr.so.1 (dependency of Qt) breaking "ls"
  • but psana1 test releases need to load alternate libraries (currently we use LD_LIBRARY_PATH)
  • proposed solution: use LD_PRELOAD which can be set on a library-by-library basis for psana1 test releases
  • when we build a binary in a development release we don't use RPATH, so that's why we had been using LD_LIBRARY_PATH (we think)
  • preliminary test-release possible proposal:
    • eliminate use of LD_LIBRARY_PATH
    • use RPATH for building development binaries
      • can be set to ONLY point to conda's sysroot (this would be just like conda)
      • could also set to first the test-release library path then conda, but it might simpler to just use LD_PRELOAD everywhere
    • use LD_PRELOAD (which can be set on per-library basis) to "jam" development .so files to run
    • to test this: checkout "psana" (builds some binaries) and "Detector" (this creates a .so)
  • modified proposal:
    • valmar and cpo found that binaries and libraries have $ORIGIN/../lib and $ORIGIN/../lib/python in the RPATH.  feels like this should work without LD_PRELOAD and LD_LIBRARY_PATH.  We'll try that and see what failure modes emerge.
  • order of library path searching:  https://en.wikipedia.org/wiki/Rpath#GNU_ld.so.  we think conda uses RPATH not RUNPATH.
  • No labels