With the new-style psana1 that supports both py2 and py3 use the following commands to create a test release:

mkdir ana-4.0.5
cd ana-4.0.5
source /reg/g/psdm/etc/psconda.sh (defaults to py2, optionally add a "-py3" flag for py3)
# If a different environment needs to be activated, it needs to be activated here
# before setup_testrel.
source /cds/sw/ds/ana/conda1/manage/bin/setup_testrel
git clone git@github.com:lcls-psana/Detector.git
scons

cpo found that (for reasons I don't currently understand, but perhaps related to conda) LD_LIBRARY_PATH is not taking precedence with test releases and I was forced to use LD_PRELOAD like this (I also have the impression LD_PRELOAD_PATH didn't work): (found in Dec. 2022 that we can use rpath and don't need LD_PRELOAD: sometimes have to check out the right external packages (see section below).

export LD_PRELOAD=/cds/home/c/cpo/ana-4.0.12/arch/x86_64-rhel7-gcc48-opt/lib/libpsddl_pds2psana.so:/cds/home/c/cpo/ana-4.0.12/arch/x86_64-rhel7-gcc48-opt/lib/libPSXtcInput.so

To compile debug code one has to put "dbg" at end of SIT_ARCH: export SIT_ARCH=x86_64-rhel7-gcc48-dbg.  Remember to set LD_PRELOAD above to point to the dbg area (instead of opt) as well.

This "dbg" variable is used in https://github.com/lcls-psana/SConsTools/blob/master/src/tools/psdm_cplusplus.py

External Packages

If working on external packages (for example, ndarray, psalg, pdsdata, python), you need to also to check out the corresponding proxy package. For example: ndarray_ext, Python, psalg_ext, pdsdata_ext, etc. The Python proxy package allows the python header files to be visible to C++ code.  Sometimes you will also get missing symbols for packages like boost and then that external package must also be checked out.

Some Notes From Dan D. On the LD_PRELOAD Problem

NOTE: as of Dec. 2022 we don't think LD_PRELOAD is needed.  A combination of the existing rpath approach plus checking out external packages (see above section) seems to be sufficient.

Hi,

The fundamental issue is that the shared libraries are being built in a way to effectively ignore LD_LIBRARY_PATH (more details below*). To get around this I made a modified branch of SConsTools to restore the old behavior of the scons build.

These are the instructions you need to follow:

  1. Run ‘scons -c’ to clean up your test release (or just nuke the ‘arch’ and ‘build’ directories)
  2. In your test release directory clone the SConsTools and checkout the  testrel-fix branch (git clone -b testrel-fix git@github.com:lcls-psana/SConsTools.git)
  3. Now if you build all the shared libraries you built will use LD_LIBRARY_PATH.
  4. Unfortunately, the shared libraries from the base release still won’t use LD_LIBRARY_PATH. To get around this I had to add  psana, PSXtcInput, and PSEvt to my test release and rebuild them. One alternative solution if we don’t want to affect the production releases is to make say a ana.X.Y.Z-dev version of all the base releases that is built with the old behavior and use it as the ‘base’ release for development.

* SConsTools changes:

At some point a line in src/tools/psdm_cplusplus.py was changed from “env.Append(LINKFLAGS = ' ' + _ld_opt.get(opt,'') + ' -Wl,--copy-dt-needed-entries -Wl,--enable-new-dtags')” to “env.Append(LINKFLAGS = ' ' + _ld_opt.get(opt,''))”

This changed the behavior of the shared libraries/executables that we build so that they set RPATH instead of RUNPATH. The generally priority on Linux for searching is RPATH -> LD_LIBRARY_PATH -> RUNPATH. This means if a matching library is in the paths defined in RPATH it will always be found there and LD_LIBRARY_PATH isn’t used. This means that you can’t override the libraries in the base release by putting them in arch/<arch>/lib directory of the test release and adding it to LD_LIBRARY_PATH.

----------------------------------

Mikhail tried the above and it didn't work:  instead Dan suggested: 

for i in ${PWD}/arch/x86_64-rhel7-gcc48-opt/lib/*.so; do export "LD_PRELOAD=$i:$LD_PRELOAD"; done

Conda LD_LIBRARY_PATH Incompatibility

  • conda has it's own "sysroot" with /lib and /include
  • these can conflict with OS default sys root, but conda avoids that by using RPATH in the executables
  • setting LD_LIBRARY_PATH can couple these two incompatible worlds: we saw this with conda's libattr.so.1 (dependency of Qt) breaking "ls"
  • but psana1 test releases need to load alternate libraries (currently we use LD_LIBRARY_PATH)
  • proposed solution: use LD_PRELOAD which can be set on a library-by-library basis for psana1 test releases
  • when we build a binary in a development release we don't use RPATH, so that's why we had been using LD_LIBRARY_PATH (we think)
  • preliminary test-release possible proposal:
    • eliminate use of LD_LIBRARY_PATH
    • use RPATH for building development binaries
      • can be set to ONLY point to conda's sysroot (this would be just like conda)
      • could also set to first the test-release library path then conda, but it might simpler to just use LD_PRELOAD everywhere
    • use LD_PRELOAD (which can be set on per-library basis) to "jam" development .so files to run
    • to test this: checkout "psana" (builds some binaries) and "Detector" (this creates a .so)
  • modified proposal:
    • valmar and cpo found that binaries and libraries have $ORIGIN/../lib and $ORIGIN/../lib/python in the RPATH.  feels like this should work without LD_PRELOAD and LD_LIBRARY_PATH.  We'll try that and see what failure modes emerge.
    • conclusion so far (Dec. 2022) is that this seems to be working if we also checkout the right external packages (see ExternalPackages)
  • order of library path searching:  https://en.wikipedia.org/wiki/Rpath#GNU_ld.so.  we think conda uses RPATH not RUNPATH.

pkg_tree.pkl Problem

When building a test release with py3 can get this error:

(ana-4.0.48-py3) scons
scons: Reading SConscript files ...
Mkdir("build/x86_64-rhel7-gcc48-opt")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 61: invalid start byte:
  File "/sdf/home/c/cpo/ana-4.0.48/SConstruct", line 186:
    loadPkgDeps(fname)
  File "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.48-py3/lib/python3.9/site-packages/SConsTools/dependencies.py", line 262:
    env['PKG_TREE_BASE'].update( pickle.load( f ) )
(ana-4.0.48-py3)

This is because files like /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.48-py3/data/anarelinfo/.pkg_tree.pkl were generated with py2.  There is a conversion script in in manage/bin/pickle2to3.py but it broke at some point, so currently Valerio has been copying the file from ana-4.0.46-py3

  • No labels