Slides from E4S meeting on June 29, 2023: https://drive.google.com/drive/folders/1u6DR04jjqgoiXfhWXOv14k0X3PCNq-0h
Cheat-Sheet of Common Commands
spack env create test
spack env activate test
spack find (shows packages)
spack add vim
spack find
spack concretize -f (force: do concretize from scratch, vs. incremental)
(knows about other environments)
(concretize normally creates a view: the equivalent of the conda solver)
("unify true" tries to force identical versions of boost?)
spack info vim (supported recipe versions)
spack remove vim
spack add vim@@8.2.2541 (specify version)
Other Tips
for laptops we should use containers
common configuration files that need to be tweaked: packages.yaml, compiler.yaml
Binary Caches
Spack's documentation on binary caches is good in providing a sufficient level of detail without being too verbose:
https://spack.readthedocs.io/en/latest/binary_caches.html
One thing to note is that however long the path is where you initially build the packages, that's the upper limit for how long the path can be when installing downstream. For this reason, one of the things we do, when possible, that isn't documented in Spack's official guide, is we make sure to use a long path length to build the binaries we intend to populate the build cache with.
We tell Spack to pad the install tree to 256 characters in length like this before doing the initial builds:
$> spack config add config:install_tree:padded_length:256
Then we do the builds, push to binary cache, update the cache index and install the public key there. Then users can install packages from the cache so long as they aren't trying to install to a path more than 256 characters in length.
Plusses And Minuses
Minuses
- For relocatability, there is an RPATH length problem. Workaround is to build your package to a path that is "long". E4S has some other workaround for this.
- not world-reused?
- many more configuration possibilities (hard to understand, failure-prone)
- binary-image caches (equivalent of conda-forge) much more machine-specific: depends on compiler, architecture, "build options" as defined by package.py
Plusses
- used on many supercomputers, so good to have some commonality with them
- can use selected OS packages (e.g. ssh) as well as spack packages
- can have many different versions of different packages in an environment
Initial Thoughts from Valerio Mariani
Jan. 26, 2023 (copied from slack)
Chris, I write this to answer some questions that you had about Spack, but also to have some circulation for the concepts of Spack so that someone else except me sees them. So ignore this if you are busy and don't "store" this anywhere in your brain. I just wanted to expose you to this
With spack, the packages are not rebuilt from scratch every time. Every package is installed in its own directory with its own dependencies and then linked to the environment. So, if you say, build epics-rogue, once, and if it never updates, it is just symlinked into the new environment. Here is a real life example. These are a few lines from an ls in the lib directory of the environment:
lrwxrwxrwx 1 valmar xu 144 Jan 26 12:30 libyaml-0.so.2.0.9 -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libyaml-0.2.5-w4drptmhnxrlzej2o6th3wrijikp5rpn/lib/libyaml-0.so.2.0.9 lrwxrwxrwx 1 valmar xu 135 Jan 26 12:30 libyaml.a -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libyaml-0.2.5-w4drptmhnxrlzej2o6th3wrijikp5rpn/lib/libyaml.a lrwxrwxrwx 1 valmar xu 136 Jan 26 12:30 libyaml.so -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libyaml-0.2.5-w4drptmhnxrlzej2o6th3wrijikp5rpn/lib/libyaml.so lrwxrwxrwx 1 valmar xu 132 Jan 26 12:31 libyasm.a -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/yasm-1.3.0-xcqkegeoczpzr7irem6li5qep7bfzwdz/lib/libyasm.a lrwxrwxrwx 1 valmar xu 130 Jan 26 12:30 libz.a -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/zlib-1.2.13-fv4i7j3qevpyzpnuz7wux5imwenhygd7/lib/libz.a lrwxrwxrwx 1 valmar xu 133 Jan 26 12:30 libzmq.a -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libzmq-4.3.4-m7d3ahowhrwwaznxpjfakqxp4wgtvasl/lib/libzmq.a lrwxrwxrwx 1 valmar xu 134 Jan 26 12:30 libzmq.so -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libzmq-4.3.4-m7d3ahowhrwwaznxpjfakqxp4wgtvasl/lib/libzmq.so lrwxrwxrwx 1 valmar xu 136 Jan 26 12:30 libzmq.so.5 -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libzmq-4.3.4-m7d3ahowhrwwaznxpjfakqxp4wgtvasl/lib/libzmq.so.5 lrwxrwxrwx 1 valmar xu 140 Jan 26 12:30 libzmq.so.5.2.4 -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libzmq-4.3.4-m7d3ahowhrwwaznxpjfakqxp4wgtvasl/lib/libzmq.so.5.2.4
Do you see that long hash in the directory name? (for example: m7d3ahowhrwwaznxpjfakqxp4wgtvasl ) That's a hash of the version of the package and the version of all its dependencies. As long as that does not change, you don't need to recompile the package
Now you will tell me: but as time goes on, the dependencies will change, say, the boost version will move from 1.74 to 1.78 and you will need to recompile, right? Not really, because each package has its own set of dependencies which are store in its installation directory and are completely independent from the other packages. This means that there can be packages in the environment, one depending on boost 1.74 and one dependent on 1.78.
Which version will be visible in the environment? None or one of them of your choice. If you try to symlink conflicting packages in the environment, Spack will error out and tell you explicitely (thank God, finally!) which packages conflict
It should be said that you can configure spack to run as a normal conda, where the dependencies are all in common, and are all symlinked into the environment, if you prefer. You can even tell spack: try to keep as much in common as possible but if a single package causes conflicts, keep the dependencies of that one private
Also, you can create a binary repository, so that you don't have to install the packages from source every time, but Spack does not use any of the conda tricks: the binary packages must be installed in the same location where they were created, no relocation
But that is just the location of the package, so the directory with the hash, not of the environment. You can create that everywhere with the symlinks
Some packages that you would think are hard to build from source, like Qt and epics-base, are actually easy, because there are already packages from them in Spack
One package that took me really a lot of time is rogue: I would say that most of the packages coming from SLAC assume in some way that you are working with conda, Spack requires that you can install a package in a directory of your choice. If you do that with rogue, you end up with the python module in the wrong place. So that required some tricks
Also, it's taking a lot of work on our own code. With its system of links spack requires that you specify explicitely the dependencies of a package. There is no build environment, that's one of the reasons it's easier, but Spack links the package you are building only to the packages you explicitly list as dependencies. We have several instances where, for example, some DAQ code depends on "readline" but without listing it in CMakeLists.txt. With conda, it gets picked automatically because it is in the environment, but Spack complains.
That is why it's taking me so long
By the way, these concepts: hashes of dependencies, hidden dependencies, etc. are not new to Spack at all. Actually spack took them from the new package installation systems that are spreading in the Linux world: Nix (https://nixos.org/), GNU's Guix (https://nixos.org/) , etc.
These are becoming popular in the Linux world, because they solve the dependency problem once and for all, and since you control the exact versions of everything, you can have perfectly reproducible environments. Spack just implements these concepts in Python, and with an eye to the scientific software world
Also, the main developers (mainy based in Berkely and Livermore) are super friendly and very very helpful. I have been interacting with them in the last few weeks
Overview
Content Tools