Thoughts from Valerio Mariano on Jan. 26, 2023 (copied from slack)
Chris, I write this to answer some questions that you had about Spack, but also to have some circulation for the concepts of Spack so that someone else except me sees them. So ignore this if you are busy and don't "store" this anywhere in your brain. I just wanted to expose you to this
With spack, the packages are not rebuilt from scratch every time. Every package is installed in its own directory with its own dependencies and then linked to the environment. So, if you say, build epics-rogue, once, and if it never updates, it is just symlinked into the new environment. Here is a real life example. These are a few lines from an ls in the lib directory of the environment:
lrwxrwxrwx 1 valmar xu 144 Jan 26 12:30 libyaml-0.so.2.0.9 -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libyaml-0.2.5-w4drptmhnxrlzej2o6th3wrijikp5rpn/lib/libyaml-0.so.2.0.9 lrwxrwxrwx 1 valmar xu 135 Jan 26 12:30 libyaml.a -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libyaml-0.2.5-w4drptmhnxrlzej2o6th3wrijikp5rpn/lib/libyaml.a lrwxrwxrwx 1 valmar xu 136 Jan 26 12:30 libyaml.so -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libyaml-0.2.5-w4drptmhnxrlzej2o6th3wrijikp5rpn/lib/libyaml.so lrwxrwxrwx 1 valmar xu 132 Jan 26 12:31 libyasm.a -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/yasm-1.3.0-xcqkegeoczpzr7irem6li5qep7bfzwdz/lib/libyasm.a lrwxrwxrwx 1 valmar xu 130 Jan 26 12:30 libz.a -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/zlib-1.2.13-fv4i7j3qevpyzpnuz7wux5imwenhygd7/lib/libz.a lrwxrwxrwx 1 valmar xu 133 Jan 26 12:30 libzmq.a -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libzmq-4.3.4-m7d3ahowhrwwaznxpjfakqxp4wgtvasl/lib/libzmq.a lrwxrwxrwx 1 valmar xu 134 Jan 26 12:30 libzmq.so -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libzmq-4.3.4-m7d3ahowhrwwaznxpjfakqxp4wgtvasl/lib/libzmq.so lrwxrwxrwx 1 valmar xu 136 Jan 26 12:30 libzmq.so.5 -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libzmq-4.3.4-m7d3ahowhrwwaznxpjfakqxp4wgtvasl/lib/libzmq.so.5 lrwxrwxrwx 1 valmar xu 140 Jan 26 12:30 libzmq.so.5.2.4 -> /cds/home/v/valmar/Software/spack/opt/spack/linux-rhel7-nehalem/gcc-12.2.0/libzmq-4.3.4-m7d3ahowhrwwaznxpjfakqxp4wgtvasl/lib/libzmq.so.5.2.4
Do you see that long hash in the directory name? (for example: m7d3ahowhrwwaznxpjfakqxp4wgtvasl ) That's a hash of the version of the package and the version of all its dependencies. As long as that does not change, you don't need to recompile the package
Now you will tell me: but as time goes on, the dependencies will change, say, the boost version will move from 1.74 to 1.78 and you will need to recompile, right? Not really, because each package has its own set of dependencies which are store in its installation directory and are completely independent from the other packages. This means that there can be packages in the environment, one depending on boost 1.74 and one dependent on 1.78.
Which version will be visible in the environment? None or one of them of your choice. If you try to symlink conflicting packages in the environment, Spack will error out and tell you explicitely (thank God, finally!) which packages conflict
It should be said that you can configure spack to run as a normal conda, where the dependencies are all in common, and are all symlinked into the environment, if you prefer. You can even tell spack: try to keep as much in common as possible but if a single package causes conflicts, keep the dependencies of that one private
Also, you can create a binary repository, so that you don't have to install the packages from source every time, but Spack does not use any of the conda tricks: the binary packages must be installed in the same location where they were created, no relocation
But that is just the location of the package, so the directory with the hash, not of the environment. You can create that everywhere with the symlinks
Some packages that you would think are hard to build from source, like Qt and epics-base, are actually easy, because there are already packages from them in Spack
One package that took me really a lot of time is rogue: I would say that most of the packages coming from SLAC assume in some way that you are working with conda, Spack requires that you can install a package in a directory of your choice. If you do that with rogue, you end up with the python module in the wrong place. So that required some tricks
Also, it's taking a lot of work on our own code. With its system of links spack requires that you specify explicitely the dependencies of a package. There is no build environment, that's one of the reasons it's easier, but Spack links the package you are building only to the packages you explicitly list as dependencies. We have several instances where, for example, some DAQ code depends on "readline" but without listing it in CMakeLists.txt. With conda, it gets picked automatically because it is in the environment, but Spack complains.
That is why it's taking me so long
By the way, these concepts: hashes of dependencies, hidden dependencies, etc. are not new to Spack at all. Actually spack took them from the new package installation systems that are spreading in the Linux world: Nix (https://nixos.org/), GNU's Guix (https://nixos.org/) , etc.
These are becoming popular in the Linux world, because they solve the dependency problem once and for all, and since you control the exact versions of everything, you can have perfectly reproducible environments. Spack just implements these concepts in Python, and with an eye to the scientific software world
Also, the main developers (mainy based in Berkely and Livermore) are super friendly and very very helpful. I have been interacting with them in the last few weeks
Overview
Content Tools