Terminology

Buildroot is the tool that we use to build the kernel, kernel modules, user tools, and root file system for our Linux RT machines.

The name we give to the artifacts that buildroot generates is "buildroot".  So, we talk about versions of buildroot as if they are releases of a Linux distribution like Red Hat or Ubuntu.  But they are unrelated to any distribution - they are just buildroot artifacts.

The machines and the buildroot running on these machines are critical to the experiment infrastructure at SLAC.  Experiments have very narrow timing requirements.  This is why we use buildroot.  Buildroot allows us to create a minimal kernel and minimal user tools.  So that for example the kernel is not needlessly interrupted to check for a system update when it should be gathering data for the experiment.

Buildroot also allows user to easily apply patches to the kernel.  We apply the Preempt RT (real time) kernel patch.  This patch makes modifications to the kernel such that kernel interruptions are time-bounded.

Links

Why create a new version?

You might want to create a new version of buildroot if you've found a bug in the current version, or if a new feature is needed.  As I write this document, we have discovered three issues with the current buildroot that must be fixed.

Where do versions exist?

There are two locations in AFS in which we keep our buildroot releases.

  • /afs/slac/g/lcls/package/linuxRT - this location seems to be deprecated
  • /afs/slac/g/lcls/tftpboot/linuxRT/boot - this seems to be the currently-accepted location

You can create directories under the "boot" directory.  You'll see that there are various directory name formats.  Please use a format like this:

  • buildroot-<year>.<month>-<version>-<architecture>

For example:

  • buildroot-2019.08-2-x86_64

The example name indicates it is the second release of buildroot (the artifacts) based off the 2019.08 version of buildroot (the tool).  Note that it does not indicate when you built this version.  You may have built it today but you based it off a version of buildroot from 2019 (or whenever).

We've never had a second release before, so you'll not see -1 or -2 in any of the names, except the version I am working on as I write this doc.

The "ipxe" directory under "boot" contains a file for every machine that boots buildroot.  The filename indicates the machine name.  In that file you will find a line like this:

  • set vers buildroot-2019.08-x86_64

You can modify that line to point at whatever version of buildroot you would like, including a test directory.  Your test directory must not be in a mounted directory (like AFS) as the machine booting won't be able to access mounted file systems.  For this reason, if you have a test directory, it's probably best for it to be in this ("boot") directory.

Versions

There are many versions that get baked in to buildroot.  There is a BusyBox version, a kernel version, a Preempt RT patch version, user tools versions, and of course a buildroot version.

The version of buildroot (the tool) you download and use defines the BusyBox and kernel versions.  The kernel version defines the Preempt RT version.  So, the buildroot (tool) version is the key to all the versions.

Buildroot versions can be found at their downloads page.  Note that if you see "rt-<version>" in the file name, that's the version of Preempt RT.

The current version that we use (2019.08) is here.

SLAC specifics

We can't use buildroot (tool) exactly as it comes from its website.  We have a git repo which contains all the SLAC-specific stuff.  The repo is here.  The master branch is not used (as I write this doc).  Instead, the branches are named by the release, e.g. br-2019.08.  This is done because each branch works with only a single version of buildroot (tool).  For example, br-2019.08 works with buildroot-2019.08, but not necessarily any other version.

The SLAC-specific stuff:

  • config files for BusyBox, kernel, and buildroot (tool)
    • so you don't have to "make menuconfig" every time you want a new version of buildroot (artifacts)
  • Preempt RT patch (big)
  • other patches (several)
  • pre-rootfs scripts for adding tools, creating users, adding a password to root

The README in the repo is useful, though vague.  I will update it to be more useful.

Note that the BusyBox config file is reset every time a "make clean" is performed in buildroot (tool).  The other config files are preserved.  I plan to fix this.

Setting up

I first tried to unzip buildroot (tool) in AFS.  I much later hit a permissions issue during the build phase.  I would avoid AFS or any mounted file system.  I ultimately used aird-b50-srv01, under the /scratch directory.  In there I created a directory for my username (/scratch/egumtow).

Under that, you should have this structure:

  • buildroot
    • buildroot-<year>.<month>-<version>
      • buildroot-<year>.<month>-<version>-<architecture> (created by unzipping the downloaded buildroot, and renaming the resulting directory)
      • site-top (created by cloning the SLAC-specific git repo)
      • host
      • download

Yes, it's repetitive.  But we will have many buildroots, and different architectures for each buildroot.

Behind the buildroot-<year>.<month>-<version>-<architecture> directory, you must symlink "site" to "../site-top".

Now you must run a SLAC-specific script (br-installconf.sh) to apply things.

And now you can run "make".  It should succeed, and if so, you can find the artifacts in the "output/images" directory.

Here is the full sequence of commands:

egumtow@aird-b50-srv01 /scratch/egumtow> mkdir buildroot
egumtow@aird-b50-srv01 /scratch/egumtow> cd buildroot
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot> mkdir buildroot-2019.08-2
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot> cd buildroot-2019.08-2
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2> mkdir host
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2> mkdir download
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2> git clone https://github.com/slaclab/buildroot-site.git site-top
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2> cd site-top
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/site-stop> git checkout origin/br-2019.08
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/site-stop> cd ..
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2> wget https://buildroot.org/downloads/buildroot-2019.08.tar.gz
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2> tar xzvf buildroot-2019.08.tar.gz
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2> mv buildroot-2019.08 buildroot-2019.08-2-x86_64
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2> cd buildroot-2019.08-2-x86_64/
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/buildroot-2019.08-2-x86_64> ln -s ../site-top site
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/buildroot-2019.08-2-x86_64> site/scripts/br-installconf.sh -a x86_64
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/buildroot-2019.08-2-x86_64> make
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/buildroot-2019.08-2-x86_64> ls -l output/images
total 58256
-rw-r--r-- 1 egumtow gu  3796944 Sep 19 16:36 bzImage
-rw-r--r-- 1 egumtow gu 62914560 Sep 19 16:36 rootfs.ext2
-rw-r--r-- 1 egumtow gu 17510981 Sep 19 16:36 rootfs.ext2.gz
drwxr-xr-x 2 egumtow gu     4096 Sep 19 16:34 syslinux/

The files in the "output/images" directory can be copied to the appropriate directory under the "boot" directory in AFS.

The "make" step is long in duration.  I recommend teeing the output so you can read the file later if the build fails.

$ make | tee make.log

Making your changes

Probably your changes will be limited to one of the config files.  In other words, you're probably not making serious changes to the kernel.  If this is a correct assumption, your work can be accomplished by running one or more of these:

  • make menuconfig - for configuration changes to buildroot
    • saved to .config
  • make linux-menuconfig - for configuration changes to the kernel
    • saved to output/build/linux-<version>/.config
  • make busybox-menuconfig - for changes to BusyBox
    • saved to output/build/busybox-<version>/.config

And then subsequently save them (your changes to the config) outside buildroot.  The instructions in the git repo's README give instructions for "make" targets, but I've not had success, I think they don't work.  Instead, for now we can manually copy the config changes to the appropriate locations in the "site" directory.

  • make savedefconfig
    • saved to site/br2-external/configs/<architecture>_defconfig
  • make linux-update-defconfig
    • saved to output/build/linux-<version>/defconfig
      • MANUALLY save to site/config/linux-<version>-<architecture>.config - note that <architecture> can be "common" if it applies to all architectures
  • cp output/build/busybox-<version>/.config site/config/bb-<version>.config

You can find the version of busybox in its menuconfig.  It will say e.g. "BusyBox 1.31.0 Configuration" at the top of the screen.

When you run the "save" commands you are saving the config files to the "site" directory where you will later push them to git.  They will be preserved in git so we don't make the same changes for  the next version of buildroot (artifacts).  The next time (maybe years later) we apply the SLAC customizations, we run the br-installconf.sh script and the changes are applied to buildroot (tool).

Under which config (buildroot, linux, busybox) a config option exists isn't always obvious until you find it.

You may have different config files for different architectures.  I haven't dealt with this yet.  Will update this document when I do.

The buildroot build system has hooks that allow you to insert scripts at various points of the build.  This allows you to modify the artifacts after various points of the build.  In the repo that you cloned into the "site-top" directory, there is a "script" directory.  This directory contains these scripts.  We have a "post-build.sh" script that performs various tasks.  You can read about the post-build scripts here and here.  Note that the config options (e.g., BR2_ROOTFS_POST_BUILD_SCRIPT) described there are found in buildroot's (tool) "make menuconfig" target.

After you've made your config file changes and saved the configs:

egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/buildroot-2019.08-2-x86_64> make clean
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/buildroot-2019.08-2-x86_64> cp output/build/busybox-1.31.0/.config site/config/bb-1.31.0.config
egumtow@aird-b50-srv01 /scratch/egumtow/buildroot/buildroot-2019.08-2/buildroot-2019.08-2-x86_64> make

SLAC properties

There is a SLAC-specific file in the file system skeleton (fs_skeleton).  The file system skeleton exists in the site directory.  The file is at /etc/SLAC_properties.  The syntax is

PROPERTY=value

One property per line.  This file can be read by human users or scripts to determine behavior.  For example, if HAVE_ROOT_PASSWORD is set, then scripts know that if they use "su" then a password is required - the implication being "su" can't be used during the boot process when humans are not involved.

Buildroot documentation

Buildroot has ample documentation.  Be careful that the version of the documentation matches the version your buildroot.  The documentation that you should read comes with the download of buildroot at docs/manual/manual.text.

Submitting your changes

You must submit whatever SLAC-specific changes you've made back to the git repo.  Create a branch the indicates which version of buildroot your changes are to be applied on top of.  The convention for the branch name is:

  • br-<year>.<month>-<version>

I will use br-2019.08-2 since my changes represent the second SLAC release of the 2019.08 version of buildroot.

Archiving your buildroot

Don't forget to copy (or move) your entire buildroot-<year>.<month>-<version>-<architecture> to /afs/slac/g/lcls/tftpboot/linuxRT/boot/.

  • No labels