What is this?

We don't have instructions on how to build our Linux RT kernels.  This document is exploratory so that we figure it out.  It's a living doc and represents what we know at the current time.

In looking at old builds, there may be relics for x86 and uclibc but these are deprecated.  I am using buildroot-2019.08-x86_64 as a reference, hoping to glean an understanding.

We have a directory /afs/slac/g/lcls/package/linuxRT (note: and /afs/slac/g/lcls/tftpboot/linuxRT/boot?).  This is our artifact area.  Subdirectory of buildroot-2019.08-x86_64, but I think we should name it buildroot-2019.08 going forward, as it contains multiple architectures (not just x86-64).  Probably x86-64 was the first, and it was simpler to go from there.  We should have architecture subdirectories under that:

  • buildroot-2019.08-x86_64
  • buildroot-2019.08-x86_64-debug (we don't have this, but should we?)
  • buildroot-2019.08-i686 (why???)
  • buildroot-2019.08-zynq (is this necessary?)

Whatever version of buildroot you download defines the version of Linux (and BusyBox, etc.).  Our (SLAC) most recent version of buildroot is http://buildroot.uclibc.org/downloads/buildroot-2019.08.tar.bz2

For building, follow the steps at SLAC's buildroot-site.  There is a README.md.  There are a couple scripts that perform the magic: creating .config for buildroot, linux, and busybox; creating the rootfs.  We need to maintain these scripts as we go forward.  New features will need to be added to (or declined from) our future kernels.

A couple issues to be aware of:

  • You can't build this on an NFS mount like AFS.  Use the /scratch directory on aird-b50-srv01.  When you are done, copy artifacts to the linuxRT directory.
  • Sometimes the download of a package fails, and after the failure the build continues on as if the download succeeded.  The failure might manifest itself much later on, like maybe the local compiler is used rather than the (supposed) downloaded compiler.
  • The SLAC instructions say to symlink site-top to ../site-top, but you should symlink site to ../site-top.
  • BR2_TARGET_GENERIC_PASSWD_ and BR2_TARGET_GENERIC_ROOT_PASSWD are the prefixes to search for in buildroot to set the default root password
    • $5$ is sha256, use sha256sum to generate the hash 
    • $5$5494acf3aae3ebe1019b1e9965534027c279d802eaa4f3cd3f981f831caf886b
    • saved to buildroot-2019.08-x86_64/.config
  • busybox config is at buildroot-2019.08-x86_64/output/build/busybox-1.31.0/.config
  • pxe file is at /afs/slac/g/lcls/tftpboot/linuxRT/boot/ipxe/cpu-b084-sp17.ipxe
    • iocConsole cpu-b084-sp17
    • telnet cpu-b084-sp17
      • reboot -f


https://github.com/slaclab/buildroot-site/blob/br-2019.08/doc/README.buildroot.modify_config

https://github.com/slaclab/buildroot-site

https://buildroot.org/downloads/manual/manual.html#_configuration_of_other_components


Building components:

  • make menuconfig           (buildroot)
  • make linux-menuconfig     (kernel)
  • make busybox-menuconfig   (busybox)


$ passwd -a sha256

$5$cUbknFhsjO1AdUNM$SQPkWyiRotZc2FHPJf3Syyc30i0Egtyc7NVWhPnI.h6


This script may be removing users and re-adding them:

/scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/support/scripts/mkusers

No. fs/common.mk creates the full_user_table.txt, and root is not allowed to be added.


./output/images/rootfs.ext2 contains correct password for root.

When successful su:

pid 7675's new scheduling priority: 84
Starting scanner at  /afs/slac.stanford.edu/g/controls/development/users/ernesto/gitSandBox/SharedPlatform/SharedPlatform-git/bin/linuxRT-x86_64/scanner using /afs/slac/g/lcls/epics/iocCommon/cpu-b084-sp17/iocSpecificRelease/cpuBoot/dev/cpu-b084-sp17/scanner.xml
[   30.150036] EtherCAT: Requesting master 0...
[   30.154807] EtherCAT: Successfully requested master 0.
[   30.188869] EtherCAT 0: Domain0: Logical address 0x00000000, 42 byte, expected working counter 6.
[   30.198783] EtherCAT 0:   Datagram domain0-0-main: Logical offset 0x00000000, 42 byte, type LRD.
[   30.209628] EtherCAT 0: Master thread exited.
[   30.214493] EtherCAT 0: Starting EtherCAT-OP thread.
Password: 
pid 7705's current scheduling policy: SCHED_OTHER
pid 7705's current scheduling priority: 0
pid 7705's new scheduling policy: SCHED_FIFO
pid 7705's new scheduling priority: 45
Starting caRepeater at /afs/slac.stanford.edu/g/controls/development/users/ernesto/gitSandBox/SharedPlatform/SharedPlatform-git/bin/linuxRT-x86_64/caRepeater on port 5067                                                                      
caRepeater successfully started from laci

When unsuccessful su:

[   30.193690] EtherCAT 0: Master thread exited.
[   30.198556] EtherCAT 0: Starting EtherCAT-OP thread.
Password: 
su: incorrect password
Starting caRepeater at /afs/slac.stanford.edu/g/controls/development/users/ernesto/gitSandBox/SharedPlatform/SharedPlatform-git/bin/linuxRT-x86_64/caRepeater on port 5067                                                                      
caRepeater successfully started from laci
Exiting cpuStart

How-to Setup EtherCAT for LinuxRT

/afs/slac/g/lcls/epics/iocCommon

su laci -c '$(which scannerStartup.sh)

Ethercat drivers loaded in common/scannerModule.sh

su chrt called in common/scannerStartup.sh

put the fix in /afs/slac/g/lcls/epics/iocCommon/cpu-b084-sp17/iocSpecificRelease/cpuBoot/dev/cpu-b084-sp17/st.cmd

Presentation

  • How to use Buildroot.
  • How to configure it with SLAC's standards.
  • Problems with booting with root password.
    • DES has been deprecated
    • buildroot config options don't work
  • Your concerns (example: the downloading of packages).
    • downloading of old packages by buildroot
    • need curators for documentation
    • scripts run as root (st.cmd) and the scripts are writeable by everybody
    • buildroot-slac doesn't have a master branch, just a bunch of branches
    • can't find old buildroot on their website
    • not clear which scripts run at boot; need to print headers and footers
    • scripts use variables which may not be defined; bad for su
    • should use CONFIG_LOCALVERSION 
    • have a better name than buildroot
  • Suggestion about how to maintain and document it so we don't need to reinvent the wheel in the future.

Network driver

i40e driver patch
https://lore.kernel.org/lkml/20230223150702.2802683-1-ivecera@redhat.com/T/

We are using i40e driver version:

   49 #define DRV_VERSION_MAJOR 2
   50 #define DRV_VERSION_MINOR 1
   51 #define DRV_VERSION_BUILD 14

Intel's website has available:

   44 #define DRV_VERSION_MAJOR 2
   45 #define DRV_VERSION_MINOR 23
   46 #define DRV_VERSION_BUILD 17
   47 #define DRV_VERSION_SUBBUILD 0

https://www.intel.com/content/www/us/en/products/details/ethernet/700-controllers/xl710-controllers/downloads.html

https://downloadmirror.intel.com/787127/Release_28.2.1.zip

https://downloadmirror.intel.com/786085/i40e-2.23.17.tar.gz

i40e driver

ethernet controller XL710 

$ make linux-menuconfig

<M>   Intel(R) Ethernet Controller XL710 Family support

Test fix on cpu-b15-sp02 SKY8101

$ source /afs/slac/g/reseng/IPMC/env.sh
$ ipmiCmd -c cpu-b15-sp02 -f off
$ ipmiCmd -c cpu-b15-sp02 -f on

[ laci@cpu-b15-sp02]$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
3: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 74:fe:48:74:b0:f0 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 74:fe:48:74:b0:f1 brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 74:fe:48:74:b0:f2 brd ff:ff:ff:ff:ff:ff
6: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 74:fe:48:74:b0:f3 brd ff:ff:ff:ff:ff:ff
7: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
    link/ether 74:fe:48:6a:19:5c brd ff:ff:ff:ff:ff:ff
8: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
    link/ether 74:fe:48:6a:19:5d brd ff:ff:ff:ff:ff:ff
9: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 74:fe:48:6a:19:5e brd ff:ff:ff:ff:ff:ff
10: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 7a:e1:05:41:9f:0b brd ff:ff:ff:ff:ff:ff

b5:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 09)
        Subsystem: Intel Corporation Device 0000
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 79
        Region 0: Memory at fa000000 (64-bit, prefetchable) [size=16M]
        Region 3: Memory at fb008000 (64-bit, prefetchable) [size=32K]
        Expansion ROM at fbd80000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00001000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
              LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [e0] Vital Product Data
                Product Name: Example VPD
                Read-only fields:
                        [V0] Vendor specific: 
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number 5d-19-6a-ff-ff-48-fe-74
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 272, stride: 1, Device ID: 37cd
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
                Region 3: Memory at 0000000000000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1a0 v1] Transaction Processing Hints
              Device specific mode supported
                No steering table available
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: i40e

b5:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 09)
        Subsystem: Intel Corporation Device 0000
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 79
        Region 0: Memory at f9000000 (64-bit, prefetchable) [size=16M]
        Region 3: Memory at fb000000 (64-bit, prefetchable) [size=32K]
        Expansion ROM at fbd00000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00001000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [e0] Vital Product Data
                Product Name: Example VPD
                Read-only fields:
                        [V0] Vendor specific: 
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number 5d-19-6a-ff-ff-48-fe-74
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
              IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
                VF offset: 335, stride: 1, Device ID: 37cd
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
                Region 3: Memory at 0000000000000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1a0 v1] Transaction Processing Hints
                Device specific mode supported
                No steering table available
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: i40e

[ root@cpu-b15-sp02]$ ethtool -i eth5
driver: i40e
version: 2.1.14-k
firmware-version: 3.31 0x80000ca6 1.1681.0
expansion-rom-version: 
bus-info: 0000:b5:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

[ root@cpu-b15-sp02]$ ethtool -i eth6
driver: i40e
version: 2.1.14-k
firmware-version: 3.31 0x80000ca6 1.1681.0
expansion-rom-version: 
bus-info: 0000:b5:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


Adding a package which is a kernel module:

https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_packages_building_kernel_modules
https://buildroot.org/downloads/manual/adding-packages-kernel-module.txt
https://stackoverflow.com/questions/40307328/how-to-add-a-linux-kernel-driver-module-as-a-buildroot-package

To rebuild a single package, remove its directory from output/build and make again.

This is the command buildroot is using to make:

/usr/bin/make
 -j33
 PATH="
  /scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin:
  /scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/sbin:
  /u/gu/egumtow/bin:/u/gu/egumtow/scratch/tools/anaconda3/condabin:
  /afs/slac/g/lcls/package/java/jdk-11.0.2/bin:
  /afs/slac/g/lcls/package/cmake/3.12.2/rhel6-x86_64/bin:
  /opt/google/chrome:
  /afs/slac/g/lcls/package/redis/redis-3.0.0/bin:
  /afs/slac/g/lcls/package/maven/3.6.0/bin:
  /afs/slac/g/lcls/package/java/jdk-11.0.2/bin:
  /afs/slac/g/lcls/epics/R3.15.5-1.1/modules/pvAccessCPP/R6.1.0-0.1.0/bin/rhel7-x86_64:
  /afs/slac/g/lcls/package/IPMC/bin/x86_64-linux-dbg:
  /afs/slac/g/lcls/tools/script:
  /afs/slac/g/lcls/epics/extensions/R0.5.0/bin/rhel7-x86_64:
  /afs/slac/g/lcls/epics/base/R3.15.5-1.1/bin/rhel7-x86_64:
  /afs/slac/g/lcls/package/git/2.18.0/rhel6-x86_64/bin:
  /usr/local/bin:
  /usr/bin:
  /usr/local/sbin:
  /usr/sbin:
  /afs/slac/g/lcls/package/eclipse/4.6"
 AR="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-ar"
 AS="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-as"
 LD="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-ld"
 NM="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-nm"
 CC="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-gcc"
 GCC="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-gcc"
 CPP="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-cpp"
 CXX="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-g++"
 FC="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-gfortran"
 F77="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-gfortran"
 RANLIB="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-ranlib"
 READELF="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-readelf"
 STRIP="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-strip"
 OBJCOPY="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-objcopy"
 OBJDUMP="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-objdump"
 AR_FOR_BUILD="/usr/bin/ar" AS_FOR_BUILD="/usr/bin/as"
 CC_FOR_BUILD="/usr/bin/gcc" GCC_FOR_BUILD="/usr/bin/gcc"
 CXX_FOR_BUILD="/usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld"
 CPPFLAGS_FOR_BUILD="-I/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/include"
 CFLAGS_FOR_BUILD="-O2 -I/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/include"
 CXXFLAGS_FOR_BUILD="-O2 -I/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/include"
 LDFLAGS_FOR_BUILD="-L/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/lib -Wl,-rpath,/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/lib"
 FCFLAGS_FOR_BUILD=""
 DEFAULT_ASSEMBLER="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-as"
 DEFAULT_LINKER="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-ld"
 CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64"
 CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g2  -Wno-stringop-truncation -Wno-format-truncation"
 CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g2  -Wno-stringop-truncation -Wno-format-truncation"
 LDFLAGS=""
 FCFLAGS=" -O2 -g2"
 FFLAGS=" -O2 -g2"
 PKG_CONFIG="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/pkg-config"
 STAGING_DIR="/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/x86_64-buildroot-linux-gnu/sysroot"
 INTLTOOL_PERL=/usr/bin/perl
 -C /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src
 noisy

This is the command buildroot uses for linking:

/scratch/egumtow/buildroot/buildroot-2019.08/host/linux-x86_64/x86_64/bin/x86_64-buildroot-linux-gnu-ld   
 -r
 -o /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e.o
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_main.o
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_ethtool.o
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_xsk.o
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_adminq.o
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_common.o
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_hmc.o
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_lan_hmc.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_nvm.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_debugfs.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_diag.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_txrx.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_ptp.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_filters.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_ddp.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_client.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_virtchnl_pf.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_dcb.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/i40e_dcb_nl.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/kcompat.o 
 /scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/intel-i40e-2.23.17/src/kcompat_vfd.o

27465 open("/lib64/libelf.so.1", O_RDONLY|O_CLOEXEC) = 3   – shelving this for now.  Seems like a dead-end because I can't compile i40e_main.c by itself outside buildroot.

The driver makefile need to define where the kernel source exists.  Define it for buildroot, then we get a compile error because a symbol is missing in the kernel source.  We need to define CONFIG_PCI_ATS, which is defined only when CONFIG_PCI_IOV is defined.  So I am changing in linux-menuconfig.

KSRC=/scratch/egumtow/buildroot/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/linux-4.14.139

Which then requires CONFIG_DCB

The issue is in the RT patch.

  │ BR2_LINUX_KERNEL_PATCH:
  │
  │ A space-separated list of patches to apply to the
  │ kernel. Each patch can be described as an URL, a local file
  │ path, or a directory. In the case of a directory, all files
  │ matching *.patch in the directory will be applied.
  │
  │ Symbol: BR2_LINUX_KERNEL_PATCH [=http://www.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.139-rt66.patch.gz $(BR2_GLOBAL_PATCH_DIR)linux/$(BR2_LINUX_KERNEL_VERSION)/*.diff]

Menuconfigs

  • buildroot 2016
    • bootloaders
      • syslinux
      • install pxelinux
      • modules to install - ldlinux.c32

Meeting 9/26/2023

Attendees:

  • Marcio - yes
  • Ernest - yes
  • Kristi Luchini - yes
  • Jeremy Mock - yes
  • An Le - yes
  • Ken Brobeck - yes
  • Garth Brown - yes
  • Debbie Rogind - 
  • Mike Zelazny - jury duty

Issues:

  • security issues (2) - telnetd is running and root is passwordless in our buildroot Linux RT
  • when adding a root password, boot hangs because "su" is used with "chrt" command
  • we can add the "chrt" to the root boot sequence (st.cmd), but we need to coordinate the change before upgrading buildroot version
    • scannerStartup.sh has the "su root" reference in /afs/slac/g/lcls/epics/iocCommon/common/
    • we can't remove the "chrt" from scannerStartup.sh until all buildroots are able to boot without asking for the root password

Topics:

  • best way to migrate all the servers to this new Linux RT release?
    • most urgent issue is in dev environment - everybody agrees
    • then we can move to production
    • we need a complete list of Linux RT machines running the insecure version of buildroot - Kristi has a list and will send it to me
    • Eric to send link to doc in the meeting minutes
    • backwards compatible?
    • which machines are calling scannerStartup.sh?
  • releases other than Linux RT?
    • CentOS
  • additional users (besides laci)?
    • are the users exclusive or can we have multiple users for a single buildroot for FACET, LCLS, and dev?
      • add all three users to buildroot  (laci (lcls), flaci (facet), acctf (testfac))
    • passwords?
  • do we need root (or su) in the boot process anywhere else?
    • nfsMount.sh
    • DHCP?
    • TCP/IP?
    • load gen3/4 frame grabber kernel modules
  • now that we have a handle on buildroot, should we revisit fixes and features we deferred?
  • are all Linux RT machines on the most recent buildroot version?  or are some on an older version?
  • other issues?
    • root SSH keys - future issue
  • how to test?
    • can the team provide test machines and test on them if I provide the buildroot?
    • contact An Le

CATERs:

  • 165458
  • 164073 – FACET
  • 164458 – LCLS
  • 165686 – DEV
  • 165817 - CentOS 7

Action items:

  1. Krist:  iSend list of IOC Applications with cpu’s that start dhcp, ethercat and pgpCard3 and 4. – Done
  2. Kristi: Send the cpu List for all facilities – Done
  3. An:   talk with Jeremy, Shawn  and Namrata today to compile a list of test-stand (linuxRT cpu’s), where can test the new buildroot, and send this list to Eric.
  4. An:   test an script changes to $EPICS_IOCS/<common,facility> or the application cpuBoot/<faciliy>/<cpu>/st.cmd files,  will be tested with the new buildroot and old buildroot. I’ve also ask An to create a cater today
  5. Eric:  Add flaci, acctf accounts to buildroot, and rsh keys for laci,flaci, acctf so we can ssh to root
  6. Eric : Add these changes to the buildroot for buildroot-2019.08-<xxx>, where xxx is
    1. x86_64
    2. i686
  7. An:   Add 4 caters one each for LCLS, Facet, Test-Facility, Dev  to reboot the remaining linuxRT cpu’s that don’t have the root -f issue. Make a job cpu’s in each application. The plan would be to reboot all cpu’s during the November downtime.   -- Done
  8. Kristi:  Submit at cater for Eric to generated a new buildroot for centOS7 with the same changes he implemented in the linuxRT buildroot – Done    cater 165817 
    1. Add sw jobs for dev and facet cpu’s
  9. An:  Repeat tests for centos7 in dev and coordinate with Eric.

Meeting  

Attendees:

  • Marcio - yes
  • Ernest - yes
  • Kristi Luchini - yes
  • Jeremy Mock - yes
  • An Le - yes
  • Ken Brobeck - yes
  • Garth Brown - yes
  • Debbie Rogind - no (ICALEPCS)
  • Mike Zelazny - no (busy)
  • Kukhee - no

Topics:

  • Linux RT testing update from An
    • Have not tested on PGP gen 3 card - waiting for Shawn Alberson
      • currently testing with gen 4
    • iocConsole.sh - can't enabled core dumps
      • -f enables core dumps
      • decision: everyone who needs core dumps should already know the root password
        • worried that people would write the password on paper 
        • Ken: or public/private key exchange for
          • ask Ken how to enable certain commands that can be run as root without being root
    • ProfileMonitorAD issue
      • VEVR created as part of CPU startup 
      • irq.*pci-evrm thread exist when the CPU boots up and the driver is loaded
        • if it does exist, then no issue
        • if it doesn't exist, then we're running as a non-root user - how to change the thread priority?
          • unless we can give non-root users the capability - see man page
  • CentOS 7 diskless boot update from Eric
    • test the additional users
    • share with An
      • needs FACET server
  • Use cpu-b15-ky01 to test
    • yes, we can use it to test, on Kristi's list
  • When can I commit the code for Linux RT and CentOS 7 diskless boot?
    • quick final meeting after everything has been tested
    • downtime is November 15 - let's update then

Action items:

  • Eric - share CentOS 7 with An
  • An - perform CentOS 7 testing
  • Marcio - check irq.*pci-evrm thread after CPU boot
  • Eric - call meeting when testing done
  • Eric - commit code / roll new releases

Meeting November 15, 2023

Attendees:

  • Marcio - yes
  • Ernest
  • Kristi - yes
  • Jeremy Mock
  • An Le - yes
  • Ken Brobeck - yes
  • Garth Brown - yes
  • Debbie Rogind - yes
  • Mike Zelazny - yes
  • Shawn Alverson - yes
  • Mike Puckett - yes
  • Jerry Katzung - yes

Topics:

  • Previous action items:
    • Testing status - An, Shawn - 
    • thread IRQ - Marcio
  • Buildroot 2016
  • Time frames

Notes:

Action items:

  • Update Jira - Eric, ???
  • investigate /etc/SLAC_properties - Eric
  • investigate thread IRQ - Marcio
  • schedule meeting for Monday - Eric
  • talk about systemd journalctl output - Mike Puckett
  • busybox menuconfig for dropbear SSH root login allowed - ???

Meeting November 20, 2023

Attendees:

  • Marcio - yes
  • Ernest -
  • Kristi - yes
  • Jeremy Mock - 
  • An Le - yes
  • Ken Brobeck - 
  • Garth Brown - 
  • Debbie Rogind - 
  • Mike Zelazny - yes
  • Shawn Alverson - declined
  • Mike Puckett - yes
  • Jerry Katzung - yes 
  • Jeremy Lorelli - yes

Topics:

  • thread IRQs - Jeremy
    • EVRM
  • root skeleton - Eric
    • CentOS 7 - yes
    • Linux RT - more complicated
    • name=value pairs, HAVE_ROOT_PASSWORD=1
  • journalctl output - Mike Puckett
  • SSH root login allowed - want consistency
    • be pushed to a January PAMM?

  • buidroot-2016 - estimate work

    • only x86-64

Action items:

  • Jeremy & Jeremy coordinate to test and think about a solution
    • rtPrioritySetup.cmd
    • irq-*pci-evrm
  • An & Mike Puckett coordinate for journalctl
    • not a high priority item
  • SSH root login
    • not a high priority item
    • Eric, Mike P, and An coordinate
  • buildroot-2016
    • ask Sonya Hoobler if "reboot" crashes - Sonya says no
    • if no reboot issue, just the root password and telnet server needed

Meeting November 29, 2023

Attendees:

  • Marcio - yes
  • Ernest -
  • Kristi - yes
  • Jeremy Mock - yes
  • An Le - yes
  • Ken Brobeck - 
  • Garth Brown - 
  • Debbie Rogind - 
  • Mike Zelazny - 
  • Shawn Alverson - yes
  • Mike Puckett - yes
  • Jerry Katzung - yes
  • Jeremy Lorelli - yes

Topics:

  • thread IRQs for EVRM
  • journalctl
    • punt until January
  • SSH root login
    • punt
  • buildroot-2016
    • not as easy as hoped; will require investigation
    • have ready first week of 2024
  • ethercat
    • call scannerStartup as root, open up (su) screen session as regular user

Action items:

  • DONE - Eric - find email with possible chrt solution
    • in /etc/sudoers file 

      • laci  ALL= NOPASSWD:  /usr/bin/chrt

  • DONE - Marcio - ask Kukhee about EVRM threads
    • find parent PID of these IRQ threads; if 0 then it's a kernel thread
    • Kukhee said that those calls related to EVRM threads can be made during the CPU boot and don't need to be part of the IOC st.cmd.
  • DONE - Eric - investigate buildroot 2016, scope out work

Meeting January 4, 2024

Attendees:

  • Marcio - no
  • Ernest - no
  • Kristi - yes
  • Jeremy Mock - no 
  • An Le - yes
  • Ken Brobeck - no
  • Garth Brown - no
  • Debbie Rogind - no
  • Mike Zelazny - no
  • Shawn Alverson - no
  • Mike Puckett - yes
  • Jerry Katzung - yes
  • Jeremy Lorelli - no

Topics:

  • Install status
    • 2016 buildroot - Eric tested root login successful
    • 2019 buildroot - Kristi will test today, testfac group
    • CentOS 7 - can boot in the superconducting area, this is 2016 - Mike/An can test in dev
    • Eric - copy images to tftpboot
    • Eric - schedule meeting on Monday

Action items:


Meeting January 8, 2024

Attendees:

  • Marcio - yes
  • Ernest - no
  • Kristi - yes
  • Jeremy Mock - yes
  • An Le - yes
  • Ken Brobeck - no
  • Garth Brown - no
  • Debbie Rogind - no
  • Mike Zelazny - no
  • Shawn Alverson - yes
  • Mike Puckett - yes
  • Jerry Katzung - yes
  • Jeremy Lorelli - no

Topics:

  • acctest GID - 2459 is correct , acctf UID - 11846
    • for CentOS 7 - nuke output directory, rebuild, and copy it over to boot directory
  • don't copy to boot directory, just let team know where it is
  • Eric - update buildroot instructions for 2016
  • Kristi - Talk with Shantha about the spear account in CentOS 7
    • Eric - add spear account CentOS 7
  • No labels