Created by Shantha Condamoor on 15-Oct-2014
----------------------------------------------------------------------
While linuxRT is a relatively established RTOS at ICD, it has mostly been used only in conjunction with the uTCA platform.
Recently linuxRT has been 'ported' and running successfully on a few COTS linux servers such as the Dell PowerEdge servers and SuperMicro Industrial PCs. A newer version of linuxRT was built by Till Straumann to support the new hardware.
The important change to the linuxRT RTOS was the addition of support for the Broadcom NIC chipsets which are ethernet network controllers found in these servers.
The other important change was the setup of these servers to boot linuxRT targets as diskless clients.
The BIOS boot order was modified to over-ride the'boot out of hard-disk' or 'CD-ROM' which are the default in most cases, and to perform network booting as the first choice.
Finally, the redirection of the console output to the 'screen' process via the 'iocConsole' script instead of the monitor.
This re-direction to the first serial port, instead of the terminal, was also done in BIOS to enable us to observe the PC boot process remotely like all our IOCs,
As we continue to test this new system and learn more, this document will also evolve.
Here's something to get started with linuxRT using the latest EPICS base.
1. What is 'new' EPICS?
The 'new' version of EPICS is R3-14-12-3_1-0
2. How do I setup my bash shell to use the new EPICS?
From a bash shell in an LCLSDEV host (Ex. lcls-dev2), type the following:
source /afs/slac/g/lcls/epics/setup/go_epics_3-14-12-3_1-0.bash
The above script sets up several EPICS environment variables in your bash shell.
Some variables are paths to ''TOP-level' directories:
EPICS_TOP=/afs/slac/g/lcls/epics/R3-14-12-3_1-0
EPICS_BASE_TOP=/afs/slac/g/lcls/epics/R3-14-12-3_1-0/base
EPICS_MODULES_TOP=/afs/slac/g/lcls/epics/R3-14-12-3_1-0/modules
EPICS_EXTENSIONS=/afs/slac/g/lcls/epics/R3-14-12-3_1-0/extensions/extensions-R3-14-12
EPICS_IOC_TOP=/afs/slac/g/lcls/epics/R3-14-12-3_1-0/iocTop
EPICS_IOC_TOP is the top level application directory for all EPICS iocs that use this EPICS version.
It is here where all new EPICS applications must be created. EPICS_IOC_TOP is the same as the APP variable.
Other environment variables setup in your bash shell are:
IOC=/afs/slac/g/lcls/epics/R3-14-12-3_1-0/iocCommon
$IOC is the directory where the startup scripts for all the iocs are located.
IOC_DATA=/nfs/slac/g/lcls/epics/ioc/data
$IOC_DATA is the data directory. This is where the outputs (and in some case inputs) from iocs are stored.
3. Where are the linuxRT kernel modules?
/afs/slac/g/lcls/package/linuxKernel_Modules
4. Where are the EPICS modules for this new base?
/afs/slac/g/lcls/epics/R3-14-12-3_1-0/modules
A handful of EPICS modules have been built for R3-14-12-3_1-0.
Most of them have been cross-compiled for several targets including the following:
linuxRT-x86
linux-x86
linux-x86_64
5. What do I do if I need a new module or a specific version for an existing module built for the new base?
Contact Murali or Ernest to have them built and installed. Murali has a python script that
automatically discovers the module dependencies and builds them in the correct order for all targets.
6. Where and how do I create a new EPICS application? How do I setup a new IOC to use linuxRT-86 OS?
(a) Create the following directories:
$IOC/ioc-b34-my01
$IOC_DATA/ioc-b34-my01
$APP/Test (look at this as an example)
Now create your EPICS application MyTest as below using standard templates and scripts:
(b) cd $APP/Test
As MyTest development continues, this is where you will be creating/checking out from CVS, several versions of your MyTest application, as you continue to develop.
Now create a sandbox area for MyTest.
mkdir Development
(c) cd Development
You are currently in:
/afs/slac/g/lcls/epics/R3-14-12-3_1-0/iocTop/Test/Development
This is the top directory for MyTest and referred as $(TOP).
(d) Using a perl script that automates the process, create MyTest:
makeBaseApp.pl -t slac MyTest
The perl script had populated your application using icdTemplates module that provides templates
that are used by ICD to support our style of IOC Application Building for the facilities we support.
The location for icdTemplates module was set as below when you sourced go_epics_3-14-12-3_1-0.bash.
EPICS_MBA_TEMPLATE_TOP=/afs/slac/g/lcls/epics/R3-14-12-3_1-0/modules/icdTemplates/icdTemplates-R1-0-1
If you would like to know more about makeBaseApp.pl, type:
makeBaseApp.pl -help
(e) Now, open the configure/RELEASE file.
(i) Notice how the following line appears towards the beginning:
include $(TOP)/RELEASE_SITE
The above sets the paths for EPICS base, modules, packages, tools etc. to point to version R3-14-12-3_1-0.
(ii) The file has the following line towards the end of it:
# Location of Linux Kernel Modules:
LINUX_KERNEL_MODULES=$(PACKAGE_SITE_TOP)/linuxKernel_Modules
(iii) configure/RELEASE file may need some modifications from what was automatically generated:
Change the various module version numbers as needed for your application.
Ensure that the specific versions included in your RELEASE file DO EXIST under the modules directory and are consistent.
They must also have been built for the linuxRT-x86 target. If something is missing, contact Ernest.
For EVENT module, Kukhee recommends us to use event-R4-1-3 or greater with this base.
EVENT_MODULE_VERSION=event-R4-1-3
(f) Open MyTestApp/src/Makefile and notice that the IOC application is referred to as 'MyTest".
(g) Now it is time to add your linuxRT-x86 virtual ioc (vioc) startup scripts. vioc-b34-bd32 is used here as an example.
From $(TOP) type the following command to create a boot directory for 'vioc-b34-bd32':
makeBaseApp.pl -i -t slac vioc-b34-bd32
When prompted to choose target architecture, choose linuxRT-x86.
Two files are modified based on this selection.
(1) vioc-b34-my01/st.cmd
(2) vioc-b34-my01/Makefile.
When prompted with 'Application name?' type 'MyTest'.
This will use the application name MyTest that was created in MyTestApp/src/Makefile as described in step (f).
If you just hit 'Enter' without specifying your application name, then your ioc name vioc-b34-bd32 will be used as the default for the dbd file in the st.cmd
Now iocBoot directory has been created under $(TOP) and underneath it, 'vioc-b34-bd32' has been created.
Open iocBoot/vioc-b34-bd32/Makefile and notice that ARCH is set to linuxRT-x86.
Open iocBoot/vioc-b34-bd32/st.cmd and notice that this script is setup to pick the binary from linuxRT-x86 directory.
(h) If you have an EVR in your system, uncomment the following lines in MyTestApp/src/Makefile:
# MyTest_DBD += evrSupport.dbd
# MyTest_DBD += devMrfEr.dbd
Ensure the following lines are not commented as well:
MyTest_LIBS += evrSupport
MyTest_LIBS += devMrfEr
....
ifeq ($(T_A),linuxRT-x86)
MyTest_LIBS += drvMrf
endif
if you have a PMC EVR230 installed in your system that you'd like to use:
To add support for PCI-express EVR300, also add the following lines to MyTestApp/src/Makefile:
MyTest_DBD += drvUioPciGen.dbd
MyTest_DBD += devBusMapped.dbd
MyTest_DBD += epicspci.dbd
.....
IPCTests_LIBS += drvUioPciGen
IPCTests_LIBS += devBusMapped
IPCTests_LIBS += epicspci
(i) Now 'make' your application from the top level directory $(TOP) to ensure your changes to RELEASE files are good:
There should be no build errors.
Having tested this phase, you can now uninstall the binaries and cleanup the make-generated files using the following command:
make clean uninstall
Replace the macros as needed.
Now cd $APP/MyApp again and 'make' the application again and ensure it builds fine.
Now cleanup the build before committing to CVS as below:
make clean uninstall
(c) Add your application to CVS and commit the source files.
7. What is PXE, DHCP, TFTP and NFS and why are they needed by my linuxRT IOC?
LinuxRT is installed on our system using the Preboot Execution Environment (PXE) method of network booting.
We enable the PXE/network-booting method in the BIOS.
In order to use PXE there is a boot-server that will allow our client system to :
(a) Request an IP address (via DHCP)
(b) Download a kernel (via TFTP)
With both of these services in place any system which supports PXE/network-booting
should be able to gain an IP address, fetch a kernel, and boot without an installed operating system.
PXE uses three distinct network protocols that map to three server processes to perform the installation.
In our case, some of the processes run on lcls-dev1 (LCLSDEV daemon)
Both DHCP and TFTP services run on the LCLSDEV host 'dhcp3' maintained by SCCS.
(a) Dynamic Host Configuration Protocol (DHCP)
PXE uses DHCP to deliver initial network configuration options to client nodes.
The DHCP server supplies the PXE boot plug-in with
(i) IP address
(ii) TFTP server address
(iii) Stage 1 image boot-loader name from which to download and execute the image.
As the supplied PXE installation environments are non-interactive and will unconditionally reinstall a client machine,
we have the client associate its MAC address with a specific OS installation before starting the PXE boot.
The configuration information, in our case, in addition to IP/MAC address, includes a hostname and a pointer to the Master Starupt script in afs for our IOC.
It has an optional root-path variable pointing to the afs area which hosts the boot image that is served via TFTP.
This can be over-ridden as will be seen later.
When the Linux server is rebooted or power-cycled, PXE will attempt the network booting method first
and as a first step it will contact the DHCP server to retrieve the network configuration information.
Hence, every new linuxRT ioc (host) needs to be added to the DHCP server configuration file in afs.
This file is in /afs/slac/service/dhcp-pxe/dhcpd.conf
Note that the DHCP service running on dhcp3 is intended only for booting embedded devices like our linuxRT servers that are connected to the LCLSDEV and SSRL subnets.
The MAC addresses for such devices must be registered in CANDO and assigned fixed IP addresses.
The IP/MAC address of the primary ethernet that will fetch the linuxRT boot image is defined here.
To add a new host to the DHCP configuration, contact Thuy.
After a new ioc is added to dhcpd.conf, the DHCP service must be restarted.
Currently only Thuy, Ernest and a couple of others have permissions to perform this restart on dhcp3.
Here's is an example - ioc-b34-bd32:
host ioc-b34-bd32 {
# SuperMicro (INTELx86)
#
hardware ethernet 00:25:90:D1:95:1E;
fixed-address 134.79.218.190;
option host-name "ioc-b34-bd32";
if ( substring( option vendor-class-identifier, 0, 5 ) = "udhcp" ) {
filename "/afs/slac/g/lcls/epics/iocCommon/ioc-b34-bd32/startup.cmd";
option root-path "afsnfs1:/afs/slac:/afs/slac";
}
}
To find out more about how our linux server boots up linuxRT, click on the following link:
How does the Linux Server boot up linuxRT?
(b) Trivial File Transfer Protocol (TFTP)
PXE uses TFTP that defines a simple UDP protocol for delivering files over a network.
PXE delivers kernels and initial bootstrap software to client nodes using TFTP.
tftpboot is mounted in afs area in LCLSDEV on the server dhcp3, which runs both the TFTP an DHCP services:
/tftpboot -> /afs/slac.stanford.edu/service/dhcp-pxe/tftpboot/
The iocs retrieve the linuxRT boot image from the TFTP server from the following location:
/afs/slac/g/lcls/tftpboot/linuxRT/boot
In this location, there are several linuxRT-x86 bootimages.
These were custom-built by T.Straumann for the various Linux Servers/IPCs that we currently have setup to boot with linuxRT OS.
Of these images, '3.14.12-rt9' is the latest and it has in-built support for the
Broadcom networking ethernet chipset that are used in our dev Poweredge Dell Servers.
(c) Network File System (NFS)
The NFS service is used by the installation kernel to read all of the packages necessary to the installation process.
NFS services run on the LCLSDEV hosts afsnfs1 and afsnfs2 maintained by SCCS.
This service makes available the boot directory to all linuxRT targets that boot as diskless clients.
All clients have read-only permissions to this directory.
The linuxRT iocs need some additional NFS Mount Points to write their data some where.
surrey04b is an NFS Appliance and is used by iocs for data and have both read and write permissions to the data directory ($IOC_DATA).
In LCLSDEV, we must obtain permissions for the iocs to write to the $IOC_DATA directory.
Fill out an online form provided by SCCS to obtain permissions for your ioc to write to this directory:
https://www-rt.slac.stanford.edu/rt3/SelfService/Forms/IocNfs.html
8. How do I start my IOC? Where is my ioc's statrup.cmd?
There are a few scripts that automate this process.
To begin with, there is the 'ipxe.ini' script in the tftp boot area /afs/slac/g/lcls/tftpboot/linuxRT/boot that PXE will run.
This is where the version of (linuxRT) kernel to run is specified as follows:
set vers 3.2.13-121108
This version number can be over-ridden by a chained, host-specific pxe init script to load an image different from the above:
chain ${hostname}.ipxe ||
For example, we have defined a script specifically for our ioc ioc-b34-bd32.ipxe which chooses to load the latest linuxRT image:
set vers 3.14.12-rt9
This is also the place to over-ride the 'root-path' option specified in the DHCP configuration file dhcpd.conf.
For example, I may decide to over-ride the afsnfs1 server and instead choose to get my boot image from afsnfs2 server:
set extra-args ROOTPATH=afsnfs2:/afs/slac:/afs/slac BOOTFILE=/afs/slac/g/lcls/epics/iocCommon/ioc-b34-bd32/startup.cmd
A few more extra arguments are specified in ioc-b34-bd32. Leave them as they are.
The 'ipxe.ini' script loads the linuxRT kernel via the TFTP protocol:
kernel --name linux tftp://${next-server}/linuxRT/boot/${vers}/bzImage && initrd tftp://${next-server}/linuxRT/boot/${vers}/rootfs.ext2 || shell
imgargs linux debug idle=halt root=/dev/ram0 console=ttyS0,115200 BOOTIF_MAC=${net0/mac:hex} ${extra-args} || boot || shell
After linuxRT boot image is downloaded to the target and linuxRT starts to run, additional nfs mounts will be done.
The afs to nfs translator service makes available the directory structure, to all clients that have mounted this nfs space.
NFS File Servers for LCLSDEV are afsnfs1 and afsnfs2
One of the arguments to the kernel process is the location of the BOOTFILE that does the mounting.
The 'filename' argument (which can be over-ridden by the BOOTFILE argument for linuxRT) is as follows:
"/afs/slac/g/lcls/epics/iocCommon/ioc-b34-bd32/startup.cmd"
This script is similar to and modelled after RTEMS startup.cmd.
When linuxRT loads and start, the kernel process is run as "root" user.
Hence it has permissions to setup the nfs mounts which is done by the following line in startup.cmd:
/afs/slac/g/lcls/epics/iocCommon/All/Dev/linuxRT_nfs.cmd
Additional NFS Mount Points for linuxRT pertaining to the ioc data directory $IOC_DATA are mounted as well
9. How do I monitor the server remotely as it boots up - after a power-cycle or after a 'reboot' command?
Thuy has setup the BIOS of the Dell Linux Servers and other servers used for linuxRT development, to re-direct the
console (monitor) output to one of the serial ports. This allows us to watch the boot process remotely via our standard 'iocConsole'
python script. iocConsole uses the 'screen' process to accomplish this re-direction.
Type the following command from any LCLSDEV host:
Example:
iocConsole ioc-b34-bd32
This will establish a serial connection with the linux box from lcls-dev1 via a DIGI Terminal Server port as follows:
: ssh -x -t -l laci lcls-dev1 bash -l -c " pyiocscreen.py -t HIOC ioc-b34-bd32 ts-lclsdev05 2001 "
You can monitor the linuxRT as it goes through the PXE network booting process.
Finally, you will get the login prompt:
Welcome to Buildroot
ioc-b34-bd32 login:
Login as 'root'. No password is required.
At the shell, type the following to ensure that you are running the correct version of liunxRT for your ioc. This has the real-time Preempt RT patch.
# uname -a
Linux ioc-b34-pm32 3.14.12-rt9 #1 SMP PREEMPT RT Sat Oct 11 17:27:39 PDT 2014 i686 GNU/Linux
To see the PCI devices in your system type the following command:
# lspci
10. What are kernel modules and how are they loaded in linuxRT?
In the linux world, kernel modules are pieces of code that can be loaded and unloaded into the kernel upon demand.
They extend the functionality of the kernel without the need to reboot the system.
For example, one type of module is the device driver, which allows the kernel to access hardware connected to the system.
Without modules, we would have to build monolithic kernels and add new functionality directly into the kernel image.
Besides having larger kernels, this has the disadvantage of requiring us to rebuild and reboot the kernel every time we want new functionality.
linuxRT too, lets you load and unload kernel modules dynamically.
Now we are ready to load some kernel modules essential to our ioc - like EVR.
In our 'startup.cmd' script, we have the following line which lets us customize and load our kernel modules:
/afs/slac/g/lcls/epics/iocCommon/ioc-b34-bd32/kernel-modules.cmd
The location for kernel modules is specified as an environment variable in linuxRT:
KERNEL_DRIVER_HOME=/afs/slac/g/lcls/package/linuxKernel_Modules
There are several linuxRT drivers in this directory.
The PCI EVR230 driver is here. The following driver version has been built for the latest linuxRT 3.14.12-rt9:
EVR230_PCI_EVR_DRIVER=$KERNEL_DRIVER_HOME/pci_mrfevr_linuxRT/buildroot-2014.08
The PCI Express EVR300 driver built for linuxRT 3.14.12-rt9, is here:
EVR300_PCI_EVR_DRIVER=$KERNEL_DRIVER_HOME/pci_mrfev300_linuxRT/buildroot-2014.08
The kernel drivers are installed (loaded) dynamically as follows:
# Load the MRF EVR230 Kernel Module for timing
insmod $EVR230_PCI_EVR_DRIVER/pci_mrfevr.ko
$EVR230_PCI_EVR_DRIVER/module_load
# Load the MRF EVR300 Kernel Module for timing
insmod $EVR300_PCI_EVR_DRIVER/pci_mrfevr300.ko
$EVR300_PCI_EVR_DRIVER/module_load
There are a couple of things to note:
1. Currently the EVR kernel modules SW has the restriction that if there are both a PMC EVR230 and a PCI-e EVR300 in a linux box,
then the PMC EVR230 MUST BE initialized as card 0 and loaded first. EVR300 must be initialized as card 1.
Additionally, due to hard-coded device names in the module, it is essential to setup the following links:
ln -s /dev/er3a0 /dev/erb0
ln -s /dev/er3a1 /dev/erb1
ln -s /dev/er3a2 /dev/erb2
ln -s /dev/er3a3 /dev/erb3
2. If only one EVR (either PMC EVR230 or PCI EVR300) is installed in your system, then the above restriction does not apply and soft links are not needed.
Take a look at the following script:
/afs/slac/g/lcls/package/linuxKernel_Modules/pci_mrfev300_linuxRT/buildroot-2014.08/module_load
Notice how kernel modules are loaded as device drivers under the /dev/ in linuxRT much like linux.
3. The Broadcom Ethernet NIC driver used to be a separate kernel module and its dirver was loaded dynamically via ;modprobe tg3' in this script.
With the latest linuxRT version 3.14.12-rt9, this step has become unnecessary as the driver has become part of this linuxRT boot image.
4. The SIS digitizers for uTCA, loads their device drivers in 'kernel-modules.cmd':
SIS8300_DRIVER=$KERNEL_DRIVER_HOME/sis8300drv/MAIN_TRUNK
modprobe uio
insmod $SIS8300_DRIVER/sis8300drv.ko
Please note that as of date, SIS8300 has NOT been rebuilt for the latest linuxRT 3.14.12-rt9.
It is currently unsupported for the Poweredge Dell servers.
11. How do I create a startup script for my ioc?
The third and final script is specifically to setup and start your EPICS ioc.
The 'startupConsole-laci-rt.cmd' lets you start your EPICS-based 'virtual' ioc as a foreground process in your host.
Your EPICS ioc must run as a real-time process. It must lock the kernel in memory under linuxRT.
The following command in your 'startupConsole-laci-rt.cmd' does that:
ulimit -l unlimited
The following line is also needed to run the ioc with real-time priorities:
ulimit -r unlimited
Finally, you will be running your 'virtual' ioc as a specific user called as 'laci' who has permissions to run this ioc.
Setup the permissions for this user 'laci':
umask 002
Now you are ready to start your IOC and have it run as a foreground process.
Create a directory called as 'vioc-b34-my01' for your 'virtual' ioc process under the following directory:
$IOC/ioc-b34-my01
cd $IOC/ioc-b34-my01/vioc-b34-my01
Set up a soft link to the 'bin' directory of your IOC app that you created in step (6):
ln -s /afs/slac/g/lcls/epics/R3-14-12-3_1-0/iocTop/MyApp/bin/linuxRT-x86 bin
In the same directory $IOC/ioc-b34-my01/vioc-b34-my01, add a startup script 'iocStartup.cmd' for vioc-b34-my01:
This script is very similar to the Soft IOC startup scripts that we are familiar with.
It setups some env variables used by all iocs, then changes to the ioc boot directory and starts the st.cmd file.
12. How do I start my ioc and monitor it?
Example, from any lcls-dev host ssh to your ioc as 'laci':
ssh laci@ioc-b34-bd32
$ cd /afs/slac/g/lcls/epics/iocCommon/ioc-b34-bd32
Ensure that 'startupConsole-laci-rt.cmd' is in your current directory.
Start the virtual ioc as a foreground process:
./startupConsole-laci-rt.cmd
.
