Page History

3. Where are the linuxRT kernels and kernel modules?

Currently there are several versions of linux kernels that have been built for several host architectures.

They are all here under BUIDLROOT_HOME: /afs/slac/g/lcls/package/linuxRT/buildroot-2014.11

The tool chain used to build these kernels is 'buildroot' and version 2014.11 was used to build these kernels.

These kernels are available for 32 bit (linuxRT_glibc-x86_64) and 64-bit (linuxRT_glibc-x86_64_64) host architectures.

Further, they have been built with either uClibc (C library for embedded LInux) or glibc (GNU C LIbrary).

uClibc or glibc are libraries which defines "system calls" and other basic facilities such as open, malloc, printf, exit etc.

Based on which libraries the kernels have been built with, currently there are three options for linuxRT kernels to choose from:

buildroot-uclibc-x86 (for 32-bit i680 architectures with uClibc)

buildroot-uclibc-x86_64 (for 64-bit x86_64 architectures with uClibc)

buildroot-glibc-x86_64 (For 64-bit x86_64 architectures with glibc)

uClibc will be phased out in the near future - hence it is recommended to use glibc for new applications.

However, for our test application, we use 32-bit kernel built with uClibc. This is for demonstration purposes only.

The various kernel modules built for the various kernels are all here:

 /afs/slac/g/lcls/package/linuxKernel_Modules

Example: EVR kernel module is pci_mrfevr_linuxRT

Under this directory, there are three different kernel modules - each built for the corresponding kernels as described above.

Ensure that you use the right version of the kernel module that matches with your kernel.

6. Where and how do I create a new EPICS application?  How do I setup a new IOC to use linuxRT-86 OS?  

(a) Create directories as shown below using cpu-b34-bd32 as an example:

  $IOC/cpu-b34-bd32

  $IOC_DATA/cpu-b34-bd32

  $IOC_DATA/vioc-b34-bd32

  $IOC_DATA/vioc-b34-bd32/autosave

  $IOC_DATA/vioc-b34-bd32/autosave-req

  $IOC_DATA/vioc-b34-bd32/iocInfo

TO DO list: Generate a script to create above directories automatically via a single python or perl script.

Note: this script is available from Ken Brobeck  on softegr@lcls-srv01:/usr/local/admin/scripts/setup-ioc-dir

$APP/MyTest  (look at this as an example)  

  Now create an EPICS application in your home directory called, 'MyTest'  , using standard templates and scripts in the above example. 
(b)  In your home directory, create a sandbox area called 'MyTest'   , which will contain your EPICS IOC application.

cd

mkdir MyTest

(c) cd MyTest

You are currently in:

 ~/MyTest

(d) Now, create a new EPICS IOC Application called,'MyTest'  as follows: 
makeBaseApp.pl -t slac MyTest

The perl script, makeBaseApp.pl,  populates your application using the icdTemplates module, providing standard ICD templates.

These templates are used by ICD to support our style of IOC Application Building for all facilities we support.

The location for icdTemplates module was defined,  as below when you sourced go_epics_3-14-12-4_1-0.bash.

EPICS_MBA_TEMPLATE_TOP=/afs/slac/g/lcls/epics/R3-14-12-4_1-0/modules/icdTemplates/icdTemplates-R1-2-0

If you would like to know more about makeBaseApp.pl, type:

makeBaseApp.pl -help

(e) Now, open the configure/RELEASE file in your new  IOC Application

(i) Notice how the following line appears towards the beginning:
include $(TOP)/RELEASE_SITE

The above sets the paths for EPICS base, modules, packages, tools etc. to point to version R3-14-12-4_1-0. 

(ii) The file has the following line towards the end of it:

# Location of Linux Kernel Modules:

LINUX_KERNEL_MODULES=$(PACKAGE_SITE_TOP)/linuxKernel_Modules

  (iii) configure/RELEASE file may need some modifications from what was automatically generated:  
 
Change the various module version numbers as needed for your application.
Ensure that the specific versions included in your RELEASE file DO EXIST under the modules directory and are consistent. 

They must also have been built for the linuxRT_glibc-x86_64 target. If something is missing, contact Ernest.

For EVENT module, Kukhee recommends us to use event-R4-1-3 or greater with this base.
 
EVENT_MODULE_VERSION=event-R4-1-3

(f) Open  MyTestApp/src/Makefile and notice that the IOC application is referred to as 'MyTest".  

(g) Now it is time to add your linuxRT_glibc-x86_64 virtual ioc (vioc) startup scripts. vioc-b34-bd32 is used here as an example.

From $(TOP) type the following command to create a boot directory for 'vioc-b34-bd32':

makeBaseApp.pl -i -t slac vioc-b34-bd32

When prompted to choose target architecture, choose linuxRT_glibc-x86_64.

Two files are modified based on this selection.

(1) vioc-b34-bd32/st.cmd

(2) vioc-b34-bd32/Makefile.

When prompted with 'Application name?'  type 'MyTest'.

This will use the application name MyTest that was created in  MyTestApp/src/Makefile as described in step (f).

If you just hit 'Enter' without specifying your application name, then your ioc name vioc-b34-bd32 will be used as the default for the dbd file in the st.cmd

Now iocBoot directory has been created under $(TOP) and underneath it, 'vioc-b34-bd32' has been created.

Open iocBoot/vioc-b34-bd32/Makefile and notice that ARCH is set to linuxRT_glibc-x86_64.
Open iocBoot/vioc-b34-bd32/st.cmd and notice that this script is setup to pick the binary from  linuxRT_glibc-x86_64 directory.

(h) icdTemplates created MyTestApp/src/Makefile. Open this file and notice how it is populated.

It has support for base, iocAdmin, autosave-restore which all IOCs must include.

It also includes support for Till Straumann's Cexp shell.

Look here for some examples that show how to invoke Cexp and how to call C-functions from this interactive shell: Cexp Shell Examples

For more information on Cexp, read What is Cexp?

Additionally, the Makefile includes 'system.dbd' - this is mainly for linux-based IOCs.

For linuxRT, 'system' command is useful for changing kernel thread priorities for kernel drivers at run-time. This is explained later under rtPrioritySetup.cmd.

(i) If you have an EVR in your system, uncomment the following lines in MyTestApp/src/Makefile :

# MyTest_DBD += evrSupport.dbd
# MyTest_DBD += devMrfEr.dbd

Ensure the following lines are not commented as well:

MyTest_LIBS += evrSupport
MyTest_LIBS += devMrfEr

....

ifeq ($(T_A),linuxRT_glibc-x86_64)
    MyTest_LIBS += drvMrf
endif

For a PMC EVR230 or PCI-e EVR300 that is installed in your system, the above is all you need.

Replace the macros in the EPICS database files, or add more databases and modify   MyTestApp/src/Makefile as needed.

(j) Now 'make' your application from the top level directory  $(TOP) to ensure your changes to  RELEASE files are good:

There should be no build errors.

Refer to notes from Ernest regarding building the application for several targets here: linuxRT targets and cross compiler notes.

Having tested this phase, you can now uninstall the binaries and cleanup the make-generated files using the following command:

make clean uninstall

  Do the above before committing the files to CVS, and make sure to cram your IOC application

TO DO list: icdTemplates module that provides default templates is a work in progress.

We have to manually make several changes to the   MyTestApp/Db/* files and iocBoot/vioc-b34-bd32/st.cmd.

Follow the example MyTest and create similar database files and st.cmd.

The EVR-related screens have been added for vioc-b34-bd32. 

 You can see them when the vioc-b34-bd32 is running as follows from lcls-dev2:

lclshome -> User Dev Displays.... -> Test IOCs/Dev Event IOCs....

The buttons for the various panels for VIOC:B34:BD32 are towards the bottom of this screen.

For easy access to your IOC's EVR panels in LCLSDEV, add the necessary buttons to this page.

The 'Pattern' button brings up the 'Pattern Diagnostics' screen.

'PMC EVR:B34:BD32' button brings up the Low Level 'Diagnostics' screen for EVR;B34:BD32

'Triggers' button brings up 'Trigger Configuration' setup screen.

(c) Add your application to CVS and commit the source files.

Here are some slides that explains a few guidelines for importing new applications into CVS:

cvs-vendor-import.txt

CVS-Importing-Merging.pdf

CVS-At-SLAC.pdf

7. What is PXE, DHCP, TFTP and NFS and why are they needed by my linuxRT IOC?

LinuxRT is installed on our system using the Preboot Execution Environment (PXE) method of network booting.

We enable the PXE/network-booting method in the BIOS.

In order to use PXE there is a boot-server that will allow our client system to :
(a) Request an IP address (via DHCP)
(b) Download a kernel (via TFTP)

With both of these services in place any system which supports PXE/network-booting
should be able to gain an IP address, fetch a kernel, and boot without an installed operating system.

PXE uses three distinct network protocols that map to three server processes to perform the installation.
In our case, some of the processes run on lcls-dev1 (LCLSDEV daemon)

Both  DHCP and TFTP services run on the LCLSDEV host 'dhcp3' maintained by SCCS.

(a) Dynamic Host Configuration Protocol (DHCP)

PXE uses DHCP to deliver initial network configuration options to client nodes.
The DHCP server supplies the PXE boot plug-in with
(i) IP address
(ii) TFTP server address
(iii) Stage 1 image boot-loader name from which to download and execute the image.

As the supplied PXE installation environments are non-interactive and will unconditionally reinstall a client machine,
we have the client associate its MAC address with a specific OS installation before starting the PXE boot.

The configuration information, in our case, in addition to IP/MAC address, includes a hostname and a pointer to the Master Starupt script in afs for our IOC.
It has an optional root-path variable pointing to the afs area which hosts the boot image that is served via TFTP.
This can be over-ridden as will be seen later.

When the Linux server is rebooted or power-cycled, PXE will attempt the network booting method first
and as a first step it will contact the DHCP server to retrieve the network configuration information.

Hence, every new linuxRT ioc (host) needs to be added to the DHCP server configuration file in afs.

This file is in /afs/slac/service/dhcp-pxe/dhcpd.conf

Note that the DHCP service running on dhcp3 is intended only for booting embedded devices like our linuxRT servers that are  connected to the LCLSDEV and SSRL subnets.

The MAC addresses for such  devices must be registered in CANDO and assigned fixed IP addresses.

The IP/MAC address of the primary ethernet that will fetch the linuxRT boot image is defined here.
To add a new host to the DHCP configuration, contact Thuy.

After a new ioc is added to dhcpd.conf, the DHCP service must be restarted.

To restart DHCP Server

From a Unix command line:

remctl dhcp3 dhcp check

remctl dhcp3 dhcp restart

For help:

remctl dhcp3 dhcp help

Currently only Thuy, Ernest and a couple of others have permissions to perform this restart on dhcp3.

Here's is an example - cpu-b34-bd32:

host cpu-b34-bd32 {
# SuperMicro (INTELx86)
#
hardware ethernet 00:25:90:D1:95:1E;
fixed-address 134.79.218.190;
option host-name "cpu-b34-bd32";
if ( substring( option vendor-class-identifier, 0, 5 ) = "udhcp" ) {
filename "/afs/slac/g/lcls/epics/iocCommon/cpu-b34-bd32/startup.cmd";
option root-path " afsnfs1:/afs/slac:/afs/slac ";
}
}

To find out more about how our linux server boots up linuxRT, click on the following link:

  How does the Linux Server boot up linuxRT?

(b) Trivial File Transfer Protocol (TFTP)

PXE uses TFTP that defines a simple UDP protocol for delivering files over a network.
PXE delivers kernels and initial bootstrap software to client nodes using TFTP.

tftpboot is mounted in afs area in LCLSDEV on the server dhcp3, which runs both the TFTP an DHCP services:

  /tftpboot -> /afs/ slac.stanford.edu/service/dhcp-pxe/tftpboot/

The iocs retrieve the linuxRT boot image from the TFTP server from the following location:

/afs/slac/g/lcls/tftpboot/linuxRT/boot

In this location, there are several linuxRT_glibc-x86_64 bootimages.
These were custom-built by T.Straumann for the various Linux Servers/IPCs that we currently have setup to boot with linuxRT OS.

Of these images, '3.14.12-rt9' is the latest and it has in-built support for the
Broadcom networking ethernet chipset that are used in our dev Poweredge Dell Servers.

(c) Network File System (NFS)

The NFS service is used by the installation kernel to read all of the packages necessary to the installation process.

NFS services run on the LCLSDEV hosts  afsnfs1 and afsnfs2   maintained by SCCS.

  This service makes available the boot directory to all linuxRT targets that boot as diskless clients.

All clients have read-only permissions to this directory.

The linuxRT iocs need some additional NFS Mount Points to write their data some where.

surrey04b is an NFS Appliance and is used by iocs for data and have both read and write permissions to the data directory ($IOC_DATA).

In LCLSDEV, we must obtain permissions for the iocs to write to the $IOC_DATA directory.

  Fill out an online form provided by SCCS to obtain permissions for your ioc to write to this directory :

https://www-rt.slac.stanford.edu/rt3/SelfService/Forms/IocNfs.html    

Jingchen has created a diagram that shows linuxRT boot process' communication flow:

(1) LinuxRT boot process

The system boot process communication flow ends when IOC startup.cmd is executed.

It is then picked up by the IOC application startup process that is further explained in following sections.

They are also captured in more diagrams as below:

(2)   liunxRT Boot Sequence and Scripts

The above flow chart is also under Question 15.

(3) For a more up-to-date and complete overview of the linuxRT system in ICD development area, refer to Alisha's block diagram below: 

linuxRT System Overview in LCLSDEV subnet

11. How do I create a startup script for my ioc?

The third and final script is specifically to setup and start your EPICS ioc. The standard startup is done entirely with 3 soft links. The commands to set up vioc-b34-bd32 would be:

> cd $IOC

> mkdir vioc-b34-bd32

> cd vioc-b34-bd32

> ln -s ../skel/startup_ioc.cmd startup.cmd

> cp ../facility/screenrc screenrc [then edit the log file path to match the vioc name]

> ln -s ../../iocTop/MyTest/Development iocSpecificRelease

If the directory name where the app-specific st.cmd is located is the same as the name of this direcotry, and its name is st.cmd, nothing more is required in this directory and you can skip the rest of this question.

These defaults may be overridden in a custom startup.cmd. Rather than creading the soft link, make a copy ("cp ../skel/startup_ioc.cmd startup.cmd") and at the top define environment variables $EPICS_IOC_APP or $ST_CMD. That is, the default values that startup.cmd passes to All/common/linuxRT_st.cmd are EPICS_IOC_APP is iocSpecificRelease/iocBoot/$VIOC, and ST_CMD is "st.cmd", but you can define them to be something else like

epicsEnvSet("EPICS_IOC_APP", "iocSpecificRelease/iocBoot/li20/vioc-b34-bd32")

epicsEnvSet("ST_CMD", "st.dev.cmd")

Note that because startup_ioc.cmd is a generic script, there is no "!#" at the top, so it is not executable. It is intended to be used by the linuxRT_startup_ioc.sh (script that linuxRT_cpu_load_iocs.sh calls for each IOC at boot) or linuxRT_viocConsole.sh scripts, which determine the executable. It may also be run directly as an argument to the executable on the command line, e.g. "./iocSpecificRelease/bin/linuxRT_glibc-x86_64/MyTest startup.cmd".

You may also create more customized scripts. For example the 'startupConsole-bd32.cmd' lets you start your EPICS ioc as a foreground process in your host. This lets you observe the ioc startup process to catch errors and interact with the ioc shell via iocConsole. Generally linuxRT_viocConsole.sh would be used instead, as it does the same basic thing.

This is useful during development for debugging. Once the ioc has been tested fully, you can automatically start and run the process in background.

Under linuxRT, your EPICS ioc must run as a real-time process. It must lock the kernel in memory.

The following command in your 'startupConsole-bd32.cmd' does that:

ulimit -l unlimited

The following line is also needed to run the ioc with real-time priorities:

ulimit -r unlimited

Finally, you will be running your ioc as a specific user called as 'laci' who has permissions to run this ioc:
Setup the permissions for this user 'laci':

umask 002

Now you are ready to start your IOC and have it run as a foreground process.

Create a directory called as 'vioc-b34-bd32' for your ioc as below:
 
mkdir $IOC/vioc-b34-bd32

cd $IOC/cpu-b34-bd32

ln -s ../vioc-b34-bd32

cd $IOC/cpu-b34-bd32/vioc-b34-bd32
 
Set up a soft link to the 'bin' directory of your IOC app that you created in step (6). This is where your executable is:

ln -s  /afs/slac/g/lcls/epics/R3-14-12-4_1-0/iocTop/MyTest/Development/bin/linuxRT_glibc-x86_64 bin

Create an ASCII text file called 'screenrc' with the following lines in it:

deflog on

logtstamp on

defscrollback 10000

logfile /data/vioc-b34-bd32/screenlog.0

'screenrc' is passed as an argument to the 'screen' process and allows us to customize a few parameters such as number of lines stored in history buffer that we can scroll back for viewing.

In the same directory $IOC/ioc-b34-my01/vioc-b34-my01, add a startup script 'startup.cmd' for vioc-b34-my01:

It setups some shell environment variables used by all iocs, and starts the st.cmd file in your ioc boot directory.

The EPICS environment variables that are set in this script can be used by your application's st.cmd script.

The default st.cmd script generated by the module icdTemplates, expects the following environment values to be defined somewhere.

iocStartup.cmd may be a good place to define it:

epicsEnvSet("IOC_NAME","VIOC:B34:BD32")

epicsEnvSet("IOC_APP","/afs/slac/g/lcls/epics/R3-14-12-4_1-0/iocTop/MyTest/Development")

epicsEnvSet("IOC_BOOT","${IOC_APP}/iocBoot/vioc-b34-bd32")

epicsEnvSet("IOC_COMMON","/afs/slac/g/lcls/epics/iocCommon")

epicsEnvSet("SYS_FACILITY","SYS0")

In addition to setting these environment variables, this script also executes some scripts common to all EPICS IOCs, such as below:

${IOC_COMMON}/All/Dev/linuxRT_pre_st.cmd

${IOC_COMMON}/All/Dev/linuxRT_post_st.cmd

The IOC application startup occurs between the pre and post scirpts:

cd ${IOC_BOOT}

< st.cmd