Page History

8. How do I start my CPU? Where is my host's statrup.cmd?

There are a few scripts that automate this process.

To begin with, there is the 'ipxe.ini' script in the tftp boot area /afs/slac/g/lcls/tftpboot/linuxRT/boot that PXE will run.

This is where the version of (linuxRT) kernel to run is specified as follows:

set vers 3.2.13-121108

This version number can be over-ridden by a chained, host-specific pxe init script to load an image different from the above:

chain ${hostname}.ipxe ||

 For example, we have defined a script specifically for our ioc cpu-b34-bd32.ipxe which chooses to load the latest linuxRT image:

set vers 3.14.12-rt9_x86_64_glibc

 This is also the place to over-ride the 'root-path' option specified in the DHCP configuration file dhcpd.conf.

 For example, we may decide to over-ride the afsnfs1 server and instead choose to get my boot image from lcls-dev3 server:

set extra-args brd.rd_size=524288 ROOTPATH=lcls-dev3:/afs/slac:/afs/slac BOOTFILE=/afs/slac/g/lcls/epics/iocCommon/cpu-b34-bd32/startup.cmd

The 'ipxe.ini' script loads the linuxRT kernel via the TFTP protocol:

kernel --name linux tftp://${next-server}/linuxRT/boot/${vers}/bzImage && initrd tftp://${next-server}/linuxRT/boot/${vers}/rootfs.ext2 || shell
imgargs linux debug idle=halt root=/dev/ram0 console=ttyS0,115200 BOOTIF_MAC=${net0/ mac:hex } ${extra-args} || boot || shell

After linuxRT boot image is downloaded to the target and linuxRT starts to run, additional nfs mounts will be done.

 The afs to nfs translator service makes available the directory structure, to all clients that have mounted this nfs space.

 NFS File Servers for LCLSDEV is primarily lcls-dev3. Alternately, it can use afsnfs1 or afsnfs2 but these will be phased out soon.

 One of the arguments to the kernel process is the location of the BOOTFILE that does the mounting.

The 'filename' argument (which can be over-ridden by the BOOTFILE argument for linuxRT) is as follows:

"/afs/slac/g/lcls/epics/iocCommon/cpu-b34-bd32/startup.cmd"

This script is similar to and modeled after RTEMS startup.cmd. The standard startup.cmd script is a soft link to ../skel/startup_cpu.cmd.

If you use the above startup_cpu.cmd script, you can skip the rest of this section.

The remaining standard host level configuration involves setting up iocConsole (question 9) and hardware specific kernel modules (question 10).

Then system startup continues with IOC application startup (questions 11 through 14)

If non-default behavior is desired, the soft link may be replaced by a different startup.cmd. The standard startup_cpu.cmd sets up the host-wide environment by calling the generic script All/common/linuxRT_startup_cpu.sh.

The linuxRT_startup_cpu.sh script defines standard environment variables such as $HOSTNAME and $T_A, and calls a script for facility specific environment setup, facility/linuxRT_env.sh.

When linuxRT loads and starts, the kernel process is run as "root" user.

 Hence it has permissions to setup the nfs mounts which is done by the following line in linuxRT_startup_cpu.sh:

linuxRT_nfs.cmd

Additional NFS Mount Points for linuxRT pertaing to the ioc data directory $IOC_DATA are mounted as well.

The next line in the linuxRT_startup_cpu.sh file loads the system specific linuxRT kernel modules.

More on kernel modules under question (10).

This can also be done only by the "root" user:

/afs/slac/g/lcls/epics/iocCommon/cpu-b34-bd32/kernel-modules.cmd

Next we must start the caRepeater process:

(http://www.aps.anl.gov/epics/base/R3-14/12-docs/CAref.html#Repeater)

"When several client processes (virtual iocs) run on the same host it is not possible for all of them to directly receive a copy of the server beacon messages when the beacon messages are sent to unicast addresses, or when legacy IP kernels are still in use. To avoid confusion over these restrictions a special UDP server, the CA Repeater, is automatically spawned by the CA client library when it is not found to be running. This program listens for server beacons sent to the UDP port specified in the EPICS_CA_REPEATER_PORT parameter and fans any beacons received out to any CA client program running on the same host that have registered themselves with the CA Repeater. If the CA Repeater is not already running on a workstation, then the "caRepeater" program must be in your path before using the CA client library for the first time."

So we have added the following lines to  linuxRT_startup_cpu.sh to start a caRepeater for all EPICS VIOCs that may be hosted by this CPU:

su $IOC_USER -c $(which caRepeater) &

Once linuxRT_startup_cpu.sh completes system-wide configuration, it is possible to automatically startup one or more EPICS IOCs and detach them using the linux screen program. The standard startup script uses linuxRT_cpu_load_iocs.sh for this. It searches screeniocs for viocs configured to run on this host and starts them.

There are other options to start IOCs when the standard automatic startup is not wanted. Rather than using screeniocs to determine what to start, a custom startup.cmd may call linuxRT_cpu_load_iocs.sh after defining environment variable $LOCAL_IOCS and optionally $IOC_EXECUTABLES to list which IOCs to start.

More directly, we can start another shell such that the ioc user account, usually "laci", can start the IOC process instead of the "root" user:

su laci -c /afs/slac/g/lcls/epics/iocCommon/cpu-b34-bd32/startup-epics-bd32.cmd

or

 su laci -c /afs/slac/g/lcls/epics/iocCommon/vioc-b34-bd32/startup.cmd

 More on startup-epics-bd32.cmd under Question (13).

 Under linuxRT, a few real time processes that need real time scheduler and kernel memory locking features, can be specifically run as such.

 The screen process does not need or have RT priority.

The _MAIN_ thread in epics application process, which is started  doesn't have the RT priority either.

But, other threads which are created by the _MAIN_ thread may need RT priority.

 Kukhee provides some information about this to find out which processes are running with real-time priorities: Command to look up thread priorities  

11. How do I create a startup script for my ioc?

The third and final script is specifically to setup and start your EPICS ioc. The standard startup is done entirely with 3 soft links. The commands to set up vioc-b34-bd32 would be:

> cd $IOC

> mkdir vioc-b34-bd32

> cd vioc-b34-bd32

> ln -s ../skel/startup_ioc.cmd startup.cmd

> cp ../facility/screenrc screenrc [then edit the log file path to match the vioc name]

> ln -s ../../iocTop/MyTest/Development iocSpecificRelease

If the directory name where the app-specific st.cmd is located is the same as the name of this direcotry, and its name is st.cmd, nothing more is required in this directory and you can skip the rest of this question.

These defaults may be overridden in a custom startup.cmd. Rather than creading the soft link, make a copy ("cp ../skel/startup_ioc.cmd startup.cmd") and at the top define environment variables $EPICS_IOC_APP or $ST_CMD. That is, the default values that startup.cmd passes to All/ common/linuxRT_st.cmd are EPICS_IOC_APP is iocSpecificRelease/iocBoot/$VIOC, and ST_CMD is "st.cmd", but you can define them to be something else like

epicsEnvSet("EPICS_IOC_APP", "iocSpecificRelease/iocBoot/accl/vioc-b34-bd32")

epicsEnvSet("ST_CMD", "st.dev.cmd")

Note that because startup_ioc.cmd is a generic script, there is no "!#" at the top, so it is not executable. It is intended to be used by the linuxRT_startup_ioc.sh (script that linuxRT_cpu_load_iocs.sh calls for each IOC at boot) or linuxRT_viocConsole.sh scripts, which determine the executable. It may also be run directly as an argument to the executable on the command line, e.g. "./iocSpecificRelease/bin/linuxRT_glibc-x86_64/MyTest startup.cmd".

You may also create more customized scripts. For example the 'startupConsole-bd32.cmd' lets you start your EPICS ioc as a foreground process in your host. This lets you observe the ioc startup process to catch errors and interact with the ioc shell via iocConsole. Generally linuxRT_viocConsole.sh would be used instead, as it does the same basic thing.

This is useful during development for debugging. Once the ioc has been tested fully, you can automatically start and run the process in background.

Under linuxRT, your EPICS ioc must run as a real-time process. It must lock the kernel in memory.

 The following command in your 'startupConsole-bd32.cmd' does that:

ulimit -l unlimited

The following line is also needed to run the ioc with real-time priorities:

ulimit -r unlimited

Finally, you will be running your ioc as a specific user called as 'laci' who has permissions to run this ioc:
Setup the permissions for this user 'laci':

umask 002

Now you are ready to start your IOC and have it run as a foreground process.

Create a directory called as 'vioc-b34-bd32' for your ioc as below:

 mkdir $IOC/vioc-b34-bd32

cd $IOC/cpu-b34-bd32

ln -s ../vioc-b34-bd32

cd $IOC/cpu-b34-bd32/vioc-b34-bd32
 
Set up a soft link to the 'bin' directory of your IOC app that you created in step (6). This is where your executable is:

ln -s  /afs/slac/g/lcls/epics/R3-14-12-4_1-0/iocTop/MyTest/Development/bin/linuxRT_glibc-x86_64 bin

Create an ASCII text file called 'screenrc' with the following lines in it:

deflog on

logtstamp on

defscrollback 10000

logfile /data/vioc-b34-bd32/screenlog.0

'screenrc' is passed as an argument to the 'screen' process and allows us to customize a few parameters such as number of lines stored in history buffer that we can scroll back for viewing.

In the same directory $IOC/ioc-b34-my01/vioc-b34-my01, add a startup script 'startup.cmd' for vioc-b34-my01:

It setups some shell environment variables used by all iocs, and starts the st.cmd file in your ioc boot directory.

The EPICS environment variables that are set in this script can be used by your application's st.cmd script.

The default st.cmd script generated by the module icdTemplates, expects the following environment values to be defined somewhere.

iocStartup.cmd may be a good place to define it:

epicsEnvSet("IOC_NAME","VIOC:B34:BD32")

epicsEnvSet("IOC_APP","/afs/slac/g/lcls/epics/R3-14-12-4_1-0/iocTop/MyTest/Development")

epicsEnvSet("IOC_BOOT","${IOC_APP}/iocBoot/vioc-b34-bd32")

epicsEnvSet("IOC_COMMON","/afs/slac/g/lcls/epics/iocCommon")

epicsEnvSet("SYS_FACILITY","SYS0")

In addition to setting these environment variables, this script also executes some scripts common to all EPICS IOCs, such as below:

 ${IOC_COMMON}/All/Dev/linuxRT_pre_st.cmd

 ${IOC_COMMON}/All/Dev/linuxRT_post_st.cmd  

The IOC application startup occurs between the pre and post scirpts:

cd ${IOC_BOOT}

< st.cmd