...
| Panel |
|---|
7. What is PXE, DHCP, TFTP and NFS and why are they needed by my linuxRT IOC?LinuxRT is installed on our system using the Preboot Execution Environment (PXE) method of network booting. With both of these services in place any system which supports PXE/network-booting Both DHCP and TFTP services run on the LCLSDEV host 'dhcp3' maintained by SCCS. (a) Dynamic Host Configuration Protocol (DHCP) Note that the DHCP service running on dhcp3 is intended only for booting embedded devices like our linuxRT servers that are connected to the LCLSDEV and SSRL subnets. The MAC addresses for such devices must be registered in CANDO and assigned fixed IP addresses. The IP/MAC address of the primary ethernet that will fetch the linuxRT boot image is defined here. After a new ioc is added to dhcpd.conf, the DHCP service must be restarted. To restart DHCP Server From a Unix command line: remctl dhcp3 dhcp check remctl dhcp3 dhcp restart For help: remctl dhcp3 dhcp help Currently only Thuy, Ernest and a couple of others have permissions to perform this restart on dhcp3. Here's is an example - ioc-b34-bd32: To find out more about how our linux server boots up linuxRT, click on the following link: How does the Linux Server boot up linuxRT? (b) Trivial File Transfer Protocol (TFTP) PXE uses TFTP that defines a simple UDP protocol for delivering files over a network. tftpboot is mounted in afs area in LCLSDEV on the server dhcp3, which runs both the TFTP an DHCP services: /tftpboot -> /afs/ slac.stanford.edu/service/dhcp-pxe/tftpboot/ The iocs retrieve the linuxRT boot image from the TFTP server from the following location: NFS services run on the LCLSDEV hosts afsnfs1 and afsnfs2 maintained by SCCS. This service makes available the boot directory to all linuxRT targets that boot as diskless clients. All clients have read-only permissions to this directory. The linuxRT iocs need some additional NFS Mount Points to write their data some where. surrey04b is an NFS Appliance and is used by iocs for data and have both read and write permissions to the data directory ($IOC_DATA). In LCLSDEV, we must obtain permissions for the iocs to write to the $IOC_DATA directory. Fill out an online form provided by SCCS to obtain permissions for your ioc to write to this directory : https://www-rt.slac.stanford.edu/rt3/SelfService/Forms/IocNfs.html
|
| Panel |
|---|
8. How do I start myIOCCPU? Where is myiochost's statrup.cmd?There are a few scripts that automate this process. set vers 3.14.12-rt9 Then system startup continues with IOC application startup (questions 11 through 14) If non-default behavior is desired, the soft link may be replaced by a different startup.cmd. The standard startup_cpu.cmd sets up the host-wide environment by calling the generic script All/common/linuxRT_startup_cpu.sh. The linuxRT_startup_cpu.sh script defines standard environment variables such as $HOSTNAME and $T_A, and calls a script for facility specific enviromnet environment setup, common/linuxRT_env.sh. Additional NFS Mount Points for linuxRT pertaining to the ioc data directory $IOC_DATA are mounted as well. The next line in the linuxRT_startup_cpu.sh file loads the system specific linuxRT kernel modules. More on kernel modules under question (10). This can also be done only by the "root" user: /afs/slac/g/lcls/epics/iocCommon/ioc-b34-bd32/kernel-modules.cmd Next we must start the caRepeater process: (http://www.aps.anl.gov/epics/base/R3-14/12-docs/CAref.html#Repeater) "When several client processes (virtual iocs) run on the same host it is not possible for all of them to directly receive a copy of the server beacon messages when the beacon messages are sent to unicast addresses, or when legacy IP kernels are still in use. To avoid confusion over these restrictions a special UDP server, the CA Repeater, is automatically spawned by the CA client library when it is not found to be running. This program listens for server beacons sent to the UDP port specified in the EPICS_CA_REPEATER_PORT parameter and fans any beacons received out to any CA client program running on the same host that have registered themselves with the CA Repeater. If the CA Repeater is not already running on a workstation, then the "caRepeater" program must be in your path before using the CA client library for the first time." So we have added the following lines to linuxRT_startup_cpu.sh to start a caRepeater for all EPICS VIOCs that may be hosted by this CPU: su $IOC_USER -c $(which caRepeater) & Once linuxRT_startup_cpu.sh completes system-wide configuration, it is possible to automatically startup one or more EPICS IOCs and detach them using the linux screen program. The standard startup script uses linuxRT_cpu_load_iocs.sh for this. It searches screeniocs for viocs configured to run on this host and starts them. There are other options to start IOCs when the standard automatic startup is not wanted. More directly, we can start another shell such that the ioc user account, usually "laci", can start the IOC process We can start another shell such that a user called "laci" can start the IOC process instead of the "root" user: su laci -c /afs/slac/g/lcls/epics/iocCommon/ioc-b34-bd32/startup-epics-bd32.cmd or su laci -c /afs/slac/g/lcls/epics/iocCommon/vioc-b34-bd32/startup.cmd More on startup-epics-bd32.cmd under Question (13). Under linuxRT, a few real time processes that need real time scheduler and kernel memory locking features, can be specifically run as such. The screen process does not need or have RT priority. The _MAIN_ thread in epics application process, which is started doesn't have the RT priority either. But, other threads which are created by the _MAIN_ thread may need RT priority. Kukhee provides some information about this to find out which processes are running with real-time priorities: |
| Panel |
|---|
9. How do I monitor the server remotely as it boots up - after a power-cycle or after a 'reboot' command?Thuy has setup the BIOS of all the linux servers used for linuxRT development, to re-direct the Type the following command from any LCLSDEV host: Example: Add iocConsole access by adding a line for your host to $IOC/screeniocs like: ioc-b34-bd32 ts-lclsdev05 2001 lcls-dev1 where the fields are cpu-name, terminal server name, terminal server port, host where the screen process will run. Type the following command from any LCLSDEV host: Example: iocConsole ioc-b34-iocConsole ioc-b34-bd32 This will establish a serial connection with the linux box from lcls-dev1 via a DIGI Terminal Server port as follows: : ssh -x -t -l laci lcls-dev1 bash -l -c " pyiocscreen.py -t HIOC ioc-b34-bd32 ts-lclsdev05 2001 " You can monitor the linuxRT as it goes through the PXE network booting process. Finally, you will get the login prompt: Welcome to Buildroot Login as 'root'. No password is required. At the shell, type the following to ensure that you are running the correct version of liunxRT for your ioc. This has the real-time Preempt RT patch. # uname -a To see the PCI devices in your system type the following command: # lspci |
...
| Panel |
|---|
11. How do I create a startup script for my ioc?The third and final script is specifically to setup and start your EPICS ioc. This lets you observe the ioc boot process to catch errors and interact with the ioc shell via iocConsole. Under linuxRT, your EPICS ioc must run as a real-time process. It must lock the kernel in memory. The following command in your 'startupConsole-bd32.cmd' does that:ulimit -l unlimited The following line is also needed to run the ioc with real-time priorities: ulimit -r unlimited Finally, you will be running your ioc as a specific user called as 'laci' who has permissions to run this ioc: Setup the permissions for this user 'laci': umask 002 Now you are ready to start your IOC and have it run as a foreground process. Create a directory called as 'vioc-b34-bd32' for your ioc as below: mkdir $IOC/ioc-b34-bd32/vioc-b34-bd32cd $IOC/ioc-b34-bd32/vioc-b34-bd32 Set up a soft link to the 'bin' directory of your IOC app that you created in step (6). This is where your executable is: ln -s /afs/slac/g/lcls/epics/R3-14-12-4_1-0/iocTop/MyTest/Development/bin/linuxRT-x86 bin Create an ASCII text file called 'screenrc' with the following lines in it: deflog on logtstamp on defscrollback 10000 'screenrc' is passed as an argument to the 'screen' process and allows us to customize a few parameters such as number of lines stored in history buffer that we can scroll back for viewing. In the same directory $IOC/ioc-b34-my01/vioc-b34-my01, add a startup script 'iocStartup.cmd' for vioc-b34-my01:It setups some shell environment variables used by all iocs, and starts the st.cmd file in your ioc boot directory. The EPICS environment variables that are set in this script can be used by your application's st.cmd script. The default st.cmd script generated by the module icdTemplates, expects the following environment values to be defined somewhere. iocStartup.cmd may be a good place to define it: epicsEnvSet("IOC_NAME","VIOC:B34:BD32") epicsEnvSet("IOC_APP","/afs/slac/g/lcls/epics/R3-14-12-4_1-0/iocTop/MyTest/Development") epicsEnvSet("IOC_BOOT","${IOC_APP}/iocBoot/vioc-b34-bd32") epicsEnvSet("IOC_COMMON","/afs/slac/g/lcls/epics/iocCommon") epicsEnvSet("SYS_FACILITY","SYS0") In addition to setting these environment variables, this script also executes some scripts common to all EPICS IOCs, such as below: ${IOC_COMMON}/All/Dev/linuxRT_pre_st.cmd ${IOC_COMMON}/All/Dev/linuxRT_post_st.cmd The IOC application startup occurs between the pre and post scirpts: cd ${IOC_BOOT} < st.cmd |
| Panel |
|---|
12. What are the guidelines for creating App-specific st.cmd?Work is still in progress regarding standardizing App-specific st.cmd. The IOC engineer has freedom to be creative with this script during development. But the following are expected as a minimum when the ioc is installed in production. Refer to IOC Preferred Practices document for further guidance on creating IOCs. That is the reason they are included in the default template provided by the icdTemplates module, as a guideline:
While the above is common guideline for all iocs, the following is specific to linuxRT. You can set up real-time priorities after iocInit() for your driver threads and this can be done with a script such as rtPrioritySetup.cmd. cd ${IOC_BOOT} system("/bin/su root -c `pwd`/rtPrioritySetup.cmd") Look at rtPrioritySetup.cmd as an example. |
. The standard startup is done entirely with 3 soft links. The commands to set up vioc-b34-bd32 would be: > cd $IOC> mkdir vioc-b34-bd32> cd vioc-b34-bd32> ln -s ../skel/startup_ioc.cmd startup.cmd> ln -s ../All/screenrc screenrc> ln -s ../../iocTop/MyTest/Development iocSpecificReleaseIf the directory name where the app-specific st.cmd is located is the same as the name of this direcotry, and its name is st.cmd, nothing more is required in this directory and you can skip the rest of this question. These defaults may be overridden in a custom startup.cmd. Rather than creading the soft link, make a copy ("cp ../skel/startup_ioc.cmd startup.cmd") and at the top define environment variables $EPICS_IOC_APP or $ST_CMD. That is, the default values that startup.cmd passes to All/common/linuxRT_st.cmd are EPICS_IOC_APP is iocSpecificRelease/iocBoot/$VIOC, and ST_CMD is "st.cmd", but you can define them to be something else like epicsEnvSet("EPICS_IOC_APP", "iocSpecificRelease/iocBoot/li20/vioc-b34-bd32") epicsEnvSet("ST_CMD", "st.dev.cmd") Note that because startup_ioc.cmd is a generic script, there is no "!#" at the top, so it is not executable. It is intended to be used by the linuxRT_startup_ioc.sh (script that linuxRT_cpu_load_iocs.sh calls for each IOC at boot) or linuxRT_startupConsole.sh scripts, which determine the executable. It may also be run directly as an argument to the executable on the command line, e.g. "./iocSpecificRelease/bin/linuxRT-x86/MyTest startup.cmd". You may also create more customized scripts. For example the 'startupConsole-bd32.cmd' lets you start your EPICS ioc as a foreground process in your host. This lets you observe the ioc startup process to catch errors and interact with the ioc shell via iocConsole. Generally linuxRT_startupConsole.sh would be used instead, as it does the same basic thing. This is useful during development for debugging. Once the ioc has been tested fully, you can automatically start and run the process in background. ulimit -l unlimited cd $IOC/ioc-b34-bd32 ln -s ../vioc-b34-bd32 cd $IOC/ioc-b34-bd32/vioc-b34-bd32 Create an ASCII text file called 'screenrc' with the following lines in it: deflog on logtstamp on defscrollback 10000 'screenrc' is passed as an argument to the 'screen' process and allows us to customize a few parameters such as number of lines stored in history buffer that we can scroll back for viewing. In the same directory $IOC/ioc-b34-my01/vioc-b34-my01, add a startup script 'startup.cmd' for vioc-b34-my01: The EPICS environment variables that are set in this script can be used by your application's st.cmd script. The default st.cmd script generated by the module icdTemplates, expects the following environment values to be defined somewhere. iocStartup.cmd may be a good place to define it: epicsEnvSet("IOC_NAME","VIOC:B34:BD32") epicsEnvSet("IOC_APP","/afs/slac/g/lcls/epics/R3-14-12-4_1-0/iocTop/MyTest/Development") epicsEnvSet("IOC_BOOT","${IOC_APP}/iocBoot/vioc-b34-bd32") epicsEnvSet("IOC_COMMON","/afs/slac/g/lcls/epics/iocCommon") epicsEnvSet("SYS_FACILITY","SYS0") In addition to setting these environment variables, this script also executes some scripts common to all EPICS IOCs, such as below: ${IOC_COMMON}/All/Dev/linuxRT_pre_st.cmd ${IOC_COMMON}/All/Dev/linuxRT_post_st.cmd The IOC application startup occurs between the pre and post scirpts: cd ${IOC_BOOT} < st.cmd |
| Panel |
|---|
12. What are the guidelines for creating App-specific st.cmd?Work is still in progress regarding standardizing App-specific st.cmd. The IOC engineer has freedom to be creative with this script during development. But the following are expected as a minimum when the ioc is installed in production. Refer to IOC Preferred Practices document for further guidance on creating IOCs. That is the reason they are included in the default template provided by the icdTemplates module, as a guideline:
While the above is common guideline for all iocs, the following is specific to linuxRT. You can set up real-time priorities after iocInit() for your driver threads and this can be done with a script such as rtPrioritySetup.cmd . cd ${IOC_BOOT} system("/bin/su root -c `pwd`/rtPrioritySetup.cmd") Look at rtPrioritySetup.cmd as an example. |
| Panel |
|---|
13. How do I start my ioc and monitor it?(a) If the ioc is defined in screeniocs and automatically started, use iocConsole. See question 14 for details on automatic startup. (b) You can directly ssh to your ioc as 'laci' from any LCLSDEV host and use linuxRT_startupConsole.sh: Alternately, you can get a console to the target ioc-b34-bd32 using iocConsole and ssh as user "laci" from there: iocConsole ioc-b34-bd32 When prompted to enter the "login" username as below, enter "laci". Welcome to Buildroot ioc-b34-bd32 login: No password is needed. Just hit Enter an you will get the shell prompt "$". Ensure the caRepeater process is running on this CPU. (b) Ensure your IOC is not already running as below: $ screen -ls No Sockets found in /tmp/uscreens/S-laci. You can also look for your application under all running processes and make sure one is not already running: If it was running already, then you may get something as below: $ ps -ef | grep MyTest 2085 laci 0:07 bin/MyTest iocStartup.cmd (c) Change to $IOC/ioc-b34-bd32 and from there start your ioc as a foreground process as below: $ pwd /home/laci $ ls -lt total 0 lrwxrwxrwx 1 root lcls 45 Nov 5 15:11 bld -> /afs/slac/g/lcls/epics/iocCommon/ioc-b34-bd32 $ cd bld $ ls -lt | grep startupConsole-bd32.cmd -rwxrwxr-x 1 8396 1020 1225 Nov 5 15:22 startupConsole-bd32.cmd You can observe the vioc If you had an EVR230 and it was initialized correctly, you may see something as below before iocInit(): ErConfigure(0, 0, 0, 0, 1) Try EvrOpen, device = /dev/era3 EVR Found with Firmware Revision 0x11000007 Found a PMC_EVR /dev/era3 Verify if all your drivers, modules and databases loaded correctly. If you used iocConsole, you can scroll back and forth in the screen using <CTRL> A-[ and <CTRL>-A-] If everything went well, you must get your interactive ioc shell prompt as below: vioc-b34-bd32> (d) Once the ioc is started, you can interact with vioc-b34-bd32 from any LCLSDEV host via ssh laci@ioc-b34-bd32 : To attach a screen session: screen -r vioc-b34-bd32 To detach from a screen session: <CTRL> A D (e) To exit an EPICS IOC Process type exit at the IOC Shell and you will be back at the "$" prompt: vioc-b34-bd32>exit If the IOC hangs too long while trying to exit, use <CTRL> C |
| Panel |
|---|
14. How do I start my ioc automatically every time my server boots up?You can start your ioc as a background process that will start up everytime your Linux server reboots, as follows: Add a line to screeniocs for your vioc, e.g. vioc-b34-bd32 ioc-b34-bd32 laci /afs/slac/g/lcls/epics/iocCommon/vioc-b34-bd32/iocSpecificRelease/bin/linuxRT_glibc-x86_64/MyTest where the fields are vioc name, host name, IOC user, executable path. At startup, the cpu/startup.cmd script calls linuxRT_cpu_load_iocs.sh, which scans screeniocs for these lines and starts the ones with the correct host name. |
| Panel |
| Panel |
14. How do I start my ioc automatically every time my server boots up?You can start your ioc as a background process that will start up everytime your Linux server reboots, as follows:The rest of this answer describes the older approach, which will still work and may be desirable in some circumstances. In $IOC/ioc-b34-bd32, you can add another script ' startup-epics-bd32.cmd ' that lets you start your EPICS-based ioc as a background process in your host. Invoke this script from $IOC/ioc-b34-bd32/startup.cmd : su laci -c /usr/local/lcls/epics/iocCommon/eioc-b34-bd32/startup-epics-bd32.cmd The above line starts another shell so that the user called "laci" can start the IOC Process as opposed to the "root" account Remember, the kernel modules can only be loaded onto the kernel with "root" permissions but iocs can be run by users. 'startup-epics-bd32.cmd' executes the EPICS-based IOC startup script and detaches it using the linux screen program It contains the following lines which are similar to what we have in startupConsole-bd32.cmd: export EPICS_IOCS=/afs/slac/g/lcls/epics/iocCommon # Lock Kernel in memory # Set umask for the laci user # EPICS IOC Application Real-time Process cd $EPICS_IOCS/ioc-b34-bd32/vioc-b34-bd32 screen -h 8192 -c screenrc -dm -S vioc-b34-bd32 bin/MyTest iocStartup.cmd The above line will automatically start your ioc and have it running in the background, every time your server reboots To view the console output from the running ioc, you can attach the screen process to it, as described below: (1) ssh into the ioc-b34-b32 as user "laci": ssh laci@ioc-b34-bd32 (2) To see the list of all EPICS iocs running on this CPU, at the console type: screen -ls (3) To attach to vioc-b34-b32 that is running MyTest application: screen -r vioc-b34-b32 |
...
