Summary
Debugging code on an embedded system is a difficult problem. The basic features of embedded environments – isolation, limited code space – usually preclude having a fully implemented shell with the entire development tool chain present on the dedicated processor. Many commercial vendors (VxWorks, Xilinx) include remote debuggers usable over various connection protocols, but for RTEMS, there wasn't a good way to run debuggers on a large cluster of embedded processors.
Till Straumann of SSRL wrote a nice GDB stub to enable remote debugging of an RTEMS system over a local network. This code has been ported to the RCE system and methods have been developed to use it in conjunction with the RCE dynamic linker developed by Steve Tether.
Requirements
The implementation developed at SSRL requires some patches to GDB in order to run. This patched version of GDB-6.8 (current on rhel5), compiled for powerpc-rtems, is available on the lab-1 network in /reg/lab1/home/panetta/bin/powerpc-rtems-gdb
. These patches define a new command rtems
and a new target
type, see below for more details. Also, the core code on the RCE must have the GDB stub compiled in and started automatically. This is true as of core 1.3 (commit 1135). Note: the GDB daemons only run on the development core, not the production core.
Starting out
The GDB process needs to be run from a machine with unrestricted access to the RCE rack. This means that one must run from a machine on the 172.21.6.XXX network, such as rdcds104, or an acta equivalent.
- Create two windows, one on the GDB host, and one for telnetting to the RCE.
- Start GDB against the version of the development core that is on the RCE
/reg/lab1/home/panetta/bin/powerpc-rtems-gdb build/rceapp/bin/ppc-rtems-rce405-dbg/core.1.2.devel
- Connect GDB to the RCE over port 2159 (Ignore the warning. Steve says that this will be normal.)
(gdb) target rtems-remote rce48:2159 Remote debugging using rce48:2159 [New Thread a01000a] [Switching to Thread a01000a] BREAKPOINT () at /reg/lab1/home/panetta/petacache/rce/gdbstub/rtems-gdb-stub-ppc-shared.h:20 20 } warning: /reg/lab1/home/panetta/petacache/build/rceapp/bin/ppc-rtems-rce405-dbg/core.1.2.devel: '.text' section of executable file doesn't match the target's -- do GDB and the target use the same file? Current language: auto; currently c++ (gdb)
At this point, we're sitting in the GDBh thread/task on the RCE, which is stopped as we are connected to it.
8 Thread a010009 ('GDBd' PRI: 20 STATE: ready) 7 Thread a010008 ('TNTD' PRI: 50 STATE: BLOCKED - evt) 6 Thread a010006 ('ntwk' PRI: 80 STATE: ready) 5 Thread a010005 ('ehf0' PRI: 70 STATE: BLOCKED - evt) 4 Thread a010004 ('ehr0' PRI: 80 STATE: BLOCKED - evt) 3 Thread a010003 ('ehc0' PRI: 80 STATE: ready) 2 Thread 9010001 ('IDLE' PRI: 255 STATE: ready) * 1 Thread a01000a ('GDBh' PRI: 200 STATE: stopped - SIGINT)
We can now use breakpoints that are defined in the core development code (such as inside the telnet protocol, or in one of the shell commands.) If we have a task available, we can run it from the shell using runTask. However, as the symbols in this task aren't in memory yet, we cannot yet set a breakpoint inside.
Breakpoints in dynamically linked libraries
To set breakpoints in these libraries, we need a bootstrap procedure. Here is one example of a GDB macro which will bootstrap the breakpoint:
define lmbreak dont-repeat set $lmbreak_file = "$arg0" set $lmbreak_point = "$arg1" break RCE::ELF::RunnableModule::run commands silent set logging file /tmp/lmbreak set logging on printf "add-symbol-file %s 0x%x\n",$lmbreak_file, this + this->textOffset() printf "break %s\n",$lmbreak_point set logging off source /tmp/lmbreak shell rm -f /tmp/lmbreak cont end end document lmbreak Break in an rce loadable module at a specified function, provided as $arg0 and $arg1 Example: lmbreak build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so quarks::service::Logger::initLogging end
This code should be loaded in the user's .gdbinit.
So, to set a breakpoint inside a test program we'll be loading from testQuarks:
(gdb) lmbreak build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so quarks::service::Logger::initLogging Breakpoint 1 at 0x6e618: file elf/RunnableModule.cc, line 17. (gdb) continue
This says: Set the breakpoint quarks::service::Logger::initLogging
using build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so
as the symbol definitions. Note, the initLogging
breakpoint is not set yet, as the symbols aren't extant on the RCE at this point. However, a breakpoint in our dynamic linker's run() method has been defined, and there's where the bootstrap happens.
From a telnet session on the RCE, assuming the proper directories are mounted, we can now run the task:
SHLL [/] # runTask -N QQQQ /build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so 1: runTask -N QQQQ /build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so runTask loaded the task to 0x78c6e00.
The task is now loaded, and the shell is now back, but the QQQQ
task is stopped:
SHLL [/] # task 3: task ID NAME PRI STATE MODES EVENTS WAITID WAITARG NOTES ------------------------------------------------------------------------------ 0a010003 ehc0 80 READY P:T:nA NONE 08424840 0x1fa088 0a010004 ehr0 80 READY P:T:nA NONE 1a010013 0x1fa088 0a010005 ehf0 70 Wevnt P:T:nA NONE 0a010006 ntwk 80 READY P:T:nA NONE 1a010013 0x1fa088 0a010008 TNTD 50 Wevnt P:T:nA NONE 1a010013 0x1fa088 0a010009 GDBd 20 Wevnt P:T:nA NONE 1a010013 0x1fa088 0a01000a GDBh 200 SUSP P:T:nA NONE f6bfd49f 0x1fa088 0a01000c RPCd 80 Wevnt P:T:nA NONE 28434856 0x1fa088 0a01002c pty0 50 READY P:T:nA NONE 1a010013 0x1fa088 0a01002d QQQQ 100 SUSP P:T:nA NONE
If we examine the GDB process window, we see that the new thread 0a1002d
has appeared, and we're now stopped in it in the first line of the quarks::service::Logger::initLogging
method.
add-symbol-file build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so 0x78cce00 break quarks::service::Logger::initLogging add symbol table from file "build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so" at .text_addr = 0x78cce00 Breakpoint 2 at 0x78d0bd4: file ../service/src/Logger.cc, line 64. [New Thread a01002d] [Switching to Thread a01002d] Breakpoint 2, quarks::service::Logger::initLogging (thresh=quarks::service::Logger::Info, imp=0x11f89b0) at ../service/src/Logger.cc:64 64 _threshold = thresh; (gdb)
At this point, we can continue debugging the task.
One important caveat: Every time you run the task, the new breakpoint must be reloaded using lmbreak
. This is because every time the task is run, the code gets loaded at a different point in memory, and the previous set of symbols is no longer valid.