You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Summary

Debugging code on an embedded system is a difficult problem. The basic features of embedded environments – isolation, limited code space – usually preclude having a fully implemented shell with the entire development tool chain present on the dedicated processor. Many commercial vendors (VxWorks, Xilinx) include remote debuggers usable over various connection protocols, but for RTEMS, there wasn't a good way to run debuggers on a large cluster of embedded processors.

Till Straumann of SSRL wrote a nice GDB stub to enable remote debugging of an RTEMS system over a local network. This code has been ported to the RCE system and methods have been developed to use it in conjunction with the RCE dynamic linker developed by Steve Tether.

Requirements

The implementation developed at SSRL requires some patches to GDB in order to run. This patched version of GDB-6.8 (current on rhel5), compiled for powerpc-rtems, is available on the lab-1 network in /reg/lab1/home/panetta/bin/powerpc-rtems-gdb. These patches define a new command rtems and a new target type, see below for more details. Also, the core code on the RCE must have the GDB stub compiled in and started automatically. This is true as of core 1.3 (commit 1135). Note: the GDB daemons only run on the development core, not the production core.

Starting out

The GDB process needs to be run from a machine with unrestricted access to the RCE rack. This means that one must run from a machine on the 172.21.6.XXX network, such as rdcds104, or an acta equivalent.

  • Create two windows, one on the GDB host, and one for telnetting to the RCE.
  • Start GDB against the version of the development core that is on the RCE 
    /reg/lab1/home/panetta/bin/powerpc-rtems-gdb build/rceapp/bin/ppc-rtems-rce405-dbg/core.1.2.devel
  • Connect GDB to the RCE over port 2159 (Ignore the warning. Steve says that this will be normal.) 
    (gdb) target rtems-remote rce48:2159
Remote debugging using rce48:2159
[New Thread  a01000a]
[Switching to Thread  a01000a]
BREAKPOINT () at /reg/lab1/home/panetta/petacache/rce/gdbstub/rtems-gdb-stub-ppc-shared.h:20
20      }
warning: /reg/lab1/home/panetta/petacache/build/rceapp/bin/ppc-rtems-rce405-dbg/core.1.2.devel: 
'.text' section of executable file doesn't match the target's -- do GDB and the target use the same file?
Current language:  auto; currently c++
(gdb)

At this point, we're sitting in the GDBh thread/task on the RCE, which is stopped as we are connected to it.

  8 Thread  a010009 ('GDBd'  PRI:  20 STATE: ready)  
  7 Thread  a010008 ('TNTD'  PRI:  50 STATE: BLOCKED -  evt)  
  6 Thread  a010006 ('ntwk'  PRI:  80 STATE: ready)  
  5 Thread  a010005 ('ehf0'  PRI:  70 STATE: BLOCKED -  evt)  
  4 Thread  a010004 ('ehr0'  PRI:  80 STATE: BLOCKED -  evt)  
  3 Thread  a010003 ('ehc0'  PRI:  80 STATE: ready)  
  2 Thread  9010001 ('IDLE'  PRI: 255 STATE: ready)  
* 1 Thread  a01000a ('GDBh'  PRI: 200 STATE: stopped - SIGINT)

We can now use breakpoints that are defined in the core development code (such as inside the telnet protocol, or in one of the shell commands.) If we have a task available, we can run it from the shell using runTask. However, as the symbols in this task aren't in memory yet, we cannot yet set a breakpoint inside.

Breakpoints in dynamically linked libraries

To set breakpoints in these libraries, we need a bootstrap procedure. Here is one example of a GDB macro which will bootstrap the breakpoint:

define lmbreak
  dont-repeat
  set $lmbreak_file  = "$arg0"
  set $lmbreak_point = "$arg1"
  break RCE::ELF::RunnableModule::run
  commands
    silent
    set logging file /tmp/lmbreak
    set logging on
    printf "add-symbol-file %s 0x%x\n",$lmbreak_file, this + this->textOffset()
    printf "break %s\n",$lmbreak_point
    set logging off
    source /tmp/lmbreak
    shell rm -f /tmp/lmbreak
    cont
  end
end
document lmbreak
Break in an rce loadable module at a specified function, provided as $arg0 and $arg1
Example: 
  lmbreak build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so quarks::service::Logger::initLogging
end

This code should be loaded in the user's .gdbinit.

So, to set a breakpoint inside a test program we'll be loading from testQuarks:

(gdb) lmbreak build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so quarks::service::Logger::initLogging
Breakpoint 1 at 0x6e618: file elf/RunnableModule.cc, line 17.
(gdb) continue

This says: Set the breakpoint quarks::service::Logger::initLogging using build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so as the symbol definitions. Note, the initLogging breakpoint is not set yet, as the symbols aren't extant on the RCE at this point. However, a breakpoint in our dynamic linker's run() method has been defined, and there's where the bootstrap happens.

From a telnet session on the RCE, assuming the proper directories are mounted, we can now run the task:

SHLL [/] # runTask -N QQQQ /build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so
1: runTask -N QQQQ /build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so
runTask loaded the task to 0x78c6e00.

The task is now loaded, and the shell is now back, but the QQQQ task is stopped:

SHLL [/] # task 
3: task
  ID       NAME           PRI  STATE MODES   EVENTS    WAITID  WAITARG  NOTES
------------------------------------------------------------------------------
0a010003   ehc0            80 READY  P:T:nA    NONE   08424840 0x1fa088 
0a010004   ehr0            80 READY  P:T:nA    NONE   1a010013 0x1fa088 
0a010005   ehf0            70 Wevnt  P:T:nA    NONE                     
0a010006   ntwk            80 READY  P:T:nA    NONE   1a010013 0x1fa088 
0a010008   TNTD            50 Wevnt  P:T:nA    NONE   1a010013 0x1fa088 
0a010009   GDBd            20 Wevnt  P:T:nA    NONE   1a010013 0x1fa088 
0a01000a   GDBh           200 SUSP   P:T:nA    NONE   f6bfd49f 0x1fa088 
0a01000c   RPCd            80 Wevnt  P:T:nA    NONE   28434856 0x1fa088 
0a01002c   pty0            50 READY  P:T:nA    NONE   1a010013 0x1fa088 
0a01002d   QQQQ           100 SUSP   P:T:nA    NONE

If we examine the GDB process window, we see that the new thread 0a1002d has appeared, and we're now stopped in it in the first line of the quarks::service::Logger::initLogging method.

add-symbol-file build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so 0x78cce00
break quarks::service::Logger::initLogging
add symbol table from file "build/quarks/mod/ppc-rtems-rce405-dbg/testQuarks.1.0.main.so" at
        .text_addr = 0x78cce00
Breakpoint 2 at 0x78d0bd4: file ../service/src/Logger.cc, line 64.
[New Thread  a01002d]
[Switching to Thread  a01002d]

Breakpoint 2, quarks::service::Logger::initLogging (thresh=quarks::service::Logger::Info, imp=0x11f89b0)
    at ../service/src/Logger.cc:64
64            _threshold = thresh;
(gdb)

At this point, we can continue debugging the task.

One important caveat: Every time you run the task, the new breakpoint must be reloaded using lmbreak. This is because every time the task is run, the code gets loaded at a different point in memory, and the previous set of symbols is no longer valid.

  • No labels