Using gdb

gdb is a tool that allows you to debug a live-running application, and also debug a crashed application that generated a core file.  The instructions in this doc are for debugging a live-running application.

Debuggable application

An application that is created with debug symbols and without optimizations are much easier to debug.  To accomplish this, EPICS documentation says to create a Makefile.Host file.

https://epics.anl.gov/EpicsDocumentation/AppDevManuals/iocAppBuildSRcontrol.html

However, that did not work for me.  Instead, on the command line, I manually add the compiler flags like this:

bash
$ make BUILD_IOCS=YES USR_CFLAGS="-ggdb -O0" USR_CXXFLAGS="-ggdb -O0"

With that, the debug symbols are included in the resulting executable (-ggdb) and all compiler optimizations are disabled (-O0).

With the debug symbols in the resulting executable, you get an error message when starting gdb and running the executable.  So you need to add this to your ~/.gdbinit file:

set auto-load safe-path /

Beware, with that you are executing python code automatically.  It is possible somebody does something bad to you.

You also may need to add the source directory to your source code path.

(gdb) directory /path/to/src/dir

Example:

(gdb) directory /afs/slac.stanford.edu/u/gu/egumtow/code/ipmiComm/src

And now you should be ok to debug.

bash
egumtow@lcls-dev3 ~/code/ipmiComm/iocs/ipmicomm-test-IOC/iocBoot/iocipmicomm-test-IOC> gdb ../../bin/rhel6-x86_64/ipmicomm-test-IOC
[...]
(gdb) b init_fru_stringin_record
Breakpoint 1 at 0x447f25: file ../devMch.c, line 1509.
(gdb) r ./st.cmd 
Breakpoint 1, init_fru_stringin_record (pstringin=0x8ab520) at ../devMch.c:1509
1509    MchRec  recPvt  = 0; /* Info stored with record */
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6_10.3.x86_64 ncurses-libs-5.7-4.20090207.el6.x86_64 readline-6.0-4.el6.x86_64
(gdb) bt
#0  init_fru_stringin_record (pstringin=0x8ab520) at ../devMch.c:1509
#1  0x00000000004b4b79 in init_record (pcommon=0x8ab520, pass=<value optimized out>) at ../rec/stringinRecord.c:119
#2  0x00000000004fc2cd in doInitRecord1 () at ../misc/iocInit.c:553
#3  iterateRecords () at ../misc/iocInit.c:485
#4  initDatabase () at ../misc/iocInit.c:561
#5  iocBuild_2 () at ../misc/iocInit.c:163
#6  0x00000000004fc505 in iocBuild () at ../misc/iocInit.c:202
#7  0x00000000004fc569 in iocInit () at ../misc/iocInit.c:107
#8  0x0000000000500be9 in iocInitCallFunc (args=<value optimized out>) at ../misc/miscIocRegister.c:25
#9  0x000000000053eee2 in iocshBody (pathname=<value optimized out>, commandLine=0x0, macros=0x0) at ../iocsh/iocsh.cpp:925
#10 0x000000000043e9d0 in main (argc=2, argv=0x7fffffffad58) at ../ipmicomm-test-IOCMain.cpp:17

Future improvements


Note that you see some "value optimized out" in the backtrace.  That is because those frames are inside libraries that we did not compile, but did link to.  Whoever compiled those did so with optimization enabled.  It is typical to have two versions of libraries, one of which is used for debugging like we want to do.  By default the non-debugging libraries are linked in, and if needed the user can link in the debugging libraries by overriding the -L options to gcc and/or setting the LD_LIBRARY_PATH environment variable.  But I don't know where the debugging libraries live or if they exist.

Also note that gdb complains:

bash
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6_10.3.x86_64 ncurses-libs-5.7-4.20090207.el6.x86_64 readline-6.0-4.el6.x86_64

This is because when we installed the OS, we did not install the debug versions of those libraries.  See here that ld wants to link against these libraries:

bash
egumtow@lcls-dev3 ~/code/ipmiComm/iocs/ipmicomm-test-IOC/iocBoot/iocipmicomm-test-IOC> ldd ../../bin/rhel6-x86_64/ipmicomm-test-IOC 
        linux-vdso.so.1 =>  (0x00007ffcc3bec000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3976348000)
        libreadline.so.6 => /lib64/libreadline.so.6 (0x00007f3976105000)
        libncurses.so.5 => /lib64/libncurses.so.5 (0x00007f3975ee3000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f3975cdb000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f3975ad7000)
        libstdc++.so.6 => /afs/slac/g/lcls/package/gcc/gcc-4.9.4/linux-x86_64/lib64/libstdc++.so.6 (0x00007f39757c5000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f3975541000)
        libgcc_s.so.1 => /afs/slac/g/lcls/package/gcc/gcc-4.9.4/linux-x86_64/lib64/libgcc_s.so.1 (0x00007f397532b000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f3974f97000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3976565000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f3974d76000)

We can override ld's preference by using LD_LIBRARY PATH environment variables, and having it first find the debug versions of those libraries.  But again we need to know where those libraries live.

  • No labels