Wednesday 26 September 2012

how to debug segfault from /var/log/message kenel message

Scenario:

Core is not enabled, before program executes, and there was a crash happened , which has logged into /var/log/message

Also make sure that abrtd service stopped.

In case of abrtd service running abrtd ( abort daemon) will not dump a core file nor write error segfault message in /var/log/message.

instantly once application crashed abort daemon will dump a core under /var/spool/abrt/ , upon identifies program is not part of default package/module of system, it will delete dumped core file .

Hence make sure that abort daemon is not running on your system.

There are some cases even if you enable core, if the core file size is  exceeds allowable limit, you may not able to find proper info when you debugging with gdb.

Coming to our discussion,

How to debug a program/process which crashed, and has logged message in /var/log/message

Consider that you have message like this in your /var/log/message

Sep 27 11:25:12 MDP-204 kernel: a.out[4302]: segfault at 0 ip 000000000040047c sp 00007fff3cdf9060 error 6 in a.out[400000+1000]

ip --> instruction pointer
sp --> stack pointer
error 6 --> no address ( means possible crash due to this error no 6)
a.out[400000+1000] --> a.out (program name) base address + virtual memory taken while loading program

hence instruction pointer should be within this range  40000 : 401000

here my test program crashed due to there is no address to store value.

how to get location, using command called "addr2line"

<addr2line> -e <exe name> <instruction pointer addr>
#addr2line -e a.out 000000000040047c
debug.c:7
#

sample program follows,

debug.c

#include <stdio.h>

int main()
{
        char *ptr;

        *ptr = 'a'; //program will crash here

        return 0;
}


Therefore we got useful information that in test program debug.c line number 7 is caused crash.

let me know if anyone has alternative methods for this scenario.


Apart from addr2line, we can use nm, objdump to debug the program

in the above same test program case

generate objdump

#objdump -DCl <exe-name>  > obj.out  -- search instruction pointer in this object file

its also hard, if you are not aware of assembly language , since we would need to map from assembly code into actual source code .. , more over its again depends on the debugging symbols options

 i  always recommend build your program with debug option, ex: gcc -g3 O2 -Wall test.c

Hopefully we could able to get some info from objdump at the final.

#another command objdump --architecture -d <exec-file>   -- this will also help  to find root caused


No comments:

Post a Comment