how to debug segfault from /var/log/message kenel message

Scenario:

Core is not enabled, before program executes, and there was a crash happened , which has logged into /var/log/message

Also make sure that abrtd service stopped.

In case of abrtd service running abrtd ( abort daemon) will not dump a core file nor write error segfault message in /var/log/message.

instantly once application crashed abort daemon will dump a core under /var/spool/abrt/ , upon identifies program is not part of default package/module of system, it will delete dumped core file .

Hence make sure that abort daemon is not running on your system.

There are some cases even if you enable core, if the core file size is  exceeds allowable limit, you may not able to find proper info when you debugging with gdb.

Coming to our discussion,

How to debug a program/process which crashed, and has logged message in /var/log/message

Consider that you have message like this in your /var/log/message

Sep 27 11:25:12 MDP-204 kernel: a.out[4302]: segfault at 0 ip 000000000040047c sp 00007fff3cdf9060 error 6 in a.out[400000+1000]

ip --> instruction pointer
sp --> stack pointer
error 6 --> no address ( means possible crash due to this error no 6)
a.out[400000+1000] --> a.out (program name) base address + virtual memory taken while loading program

hence instruction pointer should be within this range  40000 : 401000

here my test program crashed due to there is no address to store value.

how to get location, using command called "addr2line"

<addr2line> -e <exe name> <instruction pointer addr>
#addr2line -e a.out 000000000040047c
debug.c:7
#

sample program follows,

debug.c

#include <stdio.h>

int main()
{
        char *ptr;

        *ptr = 'a'; //program will crash here

        return 0;
}


Therefore we got useful information that in test program debug.c line number 7 is caused crash.

let me know if anyone has alternative methods for this scenario.


Apart from addr2line, we can use nm, objdump to debug the program

in the above same test program case

generate objdump

#objdump -DCl <exe-name>  > obj.out  -- search instruction pointer in this object file

its also hard, if you are not aware of assembly language , since we would need to map from assembly code into actual source code .. , more over its again depends on the debugging symbols options

 i  always recommend build your program with debug option, ex: gcc -g3 O2 -Wall test.c

Hopefully we could able to get some info from objdump at the final.

#another command objdump --architecture -d <exec-file>   -- this will also help  to find root caused

HowTo Shared Library in Linux

Library Name : hello

filename : hello.c

#include <stdio.h>

void Hello()
{
  printf("Hello World .. How are you 1\n");
}
======================================
Header File:

hello.h

#ifndef __HELLO_H
#define __HELLO_H

void Hello(void);

#endif

===============================

Application :

filename : app.c

#include <stdio.h>
#include "hello.h"

int main()
{

  Hello();

  return 0;
}

====================================

Makefile:

VERSION = 1
MINORVER = 1
RELEASEVER = 1

LIBNAME = libhello
DYLIBSUFFIX = so

SONAME = $(LIBNAME).$(DYLIBSUFFIX).$(VERSION)
REALNAME = $(SONAME).$(MINORVER).$(RELEASEVER)

MAJORNAME = $(SONAME)
DYLIBNAME = $(LIBNAME).$(DYLIBSUFFIX)


OBJS = hello.o

all: $(LIBNAME)

$(LIBNAME): $(OBJS)
  $(CC) -shared -Wl,-soname,$(SONAME) -o $(REALNAME) $(OBJS)
  ln -sf $(REALNAME) $(MAJORNAME) && ln -sf $(MAJORNAME) $(DYLIBNAME)

hello.o: hello.c
  $(CC) -fPIC -c -g -Wall hello.c

app: app.o
  $(CC) -o $@ app.o -L./ -lhello

clean:
  -@rm *.o

distclean:  clean
  -@rm libhello.so* app

SCP/SSH without entering password

1. Generate public key at local host
follows,

[root@hp6core1 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
/root/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
e6:46:ca:ee:10:25:1f:7c:3f:25:de:24:3d:c2:6a:20 root@hp6core1
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|     .   . .     |
|    E = . = =    |
|     = + + B .   |
|    . . S + .    |
|     o *   .     |
|    . o o        |
|     o .         |
|     .o          |
+-----------------+

2. copy pub file to remote host , follows as
[root@hp6core1 ~]# scp /root/.ssh/id_rsa.pub remoet-host:/root/.ssh/authorized_keys

3. Otherwise append the content into authorized keys at remote file in case file exists

IPV6 Simple Server Program

/**************************************************************************/
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <string.h>

#define SERVER_PORT     3005
#define BUFFER_LENGTH    250
#define FALSE              0

void main()
{
                int sd=-1, sdconn=-1;
                int rc, on=1, rcdsize=BUFFER_LENGTH;
                char buffer[BUFFER_LENGTH];
                struct sockaddr_in6 serveraddr, clientaddr;
                int addrlen=sizeof(clientaddr);
                char str[INET6_ADDRSTRLEN];

                do
                {
                                if ((sd = socket(AF_INET6, SOCK_STREAM, 0)) < 0)
                                {
                                                perror("socket() failed");
                                                break;
                                }

                                if (setsockopt(sd, SOL_SOCKET, SO_REUSEADDR,
                                                                                (char *)&on,sizeof(on)) < 0)
                                {
                                                perror("setsockopt(SO_REUSEADDR) failed");
                                                break;
                                }

                                memset(&serveraddr, 0, sizeof(serveraddr));
                                serveraddr.sin6_family = AF_INET6;
                                serveraddr.sin6_port   = htons(SERVER_PORT);

                                serveraddr.sin6_addr   = in6addr_any;
  if (bind(sd,
                                                                                (struct sockaddr *)&serveraddr,
                                                                                sizeof(serveraddr)) < 0)
                                {
                                                perror("bind() failed");
                                                break;
                                }

                                if (listen(sd, 128) < 0)
                                {
                                                perror("listen() failed");
                                                break;
                                }

                                printf("Ready for client connect().\n");

                                addrlen=sizeof(clientaddr);
                                if ((sdconn = accept(sd, (struct sockaddr *)&clientaddr, &addrlen)) < 0)
                                {
                                                perror("accept() failed");
                                                break;
                                }
                                else
                                {
                                                //getpeername(sdconn, (struct sockaddr *)&clientaddr, &addrlen);
                                                if(inet_ntop(AF_INET6, &clientaddr.sin6_addr, str, sizeof(str))) {
                                                                printf("Client address is %s\n", str);
                                                                printf("Client IP Family [%d]\n", clientaddr.sin6_family);
                                                                printf("Client port is %d\n", ntohs(clientaddr.sin6_port));
                                                }
                                }

                                /********************************************************************/
                                /* In this example we know that the client will send 250 bytes of   */
                                /* data over.  Knowing this, we can use the SO_RCVLOWAT socket      */
                                /* option and specify that we don't want our recv() to wake up      */
                                /* until all 250 bytes of data have arrived.                        */
                                /********************************************************************/
                                if (setsockopt(sdconn, SOL_SOCKET, SO_RCVLOWAT,
                                                                                (char *)&rcdsize,sizeof(rcdsize)) < 0)
                                {
                                                perror("setsockopt(SO_RCVLOWAT) failed");
                                                break;
  }

                                rc = recv(sdconn, buffer, sizeof(buffer), 0);
                                if (rc < 0)
                                {
                                                perror("recv() failed");
                                                break;
                                }

                                printf("%d bytes of data were received\n", rc);
                                if (rc == 0 ||
                                                                rc < sizeof(buffer))
                                {
                                                printf("The client closed the connection before all of the\n");
                                                printf("data was sent\n");
                                                break;
                                }

                                rc = send(sdconn, buffer, sizeof(buffer), 0);
                                if (rc < 0)
                                {
                                                perror("send() failed");
                                                break;
                                }

                } while (FALSE);

                if (sd != -1)
                                close(sd);
                if (sdconn != -1)
                                close(sdconn);
}