Saturday, December 21, 2013

Playing with segfaults , core dumps and such.

How to determine when (time-wise) you had a segfault with/from any tool,software or program:

dmesg | gawk -v uptime=$( grep btime /proc/stat | cut -d ' ' -f 2 ) '/^[[ 0-9.]*]/ { print strftime("[%Y/%m/%d %H:%M:%S]", substr($0,2,index($0,".")-2)+uptime) substr($0,index($0,"]")+1) }' |grep segf

Like so:
root@suricata:/root# dmesg | gawk -v uptime=$( grep btime /proc/stat | cut -d ' ' -f 2 ) '/^[[ 0-9.]*]/ { print strftime("[%Y/%m/%d %H:%M:%S]", substr($0,2,index($0,".")-2)+uptime) substr($0,index($0,"]")+1) }' |grep segf
[2013/12/19 02:21:49] AFPacketeth38[8874]: segfault at 17c ip 00007f1831f919a0 sp 00007f181c85b5d0 error 4 in libhtp-0.5.8.so.1.0.0[7f1831f83000+1d000]
root@suricata:/root#


The command above will give you the exact time when Suricata issued a core dump -
[2013/12/19 02:21:49] AFPacketeth38[8874]: segfault...


You could also speed up things :)
[force Suricata to core dump/crash (or many other software products) ...part of my job description :) ]
If this is what you want -

1) Start Suricata.
2) Kill it with an abort signal.
sudo kill -n ABRT `pidof suricata`


Suricata will now abort and dump core. Then issue the following command:
gdb /usr/bin/suricata /var/data/peter/crashes/suricata/core

/usr/bin/suricata  - is the location of the suricata binary (if not sure, issue the command-
which suricata)
/var/data/peter/crashes/suricata/core - this is the location/name of the core file


The location of the core dump file could be specified in suricata.yaml:
# Daemon working directory
# Suricata will change directory to this one if provided
# Default: "/"
daemon-directory: "/var/data/peter/crashes/suricata"


Once in  gdb:
thread apply all bt


NOTE: To be able to get any useful info of the core dump file, you should compile Suricata with CFLAGS, like so:
CFLAGS="-O0 -ggdb"  ./configure

instead of just
./configure



1 comment:

  1. This post was rather helpful, thanks a lot!

    One thing that I noticed was that on my Ubuntu (20.04), when I tried to kill Suricata with abort signal, I had to change the command for `kill` to accept it:
    $sudo kill -s SIGABRT `pidof suricata`
    Sharing in case any folks coming from the future also end up here, like I did.

    ReplyDelete