Saturday, January 25, 2014

Suricata - peculiarities when running on virtual guests



In this case we have a Ubuntu with kernel 3.2 as  virtual guest  OS and Surcata latest dev edition as at the moment of this writing.
[This solution blog-post would have not been possible without the help of Victor Julien - his blog]

This is a situation where xen visualization is used and Suricata can not start unless compiled in with "--disable-gccmarch-native" on the particular virtual guest.
There is no other err msg (and/or core file even when compiled with debugging) besides the:  
root@ip-xx-xxx-xxx-xxx:/# suricata -c /etc/suricata/suricata.yaml -i eth0
[14844] 23/1/2014 -- 10:26:32 - (suricata.c:942) <Notice> (SCPrintVersion) -- This is Suricata version 2.0dev (rev a77b9b3)
Illegal instruction (core dumped)

Even when tried (just for the sake of playing with it) sudo or not you can notice the diff between the two commands:
root@ip-xx-xxx-xxx-xxx:/# sudo suricata -c /etc/suricata/suricata.yaml -i eth0 -v
[15562] 23/1/2014 -- 10:58:10 - (suricata.c:942) <Notice> (SCPrintVersion) -- This is Suricata version 2.0dev (rev a77b9b3)
[15562] 23/1/2014 -- 10:58:10 - (util-cpu.c:170) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 1
root@ip-xx-xxx-xxx-xxx:/#
root@ip-xx-xxx-xxx-xxx:/#
root@ip-xx-xxx-xxx-xxx:/# suricata -c /etc/suricata/suricata.yaml -i eth0 -v
[15564] 23/1/2014 -- 10:58:15 - (suricata.c:942) <Notice> (SCPrintVersion) -- This is Suricata version 2.0dev (rev a77b9b3)
[15564] 23/1/2014 -- 10:58:15 - (util-cpu.c:170) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 1
Illegal instruction (core dumped)
root@ip-xx-xxx-xxx-xxx:/#
root@ip-xx-xxx-xxx-xxx:/# whoami
root
root@ip-xx-xxx-xxx-xxx:/#

Notice how in the first case there is not even an err message. In either case Suri never starts and never dumps a core even when it is compiled with CFLAGS (debugging enabled) aka:
CFLAGS="-O0 -ggdb"  ./configure

If we have not used the --disable-gccmarch-native option during the configure stage, can be concluded from the build-info command:
root@ip-xx-xxx-xxx-xxx:/# suricata --build-info
This is Suricata version 2.0dev (rev a77b9b3)
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.6.3, C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
compiled with LibHTP v0.5.9, linked against LibHTP v0.5.9
Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         no
  NFQueue support:                         no
  IPFW support:                            no
  DAG enabled:                             no
  Napatech enabled:                        no
  Unix socket enabled:                     no

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      no
  Prelude support:                         no
  PCRE jit:                                no
  libluajit:                               no
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  CUDA enabled:                            no

  Suricatasc install:                      yes

  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Coccinelle / spatch:                     no

Generic build parameters:
  Installation prefix (--prefix):          /usr
  Configuration directory (--sysconfdir):  /etc/suricata/
  Log directory (--localstatedir) :        /var/log/suricata/

  Host:                                    x86_64-unknown-linux-gnu
  GCC binary:                              gcc
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
root@ip-xx-xxx-xxx-xxx:/#

This above is the default behavior - for GCC march native during the configure stage.
Having run into the above described problem (basically, can't start Suricata) , I did some investigation and
root@ip-xx-xxx-xxx-xxx:/opt/oisf# dmesg |grep virt
[    0.000000] Linux version 3.2.0-54-virtual (buildd@roseapple) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #82-Ubuntu SMP Tue Sep 10 20:31:18 UTC 2013 (Ubuntu 3.2.0-54.82-virtual 3.2.50)
[    0.000000] Booting paravirtualized kernel on Xen
[1960849.933770] Initialising Xen virtual ethernet driver.
root@ip-xx-xxx-xxx-xxx:/opt/oisf#

what do you know ...a virtual machine :)


I wanted to be 100% sure that this is the case , based on a command output, otherwise I suspected it was a virtual server. I did try all of the below commands to determine if it is a virtual machine:
root@ip-xx-xxx-xxx-xxx:/opt/oisf# ethtool -i eth0
driver: vif
version:
firmware-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
root@ip-xx-xxx-xxx-xxx:/opt/oisf# dmidecode | grep -i vmware
root@ip-xx-xxx-xxx-xxx:/opt/oisf# dmidecode
# dmidecode 2.11
# No SMBIOS nor DMI entry point found, sorry.
root@ip-xx-xxx-xxx-xxx:/opt/oisf# cat /proc/scsi/scsi
root@ip-xx-xxx-xxx-xxx:/opt/oisf# lshw -class system
ip-xx-xxx-xxx-xxx        
    description: Computer
    width: 64 bits
    capabilities: vsyscall32
root@ip-xx-xxx-xxx-xxx:/opt/oisf#
root@ip-xx-xxx-xxx-xxx:/opt/oisf#
root@ip-xx-xxx-xxx-xxx:/opt/oisf#
root@ip-xx-xxx-xxx-xxx:/opt/oisf# lspci | grep -i vmware
root@ip-xx-xxx-xxx-xxx:/opt/oisf# lspci | grep -i virt
root@ip-xx-xxx-xxx-xxx:/opt/oisf# ethtool -i eth0
driver: vif
version:
firmware-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
root@ip-xx-xxx-xxx-xxx:/opt/oisf#
root@ip-xx-xxx-xxx-xxx:/opt/oisf#
root@ip-xx-xxx-xxx-xxx:/opt/oisf# dmesg |grep virt
[    0.000000] Linux version 3.2.0-54-virtual (buildd@roseapple) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #82-Ubuntu SMP Tue Sep 10 20:31:18 UTC 2013 (Ubuntu 3.2.0-54.82-virtual 3.2.50)
[    0.000000] Booting paravirtualized kernel on Xen
[1960849.933770] Initialising Xen virtual ethernet driver.
root@ip-xx-xxx-xxx-xxx:/opt/oisf#

only dmesg |grep virt (and hints from ethtool -i eth0 ) returned what I was looking for.

Disabling gcc march native during the configure stage and recompiling  did the trick and I was able to start and run Suri without a problem.
root@ip-xx-xxx-xxx-xxx:/opt/oisf# suricata --build-info
This is Suricata version 2.0dev (rev a77b9b3)
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 4.6.3, C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
compiled with LibHTP v0.5.9, linked against LibHTP v0.5.9
Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         no
  NFQueue support:                         no
  IPFW support:                            no
  DAG enabled:                             no
  Napatech enabled:                        no
  Unix socket enabled:                     no

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      no
  Prelude support:                         no
  PCRE jit:                                no
  libluajit:                               no
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  CUDA enabled:                            no

  Suricatasc install:                      yes

  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Coccinelle / spatch:                     no

Generic build parameters:
  Installation prefix (--prefix):          /usr
  Configuration directory (--sysconfdir):  /etc/suricata/
  Log directory (--localstatedir) :        /var/log/suricata/

  Host:                                    x86_64-unknown-linux-gnu
  GCC binary:                              gcc
  GCC Protect enabled:                     no
  GCC march native enabled:                no
  GCC Profile enabled:                     no
root@ip-xx-xxx-xxx-xxx:/opt/oisf#

NOTICE:
GCC march native enabled:                no
You would get the above result when compiling this way (this build is using the latest git dev edition at the moment of this writing):
git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && CFLAGS="-O0 -ggdb"  ./configure \
--prefix=/usr --sysconfdir=/etc --localstatedir=/var \
--disable-gccmarch-native \
--enable-geoip \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
&& sudo make clean \
&& sudo make \
&& sudo make install \
&& sudo ldconfig

as compared with:
git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && CFLAGS="-O0 -ggdb"  ./configure \
--prefix=/usr --sysconfdir=/etc --localstatedir=/var \
--enable-geoip \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
&& sudo make clean \
&& sudo make \
&& sudo make install \
&& sudo ldconfig


Notice "--disable-gccmarch-native" is missing in the second one.
The most important thing is to configure/compile with --disable-gccmarch-native on a virtual guest, if you get the same problem.


Saturday, January 11, 2014

Git - merging branches, rebasing and things..




I was wondering .. how (if there is a way) I can merge the latest
current Suricata dev git master and a git pull request that has not yet been
introduced into the git master?

example:
I want to git clone the latest Suricata git master... and then apply to it
Tom Decanio's github branch for ALL JSON output -
(git clone https://github.com/decanio/suricata-np.git -b dev-np-work1.3 )

What is the best way to do that?

....
well (with some invaluable help from  Regit):


 git clone git://phalanx.openinfosecfoundation.org/oisf.git
 cd oisf/
 git remote add decanio https://github.com/decanio/suricata-np.git
 git fetch decanio
 git checkout -b my-dev-np-work1.3 decanio/dev-np-work1.3
 git fetch origin
 git rebase origin/master

you're done (if there are no errors during the re-basing phase :) ) !



Tuesday, December 31, 2013

Suricata cocktails (handy one-liners)




Some of my favorite cocktails (one-liners) :)

Suricata cocktails with git master. Tested on Ubuntu and Debian.
You can just copy/paste.

Before you start, make sure you have the below packages installed.

General packages needed:
apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev \
build-essential autoconf automake libtool libpcap-dev libnet1-dev \
libyaml-0-2 libyaml-dev zlib1g zlib1g-dev libcap-ng-dev libcap-ng0 \
make flex bison git git-core subversion libmagic-dev


For MD5 support(file extraction):
apt-get install libnss3-dev libnspr4-dev


For GeoIP:
apt-get install libgeoip1 libgeoip-dev


For the first three (3) cocktails/recipes you would need:
  1. PF_RING as explained HERE 
  2. luajit as explained HERE
and use Suricata's git master - latest dev edition.


Cocktail 1

Suricata - latest dev edition plus enabled:
  1. pf_ring
  2. Luajit scripting
  3. GeoIP
  4. Filemagic/MD5

In case you get the pfring err:
    checking for pfring_open in -lpfring... no

       ERROR! --enable-pfring was passed but the library was not found
       or version is >4, go get it
       from http://www.ntop.org/PF_RING.html

The "LIBS=-lrt" infront of "./configure" below addresses that problem (err message above)


git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && LIBS=-lrt ./configure  --enable-pfring --enable-luajit --enable-geoip \
--with-libpfring-includes=/usr/local/pfring/include/ \
--with-libpfring-libraries=/usr/local/pfring/lib/ \
--with-libpcap-includes=/usr/local/pfring/include/ \
--with-libpcap-libraries=/usr/local/pfring/lib/ \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
--with-libluajit-includes=/usr/local/include/luajit-2.0/ \
--with-libluajit-libraries=/usr/lib/x86_64-linux-gnu/ \
&& sudo make clean && sudo make && sudo make install \
&& sudo ldconfig


Cocktail 2

Suricata - latest dev edition plus enabled:
  1. pf_ring
  2. Luajit scripting
  3. GeoIP
  4. Filemagic/MD5
  5. Debugging

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && CFLAGS="-O0 -ggdb"  \
./configure  \
--enable-pfring --enable-luajit --enable-geoip \
--with-libpfring-includes=/usr/local/pfring/include/ \
--with-libpfring-libraries=/usr/local/pfring/lib/ \
--with-libpcap-includes=/usr/local/pfring/include/ \
--with-libpcap-libraries=/usr/local/pfring/lib/ \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
--with-libluajit-includes=/usr/local/include/luajit-2.0/ \
--with-libluajit-libraries=/usr/lib/x86_64-linux-gnu/ \
&& sudo make clean && sudo make && sudo make install \
&& sudo ldconfig


Cocktail 3

Suricata - latest dev edition plus enabled:
  1. pf_ring
  2. Luajit scripting
  3. GeoIP
  4. Filemagic/MD5


git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && ./configure  \
--enable-pfring --enable-luajit --enable-geoip \
--with-libpfring-includes=/usr/local/pfring/include/ \
--with-libpfring-libraries=/usr/local/pfring/lib/ \
--with-libpcap-includes=/usr/local/pfring/include/ \
--with-libpcap-libraries=/usr/local/pfring/lib/ \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
--with-libluajit-includes=/usr/local/include/luajit-2.0/ \
--with-libluajit-libraries=/usr/lib/x86_64-linux-gnu/ \
&& sudo make clean && sudo make && sudo make install \
&& sudo ldconfig

Cocktail 4

Suricata - latest dev edition plus enabled:
  1. GeoIP
  2. Filemagic/MD5

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && ./configure --enable-geoip \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
&& sudo make clean && sudo make && sudo make install \
&& sudo ldconfig

Cocktail 5

Suricata - latest dev edition plus enabled:
  1. GeoIP

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && ./configure --enable-geoip \
&& sudo make clean && sudo make && sudo make install \
&& sudo ldconfig






Cocktail 6

Suricata - latest dev edition plus enabled:
  1. Filemagic/MD5

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && ./configure \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
&& sudo make clean && sudo make && sudo make install \
&& sudo ldconfig



Cocktail 7

Suricata - latest dev edition - default

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && \
git clone https://github.com/ironbee/libhtp.git -b 0.5.x && \
./autogen.sh && ./configure \
&& sudo make clean \
&& sudo make \
&& sudo make install \
&& sudo ldconfig



Issue  - suricata --build-info - to verify after compile and installation.
You could twist it anyway you want - depending on library locations and features enabled in Suricata.




Monday, December 30, 2013

Suricata - setting up flows





So looking at the suricata.log file (after starting suricata):


root@suricata:/var/data/log/suricata# more suricata.log                                                       
 [1372] 17/12/2013 -- 17:47:35 - (suricata.c:962) <Notice> (SCPrintVersion) -- This is Suricata version 2.0dev (rev e7f6107)
[1372] 17/12/2013 -- 17:47:35 - (util-cpu.c:170) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 16
[1372] 17/12/2013 -- 17:47:35 - (app-layer-dns-udp.c:315) <Info> (DNSUDPConfigure) -- DNS request flood protection level: 500
[1372] 17/12/2013 -- 17:47:35 - (defrag-hash.c:212) <Info> (DefragInitConfig) -- allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
[1372] 17/12/2013 -- 17:47:35 - (defrag-hash.c:237) <Info> (DefragInitConfig) -- preallocated 65535 defrag trackers of size 152
[1372] 17/12/2013 -- 17:47:35 - (defrag-hash.c:244) <Info> (DefragInitConfig) -- defrag memory usage: 13631336 bytes, maximum: 536870912
[1372] 17/12/2013 -- 17:47:35 - (tmqh-flow.c:76) <Info> (TmqhFlowRegister) -- AutoFP mode using default "Active Packets" flow load balancer
[1373] 17/12/2013 -- 17:47:35 - (tmqh-packetpool.c:142) <Info> (PacketPoolInit) -- preallocated 65534 packets. Total memory 229106864
[1373] 17/12/2013 -- 17:47:35 - (host.c:205) <Info> (HostInitConfig) -- allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
[1373] 17/12/2013 -- 17:47:35 - (host.c:228) <Info> (HostInitConfig) -- preallocated 1000 hosts of size 112
[1373] 17/12/2013 -- 17:47:35 - (host.c:230) <Info> (HostInitConfig) -- host memory usage: 390144 bytes, maximum: 16777216

[1373] 17/12/2013 -- 17:47:36 - (flow.c:386) <Info> (FlowInitConfig) -- allocated 1006632960 bytes of memory for the flow hash... 15728640 buckets of size 64
[1373] 17/12/2013 -- 17:47:37 - (flow.c:410) <Info> (FlowInitConfig) -- preallocated 8000000 flows of size 280

[1373] 17/12/2013 -- 17:47:37 - (flow.c:412) <Info> (FlowInitConfig) -- flow memory usage: 3310632960 bytes, maximum: 6442450944
[1373] 17/12/2013 -- 17:47:37 - (reputation.c:459) <Info> (SRepInit) -- IP reputation disabled
[1373] 17/12/2013 -- 17:47:37 - (util-magic.c:62) <Info> (MagicInit) -- using magic-file /usr/share/file/magic
[1373] 17/12/2013 -- 17:47:37 - (suricata.c:1769) <Info> (SetupDelayedDetect) -- Delayed detect disabled



We see:

[1373] 17/12/2013 -- 17:47:36 - (flow.c:386) <Info> (FlowInitConfig) -- allocated 1006632960 bytes of memory for the flow hash... 15728640 buckets of size 64
[1373] 17/12/2013 -- 17:47:37 - (flow.c:410) <Info> (FlowInitConfig) -- preallocated 8000000 flows of size 280

-> This is approximatelly 3GB of RAM
How did we get to this number... well ... I  have custom defined it in suricata.yaml under the flow section:
  hash-size: 15728640
  prealloc: 8000000

So we need to sum up->
15728640 x 64 ("15728640 buckets of size 64" = 1006632960 bytes)
+
8000000 x 280 ("preallocated 8000000 flows of size 280" = 2240000000 bytes )
=
total of 3246632960 bytes which is 3096.23MB

(15728640 x 64) + (8000000 x 280) =  3246632960 bytes


That would define our flow memcap value in suricata.yaml.
So this would work like this ->

flow:
  memcap: 4gb
  hash-size: 15728640
  prealloc: 8000000
  emergency-recovery: 30


This would work as well ->

flow:
  memcap: 3500mb
  hash-size: 15728640
  prealloc: 8000000
  emergency-recovery: 30


That's it :)

Saturday, December 21, 2013

Playing with segfaults , core dumps and such.

How to determine when (time-wise) you had a segfault with/from any tool,software or program:

dmesg | gawk -v uptime=$( grep btime /proc/stat | cut -d ' ' -f 2 ) '/^[[ 0-9.]*]/ { print strftime("[%Y/%m/%d %H:%M:%S]", substr($0,2,index($0,".")-2)+uptime) substr($0,index($0,"]")+1) }' |grep segf

Like so:
root@suricata:/root# dmesg | gawk -v uptime=$( grep btime /proc/stat | cut -d ' ' -f 2 ) '/^[[ 0-9.]*]/ { print strftime("[%Y/%m/%d %H:%M:%S]", substr($0,2,index($0,".")-2)+uptime) substr($0,index($0,"]")+1) }' |grep segf
[2013/12/19 02:21:49] AFPacketeth38[8874]: segfault at 17c ip 00007f1831f919a0 sp 00007f181c85b5d0 error 4 in libhtp-0.5.8.so.1.0.0[7f1831f83000+1d000]
root@suricata:/root#


The command above will give you the exact time when Suricata issued a core dump -
[2013/12/19 02:21:49] AFPacketeth38[8874]: segfault...


You could also speed up things :)
[force Suricata to core dump/crash (or many other software products) ...part of my job description :) ]
If this is what you want -

1) Start Suricata.
2) Kill it with an abort signal.
sudo kill -n ABRT `pidof suricata`


Suricata will now abort and dump core. Then issue the following command:
gdb /usr/bin/suricata /var/data/peter/crashes/suricata/core

/usr/bin/suricata  - is the location of the suricata binary (if not sure, issue the command-
which suricata)
/var/data/peter/crashes/suricata/core - this is the location/name of the core file


The location of the core dump file could be specified in suricata.yaml:
# Daemon working directory
# Suricata will change directory to this one if provided
# Default: "/"
daemon-directory: "/var/data/peter/crashes/suricata"


Once in  gdb:
thread apply all bt


NOTE: To be able to get any useful info of the core dump file, you should compile Suricata with CFLAGS, like so:
CFLAGS="-O0 -ggdb"  ./configure

instead of just
./configure



Suri 2.0beta2 very informative - when you need it

 With the release of Suricata 2.0beta2 one can notice right away a few of the many changes.

 root@LTS-64-1:~# suricata -c /etc/suricata/suricata.yaml -i eth0 -v
19/12/2013 -- 08:57:48 - <Notice> - This is Suricata version 2.0beta2 RELEASE
19/12/2013 -- 08:57:48 - <Info> - CPUs/cores online: 2
19/12/2013 -- 08:57:48 - <Info> - 'default' server has 'request-body-minimal-inspect-size' set to 33882 and 'request-body-inspect-window' set to 4053 after randomization.
19/12/2013 -- 08:57:48 - <Info> - 'default' server has 'response-body-minimal-inspect-size' set to 33695 and 'response-body-inspect-window' set to 4218 after randomization.
19/12/2013 -- 08:57:48 - <Info> - 'apache' server has 'request-body-minimal-inspect-size' set to 34116 and 'request-body-inspect-window' set to 3973 after randomization.
19/12/2013 -- 08:57:48 - <Info> - 'apache' server has 'response-body-minimal-inspect-size' set to 32229 and 'response-body-inspect-window' set to 4205 after randomization.
19/12/2013 -- 08:57:48 - <Info> - 'iis7' server has 'request-body-minimal-inspect-size' set to 32040 and 'request-body-inspect-window' set to 4118 after randomization.
19/12/2013 -- 08:57:48 - <Info> - 'iis7' server has 'response-body-minimal-inspect-size' set to 32694 and 'response-body-inspect-window' set to 4148 after randomization.
19/12/2013 -- 08:57:48 - <Info> - DNS request flood protection level: 500
19/12/2013 -- 08:57:48 - <Info> - Found an MTU of 1500 for 'eth0'
19/12/2013 -- 08:57:48 - <Info> - allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
19/12/2013 -- 08:57:48 - <Info> - preallocated 65535 defrag trackers of size 152
19/12/2013 -- 08:57:48 - <Info> - defrag memory usage: 13631336 bytes, maximum: 33554432
19/12/2013 -- 08:57:48 - <Info> - AutoFP mode using default "Active Packets" flow load balancer
19/12/2013 -- 08:57:48 - <Info> - preallocated 1024 packets. Total memory 3567616
19/12/2013 -- 08:57:48 - <Info> - allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
19/12/2013 -- 08:57:48 - <Info> - preallocated 1000 hosts of size 112
19/12/2013 -- 08:57:48 - <Info> - host memory usage: 390144 bytes, maximum: 16777216
19/12/2013 -- 08:57:48 - <Info> - allocated 4194304 bytes of memory for the flow hash... 65536 buckets of size 64
19/12/2013 -- 08:57:48 - <Info> - preallocated 10000 flows of size 280
19/12/2013 -- 08:57:48 - <Info> - flow memory usage: 7074304 bytes, maximum: 134217728
19/12/2013 -- 08:57:48 - <Info> - IP reputation disabled
19/12/2013 -- 08:57:48 - <Info> - using magic-file /usr/share/file/magic
19/12/2013 -- 08:57:48 - <Info> - Delayed detect disabled
19/12/2013 -- 08:57:53 - <Info> - 48 rule files processed. 14045 rules successfully loaded, 0 rules failed
19/12/2013 -- 08:57:53 - <Info> - 14053 signatures processed. 1136 are IP-only rules, 4310 are inspecting packet payload, 10513 inspect application layer, 72 are decoder event only
19/12/2013 -- 08:57:53 - <Info> - building signature grouping structure, stage 1: preprocessing rules... complete
19/12/2013 -- 08:57:54 - <Info> - building signature grouping structure, stage 2: building source address list... complete
19/12/2013 -- 08:58:00 - <Info> - building signature grouping structure, stage 3: building destination address lists... complete
19/12/2013 -- 08:58:03 - <Info> - Threshold config parsed: 0 rule(s) found
19/12/2013 -- 08:58:03 - <Info> - Core dump size set to unlimited.
19/12/2013 -- 08:58:03 - <Info> - fast output device (regular) initialized: fast.log
19/12/2013 -- 08:58:03 - <Info> - http-log output device (regular) initialized: http.log
19/12/2013 -- 08:58:03 - <Info> - dns-log output device (regular) initialized: dns.log
19/12/2013 -- 08:58:03 - <Info> - file-log output device (regular) initialized: files-json.log
19/12/2013 -- 08:58:03 - <Info> - forcing magic lookup for logged files
19/12/2013 -- 08:58:03 - <Info> - forcing md5 calculation for logged files
19/12/2013 -- 08:58:03 - <Info> - Using 1 live device(s).
19/12/2013 -- 08:58:03 - <Info> - using interface eth0
19/12/2013 -- 08:58:03 - <Info> - Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
19/12/2013 -- 08:58:03 - <Info> - Found an MTU of 1500 for 'eth0'
19/12/2013 -- 08:58:03 - <Info> - Set snaplen to 1516 for 'eth0'
19/12/2013 -- 08:58:03 - <Info> - Generic Receive Offload is set on eth0
19/12/2013 -- 08:58:03 - <Info> - Large Receive Offload is unset on eth0
19/12/2013 -- 08:58:03 - <Warning> - [ERRCODE: SC_ERR_PCAP_CREATE(21)] - Using Pcap capture with GRO or LRO activated can lead to capture problems.
19/12/2013 -- 08:58:03 - <Info> - RunModeIdsPcapAutoFp initialised
19/12/2013 -- 08:58:03 - <Info> - stream "prealloc-sessions": 2048 (per thread)
19/12/2013 -- 08:58:03 - <Info> - stream "memcap": 536870912
19/12/2013 -- 08:58:03 - <Info> - stream "midstream" session pickups: disabled
19/12/2013 -- 08:58:03 - <Info> - stream "async-oneside": disabled
19/12/2013 -- 08:58:03 - <Info> - stream "checksum-validation": disabled
19/12/2013 -- 08:58:03 - <Info> - stream."inline": disabled
19/12/2013 -- 08:58:03 - <Info> - stream "max-synack-queued": 5
19/12/2013 -- 08:58:03 - <Info> - stream.reassembly "memcap": 1073741824
19/12/2013 -- 08:58:03 - <Info> - stream.reassembly "depth": 8388608
19/12/2013 -- 08:58:03 - <Info> - stream.reassembly "toserver-chunk-size": 2447
19/12/2013 -- 08:58:03 - <Info> - stream.reassembly "toclient-chunk-size": 2489
19/12/2013 -- 08:58:03 - <Info> - stream.reassembly.raw: enabled
19/12/2013 -- 08:58:03 - <Notice> - all 4 packet processing threads, 3 management threads initialized, engine started.
19/12/2013 -- 08:58:32 - <Info> - No packets with invalid checksum, assuming checksum offloading is NOT used




Note 1)
19/12/2013 -- 08:58:03 - <Info> - Generic Receive Offload is set on eth0
19/12/2013 -- 08:58:03 - <Info> - Large Receive Offload is unset on eth0
19/12/2013 -- 08:58:03 - <Warning> - [ERRCODE: SC_ERR_PCAP_CREATE(21)] - Using Pcap capture with GRO or LRO activated can lead to capture problems.

Note 2 ...after some packets)
19/12/2013 -- 08:58:32 - <Info> - No packets with invalid checksum, assuming checksum offloading is NOT used

So  for Note 1)  if we check our interface using ethtool, (if you do not have it  -
apt-get install ethtool on Ubuntu/Debian like systems ):

root@LTS-64-1:~# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off
root@LTS-64-1:~#

We see that :
generic-receive-offload: on
large-receive-offload: off

exactly as Suricata reports.
(
do not forget to run it with the -v option ! :
suricata -c /etc/suricata/suricata.yaml -i eth0 -v
)
Do not forget  - all offloading and checksumming features should be OFF(disabled) on the network interface, so that Suricata processes correctly all the traffic !
 

Saturday, December 14, 2013

Suricata - per host/network fragmentation timeouts


Suricata can have the ip fragmentation time out values on a configurable per network/host basis in the suricata.yaml.

Through a light but time consuming research it (frag timeout) seems to be different for the different OSs . It does not matter if the system is 32 or 64 bit, but it does matter if it handles IPv4 or IPv6 addresses. Most of those values you can find under /proc/sys/net/ipv4/ipfrag_time (or using sysctl -a) on most of the Linux/Unix systems – the amount of time a fragment will be kept in memory, after that it will be discarded.

All  (default) values in seconds ->
IPv4:
Suse - 20
CentOS - 30
Ubuntu - 30
Debian - 30
Fedora - 30
Windows (all) - hard coded, can not be changed – 60

IPv6:
Suse - 60
CentOS - 60
Ubuntu - 60
Debian - 60
Fedora - 60
Windows (all) – hardcoded, can not be changed – 60

There are other ip fragmentation values that differ for the different OSs as well. Some values could not have the default value due to network/OS/application specific tuning and other reasons.

However for those hosts and networks that you are sure and know what the timeouts are in seconds - you could use the defrag timeout values in suricata.yaml section and accordingly. That way Suricata will inspect the ip fragments with the same timeouts as the receiving hosts.


Setup defrag timeouts on a per network/host type basics:
# Enable defrag per host settings
  host-config:

    - dmz:
        timeout: 30
        address: [192.168.1.0/24, 127.0.0.0/8, 1.1.1.0/24, 2.2.2.0/24, "1.1.1.1", "2.2.2.2", "::1"]

    - lan:
        timeout: 45
        address:
          - 192.168.0.0/24
          - 192.168.10.0/24
          - 172.16.14.0/24