Wednesday, December 4, 2013

Suricata (and the grand slam of) Open Source IDPS - Chapter II - PF_RING / DNA , Part One - PF_RING

 

 

Introduction


This is Chapter II (Part One - PF_RING) of a series of articles about high performance  and advance tuning of Suricata IDPS

This article will consist of two parts  - on setting up and configuring PF_RING and DNA for a 10Gbps interface monitoring.

PF_RING:
PF_RING™ is a new type of network socket that dramatically improves the packet capture speed...
PF_RING DNA:
PF_RING™ DNA (Direct NIC Access) is a way to map NIC memory and registers to userland so that there is no additional packet copy besides the DMA transfer done by the NIC NPU (Network Process Unit), unlike what happens with NAPI. This results in better performance as CPU cycles are used uniquely for consuming packets and not for moving them off the adapter...


NOTE: PF_RING™ is open source and free, for the DNA part you need a license. However the DNA license is free for non-profit organizations or education institutions (universities,colleges etc.)




Part One - PF_RING


If you have pf_ring already installed, you might want to do:
sudo rmmod pf_ring
If you are not sure if you have pf_ring installed , you can do:
sudo modinfo pf_ring


Get the latest pf_ring sources:
git clone https://github.com/ntop/PF_RING.git


Compile and install PF_RING


Next, enter the following commands for configuration and installation:
(!!! NOT AS ROOT !!!)

    cd PF_RING/kernel/
    make && sudo make install
    cd ../userland/lib
    ./configure --prefix=/usr/local/pfring && make && sudo make install
    cd ../libpcap
    ./configure --prefix=/usr/local/pfring && make && sudo make install
    cd ../tcpdump-4.1.1
    ./configure --prefix=/usr/local/pfring && make && sudo make install
    sudo ldconfig
  

Then we load the module:
sudo modprobe pf_ring
  
Elevate as root and check if you have everything you need -enter:
modinfo pf_ring && cat /proc/net/pf_ring/info
   
Increase the throttle rate of the ixgbe module:
modprobe ixgbe InterruptThrottleRate=4000



The default pf_ring setup will look something like this:
root@suricata:/var/og/suricata# cat /proc/net/pf_ring/info
PF_RING Version          : 5.6.2 ($Revision: exported$)
Total rings              : 16
Standard (non DNA) Options
Ring slots               : 4096
Slot version             : 15
Capture TX               : Yes [RX+TX]
IP Defragment            : No
Socket Mode              : Standard
Transparent mode         : Yes [mode 0]
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0


Notice the ring slots above. We would actually like to increase that in order to meet the needs of a high speed network that we are going to monitor with Suricata.

So we do:
rmmod pf_ring
modprobe pf_ring transparent_mode=0 min_num_slots=65534
root@suricata:/home/pevman/pfring-svn-latest# modprobe pf_ring transparent_mode=0 min_num_slots=65534

root@suricata:/home/pevman/pfring-svn-latest# cat /proc/net/pf_ring/info
PF_RING Version          : 5.6.2 ($Revision: exported$)
Total rings              : 0
Standard (non DNA) Options
Ring slots               : 65534
Slot version             : 15
Capture TX               : Yes [RX+TX]
IP Defragment            : No
Socket Mode              : Standard
Transparent mode         : Yes [mode 0]
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0


Notice the difference above  - Ring slots: 65534



Compile and install Suricata with PF_RING enabled


Get the latest Suricata dev branch:
git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ &&  git clone https://github.com/ironbee/libhtp.git -b 0.5.x

 Compile and install
./autogen.sh && ./configure --enable-pfring --enable-geoip \
--with-libpfring-includes=/usr/local/pfring/include/ \
--with-libpfring-libraries=/usr/local/pfring/lib/ \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
&& sudo make clean && sudo make && sudo make install && sudo ldconfig


PF_RING - suricata.yaml tune up and configuration

The following values and variables in the default suricata.yaml need to be changed ->

We make sure we use runmode workers (feel free to try other modes and experiment what is best for your specific set up):
#runmode: autofp
runmode: workers


Adjust the packet size:
# Preallocated size for packet. Default is 1514 which is the classical
# size for pcap on ethernet. You should adjust this value to the highest
# packet size (MTU + hardware header) on your system.
default-packet-size: 1522


Use custom profile in detect-engine with a lot more groups (high gives you about 15 groups per variable, but you can customize as needed depending on the network ranges you monitor ):
detect-engine:
  - profile: high
  - custom-values:
      toclient-src-groups: 200
      toclient-dst-groups: 200
      toclient-sp-groups: 200
      toclient-dp-groups: 300
      toserver-src-groups: 200
      toserver-dst-groups: 400
      toserver-sp-groups: 200
      toserver-dp-groups: 250
  - sgh-mpm-context: full
  - inspection-recursion-limit: 3000


Adjust your defrag settings:
# Defrag settings:
defrag:
  memcap: 512mb
  hash-size: 65536
  trackers: 65535 # number of defragmented flows to follow
  max-frags: 65535 # number of fragments to keep
  prealloc: yes
  timeout: 30



Adjust your flow settings:
flow:
  memcap: 1gb
  hash-size: 1048576
  prealloc: 1048576
  emergency-recovery: 30


Adjust your per protocol timeout values:
flow-timeouts:

  default:
    new: 3
    established: 30
    closed: 0
    emergency-new: 10
    emergency-established: 10
    emergency-closed: 0
  tcp:
    new: 6
    established: 100
    closed: 12
    emergency-new: 1
    emergency-established: 5
    emergency-closed: 2
  udp:
    new: 3
    established: 30
    emergency-new: 3
    emergency-established: 10
  icmp:
    new: 3
    established: 30
    emergency-new: 1
    emergency-established: 10



Adjust your stream engine settings:
stream:
  memcap: 12gb
  checksum-validation: no      # reject wrong csums
  prealloc-sesions: 500000     #per thread
  midstream: true
  asyn-oneside: true
  inline: no                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 20gb
    depth: 12mb                  # reassemble 12mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10


Make sure you enable suricata.log for troubleshooting if something goes wrong:
  outputs:
  - console:
      enabled: yes
  - file:
      enabled: yes
      filename: /var/log/suricata/suricata.log



The PF_RING section:
# PF_RING configuration. for use with native PF_RING support
# for more info see http://www.ntop.org/PF_RING.html
pfring:
  - interface: eth3
    # Number of receive threads (>1 will enable experimental flow pinned
    # runmode)
    threads: 16

    # Default clusterid.  PF_RING will load balance packets based on flow.
    # All threads/processes that will participate need to have the same
    # clusterid.
    cluster-id: 99

    # Default PF_RING cluster type. PF_RING can load balance per flow or per hash.
    # This is only supported in versions of PF_RING > 4.1.1.
    cluster-type: cluster_flow
    # bpf filter for this interface
    #bpf-filter: tcp
    # Choose checksum verification mode for the interface. At the moment
    # of the capture, some packets may be with an invalid checksum due to
    # offloading to the network card of the checksum computation.
    # Possible values are:
    #  - rxonly: only compute checksum for packets received by network card.
    #  - yes: checksum validation is forced
    #  - no: checksum validation is disabled
    #  - auto: suricata uses a statistical approach to detect when
    #  checksum off-loading is used. (default)
    # Warning: 'checksum-validation' must be set to yes to have any validation
    #checksum-checks: auto



We had these rules enabled:
rule-files:
 - md5.rules # 134 000 specially selected file md5s
 - dns.rules
 - malware.rules
 - local.rules
 - current_events.rules
 - mobile_malware.rules
 - user_agents.rules


Make sure you adjust your Network and Port variables:
  # Holds the address group vars that would be passed in a Signature.
  # These would be retrieved during the Signature address parsing stage.
  address-groups:

    HOME_NET: "[ HOME NET HERE ]"

    EXTERNAL_NET: "!$HOME_NET"

    HTTP_SERVERS: "$HOME_NET"

    SMTP_SERVERS: "$HOME_NET"

    SQL_SERVERS: "$HOME_NET"

    DNS_SERVERS: "$HOME_NET"

    TELNET_SERVERS: "$HOME_NET"

    AIM_SERVERS: "$EXTERNAL_NET"

    DNP3_SERVER: "$HOME_NET"

    DNP3_CLIENT: "$HOME_NET"

    MODBUS_CLIENT: "$HOME_NET"

    MODBUS_SERVER: "$HOME_NET"

    ENIP_CLIENT: "$HOME_NET"

    ENIP_SERVER: "$HOME_NET"

  # Holds the port group vars that would be passed in a Signature.
  # These would be retrieved during the Signature port parsing stage.
  port-groups:

    HTTP_PORTS: "80"

    SHELLCODE_PORTS: "!80"

    ORACLE_PORTS: 1521

    SSH_PORTS: 22

    DNP3_PORTS: 20000


Your app parsers:
# Holds details on the app-layer. The protocols section details each protocol.
# Under each protocol, the default value for detection-enabled and "
# parsed-enabled is yes, unless specified otherwise.
# Each protocol covers enabling/disabling parsers for all ipprotos
# the app-layer protocol runs on.  For example "dcerpc" refers to the tcp
# version of the protocol as well as the udp version of the protocol.
# The option "enabled" takes 3 values - "yes", "no", "detection-only".
# "yes" enables both detection and the parser, "no" disables both, and
# "detection-only" enables detection only(parser disabled).
app-layer:
  protocols:
    tls:
      enabled: yes
      detection-ports:
        tcp:
          toserver: 443

      #no-reassemble: yes
    dcerpc:
      enabled: yes
    ftp:
      enabled: yes
    ssh:
      enabled: yes
    smtp:
      enabled: yes
    imap:
      enabled: detection-only
    msn:
      enabled: detection-only
    smb:
      enabled: yes
      detection-ports:
        tcp:
          toserver: 139
    # smb2 detection is disabled internally inside the engine.
    #smb2:
    #  enabled: yes
    dnstcp:
       enabled: yes
       detection-ports:
         tcp:
           toserver: 53
    dnsudp:
       enabled: yes
       detection-ports:
         udp:
           toserver: 53
    http:
      enabled: yes


Libhtp body limits:
      libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 12mb
           response-body-limit: 12mb

           # inspection limits
           request-body-minimal-inspect-size: 32kb
           request-body-inspect-window: 4kb
           response-body-minimal-inspect-size: 32kb
           response-body-inspect-window: 4kb



Run it


With all that done and in place  - you can start Suricata like this (change your directory locations and such !)
 LD_LIBRARY_PATH=/usr/local/pfring/lib suricata --pfring-int=eth3 \
 --pfring-cluster-id=99 --pfring-cluster-type=cluster_flow \
 -c /etc/suricata/peter-yaml/suricata-pfring.yaml -D -v


this would also work:
suricata --pfring-int=eth3  --pfring-cluster-id=99 --pfring-cluster-type=cluster_flow \
 -c /etc/suricata/peter-yaml/suricata-pfring.yaml -D -v



After you start Suricata with PF_RING, you could use htop and the logs info of suricata.log to determine if everything is ok


EXAMPLE:
 [29966] 30/11/2013 -- 14:29:12 - (util-cpu.c:170) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 16
[29966] 30/11/2013 -- 14:29:12 - (app-layer-dns-udp.c:315) <Info> (DNSUDPConfigure) -- DNS request flood protection level: 500
[29966] 30/11/2013 -- 14:29:12 - (defrag-hash.c:212) <Info> (DefragInitConfig) -- allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
[29966] 30/11/2013 -- 14:29:12 - (defrag-hash.c:237) <Info> (DefragInitConfig) -- preallocated 65535 defrag trackers of size 152
[29966] 30/11/2013 -- 14:29:12 - (defrag-hash.c:244) <Info> (DefragInitConfig) -- defrag memory usage: 13631336 bytes, maximum: 536870912
[29966] 30/11/2013 -- 14:29:12 - (tmqh-flow.c:76) <Info> (TmqhFlowRegister) -- AutoFP mode using default "Active Packets" flow load balancer
[29967] 30/11/2013 -- 14:29:12 - (tmqh-packetpool.c:141) <Info> (PacketPoolInit) -- preallocated 65534 packets. Total memory 229106864
[29967] 30/11/2013 -- 14:29:12 - (host.c:205) <Info> (HostInitConfig) -- allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
[29967] 30/11/2013 -- 14:29:12 - (host.c:228) <Info> (HostInitConfig) -- preallocated 1000 hosts of size 112
[29967] 30/11/2013 -- 14:29:12 - (host.c:230) <Info> (HostInitConfig) -- host memory usage: 390144 bytes, maximum: 16777216
[29967] 30/11/2013 -- 14:29:12 - (flow.c:386) <Info> (FlowInitConfig) -- allocated 67108864 bytes of memory for the flow hash... 1048576 buckets of size 64
[29967] 30/11/2013 -- 14:29:13 - (flow.c:410) <Info> (FlowInitConfig) -- preallocated 1048576 flows of size 280
[29967] 30/11/2013 -- 14:29:13 - (flow.c:412) <Info> (FlowInitConfig) -- flow memory usage: 369098752 bytes, maximum: 1073741824
.....
[29967] 30/11/2013 -- 14:30:23 - (util-runmodes.c:545) <Info> (RunModeSetLiveCaptureWorkersForDevice) -- Going to use 16 thread(s)
[30000] 30/11/2013 -- 14:30:23 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth31) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30001] 30/11/2013 -- 14:30:23 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth32) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30002] 30/11/2013 -- 14:30:23 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth33) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30003] 30/11/2013 -- 14:30:23 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth34) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30004] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth35) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30005] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth36) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30006] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth37) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30007] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth38) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30008] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth39) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30009] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth310) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30010] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth311) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30011] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth312) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30012] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth313) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30013] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth314) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30014] 30/11/2013 -- 14:30:25 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth315) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30015] 30/11/2013 -- 14:30:25 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth316) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[29967] 30/11/2013 -- 14:30:25 - (runmode-pfring.c:555) <Info> (RunModeIdsPfringWorkers) -- RunModeIdsPfringWorkers initialised

.....
[29967] 30/11/2013 -- 14:30:25 - (tm-threads.c:2191) <Notice> (TmThreadWaitOnThreadInit) -- all 16 packet processing threads, 3 management threads initialized, engine started.




So after running for about 7 hrs:
root@suricata:/var/log/suricata# grep kernel stats.log |tail -32
capture.kernel_packets    | RxPFReth31                | 2313986783
capture.kernel_drops      | RxPFReth31                | 75254447
capture.kernel_packets    | RxPFReth32                | 2420204427
capture.kernel_drops      | RxPFReth32                | 23492323
capture.kernel_packets    | RxPFReth33                | 2412343682
capture.kernel_drops      | RxPFReth33                | 71202459
capture.kernel_packets    | RxPFReth34                | 2249712177
capture.kernel_drops      | RxPFReth34                | 15290216
capture.kernel_packets    | RxPFReth35                | 2272653367
capture.kernel_drops      | RxPFReth35                | 2072826
capture.kernel_packets    | RxPFReth36                | 2281254066
capture.kernel_drops      | RxPFReth36                | 118723669
capture.kernel_packets    | RxPFReth37                | 2430047882
capture.kernel_drops      | RxPFReth37                | 13702511
capture.kernel_packets    | RxPFReth38                | 2474713911
capture.kernel_drops      | RxPFReth38                | 6512062
capture.kernel_packets    | RxPFReth39                | 2299221265
capture.kernel_drops      | RxPFReth39                | 596690
capture.kernel_packets    | RxPFReth310               | 2398183554
capture.kernel_drops      | RxPFReth310               | 15623971
capture.kernel_packets    | RxPFReth311               | 2277348230
capture.kernel_drops      | RxPFReth311               | 62773742
capture.kernel_packets    | RxPFReth312               | 2693710052
capture.kernel_drops      | RxPFReth312               | 40213266
capture.kernel_packets    | RxPFReth313               | 2470037871
capture.kernel_drops      | RxPFReth313               | 406738
capture.kernel_packets    | RxPFReth314               | 2236636480
capture.kernel_drops      | RxPFReth314               | 714360
capture.kernel_packets    | RxPFReth315               | 2314829059
capture.kernel_drops      | RxPFReth315               | 1818726
capture.kernel_packets    | RxPFReth316               | 2271917603
capture.kernel_drops      | RxPFReth316               | 1200009

about 2% drops, 85% CPU usage , about 3300 rules and inspecting traffic for match on 134 000 file MD5s.


On a side note

You could also use linux-tools to do some more analyzing and performance tuning:
apt-get install linux-tools
Example: perf top
(hit enter)




Some more info found HERE and thanks to Regit HERE.

Your task of tuning up is not yet done. You could also do a dry test runs with profiling enabled in Suricata and determine the most "expensive rules" and tune them accordingly.

This is Chapter II (Part One - PF_RING) of a series of articles about high performance  and advance tuning of Suricata IDPS. The next article is Chapter II (Part Two - DNA)


1 comment:

  1. Thanks Peter, great tutorial! Everything is getting clear. And more importantly - everything works well :). 0 dropped for now

    ReplyDelete