Sunday, December 8, 2013

Suricata (and the grand slam of) Open Source IDPS - Chapter III - AF_PACKET

Introduction


This is Chapter III - AF_PACKET of a series of articles about high performance  and advance tuning of Suricata IDPS

This article will consist of series of instructions on setting up and configuring Suricata IDPS with  AF_PACKET for a 10Gbps traffic interface monitoring.



Chapter III - AF_PACKET

AF_PACKET works "out of the box " with Suricata. Please make sure your kernel level is at least 3.2 in order to get the best results.

Once you have followed all the steps in Chapter I - Preparation  The only thing left to do is adjust the suricata.yaml settings.


AF_PACKET - suricata.yaml tune up and configuration




NOTE:
AF_PACKET - Which kernel version not to use with Suricata in AF_PACKET mode
(thanks to Regit)


We make sure we use runmode workers (feel free to try other modes and experiment what is best for your specific set up):
#runmode: autofp
runmode: workers


Adjust the packet size:
# Preallocated size for packet. Default is 1514 which is the classical
# size for pcap on ethernet. You should adjust this value to the highest
# packet size (MTU + hardware header) on your system.
default-packet-size: 1520


Use custom profile in detect-engine with a lot more groups (high gives you about 15 groups per variable, but you can customize as needed depending on the network ranges you monitor ):
detect-engine:
  - profile: high
  - custom-values:
      toclient-src-groups: 200
      toclient-dst-groups: 200
      toclient-sp-groups: 200
      toclient-dp-groups: 300
      toserver-src-groups: 200
      toserver-dst-groups: 400
      toserver-sp-groups: 200
      toserver-dp-groups: 250
  - sgh-mpm-context: full
  - inspection-recursion-limit: 3000


Adjust your defrag settings:
# Defrag settings:
defrag:
  memcap: 512mb
  hash-size: 65536
  trackers: 65535 # number of defragmented flows to follow
  max-frags: 65535 # number of fragments to keep
  prealloc: yes
  timeout: 30



Adjust your flow settings:
flow:
  memcap: 1gb
  hash-size: 1048576
  prealloc: 1048576
  emergency-recovery: 30


Adjust your per protocol timeout values:
flow-timeouts:

  default:
    new: 3
    established: 30
    closed: 0
    emergency-new: 10
    emergency-established: 10
    emergency-closed: 0
  tcp:
    new: 6
    established: 100
    closed: 12
    emergency-new: 1
    emergency-established: 5
    emergency-closed: 2
  udp:
    new: 3
    established: 30
    emergency-new: 3
    emergency-established: 10
  icmp:
    new: 3
    established: 30
    emergency-new: 1
    emergency-established: 10



Adjust your stream engine settings:
stream:
  memcap: 16gb
  checksum-validation: no      # reject wrong csums
  prealloc-sesions: 500000     #per thread
  midstream: true
  asyn-oneside: true
  inline: no                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 20gb
    depth: 12mb                  # reassemble 12mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10


Make sure you enable suricata.log for troubleshooting if something goes wrong:
  outputs:
  - console:
      enabled: yes
  - file:
      enabled: yes
      filename: /var/log/suricata/suricata.log



The AF_PACKET section:
af-packet:
  - interface: eth3
    # Number of receive threads (>1 will enable experimental flow pinned
    # runmode)
    threads: 16
    # Default clusterid.  AF_PACKET will load balance packets based on flow.
    # All threads/processes that will participate need to have the same
    # clusterid.
    cluster-id: 98
    # Default AF_PACKET cluster type. AF_PACKET can load balance per flow or per hash.
    # This is only supported for Linux kernel > 3.1
    # possible value are:
    #  * cluster_round_robin: round robin load balancing
    #  * cluster_flow: all packets of a given flow are send to the same socket
    #  * cluster_cpu: all packets treated in kernel by a CPU are send to the same socket
    cluster-type: cluster_cpu
    # In some fragmentation case, the hash can not be computed. If "defrag" is set
    # to yes, the kernel will do the needed defragmentation before sending the packets.
    defrag: no
    # To use the ring feature of AF_PACKET, set 'use-mmap' to yes
    use-mmap: yes
    # Ring size will be computed with respect to max_pending_packets and number
    # of threads. You can set manually the ring size in number of packets by setting
    # the following value. If you are using flow cluster-type and have really network
    # intensive single-flow you could want to set the ring-size independantly of the number
    # of threads:
    ring-size: 200000
    # On busy system, this could help to set it to yes to recover from a packet drop
    # phase. This will result in some packets (at max a ring flush) being non treated.
    #use-emergency-flush: yes
    # recv buffer size, increase value could improve performance
    # buffer-size: 100000
    # Set to yes to disable promiscuous mode
    # disable-promisc: no
    # Choose checksum verification mode for the interface. At the moment
    # of the capture, some packets may be with an invalid checksum due to
    # offloading to the network card of the checksum computation.
    # Possible values are:
    #  - kernel: use indication sent by kernel for each packet (default)
    #  - yes: checksum validation is forced
    #  - no: checksum validation is disabled
    #  - auto: suricata uses a statistical approach to detect when
    #  checksum off-loading is used.
    # Warning: 'checksum-validation' must be set to yes to have any validation
    checksum-checks: kernel
    # BPF filter to apply to this interface. The pcap filter syntax apply here.
    #bpf-filter: port 80 or udp
   



We had these rules enabled:
rule-files:
   - trojan.rules
   - md5.rules # 134 000 specially selected file md5s
   - dns.rules
   - malware.rules
   - local.rules
   - current_events.rules
   -  mobile_malware.rules
   - user_agents.rules 



Make sure you adjust your Network and Port variables:
  # Holds the address group vars that would be passed in a Signature.
  # These would be retrieved during the Signature address parsing stage.
  address-groups:

    HOME_NET: "[ HOME NET HERE ]"

    EXTERNAL_NET: "!$HOME_NET"

    HTTP_SERVERS: "$HOME_NET"

    SMTP_SERVERS: "$HOME_NET"

    SQL_SERVERS: "$HOME_NET"

    DNS_SERVERS: "$HOME_NET"

    TELNET_SERVERS: "$HOME_NET"

    AIM_SERVERS: "$EXTERNAL_NET"

    DNP3_SERVER: "$HOME_NET"

    DNP3_CLIENT: "$HOME_NET"

    MODBUS_CLIENT: "$HOME_NET"

    MODBUS_SERVER: "$HOME_NET"

    ENIP_CLIENT: "$HOME_NET"

    ENIP_SERVER: "$HOME_NET"

  # Holds the port group vars that would be passed in a Signature.
  # These would be retrieved during the Signature port parsing stage.
  port-groups:

    HTTP_PORTS: "80"

    SHELLCODE_PORTS: "!80"

    ORACLE_PORTS: 1521

    SSH_PORTS: 22

    DNP3_PORTS: 20000


Your app parsers:
# Holds details on the app-layer. The protocols section details each protocol.
# Under each protocol, the default value for detection-enabled and "
# parsed-enabled is yes, unless specified otherwise.
# Each protocol covers enabling/disabling parsers for all ipprotos
# the app-layer protocol runs on.  For example "dcerpc" refers to the tcp
# version of the protocol as well as the udp version of the protocol.
# The option "enabled" takes 3 values - "yes", "no", "detection-only".
# "yes" enables both detection and the parser, "no" disables both, and
# "detection-only" enables detection only(parser disabled).
app-layer:
  protocols:
    tls:
      enabled: yes
      detection-ports:
        tcp:
          toserver: 443

      #no-reassemble: yes
    dcerpc:
      enabled: yes
    ftp:
      enabled: yes
    ssh:
      enabled: yes
    smtp:
      enabled: yes
    imap:
      enabled: detection-only
    msn:
      enabled: detection-only
    smb:
      enabled: yes
      detection-ports:
        tcp:
          toserver: 139
    # smb2 detection is disabled internally inside the engine.
    #smb2:
    #  enabled: yes
    dnstcp:
       enabled: yes
       detection-ports:
         tcp:
           toserver: 53
    dnsudp:
       enabled: yes
       detection-ports:
         udp:
           toserver: 53
    http:
      enabled: yes


Libhtp body limits:
      libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 12mb
           response-body-limit: 12mb

           # inspection limits
           request-body-minimal-inspect-size: 32kb
           request-body-inspect-window: 4kb
           response-body-minimal-inspect-size: 32kb
           response-body-inspect-window: 4kb



Run it

 /usr/local/bin/suricata -c /etc/suricata/suricata.yaml --af-packet=eth3 -D -v



Results


We take a look at the suricata.log file:
[13915] 4/12/2013 -- 15:38:15 - (suricata.c:962) <Notice> (SCPrintVersion) -- This is Suricata version 2.0dev (rev e7f6107)
[13915] 4/12/2013 -- 15:38:15 - (util-cpu.c:170) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 16
[13915] 4/12/2013 -- 15:38:15 - (app-layer-dns-udp.c:315) <Info> (DNSUDPConfigure) -- DNS request flood protection level: 500
[13915] 4/12/2013 -- 15:38:15 - (util-ioctl.c:99) <Info> (GetIfaceMTU) -- Found an MTU of 1500 for 'eth3'
[13915] 4/12/2013 -- 15:38:15 - (defrag-hash.c:212) <Info> (DefragInitConfig) -- allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
[13915] 4/12/2013 -- 15:38:15 - (defrag-hash.c:237) <Info> (DefragInitConfig) -- preallocated 65535 defrag trackers of size 152
[13915] 4/12/2013 -- 15:38:15 - (defrag-hash.c:244) <Info> (DefragInitConfig) -- defrag memory usage: 13631336 bytes, maximum: 536870912
[13915] 4/12/2013 -- 15:38:15 - (tmqh-flow.c:76) <Info> (TmqhFlowRegister) -- AutoFP mode using default "Active Packets" flow load balancer
[13916] 4/12/2013 -- 15:38:15 - (tmqh-packetpool.c:142) <Info> (PacketPoolInit) -- preallocated 2048 packets. Total memory 7151616
[13916] 4/12/2013 -- 15:38:15 - (host.c:205) <Info> (HostInitConfig) -- allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
[13916] 4/12/2013 -- 15:38:15 - (host.c:228) <Info> (HostInitConfig) -- preallocated 1000 hosts of size 112
[13916] 4/12/2013 -- 15:38:15 - (host.c:230) <Info> (HostInitConfig) -- host memory usage: 390144 bytes, maximum: 16777216
[13916] 4/12/2013 -- 15:38:15 - (flow.c:386) <Info> (FlowInitConfig) -- allocated 67108864 bytes of memory for the flow hash... 1048576 buckets of size 64
[13916] 4/12/2013 -- 15:38:15 - (flow.c:410) <Info> (FlowInitConfig) -- preallocated 1048576 flows of size 280
[13916] 4/12/2013 -- 15:38:15 - (flow.c:412) <Info> (FlowInitConfig) -- flow memory usage: 369098752 bytes, maximum: 1073741824
[13916] 4/12/2013 -- 15:38:15 - (reputation.c:459) <Info> (SRepInit) -- IP reputation disabled
[13916] 4/12/2013 -- 15:38:15 - (util-magic.c:62) <Info> (MagicInit) -- using magic-file /usr/share/file/magic
[13916] 4/12/2013 -- 15:38:15 - (suricata.c:1769) <Info> (SetupDelayedDetect) -- Delayed detect disabled
[13916] 4/12/2013 -- 15:38:17 - (detect-filemd5.c:275) <Info> (DetectFileMd5Parse) -- MD5 hash size 2143616 bytes


...8 rule files, 7947 rules loaded
[13916] 4/12/2013 -- 15:38:17 - (detect.c:453) <Info> (SigLoadSignatures) -- 8 rule files processed. 7947 rules successfully loaded, 0 rules failed
[13916] 4/12/2013 -- 15:38:17 - (detect.c:2568) <Info> (SigAddressPrepareStage1) -- 7947 signatures processed. 1 are IP-only rules, 1976 are inspecting packet payload, 6714 inspect application laye
r, 0 are decoder event only
[13916] 4/12/2013 -- 15:38:17 - (detect.c:2571) <Info> (SigAddressPrepareStage1) -- building signature grouping structure, stage 1: preprocessing rules... complete
[13916] 4/12/2013 -- 15:38:17 - (detect.c:3194) <Info> (SigAddressPrepareStage2) -- building signature grouping structure, stage 2: building source address list... complete
[13916] 4/12/2013 -- 15:39:51 - (detect.c:3836) <Info> (SigAddressPrepareStage3) -- building signature grouping structure, stage 3: building destination address lists... complete
[13916] 4/12/2013 -- 15:39:51 - (util-threshold-config.c:1186) <Info> (SCThresholdConfParseFile) -- Threshold config parsed: 0 rule(s) found
[13916] 4/12/2013 -- 15:39:51 - (util-coredump-config.c:122) <Info> (CoredumpLoadConfig) -- Core dump size set to unlimited.
[13916] 4/12/2013 -- 15:39:51 - (util-logopenfile.c:168) <Info> (SCConfLogOpenGeneric) -- fast output device (regular) initialized: fast.log
[13916] 4/12/2013 -- 15:39:51 - (util-logopenfile.c:168) <Info> (SCConfLogOpenGeneric) -- http-log output device (regular) initialized: http.log
[13916] 4/12/2013 -- 15:39:51 - (util-logopenfile.c:168) <Info> (SCConfLogOpenGeneric) -- tls-log output device (regular) initialized: tls.log
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:217) <Info> (AffinitySetupLoadFromConfig) -- Found affinity definition for "management-cpu-set"
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:265) <Info> (AffinitySetupLoadFromConfig) -- Using default prio 'low'
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:217) <Info> (AffinitySetupLoadFromConfig) -- Found affinity definition for "receive-cpu-set"
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:217) <Info> (AffinitySetupLoadFromConfig) -- Found affinity definition for "decode-cpu-set"
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:217) <Info> (AffinitySetupLoadFromConfig) -- Found affinity definition for "stream-cpu-set"
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:217) <Info> (AffinitySetupLoadFromConfig) -- Found affinity definition for "detect-cpu-set"
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:265) <Info> (AffinitySetupLoadFromConfig) -- Using default prio 'high'
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:217) <Info> (AffinitySetupLoadFromConfig) -- Found affinity definition for "verdict-cpu-set"
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:265) <Info> (AffinitySetupLoadFromConfig) -- Using default prio 'high'
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:217) <Info> (AffinitySetupLoadFromConfig) -- Found affinity definition for "reject-cpu-set"
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:265) <Info> (AffinitySetupLoadFromConfig) -- Using default prio 'low'
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:217) <Info> (AffinitySetupLoadFromConfig) -- Found affinity definition for "output-cpu-set"
[13916] 4/12/2013 -- 15:39:51 - (util-affinity.c:265) <Info> (AffinitySetupLoadFromConfig) -- Using default prio 'medium'
[13916] 4/12/2013 -- 15:39:51 - (runmode-af-packet.c:200) <Info> (ParseAFPConfig) -- Enabling mmaped capture on iface eth3
[13916] 4/12/2013 -- 15:39:51 - (runmode-af-packet.c:268) <Info> (ParseAFPConfig) -- Using cpu cluster mode for AF_PACKET (iface eth3)
[13916] 4/12/2013 -- 15:39:51 - (util-runmodes.c:545) <Info>


...going to use 16 threads:
(RunModeSetLiveCaptureWorkersForDevice) -- Going to use 16 thread(s)
[13918] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 0
[13918] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth31" Module to cpu/core 0, thread id 13918
[13918] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13918] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13919] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 1
[13919] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth32" Module to cpu/core 1, thread id 13919
[13919] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13919] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13920] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 2
[13920] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth33" Module to cpu/core 2, thread id 13920
[13920] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13920] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13921] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 3
[13921] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth34" Module to cpu/core 3, thread id 13921
[13921] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13921] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13922] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 4
[13922] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth35" Module to cpu/core 4, thread id 13922
[13922] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13922] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13923] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 5
[13923] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth36" Module to cpu/core 5, thread id 13923
[13923] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13923] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13924] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 6
[13924] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth37" Module to cpu/core 6, thread id 13924
[13924] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13924] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13925] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 7
[13925] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth38" Module to cpu/core 7, thread id 13925
[13925] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13925] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13926] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 8
[13926] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth39" Module to cpu/core 8, thread id 13926
[13926] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13926] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13927] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 9
[13927] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth310" Module to cpu/core 9, thread id 13927
[13927] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13927] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13928] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 10
[13928] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth311" Module to cpu/core 10, thread id 13928
[13928] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13928] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13929] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 11
[13929] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth312" Module to cpu/core 11, thread id 13929
[13929] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13929] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13930] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 12
[13930] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth313" Module to cpu/core 12, thread id 13930
[13930] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13930] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13931] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 13
[13931] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth314" Module to cpu/core 13, thread id 13931
[13931] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13931] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13932] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 14
[13932] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth315" Module to cpu/core 14, thread id 13932
[13932] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13932] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call
[13933] 4/12/2013 -- 15:39:51 - (util-affinity.c:319) <Info> (AffinityGetNextCPU) -- Setting affinity on CPU 15
[13933] 4/12/2013 -- 15:39:51 - (tm-threads.c:1332) <Info> (TmThreadSetupOptions) -- Setting prio -2 for "AFPacketeth316" Module to cpu/core 15, thread id 13933
[13933] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1554) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode
[13933] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1564) <Info> (ReceiveAFPThreadInit) -- Enabling zero copy mode by using data release call


...reading in some  memory settings from yaml:
[13916] 4/12/2013 -- 15:39:51 - (runmode-af-packet.c:529) <Info> (RunModeIdsAFPWorkers) -- RunModeIdsAFPWorkers initialised
[13934] 4/12/2013 -- 15:39:51 - (tm-threads.c:1338) <Info> (TmThreadSetupOptions) -- Setting prio 2 for "FlowManagerThread" thread , thread id 13934
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:376) <Info> (StreamTcpInitConfig) -- stream "prealloc-sessions": 375000 (per thread)
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:392) <Info> (StreamTcpInitConfig) -- stream "memcap": 17179869184
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:398) <Info> (StreamTcpInitConfig) -- stream "midstream" session pickups: enabled
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:404) <Info> (StreamTcpInitConfig) -- stream "async-oneside": disabled
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:421) <Info> (StreamTcpInitConfig) -- stream "checksum-validation": disabled
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:443) <Info> (StreamTcpInitConfig) -- stream."inline": disabled
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:456) <Info> (StreamTcpInitConfig) -- stream "max-synack-queued": 5
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:474) <Info> (StreamTcpInitConfig) -- stream.reassembly "memcap": 21474836480
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:492) <Info> (StreamTcpInitConfig) -- stream.reassembly "depth": 12582912
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:575) <Info> (StreamTcpInitConfig) -- stream.reassembly "toserver-chunk-size": 2671
[13916] 4/12/2013 -- 15:39:51 - (stream-tcp.c:577) <Info> (StreamTcpInitConfig) -- stream.reassembly "toclient-chunk-size": 2582
[13935] 4/12/2013 -- 15:39:51 - (tm-threads.c:1338) <Info> (TmThreadSetupOptions) -- Setting prio 2 for "SCPerfWakeupThread" thread , thread id 13935
[13936] 4/12/2013 -- 15:39:51 - (tm-threads.c:1338) <Info> (TmThreadSetupOptions) -- Setting prio 2 for "SCPerfMgmtThread" thread , thread id 13936
[13916] 4/12/2013 -- 15:39:51 - (tm-threads.c:2191) <Notice> (TmThreadWaitOnThreadInit) -- all 16 packet processing threads, 3 management threads initialized, engine started.


....have  a look - Suricata detects if OFFloading (discussed in  Chapter I - Preparation) is used on the network interface:
[13918] 4/12/2013 -- 15:39:51 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13918] 4/12/2013 -- 15:39:51 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13918] 4/12/2013 -- 15:39:51 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13918] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 8
[13918] 4/12/2013 -- 15:39:52 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth31 using socket 8
[13919] 4/12/2013 -- 15:39:52 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13919] 4/12/2013 -- 15:39:52 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13919] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13919] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 9
[13919] 4/12/2013 -- 15:39:52 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth32 using socket 9
[13920] 4/12/2013 -- 15:39:52 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13920] 4/12/2013 -- 15:39:52 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13920] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13920] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 10
[13920] 4/12/2013 -- 15:39:52 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth33 using socket 10
[13921] 4/12/2013 -- 15:39:52 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13921] 4/12/2013 -- 15:39:52 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13921] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13921] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 11
[13921] 4/12/2013 -- 15:39:52 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth34 using socket 11
[13922] 4/12/2013 -- 15:39:52 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13922] 4/12/2013 -- 15:39:52 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13922] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13922] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 12
[13922] 4/12/2013 -- 15:39:52 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth35 using socket 12
[13923] 4/12/2013 -- 15:39:52 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13923] 4/12/2013 -- 15:39:52 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13923] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13923] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 13
[13923] 4/12/2013 -- 15:39:52 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth36 using socket 13
[13924] 4/12/2013 -- 15:39:52 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13924] 4/12/2013 -- 15:39:52 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13924] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13924] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 14
[13924] 4/12/2013 -- 15:39:52 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth37 using socket 14
[13925] 4/12/2013 -- 15:39:52 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13925] 4/12/2013 -- 15:39:52 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13925] 4/12/2013 -- 15:39:52 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13925] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 15
[13925] 4/12/2013 -- 15:39:53 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth38 using socket 15
[13926] 4/12/2013 -- 15:39:53 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13926] 4/12/2013 -- 15:39:53 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13926] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13926] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 16
[13926] 4/12/2013 -- 15:39:53 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth39 using socket 16
[13927] 4/12/2013 -- 15:39:53 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13927] 4/12/2013 -- 15:39:53 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13927] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13927] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 17
[13927] 4/12/2013 -- 15:39:53 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth310 using socket 17
[13928] 4/12/2013 -- 15:39:53 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13928] 4/12/2013 -- 15:39:53 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13928] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13928] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 18
[13928] 4/12/2013 -- 15:39:53 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth311 using socket 18
[13929] 4/12/2013 -- 15:39:53 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13929] 4/12/2013 -- 15:39:53 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13929] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13929] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 19
[13929] 4/12/2013 -- 15:39:53 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth312 using socket 19
[13930] 4/12/2013 -- 15:39:53 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13930] 4/12/2013 -- 15:39:53 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13930] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13930] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 20
[13930] 4/12/2013 -- 15:39:53 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth313 using socket 20
[13931] 4/12/2013 -- 15:39:53 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13931] 4/12/2013 -- 15:39:53 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13931] 4/12/2013 -- 15:39:53 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13931] 4/12/2013 -- 15:39:54 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 21
[13931] 4/12/2013 -- 15:39:54 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth314 using socket 21
[13932] 4/12/2013 -- 15:39:54 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13932] 4/12/2013 -- 15:39:54 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13932] 4/12/2013 -- 15:39:54 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13932] 4/12/2013 -- 15:39:54 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 22
[13932] 4/12/2013 -- 15:39:54 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth315 using socket 22
[13933] 4/12/2013 -- 15:39:54 - (util-ioctl.c:175) <Info> (GetIfaceOffloading) -- Generic Receive Offload is unset on eth3
[13933] 4/12/2013 -- 15:39:54 - (util-ioctl.c:194) <Info> (GetIfaceOffloading) -- Large Receive Offload is unset on eth3
[13933] 4/12/2013 -- 15:39:54 - (source-af-packet.c:1189) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=10001 frame_size=1584 frame_nr=200020
[13933] 4/12/2013 -- 15:39:54 - (source-af-packet.c:1380) <Info> (AFPCreateSocket) -- Using interface 'eth3' via socket 23
[13933] 4/12/2013 -- 15:39:54 - (source-af-packet.c:439) <Info> (AFPPeersListReachedInc) -- All AFP capture threads are running.
[13933] 4/12/2013 -- 15:39:54 - (source-af-packet.c:988) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth316 using socket 23



htop - Now that we have been up  and running for a while (6-7 hrs) on a 10Gbps link ( 9.3 Gbps traffic - to be precise - at the moment of these statistics):




we have about 1-2% drops in total (on 7947 rules):





and then after 13 hrs:





we still have 1-2% drops
(1.897% to be precise - total kernel drops 1 337 487 757 out of total packets 70 491 114 835 is 1.897%) :




And that is just half the job done on Suricata's high performance tuning. Before you arrive at this point  there is much more work to be done - pre-study, HW choice, rule selection and tuning, traffic analysis , office/organization needs analysis, network location design and deployment, testing/PoCs and  more...

Next - Chapter IV - Logstash / Kibana / Elasticsearch





No comments:

Post a Comment