Introduction
This is Chapter II (Part One - PF_RING) of a series of articles about high performance and advance tuning of Suricata IDPS
This article will consist of two parts - on setting up and configuring PF_RING and DNA for a 10Gbps interface monitoring.
PF_RING:
PF_RING™ is a new type of network socket that dramatically improves the packet capture speed...PF_RING DNA:
PF_RING™ DNA (Direct NIC Access) is a way to map NIC memory and registers to userland so that there is no additional packet copy besides the DMA transfer done by the NIC NPU (Network Process Unit), unlike what happens with NAPI. This results in better performance as CPU cycles are used uniquely for consuming packets and not for moving them off the adapter...
NOTE: PF_RING™ is open source and free, for the DNA part you need a license. However the DNA license is free for non-profit organizations or education institutions (universities,colleges etc.)
Part One - PF_RING
If you have pf_ring already installed, you might want to do:
sudo rmmod pf_ringIf you are not sure if you have pf_ring installed , you can do:
sudo modinfo pf_ring
Get the latest pf_ring sources:
git clone https://github.com/ntop/PF_RING.git
Compile and install PF_RING
Next, enter the following commands for configuration and installation:
(!!! NOT AS ROOT !!!)
cd PF_RING/kernel/
make && sudo make install
cd ../userland/lib
./configure --prefix=/usr/local/pfring && make && sudo make install
cd ../libpcap
./configure --prefix=/usr/local/pfring && make && sudo make install
cd ../tcpdump-4.1.1
./configure --prefix=/usr/local/pfring && make && sudo make install
sudo ldconfig
Then we load the module:
sudo modprobe pf_ring
Elevate as root and check if you have everything you need -enter:
modinfo pf_ring && cat /proc/net/pf_ring/info
Increase the throttle rate of the ixgbe module:
modprobe ixgbe InterruptThrottleRate=4000
The default pf_ring setup will look something like this:
root@suricata:/var/og/suricata# cat /proc/net/pf_ring/info
PF_RING Version : 5.6.2 ($Revision: exported$)
Total rings : 16
Standard (non DNA) Options
Ring slots : 4096
Slot version : 15
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Transparent mode : Yes [mode 0]
Total plugins : 0
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0
Notice the ring slots above. We would actually like to increase that in order to meet the needs of a high speed network that we are going to monitor with Suricata.
So we do:
rmmod pf_ring
modprobe pf_ring transparent_mode=0 min_num_slots=65534
root@suricata:/home/pevman/pfring-svn-latest# modprobe pf_ring transparent_mode=0 min_num_slots=65534
root@suricata:/home/pevman/pfring-svn-latest# cat /proc/net/pf_ring/info
PF_RING Version : 5.6.2 ($Revision: exported$)
Total rings : 0
Standard (non DNA) Options
Ring slots : 65534
Slot version : 15
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Transparent mode : Yes [mode 0]
Total plugins : 0
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0
Notice the difference above - Ring slots: 65534
Compile and install Suricata with PF_RING enabled
Get the latest Suricata dev branch:
git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ && git clone https://github.com/ironbee/libhtp.git -b 0.5.x
Compile and install
./autogen.sh && ./configure --enable-pfring --enable-geoip \
--with-libpfring-includes=/usr/local/pfring/include/ \
--with-libpfring-libraries=/usr/local/pfring/lib/ \
--with-libnss-libraries=/usr/lib \
--with-libnss-includes=/usr/include/nss/ \
--with-libnspr-libraries=/usr/lib \
--with-libnspr-includes=/usr/include/nspr \
&& sudo make clean && sudo make && sudo make install && sudo ldconfig
PF_RING - suricata.yaml tune up and configuration
The following values and variables in the default suricata.yaml need to be changed ->We make sure we use runmode workers (feel free to try other modes and experiment what is best for your specific set up):
#runmode: autofp
runmode: workers
Adjust the packet size:
# Preallocated size for packet. Default is 1514 which is the classical
# size for pcap on ethernet. You should adjust this value to the highest
# packet size (MTU + hardware header) on your system.
default-packet-size: 1522
Use custom profile in detect-engine with a lot more groups (high gives you about 15 groups per variable, but you can customize as needed depending on the network ranges you monitor ):
detect-engine:
- profile: high
- custom-values:
toclient-src-groups: 200
toclient-dst-groups: 200
toclient-sp-groups: 200
toclient-dp-groups: 300
toserver-src-groups: 200
toserver-dst-groups: 400
toserver-sp-groups: 200
toserver-dp-groups: 250
- sgh-mpm-context: full
- inspection-recursion-limit: 3000
Adjust your defrag settings:
# Defrag settings:
defrag:
memcap: 512mb
hash-size: 65536
trackers: 65535 # number of defragmented flows to follow
max-frags: 65535 # number of fragments to keep
prealloc: yes
timeout: 30
Adjust your flow settings:
flow:
memcap: 1gb
hash-size: 1048576
prealloc: 1048576
emergency-recovery: 30
Adjust your per protocol timeout values:
flow-timeouts:
default:
new: 3
established: 30
closed: 0
emergency-new: 10
emergency-established: 10
emergency-closed: 0
tcp:
new: 6
established: 100
closed: 12
emergency-new: 1
emergency-established: 5
emergency-closed: 2
udp:
new: 3
established: 30
emergency-new: 3
emergency-established: 10
icmp:
new: 3
established: 30
emergency-new: 1
emergency-established: 10
Adjust your stream engine settings:
stream:
memcap: 12gb
checksum-validation: no # reject wrong csums
prealloc-sesions: 500000 #per thread
midstream: true
asyn-oneside: true
inline: no # auto will use inline mode in IPS mode, yes or no set it statically
reassembly:
memcap: 20gb
depth: 12mb # reassemble 12mb into a stream
toserver-chunk-size: 2560
toclient-chunk-size: 2560
randomize-chunk-size: yes
#randomize-chunk-range: 10
Make sure you enable suricata.log for troubleshooting if something goes wrong:
outputs:
- console:
enabled: yes
- file:
enabled: yes
filename: /var/log/suricata/suricata.log
The PF_RING section:
# PF_RING configuration. for use with native PF_RING support
# for more info see http://www.ntop.org/PF_RING.html
pfring:
- interface: eth3
# Number of receive threads (>1 will enable experimental flow pinned
# runmode)
threads: 16
# Default clusterid. PF_RING will load balance packets based on flow.
# All threads/processes that will participate need to have the same
# clusterid.
cluster-id: 99
# Default PF_RING cluster type. PF_RING can load balance per flow or per hash.
# This is only supported in versions of PF_RING > 4.1.1.
cluster-type: cluster_flow
# bpf filter for this interface
#bpf-filter: tcp
# Choose checksum verification mode for the interface. At the moment
# of the capture, some packets may be with an invalid checksum due to
# offloading to the network card of the checksum computation.
# Possible values are:
# - rxonly: only compute checksum for packets received by network card.
# - yes: checksum validation is forced
# - no: checksum validation is disabled
# - auto: suricata uses a statistical approach to detect when
# checksum off-loading is used. (default)
# Warning: 'checksum-validation' must be set to yes to have any validation
#checksum-checks: auto
We had these rules enabled:
rule-files:
- md5.rules # 134 000 specially selected file md5s
- dns.rules
- malware.rules
- local.rules
- current_events.rules
- mobile_malware.rules
- user_agents.rules
Make sure you adjust your Network and Port variables:
# Holds the address group vars that would be passed in a Signature.
# These would be retrieved during the Signature address parsing stage.
address-groups:
HOME_NET: "[ HOME NET HERE ]"
EXTERNAL_NET: "!$HOME_NET"
HTTP_SERVERS: "$HOME_NET"
SMTP_SERVERS: "$HOME_NET"
SQL_SERVERS: "$HOME_NET"
DNS_SERVERS: "$HOME_NET"
TELNET_SERVERS: "$HOME_NET"
AIM_SERVERS: "$EXTERNAL_NET"
DNP3_SERVER: "$HOME_NET"
DNP3_CLIENT: "$HOME_NET"
MODBUS_CLIENT: "$HOME_NET"
MODBUS_SERVER: "$HOME_NET"
ENIP_CLIENT: "$HOME_NET"
ENIP_SERVER: "$HOME_NET"
# Holds the port group vars that would be passed in a Signature.
# These would be retrieved during the Signature port parsing stage.
port-groups:
HTTP_PORTS: "80"
SHELLCODE_PORTS: "!80"
ORACLE_PORTS: 1521
SSH_PORTS: 22
DNP3_PORTS: 20000
Your app parsers:
# Holds details on the app-layer. The protocols section details each protocol.
# Under each protocol, the default value for detection-enabled and "
# parsed-enabled is yes, unless specified otherwise.
# Each protocol covers enabling/disabling parsers for all ipprotos
# the app-layer protocol runs on. For example "dcerpc" refers to the tcp
# version of the protocol as well as the udp version of the protocol.
# The option "enabled" takes 3 values - "yes", "no", "detection-only".
# "yes" enables both detection and the parser, "no" disables both, and
# "detection-only" enables detection only(parser disabled).
app-layer:
protocols:
tls:
enabled: yes
detection-ports:
tcp:
toserver: 443
#no-reassemble: yes
dcerpc:
enabled: yes
ftp:
enabled: yes
ssh:
enabled: yes
smtp:
enabled: yes
imap:
enabled: detection-only
msn:
enabled: detection-only
smb:
enabled: yes
detection-ports:
tcp:
toserver: 139
# smb2 detection is disabled internally inside the engine.
#smb2:
# enabled: yes
dnstcp:
enabled: yes
detection-ports:
tcp:
toserver: 53
dnsudp:
enabled: yes
detection-ports:
udp:
toserver: 53
http:
enabled: yes
Libhtp body limits:
libhtp:
default-config:
personality: IDS
# Can be specified in kb, mb, gb. Just a number indicates
# it's in bytes.
request-body-limit: 12mb
response-body-limit: 12mb
# inspection limits
request-body-minimal-inspect-size: 32kb
request-body-inspect-window: 4kb
response-body-minimal-inspect-size: 32kb
response-body-inspect-window: 4kb
Run it
With all that done and in place - you can start Suricata like this (change your directory locations and such !)
LD_LIBRARY_PATH=/usr/local/pfring/lib suricata --pfring-int=eth3 \
--pfring-cluster-id=99 --pfring-cluster-type=cluster_flow \
-c /etc/suricata/peter-yaml/suricata-pfring.yaml -D -v
this would also work:
suricata --pfring-int=eth3 --pfring-cluster-id=99 --pfring-cluster-type=cluster_flow \
-c /etc/suricata/peter-yaml/suricata-pfring.yaml -D -v
After you start Suricata with PF_RING, you could use htop and the logs info of suricata.log to determine if everything is ok
EXAMPLE:
[29966] 30/11/2013 -- 14:29:12 - (util-cpu.c:170) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 16
[29966] 30/11/2013 -- 14:29:12 - (app-layer-dns-udp.c:315) <Info> (DNSUDPConfigure) -- DNS request flood protection level: 500
[29966] 30/11/2013 -- 14:29:12 - (defrag-hash.c:212) <Info> (DefragInitConfig) -- allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
[29966] 30/11/2013 -- 14:29:12 - (defrag-hash.c:237) <Info> (DefragInitConfig) -- preallocated 65535 defrag trackers of size 152
[29966] 30/11/2013 -- 14:29:12 - (defrag-hash.c:244) <Info> (DefragInitConfig) -- defrag memory usage: 13631336 bytes, maximum: 536870912
[29966] 30/11/2013 -- 14:29:12 - (tmqh-flow.c:76) <Info> (TmqhFlowRegister) -- AutoFP mode using default "Active Packets" flow load balancer
[29967] 30/11/2013 -- 14:29:12 - (tmqh-packetpool.c:141) <Info> (PacketPoolInit) -- preallocated 65534 packets. Total memory 229106864
[29967] 30/11/2013 -- 14:29:12 - (host.c:205) <Info> (HostInitConfig) -- allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
[29967] 30/11/2013 -- 14:29:12 - (host.c:228) <Info> (HostInitConfig) -- preallocated 1000 hosts of size 112
[29967] 30/11/2013 -- 14:29:12 - (host.c:230) <Info> (HostInitConfig) -- host memory usage: 390144 bytes, maximum: 16777216
[29967] 30/11/2013 -- 14:29:12 - (flow.c:386) <Info> (FlowInitConfig) -- allocated 67108864 bytes of memory for the flow hash... 1048576 buckets of size 64
[29967] 30/11/2013 -- 14:29:13 - (flow.c:410) <Info> (FlowInitConfig) -- preallocated 1048576 flows of size 280
[29967] 30/11/2013 -- 14:29:13 - (flow.c:412) <Info> (FlowInitConfig) -- flow memory usage: 369098752 bytes, maximum: 1073741824
.....
[29967] 30/11/2013 -- 14:30:23 - (util-runmodes.c:545) <Info> (RunModeSetLiveCaptureWorkersForDevice) -- Going to use 16 thread(s)
[30000] 30/11/2013 -- 14:30:23 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth31) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30001] 30/11/2013 -- 14:30:23 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth32) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30002] 30/11/2013 -- 14:30:23 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth33) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30003] 30/11/2013 -- 14:30:23 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth34) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30004] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth35) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30005] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth36) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30006] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth37) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30007] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth38) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30008] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth39) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30009] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth310) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30010] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth311) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30011] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth312) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30012] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth313) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30013] 30/11/2013 -- 14:30:24 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth314) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30014] 30/11/2013 -- 14:30:25 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth315) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[30015] 30/11/2013 -- 14:30:25 - (source-pfring.c:445) <Info> (ReceivePfringThreadInit) -- (RxPFReth316) Using PF_RING v.5.6.2, interface eth3, cluster-id 99
[29967] 30/11/2013 -- 14:30:25 - (runmode-pfring.c:555) <Info> (RunModeIdsPfringWorkers) -- RunModeIdsPfringWorkers initialised
.....
[29967] 30/11/2013 -- 14:30:25 - (tm-threads.c:2191) <Notice> (TmThreadWaitOnThreadInit) -- all 16 packet processing threads, 3 management threads initialized, engine started.
So after running for about 7 hrs:
root@suricata:/var/log/suricata# grep kernel stats.log |tail -32
capture.kernel_packets | RxPFReth31 | 2313986783
capture.kernel_drops | RxPFReth31 | 75254447
capture.kernel_packets | RxPFReth32 | 2420204427
capture.kernel_drops | RxPFReth32 | 23492323
capture.kernel_packets | RxPFReth33 | 2412343682
capture.kernel_drops | RxPFReth33 | 71202459
capture.kernel_packets | RxPFReth34 | 2249712177
capture.kernel_drops | RxPFReth34 | 15290216
capture.kernel_packets | RxPFReth35 | 2272653367
capture.kernel_drops | RxPFReth35 | 2072826
capture.kernel_packets | RxPFReth36 | 2281254066
capture.kernel_drops | RxPFReth36 | 118723669
capture.kernel_packets | RxPFReth37 | 2430047882
capture.kernel_drops | RxPFReth37 | 13702511
capture.kernel_packets | RxPFReth38 | 2474713911
capture.kernel_drops | RxPFReth38 | 6512062
capture.kernel_packets | RxPFReth39 | 2299221265
capture.kernel_drops | RxPFReth39 | 596690
capture.kernel_packets | RxPFReth310 | 2398183554
capture.kernel_drops | RxPFReth310 | 15623971
capture.kernel_packets | RxPFReth311 | 2277348230
capture.kernel_drops | RxPFReth311 | 62773742
capture.kernel_packets | RxPFReth312 | 2693710052
capture.kernel_drops | RxPFReth312 | 40213266
capture.kernel_packets | RxPFReth313 | 2470037871
capture.kernel_drops | RxPFReth313 | 406738
capture.kernel_packets | RxPFReth314 | 2236636480
capture.kernel_drops | RxPFReth314 | 714360
capture.kernel_packets | RxPFReth315 | 2314829059
capture.kernel_drops | RxPFReth315 | 1818726
capture.kernel_packets | RxPFReth316 | 2271917603
capture.kernel_drops | RxPFReth316 | 1200009
about 2% drops, 85% CPU usage , about 3300 rules and inspecting traffic for match on 134 000 file MD5s.
On a side note
You could also use linux-tools to do some more analyzing and performance tuning:apt-get install linux-toolsExample: perf top
(hit enter)
Some more info found HERE and thanks to Regit HERE.
Your task of tuning up is not yet done. You could also do a dry test runs with profiling enabled in Suricata and determine the most "expensive rules" and tune them accordingly.
This is Chapter II (Part One - PF_RING) of a series of articles about high performance and advance tuning of Suricata IDPS. The next article is Chapter II (Part Two - DNA)
Thanks Peter, great tutorial! Everything is getting clear. And more importantly - everything works well :). 0 dropped for now
ReplyDelete