Saturday, March 15, 2014

Suricata (and the grand slam of) Open Source IDPS - Chapter IV - Logstash / Kibana / Elasticsearch, Part One


Introduction 


This article covers old installation instructions for Logstash 1.3.3 and prior. There is an UPDATED article - http://pevma.blogspot.se/2014/03/suricata-and-grand-slam-of-open-source_26.html that covers the new (at the time of this writing) 1.4.0 Logstash release.


This is Chapter IV of a series of 4 articles aiming at giving a general guideline on how to deploy the Open Source Suricata IDPS on a high speed networks (10Gbps) in IDS mode using AF_PACKET, PF_RING or DNA and Logstash / Kibana / Elasticsearch

This chapter consist of two parts:
Chapter IV Part One - installation and set up of logstash.
Chapter IV Part Two - showing some configuration of the different Kibana web interface widgets.

The end result should be as many and as different widgets to analyze the Suricata IDPS logs , something like :






This chapter describes a quick and easy set up of Logstash / Kibana / Elasticsearch
This set up described in this chapter was not intended for a huge deployment, but rather a conceptual proof in a working environment as pictured below:






We have two Suricata IDS deployed - IDS1 and IDS2
  • IDS2 uses logstash-forwarder (former lumberjack) to securely forward (SSL encrypted) its eve.json logs (configured in suricata.yaml) to IDS1, main Logstash/Kibana deployment.
  • IDS1 has its own logging (eve.json as well) that is also digested by Logstash.

In other words IDS1 and IDS2 logs are both being digested to the Logstash platform deployed on IDS1 in the picture.

Prerequisites

Both IDS1 and IDS2 should be set up and tuned with Suricata IDPS. This article will not cover that. If you have not done it you could start HERE.

Make sure you have installed Suricata with JSON availability. The following two packages must be present on your system prior to installation/compilation:
root@LTS-64-1:~# apt-cache search libjansson
libjansson-dev - C library for encoding, decoding and manipulating JSON data (dev)
libjansson4 - C library for encoding, decoding and manipulating JSON data
If there are not present on the system  - install them:
apt-get install libjansson4 libjansson-dev

In both IDS1 and IDS2 you should have in your suricata.yaml:
  # "United" event log in JSON format
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert
        - http:
            extended: yes     # enable this for extended logging information
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: yes   # force logging magic on all logged files
            force-md5: yes     # force logging of md5 checksums
        #- drop
        - ssh
This tutorial uses /var/log/suricata as a default logging directory.

You can do a few dry runs to confirm log generation on both systems.
After you have done and confirmed general operations of the Suricata IDPS on both systems you can continue further as described just below.

Installation

IDS2

For the logstash-forwarder we need Go installed.

cd /opt
apt-get install hg-fast-export
hg clone -u release https://code.google.com/p/go
cd go/src
./all.bash


If everything goes ok you should see at the end:
ALL TESTS PASSED


Update your $PATH variable, in  make sure it has:
PATH=$PATH:/opt/go/bin
export PATH

root@debian64:~# nano  ~/.bashrc


edit the file (.bashrc), add at the bottom:

PATH=$PATH:/opt/go/bin
export PATH

 then:

root@debian64:~# source ~/.bashrc
root@debian64:~# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/go/bin


Install logstash-forwarder:
cd /opt
git clone git://github.com/elasticsearch/logstash-forwarder.git
cd logstash-forwarder
go build

Build a debian package:
apt-get install ruby ruby-dev
gem install fpm
make deb
That will produce a Debian package in the same directory (something like):
logstash-forwarder_0.3.1_amd64.deb

Install the Debian package:
root@debian64:/opt# dpkg -i logstash-forwarder_0.3.1_amd64.deb

 NOTE: You can use the same Debian package to copy and install it (dependency free) on other machines/servers. So once you have the deb package you can install it on any other server the same way, no need for rebuilding everything again (Go and ruby)

Create SSL certificates that will be used to securely encrypt and transport the logs:
cd /opt
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt

Copy to BOTH IDS1 and IDS2:
logstash-forwarder.key in /etc/ssl/private/
logstash-forwarder.crt in /etc/ssl/certs/

Now you can try to start/restart/stop the logstash-forwarder service:
root@debian64:/opt# /etc/init.d/logstash-forwarder start
root@debian64:/opt# /etc/init.d/logstash-forwarder status
[ ok ] logstash-forwarder is running.
root@debian64:/opt# /etc/init.d/logstash-forwarder stop
root@debian64:/opt# /etc/init.d/logstash-forwarder status
[FAIL] logstash-forwarder is not running ... failed!
root@debian64:/opt# /etc/init.d/logstash-forwarder start
root@debian64:/opt# /etc/init.d/logstash-forwarder status
[ ok ] logstash-forwarder is running.
root@debian64:/opt# /etc/init.d/logstash-forwarder stop
root@debian64:/opt#
Good to go.

Create on IDS2 your logstash-forwarder config:
touch /etc/logstash-forwarder
Make sure the file looks like this (in this tutorial - copy/paste):

{
  "network": {
    "servers": [ "192.168.1.158:5043" ],
    "ssl certificate": "/etc/ssl/certs/logstash-forwarder.crt",
    "ssl key": "/etc/ssl/private/logstash-forwarder.key",
    "ssl ca": "/etc/ssl/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "/var/log/suricata/eve.json" ],
      "codec": { "type": "json" }
    }
  ]
}

This is as far as the set up on IDS2 goes....

IDS1 - indexer

Download Logstash (change or create directory names to whichever suits you best):
cd /root/Work/tmp/Logstash
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.3.3-flatjar.jar

Download the GoeIP lite data needed for our geoip location:
wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

Create your logstash conf file:
touch /etc/init.d/logstash.conf

Make sure it looks like this (change directory names accordingly):
input {
  file {
    path => "/var/log/suricata/eve.json"
    codec =>   json
    # This format tells logstash to expect 'logstash' json events from the file.
    #format => json_event
  }
 
  lumberjack {
  port => 5043
  type => "logs"
  codec =>   json
  ssl_certificate => "/etc/ssl/certs/logstash-forwarder.crt"
  ssl_key => "/etc/ssl/private/logstash-forwarder.key"
  }
}


output {
  stdout { codec => rubydebug }
  elasticsearch { embedded => true }
}

#geoip part
filter {
  if [src_ip] {
    geoip {
      source => "src_ip"
      target => "geoip"
      database => "/root/Work/tmp/Logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
  }
}


Create a startup script:
touch /etc/init.d/logstash-startup.conf

Make sure it looks like this (change directories accordingly):
# logstash - indexer instance
#

description     "logstash indexer instance using ports 9292 9200 9300 9301"

start on runlevel [345]
stop on runlevel [!345]

#respawn
#respawn limit 5 30
#limit nofile 65550 65550
expect fork

script
  test -d /var/log/logstash || mkdir /var/log/logstash
  chdir /root/Work/Logstash/
  exec sudo java -jar /root/Work/tmp/Logstash/logstash-1.3.3-flatjar.jar agent -f /etc/init/logstash.conf --log /var/log/logstash/logstash-indexer.out -- web &
end script


Then:
initctl reload-configuration   

 Rolling it out


On IDS1 and IDS2 - start Suricata IDPS. Genereate some logs
On IDS1:
service logstash-startup start

On IDS2:
root@debian64:/opt# /etc/init.d/logstash-forwarder start
You can check if it is working properly like so - > tail -f /var/log/syslog :



Go to your browser and navigate to (in this case IDS1)
http://192.168.1.158:9292
NOTE: This is http (as this is just a simple tutorial), you should configure it to use httpS

The Kibana web interface should come up.

That is it. From here on it is up to you to configure the web interface with your own widgets.

Chapter IV Part Two will follow with detail on that subject.
However something like this is easily achievable with a few clicks in under 5 min:





Troubleshooting:

You should keep an eye on /var/log/logstash/logstash-indexer.out - any troubles should be visible there.

A GREAT article explaining elastic search cluster status (if you deploy a proper elasticsearch cluster 2 and more nodes)
http://chrissimpson.co.uk/elasticsearch-yellow-cluster-status-explained.html

ERR in logstash-indexer.out - too many open files
http://www.elasticsearch.org/tutorials/too-many-open-files/

Set ulimit parameters on Ubuntu(this is in case you need to increase the number of Inodes(files) available on a system "df -ih"):
http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/

This is an advanced topic - Cluster status and settings commands:
 curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

 curl -XGET 'http://localhost:9200/_status?pretty=true'

 curl -XGET 'http://localhost:9200/_nodes?os=true&process=true&pretty=true'



Very useful links:

A MUST READ (explaining the usage of ".raw" in terms so that the terms  re not broken by space delimiter)
http://www.elasticsearch.org/blog/logstash-1-3-1-released/

Article explaining how to set up a 2 node cluster:
http://techhari.blogspot.se/2013/03/elasticsearch-cluster-setup-in-2-minutes.html

Installing Logstash Central Server (using rsyslog):
https://support.shotgunsoftware.com/entries/23163863-Installing-Logstash-Central-Server

ElasticSearch cluster setup in 2 minutes:
http://techhari.blogspot.com/2013/03/elasticsearch-cluster-setup-in-2-minutes.html






No comments:

Post a Comment