Saturday, May 31, 2014

Logs per second on eve.json - the good and the bad news on a 10Gbps IDPS line inspection

I found this one liner which gives the amount of logs per second logged in eve.json
tail -f  eve.json |perl -e 'while (<>) {$l++;if (time > $e) {$e=time;print "$l\n";$l=0}}'
I take no credit for it  - I got it from commandlinefu

tail -f  eve.json |perl -e 'while (<>) {$l++;if (time > $e) {$e=time;print "$l\n";$l=0}}'

Having in mind this is at Sat lunch time... 3 - 3,5K logs per second it turns to minimum 4 - 4,5K logs per second on a working day.
I had "only"  these logs enabled in suricata.yaml in the eve log section - dns,http,alert and ssh on a 10Gbps Suricata 2.0.1 IDS sensor:

  # "United" event log in JSON format
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
        - alert
        - http:
            extended: yes     # enable this for extended logging information
        - dns
        #- tls:
            #extended: yes     # enable this for extended logging information
        #- files:
            #force-magic: yes   # force logging magic on all logged files
            #force-md5: yes     # force logging of md5 checksums
        #- drop
        - ssh
      append: yes

If you enable "files" and "tls" it will increase it to probably about 5-6K logs per second (maybe even more , depends on the type of traffic) with that set up.

The good news:

eve.json logs in standard JSON format (JavaScript Object Notation). So there are A LOT of log analysis solutions and software both open source, free and/or commercial that can digest and run analysis on JSON logs.

The bad news:

How many log analysis solutions can "really" handle 5K logs per second -
  • indexing, 
  • query, 
  • search, 
  • report generation, 
  • log correlation, 
  • filter searches by key fields,
  • nice graphs - "eye candy" for the management and/or customer , 
all that while being fast?
(and like that on at least 20 days of data from a 10Gbps IDPS Suricata sensor)

...aka 18 mil per hour ...or 432 mil log records per day

Another perspective -> 54-70GB of logs a day...


Deploying and tuning Suricata IDS/IPS is  the first important step. Then you need to  handle all the data that comes out of the sensor.
You should very carefully consider your goals, requirements and design and do Proof of Concept and test runs before you end up in a production situation in which you can't handle what you asked for :)


  1. At least for alerts, tuning should get the number down. It's always nice to get the alert count down to something that is human manageable.

    But DNS and HTTP logging (and flow) are going to be a whole new issue that I'm not sure existing tools will be able to handle all that well.

    1. Yes I agree here. All the available outputs enabled in Suricata present a new level of challenge for the analysis management and implementation on high speed traffic inspection.