Logstash for IPFIX

Native IPFIX (Netflow V10) for Logstash is still in development but in the meantime i've been able to get it working thus:

Nprobe

I know, nprobe is commercial but i had it lying around so chose to use it. It can output in a variety of ways but in the base license model only zeromq and tcp methods allow direct output to logstash. My experience with zeromq was initially good but after a reinstall i couldn't get it working again, something wrong with the encoding or compression or something meant that logstash was seeing the data as garbage.

Switching to tcp output in json format got it working in the end so this is the nprobe config file, you'll note that local interface sniffing is disabled to it only listens for flows on port 2100:

# cat /etc/nprobe/nprobe-eth0.conf
-i none 
--collector-port 2100 
#--json-labels
#--zmq tcp://*:5000 
--tcp localhost:5000
-V10
-g=/var/run/nprobe-eth0.pid

And the nprobe init files:

touch /etc/nprobe/nprobe-eth0.start

Logstash

The logstash config is easy unless you want to filter the flows as they arrive:

# cat /etc/logstash/conf.d/tcp.conf 
input {
	tcp {
		port => 5000
		type => "netflow"
		codec => "json"
	}
}

I chose to do some filtering to assign nice names and tags to my SNMP interface ids

filter {
	if [INPUT_SNMP] == 863 { 
		mutate { 
			add_field =>  { 
				"INPUT_DESC" => "International IP Transit - Vocus Auckland" 
			} 
			add_tag => [ "International IP Transit - Vocus Auckland", "International"]
		} 
	}
        if [OUTPUT_SNMP] == 863 {
                mutate {
                        add_field =>  {
                                "OUTPUT_DESC" => "International IP Transit - Vocus Auckland"
                        }
                        add_tag => [ "International IP Transit - Vocus Auckland", "International"]
                }
        }
        if [INPUT_SNMP] == 588 {
                mutate {
                        add_field =>  {
                                "INPUT_DESC" => "Peering - APE"
                        }
                        add_tag => [ "Peering - APE", "Domestic"]
                }
        }
        if [OUTPUT_SNMP] == 588 {
                mutate {
                        add_field =>  {
                                "OUTPUT_DESC" => "Peering - APE"
                        }
                        add_tag => [ "Peering - APE", "Domestic"]
                }
        }
        if [INPUT_SNMP] == 1248 {
                mutate {
                        add_field =>  {
                                "INPUT_DESC" => "Peering - MegaIX Auckland"
                        }
                        add_tag => [ "Peering - MegaIX Auckland", "Domestic"]
                }
        }
        if [OUTPUT_SNMP] == 1248 {
                mutate {
                        add_field =>  {
                                "OUTPUT_DESC" => "Peering - MegaIX Auckland"
                        }
                        add_tag => [ "Peering - MegaIX Auckland", "Domestic"]
                }
        }
        if [INPUT_SNMP] == 609 {
                mutate {
                        add_field =>  {
                                "INPUT_DESC" => "Peering - WIX"
                        }
                        add_tag => [ "Peering - WIX", "Domestic"]
                }
        }
        if [OUTPUT_SNMP] == 609 {
                mutate {
                        add_field =>  {
                                "OUTPUT_DESC" => "Peering - WIX"
                        }
                        add_tag => [ "Peering - WIX", "Domestic"]
                }
        }
        if [INPUT_SNMP] == 572 {
                mutate {
                        add_field =>  {
                                "INPUT_DESC" => "International IP Transit - Vocus Sydney"
                        }
                        add_tag => [ "International IP Transit - Vocus Sydney", "International"]
                }
        }
        if [OUTPUT_SNMP] == 572 {
                mutate {
                        add_field =>  {
                                "OUTPUT_DESC" => "International IP Transit - Vocus Sydney"
                        }
                        add_tag => [ "International IP Transit - Vocus Sydney", "International"]
                }
        }
        if [INPUT_SNMP] == 566 {
                mutate {
                        add_field =>  {
                                "INPUT_DESC" => "Peering - MegaIX Sydney"
                        }
                        add_tag => [ "Peering - MegaIX Sydney", "Domestic"]
                }
        }
        if [OUTPUT_SNMP] == 566 {
                mutate {
                        add_field =>  {
                                "OUTPUT_DESC" => "Peering - MegaIX Sydney"
                        }
                        add_tag => [ "Peering - MegaIX Sydney", "Domestic"]
                }
        }
}