Difference between revisions of "Logstash"
From Initech Technical Wiki
| Line 3: | Line 3: | ||
== input == | == input == | ||
| − | < | + | <pre style="white-space: pre-wrap; |
| + | white-space: -moz-pre-wrap; | ||
| + | white-space: -pre-wrap; | ||
| + | white-space: -o-pre-wrap; | ||
| + | word-wrap: break-word;"> | ||
input { | input { | ||
udp { | udp { | ||
| Line 14: | Line 18: | ||
} | } | ||
} | } | ||
| − | </ | + | </pre> |
== filter == | == filter == | ||
| − | < | + | <pre style="white-space: pre-wrap; |
| + | white-space: -moz-pre-wrap; | ||
| + | white-space: -pre-wrap; | ||
| + | white-space: -o-pre-wrap; | ||
| + | word-wrap: break-word;"> | ||
filter { | filter { | ||
if [type] == "syslog" { | if [type] == "syslog" { | ||
| − | </ | + | </pre> |
=== Delete useless log messages === | === Delete useless log messages === | ||
| − | < | + | <pre style="white-space: pre-wrap; |
| + | white-space: -moz-pre-wrap; | ||
| + | white-space: -pre-wrap; | ||
| + | white-space: -o-pre-wrap; | ||
| + | word-wrap: break-word;"> | ||
if [message] =~ /Access Server SDK: No log writers/ or [message] =~ /Warning: Could not obtain lock on \/mnt\/oracle\/ohs/ or [message] =~ /last message repeated [0-9]+ times/ { | if [message] =~ /Access Server SDK: No log writers/ or [message] =~ /Warning: Could not obtain lock on \/mnt\/oracle\/ohs/ or [message] =~ /last message repeated [0-9]+ times/ { | ||
drop { } | drop { } | ||
} | } | ||
| − | </ | + | </pre> |
=== Match Cisco ACL logs === | === Match Cisco ACL logs === | ||
| − | < | + | <pre style="white-space: pre-wrap; |
| + | white-space: -moz-pre-wrap; | ||
| + | white-space: -pre-wrap; | ||
| + | white-space: -o-pre-wrap; | ||
| + | word-wrap: break-word;"> | ||
grok { | grok { | ||
match => { "message" => "%{SYSLOG5424PRI}%{NUMBER:sequence}: %{SYSLOGTIMESTAMP:log_timestamp}: \%SEC-6-IPACCESSLOGP: list %{INT:acl_number} %{WORD:action} %{WORD:protocol} %{IP:src_ip}\(%{NUMBER:src_port}\) \-\> %{IP:dst_ip}\(%{NUMBER:dst_port}\), %{INT:packets} packet" } | match => { "message" => "%{SYSLOG5424PRI}%{NUMBER:sequence}: %{SYSLOGTIMESTAMP:log_timestamp}: \%SEC-6-IPACCESSLOGP: list %{INT:acl_number} %{WORD:action} %{WORD:protocol} %{IP:src_ip}\(%{NUMBER:src_port}\) \-\> %{IP:dst_ip}\(%{NUMBER:dst_port}\), %{INT:packets} packet" } | ||
| Line 40: | Line 56: | ||
remove_field => [ "message" ] | remove_field => [ "message" ] | ||
} | } | ||
| − | </ | + | </pre> |
Revision as of 21:08, 18 November 2014
For various reasons, not least of which is timestamp normalisation and/or everything normalisation for that matter i chose to use a standard udp and tcp input method for logstash and skip the in-built syslog one. This meant that i needed to reinvent the wheel somewhat when it came to grok filters however. So here i have documented the grok filters that I built and why.
input
input {
udp {
port => 514
type => syslog
}
tcp {
port => 514
type => syslog
}
}
filter
filter {
if [type] == "syslog" {
Delete useless log messages
if [message] =~ /Access Server SDK: No log writers/ or [message] =~ /Warning: Could not obtain lock on \/mnt\/oracle\/ohs/ or [message] =~ /last message repeated [0-9]+ times/ {
drop { }
}
Match Cisco ACL logs
grok {
match => { "message" => "%{SYSLOG5424PRI}%{NUMBER:sequence}: %{SYSLOGTIMESTAMP:log_timestamp}: \%SEC-6-IPACCESSLOGP: list %{INT:acl_number} %{WORD:action} %{WORD:protocol} %{IP:src_ip}\(%{NUMBER:src_port}\) \-\> %{IP:dst_ip}\(%{NUMBER:dst_port}\), %{INT:packets} packet" }
add_tag => ["grok_match", "cisco_acl_message"]
remove_field => [ "message" ]
}