前一篇我们已经学会了logstash-input-file插件的用法,我们现在在上一篇的基础上来学习一下filter。
首先呢,还是来伪造数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[sqczm@sqczm logstash-6.7.1]$ pwd
/opt/logstash-6.7.1
[sqczm@sqczm logstash-6.7.1]$ ls demo/second/
events.txt second.conf
[sqczm@sqczm logstash-6.7.1]$ more demo/second/events.txt
2019-04-20 20:21:00 64 bytes from 8.8.8.8: icmp_seq=1 ttl=64 time=1.720 ms
2019-04-20 20:21:01 64 bytes from 8.8.8.8: icmp_seq=2 ttl=64 time=2.197 ms
[sqczm@sqczm logstash-6.7.1]$ more demo/second/second.conf
input {
file {
path => ["/opt/logstash-6.7.1/demo/second/events.txt"]
start_position => "beginning"
}
}
filter {

}
output {
stdout {}
}
[sqczm@sqczm logstash-6.7.1]$ bin/logstash -f demo/second/second.conf
……中间省略部分输出……
{
"@version" => "1",
"host" => "sqczm",
"@timestamp" => 2019-04-20T12:28:58.365Z,
"message" => "2019-04-20 20:21:01 64 bytes from 8.8.8.8: icmp_seq=2 ttl=64 time=2.197 ms",
"path" => "/opt/logstash-6.7.1/demo/second/events.txt"
}
{
"@version" => "1",
"host" => "sqczm",
"@timestamp" => 2019-04-20T12:28:58.323Z,
"message" => "2019-04-20 20:21:00 64 bytes from 8.8.8.8: icmp_seq=1 ttl=64 time=1.720 ms",
"path" => "/opt/logstash-6.7.1/demo/second/events.txt"
}

我们来看上面的数据,其实我的想法很简单,就是想完成以下功能:

  1. 将输出中的@timestamp字段设置为message字段中的时间
  2. 将message中的信息提取出来其他几个属性:bytes、ip、icmp_seq、ttl、time

带着上面的这个疑问,我们来看如何实现。
首先,我们需要先把@timestamp字段给替换掉,要替换这个字段我们就需要logstash-filter-date这个插件了,有关该插件的详细说明可以看下面的网址
logstash-filter-date详细说明
貌似我们了解了logstash-filter-date这个插件还不够,我们还需要了解另一个插件,这个插件就是logstash-filter-grok,因为要用它来写正则解析文本的信息,有关该插件的详细说明可以看下面的网址
logstash-filter-grok详细说明

了解完上面两个插件后,我们就来实现我们的需求功能

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
[sqczm@sqczm logstash-6.7.1]$ pwd
/opt/logstash-6.7.1
[sqczm@sqczm logstash-6.7.1]$ rm data/plugins/inputs/file/.sincedb_814435f84c5f13c338c4625fd2af163b
[sqczm@sqczm logstash-6.7.1]$ more demo/second/second.conf
input {
file {
path => ["/opt/logstash-6.7.1/demo/second/events.txt"]
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:logdate} %{NUMBER:bytes} by
tes from %{IP:ip}: icmp_seq=%{NUMBER:icmp_seq} ttl=%{NUMBER:ttl} time=%{NUMBER:t
ime} ms" }
}
date {
match => [ "logdate", "yyyy-MM-dd HH:mm:ss"]
}
}
output {
stdout {}
}
[sqczm@sqczm logstash-6.7.1]$ bin/logstash -f demo/second/second.conf
……省略部分输出内容……
{
"message" => "2019-04-20 20:21:01 64 bytes from 8.8.8.8: icmp_seq=2 ttl=64 time=2.197 ms",
"icmp_seq" => "2",
"host" => "sqczm",
"bytes" => "64",
"ip" => "8.8.8.8",
"time" => "2.197",
"path" => "/opt/logstash-6.7.1/demo/second/events.txt",
"@version" => "1",
"logdate" => "2019-04-20 20:21:01",
"ttl" => "64",
"@timestamp" => 2019-04-20T12:21:01.000Z
}
{
"message" => "2019-04-20 20:21:00 64 bytes from 8.8.8.8: icmp_seq=1 ttl=64 time=1.720 ms",
"icmp_seq" => "1",
"host" => "sqczm",
"bytes" => "64",
"ip" => "8.8.8.8",
"time" => "1.720",
"path" => "/opt/logstash-6.7.1/demo/second/events.txt",
"@version" => "1",
"logdate" => "2019-04-20 20:21:00",
"ttl" => "64",
"@timestamp" => 2019-04-20T12:21:00.000Z
}

至此,我们的功能已经实现完毕,插件中比较麻烦的就是grok部分,其实上该插件已经帮我们提供了很多的正则,官方文档中已经说明了,这些正则可以在以下的网址中找到
内置的正则
当然你自己写正则的话,可以使用以下网址进行验证
Grok正则验证