Top

NSD ARCHITECTURE DAY04

  1. 案例1:安装Kibana
  2. 案例2:综合练习

1 案例1:安装Kibana

1.1 问题

本案例要求:

1.2 步骤

实现此案例需要按照如下步骤进行

步骤一:安装kibana

1)在另一台主机,配置ip为192.168.1.56,配置yum源,更改主机名

2)安装kibana

[root@kibana ~]# yum -y install kibana 
[root@kibana ~]# rpm -qc kibana
/opt/kibana/config/kibana.yml
[root@kibana ~]# vim /opt/kibana/config/kibana.yml
  2 server.port: 5601		
//若把端口改为80,可以成功启动kibana,但ss时没有端口,没有监听80端口,服务里面写死了,不能用80端口,只能是5601这个端口
  5 server.host: "0.0.0.0"		//服务器监听地址
15 elasticsearch.url: http://192.168.1.51:9200	
//声明地址,从哪里查,集群里面随便选一个
23 kibana.index: ".kibana"	//kibana自己创建的索引
26 kibana.defaultAppId: "discover"	//打开kibana页面时,默认打开的页面discover
53 elasticsearch.pingTimeout: 1500	//ping检测超时时间
57 elasticsearch.requestTimeout: 30000	//请求超时
64 elasticsearch.startupTimeout: 5000	//启动超时
[root@kibana ~]# systemctl restart kibana
[root@kibana ~]# systemctl enable  kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
[root@kibana ~]# ss -antup | grep 5601  //查看监听端口

3)浏览器访问kibana,如图-1所示:

[student@room9pc01 ~]$ firefox 192.168.1.56:5601

图-1

4)点击Status,查看是否安装成功,全部是绿色的对钩,说明安装成功,如图-2所示:

图-2

5)用head插件访问会有.kibana的索引信息,如图-3所示:

[student@room9pc01 ~]$ firefox http://192.168.1.55:9200/_plugin/head

图-3

步骤二:使用kibana查看数据是否导入成功

1)数据导入以后查看logs是否导入成功,如图-4所示:

[student@room9pc01 ~]$ firefox http://192.168.1.55:9200/_plugin/head

图-4

2)kibana导入数据,如图-5所示:

[student@room9pc01 ~]$ firefox  http://192.168.1.56:5601

图-5

3)成功创建会有logstash-*,如图-6所示:

图-6

4)导入成功之后选择Discover,如图-7所示:

图-7

注意: 这里没有数据的原因是导入日志的时间段不对,默认配置是最近15分钟,在这可以修改一下时间来显示

5)kibana修改时间,选择Lsat 15 miuntes,如图-8所示:

图-8

6)选择Absolute,如图-9所示:

图-9

7)选择时间2015-5-15到2015-5-22,如图-10所示:

图-10

8)查看结果,如图-11所示:

图-11

9)除了柱状图,Kibana还支持很多种展示方式 ,如图-12所示:

图-12

10)做一个饼图,选择Pie chart,如图-13所示:

图-14

11)选择from a new serach,如图-11所示:

图-15

12)选择Spilt Slices,如图-16所示:

图-16

13)选择Trems,Memary(也可以选择其他的,这个不固定),如图-17所示:

图-17

14)结果,如图-18所示:

图-18

15)保存后可以在Dashboard查看,如图-19所示:

图-19

2 案例2:综合练习

2.1 问题

本案例要求:

2.2 步骤

实现此案例需要按照如下步骤进行。

步骤一:安装logstash

1)配置主机名,ip和yum源,配置/etc/hosts(请把es1-es5、kibana主机配置和logstash一样的/etc/hosts)

[root@logstash ~]# vim /etc/hosts
192.168.1.51 es1
192.168.1.52 es2
192.168.1.53 es3
192.168.1.54 es4
192.168.1.55 es5
192.168.1.56 kibana
192.168.1.57 logstash

2)安装java-1.8.0-openjdk和logstash

[root@logstash ~]#  yum -y install java-1.8.0-openjdk
[root@logstash ~]# yum -y install logstash
[root@logstash ~]# java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

[root@logstash ~]# touch /etc/logstash/logstash.conf
[root@logstash ~]#  /opt/logstash/bin/logstash  --version
logstash 2.3.4
[root@logstash ~]# /opt/logstash/bin/logstash-plugin  list   //查看插件
...
logstash-input-stdin	//标准输入插件
logstash-output-stdout	//标准输出插件
...
[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
    stdin{

   }
}

filter{

}

output{
    stdout{

   }
}

[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf 
//启动并测试
Settings: Default pipeline workers: 2
Pipeline main started
aa		//logstash 配置从标准输入读取输入源,然后从标准输出输出到屏幕
2018-09-15T06:19:28.724Z logstash aa

备注:若不会写配置文件可以找帮助,插件文档的位置:

https://github.com/logstash-plugins

3)codec类插件

[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
    stdin{
    codec => "json"		//输入设置为编码json
   }
}

filter{

}

output{
    stdout{
    codec => "rubydebug"		//输出设置为rubydebug
   }
}
[root@logstash ~]#  /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf 
Settings: Default pipeline workers: 2
Pipeline main started
{"a":1}
{
             "a" => 1,
      "@version" => "1",
    "@timestamp" => "2019-03-12T03:25:58.778Z",
          "host" => "logstash"
}

4)file模块插件

[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
  file {
    path          => [ "/tmp/a.log", "/tmp/b.log" ]
   sincedb_path   => "/var/lib/logstash/sincedb"	//记录读取文件的位置
   start_position => "beginning"				//配置第一次读取文件从什么地方开始
   type           => "testlog"					//类型名称
  }
}

filter{

}

output{
    stdout{
    codec => "rubydebug"
}
}

[root@logstash ~]# touch /tmp/a.log
[root@logstash ~]# touch /tmp/b.log
[root@logstash ~]#  /opt/logstash/bin/logstash -f  /etc/logstash/logstash.conf 

另开一个终端:写入数据

[root@logstash ~]#  echo a1 > /tmp/a.log 
[root@logstash ~]#  echo b1 > /var/tmp/b.log

之前终端查看:

 [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf 
Settings: Default pipeline workers: 2
Pipeline main started
{
       "message" => "a1",
      "@version" => "1",
    "@timestamp" => "2019-03-12T03:40:24.111Z",
          "path" => "/tmp/a.log",
          "host" => "logstash",
          "type" => "testlog"
}
{
       "message" => "b1",
      "@version" => "1",
    "@timestamp" => "2019-03-12T03:40:49.167Z",
          "path" => "/tmp/b.log",
          "host" => "logstash",
          "type" => "testlog"
}
	

7)filter grok插件

grok插件:

解析各种非结构化的日志数据插件

grok使用正则表达式把飞结构化的数据结构化

在分组匹配,正则表达式需要根据具体数据结构编写

虽然编写困难,但适用性极广

解析Apache的日志

[root@es5 ~]# yum -y install httpd
[root@es5 ~]# systemctl restart httpd

浏览器访问网页,在/var/log/httpd/access_log有日志出现

[root@es5 ~]# cat /var/log/httpd/access_log
192.168.1.254 - - [12/Mar/2019:11:51:31 +0800] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"

[root@logstash ~]#  vim /etc/logstash/logstash.conf

input{
    file {
      path           => [ "/tmp/a.log", "/tmp/b.log" ]
      sincedb_path   => "/var/lib/logstash/sincedb"
      start_position => "beginning"
      type           => "testlog"
   }
}

filter{
    grok{
       match => [ "message",  "(?<key>reg)" ]
    }
}

output{
    stdout{
    codec => "rubydebug"
   }
}




复制/var/log/httpd/access_log的日志到logstash下的/tmp/a.log

[root@logstash ~]# vim /tmp/a.log
192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] "GET / HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"

[root@logstash ~]#  /opt/logstash/bin/logstash -f  /etc/logstash/logstash.conf
//出现message的日志,但是没有解析是什么意思
Settings: Default pipeline workers: 2
Pipeline main started
{
       "message" => ".168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET / HTTP/1.1\" 403 4897 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
      "@version" => "1",
    "@timestamp" => "2018-09-15T10:26:51.335Z",
          "path" => "/tmp/a.log",
          "host" => "logstash",
          "type" => "testlog",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

若要解决没有解析的问题,同样的方法把日志复制到/tmp/a.log,logstash.conf配置文件里面修改grok

查找正则宏路径

[root@logstash ~]# cd  /opt/logstash/vendor/bundle/ \ 
jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/
[root@logstash ~]# vim grok-patterns  //查找COMBINEDAPACHELOG
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

[root@logstash ~]#  vim /etc/logstash/logstash.conf
...
filter{
   grok{
        match => ["message", "%{COMBINEDAPACHELOG}"]
  }
}
...

解析出的结果

 [root@logstash ~]#  /opt/logstash/bin/logstash -f  /etc/logstash/logstash.conf  
Settings: Default pipeline workers: 2
Pipeline main started
{
        "message" => "192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET /noindex/css/open-sans.css HTTP/1.1\" 200 5081 \"http://192.168.1.65/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
       "@version" => "1",
     "@timestamp" => "2018-09-15T10:55:57.743Z",
           "path" => "/tmp/a.log",
ZZ           "host" => "logstash",
           "type" => "testlog",
       "clientip" => "192.168.1.254",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "15/Sep/2019:18:25:46 +0800",
           "verb" => "GET",
        "request" => "/noindex/css/open-sans.css",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "5081",
       "referrer" => "\"http://192.168.1.65/\"",
          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\""
}
...

步骤二: 安装Apache服务,用filebeat收集Apache服务器的日志,并存入elasticsearch

1)在之前安装了Apache的主机上面安装filebeat

[root@se5 ~]#  yum -y install filebeat
[root@se5 ~]#  vim/etc/filebeat/filebeat.yml  
paths:
	- /var/log/httpd/access_log   //日志的路径,短横线加空格代表yml格式
document_type: apachelog    //文档类型
elasticsearch:		//加上注释
hosts: ["localhost:9200"]				//加上注释
logstash:					//去掉注释
hosts: ["192.168.1.57:5044"] 	//去掉注释,logstash那台主机的ip
[root@se5 ~]# systemctl start filebeat

[root@logstash ~]#  vim /etc/logstash/logstash.conf
input{
        stdin{ codec => "json" }
        beats{
            port => 5044
}
  file {
    path          => [ "/tmp/a.log", "/tmp/b.log" ]
   sincedb_path   => "/var/lib/logstash/sincedb"
   start_position => "beginning"
   type           => "testlog"
  }


filter{
if [type] == "apachelog"{
   grok{
        match => ["message", "%{COMBINEDAPACHELOG}"]
  }}
}

output{
      stdout{ codec => "rubydebug" }
      if [type] == "filelog"{
      elasticsearch {
          hosts => ["192.168.1.51:9200", "192.168.1.52:9200"]
          index => "filelog"
          flush_size => 2000
          idle_flush_time => 10
      }}
}
 [root@logstash logstash]#  /opt/logstash/bin/logstash  \ 
-f  /etc/logstash/logstash.conf

打开另一终端查看5044是否成功启动

[root@logstash ~]#  netstat -antup | grep 5044
tcp6       0      0 :::5044                 :::*                    LISTEN      23776/java

[root@se5 ~]#  firefox 192.168.1.55   //ip为安装filebeat的那台机器

回到原来的终端,有数据

2)修改logstash.conf文件

[root@logstash logstash]# vim logstash.conf
...
output{
      stdout{ codec => "rubydebug" }
      if [type] == "apachelog"{
      elasticsearch {
          hosts => ["192.168.1.51:9200", "192.168.1.52:9200"]
          index => "apachelog"
          flush_size => 2000
          idle_flush_time => 10
      }}
}

浏览器访问Elasticsearch,有apachelog,如图-20所示:

[student@room9pc01 ~]$ firefox http://192.168.1.55:9200/_plugin/head

图-20