ELK安装部署

发表于 | 分类于

ELK安装部署

标签:ELK

ELK包括三个组件：

Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据三大功能

Logstash 主要是用来日志的搜集、分析、过滤日志的工具，支持大量的数据获取方式。一般工作方式为c/s架构，client端安装在需要收集日志的主机上，server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去

Kibana 也是一个开源和免费的工具，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。

Filebeat隶属于Beats。目前Beats包含四种工具：

  • Packetbeat（搜集网络流量数据）

  • Topbeat（搜集系统、进程和文件系统级别的 CPU 和内存使用情况等数据）

  • Filebeat（搜集文件数据）

  • Winlogbeat（搜集 Windows 事件日志数据）

Elasticsearch安装

下载二进制执行文件下载地址

解压后进入Elasticsearch文件目录

1
2
3
4
5
vim config/elasticsearch.yml

http.cors.enabled: true
http.cors.allow-origin: "*"
network.host: 0.0.0.0

修改完之后

1
2
su es
./elasticearch & #elasticearch 必须以普通用户运行，文件描述符必须为65535

logstash安装

下载二进制执行文件下载地址

新建配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
vim logstash.conf
input {
  beats {
    port => 5044 # 此端口需要与 filebeat.yml 中的端口相同
  }
}
input{
tcp{
   host => "0.0.0.0" port => 9600 mode => "server" tags => ["app-log"] codec => json_lines
  }
}
input{
tcp{
   host => "0.0.0.0" port => 9610 mode => "server" tags => ["log-passenger"] codec => json_lines
  }
}
input{
tcp{
   host => "0.0.0.0" port => 9620 mode => "server" tags => ["log-driver"] codec => json_lines
  }
}
  input{
tcp{
   host => "0.0.0.0" port => 9630 mode => "server" tags => ["log-socket"] codec => json_lines
  }
}

output {
  if "passenger1" in [tags]{
  elasticsearch {
    hosts => "localhost:9200"
    index => "log-passenger1"
  }
 }
if "driver1" in [tags]{
  elasticsearch {
    hosts => "localhost:9200"
    index => "log-driver1"
  }
 }
 if "java1" in [tags]{
  elasticsearch {
    hosts => "localhost:9200"
    index => "log-java1"
  }
 }
if "java2" in [tags]{
  elasticsearch {
    hosts => "localhost:9200"
    index => "log-java2"
  }
 }
if "app-log" in [tags]{
  elasticsearch {
    hosts => "localhost:9200"
    index => "log-app"
  }  
 }
if "log-passenger" in [tags]{
  elasticsearch {
    hosts => "localhost:9200"
    index => "log-passenger"
  }
 }
if "log-driver" in [tags]{
  elasticsearch {
    hosts => "localhost:9200"
    index => "log-driver"
  }
 }
if "log-socket" in [tags]{
  elasticsearch {
    hosts => "localhost:9200"
    index => "log-socket"
  }
 }
}
#根据自己对应要求修改

启动logstash

1
bin/logstash -f config/5400.conf &

filebeat安装

下载二进制执行文件下载地址

1
2
3
4
5
6
7
8
9
10
11
12
13
filebeat.prospectors:
- input_type: log
  enabled: true
  paths:
    - /home/driver/logs/DriverApplication.log
  tags: ["java1"]
  multiline:
    pattern: '^\d{4}-\d{1,2}-\d{1,2}'
    negate: true
    match: after
output.logstash:
     hosts: ["192.168.0.33:5044"]
#与logstash文件对应

启动filebeat

1
nohup ./filebeat -e -c filebeat.yml &

kibana安装

下载二进制执行文件下载地址

1
2
3
vim config/kibana.yml
elasticsearch.url: "http://localhost:9200" #修改自己的地址
server.host: 0.0.0.0

启动kibana.yml

1
nohup ./bin/kibana > /dev/null 2>&1 &

Sentinl安装

Sentinl是kibana的一个扩展插件，实现日志匹配报警

1
./kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.3-3/sentinl-v6.2.4.zip #版本须根据安装的kibana版本来定

完成之后修改kibana配置文件

1
2
3
4
5
6
7
8
9
10
11
#在末尾添加
sentinl:
  settings:
    email:
      active: true
      user: xxxxxxx@163.com
      password: xxxxx
      host: smtp.163.com
      ssl: true
    report:
      active: true

在kibana中点击Sentinl进行配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
{
  "_index": "watcher",
  "_type": "sentinl-watcher",
  "_id": "qopmry0hym-ob9vtzk8f5j-71rek2bnhq",
  "_version": 40,
  "found": true,
  "_source": {
    "title": "ERROR-DRIVER",
    "disable": false,
    "report": false,
    "trigger": {
      "schedule": {
        "later": "every 5 mins"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "log-driver"
          ],
          "body": {
            "query": {
              "bool": {
                "must": [
                  {
                    "match": {
                      "level": "ERROR"
                    }
                  },
                  {
                    "range": {
                      "@timestamp": {
                        "gte": "now-5m",
                        "lte": "now",
                        "format": "epoch_millis"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.hits.total > 0"
      }
    },
    "actions": {
      "email_admin": {
        "throttle_period": "0h5m0s",
        "email": {
          "to": "522336309@qq.com",
          "from": "xxxxxx@163.com",
          "subject": "ERROR alarm",
          "priority": "high",
          "body": "time:{{payload.hits.hits.0._source.@timestamp}} count_error:{{payload.hits.total}} message:{{payload.hits.hits.0._source.message}}"
        }
      }
    }
  }
}

常用操作

1
2
3
#Elasticsearch查询操作
curl -u elastic:changeme 'localhost:9200/_cat/indices?v' #查看所有索引
curl -XDELETE http://localhost:9200/_all #删除所有索引