ELK集群模式部署

架构拓扑图为：

准备工作：

下载资源包：

Elasticsearch： wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz  # 这一步用docker启动，可以不用下载。

Kibana: wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86\_64.tar.gz

Logstash:wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz

Filebeat:wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86\_64.tar.gz

安装：

根据拓扑图把对应的服务安装在对应的服务器。

tar xvf kibana-6.2.3-linux-x86_64.tar.gz -C /usr/local/

tar xvf logstash-6.2.4.tar.gz -C /usr/local/

tar -xvf filebeat-6.2.4-linux-x86_64.tar.gz -C /usr/local/

部署启动:

Elasticsearch-master(192.168.11.1)：

需自行提前安装好docker 和 docker-compose。 

1vim docker-compose.yml
2
3version: '2'
4services:
5  elasticsearch:
6    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
7    container_name: elasticsearch
8    environment:
9      - cluster.name=es-cluster
10      - bootstrap.memory_lock=true
11      - "network.publish_host=192.168.11.1"
12      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
13    ulimits:
14      memlock:
15        soft: -1
16        hard: -1
17    volumes:
18      - /data/elasticsearch_data:/usr/share/elasticsearch/data
19    ports:
20      - "192.168.11.1:9200:9200"
21      - "192.168.11.1:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

Elasticsearch-node2.data(192.168.11.2)：

需自行提前安装好docker 和 docker-compose。 

1vim docker-compose.yml
2
3version: '2'
4services:
5  elasticsearch:
6    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
7    container_name: elasticsearch
8    environment:
9      - cluster.name=es-cluster
10      - bootstrap.memory_lock=true
11      - "network.publish_host=192.168.11.2"      - "discovery.zen.ping.unicast.hosts=192.168.11.1"
12      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
13    ulimits:
14      memlock:
15        soft: -1
16        hard: -1
17    volumes:
18      - /data/elasticsearch_data:/usr/share/elasticsearch/data
19    ports:
20      - "192.168.11.2:9200:9200"
21      - "192.168.11.2:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

Elasticsearch-node3.data(192.168.11.3)：

需自行提前安装好docker 和 docker-compose。 

1vim docker-compose.yml
2
3version: '2'
4services:
5  elasticsearch:
6    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
7    container_name: elasticsearch
8    environment:
9      - cluster.name=es-cluster
10      - bootstrap.memory_lock=true
11      - "network.publish_host=192.168.11.3"      - "discovery.zen.ping.unicast.hosts=192.168.11.1"
12      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
13    ulimits:
14      memlock:
15        soft: -1
16        hard: -1
17    volumes:
18      - /data/elasticsearch_data:/usr/share/elasticsearch/data
19    ports:
20      - "192.168.11.3:9200:9200"
21      - "192.168.11.3:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

查看集群节点：curl -XGET 'http://192.168.11.1:9200/\_cat/nodes?pretty'

 kibana(192.168.11.4)：

cd /usr/local/kibana-6.2.3-linux-x86_64

cp ./config/kibana.yml ./config/kibana.yml.bak

echo '' > ./config/kibana.yml

vim ./config/kibana.yml

1server.port: 5601
2server.host: "192.168.11.4"
3elasticsearch.url: "http://192.168.11.1:9200"

启动：./bin/kibana

扩展包（可装可不装）：

kibana 报警扩展: 

安装 ./bin/kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.3-2/sentinl-v6.2.3.zip

重启 kibana

logstash(192.168.11.5)

cd /usr/local/logstash-6.2.4

vim nginx.conf

1input {
2beats {
3port => 5066
4host => "192.168.11.5"
5codec => "json"
6}
7}
8filter {
9
10mutate {
11
12gsub => ["message", "\\x", "\\\x"]
13
14}
15
16json {
17
18source => "message"
19
20}
21
22}
23output {
24    elasticsearch {
25        action => "index"
26        hosts => "192.168.11.1:9200"
27        index => "nginx-json-%{+YYYY.MM.dd}"
28   }
29}

mkdir -p /data/logstash-data/nginx

启动：./bin/logstash -f ./nginx.conf --path.data=/data/logstash-data/nginx &

扩展

grok调试地址：http://grokdebug.herokuapp.com/

如需调试请修改output成以下，方便debug

1output {
2        stdout {
3        codec    => rubydebug
4    }
5}

filebeat(192.168.11.6)

cd /usr/local/filebeat-6.2.4-linux-x86_64

vim nginx.yml

1filebeat.prospectors:
2- input_type: log
3  document_type: jsonlog
4  paths:
5    - /var/log/nginx/access.log
6output:
7  logstash:
8    hosts: ["192.168.11.5:5066"]

启动：./filebeat -c ./nginx.yml &

nginx 日志的输出格式为：

1log_format json '{"@timestamp":"$time_iso8601",'
2               '"remote_addr":"$remote_addr",'
3               '"request":"$request",'
4               '"status":$status,'
5               '"body_bytes":$body_bytes_sent,'
6               '"user_agent":"$http_user_agent",'
7               '"resp_time":"$upstream_response_time",'
8               '"req_time":$request_time,'
9               '"host":"$host",'
10               '"@version":"1",'
11               '"http_x_forwarded_for":"$http_x_forwarded_for",'
12               '"upstream_addr":"$upstream_addr",'
13               '"req_body":"$request_body"}';

参考地址：

https://www.elastic.co/guide/index.html 

http://docs.docker.com/compose/

https://www.elastic.co/guide/en/logstash/current/index.html

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docker.html

https://www.elastic.co/guide/en/kibana/current/install.html