前言

本篇文章主要介绍在两台机器上使用 Docker 搭建 ELK。

正文

环境

CentOS 7.7 系统
Docker version 19.03.8
docker-compose version 1.23.2

系统设置

vim 编辑 /etc/security/limits.conf，在末尾加上：

1* soft nofile 65536
2* hard nofile 65536
3* soft nproc 4096
4* hard nproc 4096

vim 编辑 /etc/sysctl.conf，在末尾加上：

vm.max_map_count = 655360

执行 sysctl -p 命令是配置生效。

Elasticsearch 搭建

注意：如果用非 Docker 搭建，是不能用 root 用户去启动的。

由于我是用虚拟机搭建的，我的机器只能开两台，所以只有一个主节点和一个数据节点；在生产环境中最少要3台，防止脑裂问题。

注意：如果开启了防火墙，需要执行以下命令开放 9200 和 9300 端口号。
1firewall-cmd --zone=public --add-port=9200/tcp --permanent
2firewall-cmd --zone=public --add-port=9300/tcp --permanent

主节点

首先设置主节点的配置文件 elasticsearch.yml，如下：

1# ======================== Elasticsearch Configuration =========================
2#
3# NOTE: Elasticsearch comes with reasonable defaults for most settings.
4#       Before you set out to tweak and tune the configuration, make sure you
5#       understand what are you trying to accomplish and the consequences.
6#
7# The primary way of configuring a node is via this file. This template lists
8# the most important settings you may want to configure for a production cluster.
9#
10# Please consult the documentation for further information on configuration options:
11# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
12#
13# ---------------------------------- Cluster -----------------------------------
14#
15# Use a descriptive name for your cluster:
16
17cluster.name: es-cluster
18#
19# ------------------------------------ Node ------------------------------------
20#
21# Use a descriptive name for the node:
22
23node.name: es-master
24
25node.master: true
26
27node.data: false
28
29#node.ingest: false
30
31#node.ml: false
32#xpack.ml.enabled: true
33
34#cluster.remote.connect: false
35#
36# Add custom attributes to the node:
37#
38#node.attr.rack: r1
39#
40# ----------------------------------- Paths ------------------------------------
41#
42# Path to directory where to store the data (separate multiple locations by comma):
43#
44#path.data: /path/to/data
45#
46# Path to log files:
47#
48#path.logs: /path/to/logs
49#
50# ----------------------------------- Memory -----------------------------------
51#
52# Lock the memory on startup:
53#
54#bootstrap.memory_lock: true
55#
56# Make sure that the heap size is set to about half the memory available
57# on the system and that the owner of the process is allowed to use this
58# limit.
59#
60# Elasticsearch performs poorly when the system is swapping the memory.
61#
62# ---------------------------------- Network -----------------------------------
63#
64# Set the bind address to a specific IP (IPv4 or IPv6):
65
66network.host: 0.0.0.0
67network.publish_host: 192.168.239.133
68#
69# Set a custom port for HTTP:
70
71http.port: 9200
72
73transport.tcp.port: 9300
74#
75# For more information, consult the network module documentation.
76#
77# --------------------------------- Discovery ----------------------------------
78#
79# Pass an initial list of hosts to perform discovery when this node is started:
80# The default list of hosts is ["127.0.0.1", "[::1]"]
81#
82discovery.seed_hosts:
83  - 192.168.239.133
84  - 192.168.239.131
85#
86# Bootstrap the cluster using an initial set of master-eligible nodes:
87
88cluster.initial_master_nodes:
89  - es-master
90#  - es-node2
91#  - es-node3
92#
93# For more information, consult the discovery and cluster formation module documentation.
94#
95# ---------------------------------- Gateway -----------------------------------
96#
97# Block initial recovery after a full cluster restart until N nodes are started:
98#
99#gateway.recover_after_nodes: 2
100#
101# For more information, consult the gateway module documentation.
102#
103# ---------------------------------- Various -----------------------------------
104#
105# Require explicit names when deleting indices:
106#
107#action.destructive_requires_name: true
108
109
110http.cors.enabled: true
111http.cors.allow-origin: "*"

然后编写主节点的 docker-compose.yml，如下：

1version: "3"
2services:
3  es-master:
4    container_name: es-master
5    hostname: es-master
6    image: leisurexi/elasticsearch:7.1.0
7    privileged: true
8    ports:
9      - 9200:9200
10      - 9300:9300
11    volumes:
12      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
13      - ./data:/usr/share/elasticsearch/data
14      - ./logs:/usr/share/elasticsearch/logs
15    environment:
16      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
17    ulimits:
18      memlock:
19        soft: -1
20        hard: -1

注意：这个镜像是我自己 Docker Hup 上的，你可以换成官方的。（我的镜像和官方的一样，只是嫌每次下载太难，就把官方的镜像改了 tag 上传到自己的 Docker Hup 上了）

接着执行以下命令启动容器

docker-compose up -d

如果出现下图所示的错误，可以使用 chmod 777 logs 和 chmod 777 data 来修改文件夹的权限，即可正常启动。

然后进入容器，创建默认的用户和分组

1# 进入容器
2docker exec -it es-master bash
3
4
5# 激活默认用户并设置密码
6bin/elasticsearch-setup-passwords interactive

数据节点

首先设置数据节点的配置文件 elasticsearch.yml，如下：

1# ======================== Elasticsearch Configuration =========================
2#
3# NOTE: Elasticsearch comes with reasonable defaults for most settings.
4#       Before you set out to tweak and tune the configuration, make sure you
5#       understand what are you trying to accomplish and the consequences.
6#
7# The primary way of configuring a node is via this file. This template lists
8# the most important settings you may want to configure for a production cluster.
9#
10# Please consult the documentation for further information on configuration options:
11# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
12#
13# ---------------------------------- Cluster -----------------------------------
14#
15# Use a descriptive name for your cluster:
16
17cluster.name: es-cluster
18#
19# ------------------------------------ Node ------------------------------------
20#
21# Use a descriptive name for the node:
22
23node.name: es-data
24
25node.master: true
26
27node.data: true
28
29#node.ingest: false
30
31#node.ml: false
32#xpack.ml.enabled: true
33
34#cluster.remote.connect: false
35#
36# Add custom attributes to the node:
37#
38#node.attr.rack: r1
39#
40# ----------------------------------- Paths ------------------------------------
41#
42# Path to directory where to store the data (separate multiple locations by comma):
43#
44#path.data: /path/to/data
45#
46# Path to log files:
47#
48#path.logs: /path/to/logs
49#
50# ----------------------------------- Memory -----------------------------------
51#
52# Lock the memory on startup:
53#
54#bootstrap.memory_lock: true
55#
56# Make sure that the heap size is set to about half the memory available
57# on the system and that the owner of the process is allowed to use this
58# limit.
59#
60# Elasticsearch performs poorly when the system is swapping the memory.
61#
62# ---------------------------------- Network -----------------------------------
63#
64# Set the bind address to a specific IP (IPv4 or IPv6):
65
66network.host: 0.0.0.0
67network.publish_host: 192.168.239.131
68#
69# Set a custom port for HTTP:
70
71http.port: 9200
72
73transport.tcp.port: 9300
74#
75# For more information, consult the network module documentation.
76#
77# --------------------------------- Discovery ----------------------------------
78#
79# Pass an initial list of hosts to perform discovery when this node is started:
80# The default list of hosts is ["127.0.0.1", "[::1]"]
81#
82discovery.seed_hosts:
83  - 192.168.239.133
84  - 192.168.239.131
85#
86# Bootstrap the cluster using an initial set of master-eligible nodes:
87
88cluster.initial_master_nodes:
89  - es-master
90#  - es-node2
91#  - es-node3
92#
93# For more information, consult the discovery and cluster formation module documentation.
94#
95# ---------------------------------- Gateway -----------------------------------
96#
97# Block initial recovery after a full cluster restart until N nodes are started:
98#
99#gateway.recover_after_nodes: 2
100#
101# For more information, consult the gateway module documentation.
102#
103# ---------------------------------- Various -----------------------------------
104#
105# Require explicit names when deleting indices:
106#
107#action.destructive_requires_name: true
108
109
110http.cors.enabled: true
111http.cors.allow-origin: "*"

然后编写数据节点的 docker-compose.yml，如下：

1version: "3"
2services:
3  es-master:
4    container_name: es-data
5    hostname: es-data
6    image: leisurexi/elasticsearch:7.1.0
7    privileged: true
8    ports:
9      - 9200:9200
10      - 9300:9300
11    volumes:
12      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
13      - ./data:/usr/share/elasticsearch/data
14      - ./logs:/usr/share/elasticsearch/logs
15    environment:
16      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
17    ulimits:
18      memlock:
19        soft: -1
20        hard: -1

接着像上面主节点一样启动就行了，然后访问主节点的 http://192.168.239.133:9200/_cat/nodes API 地址，如下图所示就代表 Elasticsearch 集群搭建成功了。

Kibana 搭建

因为主节点负责集群范围内的轻量级操作，例如创建或删除索引，跟踪哪些节点是集群的一部分以及确定将哪些碎片分配给哪些节点；所以将 Kibana 跟主节点放在一台机器上。

注意：如果开启了防火墙，需要执行以下命令开放 5601 端口号。
firewall-cmd --zone=public --add-port=5601/tcp --permanent

首先是 Kibana 的配置文件 Kibana.yml，如下：

1# Kibana is served by a back end server. This setting specifies the port to use.
2server.port: 5601
3
4# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
5# The default is 'localhost', which usually means remote machines will not be able to connect.
6# To allow connections from remote users, set this parameter to a non-loopback address.
7server.host: "0.0.0.0"
8
9# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
10# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
11# from requests it receives, and to prevent a deprecation warning at startup.
12# This setting cannot end in a slash.
13#server.basePath: ""
14
15# Specifies whether Kibana should rewrite requests that are prefixed with
16# `server.basePath` or require that they are rewritten by your reverse proxy.
17# This setting was effectively always `false` before Kibana 6.3 and will
18# default to `true` starting in Kibana 7.0.
19#server.rewriteBasePath: false
20
21# The maximum payload size in bytes for incoming server requests.
22#server.maxPayloadBytes: 1048576
23
24# The Kibana server's name.  This is used for display purposes.
25#server.name: "your-hostname"
26
27# The URLs of the Elasticsearch instances to use for all your queries.
28elasticsearch.hosts: ["http://192.168.239.133:9200", "http://192.168.239.131:9200"]
29
30# When this setting's value is true Kibana uses the hostname specified in the server.host
31# setting. When the value of this setting is false, Kibana uses the hostname of the host
32# that connects to this Kibana instance.
33#elasticsearch.preserveHost: true
34
35# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
36# dashboards. Kibana creates a new index if the index doesn't already exist.
37#kibana.index: ".kibana"
38
39# The default application to load.
40#kibana.defaultAppId: "home"
41
42# If your Elasticsearch is protected with basic authentication, these settings provide
43# the username and password that the Kibana server uses to perform maintenance on the Kibana
44# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
45# is proxied through the Kibana server.
46#elasticsearch.username: "user"
47#elasticsearch.password: "pass"
48
49# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
50# These settings enable SSL for outgoing requests from the Kibana server to the browser.
51#server.ssl.enabled: false
52#server.ssl.certificate: /path/to/your/server.crt
53#server.ssl.key: /path/to/your/server.key
54
55# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
56# These files validate that your Elasticsearch backend uses the same key files.
57#elasticsearch.ssl.certificate: /path/to/your/client.crt
58#elasticsearch.ssl.key: /path/to/your/client.key
59
60# Optional setting that enables you to specify a path to the PEM file for the certificate
61# authority for your Elasticsearch instance.
62#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
63
64# To disregard the validity of SSL certificates, change this setting's value to 'none'.
65#elasticsearch.ssl.verificationMode: full
66
67# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
68# the elasticsearch.requestTimeout setting.
69#elasticsearch.pingTimeout: 1500
70
71# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
72# must be a positive integer.
73#elasticsearch.requestTimeout: 30000
74
75# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
76# headers, set this value to [] (an empty list).
77#elasticsearch.requestHeadersWhitelist: [ authorization ]
78
79# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
80# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
81#elasticsearch.customHeaders: {}
82
83# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
84#elasticsearch.shardTimeout: 30000
85
86# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
87#elasticsearch.startupTimeout: 5000
88
89# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
90#elasticsearch.logQueries: false
91
92# Specifies the path where Kibana creates the process ID file.
93#pid.file: /var/run/kibana.pid
94
95# Enables you specify a file where Kibana stores log output.
96#logging.dest: stdout
97
98# Set the value of this setting to true to suppress all logging output.
99#logging.silent: false
100
101# Set the value of this setting to true to suppress all logging output other than error messages.
102#logging.quiet: false
103
104# Set the value of this setting to true to log all events, including system usage information
105# and all requests.
106#logging.verbose: false
107
108# Set the interval in milliseconds to sample system and process performance
109# metrics. Minimum is 100ms. Defaults to 5000.
110#ops.interval: 5000
111
112# Specifies locale to be used for all localizable strings, dates and number formats.
113i18n.locale: "zh-CN"

然后是 docker-compose.yml 文件的编写，如下：

1version: "3"
2services:
3  kibana:
4    container_name: kibana
5    hostname: kibana
6    image: leisurexi/kibana:7.1.0
7    ports:
8      - 5601:5601
9    volumes:
10      - ./kibana.yml:/usr/share/kibana/config/kibana.yml

注意：这个镜像是我自己 Docker Hup 上的，你可以换成官方的。

接着像 Elasticsearch 几点一样启动就可以了。

我们访问 Kibana 节点的 5601 端口就可以看到界面了，接下来执行 GET _cluster/health 查看 ES 集群的健康状况，来验证 Kibana 是否可以正常工作。

如上图一样就代表你已经 kibana 已经搭建成功了。

logstash 搭建

logstash 在 ES 的数据节点上搭建。

注意：如果开启了防火墙，需要执行以下命令开放 4560 和 5044 端口号。
1firewall-cmd --zone=public --add-port=4560/tcp --permanent
2firewall-cmd --zone=public --add-port=5044/tcp --permanent

首先是 logstash 的全局配置文件 logstash.yml，如下：

1# Settings file in YAML
2#
3# Settings can be specified either in hierarchical form, e.g.:
4#
5#   pipeline:
6#     batch:
7#       size: 125
8#       delay: 5
9#
10# Or as flat keys:
11#
12#   pipeline.batch.size: 125
13#   pipeline.batch.delay: 5
14#
15# ------------  Node identity ------------
16#
17# Use a descriptive name for the node:
18#
19# node.name: test
20#
21# If omitted the node name will default to the machine's host name
22#
23# ------------ Data path ------------------
24#
25# Which directory should be used by logstash and its plugins
26# for any persistent needs. Defaults to LOGSTASH_HOME/data
27#
28# path.data:
29#
30# ------------ Pipeline Settings --------------
31#
32# The ID of the pipeline.
33#
34# pipeline.id: main
35#
36# Set the number of workers that will, in parallel, execute the filters+outputs
37# stage of the pipeline.
38#
39# This defaults to the number of the host's CPU cores.
40#
41# pipeline.workers: 2
42#
43# How many events to retrieve from inputs before sending to filters+workers
44#
45# pipeline.batch.size: 125
46#
47# How long to wait in milliseconds while polling for the next event
48# before dispatching an undersized batch to filters+outputs
49#
50# pipeline.batch.delay: 50
51#
52# Force Logstash to exit during shutdown even if there are still inflight
53# events in memory. By default, logstash will refuse to quit until all
54# received events have been pushed to the outputs.
55#
56# WARNING: enabling this can lead to data loss during shutdown
57#
58# pipeline.unsafe_shutdown: false
59#
60# ------------ Pipeline Configuration Settings --------------
61#
62# Where to fetch the pipeline configuration for the main pipeline
63#
64# path.config:
65#
66# Pipeline configuration string for the main pipeline
67#
68# config.string:
69#
70# At startup, test if the configuration is valid and exit (dry run)
71#
72# config.test_and_exit: false
73#
74# Periodically check if the configuration has changed and reload the pipeline
75# This can also be triggered manually through the SIGHUP signal
76#
77# config.reload.automatic: false
78#
79# How often to check if the pipeline configuration has changed (in seconds)
80#
81# config.reload.interval: 3s
82#
83# Show fully compiled configuration as debug log message
84# NOTE: --log.level must be 'debug'
85#
86# config.debug: false
87#
88# When enabled, process escaped characters such as \n and \" in strings in the
89# pipeline configuration files.
90#
91# config.support_escapes: false
92#
93# ------------ Module Settings ---------------
94# Define modules here.  Modules definitions must be defined as an array.
95# The simple way to see this is to prepend each `name` with a `-`, and keep
96# all associated variables under the `name` they are associated with, and
97# above the next, like this:
98#
99# modules:
100#   - name: MODULE_NAME
101#     var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
102#     var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
103#     var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
104#     var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE
105#
106# Module variable names must be in the format of
107#
108# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
109#
110# modules:
111#
112# ------------ Cloud Settings ---------------
113# Define Elastic Cloud settings here.
114# Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy
115# and it may have an label prefix e.g. staging:dXMtZ...
116# This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host'
117# cloud.id: <identifier>
118#
119# Format of cloud.auth is: <user>:<pass>
120# This is optional
121# If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password'
122# If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password'
123# cloud.auth: elastic:<password>
124#
125# ------------ Queuing Settings --------------
126#
127# Internal queuing model, "memory" for legacy in-memory based queuing and
128# "persisted" for disk-based acked queueing. Defaults is memory
129#
130# queue.type: memory
131#
132# If using queue.type: persisted, the directory path where the data files will be stored.
133# Default is path.data/queue
134#
135# path.queue:
136#
137# If using queue.type: persisted, the page data files size. The queue data consists of
138# append-only data files separated into pages. Default is 64mb
139#
140# queue.page_capacity: 64mb
141#
142# If using queue.type: persisted, the maximum number of unread events in the queue.
143# Default is 0 (unlimited)
144#
145# queue.max_events: 0
146#
147# If using queue.type: persisted, the total capacity of the queue in number of bytes.
148# If you would like more unacked events to be buffered in Logstash, you can increase the
149# capacity using this setting. Please make sure your disk drive has capacity greater than
150# the size specified here. If both max_bytes and max_events are specified, Logstash will pick
151# whichever criteria is reached first
152# Default is 1024mb or 1gb
153#
154# queue.max_bytes: 1024mb
155#
156# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
157# Default is 1024, 0 for unlimited
158#
159# queue.checkpoint.acks: 1024
160#
161# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
162# Default is 1024, 0 for unlimited
163#
164# queue.checkpoint.writes: 1024
165#
166# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
167# Default is 1000, 0 for no periodic checkpoint.
168#
169# queue.checkpoint.interval: 1000
170#
171# ------------ Dead-Letter Queue Settings --------------
172# Flag to turn on dead-letter queue.
173#
174# dead_letter_queue.enable: false
175
176# If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries
177# will be dropped if they would increase the size of the dead letter queue beyond this setting.
178# Default is 1024mb
179# dead_letter_queue.max_bytes: 1024mb
180
181# If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
182# Default is path.data/dead_letter_queue
183#
184# path.dead_letter_queue:
185#
186# ------------ Metrics Settings --------------
187#
188# Bind address for the metrics REST endpoint
189#
190# http.host: "127.0.0.1"
191#
192# Bind port for the metrics REST endpoint, this option also accept a range
193# (9600-9700) and logstash will pick up the first available ports.
194#
195# http.port: 9600-9700
196#
197# ------------ Debugging Settings --------------
198#
199# Options for log.level:
200#   * fatal
201#   * error
202#   * warn
203#   * info (default)
204#   * debug
205#   * trace
206#
207# log.level: info
208# path.logs:
209#
210# ------------ Other Settings --------------
211#
212# Where to find custom plugins
213# path.plugins: []
214#
215# ------------ X-Pack Settings (not applicable for OSS build)--------------
216#
217# X-Pack Monitoring
218# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
219xpack.monitoring.enabled: true
220#xpack.monitoring.elasticsearch.username: logstash_system
221#xpack.monitoring.elasticsearch.password: password
222xpack.monitoring.elasticsearch.hosts: ["http://192.168.239.133:9200", "http://192.168.239.131:9200"]
223#xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
224#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
225#xpack.monitoring.elasticsearch.ssl.truststore.password: password
226#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
227#xpack.monitoring.elasticsearch.ssl.keystore.password: password
228#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
229#xpack.monitoring.elasticsearch.sniffing: false
230#xpack.monitoring.collection.interval: 10s
231#xpack.monitoring.collection.pipeline.details.enabled: true
232#
233# X-Pack Management
234# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
235xpack.management.enabled: false
236#xpack.management.pipeline.id: ["main", "apache_logs"]
237#xpack.management.elasticsearch.username: logstash_admin_user
238#xpack.management.elasticsearch.password: password
239#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
240#xpack.management.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
241#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
242#xpack.management.elasticsearch.ssl.truststore.password: password
243#xpack.management.elasticsearch.ssl.keystore.path: /path/to/file
244#xpack.management.elasticsearch.ssl.keystore.password: password
245#xpack.management.elasticsearch.ssl.verification_mode: certificate
246#xpack.management.elasticsearch.sniffing: false
247#xpack.management.logstash.poll_interval: 5s
248

然后是自定义的 logstash 的配置文件 logstash.conf，如下：

1input {
2  tcp {
3    mode => "server"
4    host => "0.0.0.0"
5    port => 4560
6    codec => json_lines
7  }
8}
9output {
10  elasticsearch {
11    hosts => "http://192.168.239.133:9200"
12    index => "log-%{+YYYY.MM.dd}"
13  }
14}
15

上面文件的大概意思就是监听 4560 端口，然后写入 ES，索引名称就是 log 前缀加上日期；每天都会创建一个新的索引。

然后是 docker-compose.yml，如下：

1version: "3"
2services:
3  logstash:
4    container_name: logstash
5    hostname: logstash
6    image: leisurexi/logstash:7.1.0
7    command: logstash -f ./config/logstash.conf
8    volumes:
9      - ./logstash.conf:/usr/share/logstash/config/logstash.conf
10      - ./logstash.yml:/usr/share/logstash/config/logstash.yml
11    environment:
12      - elasticsearch.hosts=http://192.168.239.133:9200
13    ports:
14      - 4560:4560
15      - 5044:5044

最后像上面启动 ES 一样，启动 logstash 即可。

定期删除索引

如果长时间运行，会有磁盘满的而无法写入 ES 的情况，所以得定时删除不怎么重要的索引数据；如下，可以通过定时脚本来实现。

我们先写个删除15天前索引的脚本 es-index-clear.sh，如下：

1# /bin/bash
2# es-index-clear
3# 只保留15天内的日志索引
4LAST_DATA=`date -d "-15 days" "+%Y.%m.%d"`
5# 删除索引
6curl -XDELETE 'http://192.168.239.133:9200/*-'${LAST_DATA}'*'

然后利用 crontab 去添加定时任务，首先执行 crontab -e，然后添加以下内容：

0 1 * * * /opt/elk/es-index-clear.sh

该定时会在每天的凌晨1点执行，后面换成你自己脚本所在的绝对路径即可。

可以执行 tail -f /var/log/cron，查看定时任务的日志。

测试

我们新建一个 spring-boot 应用，添加 logstash 的依赖，如下：

1<dependency>
2    <groupId>net.logstash.logback</groupId>
3    <artifactId>logstash-logback-encoder</artifactId>
4    <version>5.3</version>
5</dependency>

然后新建一个 logback.xml 放在 resources 目录下，内容如下：

1<?xml version="1.0" encoding="UTF-8"?>
2<!DOCTYPE configuration>
3<configuration>
4    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
5    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
6    <!--应用名称-->
7    <property name="APP_NAME" value="log"/>
8
9    <!--输出到logstash的appender-->
10    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
11        <!--可以访问的logstash日志收集端口-->
12        <destination>192.168.239.131:4560</destination>
13        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder"/>
14    </appender>
15
16    <root level="INFO">
17        <appender-ref ref="CONSOLE"/>
18        <appender-ref ref="LOGSTASH"/>
19    </root>
20
21</configuration>

接着编写一个定时任务，Java 代码如下：

1@EnableScheduling
2@Configuration
3public class LogScheduler {
4
5    private static Logger log = LoggerFactory.getLogger(LogScheduler.class);
6
7    @Scheduled(cron = " 0/30 * * * * ? ")
8    public void doTiming() {
9        log.info("ELK测试日志");
10    }
11
12}

该定时任务每30秒输出一条日志。

最后我们查看 kibana 的界面就可以看到啦！

总结

本次只是简单的搭建了 ELK，如果要在生成环境上使用，还需要做很多修改；例如，ES 开启安全认证，端口不可直接暴露在公网上，索引最好使用模板创建等。

最后本篇文章的代码和 ELK 的配置文件，我都上传到 https://github.com/leisurexi/elk。访问新博客地址，观看效果更佳 https://leisurexi.github.io/

注意：Github 上的 docker-compose.yml 我是和在一起写的，文章中是分开写的，为了更清晰一点。

Docker 搭建 ELK 集群步骤

前言

正文

环境