Elasticsearch 7.6.2安装-RPM方式
Elasticsearch
安装准备
服务器准备
为搭建ElasticSearch集群，准备了三台服务器，主机IP分别为：
| 服务器IP | 系统版本 |
|---|---|
| 192.168.1.107 | Centos6.5 |
| 192.168.1.108 | Centos6.5 |
| 192.168.1.109 | Centos6.5 |
调整系统参数
内核参数调整
1 2 31. `vim /etc/sysctl.conf` 42. `# 增加下面的内容` 53. `fs.file-max = 65536` 64. `vm.max_map_count = 262144` 75. `# 执行命令` 86. `sysctl -p` 9 10
资源参数调整
1 2 31. `vim /etc/security/limits.conf` 42. `# 修改` 53. `* soft nofile 65536` 64. `* hard nofile 65536` 75. `* soft nproc 4096` 86. `* hard nproc 4096` 97. `elasticsearch soft memlock unlimited` 108. `elasticsearch hard memlock unlimited` 11 12
调整线程数
1 2 31. `vim /etc/security/limits.d/90-nproc.conf` 42. `找到如下内容：` 53. `* soft nproc 1024` 64. `#修改为` 75. `* soft nproc 4096` 8 9
ES安装及配置
导入Elasticsearch PGP密钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
配置yum
vim /etc/yum.repos.d/elasticsearch.repo
加入以下信息
1[elasticsearch] 2name=Elasticsearch repository for 7.x packages 3baseurl=https://artifacts.elastic.co/packages/7.x/yum 4gpgcheck=1 5gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch 6enabled=1 7autorefresh=1 8type=rpm-md
安装
1# 展示所有版本 2sudo yum --showduplicates list elasticsearch 3# 安装指定版本 4sudo yum install elasticsearch-7.6.2-1
配置开机启动
sudo chkconfig --add elasticsearch
配置
JAVA_HOME
1# 编辑配置文件 2vim /etc/sysconfig/elasticsearch 3 4# 修改JAVA_HOME的值 5JAVA_HOME=/opt/jdk1.8.0_181/
elasticsearch.yml
vim /etc/elasticsearch/elasticsearch.yml
1 2 31. `# 集群名称，各个节点的值必须一致` 42. `cluster.name: elasticsearch_production` 53. `# 节点名称，区分节点，各个节点的值不能一致` 64. `node.name: node-1` 7 86. `# 数据文件路径，需要配置datanode所有数据磁盘路径` 97. `path.data:` 108. `- /data/dfs/dfs00/dfs/elasticsearch` 119. `- /data/dfs/dfs01/dfs/elasticsearch` 1210. `- /data/dfs/dfs02/dfs/elasticsearch` 13 1412. `# 日志文件路径` 1513. `path.logs: /var/log/elasticsearch` 16 1715. `# 设置为true来锁住内存。因为当jvm开始swapping时es的效率会降低，所以要保证它不swap，可以把ES_MIN_MEM和ES_MAX_MEM两个环境变量设置成同一个值，并且保证机器有足够的内存分配给es。同时也要允许elasticsearch的进程可以锁住内存，Linux下可以通过ulimit -l unlimited命令` 1816. `bootstrap.memory_lock: true` 1917. `# 因为Centos6不支持SecComp，而ES6.1.2默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动` 2018. `bootstrap.system_call_filter: false` 2119. `# 本机IP` 2220. `network.host: 192.168.1.107` 23 2422. `# 集群种子节点` 2523. `discovery.seed_hosts:` 2624. `- 192.168.1.107:9300` 2725. `- 192.168.1.108:9300` 2826. `- 192.168.1.109:9300` 2927. `# 集群初始主节点，需要填写node.name中的值` 3028. `cluster.initial_master_nodes:` 3129. `- node-1` 3230. `- node-2` 3331. `- node-3` 34 3533. `# 存在至少2个节点（数据节点或者 master 节点）才进行数据恢复` 3634. `gateway.recover_after_nodes: 2` 3735. `# 等待10分钟，或者3个节点上线后，才进行数据恢复，这取决于哪个条件先达到` 3836. `gateway.expected_nodes: 3` 3937. `gateway.recover_after_time: 10m` 40 4139. `search.max_buckets: 200000` 4240. `action.destructive_requires_name: true` 43 44
jvm.options
1vim /etc/elasticsearch/jvm.options 2# 调整内存大小根据系统资源而定，最好不要超过总资源一半 3-Xms16g 4-Xmx16g
启动
1sudo -i service elasticsearch start 2# 查看节点状态 3curl -XGET '192.168.1.107:9200/_cat/nodes?v' 4# 查看集群状态 5curl -XGET '192.168.1.107:9200/_cat/health?v'
Kibana安装
Kibana安装在1.109上面
导入Elasticsearch PGP密钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
配置yum
vim /etc/yum.repos.d/kibana.repo
加入以下信息
1[kibana] 2name=Kibana repository for 7.x packages 3baseurl=https://artifacts.elastic.co/packages/7.x/yum 4gpgcheck=1 5gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch 6enabled=1 7autorefresh=1 8type=rpm-md
安装
1# 展示所有版本 2sudo yum --showduplicates list kibana --enablerepo=kibana kibana 3# 安装指定版本 4sudo yum install kibana-7.6.2-1 --enablerepo=kibana kibana
配置开机启动
sudo chkconfig --add kibana
修改配置
vim /etc/kibana/kibana.yml
1 2 31. `# 本机IP` 42. `server.host: "192.168.1.109"` 53. `# 访问es的地址` 64. `elasticsearch.hosts: ["http://192.168.1.107:9200","http://192.168.1.108:9200","http://192.168.1.109:9200"]` 7 8
启动
sudo -i service kibana start
Logstash安装
导入Elasticsearch PGP密钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
配置yum
vim /etc/yum.repos.d/logstash.repo
加入以下信息
1[logstash] 2name=logstash repository for 7.x packages 3baseurl=https://artifacts.elastic.co/packages/7.x/yum 4gpgcheck=1 5gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch 6enabled=1 7autorefresh=1 8type=rpm-md
安装
1mv /usr/bin/java /usr/bin/java7 2ln -s $JAVA_HOME/bin/java /usr/bin/java 3 4sudo yum --showduplicates list logstash 5sudo yum install logstash-7.6.2-1
修改配置
vim /etc/logstash/logstash.yml
1 2 31. `# 配置热加载` 42. `config.reload.automatic: true` 5 64. `# 开启持久化队列` 75. `queue.type: persisted` 86. `queue.max_bytes: 8gb` 9 10
管道配置
在/etc/logstash/conf.d下新建 xxx.conf 编写管道处理逻辑
启动
nohup /usr/share/logstash/bin/logstash --path.settings /etc/logstash >/dev/null 2>&1 &
Filebeat安装
导入Elasticsearch PGP密钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
配置yum
vim /etc/yum.repos.d/filebeat.repo
加入以下信息
1[filebeat] 2name=filebeat repository for 7.x packages 3baseurl=https://artifacts.elastic.co/packages/7.x/yum 4gpgcheck=1 5gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch 6enabled=1 7autorefresh=1 8type=rpm-md
安装
1sudo yum install filebeat-7.6.2-1 2sudo chkconfig --add filebeat
修改配置
vim /etc/filebeat/filebeat.yml
1 2 31. `filebeat.inputs:` 4 53. `# Each - is an input. Most options can be set at the input level, so` 64. `# you can use different inputs for various configurations.` 75. `# Below are the input specific configurations.` 8 97. `- type: log` 10 119. `# Change to true to enable this input configuration.` 1210. `enabled: true` 13 1412. `# Paths that should be crawled and fetched. Glob based paths.` 1513. `paths:` 1614. `- /opt/web_app/openservice/elk/elk*.log` 1715. `#- c:\programdata\elasticsearch\logs\*` 1816. `tags: ["log_api"]` 19 2018. `fields:` 2119. `log_topic: log_api` 22 2322. `-------------------------- Kafka output ------------------------------` 2423. `output.kafka:` 2524. `# initial brokers for reading cluster metadata` 2625. `hosts: ["kafka01.bitnei.cn:9092", "kafka02.bitnei.cn:9092", "kafka03.bitnei.cn:9092"]` 27 2827. `# message topic selection + partitioning` 2928. `topic: '%{[fields.log_topic]}'` 3029. `#key: '%{[interfaceName]}'` 31 3231. `required_acks: -1` 3332. `compression: gzip` 3433. `# (bytes) This value should be equal to or less than the broker’s message.max.bytes.` 3534. `max_message_bytes: 10000000` 36 37
启动
service filebeat start