京东物流：康睿姚再毅李振刘斌王北永

说明：以下全部均基于elasticsearch8.1 版本

一.跨集群检索 - ccr

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/modules-cross-cluster-search.html

跨集群检索的背景和意义

跨集群检索定义

跨集群检索环境搭建

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/modules-cross-cluster-search.html

步骤1：搭建两个本地单节点集群，本地练习可取消安全配置

步骤2：每个集群都执行以下命令

PUT _cluster/settings { "persistent": { "cluster": { "remote": { "cluster_one": { "seeds": [ "172.21.0.14:9301" ] },"cluster_two": { "seeds": [ "172.21.0.14:9302" ] } } } } }

步骤3：验证集群之间是否互通

方案1：Kibana 可视化查看：stack Management -> Remote Clusters -> status 应该是 connected！且必须打上绿色的对号。

方案2：GET _remote/info

跨集群查询演练

1# 步骤1 在集群 1 中添加数据如下
2PUT test01/_bulk
3{"index":{"_id":1}}
4{"title":"this is from cluster01..."}
5
6# 步骤2 在集群 2 中添加数据如下：
7PUT test01/_bulk
8{"index":{"_id":1}}
9{"title":"this is from cluster02..."}
10
11# 步骤 3：执行跨集群检索如下: 语法：POST 集群名称1:索引名称,集群名称2:索引名称/_search
12POST cluster_one:test01,cluster_two:test01/_search
13{
14  "took" : 7,
15  "timed_out" : false,
16  "num_reduce_phases" : 3,
17  "_shards" : {
18    "total" : 2,
19    "successful" : 2,
20    "skipped" : 0,
21    "failed" : 0
22  },
23  "_clusters" : {
24    "total" : 2,
25    "successful" : 2,
26    "skipped" : 0
27  },
28  "hits" : {
29    "total" : {
30      "value" : 2,
31      "relation" : "eq"
32    },
33    "max_score" : 1.0,
34    "hits" : [
35      {
36        "_index" : "cluster_two:test01",
37        "_id" : "1",
38        "_score" : 1.0,
39        "_source" : {
40          "title" : "this is from cluster02..."
41        }
42      },
43      {
44        "_index" : "cluster_one:test01",
45        "_id" : "1",
46        "_score" : 1.0,
47        "_source" : {
48          "title" : "this is from cluster01..."
49        }
50      }
51    ]
52  }
53}
54
55

二.跨集群复制 - ccs - 该功能需付费

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-ccr.html

如何保障集群的高可用

副本机制
快照和恢复
跨集群复制（类似mysql 主从同步）

跨集群复制概述

跨集群复制配置

准备两个集群，网络互通
开启 license 使用，可试用30天

开启位置：Stack Management -> License mangement.

3.定义好谁是Leads集群，谁是follower集群

4.在follower集群配置Leader集群

5.在follower集群配置Leader集群的索引同步规则（kibana页面配置）

a.stack Management -> Cross Cluster Replication -> create a follower index.

6.启用步骤5的配置

三索引模板

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/index-templates.html

8.X之组件模板

1.创建组件模板-索引setting相关

1# 组件模板 - 索引setting相关
2PUT _component_template/template_sttting_part
3{
4  "template": {
5    "settings": {
6      "number_of_shards": 3,
7      "number_of_replicas": 0
8    }
9  }
10}
11

2.创建组件模板-索引mapping相关

1# 组件模板 - 索引mapping相关
2PUT _component_template/template_mapping_part
3{
4  "template": {
5    "mappings": {
6      "properties": {
7        "hosr_name":{
8          "type": "keyword"
9        },
10        "cratet_at":{
11          "type": "date",
12          "format": "EEE MMM dd HH:mm:ss Z yyyy"
13        }
14      }
15    }
16  }
17}
18

3.创建组件模板-配置模板和索引之间的关联

1// **注意：composed_of 如果多个组件模板中的配置项有重复，后面的会覆盖前面的，和配置的顺序有关**
2# 基于组件模板，配置模板和索引之间的关联
3# 也就是所有 tem_* 该表达式相关的索引创建时，都会使用到以下规则
4PUT _index_template/template_1
5{
6  "index_patterns": [
7    "tem_*"
8  ],
9  "composed_of": [
10    "template_sttting_part",
11    "template_mapping_part"
12  ]
13}
14

4.测试

1# 创建测试
2PUT tem_001
3

索引模板基本操作

实战演练

需求1：默认如果不显式指定Mapping,数值类型会被动态映射为long类型，但实际上业务数值都比较小，会存在存储浪费。需要将默认值指定为Integer

索引模板，官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/index-templates.html

mapping-动态模板,官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/dynamic-templates.html

1# 结合mapping 动态模板 和 索引模板
2# 1.创建组件模板之 - mapping模板
3PUT _component_template/template_mapping_part_01
4{
5  "template": {
6    "mappings": {
7      "dynamic_templates": [
8        {
9          "integers": {
10            "match_mapping_type": "long",
11            "mapping": {
12              "type": "integer"
13            }
14          }
15        }
16      ]
17    }
18  }
19}
20
21# 2. 创建组件模板与索引关联配置
22PUT _index_template/template_2
23{
24  "index_patterns": ["tem1_*"],
25  "composed_of": ["template_mapping_part_01"]
26}
27
28# 3.创建测试数据
29POST tem1_001/_doc/1
30{
31  "age":18
32}
33
34# 4.查看mapping结构验证
35get tem1_001/_mapping
36
37

需求2：date_*开头的字段，统一匹配为date日期类型。

索引模板，官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/index-templates.html

mapping-动态模板,官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/dynamic-templates.html

1# 结合mapping 动态模板 和 索引模板
2# 1.创建组件模板之 - mapping模板
3PUT _component_template/template_mapping_part_01
4{
5  "template": {
6    "mappings": {
7      "dynamic_templates": [
8        {
9          "integers": {
10            "match_mapping_type": "long",
11            "mapping": {
12              "type": "integer"
13            }
14          }
15        },
16        {
17        "date_type_process": {
18          "match": "date_*",
19          "mapping": {
20            "type": "date",
21            "format":"yyyy-MM-dd HH:mm:ss"
22          }
23        }
24      }
25      ]
26    }
27  }
28}
29
30# 2. 创建组件模板与索引关联配置
31PUT _index_template/template_2
32{
33  "index_patterns": ["tem1_*"],
34  "composed_of": ["template_mapping_part_01"]
35}
36
37
38# 3.创建测试数据
39POST tem1_001/_doc/2
40{
41  "age":19,
42  "date_aoe":"2022-01-01 18:18:00"
43}
44
45# 4.查看mapping结构验证
46get tem1_001/_mapping
47

四.LIM 索引生命周期管理

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/index-lifecycle-management.html

什么是索引生命周期

索引的生-> 老 -> 病 -> 死

是否有过考虑，如果一个索引，创建之后，就不再去管理了？会发生什么？

什么是索引生命周期管理

索引太大了会如何?

大索引的恢复时间，要远比小索引恢复慢的多的多索引大了以后，检索会很慢，写入和更新也会受到不同程度的影响索引大到一定程度，当索引出现健康问题，会导致整个集群核心业务不可用

最佳实践

集群的单个分片最大文档数上限：2的32次幂减1，即20亿左右官方建议：分片大小控制在30GB-50GB，若索引数据量无限增大，肯定会超过这个值

用户不关注全量

某些业务场景，业务更关注近期的数据，如近3天、近7天大索引会将全部历史数据汇集在一起，不利于这种场景的查询

索引生命周期管理的历史演变

LIM前奏 - rollover 滚动索引

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/index-rollover.html

1# 0.自测前提，lim生命周期rollover频率。默认10分钟
2PUT _cluster/settings
3{
4  "persistent": {
5    "indices.lifecycle.poll_interval": "1s"
6  }
7}
8
9# 1. 创建索引，并指定别名
10PUT test_index-0001
11{
12  "aliases": {
13    "my-test-index-alias": {
14      "is_write_index": true
15    }
16  }
17}
18
19# 2.批量导入数据
20PUT my-test-index-alias/_bulk
21{"index":{"_id":1}}
22{"title":"testing 01"}
23{"index":{"_id":2}}
24{"title":"testing 02"}
25{"index":{"_id":3}}
26{"title":"testing 03"}
27{"index":{"_id":4}}
28{"title":"testing 04"}
29{"index":{"_id":5}}
30{"title":"testing 05"}
31
32# 3.rollover 滚动规则配置
33POST my-test-index-alias/_rollover
34{
35  "conditions": {
36    "max_age": "7d",
37    "max_docs": 5,
38    "max_primary_shard_size": "50gb"
39  }
40}
41
42# 4.在满足条件的前提下创建滚动索引
43PUT my-test-index-alias/_bulk
44{"index":{"_id":7}}
45{"title":"testing 07"}
46
47# 5.查询验证滚动是否成功
48POST my-test-index-alias/_search
49

LIM前奏 - shrink 索引压缩

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/ilm-shrink.html

核心步骤：

1. 将数据全部迁移至一个独立的节点

2. 索引禁止写入

3. 方可进行压缩

1# 1.准备测试数据
2DELETE kibana_sample_data_logs_ext
3PUT kibana_sample_data_logs_ext
4{
5  "settings": {
6    "number_of_shards": 5,
7    "number_of_replicas": 0
8  }
9}
10POST _reindex
11{
12  "source": {
13    "index": "kibana_sample_data_logs"
14  },
15  "dest": {
16    "index": "kibana_sample_data_logs_ext"
17  }
18}
19
20
21# 2.压缩前必要的条件设置
22# number_of_replicas :压缩后副本为0
23# index.routing.allocation.include._tier_preference 数据分片全部路由到hot节点
24# "index.blocks.write 压缩后索引不再允许数据写入
25PUT kibana_sample_data_logs_ext/_settings
26{
27  "settings": {
28    "index.number_of_replicas": 0,
29    "index.routing.allocation.include._tier_preference": "data_hot",
30    "index.blocks.write": true
31  }
32}
33
34# 3.实施压缩
35POST kibana_sample_data_logs_ext/_shrink/kibana_sample_data_logs_ext_shrink
36{
37  "settings":{
38    "index.number_of_replicas": 0,
39    "index.number_of_shards": 1,
40    "index.codec":"best_compression"
41  },
42  "aliases":{
43    "kibana_sample_data_logs_alias":{}
44  }
45}
46

LIM实战

全局认知建立 - 四大阶段

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/overview-index-lifecycle-management.html

生命周期管理阶段（Policy）：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/ilm-index-lifecycle.html

Hot阶段(生)

Set priority

Unfollow

Rollover

Read-only

Shrink

Force Merge

Search snapshot

Warm阶段（老）

Set priority

Unfollow

Read-only

Allocate

migrate

Shirink

Force Merge

Cold阶段（病）

Search snapshot

Delete阶段（死）

delete

演练

1.创建policy

Hot阶段设置，rollover: max_age:3d，max_docs:5, max_size:50gb, 优先级：100
Warm阶段设置：min_age:15s , forcemerage段合并，热节点迁移到warm节点，副本数设置0，优先级：50
Cold阶段设置: min_age 30s, warm迁移到cold阶段
Delete阶段设置：min_age 45s，执行删除操作

1PUT _ilm/policy/kr_20221114_policy
2{
3  "policy": {
4    "phases": {
5      "hot": {
6        "min_age": "0ms",
7        "actions": {
8          "set_priority": {
9            "priority": 100
10          },
11          "rollover": {
12            "max_size": "50gb",
13            "max_primary_shard_size": "50gb",
14            "max_age": "3d",
15            "max_docs": 5
16          }
17        }
18      },
19      "warm": {
20        "min_age": "15s",
21        "actions": {
22          "forcemerge": {
23            "max_num_segments": 1
24          },
25          "set_priority": {
26            "priority": 50
27          },
28          "allocate": {
29            "number_of_replicas": 0
30          }
31        }
32      },
33      "cold": {
34        "min_age": "30s",
35        "actions": {
36          "set_priority": {
37            "priority": 0
38          }
39        }
40      },
41      "delete": {
42        "min_age": "45s",
43        "actions": {
44          "delete": {
45            "delete_searchable_snapshot": true
46          }
47        }
48      }
49    }
50  }
51}
52

2.创建index template

1PUT _index_template/kr_20221114_template
2{
3  "index_patterns": ["kr_index-**"],
4  "template": {
5    "settings": {
6      "index": {
7        "lifecycle": {
8          "name": "kr_20221114_policy",
9          "rollover_alias": "kr-index-alias"
10        },
11        "routing": {
12          "allocation": {
13            "include": {
14              "_tier_preference": "data-hot"
15            }
16          }
17        },
18        "number_of_shards": "3",
19        "number_of_replicas": "1"
20      }
21    },
22    "aliases": {},
23    "mappings": {}
24  }
25}
26
27

3.测试需要修改lim rollover刷新频率

1PUT _cluster/settings
2{
3  "persistent": {
4    "indices.lifecycle.poll_interval": "1s"
5  }
6}
7

4.进行测试

1# 创建索引，并制定可写别名
2PUT kr_index-0001
3{
4  "aliases": {
5    "kr-index-alias": {
6      "is_write_index": true
7    }
8  }
9}
10# 通过别名新增数据
11PUT kr-index-alias/_bulk
12{"index":{"_id":1}}
13{"title":"testing 01"}
14{"index":{"_id":2}}
15{"title":"testing 02"}
16{"index":{"_id":3}}
17{"title":"testing 03"}
18{"index":{"_id":4}}
19{"title":"testing 04"}
20{"index":{"_id":5}}
21{"title":"testing 05"}
22# 通过别名新增数据，触发rollover
23PUT kr-index-alias/_bulk
24{"index":{"_id":6}}
25{"title":"testing 06"}
26# 查看索引情况
27GET kr_index-0001
28
29get _cat/indices?v
30

过程总结

第一步：配置 lim pollicy

横向：Phrase 阶段(Hot、Warm、Cold、Delete) 生老病死
纵向：Action 操作（rollover、forcemerge、readlyonly、delete）

第二步：创建模板绑定policy,指定别名

第三步：创建起始索引

第四步：索引基于第一步指定的policy进行滚动

五.Data Stream

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/ilm-actions.html

特性解析

Data Stream让我们跨多个索引存储时序数据，同时给了唯一的对外接口（data stream名称）

写入和检索请求发给data stream
data stream将这些请求路由至 backing index（后台索引）

Backing indices

每个data stream由多个隐藏的后台索引构成

自动创建
要求模板索引

rollover 滚动索引机制用于自动生成后台索引

将成为data stream 新的写入索引

应用场景

日志、事件、指标等其他持续创建（少更新）的业务数据
两大核心特点
时序性数据
数据极少更新或没有更新

创建Data Stream 核心步骤

官网文档地址：
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/set-up-a-data-stream.html

Set up a data stream

To set up a data stream, follow these steps:

Create an index lifecycle policy
Create component templates
Create an index template
Create the data stream
Secure the data stream

演练

1. 创建一个data stream，名称为my-data-stream

2. index_template 名称为 my-index-template

3. 满足index格式【"my-data-stream*"】的索引都要被应用到

4. 数据插入的时候，在data_hot节点

5. 过3分钟之后要rollover到data_warm节点

6. 再过5分钟要到data_cold节点

1# 步骤1 。创建 lim policy
2PUT _ilm/policy/my-lifecycle-policy
3{
4  "policy": {
5    "phases": {
6      "hot": {
7        "actions": {
8          "rollover": {
9            "max_size": "50gb",
10            "max_age": "3m",
11            "max_docs": 5
12          },
13          "set_priority": {
14            "priority": 100
15          }
16        }
17      },
18      "warm": {
19        "min_age": "5m",
20        "actions": {
21          "allocate": {
22            "number_of_replicas": 0
23          }, 
24          "forcemerge": {
25            "max_num_segments": 1
26          },
27          "set_priority": {
28            "priority": 50
29          }
30        }
31      },
32      "cold": {
33        "min_age": "6m",
34        "actions": {
35          "freeze":{}
36        }
37      },
38      "delete": {
39        "min_age": "45s",
40        "actions": {
41          "delete": {}
42        }
43      }
44    }
45  }
46}
47
48# 步骤2 创建组件模板 - mapping
49PUT _component_template/my-mappings
50{
51  "template": {
52    "mappings": {
53      "properties": {
54        "@timestamp": {
55          "type": "date",
56          "format": "date_optional_time||epoch_millis"
57        },
58        "message": {
59          "type": "wildcard"
60        }
61      }
62    }
63  },
64  "_meta": {
65    "description": "Mappings for @timestamp and message fields",
66    "my-custom-meta-field": "More arbitrary metadata"
67  }
68}
69
70# 步骤3 创建组件模板 - setting
71PUT _component_template/my-settings
72{
73  "template": {
74    "settings": {
75      "index.lifecycle.name": "my-lifecycle-policy",
76      "index.routing.allocation.include._tier_preference":"data_hot"
77    }
78  },
79  "_meta": {
80    "description": "Settings for ILM",
81    "my-custom-meta-field": "More arbitrary metadata"
82  }
83}
84
85# 步骤4 创建索引模板
86PUT _index_template/my-index-template
87{
88  "index_patterns": ["my-data-stream*"],
89  "data_stream": { },
90  "composed_of": [ "my-mappings", "my-settings" ],
91  "priority": 500,
92  "_meta": {
93    "description": "Template for my time series data",
94    "my-custom-meta-field": "More arbitrary metadata"
95  }
96}
97
98# 步骤5 创建 data stream  并 写入数据测试
99PUT my-data-stream/_bulk
100{ "create":{ } }
101{ "@timestamp": "2099-05-06T16:21:15.000Z", "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736" }
102{ "create":{ } }
103{ "@timestamp": "2099-05-06T16:25:42.000Z", "message": "192.0.2.255 - - [06/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" }
104
105POST my-data-stream/_doc
106{
107  "@timestamp": "2099-05-06T16:21:15.000Z",
108  "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
109}
110
111
112# 步骤6 查看data stream 后台索引信息
113GET /_resolve/index/my-data-stream*

ElasticSearch必知必会-进阶篇