Elasticsearch学习笔记——分词

1.测试Elasticsearch的分词

Elasticsearch有多种分词器（参考:https://www.jianshu.com/p/d57935ba514b）

Set the shape to semi-transparent by calling set_trans(5)

（1）standard analyzer：标准分词器（默认是这种）
set,the,shape,to,semi,transparent by,calling,set_trans,5

（2）simple analyzer：简单分词器
set, the, shape, to, semi, transparent, by, calling, set, trans

（3）whitespace analyzer：空白分词器。大小写，下划线等都不会转换
Set, the, shape, to, semi-transparent, by, calling, set_trans(5)

（4）language analyzer：（特定语言分词器，比如说English英语分瓷器）
set, shape, semi, transpar, call, set_tran, 5

2.为Elasticsearch的index设置分词

这样就将这个index里面的所有type的分词设置成了simple

1PUT my_index
2{
3"settings": {
4    "analysis": {
5      "analyzer": {"default":{"type":"simple"}}
6    }
7  }
8}

标准分词器 : standard analyzer

http://localhost:9200/_analyze?analyzer=standard&pretty=true&text=test测试

分词结果

1{
2  "tokens" : [
3    {
4      "token" : "test",
5      "start_offset" : 0,
6      "end_offset" : 4,
7      "type" : "<ALPHANUM>",
8      "position" : 0
9    },
10    {
11      "token" : "测",
12      "start_offset" : 4,
13      "end_offset" : 5,
14      "type" : "<IDEOGRAPHIC>",
15      "position" : 1
16    },
17    {
18      "token" : "试",
19      "start_offset" : 5,
20      "end_offset" : 6,
21      "type" : "<IDEOGRAPHIC>",
22      "position" : 2
23    }
24  ]
25}

简单分词器 : simple analyzer

http://localhost:9200/_analyze?analyzer=simple&pretty=true&text=test_测试

 结果

1{
2  "tokens" : [
3    {
4      "token" : "test",
5      "start_offset" : 0,
6      "end_offset" : 4,
7      "type" : "word",
8      "position" : 0
9    },
10    {
11      "token" : "测试",
12      "start_offset" : 5,
13      "end_offset" : 7,
14      "type" : "word",
15      "position" : 1
16    }
17  ]
18}

**IK分词器 : ik_max_word **analyzer** 和 `ik_smart **analyzer**`
**

首先需要安装

https://github.com/medcl/elasticsearch-analysis-ik

下zip包,然后使用install plugin进行安装,我机器上的es版本是5.6.10,所以安装的就是5.6.10

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.6.10/elasticsearch-analysis-ik-5.6.10.zip

然后重新启动Elasticsearch就可以了

进行测试

http://localhost:9200/_analyze?analyzer=ik_max_word&pretty=true&text=test_tes_te测试

结果

1{
2  "tokens" : [
3    {
4      "token" : "test_tes_te",
5      "start_offset" : 0,
6      "end_offset" : 11,
7      "type" : "LETTER",
8      "position" : 0
9    },
10    {
11      "token" : "test",
12      "start_offset" : 0,
13      "end_offset" : 4,
14      "type" : "ENGLISH",
15      "position" : 1
16    },
17    {
18      "token" : "tes",
19      "start_offset" : 5,
20      "end_offset" : 8,
21      "type" : "ENGLISH",
22      "position" : 2
23    },
24    {
25      "token" : "te",
26      "start_offset" : 9,
27      "end_offset" : 11,
28      "type" : "ENGLISH",
29      "position" : 3
30    },
31    {
32      "token" : "测试",
33      "start_offset" : 11,
34      "end_offset" : 13,
35      "type" : "CN_WORD",
36      "position" : 4
37    }
38  ]
39}