一、分析
1. 分析(analysis)
- 首先，标记化一个文本块为适用于倒排索引单独的词(term)
- 然后标准化这些词为标准形式，提高它们的“可搜索性”或“查全率” 分析是由分析器(analyzer)完成的。
2. 分析器(analyzer)
- 字符过滤器(character filter) 过滤处理字符串（比如去掉多余的空格之类的），让字符串在被分词前变得更加“整洁”，一个分析器可能包含零到多个字符过滤器。
- 分词器(tokenizer) 字符串被标记化成独立的词（比如按空格划分成一个个单词），一个分析器必须包含一个分词器。
- 标记过滤器(token filters) 所有的词经过标记过滤，标记过滤器可能修改，添加或删除标记。
只有字段是全文字段(full-text fields)的时候分析器才会被使用,当字段是一个确切的值(exact value)时，不会对该字段做分析。
- 全文字段：类似于string、text
- 确切值：类似于数值、日期
二、自定义分析器
1. char_filter(字符过滤器)
html_strip(html标签过滤) 参数：escaped_tags不应该从原始文本中删除的HTML标签数组
mapping(自定义映射过滤) 参数：mappings一个映射数组，每个元素的格式为key => valuemappings_path一个以UTF-8编码的文件的绝对路径或者是相对于config目录的路径，文件每一行都是一个格式为key => value映射
pattern_replace(使用正则表达式来匹配字符并使用指定的字符串替换) 参数：pattern一个java正则表达式mappings_path作替换的字符串flagsjava正则表达式标志
2. tokenizer(分词器)
这里只列出常用的几个，更多分词器请查阅官方文档
standard(标准分词，默认使用的分词。根据Unicode Consortium的定义的单词边界来切分文本，然后去掉大部分标点符号对于文本分析，所以对于任何语言都是最佳选择) 参数：max_token_length最大标记长度。如果一个标记超过这个长度，就会被分割。默认值为255
letter(遇到不是字母的字符就分割) 参数：无lowercase(在letter基础上把所分词都转为小写) 参数：无whitespace(以空格分词) 参数：无keyword(相当于不分词，接收啥输出啥) 参数：buffer_size缓冲区大小。默认为256。缓冲区将以这种大小增长，直到所有文本被消耗。建议不要改变这个设置。
3. filter(标记过滤器)
由于标记过滤器太多，这里就不一一介绍了，请查阅官方文档
4. 自定义分析器
newindex PUT
1{ 2 "settings": { 3 "analysis": { 4 "char_filter": { 5 "my_char_filter": { 6 "type": "mapping", 7 "mappings": [ 8 "&=>and", 9 ":)=>happy", 10 ":(=>sad" 11 ] 12 } 13 }, 14 "tokenizer": { 15 "my_tokenizer": { 16 "type": "standard", 17 "max_token_length": 5 18 } 19 }, 20 "filter": { 21 "my_filter": { 22 "type": "stop", 23 "stopwords": [ 24 "the", 25 "a" 26 ] 27 } 28 }, 29 "analyzer": { 30 "my_analyzer": { 31 "type": "custom", 32 "char_filter": [ 33 "html_strip", 34 "my_char_filter" 35 ], 36 "tokenizer": "my_tokenizer", 37 "filter": [ 38 "lowercase", 39 "my_filter" 40 ] 41 } 42 } 43 } 44 } 45}
然后用自定义分析器分析一段字符串：
newindex/_analyze POST
1{ 2 "analyzer": "my_analyzer", 3 "text": "<span>If you are :(, I will be :).</span> The people & a banana", 4 "explain": true 5}
可以看到分析过程：
1{ 2 "detail": { 3 "custom_analyzer": true, 4 "charfilters": [ 5 { 6 "name": "html_strip", 7 "filtered_text": [ 8 "if you are :(, I will be :). the people & a banana" 9 ] 10 }, 11 { 12 "name": "my_char_filter", 13 "filtered_text": [ 14 "if you are sad, I will be happy. the people and a banana" 15 ] 16 } 17 ], 18 "tokenizer": { 19 "name": "my_tokenizer", 20 "tokens": [ 21 { 22 "token": "if", 23 "start_offset": 6, 24 "end_offset": 8, 25 "type": "<ALPHANUM>", 26 "position": 0, 27 "bytes": "[69 66]", 28 "positionLength": 1 29 }, 30 { 31 "token": "you", 32 "start_offset": 9, 33 "end_offset": 12, 34 "type": "<ALPHANUM>", 35 "position": 1, 36 "bytes": "[79 6f 75]", 37 "positionLength": 1 38 }, 39 { 40 "token": "are", 41 "start_offset": 13, 42 "end_offset": 16, 43 "type": "<ALPHANUM>", 44 "position": 2, 45 "bytes": "[61 72 65]", 46 "positionLength": 1 47 }, 48 { 49 "token": "sad", 50 "start_offset": 17, 51 "end_offset": 19, 52 "type": "<ALPHANUM>", 53 "position": 3, 54 "bytes": "[73 61 64]", 55 "positionLength": 1 56 }, 57 { 58 "token": "I", 59 "start_offset": 21, 60 "end_offset": 22, 61 "type": "<ALPHANUM>", 62 "position": 4, 63 "bytes": "[49]", 64 "positionLength": 1 65 }, 66 { 67 "token": "will", 68 "start_offset": 23, 69 "end_offset": 27, 70 "type": "<ALPHANUM>", 71 "position": 5, 72 "bytes": "[77 69 6c 6c]", 73 "positionLength": 1 74 }, 75 { 76 "token": "be", 77 "start_offset": 28, 78 "end_offset": 30, 79 "type": "<ALPHANUM>", 80 "position": 6, 81 "bytes": "[62 65]", 82 "positionLength": 1 83 }, 84 { 85 "token": "happy", 86 "start_offset": 31, 87 "end_offset": 33, 88 "type": "<ALPHANUM>", 89 "position": 7, 90 "bytes": "[68 61 70 70 79]", 91 "positionLength": 1 92 }, 93 { 94 "token": "the", 95 "start_offset": 42, 96 "end_offset": 45, 97 "type": "<ALPHANUM>", 98 "position": 8, 99 "bytes": "[74 68 65]", 100 "positionLength": 1 101 }, 102 { 103 "token": "peopl", 104 "start_offset": 46, 105 "end_offset": 51, 106 "type": "<ALPHANUM>", 107 "position": 9, 108 "bytes": "[70 65 6f 70 6c]", 109 "positionLength": 1 110 }, 111 { 112 "token": "e", 113 "start_offset": 51, 114 "end_offset": 52, 115 "type": "<ALPHANUM>", 116 "position": 10, 117 "bytes": "[65]", 118 "positionLength": 1 119 }, 120 { 121 "token": "and", 122 "start_offset": 53, 123 "end_offset": 54, 124 "type": "<ALPHANUM>", 125 "position": 11, 126 "bytes": "[61 6e 64]", 127 "positionLength": 1 128 }, 129 { 130 "token": "a", 131 "start_offset": 55, 132 "end_offset": 56, 133 "type": "<ALPHANUM>", 134 "position": 12, 135 "bytes": "[61]", 136 "positionLength": 1 137 }, 138 { 139 "token": "banan", 140 "start_offset": 57, 141 "end_offset": 62, 142 "type": "<ALPHANUM>", 143 "position": 13, 144 "bytes": "[62 61 6e 61 6e]", 145 "positionLength": 1 146 }, 147 { 148 "token": "a", 149 "start_offset": 62, 150 "end_offset": 63, 151 "type": "<ALPHANUM>", 152 "position": 14, 153 "bytes": "[61]", 154 "positionLength": 1 155 } 156 ] 157 }, 158 "tokenfilters": [ 159 { 160 "name": "lowercase", 161 "tokens": [ 162 { 163 "token": "if", 164 "start_offset": 6, 165 "end_offset": 8, 166 "type": "<ALPHANUM>", 167 "position": 0, 168 "bytes": "[69 66]", 169 "positionLength": 1 170 }, 171 { 172 "token": "you", 173 "start_offset": 9, 174 "end_offset": 12, 175 "type": "<ALPHANUM>", 176 "position": 1, 177 "bytes": "[79 6f 75]", 178 "positionLength": 1 179 }, 180 { 181 "token": "are", 182 "start_offset": 13, 183 "end_offset": 16, 184 "type": "<ALPHANUM>", 185 "position": 2, 186 "bytes": "[61 72 65]", 187 "positionLength": 1 188 }, 189 { 190 "token": "sad", 191 "start_offset": 17, 192 "end_offset": 19, 193 "type": "<ALPHANUM>", 194 "position": 3, 195 "bytes": "[73 61 64]", 196 "positionLength": 1 197 }, 198 { 199 "token": "i", 200 "start_offset": 21, 201 "end_offset": 22, 202 "type": "<ALPHANUM>", 203 "position": 4, 204 "bytes": "[69]", 205 "positionLength": 1 206 }, 207 { 208 "token": "will", 209 "start_offset": 23, 210 "end_offset": 27, 211 "type": "<ALPHANUM>", 212 "position": 5, 213 "bytes": "[77 69 6c 6c]", 214 "positionLength": 1 215 }, 216 { 217 "token": "be", 218 "start_offset": 28, 219 "end_offset": 30, 220 "type": "<ALPHANUM>", 221 "position": 6, 222 "bytes": "[62 65]", 223 "positionLength": 1 224 }, 225 { 226 "token": "happy", 227 "start_offset": 31, 228 "end_offset": 33, 229 "type": "<ALPHANUM>", 230 "position": 7, 231 "bytes": "[68 61 70 70 79]", 232 "positionLength": 1 233 }, 234 { 235 "token": "the", 236 "start_offset": 42, 237 "end_offset": 45, 238 "type": "<ALPHANUM>", 239 "position": 8, 240 "bytes": "[74 68 65]", 241 "positionLength": 1 242 }, 243 { 244 "token": "peopl", 245 "start_offset": 46, 246 "end_offset": 51, 247 "type": "<ALPHANUM>", 248 "position": 9, 249 "bytes": "[70 65 6f 70 6c]", 250 "positionLength": 1 251 }, 252 { 253 "token": "e", 254 "start_offset": 51, 255 "end_offset": 52, 256 "type": "<ALPHANUM>", 257 "position": 10, 258 "bytes": "[65]", 259 "positionLength": 1 260 }, 261 { 262 "token": "and", 263 "start_offset": 53, 264 "end_offset": 54, 265 "type": "<ALPHANUM>", 266 "position": 11, 267 "bytes": "[61 6e 64]", 268 "positionLength": 1 269 }, 270 { 271 "token": "a", 272 "start_offset": 55, 273 "end_offset": 56, 274 "type": "<ALPHANUM>", 275 "position": 12, 276 "bytes": "[61]", 277 "positionLength": 1 278 }, 279 { 280 "token": "banan", 281 "start_offset": 57, 282 "end_offset": 62, 283 "type": "<ALPHANUM>", 284 "position": 13, 285 "bytes": "[62 61 6e 61 6e]", 286 "positionLength": 1 287 }, 288 { 289 "token": "a", 290 "start_offset": 62, 291 "end_offset": 63, 292 "type": "<ALPHANUM>", 293 "position": 14, 294 "bytes": "[61]", 295 "positionLength": 1 296 } 297 ] 298 }, 299 { 300 "name": "my_filter", 301 "tokens": [ 302 { 303 "token": "if", 304 "start_offset": 6, 305 "end_offset": 8, 306 "type": "<ALPHANUM>", 307 "position": 0, 308 "bytes": "[69 66]", 309 "positionLength": 1 310 }, 311 { 312 "token": "you", 313 "start_offset": 9, 314 "end_offset": 12, 315 "type": "<ALPHANUM>", 316 "position": 1, 317 "bytes": "[79 6f 75]", 318 "positionLength": 1 319 }, 320 { 321 "token": "are", 322 "start_offset": 13, 323 "end_offset": 16, 324 "type": "<ALPHANUM>", 325 "position": 2, 326 "bytes": "[61 72 65]", 327 "positionLength": 1 328 }, 329 { 330 "token": "sad", 331 "start_offset": 17, 332 "end_offset": 19, 333 "type": "<ALPHANUM>", 334 "position": 3, 335 "bytes": "[73 61 64]", 336 "positionLength": 1 337 }, 338 { 339 "token": "i", 340 "start_offset": 21, 341 "end_offset": 22, 342 "type": "<ALPHANUM>", 343 "position": 4, 344 "bytes": "[69]", 345 "positionLength": 1 346 }, 347 { 348 "token": "will", 349 "start_offset": 23, 350 "end_offset": 27, 351 "type": "<ALPHANUM>", 352 "position": 5, 353 "bytes": "[77 69 6c 6c]", 354 "positionLength": 1 355 }, 356 { 357 "token": "be", 358 "start_offset": 28, 359 "end_offset": 30, 360 "type": "<ALPHANUM>", 361 "position": 6, 362 "bytes": "[62 65]", 363 "positionLength": 1 364 }, 365 { 366 "token": "happy", 367 "start_offset": 31, 368 "end_offset": 33, 369 "type": "<ALPHANUM>", 370 "position": 7, 371 "bytes": "[68 61 70 70 79]", 372 "positionLength": 1 373 }, 374 { 375 "token": "peopl", 376 "start_offset": 46, 377 "end_offset": 51, 378 "type": "<ALPHANUM>", 379 "position": 9, 380 "bytes": "[70 65 6f 70 6c]", 381 "positionLength": 1 382 }, 383 { 384 "token": "e", 385 "start_offset": 51, 386 "end_offset": 52, 387 "type": "<ALPHANUM>", 388 "position": 10, 389 "bytes": "[65]", 390 "positionLength": 1 391 }, 392 { 393 "token": "and", 394 "start_offset": 53, 395 "end_offset": 54, 396 "type": "<ALPHANUM>", 397 "position": 11, 398 "bytes": "[61 6e 64]", 399 "positionLength": 1 400 }, 401 { 402 "token": "banan", 403 "start_offset": 57, 404 "end_offset": 62, 405 "type": "<ALPHANUM>", 406 "position": 13, 407 "bytes": "[62 61 6e 61 6e]", 408 "positionLength": 1 409 } 410 ] 411 } 412 ] 413 } 414}