Elastic 搜索开发实战
需求描述：
- 智能补全提示
- 结果的高亮显示
- 结果的聚合统计和过滤
- 相关搜索结果的推荐
- 短语纠错(fuzziness)
- 数据的实时同步与更新(logstash)
- 查得到
- 查得全
- 查得准
- 支持同义词
- 支持简繁体
- 支持拼音(pinyin)
- 支持 PPT 的搜索
- 支持自定义排序，按时间，按相关度
- 支持结果过滤，按分类、按标签、按时间范围等
- 搜索结果展示待加强，UI 设计
版本：
- Elasticsearch v6.2.4
- Kibana v6.2.4
- Logstash v6.2.4
- mysql-connector-java-8.0.20.tar.gz
kibana dev tools快捷键
command + i 自动缩进 command + enter 提交请求
常用 API 介绍
1、增删改查
# 创建文档 POST twitter/doc/1 { "name": "Jack", "age": 30 } # 取回文档 GET twitter/doc/1 # 完全替换 PUT twitter/doc/1 { "name": "Jack", "age": 35 } # 部分更新 POST twitter/doc/1/_update { "doc": { "name": "Mark" } } # 删除文档 DELETE twitter/doc/1
2、搜索的使用
# 创建两个索引文档 POST twitter/doc/1 { "name": "Jack", "age": 30 } POST twitter/doc/2 { "name": "Mark", "age": 35 } # 通过名称搜索 GET twitter/_search?q=Jack # 通过年龄检索 GET twitter/_search?q=35 # 限定查询年龄字段 GET twitter/_search?q=age:35 # QueryDSL查询表达式 POST twitter/_search { "query": { "match": { "age": 35 } } }
3、聚合的使用
# 再创建两个索引文档 POST twitter/doc/3 { "name":"john", "age":30 } POST twitter/doc/4 { "name":"mark", "age":40 } # 统计年龄的分布情况 POST twitter/_search { "size": 0, "aggs": { "age_stats": { "terms": { "field": "age", "size": 10 } } } }
聚合查询表达式说明
- 最外层的 size 为 0 不返回搜索命中的文档，只返回聚合的统计结果，
- aggs 就是我们描述聚合查询语句根节点，
- 使用 terms 聚合类型来对 age 字段的值进行统计，并且只返回前 10 个统计值，
- 然后这些统计结果我们命名为 age_stats
4、索引的管理
# 查看索引列表 GET _cat/indices?v # 删除索引 DELETE twitter # 查看集群的健康状况 GET _cluster/health
| 字段名 | 说明 |
|---|---|
| health | 索引健康状态，green 表示健康；yellow 表示数据完整，但是缺少副本；red 则表示有数据损坏。 |
| status | 索引工作状态，open 表示索引打开中，可以被使用；close 表示索引被关闭，不能使用。 |
| index | 索引名称。 |
| uuid | 索引的唯一 ID 标识。 |
| pri | 索引的主分片个数。 |
| rep | 索引的副本分片个数。 |
| docs.count | 索引内的文档个数。 |
| docs.deleted | 索引内已经删除的文档个数。 |
搜索示例
数据准备
创建表
CREATE TABLE `blog` ( `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键id', `title` varchar(60) DEFAULT NULL COMMENT '标题', `author` varchar(60) DEFAULT NULL COMMENT '作者', `content` text COMMENT '内容', `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', `update_time` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
获取测试数据
# -*- coding: utf-8 -*- from pprint import pprint import requests from parsel import Selector from puremysql import PureMysql def get_data(url): """ 获取古诗文网数据 eg: https://www.gushiwen.cn/ :return: list """ response = requests.get(url) sel = Selector(text=response.text) rows = sel.css(".main3 .left .sons") lst = [] for row in rows: title = row.css("b::text").extract_first() author = row.css(".source").xpath("string(.)").extract_first() content = row.css(".contson").xpath("string(.)").extract_first() if not title: continue item = { "title": title.strip(), "author": author.strip(), "content": content.replace('\n', ''), } pprint(item) lst.append(item) return lst def insert_data(lst): """ 数据入库 """ con = PureMysql(db_url="mysql://root:123456@127.0.0.1:3306/data?charset=utf8") table = con.table("blog") ret = table.insert(lst) con.close() print("成功入库", ret) def main(): # url = "https://www.gushiwen.cn/" for page in range(1, 11): url = f"https://www.gushiwen.cn/default.aspx?page={page}" lst = get_data(url) insert_data(lst) if __name__ == '__main__': main()
logstash同步数据配置
config/jdbc.conf
input { jdbc { jdbc_driver_library => "mysql-connector-java-8.0.16.jar" jdbc_driver_class => "com.mysql.cj.jdbc.Driver" jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/data" jdbc_user => "root" jdbc_password => "123456" statement => "SELECT id, title, content, author, create_time, update_time FROM blog" jdbc_paging_enabled => "true" jdbc_page_size => "5000" } } filter { } output { stdout { codec => rubydebug } elasticsearch { index => "blog", document_id => "%{id}" } }
同步数据
# 检查配置文件 $ ./bin/logstash -t -f config/jdbc.conf # 执行配置文件 $ ./bin/logstash -f config/jdbc.conf
问题及处理
处理elasticsearch跨域问题
config/elasticsearch.yml
http.cors.enabled: true http.cors.allow-origin: "*"
搜索提示
高亮结果显示
POST /blog/_search { "query": { "match": { "author": "李白" } }, "highlight": { "fields": { "author": {} } } }
搜索模板
将查询和参数分离
POST /blog/_search/template { "source": { "query": { "match": { "{{key}}": "{{value}}" } }, "size": "{{size}}" }, "params": { "key": "author", "value": "李白", "size": 10 } }
其他语句
# 调试模板渲染结果： GET _render/template # 取回模板定义的语法： GET _scripts/<templatename> # 删除模板定义的语法： DELETE _scripts/<templatename>
创建模板
POST /_scripts/blog_template_v1 { "script": { "lang": "mustache", "source": { "query": { "match": { "{{key}}": "{{value}}" } }, "highlight": { "fields": { "{{key}}": {} } }, "size": "{{size}}" } } }
使用模板
POST /blog/_search/template { "id": "blog_template_v1", "params": { "key": "author", "value": "李白", "size": 10 } }
模糊查询
GET test/_search { "query": { "match": { "doc":{ "query": "elastix", "fuzziness": "AUTO" } } } }
优化查询
POST _scripts/blog_template_v1 { "script": { "lang": "mustache", "source": { "size": "{{size}}", "query": { "bool": { "should": [ { "prefix": { "{{field}}.keyword": { "value": "{{query}}", "boost": 10 } } }, { "match_phrase_prefix": { "{{field}}": { "query": "{{query}}", "boost": 2 } } }, { "match": { "{{field}}": "{{query}}" } } ] } }, "_source": [ "title", "id", "uid", "views" ] } } }
重建索引
# 新建索引 PUT blog_v1 # 查看原索引的mapping GET blog/_mapping # 设置索引的mapping POST blog_v1/doc/_mapping { "doc": { "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "author": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "content": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "create_time": { "type": "date" }, "id": { "type": "long" }, "title": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "update_time": { "type": "date" } } } } # 索引迁移 POST _reindex { "source": {"index": "blog"}, "dest": {"index": "blog_v1"} } # 查询测试 POST /blog_v1/_search
索引别名
# 查看别名 GET _cat/aliases # 添加别名 POST /_aliases { "actions": [ { "add": { "index": "blog", "alias": "my-blog" } } ] } # 切换别名 POST /_aliases { "actions": [ { "add": { "index": "blog_v1", "alias": "my-blog" } }, { "remove": { "index": "blog", "alias": "my-blog" } } ] } # 通过别名搜索 POST my-blog/_search
拼音处理的插件
https://github.com/medcl/elasticsearch-analysis-pinyin/releases/tag/v6.3.2
添加拼音搜索字段
# 关闭索引 POST my-blog/_close # 设置索引支持拼音分析器 PUT my-blog/_settings { "index": { "analysis": { "analyzer": { "pinyin_analyzer": { "tokenizer": "my_pinyin" } }, "tokenizer": { "my_pinyin": { "type": "pinyin", "keep_first_letter": true, "keep_separate_first_letter": true, "keep_full_pinyin": true, "keep_original": false, "limit_first_letter_length": 16, "lowercase": true } } } } } # 打开索引 POST my-blog/_open # 获取原索引mapping GET my-blog/_mapping # 添加字段 PUT my-blog/doc/_mapping { "doc": { "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "author": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 }, "pinyin": { "type": "text", "analyzer": "pinyin_analyzer" } } }, "content": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "create_time": { "type": "date" }, "id": { "type": "long" }, "title": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "update_time": { "type": "date" } } } } # 更新索引 POST my-blog/_update_by_query?conflicts=proceed # 测试拼音搜索 POST my-blog/_search { "query": {"match": { "author.pinyin": "libai" }} }
前端显示
<html lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <title>Document</title> <!-- 开发环境版本，包含了有帮助的命令行警告 --> <script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script> <!-- 引入样式 --> <link rel="stylesheet" href="https://unpkg.com/element-ui/lib/theme-chalk/index.css" /> <!-- 引入组件库 --> <script src="https://unpkg.com/element-ui/lib/index.js"></script> <!-- axios --> <script src="https://unpkg.com/axios/dist/axios.min.js"></script> <style> /* 居中显示 */ #app { width: 200px; margin: 0 auto; margin-top: 300px; } /* 搜索结果高亮 */ em { color: red; } </style> </head> <body> <div id="app"> <el-autocomplete v-model="state" :fetch-suggestions="querySearchAsync" placeholder="请输入内容" @select="handleSelect" > <!-- 自定义显示 --> <template slot-scope="{ item }"> <div v-html="item.highlight.author[0]"></div> </template> </el-autocomplete> </div> <script> new Vue({ el: "#app", data() { return { list: [], state: "", }; }, methods: { async querySearchAsync(queryString, cb) { // 查询地址 const QUERY_URL = "http://localhost:9200/blog/_search/template"; // 查询语句 let query = { id: "blog_template_v1", params: { field: "author", query: queryString, size: 10, }, }; const res = await axios.post(QUERY_URL, query); console.log(res.data.hits.hits); cb(res.data.hits.hits); }, handleSelect(item) { console.log(item); }, }, }); </script> </body> </html>