=Start=
缘由:
最近用ElasticSearch的搜索比较多,但是比较麻烦的一点在于Web UI上的搜索如果匹配数量超过了10000就无法知道具体数量了,而且只支持简单的查询。基于这两个痛点,不得不转向去用ES搜索接口进行查询、检索。
初次使用,碰到了不少问题,在这里简单整理一下,方便以后快速参考。
正文:
参考解答:
## 查询 age 字段在 [10,20] 区间的信息 curl -X GET "localhost:9200/_search" -H 'Content-Type: application/json' -d' { "query": { "range" : { "age" : { "gte" : 10, "lte" : 20, "boost" : 2.0 } } } } '
&
## 查询 timestamp 字段在 [2015-01-01 00:00:00, now] 区间的信息(有时候需要在 00:00:00 后面添加一个'Z'字符) curl -X GET "localhost:9200/_search" -H 'Content-Type: application/json' -d' { "query": { "range" : { "timestamp" : { "gte": "2015-01-01 00:00:00", "lte": "now", "time_zone": "+08:00" } } } } '
&
## 查询 name.first 字段的内容是 "s.*y" 的信息 curl -X GET "localhost:9200/_search" -H 'Content-Type: application/json' -d' { "query": { "regexp":{ "name.first": "s.*y" } } } '
&
## must必须匹配 / filter限定条件 curl -X POST "localhost:9200/_search" -H 'Content-Type: application/json' -d' { "query": { "bool" : { "must" : { "term" : { "user" : "kimchy" } }, "filter": { "term" : { "tag" : "tech" } }, "must_not" : { "range" : { "age" : { "gte" : 10, "lte" : 20 } } }, "should" : [ { "term" : { "tag" : "wow" } }, { "term" : { "tag" : "elasticsearch" } } ], "minimum_should_match" : 1, "boost" : 1.0 } } } '
&
#!/usr/bin/env python # coding=utf-8 import sys, time import json import requests def main(): url = 'http://localhost:9200/index_name/_search' # 查找某个字段是否完全等于某个字符串的值 # payload = { # "query": { # "query_string": { # # "fields" : ["uniqueCode", ], # # "query": "98xTyz-x8B7-4B79-82x6-xyz68D75" # "query": "uniqueCode:\"98xTyz-x8B7-4B79-82x6-xyz68D75\"" #感觉这一行比上面两行的速度更快 # # "query": "\"search_string\"" # } # } # } # 限定检索区间 # payload = { # "query": { # "bool" : { # "must": { # "term": { "uniqueCode": "98xTyz-x8B7-4B79-82x6-xyz68D75" } # }, # "filter": { # "range" : { # "mt_datetime" : { # "gte": "2018-08-16 15:00:16Z", # "lte": "now", # "time_zone": "+08:00" # } # } # } # } # } # } # 正则匹配 payload = { "query": { "bool" : { "must": { # "term": { "data": "\"search_string\"" }, "regexp":{ "data": ".*\"search_string\".*" } }, "filter": { "range" : { "mt_datetime" : { "gte": "2018-08-16 00:00:00Z", "lte": "2018-08-17 00:00:00Z", # "lte": "now", "time_zone": "+08:00" } } } } } } r = requests.post(url, data=json.dumps(payload)) print r.text if __name__ == '__main__': time_start = time.time() try: main() except KeyboardInterrupt: print 'Killed by user' sys.exit(0) print "Spend {0} seconds.\n".format(time.time() - time_start)
参考链接:
- 范围查询
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html - Regexp Query(正则表达式查询)
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html - Bool 查询
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html - Elasticsearch查询——布尔查询Bool Query
- Elasticsearch(四)elasticsearch复杂检索
- elasticsearch 查询(match和term)
- Elasticsearch-bool组合查询
=END=
《 “ElasticSearch的搜索API学习整理” 》 有 7 条评论
浅析ES的安全模块
http://www.iigrowing.cn/qian_xi_es_de_an_quan_mo_kuai.html
https://www.jianshu.com/p/8ab4c955badc
Elasticsearch Query DSL查询入门
https://mp.weixin.qq.com/s/lvfN0sdRZ2VGKL1rMGaDXw
Elasticsearch query string query with not equal to?
https://stackoverflow.com/questions/40599262/elasticsearch-query-string-query-with-not-equal-to
`
Elasticsearch页面查询中如何实现「字符串不等于」的功能?
— 查询name字段中值不是”Fred”的值
!(name:”Fred”)
// 或
NOT (name:”Fred”)
`
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_boolean_operators
# 查询字符串查询
https://www.elastic.co/guide/en/elasticsearch/reference/6.4/query-dsl-query-string-query.html
# bool查询
https://www.elastic.co/guide/en/elasticsearch/reference/6.4/query-dsl-bool-query.html
23 Useful Elasticsearch Example Queries
https://dzone.com/articles/23-useful-elasticsearch-example-queries
Elasticsearch 7.x Cheatsheet
https://elasticsearch-cheatsheet.jolicode.com/
Terms aggregation
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
Search your data
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html#common-search-options
`
对于ES的常规用法中有一类是即席查询(Ad Hoc Search)某个字段是否存在,但频率不建议太高,因为ES不是为了这种用法而设计的,qps不高的情况下还可以接受,qps高了容易把服务搞挂。
但有一个场景看上去还比较合适——间隔几分钟做一次聚合搜索,从人工操作来看就是去 可视化Visualize 那里拿 terms 聚合后的结果,可能原始数据有几千上万条,经过一些过滤和聚合之后,只有几百条结果了,这时就可以定期去拿关键字段来实现自己的需求,不过实效性相对来说会差一点(根定时任务的间隔长短有关),但如果对实效性和准确性有要求的话,还是建议自己用mysql/redis来实现接口,准确性和时效性还有QPS会更有保证,不过就是要写点代码来实现。
搜索语法的话,可以去浏览器的Web页面上抓取Kibana的请求来拿查询请求 body 然后直接放到代码里面用。
`
Elasticsearch查询和聚合基本语法
https://juejin.cn/post/6844903865121259534
Version: 8.3.3
Download Elasticsearch
https://www.elastic.co/cn/downloads/elasticsearch
Download Kibana
https://www.elastic.co/cn/downloads/kibana
Start the Elastic Stack with security enabled automatically
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html
Add data
https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html
加载示例数据
https://www.elastic.co/guide/cn/kibana/current/tutorial-load-dataset.html
如何识别某个系统是Kibana系统?
根据域名 Domain:
%kibana%
根据链接 URI:
/app/kibana
/app/timelion
/app/monitoring
/app/ml
/bundles/kibana.style.css
/bundles/kibana.bundle.js
/plugins/kibana/assets/
根据响应内容 ResponseBody:
Kibana
kibana-body
kibanaWelcomeView