=Start=
缘由:
最近用ElasticSearch的搜索比较多,但是比较麻烦的一点在于Web UI上的搜索如果匹配数量超过了10000就无法知道具体数量了,而且只支持简单的查询。基于这两个痛点,不得不转向去用ES搜索接口进行查询、检索。
初次使用,碰到了不少问题,在这里简单整理一下,方便以后快速参考。
正文:
参考解答:
## 查询 age 字段在 [10,20] 区间的信息
curl -X GET "localhost:9200/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"range" : {
"age" : {
"gte" : 10,
"lte" : 20,
"boost" : 2.0
}
}
}
}
'
&
## 查询 timestamp 字段在 [2015-01-01 00:00:00, now] 区间的信息(有时候需要在 00:00:00 后面添加一个'Z'字符)
curl -X GET "localhost:9200/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"range" : {
"timestamp" : {
"gte": "2015-01-01 00:00:00",
"lte": "now",
"time_zone": "+08:00"
}
}
}
}
'
&
## 查询 name.first 字段的内容是 "s.*y" 的信息
curl -X GET "localhost:9200/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"regexp":{
"name.first": "s.*y"
}
}
}
'
&
## must必须匹配 / filter限定条件
curl -X POST "localhost:9200/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"bool" : {
"must" : {
"term" : { "user" : "kimchy" }
},
"filter": {
"term" : { "tag" : "tech" }
},
"must_not" : {
"range" : {
"age" : { "gte" : 10, "lte" : 20 }
}
},
"should" : [
{ "term" : { "tag" : "wow" } },
{ "term" : { "tag" : "elasticsearch" } }
],
"minimum_should_match" : 1,
"boost" : 1.0
}
}
}
'
&
#!/usr/bin/env python
# coding=utf-8
import sys, time
import json
import requests
def main():
url = 'http://localhost:9200/index_name/_search'
# 查找某个字段是否完全等于某个字符串的值
# payload = {
# "query": {
# "query_string": {
# # "fields" : ["uniqueCode", ],
# # "query": "98xTyz-x8B7-4B79-82x6-xyz68D75"
# "query": "uniqueCode:\"98xTyz-x8B7-4B79-82x6-xyz68D75\"" #感觉这一行比上面两行的速度更快
# # "query": "\"search_string\""
# }
# }
# }
# 限定检索区间
# payload = {
# "query": {
# "bool" : {
# "must": {
# "term": { "uniqueCode": "98xTyz-x8B7-4B79-82x6-xyz68D75" }
# },
# "filter": {
# "range" : {
# "mt_datetime" : {
# "gte": "2018-08-16 15:00:16Z",
# "lte": "now",
# "time_zone": "+08:00"
# }
# }
# }
# }
# }
# }
# 正则匹配
payload = {
"query": {
"bool" : {
"must": {
# "term": { "data": "\"search_string\"" },
"regexp":{
"data": ".*\"search_string\".*"
}
},
"filter": {
"range" : {
"mt_datetime" : {
"gte": "2018-08-16 00:00:00Z",
"lte": "2018-08-17 00:00:00Z",
# "lte": "now",
"time_zone": "+08:00"
}
}
}
}
}
}
r = requests.post(url, data=json.dumps(payload))
print r.text
if __name__ == '__main__':
time_start = time.time()
try:
main()
except KeyboardInterrupt:
print 'Killed by user'
sys.exit(0)
print "Spend {0} seconds.\n".format(time.time() - time_start)
参考链接:
- 范围查询
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html - Regexp Query(正则表达式查询)
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html - Bool 查询
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html - Elasticsearch查询——布尔查询Bool Query
- Elasticsearch(四)elasticsearch复杂检索
- elasticsearch 查询(match和term)
- Elasticsearch-bool组合查询
=END=
《 “ElasticSearch的搜索API学习整理” 》 有 7 条评论
浅析ES的安全模块
http://www.iigrowing.cn/qian_xi_es_de_an_quan_mo_kuai.html
https://www.jianshu.com/p/8ab4c955badc
Elasticsearch Query DSL查询入门
https://mp.weixin.qq.com/s/lvfN0sdRZ2VGKL1rMGaDXw
Elasticsearch query string query with not equal to?
https://stackoverflow.com/questions/40599262/elasticsearch-query-string-query-with-not-equal-to
`
Elasticsearch页面查询中如何实现「字符串不等于」的功能?
— 查询name字段中值不是”Fred”的值
!(name:”Fred”)
// 或
NOT (name:”Fred”)
`
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_boolean_operators
# 查询字符串查询
https://www.elastic.co/guide/en/elasticsearch/reference/6.4/query-dsl-query-string-query.html
# bool查询
https://www.elastic.co/guide/en/elasticsearch/reference/6.4/query-dsl-bool-query.html
23 Useful Elasticsearch Example Queries
https://dzone.com/articles/23-useful-elasticsearch-example-queries
Elasticsearch 7.x Cheatsheet
https://elasticsearch-cheatsheet.jolicode.com/
Terms aggregation
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
Search your data
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html#common-search-options
`
对于ES的常规用法中有一类是即席查询(Ad Hoc Search)某个字段是否存在,但频率不建议太高,因为ES不是为了这种用法而设计的,qps不高的情况下还可以接受,qps高了容易把服务搞挂。
但有一个场景看上去还比较合适——间隔几分钟做一次聚合搜索,从人工操作来看就是去 可视化Visualize 那里拿 terms 聚合后的结果,可能原始数据有几千上万条,经过一些过滤和聚合之后,只有几百条结果了,这时就可以定期去拿关键字段来实现自己的需求,不过实效性相对来说会差一点(根定时任务的间隔长短有关),但如果对实效性和准确性有要求的话,还是建议自己用mysql/redis来实现接口,准确性和时效性还有QPS会更有保证,不过就是要写点代码来实现。
搜索语法的话,可以去浏览器的Web页面上抓取Kibana的请求来拿查询请求 body 然后直接放到代码里面用。
`
Elasticsearch查询和聚合基本语法
https://juejin.cn/post/6844903865121259534
Version: 8.3.3
Download Elasticsearch
https://www.elastic.co/cn/downloads/elasticsearch
Download Kibana
https://www.elastic.co/cn/downloads/kibana
Start the Elastic Stack with security enabled automatically
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html
Add data
https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html
加载示例数据
https://www.elastic.co/guide/cn/kibana/current/tutorial-load-dataset.html
如何识别某个系统是Kibana系统?
根据域名 Domain:
%kibana%
根据链接 URI:
/app/kibana
/app/timelion
/app/monitoring
/app/ml
/bundles/kibana.style.css
/bundles/kibana.bundle.js
/plugins/kibana/assets/
根据响应内容 ResponseBody:
Kibana
kibana-body
kibanaWelcomeView