SourceOne: Searches fail on ElasticSearch when searching a broad date range.

 

Search fails on ElasticSearch based indexes when searching a high number of indexes. These may occur when using a combination of a broad date range and/or high number of mapped folders with ES indexes.

Users may see the Incomplete Results warning in the Websearch application. The query may fail searching on some of the shards and the following errors are logged:

The following error may be seen in the ExJBQuery log:
CoExESQueryPlugIn::GetHits|ERROR|Query failed on 440 shards, setting INCOMPLETE_RESULTS warning|CoExESQueryPlugIn.cpp(777)

And the following error may also be seen in the elasticsearch.log on the data nodes that fail to search some of the shards:
Failed to execute [org.elasticsearch.action.search.SearchRequest@1f3142a] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [s1index5][inet[/10.xxx.xx.xx:9300]][search/phase/scan]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@79a0d462

The threadpool.search.size and the threadpool.search.queue_size may be exceeded in the ElasticSearch nodes when searching a high number of shards/indexes. When this happens, the server rejects searching on some of the shards. In order to confirm if the search pool is being exceeded, the following command can be run. The search.rejected column will show if the search threadpool has been exceeded since the server was started. The first column identifies in which member of the cluster is rejecting searches.

curl 'localhost:9200/_cat/thread_pool?v&h=id,host,search.active,search.rejected,search.completed'


 

The threadpool.search.size can be set to unlimited (-1) and the threadpool.search.queue_size should be changed based on the number of cores installed on the server. These changes will add more search capacity to the ES servers.

Engineering has confirmed that the current recommended settings set by ElasticSearch can be used. Add or modify the following settings in the elasticsearch.yml file on the data & search nodes:

threadpool.search.type: fixed
threadpool.search.size: (number of cores * 5)
threadpool.search.queue_size: -1