Search on large number of traces can overload Elasticserch

[pavolloffay]

Requirement - what kind of business use case are you trying to solve?

Search a large number of traces via /api/traces. For example limit set to 300k can overload elasticsearch. Resulting in running GC in a loop

[2018-07-26T17:09:36,078][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][315] overhead, spent [4s] collecting in the last [4.2s]
[2018-07-26T17:09:39,889][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][316] overhead, spent [3.5s] collecting in the last [3.8s]
[2018-07-26T17:09:43,519][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][317] overhead, spent [3.4s] collecting in the last [3.6s]
[2018-07-26T17:09:46,900][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][318] overhead, spent [3.1s] collecting in the last [2.9s]
[2018-07-26T17:09:50,112][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][319] overhead, spent [2.9s] collecting in the last [3.6s]
[2018-07-26T17:09:53,126][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][320] overhead, spent [2.5s] collecting in the last [2.7s]
[2018-07-26T17:09:56,015][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][321] overhead, spent [3s] collecting in the last [3.1s]
[2018-07-26T17:09:58,613][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][322] overhead, spent [2.4s] collecting in the last [2.5s]
[2018-07-26T17:10:01,885][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][323] overhead, spent [3.1s] collecting in the last [3.2s]
[2018-07-26T17:10:04,490][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][324] overhead, spent [2.4s] collecting in the last [2.6s]
[2018-07-26T17:10:07,791][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][325] overhead, spent [3.1s] collecting in the last [3.2s]
[2018-07-26T17:10:10,437][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][326] overhead, spent [2.5s] collecting in the last [2.6s]
[2018-07-26T17:10:14,049][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][327] overhead, spent [3.5s] collecting in the last [3.6s]
[2018-07-26T17:10:16,558][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][328] overhead, spent [2.3s] collecting in the last [2.5s]

Steps to reproduce:
docker run -it --rm "ES_JAVA_OPTS=-Xms=2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" --name=elasticsearch docker.elastic.co/elasticsearch/elasticsearch:5.6.10
SPAN_STORAGE_TYPE=elasticsearch go run -tags ui cmd/standalone/main.go --collector.queue-size=300000
QUERY_FROM=jaeger-query ITERATIONS=3000 THREAD_COUNT=100 mvn clean package exec:java from https://github.com/pavolloffay/jaeger-perf-tests

Proposal - what do you suggest to solve the problem or improve the existing situation?

Maybe use scroll API https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html #696 (comment) . Or define max limit and provide pagination. UI already defines hard limit to Related issue jaegertracing/jaeger-ui#73

[NoobIsBack]

Could u pls add your query, this would provide proper insights into the problem and resolution

[pavolloffay]

You can query jaeger traces API with a large limit parameter.

[arruda]

One of the things that I thought about that could help (since there is no pagination on the ui) is to use multiple requests with a very high limit, but using a very small window of time for the search( using lookback=custom and setting the start/end parameters). Not sure if this indeed would cover the case you are talking about, and you would also need to check for duplicate traces between the different requests results, but I believe it might be workaround on this.