[BUG] Performance regression in `top_hits` aggregation

[andrross]

I believe I have isolated a regression in performance of the top_hits aggregation between ES 7.9 and ES 7.10.2 (which means the regression is present in OpenSearch 1.0 as well).

To Reproduce
I added the following operation to the default nyc_taxis track in es rally:

{
  "name": "top_hits_agg",
  "warmup-iterations": 10,
  "iterations": 50,
  "clients": 2,
  "operation": {
    "operation-type": "search",
    "body": {
      "size": 0,
      "query": {
        "range": {
          "dropoff_datetime": {
            "from": "2015-01-01 00:00:00",
            "to": "2015-01-01 12:00:00"
          }
        }
      },
      "aggs": {
        "2": {
          "terms": {
            "field": "payment_type",
            "size": 1000
          },
          "aggregations": {
            "3": {
              "date_histogram": {
                "field": "dropoff_datetime",
                "fixed_interval": "5s"
              },
              "aggregations": {
                "1": {
                  "top_hits": {
                    "size": 1,
                    "sort": [
                      {
                        "dropoff_datetime": {
                          "order": "desc"
                        }
                      }
                    ]
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

After populating both a 7.9 and 7.10.2 cluster with the nyc_taxis dataset, I then ran a race with the following commands:

esrally race --pipeline=benchmark-only --distribution-version=7.9.0 --track=nyc_taxis --target-host=127.0.0.1:9790 --report-file=es_790_report_$(date +%s) --user-tag='version:7.9,race:top_hits_and_date' --include-tasks=top_hits_agg,date_histogram_agg

esrally race --pipeline=benchmark-only --distribution-version=7.10.2 --track=nyc_taxis --target-host=127.0.0.1:9710 --report-file=es_710_report_$(date +%s) --user-tag='version:7.10.2,race:top_hits_and_date' --include-tasks=top_hits_agg,date_histogram_agg

Expected behavior
The expected behavior is that there will be very similar performance for the top_hits aggregation between the two versions of the software. The following data compares ES7.9 as baseline and ES7.10.2 as the contender. As expected, the performance of the date_histogram_agg operation is more-or-less at parity between the two. However, the top_hits aggregation performance is consistently 20-30% worse in 7.10.2 than 7.9:

Metric	Task	Baseline	Contender	Diff	Unit
Cumulative indexing time of primary shards		125.509	110.375	-15.1333	min
Min cumulative indexing time across primary shard		125.509	110.375	-15.1333	min
Median cumulative indexing time across primary shard		125.509	110.375	-15.1333	min
Max cumulative indexing time across primary shard		125.509	110.375	-15.1333	min
Cumulative indexing throttle time of primary shards		0	0	0	min
Min cumulative indexing throttle time across primary shard		0	0	0	min
Median cumulative indexing throttle time across primary shard		0	0	0	min
Max cumulative indexing throttle time across primary shard		0	0	0	min
Cumulative merge time of primary shards		41.7306	47.5379	5.80733	min
Cumulative merge count of primary shards		93	89	-4
Min cumulative merge time across primary shard		41.7306	47.5379	5.80733	min
Median cumulative merge time across primary shard		41.7306	47.5379	5.80733	min
Max cumulative merge time across primary shard		41.7306	47.5379	5.80733	min
Cumulative merge throttle time of primary shards		4.12925	4.25008	0.12083	min
Min cumulative merge throttle time across primary shard		4.12925	4.25008	0.12083	min
Median cumulative merge throttle time across primary shard		4.12925	4.25008	0.12083	min
Max cumulative merge throttle time across primary shard		4.12925	4.25008	0.12083	min
Cumulative refresh time of primary shards		2.60575	2.36	-0.24575	min
Cumulative refresh count of primary shards		82	79	-3
Min cumulative refresh time across primary shard		2.60575	2.36	-0.24575	min
Median cumulative refresh time across primary shard		2.60575	2.36	-0.24575	min
Max cumulative refresh time across primary shard		2.60575	2.36	-0.24575	min
Cumulative flush time of primary shards		5.45558	5.48922	0.03363	min
Cumulative flush count of primary shards		35	34	-1
Min cumulative flush time across primary shard		5.45558	5.48922	0.03363	min
Median cumulative flush time across primary shard		5.45558	5.48922	0.03363	min
Max cumulative flush time across primary shard		5.45558	5.48922	0.03363	min
Total Young Gen GC time		0.258	1.24	0.982	s
Total Young Gen GC count		32	136	104
Total Old Gen GC time		0	0	0	s
Total Old Gen GC count		0	0	0
Store size		25.0414	24.3655	-0.67586	GB
Translog size		5.12227e-08	5.12227e-08	0	GB
Heap used for segments		0.193222	0.0988235	-0.0944	MB
Heap used for doc values		0.0354309	0.034874	-0.00056	MB
Heap used for terms		0.0408325	0.0370789	-0.00375	MB
Heap used for norms		0	0	0	MB
Heap used for points		0	0	0	MB
Heap used for stored fields		0.116959	0.0268707	-0.09009	MB
Segment count		32	30	-2
Min Throughput	date_histogram_agg	1.50023	1.50023	-1e-05	ops/s
Median Throughput	date_histogram_agg	1.50035	1.50035	-0	ops/s
Max Throughput	date_histogram_agg	1.50054	1.50054	0	ops/s
50th percentile latency	date_histogram_agg	548.563	549.301	0.73863	ms
90th percentile latency	date_histogram_agg	557.053	559.417	2.36497	ms
99th percentile latency	date_histogram_agg	571.816	569.581	-2.23542	ms
100th percentile latency	date_histogram_agg	578.69	575.319	-3.37139	ms
50th percentile service time	date_histogram_agg	547.429	548.28	0.85141	ms
90th percentile service time	date_histogram_agg	555.608	558.266	2.65809	ms
99th percentile service time	date_histogram_agg	570.955	568.385	-2.57038	ms
100th percentile service time	date_histogram_agg	577.466	573.309	-4.1576	ms
error rate	date_histogram_agg	0	0	0	%
Min Throughput	top_hits_agg	0.720015	0.559013	-0.161	ops/s
Median Throughput	top_hits_agg	0.747501	0.577278	-0.17022	ops/s
Max Throughput	top_hits_agg	0.759315	0.582273	-0.17704	ops/s
50th percentile latency	top_hits_agg	2569.6	3373.35	803.747	ms
90th percentile latency	top_hits_agg	2609.98	3455.39	845.411	ms
99th percentile latency	top_hits_agg	2622.45	3502.29	879.836	ms
100th percentile latency	top_hits_agg	2694.13	3503.48	809.352	ms
50th percentile service time	top_hits_agg	2569.6	3373.35	803.747	ms
90th percentile service time	top_hits_agg	2609.98	3455.39	845.411	ms
99th percentile service time	top_hits_agg	2622.45	3502.29	879.836	ms
100th percentile service time	top_hits_agg	2694.13	3503.48	809.352	ms
error rate	top_hits_agg	0	0	0	%

Plugins
None

Host/Environment (please complete the following information):

• OS: Ubuntu 20.04
• Version ES 7.9 and ES 7.10.2
• Host type: c6i.8xlarge

Additional context
The elasticsearch distributions are just the stock versions downloaded as tar files, extracted and started with the following command:

ES_JAVA_OPTS="-Xms8g -Xmx8g" ./bin/elasticsearch

The only non-default setting in the yml configuration file is the port number. Both instances are running on the same machine, though there load is only driven one at a time (not concurrently). The service is bound to localhost and es rally is running on the same machine.

[reta]

@andrross do you mind to check the return of the top_hits_agg for both distributions? I suspect we may run into elastic/elasticsearch#65624 or elastic/elasticsearch#67043, thank you.

[andrross]

I do indeed get different results for the query in question here, so it looks like the referenced bug may be a factor.

[nknize]

I'm curious about the referenced bug fixes. While they are something we should consider looking at, those fixes are related to nested documents. But it looks like that doesn't apply to the nyc_taxi data?

[reta]

@nknize I was just checking the recent changes (and bugfixes) related to aggregation, you may be right, but the difference in aggregation results returned between these two runs (as confirmed by @andrross) is exactly what was observed in elastic/elasticsearch#65624

[andrross]

I think I spoke too soon about the results being wrong. The results are indeed different, but in this case the query is looking for the "top hit" on the dropoff_datetime field. In the case producing the difference, the data contains multiple entries with equivalent dropoff_datetime values and the different versions of the software end up returning a different document. I don't think that is necessarily indicative of a problem though.

[reta]

Ah ... thanks @andrross , we probably could just profile this particular aggregation using 7.9 and 7.10 via curl or alike, may be something obvious is going to surface right away.

[andrross]

I profiled ES7.9 versus OpenSearch 1.2. It looks like OS1.2 is possibly spending more time doing decompression.

[andrross]

Also, I've tested this "top hits" query against ES7.11 and ES7.15 and both appear to exhibit the regressions relative to ES7.9.

[reta]

Interesting, seems to be https://issues.apache.org/jira/browse/LUCENE-9486, ES 7.9 is using Lucene 8.6 whereas 7.10 is on 8.7 where this feature has been delivered.

Update: Basically, runs down to: LZ4(16kB) (current BEST_SPEED) vs Deflate with preset dict (new BEST_COMPRESSION) and the benchmarks on the ticket do show significant difference impact on fetch time.