Fix leafSorter optimization for ReadOnlyEngine and NRTReplicationEngine #18639

rgsriram · 2025-06-28T15:38:27Z

Description

Timestamp sort optimisations (leafSorter) were not working consistently across all engine types. While InternalEngine (primary shards) correctly applied leafSorter, other engine types like ReadOnlyEngine (searchable snapshots) and NRTReplicationEngine (segment replication) were not passing leafSorter when opening DirectoryReader instances.
This caused inconsistent performance for time-series workloads:

Primary shards: Fast time-series queries (segments ordered by timestamp)
Searchable snapshots: Slower time-series queries (segments in arbitrary order)
Segment replication replicas: Slower time-series queries (segments in arbitrary order)

Solution
Apply leafSorter optimisation consistently across all engine types by ensuring DirectoryReader.open() calls include the leafSorter parameter.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#17579

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-06-28T16:22:39Z

❌ Gradle check result for 589d285: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-06-28T20:16:33Z

✅ Gradle check result for 70b752d: SUCCESS

codecov · 2025-06-28T20:16:51Z

Codecov Report

Attention: Patch coverage is 76.92308% with 3 lines in your changes missing coverage. Please review.

Project coverage is 72.77%. Comparing base (2906600) to head (c5aff16).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
...va/org/opensearch/index/engine/ReadOnlyEngine.java	40.00%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18639      +/-   ##
============================================
- Coverage     72.78%   72.77%   -0.01%     
+ Complexity    68544    68502      -42     
============================================
  Files          5567     5569       +2     
  Lines        314911   314937      +26     
  Branches      45684    45685       +1     
============================================
- Hits         229201   229189      -12     
- Misses        67092    67143      +51     
+ Partials      18618    18605      -13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rgsriram · 2025-06-29T12:43:14Z

@msfroh - Could you please review it?.

github-actions · 2025-07-02T13:13:39Z

✅ Gradle check result for 115bd30: SUCCESS

github-actions · 2025-07-10T12:56:32Z

❌ Gradle check result for 7b394f8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-10T14:56:45Z

❌ Gradle check result for 25e79de: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-10T16:21:46Z

❌ Gradle check result for 07de023: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/engine/ReadOnlyEngine.java

server/src/test/java/org/opensearch/index/engine/LeafSorterOptimizationTests.java

github-actions · 2025-07-11T10:39:01Z

❕ Gradle check result for e32bc6b: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/test/java/org/opensearch/index/engine/LeafSorterOptimizationTests.java

github-actions · 2025-07-17T14:11:02Z

❌ Gradle check result for 20f7527: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/test/java/org/opensearch/index/engine/LeafSorterOptimizationTests.java

github-actions · 2025-07-18T07:53:29Z

✅ Gradle check result for 892401e: SUCCESS

github-actions · 2025-07-22T05:44:35Z

❌ Gradle check result for 7a03880: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-22T06:05:54Z

❌ Gradle check result for 5fdca4d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-22T06:51:43Z

❌ Gradle check result for 16c2978: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-22T11:42:19Z

❌ Gradle check result for 3d87ec3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sriram Ganesh <[email protected]>

github-actions · 2025-07-22T13:43:29Z

✅ Gradle check result for c5aff16: SUCCESS

msfroh

Looks good! Thanks for sticking it out with this one @rgsriram!

…ne (opensearch-project#18639) --------- Signed-off-by: Sriram Ganesh <[email protected]>

rgsriram requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners June 28, 2025 15:38

rgsriram force-pushed the issue-17579 branch from 70b752d to 115bd30 Compare July 2, 2025 11:55

rgsriram force-pushed the issue-17579 branch from 115bd30 to 7b394f8 Compare July 10, 2025 11:50

msfroh reviewed Jul 10, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/engine/ReadOnlyEngine.java Show resolved Hide resolved

server/src/test/java/org/opensearch/index/engine/LeafSorterOptimizationTests.java Outdated Show resolved Hide resolved

opensearch-ci-bot mentioned this pull request Jul 9, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FullRollingRestartIT #18490

Closed

msfroh reviewed Jul 15, 2025

View reviewed changes

server/src/test/java/org/opensearch/index/engine/LeafSorterOptimizationTests.java Outdated Show resolved Hide resolved

rgsriram force-pushed the issue-17579 branch from 27a8ed0 to 20f7527 Compare July 17, 2025 12:54

msfroh reviewed Jul 17, 2025

View reviewed changes

server/src/test/java/org/opensearch/index/engine/LeafSorterOptimizationTests.java Outdated Show resolved Hide resolved

rgsriram force-pushed the issue-17579 branch from 7a03880 to 5fdca4d Compare July 22, 2025 05:56

rgsriram added 4 commits July 22, 2025 17:43

Fix leafSorter optimization for ReadOnlyEngine and NRTReplicationEngine

dab8de2

Signed-off-by: Sriram Ganesh <[email protected]>

Working on fixing the unit test case

d94164d

Signed-off-by: Sriram Ganesh <[email protected]>

Adding changelog

2d53b40

Signed-off-by: Sriram Ganesh <[email protected]>

Fixed the testcases

c5aff16

Signed-off-by: Sriram Ganesh <[email protected]>

rgsriram force-pushed the issue-17579 branch from 3d87ec3 to c5aff16 Compare July 22, 2025 12:13

This was referenced Jul 23, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FullClusterRestartIT #15813

Open

[AUTOCUT] Gradle Check Flaky Test Report for EquivalenceIT #15821

Open

[AUTOCUT] Gradle Check Flaky Test Report for WorkloadManagementIT #17154

Open

msfroh approved these changes Jul 23, 2025

View reviewed changes

msfroh merged commit 5486c63 into opensearch-project:main Jul 23, 2025
31 checks passed

opensearch-ci-bot mentioned this pull request Aug 3, 2025

[AUTOCUT] Gradle Check Flaky Test Report for LeafSorterOptimizationTests #18898

Closed

tandonks pushed a commit to tandonks/OpenSearch that referenced this pull request Aug 5, 2025

Fix leafSorter optimization for ReadOnlyEngine and NRTReplicationEngi…

9d9c6c7

…ne (opensearch-project#18639) --------- Signed-off-by: Sriram Ganesh <[email protected]>

opensearch-ci-bot mentioned this pull request Aug 13, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SegmentReplicationIT #14328

Open

BrewTestBot mentioned this pull request Aug 20, 2025

opensearch 3.2.0 Homebrew/homebrew-core#234146

Merged

vinaykpud pushed a commit to vinaykpud/OpenSearch that referenced this pull request Sep 26, 2025

Fix leafSorter optimization for ReadOnlyEngine and NRTReplicationEngi…

e069e90

…ne (opensearch-project#18639) --------- Signed-off-by: Sriram Ganesh <[email protected]>

Fix leafSorter optimization for ReadOnlyEngine and NRTReplicationEngine #18639

Fix leafSorter optimization for ReadOnlyEngine and NRTReplicationEngine #18639

Uh oh!

Conversation

rgsriram commented Jun 28, 2025

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Jun 28, 2025

Uh oh!

github-actions bot commented Jun 28, 2025

Uh oh!

codecov bot commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rgsriram commented Jun 29, 2025

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 11, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 17, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 18, 2025

Uh oh!

github-actions bot commented Jul 22, 2025

Uh oh!

github-actions bot commented Jul 22, 2025

Uh oh!

github-actions bot commented Jul 22, 2025

Uh oh!

github-actions bot commented Jul 22, 2025

Uh oh!

github-actions bot commented Jul 22, 2025

Uh oh!

msfroh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Jun 28, 2025 •

edited

Loading