Fix checkpoint handling to prevent segment replication infinite loop

[ashking94]

Description

This fixes an issue in segrep flow which causes infinite segrep loop. The analysis is present in the issue - #18605.

Related Issues

Resolves #18605

Check List

• Functionality includes testing.
• [ ] API changes companion pull request created, if applicable.
• [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for ff5a898: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 33.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.84%. Comparing base (b1d6f55) to head (ed9e60a).
Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
...s/replication/SegmentReplicationTargetService.java	33.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18636      +/-   ##
============================================
+ Coverage     72.79%   72.84%   +0.05%     
- Complexity    68362    68466     +104     
============================================
  Files          5563     5563              
  Lines        314174   314176       +2     
  Branches      45554    45555       +1     
============================================
+ Hits         228703   228871     +168     
+ Misses        66887    66764     -123     
+ Partials      18584    18541      -43     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Gradle check result for 27e3218: SUCCESS

[ashking94]

I have not been able to repro the issue in ITs. I have tried reproducing it, will try to understand why doesn't it happen on local.

[ashking94]

I have not been able to repro the issue in ITs. I have tried reproducing it, will try to understand why doesn't it happen on local.

I was able to modify an unit test that was showing the behaviour where the cyclic loop of segrep is seen always.

REPRODUCE WITH: ./gradlew 'null' --tests 'org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testStartReplicationListenerSuccess' -Dtests.seed=B17175E6A29DDE01 -Dtests.locale=bg-Cyrl-BG -Dtests.timezone=Europe/Vatican -Druntime.java=21

org.mockito.exceptions.verification.TooManyActualInvocations: 
segmentReplicationTargetService.processLatestReceivedCheckpoint(
    <any>,
    <any>
);
Wanted 1 time:
-> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testStartReplicationListenerSuccess(SegmentReplicationTargetServiceTests.java:628)
But was 27 times:
-> at org.opensearch.indices.replication.SegmentReplicationTargetService.afterIndexShardStarted(SegmentReplicationTargetService.java:235)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)

[github-actions]

✅ Gradle check result for ed9e60a: SUCCESS

[ashking94]

@mch2 I have tried making the above change which should fix the recursion loop. Do review if you can.

[ashking94]

I have run following tests for 1000+ iterations and have not seen any single failure:

• PublishCheckpointActionTests
• SegmentReplicationTargetServiceTests
• SegmentReplicationIT

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-18636-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 46a0045a2301b3bc51e2da9c5e853f69ac6342c2
# Push it to GitHub
git push --set-upstream origin backport/backport-18636-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-18636-to-2.x.

[opensearch-trigger-bot]

The backport to 2.19 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.19 2.19
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.19
# Create a new branch
git switch --create backport/backport-18636-to-2.19
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 46a0045a2301b3bc51e2da9c5e853f69ac6342c2
# Push it to GitHub
git push --set-upstream origin backport/backport-18636-to-2.19
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.19

Then, create a pull request where the base branch is 2.19 and the compare/head branch is backport/backport-18636-to-2.19.

[opensearch-trigger-bot]

The backport to 3.0 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-3.0 3.0
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-3.0
# Create a new branch
git switch --create backport/backport-18636-to-3.0
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 46a0045a2301b3bc51e2da9c5e853f69ac6342c2
# Push it to GitHub
git push --set-upstream origin backport/backport-18636-to-3.0
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-3.0

Then, create a pull request where the base branch is 3.0 and the compare/head branch is backport/backport-18636-to-3.0.

[opensearch-trigger-bot]

The backport to 3.1 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-3.1 3.1
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-3.1
# Create a new branch
git switch --create backport/backport-18636-to-3.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 46a0045a2301b3bc51e2da9c5e853f69ac6342c2
# Push it to GitHub
git push --set-upstream origin backport/backport-18636-to-3.1
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-3.1

Then, create a pull request where the base branch is 3.1 and the compare/head branch is backport/backport-18636-to-3.1.

[ashking94]

Raised manual backport for 2.19 minor version releases - #18677.

[ashking94]

It appears that the test org.opensearch.indices.replication.SegmentReplicationIT.testNodeDropWithOngoingReplication may have become flaky as I saw a test failure in random iterations run. I will fix the same.

[fullykubed]

@ashking94 @gbbafna Not sure what the policy here is, but is it possible to cut a new patch release for the backported version channels now that this is merged? This bug unfortunately makes remote storage functionally unusable, and it would be nice to make it usable again.

[ashking94]

@fullykubed This change has been backported to 2.19 and should be released as part of 2.19.3. You can refer the schedule of planned releases at https://opensearch.org/releases/.

[fullykubed]

@ashking94 Got it! Thanks for the clarification.

To confirm, no plans to backport this to 3.x release branches?

[ashking94]

AFAIK, I dont think there are any planned releases for 3.x.x minor/patch versions. The fix will be released in 3.2.0 which is scheduled to be released around August 19th, 2025.