Fix: Replication of large documents breaches the size limit (2GB) of ReleasableBytesStreamOutput #1580

darjisagar7 · 2025-09-15T12:35:17Z

Description

When the follower cluster fetches the data from the leader node it breaches the 2GB limit for a single call. This PR handles that issue in the following ways

Introducing an Index level batch size, which can be maintain by the clients via the index settings.
Retry immediately by reducing the batch size. This batch size is dynamically maintained by the node. If the node is destroy then this values will be discarded.

Related Issues

Resolves #1568

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ReleasableBytesStreamOutput Signed-off-by: Sagar Darji <[email protected]>

ankitkala · 2025-09-16T06:54:39Z

src/main/kotlin/org/opensearch/replication/task/shard/ShardReplicationChangesTracker.kt

-                val fromSeq = seqNoAlreadyRequested.getAndAdd(batchSize.toLong()) + 1
-                val toSeq = fromSeq + batchSize - 1
-                logDebug("Fetching the batch $fromSeq-$toSeq")
+                val currentBatchSize = batchSizeSettings.getEffectiveBatchSize()


Can we gate this dynamic batch size feature behind a cluster/index setting? We can enable it by default based on testing.

I dont think another setting is needed as the test coverage is good.

src/main/kotlin/org/opensearch/replication/task/shard/BatchSizeSettings.kt

ankitkala · 2025-09-16T06:59:44Z

src/main/kotlin/org/opensearch/replication/task/shard/ShardReplicationTask.kt

+
+                        // Handle 2GB limit exception specifically
+                        if (e is IllegalArgumentException &&
+                            e.message?.equals("ReleasableBytesStreamOutput cannot hold more than 2GB of data") == true) {


Should we just check for ReleasableBytesStreamOutput cannot hold more than? If the 2GB limit changes in future, this exception will start failing again

src/main/kotlin/org/opensearch/replication/task/shard/BatchSizeSettings.kt

soosinha · 2025-09-18T07:08:40Z

src/main/kotlin/org/opensearch/replication/ReplicationPlugin.kt

        const val REPLICATION_EXECUTOR_NAME_FOLLOWER = "replication_follower"
        val REPLICATED_INDEX_SETTING: Setting<String> = Setting.simpleString("index.plugins.replication.follower.leader_index",
            Setting.Property.InternalIndex, Setting.Property.IndexScope)
+        // Node-level batch size setting


Nit: Cluster level

soosinha · 2025-09-18T08:40:51Z

src/main/kotlin/org/opensearch/replication/task/shard/BatchSizeSettings.kt

+    // For dynamic batch size adjustment (2GB fix)
+    @Volatile
+    private var dynamicBatchSize: Int? = null
+    private val minBatchSize = 16


Lets define this as a constant and reuse the value here and also in the settings definition in ReplicationPlugin

Have added a constant in ReplicationPlugin class

soosinha · 2025-09-18T08:44:39Z

src/main/kotlin/org/opensearch/replication/task/shard/ShardReplicationChangesTracker.kt

+     */
+    fun reduceBatchSize() {
+        batchSizeSettings.reduceBatchSize()
+        logDebug("Batch size reduced to ${batchSizeSettings.getEffectiveBatchSize()}")


log as INFO

soosinha · 2025-09-18T08:44:51Z

src/main/kotlin/org/opensearch/replication/task/shard/ShardReplicationChangesTracker.kt

+     */
+    fun resetBatchSize() {
+        batchSizeSettings.resetBatchSize()
+        logDebug("Batch size reset to ${batchSizeSettings.getEffectiveBatchSize()}")


log as INFO

soosinha · 2025-09-18T09:04:59Z

src/main/kotlin/org/opensearch/replication/task/shard/ShardReplicationChangesTracker.kt

-                val fromSeq = seqNoAlreadyRequested.getAndAdd(batchSize.toLong()) + 1
-                val toSeq = fromSeq + batchSize - 1
-                logDebug("Fetching the batch $fromSeq-$toSeq")
+                val currentBatchSize = batchSizeSettings.getEffectiveBatchSize()


I dont think another setting is needed as the test coverage is good.

Signed-off-by: Sagar Darji <[email protected]>

opensearch-trigger-bot · 2025-09-29T07:21:45Z

The backport to 2.7 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.7 2.7
# Navigate to the new working tree
cd .worktrees/backport-2.7
# Create a new branch
git switch --create backport/backport-1580-to-2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5bbe925d91816b998b9a472d2218f9102367b843
# Push it to GitHub
git push --set-upstream origin backport/backport-1580-to-2.7
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.7

Then, create a pull request where the base branch is 2.7 and the compare/head branch is backport/backport-1580-to-2.7.

opensearch-trigger-bot · 2025-09-29T07:22:42Z

The backport to 2.17 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.17 2.17
# Navigate to the new working tree
cd .worktrees/backport-2.17
# Create a new branch
git switch --create backport/backport-1580-to-2.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5bbe925d91816b998b9a472d2218f9102367b843
# Push it to GitHub
git push --set-upstream origin backport/backport-1580-to-2.17
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.17

Then, create a pull request where the base branch is 2.17 and the compare/head branch is backport/backport-1580-to-2.17.

opensearch-trigger-bot · 2025-09-29T07:22:47Z

The backport to 2.19 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.19 2.19
# Navigate to the new working tree
cd .worktrees/backport-2.19
# Create a new branch
git switch --create backport/backport-1580-to-2.19
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5bbe925d91816b998b9a472d2218f9102367b843
# Push it to GitHub
git push --set-upstream origin backport/backport-1580-to-2.19
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.19

Then, create a pull request where the base branch is 2.19 and the compare/head branch is backport/backport-1580-to-2.19.

opensearch-trigger-bot · 2025-09-29T07:22:48Z

The backport to 3.1.0 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-3.1.0 3.1.0
# Navigate to the new working tree
cd .worktrees/backport-3.1.0
# Create a new branch
git switch --create backport/backport-1580-to-3.1.0
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5bbe925d91816b998b9a472d2218f9102367b843
# Push it to GitHub
git push --set-upstream origin backport/backport-1580-to-3.1.0
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-3.1.0

Then, create a pull request where the base branch is 3.1.0 and the compare/head branch is backport/backport-1580-to-3.1.0.

Fix: Replication of large documents breaches the size limit (2GB) of …

2df39ab

…ReleasableBytesStreamOutput Signed-off-by: Sagar Darji <[email protected]>

darjisagar7 requested review from ankitkala, gbbafna, krishna-ggk, monusingh-1, saikaranam-amazon and soosinha as code owners September 15, 2025 12:35

ankitkala reviewed Sep 16, 2025

View reviewed changes

soosinha reviewed Sep 18, 2025

View reviewed changes

Addressing the comments for previous commit

46771df

Signed-off-by: Sagar Darji <[email protected]>

soosinha approved these changes Sep 19, 2025

View reviewed changes

ankitkala enabled auto-merge (squash) September 19, 2025 11:30

ankitkala approved these changes Sep 19, 2025

View reviewed changes

ankitkala merged commit 5bbe925 into opensearch-project:main Sep 19, 2025
12 of 13 checks passed

monusingh-1 added the backport 2.7 label Sep 29, 2025

monusingh-1 added backport 2.17 backport 2.19 backport 3.1.0 and removed backport 2.7 labels Sep 29, 2025

darjisagar7 mentioned this pull request Nov 6, 2025

Updating the Cross Cluster Replication documentation for the index le… opensearch-project/documentation-website#11496

Merged

1 task

soosinha removed backport 2.17 backport 2.19 backport 3.1.0 labels Nov 7, 2025

Fix: Replication of large documents breaches the size limit (2GB) of ReleasableBytesStreamOutput #1580

Fix: Replication of large documents breaches the size limit (2GB) of ReleasableBytesStreamOutput #1580

Uh oh!

Conversation

darjisagar7 commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Sep 29, 2025

Uh oh!

opensearch-trigger-bot bot commented Sep 29, 2025

Uh oh!

opensearch-trigger-bot bot commented Sep 29, 2025

Uh oh!

opensearch-trigger-bot bot commented Sep 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

darjisagar7 commented Sep 15, 2025 •

edited

Loading