Adds max_shard_size parameter to shrink API

[kolchfa-aws]

Adds max_shard_size parameter to shrink API

Expands #2352
Fixes #2044

Checklist

• [ x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
  For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[kolchfa-aws]

@gaobinlong: I've created a section for the max_shard_size parameter. Could you review it for technical accuracy? Thanks!

[gaobinlong]

@kolchfa-aws here is my comment: a total of 400 GB of memory maybe not accurate, shard size means the storage size i.e. the disk usage, so can you change it to a total of 400 GB of storage ?

[kolchfa-aws]

@gaobinlong Done, thank you for the review!

[carolxob]

Suggested change

The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm:

The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. As an example, the source index has eight primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm:

[kolchfa-aws]

@natebower Could you provide guidance on numerals vs spelled out numbers? I thought that in technical texts we use numerals, but please let me know if it's not true.

[natebower]

Spell out cardinal numbers from 1 to 9. For example, one NAT instance. Use numerals for cardinal numbers 10 and higher. Spell out ordinal numbers: first, second, and so on. In a series that includes numbers 10 or higher, use numerals for all. In this case, we should use 8 because 400 also appears in the sentence.

[carolxob]

Suggested change

The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of memory. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is lower than 6, the number of primary shards in the target index is set to 5.

The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with five primary shards that occupy a total of 600 GB of memory. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is six. However, because the number of primary shards in the source index is lower than six, the number of primary shards in the target index is set to five.

[carolxob]

LGTM with very minor suggestions.

[ariamarble]

LGTM

[cwillum]

Looks good, with a few questions/comments.

[cwillum]

1. Are these paragraphs under the heading indented? Extra space here?
2. "Smallest" factor sounds more natural to me than "lowest" factor. But this might be a convention in mathematics and widely accepted. Just asking.
3. "... of the source index's primary shard count, whose shard size should not [will not?] exceed max_shard_size." Does that mess up the meaning?

[kolchfa-aws]

1. Removed extra space, thank you. Does not affect the rendering, but still a good call.
2. Agreed. Smallest is better.
3. I feel like "whose" makes it less clear because it's not clear what it refers to.

[cwillum]

smallest versus lowest for factor. (again, I admit I may be out of the know on this)
I like having this example here.

[cwillum]

"... because the number of primary shards in the source index is smaller than 6, ..."
Number being smaller.

[natebower]

@kolchfa-aws Just a few small changes and one comment. Thanks!

[opensearch-trigger-bot]

The backport to 2.0 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.0 2.0
# Navigate to the new working tree
cd .worktrees/backport-2.0
# Create a new branch
git switch --create backport/backport-2519-to-2.0
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6e120ec4b8d6ff03b0706f56b0106a16f1ef9b42
# Push it to GitHub
git push --set-upstream origin backport/backport-2519-to-2.0
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.0

Then, create a pull request where the base branch is 2.0 and the compare/head branch is backport/backport-2519-to-2.0.

[opensearch-trigger-bot]

The backport to 2.1 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.1 2.1
# Navigate to the new working tree
cd .worktrees/backport-2.1
# Create a new branch
git switch --create backport/backport-2519-to-2.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6e120ec4b8d6ff03b0706f56b0106a16f1ef9b42
# Push it to GitHub
git push --set-upstream origin backport/backport-2519-to-2.1
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.1

Then, create a pull request where the base branch is 2.1 and the compare/head branch is backport/backport-2519-to-2.1.

[opensearch-trigger-bot]

The backport to 2.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.2 2.2
# Navigate to the new working tree
cd .worktrees/backport-2.2
# Create a new branch
git switch --create backport/backport-2519-to-2.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6e120ec4b8d6ff03b0706f56b0106a16f1ef9b42
# Push it to GitHub
git push --set-upstream origin backport/backport-2519-to-2.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.2

Then, create a pull request where the base branch is 2.2 and the compare/head branch is backport/backport-2519-to-2.2.