Skip to content

Adds max_shard_size parameter to shrink API #2519

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kolchfa-aws merged 8 commits into from
Feb 2, 2023
Merged

Adds max_shard_size parameter to shrink API #2519

Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
618089e
Adds max_shard_size parameter to shrink API
kolchfa-aws Jan 30, 2023
0817b32
Implemented tech review comment
kolchfa-aws Feb 2, 2023
7a02a31
One more rewording
kolchfa-aws Feb 2, 2023
c6e3eb3
Implemented doc review comments
kolchfa-aws Feb 2, 2023
24ed145
Update _api-reference/index-apis/shrink-index.md
kolchfa-aws Feb 2, 2023
fbc33d1
Update _api-reference/index-apis/shrink-index.md
kolchfa-aws Feb 2, 2023
42cec1c
Update _api-reference/index-apis/shrink-index.md
kolchfa-aws Feb 2, 2023
f699149
Implemented editorial comments
kolchfa-aws Feb 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions _api-reference/index-apis/shrink-index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,21 @@ timeout | Time | How long to wait for the request to return a response. Default
You can use the request body to configure some index settings for the target index. All fields are optional.

Field | Type | Description
:--- | :--- | :---
alias | Object | Sets an alias for the target index. Can have the fields `filter`, `index_routing`, `is_hidden`, `is_write_index`, `routing`, and `search_routing`. See [Index Aliases]({{site.url}}{{site.baseurl}}/api-reference/alias/#request-body).
Comment thread
kolchfa-aws marked this conversation as resolved.
Outdated
settings | Object | Index settings you can apply to your target index. See [Index Settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings).
[max_shard_size](#the-max_shard_size-parameter) | Bytes | Specifies the maximum size of a primary shard in the target index. Because `max_shard_size` conflicts with the `index.number_of_shards` setting, you cannot set both of them at the same time.

### The `max_shard_size` parameter

The `max_shard_size` parameter specifies the maximum size of a primary shard in the target index. OpenSearch uses `max_shard_size` and the total storage for all primary shards in the source index to calculate the number of primary shards and their size for the target index.

The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm:

@carolxob carolxob Feb 2, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm:
The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. As an example, the source index has eight primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm:

@kolchfa-aws kolchfa-aws Feb 2, 2023

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@natebower Could you provide guidance on numerals vs spelled out numbers? I thought that in technical texts we use numerals, but please let me know if it's not true.

@natebower natebower Feb 2, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spell out cardinal numbers from 1 to 9. For example, one NAT instance. Use numerals for cardinal numbers 10 and higher. Spell out ordinal numbers: first, second, and so on. In a series that includes numbers 10 or higher, use numerals for all. In this case, we should use 8 because 400 also appears in the sentence.

@cwillum cwillum Feb 2, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Are these paragraphs under the heading indented? Extra space here?
  2. "Smallest" factor sounds more natural to me than "lowest" factor. But this might be a convention in mathematics and widely accepted. Just asking.
  3. "... of the source index's primary shard count, whose shard size should not [will not?] exceed max_shard_size." Does that mess up the meaning?

@kolchfa-aws kolchfa-aws Feb 2, 2023

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Removed extra space, thank you. Does not affect the rendering, but still a good call.
  2. Agreed. Smallest is better.
  3. I feel like "whose" makes it less clear because it's not clear what it refers to.


1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3.
1. Calculate the number of primary shards as the lowest factor of 8 that is greater than 3. The number of primary shards is 4.

@cwillum cwillum Feb 2, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

smallest versus lowest for factor. (again, I admit I may be out of the know on this)
I like having this example here.


The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of memory. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is lower than 6, the number of primary shards in the target index is set to 5.

@carolxob carolxob Feb 2, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of memory. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is lower than 6, the number of primary shards in the target index is set to 5.
The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with five primary shards that occupy a total of 600 GB of memory. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is six. However, because the number of primary shards in the source index is lower than six, the number of primary shards in the target index is set to five.


{% comment %}
The new parameter to be called max_shard_size.
max_primary_shard_size | Bytes | Sets the maximum size of a primary shard in the target index. For example, if this field is set to 100 GB, and the source index's primary shards total to 300 GB, then the target index has 3 primary shards of 100 GB each.
{% endcomment %}
The minimum number of primary shards for the target index is 1.
{: .note}