From 618089e71665f45e0f9f342ad1c948b6d1128153 Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Mon, 30 Jan 2023 10:23:05 -0500 Subject: [PATCH 1/8] Adds max_shard_size parameter to shrink API Signed-off-by: Fanit Kolchina --- _api-reference/index-apis/shrink-index.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index 3a723914873..4f2048ee425 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -56,10 +56,21 @@ timeout | Time | How long to wait for the request to return a response. Default You can use the request body to configure some index settings for the target index. All fields are optional. Field | Type | Description +:--- | :--- | :--- alias | Object | Sets an alias for the target index. Can have the fields `filter`, `index_routing`, `is_hidden`, `is_write_index`, `routing`, and `search_routing`. See [Index Aliases]({{site.url}}{{site.baseurl}}/api-reference/alias/#request-body). settings | Object | Index settings you can apply to your target index. See [Index Settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings). +[max_shard_size](#the-max_shard_size-parameter) | Bytes | Specifies the maximum size of a primary shard in the target index. Because `max_shard_size` conflicts with the `index.number_of_shards` setting, you cannot set both of them at the same time. + +### The `max_shard_size` parameter + + The `max_shard_size` parameter specifies the maximum size of a primary shard in the target index. OpenSearch uses `max_shard_size` and the total storage for all primary shards in the source index to calculate the number of primary shards and their size for the target index. + + The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of memory. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: + + 1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3. + 1. Calculate the number of primary shards as the lowest factor of 8 that is greater than 3. The number of primary shards is 4. + + The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of memory. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is lower than 6, the number of primary shards in the target index is set to 5. -{% comment %} -The new parameter to be called max_shard_size. -max_primary_shard_size | Bytes | Sets the maximum size of a primary shard in the target index. For example, if this field is set to 100 GB, and the source index's primary shards total to 300 GB, then the target index has 3 primary shards of 100 GB each. -{% endcomment %} + The minimum number of primary shards for the target index is 1. + {: .note} \ No newline at end of file From 0817b320bf124bf9ceeef14fe5873c978f5a19de Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Thu, 2 Feb 2023 09:59:12 -0500 Subject: [PATCH 2/8] Implemented tech review comment Signed-off-by: Fanit Kolchina --- _api-reference/index-apis/shrink-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index 4f2048ee425..2a5e3581df3 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -65,7 +65,7 @@ settings | Object | Index settings you can apply to your target index. See [Inde The `max_shard_size` parameter specifies the maximum size of a primary shard in the target index. OpenSearch uses `max_shard_size` and the total storage for all primary shards in the source index to calculate the number of primary shards and their size for the target index. - The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of memory. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: + The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: 1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3. 1. Calculate the number of primary shards as the lowest factor of 8 that is greater than 3. The number of primary shards is 4. From 7a02a31637fe6940a6b3206a4bb0a23b3d0953ee Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Thu, 2 Feb 2023 10:47:37 -0500 Subject: [PATCH 3/8] One more rewording Signed-off-by: Fanit Kolchina --- _api-reference/index-apis/shrink-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index 2a5e3581df3..19dfd22147e 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -70,7 +70,7 @@ settings | Object | Index settings you can apply to your target index. See [Inde 1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3. 1. Calculate the number of primary shards as the lowest factor of 8 that is greater than 3. The number of primary shards is 4. - The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of memory. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is lower than 6, the number of primary shards in the target index is set to 5. + The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of storage. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is lower than 6, the number of primary shards in the target index is set to 5. The minimum number of primary shards for the target index is 1. {: .note} \ No newline at end of file From c6e3eb33223b78b77eda6c2ea38d785f3335b7d2 Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Thu, 2 Feb 2023 14:17:54 -0500 Subject: [PATCH 4/8] Implemented doc review comments Signed-off-by: Fanit Kolchina --- _api-reference/index-apis/shrink-index.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index 19dfd22147e..ed9e341ce6e 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -63,14 +63,14 @@ settings | Object | Index settings you can apply to your target index. See [Inde ### The `max_shard_size` parameter - The `max_shard_size` parameter specifies the maximum size of a primary shard in the target index. OpenSearch uses `max_shard_size` and the total storage for all primary shards in the source index to calculate the number of primary shards and their size for the target index. - - The primary shard count of the target index is the lowest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: +The `max_shard_size` parameter specifies the maximum size of a primary shard in the target index. OpenSearch uses `max_shard_size` and the total storage for all primary shards in the source index to calculate the number of primary shards and their size for the target index. - 1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3. - 1. Calculate the number of primary shards as the lowest factor of 8 that is greater than 3. The number of primary shards is 4. - - The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of storage. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is lower than 6, the number of primary shards in the target index is set to 5. +The primary shard count of the target index is the smallest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: - The minimum number of primary shards for the target index is 1. - {: .note} \ No newline at end of file +1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3. +1. Calculate the number of primary shards as the smallest factor of 8 that is greater than 3. The number of primary shards is 4. + +The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of storage. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is smaller than 6, the number of primary shards in the target index is set to 5. + +The minimum number of primary shards for the target index is 1. +{: .note} \ No newline at end of file From 24ed14508c9b48401ddd632d16476df7d709fcd2 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Thu, 2 Feb 2023 15:02:36 -0500 Subject: [PATCH 5/8] Update _api-reference/index-apis/shrink-index.md Co-authored-by: Nathan Bower --- _api-reference/index-apis/shrink-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index ed9e341ce6e..518d365a175 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -57,7 +57,7 @@ You can use the request body to configure some index settings for the target ind Field | Type | Description :--- | :--- | :--- -alias | Object | Sets an alias for the target index. Can have the fields `filter`, `index_routing`, `is_hidden`, `is_write_index`, `routing`, and `search_routing`. See [Index Aliases]({{site.url}}{{site.baseurl}}/api-reference/alias/#request-body). +alias | Object | Sets an alias for the target index. Can have the fields `filter`, `index_routing`, `is_hidden`, `is_write_index`, `routing`, or `search_routing`. See [Index Aliases]({{site.url}}{{site.baseurl}}/api-reference/alias/#request-body). settings | Object | Index settings you can apply to your target index. See [Index Settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings). [max_shard_size](#the-max_shard_size-parameter) | Bytes | Specifies the maximum size of a primary shard in the target index. Because `max_shard_size` conflicts with the `index.number_of_shards` setting, you cannot set both of them at the same time. From fbc33d1dda11182143aa18f6f8f90447d95646b1 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Thu, 2 Feb 2023 15:02:49 -0500 Subject: [PATCH 6/8] Update _api-reference/index-apis/shrink-index.md Co-authored-by: Nathan Bower --- _api-reference/index-apis/shrink-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index 518d365a175..5cacfac7654 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -65,7 +65,7 @@ settings | Object | Index settings you can apply to your target index. See [Inde The `max_shard_size` parameter specifies the maximum size of a primary shard in the target index. OpenSearch uses `max_shard_size` and the total storage for all primary shards in the source index to calculate the number of primary shards and their size for the target index. -The primary shard count of the target index is the smallest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. Consider the following example. Let's say the source index has 8 primary shards and they occupy a total of 400 GB of storage. If `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: +The primary shard count of the target index is the smallest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. For example, if the source index has 8 primary shards, they occupy a total of 400 GB of storage, and the `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: 1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3. 1. Calculate the number of primary shards as the smallest factor of 8 that is greater than 3. The number of primary shards is 4. From 42cec1c0d380d07a71b801869e07a439b3a571ca Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Thu, 2 Feb 2023 15:02:56 -0500 Subject: [PATCH 7/8] Update _api-reference/index-apis/shrink-index.md Co-authored-by: Nathan Bower --- _api-reference/index-apis/shrink-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index 5cacfac7654..24f1a0501c4 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -70,7 +70,7 @@ The primary shard count of the target index is the smallest factor of the source 1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3. 1. Calculate the number of primary shards as the smallest factor of 8 that is greater than 3. The number of primary shards is 4. -The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider the source index with 5 primary shards that occupy a total of 600 GB of storage. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is smaller than 6, the number of primary shards in the target index is set to 5. +The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider a source index with 5 primary shards that occupy a total of 600 GB of storage. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is smaller than 6, the number of primary shards in the target index is set to 5. The minimum number of primary shards for the target index is 1. {: .note} \ No newline at end of file From f699149c3fd459991275c127d677b3b53e18545a Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Thu, 2 Feb 2023 15:04:20 -0500 Subject: [PATCH 8/8] Implemented editorial comments Signed-off-by: Fanit Kolchina --- _api-reference/index-apis/shrink-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_api-reference/index-apis/shrink-index.md b/_api-reference/index-apis/shrink-index.md index 24f1a0501c4..5d760a74013 100644 --- a/_api-reference/index-apis/shrink-index.md +++ b/_api-reference/index-apis/shrink-index.md @@ -65,7 +65,7 @@ settings | Object | Index settings you can apply to your target index. See [Inde The `max_shard_size` parameter specifies the maximum size of a primary shard in the target index. OpenSearch uses `max_shard_size` and the total storage for all primary shards in the source index to calculate the number of primary shards and their size for the target index. -The primary shard count of the target index is the smallest factor of the source index's primary shard count, for which the shard size does not exceed `max_shard_size`. For example, if the source index has 8 primary shards, they occupy a total of 400 GB of storage, and the `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: +The primary shard count of the target index is the smallest factor of the source index's primary shard count for which the shard size does not exceed `max_shard_size`. For example, if the source index has 8 primary shards, they occupy a total of 400 GB of storage, and the `max_shard_size` is equal to 150 GB, OpenSearch calculates the number of primary shards in the target index using the following algorithm: 1. Calculate the minimum number of primary shards as 400/150, rounded to the nearest whole integer. The minimum number of primary shards is 3. 1. Calculate the number of primary shards as the smallest factor of 8 that is greater than 3. The number of primary shards is 4.