Add a guardrail to limit maximum number of shard on the cluster by RS146BIJAY · Pull Request #6143 · opensearch-project/OpenSearch

RS146BIJAY · 2023-02-01T20:51:38Z

Signed-off-by: Rishav Sagar rissag@amazon.com

Description

We have observed that as shard count on the cluster increases, it becomes more prone to instability and availability. We have observed multiple instances where user had issues in the past because of too many shards on their cluster.

As of now, OpenSearch allows us to restrict the number of shards per node using setting cluster.max_shards_per_node inside ShardLimitValidator. The maximum number of shards that can be present on the cluster is determined by maxShardsPerNode * nodeCount. Issue here is, we can increase this total shard limit on the cluster just by increasing the number of node. There is no way we can control the maximum number of shards on an entire cluster.

I would suggest to add a similar setting inside ShardLimitValidator which will control the maximum number of shards that can be present on a cluster. If the total number of shards exceeds this threshold, ShardLimitValidator will block any operation which creates new shards. The threshold value will be an optional configurable cluster setting, which user can set dynamically.

Issues Resolved

#6050

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-02-01T20:57:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10418/
CommitID: 5fcf038
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-01T21:03:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10419/
CommitID: 0ee3948
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-01T21:10:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10422/
CommitID: 0d89de4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-01T21:40:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10423/
CommitID: ad53c45

server/src/main/java/org/opensearch/indices/ShardLimitValidator.java

Bukhtawar

Overall LGTM, ensure we handle min case of node and cluster limit

github-actions · 2023-02-02T10:09:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10466/
CommitID: 9e5a63e

codecov-commenter · 2023-02-02T10:10:52Z

Codecov Report

Merging #6143 (b1397cf) into main (c5a1bdf) will decrease coverage by 0.01%.
The diff coverage is 87.09%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6143      +/-   ##
============================================
- Coverage     70.87%   70.86%   -0.01%     
+ Complexity    58830    58823       -7     
============================================
  Files          4776     4776              
  Lines        280993   281020      +27     
  Branches      40598    40599       +1     
============================================
  Hits         199141   199141              
- Misses        65488    65557      +69     
+ Partials      16364    16322      -42

Impacted Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	`91.89% <ø> (ø)`
...va/org/opensearch/indices/ShardLimitValidator.java	`92.68% <87.09%> (-3.69%)`	⬇️
.../index/shard/IndexShardNotRecoveringException.java	`0.00% <0.00%> (-100.00%)`	⬇️
...adonly/AddIndexBlockClusterStateUpdateRequest.java	`0.00% <0.00%> (-75.00%)`	⬇️
...readonly/TransportVerifyShardIndexBlockAction.java	`9.75% <0.00%> (-58.54%)`	⬇️
.../admin/cluster/reroute/ClusterRerouteResponse.java	`60.00% <0.00%> (-40.00%)`	⬇️
...h/action/ingest/SimulateDocumentVerboseResult.java	`60.71% <0.00%> (-39.29%)`	⬇️
...indices/readonly/TransportAddIndexBlockAction.java	`20.68% <0.00%> (-37.94%)`	⬇️
...luster/routing/allocation/RoutingExplanations.java	`62.06% <0.00%> (-37.94%)`	⬇️
...cluster/routing/allocation/RerouteExplanation.java	`65.00% <0.00%> (-35.00%)`	⬇️
... and 453 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

server/src/main/java/org/opensearch/indices/ShardLimitValidator.java

ashking94 · 2023-02-02T15:53:41Z

server/src/main/java/org/opensearch/indices/ShardLimitValidator.java

+            final String inValidMaxShardSettingErrorMessage = "Max shard per cluster threshold "
+                + maxShardsInCluster
+                + "should be greater than or equal to max shard per node "
+                + maxShardsPerNode;
+            return Optional.of(inValidMaxShardSettingErrorMessage);


Curious on how would this behave since the node count is dynamic. Lets says initially max shard per node is 100 and there are 10 nodes with max shard per cluster as 1010. Now another node is added to the cluster in which case 1010 (max shard per cluster) is lesser than 1000*11. May be we can just put a condition as mentioned in this comment and remove this condition here? wdyt @sachinpkale @Bukhtawar @reta?

and maxShardsInCluster = Math.min(maxShardsInCluster, maxShardsPerNode * nodeCount) can stay as a single else block.

Addressed the comment.

github-actions · 2023-02-02T15:56:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10481/
CommitID: 367840e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/indices/ShardLimitValidator.java

github-actions · 2023-02-03T11:39:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10584/
CommitID: dd5e23d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishav Sagar <rissag@amazon.com>

github-actions · 2023-02-03T12:56:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10586/
CommitID: b1397cf

Bukhtawar

LGTM

Signed-off-by: Rishav Sagar <rissag@amazon.com> (cherry picked from commit e42b76f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

(cherry picked from commit e42b76f) Signed-off-by: Rishav Sagar <rissag@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

RS146BIJAY force-pushed the cluster-shard-limit-guardrail branch from 5fcf038 to 0ee3948 Compare February 1, 2023 20:54

RS146BIJAY force-pushed the cluster-shard-limit-guardrail branch 2 times, most recently from 0d89de4 to ad53c45 Compare February 1, 2023 21:09

RS146BIJAY marked this pull request as ready for review February 2, 2023 05:36

sachinpkale reviewed Feb 2, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/ShardLimitValidator.java Show resolved Hide resolved

Bukhtawar reviewed Feb 2, 2023

View reviewed changes

RS146BIJAY force-pushed the cluster-shard-limit-guardrail branch from ad53c45 to 9e5a63e Compare February 2, 2023 09:36

sachinpkale approved these changes Feb 2, 2023

View reviewed changes

reta reviewed Feb 2, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/ShardLimitValidator.java Show resolved Hide resolved

RS146BIJAY force-pushed the cluster-shard-limit-guardrail branch from 9e5a63e to 367840e Compare February 2, 2023 15:25

ashking94 reviewed Feb 2, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/ShardLimitValidator.java Show resolved Hide resolved

RS146BIJAY force-pushed the cluster-shard-limit-guardrail branch from 367840e to dd5e23d Compare February 3, 2023 11:11

Add setting to limit total number of shards on a cluster

b1397cf

Signed-off-by: Rishav Sagar <rissag@amazon.com>

RS146BIJAY force-pushed the cluster-shard-limit-guardrail branch from dd5e23d to b1397cf Compare February 3, 2023 12:25

Bukhtawar approved these changes Feb 3, 2023

View reviewed changes

reta approved these changes Feb 3, 2023

View reviewed changes

Bukhtawar approved these changes Feb 5, 2023

View reviewed changes

Bukhtawar merged commit e42b76f into opensearch-project:main Feb 5, 2023

Bukhtawar added the backport 2.x Backport to 2.x branch label Feb 5, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 5, 2023

[Backport 2.x] Add a guardrail to limit maximum number of shard on the cluster #6190

Merged

ashking94 mentioned this pull request Feb 10, 2023

[BUG] Flaky test ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting #6287

Closed

nknize mentioned this pull request May 22, 2023

[Refactor] OpenSearchException streamables to a registry #7646

Merged

Conversation

RS146BIJAY commented Feb 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

github-actions bot commented Feb 1, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Feb 1, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Feb 1, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Feb 1, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

Bukhtawar left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 2, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov-commenter commented Feb 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

ashking94 Feb 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashking94 Feb 2, 2023

Choose a reason for hiding this comment

Uh oh!

RS146BIJAY Feb 3, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 2, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

github-actions bot commented Feb 3, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Feb 3, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Bukhtawar left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Comments

RS146BIJAY commented Feb 1, 2023 •

edited

Loading

codecov-commenter commented Feb 2, 2023 •

edited

Loading

ashking94 Feb 2, 2023 •

edited

Loading