Return NOT_PREFERRED decisions in allocation explain by DiannaHohensee · Pull Request #137228 · elastic/elasticsearch

DiannaHohensee · 2025-10-28T00:35:35Z

Adds numerous NOT_PREFERRED options to allocation decision / status types.

Adds NOT_PREFERRED option to AllocationDecision (resolving ES-12729).

A significant change is to re-order comparison of Decision.Type enum
values, such that THROTTLE is chosen over NOT_PREFERRED. Functionally
this change should not matter because simulation (DesiredBalance
computation) does not throttle and reconciliation (real shard movement)
treats not-preferred essentially as a YES: they are not compared.

Closes ES-12833, ES-13288, ES-12729

Adds numerous NOT_PREFERRED options to allocation decision / status types. Adds NOT_PREFERRED option to AllocationDecision (resolving ES-12729). Closes ES-12833, ES-13288, ES-12729

elasticsearchmachine · 2025-11-08T02:29:21Z

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

DiannaHohensee · 2025-11-08T02:33:39Z

Adding @DaveCTurner as a reviewer in case he wants to take a high level look for anything I'm missing -- I saw in git-blame that he wrote the Explanations.java file a while ago. But optional.

DiannaHohensee · 2025-11-08T02:38:01Z

I think I might be missing test coverage (if not functionality) of the allocate unassigned code path. However, I've been hacking on the rebalancing code path for a while now to get this all to work sensibly, so if agreeable I might just file another ticket to explore that.

.../java/org/elasticsearch/cluster/routing/allocation/decider/WriteLoadConstraintDeciderIT.java

.../java/org/elasticsearch/action/admin/cluster/allocation/ClusterAllocationExplainRequest.java

nicktindall · 2025-11-09T23:32:07Z

...ain/java/org/elasticsearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java

                    }
                    if (allocationDecision.type().higherThan(bestDecision)) {
+                        assert allocationDecision.type() != Type.THROTTLE
+                            : "DesiredBalance computations run in a simulation mode and should not encounter throttling";


Sorry, how do we know we're simulating here?

I added this as extra protection because I changed the Decision.Type enum ordering, such that the higherThan gate above would now prefer (return true for) THROTTLE over NOT_PREFERRED.

Because this is simulation, we should never see THROTTLE.

I updated this. Tripped an explain test, because explain uses the decideMove path, too.

I moved the !Throttle check higher up next to the canAllocate call, and added || !(IsSimulating)

nicktindall · 2025-11-09T23:37:45Z

...r/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/AllocationDeciders.java

+            assert decider.canRebalance(shardRouting, allocation).type() != Decision.Type.THROTTLE
+                : decider.getClass().getSimpleName() + " throttled unexpectedly in canRebalance";
+            return decider.canRebalance(shardRouting, allocation);
+        }, (decider, decision) -> Strings.format("Can not rebalance [%s]. [%s]: %s", shardRouting, decider, decision));


ConcurrentRebalanceAllocationDecider appears to return THROTTLE from canRebalance ?

Nit: I'd prefer if we stored the result of canRebalance then ran the assertion after it rather than running canRebalance twice?

You're right, reverted 👍

nicktindall · 2025-11-09T23:54:38Z

server/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/Decision.java

        NOT_PREFERRED,
+        // Temporarily throttled is a better choice than choosing a not-preferred node,
+        // but NOT_PREFERRED and THROTTLED are generally not comparable.
+        THROTTLE,


I don't think we can do this can we? It'll mean in the presence of a THROTTLE and a NOT_PREFERRED, canAllocate will return NOT_PREFERRED, which I think is bad for non-desired-balance allocation, because there we should wait for a THROTTLE to become a yes before allocating to a NOT_PREFERRED?

I don't believe there's currently any direct comparison between THROTTLE and NOT_PREFERRED. But we still need to specify an ordering here, to not break how one or the other gets compared to YES and NO.

It'll mean in the presence of a THROTTLE and a NOT_PREFERRED, canAllocate will return NOT_PREFERRED

I think this would be the other way around, THROTTLE would be higher value, closer to YES, and chosen over NOT_PREFERRED. So, IIUC, we both want it the current way.

I've a little blurb for the commit message:

"A significant change is to re-order comparison of Decision.Type enum
values, such that THROTTLE is chosen over NOT_PREFERRED. Functionally
this change should not matter because simulation (DesiredBalance
computation) does not throttle and reconciliation (real shard movement)
treats not-preferred essentially as a YES: they are not compared."

Ah yeah I think it all makes sense now.

Though in reconciler in allocateUnassigned we still don't handle NOT_PREFERRED but there's a ticket to fix that, and I don't think it's important.

...r/src/main/java/org/elasticsearch/cluster/routing/allocation/AllocateUnassignedDecision.java

server/src/main/java/org/elasticsearch/cluster/routing/allocation/Explanations.java

nicktindall · 2025-11-10T00:06:37Z

server/src/main/java/org/elasticsearch/cluster/routing/allocation/MoveDecision.java

            }
+            // TODO (ES-13482): clusterRebalanceDecision is set to the result of AllocationDecider#canRebalance, which does not return
+            // NOT_PREFERRED or THROTTLE. This switch statement, and how MoveDecision uses clusterRebalanceDecision, should be
+            // refactored.


I think it still does? see org/elasticsearch/cluster/routing/allocation/decider/ConcurrentRebalanceAllocationDecider.java:182

Looks like Simon committed some changes concurrently with my PR.

But I think I was wrong, anyway: I didn't realize the complexities of canRebalance.

Removed 👍 Thanks!

…decideMove

DiannaHohensee

Made a logic change / bug fix, per my comment. Your assertion suggestion got me rethinking.

server/src/main/java/org/elasticsearch/cluster/routing/allocation/MoveDecision.java

nicktindall · 2025-12-16T04:56:15Z

...ain/java/org/elasticsearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java

                        } else if (bestDecision == Type.NOT_PREFERRED) {
-                            assert remainDecision.type() != Type.NOT_PREFERRED;
+                            assert remainDecision.type() != Type.NOT_PREFERRED || allocation.isSimulating() == false;
                            // If we don't ever find a YES decision, we'll settle for NOT_PREFERRED as preferable to NO.


Suggested change

// If we don't ever find a YES decision, we'll settle for NOT_PREFERRED as preferable to NO.

// If we don't ever find a YES/THROTTLE decision, we'll settle for NOT_PREFERRED as preferable to NO.

Applied 👍

nicktindall · 2025-12-16T05:02:44Z

...ain/java/org/elasticsearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java

                            }
                        } else if (bestDecision == Type.NOT_PREFERRED) {
-                            assert remainDecision.type() != Type.NOT_PREFERRED;
+                            assert remainDecision.type() != Type.NOT_PREFERRED || allocation.isSimulating() == false;


This assertion is a bit confusing

assert remainDecision.type() != Type.NOT_PREFERRED || allocation.isSimulating() == false;

Do we need that isSimulating() == false clause? It looks like we should never get here when

allocationDecision.type() == Type.NOT_PREFERRED && remainDecision.type() == Type.NOT_PREFERRED

Or have I missed something?

You're right, that scenario is short circuited on L1049.
I forgot and got !isSimulating-happy after adding throttle below for the explain path.

nicktindall

Changes LGTM, a couple of questions

Relates: elastic#137228

Relates: #137228

Relates: elastic#137228

Relates: #137228

Relates: elastic#137228

DiannaHohensee self-assigned this Oct 28, 2025

elasticsearchmachine added the v9.3.0 label Oct 28, 2025

DiannaHohensee force-pushed the 2025/10/23/ES-12833 branch 4 times, most recently from b03f11f to be58da7 Compare October 30, 2025 20:21

DiannaHohensee force-pushed the 2025/10/23/ES-12833 branch 6 times, most recently from 13322ec to 17de33c Compare November 8, 2025 02:07

Return NOT_PREFERRED decisions in allocation explain

4a0ee8a

Adds numerous NOT_PREFERRED options to allocation decision / status types. Adds NOT_PREFERRED option to AllocationDecision (resolving ES-12729). Closes ES-12833, ES-13288, ES-12729

DiannaHohensee force-pushed the 2025/10/23/ES-12833 branch from 17de33c to 4a0ee8a Compare November 8, 2025 02:19

DiannaHohensee changed the title ~~[DRAFT] Return NOT_PREFERRED decisions in allocation explain~~ Return NOT_PREFERRED decisions in allocation explain Nov 8, 2025

DiannaHohensee marked this pull request as ready for review November 8, 2025 02:28

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Nov 8, 2025

DiannaHohensee requested review from DaveCTurner and nicktindall November 8, 2025 02:29

improve assert message

7165744

nicktindall reviewed Nov 10, 2025

View reviewed changes

DiannaHohensee added 3 commits December 15, 2025 09:45

add assertion to MoveDecision#move; clear targetNode on THROTTLE in #…

9f0fee3

…decideMove

Merge branch 'main' into 2025/10/23/ES-12833

f2d429f

fix transport version after merge

0705c5e

DiannaHohensee commented Dec 15, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/cluster/routing/allocation/MoveDecision.java Show resolved Hide resolved

DiannaHohensee added 4 commits December 15, 2025 15:36

fix assertion...

0353019

actually fixed test?

bbf524d

Merge branch 'main' into 2025/10/23/ES-12833

91c6b4c

transport version fix

4773a4f

nicktindall reviewed Dec 16, 2025

View reviewed changes

nicktindall approved these changes Dec 16, 2025

View reviewed changes

DiannaHohensee added 3 commits December 16, 2025 10:49

fix msg and remove isSimulating non-existent scenario

aac1157

Merge branch 'main' into 2025/10/23/ES-12833

1ff02aa

fix transport version

29f813e

DiannaHohensee added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Dec 16, 2025

elasticsearchmachine merged commit 95b0836 into elastic:main Dec 16, 2025
35 checks passed

DiannaHohensee deleted the 2025/10/23/ES-12833 branch December 16, 2025 20:08

This was referenced Jan 5, 2026

Revert "Return NOT_PREFERRED decisions in allocation explain (#137228)" #140165

Closed

Revert "Return NOT_PREFERRED decisions in allocation explain (#137228)" #140175

Closed

DiannaHohensee mentioned this pull request Jan 5, 2026

Undo not-preferred in allocation explain #140195

Closed

This was referenced Jan 6, 2026

Fix Decision.Type serialization BWC #140199

Merged

Allow relocation to NOT_PREFERRED node for evacuating shards #140197

Merged

ywangd mentioned this pull request Jan 7, 2026

Overall Decision for Deciders prioritizes THROTTLE #140237

Merged

ywangd added a commit to ywangd/elasticsearch that referenced this pull request Jan 12, 2026

Fix transport serialization for AllocationDecision

0905f94

Relates: elastic#137228

ywangd mentioned this pull request Jan 12, 2026

Fix transport serialization for AllocationDecision #140500

Merged

elasticsearchmachine pushed a commit that referenced this pull request Jan 13, 2026

Fix transport serialization for AllocationDecision (#140500)

e471ba2

Relates: #137228

ywangd added a commit to ywangd/elasticsearch that referenced this pull request Jan 13, 2026

Fix transport serialization for AllocationDecision (elastic#140500)

05d9c9d

Relates: elastic#137228

elasticsearchmachine pushed a commit that referenced this pull request Jan 13, 2026

Fix transport serialization for AllocationDecision (#140500) (#140532)

32fef86

Relates: #137228

eranweiss-elastic pushed a commit to eranweiss-elastic/elasticsearch that referenced this pull request Jan 15, 2026

Fix transport serialization for AllocationDecision (elastic#140500)

5ff7b36

Relates: elastic#137228

spinscale pushed a commit to spinscale/elasticsearch that referenced this pull request Jan 21, 2026

Fix transport serialization for AllocationDecision (elastic#140500)

789e567

Relates: elastic#137228

	// If we don't ever find a YES decision, we'll settle for NOT_PREFERRED as preferable to NO.
	// If we don't ever find a YES/THROTTLE decision, we'll settle for NOT_PREFERRED as preferable to NO.

Comments

Conversation

DiannaHohensee commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Nov 8, 2025

Uh oh!

DiannaHohensee commented Nov 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DiannaHohensee commented Nov 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DiannaHohensee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicktindall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DiannaHohensee commented Oct 28, 2025 •

edited

Loading

DiannaHohensee commented Nov 8, 2025 •

edited

Loading

DiannaHohensee commented Nov 8, 2025 •

edited

Loading