fix(runtime): preserve rebalancer state on shutdown

[ReubenBond]

Problem

During graceful silo shutdown, the activation rebalancer monitor requested migration of its worker but did not wait for the asynchronous migration to finish. Ordinary grain deactivation could therefore overtake the handoff, causing the worker to reactivate without its serialized state and leaving tests dependent on stale monitor reports and timing.

Solution

Move the rebalancer handoff to a lifecycle stage immediately before ordinary grain deactivation and await the worker's deactivation/migration completion. Update the regression test to synchronize initial placement and rebalancing cycles explicitly, verify the dead host is no longer used, and compare the preserved report state after migration. Placement hints are now cleared after test activation creation so they cannot leak into later calls.

Rationale

The migration manager and grain directory are still available at this stage, while the catalog has not yet begun bulk deactivation. Awaiting the handoff makes graceful shutdown ordering deterministic and preserves the worker's migration state instead of relying on retries or longer delays.

Fixes #10570

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

This PR makes activation rebalancer shutdown deterministic by migrating the rebalancer worker earlier in the silo shutdown lifecycle and awaiting the worker’s deactivation/migration so its serialized state is preserved, addressing a flaky rebalancing state preservation test.

Changes:

• Move rebalancer migration to a lifecycle stage immediately before ordinary grain deactivation and await the worker activation’s deactivation.
• Update the state-preservation regression test to use event-driven synchronization for rebalancing cycles and validate report/state preservation post-migration.
• Ensure placement hint RequestContext is always cleared after test activation creation to prevent leakage between calls.

Show a summary per file

File	Description
test/Orleans.Placement.Tests/ActivationRebalancingTests/StatePreservationRebalancingTests.cs	Refactors the regression test to coordinate on diagnostic events, validates migration/state preservation, and introduces helper for deterministic rebalancer relocation.
test/Orleans.Placement.Tests/ActivationRebalancingTests/RebalancingTestBase.cs	Wraps placement hint usage in try/finally to avoid RequestContext leakage during test activation creation.
src/Orleans.Runtime/Placement/Rebalancing/ActivationRebalancerMonitor.cs	Adds an earlier shutdown hook to migrate the rebalancer worker and await completion before bulk grain deactivation proceeds.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

test/Orleans.Placement.Tests/ActivationRebalancingTests/StatePreservationRebalancingTests.cs:145

• MigrateOnIdle() initiates migration but does not wait for the source activation to deactivate/migrate. The immediate GetReport()/Assert.Equal can therefore race and intermittently observe the old host. Capture the current activation’s deactivation task before requesting migration and await it (similar to TestCluster.MigrateAsync/WaitForDeactivationAsync).

        RequestContext.Set(IPlacementDirector.PlacementHintKey, targetHost);
        try
        {
            await rebalancer.Cast<IGrainManagementExtension>().MigrateOnIdle();
        }

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

test/Orleans.Placement.Tests/ActivationRebalancingTests/StatePreservationRebalancingTests.cs:152

• MigrateOnIdle() only initiates migration (it calls Migrate(...) without awaiting deactivation/rehydration), so immediately asserting the new host here can reintroduce timing-based flakiness. Consider waiting (with a bounded timeout) until GetReport().Host actually reflects targetHost, and perform the assert on the final attempt for a clearer failure message.

        Assert.Equal(targetHost, (await rebalancer.GetReport()).Host);

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite