Implement poisoned shard detection and handling in AzureStorageJobShardManager

[benjaminpetit]

Fix #9900

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

Implements poisoned shard detection for durable job shard managers by tracking how often shards are “stolen” from dead silos and preventing further assignment once a configurable threshold is exceeded, addressing issue #9900 for Azure Storage and the in-memory implementation.

Changes:

• Added DurableJobsOptions.MaxStolenCount to configure the poisoned-shard threshold.
• Updated AzureStorageJobShardManager and InMemoryJobShardManager to track “stolen count” and skip assignment once poisoned.
• Added InMemoryJobShardManagerTests covering orphaned/stolen/poisoned shard assignment scenarios.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

File	Description
test/NonSilo.Tests/DurableJobs/InMemoryJobShardManagerTests.cs	Adds tests for orphaned vs stolen shard reassignment and poison-threshold behavior.
src/Orleans.DurableJobs/JobShardManager.cs	Extends InMemoryJobShardManager with stolen-count tracking and poison-threshold logic.
src/Orleans.DurableJobs/Hosting/DurableJobsOptions.cs	Introduces MaxStolenCount option and documentation describing the behavior.
src/Azure/Orleans.DurableJobs.AzureStorage/AzureStorageJobShardManager.cs	Adds Azure blob metadata tracking (StolenCount, LastStolenTime) and poison-threshold enforcement.

Comments suppressed due to low confidence (1)

src/Orleans.DurableJobs/JobShardManager.cs:218

• These 'if' statements can be combined.

                if (isOrphaned || isFromDeadSilo)
                {
                    if (ownership.Shard.StartTime <= maxDueTime)
                    {
                        // If stolen from dead silo, increment stolen count
                        if (isFromDeadSilo)
                        {
                            ownership.StolenCount++;

                            // Check if shard is poisoned
                            if (ownership.StolenCount > _maxStolenCount)
                            {
                                // Shard is poisoned - don't assign it
                                continue;
                            }
                        }

                        ownership.OwnerSiloAddress = SiloAddress.ToString();
                        stolenShards.Add(ownership.Shard);
                    }
                }