Skip to content

Create the saga feature per attempt so a retried saga does not reuse a disposed store - #10362

Closed
alisan3 wants to merge 1 commit into
ChilliCream:mainfrom
alisan3:ali/mocha-saga-feature-per-attempt
Closed

alisan3 wants to merge 1 commit into
ChilliCream:mainfrom
alisan3:ali/mocha-saga-feature-per-attempt

Conversation

@alisan3

@alisan3 alisan3 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Problem

When a saga consume fails and is retried in process, every retry throws ObjectDisposedException from a disposed DbContext in DbContextSagaStore.StartTransactionAsync, before the saga runs — whenever any earlier saga consume has already run in the process. The retry cannot succeed, the in-process budget is exhausted in microseconds, and the message waits for delayed redelivery. Nothing is logged for a handler fault, so the only symptom is a saga that stops progressing.

This is the residue of #10327. That change gives each retry its own scope so saga-state contention can resolve, but the retry that is supposed to resolve it is the thing that faults. Retry policy is irrelevant: 25 attempts behave exactly like 3, because every attempt after the first is dead on arrival.

Live stack, captured in a first-chance handler against Postgres:

Microsoft.EntityFrameworkCore.DbContext.CheckDisposed()
Microsoft.EntityFrameworkCore.DbContext.get_Database()
Mocha.Sagas.EfCore.DbContextSagaStore.StartTransactionAsync(CancellationToken)
Mocha.Sagas.SagaConsumer.ConsumeAsync(IConsumeContext)
Mocha.Inbox.ConsumeInboxMiddleware.InvokeAsync(IConsumeContext, ConsumerDelegate)

Root cause

Four pieces, each individually intended, that combine into a shared mutable feature holding a scope-bound service.

1. Attempt clones read the receive context's features through a fallback. ConsumerRetryMiddleware.ExecuteAttemptAsync rents a pooled ReceiveContext per attempt, and ReceiveContext.CopyTo installs the outer context's collection as the clone's _defaults (clone._features.Initialize(_features)). PooledFeatureCollection.TryGet checks the clone's own entries and then falls through, so GetOrSet<T>() on the clone returns the outer context's instance whenever the outer has one. This is what lets the parsed message be shared across attempts.

2. SagaConsumer mutates whatever the fallback returns, with the attempt's scoped store.

var sagaFeature = context.Features.GetOrSet<SagaFeature>();
sagaFeature.Store ??= context.Services.GetRequiredService<ISagaStore>();

context.Services is the attempt's scope. If GetOrSet returned the outer's SagaFeature, the attempt's DbContextSagaStore — and its scoped DbContext — is now pinned on an object that outlives the attempt.

3. The outer context has a SagaFeature whenever its pooled instance previously served as a saga attempt clone. SagaFeature : IPooledFeature, so when a clone is returned, PooledFeatureCollection.Reset() parks it (Store nulled). When that instance is later rented as an outer receive context, ReceiveContext.Initialize() calls _features.Initialize() with no defaults, which restores every parked feature. A SagaFeature with Store == null now sits exactly where the next clone's fallback will find it.

4. Attempt 1 fails and its scope is disposed; attempt 2 reads the same feature. ??= is a no-op because Store is set — to the store from the disposed scope.

#10327's description anticipated this: "SagaConsumer caches the saga store on SagaFeature; without a reset a retry would reuse a store from the disposed scope. IScopeBoundFeature marks features that hold a scoped service; SagaFeature is the one implementor and is reset before and after the attempt." The merged revision has neither IScopeBoundFeature nor that reset. Both went when the design moved from swapping context.Services on one context to running each attempt on a pooled clone — and the clone's fallback to the receive context is what makes the cached store reachable again.

Why the existing guard did not catch it

SagaRetryScopeTests.Saga_Should_UseFreshStore_When_SaveIsRetried publishes one saga message on a fresh pool. No pooled instance has ever been a saga clone, the outer has no SagaFeature, each attempt creates its own on the clone, and the test passes without exercising the fallback. The same blind spot produced a false "requires concurrency" conclusion downstream: concurrency only makes the precondition in (3) near-certain and supplies the first failure via version conflicts. A single-threaded reproduction with one warm-up saga consume produced 294 disposals.

Fix

SagaFeature stops being pooled, and SagaConsumer creates it on the attempt instead of reading through the fallback.

// SagaConsumer.ConsumeAsync
var sagaFeature = new SagaFeature { Store = context.Services.GetRequiredService<ISagaStore>() };
context.Features.Set(sagaFeature);

Saga.cs reads the store through GetSagaFeature() → GetOrSet<SagaFeature>(), which finds the attempt's own entry first, so no reader changes. With IPooledFeature gone the feature is dropped on Reset() rather than parked, so a receive context can never carry one and the fallback can never surface one — the hazard is removed rather than patched around.

The alternative of keeping the pooled feature and making the assignment unconditional (Store =) also passes the new test, but leaves a shared, mutable, scope-holding feature in the fallback path. #10327 kept ??= because "several test harnesses inject a store through the feature and rely on the cache"; all five of those sites (SagaTester, SagaExecutionTests, SagaStateMachineTests, SagaSchedulingTests, SagaTimeoutTests) call saga.HandleEvent directly and never go through SagaConsumer, so that constraint does not apply here.

SagaFeature is the only IPooledFeature in the repository that holds a scope-resolved service; the others carry flags, the consumer set or the parsed message, all meant to be shared across attempts.

Behavior change

  • SagaFeature no longer implements IPooledFeature; its Initialize/Reset members are removed. Public type, no in-repo callers outside the collection machinery.
  • SagaConsumer resolves ISagaStore from the current attempt's scope on every consume. A SagaFeature.Store pre-seeded on the consume context is no longer honoured by SagaConsumer — only by Saga.HandleEvent called directly, which is how every existing harness does it.

Tests

SagaRetryScopeWarmupTests is SagaRetryScopeTests plus one warm-up saga consume before the message whose first save throws, with store ids recorded only once armed. The saga ends in a non-final state so the persisted count is observable.

Test Without fix With fix
Saga retried after a failed save uses a fresh store, after an earlier saga consume (in-memory) 1 store (Expected: 2, Actual: 1) 2 stores, state persisted
SagaRetryScopeTests (existing, single message) pass pass

Mocha.Sagas.Tests: 154 total, 0 failed, 1 pre-existing skip. Not run here: the EF, Postgres and RabbitMQ suites. Downstream, the same defect reproduced end-to-end against Postgres and RabbitMQ (626 ObjectDisposedException against 280 genuine version conflicts in an 11-message fan-in; 0 disposals with in-process retries disabled).

Related

@PascalSenn

Copy link
Copy Markdown
Member

resolved in #10363

@PascalSenn PascalSenn closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants