Fix ancillary store scheduled messages stuck Incoming forever (#2576)

[jeremydmiller]

Fixes #2576.

The bug

Reporter set up two Marten stores (main + ancillary), wrote a handler chain entirely owned by the ancillary store via [MartenStore(typeof(IAncillaryStore))], and had one of the handlers return ScheduledMessage<T>. After the scheduled message fired and its handler succeeded, the row stayed Status == Incoming in the ancillary store forever — the "mark as Handled" SQL was hitting the main store instead.

Why

The path:

PollForScheduledMessagesAsync (per-store)
  → loaded envelopes hand-off
  → runtime.EnqueueDirectlyAsync(envelopes)
  → ListeningAgent → DurableReceiver.EnqueueAsync(envelope)
  → handler runs, succeeds
  → DurableReceiver._markAsHandled
  → DelegatingMessageInbox.MarkIncomingEnvelopeAsHandledAsync
       (uses envelope.Store?.Inbox ?? _inner)
  → _inner == main store ✗

envelope.Store was never stamped, so dispatch fell through to the main store.

There's an existing message-type-to-ancillary-store map (MessageStoreCollection.MapMessageTypeToAncillaryStore) that should have populated envelope.Store upstream, but it's empty at runtime: it's built from chain.AncillaryStoreType, which is set by MartenStoreAttribute.Modify — and that attribute application is lazy, deferred until first handler use. So at startup, no chains have AncillaryStoreType set yet, the map is empty, and the lookup returns null. (This was also gated to MultipleHandlerBehavior.Separated, which the reporter happened to use.)

The fix (Option B from the analysis on the issue)

Each MessageDatabase.PollForScheduledMessagesAsync now stamps envelope.Store = this on every loaded envelope before handing them to runtime.EnqueueDirectlyAsync. The store knows itself; this sidesteps the type-name map entirely and survives any combination of attribute-application timing or handler-config behavior.

This mirrors the existing pattern in RecoverIncomingMessagesCommand.cs:48 (Bug-2318 fix for the recovery path).

Applied to: Postgres, SqlServer, MySql, Sqlite, Oracle.

Defense-in-depth changes:

• DurableReceiver.Enqueue / EnqueueAsync now also call assignAncillaryStoreIfNeeded (with a guard that won't overwrite a Store already set by Option B). Catches edge cases where envelopes enter the queue without going through the poll path.
• WolverineRuntime.HostService now uses Handlers.AllChains() instead of Handlers.Chains when populating the type-to-store map, so per-endpoint sticky chains under MultipleHandlerBehavior.Separated are included once attribute application happens earlier.

DLQ recovery — verified safe

Asked to also check the DLQ replay path:

1. DeadLetters.ReplayAsync flips replayable=true in the dead_letter table (per store).
2. Per-store DurabilityAgent runs MoveReplayableErrorMessagesToIncomingOperation, which is pure SQL within the same database — moves rows from dead_letter → incoming table.
3. RecoverIncomingMessagesCommand.ExecuteAsync already does envelope.Store ??= _store; (line 48, comment references Replayed Dead Letter messages from Ancillary Store are marked as processed in the Main Store #2318) before EnqueueDirectlyAsync.

So the DLQ replay → recovery cycle was already correct because of the prior GH-2318 fix. Added a sister contract test (replayed_dead_letter_recovery_stamps_envelope_with_originating_store) to pin that down so it can't accidentally regress alongside this fix.

Tests

New reproducer (MartenTests/Bugs/Bug_2576_ancillary_scheduled_message_stuck_incoming.cs, added in the prior commit on this branch). Now passes.

Two new contract tests in MessageStoreCompliance:

• scheduled_poll_stamps_envelope_with_originating_store — proves Option B's contract for any IMessageDatabase implementation.
• replayed_dead_letter_recovery_stamps_envelope_with_originating_store — pins the DLQ recovery contract so future refactors can't strip it.

Both use a Substitute.For<IWolverineRuntime>() spy on EnqueueDirectlyAsync to capture the in-memory envelope.

Required adding a project reference from Wolverine.ComplianceTests to Wolverine.RDBMS (so the contract test can refer to IMessageDatabase), and InternalsVisibleTo("Wolverine.ComplianceTests") from Wolverine.RDBMS. Stores that don't implement IMessageDatabase (RavenDb, CosmosDb) early-return from the contract test — they wire scheduled dispatch through their own durability agents.

Verification

• 4/4 Postgres compliance variants pass both new contract tests
• 1/1 Sqlite compliance passes
• Full PostgresqlTests: 365/365 pass
• Full SqliteTests: 117/118 (1 pre-existing flake — multi_tenancy_with_multiple_files.scheduled_messages_are_processed_in_tenant_files — confirmed to also fail on stashed-clean state, unrelated to this change)
• MartenTests Bug_2576 + Bug_2382 + Bug_2318 + Bug_2026: 6/6 pass

Test plan

• Reproducer test fails on main, passes after the fix
• New compliance contract tests pass for all RDBMS-backed stores
• Existing PostgresqlTests pass without regression
• DLQ recovery path verified to already be correct (Replayed Dead Letter messages from Ancillary Store are marked as processed in the Main Store #2318 fix)

🤖 Generated with Claude Code