Skip to content

Durable inbox hardening, part 1: transactions, idempotency, deduplication, GH-5123 and the CISqlServer split - #5129

Merged
jeremydmiller merged 49 commits into
mainfrom
durable-hardening
Oct 22, 2026
Merged

jeremydmiller merged 49 commits into
mainfrom
durable-hardening

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #5123.

The first of two coarse PRs from a durable inbox compliance effort. A new Bobcat rig (src/Testing/DurableComplianceRig, outside both solutions and CI) exercised the durable inbox end to end across PostgreSQL, SQL Server, Marten and EF Core over local, database, RabbitMQ and SQS endpoints. Every fix here has a regression test in a normal CI test project that was red before the fix and green after, and the rig specs that found it now pass. Groups A and B (listener lifecycle and durable batching) follow in a second PR.

Transactions and idempotency

  • R1: a handler using the documented outbox pattern on PostgreSQL could kill another handler's transaction. MessageContext.ClearState() disposed the enlisted DatabaseEnvelopeTransaction, which disposed a DbTransaction its owner had already finished, and Npgsql reuses that object on the pooled connector. DatabaseEnvelopeTransaction no longer disposes a transaction it does not own.
  • R2 / E1: eager idempotency committed a "handled" placeholder before the handler ran. A failed handler that was requeued came back as a duplicate of itself and was dead-lettered, and a rollback left the envelope flags set, so idempotency was lost after a retry. The placeholder is now written inside the handler's own transaction (Marten, EF Core, Polecat, Fisher), and the flags are restored on rollback.
  • F25: EF Core set Status = Handled before its own commit, so a failed commit followed by Discard left the row Incoming. The status is now set only after the commit succeeds.
  • E3: EF Core with a database per tenant never marked the inbox row inside the tenant transaction. It does now.
  • F13: a handler that called ReScheduleAsync and then returned normally had its scheduled row overwritten to Handled, losing the retry. It isn't now.

Logical deduplication

  • D1: on the plain RDBMS stores, a [Deduplicated] claim stranded by a crash mid-handler made inbox recovery discard the message for the whole window. Claims now record the envelope that took them (envelope_id, a new nullable column), so the same envelope can take its claim over and a different one is still refused.
    • This comes through a new opt-in IEnvelopeAwareDeduplicationStore; IDeduplicationStore is unchanged.
    • A table without the new column keeps the old behaviour and logs one warning, so upgrading never breaks an app running AutoCreate.None.
  • D2: Envelope.DeduplicationId did not cross RabbitMQ, or any broker built on the generic EnvelopeMapper or the MQTT mappers. Those mappers now map it.

GH-5123

  • Duplicates inside one batch: two copies of one envelope in a single batch made the RDBMS stores rethrow the raw provider exception. A single store then paused the listener, and a store per tenant tripped the tenant write brake and looped. The batch insert now reports it as DuplicateIncomingEnvelopeException, as RavenDB already did, across PostgreSQL, SQL Server, MySQL, SQLite and Oracle. Oracle's batch insert, which committed everything but the failing statement, is now all-or-nothing.
  • Quorum queues: on RabbitMQ 4.2.5 a requeueing reject, a requeueing nack and a channel close all spend the delivery limit, so switching to nack, as suggested in the issue, would not help. Added:
    • an opt-in x-delivery-limit for queues Wolverine declares: .DeliveryLimit(n) per listener, DeliveryLimitForDurableQuorumQueues(n) per transport. Nothing changes by default;
    • a startup warning for a durable listener on a quorum queue that Wolverine declared without a limit;
    • a new docs section, "Quorum Queues and the Durable Inbox".
  • Defer ordering: RabbitMqEnvelope.DeferAsync now publishes the copy before it acks the original.

CI

  • CISqlServer had outgrown the 20 minute cap: 658 tests and about 2300s of test time, against 405 tests and 1367s on Aug 26. It is now split by namespace like CIPolecat: CISqlServerTransport, plus CISqlServer as the catch-all.
  • SqlServerMessageStore_with_IdAndDestination_Identity dropped and rebuilt its schema before every test (314s in CI). It now does that once per process.

Local verification

  • dotnet build wolverine.slnx -c Release -f net9.0: clean.
  • The full projects each lane touched passed. The only failures were known flakes that passed on rerun.
  • The combined branch's local gate finished CoreTests (3754/0) and PostgresqlTests (909 passed, 2 flakes that passed on rerun). It was then stopped to free the machine, and CI runs the rest.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U

jeremydmiller and others added 30 commits October 22, 2026 07:28
…invariant checker

A standalone test project (outside both solutions and CI) that exercises the durable inbox end to end.
Handlers write a ledger row inside their own transaction; after each scenario the invariants are
judged one by one against the store: lost messages, duplicate effects, concurrent runs of one
message, dead letter mismatches, unsettled inbox rows, outbox leftovers, and whether every envelope
carried its message's id (which the row-level checks depend on).

First slice: the settling scenarios on PostgreSQL over a durable local queue, a PostgreSQL queue,
RabbitMQ and SQS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…ures

SqlServerStore reuses RdbmsLedger; EfCoreStore maps LedgerEntry onto rig_ledger and
lets the EF Core transactional middleware commit, with an interceptor finishing each
execution at commit/rollback. InvariantGlobalization off: SqlClient refuses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Killable PostgreSQL store (a login role per node, cut on kill), an inbox-row
blip, an inbox-row watch, and specs for graceful stop, kill, InboxStaleTime,
F15 (unlatched pause), F16 (listener rebuild) and F12 (blip during a
scheduled retry). Execution.Finish is now first-wins so a killed run stays
killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…eSQL

Batch members judged by the same invariants as single messages, over a durable
local batch queue and over RabbitMQ/SQS durable listeners feeding the default
buffered batch queue (plus a durable batch queue variant). Error policies,
ProbeIndividuallyAfter, IsolateBatchMembers, ApplyItemException, and specs for
ledger leads F1-F6 and F8. Extra checks: batch pending counts, and errors
logged settling members.

RdbmsLedger: a member's scripted failure now finishes the members begun before
it, and the RigItem[] handler can throw ApplyItemException for a scripted poison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
F25 (failed commit after the in-transaction inbox mark strands the row and
re-runs a discarded message on restart) and EF idempotency on buffered/inline
endpoints (a retry loses the Handled row, so a redelivery runs again), each with
a diagnostic switch that removes the failure (RIG_EF_UNDO_HANDLED,
RIG_EF_RESET_INBOX_FLAG).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…use stall

Pause without/with drain, stop-and-drain then start, a scheduled retry coming
due while a listener is paused (and its control), and a pause raised from a
durable local queue's own pipeline (and its DrainTimeout control).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
- MartenStore profile (AddMarten + IntegrateWithWolverine in its own schema,
  AutoApplyTransactions with an optional IdempotencyStyle); handlers queue the
  ledger row into the session's unit of work. ExactlyOnce, verified by a
  crash-after-commit spec and its RIG_SKIP_TX_MARK control
- Settling features for Marten over local, PostgreSQL queue, RabbitMQ, SQS
- Marten Eager/Optimistic idempotency on buffered and inline endpoints, plus
  the durable local and PostgreSQL queue no-op cases; deliveries reaching the
  handler chain are counted so a dropped duplicate is not vacuous
- F13 spec (handler reschedules itself, then returns normally) with the
  RIG_F13_EXPERIMENT diagnostic switches

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
SqlServerTests grew from 405 tests / 1367s (2026-08-26) to 658 tests / ~2300s (2026-10-22). On 2
lanes that is ~19m of tests plus ~2m of setup against the 20 minute cap; the job sat at 16-19.9m
from 2026-10-12 with regular cancellations. Split like CIPolecat: CISqlServerTransport lists
Transport, Agents, ScheduledJobs, Sagas and MultiTenancy (~1000s measured), and CISqlServer is the
negation of that list (root namespace + Persistence, ~1180s), so a new namespace still runs.

SqlServerMessageStore_with_IdAndDestination_Identity dropped its schema before every test, rebuilding
the whole Wolverine schema 78 times (2.7s median a test vs 0.4s for the identical compliance suite,
314s vs 60s in CI). Drop it once per test process; ClearAllAsync resets the rows. Locally the class
now runs in 23s, the same as its sibling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…rten and EF branches

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…-class wildcards match nothing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Tenancy store profiles (PostgreSQL and SQL Server static tenants, Marten
multi-database, EF Core managed multi-tenancy) over databases the rig creates
with RIG_SUFFIX in their names; per-tenant routing checks for effects, inbox
rows and dead letters; a tenant database taken offline mid-run; the per-tenant
transactional boundary; and the F19 inbox heartbeat scenario.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…one-member DLQ control

A crashed node's gated handler cannot be killed in-process; it never commits, so its open
execution is closed at the crash rather than reported as overlapping the rerun. The F2
control dead letters a one-member batch, because members whose dead letter write failed
are redelivered on their own clocks and the innocent ones then rightly succeed. Error logs
now print the first full exception of each kind for triage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…ed, real kills use IKillableStore

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Copies of one intent share a deduplication id with their own envelope ids; Outcome.Deduplicated
lets the copies of an intent be judged together. Covers duplicates, concurrency on one and two
nodes, the claim vs a failing handler, the claim vs the business transaction (including crashes),
GH-4757 collation, and the window/reaper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…bservation and experiment

EF Core managed multi-tenancy commits through CommitTenantedDbContextTransaction and
never marks the inbox row inside the tenant transaction; the crash-after-commit spec
duplicates 3 of 3 runs, the one-database control does not, and RIG_EF_TENANT_MARK_IN_TX=1
(the rig marking the row on the committing transaction) makes it pass 3 of 3. Adds a
crash-recovery spec for every tenancy store, row and blocking traces, and a pool
cleanup between scenarios so a tenant outage does not fail the next host start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
… database queue features

- A held EF Core zombie closes its connection on the crash signal instead of throwing, so its
  open claim no longer blocks the recovered copy and its error policy no longer dead letters it
- RIG_DEDUP_KEY_FROM_MESSAGE=1 reads the key off the message, taking transport header mapping
  out of the picture; RIG_DEDUP_RELEASE_ON_CRASH=1 deletes the stranded claim after a crash
- Narration of dead letter exceptions; sends from two nodes no longer race on a HashSet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…sh' specs are node stops

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…livery limit

- MessageStoreCompliance: the same envelope twice in one batch must surface as
  DuplicateIncomingEnvelopeException (RavenDB's existing fact, now held for every store). RED on
  PostgreSQL (4/4 configurations) and SQL Server (3/3): the RDBMS batch insert rethrows the raw
  provider exception when the duplicate is inside the batch rather than in the table.
- Bug_5123_quorum_queue_delivery_limit pins RabbitMQ 4.2.5's real semantics: a requeued give-back
  is dropped on the 21st attempt whether it is a reject OR a nack, so switching the listener's
  transient reject to a nack (as the report suggested) would not stop the loss.

Fixes to follow on this branch after review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…nted spec

F19 reproduces on every tenancy store (9 of 9 specs, 3 runs) while the one-database
PostgreSQL control passes; RIG_TENANT_HEARTBEAT=1 (the rig restarting the stale
clocks the receiver skips) makes all 9 pass. Marten's MultiTenantedDatabases has no
database for an untenanted message, so it only runs the tenant spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…efused sends judged

A node's own send for an offline tenant is refused to the caller on every transport,
so the first outage spec never exercised GH-4435/4658. The outage scenario now sends
through RabbitMQ from a host with no message store; the node-send case is its own spec
that requires the refusals. Unscripted handler failures are written with their stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…were graceful stops)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…e MQTT mappers

The deduplication-id header was written and read only by EnvelopeSerializer, so it
survived transports that carry the whole serialized envelope (SQS/SNS, database
queues, local queues) but was dropped by every broker built on EnvelopeMapper
(RabbitMQ, Azure Service Bus, Kafka, NATS, Pulsar, Redis; GCP Pub/Sub already mapped
it in its own subclass). A [Deduplicated] handler behind RabbitMQ dead-lettered every
copy with MissingDeduplicationIdException. The MQTT mappers write their user
properties by hand and also skipped it; their reader already goes through
EnvelopeSerializer.ReadDataElement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
This was referenced Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant