Repository navigation
Durable inbox hardening, part 1: transactions, idempotency, deduplication, GH-5123 and the CISqlServer split - #5129
Merged
Merged
Conversation
…invariant checker A standalone test project (outside both solutions and CI) that exercises the durable inbox end to end. Handlers write a ledger row inside their own transaction; after each scenario the invariants are judged one by one against the store: lost messages, duplicate effects, concurrent runs of one message, dead letter mismatches, unsettled inbox rows, outbox leftovers, and whether every envelope carried its message's id (which the row-level checks depend on). First slice: the settling scenarios on PostgreSQL over a durable local queue, a PostgreSQL queue, RabbitMQ and SQS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…ures SqlServerStore reuses RdbmsLedger; EfCoreStore maps LedgerEntry onto rig_ledger and lets the EF Core transactional middleware commit, with an interceptor finishing each execution at commit/rollback. InvariantGlobalization off: SqlClient refuses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Killable PostgreSQL store (a login role per node, cut on kill), an inbox-row blip, an inbox-row watch, and specs for graceful stop, kill, InboxStaleTime, F15 (unlatched pause), F16 (listener rebuild) and F12 (blip during a scheduled retry). Execution.Finish is now first-wins so a killed run stays killed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…eSQL Batch members judged by the same invariants as single messages, over a durable local batch queue and over RabbitMQ/SQS durable listeners feeding the default buffered batch queue (plus a durable batch queue variant). Error policies, ProbeIndividuallyAfter, IsolateBatchMembers, ApplyItemException, and specs for ledger leads F1-F6 and F8. Extra checks: batch pending counts, and errors logged settling members. RdbmsLedger: a member's scripted failure now finishes the members begun before it, and the RigItem[] handler can throw ApplyItemException for a scripted poison. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
F25 (failed commit after the in-transaction inbox mark strands the row and re-runs a discarded message on restart) and EF idempotency on buffered/inline endpoints (a retry loses the Handled row, so a redelivery runs again), each with a diagnostic switch that removes the failure (RIG_EF_UNDO_HANDLED, RIG_EF_RESET_INBOX_FLAG). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…use stall Pause without/with drain, stop-and-drain then start, a scheduled retry coming due while a listener is paused (and its control), and a pause raised from a durable local queue's own pipeline (and its DrainTimeout control). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
- MartenStore profile (AddMarten + IntegrateWithWolverine in its own schema, AutoApplyTransactions with an optional IdempotencyStyle); handlers queue the ledger row into the session's unit of work. ExactlyOnce, verified by a crash-after-commit spec and its RIG_SKIP_TX_MARK control - Settling features for Marten over local, PostgreSQL queue, RabbitMQ, SQS - Marten Eager/Optimistic idempotency on buffered and inline endpoints, plus the durable local and PostgreSQL queue no-op cases; deliveries reaching the handler chain are counted so a dropped duplicate is not vacuous - F13 spec (handler reschedules itself, then returns normally) with the RIG_F13_EXPERIMENT diagnostic switches Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
SqlServerTests grew from 405 tests / 1367s (2026-08-26) to 658 tests / ~2300s (2026-10-22). On 2 lanes that is ~19m of tests plus ~2m of setup against the 20 minute cap; the job sat at 16-19.9m from 2026-10-12 with regular cancellations. Split like CIPolecat: CISqlServerTransport lists Transport, Agents, ScheduledJobs, Sagas and MultiTenancy (~1000s measured), and CISqlServer is the negation of that list (root namespace + Persistence, ~1180s), so a new namespace still runs. SqlServerMessageStore_with_IdAndDestination_Identity dropped its schema before every test, rebuilding the whole Wolverine schema 78 times (2.7s median a test vs 0.4s for the identical compliance suite, 314s vs 60s in CI). Drop it once per test process; ClearAllAsync resets the rows. Locally the class now runs in 23s, the same as its sibling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…rten and EF branches Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…-class wildcards match nothing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Tenancy store profiles (PostgreSQL and SQL Server static tenants, Marten multi-database, EF Core managed multi-tenancy) over databases the rig creates with RIG_SUFFIX in their names; per-tenant routing checks for effects, inbox rows and dead letters; a tenant database taken offline mid-run; the per-tenant transactional boundary; and the F19 inbox heartbeat scenario. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…one-member DLQ control A crashed node's gated handler cannot be killed in-process; it never commits, so its open execution is closed at the crash rather than reported as overlapping the rerun. The F2 control dead letters a one-member batch, because members whose dead letter write failed are redelivered on their own clocks and the innocent ones then rightly succeed. Error logs now print the first full exception of each kind for triage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…ed, real kills use IKillableStore Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Copies of one intent share a deduplication id with their own envelope ids; Outcome.Deduplicated lets the copies of an intent be judged together. Covers duplicates, concurrency on one and two nodes, the claim vs a failing handler, the claim vs the business transaction (including crashes), GH-4757 collation, and the window/reaper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…bservation and experiment EF Core managed multi-tenancy commits through CommitTenantedDbContextTransaction and never marks the inbox row inside the tenant transaction; the crash-after-commit spec duplicates 3 of 3 runs, the one-database control does not, and RIG_EF_TENANT_MARK_IN_TX=1 (the rig marking the row on the committing transaction) makes it pass 3 of 3. Adds a crash-recovery spec for every tenancy store, row and blocking traces, and a pool cleanup between scenarios so a tenant outage does not fail the next host start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
… database queue features - A held EF Core zombie closes its connection on the crash signal instead of throwing, so its open claim no longer blocks the recovered copy and its error policy no longer dead letters it - RIG_DEDUP_KEY_FROM_MESSAGE=1 reads the key off the message, taking transport header mapping out of the picture; RIG_DEDUP_RELEASE_ON_CRASH=1 deletes the stranded claim after a crash - Narration of dead letter exceptions; sends from two nodes no longer race on a HashSet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…sh' specs are node stops Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…livery limit - MessageStoreCompliance: the same envelope twice in one batch must surface as DuplicateIncomingEnvelopeException (RavenDB's existing fact, now held for every store). RED on PostgreSQL (4/4 configurations) and SQL Server (3/3): the RDBMS batch insert rethrows the raw provider exception when the duplicate is inside the batch rather than in the table. - Bug_5123_quorum_queue_delivery_limit pins RabbitMQ 4.2.5's real semantics: a requeued give-back is dropped on the 21st attempt whether it is a reject OR a nack, so switching the listener's transient reject to a nack (as the report suggested) would not stop the loss. Fixes to follow on this branch after review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…nted spec F19 reproduces on every tenancy store (9 of 9 specs, 3 runs) while the one-database PostgreSQL control passes; RIG_TENANT_HEARTBEAT=1 (the rig restarting the stale clocks the receiver skips) makes all 9 pass. Marten's MultiTenantedDatabases has no database for an untenanted message, so it only runs the tenant spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…efused sends judged A node's own send for an offline tenant is refused to the caller on every transport, so the first outage spec never exercised GH-4435/4658. The outage scenario now sends through RabbitMQ from a host with no message store; the node-send case is its own spec that requires the refusals. Unscripted handler failures are written with their stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…were graceful stops) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…e MQTT mappers The deduplication-id header was written and read only by EnvelopeSerializer, so it survived transports that carry the whole serialized envelope (SQS/SNS, database queues, local queues) but was dropped by every broker built on EnvelopeMapper (RabbitMQ, Azure Service Bus, Kafka, NATS, Pulsar, Redis; GCP Pub/Sub already mapped it in its own subclass). A [Deduplicated] handler behind RabbitMQ dead-lettered every copy with MissingDeduplicationIdException. The MQTT mappers write their user properties by hand and also skipped it; their reader already goes through EnvelopeSerializer.ReadDataElement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
This was referenced Oct 23, 2026
This was referenced Oct 23, 2026
This was referenced Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5123.
The first of two coarse PRs from a durable inbox compliance effort. A new Bobcat rig (
src/Testing/DurableComplianceRig, outside both solutions and CI) exercised the durable inbox end to end across PostgreSQL, SQL Server, Marten and EF Core over local, database, RabbitMQ and SQS endpoints. Every fix here has a regression test in a normal CI test project that was red before the fix and green after, and the rig specs that found it now pass. Groups A and B (listener lifecycle and durable batching) follow in a second PR.Transactions and idempotency
MessageContext.ClearState()disposed the enlistedDatabaseEnvelopeTransaction, which disposed aDbTransactionits owner had already finished, and Npgsql reuses that object on the pooled connector.DatabaseEnvelopeTransactionno longer disposes a transaction it does not own.Status = Handledbefore its own commit, so a failed commit followed by Discard left the rowIncoming. The status is now set only after the commit succeeds.ReScheduleAsyncand then returned normally had its scheduled row overwritten to Handled, losing the retry. It isn't now.Logical deduplication
[Deduplicated]claim stranded by a crash mid-handler made inbox recovery discard the message for the whole window. Claims now record the envelope that took them (envelope_id, a new nullable column), so the same envelope can take its claim over and a different one is still refused.IEnvelopeAwareDeduplicationStore;IDeduplicationStoreis unchanged.AutoCreate.None.Envelope.DeduplicationIddid not cross RabbitMQ, or any broker built on the genericEnvelopeMapperor the MQTT mappers. Those mappers now map it.GH-5123
DuplicateIncomingEnvelopeException, as RavenDB already did, across PostgreSQL, SQL Server, MySQL, SQLite and Oracle. Oracle's batch insert, which committed everything but the failing statement, is now all-or-nothing.x-delivery-limitfor queues Wolverine declares:.DeliveryLimit(n)per listener,DeliveryLimitForDurableQuorumQueues(n)per transport. Nothing changes by default;RabbitMqEnvelope.DeferAsyncnow publishes the copy before it acks the original.CI
CISqlServerhad outgrown the 20 minute cap: 658 tests and about 2300s of test time, against 405 tests and 1367s on Aug 26. It is now split by namespace likeCIPolecat:CISqlServerTransport, plusCISqlServeras the catch-all.SqlServerMessageStore_with_IdAndDestination_Identitydropped and rebuilt its schema before every test (314s in CI). It now does that once per process.Local verification
dotnet build wolverine.slnx -c Release -f net9.0: clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U