Durable inbox hardening, part 1: transactions, idempotency, deduplication, GH-5123 and the CISqlServer split

.github/workflows/tests.yml

@@ -97,7 +97,10 @@ jobs:
           - CIRavenDb
           - CIRedis
           - CISqlite
+          # SqlServerTests is sharded by namespace: CISqlServerTransport lists its namespaces and CISqlServer
+          # is the "everything else" shard. It sat at 16-19.9m of the 20 minute cap as one job (2026-10-22).
           - CISqlServer
+          - CISqlServerTransport
           - CICircuitBreaking
           - CIClaimCheck
           - CIAotSmoke

build/CITargets.cs

@@ -767,25 +767,55 @@ AbsolutePath suite(string name) =>
             RunTestProject(inMemoryTests);
         });
 
+    // SqlServerTests outgrew one job. Profiled 2026-08-01 it was 931s of test time; by 2026-10-22 it was 658
+    // tests and ~2300s, so even a perfectly balanced 2-lane run (hosted runners clamp 4 workers to 2) is
+    // ~19m of tests plus ~2m of setup against the 20 minute cap, and it had sat at 16-19.9m with regular
+    // cancellations since 2026-10-12. Split the way CIPolecat and CIMarten are: one named shard lists its
+    // namespaces, and CISqlServer is the NEGATION of that list, so a brand new namespace still runs.
+    //
+    // Measured by namespace from the lane log of the cancelled #5122 run: Transport 660s, Agents 187s,
+    // ScheduledJobs 67s, Sagas 62s, MultiTenancy 29s (~1000s) here; the root namespace 593s and
+    // Persistence 586s (~1180s) in CISqlServer. Each shard is ~10m of tests on 2 lanes.
+
+    AbsolutePath SqlServerTests => RootDirectory / "src" / "Persistence" / "SqlServerTests" / "SqlServerTests.csproj";
+
+    static readonly string[] SqlServerTransportNamespaces =
+    [
+        "SqlServerTests.Transport",
+        "SqlServerTests.Agents",
+        "SqlServerTests.ScheduledJobs",
+        "SqlServerTests.Sagas",
+        "SqlServerTests.MultiTenancy"
+    ];
+
+    void runSqlServerShard(Func<WorkerTest, bool> testFilter)
+    {
+        // SQL Server's boot is one of the slower container starts; overlap it with the compile
+        // instead of paying the two serially.
+        LaunchDockerServices("sqlserver");
+        BuildTestProjects(SqlServerTests);
+        AwaitDockerServices("sqlserver");
+
+        // The sibling databases the multi-tenancy and NSB dedicated-db suites create are lane-scoped via
+        // LaneDatabases.Name, so lanes cannot collide on them.
+        RunTestProject(SqlServerTests, testFilter: testFilter, workers: 4, sqlServerDatabasePerLane: true);
+    }
+
+    /// <summary>
+    /// SQL Server shard 1: the database-backed transport, agents and node persistence, scheduled jobs, sagas
+    /// and multi-tenancy.
+    /// </summary>
+    Target CISqlServerTransport => _ => _
+        .ProceedAfterFailure()
+        .Executes(() => runSqlServerShard(includeNamespaces(SqlServerTransportNamespaces)));
+
+    /// <summary>
+    /// SQL Server shard 2: everything not claimed by CISqlServerTransport -- the message store compliance
+    /// suites, the root namespace, and any newly added namespace, which lands here by default.
+    /// </summary>
     Target CISqlServer => _ => _
         .ProceedAfterFailure()
-        .Executes(() =>
-        {
-            var sqlServerTests = RootDirectory / "src" / "Persistence" / "SqlServerTests" / "SqlServerTests.csproj";
-
-            // SQL Server's boot is one of the slower container starts; overlap it with the compile
-            // instead of paying the two serially.
-            LaunchDockerServices("sqlserver");
-            BuildTestProjects(sqlServerTests);
-            AwaitDockerServices("sqlserver");
-
-            // Profiled 2026-08-01 (TRX over the full suite, local M-series): 60 classes, 931s of
-            // test time, largest class 167s -> a 5.6x ceiling. Hosted runners clamp 4 workers to 2,
-            // where the LPT bound is ~470s of test wall clock against ~860s sequential. The sibling
-            // databases the multi-tenancy and NSB dedicated-db suites create are lane-scoped via
-            // LaneDatabases.Name, so lanes cannot collide on them.
-            RunTestProject(sqlServerTests, workers: 4, sqlServerDatabasePerLane: true);
-        });
+        .Executes(() => runSqlServerShard(excludeNamespaces(SqlServerTransportNamespaces)));
 
     // MartenTests was the 18m long pole of the matrix (#3752). It was first attacked with worker lanes —
     // four processes, each on its own database via its own connection string — and that halved the wall clock but

docs/guide/durability/idempotency.md

@@ -61,11 +61,21 @@ Each of those is a *different* delivery of the *same intent*, so each carries a
 `Envelope.Id` and every one of them gets through. What is needed is an identity for the intent, and
 that is what `DeduplicationId` is.
 
-`Envelope.DeduplicationId` is a `string`, it is set by your application, and it already round-trips on
-every transport Wolverine supports — it is written to and read from the `deduplication-id` wire header
-by Wolverine's own serialization, so it survives Rabbit MQ, Kafka, Azure Service Bus, SQS, and the
-rest without any transport-specific configuration. (Two transports also consume it natively: SQS/SNS
-FIFO queues and topics, and GCP Pub/Sub.)
+`Envelope.DeduplicationId` is a `string`, it is set by your application, and it round-trips on every
+transport Wolverine supports under the `deduplication-id` wire header. The transports that carry the whole
+serialized envelope (SQS, SNS, the database-backed queues, local queues, TCP) get it from Wolverine's own
+serialization, and the brokers that map Wolverine's envelope onto native headers -- Rabbit MQ, Azure Service
+Bus, Kafka, NATS, Pulsar, Redis, GCP Pub/Sub and MQTT -- write and read the same header, so you don't need
+any transport-specific configuration. (Two transports also consume it natively: SQS/SNS FIFO queues and
+topics, and GCP Pub/Sub.)
+
+::: warning
+Before this release, the header-mapping brokers dropped the id on the floor. If you put a `[Deduplicated]`
+handler behind Rabbit MQ (or any of the brokers above) on an older Wolverine version, every copy arrived
+without its id and went straight to the dead letter queue with a `MissingDeduplicationIdException`. Reading
+the id off the message itself with `[Deduplicated(ValueSource.InputMember, ...)]` or
+`[DeduplicationIdentity]` side-stepped the problem, because then the id never has to cross the wire at all.
+:::
 
 ::: info
 We probably should have added this feature ages ago, but we did so in 6.31 to get ready for Wolverine to have a first
@@ -461,9 +471,48 @@ the key back, and the release is registered to run before the response is flushe
 cannot beat it. A 2xx or a 3xx keeps it, so an idempotency key means "this succeeded once" rather than
 "this was attempted once".
 
-Either way the guarantee you see is the same. The difference is cost: the transactional path pays one
-round trip, shared with whatever read your handler was already doing, where the compensating path pays
-one to claim and another to release.
+For a handler that fails and gets retried, the guarantee you see is the same either way. The difference is
+cost: the transactional path pays one round trip, shared with whatever read your handler was already doing,
+where the compensating path pays one to claim and another to release. Where the two paths do differ is when
+the process dies in the middle of your handler, and that's the next section.
+
+### When a node dies mid-handler
+
+On the compensating path, the claim is already committed before your handler starts, and the only thing that
+gives it back is the release Wolverine runs when the handler fails. If the node is killed outright in the
+middle of your handler -- a pod eviction, an out of memory kill, somebody pulling the plug -- that release
+never runs, and the claim is left sitting in the deduplication table.
+
+If you're using the durable inbox, Wolverine isn't going to lose that message, and another node will recover it
+from the inbox and run it again. That recovered copy is the *same* envelope, with the same `Envelope.Id`, as
+the one that died. To keep the stranded claim from refusing that recovered copy as a duplicate of itself,
+Wolverine records which envelope took each claim. When a copy of that same envelope comes back, it takes the
+claim over and runs. A *different* envelope with the same deduplication id is still refused, exactly as before,
+so the operator double-click is still caught.
+
+::: tip
+This is automatic for message handlers on the PostgreSQL, SQL Server, MySQL and SQLite message stores. Marten,
+EF Core in its `Eager` mode, Polecat and Fisher don't need it at all, because their claim rides your
+transaction: a crash before the commit takes the claim away with it.
+:::
+
+A couple of things you might want to know about this:
+
+* The envelope is recorded in a new, nullable `envelope_id` column on the deduplication table (in both the
+  `CompareByString` and `CompareByHash` shapes). If you already have the table, your next schema migration
+  just adds that column, and nothing else about the table changes. Claims written before the upgrade have no
+  envelope recorded, so they keep the old behaviour and refuse everything until they expire.
+* If you run with `AutoCreate.None` and haven't applied that migration yet,
+  nothing breaks. The first time a deduplicated handler claims an id, Wolverine checks whether the table has the
+  `envelope_id` column, once per process. If it doesn't, your claims work exactly the way they did before the
+  upgrade, and Wolverine logs a single warning telling you the table should be migrated to get crash-safe
+  claims. That check isn't repeated, so after you apply the migration by hand, restart the application to pick
+  the column up.
+* Taking a claim over is about the envelope, not about whether the work finished. If the node died *after*
+  your handler did its work but before the inbox marked the message as handled, the recovered copy will take
+  the claim over and run again. That's the usual at-least-once promise of the durable inbox on a plain
+  relational store, and the same thing an idempotent handler already has to cope with.
+* HTTP endpoints and gRPC methods have no envelope, so they keep the original behaviour.
 
 ### Why a separate table
 

docs/guide/messaging/transports/rabbitmq/object-management.md

@@ -292,6 +292,95 @@ There are just a few things to know:
 * Wolverine's internal reply or control queues will still be declared as "classic" so they can be non-durable
 * Streams cannot be purged, and Wolverine ignores the `AutoPurgeOnStartup()` setting for streams
 
+## Quorum Queues and the Durable Inbox
+
+If you're using quorum queues behind a [durable inbox](/guide/durability/), there's one Rabbit MQ setting you
+really want to know about before your first broker outage, and that's the quorum queue *delivery limit*.
+
+Starting with Rabbit MQ 4, every quorum queue has a delivery limit of 20 by default. Every time a message goes
+back to the queue without being acknowledged, the broker counts it, and on the 21st return the broker dead letters
+the message -- or just drops it on the floor if the queue doesn't have a dead letter exchange. That sounds like a
+reasonable poison message guard, and for a lot of systems it is. But think about what actually hands a message back
+to the queue:
+
+* A listener that is stopping, pausing, or restarting with messages still prefetched in memory
+* A channel or connection that closes during a network blip or a broker restart
+* A reject with requeue, which counts on every Rabbit MQ 4 version we've tried
+* A nack with requeue -- at least on Rabbit MQ 4.2.5, where a reject, a nack and a closed channel all counted against
+  the limit the exact same way. Newer 4.x releases stopped counting the nack, and Wolverine now hands transient
+  deliveries back with a nack for exactly that reason, but you shouldn't bet your messages on which broker version
+  you happen to be running
+
+None of those mean your message is poison. With the durable inbox, Wolverine acknowledges a message to Rabbit MQ
+just as soon as it's safely stored in your database, and from then on Wolverine's own [error handling
+policies](/guide/handlers/error-handling) decide what happens to a message that keeps failing. So the only thing the
+broker's delivery limit is really counting for a durable listener is how many times your system was disrupted while
+that message was in flight. A long enough outage with enough reconnects, and you can lose a perfectly good message
+without a single exception in your logs.
+
+To get out in front of that, you can opt into an `x-delivery-limit` for the quorum queues that Wolverine declares,
+either for every durable listener at once or queue by queue:
+
+<!-- snippet: sample_delivery_limit_for_durable_quorum_queues -->
+<a id='snippet-sample_delivery_limit_for_durable_quorum_queues'></a>
+```cs
+var builder = Host.CreateApplicationBuilder();
+builder.UseWolverine(opts =>
+{
+    opts
+        .UseRabbitMq(builder.Configuration.GetConnectionString("rabbit")!)
+        .AutoProvision()
+        .UseQuorumQueues()
+
+        // Every quorum queue that Wolverine declares for a listener
+        // using the durable inbox gets this x-delivery-limit.
+        // -1 turns the limit off on RabbitMQ 4
+        .DeliveryLimitForDurableQuorumQueues(-1);
+
+    opts.ListenToRabbitQueue("incoming")
+        .UseDurableInbox();
+
+    opts.ListenToRabbitQueue("payments")
+        .UseDurableInbox()
+
+        // Or set the limit for just one queue, and this
+        // wins over the transport wide setting
+        .DeliveryLimit(100)
+
+        // Optional, but this lets in flight deliveries reach
+        // the inbox on a stop instead of being handed back
+        .DrainWaitForPrefetch();
+});
+```
+<sup><a href='https://github.com/JasperFx/wolverine/blob/main/src/Transports/RabbitMQ/Wolverine.RabbitMQ.Tests/Samples.cs#L599-L628' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_delivery_limit_for_durable_quorum_queues' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->
+
+A couple of things to know:
+
+* Neither setting changes anything unless you opt in. Wolverine declares your queues exactly the way it always has
+  by default.
+* `DeliveryLimitForDurableQuorumQueues()` only applies to quorum queues that are listened to with the durable inbox.
+  Inline or buffered listeners don't have the inbox to fall back on, so the broker's limit is still doing real work
+  for them.
+* The limit is only applied when Wolverine is the one declaring the queue. If you mark a queue as externally owned or
+  let some other process create it, Wolverine stays out of it.
+* `-1` turns the limit off on Rabbit MQ 4. If you'd rather keep a backstop, pick a number big enough to ride out
+  your worst outage.
+
+::: warning
+Rabbit MQ will not let you redeclare an existing queue with different arguments, so adding a delivery limit to a
+queue that already exists will make Wolverine fail at startup with a "inequivalent arg" error. For an existing queue,
+either delete the queue and let Wolverine recreate it, or set the delivery limit with a Rabbit MQ
+[policy](https://www.rabbitmq.com/docs/parameters#policies) instead, which applies to existing queues without
+redeclaring anything.
+:::
+
+::: tip
+If Wolverine declares a quorum queue for a durable listener without any delivery limit, you'll see a warning in your
+logs when that listener starts up. That's your cue to either opt in like the sample above, or confirm that a Rabbit
+MQ policy is taking care of it for you.
+:::
+
 ## Inside of Wolverine Extensions
 
 If you need to declare Rabbit MQ queues, exchanges, or bindings within a [Wolverine extension](/guide/extensions),

docs/tutorials/idempotency.md

@@ -99,9 +99,22 @@ proven to be too buggy to be useful.
 Idempotency checking within message handlers executing within `Buffered` or more likely `Inline` listeners will require
 you to "opt in." First though, the idempotency check in this case can be done in one of two modes:
 
-1. `Eager` -- just means that Wolverine will apply some middleware around the handler such that it will make an early database call to try to insert a skeleton placeholder in the transactional inbox storage
+1. `Eager` -- just means that Wolverine will apply some middleware around the handler such that it checks for, and claims, a skeleton placeholder in the transactional inbox storage *before* your handler runs
 2. `Optimistic` -- Wolverine would try to insert the skeleton message information as part of the message handling transaction to try to avoid extra database round trips
 
+::: info
+The `Eager` placeholder is written into the **same** transaction as your handler's own work, so it commits or rolls back right
+along with everything else. That matters more than it sounds. Before this, the placeholder was committed up front on its own, so
+a handler that failed and got requeued or retried came back to find a "handled" record for work that never actually committed --
+and Wolverine refused the retry as a duplicate of itself. Now a failed attempt leaves nothing behind, and only an attempt that really
+committed makes the next copy a duplicate.
+
+Marten goes one step further and takes the placeholder row as soon as the check runs, so a second copy of the same message arriving
+at the same moment waits on the first one rather than running alongside it. Polecat, Fisher, and EF Core in `Lightweight` mode can't
+open their transaction that early, so two copies landing at exactly the same moment can both run the handler -- but only one of them
+can commit.
+:::
+
 `Optimistic` was the more optimal of the two on paper, but as the warning above says, it is **not currently reachable**. Every
 provider -- EF Core, Marten, Polecat, and Fisher alike -- emits the `Eager` check for any handler configured with either style,
 so `IdempotencyStyle.Optimistic` behaves exactly like `IdempotencyStyle.Eager` at runtime. Treat the two as one option until