GH-5160, GH-5165, GH-5166: [DeduplicatedWithResponse] ClaimInTransaction, plus two Marten commit-failure fixes

docs/guide/durability/idempotency.md

@@ -456,7 +456,11 @@ commits, rather than doing its work and losing at commit time. `Lightweight` mod
 The one case this cannot decide up front is two genuinely concurrent callers under the same key.
 Neither one's claim is committed yet, so neither is visible to the other, and both proceed. The
 deduplication table's primary key settles it at commit time: one transaction wins, and the loser is
-refused exactly as if it had been caught by the up-front check — a 409 on HTTP, a discard on a handler.
+refused exactly as if it had been caught by the up-front check — a 409 on HTTP, a discard on a handler. On
+Marten, the loser's commit can also fail on its own work before it ever gets to the claim, like two
+`[WriteAggregate]` callers appending to the same stream at the same version. In that case Wolverine
+checks whether the key was claimed in the meantime and refuses the loser the same way. If the key isn't
+claimed, the original failure is rethrown unchanged.
 Worth knowing that the loser does its work and throws it away at commit rather than being refused
 before doing any, so a handler with side effects *outside* the transaction should account for that.
 

docs/guide/http/deduplication.md

@@ -169,7 +169,7 @@ For a given `Idempotency-Key`:
 | The first | Runs normally |
 | The same request again | The first response: its status, body and `Location`. The endpoint does not run |
 | A different request | **422** with a `ProblemDetails` body. Send a new key for a new request |
-| The same request while the first is still running | **409** with a `ProblemDetails` body |
+| The same request while the first is still running | **409** with a `ProblemDetails` body. With [`ClaimInTransaction`](#claiming-inside-the-transaction), it runs too ([see below](#claiming-inside-the-transaction)) |
 | No key | **400**, unless `Required = false` |
 
 A request that ends before the endpoint's work is done — it throws, answers 400 or above, or is stopped
@@ -218,6 +218,57 @@ cleanup removes it, so a repeat can be answered a little after its window.
 
 ### What is guaranteed
 
+By default the key is claimed and released on a connection of its own (below). With `ClaimInTransaction`,
+an endpoint whose store can take the claim inside its own transaction writes it there instead.
+
+#### Claiming inside the transaction
+
+```csharp
+[DeduplicatedWithResponse(DeduplicationScope.User, ClaimInTransaction = true)]
+[WolverinePost("/orders")]
+public static OrderCreated Post(CreateOrder command, IDocumentSession session)
+```
+
+Today that is an endpoint that commits through a Marten session with Wolverine's transactional middleware,
+on a single-database store, with Wolverine's PostgreSQL message store in the same database. The claim and
+its response are written together, as one row, in the same transaction as the endpoint's events,
+documents and outgoing messages. Nothing is claimed up front: the endpoint first checks for an earlier
+request's claim and answers from it.
+
+- A rollback, or a process that dies before the commit, leaves no claim, so a retry runs.
+- A commit leaves the claim answered, so a repeat is answered from it. There is no separate write that can
+  fail after the work is done. The exception is a response that cannot be serialized: the claim is then
+  committed unanswered, and repeats get 409 until it expires, so the work never runs twice.
+- Two concurrent requests with the same key both run; neither waits for the other. The first to commit
+  wins. The other's commit fails, on the claim's primary key or earlier on its own work (the same stream,
+  a versioned document); its work rolls back, and it is answered from the winner's claim: the stored
+  response for the same request, 422 for a different one. **Work outside the transaction runs for both**:
+  a call to another service, a message sent without the outbox. Claim-and-release would have refused the
+  second with 409 before it did anything. This is why the option is off by default.
+- A concurrent repeat that reaches its own answer before its commit gets that answer instead. If it reads
+  the first request's committed work and fails because of it (a validation that answers 400 "already
+  shipped", a 404, an exception), nothing is claimed and that failure goes to the client.
+- The losing request's response is cleared before it is answered, as ASP.NET Core's exception handler
+  clears a response it replaces, so a header its rolled-back work set does not go out. That clears headers
+  other middleware set directly on the response too; ones added through `OnStarting` are kept.
+- A failure inside the commit call after the database commit (on Marten, the outbox flush or a session
+  listener) is rethrown, as on claim-and-release, and a retry is answered from the claim.
+- The response is captured just before the commit, after the endpoint has set the status and `Location`.
+  A value only the commit assigns (a document version, say) is not in the stored body. An endpoint whose
+  response is only produced after the commit, such as one returning `UpdatedAggregate`, keeps
+  claim-and-release.
+
+An endpoint that asks for `ClaimInTransaction` but cannot take it — another store, database per tenant, a
+message store in another database, no commit through the middleware — keeps claim-and-release, and the
+host logs a warning naming it. The database check is repeated at runtime, so pre-generated code built
+against a different configuration fails loudly rather than writing claims where they are never read.
+
+Don't call `SaveChangesAsync` yourself in such an endpoint as well. Your own commit writes the work
+before the middleware's commit writes the claim, so a crash between them, or a concurrent repeat, runs
+the work twice.
+
+#### Claim-and-release
+
 The key is claimed before the endpoint runs. The response is recorded on the claim after the endpoint
 and any transactional commit, and before the response is written or cascaded messages flush, so a
 failure after the work was done finds the claim answered and a retry is answered from it. That holds on
@@ -228,9 +279,15 @@ Recording the response is a write of its own, not part of the endpoint's transac
 dies between the commit and that write, repeats get 409 until the claim expires. They never run twice
 within the window.
 
-Unlike `[Deduplicated]` above, the claim does not ride the endpoint's business transaction: a process
-that dies between the claim and the endpoint finishing leaves the key claimed for the whole window
-(24 hours by default) rather than releasing it with a rolled-back transaction.
+On Marten, the outbox flush and any session listeners run inside `SaveChangesAsync`, *after* the database
+commit. If one of them throws, the work is already committed, so Wolverine keeps the claim even though
+the response was never recorded. Repeats get 409 until the claim expires rather than running the work
+again. Use [`ClaimInTransaction`](#claiming-inside-the-transaction) if you want those retries answered
+with the first response instead.
+
+Here the claim does not ride the endpoint's business transaction: a process that dies between the claim
+and the endpoint finishing leaves the key claimed for the whole window (24 hours by default) rather than
+releasing it with a rolled-back transaction.
 
 ### Requirements and limits
 

src/Http/Wolverine.Http.Tests.AncillaryStore/SharedDataSourceDeduplicatedResponseEndpoints.cs

@@ -0,0 +1,45 @@
+using System.Collections.Concurrent;
+using Marten;
+using Wolverine.Marten;
+
+namespace Wolverine.Http.Tests.AncillaryStore;
+
+// GH-5160. [DeduplicatedWithResponse(ClaimInTransaction = true)] on the main store and on an ancillary store that
+// were both handed ONE NpgsqlDataSource, each with its own message schema. Here rather than in Wolverine.Http.Tests
+// because the ancillary endpoint needs INkThingStore registered; see the csproj.
+
+public record SharedSourceOrder(string Name);
+
+public record SharedSourceOrderCreated(Guid Id);
+
+public class SharedSourceOrderDocument
+{
+    public Guid Id { get; set; }
+    public string Name { get; set; } = string.Empty;
+}
+
+public static class SharedDataSourceDeduplicatedResponseEndpoints
+{
+    private static readonly ConcurrentDictionary<string, int> _runs = new();
+
+    public static int Runs(string name) => _runs.GetValueOrDefault(name);
+
+    [DeduplicatedWithResponse(DeduplicationScope.User, ClaimInTransaction = true)]
+    [WolverinePost("/shared-source-deduplicated-response/main")]
+    public static SharedSourceOrderCreated PostMain(SharedSourceOrder request, IDocumentSession session)
+        => store(request, session);
+
+    [MartenStore(typeof(INkThingStore))]
+    [DeduplicatedWithResponse(DeduplicationScope.User, ClaimInTransaction = true)]
+    [WolverinePost("/shared-source-deduplicated-response/ancillary")]
+    public static SharedSourceOrderCreated PostAncillary(SharedSourceOrder request, IDocumentSession session)
+        => store(request, session);
+
+    private static SharedSourceOrderCreated store(SharedSourceOrder request, IDocumentSession session)
+    {
+        _runs.AddOrUpdate(request.Name, 1, (_, count) => count + 1);
+        var document = new SharedSourceOrderDocument { Id = Guid.NewGuid(), Name = request.Name };
+        session.Store(document);
+        return new SharedSourceOrderCreated(document.Id);
+    }
+}