diff --git a/AI-INTEGRATION-RESEARCH.md b/AI-INTEGRATION-RESEARCH.md new file mode 100644 index 000000000..f7a86270e --- /dev/null +++ b/AI-INTEGRATION-RESEARCH.md @@ -0,0 +1,78 @@ +# AI/LLM Integration Research — Prior Art for a Wolverine/Marten Feature Set + +*Research date: 2026-07-12. Sources: deep-research workflow (25 sources fetched, 115 claims extracted, 25 top claims adversarially verified 3-0) plus a dedicated AxonIQ/KurrentDB sweep. Seed question: prior art for AI-gateway-style resilience/observability around LLM calls in messaging frameworks and event stores, per https://dotnetdigest.com/building-an-ai-gateway-in-net.* + +## Executive summary + +1. **The .NET messaging field is empty.** MassTransit v9 is purely the commercial/Massient licensing transition — zero AI features, no MassTransit.AI, no IChatClient integration (verified against public announcements as of July 2026). Nothing from NServiceBus/Rebus/Brighter surfaced either. A Wolverine AI integration would be a first mover. +2. **`IChatClient` is the sanctioned seam and Microsoft prescribes the packaging.** A `Wolverine.AI` package should reference only `Microsoft.Extensions.AI.Abstractions`, derive middleware from `DelegatingChatClient`, and expose `Use*` extensions on `ChatClientBuilder`. Caching (exact-history) and OTel GenAI-semconv telemetry are solved in-box; **routing/fallback, budget enforcement, and durable/transactional resilience are not** — that's the gap. +3. **Nobody shipped the durable LLM-callout primitive.** AxonIQ and Kurrent both pivoted their positioning hard to AI (Oct 2025 / Dec 2024) but answer "LLM call from an event handler" with generic at-least-once + retry + park/DLQ + an idempotency warning in the docs. First-class support — a memoized "AI side effect" that records the LLM response as an event so retries and replays reuse it instead of re-billing — has no prior art anywhere. +4. **The event-sourcing + AI angle is genuine white space.** The main workflow's verification pass produced *zero* surviving claims for LLM-from-projections/subscriptions prior art. The one hard principle: projections must be deterministic/replayable, which rules out LLM calls inside projection logic and points at side-effect subscriptions as the right home. + +## What Microsoft.Extensions.AI already gives us (don't rebuild) + +- `IChatClient` + `IEmbeddingGenerator` in `Microsoft.Extensions.AI.Abstractions`; framework libraries reference abstractions only, apps reference the full package. ([learn.microsoft.com/dotnet/ai/microsoft-extensions-ai](https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai)) +- `ChatClientBuilder` composes `DelegatingChatClient` decorators; first `Use*` registered = outermost. Built-ins: `UseDistributedCache`, `UseFunctionInvocation`, `UseOpenTelemetry`, `UseLogging`, `ConfigureOptions`. Microsoft explicitly frames these as "a small subset" and invites third-party middleware; the canonical worked example is a rate limiter on `System.Threading.RateLimiting`. ([learn.microsoft.com/dotnet/ai/ichatclient](https://learn.microsoft.com/en-us/dotnet/ai/ichatclient)) +- `DistributedCachingChatClient` = exact-match caching keyed on chat history + `ChatOptions` (no semantic caching; `RawRepresentation` not cached). `OpenTelemetryChatClient` emits OTel GenAI semantic conventions (still experimental; semconv v1.37). +- **Absent from built-ins:** cross-provider routing/fallback, cost/budget enforcement, durability. These are the framework-integration opportunities. +- API drift warning: preview-era `.Use(inner).AsChatClient` became `AsIChatClient` at GA — re-check current package surface before implementing. + +## Richest framework prior art: Spring AI 2.0 (GA June 2026) + +- **Advisors API** = around-style interceptor chain: `ChatClientResponse adviseCall(ChatClientRequest, CallAdvisorChain)` / `Flux adviseStream(...)`. The abstraction is **split into sync and streaming interfaces** with guidance to implement both — any IChatClient wrapper faces the same fork. ([docs.spring.io/spring-ai/reference/api/advisors.html](https://docs.spring.io/spring-ai/reference/api/advisors.html)) +- Batteries-included advisor catalog: chat memory (message + vector-store variants), RAG (`QuestionAnswerAdvisor`, `RetrievalAugmentationAdvisor`), content safety (`SafeGuardAdvisor`), auto-registered `ToolCallingAdvisor`. +- **Spring 2.0 lifted the tool-calling loop into the middleware chain** as a recursive advisor that re-enters the downstream chain until the model stops requesting tools. Ordering relative to the loop is semantically meaningful: a memory advisor *outside* the loop persists only final user/assistant messages; *inside* the loop it persists the full tool transcript, and the framework auto-disables internal history to avoid duplicate writes. This is the state-of-the-art answer to "where does the agentic loop live in a pipeline." ([spring.io blog: composable tool calling](https://spring.io/blog/2026/06/15/spring-ai-composable-tool-calling/)) + +## Durable execution prior art + +- **Microsoft Agent Framework durable workflows** (May 2026): automatic checkpointing on the Durable Task stack, per-executor granularity (each executor = a durable activity, `dafx-` prefixed) so completed LLM steps don't re-run after a crash. Criticisms (Diagrid): manual resume, no tool-level RetryPolicy, Azure dependence; the framework's *native* in-process CheckpointManager is only superstep-granular. ([devblogs: durable workflows in MAF](https://devblogs.microsoft.com/dotnet/durable-workflows-in-microsoft-agent-framework/)) +- Temporal/Restate converge on "LLM call = retryable activity inside deterministic orchestration." +- **Differentiation axis:** Wolverine's durable inbox/outbox + scheduled retries can offer per-step durability broker- and cloud-agnostically. Nobody checkpoints at sub-call granularity (per tool invocation inside an agentic loop) yet. + +## AI-gateway feature checklist (Cloudflare / LiteLLM / Portkey / Kong) + +- Canonical set: caching, rate limiting, automatic retry + **model/provider-granularity ordered fallback** (configurable backoff; telemetry reports which fallback step served the request), token/cost analytics, request logging. ([developers.cloudflare.com/ai-gateway](https://developers.cloudflare.com/ai-gateway/)) +- **Cost is a first-class telemetry dimension** — requests, tokens, dollars. Kong is dinged competitively for weak cost intelligence; LiteLLM does per-key/team budget caps. Any Wolverine observability story needs token/cost accounting alongside OTel spans. +- Fallback-at-model-granularity maps naturally onto Wolverine error policies; budget caps map onto per-endpoint/per-tenant/per-message-type policies (open question: budget-exceeded → backpressure/queue semantics has no prior art). + +## AxonIQ (deep-dive) + +- **Oct 30, 2025:** "AxonIQ Platform" launch — AF5 + Axon Server 2025.2 + "Agents"; pitch = architecture-level explainability ("persistent event-based memory," time-travel debugging, MCP extensions in Axon Server). **AI lives in the commercial platform; Axon Framework 5.x itself has no LLM API, no Spring AI integration.** (AF 5.0 GA Oct 2025; 5.2.0 Jul 9, 2026.) +- **Architectural stance worth quoting:** agents enter *through the command/query bus*, validated by normal business logic — never appending events directly. Their MCP demo drives `RegisterBike` commands via an OpenAI agent "without bypassing safeguards." +- **AxonIQ Insights** (most concrete shipped AI feature): subscribes to event streams, batches to Parquet in a DuckDB "EventLake" sidecar; SQL over Postgres wire protocol; NL chat where the LLM sees the schema and generates SQL; embedded MCP server. **AI reads a derived analytical replica, never the live store.** +- **Side-effect machinery (closest analog to Wolverine + Marten subscriptions):** + - Streaming processors retry with incremental backoff 1s→60s, stalling the segment. + - **Sequence-aware DLQ**: `SequencedDeadLetterQueue` parks the failed event *and all subsequent events with the same sequence id*, so projections never build on inconsistent state. (Dropped in AF 5.0, reintroduced 5.1.0.) + - Docs explicitly warn: enabling DLQ ⇒ at-least-once ⇒ handlers must be idempotent; token-steal re-fires side effects (emails, sagas, queue messages named). + - **Replay-awareness primitives — the most reusable idea:** `resetTokens()` triggers replay; handlers opt out of side effects during replay via `@DisallowReplay` / `@AllowReplay`, a `ReplayStatus` parameter (REGULAR vs REPLAY), and `@ReplayContext` injection. Built generically for emails years before LLMs; nothing LLM-specific shipped on top. +- **No embeddings/vector/RAG** anywhere in Axon Server or the framework. +- Unverified: Developer Agent 2.0 capabilities, multi-agent orchestration details, Axon Server "MCP extensions" specifics (press-release wording only). + +## Kurrent / KurrentDB (deep-dive) + +- **Dec 2024:** Event Store → Kurrent rebrand + $12M; "the event-native data platform"; homepage claims "if you're building agentic AI, Kurrent is the only database that actually understands history" — but names no concrete AI features. +- **KurrentDB MCP Server** (May 2025, flagship AI deliverable): Python/MIT/stdio; 8 tools — `read_stream`, `list_streams`, `build_projection`, `create_projection`, `update_projection`, `test_projection`, `write_events_to_stream`, `get_projections_status`. Signature feature: **self-correcting projection authoring** (agent writes JS projection, runs `test_projection`, iterates on faults). Dev-time positioning; modest repo activity. ([github.com/kurrent-io/mcp-server](https://github.com/kurrent-io/mcp-server)) +- **KurrentDB 26.0** (GA Jan 2026), marketed "Non-disruptive Integration for AI Systems," actually shipped: Kafka source connector, **Relational Sink** (declarative reducers → auto-maintained Postgres/SQL Server read models), user-defined secondary indexes. **No vector search, no embedding generation, no MCP in the DB** — "for AI systems" means "your events reach AI/analytics systems without custom code." +- **Callout machinery:** persistent subscriptions = ack/nack (retry/skip/**park**); park after `maxRetryCount` to `$persistentsubscription-{stream}::{group}-parked`; parked replayable with `stopAt`; checkpoint streams; documented at-least-once. **Documented pitfall: ordering is not guaranteed with persistent subscriptions**, and parking is *not* sequence-aware (a parked event's successors keep flowing) — the sequence-consistency gap Axon solved and Kurrent didn't. Connectors run in-server, at-least-once, shared retry/backoff resilience config; HTTP sink POSTs events individually (no batching). +- **Samples, not product:** event-driven multi-agent coordination via persistent subscriptions (Sept 2025, no idempotency/cost-on-replay guidance); LangGraph checkpointer persisting agent state as KurrentDB events with export-run-as-OTel-trace ("not production-ready"). +- **Capacitor** (June 2026, private preview): "shared memory for coding agents" — records agent sessions as immutable events, MCP-queryable; Kurrent dogfooding event-store-as-agent-memory as a standalone dev tool. + +## Cross-cutting takeaways → Wolverine/Marten opportunity map + +1. **Durable LLM-callout primitive (no prior art anywhere).** A memoized "AI side effect": Wolverine handler/subscription makes the IChatClient call once, records the response durably (as an event or inbox-adjacent record), and retries/replays reuse the recorded response instead of re-billing. Combines outbox, idempotency, and cost control in one feature. +2. **Replay-awareness surfaced to handler code.** Axon's `@DisallowReplay`/`ReplayStatus` is the proven shape; the Marten/Polecat equivalent is an `IsReplay`/rebuild flag on subscription context + a policy that suppresses or memoizes callouts during projection rebuilds. Marten subscriptions (at-least-once, checkpointed, retry-capable) are the right home — never inline projections (determinism). +3. **Wolverine handlers as the AI-gateway pipeline.** A `Wolverine.AI` package (abstractions-only dependency): error policies → model/provider-granularity fallback; scheduled retries → durable retry of model calls; outbox → transactional AI side effects; OTel + token/cost accounting as first-class telemetry; budget policies per endpoint/tenant/message-type. MassTransit's absence makes this a first-mover play. +4. **MCP-over-the-event-store is table stakes.** Kurrent's 8-tool server and AxonIQ's Insights both shipped it; both keep agents off the raw store (dev-time scoping / derived replica / commands-only). A Marten MCP server (read streams, author + self-test C# projections, write test events) matches the flagship deliverable of both vendors — with a stronger self-test loop than Kurrent's JS projections. Adopt Axon's policy verbatim: agents enter via Wolverine handlers/HTTP endpoints, never append events directly. +5. **"Event store as agent memory" is the shared marketing hill; implementations are thin.** Marten + pgvector could make it concrete (embedding-per-event via `IEmbeddingGenerator` from a subscription, semantic retrieval over history) in a way neither vendor shipped. The LangGraph-checkpointer + rebuild-run-as-OTel-trace sample is a cheap, high-visibility .NET equivalent (Marten-backed agent-state store). +6. **Keep OSS core AI-free; ship AI surface in add-on packages / CritterWatch-style tooling** — the monetization pattern both vendors follow. + +## Open questions (no prior art found) + +- Streaming responses (`GetStreamingResponseAsync`) vs durable outbox: semantics for a partially-consumed stream. Spring's dual-interface split frames the problem; nobody has durability answers. +- Budget-exceeded → backpressure: mapping cost caps onto queue/rate-limiting semantics. +- Sub-call checkpoint granularity (per tool invocation inside an agentic loop). +- Sequence-aware parking + expensive callouts: combining Axon-style sequence DLQ semantics with memoized AI side effects in Marten subscriptions. + +## Coverage caveats + +Angles that produced no *verified* claims in the main workflow (unverified ≠ absent): the dotnetdigest seed post itself, NServiceBus/Rebus/Brighter, Dapr Conversation API/dapr-agents, Kafka/Flink inference detail (Confluent's Flink SQL `ML_PREDICT` LATERAL-join pattern appeared in extraction but wasn't in the verified top-25), Akka.NET agentic patterns, Temporal/Restate beyond blog level, LangGraph/SK process framework. OTel GenAI semconv is experimental. MassTransit finding is an absence claim (public announcements/search, not commercial source audit). Spring AI 2.0 and MAF durable workflows are ≤2 months old — re-verify API surfaces before implementing. diff --git a/AZURESERVICEBUS-PERF-DEEP-DIVE-PLAN.md b/AZURESERVICEBUS-PERF-DEEP-DIVE-PLAN.md new file mode 100644 index 000000000..964c85ffc --- /dev/null +++ b/AZURESERVICEBUS-PERF-DEEP-DIVE-PLAN.md @@ -0,0 +1,186 @@ +# Azure Service Bus Performance Deep-Dive Plan (2026-07-18) + +Goal: measure Wolverine-over-ASB overhead vs the raw Azure.Messaging.ServiceBus SDK across +endpoint modes, validate the just-landed PrefetchCount work before release, fix the bug-shaped +session findings, and produce throughput-tuning guidance. Companion/umbrella: +`KAFKA-PERF-DEEP-DIVE-PLAN.md` (wolverine#3490) — shared metrics semantics, resolution-chain +costs, BatchedSender mechanics, back-pressure behavior live there. + +All file:line cites verified against `main` @ `a53be88c5` (post-6.20.0 — PrefetchCount is +**unreleased**). + +--- + +## 0. Transport-specific facts that frame everything + +- **Three listener shapes** (`AzureServiceBusTransport.Listening.cs:62-141`): sessions → + hand-rolled `AzureServiceBusSessionListener`; Inline → `ServiceBusProcessor`; Buffered/Durable + (default) → `ServiceBusReceiver` batch loop pulling `MaximumMessagesToReceive` = **20** per + `MaximumWaitTime` = **5s** (`AzureServiceBusEndpoint.cs:53,61`), hardcoded 250ms idle sleep. +- **The batched listener hands the receiver `Envelope[]`** (`BatchedAzureServiceBusListener.cs:160`) + → durable mode uses the batched multi-VALUES inbox insert. Good. Settlement, however, is + **per-message** `CompleteMessageAsync` through a concurrency-1 RetryBlock + (`BatchedAzureServiceBusListener.cs:44-45`, `DurableReceiver.cs:660-664`) — 20 messages per + receive, 20 settlement round trips, serialized. +- **PrefetchCount just landed** (GH-3471 / #3488, commit `a53be88c5`): endpoint + transport-wide + default, applied to processor/receiver/session-receiver options + (`AzureServiceBusEndpoint.cs:39,72-85`, `AzureServiceBusTransport.Listening.cs:150-187`). + Default remains 0. **Unreleased — this plan's Wave 1 doubles as its pre-release validation.** +- **Inline mode never sets `MaxConcurrentCalls`** → SDK default **1** concurrent handler per + processor; only reachable via `ConfigureProcessor`. Inline ASB is single-threaded per endpoint + out of the box (RabbitMQ-analogous). Also `AutoCompleteMessages` is left at SDK default *true* + with a structural double-settlement risk when the first settle attempt fails and falls to the + background retry block. +- **Sessions are hand-rolled and quadratic**: `RequireSessions(n)` sets `ListenerCount` + (`AzureServiceBusQueueListenerConfiguration.cs:170-182`), ListeningAgent builds n session + listeners (`ListeningAgent.cs:365-375`), and **each** spawns n accept loops + (`SessionSpecificListener.cs:36-47`) → **n² concurrent `AcceptNextSessionAsync` loops**. + Each accepted session does ONE `ReceiveMessagesAsync` batch, processes sequentially inline, + then disposes the session receiver (lock churn + AMQP link create/teardown per batch, + `SessionSpecificListener.cs:84-88,226-228`). No `ServiceBusSessionProcessor`, no + `MaxConcurrentSessions`/`MaxConcurrentCallsPerSession` (MassTransit has both). +- **No lock renewal on the batched path** (zero `RenewMessageLockAsync` hits): messages queued + behind `MaxDegreeOfParallelism` workers hold locks that silently expire; `CompleteAsync` + swallows lock-invalid errors (`AzureServiceBusEnvelope.cs:43-51`) → quiet redelivery + (durable dedups via inbox; buffered already completed-on-receipt). Inline's remedy is + `MaxAutoLockRenewalDuration` via `ConfigureProcessor` (SDK default 5 min). +- **Sender**: real `ServiceBusMessageBatch` + `TryAddMessage` with size-limit handling and + partial-failure accounting (`AzureServiceBusSenderProtocol.cs:55-111`) — the best batch + protocol of the four transports — but behind the 250ms debounce and + `MessageBatchMaxDegreeOfParallelism` = 1. **Inline sender = `SendMessageAsync` per envelope** + (`InlineAzureServiceBusSender.cs:46-52`) and it backs EVERY requeue/defer/retry path. + Partitioned entities: per-batch LINQ `GroupBy` on SessionId + fail-fast on first failed group + (`AzureServiceBusSenderProtocol.cs:113-185`). +- **Back-pressure stop** closes the receiver/processor; client-prefetched undelivered messages + drop with locks left to expire → DeliveryCount increments every cycle. With PrefetchCount now + real, aggressive prefetch + small `BufferingLimits` can push messages toward MaxDeliveryCount + dead-lettering (the new XML docs warn about prefetch-vs-lock aging). +- Mapper: `Body.ToArray()` copy per message, all `ApplicationProperties` copied + reserved + headers double-read, `Values.ToArray()` per outgoing message (`AzureServiceBusEnvelopeMapper.cs:27-64`, + `EnvelopeMapper.cs:391-397`); `GroupId ↔ SessionId` is automatic (`:50`). + +## 1. Theories of overhead, ranked + +- **A1 (HIGH): receive ceiling ≈ 20/RTT per listener with prefetch 0.** Prediction: the new + PrefetchCount is the single biggest throughput lever on ASB; Wave 1 produces the numbers that + justify (a) shipping it and (b) a recommended default guidance (likely 2-3× the batch size, + bounded by lock duration). +- **A2 (HIGH): per-message settlement serialization** — 20 completes per receive through a + sequential RetryBlock; completion RTT gates the durable/buffered pipeline the same way SQS + deletes do. ASB has no batch-settlement API, so the fix is concurrency (AO4), not batching. +- **A3 (HIGH, sessions): n² accept loops + one-batch-per-accept churn.** Prediction: session + throughput is dramatically below both non-session ASB and MassTransit's session processor; + fixing the quadratic alone is a headline win, moving to `ServiceBusSessionProcessor` is the + real fix. +- **A4 (MEDIUM): inline = MaxConcurrentCalls 1** — same "slow by default" story as RabbitMQ + inline; docs + surfaced knob. +- **A5 (MEDIUM): lock expiry under load** (no renewal on batched path) — shows up as duplicate + executions and DeliveryCount climb rather than latency; measure duplicate rate vs + `MaxDegreeOfParallelism` × handler time vs lock duration. +- **A6 (MEDIUM): inline sender per-message sends** on requeue/defer/retry paths — failure-heavy + workloads pay per-message RTTs (plus defer semantics differ by listener: batched re-sends + WITHOUT completing the original — double-delivery window). +- **A7 (LOW-MEDIUM): sender debounce + 1 in-flight batch** (#3490 T1 family) and the + partitioned-entity GroupBy/fail-fast path. +- **A8 (LOW): mapper allocations** — quantify in the shared microbench suite; fixes shared with + Kafka O4. + +## 2. Harness + +`TransportPerfRig` ASB adapter + native twin on raw `Azure.Messaging.ServiceBus` +(`ServiceBusProcessor` with tuned `MaxConcurrentCalls`/prefetch — the twin represents the SDK +used *well*). **Broker: a real ASB namespace, Standard AND Premium tiers** — the local emulator +is not representative (known environmentally flaky in this repo's test history; sessions and +scheduling misbehave) and throughput quotas differ by tier. Record namespace tier + region per +run. Sandbox: dedicated queue prefix + auto-teardown; ASB Standard base cost is pennies per +million ops. + +## 3. Experiment matrix + +Baseline: non-session queue, Buffered, defaults (20/5s receive, prefetch 0, batch-send +100/250ms/1), 1Kb payloads, ~10ms handler. + +| # | Experiment | Theory | Levers | +|---|---|---|---| +| AE1 | Mode sweep: Buffered / Durable(PG) / Inline(default) / Inline+MaxConcurrentCalls=20 | A2,A4 | mode, `ConfigureProcessor` | +| AE2 | **PrefetchCount sweep: 0/20/60/200** per mode — the pre-release validation | A1 | new `PrefetchCount()` | +| AE3 | Settlement concurrency prototype (AO4) vs serialized completes | A2 | branch build | +| AE4 | Receive shape: MaximumMessagesToReceive 10/20/50 × MaximumWaitTime 1/5s | A1 | endpoint config | +| AE5 | Sessions: current listener n=1/3/5 (observe n² loops) vs post-AO2-fix vs `ServiceBusSessionProcessor` prototype; vs MassTransit same-box reference | A3 | branch builds | +| AE6 | Lock-expiry stress: handler 10s × MDOP 20 × lock 30s → duplicate rate | A5 | queue config | +| AE7 | Send: MessageBatchMaxDegreeOfParallelism 1/4/8 × debounce 250/50ms; partitioned entity on/off | A7 | endpoint config | +| AE8 | Failure path: 5% failures → requeue/defer churn (batched vs inline listener defer semantics) | A6 | error policy | +| AE9 | Back-pressure cycle with prefetch 200: DeliveryCount climb + DLQ drift | A1×A5 | `BufferingLimits` | +| AE10 | Mapper: default vs minimal; 100Kb payloads; Std vs Premium (1MB) | A8 | `UseInterop`, tier | +| AE11 | Sequencing shapes (§4) | — | see below | + +Output/archival rules identical to #3490. + +## 4. Sequencing / GlobalPartitioning-equivalents + +ASB is the one transport with a first-class broker sequencing primitive Wolverine already maps: +**sessions** (`GroupId ↔ SessionId` automatic). Shapes to benchmark: +1. **Sessions (fixed listener) + inline or `PartitionProcessingByGroupId`** — note today's + buffered path completes on enqueue and releases the session lock after the batch, so strict + per-session serial execution requires the pairing; document loudly. +2. **Non-session queue + `PartitionProcessingByGroupId(slots)`** — consumer-side only, no + session tax; likely the throughput winner when cluster-wide exclusivity isn't needed. +3. **`UseShardedAzureServiceBusQueues(base, N)`** (`AzureServiceBusTransportExtensions.cs:498-509`) + — N plain queues + Wolverine hash routing (forced Durable inside `GlobalPartitioned`); + quantify vs sessions. +Deliverable: sessions-vs-sharding-vs-local-partitioning decision table — this is also the +competitive-positioning story vs MassTransit's session support (capability-doc crossover: +richer session surface is a named opportunity there). + +## 5. Optimization backlog (gated on matrix confirmation) + +- **AO1 (A3): rewrite the session listener on `ServiceBusSessionProcessor`** with + `MaxConcurrentSessions`/`MaxConcurrentCallsPerSession` surfaced. Fixes churn, the n² + explosion, lock retention, and closes the MT feature gap in one move. +- **AO2 (A3, small + immediate): de-quadratic the current session listener** (spawn the accept + loops once, not per ListenerCount instance) — shippable before AO1. +- **AO3 (A4): surface `MaxConcurrentCalls`** on inline endpoints (+ align `AutoCompleteMessages` + handling to remove the double-settlement window). +- **AO4 (A2): parallel settlement** — widen the complete-block concurrency for ASB (settlement + order doesn't matter; locks are per-message). +- **AO5 (A5): lock renewal on the batched path** (bounded renewal while in local queue), or a + documented sizing rule (BufferingLimits × handler time < lock duration) enforced with a + startup warning. +- **AO6 (A1): PrefetchCount guidance + defaults** from AE2 data; ship in the same release as + #3488 so the feature lands with numbers. +- **AO7 (A8): mapper fixes** (shared with Kafka O4). +- **AO8 (A6): batched-listener defer should settle the original** (complete-then-resend like + inline does) — correctness alignment. + +## 6. Measured-wins ledger (for release notes / blog posts) + +Same rules as #3490 §9: archived runs only (namespace tier + region recorded), same-rig +before/after, release-note phrasing; negative results logged below. **AE2's prefetch numbers +feed the #3488/GH-3471 release notes directly** — that's the first ledger entry to chase. + +| Optimization | PR | Scenario (AE-cell) | Metric | Before | After | Release-note one-liner | +|---|---|---|---|---|---|---| +| _(PrefetchCount validation (#3488), AO2 session de-quadratic, AO4 settlement concurrency, ... — rows added as measured)_ | | | | | | | + +## 7. Sequencing & exit criteria + +- **Wave 0 (now)**: rig adapter + native twin; AO2 (de-quadratic) and AO8 (defer settle) are + small enough to land from code review + existing tests, pre-box. +- **Wave 1 (box + real namespace)**: AE1-AE4 + AE2 prefetch validation; dotTrace on + AE1-durable and AE5-current. + Exit: prefetch numbers ready for the #3488 release notes; A1/A2 confirmed or killed. +- **Wave 2**: AE5-AE9 sessions + stress cells; sequencing decision table. +- **Wave 3**: AO1/AO3/AO4/AO5 landed + re-measured; ledger filled. +- **Wave 4**: update `docs/guide/messaging/transports/azureservicebus/performance.md` (seeded + 2026-07-18 with qualitative guidance; prefetch subsection badged 6.21) with measured numbers + + release notes/blog from the ledger. + +## 8. Risks / honesty notes + +- Emulator numbers are never publishable; sessions/scheduling don't behave there (known local + flake history — CI/real namespace is the signal). +- Standard vs Premium changes message size limits, throughput units, and latency floors — + every ledger row states the tier. +- Session cells depend on group-count distribution; state it per cell. +- PrefetchCount is unreleased: AE2 findings may change its defaults/docs before it ships — + that's the point, but sequence the release accordingly. diff --git a/CONJOINED-TENANCY-EPIC-PLAN.md b/CONJOINED-TENANCY-EPIC-PLAN.md new file mode 100644 index 000000000..2108e9729 --- /dev/null +++ b/CONJOINED-TENANCY-EPIC-PLAN.md @@ -0,0 +1,374 @@ +# Epic Plan: Conjoined Multi-Tenancy for EF Core in Wolverine + +**Date:** 2026-07-18 +**Status:** FILED — all GitHub issues created 2026-07-18, pending Jeremy review + +**Issue map:** +- Master tracking: [wolverine#3465](https://github.com/JasperFx/wolverine/issues/3465) +- Workstream A: [jasperfx#531](https://github.com/JasperFx/jasperfx/issues/531) +- Workstream B: [weasel#362](https://github.com/JasperFx/weasel/issues/362) +- Workstream C: Phase 1 [wolverine#3462](https://github.com/JasperFx/wolverine/issues/3462), Phase 2 [wolverine#3463](https://github.com/JasperFx/wolverine/issues/3463), Phase 3 [wolverine#3464](https://github.com/JasperFx/wolverine/issues/3464) (Phase 4 tracked on #3465) +- CritterWatch: [CritterWatch#720](https://github.com/JasperFx/CritterWatch/issues/720) +- Workstream D: [polecat#335](https://github.com/JasperFx/polecat/issues/335) +**Motivating context:** https://barretblake.dev/posts/development/2026/07/multi-tenant-part-1/ — +hand-rolled shared-database tenancy in EF Core 10 (tenant column + named query filters + raw-SQL +partition DDL inside EF migrations). Everything the author does manually, the critter stack +already automates for Marten. This epic brings that automation to EF Core users through +Wolverine, with Weasel owning partition DDL and CritterWatch owning tenant operations. + +## Locked decisions (Jeremy, 2026-07-18) + +1. **`ITenanted` lives in JasperFx.** Promote `ITenanted : IHasTenantId` into + `JasperFx.MultiTenancy`; Marten and Polecat re-point their identical local markers at it + (same dedupe motion as jasperfx#224 did for `IHasTenantId`). Wolverine EF Core keys off the + JasperFx interface, so one marker works across Marten documents, Polecat documents, and EF + entities. +2. **SQL Server gets physical partitioning in v1** of the Wolverine feature, via + `Weasel.SqlServer.ManagedTenantPartitions`. A follow-up epic brings **Polecat** to full + parity with Marten's per-tenant partitioning (today Polecat only partitions the events + table; document tables are column+filter only and explicitly block partitioning). +3. **Tenant lifecycle registry is Wolverine-owned.** A small `wolverine_tenants` table (not an + extension of Weasel's partition control tables) carries enable/disable state and is the + authoritative tenant list for conjoined mode. Weasel's partition registries + (`*_tenant_partitions` on PG, tenant-ordinal registry on SQL Server) stay implementation + details of the partitioning layer. +4. **Conjoined sagas are in scope** for this epic (tenant-scoped load/insert/update/delete + through the EF saga frames). + +## Goals + +- `ITenanted` on an EF entity ⇒ Wolverine makes it conjoined-multi-tenant with **zero** + hand-written filters: `tenant_id` column mapped, global query filter bound to the ambient + Wolverine tenant, tenant stamped on insert, cross-tenant writes rejected. +- Opt-in **Weasel-managed physical partitioning** of tenanted tables on both PostgreSQL + (list partitions, value→suffix bucketing) and SQL Server (tenant-ordinal RANGE RIGHT). +- **Behavioral compliance with Marten/Polecat** conjoined semantics: `*DEFAULT*` sentinel, + `TenantIdStyle` correction, stamp-on-write/hydrate-on-load, tenant-scoped deletes, + additive-only partition migration. +- **CritterWatch tenancy management works out of the box**: add/disable/enable/remove/list + tenants from the Tenants tab against a conjoined EF Core app, via a Wolverine-provided + `IDynamicTenantSource`. +- Conjoined **sagas**. + +## Non-goals + +- No change to the existing DB-per-tenant EF Core mode + (`AddDbContextWithWolverineManagedMultiTenancy`); conjoined is a sibling mode, and the two + are mutually exclusive per DbContext. +- No schema-per-tenant mode (the blog's Part 2 topic) — possible future work, not this epic. +- No EF-migrations authoring of partition DDL. Partitioned conjoined contexts require the + Wolverine/Weasel-managed migration path; plain conjoined (no partitioning) works with either + EF migrations or Weasel-managed migrations. +- MySQL/Oracle/SQLite partitioning (Weasel has none; conjoined column+filter mode still works + anywhere EF Core does). + +--- + +## Workstream A — JasperFx: promote `ITenanted` + +Small, but it leads the release train (critter-stack versions move in lockstep). + +- Add `public interface ITenanted : IHasTenantId {}` to `JasperFx.MultiTenancy` with the + settable-`TenantId` doc contract both libraries already imply. +- Marten: retype `Marten.Metadata.ITenanted` as the JasperFx interface via + `[TypeForwardedTo]`/alias (pattern already used for `TenancyStyle`). `TenancyPolicy` + unchanged. +- Polecat: same for `Polecat.Metadata.ITenanted`. +- Sanity check: nothing in either library depends on the marker being locally declared + (both are already empty extensions of `IHasTenantId`). + +**Releases required:** JasperFx minor → Marten + Polecat patch/minor pin bumps. + +### Draft issue (jasperfx repo) — *placeholder wording, review before filing* + +> **Title:** Promote `ITenanted` marker into JasperFx.MultiTenancy +> **Body:** Marten (`Marten.Metadata.ITenanted`) and Polecat (`Polecat.Metadata.ITenanted`) +> declare identical empty markers extending `IHasTenantId`. Wolverine is about to need the +> same marker for conjoined EF Core tenancy. Move the interface to `JasperFx.MultiTenancy` +> so one marker drives conjoined behavior across all three, and have Marten/Polecat forward +> to it. Follows the `IHasTenantId` dedupe in jasperfx#224. + +--- + +## Workstream B — Weasel.SqlServer: close the managed-partitioning gaps + +`ManagedTenantPartitions` already exists (built for Polecat events, weasel#301) and is the +right mechanism. Before Wolverine and Polecat lean on it for *many* tables per app, audit and +close these gaps against the PG `ManagedListPartitions` feature set: + +| # | Gap / audit item | PG behavior today | SQL Server today | +|---|---|---|---| +| B1 | **Drop semantics** | `DETACH PARTITION [CONCURRENTLY]` + `DROP TABLE` — tenant data is physically removed | `ALTER PARTITION FUNCTION … MERGE RANGE` — boundary disappears but **rows survive** into the neighboring partition. Decide + document: managed drop should (optionally?) `DELETE` the tenant's rows before merging, or the Wolverine/Polecat layer does the delete. Needed for CritterWatch `RemoveTenant`/hard-delete parity | +| B2 | **Bucketing** (many tenants → one partition) | `partition_value → partition_suffix` mapping is many-to-one by design | ordinal allocation is strictly `max+1` per tenant — no sharing. Add optional tenant→ordinal assignment (explicit ordinal on add) so buckets are possible; this is also the mitigation for the 15k-partition ceiling the blog post calls out | +| B3 | **Disabled-tenant awareness** | none (Marten layers it elsewhere) | none — fine; lifecycle stays in the Wolverine/Polecat registry (decision 3), no Weasel change | +| B4 | **Delta/migration ergonomics** | managed tables use `IgnorePartitionsInMigration`; additive-only runtime path documented (marten#4706/#4713) | `TableDelta` deliberately skips managed strategies — verify a *new* table added to an existing managed set gets back-filled with all existing tenant ordinals on migration (the PG additive path has explicit handling; confirm/port) | +| B5 | **Multi-table batch add** | `AddPartitionToAllTables(logger, db, dict)` returns `TablePartitionStatus[]` | overloads exist; confirm status reporting parity so Wolverine can surface per-table results to CritterWatch | + +Deliverable: short gap-closure PR(s) to Weasel + a documented contract that both Polecat and +Wolverine build on. + +**Releases required:** Weasel minor (before Wolverine Phase 2 and the Polecat epic). + +### Draft issue (weasel repo) — *placeholder wording* + +> **Title:** `ManagedTenantPartitions` (SQL Server) parity audit vs `ManagedListPartitions` +> **Body:** Polecat currently uses `ManagedTenantPartitions` for the events table only. +> Wolverine's conjoined EF Core tenancy epic and the Polecat document-partitioning follow-up +> will apply it across many tables per application. Close the gaps: (1) data-removing drop +> semantics (MERGE RANGE leaves rows behind, unlike PG detach+drop); (2) optional explicit +> ordinal assignment to allow tenant bucketing; (3) confirm new-table back-fill of existing +> ordinals under migration; (4) batch add/status parity. + +--- + +## Workstream C — Wolverine.EntityFrameworkCore: the epic proper + +### Phase 1 — Conjoined core (no partitioning yet) + +New registration mode in `WolverineEntityCoreExtensions`: + +```csharp +// name TBD — see interview questions +services.AddDbContextWithWolverineManagedConjoinedTenancy( + (builder, connectionString) => builder.UseNpgsql(connectionString)); +``` + +Components: + +1. **Model convention** — extend `WolverineModelCustomizer` (already swapped in as EF's + `IModelCustomizer` by every Wolverine registration path): for each entity implementing + `JasperFx.MultiTenancy.ITenanted`: + - map `TenantId` → `tenant_id` (`StorageConstants.TenantIdColumn`), default + `*DEFAULT*` (`StorageConstants.DefaultTenantId`); + - attach a **global query filter** `e.TenantId == context.TenantId` bound to a + `CurrentTenant` accessor on the context (see 3); + - index note: add `tenant_id` to the entity's key/index shape only in Phase 2 + (partitioning) — plain conjoined keeps the user's PK and adds a `tenant_id` index. +2. **`SaveChangesInterceptor`** (`TenantStampingInterceptor`): + - on **Added**: stamp `TenantId` from the ambient tenant (after `TenantIdStyle` + correction); explicit non-empty `TenantId` different from ambient ⇒ throw (matches + Marten's conjoined write semantics — sessions write their own tenant only); + - on **Modified/Deleted**: if the entry's `TenantId` ≠ context tenant ⇒ throw + `CrossTenantWriteException` (name TBD). This is the "no forgotten + `IgnoreQueryFilters()`" guarantee on the write side; + - rejects writes for **disabled** tenants (registry check, Phase 3). +3. **Tenant-pinned DbContext** — a conjoined `IDbContextBuilder` + (`ConjoinedDbContextBuilder`): single database (connection string from the app's + single message store), but every built context is pinned to `MessageContext.TenantId` + before handing to user code. Registering `IDbContextBuilder` means the existing + `EFCorePersistenceFrameProvider` multi-tenant codegen branch + (`CreateTenantedDbContext<>` + `StartDatabaseTransactionForDbContext`) works **unchanged** + — the builder is where conjoined vs DB-per-tenant differ, not the frames. + - Tenant flow already exists end-to-end: `Envelope.TenantId` → `MessageContext.TenantId` + with `TenantIdStyle.MaybeCorrectTenantId`, HTTP `ITenantDetection` + (`opts.TenantId.IsQueryStringValue(...)` etc.), gRPC metadata detection. No new + detection work. + - Mechanism for pinning: prefer an injected `IWolverineTenantContext` (scoped) the filter + lambda closes over, so users' own DbContext ctors don't change; fall back to a + `WolverineDbContext` base-class property only if the filter-through-service approach + fights EF's filter caching. (EF caches the filter expression per model; the + tenant value must come from per-instance state — standard pattern is a field/property + on the context populated at construction, which the builder does.) +4. **Outbox/transaction paths**: `EfCoreEnvelopeTransaction`, `IDbContextOutboxFactory. + CreateForTenantAsync`, and `DbContextOutbox.TenantId` already carry tenant ids — audit + that the conjoined builder path threads `TenantId` into all three (the factory currently + assumes tenant ⇒ different database; conjoined means tenant ⇒ same database, pinned + context). +5. **Conjoined sagas** (decision 4): + - A saga type implementing `ITenanted` gets tenant-scoped persistence through + `EFCorePersistenceFrameProvider`: + - `DetermineLoadFrame`/`LoadEntityFrame`: load by (saga id + tenant). If the global + query filter is active on the pinned context this may come for free — **verify + `FindAsync`/keyed loads respect global query filters**; if not, generate an explicit + `Where(id && tenant)` load; + - insert: stamped by the interceptor; + - update/delete + `IncrementSagaVersionIfNecessary` + `WrapSagaConcurrencyException`: + unchanged, but the cross-tenant guard applies; + - Identity stays the user's saga id (no composite-key requirement in Phase 1); a + uniqueness decision is needed for Phase 2 partitioned sagas (see interview Q3). +6. **Message storage is untouched**: conjoined = one database ⇒ the plain single + `PostgresqlMessageStore`/`SqlServerMessageStore`. No `MultiTenantedMessageStore`. + Envelopes already persist `TenantId` for context restoration. + +**Tests:** new `EfCoreTests.ConjoinedTenancy` battery + HTTP integration tests mirroring +`multi_tenancy_detection_and_integration.cs`; compliance assertions ported from Marten's +conjoined tests (default-tenant sentinel, style correction, stamping, hydration, +tenant-scoped delete, cross-tenant rejection). Both PG (5433) and SQL Server (1434), conn +strings via `Servers`. + +### Phase 2 — Weasel-managed physical partitioning (opt-in) + +```csharp +services.AddDbContextWithWolverineManagedConjoinedTenancy( + ..., o => o.UsePartitioning()); // shape TBD +``` + +- **Migration ownership**: partitioned conjoined contexts **require** + `UseEntityFrameworkCoreWolverineManagedMigrations()` / + `EntityFrameworkCoreSystemPart`. EF migrations cannot express PG declarative partitioning + or SQL Server partition schemes — this is exactly the raw-SQL hack the blog resorts to. + Guard with a clear bootstrap error if partitioning is on and EF migrations are. +- **Translation**: from the EF `IModel`, build Weasel `Table`s for each `ITenanted` entity + (via `Weasel.EntityFrameworkCore`, already referenced): + - PG: `PARTITION BY LIST (tenant_id)`, + `.UsePartitionManager(managedListPartitions)`, `IgnorePartitionsInMigration = true`; + a `ManagedListPartitions` feature with control table + `wolverine_tenant_partitions` in the durability schema (Marten uses + `mt_tenant_partitions`; ours is Wolverine-owned and separate); + - SQL Server: `PartitionByManagedTenants(managedTenantPartitions)` with the ordinal + registry table; the model convention adds an `int tenant_ordinal` **shadow property** + to `ITenanted` entities, and `TenantStampingInterceptor` stamps it from + `ManagedTenantPartitions.Ordinals[tenantId]`. +- **PK shape**: partition column must be in the PK/unique keys. The convention rewrites + `ITenanted` entity keys to composite (`tenant_id`/`tenant_ordinal` + id), with an + ordering option mirroring Marten's `PrimaryKeyTenancyOrdering` (default + `TenantId_Then_Id`, Marten's V9 default). +- **Bucketing**: expose Marten-style tenant→suffix mapping on the add-tenant API + (`AddTenantAsync(tenantId, partitionSuffix)`-shaped) so N tenants can share a partition — + the answer to SQL Server's 15k-partition ceiling and to "small tenants don't deserve their + own partition". SQL Server bucketing depends on Weasel gap B2. +- **Suffix/identifier hygiene**: reuse Weasel's `ListPartition.SanitizeSuffix` + port + Marten's PG identifier-legality and 63-byte checks. + +### Phase 3 — Tenant registry, `IDynamicTenantSource`, CritterWatch + +- **`wolverine_tenants` registry table** (Wolverine-owned, durability schema; created as a + Weasel `FeatureSchemaBase` + `IDatabaseInitializer` like the partition registries): + `tenant_id varchar PK`, `is_disabled bit`, `partition_suffix varchar null`, + `created_utc`/`modified_utc`. Authoritative tenant list for conjoined mode — exists in + **both** plain-conjoined and partitioned-conjoined (so CritterWatch tenant management + works even without partitioning). +- **`ConjoinedTenantSource : IDynamicTenantSource`**, registered in DI (that + registration alone lights up CritterWatch's satellite handlers — they resolve and fan out + over all `IDynamicTenantSource`s, no CritterWatch handler changes): + - `AddTenantAsync(tenantId)` — registry insert; if partitioned, also + `AddPartitionToAllTables` (+ per-suffix bucketing). CritterWatch's existing + auto-assign branch (empty connection string) maps to exactly this call — built for + Marten sharded tenancy, fits conjoined perfectly; + - `AddTenantAsync(tenantId, connectionString)` — invalid for conjoined ⇒ clear error + ("conjoined tenants share the application database"); + - `Disable/Enable` — registry flag; disabled tenants rejected at context-build and + interceptor time (`UnknownTenantIdException` parity with Marten's master-table + behavior); + - `RemoveTenantAsync` — registry delete + (partitioned) partition drop with data removal + (PG today; SQL Server pending Weasel B1); + - `FindAsync`/`AllActiveByTenant`/`AllDisabledAsync`/`RefreshAsync` — registry reads; + the "connection value" returned is the shared app connection (masked in descriptors, + as `TenantedDbContextUsageSource` already does). +- **Admin convenience API** mirroring Marten: + `host.AddWolverineManagedTenantsAsync(params string[] tenantIds)` / + `(Dictionary tenantToSuffix)` / `RemoveWolverineManagedTenantsAsync(...)` + returning Weasel `TablePartitionStatus[]`. +- **Descriptors**: extend `TenantedDbContextUsageSource`/`DbContextUsage` output so the + conjoined context advertises `DatabaseCardinality.DynamicMultiple` + tenant ids on the + single `DatabaseDescriptor`. +- **CritterWatch repo work** (separate issues there): + 1. tenant action-strip gating (`hasDynamicTenancy`) currently consults **event-store** + descriptors only — teach it to consult `DbContextUsage`/document-store descriptors + with `DynamicMultiple` cardinality; + 2. skip the Postgres-hardcoded `CREATE DATABASE`/`DROP DATABASE` provisioning branches + when the source is conjoined (no connection string in play; hard-delete for conjoined + = partition drop + row purge, executed app-side by the source); + 3. Tenants-tab columns already merge per-tenant metrics/DLQ counts keyed by tenant id — + verify they populate from a conjoined app (single database URI for all rows). + +### Phase 4 — Compliance battery, docs, samples + +- Compliance-test suite asserting parity with Marten conjoined semantics (shared test list + reviewed against `conjoined_tenancy_tests` in Marten and Polecat). +- Sample app: `ConjoinedMultiTenantedEfCore` (sibling of `MultiTenantedEfCoreWithPostgreSQL`), + HTTP tenant detection + partitioning + CritterWatch capabilities. +- Docs: new page under EF Core integration ("Conjoined multi-tenancy and tenant + partitioning") + blog post announcing the feature (this epic is a strong headline: it is a + direct, named answer to a community pain point). + +--- + +## Workstream D — Polecat: full Marten per-tenant partitioning parity (follow-up epic) + +Today: Polecat partitions **events only** (`UseTenantPartitionedEvents` → +`ManagedTenantPartitions`, per-tenant sequences); `DocumentTable` explicitly throws for +partitioning + conjoined ("RANGE partitioning … single-tenant tables only"); there is no +Marten-style `PartitionMultiTenantedDocumentsUsingMartenManagement` equivalent. + +Scope for the follow-up epic (own plan doc when picked up): + +1. Managed tenant partitioning for **document tables** (lift the `DocumentTable` restriction; + `tenant_ordinal` column + shared `ManagedTenantPartitions` feature, one registry per + database). +2. **Streams-table** partitioning to match Marten (`StreamsTable` takes the same managed + partitioning as events — audit what `pc_streams` does today). +3. Policy-level API parity: `AllDocumentsAreMultiTenantedWithPartitioning(...)`, + `PartitionMultiTenantedDocumentsUsingPolecatManagement(...)` naming TBD. +4. Runtime API parity: `store.Advanced.AddPolecatManagedTenantsAsync(...)` / + `Remove...` with `TablePartitionStatus[]` returns, `__default__`-style reserved partition + for global projections if the Marten behavior applies. +5. `RemoveTenant` data semantics depend on Weasel B1 (MERGE RANGE leaves rows). +6. CritterWatch: SQL Server hard-delete path is already tracked as CritterWatch#68 — this + epic is its prerequisite. + +Dependencies: Weasel Workstream B first; independent of Wolverine Phases 1–3 (parallel +track). + +### Draft issue (polecat repo) — *placeholder wording* + +> **Title:** Per-tenant managed partitioning parity with Marten (documents + streams) +> **Body:** Polecat supports Weasel `ManagedTenantPartitions` for the events table only +> (`UseTenantPartitionedEvents`, #163/#171). Marten additionally offers managed tenant +> partitioning for document tables and the streams table, policy APIs +> (`AllDocumentsAreMultiTenantedWithPartitioning`, +> `PartitionMultiTenantedDocumentsUsingMartenManagement`), and runtime tenant onboarding +> (`AddMartenManagedTenantsAsync`). Bring Polecat to parity: lift the DocumentTable +> conjoined-partitioning restriction, partition pc_streams, add the policy + runtime APIs, +> and define tenant-removal data semantics (SQL Server MERGE RANGE does not remove rows — +> see weasel parity issue). Prerequisite for CritterWatch#68 (SQL Server hard delete). + +--- + +## Sequencing & release train + +``` +A. JasperFx: ITenanted promotion ──┐ (small; rides next lockstep release) +B. Weasel.SqlServer gap closure ──┼──> Marten/Polecat pin bumps + │ +C1. Wolverine Phase 1 (conjoined core+sagas)│ needs A only +C2. Wolverine Phase 2 (partitioning) │ needs B (SQL Server), PG ready today +C3. Wolverine Phase 3 (registry + CW) │ needs C1; CW repo issues in parallel +C4. Wolverine Phase 4 (compliance/docs) │ + │ +D. Polecat parity epic ──┘ needs B; parallel to C2–C4 +``` + +Phase C1 is independently shippable and already beats the hand-rolled blog approach; C2/C3 +are each release-noteworthy on their own. + +## Risks / verify-early items + +- **EF global query filters + `FindAsync`**: confirm keyed loads respect filters (saga load + correctness depends on it) — spike in Phase 1 week 1. +- **EF filter caching vs per-instance tenant**: the filter must close over per-context + state, not a captured value at model-build time — standard pattern, but verify against + EF Core 10 named filters too (nice interplay: users can *also* declare their own named + filters without colliding with ours). +- **Composite-PK rewrite (Phase 2)** changes `FindAsync`/`Attach` call shapes for user code + that loads by key — needs explicit docs + analyzer-grade error messages. +- **SQL Server ordinal stamping** requires `Ordinals` to be loaded before first write per + tenant — the interceptor needs a synchronous-safe lookup path (pre-hydrated at startup + + refresh on add), same pattern as Polecat's `TenantEventSequenceRegistry`. +- **Marten + EF conjoined in one app**: both can be conjoined against the same database; + ensure the two partition control tables (mt_* vs wolverine_*) coexist and CritterWatch + fans out to both sources without double-adding (its fan-out is by design — document that + "add tenant" hits both). + +## Interview questions for Jeremy (naming/wording — placeholders used above) + +1. Registration API name: `AddDbContextWithWolverineManagedConjoinedTenancy` is a + mouthful. Alternatives: `AddConjoinedMultiTenantedDbContext`, + `AddDbContextWithConjoinedTenancy`. +2. Registry/control table names: `wolverine_tenants` + `wolverine_tenant_partitions` in the + durability schema — good, or prefix differently? +3. Partitioned sagas: keep saga id globally unique (simpler frames) or allow per-tenant id + reuse (composite identity — more work in `DetermineSagaIdType`/load frames)? +4. Exception naming: `CrossTenantWriteException` vs reusing an existing JasperFx type. +5. Should plain-conjoined (no partitioning) also *require* Wolverine-managed migrations, or + stay EF-migrations-friendly as drafted? diff --git a/CONNECTION-BUDGET-3397-PLAN.md b/CONNECTION-BUDGET-3397-PLAN.md new file mode 100644 index 000000000..44e7d19c5 --- /dev/null +++ b/CONNECTION-BUDGET-3397-PLAN.md @@ -0,0 +1,276 @@ +# Connection-budget awareness plan (wolverine#3397) + +> **STATUS 2026-07-18 — Phase 1 SHIPPED (6.19.0); #3376 fix SHIPPED (6.20.0); both issues CLOSED.** +> **The only unfinished Phase-1 deliverable is the CritterWatch consumer.** Phase 2 stays parked +> pending erdtsieck's re-baseline on 6.20.0 (per the closing notes on both issues; when it resumes, +> file a fresh focused issue rather than reopening #3397). +> +> | Repo | PR | State | +> |------|----|-------| +> | wolverine | [#3422](https://github.com/JasperFx/wolverine/pull/3422) — Workstreams A/B/C + P(b), docs, 22 tests | **merged, shipped in 6.19.0** | +> | jasperfx | [#514](https://github.com/JasperFx/jasperfx/pull/514) — `Port` on `DatabaseDescriptor` | **merged, released**; `DatabaseServerId.From` now reads `descriptor.Port` (consolidation done) | +> | jasperfx | [#521](https://github.com/JasperFx/jasperfx/pull/521) — resizable governors + tenant-HWM re-read + `ConcurrencyException` ctor | **merged, in V2.28.0** → Wolverine pins 2.29.0, so Wave 2 is fully released and Wave 3.4's gate is cleared (also unblocks wolverine#3444) | +> | polecat | none needed | Workstream P resolved as (b) — no sharded tenancy, so nothing to activate | +> | CritterWatch | **still to do** | pin block gone (CW is on 6.20.0/2.29.0) but no `ConnectionBudget` consumer exists in CW source — budget snapshots are published and dropped on the floor | +> +> Issue timeline: plan posted 7/14; #3376 root-caused + fixed (#3439/#3440) 7/17; 6.20.0 shipped +> 7/17; #3376 and #3397 both closed 7/17 with re-baseline deferral notes. +> +> **Open questions, now answered (Jeremy, 2026-07-13):** +> 1. Workstream P → **(b)**, as the plan leaned. SQL Server gets probe plumbing + gauge parity; activation follows Polecat's tenancy roadmap. +> 2. Config surface → **fluent per-server**: `Durability.ConnectionBudgets.ForServer(host, port, maxConnections)`. +> 3. Naming → `wolverine-database-connection-count` (used) / `wolverine-database-connection-budget` (max), tag `server`. DTO `ConnectionBudgetSnapshot`, observer member `ConnectionBudget`. +> 4. Port gap → **derived in Wolverine's key**, not waiting on the descriptor. `DatabaseServerId(Engine, ServerName, Port?)` reads the port off the store's own connection-string builder, so 6.19.0 ships on the current JasperFx pin. jasperfx#514 closes the gap at the descriptor level for the rest of the ecosystem; Wolverine consolidates onto it later. +> +> **Deviation from the plan as written:** `DatabaseServerId.Port` is `int?`, not `int`. SQL Server's +> `Data Source` already carries the port (`host,1433`) or a named instance (`host\SQLEXPRESS`), so it +> leaves Port null and keeps the DataSource whole rather than inventing a split that would be ambiguous. + + +Working plan for [wolverine#3397](https://github.com/JasperFx/wolverine/issues/3397) — +"Adaptive connection-budget awareness: probe server max_connections/numbackends and back off +daemon polling/concurrency under pressure (CritterWatch-surfaced)". Reporter: @erdtsieck, same +512-tenant-database deployment as #3375/#3384 (shipped) and #3376 (parked). + +**Decisions from Jeremy (2026-07-13, binding):** + +1. Phase 1 (measure + surface) targets **Wolverine 6.19.0** — NOT the current 6.18.0 wave. +2. **Assume a pooler (pgBouncer) in front** for the immediate work → the connection budget's + `MaxConnections` is **explicit configuration**, not a probed `pg_settings` value. Probing + `max_connections` is at most a fallback/diagnostic when no explicit value is given. +3. **Engine parity from the start**: PostgreSQL AND SQL Server → brings **Polecat** into scope. +4. Server identity derives from the existing **`DatabaseDescriptor`** metadata (JasperFx + `Descriptors/DatabaseDescriptor.cs` — `ServerName`/`DatabaseName`/`DatabaseUri()`), not a new + connection-string parser. +5. The budget machinery is **only active when using Marten's sharded-database tenancy** + (`StoreOptions.MultiTenantedWithShardedDatabases(...)`) — the many-databases-per-server shape + is the only one where per-server budgeting pays for its complexity. + +--- + +## Ground truth from code recon (2026-07-13) + +Findings that reshape the issue as filed: + +- **The knobs the issue names are NOT Wolverine's.** `SlowPollingTime`/`FastPollingTime`, + `MaxConcurrentEventLoadsPerDatabase`, `MaxConcurrentBatchWritesPerDatabase` all live in + JasperFx.Events `DaemonSettings` (`~/code/jasperfx/src/JasperFx.Events/Daemon/DaemonSettings.cs:78-133` + — the epic #486 WS2/WS3 governors). The "adapt" half is therefore mostly a **JasperFx.Events + feature with Wolverine wiring**, pinned to a JasperFx release (2.28.0+ — same release already + earmarked for the #3376 lifecycle hooks). +- **Runtime mutability is split.** + - `HighWaterAgent` re-reads `FastPollingTime`/`SlowPollingTime` on every wait + (`HighWaterAgent.cs:115,140`) → daemon cadence is adaptable *today* by mutating settings. + - The #486 governors are `SemaphoreSlim`s sized at construction (`ThrottledEventLoader.cs:16-18`) + → resizing under pressure needs real JasperFx work (adaptive gate), not a settings tweak. + - Wolverine's `DurabilityAgent` captures `ScheduledJobPollingTime` once into + `System.Threading.Timer`s (`src/Persistence/Wolverine.RDBMS/DurabilityAgent.cs:114,267`) → + needs a re-reading loop (the #3384 sweeper already models this: `PersistenceMetricsSweeper.cs:100` + re-reads `UpdateMetricsPeriod` per pass). +- **No server probe exists anywhere.** Nothing queries `pg_settings`/`pg_stat_database`; + `max_connections` appears only in comments (`AssignmentGrid.Distribution.cs:108`). + `PostgresqlMessageStore.FetchCountsAsync` (`PostgresqlMessageStore.cs:216`) already does + `pg_catalog` introspection — the template for the probe. +- **No "server" concept exists.** Grouping today is per logical database + (`AssignmentGrid.DistributeByGroupAffinity`, `EventSubscriptionAgentFamily.cs:183`). But + `IMessageStore.Describe()` (`IMessageStore.cs:131`) returns a `DatabaseDescriptor` whose + `ServerName` is filled from the connection string host (`PostgresqlMessageStore.cs:558`), and + `DatabaseId(descriptor.ServerName, descriptor.DatabaseName)` already exists + (`PostgresqlMessageStore.cs:53`) → **server key = descriptor-derived**, per decision (4). + - ⚠️ Gap: `ServerName` is host only — **no port**. Two Postgres clusters on one host on + different ports would collide. Fold port into the descriptor (or the derived key) as part of + Phase 1. +- **The CritterWatch seam** is `IWolverineObserver.PersistedCounts` published from the #3384 + sweeper (`PersistenceMetricsSweeper.cs:126`). The sweeper is sequential per node → deduping the + probe per server key within a pass is natural. OTel side: + `PersistenceMetrics` `ObservableGauge`s (`PersistenceMetrics.cs:41-46`). +- **Polecat has no sharded-tenancy analog** (no `sharded` hits in `~/code/polecat/src`). Parity + scope for SQL Server needs its own trigger-condition definition (see Workstream P). + +## Relationship to #3376 (parked) + +#3376 (same reporter) is parked awaiting his answers on (a) whether `AddAsyncDaemon` is registered +alongside managed distribution (config-bug hypothesis that would explain the ~1,400-connection +baseline) and (b) the connection breakdown in `pg_stat_activity`. Two consequences: + +- **Phase 1 of this plan is the instrument that answers (b).** A per-server used/max gauge on the + metrics feed is pure observability, independent of #3376's resolution, and gives both sides the + data. Ship it first. +- **Phase 2 (adapt) waits** until #3376 resolves — the reporter himself scoped his prototype offer + as "once #3376 lands", and the owned-agent scoping changes the baseline any tuning works against. + The JasperFx hooks for both issues should ride the same JasperFx release. + +--- + +## Phase 0 — Reply + triage (now, no code) + +- [ ] Post the drafted reply on #3397 (draft at bottom of this doc; Jeremy reviews wording first). +- [ ] Confirm the reporter's topology (pooled vs direct) — the reply asks. +- [ ] Re-point at the outstanding #3376 questions (baseline dependency). + +## Phase 1 — Measure + surface (Wolverine 6.19.0 + CritterWatch) + +Scope: **observability only.** No behavior changes. Opt-in by construction: only wired up when the +message store's tenancy reports the sharded shape. + +### Workstream A — server identity (Wolverine core) + +1. Server key type (e.g. `DatabaseServerId`) derived from `DatabaseDescriptor` + (`Engine` + `ServerName` [+ port — close the gap noted above]). Lives next to `DatabaseId`. +2. Group registered stores/databases by server key inside the sweeper pass — no new registry; + the sweeper already walks every registered store sequentially. + +### Workstream B — the probe (Wolverine.Postgresql / Wolverine.SqlServer) + +3. New capability interface (e.g. `IConnectionBudgetProbe`, sibling of `IMessageStoreAdmin`): + `ValueTask CountServerConnectionsAsync(CancellationToken)` — one cheap query per server + per sweep interval, deduped by server key. + - Postgres: `select sum(numbackends) from pg_stat_database` (visible without special grants). + - SQL Server: `select count(*) from sys.dm_exec_connections` — requires `VIEW SERVER STATE`; + degrade loudly-but-safely (log once, report budget as unknown) when permission is missing. +4. `MaxConnections` is **explicit config, per server key** (decision 2 — pooler in front makes the + probed server value misleading). Optional fallback when unconfigured: probe + `pg_settings.max_connections` / `@@MAX_CONNECTIONS` once at startup, tagged as "probed" in + diagnostics so operators can tell the two apart. +5. Config surface (shape TBD in review): something like + `DurabilitySettings.ConnectionBudgets.ForServer(host, maxConnections)` or a callback keyed by + `DatabaseServerId`. Must be settable per server because one deployment can span servers with + different budgets. + +### Workstream C — surfacing (Wolverine core + CritterWatch) + +6. New DTO (e.g. `ConnectionBudgetSnapshot { serverKey, used, max, source(probed|configured) }`) + published via a new `IWolverineObserver` method alongside `PersistedCounts` + (`IWolverineObserver.cs:42`) from the sweeper pass. Keep it a sibling of `PersistedCounts`, not + a member — the budget is server-scoped, counts are database-scoped. +7. OTel `ObservableGauge`s in `PersistenceMetrics` tagged by server key + (`wolverine_database_server_connections_used` / `_max` or similar — naming interview with + Jeremy before shipping). +8. CritterWatch: consume on the ServiceUpdates feed; used/max per server on the service view; + alert threshold. Rides whatever CW beta follows Wolverine 6.19.0. + +### Workstream P — Polecat / SQL Server parity + +9. Polecat has **no sharded-database tenancy today**. Parity decision needed at design review: + - (a) implement the probe + budget surfacing against Polecat's existing multi-tenant shapes + (master-table tenancy), keyed by the same descriptor-derived server id; or + - (b) hold SQL Server activation until Polecat grows the sharded feature, shipping only the + probe plumbing (`IConnectionBudgetProbe` on `SqlServerMessageStore`) now. + Leaning (b) for 6.19.0 — plumbing + gauge parity, activation condition follows Polecat's + tenancy roadmap. **Confirm with Jeremy.** + +### Definition of done (Phase 1) + +- [ ] Sharded-Marten host publishes per-server used/max on the metrics feed and OTel meters. +- [ ] Probe runs once per server per sweep pass regardless of database count (assert in a test + with N databases on one server → 1 probe query). +- [ ] Explicit `MaxConnections` config honored; probed fallback clearly labeled. +- [ ] SQL Server probe parity (per Workstream P resolution). +- [ ] Docs page (pooler caveat prominent: behind a transaction-pooling pgBouncer, server + `numbackends` ≠ client connections — explain what the number means in each topology). +- [ ] CritterWatch surfacing + alert. + +## Phase 2 — Adapt (opt-in; JasperFx 2.28.0+ + Wolverine 6.19.x; AFTER #3376 resolves) + +The reactive half. Everything below is **default-off**. + +1. `ConnectionBudget` policy on `DurabilitySettings`: `Enabled`, `HighWaterUtilization` (~0.70), + **`LowWaterUtilization` for hysteresis** (~0.55 — a single threshold flaps), `BackoffFactor`, + smoothing (EMA over the last few probes). +2. Node-local `ConnectionBudgetMonitor` per server key. Each node adapts independently off the + shared server signal — no cross-node consensus. (The durability sweeper/agents give every node + that hosts agents a local probe already, from Phase 1.) +3. **Probe-failure semantics: repeated probe failure = maximal pressure** (can't get a connection + IS the signal — this is exactly the reporter's db-apply failure). Engage backoff, don't treat + as missing data. +4. Wolverine actuation: migrate `DurabilityAgent`'s recovery/scheduled timers to a re-reading + loop; scale the effective `ScheduledJobPollingTime` by the backoff factor under pressure. +5. JasperFx actuation (rides the same 2.28.0 hooks planned for #3376): + - pacing multiplier consulted by `HighWaterAgent` waits (works today via settings mutation — + formalize as a hook rather than mutating shared settings); + - adaptive gate for the WS2/WS3 governors (`ThrottledEventLoader` + batch-write semaphore) — + these must be restructured to resize; real work, not a tweak. +6. Log state transitions (engaged/released, with utilization) and include budget state in + `IDescribeMyself` output. + +## Phase 3 — CritterWatch control loop (deferred, note-only) + +Explicit pacing commands over CW's existing control queue ("this server is under reporting load, +back off") — human/automated override closing the measure → show → act loop. Post-1.0 CW +territory; acknowledged in the issue reply, not scoped. + +--- + +## Risks / caveats + +- **Pooler topologies**: probed `max_connections` is meaningless behind pgBouncer → explicit + config is primary (decision 2). Docs must spell out what `numbackends` measures in + session/transaction pooling modes. +- **`numbackends` counts ALL backends**, including other applications — that's the point + (server-scoped resource) but must be documented so nobody expects Wolverine-only attribution. +- **`ServerName` lacks port** — fix in the descriptor or key derivation, else co-hosted clusters + collide. +- **SQL Server permission**: `VIEW SERVER STATE` may be absent in locked-down hosting; degrade to + "budget unknown" without failing the sweep. +- **Semaphore governors can't shrink live** — Phase 2 JasperFx work is structural. +- **Flapping** — hysteresis + EMA are mandatory, not nice-to-have. + +## Open questions + +1. Workstream P (a) vs (b) — how much Polecat activation in 6.19.0? (Leaning (b).) +2. Config surface shape for per-server `MaxConnections` (fluent per-host vs callback). +3. Metric/DTO naming — interview Jeremy before shipping (standing convention). +4. Does the budget snapshot also feed the `db-apply`-style out-of-band tools (reporter's failure + was a schema apply job, not the daemon)? Possibly a small static helper those jobs can call — + out of scope for 6.19.0 unless Jeremy wants it. + +--- + +## Draft reply for #3397 (Jeremy reviews before posting) + +> Thanks — this direction is a keeper, and the db-apply datapoint is exactly the failure mode +> worth designing against. A few notes on how we're planning to slice it, plus two questions. +> +> **We're splitting it into "measure + surface" first, "adapt" second.** +> +> The measurement half — a per-server used/max connection budget on the node metrics feed and +> through CritterWatch — doesn't depend on anything else and is slated for **Wolverine 6.19.0**. +> It also happens to be the instrument that answers the connection-breakdown question still open +> on #3376, so shipping it first helps both issues. It'll be keyed off the database server +> identity we already carry in the `DatabaseDescriptor` metadata, deduped so it's one cheap query +> per server per metrics pass no matter how many tenant databases live there, and scoped to the +> sharded-database tenancy model — that's the shape where per-server budgeting earns its keep. +> We'll do PostgreSQL and SQL Server parity from the start. +> +> One design decision to flag: **the max side of the budget will be explicit configuration, not +> the probed `pg_settings` value.** Any pooler in front (pgBouncer et al.) makes the server's +> `max_connections` misleading as a client budget, so you'll declare the budget per server and +> the probe supplies the `used` side (`sum(numbackends)`). A probed max may serve as a labeled +> fallback when nothing is configured. **Question 1: is your claims server talking to Postgres +> directly, or through pgBouncer/a pooler?** That affects which numbers we document as meaningful +> for your topology. +> +> The adaptive half needs to wait on two things. First, the knobs you named — +> `SlowPollingTime`/`FastPollingTime` and the #486 concurrency governors — actually live in +> JasperFx.Events' `DaemonSettings`, not Wolverine, so the back-off hooks have to ship in a +> JasperFx release (the same one already earmarked for the #3376 lifecycle work). Second, as you +> anticipated, #3376's resolution changes the baseline any tuning would work against — so +> **question 2 is the same two questions still open over there** (the `AddAsyncDaemon` +> registration, and the `pg_stat_activity` breakdown). Those answers gate the whole adapt phase. +> +> Two refinements we'll bake into the adaptive design when it comes: hysteresis (high/low water +> marks plus smoothing, so a single threshold doesn't flap), and treating repeated probe *failure* +> as maximal pressure rather than missing data — a probe that can't get a connection is the +> strongest possible signal, and it's precisely your db-apply scenario. +> +> The CritterWatch control-queue pacing commands are a good closing-the-loop idea; we're noting it +> as a later phase rather than scoping it now. + +--- + +## Sequencing note + +Current wave (6.18.0 / jfx 2.28.0 / CW beta.4 per `NEXT-WAVE-HANDOFF.md`) is **unaffected** — +this plan starts at 6.19.0. If the #3376 answers arrive mid-wave and turn out to need the jfx +hooks anyway, design the hook surface with Phase 2's pacing needs in mind so one JasperFx release +serves both. diff --git a/EXECUTION-PLAN-3376-3397.md b/EXECUTION-PLAN-3376-3397.md new file mode 100644 index 000000000..46d2c3ea2 --- /dev/null +++ b/EXECUTION-PLAN-3376-3397.md @@ -0,0 +1,393 @@ +# Execution plan — wolverine#3376 (daemon connection scoping) + #3397 Phase 2 (adaptive budget) + +> **STATUS 2026-07-17 — Wave 1 COMPLETE AND MERGED to main (6.20.0-alpha.1 line).** +> - #3439 — scope tenant scheduled polling to the owning node (the #3376 fix) + tests ✅ merged +> - #3440 — release daemon tracker subscriptions on the 1→0 transition ✅ merged +> - #3441 — GH-3166 DLQ test cleans Wolverine storage (unrelated hygiene found en route) ✅ merged +> - Issue comment posted: https://github.com/JasperFx/wolverine/issues/3376#issuecomment-5002738071 +> +> Root cause confirmed empirically by a two-node test, not just by reading. Waves 2 and 3 remain. +> +> Companion doc: `CONNECTION-BUDGET-3397-PLAN.md` (Phase 1, shipped in 6.19.0 via #3422). + +--- + +## ⚠️ Corrections to the 2026-07-16 analysis below (read these first) + +The mechanism was right; two supporting claims were wrong, and one of them was headed for the issue. + +1. **"Single-database deployments already do this right, on the leader only" is FALSE — do not say + this publicly.** `MessageDatabase.Agents.cs` (the `IAgentFamily` with + `AutoStartScheduledJobPolling = true` + `RunOnLeader`) is **dead code**. Agent families come from + `_container.GetAllInstances()` (`WolverineRuntime.Agents.cs:241`) and the store is + registered only as `IMessageStore`; `NodeAgentController.cs:88-91` then hands the `wolverinedb` + scheme to `MessageStoreCollection` unconditionally. Single-DB hosts fan out on every node too — + it just never hurt, because every node already talks to the main database for heartbeats and + leadership. The "irony" paragraph in the draft comment must go. + +2. **`MultiTenantedMessageStore.StartScheduledJobs` (the `CompositeAgent` over `AllActive()` with the + stale TODO) is not in the path.** `MessageStoreCollection.InitializeAsync:106-114` already flattens + every tenant database into `_services`, so `FindAllAsync()` returns them individually. Red herring. + +3. **The better framing: #3376 is #2623, unfixed for relational stores.** RavenDb and CosmosDb already + return a *non-started* agent from `StartScheduledJobs`, with a comment saying exactly why + ("NodeAgentController owns the durability agent lifecycle... do not start a second instance here"). + The RDBMS and Oracle stores were never brought along. This is verifiable, and it's a stronger story + than the false one. + +4. **The fan-out agent is never started** — `DurableScheduledJobs` is only ever `StopAsync`'d + (`WolverineRuntime.Disposal.cs:22`). The eagerly-started poll timer was its entire runtime + contribution, which is why removing it is safe and why the fix is two lines. + +5. **1.3 cannot dispose the daemon** (see Wave 1 §1.3 below, rewritten). + +--- + +## Headline finding: #3376 is not an epic — the `nodes × databases` footprint is one unscoped code path + +The design-note thesis on the issue ("JasperFx.Events needs per-database daemon lifecycle +hooks") is **superseded by recon**. The dominant mechanism behind "497 of 512 databases hold +connections from every node, last statement `COMMIT`/`ROLLBACK`, every backend < 10 minutes +old" is **node-wide durable scheduled-job polling**, which fans out to every tenant database +on every node and was never ownership-scoped. Verified call chain: + +1. `WolverineRuntime.Agents.cs:207-231` — in `Balanced` (and `Solo`) mode, + `startDurableScheduledJobs()` runs **unconditionally on every node**, before and + independent of any agent assignment. +2. `MessageStoreCollection.StartScheduledJobProcessing` (`MessageStoreCollection.cs:324-330`) + — `FindAllAsync()` returns **all** stores, one scheduled-jobs agent each. +3. `MultiTenantedMessageStore.StartScheduledJobs` (`MultiTenantedMessageStore.cs:388-394`) — + `CompositeAgent` over `Source.AllActive()` = **one poller per tenant database**. (Carries + a `// TODO -- need to start ancillary stores too` that shows this path predates the + distribution machinery.) +4. `MessageDatabase.StartScheduledJobs` (`MessageDatabase.cs:308-314`) — `new + DurabilityAgent(...)` + `StartScheduledJobPolling()` directly. +5. `DurabilityAgent.StartScheduledJobPolling` (`DurabilityAgent.cs:262-268`) — timer every + `ScheduledJobPollingTime` (**default 5s**, `DurabilitySettings.cs:209`). +6. `PostgresqlMessageStore.PollForScheduledMessagesAsync` (`PostgresqlMessageStore.cs:469-525`) + — fresh pooled connection, `BEGIN`, try advisory lock (`ScheduledJobLockId`), `SELECT`, + **`ROLLBACK` when quiet** / `COMMIT` when work found, close. + +The per-database advisory lock dedupes the *work* across nodes but **not the connection** — +every losing node still opened a connection, began a transaction, and rolled back. At a 5s +cadence each tenant data source keeps ~1 warm connection per node: 512 DBs × N nodes, churned +by Npgsql idle-lifetime pruning. That reproduces every number erdtsieck posted (~937 quiet at +2 nodes, +~350 idle from a third node, `COMMIT`/`ROLLBACK` fingerprint, < 10-min backends). + +**The contrast that proves it's an oversight, not a design:** + +- Single-database deployments: the distributed durability agent gets + `AutoStartScheduledJobPolling = true` and runs **on the leader only** + (`MessageDatabase.Agents.cs:27-31,42-48`). +- Multi-tenant deployments: the per-database durability agents ARE distributed evenly across + nodes (`MultiTenantedMessageDatabase.Agents.cs:48-52` → `MessageDatabase.BuildAgent:120-123`) + — but are built **without** scheduled polling, which instead arrives via the unscoped + node-wide fan-out above. + +So the fix is Wolverine-only, patch-sized, and **needs no JasperFx release**: move tenant +scheduled polling onto the already-distributed durability agent and stop the node-wide fan-out +from touching tenant databases. + +### What recon ruled OUT as `nodes × databases` sources (evidence in agent reports) + +- High-water polling: already ownership-scoped. `JasperFxAsyncDaemon.StopAgentAsync` stops the + `HighWaterAgent` at the 1→0 agent transition (`JasperFxAsyncDaemon.cs:496-500`), and daemons + are only materialized for **assigned** agents (`EventStoreAgents.FindDaemonAsync`). +- Neither Marten's sharded tenancy, `MessageDatabaseDiscovery`, leader assignment enumeration, + nor node heartbeats connect per-tenant — all read the master/pool DB only. +- The #3384 metrics sweeper is correctly owned-only (registers on agent start, unregisters on + stop) and single-connection-in-flight. +- **No long-lived daemon connection exists to "release."** Marten's `HighWaterDetector` opens + and disposes a pooled connection per probe; Polecat's takes a bare connection string. The + connection footprint is governed entirely by *whether polling loops run* — which is exactly + what the scheduled-polling fix addresses. The original "release the database's pool" idea + stays dead (pool is shared with app sessions, per the issue's earlier correction). + +### Real leaks confirmed (hygiene, not the connection story) + +| Leak | Where | Fix side | +|------|-------|----------| +| `EventStoreAgents._daemons` append-only; daemon never released on 1→0 | `EventStoreAgents.cs:16,84` | Wolverine | +| Observer subscriptions to `daemon.Tracker` never disposed (`// TODO -- do we need to care about un-subscribing?`) | `EventStoreAgents.cs:78-82` | Wolverine | +| `_tenantHighWaterTimer` (tenant-partitioned stores only) started in ctor, stopped only in `Dispose()` | `JasperFxAsyncDaemon.cs:91-95,169-170,790-796` | JasperFx | +| `_deadLetterBlock` + throttle semaphores survive 1→0 | `JasperFxAsyncDaemon.cs:99-107` | JasperFx | + +Stale claim corrected: the daemon's own `_breakSubscription` IS disposed in `Dispose()` +(`JasperFxAsyncDaemon.cs:171`) — earlier recon predates that fix. + +--- + +## Wave 1 — #3376 fix (Wolverine only) — ✅ DONE, PR #3439 / #3440 + +Jeremy's calls 2026-07-17: fold into the next planned wave (not its own 6.20.0); no 5.x backport; +immediate release on 1→0; fix + tests in one PR, hygiene separate. + +### 1.1 Ownership-scope tenant scheduled polling — ✅ as built + +- `MessageDatabase.BuildAgent` and `OracleMessageStore.BuildAgent`: set + `AutoStartScheduledJobPolling = true` on the distributed agent. +- `MessageDatabase.StartScheduledJobs` / `OracleMessageStore.StartScheduledJobs`: only start a poller + when `!DurabilityAgentEnabled`. (NOT `MultiTenantedMessageStore.StartScheduledJobs` — see + correction 2 above. And no "Main store only" carve-out was needed: Main's own distributed agent + now polls it, on whichever single node owns it.) +- **Back-compat gates (must preserve):** + - `DurabilityAgentEnabled == false` hosts (`DurabilitySettings.cs:134`; set by mediator-mode + helpers, `HostBuilderExtensions.cs:486`) have no agent controller — the node-wide fan-out + is their only scheduled-message pump. Keep the full fan-out when agents are disabled. + - `Solo` mode starts every agent locally, so tenant polling still runs — verify no + double-poll window (advisory lock makes overlap harmless; avoid making it permanent). + - Dynamic tenants: a DB added at runtime gets its agent via `AllKnownAgentsAsync` refresh — + scheduled polling now follows automatically (today's fan-out only covered stores present + at startup; the fix makes late tenants *better*, worth a release-note line). + - `Serverless`/`MediatorOnly`: no change (never started scheduled jobs). + +### 1.2 Multi-node connection-scoping test — ✅ as built + +`src/Persistence/PostgresqlTests/MultiTenancy/multi_node_tenant_database_connections.cs` (PostgresqlTests, +not SlowTests — that project has no Postgres reference). 2 Balanced nodes, 3 tenant DBs. **Verified red +on main, green with the fix.** + +Two hard-won test lessons: + +- **`pg_stat_activity` cannot tell two Wolverine nodes apart.** Wolverine never sets + `application_name`. Under static tenancy the connection strings are ours, so stamp a per-host + `ApplicationName` and the node→database map falls out. +- **Connection presence ≠ polling.** `AddResourceSetupOnStartup` migrates every tenant DB from *every* + node at startup and Npgsql parks those in the pool for minutes. A presence-based assertion goes red + on `main` for the *wrong reason*. Assert that `query_start` advances inside a measurement window, + using the server's own `clock_timestamp()`. + +Exactly-once on a tenant DB is covered. Failover-moves-polling was not built (the reassignment path is +already covered by the 39 green Marten distribution tests). + +### 1.3 Leak hygiene — ✅ PR #3440, but NOT as specced + +**The daemon cannot be disposed on 1→0.** It is shared: `AllDaemonsAsync()` hands it to Polecat's +`IProjectionCoordinator` (`WolverineProjectionCoordinator.cs:50`) and `TryRebuildRegisteredProjectionAsync` +holds it across a rebuild → disposing is a use-after-dispose for anyone mid-borrow, and per the Marten +9.14 catch-up note driving projections through `IProjectionCoordinator` is the supported path. +Removing it from `_daemons` without disposing is also wrong — `DisposeAsync` sweeps that map at +shutdown, so it would never be stopped at all. + +As built: dispose the observer subscriptions on 1→0 and **re-subscribe on reuse** (a cached daemon +handed back after reassignment would otherwise leave observers deaf for the process lifetime); leave +the daemon cached. Retained daemons are bounded by databases-this-node-has-owned and are already +quiesced at 1→0. Full release still wants Wave 2's jfx `StopAndReleaseAsync`. + +### 1.4 Issue communication + +- Post the draft comment (bottom of this doc) amending the design note — third public + correction on this issue, same spirit as the previous two. +- Ask erdtsieck to re-measure on the release; predicted outcome: tenant-DB connections from + non-owner nodes drop to ~zero; steady state ≈ `databases + main-store + app traffic`. +- Optional pre-code confirmation he can run today: `select database, count(*) from pg_locks + where locktype = 'advisory'` fingerprints the scheduled-poll lock; or raise + `Durability.ScheduledJobPollingTime` to 5 minutes and watch the parked-connection count + collapse (also his cheapest interim mitigation, alongside any `Connection Idle Lifetime` + tuning). + +### Explicit non-goals for Wave 1 + +- No JasperFx.Events lifecycle API (moved to Wave 2, downgraded to hygiene). +- No change to app-session connection demand — the second axis from the issue's correction + stands; agent scoping never touches it. The Phase-1 budget gauges are the honest measure of + what remains. + +--- + +## Wave 2 — JasperFx hygiene + pacing seams — ⏳ PR jasperfx#521 OPEN, PARKED awaiting release + +**Item 1 (`StopAndReleaseAsync`) was DROPPED as mostly obsolete — the recon behind it was stale.** +`StopAllAsync` already stops high-water, stops/drains the agents, drains `_deadLetterBlock` **and +rebuilds it**, and resets the cancellation source: it is already a restartable quiesce. The only real +leftover was the tenant timer (folded into item 2). And a dispose-everything release would make the +daemon single-use, which collides with callers that cache and reuse it across reassignment +(`IProjectionCoordinator` hands out the same instance; rebuilds hold it across a replay) — the same +constraint that shaped Wave 1 §1.3. + +**Item 2 shipped in jasperfx#521**, along with the `ConcurrencyException(string, Exception)` ctor: + +- Governors resizable via swap-on-set. Note the asymmetry, documented on each property: + `BatchWriteThrottle` is a **live pass-through** so it reaches running agents; `_loadThrottle` is + **captured** into a `ThrottledEventLoader` at agent-build time, so it only applies to agents built + afterwards. Neither setter disposes the semaphore it replaces. +- `_tenantHighWaterTimer` re-reads `SlowPollingTime` per tick (was captured at construction) and now + idles when the daemon has no agents. +- `ConcurrencyException(string, Exception)` — unblocks wolverine#3444 + (`SagaConcurrencyException : ConcurrencyException`), which the sagas docs already promise. + +Also consolidate `DatabaseServerId` onto `DatabaseDescriptor.Port` (jasperfx#514, merged +2026-07-15) when the JasperFx pin bumps — noted in the Phase-1 plan as deferred. + +--- + +## Wave 3 — #3397 Phase 2: adapt (Wolverine 6.20.x+, AFTER erdtsieck re-baselines on Wave 1) + +Wave 1 removes most of the pressure #3397 exists to relieve — re-measure before tuning +(erdtsieck said exactly this on the issue). Everything below is **default-off**; ship order +within the wave is 3.1 → 3.2 → 3.3, and 3.4 can lag. + +### 3.1 `ConnectionBudgetMonitor` (Wolverine core) + +- Lives in `PersistenceMetricsSweeper` — the per-node singleton every snapshot already flows + through (`PersistenceMetricsSweeper.For(runtime)`, `PersistenceMetricsSweeper.cs:23-28`). + Attach at the publish point (`probeConnectionBudgetsAsync`, lines 246-252); extend the + existing per-server `ServerBudgetState` (line 47) with EMA + water-line state. Single-loop + thread ⇒ no locking (existing invariant). +- Config on `DurabilitySettings` beside `ConnectionBudgets` (line 300): `HighWaterUtilization` + (~0.70), `LowWaterUtilization` (~0.55), smoothing window, `BackoffFactor`, `Enabled`. + Hysteresis + EMA are mandatory (single threshold flaps). +- **Repeated probe failure = maximal pressure** — the marked catch block + (`PersistenceMetricsSweeper.cs:229-241`) engages backoff instead of only logging. This is + the reporter's db-apply failure mode. +- Exposes `PacingFactorFor(DatabaseServerId)`; keyed off `ConnectionBudgetSnapshot.Utilization` + (`ConnectionBudgetSnapshot.cs:53`, comment already points here). Log state transitions; + include budget state in `IDescribeMyself`. + +### 3.2 Wolverine actuation — DurabilityAgent re-reading loops + +Convert the recovery timer (`DurabilityAgent.cs:94-114`) and scheduled-job timer (`:262-268`) +from fixed-period `System.Threading.Timer` to the sweeper's re-read-per-pass loop model +(`PersistenceMetricsSweeper.cs:112-166`): each iteration, +`effective = ScheduledJobPollingTime × PacingFactorFor(_database.ServerId)`. The agent already +holds `_runtime` and the store resolves its `ServerId` via `IConnectionBudgetProbe`. +(Expiration and handled-cleanup timers stay as-is — hourly/minutely, not pressure drivers.) +Note: after Wave 1, scheduled polling runs inside these same distributed agents, so pacing +automatically covers it. + +### 3.3 Daemon cadence actuation — no JasperFx release required + +Two live seams already exist: + +- `FastPollingTime`/`SlowPollingTime` are re-read by the high-water loop **every wait** + (`HighWaterAgent.cs:115,140,...`) on the live `ProjectionGraph` instance. +- `DaemonSettings.Wakeup` (`IDaemonWakeup`, `DaemonSettings.cs:109`) wraps every inter-poll + wait — a Wolverine-supplied implementation can scale the effective delay by the pacing + factor without mutating shared settings. + +Prefer the `IDaemonWakeup` route (formal hook, no cross-daemon settings mutation). Gap to +document: the tenant high-water timer doesn't consult either seam until Wave 2 item 2 lands. + +### 3.4 Governor pacing (needs Wave 2 item 2) + +Scale `MaxConcurrentEventLoadsPerDatabase`/`MaxConcurrentBatchWritesPerDatabase` down under +sustained pressure via the resizable gates. Cadence pacing (3.2/3.3) ships first and alone if +the JasperFx release lags — it delivers most of the relief (connections are held by polling, +not by governor width). + +### 3.5 CritterWatch + +Consume budget state transitions (engaged/released) on ServiceUpdates; alert on sustained +high-water. The Phase-3 control-queue pacing commands stay parked (post-1.0, as told to +erdtsieck). + +### Definition of done (Phase 2) + +- [ ] Utilization ≥ high-water for M consecutive probes ⇒ polling cadence stretches by + `BackoffFactor` on that server's databases only; ≤ low-water ⇒ restores. Pinned by test + with a fake probe. +- [ ] Probe failure streak ⇒ same engagement (test). +- [ ] No flapping across the hysteresis band under a noisy fake probe (test). +- [ ] Default-off; zero behavior change when disabled (test: pacing factor pinned at 1.0). +- [ ] Docs: budget page grows an "adaptive back-off" section; pooler caveat repeated. + +--- + +## Sequencing & release map + +| Wave | Ships in | Gate | +|------|----------|------| +| 1 — #3376 scheduled-polling scoping + hygiene + test | Wolverine 6.20.0 | none — ready to build on plan approval | +| 2 — JasperFx quiesce + resizable governors | next JasperFx (2.28.0?) | independent; only gates 3.4 | +| 3 — #3397 Phase 2 adapt | Wolverine 6.20.x/6.21.0 | erdtsieck re-baseline on Wave 1 (his own sequencing ask) | + +## Risks + +- **Wave 1 diagnosis risk**: the chain is code-verified, but production has surprised us + before (BASELINE BEFORE YOU BLAME). Mitigation: the multi-node test in 1.2 reproduces the + both-nodes fingerprint on `main` *before* the fix and shows it gone after; plus erdtsieck's + optional `pg_locks` confirmation. +- **Scheduled-message latency on reassignment**: after a node dies, a tenant DB's scheduled + poll pauses until its durability agent is reassigned (heartbeat-order delay). Today's + redundant pollers masked that. Acceptable — same semantics recovery already has — but + release-note it. +- **`DurabilityAgentEnabled=false` + multi-tenant**: keep-fan-out path must be tested or the + fix silently kills scheduled messages for mediator-style hosts. +- **Phase 2 flapping/EMA tuning**: fake-probe tests, not production, are where the constants + get exercised first. + +## Open questions for Jeremy + +1. Wave 1 target: 6.20.0 as its own release, or fold into the next planned wave? +2. On 1→0 daemon release (1.3): release immediately on revocation, or after a grace period + (heartbeat interval) to avoid rebalance thrash? Design note promised "short grace" — + immediate is simpler and `FindDaemonAsync` rebuilds cheaply; recommend immediate. +3. Phase 2 naming interview (standing convention) before 3.1 ships: `BackoffFactor` vs + `PacingFactor`, config section name, transition log wording. +4. Does the scheduled-polling fix warrant a 5.x backport? Same code shape exists on the 5.0 + branch, and it's arguably a defect, not a feature. + +--- + +## Comment for #3376 — ✅ POSTED 2026-07-17 + +https://github.com/JasperFx/wolverine/issues/3376#issuecomment-5002738071 + +Rewritten after the two-node test, then posted on Jeremy's go-ahead. The old draft's "single-database +already does this right" paragraph was based on dead code and was cut — see correction 1 at the top of +this doc. Everything below is backed by a test that is red on `main` and green with the fix (PR #3439). +As posted, it also links #3439 and #3440. + +> ## Root cause found, fixed, and it's a patch rather than an epic +> +> Thanks for the registration and `pg_stat_activity` answers — they ruled out the coordinator footgun +> and pointed us straight at what else touches every tenant database from every node. I owe this issue +> a third correction: the per-database daemon lifecycle hooks I sketched are **not** the fix. The +> daemon side was already ownership-scoped, and neither the daemon nor the high-water detector holds a +> long-lived connection to release. +> +> **The mechanism is durable *scheduled-message* polling.** Under multi-tenancy Wolverine started a +> scheduled-job poller for **every tenant database on every node**, entirely outside the +> agent-distribution machinery. Every `ScheduledJobPollingTime` (default 5s), per database, per node: +> open a pooled connection, `BEGIN`, try a per-database advisory lock, `SELECT` for due messages, then +> `ROLLBACK` (quiet) or `COMMIT` (work found). The advisory lock dedupes the *work* across nodes but +> not the *connection* — the losing node still opened, lock-failed, and rolled back. That is your +> fingerprint exactly: `COMMIT`/`ROLLBACK` last-statements, ~every database connected from every node, +> every backend younger than your idle-lifetime pruning, and each added node contributing another ~512 +> parked connections while "idle". +> +> Rather than take that from a code reading, we pinned it: a two-node test against sharded tenant +> databases, with each node stamping its own `application_name` so `pg_stat_activity` can attribute +> every connection. On `main`, agent distribution is perfectly clean — each tenant database is owned by +> exactly one node — and **every tenant database is still queried by both nodes anyway**. The +> connections were never coming from the agents. +> +> **What it turned out to be:** this is [#2623](https://github.com/JasperFx/wolverine/issues/2623), +> never applied to the relational stores. The RavenDb and CosmosDb stores already return a *non-started* +> agent from this path, with a comment explaining that the node agent controller owns the durability +> agent's lifecycle and starting a second poller here would double up. The Postgres/SQL Server/Oracle +> family kept starting its own. The fan-out's agent is never actually started by the runtime, so that +> eager poll timer was its entire contribution — which is why the fix is small. +> +> **The fix** (Wolverine-only, no JasperFx release needed): scheduled polling rides the per-database +> durability agent that managed distribution already assigns to exactly one node. The node-wide fan-out +> stops starting pollers, except for hosts running without durability agents, where it's still the only +> pump. Expected steady state on your topology: tenant-DB connections from non-owner nodes ≈ 0, total ≈ +> `databases + main store + your application traffic` — and adding a node finally *divides* the polling +> load instead of multiplying it. Dynamic tenants get polling automatically now, where the old fan-out +> only covered databases present at startup. +> +> The application-traffic axis from my earlier correction still stands and is untouched by this; the +> per-server budget gauges from 6.19.0 should give us both an honest before/after. +> +> One tradeoff worth naming: when a node dies, a tenant database's scheduled poll now pauses until its +> durability agent is reassigned, where the redundant pollers used to mask that. It's the same +> semantics message recovery already has, but it's a real change. +> +> If you want to confirm before the release: `select count(*) from pg_locks where locktype = 'advisory'` +> across your tenant DBs fingerprints the pollers, and raising `opts.Durability.ScheduledJobPollingTime` +> is a crude interim relief valve (at the cost of scheduled-message latency). +> +> We're also fixing a smaller leak the recon confirmed — tracker subscriptions that were never disposed +> when a node stopped owning a database — and the #3397 adaptive budget work resumes once this lands and +> you've re-baselined, per your own sequencing. diff --git a/GLOBAL-PARTITIONING-ROLLOUT-PLAN.md b/GLOBAL-PARTITIONING-ROLLOUT-PLAN.md new file mode 100644 index 000000000..ff5d0fa7e --- /dev/null +++ b/GLOBAL-PARTITIONING-ROLLOUT-PLAN.md @@ -0,0 +1,144 @@ +# GlobalPartitioning Rollout Plan (2026-07-18) + +Goal: bring GlobalPartitioning topology support to every Wolverine messaging transport where it +makes sense, close the docs/test gaps on the transports that already have it, and sketch a +"native mode" v2 that leans on broker-native partitioning primitives. + +## Current state (verified in code) + +The core machinery lives in `src/Wolverine/Runtime/Partitioning/`: +- `GlobalPartitionedMessageTopology` pairs an external `PartitionedMessageTopology` (N broker + endpoints named `{base}1..N`, exclusive listeners, forced `EndpointMode.Durable`) with a + companion `LocalPartitionedMessageTopology` (`global-{base}` durable local queues). +- Slot routing is central: `Envelope.SlotForSending` / `SlotForProcessing` hash the resolved + `GroupId` mod N (valid slot counts 3/5/7/9). Transports do **not** implement routing. +- `GlobalPartitionedRoute` shortcuts to the companion local queue when this node owns the + exclusive listener for the slot; otherwise sends through the broker to the owning node. +- `GlobalPartitionedReceiverBridge` + `GlobalPartitionedInterceptor` handle inbound bridging and + re-slotting of messages that arrive on non-sharded endpoints. + +**A transport participates with exactly two pieces** (see Kafka/RabbitMQ as reference impls): +1. `PartitionedMessageTopologyWith{X} : PartitionedMessageTopology` + implementing `buildEndpoint`, `buildListener`, `buildSubscriber` + (e.g. `src/Transports/Kafka/Wolverine.Kafka/Internal/PartitionedMessageTopologyWithTopics.cs`). +2. A pair of extension methods: `UseSharded{X}(this GlobalPartitionedMessageTopology, ...)` + calling `SetExternalTopology(...)`, and `PublishToSharded{X}(this MessagePartitioningRules, ...)` + (e.g. `KafkaTransportExtensions.cs:300-337`, `RabbitMqTransportExtensions.cs:479-515`). + +### Support matrix + +| Transport | `UseSharded*` | Dedicated global tests | In docs table (`partitioning.md`) | +|---|---|---|---| +| RabbitMQ | ✅ `UseShardedRabbitQueues` | ✅ | ✅ | +| Kafka | ✅ `UseShardedKafkaTopics` | ✅ (3 suites) | ✅ | +| Amazon SQS | ✅ `UseShardedAmazonSqsQueues` | ✅ | ✅ | +| Pulsar | ✅ `UseShardedPulsarTopics` | ❌ | ✅ | +| Azure Service Bus | ✅ `UseShardedAzureServiceBusQueues` | ❌ | ❌ | +| GCP Pub/Sub | ✅ `UseShardedPubsubTopics` | ❌ | ❌ | +| NATS | ✅ `UseShardedNatsSubjects` | ❌ | ❌ | +| Redis Streams | ✅ `UseShardedRedisStreams` | ❌ | ❌ | +| PostgreSQL DB queues | ❌ | — | — | +| SQL Server DB queues | ❌ | — | — | +| Amazon SNS | ❌ (publish-only transport) | — | — | +| MQTT | ❌ | — | — | +| SignalR | ❌ (non-goal, see below) | — | — | +| TCP / RavenDb control | ❌ (non-goals) | — | — | + +## Wave 1 — Docs + test catch-up on the existing eight (cheap, ship first) + +1. **Docs**: `docs/guide/messaging/partitioning.md` transports table (~line 501) only lists + RabbitMQ, Kafka, SQS, Pulsar. Add Azure Service Bus, GCP Pub/Sub, NATS, and Redis with + per-transport snippets (`UseShardedAzureServiceBusQueues`, `UseShardedPubsubTopics`, + `UseShardedNatsSubjects`, `UseShardedRedisStreams`). Release-notes-worthy: half the support + surface is currently invisible to users. +2. **Tests**: add a `global_partitioned_sharded_processing` suite (modeled on the Kafka/RabbitMQ + ones) to ASB, GCP Pub/Sub, NATS, Redis, and Pulsar test projects. These are the transports + where the extension exists but nothing exercises the full multi-node + route→bridge→local-queue path against the real broker. Reuse the shared scenario shape from + `src/Transports/Kafka/Wolverine.Kafka.Tests/global_partitioned_sharded_processing.cs`; if the + duplication is annoying, consider lifting a shared scenario harness into + `Wolverine.ComplianceTests`. + +## Wave 2 — PostgreSQL + SQL Server database queues (the real feature work) + +Best candidates: both already model multiple named durable queues +(`ListenToPostgresqlQueue(name)` / `ListenToSqlServerQueue(name)`), are inherently durable +(global partitioning forces `EndpointMode.Durable` anyway), have sticky per-node listener agents, +and give a zero-extra-infrastructure partitioning story for the "Postgres is my message broker" +crowd — a very on-brand critter-stack pitch. + +Per database (Postgres first, SQL Server is a near-copy): + +1. `PartitionedMessageTopologyWithDatabaseQueues : + PartitionedMessageTopology` + in `src/Persistence/Wolverine.Postgresql/Transport/`. + - `buildEndpoint` → `transport.QueueByName("{base}{n}")` equivalent (creates + `wolverine_queue_{base}{n}` + scheduled table). + - `buildListener` / `buildSubscriber` → wrap the existing listener/subscriber configurations. +2. Extensions `UseShardedPostgresqlQueues(...)` + `PublishToShardedPostgresqlQueues(...)` in + `PostgresqlConfigurationExtensions` (and SqlServer twins). +3. **Verify exclusive listener semantics**: `ListenerScope.Exclusive` shard endpoints must be + assigned/relocated by the agent coordinator the same way broker shards are. The sticky + listener agents (`StickyPostgresqlQueueListenerAgent`) already exist — confirm they compose + with `UsedInShardedTopology` endpoints and the `GlobalPartitionedRoute` "do I own the + listener" check (`FindListeningAgent(...).Status == Accepting`). +4. **Table-name length**: shard suffixes push against the Postgres NAMEDATALEN identifier + shortening (`PostgresqlQueue` line ~40) — add a test with a long base name. +5. **Multi-tenancy interaction**: DB-per-tenant queues (`MultiTenantedQueueListener`) × sharded + topology — decide and document (simplest: shard tables exist per tenant database, slot + routing unchanged; exclusivity is per-database). +6. Tests in `PostgresqlTests` / `SqlServerTests`: the standard global sharded-processing scenario + plus a two-node concurrency test (mirror `Bug_concurrency_with_global_partitioning.cs`). + +## Wave 3 — Decisions for the remainder + +- **Amazon SNS**: recommend **no standalone implementation**. SNS is publish-only and always + pairs with SQS; the sharded consumption story is already `UseShardedAmazonSqsQueues`. If a + fan-out-then-shard story is wanted, the right shape is a convenience that publishes to one SNS + topic and subscribes N sharded SQS queues — i.e. an SQS-side feature, not an SNS topology. + Document this explicitly (non-goal with a pointer). +- **MQTT**: feasible as N sharded topics (`{base}1..N` topic names, exclusive Wolverine-assigned + listeners), but two caveats to resolve first: + 1. Global partitioning forces `EndpointMode.Durable` on external slots; MQTT topics default to + `BufferedInMemory` — verify Durable mode (DB-backed inbox) actually works on MQTT endpoints + or the forced mode will need a transport-specific carve-out. + 2. Broker offers no exclusive-consumer primitive; exclusivity rests entirely on Wolverine's + agent assignment (same as Redis, so acceptable — but QoS ≥ AtLeastOnce should be forced on + shard topics). + Priority: low. Do it only if user demand shows up; MQTT users are mostly IoT-ingest, not + ordered-work-queue. +- **SignalR**: explicit **non-goal** — broadcast hub to browser clients, no competing-consumer or + work-queue semantics to partition. Say so in the docs. +- **TCP / RavenDb**: non-goals (point-to-point / control-plane only). + +## Wave 4 (v2, feeds off the capability research) — "native mode" partitioning + +Today every transport implements global partitioning as N distinct endpoints with +Wolverine-computed hash slots — even Kafka uses N *topics*, not Kafka partitions. Several brokers +have native primitives that could back a `UseNative...` variant with fewer moving parts and +broker-managed rebalancing/failover: + +| Transport | Native primitive | Notes | +|---|---|---| +| Kafka | topic partitions + consumer group (GroupId → partition key) | broker-managed assignment replaces exclusive-agent choreography; KIP-848 makes rebalances cheap | +| RabbitMQ | **super streams + single-active-consumer** | needs `RabbitMQ.Stream.Client`; no .NET framework exposes this today | +| Azure Service Bus | **sessions** (`SessionId` = GroupId) | already hinted in docs (`partitioning.md:372`); session state is a bonus | +| Amazon SQS | FIFO `MessageGroupId` | per-group ordering + exclusive in-flight group, high-throughput mode | +| GCP Pub/Sub | ordering keys (GroupId → `OrderingKey`) | already partially wired (`PubsubEndpoint` maps GroupId → OrderingKey) | +| NATS | deterministic subject mapping `{{partition(N,...)}}` + pinned pull consumers (2.11+) | the Orbit `pcgroups` model | +| Pulsar | `KeyShared` subscription (GroupId → key) | already supported as a subscription type; DotPulsar supports KeyShared | + +This is a bigger design conversation (per-key vs per-slot ordering, poison-message head-of-line +blocking, how `GlobalPartitionedRoute`'s local-shortcut interacts with broker-managed +assignment) — park as a design doc after Waves 1–2 ship. + +## Sequencing & sizing + +| Wave | Size | Risk | +|---|---|---| +| 1 docs | small | none | +| 1 tests | medium (5 broker test suites, CI time) | flaky-broker risk; gate on per-transport CI jobs | +| 2 Postgres | medium | agent-assignment edge cases in multi-node tests | +| 2 SQL Server | small (clone of Postgres) | low | +| 3 decisions/docs | small | none | +| 4 native mode | large | design-first, separate effort | diff --git a/KAFKA-PERF-DEEP-DIVE-PLAN.md b/KAFKA-PERF-DEEP-DIVE-PLAN.md new file mode 100644 index 000000000..abbc044e2 --- /dev/null +++ b/KAFKA-PERF-DEEP-DIVE-PLAN.md @@ -0,0 +1,450 @@ +# Kafka Performance Deep-Dive Plan (2026-07-18) + +Goal: explain and then shrink the gap a client measured between "native Confluent.Kafka" and +Wolverine-over-Kafka (p50 transit 7.9ms vs 25.8ms on 1Kb; 8.4ms vs 102.4ms on 100Kb; execution +20.1ms vs native handler 8.9ms), build a reusable load harness + dotTrace protocol, produce +throughput-tuning guidance, and evaluate a "single-message-type fast-path listener" +optimization. Companion docs: `TRANSPORT-CAPABILITY-RESEARCH-2026-07-18.md`, +`GLOBAL-PARTITIONING-ROLLOUT-PLAN.md`. + +All file:line cites verified against `main` @ 6.20.0. + +--- + +## 0. What Wolverine's numbers actually measure (read this before comparing anything) + +The client is comparing their own native-harness timestamps against Wolverine's metrics. These +are **not the same intervals**, and two of the deltas may be partly definitional: + +- **`wolverine-execution-time`** (`MetricsConstants.cs:6`): monotonic Stopwatch, per handler + *attempt*. Starts at the top of `Executor.ExecuteAsync` (`Executor.cs:219-223`) — i.e. AFTER + queue dwell and AFTER deserialization — stops right after `Handler.HandleAsync` returns + (`Executor.cs:244`). Includes: middleware, the codegen chain, per-message CTS setup, and — + critically — **any time spent blocked inside custom middleware** (see T2 below). Excludes: + deserialization, cascading-message flush (listener path — `MessageSucceededContinuation.cs:22-26` + flushes *after* `ExecutionFinished`), transport ack, inbox mark-handled. + **Gotcha:** sub-1ms samples are dropped entirely (`WolverineRuntime.Tracking.cs:130` — + `if (time > 0)` after `(long)` ms truncation), so the histogram is biased upward for fast + handlers. +- **`wolverine-effective-time`** (`MetricsConstants.cs:10`): wall clock, `UtcNow − envelope.SentAt` + recorded at `MessageSucceeded` (`Tracking.cs:146-165`) — *after* cascading flush AND after + `CompleteAsync()` (transport ack / durable mark-handled UPDATE). So effective time = + producer-side stamp → broker transit → consumer dwell → (durable: inbox INSERT) → queue dwell → + deserialize → handler → cascading flush → ack. It is **cross-machine wall clock**: skew between + service A/B and C shifts every sample. If a `sent-at` header is ever missing/unparseable the + sample is ~year-1-based garbage (`EnvelopeMapper.cs:531-542` returns `default`), and raw-JSON + Kafka endpoints take SentAt from the **broker record timestamp** instead + (`IKafkaEnvelopeMapper.cs:85-88`). +- Wolverine already has the decomposition the comparison needs, opt-in: + `wolverine.envelope.transport_lag_ms` and `receive_dwell_ms` Activity tags + (`WolverineTracing.cs:238-249,324-344`, via `Tracking.HandlerExecutionDiagnosticsEnabled`). + +**Baseline rule for every experiment below:** measure with our own histogram capture at fixed +stages (see H1) rather than trusting either side's "effective" number; record Wolverine's +metrics *alongside* so we can also explain the client's dashboards to them. + +--- + +## 1. Theories of overhead, ranked (pre-registered hypotheses) + +Each theory gets an experiment tag (E#) in the matrix in §4. Prediction written down before +running = honest science. + +### T1 — Sender-side batch debounce dominates "transit" (HIGH confidence, explains p50 transit gap) +`BatchedSender` is the default for Buffered/Durable Kafka subscribers (`KafkaTopic.cs:302-330`). +Its `BatchingChannel` timer is a **debounce, not a fixed-interval flush**: every posted message +resets the timer to the full `MessageBatchTimeout` +(JasperFx `BatchingChannel.cs:97-122`). Consequences: +- A lone message waits the *full* timeout before hitting the wire. +- A steady trickle arriving faster than the timeout keeps postponing the flush until + `MessageBatchSize` accumulates. + +Defaults are **100 / 250ms** (`Endpoint.cs:275,288`) — the client's "batch size = 10, batch +timeout = 10ms" is **not a Wolverine default anywhere**; they configured it (or it's another +framework's number — verify with them, §7). With 10/10ms and a steady match-feed stream, p50 +transit +10-18ms vs native fire-and-forget `ProduceAsync` is exactly what this mechanism +predicts for the 1Kb case (25.8 vs 7.9). For the 100Kb flow (~0.6 msg/s), every message is a +"lone message" → full timeout + a second delivery-side hop, consistent with the much larger +102 vs 8.4 gap if their timeout there was larger (or still 250ms default on that endpoint — +worth asking). +Also in this family: `MessageBatchMaxDegreeOfParallelism` default **1** (`Endpoint.cs:281`) — +one batch in flight at a time per endpoint. +**Prediction:** `MessageBatchSize(1)`/tiny timeout or Inline-mode sending (minus T5's flush bug) +collapses most of the transit gap; `linger.ms`-style tuning on the native side is what Wolverine's +batching is duplicating badly. + +### T2 — Their sequential-by-key semaphore middleware inflates "execution time" (HIGH confidence) +Execution time starts before middleware runs. A "sequential by partition key" semaphore +middleware **waits inside the measured window** — same-stream convoying is billed as handler +execution. Their native consumer achieves per-stream sequencing structurally (partition +assignment), so its "handler" number contains no wait. 20.1ms vs 8.9ms p50, and especially +235.7 vs 51.4 p99, smells like convoy wait + worker-slot occupation: a blocked semaphore +holds one of the `MaxDegreeOfParallelism` Block workers (default `max(ProcessorCount,5)`, +`Endpoint.cs:212`; `BufferedReceiver.cs:62-64`), so unrelated streams queue behind it → +head-of-line blocking that ALSO inflates dwell/effective tails. +**Prediction:** replacing the semaphore middleware with Wolverine's own +`PartitionProcessingByGroupId(...)` (`ListenerConfiguration.cs:110-113`, `ShardedExecutionBlock` +with per-slot serial execution, `ShardedExecutionBlock.cs:7`) moves the wait *out* of the +execution metric and removes worker-slot occupation; execution-time converges toward native +handler time. + +### T3 — Buffered/queue dwell + back-pressure oscillation drives p95/p99 (MEDIUM-HIGH) +Service C runs no durable inbox → Buffered mode. Two mechanisms: +1. Queue dwell in the 10k-capacity worker channel is invisible but fully inside effective time. +2. `BackPressureAgent` (2s poll, `BackPressureAgent.cs:29-70`) stops the listener above + `BufferingLimits.Maximum` = **1000** queued and restarts under **500** (`Endpoint.cs:259`). + For Kafka, "stop" means `_consumer.Close()` — **leaving the consumer group → rebalance** — + and restart re-joins → another rebalance (`ListeningAgent.cs:451-492`, `KafkaListener.cs:167-185`). + Under sustained load this can oscillate every few seconds; each cycle = multi-second partition + stall showing up as periodic effective-time spikes while execution time stays flat. +**Prediction:** dwell-stage histograms show sawtooth; raising `BufferingLimits` (or lowering +`MaxDegreeOfParallelism` contention per T2) flattens p99. Log listener stop/restart events during +the run to correlate. + +### T4 — Durable mode: per-message inbox INSERT gates the consume loop (HIGH, for the durable leg) +Kafka delivers one envelope at a time, so `DurableReceiver` always takes the single-envelope +path: `StoreIncomingAsync(envelope)` = one connection + one INSERT **inline on the consume loop** +before the next `Consume()` (`DurableReceiver.cs:415-520` → `MessageDatabase.Incoming.cs:142-172`). +The batched multi-VALUES insert path exists (`ProcessReceivedMessagesAsync`, +`DurableReceiver.cs:608-668`; `MessageDatabase.Incoming.cs:174-197`) but **Kafka never uses it**. +Then success pays a second DB round trip (mark-handled UPDATE runs inline on the handler thread +*before* effective time is recorded — `DurableReceiver.cs:196-215`, RetryBlock first-attempt-inline). +Durable throughput ceiling ≈ 1/insert-latency per listener. This is both the main measurement +and the main optimization target for durable Kafka (O1 below). + +### T5 — `InlineKafkaSender` blocking `Flush()` per message (HIGH confidence, Inline-mode only) +`InlineKafkaSender.SendAsync`: `await ProduceAsync(...)` then **synchronous `_producer.Flush()` +after every single message** (`InlineKafkaSender.cs:48-55`). Defeats librdkafka's linger/batching +entirely, blocks a thread, and applies to ALL broker-per-tenant sends (`KafkaTopic.cs:315-324`) +and the native DLQ sender. If any of the client's publish endpoints are Inline, this is +first-order. Independent of measurement: this looks like a straight bug to fix (O2). + +### T6 — Envelope-mapper + resolution-chain per-message costs (Jeremy's theory; MEDIUM for +latency, HIGH for CPU/allocation at scale — quantify via microbench) +Per received message, warm path: +- **Mapper**: full header enumeration UTF8-decoded into a fresh `Dictionary` + (`KafkaEnvelopeMapper.cs:32-40` + `Envelope.cs:55-64`), then **19 typed readers each do a + reverse linear scan of the Kafka header list and re-decode the same bytes** + (`EnvelopeMapper.cs:88-111,273-328`) — double decode of every reserved header — plus 2× + `Guid.TryParse`, 3× `DateTimeOffset.TryParseExact`, 2× bool, 1× int, 1× `new Uri`, 1× + `string.Split` (`EnvelopeMapper.cs:413-418,531-556`). Outgoing twin allocates + `_envelopeToHeader.Values.ToArray()` **per message** (`EnvelopeMapper.cs:391-397`) and writes + ~18 UTF8 headers. +- **Resolution chain**: 2× string-keyed `ImHashMap` message-type lookups (one hiding inside the + `RequiresEncryption` check that runs even with no encryption configured — + `HandlerPipeline.cs:190,286-300` + `WolverineOptions.Encryption.cs:96-100`), 1× content-type + serializer lookup (`EnvelopeMapper.cs:119` → `Endpoint.cs:593-613`), 1× Type-keyed executor + lookup (`HandlerPipeline.cs:334`), 1× reverse `ToMessageTypeName` lookup in the + `Envelope.Message` setter (`Envelope.cs:225-233`), plus a wasted `NewId.NextSequentialGuid` + per receive that the wire `id` header immediately clobbers (`KafkaTransportExtensions.cs:252`). +- **Executor fixed cost**: 2 CTS allocations + linked-token timer registration per message + (`Executor.cs:227-228`), and the **"Successfully processed" log at Information level by + default** (`HandlerChain.cs:177`). + +Individually these are sub-microsecond to tens-of-microseconds; at 60k+ msg/run they are CPU, +allocation pressure, and GC — likely a second-order latency contributor but the *primary* input +to the fast-path listener design (O3). Note: the CLAUDE.md claim that `Endpoint._serializers` +does a hot-path `AddOrUpdate` is **stale** — already fixed by pre-seeding in `Endpoint.Compile` +(`Endpoint.cs:524-537`); update CLAUDE.md. + +### T7 — Kafka-side config mismatches (MEDIUM; cheap to test) +- **Offset commit**: default `CommitMode.StoreThenAutoFlush` (`KafkaTopic.cs:64`) is fine; + verify the client isn't on `PerMessage` (sync broker RTT per message) or an old version. +- **100Kb flow**: check `fetch.max.bytes` / `max.partition.fetch.bytes` / `queued.max.messages.kbytes` + passthrough on their ConsumerConfig, and producer `linger.ms`/`batch.size` — "Kafka + configuration identical for both flows" only covers what Wolverine passes through, and + the effective-config inheritance only copies BootstrapServers/GroupId/SASL into per-topic + overrides (`KafkaTopic.cs:145-213`) — a per-topic `ConfigureConsumer` silently DROPS other + transport-level settings. +- **`StampConsumerGroupIdOnEnvelope` default true** (`KafkaTopic.cs:124`): inbound + `envelope.GroupId` = consumer group name unless disabled — silently breaks GroupId-based + partitioning/middleware keyed on GroupId (footgun for their semaphore keying and any GP work). +- Buffered mode acks-on-receive (`BufferedReceiver.cs:233-256`) — offsets commit before + processing; at-most-once on crash. Not a latency issue but must be stated in guidance. + +### T8 — Committer lock + consume-loop architecture (LOW-MEDIUM; measure, don't assume) +One blocking `Consume()` per iteration, one awaited `ReceivedAsync` per record +(`KafkaListener.cs:72-129`); Track+Complete each take a global lock per message +(`KafkaOffsetCommitter.cs:131-188`). Fine at thousands/s; matters at tens of thousands/s. +dotTrace will tell us. + +--- + +## 2. Harnesses to build + +### H1 — `KafkaPerfRig` (primary macro harness) +New console solution folder `src/Testing/KafkaPerfRig/` (kept out of CI like `Benchmarks`), three +processes mirroring the client topology, plus a native twin: + +- **`Rig.ServiceA` / `Rig.ServiceB`**: replay a generated corpus (soccer-match-shaped: N + concurrent "games", per-game monotonic event streams; 1Kb and 100Kb payload types) at a + configurable rate/burst profile. Publish at end of a trivial Wolverine handler (like the + client), stamping `t0` in a header. +- **`Rig.ServiceC`**: Wolverine consumer, handler simulates Marten work with a configurable + `Task.Delay`/CPU-spin mix (calibrated ~9ms p50 to match their native handler number), then + cascades a follow-on message (self-publish leg). +- **`Rig.NativeC` / `Rig.NativeAB`**: raw Confluent.Kafka twin of the same shape (their + experiment reproduced) so both sides are measured by the SAME instrumentation. +- **Instrumentation — stage clock, not framework metrics**: one shared library stamping + monotonic-ish stage timestamps as headers/in-proc records: + `t0 publish-call → t1 broker-produce-ack → t2 consume-return → t3 handler-entry → + t4 handler-exit → t5 ack/offset-stored`. HdrHistogram-style capture + (`HistogramEx`/`HdrHistogram` NuGet), CSV dump per run, plus Wolverine's own + `wolverine-execution-time`/`wolverine-effective-time` scraped via `dotnet-counters` for + side-by-side "what the client's dashboard would say". Enable + `HandlerExecutionDiagnosticsEnabled` to capture `transport_lag_ms`/`receive_dwell_ms`. + Single box ⇒ no clock skew for cross-process wall-clock stages. +- **Run controller**: a small script (`rig.sh`) that sets the scenario via env vars, runs warmup + (≥60s — codegen, JIT, consumer-group stabilization), then a fixed 10-min measurement window, + and archives config + CSV + counter dumps per run ID. 2-hour soak reserved for the finale. +- **Topology**: local docker-compose Kafka (single broker — fine for relative comparisons; note + absolute transit will beat the client's real cluster). Topics explicitly provisioned with + ≥ 12 partitions (`.Specification(spec => spec.NumPartitions = N)` — auto-provision defaults to + **1 partition** which invalidates every parallelism test; `KafkaTopic.cs:43,435`). + +Reusable scaffolding: `src/Testing/Benchmarks/Driver.cs` + `targets.json` corpus, +`src/Persistence/LoadTesting` publisher shape, `src/Samples/AspireWithKafka` for wiring. + +### H2 — Microbenchmark suite (BenchmarkDotNet) +Add `KafkaHotPathBenchmarks` to `src/Testing/Benchmarks` (resurrect project; it's not in either +.slnx — add to full solution only if it builds clean). Benchmarks, each general-path vs +hypothetical fast-path: +1. `KafkaEnvelopeMapper.MapIncomingToEnvelope` with a realistic 12-15-header message vs a + hand-rolled fixed-schema reader (no dict, lazy decode, no re-scan). +2. `new Envelope()` (incl. wasted sequential-Guid) + header dictionary vs pooled/minimal envelope. +3. `TryFindMessageType` warm hit ×2 + `RequiresEncryption` in isolation. +4. `TryFindSerializer("application/json")` vs cached field. +5. `_executors[type]` + `GetType()` vs constant executor field. +6. `Envelope.Message` setter re-stamp vs raw field assignment. +7. `Executor.ExecuteAsync` fixed overhead: per-message CTS pair vs shared-deadline scheme. +8. `writeOutgoingOtherHeaders` (`Values.ToArray()` per send) vs precomputed set. +End-to-end micro: in-proc `Consume→handler-entry` latency, general pipeline vs fast-path +prototype. + +### H3 — dotTrace protocol (when the box frees up) +Box: Apple M5 Max, 18 cores, 128GB. Rider 2025.1 bundles dotTrace; for scripted capture install +CLI tools first: +```bash +dotnet tool install -g JetBrains.dotTrace.GlobalTools # dottrace attach/save-to-snapshot +dotnet tool install -g dotnet-counters dotnet-trace dotnet-gcdump +``` +Protocol per scenario: start rig → warmup → `dottrace attach ` in **Timeline** +mode for 120s of steady state (Timeline gives thread-state + lock-contention + GC lanes, which is +what T2/T3/T8 need; sampling mode as a second pass for pure CPU attribution). Also capture one +`dotnet-gcdump` mid-run for allocation census (T6) and `dotnet-counters monitor` for +`ThreadPool` queue length + GC pause. Name snapshots `-.dtp` and keep with the +CSV archive. Analysis targets: time split across +`Consume / mapper / inbox-insert / queue-dwell(thread-wait) / handler / flush / commit`. + +--- + +## 3. Intermediate steps (before the box is free) + +1. **Build H1 + H2 skeletons now** — they don't need the perf box; smoke-run at low rate against + docker Kafka on any machine to validate instrumentation plumbing (stage timestamps survive the + mapper, histograms non-empty, native twin honest). +2. **Fix CLAUDE.md** stale `_serializers` note (T6). +3. **Client questionnaire** (§7) — answers change the matrix weights. +4. **Pre-registered predictions**: keep §1 predictions as-is; the write-up will score them. +5. Decide branch hygiene: rig + benchmarks land on `main` (inert, not in CI); any scratch + instrumentation inside Wolverine itself stays on branch `perf/kafka-deep-dive`. + +## 4. Experiment matrix (macro rig) + +Axes, one change at a time from a fixed baseline +(**baseline** = client-shaped: Buffered everywhere, batch 10/10ms, semaphore-middleware +sequencing, 1Kb flow at ~8/s + 100Kb at ~0.6/s, self-publish leg on): + +| # | Experiment | Theory | Levers | +|---|---|---|---| +| E1 | Endpoint mode sweep C: Buffered / Durable(PG) / Inline | T3,T4 | `UseDurableInbox()`, `ProcessInline()` | +| E2 | Sender batching sweep A/B: (100,250ms) default / (10,10ms) / (1,1ms) / Inline send | T1,T5 | `MessageBatchSize/Timeout`, endpoint mode | +| E3 | Sequencing: semaphore-middleware vs `PartitionProcessingByGroupId(Five/Seven/Nine)` vs none | T2 | `ListenerConfiguration.cs:110` | +| E4 | Parallelism: `MaxDegreeOfParallelism` 1/5/18/36 × `ListenerCount` 1/3/6 | T3 | `Endpoint.cs:212,424` | +| E5 | Back-pressure: `BufferingLimits` (1000,500) / (10000,5000) / effectively-off | T3 | `Endpoint.cs:259` | +| E6 | Commit mode: StoreThenAutoFlush / BatchCount(100) / BatchInterval(5s) / PerMessage | T7,T8 | `KafkaListenerConfiguration.cs:42-74` | +| E7 | Mapper: default 19-header mapper vs `JsonOnlyMapper`-style minimal interop mapper | T6 | `UseInterop` | +| E8 | Payload: 1Kb vs 100Kb × fetch/linger ConsumerConfig/ProducerConfig tuning | T7 | config passthrough | +| E9 | Self-publish leg: Kafka round-trip vs durable local queue vs buffered local queue | — | routing | +| E10 | Multi-handler: single combined chain vs `MultipleHandlerBehavior.Separated` (fanout via `FanoutMessageHandler` re-dispatch) vs 2 sticky local queues | client shape | `WolverineOptions.cs:27-41`, `[StickyHandler]` | +| E11 | Telemetry cost: default vs `TelemetryEnabled(false)` + success-log at Information vs Debug | T6 | `ListenerConfiguration.cs:325` | +| E12 | Durable batch-insert prototype (O1) vs current per-message insert | T4 | branch build | + +Output per cell: p50/p95/p99 of each stage interval + throughput + GC/CPU counters; each cell +10-min window, 3 repetitions, report medians-of-percentiles. + +## 5. GlobalPartitioning-equivalent experiments (E13 block) + +The client wants same-key sequencing. Three Wolverine shapes to benchmark against their +co-partitioning + semaphore approach, cheapest first: + +1. **Kafka co-partitioning + `PartitionProcessingByGroupId`** (Buffered): keep their topology, + replace the semaphore middleware with the `ShardedExecutionBlock`. Needs + `GroupByMessageKey()` or `DisableConsumerGroupIdStamping()` + explicit GroupId so the + business key actually lands on `envelope.GroupId` (T7 footgun). No durability tax. This is + the likely recommendation. +2. **`ProcessConcurrentlyByKey(slots)`** (`KafkaListenerConfiguration.cs:179-194`): same block + but forces the durable inbox — measures the durability tax explicitly vs shape 1. +3. **Full `UseShardedKafkaTopics` GP** (`KafkaTransportExtensions.cs:300-338`): N topics + forced + Durable on external AND companion local slots (`GlobalPartitionedMessageTopology.cs:49-58`) + + bridge hop. Cluster-wide exclusivity guarantee, highest cost. Also note the May header bug + they hit (`775a67373`, GP interceptor dropped correlation/tenant headers) shipped fixed in + **5.39.0** — they can un-park; and `GlobalPartitionedInterceptor.ShouldIntercept` runs a LINQ + check on EVERY envelope of every non-sharded listener once any GP topology exists + (`GlobalPartitionedInterceptor.cs:135-151`) — include a "GP configured but message not + GP-routed" cell to price that. + +Deliverable: a decision table "which sequencing shape at which throughput/durability need" for +the docs. + +## 6. Optimization candidates (post-measurement backlog, pre-registered) + +Ordered by expected value; each gated on the matrix/microbench confirming its theory: + +- **O1 (T4): batch the durable inbox for single-envelope transports.** Micro-batch + `StoreIncomingAsync` behind the existing `_receivingOne` block (accumulate N/T like the + committer does) and/or let `KafkaListener` consume-many → `ProcessReceivedMessagesAsync` + (the multi-VALUES path already exists). Biggest structural win for durable Kafka. +- **O2 (T5): delete the per-message `Flush()` in `InlineKafkaSender`** — `ProduceAsync`'s ack is + already awaited; flush belongs in `Dispose`/drain. Near-free fix, arguably a bug. +- **O3 (T6): single-type fast-path listener.** Seams already exist: `Endpoint.MessageType` + + `DefaultIncomingMessage()` (`Endpoint.cs:416`, `ListenerConfiguration.cs:456`), + `Endpoint.DefaultSerializer`, per-endpoint `HandlerPipeline` construction + (`ListeningAgent.cs:82-95`), `IHandlerPipeline` interface, `KafkaTopic.BuildListenerAsync` + (`KafkaTopic.cs:224-269`). Design sketch: when an endpoint declares one message type + one + serializer, build a `SingleTypeHandlerPipeline` holding a pre-resolved + `(Type, IMessageSerializer, IExecutor, encryption-verdict)` tuple — skipping both string-keyed + type lookups, the serializer lookup, the executor lookup, and the `Message`-setter re-stamp — + paired with a minimal fixed-schema mapper (lazy header decode, no dictionary, no double decode, + no wasted Guid) and receive-side envelope pooling (send side already pools, #2726/#2955; + receive side does not — `KafkaTransportExtensions.cs:252`). Fall back to the general pipeline + if the type header disagrees. Microbench first (H2 #1-6 quantify the ceiling), then prototype + behind `ListenerConfiguration.OptimizeForSingleMessageType()`. +- **O4 (T6): mapper fixes independent of O3** — kill the incoming double-decode (decode once + into locals, populate dict lazily), precompute the reserved-header set instead of + `Values.ToArray()` per outgoing message, skip the `NewId` when the wire id will overwrite it. +- **O5 (T2/T6): cheap executor trims** — success log Information→Debug default (or sampled); + CTS pooling/deadline scheme; record sub-1ms executions (fix the `> 0` drop — measurement bug). +- **O6 (T1): batching ergonomics** — document the debounce semantics loudly; consider a + max-age flush (timer NOT reset per post, i.e. cap total wait at `MessageBatchTimeout`) as an + opt-in `MessageBatchMaxAge`; revisit `MessageBatchMaxDegreeOfParallelism=1` default for Kafka + (librdkafka is happy with concurrent produce). +- **O7 (T3): back-pressure without leaving the group** — Kafka `Pause()`/`Resume()` on assigned + partitions instead of listener dispose/rebuild (avoids rebalance storms). Bigger change; + price it only if E5 shows oscillation. +- **O8 (T8): committer lock → per-partition striping** if dotTrace shows contention. + +## 7. Questions back to the client (send early) + +1. Wolverine version? (Determines GP header fix ≥5.39.0, commit-mode defaults ≥ the #3134 fix, + raw-JSON mapper fixes ≥6.20.0.) +2. Endpoint modes per service, exactly — and where did "batch size = 10, batch timeout = 10ms" + come from / is it set per subscriber endpoint? (Not a Wolverine default.) +3. Where do their "transit/effective" timestamps start/stop in BOTH harnesses — and are they + comparing their own stamps or Wolverine's `wolverine-effective-time` (which includes flush + + ack and is clock-skew sensitive)? +4. Is the semaphore middleware inside the handler chain (⇒ counted as execution time)? +5. Avro serializer: registered as endpoint `DefaultSerializer`, or via a custom + `IKafkaEnvelopeMapper`? Do they rely on `envelope.GroupId`, and do they know inbound GroupId + defaults to the consumer-group name (`StampConsumerGroupIdOnEnvelope`)? +6. Producer/consumer configs: `linger.ms`, `batch.size`, fetch sizes — and any per-topic + `ConfigureConsumer` overrides (which silently drop transport-level settings other than + bootstrap/group/SASL). + +## 8. Sequencing & exit criteria + +- **Wave 0 (now, no perf box)**: H1+H2 skeletons, smoke runs, client questionnaire, CLAUDE.md + fix. Exit: rig produces believable stage histograms for both Wolverine and native twins at + low rate. +- **Wave 1 (box free)**: baseline + E1-E8 singles; dotTrace on baseline, E1-durable, E3. + Exit: ≥80% of the client's p50 transit and execution gaps attributed to named mechanisms + with numbers. +- **Wave 2**: E9-E11 (self-publish + multi-handler + telemetry), E13 GP-equivalents. + Exit: sequencing-shape decision table drafted. +- **Wave 3**: O2 + O4 + O5 quick wins implemented and re-measured (E-cells rerun); O1 and O3 + prototyped on `perf/kafka-deep-dive` with E12 verdicts. + Exit: PRs for confirmed wins; microbench deltas recorded in PR descriptions. +- **Wave 4**: update the "Performance Tuning" section of + `docs/guide/messaging/transports/kafka.md` (seeded 2026-07-18 with qualitative guidance) with + measured numbers from the ledger + reply to the client mapping each of their numbers to a + mechanism and a config change; 2-hour soak on the final recommended configuration. + +## 9. Measured-wins ledger (for release notes / blog posts) + +Every confirmed optimization gets a row here as it lands — filled in during Waves 3-4, kept +current so release notes and blog drafts can lift numbers directly instead of re-deriving them. +Rules: numbers only from rig/microbench runs archived with a run ID; record the exact scenario +(E-cell) and config so the claim is reproducible; before/after from the SAME rig version; phrase +the one-liner the way a release note would say it (user-visible effect, not internals). + +| Optimization | PR | Scenario (E-cell) | Metric | Before | After | Release-note one-liner | +|---|---|---|---|---|---|---| +| T1/O6 root cause: `BatchingChannel` debounce → max-age (JasperFx) | jasperfx PR (fix/batching-channel-max-age) | E2 batch-default (100/250ms) vs batch-1-1, local rig 2026-07-19 | transit p50 1Kb@8/s | **5,767ms** (default 100/250ms; debounce never fires under steady trickle); 263ms lone-msg 100Kb | bounded ≤ batch timeout by construction (re-measure after jfx pin bump) | "Kafka/transport sender batching now flushes within the batch timeout — a steady message trickle no longer postpones sends for seconds" | +| Client-shaped gap attribution (T1) | — | baseline (10/10ms) vs native-anchor | transit p50 1Kb@8/s | 19.9-21.2ms vs native 8.7-9.3ms (+11ms = client's 25.8 vs 7.9 reproduced) | batch(1,1ms): 7.6ms ≈ native; send-inline: 8.4-8.9ms ≈ native | "batch size/timeout tuning collapses the transit gap to native" | +| T4 durable inbox ceiling | — (O1 pending) | thru-durable 2000/s vs thru-buffered | transit p50 | durable **14,127ms** (backlog; per-msg INSERT gates consume loop) vs buffered 32ms vs native 8.5ms | O1 batched insert TBD | — | +| O2 inline-sender per-message Flush() removed | quick-wins PR | send-inline @8/s | transit p50 | 8.4ms (main) | 8.9ms (branch, noise—penalty is a concurrency tax, not visible single-threaded @8/s) | "Inline Kafka senders no longer block on a full producer flush after every send" | +| O4 mapper fixes (outgoing reserved-set cache, Kafka dict-first incoming) | quick-wins PR | H2 microbench 2026-07-19 | MapIncoming / MapOutgoing mean+alloc | 1,240ns/3,848B; 619ns/2,392B | **983ns/2,768B (-21%/-28%); 514ns/2,056B (-17%/-14%)** | "every Kafka receive maps ~21% faster with ~28% less allocation; every send ~17% faster" | +| O5 sub-1ms executions recorded | quick-wins PR | metrics | wolverine-execution-time | sub-1ms samples silently dropped (long, truncate, >0) | double histogram, all samples | "execution-time metrics no longer drop sub-millisecond handlers" | + +### Pre-fix throughput baselines (2026-07-19, fresh broker, worktree @f6f125f10: jfx 2.30.0, pre-O1) + +Max-throughput cells (uncapped publisher, no handler work, 12 partitions; consumed_per_sec over +trimmed receive window). Pre-registered expectations for the after-run in parentheses: + +| cell | BEFORE (jfx 2.30.0, pre-O1) | AFTER (jfx 2.30.1 + O1, measured 2026-07-19) | prediction verdict | +|---|---|---|---| +| max-native | **107,669/s** | 106,296/s | anchor stable ✓ | +| max-buffered | **10,726/s** | 10,872/s | unchanged as predicted ✓ (gap to native = pipeline/publisher cost → O3) | +| max-durable | **1,460/s** | **2,671/s (+83%)** | O1 real but < the 5-10x hoped; remaining ceiling = per-message mark-handled UPDATE on completion (**O1b follow-up: batch the handled updates**) | +| thru-durable @2000/s transit p50 | **14,127ms, unbounded backlog** | **31.5ms, keeps up** | **O1 user-visible headline** ✓ | +| thru-buffered @2000/s transit p50 | 32.4ms | 33.7ms | unchanged as predicted ✓ (size-flush dominates) | +| batch-default (100,250ms @8/s) transit p50 | **5,767ms** | **136ms (p99 262ms)** | timeout-bounded as predicted ✓ — the jfx max-age headline | +| batch-default lone msgs (100Kb @0.6/s) | 263ms | 264ms | unchanged as predicted ✓ (always paid full timeout) | +| baseline (10,10ms @8/s) transit p50 | 19.9ms | 20.0ms | unchanged as predicted ✓ | + +### Release-notes draft (6.21, Kafka/messaging performance — numbers from the ledger above) + +> **Message batching now flushes within the configured timeout.** The shared sender-batching +> channel treated its timeout as a quiet-period debounce: every published message reset the +> timer, so a steady stream could postpone sends until a full batch (default 100) accumulated. +> Measured on a 1Kb stream at 8 msg/s with default settings, publish-to-consume p50 latency was +> **5.8 seconds before the fix and 136ms after** (bounded by the 250ms default batch timeout; +> tighten `MessageBatchSize`/`MessageBatchTimeout` for latency-sensitive routes). Requires +> JasperFx 2.30.1. Applies to every transport that sends through Wolverine's batched sender. +> +> **Durable Kafka listeners persist incoming batches.** A durable (inbox-backed) Kafka listener +> used to make one database insert per record inline on the consume loop, capping consumption +> around **1,500 msg/s** locally and falling unboundedly behind at 2,000 msg/s (14s+ delivery +> latency in a 2-minute window). The listener now drains up to `MaximumMessagesToReceive` +> (default 100) already-fetched records into a single batched inbox insert: the same 2,000 +> msg/s load now runs at a steady **32ms** delivery p50, and maximum sustained durable +> throughput measured **+83%** (1,460 → 2,671 msg/s on the local rig). +> +> Also in this wave (#3501): every Kafka receive maps ~21% faster with ~28% less allocation and +> every send ~17% faster; inline Kafka senders no longer block on a full producer flush per +> message; `wolverine-execution-time` no longer silently drops sub-millisecond executions; the +> per-message success log now defaults to Debug (`Policies.MessageSuccessLogLevel` restores +> Information). + +Negative results so far (2026-07-19 local rig): +- **T2 semaphore sequencing**: no measurable execution-time inflation at 8/s across 20 games + (semaphore essentially uncontended at client-shaped rates). The client's 20.1 vs 8.9ms + execution gap needs their real contention profile — likely hot streams; not reproducible + at rig rates. Keep the PartitionProcessingByGroupId recommendation but don't promise numbers. +- **T5 inline Flush()**: invisible at 8/s single-threaded sends (~0.5ms noise). It's a + concurrency/throughput tax; the fix stands on inspection + code semantics, not a rig delta. +- **T3 back-pressure oscillation**: never triggered — buffered dwell stayed <1ms at 2000/s + (fast handler drains ahead of the 1000-message limit). Needs a slow-handler high-rate cell. +- **Resolution-chain lookups (part of T6)**: microbench shows warm TryFindMessageType 11.7ns, + TryFindSerializer 5.4ns, Message-setter re-stamp 4.7ns — trivial. The fast-path listener's + real ceiling is the mapper + allocations, not the ImHashMap reads. + +## 10. Risks / honesty notes + +- Single-box, single-broker rig understates broker transit and network effects; all conclusions + are relative (Wolverine vs native on identical infra), not absolute. +- The client's numbers include Marten writes we only simulate; durable-mode conclusions about + *their* DB contention need their schema (inbox and Marten sharing a PG instance compounds T4). +- M5 Max (ARM) GC/JIT behavior differs from their production x64 Linux — CPU-bound + microbenchmark ratios transfer, absolute numbers don't. +- Buffered-mode results must always carry the at-most-once caveat (offset stored on receive). diff --git a/NEXT-WAVE-HANDOFF.md b/NEXT-WAVE-HANDOFF.md new file mode 100644 index 000000000..0d9af9efa --- /dev/null +++ b/NEXT-WAVE-HANDOFF.md @@ -0,0 +1,227 @@ +# Next-wave execution handoff (state as of 2026-07-13 ~00:25 UTC) + +Self-contained handoff. Supersedes `NEXT-WAVE-RELEASES-PLAN.md` (kept for background). +**Track A is DONE — Wolverine 6.17.3 is cut.** Track B is **parked pending a reporter answer** (the +#3376 design does not survive contact with the code — see below). Track D has started. + +**Publishing policy (standing, from Jeremy):** JasperFx = automatic NuGet publish on green merge. +Marten = local-feed verification gate (verify Wolverine AND CritterWatch against the packed +candidate) before publishing. Wolverine ships via `publish_nugets.yml` + `V` tag; CritterWatch +via `publish-nuget.yml`; JasperFx/Marten via `on-manual-do-nuget-publish.yml` (`--ref main` / +`--ref master`). Merge gate everywhere: watch `gh pr checks` manually, **never `gh pr merge --auto`**. + +**Announcements (standing, from Jeremy, 2026-07-12):** +- **Discord**: announce every release. There is **no webhook/tool available** — Claude DRAFTS the + message, **Jeremy pastes it**. Announce **as soon as the publish workflow is green** (do NOT wait + for the nuget.org index to catch up). +- **GitHub release notes**: for every release, call out **all** issues and PRs that were part of it. + The V6.17.3 notes are the established shape (Closed issues / Fixes from review / Docs, with + contributor attribution). + +**Scope decisions from Jeremy (do not relitigate):** +- **wolverine PR #3387** ([Entity] load profiles, issue #3367): **NOT taking in at this time.** + Leave PR + issue open. The contributor has not been told — check with Jeremy before posting. +- CritterWatch 1.1-milestone items and post-1.0-labeled epics stay untouched. + +## Repos + +| Repo | Path | Notes | +|---|---|---| +| Wolverine | `~/code/wolverine` | `main`; push to `ghhttps` (`git fetch ghhttps main && git merge --ff-only ghhttps/main`) | +| JasperFx | `~/code/jasperfx` | `main` | +| Marten | `~/code/marten` | shared tree on a feature branch — **worktrees only** | +| CritterWatch | `~/code/CritterWatch` | `main` | + +--- + +## Track A — DONE. Wolverine 6.17.3 shipped + +**9 PRs merged, `V6.17.3` tagged, `publish_nugets.yml` run 29214705779, GitHub release notes published.** + +| Merged | What | +|---|---| +| #3364 | SNS per-tenant LocalStack fix (closed #3332) | +| #3384 | Metrics sweeper (closed #3375) | +| #3386 | gRPC saga coverage (refs #3385) | +| #3370 | RabbitMQ listener ghosting | +| #3389 | gRPC + Sagas docs page | +| #3390 | TrackedSession ignores `INotToBeRouted` (ProductSupport#33) | +| #3393 | Sweeper unregistration race + `UpdateMetricsPeriod` guard | +| #3394 | #3388 cold-path coverage + honor tracked-session timeout | +| #3395 | Testing-docs note | + +Also: CritterWatch PR #701 merged (#689 docs half; **#689 stays open** for the second half). +Filed **#3391** (RabbitMQ follow-ups: tracking-invariant test + eager-restart re-declare gap). +Closed #3392 as a duplicate of #3391. + +### Remaining Track A tail +- [ ] **Discord announcement for 6.17.3** — draft written in-session; hand to Jeremy on green publish. +- [ ] **Reply + close [ProductSupport#33](https://github.com/JasperFx/ProductSupport/issues/33)** + once 6.17.3 is resolvable. Fixed by #3390; `IgnoreMessagesMatchingType` remains the workaround + for ≤ 6.17.2. + +### 6.17.3 behavior change worth watching +`PauseThenCatchUpOnMartenDaemonActivity` now **honors the tracked session's timeout**. A test that +previously waited silently up to 60s now fails fast, telling the user to raise +`TrackActivity().Timeout(...)`. Correct, but expect questions. + +--- + +## Track B / C — #3376 is PARKED. The design note is partly wrong. + +**Do not start building JasperFx 2.28.0 lifecycle hooks until @erdtsieck answers on #3376.** +Findings (posted as a comment on the issue, 2026-07-13): + +1. **"Release the database's connection pool on revocation" is NOT implementable as specified.** + Marten's `NpgsqlDataSource` is owned by the tenancy's `MartenDatabase`, and under + database-per-tenant that same data source serves **ordinary application sessions** for that + tenant on that node. Disposing it on agent revocation would abort live app connections. There is + no "is anything else on this node using this DB" signal, and Marten has no API to evict a single + tenant database (`RefreshAsync()` blanks the cache **without disposing** → would leak data sources). +2. **Jeremy's correction, and it generalizes:** command/message processing opens connections to + **any** tenant database regardless of async-daemon affinity. So daemon scoping alone **cannot** + reach `databases + overlap` — the app's own tenant traffic is a second, independent axis of + connection demand. +3. **The high-water polling loop already stops** on last-agent-stop + (`JasperFxAsyncDaemon.StopAgentAsync`). The "only the owning node polls this database" half + largely exists today. +4. **Live footgun that would fully explain the reported 1,300 connections:** + `EventStoreAgents.StartAllAsync()` / `AllDaemonsAsync()` blanket-materialize a daemon for **every** + database and start **every** shard — no ownership check. `WolverineProjectionCoordinator` is + deliberately registered as a plain singleton so this never runs at bootstrap — **but + `IProjectionCoordinator` is itself an `IHostedService`**, so an app that also calls + `AddAsyncDaemon(Solo|HotCold)` alongside managed distribution gets `StartAsync` → `StartAllAsync` + → every node starts every shard on every database. That is exactly `nodes × databases`. + +**Asked @erdtsieck for:** (a) his registration — is `AddAsyncDaemon(...)` present alongside +`UseWolverineManagedEventSubscriptionDistribution`? (b) what fraction of the ~1,300 connections in +`pg_stat_activity` are the high-water query vs ordinary application traffic. + +If (a) is **yes** → this is a **config bug**: fix = startup guard + diagnostic, no JasperFx release +needed. If **no** → re-scope the hooks around what is actually achievable (daemon quiesce, **not** +pool release). + +**Real leaks worth fixing regardless** (found in recon, currently unticketed): +- `EventStoreAgents._daemons` is append-only — daemons are never released. +- `JasperFxAsyncDaemon`'s per-database `System.Timers.Timer` is started in the **constructor** and + only stopped in `Dispose()` — `StopAllAsync` does not touch it. +- The daemon's subscription to the database's `ShardStateTracker` is never disposed; a rebuilt daemon + re-subscribes and both observers stay attached. + +### Other Track C items (independent of #3376, still valid for 6.18.0) +- **#3385 scoped diagnostic**: replace the opaque `IndeterminateSagaStateIdException` over a gRPC hop + with "header-identified saga over gRPC is not supported; put the saga identity on the message + body", and flip the characterization test + `starting_a_header_identified_saga_over_grpc_fails_with_opaque_status_today` + (`src/Wolverine.Grpc.Tests/SagaOverGrpc/saga_over_grpc_tests.cs`). Link the gRPC sagas docs page + (shipped in #3389). +- **#698 upstream half**, if Track D's investigation confirms one. + +--- + +## Track D — CritterWatch beta.4 + +The **#698 investigation is IN FLIGHT** (leading hypothesis: an agent-store vs restriction-store +mismatch — event-subscription agents live in an ancillary Marten store while restrictions write to +the main store, which would explain `wolverine_agent_restrictions` having 0 rows). + +1. **#698 Pause-acks-success-but-never-pauses**: fix the Wolverine side if broken (→ Track C / + 6.18.0). **CritterWatch side regardless**: the pause handler must verify observed reality + (restriction row present, or agent actually stopped) before acking Succeeded. A green ack for a + no-op is worse than a visible failure. +2. **#697**: mirror the `ClearAlert` pattern for Acknowledge/Snooze in `AlertCommandHandler` (actor + from envelope principal w/ UI fallback, on the events + `AlertRecord`, plus `auditLog.LogAsync`). + Add the `/audit` → `/audit-log` route alias (same class as #693's `/dead-letters` → `/dlq`). +3. **#699 + #636 together** (same timeline/event-feed surface): materialize a timeline entry only on + (agent → node) assignment CHANGE; dedupe consecutive identical entries; retention for + `TimelineEntry` docs (reuse the #468/#695 approach — and per the #685 lesson, any new table shape + must handle upgraded stores loudly). Then #636's Recent Events widget defects on top. +4. **#689 second half**: evaluate deferring the capability snapshot's ApiExplorer read + (`ServiceCapabilities.ReadFrom` → `OpenApiDescriptorBuilder.TryBuildForWolverine`) until + `ApplicationStarted`. Close #689 either way (docs half was PR #701). +5. **#610**: audit COMPLETE → **close with evidence + file the follow-ups** (see addendum below). +6. **Pull-in candidates if room**: #632 explorer navigability; #670 sequence-diagram click-through; + #347 manual-test walkthrough (PRIORITY-tagged, pure writing). +7. Pins (Wolverine 6.18.0, JasperFx 2.28.0 if it ships, Marten if released) → **beta.4** → ask + erdtsieck to re-verify #697/#698/#699 + the round-1 fixes; nudge him onto the beta.3/6.17.2+ + baseline first (his round-2 reports were against beta.2). + +--- + +## Sweep result (2026-07-12, pre-release, all 5 repos) + +Marten, Polecat, CritterWatch, ProductSupport: **clean** — no regressions from Marten 9.15.0 / +JasperFx 2.27.0 / CW beta.3, and no new customer reports. ProductSupport has exactly **one** open +issue (#33, fixed by #3390). + +Wolverine new-but-not-blocking: **#3380** (OpenAPI: route params bound only by compound-handler +`LoadAsync`/`Before` are missing from the operation — same class as the #3135 audit), **#3366** +(`UseAzureServiceBusTesting()` documented but test-suite-only), **#3365** (Polecat primary +`IEventStore` bridge registers twice). Polecat **#320** (`IEventStore.Subject` is the DB URI, so +primary + ancillary stores on one database are indistinguishable → CritterWatch HWM buckets collide) +is small and already has a pinned skipped test waiting. + +Marten **#4920** (Guid `CompareTo` in LINQ, community) merged 07-12 — confirm whether it made 9.15.0 +or is unreleased on master; it is the only candidate reason for a Marten patch this wave. + +--- + +## Communications checklist + +- [ ] Discord: 6.17.3 (draft ready, awaiting green publish) — then jfx 2.28.0 / 6.18.0 / CW beta.4 +- [ ] ProductSupport#33 reply + close after 6.17.3 is resolvable +- [ ] **wolverine#3376: awaiting @erdtsieck's registration + connection breakdown — Track B is + blocked on this** +- [ ] **wolverine#3388: awaiting @uniquelau's re-verify** — asked him to raise + `TrackActivity().Timeout(...)` on his real monitored host. Keep open until he reports back. +- [ ] #3387/#3367 contributor: deferral message is JEREMY'S call — ask him, don't send +- [ ] CW #688 design exchange with erdtsieck; keep the #699 coalescing consistent with it +- [ ] erdtsieck beta.4 verification ask + +## Conventions (unchanged, binding) + +Full `wolverine.slnx` Release build before pushing; `--framework net9.0` for fast test iteration; +`Servers` for connection strings (compose Postgres = 5433); rebuild after every stash push/pop; +ImHashMap for hot-path lookups; private members camelCase; TCS-gated concurrency tests; version bump +before every publish (`--skip-duplicate` silently no-ops); `say` checkpoints; draft user-facing +wording and interview Jeremy at the end. + +## Definition of done + +- [x] Track A queue merged; Wolverine 6.17.3 shipped; GitHub release notes published +- [ ] Discord announcement + ProductSupport#33 closed +- [ ] #3376 direction resolved with the reporter (config bug vs architecture change) +- [ ] #3385 diagnostic shipped; characterization test flipped +- [ ] CW #697, #698 (both halves), #699+#636, #689 (closed), #610 (closed with evidence); beta.4 shipped +- [ ] All communications checklist items done + +--- + +## ADDENDUM: CW #610 verification verdict (completed 2026-07-12, vs origin/main @ 63519193) + +**Recommendation: CLOSE #610 with follow-ups.** The #694 (beta.3) read-side work answers the +substantive concerns — every shard state carries its own `(storeUri, tenantId, databaseIdentifier)` +and all gap math + action dispatch flows through that. Per-question: + +| Q | Status | +|---|---| +| 1. Row attributed to store AND tenant | **RESOLVED** for Marten (`projections-store.ts:331`, `belongsToSubscription :273-279`, three-map sourcing `:335-357`). Polecat caveat: primary + ancillary sharing one DB are indistinguishable (`IEventStore.Subject` = database URI) — already filed upstream as **polecat#320** (pin test skipped in `src/SqlServerTests/polecat_ancillary_ieventstore_registration.cs`), plus **wolverine#3365** (double bridge registration → double poll). | +| 2. HWM/gap per-store-per-tenant | **RESOLVED** — HWM maps keyed `service → store → tenant` (`:958-971`) and per-database (`highWaterMarkForDatabase :643-645`); `hwmForShardState :567-584` picks tenant → own-database → store, never the cross-store last-writer mark. Minor edge: `perTenantGap :1675-1698` lacks a fallback store URI, only matters for pre-store-tagging satellites with the same tenant id on two partitioned stores. | +| 3. Grouping composes store × tenant | **PARTIAL** — flat view composes fully; the opt-in tenant-grouped view (#263 Phase 3c) has NO store axis: `PerTenantProjectionRow` lacks `storeUri` (`ProjectionsPage.vue:827-847`), rows bucket by projection base name (`:881-886`), `agentUri` = first name match across all stores (`:889`) → row fusion + possible wrong-store agentUri when two tenant-partitioned stores share tenant ids or projection names. | +| 4. Actions target correct (store, tenant) DB | **MOSTLY RESOLVED** — dispatch carries `(agentUri, tenantId)`; `AgentUriResolution.ResolveAsync` → `FindAgentUriAsync(shardIdentity, tenantId)` returns the per-tenant agent at the tenant's own DB; `RebuildScope.Resolve` rejects scope conflicts. Residual: `ShardIdentityFromLiveAgentUri` discards the store segment (`AgentUriResolution.cs:76-95`) and both resolver loops ask every family first-match-wins (`:105-118`; `RebuildProjectionHandler.cs:104-112`) → cross-store shard-identity collisions can resolve against the wrong store. jasperfx#502 registry-baseline gap on db-per-tenant also remains. | + +**Coverage gap confirmed:** no test anywhere exercises ancillary stores AND db-per-tenant in the +same service (`projections-store-682-read-side.test.ts` is single-store; MultiTenancyTests all +single-store; MTTrips = db-per-tenant/single-store, MultiStoreHost = multi-store/no db-per-tenant). + +**Follow-ups to file on close (smallest-first — fold 1-2 into beta.4, rest as issues):** +1. Combined-composition regression test: add a single-DB ancillary store to + `projections-store-682-read-side.test.ts`; assert no cross-store mark borrowing. (S) +2. Store-scope the family loops: filter `IEventSubscriptionAgentFamily` by the store segment + already in the live agent URI, in `AgentUriResolution.ResolveAsync` and + `RebuildProjectionHandler.TryRebuildRegisteredProjectionAsync`. (S-M) +3. Store axis in the tenant-grouped view: `storeUri` on `PerTenantProjectionRow`, bucket by + store × tenant, resolve agentUri from the owning store's ProjectionView. (M) +4. Sample + e2e for the composition: ancillary store in MTTrips (or db-per-tenant in + MultiStoreHost) + one backend per-tenant-action test in that host. (M) +5. Link on close: polecat#320, wolverine#3365, jasperfx#502 (all already filed). diff --git a/NEXT-WAVE-RELEASES-PLAN.md b/NEXT-WAVE-RELEASES-PLAN.md new file mode 100644 index 000000000..9f0eaceeb --- /dev/null +++ b/NEXT-WAVE-RELEASES-PLAN.md @@ -0,0 +1,247 @@ +# Next-wave release plan — post-sweep follow-through (2026-07-12 evening state) + +> **SUPERSEDED (2026-07-12 ~23:30 UTC)** by `NEXT-WAVE-HANDOFF.md`, which reflects the mid-wave +> execution state (Track 1 queue nearly complete, PRs #3389/#3390/CW#701 open, #3387 deferred, +> ProductSupport#33 + CritterWatch unblocked-now items folded in). Hand THAT file to the agent; +> keep this one for background rationale. + +Successor to `ERDTSIECK-EPIC-PLAN.md` (that epic is COMPLETE — see "What landed" below). +Scope: the open community PR queue, the #3376 implementation, three new CritterWatch issues +from @erdtsieck, and the release train that ships all of it. + +--- + +## What landed (previous wave — all same-day 2026-07-12) + +**Releases shipped:** JasperFx **2.27.0** (V2.27.0), Marten **9.15.0**, Wolverine **6.17.2** +(pins already bumped via #3383), CritterWatch **1.0.0-beta.3** (pins via #696). + +**Issues fixed & closed (15):** + +| Repo | Closed | Via | +|---|---|---| +| jasperfx | #505, #506, #507 | PR #508 (evolver cast), PR #509 (Block observability + fault semantics + narrowed teardown catches) | +| marten | #4941, #4942, #4943 | PR #4945 (idempotent provisioning repair on auto-assign path); #4941/#4943 closed by combination/reporter | +| wolverine | #3371, #3372, #3374, #3368 | PR #3373 (uniquelau, HttpGraph provider), #3379 (opt-in strict query binding), #3381 (binding frames once per chain), #3382+#3369 (gRPC tenant detection + envelope propagation) | +| CritterWatch | #682–#687 | PRs #690–#695, #700 (wording), shipped in beta.3 | + +**Issue-closure audit — nothing to close manually right now.** Every remaining open issue is +legitimately open, and four of them close automatically when their in-flight PR merges: + +| Open issue | Closes via | State | +|---|---|---| +| wolverine#3375 | **PR #3384** (erdtsieck's sweeper — "Closes #3375") | PR open, needs review | +| wolverine#3367 | **PR #3387** (outofrange-consulting's load-profile POC) | PR open, needs decision+review | +| wolverine#3332 (CIAWS timeout) | **PR #3364** (Steve-XYZ — "Fixes #3332") | PR open, needs review | +| jasperfx#510 (new, uniquelau) | **PR #511** (uniquelau — "Closes #510") | PR open, needs review | +| wolverine#3376 | design note posted by maintainer; implementation is THIS wave's headline | open | +| wolverine#3385 (new, erikshafer) | direction decision needed; test PR #3386 pins the gap | open | +| CritterWatch #688 | design epic (per-tenant at scale) — ongoing | open | +| CritterWatch #697/#698/#699 (new, erdtsieck) | this wave, beta.4 | open | + +--- + +## New inbound since the sweep + +### erdtsieck round 2 — CritterWatch operator-trust issues (filed ~16:00–17:00, vs beta.2) + +All three were found on **beta.2** — first triage step: confirm none were incidentally fixed in +beta.3 (they weren't in its PR list, but verify behavior before coding), and ask him to upgrade +to beta.3 + Wolverine 6.17.2 so round-2 fixes are verified on current bits. + +- **[#697](https://github.com/JasperFx/CritterWatch/issues/697)** — Acknowledge/Snooze alert writes + no audit entry and records no actor; `ClearAlert` does both. Fix: mirror the ClearAlert pattern in + `AlertCommandHandler` (`AcknowledgedBy`/`SnoozedBy` from envelope principal with UI fallback, on + the events + `AlertRecord`, plus `auditLog.LogAsync`). Also `/audit` → `/audit-log` route alias + (same class as the `/dead-letters`→`/dlq` alias from #693). **S** +- **[#698](https://github.com/JasperFx/CritterWatch/issues/698)** — Pause projection acks Succeeded + (+audit +lifecycle event) **but the agent never pauses**; `wolverine_agent_restrictions` has 0 rows. + Two-part: + 1. **Wolverine-side investigation (do first — possibly an upstream bug):** does + `IAgentRuntime.ApplyRestrictionsAsync` actually persist a restriction row for + event-subscription agents on 6.17.x, and does the leader-forwarding path + (#478 `LeaderExecution.ShouldExecuteHereAsync`) execute it on a node that can write the + restriction store? Reproduce with a 2-node cluster + Marten ancillary store agent. If broken + upstream, that's a Wolverine issue + fix in this wave's Wolverine release. + 2. **CritterWatch-side regardless of root cause:** `PauseProjectionHandler` must verify observed + reality (restriction row present, or agent actually stopped) before acking Succeeded — a green + ack + audit trail for a no-op is worse than a failure. **M overall** +- **[#699](https://github.com/JasperFx/CritterWatch/issues/699)** — Timeline flooded by keep-alive + agent re-reports rendered as "Agent started" (3.5k/hour, 671k rows, no retention). Fix: the + console materializes a timeline entry only on **(agent → node) assignment change** (ServiceSummary + knows the previous state); dedupe identical consecutive entries; add retention for + `TimelineEntry` docs (same growth-pressure class as #468 metrics samples — reuse that approach). + Keep the keep-alive as state refresh, it's correct for health. **M** + +### Other new community items + +- **jasperfx#510 / PR #511** (uniquelau) — `RunCommand` double-starts a host already started under + `JasperFxEnvironment.AutoStartHost` (Alba/WebApplicationFactory harnesses), re-running every + `IHostedService.StartAsync`. Repro against 2.27.0 included. Review checklist: fix must skip + `StartAsync` only for the pre-started `PreBuiltHostBuilder` path, not change cold `run` semantics; + needs a regression test with a counting `IHostedService`. +- **wolverine#3385** (erikshafer) — gRPC can't start/continue **header-identified** sagas + (`saga-id` never crosses the hop: interceptors don't carry it, `Executor.InvokeAsync` doesn't + seed `envelope.SagaId` from context). Message-body-identified sagas already work (proven by his + test PR #3386). Decision per his own framing: **ship the scoped first cut** — clear diagnostic + instead of opaque `IndeterminateSagaStateIdException`, document body-identity as the supported + path; defer full three-point propagation until someone shows a concrete header-identity need. + +--- + +## The wave itself + +### Track 1 — Wolverine PR review queue → ship as 6.17.3 (patch, fast) + +**Added scope (2026-07-12 evening):** +- **[ProductSupport#33](https://github.com/JasperFx/ProductSupport/issues/33)** — CritterWatch + telemetry caught by tracked-session waits. Fix (branch `trackedsession-ignore-telemetry`): + TrackedSession's default ignore rule extended from `IAgentCommand` to all `INotToBeRouted` + (covers `ICritterWatchMessage` telemetry) with an explicit carve-out for `Acknowledgement` / + `FailureAcknowledgement`, which the session's ack APIs depend on. Ships in **6.17.3**. Also + document `IgnoreMessagesMatchingType` as the workaround for older Wolverine versions in the + testing docs, and reply/close on ProductSupport#33 once released. +- **Sweeper follow-up from the #3384 post-merge review** — one-line unregistration-race fix + (`TryRemove` with exact KVP instance in `PersistenceMetricsSweeper`) + `UpdateMetricsPeriod` + zero-guard. Fold into 6.17.3. + +Bug/scale fixes only; erdtsieck's production benefits immediately. Review in this order: + +1. **PR #3384** (erdtsieck — metrics sweeper, closes #3375). Review focus: `DurabilityAgent` + register/unregister lifecycle vs node shutdown races; the dynamic re-read of the registration + set (he explicitly built it to compose with #3376's owned-agent scoping — verify that claim); + at-most-one-in-flight concurrency test quality; CosmosDb/RavenDb single-store agents correctly + left on `StartPolling`. Confirm OTel gauge tags and `PersistedCounts` feed are byte-identical + (CritterWatch depends on them). +2. **PR #3370** (kconfesor — RabbitMQ listener ghosting after broker restart). Directly adjacent to + the #3171/#3187 channel-only-shutdown work — review against that: removing `_monitor.Remove(this)` + must not reintroduce the latched-Disconnected state #3187 fixed; the agent must stay tracked so + `connectionOnRecoverySucceededAsync` rebuilds it. Ask for/verify a test in the compliance-test + style; be mindful the RabbitMQ suite has known shared-broker flakes. +3. **PR #3364** (Steve-XYZ — SNS per-tenant LocalStack fix, re-enables CIAWS, fixes #3332). + Low-risk test-only change; the authoritative validation is the CI run itself. Merging also + partially addresses #3350 (leave #3350 open for the CIPolecat half). +4. **PR #3386** (erikshafer — gRPC saga coverage, test-only, refs #3385). Merge as no-regret + groundwork; the characterization test's assertions flip when the #3385 diagnostic lands. + +Merge gate for all: green CI watched manually (`gh pr checks`), **never `--auto`**. Then ship +**Wolverine 6.17.3** (bump version, `publish_nugets.yml` + `V6.17.3` tag). + +### Track 2 — JasperFx 2.28.0 + +1. **PR #511** (uniquelau) — review + merge (closes #510). +2. **#3376 companion: per-database daemon lifecycle hooks.** Implement the JasperFx.Events side of + the design note posted on wolverine#3376: start/stop hooks for per-database daemon + infrastructure (HighWaterAgent lifecycle) that Wolverine's distribution layer can drive on agent + assignment/revocation. Follow the design note on the issue; erdtsieck was invited to comment — + check the issue thread for his feedback before finalizing the hook shape. +3. Ship **JasperFx 2.28.0** — AUTOMATIC publish on green merge (policy carried over): + `gh workflow run on-manual-do-nuget-publish.yml --repo JasperFx/jasperfx --ref main`, bump + `` first, poll the flat container after. + +### Track 3 — Wolverine 6.18.0 (the headline release) + +1. **#3376 implementation** — owned-agent connection scoping on the Wolverine side, consuming the + 2.28.0 hooks: managed distribution decides which databases a node materializes daemon infra for; + assignment starts it, revocation stops it and releases pools; grace overlap during rebalancing. + Target: steady-state connections ≈ `databases + small overlap`, not `nodes × databases`. + Multi-node tests per the original plan (assert zero pools for unowned databases post-rebalance; + TCS-gated, no sleeps). Composes with the #3384 sweeper's dynamic set — add a joint test. +2. ~~PR #3387 ([Entity] load profiles)~~ — **DEFERRED per Jeremy (2026-07-12): not taking this in + at this time.** Leave the PR and #3367 open; no review/merge work this wave. Revisit later. +3. **#3385 scoped diagnostic** — replace the opaque exception with a clear "header-identified saga + over gRPC is not supported; put the id on the message body" diagnostic + docs note; flip the + #3386 characterization test. NOTE: the gRPC + Sagas docs page (PR #3389, this wave) already + documents the limitation — the diagnostic work should link to it. +4. **#698 upstream half** — if Track 4's investigation shows `ApplyRestrictionsAsync` doesn't + persist restrictions for event-subscription agents (or leader-forwarding writes to the wrong + store), fix here. +5. Marten/JasperFx pin bumps to 2.28.0 (+ Marten if it releases). Full + `dotnet build wolverine.slnx -c Release` before pushing, as always. +6. Ship **Wolverine 6.18.0**. + +### Track 4 — CritterWatch beta.4 + +**Unblocked-now additions (2026-07-12 sweep of the non-1.1 backlog; 1.1-milestone items and +post-1.0-labeled epics deliberately untouched):** + +- **[#689](https://github.com/JasperFx/CritterWatch/issues/689)** — UNBLOCKED: Wolverine 6.17.2 + shipped the #3371/#3373 OpenAPI fix. Write the upgrade-notes/compat entry (monitored hosts using + `AddOpenApi()` should be on Wolverine ≥ 6.17.2), and evaluate the issue's second ask — deferring + the capability snapshot's ApiExplorer read until after app start so CritterWatch isn't the + trigger on unfixed hosts. Small; do now, close with beta.4. +- **[#610](https://github.com/JasperFx/CritterWatch/issues/610)** — likely LARGELY RESOLVED by the + #682/#694 dimension-aware read-side work (which was driven by exactly the composition #610 worries + about: db-per-tenant main store + ancillary stores). Verification in progress; expected outcome is + close-with-evidence plus at most a small follow-up (the (store × tenant) action-targeting check and + the missing combined-axes sample/test noted in the issue). +- **[#636](https://github.com/JasperFx/CritterWatch/issues/636)** — Recent Events widget live-feed + defects: bundle with #699 (same timeline/event-feed surface; the assignment-change-only + materialization from #699 directly changes what this widget shows). Do together in beta.4. +- **Candidates, not committed** (pull in if beta.4 has room): #632 (explorer navigability quick + wins — breadcrumbs/deep-links; parent epic #631 stays post-1.0), #670 (sequence-diagram + click-through — check whether the #687 renderer rework in beta.3 changed feasibility), and the + manual-test walkthrough docs (#347 is marked PRIORITY; pure writing, no code dependency). + +Order: #698 investigation first (it may feed Track 3), then fixes. + +1. **#698** — reproduce pause-no-op against beta.3 + 6.17.2 on a 2-node cluster. Split the fix: + verify-before-ack in `PauseProjectionHandler` (CritterWatch, always) + whatever Track 3 item 4 + finds (Wolverine, maybe). The reporter's environment is "freely pokeable" — take him up on it. +2. **#697** — actor + audit entries for Acknowledge/Snooze, `/audit` route alias. +3. **#699** — assignment-change-only timeline materialization + consecutive dedupe + retention + policy for `TimelineEntry` (reuse the #468/#695 partitioning/retention approach — and check the + beta.1-drift lesson from #685: any new table shape must handle upgraded stores loudly). +4. **#688 design** — continue the per-tenant-at-scale design exchange with erdtsieck; no code + gated on it for beta.4, but the #699 roll-up thinking (dedupe, coalesce) should be consistent + with the #688 direction (alert coalescing). +5. Pin bumps (Wolverine 6.18.0, JasperFx 2.28.0, Marten if released) → ship **beta.4**; ask + erdtsieck to re-verify #697/#698/#699 plus the round-1 fixes on his environment. + +### Marten this wave + +No open community issues. Work item: consume JasperFx 2.28.0 (daemon lifecycle hooks may require +daemon-hosting adaptation in Marten — assess once the hook shape is final). If a Marten release is +needed, the **local NuGet verification gate applies unchanged**: pack `-local.N` → +verify Wolverine AND CritterWatch against the local feed → only then publish +(`on-manual-do-nuget-publish.yml --ref master`). Worktrees only, never the shared tree. + +--- + +## Sequencing + +``` +Track 1 PR queue reviews (#3384, #3370, #3364, #3386) ──► Wolverine 6.17.3 (independent, do first) +Track 2 jasperfx PR #511 + #3376 lifecycle hooks ──► JasperFx 2.28.0 (auto-publish) + │ +Track 3 #3376 impl + PR #3387 + #3385 diagnostic ────────┴──► Wolverine 6.18.0 + ▲ │ +Track 4 #698 investigation ──(upstream half, if any)──┘ │ + #697/#699 fixes ──────────────────────────────► CW beta.4 (pins 6.18.0/2.28.0) +Marten consume 2.28.0; release only if needed (local-feed gate: Wolverine + CritterWatch) +``` + +Track 1 has zero dependencies — start immediately; it also de-risks erdtsieck's production while +Tracks 2/3 build. The #698 investigation should start early since its outcome shapes Track 3. + +## Working conventions + +Unchanged from `ERDTSIECK-EPIC-PLAN.md` — full-solution builds, worktrees for Marten, manual +merge-on-green (never `--auto`), version bump before every publish, `--skip-duplicate` no-op trap, +nuget index-lag polling, TCS-gated concurrency tests, `ImHashMap` on hot paths, `Servers` for +connection strings, `say` checkpoints, wording interview with Jeremy for user-facing text. + +## Definition of done + +- [ ] PRs #3384, #3370, #3364, #3386 reviewed + merged; issues #3375, #3332 auto-closed +- [ ] Wolverine 6.17.3 shipped and resolvable +- [ ] jasperfx PR #511 merged (#510 closed); #3376 lifecycle hooks merged; JasperFx 2.28.0 auto-published +- [ ] wolverine #3376 implemented + multi-node tests green; joint test with the #3384 sweeper +- [ ] PR #3387 reviewed + merged (#3367 closed); docs page for load profiles +- [ ] #3385 scoped diagnostic merged; #3386 characterization test flipped; docs updated +- [ ] #698 root cause identified (Wolverine vs CritterWatch or both); fixes merged on the right side(s); verify-before-ack in place +- [ ] Wolverine 6.18.0 shipped (pins at 2.28.0/latest Marten); full wolverine.slnx Release build green +- [ ] CritterWatch #697, #698, #699 fixed + tested; beta.4 shipped with bumped pins +- [ ] Marten: 2.28.0 consumption assessed; if released, local-feed gate passed first +- [ ] erdtsieck replied to on #697/#698/#699 + asked to upgrade to beta.3/6.17.2 baseline, then re-verify on beta.4/6.18.0 +- [ ] #688 design exchange has a concrete next artifact (wireframe/design note), consistent with the #699 coalescing work diff --git a/RABBITMQ-PERF-DEEP-DIVE-PLAN.md b/RABBITMQ-PERF-DEEP-DIVE-PLAN.md new file mode 100644 index 000000000..faf384df2 --- /dev/null +++ b/RABBITMQ-PERF-DEEP-DIVE-PLAN.md @@ -0,0 +1,192 @@ +# RabbitMQ Performance Deep-Dive Plan (2026-07-18) + +Goal: measure Wolverine-over-RabbitMQ overhead vs raw RabbitMQ.Client 7.x across endpoint modes, +produce throughput-tuning guidance, and land the confirmed optimizations. Companion/umbrella: +`KAFKA-PERF-DEEP-DIVE-PLAN.md` (wolverine#3490) — the metrics semantics (§0 there: +execution-time vs effective-time definitions, sub-1ms sample drop, middleware-blocking +inclusion), the resolution-chain/mapper base costs, the BatchedSender debounce, and the +back-pressure agent mechanics are shared and not restated in full here. + +All file:line cites verified against `main` @ 6.20.0. Client lib: RabbitMQ.Client **7.1.2** +(fully async). + +--- + +## 0. Transport-specific facts that frame everything + +- **Default endpoint mode is `Inline`** (`RabbitMqEndpoint.cs:21`, re-asserted + `RabbitMqQueue.cs:43`) — unlike the core default of BufferedInMemory. Combined with + `ConsumerDispatchConcurrency` default **1** (`WolverineRabbitMqChannelOptions.cs:28`), an + out-of-the-box RabbitMQ listener consumes **one message at a time, end-to-end through the + handler**, per endpoint. `MaxDegreeOfParallelism` is irrelevant for inline; the only scaling + knobs are `ListenerCount` (N channels+consumers, `ListeningAgent.cs:365-375`) and the + transport-wide dispatch concurrency. +- **RabbitMQ never hands the receiver `Envelope[]`** — the consumer's only call is the + single-envelope `ReceivedAsync` (`WorkerQueueMessageConsumer.cs:100`). The batched + multi-VALUES durable-inbox insert (`DurableReceiver.cs:608-668`) is therefore **unreachable**; + durable RabbitMQ pays one `StoreIncomingAsync` INSERT per message (`DurableReceiver.cs:494`) + even though prefetch delivers natural batches. Same structural gap as Kafka (#3490 T4); SQS + and ASB do *not* have it. +- **Prefetch defaults** (`RabbitMqQueue.cs:64-83`): Buffered/Durable → `MaxDegreeOfParallelism + × 2`; Inline → **100** (mostly useless for inline beyond hiding per-message network latency). +- **Publisher confirms default OFF** (`WolverineRabbitMqChannelOptions.cs:14,21`) — fast but + fire-and-forget; when enabled, RabbitMQ.Client 7 awaits the broker ack inside + `BasicPublishAsync` = one RTT per publish with **no confirm windowing** in Wolverine. + `mandatory` is hardcoded false with no `BasicReturn` handler (`RabbitMqSender.cs:91`) — + unroutable messages silently drop. +- **No BatchedSender**: `RabbitMqSender` is a plain `ISender`; every mode funnels to + per-envelope `BasicPublishAsync` on one shared channel per endpoint (`RabbitMqSender.cs:67-92`). + So the Kafka debounce theory does NOT apply here; per-publish costs do. +- **Acks**: manual, per message, via RetryBlock → `BasicAckAsync(deliveryTag, multiple: true)` + (`RabbitMqListener.cs:317-320`). `multiple:true` makes most acks cumulative-redundant — a + coalescing opportunity, and worth an eyebrow for out-of-order completion semantics. +- **Back-pressure stop disposes the channel** (`ListeningAgent.cs:451-484` → + `RabbitMqChannelAgent.cs:218-232`): all unacked/prefetched messages redeliver. Sawtooth load + at the 1000/500 `BufferingLimits` thresholds = redelivery churn (RabbitMQ analogue of Kafka's + rebalance oscillation). + +## 1. Theories of overhead, ranked + +- **R1 (HIGH): inline-by-default single-file consumption.** Most "Wolverine RabbitMQ is slow" + reports should reproduce as: default endpoint → 1 msg at a time × handler latency. + Prediction: throughput scales ~linearly with `ListenerCount` and with switching to + Buffered; docs need a loud "scaling RabbitMQ consumption" section. +- **R2 (HIGH, durable leg): per-message inbox INSERT** despite prefetched natural batches. + Prediction: durable throughput ceiling ≈ 1/insert-RTT per listener; micro-batching the + inserts (RO1) is the biggest structural win. +- **R3 (MEDIUM): per-message mapper + publish allocations** — incoming double-decode of every + reserved header (`RabbitMqEnvelopeMapper.cs:74-100`), `body.ToArray()` copy per delivery + (`WorkerQueueMessageConsumer.cs:49`), outgoing `Values.ToArray()` + O(n×m) `Contains` + (`EnvelopeMapper.cs:391-397`), `new BasicProperties` + header `Dictionary` per publish + (`RabbitMqSender.cs:82-86`), Guid parse/ToString per message. CPU/GC pressure at high rates; + quantify via microbench (shared suite from #3490 H2, RabbitMQ variants). +- **R4 (MEDIUM): publisher-confirm mode cliff.** Confirms off = fastest but silently lossy on + broker failure; confirms on = per-publish RTT serialization on the caller. Measure both; + guidance must state the cliff and the durable-outbox interaction (outbox + confirms = paying + twice for the same guarantee?). +- **R5 (MEDIUM): back-pressure channel teardown** redelivery churn under sustained load — + p99 spikes + duplicate executions (buffered mode already acked, so its buffered backlog is + lost instead; #3137 family). +- **R6 (LOW-MEDIUM): ack RPC per message** through a RetryBlock (`RabbitMqChannelCallback.cs:16-34`) + — Task + try/catch per ack; `multiple:true` means a watermark committer (like Kafka's + `KafkaOffsetCommitter`) could ack once per N. +- **R7 (LOW): topic-routed sends may deserialize the body per send** to compute the routing key + (`RabbitMqSender.cs:42-61`) — only for topic-exchange publishing; check with a targeted cell. +- **R8 (LOW): `DeferAsync` = ack + full republish** (`RabbitMqEnvelope.cs:34-47`) — every + retry-requeue pays a publish; only matters under high failure rates. + +## 2. Harness + +Extend the `KafkaPerfRig` from #3490 into a transport-pluggable rig (`TransportPerfRig`): same +three-service topology, same stage-clock instrumentation (t0 publish-call → t1 broker-ack → +t2 consume-callback → t3 handler-entry → t4 handler-exit → t5 ack), same HdrHistogram CSV + +`dotnet-counters` capture, same dotTrace Timeline protocol. RabbitMQ adapter + a **native twin** +on raw RabbitMQ.Client 7 (`AsyncEventingBasicConsumer`, manual acks, same prefetch). Broker: +docker-compose RabbitMQ (5672). Classic AND quorum queues in the matrix — quorum's fsync path +changes everything and RabbitMQ 4.x defaults quorum `delivery-limit=20` (poison interplay noted +in `TRANSPORT-CAPABILITY-RESEARCH-2026-07-18.md`). + +## 3. Experiment matrix + +Baseline: default endpoint (Inline, prefetch 100, confirms off, classic queue), 1Kb payloads, +steady 500/s, ~10ms simulated handler. + +| # | Experiment | Theory | Levers | +|---|---|---|---| +| RE1 | Mode sweep: Inline / Buffered / Durable(PG) | R1,R2 | `ProcessInline()`, `BufferedInMemory()`, `UseDurableInbox()` | +| RE2 | Inline scaling: ListenerCount 1/5/10 × ConsumerDispatchConcurrency 1/5/20 | R1 | `ListenerCount`, `ConfigureChannelCreation` | +| RE3 | Prefetch sweep per mode: 10/100/500/2×MDOP | R1,R6 | `PreFetchCount(ushort)` | +| RE4 | Publisher confirms: off / on / on+durable-outbox | R4 | `WolverineRabbitMqChannelOptions` | +| RE5 | Queue type: classic vs quorum vs stream-as-queue | — | `UseQuorumQueues()` etc. | +| RE6 | Back-pressure: default (1000/500) vs raised limits under 2× overload burst | R5 | `BufferingLimits` | +| RE7 | Mapper: default vs minimal interop mapper; 100Kb payloads | R3 | `UseInterop` | +| RE8 | Topic-exchange routing vs direct queue sends | R7 | routing config | +| RE9 | Durable batch-insert prototype (RO1) vs per-message | R2 | branch build | +| RE10 | Sequencing shapes (§4) | — | see below | +| RE11 | Failure-path cost: 5% handler failures → Defer/republish churn | R8 | error policy | + +Output per cell: stage p50/p95/p99 + throughput + GC/CPU; 3 reps; medians-of-percentiles. +Same measured-run archival rules as #3490. + +## 4. Sequencing / GlobalPartitioning-equivalents + +RabbitMQ has **no** broker-native sequencing surface in Wolverine today: no +single-active-consumer (`x-single-active-consumer` unsupported — settable only via raw +`Queue.Arguments`), no consistent-hash exchange, streams have no offset surface. Shapes to +benchmark: +1. **One queue + `PartitionProcessingByGroupId(slots)`** (Buffered) — consumer-side sequencing, + no topology change. Likely the cheapest recommendation. +2. **`UseShardedRabbitQueues(base, N)`** (`RabbitMqTransportExtensions.cs:503-515`) — N plain + queues, Wolverine-side hash routing; inside a `GlobalPartitioned(...)` topology the slots are + forced Durable (adds the R2 tax — quantify). +3. **Manual `x-single-active-consumer` via Arguments + ListenerCount>1** — exclusive-consumer + failover semantics without Wolverine support; measure to motivate (or kill) the capability-doc + item for a first-class SAC helper. + +## 5. Optimization backlog (gated on matrix confirmation) + +- **RO1 (R2): micro-batch the durable inbox for push transports.** Either a small + accumulate-window in `DurableReceiver.ReceivedAsync` (shared with Kafka O1 — design once, + benefit both) or consumer-side aggregation into `Envelope[]`. +- **RO2 (R6): ack watermark coalescing** — deliberate use of `multiple:true`: ack once per + N/interval from a committer-style tracker instead of per message. +- **RO3 (R3): mapper/publish allocation fixes** — kill incoming double-decode, precompute the + reserved-header set (shared with Kafka O4), investigate pooling the header dictionary, + skip `body.ToArray()` where the pipeline consumes synchronously (risky — client recycles + memory; measure first). +- **RO4 (R4): publisher-confirm windowing** — allow K outstanding confirms instead of + await-per-publish, if confirms-on proves popular. +- **RO5 (R5): back-pressure without channel teardown** — `BasicCancelAsync` (consumer cancel, + channel stays open; already what `StopAsync` does at `RabbitMqListener.cs:99-115`) without the + `DisposeAsync` channel close; resume = re-consume on the same channel. +- **RO6 (R1): ergonomics/docs** — surface `ConsumerDispatchConcurrency` per endpoint (currently + transport-wide), and a prominent "scaling consumption" docs section covering the + Inline-default + dispatch-1 reality. +- **RO7: single-active-consumer helper** (capability-doc crossover) if RE10.3 motivates it. + +## 6. Measured-wins ledger (for release notes / blog posts) + +### Measured 2026-07-19 (local rig; BEFORE = worktree @f6f125f10/jfx 2.30.0, AFTER = RO1 + mapper flag + jfx 2.30.1) + +| Cell | BEFORE | AFTER | Verdict | +|---|---|---|---| +| r-default (out-of-box, 1Kb @8/s) transit p50 | 2.385ms | 2.396ms | native parity (r-native 2.1ms); mapper flag no regression | +| r-batch-buffered (BufferedInMemory send, 100/250ms) transit p50 | 2.015ms | 1.684ms | **Rabbit sends never routed through the debounced batching channel — unaffected by the GH-3490 bug both sides** | +| r-thru-inline-1 @2000/s, 5ms handler | 178.7/s | 177.9/s | R1 confirmed: inline single-file; native single-consumer twin = 166/s → NOT Wolverine overhead. Scaling: ListenerCount(4)=720/s, buffered=1,999/s @2ms p50 | +| r-thru-durable @2000/s | 1,688/s, 14.7s backlog p50 | **1,999.9/s, 0.8ms p50** | **RO1 headline** | +| r-max-durable | 1,086/s | **3,101/s (+186%)** | RO1; remaining ceiling = per-message mark-handled (O1b, shared w/ Kafka) | +| r-max-inline | 29,638/s | 27,771/s | ~unchanged (run variance) | +| r-max-buffered | 16,441/s | 15,330/s | ~unchanged; buffered max < inline max both sides — back-pressure churn suspected (R5/RO5, NOT addressed this release) | + +Negative results: R3 mapper flag shows no ms-scale rig delta (µs-scale, matches Kafka µbench +transfer); the GH-3490 debounce never applied to Rabbit's sender path (default inline AND +buffered both unaffected — checked both directions); R5 buffered-below-inline max anomaly +recorded but unfixed (RO5 follow-up). + +Same rules as #3490 §9: rows only from archived rig runs, exact E-cell + config recorded, +before/after from the same rig version, one-liner phrased for release notes. Log negative +results below the table. + +| Optimization | PR | Scenario (RE-cell) | Metric | Before | After | Release-note one-liner | +|---|---|---|---|---|---|---| +| _(RO1 batched inbox, RO2 ack coalescing, RO3 mapper fixes, ... — rows added as measured)_ | | | | | | | + +## 7. Sequencing & exit criteria + +- **Wave 0 (now)**: transport adapter + native twin in the rig; smoke-run. +- **Wave 1 (box free)**: RE1-RE8; dotTrace on RE1-durable and RE2. + Exit: mode-by-mode overhead vs native quantified; R1/R2 confirmed or killed. +- **Wave 2**: RE10 sequencing shapes + RE9/RE11 prototypes. +- **Wave 3**: land confirmed RO items; re-run touched cells; fill the ledger. +- **Wave 4**: update `docs/guide/messaging/transports/rabbitmq/performance.md` (seeded + 2026-07-18 with qualitative guidance) with measured numbers + release-note bullets from the + ledger. + +## 8. Risks / honesty notes + +- Single-node docker broker: no cluster/mirroring effects; quorum-queue numbers on one node + understate replication cost — relative comparisons only. +- Confirms-off cells measure a fire-and-forget publish; never quote them against a native twin + running confirms-on. +- RabbitMQ full test suite is load-sensitive on CI (~10 min, chronic timeout history) — the rig + must not join CI. diff --git a/RELEASE-NOTES-DRAFT-MESSAGING-PERF.md b/RELEASE-NOTES-DRAFT-MESSAGING-PERF.md new file mode 100644 index 000000000..868d9c060 --- /dev/null +++ b/RELEASE-NOTES-DRAFT-MESSAGING-PERF.md @@ -0,0 +1,113 @@ +# Release-notes draft — messaging performance wave (GH-3490 / GH-3492 / GH-3493) + +> Draft for the next Wolverine release's notes (slots alongside the conjoined-tenancy epic and +> the 7/18 transport quick wins already queued for the same release). All numbers are from the +> GH-3490/3492 load rigs (single box, single broker, M5 Max) — relative comparisons against a +> raw-client "native twin" measured by the same instrumentation, not absolute promises. + +--- + +## Headline: sender batching no longer sits on your messages + +Wolverine's shared sender-batching channel (used by every transport when a subscriber endpoint +is Buffered or Durable) treated its `MessageBatchTimeout` as a *quiet-period debounce*: every +published message reset the flush timer, so any steady stream faster than the timeout postponed +the send until a full `MessageBatchSize` accumulated. + +With the default `(100, 250ms)` settings, a modest 8 msg/s stream measured **5.8 seconds** of +publish-to-consume p50 latency. After the fix (JasperFx 2.30.1, picked up in this release) the +timeout is the **maximum age of a batch**: the same stream measures **136ms p50 / 262ms p99**, +bounded by the configured timeout. Rate-controlled and high-throughput paths are unchanged — +full batches always flushed immediately and still do. + +Notes per transport: +- **Kafka**: applies to all Buffered/Durable subscriber routes (the default). This was the + root cause of the "Wolverine-over-Kafka is 3-12x slower than native" report in GH-3490 — + with tuned batching `(1, 1ms)` or `SendInline()`, Wolverine now measures at parity with a + raw Confluent.Kafka producer (7.6-8.9ms vs 8.7ms transit p50 on the rig). +- **RabbitMQ**: unaffected. Subscriber endpoints default to *inline* sending, which never + batched, and buffered RabbitMQ sends don't route through the debounced batching channel + either — measured at 2.0ms transit p50 before the fix and 1.7ms after with `(100, 250ms)` + batching configured. Default RabbitMQ routes already measure at native parity (2.4ms vs + 2.1ms p50 on the rig). +- Latency-sensitive, low-rate routes should still tune `MessageBatchSize`/`MessageBatchTimeout` + down — the timeout is now an honest worst-case latency floor, but it is still a floor. + +## Durable (inbox-backed) listeners: batched persistence + +Durable endpoints used to make **one database insert per message, inline on the transport's +consume loop** — the inbox write RTT was the consumption ceiling. + +- **Kafka** (`GH-3490`): durable listeners now drain up to `MaximumMessagesToReceive` + (default 100) already-fetched records per consume pass and persist them with a single + multi-VALUES insert. Measured: a 2,000 msg/s stream went from **unbounded backlog (14s+ and + climbing)** to a steady **32ms** delivery p50; maximum sustained durable throughput went from + **1,460 to 2,671 msg/s (+83%)**. +- **RabbitMQ** (`GH-3492`): durable listeners now coalesce prefetched deliveries for up to 5ms + (max-age, not debounce) into the same batched insert path, with per-message acks still issued + only after persistence. Before: **1,086 msg/s** ceiling, and a 2,000 msg/s stream fell + unboundedly behind (14.7s delivery p50 in a 2-minute window). After: the same 2,000 msg/s + stream runs at **0.8ms** delivery p50, and maximum sustained durable throughput measured + **3,101 msg/s (+186%)**. +- The batched-arrival path now applies the exact same per-envelope semantics as one-at-a-time + arrival (serializer unwrap for MassTransit-style interop, dead-lettering of unidentifiable + messages, expiry, drain-time latching) — previously these guards only ran on the + single-message path. +- The remaining durable ceiling is the per-message "mark handled" update after execution; + batching that is a known follow-up. + +## Scaling RabbitMQ consumption (docs + measured guidance) + +Out-of-the-box RabbitMQ listeners are *Inline*: one message at a time, fully processed and +acked before the next — measured at **179 msg/s** against a 2,000 msg/s load with a 5ms +handler. That is not Wolverine overhead: an equivalent single-consumer raw RabbitMQ.Client +loop measures **166 msg/s** on the same rig. It's the single-file consumption pattern. +`BufferedInMemory()` holds the same load at **2ms** p50, and `ListenerCount(4)` scales inline +consumption ~4x. The new "Performance Tuning" page in the RabbitMQ docs covers the trade-offs +(including buffered mode's ack-on-receive loss window). + +## Kafka receive/send hot path + +- The envelope mapper no longer decodes every incoming header twice: **~21% faster mapping, + ~28% less allocation per received message** (1,240ns/3,848B → 983ns/2,768B); outgoing + mapping is ~17% faster / 14% lighter. RabbitMQ's mapper gets the same incoming optimization. +- `SendInline()` Kafka senders no longer issue a blocking full-producer `Flush()` after every + send (it also blocked on every other in-flight message sharing the producer — inline sends, + broker-per-tenant sends, and liveness pings all paid it). + +## SQS: batched sends no longer silently drop rejected entries (GH-3493) + +`SendMessageBatch` responses report per-entry failures (throttling, oversize); Wolverine never +inspected them, so rejected entries were treated as sent and **silently lost**. Failed entries +are now routed back through the sender's retry machinery individually, and an exception on a +later chunk of a large batch no longer re-sends chunks SQS had already accepted. + +## Metrics and logging changes to be aware of + +- **`wolverine-execution-time` is now a `double` histogram and records sub-millisecond + executions.** Previously durations were truncated to whole milliseconds and sub-1ms samples + were silently dropped — biasing the histogram upward for fast handlers (and skewing exactly + the kind of framework-vs-native comparison GH-3490 reported). Dashboards keyed to the metric + name/unit are unaffected; the point type changes from integer to floating point. +- **The per-message "successfully processed" log now defaults to `Debug`** (was `Information`) + — a per-message Information log is a measurable tax on hot listeners. Restore the old + behavior with `opts.Policies.MessageSuccessLogLevel(LogLevel.Information)`. +- `wolverine-execution-time` still measures the handler *plus all middleware* — including time + blocked in custom middleware (semaphores, locks). If you sequence by key, prefer + `PartitionProcessingByGroupId()` / `ProcessConcurrentlyByKey()` over hand-rolled gating; the + wait then happens outside worker slots and outside the execution metric. + +## New knobs + +| API | Transport | Purpose | +|---|---|---| +| `ListenToKafkaTopic(...).MaximumMessagesToReceive(n)` | Kafka | Cap (or disable with 1) the durable consume-loop drain batch. Default 100. | +| `ListenToRabbitQueue(...).MaximumMessagesToReceive(n)` | RabbitMQ | Cap (or disable with 1) durable delivery coalescing. Default 100. | + +## For contributors + +The measurement harnesses ship in-repo (outside the solutions/CI): `src/Testing/KafkaPerfRig/` +(Kafka + RabbitMQ Wolverine/native twin rigs, stage-clock instrumentation, `cells*.sh` +experiment sweeps) and `src/Testing/Benchmarks/` (`KafkaHotPathBenchmarks`). The experiment +ledgers with every measured cell — including the negative results — live in the GH-3490 and +GH-3492 plan documents. diff --git a/RELEASE-PLAN-2026-07-13.md b/RELEASE-PLAN-2026-07-13.md new file mode 100644 index 000000000..445aab8f9 --- /dev/null +++ b/RELEASE-PLAN-2026-07-13.md @@ -0,0 +1,165 @@ +# Release plan — 2026-07-13 + +Built from a 36-hour sweep (since 2026-07-11T12:00Z) of **Marten, Polecat, JasperFx, Weasel, +Wolverine, ProductSupport, CritterWatch**, filtered to still-open items. + +**Already shipped today:** Wolverine **6.17.3** (9 PRs; live on nuget.org, release notes published). +**ProductSupport is empty** — zero open issues, zero open PRs. + +Decisions taken with Jeremy: guard **throws** at startup; Marten **9.15.1 ASAP** for #4947 alone; +**#3376 = design now, build in 6.18.0**; **#3399 batches into 6.18.0** (no fast 6.17.4). + +--- + +## Release 1 — Marten 9.15.1 (URGENT, ships first) + +**Driver: #4947 — silent data-correctness regression, broken since 9.13.0.** + +`ForTenant()` on an identity/dirty-tracked session stopped seeing tenancy-neutral (global) +documents. `LoadAsync` returns **null** for a document that is there. Reported working on 9.12.0, +broken on 9.15.0. Three releases of blast radius. + +| Item | State | +|---|---| +| **#4947** | **PR #4948 OPEN.** Root cause: `74b11a461` (PR #4807, the fix for #4801) tenant-scoped the identity map **per session** instead of **per document type** — right for conjoined docs, wrong for tenancy-neutral ones. Fix gates sharing per type: identity-mapped AND not `Conjoined` AND same `Database` instance. 5 of 6 new tests fail on unmodified master; `Bug_4801` still passes; `DocumentDbTests` 1047/0. | + +**Do not hold this for anything else.** Merge on green → bump → publish +(`on-manual-do-nuget-publish.yml --ref master`). Local-feed verification gate applies +(pack `-local.N`, verify Wolverine AND CritterWatch, then publish). + +**Not in this patch:** #4946 (`BulkInsertEventsAsync` calls `ApplyAllConfiguredChangesToDatabaseAsync()` +on **every** call — 17 events/s vs >3,000/s in erdtsieck's 512-DB store). Accepted, he is PR-ing it; +rides the *next* Marten patch rather than gating a correctness fix. + +**Confirmed not a driver:** #4920 (Guid `CompareTo` in LINQ) already shipped in **9.14.1**. + +--- + +## Release 2 — Weasel 9.16.4 (contributor-driven, not urgent) + +| Item | State | +|---|---| +| **#356** | erdtsieck. `db-apply` never releases each database's pool, so a 512-DB walk drags a tail of idle pools and then **failed a real production deploy today** on `53300`. Asked for pool-release + backoff-retry + `n/total` progress. **Accepted; he is PR-ing (1)+(2).** Note this IS implementable, unlike #3376's ask — a one-shot CLI owns its data sources. | + +**Housekeeping (done):** #353 closed as superseded by the merged #354. +**Still to do:** publish a GitHub Release entry for **9.16.3** — the package is on NuGet but the +Releases page still says 9.16.2, which will mislead anyone checking versions. + +--- + +## Release 3 — Wolverine 6.18.0 (the batch) + +Nothing ships as 6.17.4; per Jeremy, everything batches here. + +### Already merged, unreleased +| Item | What | +|---|---| +| **#3396** | `ApplyRestrictionsAsync` discarded the `StopRemoteAgent` that `EvaluateAssignmentsAsync` returned, so pausing an agent persisted a restriction and had **no immediate effect**. Plus the in-memory `Restrictions` was never refreshed, so paused *listeners* restarted themselves. Zero prior test coverage. | + +### Open PRs +| Item | What | +|---|---| +| **#3400** | **GH-3388 guard.** Refuses managed distribution + an explicit Marten daemon (`MartenDaemonModeIsSolo()` / `AddAsyncDaemon(Solo\|HotCold)`) at host start. Two coordinators competing → a **hang**, not an error. Reverses the GH-3290 "never overwrite the user's choice" contract, because what it preserved was a deadlock. **It immediately caught five of our own TestHelpers fixtures** in exactly that state — which is why the #3388 cold path went unnoticed. MartenTests 518/518. | + +### To build +| Item | Effort | What | +|---|---|---| +| **#3399** | S | Codegen emits an **invalid C# class name** for batched (`T[]`) message types when duplicate handler `TypeName`s are disambiguated → `ItemDeleted[]1177234954_...` → compile failure → **app dies at startup**. Only fires with `MultipleHandlerBehavior.Separated` + a class handling two types where one is batched, which is why existing batch tests pass. Fix ≈ sanitize the identifier. | +| **#3398** | M | `[AsParameters]`: an unparseable value in a **collection** query param silently binds `null` instead of 400. Finishes the #3372 job (which fixed only the scalar case). On a filter endpoint this **silently drops the predicate and returns an unfiltered 200** — FluentValidation can't compensate because it sees `null`. | +| **#3385** | S/M | gRPC header-identified saga: replace the opaque `StatusCode.Internal` with an actionable diagnostic; flip the characterization test. **@erikshafer greenlit for option (a)** and offered to take it. | +| **#3376** | **L** | **Owned-agent daemon scoping — the headline.** See below. | + +### #3376 — what the reporter's answer changed + +erdtsieck answered: **no `AddAsyncDaemon` anywhere near managed distribution.** So the cheap config +fix is off the table for his deployment. His measurements: + +- 937 connections across 512 DBs / 2 nodes; **~808 are daemon `COMMIT`/`ROLLBACK` per-database work**, ~116 ordinary app traffic. +- **475–497 of 512 databases hold connections from BOTH nodes.** +- **A third node added ~350 connections while fully caught up and idle.** Adding capacity *increases* connection pressure. +- His `db-apply` step **failed for lack of connections** while the cluster was idle; scaling the API *down* fixed it. + +Two constraints the design must respect (from recon, and confirmed by Jeremy): + +1. **Pool release as originally specified is not implementable.** Marten's `NpgsqlDataSource` is owned + by the tenancy's `MartenDatabase` and is **shared with ordinary application sessions** for that + tenant on that node. Disposing it on agent revocation would abort live app connections. +2. **Command processing opens connections to any tenant regardless of daemon affinity.** So daemon + scoping alone **cannot** reach `databases + overlap` — the app's own tenant traffic is a second, + independent axis. Any honest target must say so. + +**Revised direction to design (not pool release):** ownership-scoped *materialization* + true +**daemon quiesce**. Three real leaks already found, all worth fixing regardless: +- `EventStoreAgents._daemons` is **append-only** — daemons are never released. +- `JasperFxAsyncDaemon`'s per-database `System.Timers.Timer` starts in the **constructor** and is only + stopped in `Dispose()` — `StopAllAsync` doesn't touch it. +- The daemon's subscription to the database's `ShardStateTracker` is **never disposed**; a rebuilt + daemon re-subscribes and both observers stay attached. + +**Next artifact:** post the revised design on #3376 (explicitly retracting the pool-release claim), +then build. + +### Deferred +- **#3397** (adaptive connection budget) — erdtsieck **deferred it himself** behind #3376. Do not schedule. +- **#3387 / #3367** ([Entity] load profiles) — deferred by Jeremy. **The contributor still has not been told.** +- #3391, #3380, #3366, #3365, #3350, #3137, #3237 — no new signal. + +--- + +## Release 4 — CritterWatch 1.0.0-beta.4 (pins Wolverine 6.18.0) + +### Done since beta.3 +#697 (PR #703), #698 CritterWatch half (PR #704), #701 (#689 docs half), #699 flooding half +(PR #705), #610 closed with evidence (residuals → #702). + +### MUST +| Item | Effort | What | +|---|---|---| +| **#706** | M | **NEW, erdtsieck's beta.3 walkthrough.** DLQ fetch failures are **invisible on multi-DB/multi-tenant**: the pre-fanout enumeration is unguarded so one bad store aborts everything, and the per-store catches swallow-and-log — so a partial failure renders as **"No dead letter queue entries."** An operator cannot tell *empty* from *broken*. Same class #683 set out to kill. Needs a partial-failure result + "N of M databases failed to answer" banner. | +| **#699 retention** | M | PR #705 fixed only the flooding. `TimelineEntry` still has **no retention** — 671k rows, 65–85k/day for one small service. Reuse the #468/#695 partitioning/retention approach. | +| **#702** | M | The #610 residuals: `PerTenantProjectionRow` has no `storeUri` (rows fuse across stores); `ShardIdentityFromLiveAgentUri` discards the store segment (cross-store collisions resolve to the wrong store); **no test composes ancillary stores AND db-per-tenant**. | +| **#707** | S | NEW. Two nits: DLQ "Query Messages" disabled while the empty state tells you to click it; `GET /alerts/config/metrics/services/{name}` 404s for an unconfigured service. | + +### Not beta.4 +#689 second half (belt-and-braces only now — the real fix shipped in 6.17.2); #636 (remaining piece +is a fleet-driven E2E blocked on programmatic replica control); #632; #670; #347. +**#688** — owed a design sketch on the thread; no code gated on it. + +### Blocked on the reporter +**#698 stays open.** Both halves are merged, but erdtsieck **has not re-verified**, and his +`wolverine_agent_restrictions` = 0 rows is **still unexplained** — the persist path is unconditional +and our cluster tests show pause working end to end. Asked him to check the **console's** store. + +--- + +## Not releasing + +- **JasperFx 2.28.0** — **hold.** #3376 was its only driver and the hook design it was going to carry + turned out to be partly wrong. Nothing else in the sweep needs a JasperFx change. +- **Polecat** — no release. Zero activity in the window, zero open PRs. **#320** (`IEventStore.Subject` + is the DB URI, so primary + ancillary on one database are indistinguishable → CritterWatch HWM + buckets collide) is small and has a pinned skipped test; it rides the next Polecat release. + +--- + +## Sequencing + +``` +Marten 9.15.1 (#4947) ── ship FIRST, alone, ASAP +Weasel 9.16.4 (#356, contributor) ── independent; when his PR lands +Wolverine 6.18.0 ── #3396 + #3400 + #3399 + #3398 + #3385 + #3376 + │ +CritterWatch beta.4 (pins 6.18.0) ── #706 + #699-retention + #702 + #707 +``` + +## Communications + +- [ ] **Discord**: 6.17.3 draft is written and waiting on Jeremy to paste. Then 9.15.1 / 6.18.0 / beta.4. +- [ ] **#3376**: post the revised design (retract pool-release; state the two-axis reality). +- [x] **#3385**: erikshafer greenlit for option (a). +- [x] **Marten #4946 / Weasel #356**: erdtsieck's contributions accepted. +- [x] **Weasel #353**: closed as superseded. +- [ ] **Weasel 9.16.3**: publish the missing GitHub Release entry. +- [ ] **#3387/#3367**: the contributor still has not been told it is deferred — **Jeremy's call**. +- [ ] **CW #698**: nudge erdtsieck to re-verify and to check the console's store. +- [ ] **CW #688**: owed a design sketch. diff --git a/RELEASE-PLAN-2026-07-14.md b/RELEASE-PLAN-2026-07-14.md new file mode 100644 index 000000000..db72dc52a --- /dev/null +++ b/RELEASE-PLAN-2026-07-14.md @@ -0,0 +1,198 @@ +# Release plan — wave of 2026-07-14 + +Scope: every **open community issue** plus **recently-opened maintainer issues** across JasperFx, +Weasel, Marten, Polecat, Wolverine, ProductSupport, CritterWatch, ai-skills. + +ProductSupport and ai-skills have **zero open issues** — nothing to do there. + +Wolverine #3376 is deliberately excluded pending a separate design discussion. #3397 is parked +behind it (its tuning baseline changes once #3376 lands). + +--- + +## Status (live) + +**Merged:** wolverine #3409 (ASB emulator) · CritterWatch #709 (DLQ partial failure), #711 (OpenAPI docs) +**In CI, no failures:** wolverine #3411, #3412, #3417, #3418, #3419 · CritterWatch #710, #712 +**Changes requested (community):** wolverine #3407 (Kafka), #3410 (Cosmos F#) +**Awaiting erdtsieck's PRs:** weasel #356, marten #4946 +**Running:** CritterWatch #702 + +### Issues filed during this wave + +Work done here surfaced eight new issues, several more serious than what we set out to fix: + +| Issue | Why it matters | +|-------|----------------| +| **wolverine #3408** | **Security-relevant.** Reserved envelope headers spoofable through the durable inbox. **Live on shipped code** via MassTransit interop — not merely reachable through the pending Kafka PR. Fixed in #3411 | +| **wolverine #3421** | The `AddOpenApi()` freeze is only *half*-closed on hybrid hosts by 6.17.2 — the document goes from "empty" to "Wolverine routes present, every minimal-API route missing", which is **strictly more deceptive** | +| **wolverine #3414** | Cosmos saga persistence has no optimistic concurrency — blind `UpsertItemAsync` silently loses updates under concurrent messages. Pre-existing, C# path | +| **wolverine #3415** | Every Cosmos saga lands in one logical partition → 20GB / 10k RU/s ceiling for the whole app's saga state | +| **wolverine #3416** | Cosmos requires a camelCase serializer policy and nothing says so; candidate for a bootstrap guard like #3400 | +| **wolverine #3413** | Test suites pollute each other through the shared `wolverine` schema. Hidden inside it: should the durability agent throw on an unresolvable transport, or dead-letter it? That's a real production shape | +| **wolverine #3420** | OpenAPI: Marten-aggregate-bound route ids render as `string`, not `uuid`, on unconstrained routes | +| **jasperfx #513** | `isNull x` codegen forces `[]` on every F# saga, framework-wide | + +### Corrections to my own issues + +Three of my own diagnoses were **wrong**, and the investigations are worth more than the fixes: + +- **#3365** — I guessed `IntegrateWithWolverine()` ran twice. It runs once. Polecat's `AddPolecat()` had *started* registering `IEventStore` and our bridge kept doing it too. A stale code comment asserted the exact precondition that had silently changed. +- **#3380** — the `parameters: []` symptom does not reproduce at all. The thesis was right, the aim was wrong; two *different* real defects found and fixed. +- **#3350** — the SNS per-tenant tests I fingered are 0.5s for 3 tests. The real hog is SQS. And Polecat's cost is **per-class, not per-test** — sharding by test duration would have skewed badly. + +--- + +## Decisions taken (2026-07-14) + +- **6.18.0 is held open** and the maintainer backlog folds into it. It is one bigger release, not + 6.18.0-now + 6.19.0-later. +- **#3366** → ship a real `UseAzureServiceBusEmulator()` API (not a docs-only fix). +- **#3350** → split the slow suites. Do not just raise `timeout-minutes`. +- **CritterWatch #77 / #74** (blue/green) → leave for now. Revisit after 1.0. + +--- + +## Wave 1 — release mechanics that do not depend on 6.18.0 (today) + +| Step | Detail | +|------|--------| +| 1.1 | Weasel: publish the missing **9.16.3** GitHub Release entry (package is live; page stops at 9.16.2) | +| 1.2 | Discord: 6.17.3 (drafted, unposted) and Marten 9.15.1 | + +--- + +## Wave 2 — Wolverine 6.18.0 (held open; ships when everything below is in) + +### Already merged to `main` + +- #3396 — `ApplyRestrictionsAsync` never dispatched the commands it computed (fixes CritterWatch #698) +- #3400 — refuse a competing Marten daemon under managed distribution (GH-3388) +- #3401 — bump Marten to 9.15.1 +- #3403 — actionable diagnostic for a header-identified saga over gRPC (GH-3385) +- #3404 — `[AsParameters]`: reject unparseable values in collection query parameters (GH-3398) +- #3406 — fix invalid generated class name for batched (array) message types (GH-3399) *(startup-fatal)* + +### Still to build + +Sequence **#3350 first** — it pays for itself across everything after it. + +| Issue | Work | +|-------|------| +| **#3350** | **CI, do this first.** Split the AWS suite into parallel SNS/SQS jobs and shard CIPolecat, then re-enable **CIAWS** (currently commented out of the matrix in `.github/workflows/tests.yml`). Attack the wall-clock so the 20m cap stays a real signal rather than raising it into meaninglessness | +| **#3365** | Polecat primary `IEventStore` bridge registers twice — `GetServices()` returns the same `DocumentStore` instance twice. Fix with `TryAddEnumerable`/dedupe. CritterWatch's `EventProgressionPoller` currently polls the primary store twice per pass | +| **#3366** | Promote a real `UseAzureServiceBusEmulator(...)` into `WolverineFx.AzureServiceBus` — sets `ManagementConnectionString`, standard emulator defaults, returns the config for chaining. Destructive delete-all-objects cleanup stays **opt-in**. Then rewrite `index.md`, `session-identifiers.md`, `conventional-routing.md` around the real API | +| **#3380** | OpenAPI: route parameters bound only by a compound-handler `LoadAsync`/`Before` are missing from the operation (`parameters: []`). Minimum bar: every `RoutePattern.Parameters` entry declared as a required path parameter, typed from whichever frame binds it. **Build the missing OpenAPI shape-test harness as part of this** — its absence is why this class of omission keeps shipping (same finding as the #3135 audit) | +| **#3391** | RabbitMQ: (a) regression test pinning the `ConnectionMonitor` tracking invariant across a callback-exception restart; (b) a successful eager restart of a listener never re-declares/`BasicConsumeAsync` — open channel, zero consumers, `State = Connected`. Revisit the #3137 quarantined circuit-breaker tests in the same pass | +| **#3408** | **Security-relevant, found reviewing #3407.** `EnvelopeSerializer.writeHeaders` appends every `env.Headers` entry unfiltered *after* the typed props, and the reader parses reserved keys back into typed properties — so a `Headers` entry under a reserved key (`tenant-id`, `saga-id`, `id`, `message-type`) **silently overwrites the real property** on any durable round trip. Inert in memory, live the moment it crosses the inbox/outbox. Filter reserved keys (skip, don't throw). Prerequisite for taking #3407 | +| **#3407** | Merge the Kafka community PR — **changes requested**, gated on #3408 (see Wave 3) | +| **#3410** | Review + merge the Cosmos F# saga codegen community PR (thechucklingatom, see Wave 3) | + +### Ship + +1. Bump `Directory.Build.props` → `6.18.0` +2. `dotnet build wolverine.slnx -c Release` (FULL solution, not `_slim`) +3. Tag `V6.18.0`, run `publish_nugets.yml`, poll the nuget flat container +4. GitHub release notes calling out every issue + PR +5. Discord announcement +6. Ask erdtsieck to re-verify **CritterWatch #698** on 6.18.0, then close it + +--- + +## Wave 3 — community PRs (gated on contributors, 1–3 days) + +We are not writing this code. We are reviewing and shipping it. + +### 3.1 Weasel #356 — `db-apply` connection discipline → **Weasel 9.16.4** +erdtsieck PR-ing. Accepted asks: release each database's `NpgsqlDataSource` after its apply; +bounded backoff-retry on `53300`; per-database `n/total` progress logging. + +Cleanly implementable *because* `db-apply` is a one-shot CLI that owns its data sources — the exact +property the daemon side lacks (which is why #3376 is hard). + +### 3.2 Marten #4946 — `BulkInsertEventsAsync` schema apply per batch → **Marten 9.15.2** +erdtsieck PR-ing. The batch overload opens with an unconditional +`ApplyAllConfiguredChangesToDatabaseAsync()`; the streaming overload has no such call. Direction +given: make it **once-per-database**, not per-call (floor: skip entirely under `AutoCreate.None`). + +Not a 9.15.0 regression — rides the patch, does not gate it. + +### 3.3 Wolverine #3407 — Kafka timestamps + record headers (jakub-petrylak-onerail) +**Reviewed 2026-07-14 — changes requested.** Direction is right (a raw-JSON listener has no Wolverine +metadata on the wire, so the record timestamp and record headers are the only source), and it is +correctly scoped to `JsonOnlyMapper`, leaving the Wolverine↔Wolverine path alone. Two findings: + +1. **Blocking — unfiltered header copy.** Kafka header keys are chosen by the producer, which on a + raw-JSON topic is by definition not Wolverine and not necessarily trusted. The copy loop does not + exclude `EnvelopeConstants` reserved names, so an external `tenant-id` / `saga-id` / `id` lands in + `envelope.Headers` and then gets promoted into the typed property on the first durable round trip + (see **#3408**). The PR's own example header is `tenant-id` — the trap in miniature. +2. **Vacuous test.** `stamps_envelope_sent_at_from_the_kafka_record_timestamp` asserts + `SentAt > now - 5min`, but `Envelope.SentAt` is initialized to `DateTimeOffset.UtcNow` at + construction (`Envelope.cs:248`) — it passes on `main` today, with or without the change. Needs a + raw producer writing an explicit past `CreateTime` and an exact-equality assertion. + +Ships **in Wolverine 6.18.0** (Wave 2), after #3408 lands. + +### 3.4 Wolverine #3410 — Cosmos F# saga codegen (thechucklingatom) +Adds `CosmosDbPersistenceFrameProvider` saga frames + a new `LoadDocumentFrame`, with an F# sample and +a checked-in `Generated.fs`. Under review. Things that decide it: whether the Cosmos point-read supplies +a correct **partition key** for saga identity (wrong = cross-partition query or a silent miss), whether +it diverges from the Marten/EF Core/RavenDb `IPersistenceFrameProvider` shape (`MarkCompleted` → delete, +optimistic concurrency), and whether the F#/Cosmos test projects are in `wolverine.slnx` at all — if they +aren't, the PR has no CI coverage, which is its own finding. + +Ships **in Wolverine 6.18.0** (Wave 2) if it holds up. + +**Pin cascade:** Weasel 9.16.4 → Marten 9.15.2 → Wolverine + CritterWatch pin bumps. If Marten +9.15.2 lands before 6.18.0 ships, fold its pin bump into 6.18.0 rather than cutting a follow-up. + +--- + +## Wave 4 — CritterWatch beta.4 + +Pins Wolverine 6.18.0. + +| Issue | Priority | Work | +|-------|----------|------| +| **#706** | **MUST** | DLQ fetch failures are invisible on multi-database/multi-tenant persistence: `dlq-operation` throws ~1/hr yet the page renders "No dead letter queue entries" with no partial-failure indication. A silent-empty on a partial failure is the worst possible rendering | +| **#699** | HIGH | Retention half of the "Agent started" flood (671k rows, 3.5k/hr). Emit half already fixed (CW #705) — this is the retention/compaction policy for the existing rows | +| **#707** | MED | beta.3 nits: DLQ "Query Messages" disabled contradicts its own empty state; metrics-overrides GET 404s for unconfigured services (console noise) | +| **#702** | MED | Store-axis gaps in the tenant-grouped view + agent-URI resolution (follow-ups from the #610 audit) | +| **#689** | LOW | Docs/release note: monitored hosts using `AddOpenApi()` need the wolverine#3371 fix | +| **#698** | — | Close after erdtsieck re-verifies on 6.18.0 | + +--- + +## Wave 5 — design track (no code yet) + +These need a decision or a written design before they can be scheduled. + +- **wolverine #3376** — daemon connection scoping. *Separate discussion with Jeremy.* Known blockers: + Marten's `NpgsqlDataSource` is owned by the tenancy's `MartenDatabase` and shared with app sessions, + and command processing opens connections to any tenant regardless of daemon affinity — so daemon + scoping alone cannot reach the target. Three leaks already identified: + `EventStoreAgents._daemons` is append-only; `JasperFxAsyncDaemon`'s per-DB `System.Timers.Timer` + starts in the ctor and only stops in `Dispose()`; the `ShardStateTracker` subscription is never disposed. +- **wolverine #3397** — adaptive connection budget. Parked behind #3376 (erdtsieck himself said the + tuning baseline changes once owned-agent scoping lands). Post that as the holding position. +- **CritterWatch #688** — per-tenant projection views at hundreds of tenants. Needs decomposition into + roll-up rows / problem-tenants filter / tenant pivot / alert coalescing before it is schedulable. +- **CritterWatch #77 + #74** — erdtsieck's blue/green deployment asks. **Decided 2026-07-14: leave for + now**, revisit after 1.0. The wave stays focused on the live 512-tenant production issues. + +--- + +## Explicitly out of scope this wave + +Not gaps — deliberate omissions. + +- CritterWatch's ~35 June-2026 epics (Event Modeling, Workflow Visualization, Explorer IA, …) +- Marten #4685 / #4682 / #4684 — the rebuild-perf epic (two draft PRs already open) +- JasperFx #480 / #459 / #435 / #430 +- Polecat #180, #318, #320 — real, but they trail the Wolverine/Marten work they depend on +- Marten #4944 — sharded-tenancy `pg_inherits` partition sweep +- JasperFx #503 — tenant-scoped `IEventStore` explorer read overloads + +(#318, #320, #4944 and #503 are all mine and recent; they're deferred rather than dropped — they +belong to the CritterWatch multi-store/multi-tenant explorer arc, which is post-beta.4.) diff --git a/SQS-PERF-DEEP-DIVE-PLAN.md b/SQS-PERF-DEEP-DIVE-PLAN.md new file mode 100644 index 000000000..212d200df --- /dev/null +++ b/SQS-PERF-DEEP-DIVE-PLAN.md @@ -0,0 +1,164 @@ +# Amazon SQS Performance Deep-Dive Plan (2026-07-18) + +Goal: measure Wolverine-over-SQS overhead vs the raw AWS SDK across endpoint modes, fix the +bug-shaped findings uncovered during research, produce throughput-tuning guidance, and land +confirmed optimizations. Companion/umbrella: `KAFKA-PERF-DEEP-DIVE-PLAN.md` (wolverine#3490) — +shared metrics semantics, resolution-chain costs, BatchedSender mechanics, and back-pressure +behavior live there and are not restated. + +All file:line cites verified against `main` @ 6.20.0. + +--- + +## 0. Transport-specific facts that frame everything + +- **SQS is the good citizen on receive**: one `ReceiveMessageAsync` returns up to + `MaxNumberOfMessages` = **10** (`AmazonSqsQueue.cs:107`) with long polling ON by default + (`WaitTimeSeconds` = 5, `:100`), and the listener hands the receiver a real **`Envelope[]`** + (`SqsListener.cs:112-115`) → durable mode DOES use the batched multi-VALUES inbox insert + (`DurableReceiver.cs:608-668`). The Kafka/RabbitMQ per-message-insert gap does not exist here. +- **…but per-message on delete**: completion = one `DeleteMessageAsync` HTTP round trip per + message (`SqsListener.cs:191-194`); `DeleteMessageBatch` is **never used** (zero hits + repo-wide). 10 delete RTTs per 1 receive RTT, flowing through a sequential RetryBlock. This + is SQS's analogue of Kafka's per-message flush — the presumptive #1 ceiling. +- **Send batching is real but throttled by defaults**: `SqsSenderProtocol` chunks by 10 and uses + `SendMessageBatchAsync` (`SqsSenderProtocol.cs:38-46`); `MessageBatchSize` is overridden to 10 + (`AmazonSqsQueue.cs:44`) but `MessageBatchMaxDegreeOfParallelism` default **1** + (`Endpoint.cs:281`) caps sustained sends at ~10 msgs per SQS RTT per endpoint, behind the + 250ms debounce (#3490 T1). +- **Correctness bug, fix regardless of perf**: `SendMessageBatchResponse.Failed` is never + inspected — per-entry failures (throttling, oversize) are silently marked successful + (`SqsSenderProtocol.cs:42-48`; the purpose-built `OutgoingSqsBatch.TryGetEnvelope` at + `:119-122` is dead code). Silent message loss under throttling. +- **Wire format**: default mapper embeds the whole serialized envelope **base64** in the body + (`ISqsEnvelopeMapper.cs:31-33`) — ~33% inflation against the 256KB limit + 3 large + allocations per message per direction. Only one message attribute is used, so the SQS + 10-attribute cap is a non-issue. +- **No visibility-timeout renewal**: no `ChangeMessageVisibility` anywhere; `VisibilityTimeout` + fixed at receive (default 120s, `AmazonSqsQueue.cs:29,81-92`). Handlers (or buffered backlog + dwell) past 120s → redelivery; durable dedups via the inbox, buffered already deleted + (delete-on-receive, `BufferedReceiver.cs:219-228` — the #3137 loss window). +- **Native `DelaySeconds` scheduling SHIPPED** (#3472): `IConditionalNativeScheduling`, ≤900s on + standard queues (`SqsSenderProtocol.cs:11,28-31`, `AmazonSqsQueue.cs:259-272`) — the + capability-research doc's "DelaySeconds unused" note is stale; update it. +- Requeue/defer = delete + un-batched inline re-send (2 API calls, `SqsListener.cs:47-55`); + count-only batch chunking with no byte-size accounting (whole-request bounce risk on large + batches); no SQS-specific throttle handling beyond the generic 100ms×n backoff (cap 1s). + +## 1. Theories of overhead, ranked + +- **S1 (HIGH): per-message delete round trips.** Prediction: receive-side throughput ceiling + ≈ concurrency-limited delete RTTs, not receive RTTs; batching deletes (SO1) is the biggest + win. LocalStack RTTs (~1-3ms) will understate this badly vs real SQS (~5-15ms + TLS) — + see harness note. +- **S2 (HIGH): send path = 10-per-RTT × 1 in-flight × 250ms debounce.** Prediction: raising + `MessageBatchMaxDegreeOfParallelism` is the single cheapest user-side send-throughput lever; + tuning guidance must lead with it. +- **S3 (MEDIUM): base64 + double serialization** — CPU/alloc cost and wire inflation; matters + most at 100Kb+ payloads where inflation forces claim-check territory. +- **S4 (MEDIUM): poll-loop shape** — single poller per endpoint (`ListenerCount` default 1), + 250ms idle sleep, `WaitTimeSeconds`=5. Prediction: `ListenerCount` scaling is near-linear + until delete RTTs saturate; longer wait (20s) cuts empty-receive costs at low traffic. +- **S5 (MEDIUM): visibility-timeout interplay** — back-pressure stop + 120s reappearance, + buffered delete-on-receive, no renewal for long handlers. Mostly a correctness/duplicates + story; measure duplicate rates under overload, not just latency. +- **S6 (LOW-MEDIUM): FIFO throughput** — FIFO caps (300 tps/group, 3000 batched without + high-throughput mode), no high-throughput-FIFO helper (`FifoThroughputLimit` unexposed; + reachable only via raw `ConfigureQueueCreation`), null-GroupId FIFO sends rejected with no + fallback. Matters only for FIFO users but the sequencing story (§4) runs through it. + +## 2. Harness + +Same `TransportPerfRig` adapter pattern as the Kafka/RabbitMQ plans; native twin on raw +`AmazonSQSClient` (`ReceiveMessageAsync` batch 10 + `DeleteMessageBatchAsync` — twin should use +the *optimal* native pattern, since that's the ceiling we're chasing). + +**Broker fidelity is THE risk for SQS**: LocalStack (compose port 4566) has toy latencies and no +real throttling. Protocol: develop + smoke on LocalStack, but **run all measured cells against a +real SQS account** (dedicated queues, same region as the box's egress; record region + observed +base RTT per run). Budget note: SQS API calls are ~$0.40-0.50/million requests — a full matrix +run is dollars, not real money; batching experiments literally measure the cost savings too +(delete batching cuts the AWS bill 10×, a release-note-friendly angle). + +## 3. Experiment matrix + +Baseline: standard queue, Buffered, defaults (10/5s receive, 120s visibility, batch-send 10/250ms/1). + +| # | Experiment | Theory | Levers | +|---|---|---|---| +| QE1 | Mode sweep: Buffered / Durable(PG) / Inline | S1,S5 | endpoint mode | +| QE2 | Delete batching prototype (SO1) vs per-message | S1 | branch build | +| QE3 | Send: MessageBatchMaxDegreeOfParallelism 1/4/16 × MessageBatchTimeout 250/50/10ms | S2 | endpoint config | +| QE4 | Poller scaling: ListenerCount 1/4/8 × WaitTimeSeconds 1/5/20 | S4 | listener config | +| QE5 | Payload: 1Kb/100Kb/200Kb (near-limit) × default vs RawJson mapper | S3 | `ISqsEnvelopeMapper` | +| QE6 | Overload burst 2×: duplicate-rate + back-pressure + visibility interplay | S5 | `BufferingLimits`, visibility | +| QE7 | FIFO: groups sweep (10/100/1000 groups) × with/without high-throughput attrs | S6 | raw queue attributes | +| QE8 | Batch-send failure injection (throttle simulation): loss measurement pre/post SO2 fix | S4-bug | branch build | +| QE9 | Sequencing shapes (§4) | — | see below | + +Output/archival rules identical to #3490. + +## 4. Sequencing / GlobalPartitioning-equivalents + +SQS has a real broker-native primitive: **FIFO message groups** (sequential per `MessageGroupId`, +parallel across groups; Wolverine already maps `Envelope.GroupId` → `MessageGroupId`, +`ISqsEnvelopeMapper.cs:24`). But nothing consumer-side respects group ordering across a +10-message receive array today — buffered/durable execute the array concurrently. Shapes to +benchmark: +1. **FIFO groups + `PartitionProcessingByGroupId(slots)`** — broker ordering + consumer-side + per-group serialization. The natural GP-equivalent; verify end-to-end ordering under + `MaxNumberOfMessages`=10 and document the required pairing (FIFO alone is NOT enough). +2. **Standard queue + `PartitionProcessingByGroupId`** — consumer-only sequencing, no FIFO tax. +3. **`UseShardedAmazonSqsQueues(base, N)`** (`AmazonSqsTransportExtensions.cs:231-243`) — N + standard queues + Wolverine hash routing (forced Durable inside a `GlobalPartitioned` + topology — quantify that tax). +4. **Fair queues** (`EnableFairQueueMessageGroups`) — tenant fairness only, no ordering; include + one cell to characterize its cost since we're ahead of every competitor on it. +Deliverable: ordering-guarantee × throughput × cost decision table for the docs. + +## 5. Optimization backlog (gated on matrix confirmation) + +- **SO1 (S1): `DeleteMessageBatchAsync` coalescing** — accumulate receipt handles (N=10 or + T=50-100ms window) behind the complete block; also batches the requeue-path deletes. Cuts + API calls (and AWS spend) ~10× on the receive side. +- **SO2 (S4-bug, do first regardless): inspect `SendMessageBatchResponse.Failed`** and route + per-entry failures through `TryGetEnvelope` → sender callback retry. Correctness fix; + release-note headline material. +- **SO3: size-aware batch chunking** (respect the 256KB/1MiB request cap, count + bytes). +- **SO4 (S2): raise `MessageBatchMaxDegreeOfParallelism` default for SQS** (or at least loud + docs); consider trimming the debounce for SQS where batches are capped at 10 anyway. +- **SO5 (S5): `ChangeMessageVisibility` renewal** for in-flight messages (durable/buffered + backlog + long handlers), or at minimum a documented BufferingLimits-vs-visibility sizing rule. +- **SO6 (S6): high-throughput FIFO helper** (`FifoThroughputLimit`/`DeduplicationScope` + first-class) + explicit error on null-GroupId FIFO sends. +- **SO7 (S3): claim-check / payload offloading story** for >256KB (capability-doc crossover; + SQS max is now 1MiB — verify SDK support and update docs either way). + +## 6. Measured-wins ledger (for release notes / blog posts) + +Same rules as #3490 §9: rows only from archived measured runs (real SQS, region recorded), +before/after on the same rig version, release-note phrasing. **Also record API-call counts per +1k messages** — for SQS, "10× fewer AWS API calls" is a cost win worth quoting alongside +latency. Log negative results below the table. + +| Optimization | PR | Scenario (QE-cell) | Metric | Before | After | Release-note one-liner | +|---|---|---|---|---|---|---| +| _(SO1 delete batching, SO2 batch-failure handling, SO4 send parallelism, ... — rows added as measured)_ | | | | | | | + +## 7. Sequencing & exit criteria + +- **Wave 0 (now)**: rig adapter + native twin; LocalStack smoke; SO2 correctness fix can land + immediately (test via QE8 failure injection, no perf box needed). +- **Wave 1 (box + real SQS)**: QE1-QE5 baseline sweeps; dotTrace on QE1-durable. + Exit: delete-RTT ceiling quantified; S1/S2 confirmed or killed. +- **Wave 2**: QE6-QE9; sequencing decision table. +- **Wave 3**: SO1/SO3/SO4 landed + re-measured; ledger filled. +- **Wave 4**: update `docs/guide/messaging/transports/sqs/performance.md` (seeded 2026-07-18 + with qualitative guidance) with measured numbers + release notes from the ledger. + +## 8. Risks / honesty notes + +- LocalStack numbers are NOT publishable — real-SQS runs only for the ledger. +- Real-SQS latency varies by region/network; record base RTT per run and prefer ratios. +- FIFO cells must state group-count assumptions; FIFO throughput is per-group. +- The rig's AWS credentials/queues must be sandboxed (dedicated prefix, auto-teardown). diff --git a/TRANSPORT-CAPABILITY-RESEARCH-2026-07-18.md b/TRANSPORT-CAPABILITY-RESEARCH-2026-07-18.md new file mode 100644 index 000000000..e6eb3636e --- /dev/null +++ b/TRANSPORT-CAPABILITY-RESEARCH-2026-07-18.md @@ -0,0 +1,283 @@ +# Wolverine Transport Capability Research (2026-07-18) + +Gap analysis: what each underlying broker offers that Wolverine doesn't exploit, what competitor +frameworks expose that Wolverine lacks, and a ranked candidate list. Companion doc: +`GLOBAL-PARTITIONING-ROLLOUT-PLAN.md`. + +Method: codebase inventory of every transport's actual surface (file refs verified), plus +web research on broker capabilities (2025–2026 state, official docs) and ~20 competitor +frameworks (.NET, JVM, Go, Python, Elixir). + +## Strategic context + +- **MassTransit v9 went commercial** (~$400–1,200/mo); v8 stays OSS but maintenance-only through + ~end 2026, then EOL. The .NET competitor with the largest feature overlap is exiting free OSS — + gaps closed in the next year land exactly when its users re-evaluate. +- **NServiceBus** has no Kafka transport, no GCP Pub/Sub transport, and explicitly does not + support ASB sessions. Its moat is ops tooling (ServicePulse redrive UX) — a CritterWatch target + list, not a framework gap. +- **Wolverine's uncontested lanes**: GCP Pub/Sub (neither MT nor NSB has one), Pulsar (no major + .NET framework has one), NATS JetStream (only SlimMessageBus core-NATS + an embryonic Rebus + port), Redis Streams (SlimMessageBus is at-most-once lists/pub-sub only). Deepening these + extends leads nobody contests. + +--- + +## Per-transport gap analysis + +### RabbitMQ + +Already strong: queue types classic/quorum/stream, full x-arguments, native DLX + recovery +bridge, MT/NSB interop, vhost-per-tenant, conventional routing. + +Unexploited broker capabilities: +1. **Streams as streams** — queue-type `stream` is declarable but there is no offset/replay + surface (`x-stream-offset` first/last/next/timestamp), no `RabbitMQ.Stream.Client`, and no + **super streams + single-active-consumer** (Rabbit's Kafka-style partitioned ordered + consumption). Spring AMQP has all of this; **no .NET framework does** → differentiator. + Also stream filtering: Bloom (3.13), AMQP property filters (4.1), SQL filters (4.2, AMQP 1.0 + only — unreachable from the current AMQP 0.9.1 client). +2. **Poison-handling collision**: RabbitMQ 4.0 defaults quorum `delivery-limit=20`; broker + dead-letters/drops before Wolverine's error policies finish. NServiceBus explicitly manages + this (their issue #1550). Wolverine needs a deliberate ownership decision + docs. +3. **Delayed delivery**: the delayed-message-exchange plugin (MassTransit's scheduler basis) is + **unmaintained/dead** (Mnesia removed in 4.3). Options: NSB-style TTL+DLX delay-level + topology (unbounded native delay, survives without a DB), or 4.3's native quorum delayed + retry (retry backoff only). Wolverine's DB scheduler stays the durable default; a + broker-native option is competitive parity. +4. Smaller: publisher confirms default **off** (`WolverineRabbitMqChannelOptions`); no priority + queue helper (`x-max-priority`); no single-active-consumer helper; no consistent-hash + exchange support; headers-exchange has no binding-argument fluent API; direct-reply-to unused + for request/reply. + +### Kafka + +Already strong: retry topics with tiered delays (`MoveToKafkaRetryTopic`), commit modes, native +DLQ topic, Schema Registry Avro/JSON, static membership, cooperative sticky, broker-per-tenant, +`ProcessConcurrentlyByKey`. + +Unexploited: +1. **Transactions / exactly-once** (consume-transform-produce, offsets in producer txn) — table + stakes in Spring Kafka (EOSMode.V2), SmallRye, Silverback. Explicitly absent + (`KafkaTransportExpression.cs:140`). Related, arguably better fit for Wolverine: + **offsets-committed-in-application-DB** (Silverback/SmallRye checkpoint pattern) — Wolverine + already owns a DB envelope store; committing offsets transactionally with app data gives + effective exactly-once without Kafka transactions. +2. **KIP-848 next-gen rebalance** — available today in Confluent.Kafka 2.12+ via `GroupProtocol` + config; cheap to expose/document/test. +3. **Share groups (KIP-932, GA in Kafka 4.2)** — per-message ack/nack/reject queue semantics. + Spring Kafka 4.1 ships it first-class. **Blocked for .NET until Confluent ships share + consumers (~H2 2026)** — track librdkafka; be first in .NET when it lands. +4. **Replay/seek surface** — no `seekToTimestamp`/offset-reset API on listeners (Spring's + `ConsumerSeekAware`); tiered storage (3.9 GA) makes replay-from-history mainstream. +5. Smaller: Protobuf serdes missing (Avro/JSON only); no manual partition `Assign`; + exception-type-routed DLTs (Spring Kafka 3.2) on top of the existing retry-topic machinery. + +### Azure Service Bus + +Already strong: sessions (basic), native scheduled enqueue, full CreateQueue/Subscription/Rule +options incl. SQL filters + reconciliation, native DLQ + recovery listener, sharded queues, +named brokers, MT/NSB interop. + +Unexploited: +1. **PrefetchCount** — not surfaced at all; cheapest throughput win in the transport. +2. **Deferral** (`DeferMessageAsync` + receive-by-sequence) — native "park until ready"; + natural fit for out-of-order saga messages. No framework exposes it. +3. **Richer session support** — MT has `MaxConcurrentSessions`/`MaxConcurrentCallsPerSession`/ + session-id formatters; NSB has nothing → surface richly and it's a competitive stick. + **Session state** (broker-hosted per-key blob) is unexploited by everyone but MT's saga repo. +4. **Cross-entity transactions** (send-via) — atomic settle+send, the broker-side outbox; NSB + has it (`SendsAtomicWithReceive`), Wolverine doesn't. +5. **Scheduled-message cancellation** — ASB is the only major broker with native cancel; + Wolverine maps `ScheduledEnqueueTime` but never the returned sequence number. +6. Untouched by anyone (greenfield): auto-forwarding topologies, Event Grid-triggered + scale-to-zero listeners, Geo-Replication (data, GA 2025) awareness, dedup + (`RequiresDuplicateDetection` + `Envelope.DeduplicationId`). +7. **Azure Event Hubs** — no Wolverine transport; MT covers it as a rider. Candidate new + transport (partitions/consumer groups/checkpointing → pairs with the offsets-in-DB pattern). + +### Amazon SQS / SNS + +Already strong (SQS): FIFO groups + dedup, full attribute passthrough, native DLQ + recovery, +**fair queues already exposed** (`EnableFairQueueMessageGroups` — ahead of every competitor), +broker-per-tenant, CloudEvents/MT/NSB mappers. + +Unexploited: +1. **Native `DelaySeconds`** (≤15 min) unused — `SupportsNativeScheduledSend => false`. Also the + NSB pattern for **unbounded** native delay (FIFO staging queue + re-loop), and **EventBridge + Scheduler** as a cancellable, timezone-aware scheduled-send backend (greenfield — nobody + integrates it). +2. **Programmatic DLQ redrive** — `StartMessageMoveTask`/`ListMessageMoveTasks` as a one-call + "replay dead letters" verb (progress + cancel). No framework wraps it; CritterWatch synergy. +3. **Payload limits stale**: SQS max is now 1 MiB (Aug 2025). No S3 claim-check story (no + official .NET extended client — open niche, see cross-cutting #3). +4. **SNS**: payload-based filter policies (`FilterPolicyScope=MessageBody`) unclaimed by any + framework; non-SQS subscription protocols (lambda, http, firehose…) unimplemented + (`NotImplementedException`); `PurgeAsync`/`GetAttributesAsync` TODOs. +5. **EventBridge** (bus/rules/pipes/archive+replay) — candidate adjacent transport, uncontested. + +### GCP Pub/Sub + +Already strong: exactly-once flag, ordering keys (GroupId→OrderingKey), native DLQ + retry +policy, filters, flow control, project-per-tenant. Uncontested lane — deepen it. + +Unexploited: **snapshots + seek** (replay/rewind, and seek-to-now as a proper `PurgeAsync` — +current purge pulls/acks max 50 msgs); push subscriptions (`PushConfig`); topic schemas +(Avro/proto); BigQuery/Cloud Storage export subscriptions (provision from Wolverine config); +Single Message Transforms (GA 2025); `GetAttributesAsync` returns empty. + +### NATS + +Already strong: core + JetStream endpoints, full StreamConfiguration provisioning, DeliverPolicy, +queue groups, native scheduled send gated on server 2.12+, DLQ subjects, tenancy via connection +or subject prefix. + +Unexploited (biggest untapped pool of any transport): +1. **JetStream KV store** — OCC/CAS via revision numbers → saga storage, dedup tables, + distributed locks, leader election (Rebus.Nats prior art); **Object store** → claim-check + backend. NATS.Net fully supports both; Wolverine uses neither. +2. **2.11 pull-consumer priority groups** — `pinned_client` = broker-native exclusive consumer + with automatic failover (relevant to GlobalPartitioning native mode and to Wolverine's + exclusive-listener choreography); overflow groups = standby consumers. +3. **Consumer pausing** (`PauseUntil`) — maps directly onto Wolverine's pause-listener ops. +4. **`Nats-Msg-Id` dedup + double-ack** — broker-side idempotent publish keyed by envelope ID + (Watermill's exactly-once recipe); free outbox-style dedup. +5. **2.12 atomic batch publish** — all-or-nothing multi-message publish; a broker-side + transactional outbox primitive. **2.14 recurring cron schedules** (`@every`/crontab in + `Nats-Schedule`) — broker-native recurring messages. +6. **Deterministic subject mapping** `{{partition(N,...)}}` + Orbit pcgroups — native + partitioning (GlobalPartitioning v2 candidate). +7. Smaller: AckPolicy fixed at Explicit (AckAll is a cheap batch-ack win for ordered inline + listeners); mirrors/sources (tenant fan-in, DR); accounts-based tenancy; capture + `MAX_DELIVERIES` advisories into the DLQ stream; micro/service API for handler observability. + +### Redis Streams + +Already strong: consumer groups, XAUTOCLAIM recovery loop, native DLQ stream, sorted-set +scheduled/retry, broker-per-tenant. Already the best .NET Redis transport (SlimMessageBus is +at-most-once lists/pub-sub); Watermill is the cross-language model and Wolverine matches its +core loop. + +Unexploited: **Redis 8.2 `XACKDEL`/`XDELEX`** (atomic ack+delete work-queue semantics, kills the +ack-then-trim race — needs raw `Execute` until SE.Redis surfaces it); `MAXLEN`/`MINID` trimming +policy (unbounded stream growth today; purge = full `KeyDelete`); consumer-group lag metrics +(`XINFO GROUPS`) for CritterWatch; sharded pub/sub or keyspace notifications as a low-latency +wakeup to shorten the polling gap (SE.Redis forbids blocking reads by design — polling is +structural; a dedicated-connection option is the escape hatch). + +### Pulsar + +Already strong: all four subscription types incl. KeyShared, native DLQ + retry-letter topics +with tiered delays, native redelivery, ack strategies, schemas + producer dedup, regex +subscriptions, hot-tail readers. No major .NET competitor has a Pulsar transport at all. + +Unexploited: +1. **Native delayed delivery (`DeliverAt`/`DeliverAfter`)** — DotPulsar supports it; + Wolverine doesn't map it to `SupportsNativeScheduledSend`. Cheapest headline win in the + whole analysis: Pulsar is *the* broker with true arbitrary-timestamp scheduling. +2. Client-blocked by DotPulsar (broker-native but unreachable): transactions, negative acks + + backoff, batch-index ack, chunking, TableView. Options: contribute upstream, emulate + (retry-letter + delay already emulates nack-with-backoff), or evaluate the F# + `pulsar-client-dotnet` which covers all of these. +3. Smaller: partitioned-topic admin (create/expand partitions); Pulsar-native tenant/namespace + mapping for Wolverine tenancy (currently tenants = separate clusters only). + +### MQTT + +Current state: v5 protocol forced, QoS + retain per topic, managed client passthrough, JWT +re-auth, broker-per-tenant. Benchmark competitor: Silverback (outbox, dedup/exactly-once, +chunking, batching, in-memory mock). + +Unexploited (all MQTT 5, all supported by MQTTnet): **user properties** = real headers (today +interop requires envelope serialization); **response topic + correlation data** = protocol-level +request/reply mapping onto Wolverine's reply semantics (Spring Integration precedent); +**message expiry** (→ `DeliverBy`); **shared subscriptions** `$share/{group}/{topic}` = +competing consumers (MQTTnet has no first-class API but raw filter subscription works — +verify per broker); session-expiry control; LWT helper (presence/AgentState signal to +CritterWatch); EMQX `$delayed/{seconds}/` native delayed publish as an opt-in scheduling +backend; HiveMQ `$dead/$dropped` feeds as a DLQ source. + +### PostgreSQL / SQL Server queues + +Current state: durable named queues + scheduled tables, sticky per-node listeners, FIFO, +MT+NSB interop transports (Postgres), NSB only (SQL Server), DB-per-tenant. + +Unexploited: +1. **Postgres `LISTEN`/`NOTIFY`** — dequeue is poll-only; latency floor = `PollingInterval`. + MassTransit's SQL transport (their strategic v9 bet) uses LISTEN/NOTIFY. Matching this makes + the "Postgres is my broker" story fully competitive. SQL Server analogue: Service Broker / + `SqlDependency` (heavier; evaluate, don't promise). +2. **MassTransit interop asymmetry** — Postgres has MT+NSB transports, SQL Server NSB only. +3. **GlobalPartitioning** — missing on both; Wave 2 of the rollout plan (best candidates). + +### SignalR + +Hub integration, not a broker. Gaps that matter: Azure SignalR package referenced but no +service-mode features wired; no client-result invocations; no streaming hub methods. Best +strategic use: the push channel for **subscription queries** (cross-cutting #9). + +--- + +## Cross-cutting candidate features (ranked) + +**T** = table stakes (2+ competitor frameworks have it), **U** = unique differentiator. + +1. **DLQ depth cluster (U/T)** — the highest-leverage cluster; builds on Wolverine's existing DB + dead-letter store + GlobalPartitioning GroupId, and nobody in .NET has Axon-grade DLQ: + - Order-preserving DLQ: same-GroupId messages queue behind a dead-lettered one (Axon + `SequencedDeadLetterQueue` is the only prior art) (U). + - Exception-type-routed dead-letter destinations (Spring Kafka 3.2) (U in .NET). + - Standardized diagnostic headers on every DLQ move: attempt count, original + endpoint/topic/partition/offset, exception FQCN + stacktrace (T, cheap). + - Redrive verbs: wrap SQS `StartMessageMoveTask`; a uniform `IDeadLetterAdmin` "replay" + across transports; ServicePulse-style UX belongs to CritterWatch (T). +2. **Broker-native scheduling backends (T)** — keep the DB scheduler as durable default, add + opt-in native delay per transport: ASB scheduled(+cancel, already mapped minus cancel), + Pulsar `DeliverAt` (cheapest win), NATS 2.12 (done), SQS DelaySeconds/FIFO-loop/EventBridge + Scheduler, Rabbit delay-level topology, EMQX `$delayed`. NSB treats native delay as the + default everywhere; MT exposes it per transport. +3. **Claim check / large-message offload (T)** — MT, NSB, Brighter, SlimMessageBus all have one; + Wolverine has none. One middleware + pluggable stores (S3, Azure Blob, GCS, **NATS Object + store**, Marten/DB LOB). SQS 1 MiB / ASB 100 MB limits make this concrete. Silverback-style + chunking is the alternative for Kafka/MQTT. +4. **Kafka exactly-once cluster (T)** — offsets-in-application-DB commit strategy first (unique + fit with Wolverine's envelope storage), Kafka transactions second, share groups when the + .NET client lands (~H2 2026). +5. **Replay/seek surface (T)** — a uniform "rewind this listener" API: Kafka offsets/timestamp, + Pulsar reader/seek, JetStream DeliverPolicy, Pub/Sub snapshots+seek, Rabbit stream offsets. + All five brokers now support it natively; no .NET framework unifies it. +6. **AsyncAPI export (T)** — FastStream/SlimMessageBus prior art; Wolverine already introspects + its full endpoint/handler graph (`describe`, OpenAPI precedent) — natural extension, cheap, + very visible. +7. **Listener runtime controls (T)** — pause/resume (NATS `PauseUntil` native; Kafka + pause-without-rebalance; others via agent stop), rate limiting per endpoint (MT, Broadway, + Watermill, Celery), pause-instead-of-sleep during error backoff. +8. **Dedup middleware (T)** — time-window ID dedup distinct from the durable inbox: surface ASB + `RequiresDuplicateDetection`, NATS `Nats-Msg-Id`, SQS FIFO dedup, Pulsar producer dedup under + one `Envelope.DeduplicationId` story (partially wired today: SQS + Pulsar only). +9. **Subscription queries over SignalR (U)** — Axon-only feature; query returns current result + + live updates pushed as projections/handlers update. Wolverine uniquely has the pieces + (HTTP + SignalR transport + Marten projections). Strong critter-stack demo material. +10. **Batch refinements (T)** — batch-key routing (Broadway `put_batch_key`), byte-based batch + sizing, per-message failure inside a batch (Spring `BatchListenerFailedException` pattern). +11. **Second-level failure handlers (T)** — dispatch exhausted messages as `IFailed` to a + compensation handler (Rebus/Broadway/Axon) as an alternative to dead-lettering. +12. **New transports (U)** — Azure Event Hubs (MT rider exists; pairs with #4 checkpointing) and + EventBridge (rules/scheduler/pipes; uncontested). Evaluate demand before committing. + +Already covered, no action: in-proc per-key serialized execution (local partitioned topologies + +`ProcessConcurrentlyByKey`), SQS fair queues (shipped), retry topics on Kafka/Pulsar (shipped), +test harness (TrackedSession covers the core; per-transport in-memory stubbing is the only delta), +distributed jobs (sagas + scheduled messages; revisit only if demand), routing slips (sagas). + +## Suggested sequencing + +| Phase | Items | Notes | +|---|---|---| +| Quick wins (each ≤ a few days) | Pulsar `DeliverAt` native scheduling; ASB `PrefetchCount`; SQS `DelaySeconds`; Kafka KIP-848 doc/expose; DLQ diagnostic headers; ASB scheduled-cancel | Independent, release-note friendly | +| GlobalPartitioning plan Waves 1–2 | docs/tests catch-up + Postgres/SQL Server sharded queues | see `GLOBAL-PARTITIONING-ROLLOUT-PLAN.md` | +| Feature cluster 1 | DLQ depth (order-preserving, exception routing, redrive verbs) | biggest differentiation-per-effort | +| Feature cluster 2 | Claim check middleware + stores | closes the most-cited table-stakes gap | +| Feature cluster 3 | Native scheduling backends (Rabbit delay topology, EventBridge Scheduler, EMQX) | per-transport opt-ins | +| Deep lanes | NATS KV/ObjStore + priority groups; Rabbit super streams; Kafka offsets-in-DB; Pub/Sub seek/snapshots; MQTT v5 properties | one epic per transport | +| Watch list | Kafka share groups (.NET client ~H2 2026); MQTTnet shared-subscription API; SE.Redis XACKDEL surfacing; DotPulsar transactions/nack | re-check quarterly | diff --git a/docs/blog/2026-06-18-wolverine-kafka-overhaul-draft.md b/docs/blog/2026-06-18-wolverine-kafka-overhaul-draft.md new file mode 100644 index 000000000..feade1b1d --- /dev/null +++ b/docs/blog/2026-06-18-wolverine-kafka-overhaul-draft.md @@ -0,0 +1,174 @@ +# Wolverine's Kafka Transport Just Grew Up + +*Draft — published as Wolverine 6.13.0 on 2026-06-18.* + +Wolverine has supported Kafka for a while, but until yesterday the integration carried a few embarrassing seams: every consumed message paid for a synchronous `Commit()` round-trip to the broker, there was no first-class way to scale a single partition's processing, replay was a "stop the world" affair, and the non-blocking retry pattern everyone copies from Spring/Uber simply wasn't there. We tracked all of this under the [#3134 "Re-Evaluate Kafka Integration"](https://github.com/JasperFx/wolverine/issues/3134) umbrella, and on **June 18 we shipped nine merged PRs against it in a single day**. Here's the tour. + +--- + +## The headline: a ~20x throughput cliff is gone + +The original Wolverine Kafka listener called `_consumer.Commit()` — argument-less, blocking — after every successfully handled message. That is a synchronous network round-trip per message, and on a busy partition it dominated the cost of consuming. Internal benchmarks pegged the overhead at roughly **20× slower than the idiomatic Kafka model**. + +[**#3150** — *Kafka: commit-strategy overhaul with `CommitMode`*](https://github.com/JasperFx/wolverine/pull/3150) replaces that with an explicit, opt-in strategy that defaults to the idiomatic non-blocking path: + +```csharp +opts.UseKafka(connectionString) + .ConfigureListeners(l => l.CommitOffsets(CommitMode.StoreThenAutoFlush)); +``` + +The four modes: + +| Mode | What it does | When to reach for it | +|---|---|---| +| `StoreThenAutoFlush` *(default)* | `EnableAutoOffsetStore=false` + `StoreOffset` per completed message; Kafka's background committer flushes on `AutoCommitIntervalMs` | The new default — idiomatic Kafka throughput | +| `PerMessage` | Synchronous commit of the message's own offset | Strict at-least-once on low-volume topics | +| `BatchCount(n)` | Commit watermark every N messages | High-volume topics where you want a tunable lever | +| `BatchInterval(t)` | Commit watermark every T elapsed | Bursty traffic | + +A subtle but important correctness fix rides along: `CompleteAsync` and the DLQ paths now commit the message's **specific `TopicPartitionOffset`** (offset + 1), not the consumer's global position. That was a prerequisite for every concurrency feature below. + +If you'd already set `EnableAutoCommit=true` on the Kafka client, Wolverine now respects that and issues no manual commits at all — the previous transport blanket-overrode it. + +### And: in-flight-safe watermarks for every mode + +In Wolverine's default buffered listener (handlers running at `MaxDegreeOfParallelism`), messages can complete out of order. The original Batch strategy tracked an in-flight watermark; the new `StoreThenAutoFlush` and `PerMessage` strategies initially did not, which meant a fast-completing offset 11 could advance the committed position past a still-in-flight offset 10 — and on a crash, that 10 would be silently dropped. + +[**#3161** — *in-flight-safe offset watermark for all commit strategies*](https://github.com/JasperFx/wolverine/pull/3161) routes all three manual strategies through a per-partition `OffsetWatermark`. The committable position is now the lowest still-in-flight offset, or high-water + 1 when nothing is in flight. **It never advances past in-flight work**, it's monotonic across re-seeks, and it tolerates the offset gaps that compacted or `read_committed` transactional topics produce. + +--- + +## Scale-out, the way Kafka actually wants you to do it + +The next two PRs make Kafka's own group coordinator the recommended path to scale Wolverine handlers across nodes. + +### [**#3139** — Cooperative-sticky rebalancing + static membership](https://github.com/JasperFx/wolverine/pull/3139) + +Two opt-in knobs that any production Kafka deployment will recognize: + +```csharp +opts.UseKafka(connectionString) + .UseCooperativeStickyAssignment() // incremental rebalances + .UseStaticMembership(); // POD_NAME → HOSTNAME → machine name +``` + +- **`UseCooperativeStickyAssignment()`** sets `partition.assignment.strategy = CooperativeSticky`, so a rebalance only moves the partitions that *need* to move — the rest of the group keeps working uninterrupted. +- **`UseStaticMembership()`** sets `group.instance.id` so a rolling restart of the same pod doesn't churn the partition map. Instance id is resolved from `POD_NAME` → `HOSTNAME` → machine name (the k8s StatefulSet idiom), and Wolverine logs the resolved id at startup so you can verify per-node uniqueness. + +Both are opt-in so you don't break a live rolling upgrade by silently switching assignment strategies. The Kafka docs section now spells out the two-step rolling onto cooperative-sticky. + +### [**#3140** — Opt-in intra-partition concurrency by key](https://github.com/JasperFx/wolverine/pull/3140) + +The second concurrency lever. Within a single partition assigned to your node, process messages with **different keys concurrently while preserving strict ordering per key**: + +```csharp +opts.ListenToKafkaTopic("orders") + .ProcessConcurrentlyByKey(PartitionSlots: 8); +``` + +The trick is that this reuses Wolverine's existing durable sharded execution — it forces the durable inbox, persists each envelope in consumption order, commits the Kafka offset on persist (the specific-offset fix from #3150), and shards inbox processing by the message key. The inbox is the reliability boundary, so a crash or rebalance can't lose in-flight work. + +### [**#3146** — First-class `AutoOffsetReset` + ephemeral hot-tail](https://github.com/JasperFx/wolverine/pull/3146) + +Cold start and live-tail consumption are now first-class: + +```csharp +opts.ListenToKafkaTopic("metrics").BeginAtEarliest(); // or .BeginAtLatest() +opts.ListenToKafkaTopic("events").TailFromLatest(); // broadcast/fan-out +``` + +`TailFromLatest()` is the interesting one — the listener joins a **unique per-process consumer group** (`{ServiceName}-hot-tail-{guid}`) at the tail with `EnableAutoCommit=true`. Every node receives every message, no commits, no replay. This is the Kafka-local equivalent of a broadcast subscription, and it's perfect for cache invalidation, ephemeral notifications, or dashboards. The trade-off (throwaway consumer groups left behind on the broker) is called out in the docs. + +--- + +## Replay, finally as a discrete operation + +[**#3147** — *Bounded one-shot replay via `Assign`*](https://github.com/JasperFx/wolverine/pull/3147) lets you replay a window of a topic's history back through the normal Wolverine handler pipeline **without disturbing the live consumer group**: + +```csharp +// Programmatic +await host.ReplayKafkaTopicAsync(new KafkaReplayRequest +{ + Topic = "orders", + FromTimestamp = DateTimeOffset.UtcNow.AddHours(-2), +}); +``` + +```bash +# CLI +dotnet run -- kafka-replay orders --from-timestamp 2026-06-18T10:00:00Z +``` + +Under the covers, `KafkaReplay` spins up a throwaway `Assign()`-based consumer with a unique group id and `EnableAutoCommit=false`, resolves per-partition start/end from explicit offsets or `OffsetsForTimes`, seeks to the start, and feeds every record through `runtime.Pipeline.InvokeAsync` — the **same** envelope mapping and handlers as live consumption. Each partition pauses at its end boundary. The live group's committed offsets are untouched. + +Live seek of a running group-subscribed listener and a CritterWatch control pane are explicit follow-ups. + +--- + +## Non-blocking tiered retries — the Spring/Uber pattern, native + +This is the one a lot of users have been asking for. [**#3148** — *Non-blocking tiered retry topics via `OnException` DSL*](https://github.com/JasperFx/wolverine/pull/3148): + +```csharp +opts.OnException() + .MoveToKafkaRetryTopic(1.Seconds(), 30.Seconds(), 5.Minutes()); +``` + +On a matching failure the message is produced to a tiered fixed-delay retry topic (`{source}.retry.{delay}`), the source offset is committed so the partition **keeps flowing — no head-of-line blocking**, and a delayed consumer reprocesses it through the normal handler pipeline once the tier delay elapses. After the last tier it lands in the existing Kafka DLQ. Tier, attempt, and exception metadata travel in headers. + +Two design notes worth calling out: + +- The continuation **self-guards**: if a non-Kafka listener somehow hits this rule it falls back to a normal inline retry, so the policy can never cross transports. The Kafka transport scans `opts.Policies.Failures` at `ConnectAsync` and warns at startup if non-Kafka listeners are present. +- The core got one small generic hook — `IFailureActions.ContinueWith(IContinuationSource)` — so transport-specific continuations can plug into the standard error DSL discoverably. This was the gap Pulsar's resiliency support had to work around; that pattern is now first-class. + +--- + +## Exactly-once building blocks (the cheap ones) + +[**#3149** — *Idempotent producer + `read_committed` + EOS docs*](https://github.com/JasperFx/wolverine/pull/3149) ships the cheap, opt-in pieces and — just as importantly — documents Wolverine's actual exactly-once story so you reach for the right tool: + +```csharp +opts.UseKafka(connectionString) + .UseIdempotentProducer() // producer→broker dedupe + .UseReadCommitted(); // skip records from aborted Kafka txns +``` + +The new docs section leads with the **durable inbox/outbox as the recommended path** for DB-backed apps — that's effectively-once across DB + Kafka, which Kafka transactions can't span — then covers the idempotent producer, `read_committed`, the handler-idempotency reality, and a clear non-goal callout pointing DB-free Kafka→Kafka EOS users at Kafka Streams. + +A transactional read-process-write EOS engine remains an explicit non-goal for Wolverine. + +--- + +## One small but annoying bug + +[**#3151** — *Fix `ExtendConsumerConfiguration` inheritance*](https://github.com/JasperFx/wolverine/pull/3151), contributed by [@Ferchke7](https://github.com/Ferchke7): a regression from a recent PR where `ExtendConsumerConfiguration()` created an empty topic-level `ConsumerConfig`, which Kafka then preferred over the parent, silently dropping any global consumer settings configured via `UseKafka(...).ConfigureClient(...)`. Now the topic config is layered properly: parent → existing topic → extension callback. + +--- + +## Where we are vs. where this leaves the .NET Kafka story + +A year ago you would reasonably have looked at the Wolverine Kafka transport and concluded that the .NET story for Kafka tops out at "manual `confluent-kafka-dotnet`." After yesterday, Wolverine has: + +- ✅ Idiomatic non-blocking commits, four selectable strategies, in-flight-safe watermarks +- ✅ Native scale-out via cooperative-sticky + static membership +- ✅ Second-tier concurrency by message key within a partition +- ✅ First-class cold-start / hot-tail consumption +- ✅ Bounded replay through the normal handler pipeline, without touching the live group +- ✅ Non-blocking tiered retry topics wired into the standard error DSL +- ✅ Idempotent producer + `read_committed` + an honest EOS story built on the durable inbox/outbox + +The remaining gap the umbrella tracks is the transactional read-process-write EOS engine — explicitly a non-goal — and live seek of a running group-subscribed listener, which is a near-term follow-up. Everything else on the original re-evaluation list shipped yesterday. + +Upgrade to **Wolverine 6.13.0** (or 6.13.1 for the unrelated RDBMS DLQ fix that followed), tweak nothing, and you'll already see the throughput bump from the new default commit strategy. Then pick the levers that match your topic shape. + +--- + +*— Jeremy* + + diff --git a/docs/blog/2026-06-29-critter-stack-week.md b/docs/blog/2026-06-29-critter-stack-week.md new file mode 100644 index 000000000..980a14e88 --- /dev/null +++ b/docs/blog/2026-06-29-critter-stack-week.md @@ -0,0 +1,250 @@ +# A Big Week for the Critter Stack + +The post-GA cadence has not let up. Between **June 22 and June 29**, we shipped **three Wolverine releases, three Marten releases, and three Polecat releases** — a week heavy on database-backed messaging, brand-new interoperability with the rest of the .NET messaging ecosystem, and a steady drumbeat of work to make every part of the stack more observable and more manageable from CritterWatch. + +Here's a tour of what landed. + +--- + +## Release Timeline + +| Day | Wolverine | Marten | Polecat | +|-----|-----------|--------|---------| +| Jun 22 | — | 9.10.0 | — | +| Jun 23 | 6.14.0 | — | 4.5.2 | +| Jun 25 | — | — | 4.6.0 | +| Jun 26 | 6.15.0 | 9.11.0 | — | +| Jun 29 | 6.16.0 | 9.12.0 | 4.7.0 | + +--- + + + +## CritterWatch + +_(Coming — written separately.)_ + +--- + +## Wolverine + +Three releases this week, but the headline is clear: **database-backed messaging got faster, and Wolverine now speaks the queueing protocols of the two biggest .NET messaging frameworks.** + +### 🚀 Database queue performance + +Both the PostgreSQL and SQL Server transports got a focused performance pass that targets the hottest part of any database-backed queue: the dequeue path. + +- **PostgreSQL transport** — indexed dequeue path plus more robust idempotency handling ([#3278](https://github.com/JasperFx/wolverine/pull/3278)). +- **SQL Server transport** — indexed dequeue path plus an **opt-in clustered queue layout** so the physical table organization matches how the queue is actually read ([#3277](https://github.com/JasperFx/wolverine/pull/3277)). + +On SQL Server the new layout is one fluent call. Clustering the queue and scheduled tables on a monotonic `seq` identity (instead of the previous random-`Guid` clustered key) turns FIFO dequeue into a clustered seek with physically contiguous deletes: + +```csharp +opts.UseSqlServerPersistenceAndTransport(connectionString) + .OptimizeQueueThroughput(); +``` + +The raw-DDL benchmark behind the PR tells the story — same hardware, same workload: + +| Layout | Throughput | p50 latency | p99 latency | +|--------|-----------:|------------:|------------:| +| baseline (clustered `Guid`, no index) | 98/s | 845 ms | 1,860 ms | +| **`OptimizeQueueThroughput()`** (clustered `seq`) | **34,612/s** | **2.4 ms** | **3.7 ms** | + +If you lean on Wolverine's database queues — whether as a no-broker option or to keep messaging transactionally consistent with your business data — the indexed dequeue path is a free win on upgrade. `OptimizeQueueThroughput()` is opt-in specifically because enabling it on an existing database triggers a one-time queue-table rebuild, so it's a maintenance-window change for existing systems and a no-brainer for new apps. + +📖 [SQL Server transport docs](https://wolverinefx.io/guide/messaging/transports/sqlserver.html) · 📖 [PostgreSQL transport docs](https://wolverinefx.io/guide/messaging/transports/postgresql.html) + +### 🆕 Interop with MassTransit and NServiceBus over SQL Server and PostgreSQL + +This is the big one. Wolverine can now **send to and receive from MassTransit and NServiceBus applications using each framework's own SQL Server or PostgreSQL queueing** — reading and writing their native tables directly, no shared broker required. + +Landed across 6.14.0 and 6.16.0: + +- **NServiceBus over SQL Server** ([#3198](https://github.com/JasperFx/wolverine/pull/3198)) +- **NServiceBus over PostgreSQL** ([#3201](https://github.com/JasperFx/wolverine/pull/3201)) +- **MassTransit over PostgreSQL** ([#3203](https://github.com/JasperFx/wolverine/pull/3203)) +- Each interop transport is pinned to a dedicated database under multi-tenanted storage ([#3271](https://github.com/JasperFx/wolverine/pull/3271)), `Seq` is indexed on the NServiceBus PostgreSQL queue table ([#3205](https://github.com/JasperFx/wolverine/pull/3205)), and a **shared `DatabaseListener` base** now backs the polling loop across all of these ([#3206](https://github.com/JasperFx/wolverine/pull/3206)). + +For **NServiceBus**, Wolverine reads and writes the NServiceBus queue tables directly — one table per queue with a JSON `Headers` column and a raw `Body` column: + +```csharp +using Wolverine.SqlServer.Transport.NServiceBus; + +builder.UseWolverine(opts => +{ + // Wolverine's own durable inbox/outbox still lives in SQL Server + opts.PersistMessagesWithSqlServer(connectionString, "wolverine"); + + opts.UseNServiceBusSqlServerInterop(); + + // Publish to an NServiceBus endpoint's queue table + opts.PublishMessage().ToNServiceBusSqlServerQueue("nsb"); + + // Listen to Wolverine's own queue table and use it for replies + opts.ListenToNServiceBusSqlServerQueue("wolverine").UseForReplies(); + + // Bind NServiceBus interface-typed messages to Wolverine's concrete types + opts.Policies.RegisterInteropMessageAssembly(typeof(IOrderContract).Assembly); +}); +``` + +PostgreSQL is identical with the `UseNServiceBusPostgresqlInterop()` / `ListenToNServiceBusPostgresqlQueue()` / `ToNServiceBusPostgresqlQueue()` trio. **MassTransit** is a different shape — its SQL transport is a function-driven, two-table model (`transport.message` + `transport.message_delivery`) that MassTransit owns and migrates, so Wolverine calls its stored functions rather than touching a table: + +```csharp +using Wolverine.Postgresql.Transport.MassTransit; + +builder.UseWolverine(opts => +{ + opts.PersistMessagesWithPostgresql(connectionString, "wolverine"); + + opts.UseMassTransitPostgresqlInterop(autoProvision: true); + + opts.PublishMessage().ToMassTransitPostgresqlQueue("masstransit"); + opts.ListenToMassTransitPostgresqlQueue("wolverine").UseForReplies(); + + opts.Policies.RegisterInteropMessageAssembly(typeof(IOrderContract).Assembly); +}); +``` + +These join the existing Amazon SQS interop options (which also picked up two bug fixes this week, [#3190](https://github.com/JasperFx/wolverine/pull/3190)) and a fix to map Wolverine's `TenantId` from incoming MassTransit messages ([#3192](https://github.com/JasperFx/wolverine/pull/3192)). The practical upshot: you can introduce Wolverine into an existing MassTransit or NServiceBus shop **incrementally**, service by service, over infrastructure both sides already trust. + +📖 [Interop with NServiceBus over database transports](https://wolverinefx.io/tutorials/interop.html#interop-with-nservicebus-over-database-transports) · 📖 [Interop with MassTransit over database transports](https://wolverinefx.io/tutorials/interop.html#interop-with-masstransit-over-database-transports) + +### 🔭 Observability & health + +A large share of the week's Wolverine work exists to make running systems legible — much of it surfaced directly through CritterWatch: + +- A shared **`BackgroundReceiveLoop`** with receive-loop health reporting, now adopted across SQS, Redis, the PostgreSQL queue, the SQL Server queue, and Kafka ([#3236](https://github.com/JasperFx/wolverine/pull/3236)). +- **Transport connection state** surfaced in `EndpointHealthSnapshot`, with a new `IReportConnectionState` implemented for NATS, MQTT, Pulsar, and Redis ([#3231](https://github.com/JasperFx/wolverine/pull/3231)), plus a **force-restart path for stuck listeners** ([#3232](https://github.com/JasperFx/wolverine/pull/3232)). +- A **sanitized, credential-free broker connection summary** on `BrokerDescription` ([#3272](https://github.com/JasperFx/wolverine/pull/3272)) — so the dashboard can show you *where* a broker points without ever leaking secrets. +- Richer **metrics**: every instrument tagged with a `source` service name ([#3221](https://github.com/JasperFx/wolverine/pull/3221)), dimensional inbox/outbox/scheduled gauges, and configurable histogram buckets ([#3224](https://github.com/JasperFx/wolverine/pull/3224)). +- The discovered **gRPC endpoint manifest** is now exposed via a `ServiceCapabilities` descriptor source ([#3268](https://github.com/JasperFx/wolverine/pull/3268), [#3266](https://github.com/JasperFx/wolverine/pull/3266)), and RabbitMQ sending endpoints are now properly named in health snapshots ([#3273](https://github.com/JasperFx/wolverine/pull/3273)). + +📖 [Instrumentation and Metrics](https://wolverinefx.io/guide/logging.html) · 📖 [Diagnostics](https://wolverinefx.io/guide/diagnostics.html) + +### 🐛 Reliability fixes & Pulsar + +6.14.0 also closed out a **major Pulsar re-evaluation effort** — DLQ/retry precedence, initial subscription position, multi-topic and regex subscriptions, native per-message redelivery, acknowledgment-strategy choice, a Reader interface for bounded replay and non-durable hot-tail, a tiered retry-letter error policy, producer deduplication, and both JSON and Avro schema support with broker-side registration ([#3194](https://github.com/JasperFx/wolverine/pull/3194)–[#3215](https://github.com/JasperFx/wolverine/pull/3215)). + +Two of those are worth showing. Pulsar's defining feature is broker-side **schema** registration and compatibility checking — now a single fluent call, with the message body still owned by Wolverine's serialization: + +```csharp +opts.PublishMessage() + .ToPulsarTopic("persistent://public/default/orders") + .UseJsonSchema(); // or UseAvroSchema() for Avro on the wire +``` + +And the new **tiered retry-letter policy** — the Pulsar analogue of the Kafka transport's `MoveToKafkaRetryTopic` — expresses native redelivery delays as a first-class, discoverable error policy: + +```csharp +// On failure: redeliver after 5s, then 30s, then 2m, then dead-letter. +opts.OnException() + .MoveToPulsarRetryTopic(5.Seconds(), 30.Seconds(), 2.Minutes()); +``` + +📖 [Pulsar schema support](https://wolverinefx.io/guide/messaging/transports/pulsar.html#schema-support) · 📖 [Tiered retry-letter policy](https://wolverinefx.io/guide/messaging/transports/pulsar.html#tiered-retry-letter-policy) · 📖 [Producer deduplication](https://wolverinefx.io/guide/messaging/transports/pulsar.html#producer-deduplication) + +Plus targeted reliability fixes: a RabbitMQ agent that could latch `Disconnected` after a channel-only shutdown ([#3187](https://github.com/JasperFx/wolverine/pull/3187)), stable node identity for storeless Solo hosts ([#3189](https://github.com/JasperFx/wolverine/pull/3189)), and re-attaching the sender wire tap to recovered envelopes ([#3276](https://github.com/JasperFx/wolverine/pull/3276)). + +--- + +## Polecat — Making It More Robust + +Polecat shipped three releases this week (4.5.2, 4.6.0, 4.7.0), and the through-line is **hardening**: fewer sharp edges, more parity with Marten's behavior, and a real document-metadata story. + +### 🛡️ Robustness & correctness fixes + +- **Repopulate the natural-key lookup table on projection rebuild** ([#261](https://github.com/JasperFx/polecat/pull/261)) — rebuilds no longer leave natural-key lookups stale (mirrored by the same fix in Marten, below). +- **`Patch().Set()` now honors `EnumStorage`** ([#264](https://github.com/JasperFx/polecat/pull/264)) and **supports `DateTime`/`DateTimeOffset`/`DateOnly`/`TimeOnly`** ([#265](https://github.com/JasperFx/polecat/pull/265)). +- **Sequential GUIDs for auto-assigned document ids** ([#245](https://github.com/JasperFx/polecat/pull/245)) — far friendlier to index locality than random GUIDs. +- `AsString` enum LINQ predicates honor the `JsonNamingPolicy` ([#224](https://github.com/JasperFx/polecat/pull/224)), computed-column indexes are usable by the LINQ translator ([#225](https://github.com/JasperFx/polecat/pull/225)), on-the-fly event-store schema creation and `InitialData` seeding work on startup ([#233](https://github.com/JasperFx/polecat/pull/233)), and `IEventStore.Identity` now varies by `StoreName` so multiple stores stay distinct ([#208](https://github.com/JasperFx/polecat/pull/208)). + +### 🆕 Document metadata + +A genuinely new capability area: opt-in document metadata, end to end — mirroring Marten's metadata model so the two stores behave alike. Enable the columns you want with a fluent DSL (or attributes) ([#251](https://github.com/JasperFx/polecat/pull/251), [#252](https://github.com/JasperFx/polecat/pull/252)): + +```csharp +opts.Schema.For().Metadata(m => +{ + m.LastModifiedBy.Enabled = true; + m.CorrelationId.Enabled = true; + m.CreatedAt.MapTo(x => x.CreatedDate); // project a column onto your own member +}); +``` + +Then read just the metadata for a row — no document body deserialization — via the new `MetadataForAsync` API ([#253](https://github.com/JasperFx/polecat/pull/253)): + +```csharp +DocumentMetadata metadata = await session.MetadataForAsync(order); +// metadata.Version, .LastModified, .LastModifiedBy, .CorrelationId, .CausationId, ... +``` + +Rounding it out: an opt-in `user_name` (`LastModifiedBy`) event-metadata column ([#248](https://github.com/JasperFx/polecat/pull/248)), **auto-seeding of `CorrelationId`/`CausationId` from `Activity.Current`** on session open ([#250](https://github.com/JasperFx/polecat/pull/250)), and session-level `Headers` with `SetHeader`/`GetHeader` ([#249](https://github.com/JasperFx/polecat/pull/249)). + +### 🔭 Observability & CritterWatch + +- An opt-in `polecat.event.append` **OpenTelemetry counter** ([#247](https://github.com/JasperFx/polecat/pull/247)) and runtime event-append observations via `IEventStoreInstrumentation.AppendObserver` ([#215](https://github.com/JasperFx/polecat/pull/215)). +- **`IDocumentStoreDiagnostics`** with an enriched mapping descriptor ([#210](https://github.com/JasperFx/polecat/pull/210)), structured partitioning in the `DocumentMappingDescriptor` ([#214](https://github.com/JasperFx/polecat/pull/214)), and **metadata capabilities + an `IEventStore` bridge with tenant-scoped document diagnostics** ([#254](https://github.com/JasperFx/polecat/pull/254)) — the same descriptor surface Marten exposes, so CritterWatch sees Polecat stores the same way it sees Marten. + +### 🆕 Range partitioning + +Declarative range partitioning for document tables ([#257](https://github.com/JasperFx/polecat/pull/257), [#212](https://github.com/JasperFx/polecat/pull/212)), now with a Marten-parity fluent surface — the classic time-series retention pattern: + +```csharp +// Marten manages the boundaries: +opts.Schema.For().PartitionOn(x => x.BucketEnd).ByRange(jan, feb, mar); + +// Or let a DBA / pg_partman own SPLIT/SWITCH/DROP at runtime for retention: +opts.Schema.For().PartitionOn(x => x.BucketEnd).ByExternallyManagedRange(jan, feb); +``` + +`ByExternallyManagedRange(...)` provisions the partitions once and then never reconciles them, so a later schema apply won't clobber the months your retention job has been splitting and dropping. + +📖 [Wolverine + Polecat integration guide](https://wolverinefx.io/guide/durability/polecat/) · 📦 [Polecat on GitHub](https://github.com/JasperFx/polecat) + +--- + +## Marten + +Three releases (9.10.0, 9.11.0, 9.12.0), with a mix of concurrency-hardening, new partitioning options, and — again — observability work feeding CritterWatch. + +### 🐛 Concurrency & correctness + +- **Close the `mt_events_sequence` gap on concurrent Quick OCC failures** ([#4771](https://github.com/JasperFx/marten/pull/4771)) — a first contribution from [@KMDjkb](https://github.com/KMDjkb). Under truly concurrent `FetchForWriting` + Quick-append writes to the same stream, a losing transaction could burn a sequence value it never rolled back, leaving a permanent gap that stalls the async daemon's high-water mark. A new opt-in option takes a `FOR UPDATE` lock in the OCC path so the loser blocks and raises a clean concurrency error *before* consuming a sequence value — no schema migration required: + + ```csharp + opts.Events.UseExclusiveLockOnConcurrentAppends = true; + ``` +- **Fix a false `ConcurrencyException`** from non-`RETURNING` event ops in a batched `SaveChanges` ([#4784](https://github.com/JasperFx/marten/pull/4784)). +- **Repopulate `mt_natural_key` on projection rebuild** ([#4793](https://github.com/JasperFx/marten/pull/4793)) — the Marten side of the same natural-key fix that landed in Polecat. + +### 🆕 Partitioning & queries + +- **Range-partition a document table by a non-tenant date column** ([#4780](https://github.com/JasperFx/marten/pull/4780)) — the `PartitionOn(member, cfg)` API already existed; a Weasel 9.3.0 fix makes the date-keyed retention path stable across deployments and time zones (partition bounds are now compared by normalized instant rather than raw SQL literal, so migrations no longer report a spurious destructive rebuild). +- **Metadata-filtered document and event queries** ([#4792](https://github.com/JasperFx/marten/pull/4792)) — the diagnostics surface can now filter documents and events by `correlation_id` / `causation_id` / `last_modified_by`, honored only when the store actually captures that metadata column. + +📖 [Document storage & date range partitioning](https://martendb.io/documents/storage.html) · 📖 [Document & event metadata](https://martendb.io/documents/metadata.html) + +### 🔭 Observability & CritterWatch + +- **`IDocumentStoreDiagnostics`** with an enriched mapping descriptor ([#4776](https://github.com/JasperFx/marten/pull/4776)) and populated event/document **metadata capabilities** with tenant-scoped document diagnostics ([#4790](https://github.com/JasperFx/marten/pull/4790)). +- **Runtime event-append observations via `IEventStoreInstrumentation`** ([#4783](https://github.com/JasperFx/marten/pull/4783)) and an exact-identity `DeleteProjectionProgressByShardNameAsync` for surgical projection-progress management ([#4786](https://github.com/JasperFx/marten/pull/4786)). + +--- + +## The Common Thread + +Three themes ran through all nine releases this week: + +1. **Database-backed messaging matured** — Wolverine's PostgreSQL and SQL Server queues got faster, and now interoperate directly with MassTransit and NServiceBus over the same databases. +2. **Polecat got tougher** — a stack of correctness fixes, sequential GUIDs, a full document-metadata story, and range partitioning. +3. **Everything got more observable** — diagnostics descriptors, instrumentation hooks, OpenTelemetry counters, connection-state reporting, and credential-safe broker summaries across Wolverine, Marten, and Polecat — all converging on a single, consistent surface for CritterWatch to manage. + +As always: upgrade, and [tell us what breaks](https://github.com/JasperFx/wolverine/issues). This week's patch cadence is the proof that we listen. diff --git a/docs/blog/2026-07-09-wolverine-6-17-messaging.md b/docs/blog/2026-07-09-wolverine-6-17-messaging.md new file mode 100644 index 000000000..c0ba7c0e0 --- /dev/null +++ b/docs/blog/2026-07-09-wolverine-6-17-messaging.md @@ -0,0 +1,181 @@ +# Wolverine 6.17: The Community-Powered Messaging Release + +[Wolverine 6.17](https://github.com/JasperFx/wolverine/releases/tag/V6.17.0) just went out the door, and it's a *big* one — 40+ pull requests, four first-time contributors, a brand new transport option, and a sweep that brought advanced multi-tenancy support to essentially every messaging transport Wolverine ships. + +Why so big? Partially because I went on a three night vacation and the community decided that was the perfect moment to throw in issues and pull requests left and right. And honestly, that's the story I want to tell here. + +Just yesterday I published [Things That Have Worked for Our OSS Community](https://jeremydmiller.com/2026/07/08/things-that-have-worked-for-our-oss-community/), about the practices that have made the Critter Stack easier to grow: reusable **compliance test suites**, **orthogonal code** that composes instead of duplicating, and **standardized test automation** that lets contributors (and yes, AI agents) follow consistent, proven patterns. Wolverine 6.17 is what those practices look like in release-note form. Nearly every headline item below either came from the community or was only feasible on this timeline *because* of that groundwork. + +Let's take the tour. + +--- + +## Community Contributions Front and Center + +### 🆕 A native RavenDB control queue — from the community, hardened by compliance tests + +[Daniel Winkler](https://github.com/danielwinkler) contributed a **native RavenDB-backed control queue** for Wolverine ([#3285](https://github.com/JasperFx/wolverine/pull/3285)). If you're using RavenDB for message persistence, Wolverine's internal node-to-node communication (leader election, agent assignment, health checks) now runs through RavenDB itself — no external broker and no database polling fallback required. It's registered automatically when you call `UseRavenDbPersistence()`. + +Here's the part that speaks directly to the OSS-community post: after Daniel's PR landed, making the `ravendb://` transport truly first-class was mostly a matter of **bolting on the existing compliance suites** — the full `TransportCompliance` battery plus the leadership/control-queue compliance tests that every other control transport already passes ([#3294](https://github.com/JasperFx/wolverine/pull/3294)). A community contributor built the feature; the standardized test infrastructure told us exactly what "done and trustworthy" means. That's the multiplier effect in action. + +### 🆕 NATS: dynamic subjects, JetStream dedup, and per-tenant connections + +[thedonmon](https://github.com/thedonmon) delivered a substantial upgrade to the NATS transport ([#3283](https://github.com/JasperFx/wolverine/pull/3283)): **dynamically computed subjects** via an `ISubjectResolver` hook, **JetStream deduplication windows** (`WithDeduplicationWindow()`), and **per-tenant NATS connections**. This PR did double duty — it also seeded the transport-wide multi-tenancy sweep described below, because once one transport shows the pattern, the compliance tests make it cheap to demand parity everywhere. + +### 🐛 Sharp-eyed fixes from returning and first-time contributors + +- [lahma](https://github.com/lahma) fixed `CircuitWatcher.Dispose()` failing to stop the ping-until-reconnected loop ([#3326](https://github.com/JasperFx/wolverine/pull/3326)) *and* caught the per-topic Kafka builder methods dropping SASL_SSL credentials from `ConsumerConfig`/`ProducerConfig` ([#3344](https://github.com/JasperFx/wolverine/pull/3344)) — the kind of production-hardening fix that only comes from people running this stuff for real. +- [robertdusek](https://github.com/robertdusek) (first contribution!) fixed the outbox so that discarding after a rolled-back outbox commit properly clears the incoming envelope ([#3327](https://github.com/JasperFx/wolverine/pull/3327)). +- [Steve-XYZ](https://github.com/Steve-XYZ) (first contribution!) fixed the shared-memory transport to copy envelopes at the transport handoff so in-process test transports can't accidentally share mutable state ([#3333](https://github.com/JasperFx/wolverine/pull/3333)). +- [knotekbr](https://github.com/knotekbr) (first contribution!) contributed a whole new package: **WolverineFx.Http.AspVersioning**, integrating [Asp.Versioning.Http](https://github.com/dotnet/aspnet-api-versioning) with Wolverine.HTTP endpoints ([#3324](https://github.com/JasperFx/wolverine/pull/3324)). +- [meyc-v1](https://github.com/meyc-v1) (first contribution!) linked up the community-built **Salesforce Pub/Sub transport** from the Wolverine docs ([#3337](https://github.com/JasperFx/wolverine/pull/3337)) — a whole transport built *outside* the Wolverine repository, which is exactly what the orthogonal transport model is supposed to enable. + +### 🤝 Community-initiated, finished in collaboration + +Several more 6.17 features started as community pull requests that I finished up and merged with additional tests — credit where it's due: + +- **Configuring `ServiceBusProcessorOptions` for Azure Service Bus listeners** was initiated by [jorik](https://github.com/jorik) ([#3286](https://github.com/JasperFx/wolverine/pull/3286) → [#3293](https://github.com/JasperFx/wolverine/pull/3293)). You can now tune the full processor options (prefetch, max concurrent calls, etc.) on any Azure Service Bus listening endpoint. +- **Explicit transactional `DbContext` selection for multi-`DbContext` handlers** was initiated by [KhaledZaabat](https://github.com/KhaledZaabat) ([#3284](https://github.com/JasperFx/wolverine/pull/3284) → [#3295](https://github.com/JasperFx/wolverine/pull/3295)) — when a handler touches more than one EF Core `DbContext`, you can now say which one owns the transaction and the Wolverine outbox. +- **Per-tenant agent fan-out with database-affine assignment** was initiated by [erdtsieck](https://github.com/erdtsieck) ([#3281](https://github.com/JasperFx/wolverine/pull/3281) → [#3328](https://github.com/JasperFx/wolverine/pull/3328)) — more on the event-subscription side of the house, but a big deal for folks running sharded, multi-tenanted Marten stores under Wolverine-managed projection distribution. + +--- + +## 🚀 Named Brokers and Broker-per-Tenant, Everywhere + +The biggest single theme of 6.17: **filling in the remaining gaps in "named broker" and "broker per tenant" support across every external messaging transport where it makes sense.** These capabilities used to be solid for Rabbit MQ and Azure Service Bus and hit-and-miss everywhere else. As of 6.17, the matrix is full: + +| Transport | Named brokers | Broker per tenant | PR | +|-----------|:-:|:-:|----| +| Kafka | — | ✅ new | [#3315](https://github.com/JasperFx/wolverine/pull/3315) | +| AWS SQS | — | ✅ new | [#3316](https://github.com/JasperFx/wolverine/pull/3316) | +| AWS SNS | ✅ new | ✅ new | [#3317](https://github.com/JasperFx/wolverine/pull/3317) | +| GCP Pub/Sub | ✅ new | ✅ new | [#3318](https://github.com/JasperFx/wolverine/pull/3318) | +| MQTT | ✅ new | ✅ new | [#3319](https://github.com/JasperFx/wolverine/pull/3319) | +| Pulsar | ✅ new | ✅ new | [#3320](https://github.com/JasperFx/wolverine/pull/3320) | +| Redis | ✅ new | ✅ new | [#3321](https://github.com/JasperFx/wolverine/pull/3321) | +| NATS | ✅ new | ✅ (community, [#3283](https://github.com/JasperFx/wolverine/pull/3283)) | [#3314](https://github.com/JasperFx/wolverine/pull/3314) | + +Quick refresher on what these mean: + +**Named brokers** let one application talk to *multiple distinct brokers of the same type* — say, your team's Redis plus a legacy system's Redis: + +```csharp +var analytics = new BrokerName("analytics"); + +builder.UseWolverine(opts => +{ + // The "main" broker + opts.UseRedisTransport("localhost:6379"); + + // A completely separate, additional broker + opts.AddNamedRedisBroker(analytics, "analytics-server:6379"); + + opts.PublishMessage() + .ToRedisStreamOnNamedBroker(analytics, "pageviews"); + + opts.ListenToRedisStreamOnNamedBroker(analytics, "clicks", "wolverine"); +}); +``` + +**Broker per tenant** is full physical tenant isolation: each tenant gets its *own cluster*, and Wolverine routes messages to the right broker based on the tenant id of the current message or operation — same topology, zero code changes in your handlers: + +```csharp +builder.UseWolverine(opts => +{ + opts.UseKafka("shared-cluster:9092"); + + // Dedicated Kafka cluster per tenant + opts.UseKafka("shared-cluster:9092") + .AddTenant("acme", "acme-cluster:9092") + .AddTenant("initech", "initech-cluster:9092"); + + opts.PublishMessage().ToKafkaTopic("orders"); +}); + +// Publishing for a tenant just works — this lands on acme-cluster:9092 +await bus.PublishAsync(new OrderPlaced(...), new DeliveryOptions { TenantId = "acme" }); +``` + +Now, the "how did eight transports get this in one release?" question is exactly what [the OSS-community post](https://jeremydmiller.com/2026/07/08/things-that-have-worked-for-our-oss-community/) is about: + +1. **Compliance tests** — every transport already passes the same reusable `TransportCompliance` suites, so "does the tenant-routed endpoint behave exactly like a normal endpoint?" is a question the test infrastructure answers mechanically, per transport. +2. **Orthogonal code** — multi-broker and multi-tenant routing are modeled *once* in Wolverine's core endpoint/routing model. Each transport only supplies the "give me a connection for this broker/tenant" piece; the routing, fallback-to-default semantics, and lifecycle management are shared. +3. **Standardized test automation** — the per-transport test projects follow the same harness recipes, so the pattern proven in the NATS community PR ([#3283](https://github.com/JasperFx/wolverine/pull/3283)) could be replicated across Kafka, SQS, SNS, Pub/Sub, MQTT, Pulsar, and Redis quickly and *safely*. + +The [transport multi-tenancy issue sweep](https://github.com/JasperFx/wolverine/issues/3303) (#3303–#3310) went from filed to shipped in under two weeks. That's not heroics; that's infrastructure paying rent. + +--- + +## 📦 Message Batching Grew Up + +Wolverine's [message batching](https://wolverinefx.io/guide/handlers/batching.html) got a focused, multi-phase overhaul ([GH-3289](https://github.com/JasperFx/wolverine/issues/3289)) aimed at the two questions everyone eventually hits in production: *"can I de-duplicate within a batch?"* and *"what happens when one poison message fails the whole batch?"* + +**De-duplication with `CoalesceBy`** ([#3300](https://github.com/JasperFx/wolverine/pull/3300)) — when a burst of messages for the same logical key arrives, the handler now only sees the last one per key, while every original message still settles (acks) with the batch: + +```csharp +opts.BatchMessagesOf(batching => +{ + // 500 queued recalcs for the same aggregate → the handler sees 1 + batching.CoalesceBy((RecalculateScores x) => x.AggregateId); +}); +``` + +**Poison-item isolation** — a family of tools for keeping one bad message from poisoning its whole batch, each fitting a different failure shape: + +- If your handler can *name* the bad item, throw `ApplyItemException` and Wolverine dead-letters just that member while replaying or acking the rest ([#3302](https://github.com/JasperFx/wolverine/pull/3302)): + +```csharp +public static void Handle(ImportRecord[] batch) +{ + var poison = batch.Where(x => !x.IsValid).ToArray(); + if (poison.Any()) + { + throw ApplyItemException.DeadLetterAndReplayOthers(poison); + } + + // process the batch... +} +``` + +- If a specific *exception type* means "one member is bad but I don't know which," the `IsolateBatchMembers()` error policy re-runs members individually so only the true culprit is dead-lettered ([#3311](https://github.com/JasperFx/wolverine/pull/3311)): + +```csharp +opts.OnException().IsolateBatchMembers(); +``` + +- And for fully *opaque* failures, `ProbeIndividuallyAfter(n)` kicks in after the whole batch has failed *n* times ([#3312](https://github.com/JasperFx/wolverine/pull/3312)): + +```csharp +opts.BatchMessagesOf(batching => +{ + batching.ProbeIndividuallyAfter(2); +}); +``` + +Rounding it out: a startup diagnostic that warns (or asserts) when a direct `Handle(T)` handler silently shadows your batch handler ([#3301](https://github.com/JasperFx/wolverine/pull/3301)), `ApplyItemException` correctly poisoning *every* member behind a coalesced key ([#3313](https://github.com/JasperFx/wolverine/pull/3313)), and properly documented settlement/durability semantics ([#3299](https://github.com/JasperFx/wolverine/pull/3299)). + +Notice the shape of the error-handling work: `IsolateBatchMembers()` is just another continuation in Wolverine's *composable* error-handling policy model — the same `OnException()` grammar you already use for retries, requeues, and dead-lettering. That's the "orthogonal code" point from [the community post](https://jeremydmiller.com/2026/07/08/things-that-have-worked-for-our-oss-community/): because error handling is a policy pipeline rather than transport-specific spaghetti, a new batching-specific strategy slots in without touching any transport. + +--- + +## 🐛 Messaging Reliability, Odds and Ends + +A few more messaging items worth your attention in 6.17: + +- **Buffered local queues no longer drop cascaded messages** under extreme load ([#3322](https://github.com/JasperFx/wolverine/pull/3322)) — the tail end of tracking down a silent-drop past 10K queued messages, fixed jointly with JasperFx core. +- **The dead letter queue table is now indexed for replay and cleanup scans** ([#3323](https://github.com/JasperFx/wolverine/pull/3323)). If you've ever accumulated a *large* `wolverine_dead_letters` table, the durability agent's replay and expiration polling no longer full-scans it. +- **Open Telemetry: the `InlineReceiver` no longer stomps the pipeline's `Error` activity status** ([#3292](https://github.com/JasperFx/wolverine/pull/3292)), so failed inline message processing shows up honestly in your traces. + +## Beyond Messaging + +Not messaging, but too good to skip: Wolverine.HTTP picked up support for the brand-new **HTTP `QUERY` verb** ([RFC 10008](https://www.rfc-editor.org/rfc/rfc10008), [#3296](https://github.com/JasperFx/wolverine/pull/3296)) — think "GET with a body" for complex query criteria — plus support for special characters in route templates ([#3297](https://github.com/JasperFx/wolverine/pull/3297)). And there was significant work on Wolverine-managed event subscription distribution for multi-tenanted Marten and Polecat stores that deserves its own post. + +--- + +## The Takeaway + +I'll say it one more time: go read [Things That Have Worked for Our OSS Community](https://jeremydmiller.com/2026/07/08/things-that-have-worked-for-our-oss-community/) with this release in mind. A community member shipped a whole control transport, and the compliance suites certified it. Another community member shipped advanced NATS multi-tenancy, and the orthogonal transport model let us propagate that capability across seven more transports in days. Four people made their first contribution to Wolverine in a single release — and one of them was documentation pointing at a transport the community built entirely outside our repository. + +Compliance tests, orthogonal code, and standardized test automation aren't glamorous. But they're why a three-night vacation produced the biggest Wolverine release in months instead of a merge-conflict pile. + +As always: upgrade, kick the tires, and [tell us what breaks](https://github.com/JasperFx/wolverine/issues). Clearly, we listen. diff --git a/docs/blog/2026-07-15-wolverine-http-query-verb.md b/docs/blog/2026-07-15-wolverine-http-query-verb.md new file mode 100644 index 000000000..b07227759 --- /dev/null +++ b/docs/blog/2026-07-15-wolverine-http-query-verb.md @@ -0,0 +1,183 @@ +# Wolverine.HTTP Learns the QUERY Verb + +*Draft — targeting a Wolverine 6.17 write-up. Author: Jeremy D. Miller.* + +Every so often a feature lands that is almost embarrassingly small in the diff but scratches a genuine, long-standing itch. Wolverine.HTTP now speaks the [HTTP `QUERY` method (RFC 10008)](https://www.rfc-editor.org/rfc/rfc10008.html) through a single new `[WolverineQuery]` attribute — and I want to walk through what it is, why you'd reach for it, and how it behaves inside Wolverine's middleware model. + +--- + +## What is the QUERY method, and why should I care? + +If you've ever built a search endpoint, you've probably felt the tension. Search criteria want to be a **body** — nested filters, arrays of facets, date ranges, a big structured DTO. But "read-only, cacheable, idempotent" wants to be a **`GET`**. And `GET` famously does not carry a request body in any way you can rely on. + +So we all compromise. Either you cram everything into an ever-growing query string (and start bumping into URL length limits and gnarly encoding), or you `POST` your search — quietly giving up the semantic promise that this call is safe and idempotent, and confusing every proxy, cache, and reader of your API along the way. + +`QUERY` is the method that resolves that tension. It is **safe and idempotent — like `GET`** — but it is **allowed to carry a request body — like `POST`**. It's purpose-built for exactly the search/query endpoints whose criteria are too large or too structured to encode in a URL. + +--- + +## The API surface: one attribute + +The entire feature is a single attribute, `[WolverineQuery]`, that sits right alongside the verb attributes you already know — `[WolverineGet]`, `[WolverinePost]`, `[WolverinePut]`, and friends. There's no new fluent method to learn and no configuration to flip on. + +Here's a complete search endpoint: + +```csharp +using Wolverine.Http; + +public record SearchRequest(string Term, int Page); +public record SearchResults(string Term, int Page, string[] Hits); + +// QUERY (RFC 10008) is safe and idempotent like GET, but carries a request body — ideal for +// search endpoints whose criteria are too large or structured for the query string. Wolverine +// binds the request body just like it would for POST. +[WolverineQuery("/search")] +public static SearchResults Search(SearchRequest request) +{ + var hits = Enumerable.Range(1, request.Page) + .Select(i => $"{request.Term}-{i}") + .ToArray(); + + return new SearchResults(request.Term, request.Page, hits); +} +``` + +That's it. The `SearchRequest` binds from the request body **exactly as it would for a `POST` endpoint** — same JSON deserialization, same everything. The only difference on the wire is the HTTP method, which flows straight through to ASP.NET Core as route metadata. + +--- + +## Middleware rules are dependency-based, not verb-based + +This is the part I want to be very precise about, because it's the most common wrong assumption. + +You might expect a "safe" verb like `QUERY` to be automatically exempt from transactional or outbox middleware. **It is not — and that's by design.** Wolverine has never keyed its middleware decisions off the HTTP verb; it keys them off the **dependencies your handler actually takes**: + +- **Outbox** middleware is applied when your handler depends on `IMessageBus` / `IMessageContext`. +- **Transactional** middleware is applied when your handler takes a persistence dependency like Marten's `IDocumentSession` or an EF Core `DbContext`. + +So the `/search` endpoint above stays free of transactional middleware because it takes no persistence dependency — **not** because it's a `QUERY`. The rule cuts both ways. Take an `IDocumentSession` on a `QUERY` endpoint under `AutoApplyTransactions()` and you'll get transactional middleware wrapped around it, exactly as you would on a `POST`: + +```csharp +using Marten; +using Wolverine.Http; + +// Taking an IDocumentSession attracts AutoApplyTransactions on a QUERY endpoint +// exactly as it would on a POST — there is no verb-based exemption. +[WolverineQuery("/search/audited")] +public static SearchResults SearchAudited(SearchRequest request, IDocumentSession session) +{ + session.Store(new SearchAudit(Guid.NewGuid(), request.Term)); + return new SearchResults(request.Term, request.Page, []); +} + +// IQuerySession is Marten's read-only session and does NOT trigger transactional +// middleware — the right dependency for a QUERY endpoint that reads the database. +[WolverineQuery("/search/readonly")] +public static SearchResults SearchReadonly(SearchRequest request, IQuerySession session) +{ + return new SearchResults(request.Term, request.Page, []); +} +``` + +The practical guidance: for a `QUERY` endpoint that reads the database and should stay non-transactional, take Marten's read-only `IQuerySession` instead of an `IDocumentSession` — or, on EF Core, decorate the endpoint with `[NonTransactional]`. + +--- + +## ⚠️ One caveat: OpenAPI 3.1 + +`QUERY` only became a first-class operation in **OpenAPI 3.2**. The OpenAPI 3.1 document produced by the Swashbuckle / `Microsoft.OpenApi` stack can't represent it, and naively handing it a `QUERY` operation throws and breaks document generation *for your whole application*. + +So — matching ASP.NET Core's own behavior on OpenAPI 3.1 — Wolverine **gracefully omits `QUERY` endpoints from the generated OpenAPI document** rather than break generation for everything else. Your `QUERY` endpoints are fully routable and functional; they're simply not *described* in the OpenAPI 3.1 output. First-class OpenAPI docs can follow once the underlying stack emits 3.2. + +--- + +## Testing QUERY endpoints (and an honest Alba caveat) + +Wolverine's HTTP tests lean heavily on [Alba](https://jasperfx.github.io/alba/), and Alba's `Scenario()` helpers are wonderful — but they assume the *standard* verbs and can't express a `QUERY` request with a body. That's not a knock on Alba; `QUERY` is niche enough that the ergonomic helpers just haven't grown a case for it yet. + +The good news is you don't need it to. You can stay inside the same Alba-based `IntegrationContext` host and drive a genuine `QUERY` straight through the test server's `HttpClient`: + +```csharp +public class query_verb_support : IntegrationContext +{ + public query_verb_support(AppFixture fixture) : base(fixture) { } + + [Fact] + public async Task query_endpoint_reads_request_body_and_returns_result() + { + // QUERY carries a request body (unlike GET). Alba's scenario helpers assume standard + // verbs, so drive a genuine QUERY request through the test server's HttpClient. + var client = Host.GetTestServer().CreateClient(); + + var request = new HttpRequestMessage(new HttpMethod("QUERY"), "/search") + { + Content = JsonContent.Create(new SearchRequest("widget", 3)) + }; + + var response = await client.SendAsync(request); + response.StatusCode.ShouldBe(HttpStatusCode.OK); + + var results = await response.Content.ReadFromJsonAsync(); + results.ShouldNotBeNull(); + results.Term.ShouldBe("widget"); + results.Page.ShouldBe(3); + results.Hits.ShouldBe(["widget-1", "widget-2", "widget-3"]); + } +} +``` + +`Host.GetTestServer()` comes from the same Alba/`TestServer` plumbing your other tests already use — you're just hand-building the one request whose verb Alba can't spell for you. + +You'll often also want to assert on routing and middleware wiring directly, without an HTTP round trip. Because the interesting behaviors here are about *metadata* and *middleware*, those checks read cleanly against the endpoint graph: + +```csharp +[Fact] +public void query_route_is_registered_with_QUERY_method_metadata() +{ + var endpoint = EndpointFor("/search"); + var methods = endpoint.Metadata.GetMetadata(); + methods.ShouldNotBeNull(); + methods.HttpMethods.ShouldContain("QUERY"); +} + +[Fact] +public void query_endpoint_is_not_wrapped_in_transactional_middleware() +{ + // Non-transactional because it takes no persistence dependency — NOT because QUERY is "safe". + var chain = HttpChains.Chains.Single(x => x.RoutePattern!.RawText == "/search"); + chain.RequiresOutbox().ShouldBeFalse(); + chain.IsTransactional.ShouldBeFalse(); +} + +[Fact] +public void query_endpoint_with_document_session_is_transactional() +{ + // The dependency-based rule cuts both ways: an IDocumentSession dependency + // attracts AutoApplyTransactions on a QUERY endpoint exactly as on a POST. + var chain = HttpChains.Chains.Single(x => x.RoutePattern!.RawText == "/search/audited"); + chain.IsTransactional.ShouldBeTrue(); + chain.RequiresOutbox().ShouldBeFalse(); +} +``` + +And, closing the loop on the OpenAPI caveat above, you can pin the "don't break the document" guarantee: + +```csharp +[Fact] +public void swagger_generation_still_succeeds_with_a_query_endpoint() +{ + var generator = Host.Services.GetRequiredService(); + var doc = generator.GetSwagger("default"); + + // The QUERY endpoint is gracefully omitted, not thrown on. + doc.Paths.ContainsKey("/search").ShouldBeFalse(); +} +``` + +--- + +## The bottom line + +`QUERY` support in Wolverine.HTTP is deliberately small: one attribute, no new configuration surface, and it reuses the same body binding and the same dependency-based middleware rules you already rely on for every other verb. If you've been `POST`-ing your searches and feeling slightly dirty about it, `[WolverineQuery]` is the honest verb you've been wanting. + +Full documentation lives in the [HTTP Endpoints guide → The HTTP QUERY Method](https://wolverinefx.net/guide/http/endpoints.html#the-http-query-method). diff --git a/docs/blog/2026-07-conjoined-efcore-tenancy-draft.md b/docs/blog/2026-07-conjoined-efcore-tenancy-draft.md new file mode 100644 index 000000000..2c16f7634 --- /dev/null +++ b/docs/blog/2026-07-conjoined-efcore-tenancy-draft.md @@ -0,0 +1,74 @@ +# Draft: Conjoined Multi-Tenancy for EF Core, the Critter Stack Way + +> DRAFT for Jeremy's edit — announcement post for the GH-3465 epic, targeted at the Wolverine +> 6.21 release. Code samples reference the `ConjoinedMultiTenantedEfCore` sample app. + +There's a well-traveled blog-post genre: "how we built shared-database multi-tenancy in EF +Core." A recent, well-written entry in that genre walks through the whole checklist by hand — a +tenant id column on every table, a global query filter everyone on the team has to remember to +configure (and to *not* accidentally bypass with `IgnoreQueryFilters()`), interceptors to stamp +the tenant id on writes, and raw partition DDL smuggled into EF migrations. It works. It's also +a lot of sharp-edged infrastructure code that every team rebuilds slightly differently, where +one forgotten filter means Party A is reading Party B's mail. + +Marten users have had all of that as a one-liner ("conjoined tenancy") for a decade. As of +Wolverine 6.21, EF Core users get the same thing: + +```csharp +builder.Services.AddDbContextWithWolverineManagedConjoinedTenancy( + (services, opts) => opts.UseNpgsql(connectionString)); +``` + +Mark your entities with `ITenanted` (a marker shared across the whole critter stack from +`JasperFx.MultiTenancy` — Marten and Polecat use the same one): + +```csharp +public class Invoice : ITenanted +{ + public Guid Id { get; set; } + public string TenantId { get; set; } = null!; + // ... +} +``` + +And Wolverine takes it from there: + +- **Mapped `tenant_id` column + composite indexing** — model conventions applied for you, no + fluent-API boilerplate per entity. +- **Tenant-bound global query filter** — every query through the context is automatically + scoped to the message's (or HTTP request's) tenant. There is no filter to forget. +- **Stamp-on-insert** — new entities get the ambient tenant id; your handlers never touch + `TenantId`. +- **Cross-tenant write rejection** — a write to an entity from another tenant throws + `CrossTenantWriteException` instead of silently corrupting a neighbor's data. +- **Tenant detection you already have** — the same Wolverine HTTP tenant-detection and message + `TenantId` propagation that all the other Wolverine multi-tenancy features use. +- **Conjoined sagas** — stateful workflows are tenant-scoped too. + +## Physical partitioning, if and when you want it + +Logical isolation is where most systems start; some end up wanting physical isolation for the +big tables without changing the programming model. The epic ships opt-in **Weasel-managed +tenant partitioning** — PostgreSQL list partitions (with optional bucketing) and SQL Server +tenant-ordinal partitioning — managed as schema objects the way Weasel manages everything else, +not hand-written DDL in a migration. Same entities, same queries, same code. + +## An authoritative tenant registry + +Tenancy metadata lives in a Wolverine-owned `wolverine_tenants` table: an authoritative list of +tenants (enable/disable included) that doubles as a dynamic tenant source for the rest of +Wolverine — and that CritterWatch surfaces for tenant management out of the box. + +## Marten parity, on purpose + +The test battery for this feature is a port of Marten's conjoined-tenancy compliance suite — +sentinel values, `TenantIdStyle` handling, stamping, hydration, tenant-scoped deletes, +cross-tenant rejection. If you know how conjoined tenancy behaves in Marten, you know how it +behaves here, because it's checked against the same expectations. + +## Where to start + +The `ConjoinedMultiTenantedEfCore` sample app in the Wolverine repo is the full tour: tenanted +entities, HTTP tenant detection, stamping and rejection in action, the partitioning opt-in, and +the tenant registry. Docs: [link]. Ships in Wolverine 6.21 with JasperFx 2.30.x, Weasel 9.18.x, +and Marten 9.16.x. diff --git a/docs/blog/pulsar-linkedin-post-draft.md b/docs/blog/pulsar-linkedin-post-draft.md new file mode 100644 index 000000000..f043c2782 --- /dev/null +++ b/docs/blog/pulsar-linkedin-post-draft.md @@ -0,0 +1,71 @@ + + +# LinkedIn post — draft (v2, with samples) + +--- + +A few weeks ago we gave Wolverine's **Kafka** transport a top-to-bottom overhaul. This week we did the same for **Apache Pulsar** — and it just landed in `main` for the upcoming 6.14.0 release. 🛰️ + +First, the part people don't always realize: in the .NET world, Wolverine is essentially the *only* high-level application framework with a real, first-class Pulsar transport. MassTransit and NServiceBus don't have one. + +And the baseline was already solid — all four subscription types, native retry-letter **and** dead-letter topics, scheduled delivery, persistent/non-persistent topics, multi-tenancy, CloudEvents interop, and everything that makes Wolverine *Wolverine*: the mediator + message bus, durable inbox/outbox, middleware, and uniform error-handling policies. + +But "solid" isn't "idiomatic." Pulsar has a personality of its own, and this release leans into it. A tour, with code: + +**🔹 Strongly-typed schemas (JSON + Avro), with broker-side enforcement** — no more raw bytes: +```csharp +opts.PublishMessage().ToPulsarTopic("orders").UseJsonSchema(); +opts.ListenToPulsarTopic("orders").UseAvroSchema(); +``` + +**🔹 Bounded, one-shot replay** through your normal handlers — without disturbing the live subscription's cursor: +```csharp +await host.ReplayPulsarTopicAsync(new PulsarReplayRequest { Topic = "orders" }); +``` + +**🔹 Multi-topic & regex subscriptions** — one consumer over many topics: +```csharp +opts.ListenToPulsarTopic("orders") + .TopicsPattern(new Regex("orders-.*"), RegexSubscriptionMode.All); +``` + +**🔹 Non-blocking tiered retry topics** — the Spring/Uber pattern, as a first-class error policy: +```csharp +opts.OnException() + .MoveToPulsarRetryTopic(2.Seconds(), 10.Seconds(), 1.Minutes()); +``` + +**🔹 Acknowledgment strategies + a "hot tail" broadcast mode**: +```csharp +opts.ListenToPulsarTopic("events").AcknowledgeInBatches(50, 2.Seconds()); +opts.ListenToPulsarTopic("notifications").TailFromLatest(); // every node sees every message +``` + +Plus: subscription start positions (`BeginAtEarliest`/`BeginAtLatest`), per-consumer/producer tuning hooks, native per-message redelivery, and producer-side deduplication. + +One thing I appreciated while building this: we verified every feature against the actual DotPulsar client API *before* committing to the plan — which surfaced a couple of honest constraints (the .NET client has no native negative-ack and no transactions API yet) and let us right-size the work instead of overpromising. Engineering in the open, constraints included. + +If you're running Pulsar on .NET — or thinking about it — I'd love your feedback once 6.14.0 drops. + +👉 [link to epic #3176] +👉 [link to Wolverine Pulsar docs] + +#dotnet #ApachePulsar #eventdriven #messaging #opensource #distributedsystems + +--- + +## Notes for Jeremy (remove before posting) + +- **Status is honest:** everything shown is **merged to `main`**, shipping in **6.14.0** (not yet released — latest tag is V6.13.1). The post says "just landed / once 6.14.0 drops" rather than implying it's on NuGet today. If 6.14.0 is published before you post, change to present tense. +- **All samples are real**, lifted from the shipped API + `Wolverine.Pulsar.Tests` (e.g. `UseJsonSchema()`/`UseAvroSchema()`, `ReplayPulsarTopicAsync`, `MoveToPulsarRetryTopic`, `AcknowledgeInBatches`, `TailFromLatest`, `TopicsPattern`). Double-check `RegexSubscriptionMode.All` is the member name you want to showcase. +- **Length:** ~320 words + 5 short snippets — long for LinkedIn but fine for a dev audience; a carousel/screenshots of the snippets often performs better than inline code. A ~120-word punchy variant is easy to cut. +- **Voice:** first person singular. Adjust to your usual posting voice. +- **Two links to fill:** epic https://github.com/JasperFx/wolverine/issues/3176 and the Pulsar docs page. +- **Optional:** tie visually to the Kafka "grew up" post so the two read as a series. diff --git a/docs/blog/pulsar-reevaluation-umbrella-draft.md b/docs/blog/pulsar-reevaluation-umbrella-draft.md new file mode 100644 index 000000000..28ad36030 --- /dev/null +++ b/docs/blog/pulsar-reevaluation-umbrella-draft.md @@ -0,0 +1,112 @@ + + + + +# Re-Evaluate Pulsar Integration + +> **Umbrella / epic issue.** Tracks a re-evaluation of Wolverine's Apache Pulsar +> integration so it embraces Pulsar idioms (typed schemas, negative-ack, the +> Reader interface, multi-topic subscriptions) and reaches parity with the +> operational maturity the Kafka transport gained under #3134. The detailed +> current-state audit (with file refs) and cross-framework comparison are in the +> analysis section at the bottom of this issue. + +The Kafka transport just went through a nine-PR overhaul (#3134, shipped in +6.13.0) that turned it from "shaped like the RabbitMQ transport" into something +that embraces Kafka idioms. The Pulsar transport never got the equivalent pass. +It's a solid DotPulsar wrapper — four subscription types, native retry-letter + +dead-letter topics, scheduled send, tenants/namespaces, CloudEvents interop — +but it lags both idiomatic Pulsar and Wolverine's own Kafka transport on schema +support, acknowledgment semantics, replay/broadcast, and consumer/producer +tunability. + +## Child issues (the plan) + +| # | Issue | Notes / dependencies | +|---|-------|----------------------| +| PUL-1 | **Finish the DLQ sender + resolve transport-vs-endpoint default TODOs** | Closes the `PulsarEndpoint.cs:118` DLQ-sender stub and the `PulsarTransport.cs:31` transport-level-default TODO. **Foundational, do first** — the DLQ paths are half-wired today. | +| PUL-2 | **Negative acknowledgment + redelivery backoff** (`nack`, nack-redelivery-backoff, ack-timeout-backoff) | Independent, cheap. The single most idiomatic Pulsar primitive currently missing — today we only `Acknowledge` + `RedeliverUnacknowledgedMessages`. | +| PUL-3 | **Subscription initial position** (`Earliest`/`Latest`) | Independent, cheap. Direct analogue of Kafka #3146's `BeginAtEarliest/Latest`. | +| PUL-4 | **Per-consumer / per-producer customization hooks** (`ConfigureConsumer`/`ConfigureProducer`) | Independent. Today only the global `IPulsarClientBuilder` is exposed; this unblocks fluent compression/batching tuning and matches the Kafka surface. | +| PUL-5 | **Acknowledgment-strategy choice** (cumulative + batched ack by count/interval) | Pulsar's analogue of Kafka's `CommitMode` overhaul (#3150). After PUL-4. | +| PUL-6 | **Multi-topic & regex/pattern subscriptions** | One DotPulsar consumer over many topics. Analogue of Kafka topic groups; Pulsar supports this natively. | +| PUL-7 | **Align retry-letter-topic DSL** (`MoveToPulsarRetryTopic`, non-blocking, discoverable) | Mirrors Kafka #3148. Build as a standard error policy via `IHandlerPolicy`/chain-config, startup-validate + warn on non-Pulsar endpoints, never cross-transport. Also document `Key_Shared` as the recommended by-key concurrency path (free analogue of #3140). | +| PUL-8 | **Pulsar Schema support** (JSON → Avro → `AUTO_CONSUME`) | **The big one.** Closes the gap with both Kafka's schema registry and Spring Pulsar. Likely a schema-aware producer/consumer redesign around typed `IProducer`/`IConsumer`. | +| PUL-9 | **Reader interface** → bounded replay (`ReplayPulsarTopicAsync`) + non-durable broadcast / hot-tail | Combined analogue of Kafka #3147 (replay) and #3146 (hot-tail). Doesn't disturb live durable subscriptions. | +| PUL-10 | **Producer dedup + Pulsar transactions** | Analogue of Kafka #3149's EOS building blocks. Lowest priority — Pulsar txns are heavier and lower-demand. | + +## Declared non-goals (ported from #3134, Pulsar-adjusted) + +- **Cooperative-sticky rebalancing + static membership (#3139 analogue).** N/A — the Pulsar broker owns subscription/partition assignment. Nothing to build; listed here so it isn't re-raised as a gap. +- **In-flight-safe offset watermark (#3161 analogue).** Largely moot — Pulsar acknowledges per-message-id rather than by a single monotonic partition offset, so the "fast offset 11 advances past in-flight offset 10" hazard doesn't exist the same way. PUL-5 should still confirm cumulative-ack ordering is safe under the buffered listener. +- **Transactional read-process-write EOS engine.** Same stance as Kafka: Wolverine's durable inbox/outbox already gives effectively-once for DB-backed apps. The cheap layer (producer dedup) is in PUL-10; a full Pulsar-transaction read-process-write engine is out of lane. + +## Suggested sequencing + +1. **PUL-1** (DLQ sender + default TODOs) — finishes half-wired code, lowest risk. +2. **PUL-2** (negative-ack) and **PUL-3** (initial position) — cheap, independent, high idiomatic value. +3. **PUL-4** (consumer/producer hooks) — unblocks tuning and PUL-5. +4. **PUL-5** (ack strategy) and **PUL-6** (multi-topic) — streaming-grade operations. +5. **PUL-7** (retry-topic DSL alignment) — interacts with PUL-1/PUL-2 ack semantics. +6. **PUL-8** (schema), **PUL-9** (Reader/replay/broadcast), **PUL-10** (dedup/txn) — larger / parallelizable; schema is the headline differentiator. + +## For implementers (hand-off) + +This epic is self-contained — start from this issue. Each child issue should carry its own design decisions, file references, and acceptance criteria. + +- **Start with PUL-1**: the DLQ sender at `src/Transports/Pulsar/Wolverine.Pulsar/PulsarEndpoint.cs:118` is a stub, and `PulsarTransport.cs:31` has an open question about transport-level vs per-endpoint DLQ/retry defaults. +- **Independent / parallelizable:** PUL-2, PUL-3, PUL-4, PUL-6, PUL-9. +- **Dependency edges:** PUL-5 after PUL-4; PUL-7 interacts with PUL-1/PUL-2. +- **Client library:** DotPulsar 5.1.2 (`Directory.Packages.props`). Verify its schema, `Reader`, and `nack` API surface before locking PUL-8/PUL-9 estimates. +- **Error-policy convention:** transport-specific error continuations (PUL-7) are built as standard error policies via `IHandlerPolicy`/chain-config, startup-validate + warn on non-matching endpoints, never cross-transport, and the continuation must be discoverable (not an opaque `CustomAction` func). +- **Build/test:** build the full `wolverine.slnx` (not `wolverine_slim.slnx`) before pushing; Pulsar tests require the docker-compose infra (`docker compose up -d pulsar`); prefer `--framework net9.0` for faster single-TFM runs. + +--- + +## Current-state audit (analysis) + +### What the Pulsar transport supports today + +- **Connection:** `UsePulsar(Action)` — auth/TLS delegated to DotPulsar's builder. +- **Subscription types:** Exclusive (default), Shared, Failover, Key_Shared — `WithXxxSubscriptionType()`. +- **Scheduled/delayed send:** `SupportsNativeScheduledSend = true` via `MessageMetadata.DeliverAtTime`. +- **Native retry-letter topics:** `RetryLetterQueueing(RetryLetterTopic)` (Shared/Key_Shared only — DotPulsar limitation). +- **Native + Wolverine-storage DLQ:** `DeadLetterQueueing(DeadLetterTopic)` with `Native` / `WolverineStorage` modes. +- **Requeue:** `DeferAsync` re-sends to the source topic; `DisableRequeue()` / `DisablePulsarRequeue()`. +- **Topics:** persistent + non-persistent; tenants/namespaces parsed from the `pulsar://...` URI; sharded-topic helpers (`PublishToShardedPulsarTopics`). +- **CloudEvents interop:** `UsePulsarWithCloudEvents(...)`. +- **Headers:** full envelope ↔ Pulsar property mapping via `PulsarEnvelopeMapper`. + +### Gaps vs idiomatic Pulsar and vs Wolverine's Kafka transport + +| Capability | Kafka | Pulsar | Child issue | +|---|---|---|---| +| Schema / typed messages | ✅ Avro + JSON Schema Registry | ❌ raw bytes only | PUL-8 | +| Negative acknowledgment | n/a | ❌ | PUL-2 | +| Reader / bounded replay | ✅ `ReplayKafkaTopicAsync` (#3147) | ❌ | PUL-9 | +| Ephemeral hot-tail / broadcast | ✅ `TailFromLatest` (#3146) | ❌ | PUL-9 | +| Cold-start position | ✅ `BeginAtEarliest/Latest` (#3146) | ❌ | PUL-3 | +| Ack/commit strategy choice | ✅ `CommitMode` ×4 (#3150) | ❌ per-message ack only | PUL-5 | +| Per-consumer/producer config hooks | ✅ `ConfigureConsumer/Producer` | ❌ global builder only | PUL-4 | +| Multi-topic / regex subscription | ✅ topic groups | ❌ single topic/endpoint | PUL-6 | +| Producer dedup / transactions | ⚠️ idempotent producer + read-committed (#3149) | ❌ | PUL-10 | +| Non-blocking tiered retry DSL | ✅ `MoveToKafkaRetryTopic` (#3148) | ⚠️ retry-letter topic, different model | PUL-7 | +| DLQ sender wiring | ✅ + `ExternallyOwned` (#3104) | ⚠️ stub TODO | PUL-1 | + +### Cross-framework comparison + +- **MassTransit:** no Pulsar transport; the "riders" architecture could host it but the team has stated no current plans. +- **NServiceBus / Brighter:** no Pulsar support. +- **Spring for Apache Pulsar (Java):** the reference integration — `@PulsarListener`/`@PulsarReader`, schema inference, nack + ack-timeout redelivery backoff, `DeadLetterPolicy`, batch consumption, pattern subscriptions, pause/resume. PUL-2/5/6/8/9 are the items that bring Wolverine toward this surface. + +**Net:** Wolverine is effectively the only high-level .NET application framework with a real Pulsar transport, so this re-evaluation is about reaching idiomatic-Pulsar / Spring-Pulsar parity and matching the bar the Kafka transport just set — not about catching a .NET competitor. diff --git a/docs/guide/messaging/transports/azureservicebus/performance.md b/docs/guide/messaging/transports/azureservicebus/performance.md new file mode 100644 index 000000000..7ea3ba151 --- /dev/null +++ b/docs/guide/messaging/transports/azureservicebus/performance.md @@ -0,0 +1,113 @@ +# Performance Tuning + +This page collects the levers that matter most for throughput and latency with the Azure +Service Bus transport, and the factors behind them. + +## The receive side + +Buffered and Durable endpoints (the default is buffered) pull messages in batches of +`MaximumMessagesToReceive` (default **20**) per receive call, waiting up to `MaximumWaitTime` +(default 5 seconds). Durable endpoints write each received batch to the database inbox in a +**single** batched insert, which makes durable ASB endpoints comparatively cheap per message. +Message *settlement* (complete) is one service call per message. + +### Prefetch + +`PrefetchCount` lets the Service Bus client stream messages ahead of your receive calls and is +the single biggest receive-throughput lever — without it, a listener's ceiling is roughly one +batch per network round trip: + +```cs +// Transport-wide default +opts.UseAzureServiceBus(connectionString).PrefetchCount(100); + +// Or per endpoint +opts.ListenToAzureServiceBusQueue("orders") + .PrefetchCount(60) + .ListenerCount(2) + .MaximumParallelMessages(10); +``` + +A good starting point is 2–3× `MaximumMessagesToReceive` × `ListenerCount`. **Do not set +prefetch higher than what your workers can settle within the queue's lock duration**: +prefetched messages age against their locks while waiting client-side, and an expired lock +means silent redelivery and a rising delivery count. + +### Inline endpoints process one message at a time by default + +Inline ASB endpoints use a `ServiceBusProcessor`, whose `MaxConcurrentCalls` defaults to **1**. +Wolverine does not change that default, so an inline listener is single-threaded unless you +raise it: + +```cs +opts.ListenToAzureServiceBusQueue("orders") + .ProcessInline() + .ConfigureProcessor(o => o.MaxConcurrentCalls = 10); +``` + +## Lock duration vs. processing window + +For Buffered/Durable endpoints, Wolverine does not renew message locks while messages wait in +the local worker queue. If (buffered backlog × handler time ÷ `MaximumParallelMessages`) can +exceed the queue's lock duration, locks expire silently and the broker redelivers: + +- **Durable** endpoints deduplicate redelivery through the inbox (wasted work, no duplicate + side effects). +- **Buffered** endpoints settle messages as soon as they are buffered — so lock expiry is moot + for them, but an ungraceful crash loses the buffered backlog (at-most-once on crash). +- **Inline** endpoints can rely on the processor's automatic lock renewal + (`ConfigureProcessor(o => o.MaxAutoLockRenewalDuration = ...)`, SDK default 5 minutes). + +Keep `BufferingLimits` sized so the backlog clears within the lock duration, or lengthen the +queue's `LockDuration`. + +## The send side + +Wolverine batches outgoing messages into real `ServiceBusMessageBatch`es, respecting the +broker's size limits (256 KB per message on Standard, 1 MB on Premium). Two defaults to +revisit for high-volume publishers: + +```cs +opts.PublishMessage().ToAzureServiceBusQueue("orders") + // Default 1: one batch in flight at a time per endpoint. + .MessageBatchMaxDegreeOfParallelism(4) + + // The batch timeout is a debounce (each new message resets it) — + // shrink it for low-rate, latency-sensitive routes. + .MessageBatchTimeout(50.Milliseconds()); +``` + +Inline sending (and every internal requeue/retry path) sends one message per service call — +prefer batched sending for high-volume routes. When publishing to *partitioned* entities, +outgoing batches are additionally grouped by session id so each batch shares a partition key. + +## Sessions and ordered processing + +Sessions give broker-enforced ordering per `SessionId` (mapped automatically from Wolverine's +`Envelope.GroupId`) with cluster-wide exclusivity — but session processing is inherently more +expensive than plain consumption: each session must be accepted, locked, drained, and released. +Keep `RequireSessions(n)` counts modest, and note that strict per-session *processing* order on +Buffered/Durable endpoints also needs `PartitionProcessingByGroupId(...)` (or inline +execution), since the local worker queue otherwise executes a session's batch in parallel — +this pairing is what `ExclusiveNodeWithSessionOrdering(...)` sets up for you. + +When you need per-key ordering but not broker-enforced cross-node exclusivity, a non-session +queue with `PartitionProcessingByGroupId(...)` is significantly cheaper. For cluster-wide +partitioned ordering without sessions, `UseShardedAzureServiceBusQueues(...)` in a global +partitioned topology spreads groups across N queues with exclusive listeners (forced durable — +budget for the inbox writes). + +## Namespace tier and client options + +Standard vs. Premium changes message size limits (256 KB vs. 1 MB), latency consistency, and +throughput headroom — benchmark on the tier you will run. The transport uses AMQP over TCP by +default; use the client-options hook on `UseAzureServiceBus(...)` to configure web sockets, +proxies, or `ServiceBusRetryOptions` (`TryTimeout`, retry counts and delays) when operating +through restrictive networks. + +## Interpreting Wolverine's metrics + +`wolverine-execution-time` measures the handler *plus all middleware* (including time blocked +inside middleware); `wolverine-effective-time` is wall-clock from the sender's `SentAt` stamp +through handling, cascading-message flush, and settlement, and is sensitive to clock skew +across machines. diff --git a/docs/guide/messaging/transports/sqs/performance.md b/docs/guide/messaging/transports/sqs/performance.md new file mode 100644 index 000000000..f8c4fc69a --- /dev/null +++ b/docs/guide/messaging/transports/sqs/performance.md @@ -0,0 +1,103 @@ +# Performance Tuning + +This page collects the levers that matter most for throughput and latency with the Amazon SQS +transport, and the factors behind them. + +## The receive side: batches in, singles out + +Wolverine receives from SQS in batches — each poll asks for up to `MaxNumberOfMessages` +(default **10**, the SQS maximum) with long polling enabled by default (`WaitTimeSeconds` = 5). +Durable endpoints benefit doubly: the whole received batch is written to the database inbox in +a **single** batched insert, so durable SQS endpoints are considerably cheaper per message than +push-based transports. + +Message *completion*, however, is currently one `DeleteMessage` API call per message — ten +deletes per receive at full batches. At high throughput, delete round trips (not receives) are +usually the ceiling for a single listener. The practical levers: + +```cs +opts.ListenToSqsQueue("orders") + // Parallel pollers: N independent receive loops on the same queue. + // The main receive-side throughput lever today. + .ListenerCount(4) + + // Long-poll duration. Raise toward 20s for low-traffic queues to cut + // empty-receive API calls (and cost); keep short only if you need + // faster listener shutdown. + .ConfigureListener(l => l.WaitTimeSeconds = 20) + + .MaximumParallelMessages(10); +``` + +## Visibility timeout: size it against your processing window + +Wolverine sets each received message's visibility timeout at receive (default **120 seconds**) +and does **not** extend it while messages wait in the local worker queue or execute. If +(queued messages ÷ processing rate) + handler time can exceed the visibility timeout, SQS +redelivers messages that are still in flight: + +- **Durable** endpoints deduplicate redeliveries through the inbox — you pay wasted work, not + duplicate side effects. +- **Buffered** endpoints delete messages from SQS *as soon as they are buffered*, before + handling — so instead of duplicates you get at-most-once semantics: an ungraceful crash + loses the buffered backlog. +- **Inline** endpoints delete after successful handling — safest, and the visibility timeout + only needs to cover a single handler execution. + +Rule of thumb: keep `BufferingLimits.Maximum × average handler time ÷ MaximumParallelMessages` +comfortably under the visibility timeout, or raise the timeout on the queue. + +## The send side: batch API, one batch in flight + +Wolverine sends with `SendMessageBatch` (10 messages per API call, the SQS maximum) through its +batched sender. Two defaults are worth revisiting for high-volume publishers: + +```cs +opts.PublishMessage().ToSqsQueue("orders") + // Default is 1: only one batch API call in flight at a time per endpoint. + // Raising this is the single cheapest send-throughput lever for SQS. + .MessageBatchMaxDegreeOfParallelism(8) + + // The batch timeout is a debounce (each new message resets it) — + // for low-rate latency-sensitive routes, shrink it from the 250ms default. + .MessageBatchTimeout(50.Milliseconds()); +``` + +With the defaults, sustained sending tops out around 10 messages per SQS round trip per +endpoint. Since SQS bills per API call, batching efficiency is also directly a cost lever. + +## Payload size + +The default envelope mapper embeds the serialized Wolverine envelope in the message body as +Base64, which inflates the wire size by roughly a third — budget against the 256 KB SQS message +limit accordingly. For large or high-volume payloads where you control both ends, the raw JSON +mapper (or a custom `ISqsEnvelopeMapper`) avoids the Base64 wrapping. + +## FIFO queues + +FIFO queues give broker-side ordering per `MessageGroupId` (mapped automatically from +Wolverine's `Envelope.GroupId`), but two caveats: + +1. FIFO throughput is limited **per message group** — total throughput scales with the number + of distinct group ids, so a workload funneled into a few groups will hit SQS's FIFO caps + long before the transport is the bottleneck. +2. Broker-side ordering does not by itself serialize *processing*: Buffered/Durable endpoints + execute a received batch in parallel. Pair FIFO listening with + `PartitionProcessingByGroupId(...)` (or Inline mode) to preserve per-group ordering + end to end. + +On standard queues, `PartitionProcessingByGroupId(...)` alone gives per-key sequential +processing within a node without the FIFO throughput caps, and +`UseShardedAmazonSqsQueues(...)` in a global partitioned topology adds cluster-wide ordering +at the cost of forced durable endpoints. + +Native scheduled delivery via `DelaySeconds` is used automatically for delays up to 15 minutes +on standard queues; longer delays or FIFO queues fall back to Wolverine's database-backed +scheduling. + +## Interpreting Wolverine's metrics + +`wolverine-execution-time` measures the handler *plus all middleware* (including time blocked +inside middleware); `wolverine-effective-time` is wall-clock from the sender's `SentAt` stamp +through handling, cascading-message flush, and completion, and is sensitive to clock skew +across machines. diff --git a/pubsub-3258-uncommitted.patch b/pubsub-3258-uncommitted.patch new file mode 100644 index 000000000..2a38c7904 --- /dev/null +++ b/pubsub-3258-uncommitted.patch @@ -0,0 +1,18 @@ +diff --git a/src/Transports/GCP/Wolverine.Pubsub/PubsubEndpoint.cs b/src/Transports/GCP/Wolverine.Pubsub/PubsubEndpoint.cs +index c441042c1..dac7f2828 100644 +--- a/src/Transports/GCP/Wolverine.Pubsub/PubsubEndpoint.cs ++++ b/src/Transports/GCP/Wolverine.Pubsub/PubsubEndpoint.cs +@@ -130,7 +130,12 @@ public class PubsubEndpoint : Endpoint + + + + net9.0 + + + + + + + + + + + + + + + + diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Demos/CrossTenantWriteDemo.cs b/src/Samples/ConjoinedMultiTenantedEfCore/Demos/CrossTenantWriteDemo.cs new file mode 100644 index 000000000..ddd300f55 --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Demos/CrossTenantWriteDemo.cs @@ -0,0 +1,63 @@ +using ConjoinedMultiTenantedEfCore.Invoicing; +using Microsoft.EntityFrameworkCore; +using Wolverine.EntityFrameworkCore; +using Wolverine.Http; + +namespace ConjoinedMultiTenantedEfCore.Demos; + +public record HijackInvoice(Guid InvoiceId, string NewDescription); + +public record CrossTenantWriteAttempted( + bool Rejected, + string Explanation, + string? EntityTenantId = null, + string? ContextTenantId = null); + +// **4. Cross-tenant write rejection** +// +// The tenant-bound query filter already makes it hard to *reach* another +// tenant's rows, but a determined (or just buggy) piece of code can always +// smuggle one out with IgnoreQueryFilters(). This endpoint does exactly that on +// purpose to show the second line of defense: Wolverine's stamping interceptor +// inspects every modified/deleted ITenanted entity at SaveChanges time and +// refuses to flush a row that belongs to a different tenant, throwing +// CrossTenantWriteException before anything hits the database. +// +// Try it: create an invoice as tenant "acme", then call this endpoint with the +// invoice id as tenant "initech" +public static class CrossTenantWriteDemo +{ + [WolverinePost("/demos/cross-tenant-write")] + public static async Task Attempt(HijackInvoice command, InvoicingDbContext db) + { + // IgnoreQueryFilters() is the "one forgotten filter" from the motivating + // blog post, weaponized: it lets us see (and track) rows from every tenant + var smuggled = await db.Invoices.IgnoreQueryFilters() + .SingleOrDefaultAsync(x => x.Id == command.InvoiceId); + if (smuggled == null) + { + return new CrossTenantWriteAttempted(false, + $"No invoice with id {command.InvoiceId} exists for any tenant"); + } + + smuggled.Description = command.NewDescription; + + try + { + await db.SaveChangesAsync(); + + // Only reachable when the invoice already belongs to the calling tenant + return new CrossTenantWriteAttempted(false, + "The write succeeded because the invoice belongs to the calling tenant. " + + "Call this endpoint again with a different tenant-id header to see the rejection."); + } + catch (CrossTenantWriteException e) + { + // Nothing was written. Clear the poisoned change tracker so the + // transactional middleware's own SaveChangesAsync stays a no-op + db.ChangeTracker.Clear(); + + return new CrossTenantWriteAttempted(true, e.Message, e.EntityTenantId, e.ContextTenantId); + } + } +} diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/Invoice.cs b/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/Invoice.cs new file mode 100644 index 000000000..52ab4dc9b --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/Invoice.cs @@ -0,0 +1,48 @@ +using JasperFx.MultiTenancy; + +namespace ConjoinedMultiTenantedEfCore.Invoicing; + +public enum InvoiceStatus +{ + Pending, + Approved +} + +// **1. An ITenanted entity** +// +// Implementing JasperFx.MultiTenancy.ITenanted -- the very same marker interface +// that Marten uses for its conjoined tenancy -- is the *entire* opt-in for +// Wolverine's conjoined EF Core multi-tenancy. At bootstrapping time Wolverine: +// +// * maps TenantId to a `tenant_id` column (with an index) +// * adds a global query filter binding every query to the current tenant, +// so nobody has to remember to add `.Where(x => x.TenantId == ...)` -- +// "one forgotten filter and Party A is reading Party B's mail" can't happen +// * stamps TenantId with the ambient tenant id on insert +// * rejects any cross-tenant update or delete with CrossTenantWriteException +// +// Note that this class has zero tenancy logic of its own, and neither does the +// DbContext mapping below. TenantId is framework-managed -- application code +// should never write to it. +public class Invoice : ITenanted +{ + public Guid Id { get; set; } + public string Description { get; set; } = null!; + public decimal Amount { get; set; } + public InvoiceStatus Status { get; set; } = InvoiceStatus.Pending; + public DateTime CreatedAt { get; set; } = DateTime.UtcNow; + + // Wolverine maps, stamps, and hydrates this for you. Treat the + // value as framework-managed + public string? TenantId { get; set; } +} + +// Deliberately NOT ITenanted. Entities that don't implement the marker are left +// completely alone -- no tenant_id column, no query filter, no guard. Perfect +// for reference data shared by every tenant (think a common product catalog) +public class Product +{ + public Guid Id { get; set; } + public string Name { get; set; } = null!; + public decimal ListPrice { get; set; } +} diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoiceCreatedHandler.cs b/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoiceCreatedHandler.cs new file mode 100644 index 000000000..9e6d6edcd --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoiceCreatedHandler.cs @@ -0,0 +1,43 @@ +using Microsoft.Extensions.Logging; + +namespace ConjoinedMultiTenantedEfCore.Invoicing; + +// **5 (continued). Tenant-scoped queries through message handlers** +// +// This handler runs on a durable local queue, *outside* the HTTP request, and +// still has zero tenant plumbing. The InvoiceCreated message cascaded from the +// POST /invoices endpoint carries the tenant id on its envelope, so: +// +// * the InvoicingDbContext injected here is pinned to that tenant +// * FindAsync below can only ever see that tenant's invoice +// * any write is stamped/guarded exactly like in the endpoint +// +// The transactional middleware saves and commits when the handler succeeds +public static class InvoiceCreatedHandler +{ + // Toy business rule: small invoices are approved automatically + public const decimal AutoApprovalLimit = 500; + + public static async Task Handle(InvoiceCreated message, InvoicingDbContext db, ILogger logger) + { + // Tenant-scoped load -- a message for tenant "acme" can never touch + // an "initech" invoice, even though both live in the same table + var invoice = await db.Invoices.FindAsync(message.InvoiceId); + if (invoice == null) + { + return; + } + + if (invoice.Amount <= AutoApprovalLimit) + { + invoice.Status = InvoiceStatus.Approved; + logger.LogInformation("Auto-approved invoice {InvoiceId} for tenant {TenantId}", + invoice.Id, invoice.TenantId); + } + else + { + logger.LogInformation("Invoice {InvoiceId} for tenant {TenantId} needs manual approval", + invoice.Id, invoice.TenantId); + } + } +} diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoiceEndpoints.cs b/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoiceEndpoints.cs new file mode 100644 index 000000000..d8b5f6bbc --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoiceEndpoints.cs @@ -0,0 +1,66 @@ +using Microsoft.EntityFrameworkCore; +using Wolverine.Http; + +namespace ConjoinedMultiTenantedEfCore.Invoicing; + +public record CreateInvoice(string Description, decimal Amount); + +public record InvoiceCreated(Guid InvoiceId, decimal Amount); + +public static class InvoiceEndpoints +{ + // **3. Stamp-on-insert** + // + // This endpoint has ZERO tenant awareness -- it never reads a header, never + // touches Invoice.TenantId, and never calls SaveChangesAsync(): + // + // * Wolverine.Http detects the tenant from the request (see + // MapWolverineEndpoints in Program.cs) and hands this endpoint an + // InvoicingDbContext already pinned to that tenant + // * the tenant stamping interceptor writes the tenant id into the new + // row on insert + // * the EF Core transactional middleware (Policies.AutoApplyTransactions) + // calls SaveChangesAsync and commits the outgoing InvoiceCreated + // message through the durable outbox in the same transaction + // + // The second tuple value is a cascaded message. It's published only after + // the transaction commits, and it *carries the tenant id with it*, so the + // message handler below is tenant-scoped too + [WolverinePost("/invoices")] + public static (CreationResponse, InvoiceCreated) Create( + CreateInvoice command, + InvoicingDbContext db) + { + var invoice = new Invoice + { + Id = Guid.NewGuid(), + Description = command.Description, + Amount = command.Amount + }; + + db.Invoices.Add(invoice); + + var created = new InvoiceCreated(invoice.Id, invoice.Amount); + return (CreationResponse.For(created, $"/invoices/{invoice.Id}"), created); + } + + // **5. Tenant-scoped queries through HTTP endpoints** + // + // No Where(x => x.TenantId == ...) in sight. The global query filter that + // Wolverine added to every ITenanted entity binds this query to the tenant + // detected from the request. Call it as tenant "acme" and you only ever see + // acme's invoices + [WolverineGet("/invoices")] + public static Task GetAll(InvoicingDbContext db) + { + return db.Invoices.OrderBy(x => x.CreatedAt).ToArrayAsync(); + } + + // FindAsync respects the tenant filter as well -- asking for another + // tenant's invoice id returns null, which Wolverine.Http turns into a 404 + [WolverineGet("/invoices/{id}")] + public static Task GetById(Guid id, InvoicingDbContext db) + { + return db.Invoices.FindAsync(id).AsTask(); + } +} diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoicingDbContext.cs b/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoicingDbContext.cs new file mode 100644 index 000000000..3739d1c91 --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Invoicing/InvoicingDbContext.cs @@ -0,0 +1,38 @@ +using Microsoft.EntityFrameworkCore; + +namespace ConjoinedMultiTenantedEfCore.Invoicing; + +// A completely vanilla DbContext. Notice what's *not* here: +// +// * no mapping for Invoice.TenantId +// * no HasQueryFilter() anybody has to remember for every new entity +// * no SaveChanges override stamping tenant ids +// * no interceptors +// +// Wolverine's conjoined tenancy model customizer applies all of that +// automatically to every entity implementing ITenanted when this context is +// registered with AddDbContextWithWolverineManagedConjoinedTenancy() +public class InvoicingDbContext : DbContext +{ + public InvoicingDbContext(DbContextOptions options) : base(options) + { + } + + public DbSet Invoices { get; set; } = null!; + public DbSet Products { get; set; } = null!; + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(map => + { + map.ToTable("invoices", "invoicing"); + map.HasKey(x => x.Id); + }); + + modelBuilder.Entity(map => + { + map.ToTable("products", "invoicing"); + map.HasKey(x => x.Id); + }); + } +} diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Program.cs b/src/Samples/ConjoinedMultiTenantedEfCore/Program.cs new file mode 100644 index 000000000..93fa0b8b0 --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Program.cs @@ -0,0 +1,126 @@ +using ConjoinedMultiTenantedEfCore.Invoicing; +using ConjoinedMultiTenantedEfCore.Tenants; +using JasperFx; +using JasperFx.Resources; +using Microsoft.EntityFrameworkCore; +using Wolverine; +using Wolverine.EntityFrameworkCore; +using Wolverine.Http; +using Wolverine.Postgresql; + +// Conjoined EF Core multi-tenancy (GH-3465): many tenants, ONE shared PostgreSQL +// database. Every entity implementing JasperFx.MultiTenancy.ITenanted gets a +// tenant_id column, a tenant-bound global query filter, tenant stamping on +// insert, and cross-tenant write rejection -- all applied by Wolverine, with +// zero tenancy code in the entities, the DbContext, the endpoints, or the +// message handlers. +// +// This is the scenario from +// https://barretblake.dev/posts/development/2026/07/multi-tenant-part-1/ +// (hand-rolled shared-database tenancy in EF Core: a tenant column on every +// table, named query filters everyone must remember, raw partition DDL smuggled +// into migrations) -- with every one of those pain points automated away. + +var builder = WebApplication.CreateBuilder(args); + +// See appsettings.json -- this defaults to the dockerized PostgreSQL from +// Wolverine's own docker-compose file (port 5433): +// +// docker compose up -d postgresql +var connectionString = builder.Configuration.GetConnectionString("postgres")!; + +builder.Services.AddWolverineHttp(); + +// **2. Registration** +// +// One call opts the InvoicingDbContext into Wolverine-managed conjoined +// tenancy. The DbContext shares the application's Wolverine message store +// database, so you configure a provider but never a connection string here -- +// Wolverine hands you the shared database's connection string. +// +// This also registers the IDbContextOutboxFactory, the transactional outbox +// code generation support, and the IDynamicTenantSource tenant +// registry used by the /tenants endpoints +builder.Services.AddDbContextWithWolverineManagedConjoinedTenancy( + (options, connection) => options.UseNpgsql(connection.Value), + + // Create the invoicing schema objects (and apply model changes) on startup + AutoCreate.CreateOrUpdate + + // **6. OPTIONAL: Weasel-managed physical partitioning** + // + // Uncomment the option below and every non-saga ITenanted entity table is + // physically partitioned per tenant -- PostgreSQL LIST partitions on + // tenant_id, managed by Weasel through the wolverine_tenant_partitions + // control table. No hand-written partition DDL hidden inside EF migrations. + // Partitions are created/dropped through the tenant registry (see the + // /tenants endpoints) or the IConjoinedTenantPartitions + // service, which also supports sharing one partition between small tenants. + // + // Requires UseEntityFrameworkCoreWolverineManagedMigrations() below, since + // EF migrations cannot express the partition DDL. + // + // , tenancy => tenancy.PartitionPerTenant() +); + +builder.Host.UseWolverine(opts => +{ + // The Wolverine message store IS the shared application database for + // conjoined tenancy. Using durable PostgreSQL storage gives this app the + // transactional inbox/outbox *and* the wolverine_tenants registry table + opts.PersistMessagesWithPostgresql(connectionString, "wolverine"); + + // EF Core backs Wolverine's transactional middleware... + opts.UseEntityFrameworkCoreTransactions(); + + // ...and Wolverine/Weasel manage the schema instead of EF migrations. + // This is what lets AutoCreate.CreateOrUpdate above build the invoicing + // tables, and it's required if you opt into PartitionPerTenant() + opts.UseEntityFrameworkCoreWolverineManagedMigrations(); + + // Wrap every handler and HTTP endpoint that writes through a DbContext in + // a transaction that spans the entity work and the outgoing messages + opts.Policies.AutoApplyTransactions(); + + // The InvoiceCreated message cascaded from POST /invoices is processed on a + // durable (inbox-backed) local queue + opts.Policies.UseDurableLocalQueues(); + + // Build out the database schema (message store + invoicing tables) on startup + opts.Services.AddResourceSetupOnStartup(); + + // Demo convenience: seed the "acme" and "initech" tenants in the + // wolverine_tenants registry. Registered AFTER AddResourceSetupOnStartup so + // the registry table exists before the seeder runs + opts.Services.AddHostedService(); +}); + +builder.Services.AddOpenApi(); + +var app = builder.Build(); + +if (app.Environment.IsDevelopment()) +{ + app.MapOpenApi(); +} + +// **2 (continued). HTTP tenant detection** +// +// Wolverine.Http detects the tenant id from each request and flows it through +// the endpoint, the DbContext, and any cascaded messages. The endpoints +// themselves never look at headers or query strings +app.MapWolverineEndpoints(opts => +{ + // Try headers first... + opts.TenantId.IsRequestHeaderValue("tenant-id"); + + // ...then fall back to a query string value, e.g. GET /invoices?tenant=acme + opts.TenantId.IsQueryStringValue("tenant"); + + // Any tenanted endpoint called without a detectable tenant id gets a 400 + // with ProblemDetails instead of quietly running against the default + // tenant. The /tenants administrative endpoints opt out with [NotTenanted] + opts.TenantId.AssertExists(); +}); + +return await app.RunJasperFxCommands(args); diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Properties/launchSettings.json b/src/Samples/ConjoinedMultiTenantedEfCore/Properties/launchSettings.json new file mode 100644 index 000000000..4d4cdb3fd --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Properties/launchSettings.json @@ -0,0 +1,14 @@ +{ + "$schema": "http://json.schemastore.org/launchsettings.json", + "profiles": { + "ConjoinedMultiTenantedEfCore": { + "commandName": "Project", + "dotnetRunMessages": true, + "launchBrowser": false, + "applicationUrl": "http://localhost:5581", + "environmentVariables": { + "ASPNETCORE_ENVIRONMENT": "Development" + } + } + } +} diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/README.md b/src/Samples/ConjoinedMultiTenantedEfCore/README.md new file mode 100644 index 000000000..80a952f98 --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/README.md @@ -0,0 +1,85 @@ +# ConjoinedMultiTenantedEfCore + +Sample application for **Wolverine-managed conjoined EF Core multi-tenancy** +([GH-3465](https://github.com/JasperFx/wolverine/issues/3465)): many tenants +sharing **one** PostgreSQL database, where every entity implementing +`JasperFx.MultiTenancy.ITenanted` is automatically: + +* mapped with a `tenant_id` column (plus index) +* filtered by the current tenant through a global query filter on every query +* stamped with the ambient tenant id on insert +* guarded against cross-tenant updates and deletes (`CrossTenantWriteException`) + +This mirrors the motivating scenario from +[Barret Blake's multi-tenancy series](https://barretblake.dev/posts/development/2026/07/multi-tenant-part-1/) — +hand-rolled shared-database tenancy in EF Core with a tenant column on every +table, query filters everyone must remember, and raw partition DDL smuggled +into EF migrations. Wolverine automates every one of those pain points, with the +same conjoined-tenancy semantics Marten has always had. + +## What to look at + +| Concern | File | +|---|---| +| `ITenanted` entity, vanilla `DbContext` | `Invoicing/Invoice.cs`, `Invoicing/InvoicingDbContext.cs` | +| Registration + HTTP tenant detection | `Program.cs` | +| Stamp-on-insert, tenant-scoped HTTP queries | `Invoicing/InvoiceEndpoints.cs` | +| Tenant-scoped message handler | `Invoicing/InvoiceCreatedHandler.cs` | +| Cross-tenant write rejection | `Demos/CrossTenantWriteDemo.cs` | +| Opt-in per-tenant physical partitioning | commented option in `Program.cs` | +| `wolverine_tenants` registry | `Tenants/TenantEndpoints.cs` | + +## Running it + +```bash +# from the wolverine repo root: dockerized PostgreSQL on port 5433 +docker compose up -d postgresql + +dotnet run --framework net9.0 --project src/Samples/ConjoinedMultiTenantedEfCore +``` + +The app listens on `http://localhost:5581` and seeds two fictional tenants, +`acme` and `initech`, in the `wolverine_tenants` registry at startup. + +## A guided tour with curl + +```bash +# The authoritative tenant registry (wolverine_tenants table) +curl http://localhost:5581/tenants + +# Create an invoice as acme -- note the endpoint code never touches TenantId, +# and small invoices get auto-approved by a tenant-scoped message handler +curl -s -X POST http://localhost:5581/invoices \ + -H 'content-type: application/json' -H 'tenant-id: acme' \ + -d '{"description": "widgets", "amount": 120}' + +# acme sees its invoice... +curl -s 'http://localhost:5581/invoices?tenant=acme' + +# ...initech sees nothing, same table, no Where() clauses anywhere +curl -s 'http://localhost:5581/invoices?tenant=initech' + +# Forgetting the tenant entirely is a 400, not a data leak +curl -s http://localhost:5581/invoices + +# Try to hijack acme's invoice as initech (use the id from the create call): +# the write is rejected with CrossTenantWriteException before touching the db +curl -s -X POST http://localhost:5581/demos/cross-tenant-write \ + -H 'content-type: application/json' -H 'tenant-id: initech' \ + -d '{"invoiceId": "", "newDescription": "hijacked!"}' + +# Tenant lifecycle: disable (soft), enable, remove (hard) +curl -s -X POST http://localhost:5581/tenants/globex +curl -s -X POST http://localhost:5581/tenants/globex/disable +curl -s -X DELETE http://localhost:5581/tenants/globex +``` + +## Physical partitioning (optional) + +Uncomment `tenancy => tenancy.PartitionPerTenant()` in `Program.cs` and every +non-saga `ITenanted` table becomes PostgreSQL LIST-partitioned per tenant, +managed by Weasel through the `wolverine_tenant_partitions` control table — +adding/removing tenants through the `/tenants` endpoints creates/drops the +partitions. No partition DDL in your migrations. (Drop the `invoicing` schema +first when toggling this on an existing database — a plain table can't be +converted to a partitioned one in place.) diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Tenants/TenantEndpoints.cs b/src/Samples/ConjoinedMultiTenantedEfCore/Tenants/TenantEndpoints.cs new file mode 100644 index 000000000..cce8b3ffa --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Tenants/TenantEndpoints.cs @@ -0,0 +1,71 @@ +using JasperFx.MultiTenancy; +using Wolverine.Http; + +namespace ConjoinedMultiTenantedEfCore.Tenants; + +public record TenantDirectory(string[] Active, string[] Disabled); + +// **7. The wolverine_tenants registry** +// +// Conjoined tenancy keeps an authoritative tenant list in the wolverine_tenants +// table inside the Wolverine durability schema, surfaced through the +// IDynamicTenantSource service that +// AddDbContextWithWolverineManagedConjoinedTenancy() registers. This is the +// same registry that lights up CritterWatch's tenant management UI. +// +// These administrative endpoints operate on the system as a whole rather than +// on any one tenant's data, so they're marked [NotTenanted] to opt out of the +// TenantId.AssertExists() rule in Program.cs +public static class TenantEndpoints +{ + [NotTenanted] + [WolverineGet("/tenants")] + public static async Task GetAll(IDynamicTenantSource tenants) + { + // Re-read the registry table so this node sees tenants added elsewhere + await tenants.RefreshAsync(); + + var active = tenants.AllActiveByTenant() + .Select(x => x.TenantId) + .OrderBy(x => x) + .ToArray(); + var disabled = (await tenants.AllDisabledAsync()).OrderBy(x => x).ToArray(); + + return new TenantDirectory(active, disabled); + } + + // Registers the tenant in wolverine_tenants. When Weasel-managed + // partitioning is enabled (see Program.cs), this is also what creates the + // tenant's physical partition on every ITenanted table + [NotTenanted] + [WolverinePost("/tenants/{tenantId}")] + public static async Task Add(string tenantId, IDynamicTenantSource tenants) + { + return await tenants.AddTenantAsync(tenantId, CancellationToken.None); + } + + // Soft delete: the tenant's rows stay put, but any further work for the + // tenant is rejected with UnknownTenantIdException until re-enabled + [NotTenanted] + [WolverinePost("/tenants/{tenantId}/disable")] + public static Task Disable(string tenantId, IDynamicTenantSource tenants) + { + return tenants.DisableTenantAsync(tenantId); + } + + [NotTenanted] + [WolverinePost("/tenants/{tenantId}/enable")] + public static Task Enable(string tenantId, IDynamicTenantSource tenants) + { + return tenants.EnableTenantAsync(tenantId); + } + + // Hard delete: removes the registry record. With partitioning enabled the + // tenant's partition -- and every row in it -- is dropped as well + [NotTenanted] + [WolverineDelete("/tenants/{tenantId}")] + public static Task Remove(string tenantId, IDynamicTenantSource tenants) + { + return tenants.RemoveTenantAsync(tenantId); + } +} diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/Tenants/TenantSeeder.cs b/src/Samples/ConjoinedMultiTenantedEfCore/Tenants/TenantSeeder.cs new file mode 100644 index 000000000..051267f64 --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/Tenants/TenantSeeder.cs @@ -0,0 +1,35 @@ +using JasperFx.MultiTenancy; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace ConjoinedMultiTenantedEfCore.Tenants; + +// Purely for demo convenience: register our two fictional tenants in the +// wolverine_tenants registry at startup so the sample is immediately usable. +// AddTenantAsync is an upsert, so restarting the application is harmless. +// +// This hosted service is registered *after* AddResourceSetupOnStartup() in +// Program.cs, so the registry table is guaranteed to exist by the time it runs +public class TenantSeeder : IHostedService +{ + private readonly IDynamicTenantSource _tenants; + private readonly ILogger _logger; + + public TenantSeeder(IDynamicTenantSource tenants, ILogger logger) + { + _tenants = tenants; + _logger = logger; + } + + public async Task StartAsync(CancellationToken cancellationToken) + { + await _tenants.AddTenantAsync("acme", cancellationToken); + await _tenants.AddTenantAsync("initech", cancellationToken); + _logger.LogInformation("Seeded conjoined tenants 'acme' and 'initech'"); + } + + public Task StopAsync(CancellationToken cancellationToken) + { + return Task.CompletedTask; + } +} diff --git a/src/Samples/ConjoinedMultiTenantedEfCore/appsettings.json b/src/Samples/ConjoinedMultiTenantedEfCore/appsettings.json new file mode 100644 index 000000000..843c4d895 --- /dev/null +++ b/src/Samples/ConjoinedMultiTenantedEfCore/appsettings.json @@ -0,0 +1,13 @@ +{ + "ConnectionStrings": { + "postgres": "Host=localhost;Port=5433;Database=postgres;Username=postgres;password=postgres" + }, + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning", + "Microsoft.EntityFrameworkCore": "Warning" + } + }, + "AllowedHosts": "*" +} diff --git a/wolverine.slnx b/wolverine.slnx index b24a462c3..e7752e742 100644 --- a/wolverine.slnx +++ b/wolverine.slnx @@ -139,6 +139,7 @@ +