Repository navigation
feat(durable-messaging): add durable outbox delivery - #11285
ReubenBond wants to merge 64 commits into
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
In-flight inbox and outbox pump turns do not revalidate callback generation and physical ownership after recovery, allowing stale callbacks to mutate current state.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Adds the durable outbox layer above the journaled inbox, including durable ownership, delivery pumping, retries, dead letters, and extensive contract/functional tests.
Changes:
- Adds journaled outbox scheduling and delivery coordination.
- Extends journaling with observer and participant lifecycle APIs.
- Adds durable messaging routing, serialization, diagnostics, and test infrastructure.
| File | Description |
|---|---|
src/Orleans.DurableMessaging/* |
Durable messaging runtime, contracts, routing, ownership, and delivery. |
src/Orleans.DurableMessaging/README.md |
Documents inbox/outbox behavior and layering. |
src/Orleans.DurableMessaging/Configuration/DurableInboxOptions.cs |
Adds messaging configuration and validation. |
src/Orleans.Journaling/IJournaledStateObserver.cs |
Adds journal boundary observer contract. |
src/Orleans.Journaling/IJournaledStateManager.cs |
Adds observer registration. |
src/Orleans.Journaling/IJournaledGrainParticipant.cs |
Adds feature participant contract. |
src/Orleans.Journaling/DurableGrain.cs |
Initializes journaled participants. |
src/api/Orleans.Journaling/Orleans.Journaling.cs |
Updates generated journaling API surface. |
test/Orleans.DurableMessaging.Tests/* |
Adds durable messaging contract, component, and functional tests. |
Orleans.slnx |
Includes durable messaging source and test projects. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Code coverage
Report-only conclusion: improved. The current-main baseline is commit Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
A recovery race can commit outbox work without scheduling a durable owner.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
Resolved since last review (1)
55fd4af to
d23dabb
Compare
d23dabb to
8b4342c
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Inbox deletion does not block active interleaved pump or gate operations, allowing post-delete work or stale-generation failures.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
Resolved since last review (1)
8b4342c to
e430bf3
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Remote delivery cancellation disposes its token source while in-flight attempts may still use it, creating a concurrency failure during ownership transitions.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
Resolved since last review (1)
a30bdee to
4470a33
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Inbox owner clearing can race interleaved acceptance preparation and fence the activation instead of persisting the incoming message.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
Resolved since last review (1)
4470a33 to
e9f7764
Compare
Adds the internal durable outbox above #11284, using explicit message preparation, synchronous safe-to-commit updates and dispatch after the corresponding journal acknowledgement.
Dependency: exact inbox parent
b3bca7bd6ba4e54bb9cd84a76eb0eba506d23b56, rebased onto main8bc9fd244427351ad24ccc039a9e7642a42c1cc4with the merged foundation. Inbox carries the eleven patch-equivalent contracts commits through2b8a7b072f254767c85789bd096e835516a6b2cc; the open contracts PR #11282 retains its published headfb911fa47667b91215371d9f43298c5178619de0. This PR targetsmain; review the owned layer using the immutable incremental comparison.PrepareSendAsyncsnapshots and validates envelopes, reserves message identities and confirms a viable durable self-wakeup. Live batches share the exact provider-returned job handle, with the acquisition gate released before a handle returns.Send(batch)applies prepared intents in the same synchronous turn as safe business changes, followed by an ordinary manager write. The handler facade owns attempt batches; application callers await preparation and dispose their handles after their staging/write scope.Six canonical standard durable collections provide the message, attempt, dead-letter and owner data. A small non-generic state stores the existing job-sequence value and associates message/owner snapshots with the ordinary capture/ACK protocol. All seven stream names and wire identities remain. The construction owner supplies the selected-format long codec from its actual activation or standalone dependency scope. Standard keyed registrations provide the other six states.
The sequence state's capture seals only that cohort's pending message identities and exact owner snapshot. Atomic storage acknowledgement releases those dispatch fences; C+1 remains pending through a later write. This works for ordinary application writes as well as feature-owned writes. Outbox operations await actual manager results directly, including zero-byte operations and coalesced owner repair.
Feature preparation and checks precede shared mutation. Standard dictionaries encode command entries during staging, while changed values encode during capture. Actual persistence failures retain manager-internal fencing/deactivation and the original feature-observed failure. Staging errors surface directly and retire the feature. Fresh owners replay the authoritative outcome, including lost acknowledgements.
The owning grain or standalone host stops and drains both messaging endpoints and its application operations before awaiting actual journal deletion, then disposes or deactivates the owner. Subsequent use creates a fresh owner. Shutdown drains delivery, owned preparation and writes; batch cancellation logs callback failures and keeps CTS resources alive through actual transport outcomes. Reset preserves the stopped lifetime and invalidates old prepared handles.
Explicit initialization retry is inherited from the foundation. Outbox recovery callbacks refresh cached state; repair scheduling starts when the owner invokes
OnStartafter successful manager recovery. Healthy persisted owners retain their physical IDs and shards, while wholly absent owner pairs are repaired before their next ordinary write.The specialized bootstrap fixture exercises the production outbox with actual DurableJobs and Journaling, including schedule-before-business, post-ACK remote delivery, deduplication, fresh replay and explicit recovery retry. Receiver fixtures retain their isolated outbox collaborator. Bounded metric dimensions, actual placement validation, constant-time depth and finite retry synchronization are preserved.
Scope: this intermediate project remains non-packable. #10693 supplies final public hosting, provider-cutover coverage, packaging and documentation-site integration.