Fault Events — Auto-Publish Fault<T> on Terminal Handler Failure

[BlackChepo]

Fault Events — Auto-Publish Fault<T> on Terminal Handler Failure

Closes #2658.

Summary

Adds an opt-in mechanism that publishes a strongly-typed Fault<T> envelope whenever a handler for T terminally fails — i.e. retries are exhausted, the message is moved to the dead-letter queue, or (with explicit opt-in) discarded. Operators get a queryable, durable event stream of handler failures without scattering try/catch logic through every handler.

The feature is fully opt-in and additive. Hosts that don't enable it see no behaviour change. The hot-path cost when disabled is one Dictionary<Type, …> lookup.

Motivation

DLQ inspection alone forces operators into a generic, untyped store. Distributed consumers that want to react programmatically to failures (alerting, compensation, projections of "what failed and why") need a typed, queryable signal. This branch provides that signal as a regular Wolverine message: any handler can subscribe to Fault<T> for any T it cares about.

Public API surface

// Global opt-in (with optional redaction defaults).
opts.PublishFaultEvents(includeExceptionMessage: true, includeStackTrace: true);

// Per-type override.
opts.Policies.ForMessagesOfType<OrderPlaced>()
    .PublishFault(includeExceptionMessage: true, includeStackTrace: false);

opts.Policies.ForMessagesOfType<HighVolumeChatter>()
    .DoNotPublishFault();

// Per-type encryption auto-pairs Fault<T>.
opts.Policies.ForMessagesOfType<PaymentDetails>().Encrypt();
//   → Fault<PaymentDetails> is also encrypted on the wire and on receive.

public record Fault<T>(
    T Message,
    ExceptionInfo Exception,
    int Attempts,
    DateTimeOffset FailedAt,
    string? CorrelationId,
    Guid ConversationId,
    string? TenantId,
    string? Source,
    IReadOnlyDictionary<string, string?> Headers
) where T : class;

public static class FaultHeaders
{
    public const string AutoPublished = "wolverine.fault.auto";
    public const string OriginalId    = "wolverine.fault.original_id";
    public const string OriginalType  = "wolverine.fault.original_type";
}

Subscribers are normal Wolverine handlers — no special attribute, no opt-in registration:

public static class OrderPlacedFaultHandler
{
    public static void Handle(Fault<OrderPlaced> fault) { /* … */ }
}

For tests, auto-published faults are surfaced via ITrackedSession.AutoFaultsPublished. Hand-published bus.PublishAsync(new Fault<T>(...)) calls do not appear there — the FaultHeaders.AutoPublished distinguishes auto from manual.

What's covered (and what isn't)

Fault is published when:

• Message moved to error queue (DLQ) — every retry policy that ends in DLQ.
• Discarded — only when the failure rule was configured with discardWithFaultPublish: true.
• Expired envelope — handler entry observes the envelope past its DeliverBy.

Fault is not published in these bypass paths (intentional):

• Send-side failures (broker rejects the outbound publish).
• Unknown message type at the receiver (cannot synthesize a T).
• Pre-handler crypto failures (EncryptionPolicyViolationException, EncryptionMissingHeaderException, EncryptionDecryptionException).
• Fault<T>-handler failures (recursion guard suppresses Fault<Fault<T>>; emits wolverine.fault.recursion_suppressed activity event instead).

Atomicity caveat (documented): fault publish is best-effort, not transactionally co-committed with the DLQ insert. The receive-side outbox does not enrol the fault enqueue in the DLQ row's transaction. Subscribers must be resilient to gaps; faults are not a strict audit log. Wired explicitly into the public XmlDoc on WolverineOptions.PublishFaultEvents.

Internal architecture

• IFaultPublisher — registered via DI, resolved lazily through IWolverineRuntimeInternal to avoid circular dependency on WolverineRuntime.
• FaultPublisher.PublishIfEnabledAsync — single entry point. Enforces a MUST NOT throw contract: try/catch wraps the entire path, with logger + counter + activity event + SetStatus(Error) on failure. No path can re-enter the failure pipeline.
• FaultPublishingPolicy — per-type override store. Snapshotted into a FrozenDictionary at Freeze() time (called by WolverineRuntime.StartAsync) and read via Volatile.*, so cross-thread visibility of pre-Freeze override writes is guaranteed by the memory model rather than implicit host-startup synchronization.
• Encryption pairing — Policies.ForMessagesOfType<T>().Encrypt() reflectively registers EncryptMessageTypeRule<Fault<T>> and adds typeof(Fault<T>) to the receive-side RequiredEncryptedTypes set. Skipped for value-type T (because Fault<T> requires T : class). wolverine.encryption.* headers are stripped from Fault<T>.Headers so encryption decisions are made fresh on the outbound fault hop.
• Tracking integration — EnvelopeHistory records AutoFaultPublished events with IsComplete = true so tracked sessions complete instead of timing out.
• DiscardEnvelope — fault publish runs before CompleteAsync (terminal-sweep race fix); DiscardedEnvelope tracking event is emitted from a finally block so it always fires, even when CompleteAsync throws.

Observability

Activity event	Fires when
wolverine.fault.published	A fault is enqueued for routing.
wolverine.fault.no_route	No route exists for Fault<T>. Tagged with wolverine.fault.message_type.
wolverine.fault.recursion_suppressed	Recursion guard short-circuits a Fault<Fault<T>>.
wolverine.fault.publish_failed	The MUST NOT throw contract caught a publish-time exception.

Counter	What
wolverine.fault.events_published	Counter<int>, incremented per fault enqueued. Recursion-suppressed faults do not increment.

Outbound Fault<T> envelopes inherit ConversationId, CorrelationId, and TraceParent from the failing envelope, so distributed traces stay connected across the failure → fault hop.

Test coverage

Three layers, all green on dotnet test src/Testing/CoreTests/CoreTests.csproj (1617/1617 passing):

• Unit tests — FaultPublisher, FaultPublishingPolicy, DiscardEnvelope tracking-on-throw, recursion guard, redaction recursion through inner exceptions, encryption header strip, frozen-policy contract.
• Integration tests — PublishFaultEventsIntegrationTests, FaultRedactionIntegrationTests, FaultEncryptionRoundTripTests (byte-level wire-bytes assertion proves "ciphertext on the wire," not just "tagged for encryption"), FaultBypassTracingTests, FaultCryptoExceptionGuardTests (drives HandlerPipeline.TryDeserializeEnvelope directly with garbled bytes above the AEAD min-length so the AEAD failure path is exercised, not the early "too short" guard).
• Compliance tests — DurableFaultPublishingCompliance (Marten/Postgres, SQL Server, RavenDB) and TransportFaultRoutingCompliance (RabbitMQ, Kafka). Per-test isolation via Guid.NewGuid() queue/topic names.

Documentation

• New page: docs/guide/handlers/fault-events.md — canonical reference covering API, anatomy, delivery semantics, subscribing, per-type config, redaction, encryption pairing, observability, ITrackedSession integration, pitfalls. Wired into the Vitepress sidebar under Handlers between Error Handling and Rate Limiting.
• Cross-link: existing docs/guide/runtime/encryption.md already had a ### Fault events subsection from the encryption-pairing work; the new page links into it for byte-level mechanics.
• Sample: src/Samples/FaultEventsDemo/ — runnable single-process console demo modelled on EncryptionDemo. In-memory transport, no Docker required. Demonstrates global opt-in, per-type override (PublishFault / DoNotPublishFault), the Encrypt() ↔ Fault<T> auto-pairing, and a fault subscriber that reads FaultHeaders.AutoPublished.

Backwards compatibility

• 100% additive. No schema migration, no API break, no behaviour change for hosts that don't call PublishFaultEvents().
• MoveToErrorQueue and DiscardEnvelope continuations gain the fault-publish call but are no-ops when the global mode is None.

[jeremydmiller]

@BlackChepo Hey, I'm going to bump this back. Just out of time to make it for 5.38.

[BlackChepo]

Sure, no problem, there's no rush.