[jeremydmiller]

Closes #5209

Store-level support for managing dead letters as groups of the same failure: a stable failure fingerprint, grouped summaries, affected-row counts from bulk operations, replay history across replay → re-failure, archive/restore, exception-message match modes and tenant scoping.

Strictly additive for 6.x. No existing public member changes signature. The new IDeadLetters members are default interface implementations, so third-party stores keep compiling. The schema changes are additive Weasel migrations.

New public API

Fingerprinting (Wolverine.Persistence.Durability.DeadLetterManagement)

• IDeadLetterFingerprinter.Fingerprint(Envelope, Exception?) → DeadLetterFingerprint
• DeadLetterFingerprint(string Group, string Title), with Hashed(identity, title), UnfingerprintedGroupPrefix ("unfingerprinted:"), UnfingerprintedGroupFor(...), UnfingerprintedTitleFor(...), TryParseUnfingerprintedGroup(...), MaxGroupLength, MaxTitleLength
• DefaultDeadLetterFingerprinter, with FrameworkNamespacePrefixes, FindApplicationFrame(stackTrace, exceptionMessage) and IsFrameworkFrame(frame)
• DeadLetterFingerprinting.Apply(fingerprinter, envelope, exception) and DeadLetterFingerprinting.Default
• DurabilitySettings.DeadLetterFingerprinter. An IDeadLetterFingerprinter registered in the container replaces it at startup.
• DeadLetterQueueConstants.FailureGroupHeader ("failure-group") and FailureTitleHeader ("failure-title"), both added to DiagnosticHeaders
• DeadLetterQueueConstants.StampFailureMetadata(Envelope, Exception, IDeadLetterFingerprinter?) overload. The existing overload now also stamps the failure group, using the default fingerprinter.

Query and results

• DeadLetterEnvelopeQuery: FailureGroup, ExceptionMessageMatch (DeadLetterMessageMatch.Exact / Contains; null keeps the original LIKE behavior), TenantId, Archived (DeadLetterArchiveFilter.Exclude / Only / Any)
• DeadLetterEnvelopeQuery helpers: EffectiveArchiveFilter(), CopyForPaging(int), MatchesExceptionMessage(...), MatchesFailureGroup(...), MatchesArchived(...)
• DeadLetterEnvelope: FailureGroup, FailureTitle, FailedAt, FirstFailedAt, ReplayCount, LastReplayedAt, ArchivedAt
• DeadLetterEnvelopeGetRequest (HTTP / MessageStoreCollection): ExceptionMessageMatch, FailureGroup, Archived
• DeadLetterOperationResult { int? Affected }, with NotReported and Add(...)
• DeadLetterCapabilities flags: FailureGroups, OperationCounts, ArchiveAndRestore, ReplayHistory, ExceptionMessageMatching, All
• DeadLetterGrouping (FailureGroup, MessageType, Endpoint, ExceptionType), DeadLetterSummaryOptions { Grouping, Query }, DeadLetterGroupSummary(ServiceName, Database, Grouping, Key, Count) { Title, ExceptionType, PredatesFingerprinting, FirstFailedAt, LastFailedAt }
• DeadLetterSummaries.Group(...), KeyFor(...), SummarizeByQueryingAsync(...), ContainsPattern(...), LikeEscapeClause

IDeadLetters (all with default implementations)

• DeadLetterCapabilities Capabilities (default None)
• Task<IReadOnlyList<DeadLetterGroupSummary>> SummarizeAllAsync(string serviceName, DeadLetterSummaryOptions options, CancellationToken token) (default: page through QueryAsync and group in memory)
• Task<DeadLetterOperationResult> ReplayMatchingAsync(DeadLetterEnvelopeQuery, CancellationToken) (default: ReplayAsync, then Affected = null)
• Task<DeadLetterOperationResult> DiscardMatchingAsync(DeadLetterEnvelopeQuery, CancellationToken) (default: DiscardAsync, then Affected = null)
• Task<DeadLetterOperationResult> ArchiveMatchingAsync(...) / RestoreMatchingAsync(...) (default: NotSupportedException)

Settings

• DurabilitySettings.ArchivedDeadLetterRetention (TimeSpan?, default null)

RDBMS (Wolverine.RDBMS)

• DatabaseConstants: DeadLetterHistoryTable, FailureGroup, FailureTitle, FailedAt, FirstFailedAt, ReplayCount, LastReplayedAt, ArchivedAt, DeadLetterExtendedFields, DeadLetterQueryFields, DeadLetterQueryFieldCount
• DatabasePersistence: ReadDeadLetterExtendedFieldsAsync, DeadLetterGroupingColumns, DeadLetterGroupingColumnCount, DeadLetterSummaryAggregates, ReadDeadLetterGroupSummaryAsync
• DeleteExpiredDeadLetterMessagesOperation(database, logger, utcNow, DurabilitySettings) constructor overload (the type itself is internal)

Behavior

• Fingerprint. The default is a SHA-256 (first 128 bits, hex) of exceptionType|first application frame. Frames from System., Microsoft., Npgsql., MySqlConnector., Oracle., Wolverine., JasperFx., Weasel., Marten., Polecat., Fisher., Lamar., Internal.Generated. and lambda_method are skipped, and the list is configurable. If nothing qualifies, the first frame is used. File and line are stripped, and the exception message is removed from the stack text before parsing. With no stack trace the key is "<ExceptionType>: no stack trace". The title is ExceptionType: Type.Method(...).
• Where it is computed. On every path into dead-letter storage, inside each store's insert (relational, Oracle, RavenDb, CosmosDb), so MessageContext, DurableReceiver, BufferedReceiver, outbox / sending failures, batching and native-DLQ recovery are all covered. StampFailureMetadata stamps it too. When the exception has no stack trace and the envelope already carries a failure-group header, that header is kept. This keeps the real group across an IDeadLetterInterceptor that substitutes a redacted exception, and across a native-DLQ recovery that kept the headers. A throwing custom fingerprinter falls back to the default.
• Rows written before this change have a null failure_group. Summaries group them by exception type under unfingerprinted:<type>, titled "<type> (recorded before failure fingerprinting)", and that key works as a FailureGroup filter. No backfill.
• Counts. The relational stores return the row count of the single set-based statement. The header-filter path sums its 500-id chunks. RavenDb uses the bulk-operation result; CosmosDb counts the documents it touched.
• Replay history. The durability agent's replay move now runs delete + insert … select … group by id into wolverine_dead_letter_history before deleting the dead letter rows. This is portable across all five engines, with no provider-specific upsert, and safe under MessageIdentity.IdAndDestination. The dead-letter insert reads the history back with scalar subqueries. History older than the expiration window is swept by the hourly expiration operation, which is now always scheduled.
• Archive. Archiving also clears replayable. Replay always skips archived rows. A query with MessageIds defaults to Archived = Any, so discarding by id still reaches an archived row. The original SummarizeAllAsync(serviceName, TimeRange, token) now leaves archived rows out.
• Retention. When ArchivedDeadLetterRetention is set, archived rows are deleted once that long has passed since archived_at, whether or not DeadLetterQueueExpirationEnabled is set, and the expires sweep then only touches unarchived rows. When it is null, nothing changes.
• Tenants. MultiTenantedMessageStore routes query.TenantId to the tenant's database for every dead-letter operation. Without a tenant, the counted operations and the grouped summary fan out across every database. The original query, replay and discard without a tenant still throw NotSupportedException, unchanged. A single-database store ignores TenantId.

Schema (additive)

Each relational wolverine_dead_letters table gains seven nullable columns: failure_group, failure_title, failed_at, first_failed_at, replay_count, last_replayed_at and archived_at. It also gains an index on failure_group, and a new wolverine_dead_letter_history table (id PK, replay_count, last_replayed_at, first_failed_at, index on last_replayed_at) appears. The existing deadletter_admin_compliance suites ran against schemas created before this branch, so the migration path itself was exercised.

Store support

Store	Failure groups	Counts	Archive / restore	Replay history	Message match
PostgreSQL, SQL Server, SQLite, MySQL, Oracle	✅	✅	✅	✅	✅
RavenDb, CosmosDb	✅	✅	✅	❌ (flag off)	✅
Third-party IDeadLetters	defaults (Capabilities = None)	null	throws	❌	store-dependent

ArchivedDeadLetterRetention is enforced by the relational stores.

Tests run locally (each suite on its own, Release, net9.0)

• CoreTests, full suite: 3909 total, 3901 passed, 8 skipped, 0 failed. Includes the new dead_letter_fingerprinting and dead_letter_query_helpers classes.
• PostgreSQL: dead_letter_failure_group_compliance, deadletter_admin_compliance, the new MultiTenancy.dead_letters_scoped_to_a_tenant, PostgresqlMessageStoreTests, …_with_IdAndDestination_Identity, …_with_DeadLetter_Expiration, compliance_using_table_partitioning and the DLQ Bug_* classes. 350 tests, green.
• SQL Server, with DLQ expiration on for the new compliance class: the new compliance class, deadletter_admin_compliance, SqlServerMessageStoreTests, …_IdAndDestination_Identity, …_DeadLetter_Expiration, …_DQL_expiration and SqlServerBackedMessageStoreTests. 264 + 6, green.
• MySQL: the new compliance class (12), deadletter_admin_compliance, MySqlMessageStoreTests and message_store_compliance (107 together), and …_IdAndDestination_Identity (74). Green.
• SQLite: the new compliance class (12) and the existing DLQ / message-store classes (173). Green.
• Oracle: the new compliance class, deadletter_admin_compliance, oracle_durability_command_translation (statement counts updated for the two new statements in the replay move) and oracle_durability_agent_recovery (40 together), plus OracleMessageStoreTests and …_IdAndDestination_Identity (152). Green.
• RavenDb, full suite: 343 passed. Includes the new compliance class and a test for documents stored before these fields existed. That test caught a real bug, fixed here: RavenDb's ArchivedAt == null does not match a missing field, so the archived filter is a date-range predicate.
• CosmosDb: compiles, but not run locally (no emulator here). The implementation follows the existing per-document pattern, with IS_DEFINED-guarded predicates.
• dotnet build wolverine.slnx -c Release -f net9.0: clean.

The compliance coverage (DeadLetterFailureGroupCompliance) runs on every store above and includes: stable fingerprints across message text, distinct groups for distinct frames, grouped summaries (failure group / message type / endpoint) with timestamps, the FailureGroup filter on query, replay and discard, counts including the >500-row header-filter chunk path, Contains vs Exact with wildcard characters in the value, archive / restore (excluded from query, summaries and replay by default; discard by id reaches archived rows), pre-fingerprint rows, and replay history across two replay → re-failure cycles. The relational subclass adds archived retention and the history sweep. Framework-frame skipping, line-number stability, message removal, stackless fallback, header preservation and DI registration are unit tested in CoreTests.

Docs

docs/guide/durability/dead-letter-storage.md gains new sections: Failure Groups (fingerprints and the IDeadLetterFingerprinter extension point), Managing Dead Letters Programmatically (the query fields, summaries, counts, replay history, archiving / restoring and retention, store support), plus the new HTTP request fields. The samples are compiled #regions in CoreTests/Persistence/dead_letter_failure_group_samples.cs.

Release note

Dead letter failure groups. Wolverine now fingerprints every message as it moves to dead letter storage, so you can work with dead letters one failure group at a time. The default IDeadLetterFingerprinter hashes the exception type and the first stack frame from application code. DeadLetterEnvelopeQuery gains FailureGroup, ExceptionMessageMatch (Exact / Contains), TenantId and Archived. A new SummarizeAllAsync overload groups by failure group, message type or endpoint, with first and last failure times. ReplayMatchingAsync, DiscardMatchingAsync, ArchiveMatchingAsync and RestoreMatchingAsync report how many dead letters they affected. A message that fails again after a replay now reports its ReplayCount, LastReplayedAt and FirstFailedAt. Archived dead letters drop out of summaries, queries and replays until restored, and can be expired separately with DurabilitySettings.ArchivedDeadLetterRetention. IDeadLetters.Capabilities tells you what a store supports. All changes are additive. The relational stores pick up seven nullable columns, an index and a wolverine_dead_letter_history table through the normal schema migration.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U