CritterWatch Integration: DLQ Recovery, Circuit Breaker Hooks, Endpoint Descriptors#2372
Merged
jeremydmiller merged 4 commits intomainfrom Mar 29, 2026
Merged
CritterWatch Integration: DLQ Recovery, Circuit Breaker Hooks, Endpoint Descriptors#2372jeremydmiller merged 4 commits intomainfrom
jeremydmiller merged 4 commits intomainfrom
Conversation
Extend IWolverineObserver with CircuitBreakerTripped and CircuitBreakerReset callbacks. When a listener's circuit breaker trips due to excessive failures, the observer is now notified with the endpoint and circuit breaker options. CritterWatch uses these hooks to raise alerts in the monitoring UI. - Add observer parameter to CircuitBreaker constructor - Fire CircuitBreakerTripped from UpdateTotalsAsync - Pass runtime.Observer when constructing CircuitBreaker in ListeningAgent and DurableLocalQueue - Update XUnitObserver with no-op implementations Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…atabase New opt-in background service that listens to the RabbitMQ dead letter queue and recovers messages into Wolverine's persistent dead letter storage (wolverine_dead_letters table). This bridges RabbitMQ's native DLX mechanism with Wolverine's database-backed DLQ management. Enable with: opts.UseRabbitMq().EnableDeadLetterQueueRecovery() The listener: - Subscribes to the wolverine-dead-letter-queue RabbitMQ queue - Reconstructs Wolverine envelope from RabbitMQ message properties - Extracts x-death headers (original queue, reason, death count) - Recovers exception info from Wolverine headers when available (InteropFriendly mode stamps these; Native mode uses x-death) - Writes to wolverine_dead_letters database table via IMessageStore - Requeues messages that fail to write (no data loss) New types: - DeadLetterQueueListener: BackgroundService implementation - DeadLetterRecoveredException: synthetic exception for recovered DLQ - RabbitMqTransport.EnableDeadLetterQueueRecovery flag Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…d E2E tests - Add params string[] overload to EnableDeadLetterQueueRecovery for listening to custom-named DLQ queues beyond the default - Introduce DeadLetterQueueRecoverySettings to hold queue name config - Update DeadLetterQueueListener to subscribe to multiple queues from the settings, falling back to the default DLQ queue name - Fix NullReferenceException: set Destination and SentAt on recovered envelopes (required by MoveToDeadLetterStorageAsync) - Extract original queue from RabbitMQ x-death headers for Destination E2E tests (3 passing, 1 marked Flaky): - recovers_native_dlq_message_to_database: single message recovery - recovers_from_custom_named_dlq: custom DLQ queue name - settings_contain_custom_queue_name: config verification - recovers_multiple_messages: bulk recovery (timing-sensitive) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Maps concrete endpoint types to human-readable labels for CritterWatch UI display (RabbitMQ Queue, Azure Service Bus Topic, Local Queue, etc.). Falls back to PascalCase splitting for unknown types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This was referenced Mar 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four changes to support CritterWatch monitoring features:
Circuit breaker observer hooks —
CircuitBreakerTrippedandCircuitBreakerResetmethods onIWolverineObserver. When a listener's circuit breaker trips, the observer is notified with the endpoint and options. CritterWatch uses these to raise/resolve alerts.Dead letter queue recovery listener —
EnableDeadLetterQueueRecovery()on the RabbitMQ transport starts a background listener that moves messages from RabbitMQ's native DLQ back into Wolverine's PostgreSQLwolverine_dead_letterstable. This enables CritterWatch to query and manage dead letters via its existing DLQ explorer UI.Custom queue name overload —
EnableDeadLetterQueueRecovery(params string[] queueNames)for services with custom DLQ queue names.Endpoint type descriptor —
TransportTypeproperty onEndpointDescriptorthat maps concrete endpoint types to human-readable labels (RabbitMQ Queue, Azure Service Bus Topic, Local Queue, Redis Stream, etc.). Used by CritterWatch's Listeners page to show what kind of endpoint each URI represents.Test plan
XUnitObserverupdated,BackPressureTestsbuildWolverine.RabbitMQ.Testsverifies messages flow through native NACK → RabbitMQ DLX → recovery listener → PostgreSQL dead letters table🤖 Generated with Claude Code