Fix InvalidCastException starting a NATS listener with a dead-letter subject (GH-3739) - #3742
Merged
Merged
Conversation
…subject (GH-3739) NatsEndpoint.BuildListenerAsync resolved the dead-letter sender by casting the result of IEndpointCollection.GetOrBuildSendingAgent(...) straight to ISender. That method returns an ISendingAgent -- a BufferedSendingAgent, DurableSendingAgent or InlineSendingAgent -- and none of those implement ISender, so the cast threw InvalidCastException during WolverineRuntime.StartAsync for every listener configured with a dead-letter subject, regardless of the destination endpoint's sending mode. The whole host failed to start; the only workaround was to drop the dead-letter subject and let poison messages be silently discarded. Reach through to the sender the agent wraps instead, the same way EndpointCollection already does when it resolves connection state. The underlying sender is deliberately what we want here rather than the agent: NatsListener.MoveToErrorsAsync publishes the poison message and only then terminates JetStream delivery, so the forward has to reach the broker before the terminate. Enqueuing on a buffered agent would return before the message was on the wire, and a crash in between would lose it. The NATS suite had no dead-letter coverage at all, which is how this survived two releases. NatsDeadLetterSubjectTests adds both the startup regression and an end-to-end check that a poison message really lands on the configured dead-letter subject with its failure metadata. Both fail with the reported InvalidCastException without the fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3739.
Reported against 6.16.0 and 6.23.1, and still present on
main. Configuring any NATS JetStream listener with a dead-letter subject:fails the whole host during
WolverineRuntime.StartAsync:Root cause
NatsEndpoint.BuildListenerAsyncresolved the dead-letter sender by casting the result ofIEndpointCollection.GetOrBuildSendingAgent(...)straight toISender. That method returns anISendingAgent— aBufferedSendingAgent,DurableSendingAgentorInlineSendingAgent— and none of those implementISender, so the cast threw for every sending mode, not just some. The listener never started, and the only workaround was to drop the dead-letter subject entirely and let poison messages be acknowledged and discarded.What changed
Reach through to the
ISenderthe agent wraps rather than casting the agent, the same wayEndpointCollectionalready does when it resolves connection state (bothSendingAgentandInlineSendingAgentexposeSender).This is the reporter's Option B, and the underlying sender is deliberately the right choice here rather than routing through the agent:
NatsListener.MoveToErrorsAsyncpublishes the poison message and only then terminates JetStream delivery — the comment there is explicit that the forward happens first so a terminate failure can't lose the message. Enqueuing on a buffered agent would return before the message was on the wire, so a crash in between would drop the copy and terminate the original.An unrecognized agent type now throws a named
InvalidOperationExceptionidentifying the listener, dead-letter subject and actual agent type instead of a bare cast failure.Tests
The NATS suite had no dead-letter coverage at all, which is how this survived two releases.
NatsDeadLetterSubjectTestsadds:listener_with_a_dead_letter_subject_starts_up— the startup regression, plus a pin that the resolved sender is the transport-levelNatsSenderand that the agent is genuinely not anISender(so the test can't silently pass again if the cast is reintroduced).poison_message_is_forwarded_to_the_configured_dead_letter_subject— end to end: a handler throws, and a raw NATS subscriber on the dead-letter subject observes the forwarded copy carrying the original body and thex-dlq-original-subjectfailure metadata.Verified as a real red baseline — both tests fail with exactly the reported
InvalidCastExceptionwhen the fix is reverted. Full NATS suite green locally (152 tests).🤖 Generated with Claude Code