Skip to content

Fix InvalidCastException starting a NATS listener with a dead-letter subject (GH-3739) - #3742

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-3739/nats-dlq-sender
Jul 30, 2026
Merged

Fix InvalidCastException starting a NATS listener with a dead-letter subject (GH-3739)#3742
jeremydmiller merged 1 commit into
mainfrom
gh-3739/nats-dlq-sender

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #3739.

Reported against 6.16.0 and 6.23.1, and still present on main. Configuring any NATS JetStream listener with a dead-letter subject:

opts.ListenToNatsSubject(subject)
    .UseJetStream(streamName, consumerName)
    .ConfigureDeadLetterQueue(maxDeliveryAttempts, deadLetterSubject);

fails the whole host during WolverineRuntime.StartAsync:

System.InvalidCastException: Unable to cast object of type
'Wolverine.Transports.Sending.BufferedSendingAgent' to type
'Wolverine.Transports.Sending.ISender'.
   at Wolverine.Nats.Internal.NatsEndpoint.BuildListenerAsync
   at Wolverine.Transports.ListeningAgent.StartAsync()

Root cause

NatsEndpoint.BuildListenerAsync resolved the dead-letter sender by casting the result of IEndpointCollection.GetOrBuildSendingAgent(...) straight to ISender. That method returns an ISendingAgent — a BufferedSendingAgent, DurableSendingAgent or InlineSendingAgent — and none of those implement ISender, so the cast threw for every sending mode, not just some. The listener never started, and the only workaround was to drop the dead-letter subject entirely and let poison messages be acknowledged and discarded.

What changed

Reach through to the ISender the agent wraps rather than casting the agent, the same way EndpointCollection already does when it resolves connection state (both SendingAgent and InlineSendingAgent expose Sender).

This is the reporter's Option B, and the underlying sender is deliberately the right choice here rather than routing through the agent: NatsListener.MoveToErrorsAsync publishes the poison message and only then terminates JetStream delivery — the comment there is explicit that the forward happens first so a terminate failure can't lose the message. Enqueuing on a buffered agent would return before the message was on the wire, so a crash in between would drop the copy and terminate the original.

An unrecognized agent type now throws a named InvalidOperationException identifying the listener, dead-letter subject and actual agent type instead of a bare cast failure.

Tests

The NATS suite had no dead-letter coverage at all, which is how this survived two releases. NatsDeadLetterSubjectTests adds:

  • listener_with_a_dead_letter_subject_starts_up — the startup regression, plus a pin that the resolved sender is the transport-level NatsSender and that the agent is genuinely not an ISender (so the test can't silently pass again if the cast is reintroduced).
  • poison_message_is_forwarded_to_the_configured_dead_letter_subject — end to end: a handler throws, and a raw NATS subscriber on the dead-letter subject observes the forwarded copy carrying the original body and the x-dlq-original-subject failure metadata.

Verified as a real red baseline — both tests fail with exactly the reported InvalidCastException when the fix is reverted. Full NATS suite green locally (152 tests).

🤖 Generated with Claude Code

…subject (GH-3739)

NatsEndpoint.BuildListenerAsync resolved the dead-letter sender by casting the
result of IEndpointCollection.GetOrBuildSendingAgent(...) straight to ISender.
That method returns an ISendingAgent -- a BufferedSendingAgent,
DurableSendingAgent or InlineSendingAgent -- and none of those implement
ISender, so the cast threw InvalidCastException during
WolverineRuntime.StartAsync for every listener configured with a dead-letter
subject, regardless of the destination endpoint's sending mode. The whole host
failed to start; the only workaround was to drop the dead-letter subject and
let poison messages be silently discarded.

Reach through to the sender the agent wraps instead, the same way
EndpointCollection already does when it resolves connection state. The
underlying sender is deliberately what we want here rather than the agent:
NatsListener.MoveToErrorsAsync publishes the poison message and only then
terminates JetStream delivery, so the forward has to reach the broker before
the terminate. Enqueuing on a buffered agent would return before the message
was on the wire, and a crash in between would lose it.

The NATS suite had no dead-letter coverage at all, which is how this survived
two releases. NatsDeadLetterSubjectTests adds both the startup regression and
an end-to-end check that a poison message really lands on the configured
dead-letter subject with its failure metadata. Both fail with the reported
InvalidCastException without the fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NATS transport: listener startup throws InvalidCastException when a dead-letter subject is configured

1 participant