GH-4398: link each retry attempt to the failed attempt before it

[jeremydmiller]

Closes #4398. Supersedes #4402 by @outofrange-consulting, who is credited as co-author on the commits. That PR got the design right: link the next attempt to the failed one with an ActivityLink instead of reparenting it, which matches the OpenTelemetry messaging conventions and what MassTransit and Brighter do. This PR keeps that design and changes two things: where the link lands, and how it is carried.

What changed compared with #4402

1. The link goes on the span the handler runs in. #4402 used up the link in StartEnvelopeActivity. That method serves send, receive and execute spans, and every receiver starts its receive span before the pipeline starts the execute span:

• BufferedReceiver
• DurableReceiver
• NativeAckReceiver
• InlineReceiver

So on a buffered, durable or NativeAck listener, the short receive span took the link and cleared it, and the process span got nothing. Here the link is attached in HandlerPipeline.InvokeAsync(envelope, channel, activity), the one place every attempt goes through, using .NET 9's Activity.AddLink. On Inline and NativeAck listeners the receiver passes its receive span in as the processing span, so this covers those too.

2. A loose header, not a mapped Envelope property, so messages that are never retried pay nothing. A mapped property adds one more reserved-header read for every incoming message in every transport mapper, plus a new Envelope field and serializer work. Custom headers already round-trip through every mapper and through EnvelopeSerializer. The pipeline check is HasHeaders first, so it never creates the lazy headers dictionary. It also makes Envelope.Reset() pool-safe without a new line, because _headers is already cleared there.

The header is removed before the handler runs. So a handler, and anything it cascades or propagates with PropagateIncomingHeadersToOutgoing, never sees it.

Known limitation

A durable-inbox reschedule (RescheduleExistingEnvelopeForRetryAsync) only UPDATEs the row's time, attempts and status. It does not rewrite the stored body, so the link does not survive that path. Supporting it would mean rewriting the body on every durable retry across all the message stores. That is a separate decision about cost, and this PR does not make it. The in-memory scheduler, local queues and broker requeues all carry the link.

Tests

• End to end: TracingTests/retry_attempt_links. A real Wolverine host runs through the real OpenTelemetry SDK into OpenTelemetry.Exporter.InMemory, a new test-only dependency pinned to 1.15.3 like the other OTel packages. The assertions are on the spans that were actually exported.

  • The matrix covers every failure policy on every listener shape:

    	Local queue (Buffered)	RabbitMQ Inline	RabbitMQ Buffered
    Retry now	✓	✓	✓
    Scheduled retry	✓	✓	✓
    Requeue	✓	✓	✓

    Inline and Buffered reach the pipeline by different routes: Inline hands its receive span over as the handler's span, while Buffered starts a separate receive span first. Durable behaves like Buffered here, so Buffered stands in for both. Local queues can't run Inline, so the Inline column is RabbitMQ.

  • Each cell checks that:

    • each attempt's span links to the previous attempt's span
    • no other span for that message carries a link, which is the Link retry attempts to the prior failed attempt's Activity via ActivityLink #4402 defect
    • the handler never sees the header

  • Attempts are identified by the span the handler actually ran in, not by span name. A retry-now on an Inline listener reruns the pipeline in a new process span, and with no message store a scheduled retry comes back through the local replies queue.

  • A first attempt that succeeds has no links, on each listener.

  • Negative control: against origin/main's HandlerPipeline, exactly the 9 link cells fail and the 3 first-attempt cells pass.

• Unit tests, adapted from Link retry attempts to the prior failed attempt's Activity via ActivityLink #4402: each continuation records the failed attempt, and LinkToPreviousAttempt links, uses up the header, ignores malformed ids and handles a missing activity.
• Local runs:
  • CoreTests: 2910 passed, 2 skipped, 0 failed
  • TracingTests: 19 of 19 passed
  • dotnet build wolverine.slnx -c Release -f net9.0: 0 errors, 0 warnings

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj