fix(runtime): resolve endpoint-scoped serializer on replay/retry path

[outofrange-consulting]

Problem

When consuming MassTransit-produced messages through UseMassTransitInterop(), the first delivery deserializes correctly, but any retry / replay (scheduled retry, durable recovery) deserializes into an all-default message — every field 0/null (e.g. OrderId comes back 0, leading to spurious NotFound-style failures).

Root cause — the interop serializer is endpoint-scoped, the retry path resolves globally

MassTransit (and NServiceBus) interop wraps the real message inside an envelope and serializes it under a message property with content type application/vnd.masstransit+json. The unwrapping is done by MassTransitJsonSerializer, which UseMassTransitInterop() registers only on the listener endpoint (EnvelopeMapper.InteropWithMassTransit → _endpoint.DefaultSerializer = serializer) — never in the global content-type registry.

Two deserialization paths then disagree:

Path	Serializer resolution	Unwraps message?
First receipt	envelope.Serializer, set by EnvelopeMapper.MapIncomingToEnvelope from the endpoint	✅
Replay / retry	envelope.Serializer is null (runtime-only, not persisted) → falls back to WolverineOptions.DetermineSerializer, which only checks the global registry	❌

On replay, HandlerPipeline.TryDeserializeEnvelope did:

var serializer = envelope.Serializer ?? _runtime.Options.DetermineSerializer(envelope);

The global registry has no entry for application/vnd.masstransit+json, so it falls back to the default System.Text.Json serializer, which deserializes the un-unwrapped MassTransit envelope root (messageId, messageType, host, …) into the target type — leaving every real field at its default value.

Fix

Resolve the serializer from the originating endpoint first, then fall back to the global registry — exactly mirroring the existing DeadLetterEnvelope.TryReadData precedent (and restoring parity with first-receipt resolution in EnvelopeMapper.MapIncomingToEnvelope):

var serializer = envelope.Serializer ?? serializerFor(envelope);

serializerFor uses the per-listener pipeline's _endpoint when available, otherwise EndpointFor(envelope.Destination) (the persisted Destination set by Envelope.MarkReceived), and only then falls back to DetermineSerializer.

This fixes the issue for any endpoint-scoped serializer on the replay path — MassTransit interop, NServiceBus interop, and custom per-endpoint serializers — with no change to the global registry and no cross-endpoint content-type collisions.

Why endpoint-first (vs. registering the interop serializer globally)

• The interop serializer is intentionally endpoint-scoped (EnvelopeMapper.InteropWithMassTransit); promoting it to the global registry would risk content-type collisions across multiple interop endpoints (last writer wins its TranslateMassTransitToWolverineUri).
• Endpoint.TryFindSerializer already falls back to the global registry, so "endpoint-first, then global" is the established read-path convention in the codebase.

Tests

masstransit_interop_serializer_on_retry (broker-free, StubAllExternalTransports): forges a real MassTransit wire payload (message nested under message), rebuilds the envelope the way durable storage hands it back on a retry (ContentType + Destination + Data, no Serializer), runs it through HandlerPipeline.TryDeserializeEnvelope, and asserts the message is unwrapped (OrderId == 92883). Confirmed RED before the fix (OrderId was 0) and GREEN after, on net9.0 and net10.0.

No regression in the existing serialization / pipeline / interop / error-handling / scheduling suites in CoreTests.