fix(mesh): frame loss when recv_validated is cancelled mid-validation (#2458)

[brocoppler]

Summary

Root-causes the #2458 mesh_demo flake to a cancel-safety bug in production code and fixes it, plus one piece of endpoint hygiene the investigation surfaced.

The bug: run_demo_echo races recv_validated against its 100 ms drain tick in a select!. recv_validated reads a frame off the stream, then awaits Redis fence validation. When the tick fires inside that await, the future is dropped with the frame it already consumed — the stream then looks idle forever, the forwarding peer waits for an echo that can no longer happen, and the probe times out (the familiar 504). Loss probability per exchange ≈ validation latency ÷ 100 ms, which is why it worsens exactly when Redis is slow — i.e. under full-suite parallelism, matching #2458's "fails under cargo test, passes standalone" signature. Reproduced at 40–80 % locally, on an in-memory stream pair and live iroh endpoints alike, which rules the transport out as the cause.

Changes

• tunnel/reliable.rs — split validate_received out of recv_validated so callers can extract frames cancel-safely and validate outside any select!/timeout region; the cancel-safety contract is documented on both.
• mesh_boot.rs — run_demo_echo now selects only over frame extraction; decoding + fence validation run outside the cancellable region. Drain semantics unchanged.
• buzz-relay-mesh/peer.rs — IrohRecvHalf::recv_frame is now resumable: partial-read state lives in the half rather than the future (each underlying RecvStream::read await is itself cancel-safe), so racing recv_frame cannot desync the stream either.
• buzz-relay-mesh/endpoint.rs — loopback binds disable the iroh portmapper. A loopback-bound socket can never transmit to a WAN mapping, but the gateway probe (UPnP/PCP/NAT-PMP) advertised one as a Portmapped candidate and path attempts to it fail (sendmsg → EADDRNOTAVAIL) — observed live during the investigation on a NAT-PMP-capable network. Production binds (0.0.0.0) are unaffected. Regression test included (trivially green on hosts without a portmap-capable gateway).
• api/mesh_demo.rs — the forwarded-arm round trip now runs over an in-memory MeshStream pair by default: deterministic, fast, and it reproduced the frame loss identically, making it the regression test for the fix. The live-iroh variant is kept as demo_join_forwarded_arm_round_trips_echo_live_mesh under #[ignore] for explicit transport evidence runs.

Verification

build	variant	runs	result
pre-fix	in-memory pair	10	6 fail (504)
pre-fix	live iroh	5	3 fail (504)
post-fix	in-memory pair	15	15 pass
post-fix	live iroh (--ignored)	15	15 pass
post-fix	full cargo test -p buzz-relay --lib	3	744 passed ×3

cargo clippy -p buzz-relay -p buzz-relay-mesh --lib --tests: clean. buzz-relay-mesh suite: 33 passed.

Relation to existing work

• Mitigates/closes the remaining mechanism behind test(relay): mesh_demo echo test flaky under parallel suite — passes standalone, fails full-suite on clean main #2458. The lease-TTL mechanism proven earlier in that thread is real but distinct; fix: project agent mentions, mesh_demo flake, Builderlab sign-in timeout #2498's TTL raise addresses it for the live variant and composes cleanly with this change (this PR deliberately does not touch the TTL line).
• Supersedes my earlier "≤1 s threshold / pre-existing transport mechanism" interpretation posted on test(relay): mesh_demo echo test flaky under parallel suite — passes standalone, fails full-suite on clean main #2458/fix: project agent mentions, mesh_demo flake, Builderlab sign-in timeout #2498 — the observations there were real, but the threshold framing was wrong: single runs were sampling a tick-alignment race. Corrections posted on those threads.

Fixes #2458

🤖 Generated with Claude Code

[mo0ooo]

Fresh latest-main validation: CLEAR.

I re-applied both PR commits (e9e4ad31, eff239c0) without modification on
top of origin/main@c3132c3. The five-file patch applied cleanly.

Reproduction on unmodified current main, with Redis enabled:

• demo_join_forwarded_arm_round_trips_echo repeatedly returned 504 even
  standalone and single-threaded.
• A direct owner receive surfaced the Data frame; the shipped
  run_demo_echo path opened the stream but lost the receive when its 100 ms
  drain tick cancelled recv_validated during Redis fence validation.

The PR's cancellation contract is also supported by the currently locked
transport source: noq::RecvStream::read is documented cancel-safe, while
read_exact is explicitly not. Persisting prefix/body bytes in
IrohRecvHalf and filling them with read is therefore the correct primitive
fix. It also protects audio/join.rs::serve_control_loop, which races raw
recv_frame against nonterminal roster events.

Post-fix verification on Rust 1.95.0, isolated Postgres + Redis:

• cargo fmt --all -- --check: pass
• cargo test --locked -p buzz-relay-mesh -p buzz-relay:
  • relay lib: 994 passed, 0 failed, 88 ignored
  • relay binary: 13 passed
  • relay-mesh: 33 passed
  • doc tests: pass
• live iroh + Redis ignored smoke
  demo_join_forwarded_arm_round_trips_echo_live_mesh: 5/5 passed
• cargo clippy --locked -p buzz-relay-mesh -p buzz-relay --all-targets -- -D warnings: pass

The original Broc Oppler author/DCO sign-off and Claude co-author attribution
remain correct. I recommend merging this PR rather than opening a duplicate
refresh PR; GitHub currently reports it MERGEABLE with review as the remaining
gate.