ep: gate intranode worst-token notify host-counter writes for CUDA-graph replay safety

[fergusfinn]

Summary

Makes the DeepEP high-throughput intranode dispatch CUDA-graph replay-safe
when num_worst_tokens > 0 — the single-node analog of the existing internode
worst-token graph-safety. The num_worst_tokens == 0 path is byte-for-byte
unchanged.

Problem

intranode::notify_dispatch writes the host-pinned receive counters
(moe_recv_counter_mapped, moe_recv_expert_counter_mapped) unconditionally.
In worst-token mode the host never reads these — intranode_prepare returns
num_worst_tokens directly — but the kernel still writes them on every call,
including CUDA-graph replays, which run with no host pacing.

The host-pinned counter is a single location shared by every dispatch on a
Buffer. When a graph-replayed (worst-token) decode dispatch's counter write
lands mid-flight — after a subsequent host-synced eager dispatch has reset the
counter to −1 but before that eager dispatch's own notify kernel completes — the
eager dispatch's CPU spin-wait observes the stale value and breaks early with a
wrong num_recv_tokens. Downstream this surfaces non-deterministically as either
a combine-receiver hang (the receiver waits on a tail for tokens that were never
dispatched, then traps) or an illegal memory access once the wrongly-sized
receive buffer feeds the MoE/attention path.

Fix

Thread num_worst_tokens into intranode::notify_dispatch (kernel, host
wrapper, header, and the intranode_prepare call site) and gate both
host-counter writes on num_worst_tokens == 0, so graph replays never touch the
host-mapped counters and cannot leave a stale value for a later eager dispatch.

Safe because:

• the host reads these counters only on the num_worst_tokens == 0 path;
• the worst-token path already returns num_recv_tokens = num_worst_tokens and
  an empty per-expert list, so nothing consumes the gated writes.

Validation

Single 4×GH200 node, DeepSeek-V4-Flash, EP=4, deepep_high_throughput, full
decode CUDA graphs. Mixed prefill+decode serving benchmark (random dataset,
ISL/OSL 1024/1024, --ignore-eos):

• Without the patch: the engine dies after ~250–670 requests with an illegal
  memory access (surfacing variously at the MoE combine or a downstream GEMM).
• With the patch: 2048/2048 successful requests across back-to-back waves,
  ~9.5k tok/s sustained, no wedge.

[fergusfinn]

Superseded by #18 — the intranode notify gating is folded into the single graph-safe worst-token HT PR (internode + intranode).