ep/cxi: async write path with completion-gated control atomics

[fergusfinn]

Summary

Completes the CXI transport in #1: replaces the blocking
post/wait-all/then-atomics design with an asynchronous write path, and fixes
the correctness and robustness gaps found in review and stress testing.
+16–22% on the EP dispatch/combine benchmarks on top of #1, and closes
three review findings (EAGAIN-as-fatal, write→atomic ordering, barrier
carve-out sizing).

• Async write path: fi_writes post without waiting; ring slots retire
  on CQ completion via a per-loop sweep (mirroring the verbs design), with a
  per-transport outstanding cap as backpressure (UCCL_CXI_MAX_OUTSTANDING,
  default 512). UCCL_CXI_SYNC_WRITES=1 restores the previous blocking
  behavior as a fallback.
• Completion-gated control atomics (the load-bearing correctness fix):
  with FI_CXI_RDZV_THRESHOLD=0 every write goes through the rendezvous
  protocol (target pulls the data), so same-TX-context submission order does
  not order an eager atomic's placement against a write's placement —
  immediate injection corrupted dispatch payloads at ≥1024 tokens. Control
  atomics now queue per peer with a threshold equal to the writes posted
  before them and are injected as those writes complete. Standalone atomics
  (no writes in flight) still inject immediately. The empirical
  submission-order results discussed in [Proposal] libfabric-CXI backend for UCCL-EP on HPE Slingshot uccl-project/uccl#956 do not
  transfer to rendezvous-forced environments.
• -FI_EAGAIN is handled by polling the CQ and retrying on every post
  path (write, atomic, barrier) instead of terminating the proxy — TX-queue
  exhaustion under load is expected, not fatal.
• quiet/barrier/shutdown drain outstanding writes and deferred atomics;
  barrier injection orders behind prior traffic to the peer.
• Optional UCCL_CXI_DELIVERY_COMPLETE=1 sets FI_DELIVERY_COMPLETE on TX
  (measured perf-neutral over a 35-minute A/B soak).
• The atomic-buffer sizing assert now accounts for the CXI barrier
  carve-out (top 4 KiB), making the slot region safe by construction.
• New ep/bench/dispatch_loop.py: sustained-dispatch wire-utilization
  measurement tool.

Performance (2×4 GH200, Slingshot-11/CXI, Isambard)

EP8, 4096 tokens, hidden 7168, 288 experts, top-k 8 (fixed chunks; vs the
sync path in #1 measured the same night on the same nodes):

Phase	#1 (sync)	this PR	Δ
FP8 dispatch	32.0 GB/s	38.2 GB/s	+19%
BF16 dispatch	35.6 GB/s	41.4 GB/s	+16%
Combine	40.7 GB/s	43.0 GB/s	+6%

Wire-level: sustained single-stream dispatch reaches 21.15 GB/s per NIC
= 87% of the measured practical ceiling (24.27 GB/s by raw fi_write,
uni- and bidirectional); verified against NIC packet-histogram counters.
The residual is iteration-boundary pipeline fill in the kernel's chunk
staging, which production microbatch overlap hides.

Validation

• Correctness gates: zero-layout guard, internode smoke, full
  test_internode.py correctness (all variants) at 1024 and 4096 tokens.
• Soaks: 35-min HT pressure (83 seeds), LL pressure (12k+ iterations,
  --debug-hash), EAGAIN-torture (UCCL_CXI_MAX_OUTSTANDING=4, full
  throughput, zero failures), sync-mode fallback regression.
• EP16 (4 nodes × 4 GH200): zero-guard, smoke, 4096-token benchmark
  (FP8 21.8 / BF16 22.4 / combine 23.8 GB/s), HT+LL pressure — all green.
• End-to-end vLLM serving (with ep: vLLM serving integration fixes (device selection, disagg proxy, test controls) #2 + ep: CUDA-graph-safe HT internode dispatch+combine (worst-tokens mode) #3 merged in a validation branch):
  DeepSeek-V4-Flash, 2 nodes, DBO, deepep_high_throughput,
  concurrency 1024 — 2048/2048 requests, 7364 out-tok/s, zero EP/CXI
  errors in server logs.

Produced with heavy coding-agent assistance.