fix(ep): keep D2H handles in device memory

[fergusfinn]

Summary

Allocate the EP runtime's device-read D2H handle arrays (d_handle_objs and the
d_handles pointer table) with plain cudaMalloc + a one-time host-staged
cudaMemcpy, instead of cudaMallocManaged + cudaMemPrefetchAsync +
cudaDeviceSynchronize. These arrays are written once at init and then read hot by
the device on every dispatch (D2HHandle::ring is dereferenced to enqueue
device-to-host commands). The device-visible contents are unchanged; only the
allocation type changes.

Classification

Bug fix / hardening of device-read-hot control state (GH200 managed-memory
corruption).

Mechanism

A cudaMallocManaged table of ring pointers is subject to Unified-Memory page
management (migration, eviction, access-counter heuristics, read-duplication). On
GH200 under sustained load this table has been observed to read back zeroed on
the device while a kernel is reading it hot, turning a handle into a null ring
pointer: the device then enqueues D2H commands into nothing and the host proxy never
receives them — surfacing downstream as a dispatch-side CPU recv timeout, null-pointer
faults in the combine/notify readers, and small-batch illegal accesses. It is a
Heisenbug: any host-side engagement with the pages (host scans, host-pinned mapping,
launch-blocking) suppresses it. cudaMalloc pins the table in HBM and removes the
managed page-management surface entirely.

Validation

Two complementary results:

1. In-situ A/B (this hardware, full serving load): with the handle table in
   managed memory the engine is killed within a few high-concurrency waves; the
   identical code with the table in plain cudaMalloc device memory runs clean across
   repeated waves. The allocation type was the only variable. This is the evidence
   that the change fixes a real under-load failure.

2. Isolated single-GPU probe (this change, in detail): a faithful model of the
   access pattern — a small managed array of ring pointers, device-read-hot inside a
   replayed CUDA graph, checked device-side each cycle, against a cudaMalloc
   control — was stressed with HBM oversubscription, host<->device page-location
   thrashing (60k iterations), concurrent multi-stream readers, and host-engagement
   on/off. The isolated probe did not reproduce the zeroing. This is consistent
   with the Heisenbug: the probe necessarily engages the host each iteration (graph
   launch, periodic device-side checks), and host engagement is exactly what
   suppresses the corruption. The negative isolated result therefore does not weaken
   the in-situ A/B; it matches the documented suppression behaviour.

Verdict

confirmed-fix (in-situ A/B), with the isolated probe consistent via the documented
host-engagement suppression. Keep / land. Device-read-hot, write-once control
state (handle and pointer tables) should not live in managed memory on GH200; this
change moves it to pinned device memory, which is both correct and strictly more
predictable. A fresh single-node graphed-serve A/B of this minimal branch alone
(managed base vs this change) is the one remaining step to re-confirm the extracted
change in isolation end-to-end.

[Copilot]

Pull request overview

This PR updates the EP runtime’s device-read D2H handle storage to live fully in device memory (instead of managed memory + prefetch), aiming to eliminate a suspected source of device-side handle corruption under load.

Changes:

• Switch d_handle_objs and d_handles from cudaMallocManaged to cudaMalloc.
• Build matching host-side staging vectors for handle objects and pointer tables.
• Initialize on host once and copy the fully initialized arrays to device via cudaMemcpy.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.