[isytwu]

Why

With the GPU nearly full, creating a hardware queue can stall a process for 30-92 s at a time on amdgpu 6.16 (reproduced on DKMS 6.16.6 and 6.16.13). libhsakmt allocates a queue's context-save area as an SVM range and migrates it into VRAM. The migration holds the process's mmap lock while TTM evicts one of the process's other SVM buffers; that eviction's worker (svm_range_evict_svm_bo_worker) needs the same lock and queues behind the process's other threads' mmap writers (munmap, mprotect, kfd_ioctl_alloc_memory_of_gpu), so nothing moves until the fence wait times out. Every other HIP call in the server backs up behind it, on clr's SDMA engine allocator lock and libhsakmt's FMM lock.

The server created its queues lazily, mid-serving, when sglang has already filled the GPU: a device's first gather streams create its compute queues, and the first copy on each copy engine creates that engine's SDMA queue. SDMA queues get a context-save area too (~173 MiB of VRAM on MI355X), because libhsakmt's handle_concrete_asic skips it only for KFD_IOC_QUEUE_TYPE_SDMA and SDMA_XGMI, not for the SDMA_BY_ENG_ID queues ROCr 7.2 creates (develop still does this). With #799 alone, a TP8 hour stalled for 926 s and 108 s. The first launch of the gather kernel on a device is another such stall: HIP loads the code object then, and with the GPU full that blocked every launch for ~28 s.

Change

• HbmCopyEngine::Prewarm(n), called by the server before it constructs its backend (the host pool's background registration would otherwise slow each queue creation to about a second):

  • creates n pooled gather streams per GPU, which creates the compute queues;
  • runs an empty gather on each: sizes its descriptor staging, uploads the whole buffer (large enough to take a copy engine, as a real batch's upload does) and launches the kernel, which loads its code object;
  • while all of a device's streams are still held, runs a hipMemcpy each way on the null stream, which the fallback copies use.

  With the pool capped at n, at most n gathers and one null-stream copy hold a copy engine at once, and clr gives each the lowest free one, so these copies take exactly the engines the server can be given and create only those SDMA queues (2 + 7 per GPU for n = 8, out of 16 engines).

• Each device's pool is capped at n streams: past that, a gather waits for a stream instead of allocating another one's device memory mid-serving. The gather path holds one stream at a time, so the wait cannot deadlock. Processes that never call Prewarm keep the uncapped pool.

• UMBP_HBM_GATHER_PREWARM_STREAMS sets n: default 8, the TP8 peak of concurrent gathers per GPU; 0 turns the prewarm off. With the gather kernel disabled, the null stream is still warmed.

• Test HbmCopyEngine.GathersRunOnPrewarmedStreams: after Prewarm(3), six concurrent gathers create no stream and copy correctly.

The second commit removes a step of the first: copying once on every copy engine created SDMA queues for all 16 engines, 2.7 GiB of VRAM per GPU, 1.2 GiB of it for engines the server is never given.

Verification

• After Prewarm(8) a process has 5 compute and 2 + 7 SDMA queues per GPU and 2.8 GiB of VRAM (4.0 GiB with an SDMA queue for every engine). Its first null-stream copies in both directions, a first 8192-fragment gather, and 16 threads of gathers (64 and 8192 fragments) mixed with fallback copies in both directions then create no queue and allocate no VRAM. Without the prewarm the same steps create the queues as they go.
• Queue-creation probe (one GPU filled to 256 MiB free, 4 threads allocating and freeing, AQL queues created and destroyed in a loop): 5 of 152 creations took 46 s on DKMS 6.16.6, and 4 of 232 took 46-92 s on 6.16.13. With the GPU empty, the slowest took 28 ms.
• Server emulator on seven GPUs (one client per GPU registering a 2 GiB KV buffer, GPU filled to ~250 MiB free, 16 threads of 8192-range batch puts and gets, twice the TP8 concurrency): without the prewarm the server froze within seconds, with calls taking up to 342 s. With it, the queues stayed the same for the whole run, and after the first half minute no call took over 5 s; the stall left in that first half minute is a KFD eviction of the server's queues as the clients fill their GPUs (below).
• TP8 Kimi-K3 c14 hour on the node that had stalled, with both commits: 1367 requests, 0 errors. The server logged no copy over 1 s all hour, and kept 64 gather streams and 5 + 2 + 7 queues per GPU from start to end, across 861 gRPC threads, 42.9k gathers and 61.8k fallback copies; the prewarm took 1.1 s for 8 GPUs. 6134 tok/s per GPU and a 96.1% overall cache hit rate, against 6247 and 96.1% without this change on the same node (that run stalled for 926 s and 108 s). An earlier hour with the first commit alone was also stall-free (6191 tok/s per GPU).
• test_transfer_engine: 24 tests pass. With UMBP_DRAM_GATHER_KERNEL=0, the two gather tests skip and the rest pass.

Not fixed here: KFD still evicts processes' queues under memory pressure on that node. In each TP8 hour there was one node-wide eviction, which stopped the sglang ranks for 3-6 s each at staggered times and so paused serving for 12 s and 35 s; the server had no copy waiting on it the second time. In the emulator the server's queues were evicted once per run as the clients filled their GPUs, for 0.6-46 s. The KFD lock interaction and the libhsakmt SDMA_BY_ENG_ID context-save area will be reported to ROCm separately.