fix(b12x): make profiling warmup lifetime-safe - #516
Conversation
|
Warning Review limit reachedNext included review available in 48 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
|
Resolve model-provided B12X launch modules after the activation profile and before CUDA graph memory descriptors are captured. Cold B12X compile-cache starts can otherwise initialize NVFP4 MoE CUDA modules between breakable descriptors and trigger an asynchronous illegal-memory-access failure. The call is a no-op outside CUDA SM120 and for models without B12X warmup providers. Existing kernel warmup remains responsible for the complete post-KV-cache warmup sequence. Validated with a four-rank GLM-5.3-Flash-NVFP4 DCP4 cold-cache startup on physical GPUs 4-7, full profiling and production graph capture, one 32k prefill request, and the targeted worker/attention/CUDA-graph test suite. (cherry picked from commit ce66580e4ed96abe8b789b0dda5e08f8ab6813a5)
B12X MoE warmup submits asynchronous kernels against caller-owned input, output, routing, and scratch tensors. Retain every launch allocation until device completion so CUDA graph profiling cannot reuse an address while warmup work still references it. Serving kernel selection, execution plans, and steady-state inference are unchanged. The synchronization applies only during model startup. Validation: the lifetime unit test passes; TP4/DCP4 GLM-5.3-Flash NVFP4 completes three warm-cache starts and one empty-cache start with full piecewise and full CUDA graph capture on physical GPUs 4-7. A 32k prefill smoke request succeeds after the empty-cache start. (cherry picked from commit 7497bd43ddb5adb4d497f24cecf0b8bc698fff99)
b8b7c5a to
3bcb901
Compare
089ce1e to
db0f144
Compare
|
@coderabbitai review |
|
552a1fc
into
fix/cudagraph-profiling-resource-lifetime
2 participants
Purpose
Resolve B12X launch modules before CUDA-graph memory profiling and keep every
caller-owned MoE warmup tensor alive until its asynchronous device work has
completed. An empty B12X kernel cache can otherwise initialize CUDA modules
between graph descriptors or reuse a warmup allocation while a submitted
kernel still references it.
Resulting behavior
CUDA-graph memory descriptors are captured.
torch.accelerator.synchronize().unchanged. The added synchronization executes only during model startup.
Review-stack boundary
This pull request is based on #515 because profiling teardown must preserve
backend resources until graph destruction completes. Its diff contains only
B12X pre-profile warmup ordering and MoE warmup allocation ownership. The
GLM5Next full-CKV implementation is intentionally excluded.
Duplicate-work check
Searches for B12X MoE warmup buffers, CUDA-graph memory profiling warmup, and
asynchronous warmup lifetime found no matching open pull request in
local-inference-lab/vllmorvllm-project/vllm.Validation
tests/model_executor/test_b12x_warmup.py: 14 passed.git diff --check: passed.cache and one start with an empty kernel cache on physical GPUs 4, 5, 6, and
7. Every start completed profiling and production graph capture without
CUDA_LAUNCH_BLOCKING.prompt tok/s and left the service healthy.
AI assistance disclosure
OpenAI Codex assisted with diagnosis, implementation, tests, runtime
qualification, and pull-request preparation. A human maintainer must review
every changed line and understand and defend the behavior before merge.