Repository navigation
[Core] Disable JIT warmup in eager mode - #58197
Conversation
Signed-off-by: mgoin <mgoin64@gmail.com>
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
|
/ci run |
|
✅ Triggered Buildkite CI #90496 for commit |
This reverts commit 5c3b61e.
vllm-project#58197 disabled JIT warmup in eager mode. Fault-tolerant deployments run peer-fault detection against deadlines, and in-inference Triton compilation latency spikes can delay detection past the deadline: the H100 Fault Tolerance E2E job became flaky right after vllm-project#58197 landed (rank 0 misses the test's 45s detection deadline when the injected fault trips while kernels are still compiling mid-serving). Keep enable_jit_warmup on when parallel_config.enable_fault_tolerance is set, so kernels compile during startup warmup instead of during serving. Co-authored-by: Kimi <noreply@moonshot.cn> Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
Can we discuss this? I think JIT warmup should stay on by default under Today an explicit Since this is a silent latency regression for deployments, I'd rather default to safe serving behavior and let short-lived processes opt out with |
Purpose
Make
--enforce-eageralso disable JIT kernel warmup. Eager mode already opts out of torch compilation and CUDA graph capture, so paying the up-front JIT compilation cost is unexpected and particularly expensive for short-lived processes.AI assistance was used to investigate the startup regression and prepare this change. The submitter reviewed the final diff.
Test Plan
No model evaluation is needed because this only changes eager-mode startup warmup behavior; model execution and outputs are unchanged.
Test Result