Skip to content

[Core] Disable JIT warmup in eager mode - #58197

Merged
mgoin merged 2 commits into
mainfrom
mgoin/disable-jit-warmup-enforce-eager
Sep 22, 2026
Merged

mgoin merged 2 commits into
mainfrom
mgoin/disable-jit-warmup-enforce-eager

Conversation

@mgoin

@mgoin mgoin commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Purpose

Make --enforce-eager also disable JIT kernel warmup. Eager mode already opts out of torch compilation and CUDA graph capture, so paying the up-front JIT compilation cost is unexpected and particularly expensive for short-lived processes.

AI assistance was used to investigate the startup regression and prepare this change. The submitter reviewed the final diff.

Test Plan

/home/mgoin/.cache/pre-commit/repoba318kxi/py_env-python3.12/bin/ruff check vllm/config/vllm.py
git diff --check

No model evaluation is needed because this only changes eager-mode startup warmup behavior; model execution and outputs are unchanged.

Test Result

All checks passed!
git diff --check: passed

Signed-off-by: mgoin <mgoin64@gmail.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@LucasWilkinson LucasWilkinson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@mgoin

mgoin commented Sep 22, 2026

Copy link
Copy Markdown
Member Author

/ci run

@mgoin mgoin added ready ONLY add when PR is ready to merge/full CI is needed startup-ux labels Sep 22, 2026
@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #90496 for commit 209f4e2c95fa.

@mgoin
mgoin merged commit 5c3b61e into main Sep 22, 2026
158 checks passed
@mgoin
mgoin deleted the mgoin/disable-jit-warmup-enforce-eager branch September 22, 2026 21:35
vllm-agent pushed a commit to vllm-agent/vllm that referenced this pull request Sep 23, 2026
njhill added a commit to njhill/vllm that referenced this pull request Sep 24, 2026
vllm-project#58197 disabled JIT warmup in eager mode. Fault-tolerant deployments run
peer-fault detection against deadlines, and in-inference Triton
compilation latency spikes can delay detection past the deadline: the
H100 Fault Tolerance E2E job became flaky right after vllm-project#58197 landed
(rank 0 misses the test's 45s detection deadline when the injected
fault trips while kernels are still compiling mid-serving).

Keep enable_jit_warmup on when parallel_config.enable_fault_tolerance
is set, so kernels compile during startup warmup instead of during
serving.

Co-authored-by: Kimi <noreply@moonshot.cn>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
@gau-nernst

Copy link
Copy Markdown
Contributor

Can we discuss this? I think JIT warmup should stay on by default under --enforce-eager. The two flags are orthogonal: e.g. in PD serving, P workers often run with --enforce-eager, but still need JIT warmup (e.g. for CuTeDSL kernels) to avoid compile latency spikes while serving.

Today an explicit --kernel-config.enable-jit-warmup=True is overridden under eager, so it's not possible to opt-in again. The workaround is -cc.mode=none -cc.cudagraph_mode=NONE instead of --enforce-eager, but that's verbose and easy to miss.

Since this is a silent latency regression for deployments, I'd rather default to safe serving behavior and let short-lived processes opt out with --kernel-config.enable_jit_warmup=False (tests/CI can set it). That would revert this change and drop the fault-tolerance special case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed startup-ux

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants