[Kernel][LoRA] Add deterministic split-K=8 LoRA shrink for batch invariance by ShengleiFu · Pull Request #59377 · vllm-project/vllm

ShengleiFu · 2026-09-30T04:54:57Z

Purpose

With VLLM_BATCH_INVARIANT=1 the LoRA shrink kernel runs with SPLIT_K=1, because split-K normally reduces with float atomics. This PR makes the batch-invariant shrink keep K parallelism without atomics: _lora_shrink_kernel stores unscaled FP32 partials for 8 fixed K splits (BLOCK_K=256 for every M) and a small reducer adds them in a fixed order and applies scaling once.

This PR:

adds a STORE_PARTIALS branch to do_shrink_kernel / _lora_shrink_kernel and a fixed-order reducer, used whenever VLLM_BATCH_INVARIANT=1 (no new env var);
allocates the FP32 partials per call with torch.empty, so each CUDA graph keeps its scratch in its own pool;
leaves the non-batch-invariant path unchanged; the split-K path does not use PDL.

Batch-invariant LoRA outputs are still invariant across batch composition, but they change relative to the previous SPLIT_K=1 path because the reduction order differs.

Not a duplicate: #50278 forces split_k=1 on the default (non-BI) path; #58533 changes PDL overlap on the dual-stream path.

Test Plan

pytest tests/lora/test_punica_ops.py -k batch_invariant (FP64 reference over K tail / empty split / 3 slices / no-LoRA rows; bitwise-equal target row across M = 31…129)
pytest tests/lora/test_punica_ops.py -k "test_kernels and shrink" (non-BI path)
lm_eval gsm8k and vllm bench serve, VLLM_BATCH_INVARIANT=1, Qwen3-4B + LoRA adapter, requests targeting the adapter, this PR vs main

Test Result

pytest tests/lora/test_punica_ops.py -k batch_invariant: 6 passed; -k "test_kernels and shrink": 2136 passed (H100 and B200)
Raw lm_eval and vllm bench serve logs (B200 vs main, H100 vs the previous base) are in the comment below.

AI assistance (Claude) was used to write this change; the submitter reviewed every line and ran the e2e evaluations.

🤖 Generated with Claude Code

…riance Under VLLM_BATCH_INVARIANT=1 the LoRA shrink kernel runs with SPLIT_K=1 because split-K normally reduces with float atomics. The opt-in VLLM_LORA_DETERMINISTIC_SPLIT_K=8 keeps K parallelism without atomics: _lora_shrink_kernel stores unscaled FP32 partials for 8 fixed K splits (BLOCK_K=256 for every M) through a STORE_PARTIALS branch in do_shrink_kernel, and a reducer adds the splits in a fixed order and applies scaling once. Partials are allocated per call with torch.empty. The default path (split-K 0, BI on or off) launches exactly as before. S=8 is rejected at import without VLLM_BATCH_INVARIANT=1 or together with VLLM_LORA_ENABLE_DUAL_STREAM=1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shenglei Fu <117230642+ShengleiFu@users.noreply.github.com>

mergify · 2026-09-30T04:55:33Z

Documentation preview: https://vllm--59377.org.readthedocs.build/en/59377/

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

yewentao256

Thanks for the work!

Please do the following:

shrink the diff, idealy < 200 LOC
e2e accuracy using lm_eval...
e2e benchmark using vllm bench serve...

When doing e2e test, please attach with raw output log in case some agents fake the data. DO NOT use AI generated Table here

For example:

"lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k --trust_remote_code"

# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9212|±  |0.0074|
|     |       |strict-match    |     5|exact_match|↑  |0.9212|±  |0.0074|

# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9151|±  |0.0077|
|     |       |strict-match    |     5|exact_match|↑  |0.9158|±  |0.0076|

vllm bench serve --model zai-org/GLM-5.2-FP8  --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16

# Now

============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.08      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              41.50     
Output token throughput (tok/s):         41.50     
Peak output token throughput (tok/s):    76.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          340041.81 
---------------Time to First Token----------------
Mean TTFT (ms):                          1669.15   
Median TTFT (ms):                        1669.02   
P99 TTFT (ms):                           2724.38   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.23      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              39.63     
Output token throughput (tok/s):         39.63     
Peak output token throughput (tok/s):    50.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          324671.20 
---------------Time to First Token----------------
Mean TTFT (ms):                          1668.69   
Median TTFT (ms):                        1649.00   
P99 TTFT (ms):                           3126.43   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

yewentao256 · 2026-09-30T21:13:22Z

+    VLLM_LORA_DETERMINISTIC_SPLIT_K: int = 0
+    """``8`` runs the batch-invariant LoRA shrink as a fixed split-K=8 two-pass
+    kernel (requires ``VLLM_BATCH_INVARIANT=1``); ``0`` (default) disables it."""


Usually we don't want to introduce more envs, could we make it by default?

yewentao256 · 2026-09-30T21:13:32Z

+    "VLLM_LORA_DETERMINISTIC_SPLIT_K": lambda: _parse_lora_deterministic_split_k(
+        os.getenv("VLLM_LORA_DETERMINISTIC_SPLIT_K", "0")
+    ),


Drop VLLM_LORA_DETERMINISTIC_SPLIT_K and its import-time guards: the fixed 8-way split-K shrink is now the batch-invariant LoRA shrink path. It skips PDL, so LoRA dual-stream no longer needs to be rejected. Move the tests into test_punica_ops.py as in-process checks against an FP64 reference and across batch sizes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shenglei Fu <117230642+ShengleiFu@users.noreply.github.com>

mergify · 2026-09-30T22:26:55Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ShengleiFu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

ShengleiFu · 2026-09-30T23:09:19Z

Thanks for the review! Addressed:

Removed VLLM_LORA_DETERMINISTIC_SPLIT_K; the split-K=8 shrink is now the default batch-invariant path.
Diff is down to +263/−10

E2E on H100 80GB, Qwen3-4B + LoRA adapter (nissenj/Qwen3-4B-lora), VLLM_BATCH_INVARIANT=1, this PR vs its base (1085b64); all requests target the adapter. For each bench, the first run warms up the LoRA kernels and the second run is shown.

VLLM_BATCH_INVARIANT=1 vllm serve Qwen3-4B --enable-lora --lora-modules bench-adapter=<adapter> \
  --max-loras 1 --max-lora-rank 16 --max-model-len 2048 --max-num-seqs 32 \
  --no-enable-prefix-caching --gpu-memory-utilization 0.82 --seed 42 --port 8000
# (--max-model-len 9216 for the 8192-token prefill bench)

lm_eval

lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:8000/v1/completions,model=bench-adapter,tokenizer=<Qwen3-4B>,num_concurrent=32,max_gen_toks=256" --tasks gsm8k --num_fewshot 5 --seed 42

# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7961|±  |0.0111|
|     |       |strict-match    |     5|exact_match|↑  |0.7930|±  |0.0112|
# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7930|±  |0.0112|
|     |       |strict-match    |     5|exact_match|↑  |0.7923|±  |0.0112|

vllm bench serve, 128 in / 128 out

vllm bench serve --model Qwen3-4B --lora-modules bench-adapter --port 8000 --dataset-name random \
  --random-input-len 128 --random-output-len 128 --num-prompts 64 --max-concurrency 8 --ignore-eos --seed 42

# Now
  ============ Serving Benchmark Result ============
  Successful requests:                     64
  Failed requests:                         0
  Maximum request concurrency:             8
  Benchmark duration (s):                  17.16
  Total input tokens:                      8192
  Total generated tokens:                  8192
  Request throughput (req/s):              3.73
  Output token throughput (tok/s):         477.35
  Peak output token throughput (tok/s):    515.00
  Peak concurrent requests:                16.00
  Total token throughput (tok/s):          954.70
  ---------------Time to First Token----------------
  Mean TTFT (ms):                          127.25
  Median TTFT (ms):                        129.14
  P99 TTFT (ms):                           144.29
  -----Time per Output Token (excl. 1st token)------
  Mean TPOT (ms):                          15.88
  Median TPOT (ms):                        15.95
  P99 TPOT (ms):                           15.98
  ---------------Inter-token Latency----------------
  Mean ITL (ms):                           15.88
  Median ITL (ms):                         15.89
  P99 ITL (ms):                            18.22
  ==================================================

# main
  ============ Serving Benchmark Result ============
  Successful requests:                     64
  Failed requests:                         0
  Maximum request concurrency:             8
  Benchmark duration (s):                  19.60
  Total input tokens:                      8192
  Total generated tokens:                  8192
  Request throughput (req/s):              3.27
  Output token throughput (tok/s):         418.05
  Peak output token throughput (tok/s):    440.00
  Peak concurrent requests:                16.00
  Total token throughput (tok/s):          836.11
  ---------------Time to First Token----------------
  Mean TTFT (ms):                          120.20
  Median TTFT (ms):                        122.15
  P99 TTFT (ms):                           138.25
  -----Time per Output Token (excl. 1st token)------
  Mean TPOT (ms):                          18.33
  Median TPOT (ms):                        18.33
  P99 TPOT (ms):                           18.38
  ---------------Inter-token Latency----------------
  Mean ITL (ms):                           18.33
  Median ITL (ms):                         18.33
  P99 ITL (ms):                            19.89
  ==================================================

vllm bench serve, 8192 in / 1 out

vllm bench serve --model Qwen3-4B --lora-modules bench-adapter --port 8000 --
  --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --seed 42

# Now
============ Serving Benchmark Result ============
Successful requests:                     128
Failed requests:                         0
Benchmark duration (s):                  27.29
Total input tokens:                      1048576
Total generated tokens:                  128
Request throughput (req/s):              4.69
Output token throughput (tok/s):         4.69
Peak output token throughput (tok/s):    5.00
Peak concurrent requests:                128.00
Total token throughput (tok/s):          38427.01
---------------Time to First Token----------------
Mean TTFT (ms):                          13749.14
Median TTFT (ms):                        13737.77
P99 TTFT (ms):                           26979.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00
Median TPOT (ms):                        0.00
P99 TPOT (ms):                           0.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P99 ITL (ms):                            0.00
==================================================
# main
============ Serving Benchmark Result ============
Successful requests:                     128
Failed requests:                         0
Benchmark duration (s):                  27.96
Total input tokens:                      1048576
Total generated tokens:                  128
Request throughput (req/s):              4.58
Output token throughput (tok/s):         4.58
Peak output token throughput (tok/s):    5.00
Peak concurrent requests:                128.00
Total token throughput (tok/s):          37504.56
---------------Time to First Token----------------
Mean TTFT (ms):                          14101.88
Median TTFT (ms):                        14097.32
P99 TTFT (ms):                           27649.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00
Median TPOT (ms):                        0.00
P99 TPOT (ms):                           0.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P99 ITL (ms):                            0.00
==================================================

B200, this PR vs main (`8fb16ea`)

lm_eval

lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:8000/v1/completions,model=bench-adapter,tokenizer=<Qwen3-4B>,num_concurrent=32,max_gen_toks=256" --tasks gsm8k --num_fewshot 5 --seed 42

# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7938|±  |0.0111|
|     |       |strict-match    |     5|exact_match|↑  |0.7930|±  |0.0112|

# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7877|±  |0.0113|
|     |       |strict-match    |     5|exact_match|↑  |0.7885|±  |0.0112|

vllm bench serve, 128 in / 128 out

vllm bench serve --model Qwen3-4B --lora-modules bench-adapter --port 8000 --dataset-name random \
  --random-input-len 128 --random-output-len 128 --num-prompts 64 --max-concurrency 8 --ignore-eos --seed 42

# Now
============ Serving Benchmark Result ============
Successful requests:                     64
Failed requests:                         0
Maximum request concurrency:             8
Benchmark duration (s):                  15.77
Total input tokens:                      8192
Total generated tokens:                  8192
Request throughput (req/s):              4.06
Output token throughput (tok/s):         519.62
Peak output token throughput (tok/s):    561.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          1039.24
---------------Time to First Token----------------
Mean TTFT (ms):                          159.06
Median TTFT (ms):                        155.13
P99 TTFT (ms):                           188.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.25
Median TPOT (ms):                        14.25
P99 TPOT (ms):                           14.31
---------------Inter-token Latency----------------
Mean ITL (ms):                           14.25
Median ITL (ms):                         14.25
P99 ITL (ms):                            18.66
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     64
Failed requests:                         0
Maximum request concurrency:             8
Benchmark duration (s):                  18.50
Total input tokens:                      8192
Total generated tokens:                  8192
Request throughput (req/s):              3.46
Output token throughput (tok/s):         442.76
Peak output token throughput (tok/s):    472.00
Peak concurrent requests:                16.00
Total token throughput (tok/s):          885.53
---------------Time to First Token----------------
Mean TTFT (ms):                          147.17
Median TTFT (ms):                        142.94
P99 TTFT (ms):                           197.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.04
Median TPOT (ms):                        17.04
P99 TPOT (ms):                           17.10
---------------Inter-token Latency----------------
Mean ITL (ms):                           17.04
Median ITL (ms):                         17.03
P99 ITL (ms):                            20.58
==================================================

vllm bench serve, 8192 in / 1 out

vllm bench serve --model Qwen3-4B --lora-modules bench-adapter --port 8000 --dataset-name random \
  --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --seed 42

# Now
============ Serving Benchmark Result ============
Successful requests:                     128
Failed requests:                         0
Benchmark duration (s):                  27.31
Total input tokens:                      1048576
Total generated tokens:                  128
Request throughput (req/s):              4.69
Output token throughput (tok/s):         4.69
Peak output token throughput (tok/s):    6.00
Peak concurrent requests:                128.00
Total token throughput (tok/s):          38404.04
---------------Time to First Token----------------
Mean TTFT (ms):                          13921.23
Median TTFT (ms):                        13917.67
P99 TTFT (ms):                           27080.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00
Median TPOT (ms):                        0.00
P99 TPOT (ms):                           0.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P99 ITL (ms):                            0.00
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     128
Failed requests:                         0
Benchmark duration (s):                  27.56
Total input tokens:                      1048576
Total generated tokens:                  128
Request throughput (req/s):              4.64
Output token throughput (tok/s):         4.64
Peak output token throughput (tok/s):    6.00
Peak concurrent requests:                128.00
Total token throughput (tok/s):          38052.37
---------------Time to First Token----------------
Mean TTFT (ms):                          14051.87
Median TTFT (ms):                        14045.95
P99 TTFT (ms):                           27383.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00
Median TPOT (ms):                        0.00
P99 TPOT (ms):                           0.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P99 ITL (ms):                            0.00
==================================================

…nistic-splitk # Conflicts: # docs/features/batch_invariance.md

yewentao256

LGTM, thanks for the work!

Nit: I think it is better to put batch invariance tests in determinism tests folder

github-actions · 2026-10-02T20:28:57Z

✅ @ShengleiFu, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

mergify · 2026-10-02T20:36:41Z

Hi @ShengleiFu, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

…nistic-splitk

…rminism Run them in the H100 and B200 batch-invariance steps and trigger those steps on vllm/lora/ops/triton_ops changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shenglei Fu <117230642+ShengleiFu@users.noreply.github.com>

yewentao256 · 2026-10-03T14:43:58Z

/ci run

github-actions · 2026-10-03T14:44:17Z

❌ This PR is 15 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

yewentao256 · 2026-10-03T14:44:29Z

/ci run

github-actions · 2026-10-03T14:44:48Z

✅ Triggered Buildkite CI #92763 for commit 3ffd13b3592b.

vllm-agent · 2026-10-03T14:54:39Z

CI selector (shadow): 33 test steps (50 jobs) instead of 54 (76 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

steps (jobs)	Today's rules	Selector	Would skip	Would add
NVIDIA, CPU and others	54 (76)	33 (50)	25 (35)	4 (9)
AMD mirrors	46 (63)	27 (44)	23 (28)	4 (9)

Selector would run (33)

async-engine-inputs-utils-worker
basic-correctness-sleep-mode
batch-invariance-b200
batch-invariance-h100
cpu-multimodal-config
cpu-params-env-tokenizers-parser
cpu-reasoning-renderers
cpu-tool-parsers
distributed-model-tests-2-gpus ×3
engine-1-gpu
entrypoints-integration-api-server ×4
entrypoints-integration-api-server-generate
entrypoints-integration-api-server-openai-chat_completion
entrypoints-integration-api-server-openai-completion
entrypoints-integration-multimodal
entrypoints-integration-speech_to_text
examples
extract-hidden-states-integration
extract-hidden-states-integration-2-gpus
lora ×4
lora-tp-distributed ×4
metrics-tracing-2-gpus
model-executor
regression
spec-decode-draft-model ×4
v1-core
v1-executor-worker
v1-kv-connectors ×4
v1-kv-offload
v1-logits-oracle
v1-metrics-lmeval
v1-others-cpu
v1-sample

Would skip (today's rules run them) (25)

ascend-npu-test
basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-models-test-other-cpu
basic-models-tests-initialization
basic-models-tests-other
benchmarks-cli-test
cpu-language-generation-and-pooling-model-tests ×3
entrypoints-integration-llm
entrypoints-integration-pooling
entrypoints-integration-responses-api
kernels-fla-ops-test-b200
kernels-mhc-test-b200
kernels-root-misc-test-b200
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
multi-modal-models-standard-4-other-whisper
multi-modal-processor ×4
multi-modal-processor-cpu ×4

Would add (today's rules do not run them) (4)

distributed-model-tests-2-gpus ×3 (Python record)
engine-1-gpu (code map)
model-executor (Python record)
spec-decode-draft-model ×4 (Python record)

AMD mirrors: would skip (23)

basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-models-tests-initialization
basic-models-tests-other
benchmarks-cli-test
entrypoints-integration-llm
entrypoints-integration-pooling
entrypoints-integration-responses-api
kernels-fla-ops-test-b200
kernels-mhc-test-b200
kernels-root-misc-test-b200
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
multi-modal-models-standard-4-other-whisper
multi-modal-processor ×4
platform-tests
pytorch-compilation-passes-unit-tests

AMD mirrors: would add (4)

distributed-model-tests-2-gpus ×3 (Python record)
engine-1-gpu (code map)
model-executor (Python record)
spec-decode-draft-model ×4 (Python record)

_{5 changed files · base e319f86f15 · head 3ffd13b359 · Python record: build 92706 at 6e517b15c1 · kernel record: table 6e517b1 (build 92706), map 6e517b1 · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 15 optional steps the selector would also run}

ShengleiFu · 2026-10-03T22:03:40Z

Hi @yewentao256 , it seems the CI failures are not related to this pr, could you help take a look and see if we can merge it, thanks!

yewentao256 · 2026-10-03T22:32:35Z

/ci retry

github-actions · 2026-10-03T22:32:54Z

✅ Queued 3 failed job(s) for retry in Buildkite CI #92763.

ShengleiFu · 2026-10-04T20:12:09Z

/ci retry

github-actions · 2026-10-04T20:12:27Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #92763.

mergify Bot added the documentation Improvements or additions to documentation label Sep 30, 2026

ShengleiFu marked this pull request as ready for review September 30, 2026 20:55

ShengleiFu requested a review from jeejeelee as a code owner September 30, 2026 20:55

claude Bot reviewed Sep 30, 2026

View reviewed changes

yewentao256 reviewed Sep 30, 2026

View reviewed changes

mergify Bot added the needs-rebase label Sep 30, 2026

Merge remote-tracking branch 'upstream/main' into lora-shrink-determi…

e78684e

…nistic-splitk # Conflicts: # docs/features/batch_invariance.md

mergify Bot removed the needs-rebase label Sep 30, 2026

yewentao256 approved these changes Oct 2, 2026

View reviewed changes

yewentao256 added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 2, 2026

ShengleiFu and others added 2 commits October 2, 2026 13:58

Merge remote-tracking branch 'upstream/main' into lora-shrink-determi…

231770c

…nistic-splitk

ShengleiFu requested review from Harry-Chen and khluu as code owners October 2, 2026 21:12

mergify Bot added the ci/build label Oct 2, 2026

Merge branch 'main' into lora-shrink-deterministic-splitk

3ffd13b

yewentao256 merged commit 1388100 into vllm-project:main Oct 4, 2026
140 checks passed

joanvelja mentioned this pull request Oct 13, 2026

[Bug]: DP > 1 + MoE expert LoRA: each DP replica applies its own LoRA state to tokens gathered from all replicas (dropped or cross-applied expert delta) #61453

Open

Conversation

ShengleiFu commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

mergify Bot commented Sep 30, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

yewentao256 left a comment

Choose a reason for hiding this comment

Uh oh!

yewentao256 Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

yewentao256 Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Sep 30, 2026

Uh oh!

ShengleiFu commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

B200, this PR vs main (8fb16ea)

Uh oh!

yewentao256 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

mergify Bot commented Oct 2, 2026

Uh oh!

yewentao256 commented Oct 3, 2026

Uh oh!

github-actions Bot commented Oct 3, 2026

Uh oh!

yewentao256 commented Oct 3, 2026

Uh oh!

github-actions Bot commented Oct 3, 2026

Uh oh!

vllm-agent commented Oct 3, 2026

CI selector (shadow): 33 test steps (50 jobs) instead of 54 (76 jobs)

Uh oh!

ShengleiFu commented Oct 3, 2026

Uh oh!

yewentao256 commented Oct 3, 2026

Uh oh!

github-actions Bot commented Oct 3, 2026

Uh oh!

ShengleiFu commented Oct 4, 2026

Uh oh!

github-actions Bot commented Oct 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ShengleiFu commented Sep 30, 2026 •

edited

Loading

ShengleiFu commented Sep 30, 2026 •

edited

Loading

B200, this PR vs main (`8fb16ea`)