[Quant] Add EXL3 Trellis backend for rank-sliced GLM/DeepSeek MoE - #139
brandonmmusic-max wants to merge 19 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughEXL3 quantization support is added for dense and routed-expert layers, including metadata handling, checkpoint loading, CUDA execution, Trellis dispatch, and tests. Structured-output speculative advancement, DeepSeek attention gating, MLA flattening, and optional Docker cache installation are also updated. ChangesEXL3 quantization
Structured-output advancement
DeepSeek attention gating
Build installation control
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant ModelLoader
participant Exl3Config
participant Exl3MoEMethod
participant SparkinferTrellis
participant exllamav3_ext
ModelLoader->>Exl3Config: normalize rank-sliced checkpoint names
Exl3Config->>Exl3MoEMethod: configure EXL3 MoE tensors
Exl3MoEMethod->>SparkinferTrellis: plan rank-sliced dispatch
SparkinferTrellis->>exllamav3_ext: execute Trellis or parity path
exllamav3_ext-->>Exl3MoEMethod: return routed expert outputs
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@vllm/model_executor/layers/quantization/exl3.py`:
- Around line 464-472: The _moe_prefix_is_exl3 method currently hardcodes
projection names, so remapped MoE checkpoints are not detected. Update this
check to use the RoutedExperts layer’s ckpt_gate_proj_name, ckpt_up_proj_name,
and ckpt_down_proj_name values, matching the names consumed by
_validate_codebooks, and thread those names through the method as needed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 87dd2ded-af12-4ca3-992c-dad738c0d7be
📥 Commits
Reviewing files that changed from the base of the PR and between 0d1e91b and a846c2d76ad7e5679166d3818abc9aab8ec4a21a.
📒 Files selected for processing (3)
vllm/model_executor/layers/fused_moe/routed_experts.pyvllm/model_executor/layers/quantization/__init__.pyvllm/model_executor/layers/quantization/exl3.py
3f575f5 to
60c82d9
Compare
brandonmmusic-max
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@vllm/v1/worker/workspace.py`:
- Around line 36-43: Update the use_workspace_lane() docstring with Google-style
Args: documentation for lane and a Raises: section documenting the ValueError
for negative lanes, while preserving the existing behavior and explanatory text.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1692d79f-0266-4848-898d-b40a56b3e55b
📒 Files selected for processing (9)
tests/v1/structured_output/test_reasoning_structured_output.pytests/v1/worker/test_workspace.pyvllm/models/deepseek_v32/nvidia/attention.pyvllm/v1/core/sched/scheduler.pyvllm/v1/structured_output/__init__.pyvllm/v1/worker/gpu/model_runner.pyvllm/v1/worker/gpu/warmup.pyvllm/v1/worker/gpu_worker.pyvllm/v1/worker/workspace.py
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in local-inference-lab#139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/b12x#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The vllm-project#139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact.
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from local-inference-lab#139.
2fdc6d3 to
31ff5a2
Compare
Rebased onto Gilded Gnosis v20 canonical (
|
| Test | v20 | Prior base |
|---|---|---|
| Code generation (Python/HTML, temp 0) | 94–100% valid — reported ~50% syntax-error symptom eliminated | ~50% (reported) |
| LAVD r10/c5 | 10/10 (6 exact, 4 near, 0 fail) | 8/10 (2 fail) |
The LAVD fail count 2 → 0 reflects the v20 canonical MTP fixes plus the flattening fix. Tested image: verdictai/glm52-exl3-sparkinfer:v20-gg6722c1d-si1a88b38-cu132-sm120a.
Re #163: the dual-plan Trellis prefill from #163 is incorporated in this rebase, so #163 can be closed or rebased onto v20 as preferred.
Full results on current base for a full validation sweet a/b for kv cache, set on reasoning high for the PR’s serve.sh, docker compose |Test |nvfp4_ds_mla |fp8 |
|---------------------------|--------------------|--------------------|
|Estonia (133K needle) |PASS 30 / FAIL 0|PASS 30 / FAIL 0|
|LAVD (ledger) |18E/11N/1F — 97%|15E/13N/2F — 93% |
|Hotel-lights (low tier)|15E/15F — 50% |18E/12F — 60% |
|KLD vs BF16 |0.116138 |0.101198 |
|Decode C1 / C8 |87.5 / 308.1 |86.6 / 312.2 |
|Prefill 8k / 128k |2,551 / 1,833 |2,660 / 1,771 |
Full server.sh, docker compose yaml and image and full results. Al run on four in pro 6ks, capped at 300 watts each, so on full workstation or higher power, decode and prefill would be faster. here https://huggingface.co/brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw/blob/main/RELEASE_TEST_SUITE.md
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@vllm/v1/attention/backends/mla/indexer.py`:
- Around line 399-402: Update the B12X activation condition in the relevant
build path around the sparse-indexer capability check to use the canonical
backend-aware predicate, including attention_config.backend == "B12X_MLA_SPARSE"
when the environment variable is unset. Preserve the existing SM120 and next_n
behavior while ensuring the native (B, next_n) path remains disabled for all
supported B12X configurations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e1796822-6dd0-4a09-a332-cead2920cc76
📥 Commits
Reviewing files that changed from the base of the PR and between 2fdc6d3 and 31ff5a2a4c036f6f2b28658fd4d42c4537611c35.
📒 Files selected for processing (14)
docker/Dockerfiletests/quantization/test_exl3.pytests/quantization/test_exl3_prefill_plan.pytests/v1/structured_output/test_reasoning_structured_output.pyvllm/config/model.pyvllm/model_executor/layers/fused_moe/routed_experts.pyvllm/model_executor/layers/quantization/__init__.pyvllm/model_executor/layers/quantization/exl3.pyvllm/model_executor/models/deepseek_v2.pyvllm/model_executor/models/glm4_moe.pyvllm/models/deepseek_v32/nvidia/attention.pyvllm/v1/attention/backends/mla/indexer.pyvllm/v1/core/sched/scheduler.pyvllm/v1/structured_output/__init__.py
🚧 Files skipped from review as they are similar to previous changes (5)
- vllm/v1/core/sched/scheduler.py
- vllm/model_executor/layers/fused_moe/routed_experts.py
- vllm/model_executor/layers/quantization/init.py
- vllm/models/deepseek_v32/nvidia/attention.py
- vllm/v1/structured_output/init.py
|
Credit goes to davidsyoung for prefill improvement |
|
Note on the incorporated prefill work (cross-ref #163): As of the v20 rebase, this branch carries the dual-plan Trellis prefill dispatch verbatim from #163:
so reviewing/merging this PR covers that change completely. #163 stays open only as the isolated review scope against the pre-rebase base ( Validation data for the incorporated change, measured on the pre-v20 stack (pinned digest
The public test image I published earlier ( |
31ff5a2 to
6a1a9f9
Compare
|
@coderabbitai review Fixes are in — the flattening gate now uses the canonical backend-aware |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Target module outline =="
ast-grep outline vllm/v1/attention/backends/mla/indexer.py --items all --type class,function --view expanded || true
echo
echo "== Targeted flattening gate and imports =="
rg -n -C 12 'use_b12x_sparse_indexer|use_flattening|B12X_MLA_SPARSE|VLLM_USE_B12X_SPARSE_INDEXER' \
vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Predicate definition and call sites =="
rg -n -C 8 'def use_b12x_sparse_indexer\b|use_b12x_sparse_indexer\s*\(' vllm --glob '*.py'
echo
echo "== Current PR diff for the relevant module =="
git diff --unified=40 'HEAD^' -- vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Reported commit diff (when present in local history) =="
git show --format=fuller --stat --oneline 6a1a9f9b3a -- vllm/v1/attention/backends/mla/indexer.py || true
git show --format= --unified=40 6a1a9f9b3a -- vllm/v1/attention/backends/mla/indexer.py || trueLength of output: 43841 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== B12X request-detection implementation =="
sed -n '600,665p' vllm/model_executor/layers/sparse_attn_indexer.py
echo
echo "== Backend configuration symbols and B12X selection references =="
rg -n -C 6 'B12X_MLA_SPARSE|_b12x_sparse_indexer_requested|attention_backend' \
vllm/model_executor/layers/sparse_attn_indexer.py \
vllm/v1/attention/backends/mla/indexer.py \
vllm --glob '*.py'
echo
echo "== Direct env-gated B12X branches in the metadata builder =="
sed -n '600,665p' vllm/v1/attention/backends/mla/indexer.py
sed -n '750,805p' vllm/v1/attention/backends/mla/indexer.py
sed -n '910,955p' vllm/v1/attention/backends/mla/indexer.py
sed -n '1080,1145p' vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Existing focused tests for B12X flattening/backend selection =="
rg -n -C 5 'use_flattening|use_b12x_sparse_indexer|B12X_MLA_SPARSE|next_n' \
tests --glob '*.py' | head -n 300Length of output: 142171
Thus, with If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! ✅ Action performedReview finished.
|
Validation — rank-sliced tr3 MTP layer now loads & servesThe two fixes in 2cb2e08 let the MTP draft head load a rank-sliced EXL3 ( Image: Runtime note: a tr3 MTP layer also needs LoadClean load, MTP-3 active. Before the fixes: Throughput (gpu-util 0.96, auto-profiled KV)
Accuracy (unchanged — MTP is lossless)
|
2cb2e08 to
dc0286f
Compare
|
@coderabbitai review Correcting my earlier wording on this comment, which understated the change and could have led to a shallower review. This branch was rebased onto
So please review the rebased state on its merits rather than assuming a no-op rebase — particularly the indexer auto-merge and any interaction between the EXL3 Trellis path and the newer DCP/prefill work. |
|
✅ Action performedReview finished.
|
Rebased for the finalized Gilded Gnosis v20 baseForce-pushed Diff size is unchanged (16 files, +2452/-41) and all 14 commits replayed without conflicts — but this is not a no-op rebase: the branch now sits on 20 newer upstream commits. In particular Also validated against the finalized v20 common basePer the v20 release note, EXL3 is enabled by rebasing the EXL3 source layer onto the pinned stack (the base ships no EXL3/Trellis loader), not by pointing the base tag at an EXL3 checkpoint. So in addition to the branch above, the EXL3 layer was rebased onto the v20-final pins and built:
Of the 12 runtime overlay files, only Validated image: Runtime validation (4x RTX PRO 6000 Blackwell SM120a, TP4/DCP4, MTP-3, NVFP4 DS-MLA KV)
Two notes for the runtime side:
|
|
One CI note: This is the repo's label gate rather than anything in the diff — it was already failing on the previous head ( Also noting CodeRabbit reports |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== PR/base topology and changed files =="
git status --short
git branch --show-current
git merge-base HEAD origin/dev/gilded-gnosis || true
git diff --stat "$(git merge-base HEAD origin/dev/gilded-gnosis)" HEAD
git diff --name-status "$(git merge-base HEAD origin/dev/gilded-gnosis)" HEAD
echo
echo "== Target module structure =="
ast-grep outline vllm/v1/attention/backends/mla/indexer.py --items all --type class,function --view expanded
ast-grep outline vllm/model_executor/layers/quantization/exl3.py --items all --type class,function --view expanded
for f in mla_attention.py b12x_mla_sparse.py b12x_attn.py; do
fd -a -t f "^${f}$" vllm -x ast-grep outline {} --items all --type class,function --view expanded \;
done
echo
echo "== PR diff: indexer auto-merge target =="
base="$(git merge-base HEAD origin/dev/gilded-gnosis)"
git diff --unified=60 "$base" HEAD -- vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Current flattening/alignment context and B12X predicate =="
rg -n -C 18 '_align_block_table_width|use_flattening|use_b12x_sparse_indexer|next_n' \
vllm/v1/attention/backends/mla/indexer.py \
vllm/model_executor/layers/sparse_attn_indexer.py
echo
echo "== EXL3 rank-sliced planning, capture, DCP/prefill, and workspace references =="
rg -n -C 10 'rank_sliced|_apply_rank_sliced|prefill|plan|capture|workspace|dcp|DCP|block_m|Trellis' \
vllm/model_executor/layers/quantization/exl3.py \
vllm/model_executor/models/deepseek_v2.py \
vllm/model_executor/models/glm4_moe.py \
vllm/v1/attention/backends/mla/indexer.py \
vllm/v1/attention/backends/mla \
--glob '*.py'
echo
echo "== Focused test coverage and current diff for relevant tests =="
rg -n -C 8 'align_block_table|use_flattening|use_b12x_sparse_indexer|rank_sliced|prefill.*trellis|Trellis|DCP' \
tests --glob '*.py'
git diff --unified=30 "$base" HEAD -- \
tests/quantization/test_exl3.py \
tests/quantization/test_exl3_prefill_plan.py \
vllm/model_executor/layers/quantization/exl3.pyLength of output: 2675 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Available repository refs and remotes =="
git remote -v
git branch -a
git show -s --format='HEAD %H%nparents %P%nsubject %s' HEAD
git for-each-ref --format='%(refname:short) %(objectname) %(subject)' \
refs/heads refs/remotes | head -n 200
echo
echo "== Pull request metadata =="
gh pr view 139 --repo local-inference-lab/vllm \
--json number,headRefName,headRefOid,baseRefName,baseRefOid,mergeStateStatus \
|| true
echo
echo "== Resolve a usable comparison base and inspect rebase-sensitive paths =="
base=""
for candidate in \
"origin/dev/gilded-gnosis" \
"dev/gilded-gnosis" \
"origin/89b4a98d" \
"89b4a98d"; do
if git rev-parse --verify --quiet "${candidate}^{commit}" >/dev/null; then
base="$(git merge-base HEAD "$candidate")"
echo "Resolved candidate: $candidate -> merge base: $base"
break
fi
done
if [ -z "$base" ]; then
echo "No local base ref was found; showing HEAD source context without a diff."
else
echo "== Changed files and relevant PR diffs =="
git diff --stat "$base" HEAD
git diff --name-status "$base" HEAD
git diff --unified=60 "$base" HEAD -- \
vllm/v1/attention/backends/mla/indexer.py \
vllm/model_executor/layers/quantization/exl3.py \
tests/quantization/test_exl3.py \
tests/quantization/test_exl3_prefill_plan.py
fi
echo
echo "== Module maps =="
ast-grep outline vllm/v1/attention/backends/mla/indexer.py --items all --type class,function --view expanded
ast-grep outline vllm/model_executor/layers/quantization/exl3.py --items all --type class,function --view expanded
echo
echo "== Current indexer alignment/flattening and B12X detection =="
rg -n -C 20 '_align_block_table_width|use_flattening|use_b12x_sparse_indexer|next_n' \
vllm/v1/attention/backends/mla/indexer.py \
vllm/model_executor/layers/sparse_attn_indexer.py
echo
echo "== EXL3 planned dispatch, capture safety, DCP and workspace interactions =="
rg -n -C 12 'def _apply_rank_sliced|def process_weights_after_loading|def apply\(|prefill|capture|workspace|DCP|dcp|plan|block_m|trellis' \
vllm/model_executor/layers/quantization/exl3.py \
vllm/v1/attention/backends/mla \
--glob '*.py'
echo
echo "== Relevant test definitions and assertions =="
rg -n -C 8 'def test_.*(flatten|align|b12x|rank_sliced|prefill|capture)|use_flattening|_align_block_table_width|_apply_rank_sliced' \
tests --glob '*.py'Length of output: 202941 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== GitHub PR base/head and changed-file list =="
gh api repos/local-inference-lab/vllm/pulls/139 \
--jq '{base: (.base.ref + "@" + .base.sha), head: (.head.ref + "@" + .head.sha), mergeable: .mergeable, changed_files: .changed_files}'
gh api --paginate repos/local-inference-lab/vllm/pulls/139/files \
--jq '.[] | [.filename, .status, .additions, .deletions] | `@tsv`'
echo
echo "== Local commits and all available parent diff summaries =="
git show -s --format='HEAD=%H%nparents=%P%nsubject=%s' HEAD
git log --oneline -n 20
git diff --stat HEAD^ HEAD
echo
echo "== Current indexer: imports, alignment helper, and flattening gate =="
rg -n '^(def _align_block_table_width|class DeepseekV32IndexerMetadataBuilder| def |from .*sparse_attn_indexer import|import .*sparse_attn_indexer)' \
vllm/v1/attention/backends/mla/indexer.py
sed -n '1,110p' vllm/v1/attention/backends/mla/indexer.py
sed -n '600,700p' vllm/v1/attention/backends/mla/indexer.py
sed -n '730,830p' vllm/v1/attention/backends/mla/indexer.py
sed -n '1080,1165p' vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Current B12X request predicate =="
rg -n -C 15 'def use_b12x_sparse_indexer|def _b12x_sparse_indexer_requested|B12X_MLA_SPARSE' \
vllm/model_executor/layers/sparse_attn_indexer.py
echo
echo "== EXL3 rank-sliced method boundaries =="
rg -n '^\s*def (_apply_rank_sliced|_ensure_rank_sliced|_plan|_get|apply|process_weights_after_loading)|^\s*class Exl3MoEMethod' \
vllm/model_executor/layers/quantization/exl3.py
rg -n -C 14 'def _apply_rank_sliced|is_current_stream_capturing|capture|prefill_plan|decode_plan|workspace|bind\(' \
vllm/model_executor/layers/quantization/exl3.py
echo
echo "== Exact PR patches from GitHub for the relevant files =="
for path in \
vllm/v1/attention/backends/mla/indexer.py \
vllm/model_executor/layers/quantization/exl3.py \
tests/quantization/test_exl3_prefill_plan.py \
tests/quantization/test_exl3.py; do
echo "--- $path ---"
gh api "repos/local-inference-lab/vllm/pulls/139/files?per_page=100" \
--jq ".[] | select(.filename == \"$path\") | .patch" \
| sed -n '1,260p'
doneLength of output: 59085 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Indexer constructor: final auto-merged context =="
sed -n '337,450p' vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Indexer alignment use sites and focused regression test =="
rg -n -C 10 '_align_block_table_width' \
vllm/v1/attention/backends/mla/indexer.py \
tests/model_executor/layers/test_sparse_attn_indexer_b12x.py
sed -n '1088,1165p' tests/model_executor/layers/test_sparse_attn_indexer_b12x.py
echo
echo "== EXL3 runtime cache key, plan allocation, and all binding paths =="
sed -n '1325,1575p' vllm/model_executor/layers/quantization/exl3.py
sed -n '1570,1708p' vllm/model_executor/layers/quantization/exl3.py
rg -n -C 6 '_RANK_SLICED_RUNTIMES|exl3_trellis_weights|exl3_pointer_tables|exl3_expert_map' \
vllm/model_executor/layers/quantization/exl3.py
echo
echo "== Final MTP-layer loading patch and supporting weight-name/model loading paths =="
git diff --unified=50 HEAD^ HEAD -- \
vllm/model_executor/models/deepseek_mtp.py \
vllm/model_executor/models/utils.py
rg -n -C 12 'normalize_rank_sliced_weight_name|mtp_block|load_weights' \
vllm/model_executor/models/deepseek_mtp.py \
vllm/model_executor/models/deepseek_v2.py \
vllm/model_executor/models/glm4_moe.py \
vllm/model_executor/models/utils.py
echo
echo "== Test collection references for target/draft resource separation and EXL3 MTP loading =="
rg -n -C 5 'target.*draft|draft.*target|rank.sliced|hybrid_tr3_tail|mtp_block|_RANK_SLICED_RUNTIMES' \
tests/quantization/test_exl3.py \
tests/quantization/test_exl3_prefill_plan.py \
tests/model_executor \
--glob '*.py' | head -n 600Length of output: 96012 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Line-numbered EXL3 runtime key and mutable buffers =="
nl -ba vllm/model_executor/layers/quantization/exl3.py | sed -n '1355,1665p'
echo
echo "== All cache lifecycle/ownership references =="
rg -n -C 10 '_RANK_SLICED_RUNTIMES|rank_sliced_runtime|trellis_scratch|prefill_scratch|token_sorted|expert_count' \
vllm tests --glob '*.py'
echo
echo "== Existing model/draft execution and graph resource isolation hooks =="
rg -n -C 8 'draft.*(workspace|scratch|graph|capture)|target.*(workspace|scratch|graph|capture)|(workspace|scratch|graph|capture).*draft' \
vllm tests --glob '*.py' | head -n 800Length of output: 1042 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Line-numbered EXL3 runtime key and mutable-buffer use =="
awk 'NR >= 1355 && NR <= 1665 { printf "%6d %s\n", NR, $0 }' \
vllm/model_executor/layers/quantization/exl3.py
echo
echo "== Cache lifecycle and ownership tests (bounded) =="
rg -n -C 8 '_RANK_SLICED_RUNTIMES|rank_sliced_runtime|trellis_scratch|prefill_scratch|token_sorted|expert_count' \
vllm/model_executor/layers/quantization/exl3.py \
tests/quantization \
tests/model_executor \
--glob '*.py' | head -n 500
echo
echo "== Target/draft graph-resource isolation mechanisms relevant to this runtime =="
rg -n -C 6 'draft.*(workspace|scratch|graph|capture)|target.*(workspace|scratch|graph|capture)|(workspace|scratch|graph|capture).*draft' \
vllm/v1 \
vllm/model_executor \
tests/model_executor \
tests/distributed \
--glob '*.py' | head -n 500Length of output: 82057
The I found one issue in the EXL3 runtime integration:
I did not run CUDA tests in this sandbox. |
|
Superseded by #190. The original head fork rejects maintainer pushes despite |
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in local-inference-lab#139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/b12x#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The vllm-project#139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact. (cherry picked from commit d57d775)
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from local-inference-lab#139. (cherry picked from commit f80a56d)
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in #139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/b12x#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The #139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact. (cherry picked from commit d57d775)
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from #139. (cherry picked from commit f80a56d)
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in local-inference-lab#139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/b12x#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The vllm-project#139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact. (cherry picked from commit d57d775)
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from local-inference-lab#139. (cherry picked from commit f80a56d)
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in local-inference-lab#139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/b12x#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The vllm-project#139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact. (cherry picked from commit d57d775)
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from local-inference-lab#139. (cherry picked from commit f80a56d)
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in local-inference-lab/vllm#139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/b12x#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The #139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact. (cherry picked from commit d57d77534630127e703867d2597a505030079d1b) (cherry picked from commit 37b3443edf9eafd479ee2f56c99c62a452c0c4f4)
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from local-inference-lab/vllm#139. (cherry picked from commit f80a56df14cf2941d8539037392f74c974f8e6f4) (cherry picked from commit eb3b217da759b7efd4bbe508bb3bdfb406ef568c)
4 participants
Summary
Adds native EXL3 (exllamav3 Trellis) loading and execution to Gilded Gnosis
vLLM. The backend serves native rank-sliced checkpoints without reconstructing
or requantizing routed-expert weights at load time.
The implementation provides:
--quantization exl3detection and tensor-storage parsingRelease validation also exposed target/draft lifetime hazards in DCP speculative
execution. This branch now includes:
The Sparkinfer kernel/API dependency is
local-inference-lab/b12x#49.
Rebase and release pins
dev/gilded-gnosisat89b4a98d1ffebb2dda1e1ac5e55238e3a9cfbd5826c4bfdd3ff2be0433e6fe07e0c3be535f5bb318d4438d490691f79022fdfc8149e1c5f161d15445The exact validated image is:
What this rebase actually changed (please review on its merits)
This branch was force-pushed after a rebase onto
dev/gilded-gnosis@89b4a98d(previous head
2cb2e085->dc0286f). The PR's own diff is the same size(16 files, +2452/-41) and all 14 commits replayed without conflicts, but this is
not a no-op rebase -- the branch now sits on 20 newer upstream commits, so
the code these edits apply to has moved. Concretely:
vllm/v1/attention/backends/mla/indexer.pywas modified by both sides.Upstream changed it +31/-2 in that span (including the new
_align_block_table_width), and this PR'suse_flatteningpredicate change(
and not use_b12x_sparse_indexer()) was auto-merged onto that newer file bygit rather than re-authored. Git reported no conflict because the two hunks are
~20 lines apart, but the semantics of the merged result were not human-reviewed --
this is the single highest-value thing to look at in this rebase.
mla_attention.py,b12x_mla_sparse.py,b12x_attn.pyandthe DCP workspace tests in the same span. Those are not in this PR's diff, but the
EXL3 Trellis MoE and indexer paths interact with them at runtime, so behavioural
review of that boundary is welcome.
unchanged.
For traceability: the EXL3 delta carried by this branch is byte-identical to the
tree the validated v23 image was built from (verified per file against each base --
10 changed lines in
deepseek_v2.py, 20 inindexer.py), so the validation resultsbelow correspond to exactly this contribution.
Finalized Gilded Gnosis v20 common base (2026-07-25)
This branch is rebased onto its own base branch for review and merge. It has also
been built and validated as an EXL3 source layer on the finalized v20 common
base. Per the v20 release note the base packages no EXL3/Trellis loader or
kernels, so EXL3 is enabled by rebasing the EXL3 layer onto the pinned stack --
not by pointing the
voipmonitor/vllmtag at an EXL3 checkpoint.voipmonitor/vllm:gilded-gnosis-v20-vllm5517197-sibe0edca-fi801d57a-cu132-20260725551719766029e78824a30d97ae6ac63917405b5fbe0edcaae6f5d284bb29a82325aba7a0ead6960fmodels/deepseek_v2.pyandv1/attention/backends/mla/indexer.pydiffer between the previous and thefinalized v20 tree. Both were re-derived from the v20-final versions with the
EXL3 edits replayed on top, so the new DCP/indexer/prefill work is preserved
rather than overwritten by the overlay.
Validated image:
Runtime validation, 4x RTX PRO 6000 Blackwell (SM120a), TP4/DCP4, MTP-3, NVFP4 DS-MLA KV:
v0.11.2.dev280+gilded.gnosis.v20.vllm5517197.sibe0edca.fi801d57a.cu132.20260725vLLM is using nccl==2.30.4Trellis m=1..32 block_m=8, prefill trellis block_m=64 arena=1054.2MiB capacity=3072 chunk=128 topk=8DCP_*=auto): query split 1,full CKV gather 1, top-k owner merge 1, indexer shards 0, CKV prefetch depth 1,
and
DCP_PREFILL_WORKSPACE=1(VLLM_DCP_PROJECT_BEFORE_MERGE=1+VLLM_B12X_MLA_DCP_GATHER_IN_WORKSPACE=1)Preallocated 30.8 MiB for 2 persistent CKV execution lane(s)andUsing native CKV layer prefetch with depth=1 and 2 workspace slotsthan the pre-fix image because the draft model now gets its own rank-sliced
runtime arena instead of sharing the target's -- the intended cost of the
target/draft scratch isolation.
Earlier images in this series measured LAVD at 2E/3N/0F (pre-fix) and, on one
5-run sample of the first scoped build, 2E/2N/1F; a second sample of that build
returned 2E/3N/0F. The single failure there was a wrong ledger total at 7,114
completion tokens against a 24,576 cap -- an answer-quality miss rather than
truncation, and within this profile's run-to-run spread. This build shows no
failures on either suite.
Note: the base's
Loading safetensors using InstantTensor loaderassertion doesnot appear on this path -- the EXL3 checkpoint is loaded by the EXL3 rank-sliced
loader, not the base InstantTensor path. All other boot assertions hold.
Production validation
Validated on a 332.19 GB GLM-5.2 checkpoint with 256 routed experts and four
RTX PRO 6000 Blackwell 96 GB GPUs.
Validated serving configuration:
Correctness and quality:
0.1124021, sample SD0.00259480.1036666, sample SD0.0018374Exact-image performance:
Release constraint
Asynchronous scheduling remains disabled in the published configuration. The
source-level target/draft isolation fixes are necessary, but hardware testing
still showed a quality regression when async retirement overlapped speculative
side-stream work and the module-global CKV gather state. The validated release
therefore uses synchronous scheduling, MTP1, DCP draft sharding enabled, global
top-k enabled, and
VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE=0.Notes and limits
unsupported shards fail closed to the parity implementation.
retain generic/parity behavior and are not claimed as performance targets.
does not expose a compatible cache builder.
g60c82d972string. Theimage's
ai.verdict.vllm.revisionOCI label records the installed PR head.CI status
CodeRabbit passes. The repository pre-run gate currently blocks the normal
pre-commit job because it requires a
verified,ready, orready-run-all-testslabel (or four prior merged PRs). None of those labelsexists in the repository, and the contributor token cannot create one, so a
maintainer label/configuration action is required to start that job.
AI assistance disclosure
Implementation, debugging, and documentation were developed with Codex and
Claude Code assistance. The author selected the design, reviewed the changes,
and ran the reported hardware validation.
Summary by CodeRabbit
New Features
Bug Fixes