Skip to content

[Bugfix][TurboQuant] Add KV quant mode for turboquant - #50533

Merged
mgoin merged 2 commits into
vllm-project:mainfrom
skavulya:skavulya/dev/turboquant_kvquantmode
Jul 31, 2026
Merged

[Bugfix][TurboQuant] Add KV quant mode for turboquant #50533
mgoin merged 2 commits into
vllm-project:mainfrom
skavulya:skavulya/dev/turboquant_kvquantmode

Conversation

@skavulya

@skavulya skavulya commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Purpose

Adds KV quant mode for turboquant KV cache. This fix prevents the KV cache dtype string being set to auto which causes the following failure in TurboQuant observed in #47609 #48177 #48907

vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507  --kv-cache-dtype turboquant_4bit_nc
(EngineCore pid=21869)   File "/tmp/vllm/vllm/v1/worker/gpu_model_runner.py", line 6499, in profile_cudagraph_memory
(EngineCore pid=21869)     self._init_minimal_kv_cache_for_profiling()
(EngineCore pid=21869)   File "/tmp/vllm/vllm/v1/worker/gpu_model_runner.py", line 6383, in _init_minimal_kv_cache_for_profiling
(EngineCore pid=21869)     self.initialize_kv_cache(minimal_config, is_profiling=True)
(EngineCore pid=21869)   File "/tmp/vllm/vllm/v1/worker/gpu_model_runner.py", line 7439, in initialize_kv_cache
(EngineCore pid=21869)     kv_caches = self.initialize_kv_cache_tensors(
(EngineCore pid=21869)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=21869)   File "/tmp/vllm/vllm/v1/worker/gpu_model_runner.py", line 7356, in initialize_kv_cache_tensors
(EngineCore pid=21869)     kv_caches = self._reshape_kv_cache_tensors(
(EngineCore pid=21869)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=21869)   File "/tmp/vllm/vllm/v1/worker/gpu_model_runner.py", line 7201, in _reshape_kv_cache_tensors
(EngineCore pid=21869)     kv_cache_shape = attn_backend.get_kv_cache_shape(
(EngineCore pid=21869)                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=21869)   File "/tmp/vllm/vllm/v1/attention/backends/turboquant_attn.py", line 161, in get_kv_cache_shape
(EngineCore pid=21869)     tq_config = TurboQuantConfig.from_cache_dtype(cache_dtype_str, head_size)
(EngineCore pid=21869)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=21869)   File "/tmp/vllm/vllm/model_executor/layers/quantization/turboquant/config.py", line 222, in from_cache_dtype
(EngineCore pid=21869)     raise ValueError(
(EngineCore pid=21869) ValueError: Unknown TurboQuant cache dtype: 'auto'. Valid presets: turboquant_k8v4, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_3bit_nc

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@skavulya skavulya changed the title [Bugfix][TurboQuant] Add KV quant mode for turboquant KV cache [Bugfix][TurboQuant] Add KV quant mode for turboquant Jul 31, 2026
@mergify mergify Bot added quantization mrv2 Model Runner V2 specific bug Something isn't working labels Jul 31, 2026
Signed-off-by: Soila Kavulya <soila.p.kavulya@intel.com>
Co-authored-by: Claude <noreply@anthropic.com>
@skavulya
skavulya force-pushed the skavulya/dev/turboquant_kvquantmode branch from 8ee65ef to 387e992 Compare July 31, 2026 06:20
@mgoin
mgoin enabled auto-merge (squash) July 31, 2026 15:38
@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 31, 2026
@mgoin
mgoin merged commit aef85ae into vllm-project:main Jul 31, 2026
128 of 131 checks passed
jsboige added a commit to jsboige/vllm that referenced this pull request Aug 20, 2026
…tream stock

Rejoue sur GPU 2 le scénario qui avait tué l'EngineCore le 2026-05-06
(vllm#41726, AssertionError turboquant_attn.py:720:_continuation_prefill),
mais sur une image STOCK (sans l'arbre de patches Genesis).

Image: vllm/vllm-openai:nightly-b22afe45ac797ae58e67a7a3ad79ee5714024420
(v0.26.1rc1.dev542+gb22afe45a). Les 4 bugfixes TurboQuant mergés depuis juin
(vllm-project#44053, vllm-project#47609, vllm-project#39988, vllm-project#50533) sont confirmés ancêtres git de ce commit.

Proxy: cyankiwi/Qwen3.5-9B-AWQ-4bit — même attention hybride GDN+full, même
quantification compressed-tensors AWQ, même model_type qwen3_5. Le bug n'est
pas spécifique au modèle (bissecté upstream sur Qwen3-4B et Llama-3.1-8B).

Résultat: prefill de 31 376 tokens chunké en ~8 morceaux (0 % de cache),
puis requête de survie — les deux aboutissent, zéro trace `Workspace is locked`
/ `turboquant_attn.py` / `EngineDeadError`, en turboquant_k8v4 comme en fp8.
KV 812 373 tokens (19,83x) vs 598 016 en fp8. Débit prefill équivalent à chaud
(~8000 tok/s), mais TurboQuant paie un JIT Triton à froid (328s -> 3,8s).

Réserves documentées: gels intermittents du banc indépendants du dtype (donc
aucune conclusion de perf par dtype au-delà de l'équivalence à chaud), et
nvidia-smi côté hôte aveugle à la mémoire du conteneur sur GPU 2 (le garde-fou
> 500 MiB de run.sh ne protège pas d'un job CoursIA concurrent).

Incident prod concomitant (14:17:34Z -> 15:02:02Z, 3 restarts watchdog) consigné
dans NOTES.md : ni OOM, ni phantom mount, ni RAM épuisée ; hypothèse principale
non prouvée = pagination WDDM avec GPU 0 relevée à 23 386 MiB (seuil 23 000).

La phase 2 (fenêtre prod, TP=2, vrai modèle, image release v0.27.0) reste à faire
et nécessite un GO séparé.

Co-Authored-By: Claude-Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working mrv2 Model Runner V2 specific quantization ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants