[Bugfix][TurboQuant] Add KV quant mode for turboquant - #50533
Merged
mgoin merged 2 commits intoJul 31, 2026
Conversation
skavulya
requested review from
AndreasKaratzas,
LucasWilkinson,
MatthewBonanni,
WoosukKwon,
heheda12345,
ivanium,
mgoin,
njhill,
pavanimajety,
robertgshaw2-redhat,
yewentao256 and
zyongye
as code owners
July 31, 2026 05:59
4 tasks
Signed-off-by: Soila Kavulya <soila.p.kavulya@intel.com> Co-authored-by: Claude <noreply@anthropic.com>
skavulya
force-pushed
the
skavulya/dev/turboquant_kvquantmode
branch
from
July 31, 2026 06:20
8ee65ef to
387e992
Compare
mgoin
approved these changes
Jul 31, 2026
jsboige
added a commit
to jsboige/vllm
that referenced
this pull request
Aug 20, 2026
…tream stock Rejoue sur GPU 2 le scénario qui avait tué l'EngineCore le 2026-05-06 (vllm#41726, AssertionError turboquant_attn.py:720:_continuation_prefill), mais sur une image STOCK (sans l'arbre de patches Genesis). Image: vllm/vllm-openai:nightly-b22afe45ac797ae58e67a7a3ad79ee5714024420 (v0.26.1rc1.dev542+gb22afe45a). Les 4 bugfixes TurboQuant mergés depuis juin (vllm-project#44053, vllm-project#47609, vllm-project#39988, vllm-project#50533) sont confirmés ancêtres git de ce commit. Proxy: cyankiwi/Qwen3.5-9B-AWQ-4bit — même attention hybride GDN+full, même quantification compressed-tensors AWQ, même model_type qwen3_5. Le bug n'est pas spécifique au modèle (bissecté upstream sur Qwen3-4B et Llama-3.1-8B). Résultat: prefill de 31 376 tokens chunké en ~8 morceaux (0 % de cache), puis requête de survie — les deux aboutissent, zéro trace `Workspace is locked` / `turboquant_attn.py` / `EngineDeadError`, en turboquant_k8v4 comme en fp8. KV 812 373 tokens (19,83x) vs 598 016 en fp8. Débit prefill équivalent à chaud (~8000 tok/s), mais TurboQuant paie un JIT Triton à froid (328s -> 3,8s). Réserves documentées: gels intermittents du banc indépendants du dtype (donc aucune conclusion de perf par dtype au-delà de l'équivalence à chaud), et nvidia-smi côté hôte aveugle à la mémoire du conteneur sur GPU 2 (le garde-fou > 500 MiB de run.sh ne protège pas d'un job CoursIA concurrent). Incident prod concomitant (14:17:34Z -> 15:02:02Z, 3 restarts watchdog) consigné dans NOTES.md : ni OOM, ni phantom mount, ni RAM épuisée ; hypothèse principale non prouvée = pagination WDDM avec GPU 0 relevée à 23 386 MiB (seuil 23 000). La phase 2 (fenêtre prod, TP=2, vrai modèle, image release v0.27.0) reste à faire et nécessite un GO séparé. Co-Authored-By: Claude-Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Adds KV quant mode for turboquant KV cache. This fix prevents the KV cache dtype string being set to auto which causes the following failure in TurboQuant observed in #47609 #48177 #48907
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.