feat(vllm): bump to v0.32.0 — KV 429 319 (+1.1%), full-window 400 eliminated

myia_vllm/_iteration_log/v0320_bump/NOTES.md

@@ -0,0 +1,120 @@
+# v0.31.0 → v0.32.0 image bump (2026-10-19, GO user « Go pour le bump maintenant ») — EN COURS
+
+- previous image: `vllm/vllm-openai:v0.31.0` (promoted 05/10, PR #72)
+- new image: `vllm/vllm-openai:v0.32.0` (published 2026-10-18T06:50Z)
+- profile changes (exactly three): `image:`, compile-cache mount `…-v0320`, volume declaration. `…-v0310` and `…-v1` volumes kept intact for rollback.
+- rollback: restore the three lines to v0.31.0 + the `…-v0310` volume, then `down && up -d --env-file`.
+
+## Why (GO user 2026-10-19, veille PR #120)
+
+For us: WSL2 non-blocking H2D copies (#60215), hybrid+prefix-caching fixes
+(#59103, #55601, #60533, #58083 quantized Mamba-2 with TP), faster startup
+(shm tensor arena default #51207, daemon-held weights counted in gpu-util
+#57298), security (chat-template DoS GHSA #50300, tokenizer #59491, P2P ZMQ
+#51504, CORS restricted #61009), and **`max_tokens` no longer reserved from
+the context window (#57035)** — the exact class of our 400 on full-window
+requests (#431). Breaking changes checked: `mamba_cache_mode="all"` removed
+(we run `align`), cascade attention removed (unused), transformers pin is
+in-image. Full triage: `v0320_triage/NOTES.md`.
+
+## Window
+
+Annonce dashboards ~20:35 locale (fenêtre annoncée AVANT) · baseline bench
+`bench_concurrent_scaling.py` sur v0.31 (moteur en service ~5 h, « âgé ») ·
+pull image (33,1 GB, `befccf0e7e40`) · image v0.32.0 confirmée dans le
+conteneur · `down` ~20:13Z · **`StartedAt 20:18:09Z`** · health 200 après
+`init engine 215,06 s` (fini **20:23:36Z**) · sidecars up ~20:24Z.
+**Downtime total ≈ 11 min** (boot froid sur volume `…-v0320` neuf — AUCUN stall
+cette fois : warmup 71,4 s).
+
+## Post-boot state
+
+- **`vLLM 0.32.0`** (vérifié dans le conteneur) · **`Using V2 Model Runner`**
+- args **byte-identiques** à v0.31 (fp8, offload 24 GiB, batch 8192/seqs 48,
+  prefix_match_unit 16, mamba fp16, gpu-util 0.70) — diff = 3 lignes de profil only
+- **KV cache : 429 319 tokens** (v0.31 : 424 610) → **+4 709 = +1,1 % à config
+  identique** (premier gain mesuré de la v0.32)
+- VRAM **GPU 0 19 359 / GPU 1 18 930 MiB** (bande froide référence 19 937/18 930
+  — GPU 0 légèrement SOUS la référence)
+- FlashInfer : `kv_cache_dtype=torch.float8_e4m3fn, arch=sm89` résolu au boot
+- `shm_broadcast.py` renuméroté (812 → 871) — code changé, comportement identique
+
+## Battery — 11/11 effectifs ([gates_v0320.py](gates_v0320.py), JSON `gates_v0320_1792441610.json`)
+
+| Gate | Résultat |
+|---|---|
+| smoke | PASS after fix (\`thinking off\` — le FAIL initial était un bug de MON gate : `max_tokens=8` + thinking par défaut = budget mangé par le reasoning, piège documenté ; avec thinking off → `PONG`) |
+| vision | 4/4 couleurs (`Red Green Blue Yellow`) 0,6 s |
+| tool-calling | `get_weather({"city":"Lyon"})` 0,9 s |
+| thinking | 154 chars reasoning, finish=stop |
+| preserve_thinking | OK (`17`) 0,3 s |
+| prefill-30k | 27 580 tok en 14,3 s → **1 931 tok/s** + survie (inclut le JIT du 1ᵉʳ forward) |
+| prefill-95k | 89 268 tok en 37,0 s → **2 414 tok/s** + survie |
+| prefill-235k | 221 830 tok en 102,0 s → **2 176 tok/s** + survie |
+| concurrent N=16 ×2 | **355 → 438 tok/s** au repeat (prefix cache actif) |
+| FORCING (batch (4096, 8192]) | 6× ≈4 618 tok = 24 348 tok en 2,9 s, **all-200** |
+
+Prefill **au-dessus** de la référence v0.31 (1 595-1 608 tok/s @157K, A/B du
+08/10) — à revérifier au bench (longueurs de prompt différentes ici).
+
+## Error scan
+
+**0 signature** (`Workspace is locked|turboquant|EngineDead|out of memory|Traceback|CUDA error`)
+sur tout le segment post-boot. Watchdog v5 : sondes OK dès les premières post-boot.
+
+## Bench A/B same-window ([bench_concurrent_scaling.py](../bench_concurrent_scaling.py), même script, mêmes prompts, temp 0)
+
+| N | v0.31 baseline (moteur ~5 h) | v0.32 passe 1 | v0.32 passe 2 |
+|---:|---:|---:|---:|
+| 1 | 25,9 | 42,1 | 36,6 |
+| 2 | 28,6 | 49,6 | 50,8 |
+| 4 | 59,3 | 103,6 | 104,3 |
+| 8 | 113,8 | 200,7 | 213,5 |
+| 16 | 243,0 | 409,8 | **463,2** |
+| 24 | 293,7 | 601,9 | 618,9 |
+| 32 | 398,5 | 718,1 | 786,4 |
+| 48 | 461,8 | 861,0 | **928,4** |
+
+**Lecture honnête — le « +86 % » apparent n'est PAS attribuable à la v0.32.**
+Deux confondants identifiés sur le baseline du jour :
+1. **Le téléchargement de l'image v0.32 (33 GB) tournait pendant le bench
+   baseline** (I/O disque/hôte concurrent) ;
+2. moteur âgé de ~5 h (l'effet documenté joue ~±8 %, pas +80 %).
+
+La référence fraîche historique sur ce type de harnais — **907 t/s à N=48
+(08/10, v0.30, batch 8192 frais)** — suggère que le baseline du jour (461,8)
+est anormalement bas. Ce qui est **solidement établi** : la v0.32 à chaud
+atteint **928,4 t/s à N=48 et 463,2 à N=16, au-dessus de la meilleure référence
+historique** ; le plateau passe 1 → passe 2 confirme le pattern de warm-up
+documenté (la passe 1 paie la taxe JIT post-boot).
+
+**Conséquence** : la promotion v0.32 ne s'appuie PAS sur le delta apparent
+contre le baseline du jour (non concluant) mais sur : gates 11/11, config
+byte-identique, KV +1,1 %, zéro erreur, et le **niveau de performance post-boot
+≥ meilleure référence historique**.
+
+## #57035 — le 400 de fenêtre-pleine est ÉLIMINÉ (test dédié, [window_full_test.py](window_full_test.py))
+
+Cas mesuré, calibré par `/tokenize` : **prompt 260 012 tok + `max_tokens=4096`
+= 264 108 > 262 144** (la classe exacte du 400 mesuré 3× le 09/10 sur v0.31,
+PR #110 — « maximum context length » sans rognage).
+
+- **v0.31 : HTTP 400.** **v0.32 : HTTP 200 en 17,6 s**, `finish=stop`.
+- `max_tokens` n'est plus réservé de la fenêtre (#57035) — confirmé en direct
+  sur notre serveur, pas seulement dans le changelog.
+- Reste licite : un prompt **seul** > 262 144 continuera de donner 400 (vrai
+  dépassement). La manœuvre de contournement de #431 (route cloud sur 400)
+  devient donc moins nécessaire — à répercuter sur claudish #431.
+
+## Verdict: **PROMOTED** (2026-10-19, GO user explicite « Go pour le bump maintenant »)
+
+Gates 11/11 effectifs, FORCING all-200, zéro signature d'erreur, config
+byte-identique, **KV 429 319 (+1,1 %)**, VRAM sous la référence froide, boot
+froid 11 min sans stall, bench post-boot **≥ meilleure référence historique
+(928,4 @ N=48)**, et #57035 vérifié en direct. Le delta apparent vs le baseline
+du jour n'est pas retenu comme preuve (confondants documentés ci-dessus).
+
+Watch items : cycle de surveillance suivant = premier cycle v0.32 (vérifier
+stabilité, sondes, conso VRAM sous trafic) — la promotion est une clairance de
+correction + performance, pas encore un soak. **Phase 2 (GO user) : éval
+NVFP4-KV sur SM89 — à suivre dans `v0320_nvfp4_kv/`.**

myia_vllm/_iteration_log/v0320_bump/gates_v0320.py

@@ -0,0 +1,198 @@
+"""v0.32.0 bump gate battery — reproduit la batterie 13 gates des bumps v0.29/v0.30.
+
+Usage: python gates_v0320.py
+Sortie: PASS/FAIL par gate, récapitulatif texte + JSON (gates_v0320_<epoch>.json).
+
+Gates: smoke · vision (PNG 4 quadrants généré en stdlib) · tool-calling ·
+thinking · preserve_thinking · prefill 30k/95k/235k (+ survie) · concurrent N=16 ×2
+(prefix cache) · FORCING (6× ~4.6k tok concurrents → forwards dans (4096, 8192]).
+"""
+import base64
+import concurrent.futures
+import json
+import os
+import struct
+import time
+import urllib.request
+import zlib
+
+API = "http://localhost:5002/v1/chat/completions"
+MODEL = "qwen3.6-35b-a3b"
+KEY = None
+for line in open("d:/vllm/myia_vllm/.env"):
+    if line.startswith("VLLM_API_KEY_MEDIUM="):
+        KEY = line.split("=", 1)[1].strip()
+assert KEY, "VLLM_API_KEY_MEDIUM not found in .env"
+
+
+def post(payload: dict, timeout: int = 600) -> tuple[dict, float]:
+    req = urllib.request.Request(
+        API,
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json", "Authorization": f"Bearer {KEY}"},
+    )
+    t0 = time.time()
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        body = json.loads(r.read().decode())
+    return body, time.time() - t0
+
+
+def text_of(body: dict) -> str:
+    try:
+        return body["choices"][0]["message"]["content"] or ""
+    except Exception:
+        return ""
+
+
+def png_4quadrants(w: int = 64, h: int = 64) -> bytes:
+    """Minimal PNG: 4 solid-color quadrants (TL red, TR green, BL blue, BR yellow)."""
+    rows = b""
+    for y in range(h):
+        row = b"\x00"
+        for x in range(w):
+            tl, tr = x < w // 2, y < h // 2
+            px = (255, 0, 0) if (tl and tr) else (0, 255, 0) if tr else (0, 0, 255) if tl else (255, 255, 0)
+            row += bytes(px)
+        rows += row
+
+    def chunk(typ: bytes, data: bytes) -> bytes:
+        return struct.pack(">I", len(data)) + typ + data + struct.pack(">I", zlib.crc32(typ + data) & 0xFFFFFFFF)
+
+    ihdr = struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0)
+    return b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr) + chunk(b"IDAT", zlib.compress(rows)) + chunk(b"IEND", b"")
+
+
+RESULTS: list[dict] = []
+
+
+def record(gate: str, ok: bool, detail: str) -> None:
+    RESULTS.append({"gate": gate, "ok": ok, "detail": detail})
+    print(f"[{'PASS' if ok else 'FAIL'}] {gate}: {detail}", flush=True)
+
+
+def fill_text(target_tokens: int) -> str:
+    # ~4 chars/token English filler; final prompt_tokens read from the response usage.
+    return ("The quick brown fox jumps over the lazy dog near the riverbank. " * (target_tokens // 11 + 1))[: target_tokens * 4]
+
+
+def main() -> None:
+    # 1. smoke
+    try:
+        body, dt = post({"model": MODEL, "messages": [{"role": "user", "content": "Reply with exactly: PONG"}], "max_tokens": 8, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}})
+        ok = "PONG" in text_of(body).upper()
+        record("smoke", ok, f"{dt:.2f}s → {text_of(body).strip()[:40]!r}")
+    except Exception as e:
+        record("smoke", False, f"EXC {e}")
+        return  # no point continuing if the engine is dead
+
+    # 2. vision — 4-quadrant PNG
+    try:
+        b64 = base64.b64encode(png_4quadrants()).decode()
+        body, dt = post(
+            {
+                "model": MODEL,
+                "messages": [
+                    {
+                        "role": "user",
+                        "content": [
+                            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
+                            {"type": "text", "text": "This image shows 4 solid color quadrants. List the 4 colors in order top-left, top-right, bottom-left, bottom-right. One word each."},
+                        ],
+                    }
+                ],
+                "max_tokens": 200,
+                "temperature": 0,
+                "chat_template_kwargs": {"enable_thinking": False},
+            }
+        )
+        t = text_of(body).lower()
+        # French or English color names both accepted
+        hits = sum(any(w in t for w in words) for words in [("red", "rouge"), ("green", "vert"), ("blue", "bleu"), ("yellow", "jaune")])
+        record("vision", hits == 4, f"{hits}/4 colors in {dt:.1f}s → {text_of(body).strip()[:80]!r}")
+    except Exception as e:
+        record("vision", False, f"EXC {e}")
+
+    # 3. tool-calling
+    try:
+        tools = [{"type": "function", "function": {"name": "get_weather", "description": "Get weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
+        body, dt = post({"model": MODEL, "messages": [{"role": "user", "content": "What is the weather in Lyon right now? Use the tool."}], "tools": tools, "max_tokens": 300, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}})
+        tc = body["choices"][0]["message"].get("tool_calls") or []
+        ok = bool(tc) and tc[0]["function"]["name"] == "get_weather" and "lyon" in tc[0]["function"]["arguments"].lower()
+        record("tool-calling", ok, f"{dt:.1f}s → {tc[0]['function']['arguments'][:60] if tc else 'no tool_calls'}")
+    except Exception as e:
+        record("tool-calling", False, f"EXC {e}")
+
+    # 4. thinking
+    try:
+        body, dt = post({"model": MODEL, "messages": [{"role": "user", "content": "A bat and a ball cost 1.10 together. The bat costs 1.00 more than the ball. How much is the ball? Answer briefly."}], "max_tokens": 2000, "temperature": 0})
+        reasoning = body["choices"][0]["message"].get("reasoning") or ""
+        ok = len(reasoning) > 50 and body["choices"][0]["finish_reason"] == "stop"
+        record("thinking", ok, f"{len(reasoning)} chars reasoning, finish={body['choices'][0]['finish_reason']}")
+    except Exception as e:
+        record("thinking", False, f"EXC {e}")
+
+    # 5. preserve_thinking — assistant history keeps its thinking block
+    try:
+        msgs = [
+            {"role": "user", "content": "Remember: my favorite number is 17."},
+            {"role": "assistant", "content": "Sure, I will remember that your favorite number is 17. Is there anything else?"},
+            {"role": "user", "content": "What is my favorite number? One word."},
+        ]
+        body, dt = post({"model": MODEL, "messages": msgs, "max_tokens": 300, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}})
+        ok = "17" in text_of(body)
+        record("preserve_thinking", ok, f"{dt:.1f}s → {text_of(body).strip()[:50]!r}")
+    except Exception as e:
+        record("preserve_thinking", False, f"EXC {e}")
+
+    # 6-8. prefill ladders + survival
+    for name, target in [("prefill-30k", 31_500), ("prefill-95k", 102_000), ("prefill-235k", 253_500)]:
+        try:
+            prompt = fill_text(target) + "\n\nReply with exactly: OK"
+            body, dt = post({"model": MODEL, "messages": [{"role": "user", "content": prompt}], "max_tokens": 24, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}, timeout=900)
+            pt = body.get("usage", {}).get("prompt_tokens", 0)
+            ok = "OK" in text_of(body).upper()
+            record(name, ok, f"{pt} tok in {dt:.1f}s → {pt / dt:.0f} tok/s; survival={'OK' in text_of(body).upper()}")
+        except Exception as e:
+            record(name, False, f"EXC {e}")
+
+    # 9. concurrent N=16 ×2 (distinct prompts; repeat shows prefix-cache warmth)
+    prompts16 = [f"Explain in 3 sentences why {topic} matters for LLM inference. Be concise." for topic in ["paged attention", "KV cache quantization", "tensor parallelism", "chunked prefill", "speculative decoding", "prefix caching", "CUDA graphs", "continuous batching", "flash attention", "expert parallelism", "MoE routing", "quantization error", "context length", "prefill/decode split", "sampling temperature", "beam search"]]
+
+    def one(p: str) -> float:
+        body, _ = post({"model": MODEL, "messages": [{"role": "user", "content": p}], "max_tokens": 150, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}, timeout=600)
+        return body.get("usage", {}).get("completion_tokens", 0)
+
+    for round_name in ["concurrent-N16", "concurrent-N16-repeat"]:
+        try:
+            t0 = time.time()
+            with concurrent.futures.ThreadPoolExecutor(16) as ex:
+                comps = list(ex.map(one, prompts16))
+            dt = time.time() - t0
+            record(round_name, all(c > 0 for c in comps), f"{sum(comps)} tokens in {dt:.1f}s → {sum(comps) / dt:.0f} tok/s aggregate")
+        except Exception as e:
+            record(round_name, False, f"EXC {e}")
+
+    # 10. FORCING — 6× ~4618-token prompts concurrently → forwards in (4096, 8192]
+    try:
+        fp = fill_text(4_618) + "\n\nReply with exactly: OK"
+        t0 = time.time()
+        with concurrent.futures.ThreadPoolExecutor(6) as ex:
+            outs = list(ex.map(lambda _: post({"model": MODEL, "messages": [{"role": "user", "content": fp}], "max_tokens": 8, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}, timeout=600), range(6)))
+        dt = time.time() - t0
+        oks = ["OK" in text_of(b).upper() for b, _ in outs]
+        tot = sum(b.get("usage", {}).get("prompt_tokens", 0) for b, _ in outs)
+        record("FORCING", all(oks), f"6×(≈4618 tok) = {tot} tok in {dt:.1f}s, all-200={all(oks)}")
+    except Exception as e:
+        record("FORCING", False, f"EXC {e}")
+
+    # summary
+    n_pass = sum(1 for r in RESULTS if r["ok"])
+    print(f"\nBATTERY: {n_pass}/{len(RESULTS)} PASS", flush=True)
+    out = f"d:/vllm/myia_vllm/_iteration_log/v0320_bump/gates_v0320_{int(time.time())}.json"
+    with open(out, "w") as f:
+        json.dump({"ts": int(time.time()), "pass": n_pass, "total": len(RESULTS), "results": RESULTS}, f, indent=2)
+    print(f"JSON: {out}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

myia_vllm/_iteration_log/v0320_bump/gates_v0320_1792441610.json

@@ -0,0 +1,62 @@
+{
+  "ts": 1792441610,
+  "pass": 10,
+  "total": 11,
+  "results": [
+    {
+      "gate": "smoke",
+      "ok": false,
+      "detail": "0.48s \u2192 ''"
+    },
+    {
+      "gate": "vision",
+      "ok": true,
+      "detail": "4/4 colors in 0.6s \u2192 'Red\\nGreen\\nBlue\\nYellow'"
+    },
+    {
+      "gate": "tool-calling",
+      "ok": true,
+      "detail": "0.9s \u2192 {\"city\": \"Lyon\"}"
+    },
+    {
+      "gate": "thinking",
+      "ok": true,
+      "detail": "154 chars reasoning, finish=stop"
+    },
+    {
+      "gate": "preserve_thinking",
+      "ok": true,
+      "detail": "0.3s \u2192 '17'"
+    },
+    {
+      "gate": "prefill-30k",
+      "ok": true,
+      "detail": "27580 tok in 14.3s \u2192 1931 tok/s; survival=True"
+    },
+    {
+      "gate": "prefill-95k",
+      "ok": true,
+      "detail": "89268 tok in 37.0s \u2192 2414 tok/s; survival=True"
+    },
+    {
+      "gate": "prefill-235k",
+      "ok": true,
+      "detail": "221830 tok in 102.0s \u2192 2176 tok/s; survival=True"
+    },
+    {
+      "gate": "concurrent-N16",
+      "ok": true,
+      "detail": "1229 tokens in 3.5s \u2192 355 tok/s aggregate"
+    },
+    {
+      "gate": "concurrent-N16-repeat",
+      "ok": true,
+      "detail": "1234 tokens in 2.8s \u2192 438 tok/s aggregate"
+    },
+    {
+      "gate": "FORCING",
+      "ok": true,
+      "detail": "6\u00d7(\u22484618 tok) = 24348 tok in 2.9s, all-200=True"
+    }
+  ]
+}