From 84622cbbfa72c41cbc4404e66aa4d22b9017e4ba Mon Sep 17 00:00:00 2001 From: jsboige Date: Mon, 19 Oct 2026 22:30:52 +0200 Subject: [PATCH] =?UTF-8?q?feat(vllm):=20bump=20to=20v0.32.0=20=E2=80=94?= =?UTF-8?q?=20KV=20429=20319=20(+1.1%),=20full-window=20400=20eliminated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GO user 19/10 (« Go pour le bump maintenant »). Fenêtre : down ~20:13Z, StartedAt 20:18:09Z, healthy ~20:24Z (~11 min, boot froid sans stall). Gates 11/11 effectifs, FORCING all-200, zéro erreur, config byte-identique. KV 429 319 vs 424 610 (+1,1 %) à config égale. Bench post-boot 928,4 t/s @N=48 (≥ meilleure référence historique) — le delta apparent vs le baseline du jour n'est PAS retenu (téléchargement d'image concurrent + moteur âgé, cf. NOTES). #57035 vérifié en direct : prompt 260 012 + max_tokens 4096 > 262 144 → HTTP 200 (400 en v0.31) — la classe du 400 #431 est éliminée côté moteur. Nouveaux harnais réutilisables : gates_v0320.py, window_full_test.py. Co-Authored-By: Claude Sonnet 5.5 --- myia_vllm/_iteration_log/v0320_bump/NOTES.md | 120 +++++++++++ .../_iteration_log/v0320_bump/gates_v0320.py | 198 ++++++++++++++++++ .../v0320_bump/gates_v0320_1792441610.json | 62 ++++++ .../v0320_bump/window_full_test.py | 81 +++++++ .../docker/profiles/medium-swift15-27b.yml | 7 +- 5 files changed, 465 insertions(+), 3 deletions(-) create mode 100644 myia_vllm/_iteration_log/v0320_bump/NOTES.md create mode 100644 myia_vllm/_iteration_log/v0320_bump/gates_v0320.py create mode 100644 myia_vllm/_iteration_log/v0320_bump/gates_v0320_1792441610.json create mode 100644 myia_vllm/_iteration_log/v0320_bump/window_full_test.py diff --git a/myia_vllm/_iteration_log/v0320_bump/NOTES.md b/myia_vllm/_iteration_log/v0320_bump/NOTES.md new file mode 100644 index 000000000000..6cdf146f6658 --- /dev/null +++ b/myia_vllm/_iteration_log/v0320_bump/NOTES.md @@ -0,0 +1,120 @@ +# v0.31.0 → v0.32.0 image bump (2026-10-19, GO user « Go pour le bump maintenant ») — EN COURS + +- previous image: `vllm/vllm-openai:v0.31.0` (promoted 05/10, PR #72) +- new image: `vllm/vllm-openai:v0.32.0` (published 2026-10-18T06:50Z) +- profile changes (exactly three): `image:`, compile-cache mount `…-v0320`, volume declaration. `…-v0310` and `…-v1` volumes kept intact for rollback. +- rollback: restore the three lines to v0.31.0 + the `…-v0310` volume, then `down && up -d --env-file`. + +## Why (GO user 2026-10-19, veille PR #120) + +For us: WSL2 non-blocking H2D copies (#60215), hybrid+prefix-caching fixes +(#59103, #55601, #60533, #58083 quantized Mamba-2 with TP), faster startup +(shm tensor arena default #51207, daemon-held weights counted in gpu-util +#57298), security (chat-template DoS GHSA #50300, tokenizer #59491, P2P ZMQ +#51504, CORS restricted #61009), and **`max_tokens` no longer reserved from +the context window (#57035)** — the exact class of our 400 on full-window +requests (#431). Breaking changes checked: `mamba_cache_mode="all"` removed +(we run `align`), cascade attention removed (unused), transformers pin is +in-image. Full triage: `v0320_triage/NOTES.md`. + +## Window + +Annonce dashboards ~20:35 locale (fenêtre annoncée AVANT) · baseline bench +`bench_concurrent_scaling.py` sur v0.31 (moteur en service ~5 h, « âgé ») · +pull image (33,1 GB, `befccf0e7e40`) · image v0.32.0 confirmée dans le +conteneur · `down` ~20:13Z · **`StartedAt 20:18:09Z`** · health 200 après +`init engine 215,06 s` (fini **20:23:36Z**) · sidecars up ~20:24Z. +**Downtime total ≈ 11 min** (boot froid sur volume `…-v0320` neuf — AUCUN stall +cette fois : warmup 71,4 s). + +## Post-boot state + +- **`vLLM 0.32.0`** (vérifié dans le conteneur) · **`Using V2 Model Runner`** +- args **byte-identiques** à v0.31 (fp8, offload 24 GiB, batch 8192/seqs 48, + prefix_match_unit 16, mamba fp16, gpu-util 0.70) — diff = 3 lignes de profil only +- **KV cache : 429 319 tokens** (v0.31 : 424 610) → **+4 709 = +1,1 % à config + identique** (premier gain mesuré de la v0.32) +- VRAM **GPU 0 19 359 / GPU 1 18 930 MiB** (bande froide référence 19 937/18 930 + — GPU 0 légèrement SOUS la référence) +- FlashInfer : `kv_cache_dtype=torch.float8_e4m3fn, arch=sm89` résolu au boot +- `shm_broadcast.py` renuméroté (812 → 871) — code changé, comportement identique + +## Battery — 11/11 effectifs ([gates_v0320.py](gates_v0320.py), JSON `gates_v0320_1792441610.json`) + +| Gate | Résultat | +|---|---| +| smoke | PASS after fix (\`thinking off\` — le FAIL initial était un bug de MON gate : `max_tokens=8` + thinking par défaut = budget mangé par le reasoning, piège documenté ; avec thinking off → `PONG`) | +| vision | 4/4 couleurs (`Red Green Blue Yellow`) 0,6 s | +| tool-calling | `get_weather({"city":"Lyon"})` 0,9 s | +| thinking | 154 chars reasoning, finish=stop | +| preserve_thinking | OK (`17`) 0,3 s | +| prefill-30k | 27 580 tok en 14,3 s → **1 931 tok/s** + survie (inclut le JIT du 1ᵉʳ forward) | +| prefill-95k | 89 268 tok en 37,0 s → **2 414 tok/s** + survie | +| prefill-235k | 221 830 tok en 102,0 s → **2 176 tok/s** + survie | +| concurrent N=16 ×2 | **355 → 438 tok/s** au repeat (prefix cache actif) | +| FORCING (batch (4096, 8192]) | 6× ≈4 618 tok = 24 348 tok en 2,9 s, **all-200** | + +Prefill **au-dessus** de la référence v0.31 (1 595-1 608 tok/s @157K, A/B du +08/10) — à revérifier au bench (longueurs de prompt différentes ici). + +## Error scan + +**0 signature** (`Workspace is locked|turboquant|EngineDead|out of memory|Traceback|CUDA error`) +sur tout le segment post-boot. Watchdog v5 : sondes OK dès les premières post-boot. + +## Bench A/B same-window ([bench_concurrent_scaling.py](../bench_concurrent_scaling.py), même script, mêmes prompts, temp 0) + +| N | v0.31 baseline (moteur ~5 h) | v0.32 passe 1 | v0.32 passe 2 | +|---:|---:|---:|---:| +| 1 | 25,9 | 42,1 | 36,6 | +| 2 | 28,6 | 49,6 | 50,8 | +| 4 | 59,3 | 103,6 | 104,3 | +| 8 | 113,8 | 200,7 | 213,5 | +| 16 | 243,0 | 409,8 | **463,2** | +| 24 | 293,7 | 601,9 | 618,9 | +| 32 | 398,5 | 718,1 | 786,4 | +| 48 | 461,8 | 861,0 | **928,4** | + +**Lecture honnête — le « +86 % » apparent n'est PAS attribuable à la v0.32.** +Deux confondants identifiés sur le baseline du jour : +1. **Le téléchargement de l'image v0.32 (33 GB) tournait pendant le bench + baseline** (I/O disque/hôte concurrent) ; +2. moteur âgé de ~5 h (l'effet documenté joue ~±8 %, pas +80 %). + +La référence fraîche historique sur ce type de harnais — **907 t/s à N=48 +(08/10, v0.30, batch 8192 frais)** — suggère que le baseline du jour (461,8) +est anormalement bas. Ce qui est **solidement établi** : la v0.32 à chaud +atteint **928,4 t/s à N=48 et 463,2 à N=16, au-dessus de la meilleure référence +historique** ; le plateau passe 1 → passe 2 confirme le pattern de warm-up +documenté (la passe 1 paie la taxe JIT post-boot). + +**Conséquence** : la promotion v0.32 ne s'appuie PAS sur le delta apparent +contre le baseline du jour (non concluant) mais sur : gates 11/11, config +byte-identique, KV +1,1 %, zéro erreur, et le **niveau de performance post-boot +≥ meilleure référence historique**. + +## #57035 — le 400 de fenêtre-pleine est ÉLIMINÉ (test dédié, [window_full_test.py](window_full_test.py)) + +Cas mesuré, calibré par `/tokenize` : **prompt 260 012 tok + `max_tokens=4096` += 264 108 > 262 144** (la classe exacte du 400 mesuré 3× le 09/10 sur v0.31, +PR #110 — « maximum context length » sans rognage). + +- **v0.31 : HTTP 400.** **v0.32 : HTTP 200 en 17,6 s**, `finish=stop`. +- `max_tokens` n'est plus réservé de la fenêtre (#57035) — confirmé en direct + sur notre serveur, pas seulement dans le changelog. +- Reste licite : un prompt **seul** > 262 144 continuera de donner 400 (vrai + dépassement). La manœuvre de contournement de #431 (route cloud sur 400) + devient donc moins nécessaire — à répercuter sur claudish #431. + +## Verdict: **PROMOTED** (2026-10-19, GO user explicite « Go pour le bump maintenant ») + +Gates 11/11 effectifs, FORCING all-200, zéro signature d'erreur, config +byte-identique, **KV 429 319 (+1,1 %)**, VRAM sous la référence froide, boot +froid 11 min sans stall, bench post-boot **≥ meilleure référence historique +(928,4 @ N=48)**, et #57035 vérifié en direct. Le delta apparent vs le baseline +du jour n'est pas retenu comme preuve (confondants documentés ci-dessus). + +Watch items : cycle de surveillance suivant = premier cycle v0.32 (vérifier +stabilité, sondes, conso VRAM sous trafic) — la promotion est une clairance de +correction + performance, pas encore un soak. **Phase 2 (GO user) : éval +NVFP4-KV sur SM89 — à suivre dans `v0320_nvfp4_kv/`.** \ No newline at end of file diff --git a/myia_vllm/_iteration_log/v0320_bump/gates_v0320.py b/myia_vllm/_iteration_log/v0320_bump/gates_v0320.py new file mode 100644 index 000000000000..58f9b96dd452 --- /dev/null +++ b/myia_vllm/_iteration_log/v0320_bump/gates_v0320.py @@ -0,0 +1,198 @@ +"""v0.32.0 bump gate battery — reproduit la batterie 13 gates des bumps v0.29/v0.30. + +Usage: python gates_v0320.py +Sortie: PASS/FAIL par gate, récapitulatif texte + JSON (gates_v0320_.json). + +Gates: smoke · vision (PNG 4 quadrants généré en stdlib) · tool-calling · +thinking · preserve_thinking · prefill 30k/95k/235k (+ survie) · concurrent N=16 ×2 +(prefix cache) · FORCING (6× ~4.6k tok concurrents → forwards dans (4096, 8192]). +""" +import base64 +import concurrent.futures +import json +import os +import struct +import time +import urllib.request +import zlib + +API = "http://localhost:5002/v1/chat/completions" +MODEL = "qwen3.6-35b-a3b" +KEY = None +for line in open("d:/vllm/myia_vllm/.env"): + if line.startswith("VLLM_API_KEY_MEDIUM="): + KEY = line.split("=", 1)[1].strip() +assert KEY, "VLLM_API_KEY_MEDIUM not found in .env" + + +def post(payload: dict, timeout: int = 600) -> tuple[dict, float]: + req = urllib.request.Request( + API, + data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json", "Authorization": f"Bearer {KEY}"}, + ) + t0 = time.time() + with urllib.request.urlopen(req, timeout=timeout) as r: + body = json.loads(r.read().decode()) + return body, time.time() - t0 + + +def text_of(body: dict) -> str: + try: + return body["choices"][0]["message"]["content"] or "" + except Exception: + return "" + + +def png_4quadrants(w: int = 64, h: int = 64) -> bytes: + """Minimal PNG: 4 solid-color quadrants (TL red, TR green, BL blue, BR yellow).""" + rows = b"" + for y in range(h): + row = b"\x00" + for x in range(w): + tl, tr = x < w // 2, y < h // 2 + px = (255, 0, 0) if (tl and tr) else (0, 255, 0) if tr else (0, 0, 255) if tl else (255, 255, 0) + row += bytes(px) + rows += row + + def chunk(typ: bytes, data: bytes) -> bytes: + return struct.pack(">I", len(data)) + typ + data + struct.pack(">I", zlib.crc32(typ + data) & 0xFFFFFFFF) + + ihdr = struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0) + return b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr) + chunk(b"IDAT", zlib.compress(rows)) + chunk(b"IEND", b"") + + +RESULTS: list[dict] = [] + + +def record(gate: str, ok: bool, detail: str) -> None: + RESULTS.append({"gate": gate, "ok": ok, "detail": detail}) + print(f"[{'PASS' if ok else 'FAIL'}] {gate}: {detail}", flush=True) + + +def fill_text(target_tokens: int) -> str: + # ~4 chars/token English filler; final prompt_tokens read from the response usage. + return ("The quick brown fox jumps over the lazy dog near the riverbank. " * (target_tokens // 11 + 1))[: target_tokens * 4] + + +def main() -> None: + # 1. smoke + try: + body, dt = post({"model": MODEL, "messages": [{"role": "user", "content": "Reply with exactly: PONG"}], "max_tokens": 8, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}) + ok = "PONG" in text_of(body).upper() + record("smoke", ok, f"{dt:.2f}s → {text_of(body).strip()[:40]!r}") + except Exception as e: + record("smoke", False, f"EXC {e}") + return # no point continuing if the engine is dead + + # 2. vision — 4-quadrant PNG + try: + b64 = base64.b64encode(png_4quadrants()).decode() + body, dt = post( + { + "model": MODEL, + "messages": [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}, + {"type": "text", "text": "This image shows 4 solid color quadrants. List the 4 colors in order top-left, top-right, bottom-left, bottom-right. One word each."}, + ], + } + ], + "max_tokens": 200, + "temperature": 0, + "chat_template_kwargs": {"enable_thinking": False}, + } + ) + t = text_of(body).lower() + # French or English color names both accepted + hits = sum(any(w in t for w in words) for words in [("red", "rouge"), ("green", "vert"), ("blue", "bleu"), ("yellow", "jaune")]) + record("vision", hits == 4, f"{hits}/4 colors in {dt:.1f}s → {text_of(body).strip()[:80]!r}") + except Exception as e: + record("vision", False, f"EXC {e}") + + # 3. tool-calling + try: + tools = [{"type": "function", "function": {"name": "get_weather", "description": "Get weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}] + body, dt = post({"model": MODEL, "messages": [{"role": "user", "content": "What is the weather in Lyon right now? Use the tool."}], "tools": tools, "max_tokens": 300, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}) + tc = body["choices"][0]["message"].get("tool_calls") or [] + ok = bool(tc) and tc[0]["function"]["name"] == "get_weather" and "lyon" in tc[0]["function"]["arguments"].lower() + record("tool-calling", ok, f"{dt:.1f}s → {tc[0]['function']['arguments'][:60] if tc else 'no tool_calls'}") + except Exception as e: + record("tool-calling", False, f"EXC {e}") + + # 4. thinking + try: + body, dt = post({"model": MODEL, "messages": [{"role": "user", "content": "A bat and a ball cost 1.10 together. The bat costs 1.00 more than the ball. How much is the ball? Answer briefly."}], "max_tokens": 2000, "temperature": 0}) + reasoning = body["choices"][0]["message"].get("reasoning") or "" + ok = len(reasoning) > 50 and body["choices"][0]["finish_reason"] == "stop" + record("thinking", ok, f"{len(reasoning)} chars reasoning, finish={body['choices'][0]['finish_reason']}") + except Exception as e: + record("thinking", False, f"EXC {e}") + + # 5. preserve_thinking — assistant history keeps its thinking block + try: + msgs = [ + {"role": "user", "content": "Remember: my favorite number is 17."}, + {"role": "assistant", "content": "Sure, I will remember that your favorite number is 17. Is there anything else?"}, + {"role": "user", "content": "What is my favorite number? One word."}, + ] + body, dt = post({"model": MODEL, "messages": msgs, "max_tokens": 300, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}) + ok = "17" in text_of(body) + record("preserve_thinking", ok, f"{dt:.1f}s → {text_of(body).strip()[:50]!r}") + except Exception as e: + record("preserve_thinking", False, f"EXC {e}") + + # 6-8. prefill ladders + survival + for name, target in [("prefill-30k", 31_500), ("prefill-95k", 102_000), ("prefill-235k", 253_500)]: + try: + prompt = fill_text(target) + "\n\nReply with exactly: OK" + body, dt = post({"model": MODEL, "messages": [{"role": "user", "content": prompt}], "max_tokens": 24, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}, timeout=900) + pt = body.get("usage", {}).get("prompt_tokens", 0) + ok = "OK" in text_of(body).upper() + record(name, ok, f"{pt} tok in {dt:.1f}s → {pt / dt:.0f} tok/s; survival={'OK' in text_of(body).upper()}") + except Exception as e: + record(name, False, f"EXC {e}") + + # 9. concurrent N=16 ×2 (distinct prompts; repeat shows prefix-cache warmth) + prompts16 = [f"Explain in 3 sentences why {topic} matters for LLM inference. Be concise." for topic in ["paged attention", "KV cache quantization", "tensor parallelism", "chunked prefill", "speculative decoding", "prefix caching", "CUDA graphs", "continuous batching", "flash attention", "expert parallelism", "MoE routing", "quantization error", "context length", "prefill/decode split", "sampling temperature", "beam search"]] + + def one(p: str) -> float: + body, _ = post({"model": MODEL, "messages": [{"role": "user", "content": p}], "max_tokens": 150, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}, timeout=600) + return body.get("usage", {}).get("completion_tokens", 0) + + for round_name in ["concurrent-N16", "concurrent-N16-repeat"]: + try: + t0 = time.time() + with concurrent.futures.ThreadPoolExecutor(16) as ex: + comps = list(ex.map(one, prompts16)) + dt = time.time() - t0 + record(round_name, all(c > 0 for c in comps), f"{sum(comps)} tokens in {dt:.1f}s → {sum(comps) / dt:.0f} tok/s aggregate") + except Exception as e: + record(round_name, False, f"EXC {e}") + + # 10. FORCING — 6× ~4618-token prompts concurrently → forwards in (4096, 8192] + try: + fp = fill_text(4_618) + "\n\nReply with exactly: OK" + t0 = time.time() + with concurrent.futures.ThreadPoolExecutor(6) as ex: + outs = list(ex.map(lambda _: post({"model": MODEL, "messages": [{"role": "user", "content": fp}], "max_tokens": 8, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}, timeout=600), range(6))) + dt = time.time() - t0 + oks = ["OK" in text_of(b).upper() for b, _ in outs] + tot = sum(b.get("usage", {}).get("prompt_tokens", 0) for b, _ in outs) + record("FORCING", all(oks), f"6×(≈4618 tok) = {tot} tok in {dt:.1f}s, all-200={all(oks)}") + except Exception as e: + record("FORCING", False, f"EXC {e}") + + # summary + n_pass = sum(1 for r in RESULTS if r["ok"]) + print(f"\nBATTERY: {n_pass}/{len(RESULTS)} PASS", flush=True) + out = f"d:/vllm/myia_vllm/_iteration_log/v0320_bump/gates_v0320_{int(time.time())}.json" + with open(out, "w") as f: + json.dump({"ts": int(time.time()), "pass": n_pass, "total": len(RESULTS), "results": RESULTS}, f, indent=2) + print(f"JSON: {out}", flush=True) + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/myia_vllm/_iteration_log/v0320_bump/gates_v0320_1792441610.json b/myia_vllm/_iteration_log/v0320_bump/gates_v0320_1792441610.json new file mode 100644 index 000000000000..9497ff9e57de --- /dev/null +++ b/myia_vllm/_iteration_log/v0320_bump/gates_v0320_1792441610.json @@ -0,0 +1,62 @@ +{ + "ts": 1792441610, + "pass": 10, + "total": 11, + "results": [ + { + "gate": "smoke", + "ok": false, + "detail": "0.48s \u2192 ''" + }, + { + "gate": "vision", + "ok": true, + "detail": "4/4 colors in 0.6s \u2192 'Red\\nGreen\\nBlue\\nYellow'" + }, + { + "gate": "tool-calling", + "ok": true, + "detail": "0.9s \u2192 {\"city\": \"Lyon\"}" + }, + { + "gate": "thinking", + "ok": true, + "detail": "154 chars reasoning, finish=stop" + }, + { + "gate": "preserve_thinking", + "ok": true, + "detail": "0.3s \u2192 '17'" + }, + { + "gate": "prefill-30k", + "ok": true, + "detail": "27580 tok in 14.3s \u2192 1931 tok/s; survival=True" + }, + { + "gate": "prefill-95k", + "ok": true, + "detail": "89268 tok in 37.0s \u2192 2414 tok/s; survival=True" + }, + { + "gate": "prefill-235k", + "ok": true, + "detail": "221830 tok in 102.0s \u2192 2176 tok/s; survival=True" + }, + { + "gate": "concurrent-N16", + "ok": true, + "detail": "1229 tokens in 3.5s \u2192 355 tok/s aggregate" + }, + { + "gate": "concurrent-N16-repeat", + "ok": true, + "detail": "1234 tokens in 2.8s \u2192 438 tok/s aggregate" + }, + { + "gate": "FORCING", + "ok": true, + "detail": "6\u00d7(\u22484618 tok) = 24348 tok in 2.9s, all-200=True" + } + ] +} \ No newline at end of file diff --git a/myia_vllm/_iteration_log/v0320_bump/window_full_test.py b/myia_vllm/_iteration_log/v0320_bump/window_full_test.py new file mode 100644 index 000000000000..ae8fd45f8db9 --- /dev/null +++ b/myia_vllm/_iteration_log/v0320_bump/window_full_test.py @@ -0,0 +1,81 @@ +"""#57035 behaviour test: prompt + max_tokens reservation exceeding max_model_len. + +v0.31 (measured 2026-10-09, PR #110): prompt+max_tokens > 262 144 → HTTP 400 +"maximum context length" WITHOUT trimming. v0.32 changelog: "max_tokens is an +upper bound instead of a reservation, so long prompts are no longer rejected" +(#57035). + +Target case: prompt ≈ 260K tokens (under the window) + max_tokens=4096 → +reservation sum ≈ 264K > 262 144. v0.32 expected: HTTP 200; v0.31 behaviour +would be HTTP 400. Prompt size is calibrated via /tokenize before sending. +""" +import json +import time +import urllib.error +import urllib.request + +API = "http://localhost:5002/v1/chat/completions" +TOKENIZE = "http://localhost:5002/tokenize" +MODEL = "qwen3.6-35b-a3b" +KEY = None +for line in open("d:/vllm/myia_vllm/.env"): + if line.startswith("VLLM_API_KEY_MEDIUM="): + KEY = line.split("=", 1)[1].strip() + +PHRASE = "The quick brown fox jumps over the lazy dog near the riverbank. " # 15 tok +MAX_TOKENS = 4096 +TARGET = 260_000 # prompt tokens target (under 262 144; sum with MAX_TOKENS exceeds it) + + +def count_tokens(prompt: str) -> int: + req = urllib.request.Request( + TOKENIZE, + data=json.dumps({"model": MODEL, "prompt": prompt}).encode(), + headers={"Content-Type": "application/json", "Authorization": f"Bearer {KEY}"}, + ) + with urllib.request.urlopen(req, timeout=120) as r: + return json.loads(r.read().decode())["count"] + + +def build(n_phrases: int, suffix: str) -> str: + return PHRASE * n_phrases + suffix + + +suffix = "\n\nReply with exactly: OK" +n = TARGET // 14 # ~14.0 tok/phrase in-sequence (measured 2026-10-19) +prompt = build(n, suffix) +count = count_tokens(prompt) +print(f"calibration: {n} phrases -> {count} tokens (target ~{TARGET})", flush=True) +for _ in range(4): # proportional refinement loop + if 259_000 <= count <= 261_500: + break + n = max(1, int(n * TARGET / count)) + prompt = build(n, suffix) + count = count_tokens(prompt) + print(f"refined: {n} phrases -> {count} tokens", flush=True) + +print(f"RESERVATION TEST: prompt={count} tok + max_tokens={MAX_TOKENS} = {count + MAX_TOKENS} > 262144 ? {count + MAX_TOKENS > 262144}", flush=True) + +req = urllib.request.Request( + API, + data=json.dumps( + { + "model": MODEL, + "messages": [{"role": "user", "content": prompt}], + "max_tokens": MAX_TOKENS, + "temperature": 0, + "chat_template_kwargs": {"enable_thinking": False}, + } + ).encode(), + headers={"Content-Type": "application/json", "Authorization": f"Bearer {KEY}"}, +) +t0 = time.time() +try: + with urllib.request.urlopen(req, timeout=900) as r: + body = json.loads(r.read().decode()) + usage = body.get("usage", {}) + print(f"HTTP {r.status} in {time.time() - t0:.1f}s") + print(f"prompt_tokens={usage.get('prompt_tokens')} completion_tokens={usage.get('completion_tokens')} finish={body['choices'][0].get('finish_reason')}") +except urllib.error.HTTPError as e: + print(f"HTTP {e.code} in {time.time() - t0:.1f}s") + print(f"body: {e.read().decode()[:600]}") \ No newline at end of file diff --git a/myia_vllm/configs/docker/profiles/medium-swift15-27b.yml b/myia_vllm/configs/docker/profiles/medium-swift15-27b.yml index ace0888ba157..c6837754982f 100644 --- a/myia_vllm/configs/docker/profiles/medium-swift15-27b.yml +++ b/myia_vllm/configs/docker/profiles/medium-swift15-27b.yml @@ -66,7 +66,7 @@ services: vllm-medium-swift15-27b: - image: ${VLLM_IMAGE_SWIFT15:-vllm/vllm-openai:v0.31.0} + image: ${VLLM_IMAGE_SWIFT15:-vllm/vllm-openai:v0.32.0} container_name: myia_vllm-medium-swift15-27b entrypoint: ["/bin/sh", "-c"] command: @@ -157,7 +157,7 @@ services: target: /root/.cache/huggingface bind: create_host_path: false - - vllm-compile-cache-swift15-27b-v0310:/root/.cache/vllm + - vllm-compile-cache-swift15-27b-v0320:/root/.cache/vllm - ${VLLM_LOG_PATH:-./../../../logs}:/logs - ./../../../middleware:/middleware:ro ports: @@ -330,5 +330,6 @@ services: restart: unless-stopped volumes: - vllm-compile-cache-swift15-27b-v0310: + vllm-compile-cache-swift15-27b-v0320: + vllm-compile-cache-swift15-27b-v0310: # kept: rollback volume for v0.31.0 vllm-compile-cache-swift15-27b-v1: # kept: rollback volume for v0.30.0