[noonghunna]

The problem

Turning the speculative-decoding drafter off is the documented mitigation for vllm#50021 — the GDN spec-decode wild write that kills workers under sustained multi-turn traffic. It is referenced as such in DUAL_CARD.md, MULTI_CARD.md, FAQ.md and a dozen compose headers.

On every one of the 27 composes that ship a drafter, that mitigation did not work as documented. The knob had accreted across three generations and was never reconciled:

mode	composes	what actually happened
hardcoded	10	no runtime escape at all — the header told you to delete a line from the YAML
SPEC only	5	SPEC_N=0 silently ignored
SPEC_N only	9	SPEC=off silently ignored
n=0 passthrough	3	flag still sent, with num_speculative_tokens:0

Two more names were in play beyond those: SPEC_N_MAX (3 Gemma composes) and NUM_SPEC_TOKENS (Nemotron).

The silent half is what makes this a safety bug rather than an annoyance. A user who learns SPEC=off on one slug, applies it to another, and sees a healthy server has not mitigated anything. Qwen3.8 is where it surfaced because it is the one model whose slugs span two conventions.

Worst case is vllm/dual — the blessed (qwen3.6-27b, vllm, dual) default. Its own header read:

MITIGATION: run the drafter off — delete the --speculative-config line below.

That is not scriptable, not .env-able, lost on git pull, and it dirties the working tree so the next pull conflicts.

The change

One contract, on all 27:

SPEC_N=0 bash scripts/switch.sh vllm/dual    # drafter off
SPEC_N=6 bash scripts/switch.sh vllm/dual    # depth 6
SPEC=off bash scripts/switch.sh vllm/dual    # alias, still works

• Every compose keeps its own default depth — n=4 where n=4 was measured (the ⚡ Qwen3.8-27B part 2 — the fast tier lands at 105 tok/s, plus NVFP4 #1024 knee), n=3/n=2 elsewhere. Asserted per-file before/after; no default moved.
• SPEC_N_MAX / NUM_SPEC_TOKENS still resolve, so nothing anyone has pinned breaks.
• A non-numeric SPEC_N is a hard boot error, not a silent fall back to "off". A typo can never quietly leave the drafter running, nor quietly disable it.
• The default lives in the entrypoint only — never duplicated into the environment: declaration, so the two cannot drift.

scripts/spec-sweep.sh now uses one variable for every arm including n=0, replacing the two-variable dance and the SPEC-OFF-IGNORED workaround it carried for exactly this bug.

swap_apply.py (brought-weights / c3 ② Serve) recovers the drafter from the sibling's entrypoint now that command: no longer carries it. It patches that script in place rather than rebuilding it — the naive version would have replaced the entrypoint wholesale and silently dropped NVLink detection and patch installs from every brought-weights compose. Verified: vllm/dual → mtp/n=3 and vllm/qwen38-27b-dual-fast → mtp/n=4, both retaining detect_nvlink and their install.sh calls.

The gate: test-spec-toggle-contract.sh (suite 103 → 104)

It does not grep. For each compose it extracts the entrypoint and command:, applies compose interpolation, filters the env exactly as docker does, runs the script with vllm stubbed, and inspects the argv the engine would receive — then parses --speculative-config as JSON. No GPU, no weights, no image.

Seven self-tests run before the real scan, each a shape this repo actually shipped: hardcoded, SPEC-only, SPEC_N-only, n=0 passthrough, undeclared env, malformed payload, plus a passing control. They fail as "the gate itself is broken".

Two bugs the process caught, worth recording

1. A green gate over a dead feature. My first version passed 27/27 while every compose was still broken — it applied env directly to bash, but docker only forwards variables declared under environment:, and 21 composes declared neither knob. A live boot of vllm/dual with SPEC_N=0 logged drafter ON n=3 and exposed it. The gate now reproduces docker's filtering, and a self-test covers it.

2. A payload that looked right and wasn't. 16 composes briefly carried a stray list-item prefix ("- '{...}") that kept a readable token count under a regex while being unparseable JSON — vLLM would have failed at startup. The gate now json.loads the payload and checks its keys.

Both are the same lesson: a check that only ever passes is indistinguishable from no check. Each fix has a negative control that was confirmed to fail before it was trusted.

Verification

• test-spec-toggle-contract — self-test 7/7, 27/27 composes, and confirmed to fail when the bug is reintroduced (all three classes tested individually).
• Full suite green.
• Live, vllm/dual on 2× 3090 — the compose that previously had no escape:
  • SPEC_N=0 → [spec] drafter OFF, no --speculative-config in engine argv, serves
  • default → [spec] drafter ON - mtp n=3, drafter present, serves
• Behaviour at default settings is unchanged on all 27 — this is additive.

Not covered

The llama.cpp family has the same bug and is untouched here. No llama.cpp compose gates on SPEC, so spec-sweep.sh's n=0 arm for that engine has always booted spec-ON (its acceptance guard flags it). Those use a different mechanism (MTP_DRAFT_N_MAX / --spec-draft-model) and I could not verify a change without booting them, so it is documented in place rather than guessed at. Tracked as #1049.

The n=3 vs n=4 question — n=4 is measured on the AutoRound tier, n=3 on fp8/nvfp4 is inherited and unswept — is a separate measurement job, not touched here.