[noonghunna]

The header block asserting an "accumulated-ctx decode cliff" was retracted on 2026-09-08 — but only in learnings/. It survived verbatim in all 20 qwen3.8-27b vLLM composes, which is where operators actually read it.

There was never a cliff

bench-agentic set TTFT on the first content delta. Every model in that matrix is a thinking model, so the server streams reasoning_content deltas first — the entire reasoning phase was charged to prefill, the decode window collapsed below the harness's own 5%-of-wall "degenerate" threshold, and it printed single-block emission … wall 2.0 tok/s. Fixed in #1218.

Measured live at ~21K ctx while the old harness was calling it collapsed:

first reasoning delta	28.64 s — 278 deltas, streaming normally
first content delta	34.69 s
last delta	35.95 s
old TTFT / window	34.69 s / 1.25 s = 3.5% of wall → FLAGGED
true window	7.31 s = 20% → fine

That rule flags whatever has slow prefill relative to decode, which also explains the "hardware severity split" the block reported: Blackwell prefills fast → large ratio → "clean"; Ampere + CPU offload → prefill-dominated wall → "collapsed". A denominator effect, not a hardware property.

Corrected re-test

Both arms, same v0.27.1 pin, cross-checked against vLLM's own histograms (vllm:request_generation_tokens_sum / vllm:request_decode_time_seconds_sum — recorded in-engine, immune to parser cost or SSE coalescing):

slug	recorded threshold	client	engine
qwen38-27b-dual-max (MTP n=3)	collapse ~21k → 1.5–6 tok/s	110.2 @ 21,443	109.2
qwen38-27b-dual-supermax (DFlash2 n=7)	collapse ~12k → 4–6 tok/s	159.4 @ 12,275	154.1

Decode flat across 1.2K → 35.3K on both arms, zero degenerate turns, zero preemptions, two instruments agreeing within 1–2%.

Why this is more than tidying

The block told operators:

Interim: MTP tier (higher cliff) or SPEC_N=0 for sustained >~20k

That is advice to throw away the drafter to dodge a problem that does not exist — on the tier whose entire purpose is the drafter.

What survives is kept, not deleted

vllm#35387 is a real spec tax that scales with drafter depth, not context (29K: no-spec 57.0 → n=1 40.6 / n=2 36.0 / n=3 32.7 tok/s), profiled to the cudagraph downgrade FULL_AND_PIECEWISE → PIECEWISE. The replacement text carries it, framed as a steady tax rather than a cliff.

Full guard suite green: 150 pass / 0 fail.

🤖 Generated with Claude Code

[sudeposutemizligi]

hello, i had some difficulties to keep up with the latest progress in the repo, but as far as i understand from above writings, it is said that there's no decode tps degradation at long ctx but it was a bench misread.

but the situation is different I guess . (or I really do smt wrong) thats why i made a small screen recording of a deepseek harness using vllm 0271 with all patches and settings done from the repo compose, running as a standalone server.
the ctx is set at ~200k and the context is %70 in the harness.
you can see in that screen recording, words are coming like 10 tps.

#1052 (comment)

video and config is in this comment .

what i think is: if i had made smt wrong it wouldn't pass the agentic bench. and if its not degrading, i wouldn't see the slow typing as in the video. this is a daily usage with file reads, tool uses, web searches etc.

if these mean anything i would happily make further tests you point out.

Thank youuu ❤️🤲

[noonghunna]

Thank you for this — the screen recording plus a full config plus a bench table is the most useful
shape a report can take, and you have not done anything wrong. Let me separate what we retracted
from what we did not, because the retraction was narrower than it probably reads.

What #1096 retracted, precisely

We retracted the cliff — the claim that there is a context threshold past which decode
suddenly collapses to 1.5–6 tok/s, and the mechanism we proposed for it. That turned out to be our
own harness: bench-agentic.sh set TTFT on the first content delta, so on a thinking model the
entire reasoning phase was charged to prefill, and a decode_window / wall < 5% rule then flagged
the turn as degenerate. That rule flags whatever prefills slowly relative to how fast it decodes
— which is why it appeared only with spec on, why SPEC_N=0 looked like a clean control, and why
Ampere looked worse than Blackwell. None of it was the engine.

What we did not retract, and what is still true:

• Decode does get slower as context grows. That is attention cost, and it is expected.
• There is a real spec-decode tax on this architecture — vllm#35387, filed by a vLLM maintainer with an independent reporter on A100/v0.27.1/FlashInfer, measured Qwen3.8-27B at 29K: no-spec 57.0 → MTP n=1 40.6 → n=2 36.0 → n=3 32.7 tok/s, with acceptance 87/78/68%. They profiled it to a cudagraph downgrade (FULL_AND_PIECEWISE → PIECEWISE, 48 GDN layers launching eagerly). It is a steady tax that scales with drafter depth, not a context-triggered cliff. That measurement stands.

So "there is no decode degradation at long context" is not what we corrected to. The corrected claim
is narrower: there is no sudden cliff, and the numbers that suggested one were ours, not the
engine's.

Your bench table actually agrees with our corrected numbers

Look at your own two sessions — decode goes 250 → 92 tok/s between 1.2K and 35.3K, smoothly, with
no collapse. Our corrected re-test on the same pin:

slug	drafter	client	engine
dual-max	MTP n=3 + align	110.2 @ 21,443	109.2
dual-supermax	DFlash2 n=7	159.4 @ 12,275	154.1

Flat across the ramp, cross-checked against vLLM's own vllm:request_generation_tokens_sum /
vllm:request_decode_time_seconds_sum histograms, which are recorded inside the engine and cannot
be affected by our parser. Your turn-12 figure of 92–93 tok/s at 35,304 sits inside that band. Your
bench is not contradicting us — it reproduces us.

Why your harness experience is also not contradicting us

Because it is four times deeper than anything either of us has measured. Your video is at ~70%
of a 200K window — call it ~140K. Our deepest trustworthy decode measurements are the corrected
agentic ramp to 35K and single-turn bench at short prompts. We publish prefill at 90K, but we do
not publish a decode-at-depth curve for your tier, and I would rather say that plainly than
extrapolate one for you.

⚠️ Worth stating: our own prefill-90k bench leg generates only 16 tokens, so its "decode @ 90K
depth" figure is a ~0.15 s window. I retracted a claim built on that number earlier this month. It
is not evidence about your situation and I am not going to quote it at you.

So: 10 tok/s at ~140K is in unmeasured territory. It is not predicted by our data and it is not
excluded by it.

Three things in your config that I would check first, in order

Your setup is essentially our dual-ultrafast shape — DFlash2 + bf16 KV + FA2 at ~200K — with three
deliberate differences from what we ship. Each is defensible on its own; together at 140K they are
the first place I would look.

1. --max-num-seqs 3 with a pool sized for roughly one long sequence. bf16 KV on this model is
65,536 B/token, so 204,800 tokens is ~12.5 GiB of KV, ~6.25 GiB/card at TP=2. That pool holds
one 200K sequence. An agentic harness that issues concurrent tool calls will ask for more than
that, and vLLM's answer is preemption + recompute, which looks exactly like "words arriving at
10 tok/s". We ship max_num_seqs=1 on dual-ultrafast for this reason.

This is cheap to confirm — it is a counter, not a guess:

curl -s localhost:8020/metrics | grep -E 'vllm:num_preemptions|vllm:kv_cache_usage_perc|vllm:num_requests_(running|waiting)'

If vllm:num_preemptions is climbing during the slow typing, that is your answer and nothing else
matters. The engine log says the same thing in words — watch for Running: N reqs / Waiting: M reqs
with M non-zero.

2. SPEC_N=7. This is the one I would A/B. vllm#35387's tax scales with drafter depth, and n=7
is deeper than anything we ship on a dual. At your real working depth, with the drafter off:

SPEC_N=0 <your launch script>

then repeat the same harness session. If 10 tok/s becomes 25–30, the drafter is the cost and
SPEC_N=0 is your fix at depth even though it is clearly the wrong choice at 2K. (Note #35387's
numbers are MTP, not DFlash2 — the cudagraph mechanism is shared, the magnitude at n=7 is not
something anyone has published.)

3. --gpu-memory-utilization 0.95. We ship 0.90 on this model deliberately, and it is
counter-intuitive: the knob is a ceiling on what vLLM claims, so higher means a bigger KV pool and
less slack for transient prefill peaks. At 140K with chunked prefill and a drafter, those peaks
are the largest they ever get. 0.95 on the tightest tier we have is where I would expect trouble.

The measurement that would settle it

Perceived typing speed in a harness mixes prefill, reasoning, tool round-trips and decode. One clean
number beats any amount of video:

# at your real working depth, single stream, nothing else running
bash scripts/bench.sh    # or just: one long-prompt request, time the decode phase only
docker logs <container> 2>&1 | grep -oE 'gen throughput: [0-9.]+' | tail -20

The engine's own gen throughput line is the ground truth — it cannot be distorted by a client-side
parser, which is precisely the mistake we made in #1096.

If you can give me gen throughput at ~140K with SPEC_N=7 and again with SPEC_N=0, plus the
vllm:num_preemptions delta across each run, that is a complete diagnosis and it is genuinely
useful to us — nobody has a decode-at-depth curve for this tier, and you would be the first.

Your upgrade question

No, don't chase upstream for this. Your config already notes it: DFlash2 is native in vLLM
0.28.0 (#52816), so bumping would let you drop the
backport and the topk sampler shim — a maintenance win, not a performance one. Meanwhile
vllm#50021 is not fixed in any release,
including main, and we have not validated 0.28.0 on this rig. Nothing in that bump addresses what
you are seeing. Stay on 0.27.1 until we move the pin.

---

Your instinct that "it wouldn't pass the agentic bench if I'd done something wrong" is sound, and the
bench agrees with you — but the bench tops out at 35K and your daily usage is at 140K. That gap is
ours to close, not your mistake. If you run the two gen throughput numbers above I will add the row
and credit you. ❤️

[sudeposutemizligi]

Selam, thank you for detailed explanations again brother. and again sorry to have posted this video evidence thing on #1052, wrong concept.
i got the point now.

spec=7 was what i thought you had in the composes already (my bad then ), and numseq 3 i tried to make a cap but didnt use more than 1 seq on that 140k daily test(didn't think it would harm decode though) and -gpu-memory-utilization 0.95 was, I think, the idea of codex when i raised numseq 3, i wasn't really aware of the consequences.

i will definitely make the tests you ask for, as far as i got (i mean codex will make them though:))

just i need some time from you , i am away from home. near Iranian border here, that far:)

Thank you for the deep explanations as usual, they help me get the idea and learn a lot.

May God bless you 🤲

[noonghunna]

No rush at all — whenever you're back, and only if you still feel like it.

And your config wasn't wrong. SPEC_N=7 is a reasonable reading of what we ship (the dflash2
tiers do default deep), --max-num-seqs 3 is a sensible cap, and 0.95 is the intuitive
direction for that knob — it just happens to be backwards on this model, which is a thing we
only learned by OOMing at 16K prompts. None of that is on you; it's on us for documenting it in
a compose header instead of somewhere you'd actually hit it.

When you do get to it, the two numbers that settle it:

# at your real working depth, single stream, nothing else running
docker logs <container> 2>&1 | grep -oE 'gen throughput: [0-9.]+' | tail -20
curl -s localhost:8020/metrics | grep -E 'vllm:num_preemptions'

once with SPEC_N=7, once with SPEC_N=0. If the preemption counter is climbing during the
slow typing, that's the whole answer and the drafter is irrelevant.

Safe travels. 🙏