[dghosef]

Pending GPU testing.

Follow-up to #970: Gaurav found that model_kwargs.prefill_token_buckets only reached Qwen3.5, the Qwen3-Omni Thinker and GLM-5.2. Every other AR model that reads PREFILL_TOKEN_BUCKETS (Orpheus, Whisper, Qwen3-ASR, Qwen3-TTS, Chatterbox, Fish S2, Higgs, Pi0.5) dropped it in **kwargs without an error.

What this does

• prefill_token_buckets is now a top-level serving-config key, per node and capture walk, shaped like max_batch_tokens:

  prefill_token_buckets:
    LLM:
      prefill_text: [32, 64, 128, 256, 512, 1024, 2048]
      mixed: [32, 64, 128, 256, 512, 1024, 2048]

  The engine applies it to any model's packed captures, so no model plumbs the key any more.
• The old model_kwargs.prefill_token_buckets is refused at load with a pointer to the new key, before any weights load. So is a ladder in the wrong shape.
• A walk with no packed capture is refused. Rungs past the first one that holds the walk's step budget are dropped, with a log line, because a step never reaches them.
• The Qwen3.5 9B configs move to the new key. Their 2176 rung goes, since the 2048-token budget already caps the mixed step it was there for. Default ladders don't change when the key is absent. The exception is GLM-5.2's test-scale reduced() variant, which now captures [32, 64] instead of [64].

How it works

• EngineManager reads the key beside max_batch_tokens and checks it before any submodule loads. It's stored on each node's SubmoduleManagement.
• At warmup, each node's AcceleratorGraphRunner calls apply_token_ladders (accelerator_graph_config.py). That rewrites capture_token_lengths on the packed configs captured on each named walk, then trims each ladder to the walk's step budget.
• The budget comes from ChunkingManagement.step_budget. A walk scheduled inside a combined walk (Qwen3.5's prefill_text under mixed) takes the combined walk's budget.
• Deleted: ARNodeSubmodule.__init__'s ladder argument, Qwen3.5's, Qwen3-Omni's and GLM-5.2's prefill_token_buckets arguments, and their tests.

Test	Where	Result
ruff check .	local	clean
CPU Core vs main 9c6f1ad75	dev box	11,174 passed, 0 failing only on this branch (16 not run are the deleted tests)
Changed test files repeated 20×	dev box	0 flakes. One failure outside them under load, test_step_threads::test_the_worker_takes_its_row_and_logs_it, which plans CPUs from the host's state; it passes alone and this PR touches no CPU-pinning code
A ladder reaches Orpheus's prefill capture; rungs past the step budget drop, by the combined walk's budget; a walk with no packed capture and a malformed ladder are refused (test_chunked_prefill)	CPU	pass; each fails without its change
The old model_kwargs key, a top-level list and a non-mapping node value are refused by EngineManager.build before any weights load	CPU	pass; fail without the change
Qwen3.5-9B with the new configs: boot, capture count, c=1/c=64 tok/s against main	H100/H200	pending

Follow-up: OmniVoice's model_kwargs.backbone_token_buckets is the same kind of per-model knob, on a piecewise capture.

🤖 Generated with Claude Code