kv: page footprints from one layout per request by Gaurav-Shah05 · Pull Request #832 · mstar-project/mstar

Gaurav-Shah05 · 2026-10-22T18:23:14Z

This PR gives each request a footprint, the most KV pages each of its labels can hold, from a layout stated by the model.

This is a follow up on admission control (#400), where the hand-written counts were brittle and wrong for some models. With this change, every model with a paged KV pool now lays out what each request's walks write per label, and the KV manager checks the pages requests take against the footprint. Whatever the scheduler admits doesn't change yet (follow up soon).

How it works

process_prompt returns a LabelLayout per label the request opens: the spans its walks commit, the scratch a non-committing step writes, and the growth per loop iteration.
get_loop_caps gives each loop the iterations its stop check allows, and neither graph runtime runs a loop past its cap.
The KV manager computes each label's footprint at ingest, and logs an error the first time a label takes a page past it.
Under MSTAR_KV_DEBUG_ASSERTS=1, each commit must match the layout, and a removed request logs each label that held fewer pages than its footprint.
kv_labels declares each paged label once. Loading and intake refuse a model or a layout that doesn't match the declaration.
PrefixStream becomes a mark on a label, KVLabel.keys. Keying is unchanged, since keyed_spans limits it to the spans main keys.

Testing

On one H100 80GB, each model's single-GPU config, greedy, fixed lengths with --ignore-eos, fixed seeds for denoising, p50 over three to twelve alternating boots per arm:

	`main`	This PR	Δ
Qwen3.5 0.8B: ITL c1/c16	2.18/2.31 ms	2.18/2.31 ms	1.00x/1.00x
Orpheus: ITL c1/c16	98.1/128 ms	98.2/127 ms	1.00x/1.01x
Whisper large-v3: ITL c1/c16	4.33/7.66 ms	4.33/8.26 ms	1.00x/0.93x
Higgs Audio: TTFT c1/c16	41.0/658 ms	42.8/668 ms	0.96x/0.99x
Qwen3-TTS 0.6B: ITL c1/c16	116/220 ms	116/214 ms	1.00x/1.03x
Chatterbox: ITL c1/c16	196/1,006 ms	201/922 ms	0.98x/1.09x
V-JEPA2-AC: step c1/c4	7.83/8.70 ms	8.32/8.93 ms	0.94x/0.97x
Pi0.5: TTFT c1/c16	58.7/582 ms	58.1/598 ms	1.01x/0.97x
Bagel: ITL c1/c16	6.51/9.40 ms	6.50/9.45 ms	1.00x/0.99x
Bagel: text to image, CFG on/off	4.19/4.22 s	4.19/4.21 s	1.00x/1.00x
Qwen3-Omni: ITL c1/c16	6.45/17.5 ms	5.53/17.7 ms	1.17x/0.99x
Qwen3-Omni: speech TTFA	397 ms	394 ms	1.01x
Cosmos3 Edge: image/video	1.31/13.8 s	1.37/12.7 s	0.96x/1.09x

Every c1 TTFT the table leaves out is 0.97x to 1.02x, and boot time is 0.97x to 1.04x for every model.
V-JEPA2-AC's c1 step sits inside main's spread between boots, and Cosmos3 Edge's image row comes from one slow boot.
Higgs Audio's c1 TTFT sits above main's spread.
Whisper's c16 e2e latency matches main's, since its c16 TTFT is 1.08x.

Under MSTAR_KV_DEBUG_ASSERTS=1, no request took a page past its footprint.

mstar-review · 2026-10-22T18:29:11Z

An earlier review run didn't complete; a later run finished: #832 (comment)

Gaurav-Shah05 · 2026-10-22T18:34:18Z

/review continue

mstar-review · 2026-10-22T18:35:32Z

Automated review

🟡 No findings on 0caa314, light review

""

Description

The Testing section lists no test commands or results: ruff, the CPU Core pytest set, cargo test plus pytest test/rust/ for the Rust graph-runtime change, and any GPU or integration tests run on the head commit, as AGENTS.md requires for every PR.
The benchmark table doesn't say how it was produced. A maintainer would ask for the harness (benchmark.runner or something else), the commands, flags and concurrency settings, the torch/CUDA build, and where the debug-assert run that saw no overruns comes from.
The PR changes behaviour the description doesn't mention: an understanding request on a BAGEL CFG-parallel deployment now opens KV labels only on LLM, not on all three LLM nodes.
The description doesn't mention the new hard asserts in serving paths: BAGEL's VAE prepare_inputs, Chatterbox's condition, Cosmos3's _build_static/_build_action_static, Qwen3-TTS talker prefill, the Qwen3-Omni vision prefill, and Nemotron talker load. If a layout predictor and a walk ever disagree, the request now fails instead of only logging an overrun.
The description doesn't say that the conductor now gives every declared paged pool a KVReqConfig, even where the model returned none (before, only keyed streams got one). It also doesn't say that both graph runtimes now take a per-request loop_caps on NewRequest, which changes how loops end for models that override get_loop_caps.
The description doesn't explain why Whisper c16 ITL is 0.93x and Higgs Audio c1 TTFT is 0.96x. It calls Higgs's regression "above main's spread" but gives no mechanism. Since this PR adds work to ingest, _alloc and commit, a worker_phases breakdown would show whether the regression is real.

What the review checked

Loop caps reach both graph runtimes the same way: Python sets max_iters on each request's copy of the section, and Rust's cap_loop does the same. Speculation is blocked at the cap on both sides (_can_continue_loop and loop_max_iters), and the cap survives the end-of-forward reset on both sides (WorkerGraphIO.clear and RequestState::reset).
The conductor clamps each cap to its graph's max_iters before anything converts it to u32. The caps come from the same conductor values on every rank, so TP lockstep is not affected.
If refuse_undeclared_labels raises, it does so before dispatched=True, so _admit_request rejects the request with a 500 and cleans it up, as the docs say.
without_layouts runs only after engine_manager.add_request. NewRequest is the only path that calls ingest_request, and the KV manager keeps the original config object, so later frames without layouts never reach an ingest.
Every model that builds a PagedKVConfig also defines kv_labels. Non-AC V-JEPA2 declares no paged pool, so _refuse_undeclared_pools doesn't refuse any shipped model.
data_worker._process_input always passes tensors to process_prompt, so the BAGEL and Qwen3-Omni early returns without a layout are reachable only from tests.
The SpanCursor.advance touched-span logic matches the old chain-advance loop in _index_filled_pages, including spans the cache served through complete_cached_walk.
Retention removes released pages from page_indices, so the len(page_indices) > footprint check in _alloc compares like with like for windowed streams.
BAGEL's _has_cfg_parallel is set in get_worker_graphs, which the conductor calls, so the new nodes selection in get_request_resource_configs is defined there.
max_output_tokens is validated as an int >= 1 before get_loop_caps runs, so the per-model max_output_tokens - 1 caps can't receive None, and the max(1, ...) floors keep every cap at 1 or more.

Not reviewed: test/modular/test_kv_footprint.py, test/modular/test_kv_footprint_checks.py, test/modular/test_kv_layout_intake.py, test/modular/test_kv_layout_refusals.py, test/modular/test_model_kv_layouts.py, test/modular/test_kv_prefix_*.py, test/rust/test_runtime_parity.py, test/modular (other mechanical call-site updates), mstar/model/bagel/submodules.py (beyond the VAE assert), mstar/model/cosmos3/submodules.py (beyond the helpers moved into it).

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}

…nd cap decode where its stop check ends it (mstar-project#832, mstar-project#808) Main's engine manager refuses to load a model whose paged pools have no labels, so the worker would fail at init. Both LMs write the prompt build_prompt makes, then a row per decode step. get_max_output_tokens_limit is gone from main; the loop cap takes its place.

… and cap decode where its stop check ends it (mstar-project#832, mstar-project#808) Main's engine manager refuses to load a model whose paged pool has no labels, so the worker would fail at init. The layout is [sos | text | task_id | prompt speech] plus a row per decode step; reference_tokens gives the prompt-token count the speech tokenizer makes of a clip, and the speaker encoder asserts it.

…nd cap decode where its stop check ends it (mstar-project#832, mstar-project#808) Main's engine manager refuses to load a model whose paged pools have no labels, so the worker would fail at init. Both LMs write the prompt build_prompt makes, then a row per decode step. get_max_output_tokens_limit is gone from main; the loop cap takes its place.

… and cap decode where its stop check ends it (mstar-project#832, mstar-project#808) Main's engine manager refuses to load a model whose paged pool has no labels, so the worker would fail at init. The layout is [sos | text | task_id | prompt speech] plus a row per decode step; reference_tokens gives the prompt-token count the speech tokenizer makes of a clip, and the speaker encoder asserts it.

…nd cap decode where its stop check ends it (mstar-project#832, mstar-project#808) Main's engine manager refuses to load a model whose paged pools have no labels, so the worker would fail at init. Both LMs write the prompt build_prompt makes, then a row per decode step. get_max_output_tokens_limit is gone from main; the loop cap takes its place.

mstar-project#832's test pinned the conductor's ingest RuntimeError text in the client message. It is a model bug, so the client now gets an error id and the test finds the reason in the log under it.

… and cap decode where its stop check ends it (mstar-project#832, mstar-project#808) Main's engine manager refuses to load a model whose paged pool has no labels, so the worker would fail at init. The layout is [sos | text | task_id | prompt speech] plus a row per decode step; reference_tokens gives the prompt-token count the speech tokenizer makes of a clip, and the speaker encoder asserts it.

…ect#799) and loop_caps (mstar-project#832), and mstar-project#799's host-copies stub gets the scheduler the in-flight filter reads

Conflicts, and main's newer contracts this branch has to meet: - entrypoint.py, tensor_store.py, worker.py: main's #837 (collect_results woken on finish) and #799 (host copies dropped after the step's sends) replace this branch's versions of the same changes; _pass_wait stays. - communication/tensors.py: drop this branch's drop_host_copies(uuids). It auto-merged beside main's drop_host_copies() and shadowed it, so every step raised TypeError (ruff F811). - #832/#808: Qwen3-ASR declares its paged KV labels, returns a kv_layout from process_prompt and caps decode_loop in get_loop_caps; main refuses to load a paged pool without labels. Whisper keeps main's labels and caps; its layout now covers detect_language (the prompt up to <|sot|>) then prefill_prompt (the detected token and the tail), and an encoder context of max_source_positions rows for any clip. - Tests: the TP worker fakes set _enable_pre_plan, which run() now reads; main's Whisper layout tests cover the padded window and the detection spans; this branch's tests read process_prompt's new_input_tensors and allow kv_layout in its metadata. ruff clean. CPU Core: 6835 passed, 5 failed, the same 5 environment failures as main. On an H200 with MSTAR_KV_DEBUG_ASSERTS=1 both models boot and transcribe at this branch's WER, with no request past its KV footprint.

…ect#799) and loop_caps (mstar-project#832), and mstar-project#799's host-copies stub gets the scheduler the in-flight filter reads

Main refuses a paged pool with no labels at engine build (mstar-project#832), so neither model started. Canary follows Whisper: the decoder's cache grows a slot per decode step and the encoder context is one rows span of the clip's frames. Canary-Qwen's prompt is a rows span: its audio placeholders are the same for every clip of a length, so it must never key a cached prefix.

…ect#799) and loop_caps (mstar-project#832), and mstar-project#799's host-copies stub gets the scheduler the in-flight filter reads

mstar-project#832's test pinned the conductor's ingest RuntimeError text in the client message. It is a model bug, so the client now gets an error id and the test finds the reason in the log under it.

…ect#799) and loop_caps (mstar-project#832), and mstar-project#799's host-copies stub gets the scheduler the in-flight filter reads

… and cap decode where its stop check ends it (mstar-project#832, mstar-project#808) Main's engine manager refuses to load a model whose paged pool has no labels, so the worker would fail at init. The layout is [sos | text | task_id | prompt speech] plus a row per decode step; reference_tokens gives the prompt-token count the speech tokenizer makes of a clip, and the speaker encoder asserts it.

* transcriptions: windows for bounded ASR models Long uploads are cut by the adapter's clip limit, uploads with no length in their header are measured, every decode stops at the route's cap, and a window without timestamps is one segment (Higgs and untimed Whisper captions). * fastconformer: shared encoder and log-mel NVIDIA's FastConformer encoder, offline or chunked for cache-aware checkpoints, NeMo's log-mel, and the encoder node, captured per mel bucket. * parakeet: Parakeet-TDT and Nemotron ASR Greedy TDT and RNN-T decoding in captured chunks of a compiled step; RNN-T loops over labels. * canary: Canary-1B-v2 Speech recognition in 25 languages and translation to or from English. Whisper and Canary share one KV-fit check. * canary_qwen: Canary-Qwen-2.5B Canary's encoder projected into a LoRA-merged Qwen3-1.7B, English only. * components: one encoder-decoder node for Whisper, Canary Whisper's and Canary's decoder nodes share the prefill/decode flow; Whisper keeps its token suppression as a process_logits hook. * asr: review fixes on the rebased stack - TokenStepNode.check_stop_batched takes step_walk, which the engine passes since #362; without it every call raised and fell back to the per-request stop check. - The FastConformer ASR adapters stream text without SentencePiece's leading space, and the route sends a window-joining space only before a window's first text, so the deltas add up to the done event's text. - Parakeet, Nemotron and Canary-Qwen refuse a task other than transcribe instead of returning a transcript for a translation. * perf(fastconformer): plan a batch's encoder replays by cost, not one series per bucket * perf(fastconformer): log-mel features for the whole batch in preprocess * perf(fastconformer): the position term as one GEMM per head, half-step residuals in the add * perf(fastconformer): Triton kernels for the conv module and the subsampling's bias, mask and ReLU * perf(fastconformer): each residual add in the kernel of the LayerNorm after it * perf(fastconformer): refit the replay cost model to the fused encoder * docs: Omnilingual ASR gets a segment and a caption cue per window A window without timestamps is one segment spanning it (merge_windows), so the CTC model's verbose_json and captions follow its windows, as Whisper's and Higgs-Audio's untimed windows do. * perf(transcriptions): skip the thread hop for an upload prepare() leaves alone * perf(media_io): skip the channel mean for mono audio in read_soundfile_capped * Canary-Qwen: take a custom instruction from model_kwargs, not /generate's text process_prompt used /generate's text prompt as the instruction, so the ASR harnesses' default prompt ("Transcribe the speech in this audio clip.") replaced the trained one, "Transcribe the following:", and Canary-Qwen was benchmarked under an instruction it was not trained on. The text prompt is now unused, as on the other FastConformer ASR models, and a custom instruction comes from model_kwargs' "instruction". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: untimed transcription models get one segment per window The Omnilingual ASR adapter's docstring still said srt/vtt get one cue spanning the file and verbose_json an empty segment list. merge_windows now makes each window with text one segment spanning it, so each format gets one cue or segment per window, as docs/models.rst and docs/clients.rst already say. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * media_io: decode_audio is load_speech_file, so a missing FFmpeg stays a 500 The transcription route decoded uploads with its own copy of the libsndfile-then-FFmpeg chain, which turned every torchcodec RuntimeError into a 400, a missing FFmpeg included. load_speech_file already tells the two apart; the copy and its numpy resample fallback go (torchaudio is a dependency). * fastconformer: log-mel per replay group, not padded to the batch's longest clip A 600 s Parakeet upload batched with short clips padded every clip's STFT to 600 s. preprocess now plans the replay groups and computes each group's features together (a clip past every bucket alone), and hands the plan to the encoder so it isn't computed twice. * canary, canary_qwen: declare their KV labels and lay out each request Main refuses a paged pool with no labels at engine build (#832), so neither model started. Canary follows Whisper: the decoder's cache grows a slot per decode step and the encoder context is one rows span of the clip's frames. Canary-Qwen's prompt is a rows span: its audio placeholders are the same for every clip of a length, so it must never key a cached prefix. * parakeet, nemotron_asr: out of this PR, into a follow-up on top of it Review asked for the transducer to step through the graph as a loop, with its LSTM state and cursor in a resource, rather than run the whole greedy loop inside one forward. That rework lands with the two transducer models in their own PR, stacked on this one. This PR keeps Canary, Canary-Qwen, the shared FastConformer encoder, the encoder-decoder and token-step nodes, the Whisper refactor and the transcription route changes. Removed: mstar/model/parakeet/, components/transducer.py, their configs, registry, CLI and OpenAI adapter entries, benchmark entries, docs and tests. The streamed-deltas test of the transcription route now runs on Canary, the other windowed SentencePiece model. The encoder keeps its causal path, which its own tests cover. * audio_features: use the engine's fp32_precision_scope for the mel matmul Drops mstar/utils/precision.py, a second copy of what mstar/engine/fp32_precision.py already does, which restored only the legacy matmul flag and not the per-backend ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * canary, canary_qwen: cap the decode loop at the request's max_output_tokens Without get_loop_caps the loop cap fell back to the graph's max_iters, so speculation ran past a max-token stop and a smaller kv_cache override refused every request. Same override as Whisper's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * transcriptions: no compression-ratio retry for the untimed ASR adapters The 2.4 threshold is Whisper's sequential-long-form fallback. Canary and Canary-Qwen decode greedily with no conditioning text, so a forced sequential retry could only replace a correct transcript with a sampled one. Omnilingual already opts out the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * audio_features: only a failed decode is the upload's fault load_speech_file wrapped the resample too, so a server-side failure there (e.g. a CPU allocation failure, a RuntimeError) came back as a 400 'could not decode the audio upload'. Split the decode from the resample and wrap only the decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * canary: the prompt's KV is a rows span From the second decoder layer on, the prompt's self-attention KV depends on the clip through cross-attention, so its token ids don't key it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * canary: refuse a checkpoint whose mel front end NemoLogMel doesn't compute Canary-Qwen already checks its preprocessor settings at load; Canary assumed them. Read processor_config.json's feature extractor and refuse a mismatch at startup instead of decoding wrong features. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * token_step: say decode is uncapped with accelerator graphs off Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: load_speech_file's blame split goes through _decode, and a resample failure stays the server's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * canary, canary_qwen: one audio file per request, refused before decoding Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * asr extra installs soundfile; Higgs segments and a stale test comment documented Without soundfile every WAV/FLAC upload went through torchcodec and needed FFmpeg, which the docs said only M4A/WebM do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Kirill Vasilev <290354918+vasilevklart@users.noreply.github.com> Co-authored-by: Joseph Tan <jdtan638@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Conflicts: - mstar/model/base.py: kept this branch's ClientChunk; dropped PrefixStream, which main replaced with KVLabel/PrefixKeys (#832/#834). - mstar/api_server/data_worker.py: imports take ClientChunk plus main's PrefixKeys/keyed_labels; this branch's _load_media refactor kept with main's _prefix_streams -> _keyed_labels rename; _finish_reading drains when the rid is already draining (this branch) or still has a failed read's transfers in flight (main #419), docstring merged. - mstar/api_server/entrypoint.py: PendingRequest takes main's output_totals (#844), which counts outputs as they are accepted; this branch's `delivered` counter and _count_outputs, the same fix for the same leak, are deleted. Stream drain keeps this branch's last_activity read with the swap-out; the done path clears req.chunks as main does. /generate/ws: main's empty-id-as-None and check_request_id (#618) move into the receive loop beside this branch's isinstance check, since live-call routing reads the id there; the error reply keeps this branch's call/turn release with main's client_error_text scrubbing (#646), under this branch's `payload` name. - mstar/conductor/conductor.py: imports from both; worker graphs from main's validate_deployment (#640), then this branch's _find_client_streams. - test/modular/test_kv_prefix_request_keys.py: kept this branch's get_partition_topology stub and main's kv_labels. Adapted to main (no conflict markers): - test/modular/test_client_stream_api.py: stub renamed prefix_key_streams -> kv_labels; the tensor-manager fake gains take_failed_reads; the bad-append test expects #646's "internal error" text in place of the model's exception; test_a_streaming_request_lets_go_of_the_chunks_it_has_delivered dropped, as main's test_api_result_delivery.py covers the release and the totals. - test/modular/test_request_id_limit.py: the fake server gains takes_stream_input, submission_key and abort_submission, which this branch's /generate/ws reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Conflicts: - mstar/model/nemotron_duplex/nemotron_duplex_model.py: imports take this branch's ClientChunk with main's KVLabel/LabelLayout/ProcessPromptOutput/Span (#832); process_prompt keeps this branch's body and comment and returns main's ProcessPromptOutput with the kv_layout, beside main's kv_labels and this branch's process_stream_chunk. - test/modular/test_audio_decode_limit.py: this branch's (load, error, match, seconds) rows plus main's OmniVoice row (#779). - test/modular/test_nemotron_duplex_model.py: this branch's tests (audio_in, session cap, appends) kept; every process_prompt result they read goes through .new_input_tensors, as main's #832 returns ProcessPromptOutput. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Conflicts: - mstar/model/nemotron_duplex/nemotron_duplex_model.py: _create_talker_submodule builds the talker with this branch's codec and keeps main's warm-up assert (#832) before returning it; _create_codec_submodule stays deleted, as this branch moved the codec into the talker. Adapted to main (no conflict markers): - mstar/model/nemotron_duplex/submodules.py: codec_step reads and writes CODEC_STATE with main's gather / scatter_ (#891), which follow the step's plan; its rows parameter is deleted. A fresh slot still reads zeros, the decoder's causal padding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… graph loop (#892) * transcriptions: windows for bounded ASR models Long uploads are cut by the adapter's clip limit, uploads with no length in their header are measured, every decode stops at the route's cap, and a window without timestamps is one segment (Higgs and untimed Whisper captions). * fastconformer: shared encoder and log-mel NVIDIA's FastConformer encoder, offline or chunked for cache-aware checkpoints, NeMo's log-mel, and the encoder node, captured per mel bucket. * parakeet: Parakeet-TDT and Nemotron ASR Greedy TDT and RNN-T decoding in captured chunks of a compiled step; RNN-T loops over labels. * canary: Canary-1B-v2 Speech recognition in 25 languages and translation to or from English. Whisper and Canary share one KV-fit check. * canary_qwen: Canary-Qwen-2.5B Canary's encoder projected into a LoRA-merged Qwen3-1.7B, English only. * components: one encoder-decoder node for Whisper, Canary Whisper's and Canary's decoder nodes share the prefill/decode flow; Whisper keeps its token suppression as a process_logits hook. * asr: review fixes on the rebased stack - TokenStepNode.check_stop_batched takes step_walk, which the engine passes since #362; without it every call raised and fell back to the per-request stop check. - The FastConformer ASR adapters stream text without SentencePiece's leading space, and the route sends a window-joining space only before a window's first text, so the deltas add up to the done event's text. - Parakeet, Nemotron and Canary-Qwen refuse a task other than transcribe instead of returning a transcript for a translation. * perf(fastconformer): plan a batch's encoder replays by cost, not one series per bucket * perf(fastconformer): log-mel features for the whole batch in preprocess * perf(fastconformer): the position term as one GEMM per head, half-step residuals in the add * perf(fastconformer): Triton kernels for the conv module and the subsampling's bias, mask and ReLU * perf(fastconformer): each residual add in the kernel of the LayerNorm after it * perf(fastconformer): refit the replay cost model to the fused encoder * docs: Omnilingual ASR gets a segment and a caption cue per window A window without timestamps is one segment spanning it (merge_windows), so the CTC model's verbose_json and captions follow its windows, as Whisper's and Higgs-Audio's untimed windows do. * perf(transcriptions): skip the thread hop for an upload prepare() leaves alone * perf(media_io): skip the channel mean for mono audio in read_soundfile_capped * Canary-Qwen: take a custom instruction from model_kwargs, not /generate's text process_prompt used /generate's text prompt as the instruction, so the ASR harnesses' default prompt ("Transcribe the speech in this audio clip.") replaced the trained one, "Transcribe the following:", and Canary-Qwen was benchmarked under an instruction it was not trained on. The text prompt is now unused, as on the other FastConformer ASR models, and a custom instruction comes from model_kwargs' "instruction". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: untimed transcription models get one segment per window The Omnilingual ASR adapter's docstring still said srt/vtt get one cue spanning the file and verbose_json an empty segment list. merge_windows now makes each window with text one segment spanning it, so each format gets one cue or segment per window, as docs/models.rst and docs/clients.rst already say. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * media_io: decode_audio is load_speech_file, so a missing FFmpeg stays a 500 The transcription route decoded uploads with its own copy of the libsndfile-then-FFmpeg chain, which turned every torchcodec RuntimeError into a 400, a missing FFmpeg included. load_speech_file already tells the two apart; the copy and its numpy resample fallback go (torchaudio is a dependency). * fastconformer: log-mel per replay group, not padded to the batch's longest clip A 600 s Parakeet upload batched with short clips padded every clip's STFT to 600 s. preprocess now plans the replay groups and computes each group's features together (a clip past every bucket alone), and hands the plan to the encoder so it isn't computed twice. * canary, canary_qwen: declare their KV labels and lay out each request Main refuses a paged pool with no labels at engine build (#832), so neither model started. Canary follows Whisper: the decoder's cache grows a slot per decode step and the encoder context is one rows span of the clip's frames. Canary-Qwen's prompt is a rows span: its audio placeholders are the same for every clip of a length, so it must never key a cached prefix. * parakeet, nemotron_asr: out of this PR, into a follow-up on top of it Review asked for the transducer to step through the graph as a loop, with its LSTM state and cursor in a resource, rather than run the whole greedy loop inside one forward. That rework lands with the two transducer models in their own PR, stacked on this one. This PR keeps Canary, Canary-Qwen, the shared FastConformer encoder, the encoder-decoder and token-step nodes, the Whisper refactor and the transcription route changes. Removed: mstar/model/parakeet/, components/transducer.py, their configs, registry, CLI and OpenAI adapter entries, benchmark entries, docs and tests. The streamed-deltas test of the transcription route now runs on Canary, the other windowed SentencePiece model. The encoder keeps its causal path, which its own tests cover. * RecurrentStatePool: gather / scatter_ over the step's addressing A forward whose recurrent state has no backend kernel to plan against reads and writes its rows' slots through the pool instead of indexing pool.block() itself, so its eager and captured paths address the same slots and a replay's padding rows land on the sink. num_rows is the rows the step declared, padding rows included, which is what both slice by. From #383 (e9b665a, c1813dc), without its MiniCPM-o changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: gather / scatter_ round-trip, replay vs eager addressing, padding rows on the sink On CPU, admit / plan / commit driven by hand: a shorter eager step reads only its rows from the grown eager buffers; a replay padded to its bucket leaves the real slots as the eager forward does and writes nothing but the sink; the replay buffers are one per (bucket, capture slot) and the same storage every step; a freed slot reads as zeros. On CUDA, a graph captured over a bucket's addressing and replayed for four different batches matches the eager forward. * docs: RecurrentStatePool.gather / scatter_ are on main, not an open PR * Revert "parakeet, nemotron_asr: out of this PR, into a follow-up on top of it" This reverts commit 32c4258. Parakeet and Nemotron ASR come back as they are on #796's head (bf9f067), with the transducer's greedy loop still inside one forward. This is the follow-up's starting point; the graph loop with its state in the recurrent pool goes on top. * parakeet: the transducer as a decode loop, its state in a recurrent pool The transducer node no longer decodes a whole clip in one forward. prefill projects the encoder output once (with Nemotron's prompt) and persists each request's frames; decode is a Loop over the node that re-injects them every iteration, with a data-free clock loop-back so the node is a same-node speculation target. An iteration is 16 greedy steps: gather the rows' h, c, dec_out, cursor and symbols from a RecurrentStatePool slot, advance, scatter them back, emit the tokens (-1 where a step emitted none). check_stop ends a row's loop off the side-stream copy of its active flag, so the per-chunk host read is gone; the never-binding step bound is the loop cap and a stop-check guard. Capture moves to the engine's main runner: a BatchedAcceleratorGraphConfig per frame bucket on the decode walk, keyed by cg_key_info from the clip's frame count, which process_prompt computes and step_metadata carries. A fresh slot and a padding row start from the start token's state, computed once at load. The node asks the engine for full fp32 instead of setting it. The YAMLs size the pool at 129 slots for max_concurrent_requests 128, and startup refuses a cap the pool can't hold. * test: the transducer loop through the recurrent pool, against #796's decoder A CPU harness in the shape of test_gdn_cuda_graph.py: tiny TDT and RNN-T heads, the model's pool declaration under a YAML max_slots, a real step runner, and the node driven by hand from prepare_inputs to check_stop. Requests of 1 to 30 frames join and leave at different iterations, in both frame buckets, in eager steps and in steps padded to a capture bucket; a stopped request runs the speculated extra iteration and emits nothing. Each request's tokens equal the one-request reference loop's and a frozen copy of bf9f067's greedy_decode, for random heads, a head that never emits blank (NeMo's per-frame guard) and blank runs longer than a window. On CUDA the engine's runner captures and replays the decode iteration at 1, 3 and 64 rows, and must match eager. * docs: Parakeet and Nemotron keep their decoder state in a recurrent pool * RecurrentStatePool: gather / scatter_ refuse a pool without a sink slot With disable_sink_slot, rows with no slot address NO_SLOT (-1), which index_select and index_copy_ refuse: a bare IndexError on the CPU, and on CUDA a device-side assert that takes the worker down. Both now raise a ValueError naming the pool, as the bf16 GDN backend does for its kernel; the check is a bool per call, once at capture for a captured forward. * audio_features: use the engine's fp32_precision_scope for the mel matmul Drops mstar/utils/precision.py, a second copy of what mstar/engine/fp32_precision.py already does, which restored only the legacy matmul flag and not the per-backend ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: the engine's fp32_precision_scope in place of the removed helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * canary, canary_qwen: cap the decode loop at the request's max_output_tokens Without get_loop_caps the loop cap fell back to the graph's max_iters, so speculation ran past a max-token stop and a smaller kv_cache override refused every request. Same override as Whisper's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * transcriptions: no compression-ratio retry for the untimed ASR adapters The 2.4 threshold is Whisper's sequential-long-form fallback. Canary and Canary-Qwen decode greedily with no conditioning text, so a forced sequential retry could only replace a correct transcript with a sampled one. Omnilingual already opts out the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * parakeet: the decode tokens reach the client from the stop check's host copy The tokens join the active flag in the stop buffers. The stop check doesn't read them, but the worker then stages the batch's tokens to the host once, behind the step, and the shared-memory send takes each request's row from there instead of a device-to-host copy per request: register_outputs went from 0.57 to 0.25 ms per decode step at c=64 on 1x H100 (worker_phases). * audio_features: only a failed decode is the upload's fault load_speech_file wrapped the resample too, so a server-side failure there (e.g. a CPU allocation failure, a RuntimeError) came back as a 400 'could not decode the audio upload'. Split the decode from the resample and wrap only the decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * canary: the prompt's KV is a rows span From the second decoder layer on, the prompt's self-attention KV depends on the clip through cross-attention, so its token ids don't key it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * canary: refuse a checkpoint whose mel front end NemoLogMel doesn't compute Canary-Qwen already checks its preprocessor settings at load; Canary assumed them. Read processor_config.json's feature extractor and refuse a mismatch at startup instead of decoding wrong features. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * token_step: say decode is uncapped with accelerator graphs off Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: load_speech_file's blame split goes through _decode, and a resample failure stays the server's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * canary, canary_qwen: one audio file per request, refused before decoding Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * asr extra installs soundfile; Higgs segments and a stale test comment documented Without soundfile every WAV/FLAC upload went through torchcodec and needed FFmpeg, which the docs said only M4A/WebM do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * parakeet: one walk, the encoder output decoded where it lands The transducer no longer projects a clip's frames in a step of its own and persists them for a second walk. One walk runs the encoder and then the decode loop, which re-injects the encoder output every iteration; each greedy step projects only the frames under the cursors (with Nemotron's prompt), in fp32. That drops a prefill step and a conductor round trip per request, about 0.5 ms of the worker's 3.5 ms a request at c=64 on 1x H100 (worker_phases: the step's GPU-thread and postprocess work, and the decode walk's message). The loop has no loop-back edge any more. The clock only made the node a same-node speculation target, and a speculated iteration past a clip's end is a whole chunk of steps for nothing; the worker still speculates a request's first iteration behind the encoder's step. Nemotron's prompt id travels in the step metadata, a host value, rather than as a tensor the loop would move to the device every iteration. * parakeet: 64 greedy steps a decode iteration, not 16 Every iteration costs each of its rows the engine's and the worker's per-step work: prepare_inputs, the store, route and send of its tokens, the stop check. At 16 steps a LibriSpeech test-clean clip ran 3.6 decode steps of the node at c=64 on 1x H100 (worker_phases; the two-walk loop, its speculated step past the end included), and that host work, not the GPU, bounded throughput. At 64 it runs 1.2, at the price of the idle steps of a row that ends early in an iteration. The loop cap and the stop check's guard count iterations of the node's chunk, which the loop test sets to 16 so that its short requests still span several iterations. * parakeet: a 2,048-frame decode bucket, 164 s of audio Past 41 s a clip decoded in the 8,192-frame bucket, whose captures hold at most 4 rows, so a batch of more than 4 minute-long clips ran its iterations eagerly. The middle bucket's captures hold 16. On the mixed workload (10% of the clips 60-120 s), 1x H100, 4 passes each: RTFx 1,862 -> 2,224 at c=32 and 3,090 -> 3,199 at c=64 (#796's transducer: 2,047 and 3,057). * TransducerAdapter: the compression-ratio opt-out now comes from _UntimedTranscriptionAdapter Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * parakeet, nemotron_asr: one audio file per request, refused before decoding Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * parakeet: one audio file per request, refused before decoding * RecurrentStatePool: gather / scatter_ read the step's plan, and gather takes a start state Every caller fetched the addressing and wrote its own torch.where over has_state for rows with no saved state (a request's first step, or a padding row on the sink, which held leftovers). gather now does that select itself, zeros by default or the given start, and scatter_ checks the row count with a message naming the pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * parakeet: read and write the transducer state through the pool's gather(start=) / scatter_ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * parakeet: refuse a transducer state pool with fewer than 2 slots max_slots 1 leaves no usable slot (slot 0 is the sink), so every request hung waiting for one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * parakeet: comments on the clock edge and slot lifetime match the code The clock carries each row's active flag, which check_stop reads, and a request's state slot frees at teardown, not at its last iteration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: the recurrent state pool's gather / scatter_ for models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * RecurrentStatePool: say what a zero-span row addresses Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * parakeet: drop the stop check's copy of the loop cap get_loop_caps already has the runtime end each request's loop at that cap, so the stop check reads only the active flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: drop the frozen copy of the transducer step; the independent reference covers it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * RecurrentStatePool: cast the scatter index once per step gather passes the int32 slot indices to index_select as they are, and scatter_ casts them to the int64 index_copy_ needs once per step, cached on the step's addressing (keyed by whether a graph capture made it, so a capture records its own cast and each replay recomputes it from its own plan). Option (a) of the two: a staged int64 copy in plan would add an H2D for every pool user, GDN and Mamba-2 included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Kirill Vasilev <290354918+vasilevklart@users.noreply.github.com> Co-authored-by: Joseph Tan <jdtan638@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Naomi Sagan <nsagan@stanford.edu>

* engine: MLA, fp8 MoE, step budget, symm all-reduce attn: MLA on a latent KV layout (#321). fused_moe: fp8 block-scale experts (#270). worker: per-step token budget; api_server: role on the first token. distributed: group timeout, compile_mode, symmetric-memory all-reduce. * glm52: GLM-5.2 with MLA and fp8 MoE GLM-5.2 at k=0: MLA with the DSA indexer, fp8 MoE, weight loader, configs. Replaces #272. Open: the configs' new max_concurrent_requests caps. * glm52: refuse a prompt without room, as a 400 A prompt of limit - 1 tokens failed its whole decode batch (the speculative step overran the window), and a long-context prompt past index_topk failed its prefill batch. process_prompt now refuses both; prepare_inputs agrees. * serving_chat: decode text across chunk boundaries Byte-level BPE models emit each token's raw bytes, which can end inside a character; decoding per chunk turned those into U+FFFD. * glm52: emit each token's raw bytes Per-token decode turned a character split across byte-level BPE tokens into U+FFFD; the 10-07 GSM8K answers have them at ÷ and √. * attn: MLA wrappers per slot, graph indices per row As FlashInferManager since #243/#280: an eager plan staged into the one wrapper per label while the last step's copy could still be queued, and a captured step whose rows share a cached prefix overran the kv_indices buffer. * glm52: ask for the MoE op pre-build The conductor now builds it only for a model that sets prebuild_fused_moe. * fused_moe: prefill experts as one custom op As fused_experts_fp8: dynamo traced the launch path inline, which breaks the graph and fails on a symbolic token count under torch 2.9.1. * glm52: fail loading loudly, not serve garbage An fp8 checkpoint whose quant config went undetected loaded clean with unscaled e4m3 weights; a failed Hub fetch came back as a local path. * micro_scheduler: step budget takes the oldest The runtime's ready order is handle order in Rust, and handles are recycled: a large prefill could be passed over indefinitely. Backlogged rows count too, since the batch builder tops a backlog up with fresh rows. * api_server: TextChatAdapter checks what it passes A non-numeric max_output_tokens from extra_body reached the conductor's min() and took its message batch down; image or audio parts were decoded and dropped. Both are 400s now. * fused_moe: decode router takes at most 64 tokens Its logits kernel holds every token in one tile; past 128 the first launch raised OutOfResources. The models never send more; now it says so. * glm52: smaller review fixes top_k and min_p reach the sampler; a fused-kernel import failure is logged; MSTAR_GLM52_GRAPH_COMPILE=0 also leaves uncaptured steps uncompiled; the single-rank test config is dummy mode, not a 750 GB pull. * attn: FlashMLA only for eager MLA plans Chosen per plan, a captured bucket could re-plan into the other kernel while the graph replayed the captured one on stale buffers. * glm52: refuse a scaled RoPE from_hf_config read only rope_theta, so a YaRN or dynamic variant ran as plain RoPE, its frequencies and softmax scale wrong. * fused_moe: an empty step skips the align op The CUDA op launched a 0-block grid unchecked; the latched error surfaced in the next unrelated kernel. * test: CI runs the new CPU tests Each new test file with a CPU case, listed in test/cpu-core.txt. * distributed: symm all-reduce on every rank or none A rank whose setup failed fell back to NCCL alone, and its peers' first all-reduce waited on it forever. tp_allreduce_max_kb: 0 now turns it off instead of reading as unset (512). * glm52: tokenizer registers special tokens Built from tokenizer.json alone (transformers 4.x), the tokenizer had no special tokens, so the detokenizer printed <|user|> at the end of replies. * cuda_graph_runner: refuse empty/cudagraph modes An empty MSTAR_GRAPH_COMPILE_MODE compiled in Inductor's default mode silently; reduce-overhead and max-autotune capture inside the runner's own graph, fail it, and the bucket serves eager. * glm52: a prefill stop token ends the reply The prefill walk has no decode_loop: the runtime dropped (and logged) its stop, and decode wrote on past the stop token. The first decode step now stops and emits nothing; the CPU stream harness runs through the runtime. * fused_moe: check block scales and inputs The decode and prefill kernels index scales, x and shared as dense rows; a strided view was mis-scaled or mis-added silently. The prefill test now covers GLM-5.2's hidden 6144, the fused sum-and-add branch. * glm52: a bad sampling kwarg is a 400 None now means unset. A string or null reached the conductor's ingest and raised there, outside any request; process_prompt reads the same kwargs first, where a bad one is the client's 400. * kv: SHM transfer reads an MLA cache by token The reader sliced NHD's K/V axis first; on an MLA cache that took latent columns, not tokens. * glm52: refuse min_p as a 400 The sampler is built without min-p, so the worker refused it at ingest as a 500, after every rank had the request. * conductor: pre-build the MoE op only when asked Every deployment JIT-built the fused-MoE align op before spawning its workers: non-MoE models paid the nvcc build, or hung on a stuck build lock. A model now asks with prebuild_fused_moe. * api_server: TextChatAdapter refuses what it drops n, stop, logprobs, logit_bias and the presence and frequency penalties returned 200 and were dropped. Left off they pass; set, they are a 400. * attn: refuse what only the MLA backend reads sm_scale on another backend was dropped silently; an MLA cache without num_qo_heads defaulted to the one latent KV head, and the kernel planned 1. * glm52: no max_tokens runs to the window The default budget was 1024 on every window, so a thinking reply was cut with length; vLLM runs it to what the prompt leaves, as the submodule's cap now does. * test: CI runs the new CPU tests Each new test file with a CPU case, listed in test/cpu-core.txt. * cli: mstar serve glm52 It answered unknown model; the default is the TP8 yaml. * engine: select TP async scheduling from config The deployment config's tp_async_sched key (true, false or a node list) turns it on so a shipped yaml can opt in; MSTAR_TP_ASYNC_SCHED still overrides it. Default stays off. * pyproject: glm extra Every other model family has one; GLM's runtime came only with all. * api: warmup_requests can repeat their text A long prompt for a path only long prompts take, such as a long-context prefill, without pasting it into the yaml. * glm52: an oversized top_k or seed is a 400 The sampler keeps them in int32 / int64 rows. Past that its step raised on every rank, failing the whole batch; under TP async the followers then hung. * worker: tp_async_sched reads boolean words A quoted yaml true or false arrives as a string and was taken as a node name, so the switch silently did nothing. * glm52: an oversized max_tokens is a 400 Past 2**64 it failed the wire to the conductor as a 500. The bound is int64, what the Rust server's max_tokens holds. * cuda_graph_runner: empty compile mode env is unset Raising at import failed every model's boot on an empty export. * glm52: an out-of-range sampling value is a 400 A negative temperature or a top_p of 0 returned 200 with garbage: GLM-5.2's eager prefill inverted the logits or sampled token 0. * distributed: load timeout ends after warmup dist_timeout_s held for every serving collective, so a wedged rank hung its peers for the whole load timeout. Serving gets torch's default. * api_server: three more dropped fields are a 400 best_of, response_format and stop_token_ids returned 200 and were dropped, as n and stop were. * micro_scheduler: drop the step token budget Chunked prefill (#365) brings a per-(node, walk) max_batch_tokens, and the two would fight over the same batch; GLM moves to it once it lands. The round-robin pass over walks with no room stays, and so do its tests. * attn: refuse context_only on the backends that don't read it Every backend but MLA planned it as an ordinary step, so the queries also attended to the tokens the step appends. * attn: drop FlashMLA Off by default and measured no faster than FlashInfer's MLA kernel; it needs 64-token pages, which no GLM config uses, and served eager plans only. * cuda_graph_runner: drop MSTAR_GRAPH_COMPILE_MODE It set the mode of every capture of every model at once. A capture config's compile_mode, declared per (node, walk), is now checked when the config is built, so a bad one fails warmup instead of serving its buckets eager. * distributed: tp_allreduce and dist_timeout_s from the config only MSTAR_TP_ALLREDUCE, MSTAR_TP_SYMM_AR_MAX_KB and MSTAR_DIST_TIMEOUT_S were a second way to set three serving-config keys, and nothing sets them. * distributed: all-reduce backends as classes AllReduce is NCCL; SymmMemAllReduce takes the small messages it can and hands the rest to it; AllReduce.new picks one from tp_allreduce. CommGroup holds one. resolve_dist_timeout moves to distributed/utils.py. * distributed: one multimem kernel on every rank The one-shot kernel ran only where the caller's tensor started 16-byte aligned, which each rank decided for itself, and ranks on different kernels hang the collective. A misaligned tensor gets the sum in an aligned one and a copy. * docs: the MLA attention resource in adding_models The MLA cache layout, the FLASHINFER_MLA backend and AttentionStep's context_only, beside the resources they extend. * distributed: a class per symmetric-memory kernel SymmMemAllReduce keeps the buffer and the routing; OneShotAllReduce and MultimemAllReduce are its two kernels, in place of a mode string checked on every call. AllReduce.new refuses a tp_allreduce it doesn't know, which used to serve on NCCL behind a warning. * glm52: the accelerator graph names from #262, and capture takes #341's planned * test: the GLM-5.2 tests install the CPU FlashInfer per test, not at import * test: GLM-5.2's attention spec takes #603's AUTO backend * pyproject: the glm extra takes main's Mooncake floor * distributed: compiled one-shot all-reduce stays on NCCL Inductor gives one_shot_all_reduce's input a persistent symmetric buffer sized by the first call, so BAGEL at TP2 with symm_oneshot failed decode capture at its second batch size. * test: #755's looping worker has the timeout restore warmup ends on * test: rust-transport lists TextChatAdapter's chat as not ported mstar-server does not serve GLM's flattened text chat yet (#798). * test: the GLM-5.2 serve harnesses stop as the runtime does They ran decode on the prefill walk's info, whose check_stop ends no loop since 0dedee4, so the loop never stopped. They now check the prefill's stop, then decode on the decode walk, as the capture test does. * glm52: request kwargs through normalize_generation_kwargs The model's own sampling_kwargs duplicated main's check. As main has it, a number sent as a string is a 400, and a top_k past int32 is served as off. * glm52: checkpoints resolve through hf_snapshot.resolve_snapshot_dir The model's copy is gone; a repo id the Hub cannot fetch still raises. * glm52: a request's token budget and prefill stop in its request_state The engine drops the state with the request, so cleanup_request only evicts the DSA k-store. * glm52: positions from a PositionSpec The step's tokens land where the position resource's counter says, not at the KV stream's stored length, as every other model does. The layers still rotate q_pe / k_pe themselves. KVManager.stored_len had no other user. * fp8 block-scale checkpoints in mstar/utils/fp8_block.py Moved from mstar/model/glm52/quantization.py: nothing in it is GLM's. quant_fp8 takes its FP8_DTYPE from there, and the post-load hook walk is in mstar/model/utils.py. * moe: the noaux_tc router and fp8 expert loaders are shared Moved from mstar/model/glm52/components/moe.py into components/moe.py, nothing in them is GLM's. The router keeps main's router contract (a third value, None: it is stateless). * rope: the interleaved fp32 RoPE is shared Moved from mstar/model/glm52/components/rope.py into components/rope.py as InterleavedRotaryEmbedding, same math: one fp32 cos/sin table per forward. * mla: the shared MLAAttention; GLM-5.2's layer adds DSA MLA with weight absorption (projections, the w_kc / w_vc fold, the naive and absorbed paths) moves to components/mla.py. Glm52MLAAttention is that plus its DSA, through two hooks: _select and _attend_selected. * glm52: stage host tensors through main's PinnedStager mstar/utils/pinned_staging.py duplicated mstar.utils.h2d.PinnedStager with a fresh pinned allocation per call. GLM-5.2's preprocess now uses a PinnedStager like Qwen3-TTS; PinnedStager.to_device returns a new device tensor of the values' shape, for the callers that build one. * dsa: GLM-5.2's DSA as an engine resource mstar/engine/resources/dsa holds what glm52/dsa.py and the attention layer did: the index key store, each step's rows off the KV plan, the indexer's scores and top-k, IndexShare's shared pick and sparse MLA over the picks. The model keeps the indexer projections and declares a DSASpec under dsa_long_context. Still eager only; the pages are staged pinned and the selection no longer syncs on positions.max(). * glm52: drop the unmeasured MoE fused all-reduce switch MSTAR_GLM52_MOE_FUSED_ALLREDUCE came with no measurement, which AGENTS.md invariant 9 asks for. The block reduces the routed and the shared expert's outputs separately, as before the switch; ParallelGatedMLP is main's again. * docs: DSASpec in adding_models.rst What the DSA resource builds, what it depends on and what a step declares, with its limits in this PR: eager only, prompts within topk, keys outside the paged pool. * position: refuse rope_table_dtype widths the CUDA kernel can't rotate FlashInfer's cos/sin-cache kernel moves 16 elements a thread; at rotary_dim 8 it returned garbage, different from run to run, instead of failing. Build now refuses a CUDA table whose width is not a multiple of 16. The comment no longer calls the in-kernel angles exact: they come from a fast sin/cos, about a bf16 ulp from a float32 table. * glm52: positions and RoPE from the position resource, softmax_scale from config The forward reads PositionManager.plan's ids (pos_ids("main")) instead of building its own in preprocess. The layers and the DSA indexer rotate with apply_qk, which takes explicit positions (pos_ids=) and rotates q alone (k=None); GLM-5.2's PositionConfig is interleaved over qk_rope_head_dim from an fp32 table, the angles as before. components/rope.py is gone. MLAAttention reads config.softmax_scale. The reduced test config rotates 16 dims (8 before), the narrowest the CUDA table kernel takes. * dsa: a selecting step's page tables go to the device once, in plan attend copied each request's page list to the device in every layer, and the stager's ring waited on a copy eight calls back, so the CPU ran at most eight layers ahead of the GPU. plan() now stages the tables once per step and attend gathers every row's picks in one batch. The decode-only check moves to plan, before any layer runs. The docs row says what the paged follow-up changes. * api_server: TextChatAdapter refuses tool calls A chat with tools reached the model as plain text and came back with a 200, as if the model had declined every tool; _refuse_tool_calls makes it a 400, as for the other chat adapters. A test pins that each turn keeps its role. * glm52: chat turns in their roles, main's intake checks, batched check_stop - process_prompt rendered every message's text as one user turn, the system prompt and earlier replies included; it now builds one message per turn from prompt_parts, as Qwen3.5 does, and a chat the template refuses is a 400. - The sampling-kwarg and min_p checks in process_prompt repeated what the conductor already does at intake since #402 (SamplingReqConfig.validate, and a ValueError before dispatch is a 400); they are gone, and the tests drive the conductor's own methods. - get_max_output_tokens_limit publishes the context window, so a larger max_tokens is the conductor's 400 rather than a silent clamp. - check_stop_batched reads the step's tokens in one tolist; the sampler returns them as row outputs, one clone and one D2H per step. - MSTAR_GLM52_GRAPH_COMPILE becomes model_kwargs.graph_compile. * glm52: GLM-5.2 declares its KV layout Since #832 a paged pool needs its labels declared, and each request's layout returned from process_prompt; without them every GLM-5.2 request was refused. - kv_labels: the one "main" label, growing on "decode" under decode_loop. - process_prompt returns the prompt's ids span and one slot per decode iteration under "kv_layout". - get_loop_caps: max_output_tokens counts the token the prefill emits, so decode_loop runs max_output_tokens - 1 iterations, where check_stop ends it. #808 refuses a cap past the loop's max_iters (context_limit - 1), which is the bound get_max_output_tokens_limit published; #808 removed that hook, so GLM-5.2's override goes too. - The prefill's prompt-aware clamp stays: the conductor holds prompt plus max_tokens to the KV pool's max_seq_len, the clamp to GLM-5.2's own limit, one row short for a step still in flight when the stop lands. - The walk harness passes the model's loop caps to the graph runtime. * glm52: no token-budget clamp; the KV pool is the context window Since #808 and #832 the conductor holds every request to the room its prompt leaves in the KV pool (it refuses a max_tokens past it, and fits a request that names none), and the decode loop's cap ends decode where check_stop does, so no step runs past it. GLM-5.2's own prompt-aware clamp duplicated that, one row short, so a request sized exactly to the window got one token fewer than it asked for. check_stop now reads the request's max_tokens. The KV pool's max_seq_len is now Glm52Model.context_limit(), so the conductor sizes requests against the window the preprocess guard holds contexts to. Before, the two agreed only by convention (index_topk equals max_seq_len in the checkpoint's config); the reduced variant's did not (64 against 512). Claude-Session: https://claude.ai/code/session_01FeseQ9qqtZRk2CqVpUJ6MH * loader: pread into private buffers, an index for stacked rules (from #389) Tensors are read with pread by a small thread pool a bounded window ahead of the consumer instead of faulting through safetensors' mmap, which moves a few pages per fault on ZFS and network filesystems. Per-expert stacked rule lists are matched through a token index instead of trying every rule on every key. Moved here from #389 (caeea87, 1ded60c) so #387 boots without it. * configs: glm52_tp8_longctx points its DSA limits at the resource * test: the GLM-5.2 layout test asks main's validation checks #640 moved the paged-pool and prefix-walk refusals out of mstar/worker/engine_manager.py into mstar/validation/resources.py, as generators of problems, so test_glm52_kv_layout.py failed to import after the merge of main. --------- Co-authored-by: Kirill Vasilev <290354918+vasilevklart@users.noreply.github.com>

#387 dropped GLM-5.2's own prompt-aware clamp without MTP: the conductor holds prompt plus max_tokens to the window (#808, #832), so the k=0 arm asked for 40 tokens in a 24-row window and tripped the context guard. Both arms now ask for the most the conductor admits (limit - prompt + 1): the k=0 arm fills the window exactly, and under MTP the clamp still leaves two verify blocks.

Since #832 a paged pool needs its labels declared and each request its layout from process_prompt, and since #640 validate_deployment refuses a paged pool with no labels before any worker starts, so GLM-5.3 could not start once main was merged in. As GLM-5.2 does (#387): - kv_labels: the one "main" label, growing on "decode" under decode_loop. - process_prompt returns the prompt's ids span and, per decode iteration, one row, or under MTP the whole k + 1 block a verify step keeps (its rejected drafts stay as holes against kv_rows), so the pool sizes a request at its worst case. - get_loop_caps: decode_loop runs max_output_tokens - 1 iterations; the prefill emits the first token. - A request that names no max_tokens starts from what the decode loop can run (under MTP the cache rows bind before index_topk), which the conductor then fits to its prompt. Asking for index_topk, every such MTP request would need more iterations than the loop has, a 400.

… 24 kHz audio (#495) * cosyvoice3: config and ONNX front-end weight loaders (WIP) * cosyvoice3: LM, flow, HiFT components, submodules and model (WIP) * cosyvoice3: registry entry and serving config (WIP) * cosyvoice3: OpenAI adapter with #411 media rules, preset-name check, CPU tests (contract, LM stop rules, RAS, flow chunks, exact HiFT and Token2Wav stream stitching) * cosyvoice3: real-weights CPU test (exact loads, computed buffers, front-end and Token2Wav geometry), skips without the cached checkpoint * cosyvoice3: run the flow solve without TF32 (the engine enables it process-wide; it moved the B200 streaming mel by up to 2 vs float64) * cosyvoice3: ruff format the new files * cosyvoice3: [cosyvoice3] extra, mstar serve cosyvoice3, models table entry * cosyvoice3: a request ended by its prefill token adds no speech, stop checks count their own step * cosyvoice3: overlong preset names and failed ref_audio fetches are bad requests * cosyvoice3: the LM head stays float32 under the engine's bfloat16 autocast; a truncated ref_audio fetch is a bad request * cosyvoice3: the LM head is float32 before the checkpoint loads, so its weights are not rounded through bfloat16 * cosyvoice3: Apache-2.0 headers credit the CosyVoice files each port follows * cosyvoice3: the flow estimator defaults to bfloat16; flow_estimator_dtype: float32 keeps the reference's precision * cosyvoice3: a bfloat16 flow estimator keeps float32 rotary frequencies * cosyvoice3: a bfloat16 flow estimator takes the float32 time grid into its sinusoid; the DiT casts its own inputs * cosyvoice3: LM sampling inputs reach the device in one pinned copy per dtype; request knobs, token limits and the voice key are host values in step metadata Each row carried nine one-element device tensors built per step, the token limits and voice key were read back from device tensors, and the EOS forced after a prefill stop read the prefill token with .item(). The knobs are now resolved once per request, preprocess stages the batch's columns with PinnedStager, and the forced EOS is a device comparison kept per request. The captured decode graph's keys, dtypes and shapes are unchanged. * cosyvoice3: a flow step reads its tokens back once and plans on the host; row counts and the end-of-stream flag stay host values filter_tokens ran on the device (a boolean index, a pageable copy of the silent ids, a .tolist()), solve_rows built its per-row counts with pageable copies and read the frame totals back, and MEL_FINAL went to the device for HiFT to read back. The token values are needed to plan what settles, so one copy to the host per flow step stays; the filtered tokens and the row counts go back in one pinned copy each. * cosyvoice3: a request's ras_window counts repeats in its own last ras_window tokens; the conductor picks the LM walk from the mode process_prompt built * cosyvoice3: say a refused ref_audio is not readable audio, and keep reference clips short in the docs * test: build CosyVoice3 in the shipped-config startup sweep without its checkpoint * cosyvoice3: HiFT keeps its excitation noise on the device and overlap-adds the iSTFT itself, bit-identical to torch.istft without its per-call host check of the window envelope * cosyvoice3: hand speech tokens to the flow node every 25 tokens, the smallest hop, instead of every 5 * cosyvoice3: synthesize long text in the reference frontend's pieces (about 80 tokens, or 80 characters with Chinese) through an adapter hook on speech chunking, join CJK sentences without a space, and take ref_audio through the shared _resolve_ref_audio * chatterbox, cosyvoice3: a reference clip neither decoder reads is a 400 that does not name the server's upload path; docs on long-text pieces * cosyvoice3: the URL ref_audio test stubs check_ref_audio, which now runs on fetched clips * cosyvoice3: the flow's look-ahead convolutions bypass cuDNN, whose float32 plan without TF32 reserved about 40 GiB of workspace and took about 155 ms a call at small batch x length * cosyvoice3: the cosyvoice3 extra requires a patched Mooncake, like every other extra * cosyvoice3: the flow and HiFT graph hooks under main's accelerator graph names * cosyvoice3: size long-text pieces from the script's characters per token, so Hangul, kana and Cyrillic pieces stay near the reference's 80 tokens * cosyvoice3: name the flow row and HiFT stream dataclasses apart from chatterbox's, so the keyword-arity guard resolves each call to its own class * cosyvoice3: the adapter docstring says how pieces without Chinese are sized * CosyVoice3 docs: kana-only Japanese with spaces between words is read poorly, as by the official model * cosyvoice3: declare the LM's KV label, lay out each request's prefill and cap decode where its stop check ends it (#832, #808) Main's engine manager refuses to load a model whose paged pool has no labels, so the worker would fail at init. The layout is [sos | text | task_id | prompt speech] plus a row per decode step; reference_tokens gives the prompt-token count the speech tokenizer makes of a clip, and the speaker encoder asserts it. * cosyvoice3: list process_prompt's facts in INTERNAL_KWARGS, so the data worker drops a client's values for them * cosyvoice3: cut text with Chinese as the reference's split_paragraph does The shared splitter wrapped a sentence over 80 characters at its commas and joined the parts with spaces, so the LM read spaces the text never had, and it cut 、-separated Japanese by length. Text with Chinese now breaks at 。？！；：、 (a Latin mark only before a space), groups sentences until a piece is past 60 characters and the next would pass 80, and keeps a longer sentence whole, as split_paragraph(lang="zh") does. * cosyvoice3: list CosyVoice3Adapter's speech route in NOT_PORTED, since the Rust frontend does not serve it yet * chatterbox, cosyvoice3: a ref_audio the server cannot save is a 500 that names no path _resolve_ref_audio caught every OSError as a failed fetch, so disk full or no permission while writing the upload came back as a 400, and its repr could name the upload path. Only fetch errors (URLError, timeouts, resets, TLS, short bodies) are a 400 now, with the error type or HTTP status as the reason. A write that fails is RefAudioStoreError, a 500 whose message is "could not store ref_audio on the server"; the cause goes to the server log. * cosyvoice3: a Chinese piece over 300 characters is cut again at its commas The reference keeps a long sentence whole, so a run-on Chinese passage with only commas became one piece past the LM's 512 tokens of prompt text and text: a 400 when it was the first piece of a stream, audio cut short when it was a later one. A piece over 300 characters (one token per Han character, which leaves room for the system prompt and a 30 s clip's transcript) is now cut again at commas too, as split_paragraph(comma_split=True) does, after any clause over 80 characters is cut into even parts. Pieces up to 300 characters are unchanged, which the tests check against a port of the reference's split_paragraph. * test: cosyvoice3's KV label load check calls main's validate_deployment checks (#640 moved them out of the engine manager)

…nd cap decode where its stop check ends it (mstar-project#832, mstar-project#808) Main's engine manager refuses to load a model whose paged pools have no labels, so the worker would fail at init. Both LMs write the prompt build_prompt makes, then a row per decode step. get_max_output_tokens_limit is gone from main; the loop cap takes its place.

…VAE window decode (#659) * Models: VoxCPM2 port (WIP): config, MiniCPM-4 stacks, local DiT/encoder, AudioVAE V2, submodules, contract, adapter * VoxCPM2: CPU tests (contract, prompt modes, Talker batching/stop, exact VAE stream stitching, adapter); fp64-capable VAE node * VoxCPM2: unsaturated random VAE in the stream tests, short-context negative test, #411 ref_audio rules in the adapter * VoxCPM2: CPU tests run without soundfile/soxr (skip or stdlib WAV), list them in cpu-core.txt * VoxCPM2: tokenize continuation transcript + target as one string and keep the reference's whitespace rule (official prompt layout) * VoxCPM2: round the LongRoPE tables through the model dtype, as the reference's cast does * VoxCPM2: real-weight tests (official prompt layout, exact VAE stream in fp64, fp32 teacher-forced steps vs an official trajectory); skip without weights * VoxCPM2: compile the talker decode step before CUDA graph capture The decode step is dominated by the 10-step CFM DiT; compiling it before capture drops served c1 RTF from 0.109 to 0.072 on a B200, at the cost of ~9 min of max-autotune on a cold start. * VoxCPM2: real-weight VAE stream test compares fp64 within 1e-12 (bit equality depends on the CPU thread count) * VoxCPM2: modular VAE stream tests compare fp64 within 1e-13 (bit equality fails by about 6e-15 on AMD CPUs at any thread count) * VoxCPM2: ruff format the new files (the recorded prompt table keeps its layout) * VoxCPM2: [voxcpm2] extra, mstar serve voxcpm2, models table entry * VoxCPM2: a one-patch request ends after prefill, a refused step re-prepares the same patch, stop checks use their own step's index, max_output_tokens caps patches * VoxCPM2: a failed ref_audio fetch is a bad request * VoxCPM2: tests load their test/voxcpm2 helpers by path, as other tests do (import test can find the standard library's package) * VoxCPM2: the batching test postprocesses each step and checks that steps advance the patch index and the noise; a truncated ref_audio fetch is a bad request * VoxCPM2: Apache-2.0 headers credit the VoxCPM files each port follows * VoxCPM2: compile_talker: false captures the eager decode step (compiled stays the default) * VoxCPM2: prompt mode, patch budget, min_len and the reference digest ride step_metadata; step indices and prefill offsets reach the device in one pinned copy per batch (no host syncs in prepare_inputs or preprocess) * VoxCPM2: the clone prompt's ref tokens and the VAE's output-rate index are device fills, not pageable copies * VoxCPM2: the clone prompt's text mask is an index_fill_, not a scalar index assignment that copies H2D * test: build VoxCPM2 in the shipped-config startup sweep without its checkpoint * VoxCPM2: set the VAE stream lead in every prompt mode process_prompt only wrote vae_stream_lead_patches for a reference continuation, and the data worker lays the prompt metadata over the request's own kwargs, so in the other modes a client value reached the VAEDecoder's stream priming and an out-of-range one raised in the worker's ingest. The server's value (0 outside continuation) now always wins. * VoxCPM2: take ref_audio through the shared _resolve_ref_audio, which now reports a failed fetch as a 400 for every model that uses it * VoxCPM2: cap max_concurrent_requests at what both KV caches hold and size them for 32 full-context requests; bound text at max_text_tokens and cfg_value at 10; a clip soundfile cannot read is a 400 without the upload path; collapse whitespace as the reference does; correct the cold-start time * VoxCPM2: the data URL and fetched URL adapter tests stub check_ref_audio, which now runs on those clips * VoxCPM2: a ref_audio upload that never swings past -60 dBFS is a 400, as for Chatterbox, not seconds of near-silent audio * VoxCPM2: [all] carries soxr from the voxcpm2 extra * VoxCPM2: the voxcpm2 extra requires a patched Mooncake, like every other extra * VoxCPM2: a text with no letter or digit (emoji or punctuation alone) is a 400, like an empty one * VoxCPM2: the Talker's graph hooks under main's accelerator graph names * VoxCPM2 docs: one request with long text can end early, as in the official package; split long text * voxcpm2: declare both LMs' KV labels, lay out each request's prompt and cap decode where its stop check ends it (#832, #808) Main's engine manager refuses to load a model whose paged pools have no labels, so the worker would fail at init. Both LMs write the prompt build_prompt makes, then a row per decode step. get_max_output_tokens_limit is gone from main; the loop cap takes its place. * voxcpm2: list process_prompt's facts in INTERNAL_KWARGS, so the data worker drops a client's values for them * voxcpm2: list VoxCPM2Adapter's speech route in NOT_PORTED, since the Rust frontend does not serve it yet * voxcpm2: the conductor picks the Talker walk from the prompt mode process_prompt built; a FLAC written to a pipe is counted as it decodes On /generate the client sets input_modalities, so a clip sent with ["text"] built a cloning prompt while the conductor skipped the RefEncoder walk, and the Talker raised for the missing reference latents. A FLAC written to a pipe has no sample count in its header; load_audio now reads its length with sound_file_seconds, as Qwen3-TTS does, instead of refusing it as a 1.9e14 s clip. * voxcpm2: the one-patch prefill test sends the prompt mode the conductor now reads * VoxCPM2: capture the prefill's per-row CFM solve and stop head, and the VAE decode per stream window width, as piecewise CUDA graph regions * VoxCPM2: GPU test that both captured regions replay the eager result * VoxCPM2: capture the VAE window decode at 1, 2 and 4 rows only; wider batches decode eagerly * VoxCPM2: stream_window_widths takes an empty stream_schedule * VoxCPM2: the prefill tail and VAE window regions under main's accelerator graph hook names, with a test that each node overrides the hooks the engine calls * VoxCPM2: the CUDA region test captures with no plan, as main's piecewise runner now takes one

Gaurav-Shah05 added this pull request to stack #833 October 22, 2026 18:23

mstar-review Bot added area/api area/conductor area/engine area/graph area/model-shared area/resources area/rust area/worker model/many size/L labels Oct 22, 2026

dghosef self-assigned this Oct 22, 2026

Gaurav-Shah05 mentioned this pull request Oct 22, 2026

kv: a host index keyed by prefix #834

Merged

mstar-review Bot added type/feature type/refactor labels Oct 22, 2026

Gaurav-Shah05 mentioned this pull request Oct 22, 2026

fix: refuse a request that asks for more tokens than its decode loop runs #808

Merged

Base automatically changed from fix/bagel-guidance-sink to main October 22, 2026 21:04

Gaurav-Shah05 added 10 commits October 22, 2026 17:04

kv: a page footprint from a label's layout

441e94c

model, runtime: a loop's per-request cap becomes its max_iters

8484a1a

model, kv: process_prompt's layout reaches the KV manager

c195820

kv: PrefixStream becomes a mark with one span cursor

fe10c6b

kv: always-on page check and debug span checks

79ca1bf

model: orpheus caps decode where its stop check ends it

e5d052e

model: bagel lays out its guidance labels and caps decode

760eba7

model: chatterbox declares its KV layout

fb6d718

model: cosmos3 declares its KV layout

c62d8e7

model: fish_s2_pro declares its KV layout

ce8cd23

merceod mentioned this pull request Oct 25, 2026

pi0.5 DROID: serve the DROID fine-tune with openpi's state normalization and inputs #804

Merged

merceod mentioned this pull request Oct 26, 2026

[Model] Add Fun-CosyVoice3: zero-shot voice-cloning TTS with streamed 24 kHz audio #495

Merged

2 tasks done

Navigation Menu

Repository navigation

kv: page footprints from one layout per request - #832

kv: page footprints from one layout per request#832
Gaurav-Shah05 merged 27 commits into
mainfrom
kv-footprints

Gaurav-Shah05 commented Oct 22, 2026 •

edited

Loading

Uh oh!

mstar-review Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Gaurav-Shah05 commented Oct 22, 2026

Uh oh!

mstar-review Bot commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Gaurav-Shah05 commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

Testing

Uh oh!

mstar-review Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gaurav-Shah05 commented Oct 22, 2026

Uh oh!

mstar-review Bot commented Oct 22, 2026

Automated review

Description

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Gaurav-Shah05 commented Oct 22, 2026 •

edited

Loading

mstar-review Bot commented Oct 22, 2026 •

edited

Loading