[merceod]

Stacked on #691, the second docs PR. Needs #688 (media) for its loaders (benchmark/dataset.py and the Omni correctness family import media_io). Nothing under mstar/ changes.

Correctness runner (benchmark/correctness/). Serves one config, sends a fixed sample set, scores the outputs against a reference (transformers, diffusers or the model's own package) and writes a verdict per model family.

• Text divergences have to be near ties, measured against a same-code noise floor, plus long-context needles.
• ASR and TTS use Whisper WER, duration and sanity checks. Image and video use PSNR, with a ROCm floor from the reference's own spread when only its attention runs math SDPA. Models with no reference (LTX-2.5 audio+video, for example) get self-consistency across two server instances.
• Sample sets are built once from pinned sources with seeded picks and checked by sha256. Reports record the served commit per server instance, and rescoring keeps the run's own options.
• GPUs are checked for memory in use by HIP index before a run.
• Configs without a family yet (LingBot-World and the Omnilingual ASR sizes from main) are listed with the reason, so a new config can't go unnoticed.

AMD tables (benchmark/amd/). Launches competitor arms (vLLM, vLLM with AITER, SGLang, vllm-omni, sglang-omni, SGLang Diffusion) from pinned containers with cited recipes, gates each arm on node health, runs warmup plus 3 runs and writes tables with versions, the attention backend each engine picked, the fused-MoE configs each server loaded, co-tenants and a correctness column. GPU numbering follows HIP (amd-smi's order differs on 8-GPU nodes, so it's mapped by PCI address), busy GPUs are found by memory in use, and TP rows run gloo on loopback. long_prompt.py runs worker_phases for concurrent ~1k-token prompts against any serving config. A row can name a different mstar deployment config per arm, so the Qwen3-TTS 1.7B row runs mstar on qwen3tts_1p7b_split_bf16 and an mstar-colocated arm on qwen3tts_1p7b, and the table shows which config each engine column used.

Smaller changes. benchmark.runner gains vLLM and SGLang text clients, sample-set prompts, unique prompt tags, ITL and token counts in results.json. GPU memory queries take amd-smi too, in the Wan2.2 and Waypoint harnesses and the benchmarking skill's server.sh. The Kokoro ONNX baselines take the MIGraphX and ROCm execution providers and refuse a missing one. No more /scratch prompt-file defaults. run_benchmark.sh loses two unbound variables.

NVIDIA: nothing under mstar/ changes. The runner and the vLLM/SGLang clients work against any server.

Tests

What	Result
ruff check .	clean
test_correctness_metrics.py, test_correctness_runner.py, test_amd_bench.py (new), plus test_bench_summary.py, test_video_frame_protocol.py, test_psnr_tool.py	pass in the run of the whole integration branch on MI300X below
test/modular, test/distributed and test/sampling_test on an MI300X node with GPUs hidden, on the integration branch that carries this whole stack, rebased on main	only the 7 environment failures main has there too (rsync, ftfy)
test/rust there	359 passed
test/rocm on MI300X, same branch	674 passed, plus the fused all-reduce harness at 2, 4 and 8 GPUs
Rebase onto main 54f29da (2026-10-26, the whole stack)	CPU on a devel node, GPUs hidden: test/modular only fails the two test_dump_stacks environment cases main fails too (7174 passed before the last three test fixes, targeted rerun on the tip 398 passed), other CPU suites 248 passed, TP lockstep sim 7 passed, test/rust 368 passed. One MI300X: test/rocm 675 passed, fused-MoE and sampler GPU tests 228 passed, served smoke rows for Qwen3.5 0.8B, Kokoro, Whisper large-v3 and Cosmos3 DROID match the earlier runs

What the runner has found so far on MI300X and MI325X (all with the rest of this stack). Pass: Qwen3.5 0.8B-9B (text and image), Whisper, Orpheus, Kokoro, Qwen3-TTS 0.6B, Chatterbox, OmniVoice, Qwen3-Omni media and speech, the Cosmos3 DROID and action configs, Cosmos3 nano t2v, V-JEPA2-AC, Wan2.2. Pass with a caveat: Qwen3-TTS 1.7B and Chatterbox multilingual (sampled, the extra errors are proper nouns or reference artifacts), and Qwen3-Omni's cold start, which is over the runner's 2400 s limit without a saved MIOpen database. Fail: pi05 (all-NaN actions) and V-JEPA2 (embedding drift), which turned out to be a weight-loading bug and a preprocessing difference, neither ROCm-specific, with fixes going up as their own PRs. Also Cosmos3 nano t2i (22.2 dB against a 23.5 dB floor) and LTX-2.5 audio self-consistency, still open.

Tables from this harness on MI300X and MI325X follow.

Open before review

• Several defaults point at our cluster (/work1/... paths, our job marker files). They have to become options.
• It's big (59 files). The correctness runner and the AMD tables could go up as two PRs.

Since the first cut

• --tree base=PATH serves a second mstar tree for the mstar-base arms, run back to back with the head's on each row, so a change is measured in one session. Without the flag those arms are skipped and nothing else changes.
• The fused-MoE column names Qwen3-Omni's expert sizes the way the tables do now (E=..,N=..,K=.., N as w1's rows), after the loader unification in MoE: tuned Triton configs per GPU, the align op on ROCm, AITER fused MoE as an opt-in backend #711.
• versions.json records where mstar_rust imports from. An arm whose tree carries its own build fails at start if that build isn't the one loaded, and the table says whether mstar_rust came from the tree or the venv.
• benchmark/amd/phase_cells.py: worker phases of one mstar row at a workload's concurrencies, sharing long_prompt's server wrapper.
• Video families count videos with identical decoded frames instead of identical mp4 bytes.

Since r8

Since the r8 cut: main's nine new ASR, Command A+ and GLM-5.2 configs are marked unsupported in the correctness map until they have references here, and the harness's own node marker has a neutral name (older result files keep the previous one and are still filtered).

Checks on this rebase

The stack now sits on main f7cb5ca (2026-10-29), every branch cut from one integration tree.

Check	Result
CPU on a devel node with the GPUs hidden: ruff, test/ci, the TP lockstep sim, test/rust, CPU Core	ruff clean, sim 14 passed, test/rust 436 passed, CPU Core fails nothing main doesn't (main's own 34: test_dump_stacks x2, loop_output_ownership x32)
test/rocm on one MI325X (a VM slice, torch 2.12.1+rocm7.2)	715 passed, 9 skipped, 0 failed
MoE and sampler tests there	MoE EP 53 passed, sampler 145 passed. The 75 test_*moe* failures are main's GLM-5.2 kernel tests, the same set on the previous head
Served rows there through the correctness runner	all PASS: Qwen3.5 0.8B and 9B text (no divergence away from near ties, 4/4 needles), Whisper large-v3 (WER 1.64%, 99/100 transcripts equal to the reference), Kokoro (WER delta 0), Qwen3-Omni colocated (50/50, spoken WER 0), pi0.5 and Cosmos3 DROID (two instances bit-same)
Qwen3.5 9B TP2 on two GPUs there	PASS (21/32 exact, 11 near ties, 0 unexplained), the fused all-reduce on both ranks

Nothing fails on this tree that the previous head doesn't: the failing-set diff against it is empty for test/rocm, MoE, the sampler and the rows.