[AndreasKaratzas]

• monokernels.yaml: new test area with one group per AMD monokernel (DeepSeek-V4.1 gfx950 and gfx942, Kimi-K3 specialized decode, MiniMax-M3 MonoKernel), curated source_file_dependencies per step, and the existing steps moved out of models_basic.yaml and kernels.yaml
• DeepSeek-V4.1 tests: DSpark draft-layer numerics and an engine run with DSpark, FULL CUDA graphs with padded rows, step-width and short-context sweeps, KV-record and index-source checks, refusal cases, and a gfx942 arch skip plus CPU weight-transform tests
• Kimi-K3 tests: e2e graph replay that proves every verify step ran the kernels, per-layer checks against vLLM's layers, multi-width/multi-step K1, hostile routing for K2, and compile coverage of all 30 K3 builds
• MiniMax-M3 tests: launch/epoch engagement on every rank, EAGLE3 aux hidden states, width sweeps, stock/mono transitions, and the TP4 rank-skew stress made blocking
• Nightly gsm8k configs on the real checkpoints with a new final_rpc_checks hook and monokernel_checks.py, so the eval fails if the monokernel never ran
• AGENTS.md: a PR that adds an AMD monokernel must add a group to monokernels.yaml, or extend the model's existing group

Monokernels landed on AMD in the last few weeks for DeepSeek-V4.1 (#60397, #61082, #62131, #60731), Kimi-K3 (#60683) and MiniMax-M3 (#62712, #62994). Each came with some CI, but it was spread over three test areas, missed files the kernels depend on, and never checked accuracy on real weights. The engine tests also could not tell whether the monokernel ran at all. This puts every monokernel's coverage in one place with the same shape: blocking kernel/layer numerics against vLLM's stock path, an engine test that asserts the monokernel's own counters moved, and a nightly gsm8k run on the real checkpoint with a post-eval engagement check.

The new tests found a few problems, kept visible rather than hidden:

• DeepSeek-V4.1 index-source K2 reads a stale klen: stage_combine in mono/attention/back.py loads it without waiting on krdy, unlike stage_split. test_mono_index_source_matches_vllm is xfail(strict=True) until it is fixed. An intermittent hang in the same launches was seen earlier but not reproduced, so the DSv4.1 steps run under --timeout.
• MiniMax-M3's TP4 rank-skew stress hit its fail-stop twice with no fault injected, only in whole-directory runs; the stress now runs in its own pytest process and the root cause is still open.
• Kimi-K3's 2.4% layer gap against stock vLLM is not a kernel bug: stock MoE decode is not batch invariant at 8 rows, so the test compares against stock run one row at a time.

Hardware. Blocking steps use synthetic or dummy weights and add about 1.3-1.6 MI355 GPU-h per fully triggered run (≈2.4 → 3.6-4.0), plus an unmeasured MI300 delta. The Kimi engine-fault step needs a whole mi355_8 (≈259 GiB per GPU), and none of these steps can run on mi355_dpx. The nightly gsm8k steps are all new: ≈11-14 MI355 GPU-h per night, and about 2.11 TiB of additional checkpoint storage per HF cache.

Checkpoint	Size	Used by AMD CI today	Nightly device
moonshotai/Kimi-K3	1453.8 GiB	no	mi355_8 (does not fit MI300X)
deepseek-ai/DeepSeek-V4.1-Flash	475.3 GiB	config only	mi355_4
amd/MiniMax-M3-MXFP4	226.0 GiB	no	mi355_4
Inferact/MiniMax-M3-EAGLE3	6.1 GiB	no	mi355_4

The native (dind: false) path of run-amd-test.sh does not disable core dumps, and a crashed rank writes about 40 GB, so the steps that inject faults run under ulimit -c 0.

Not yet validated (why this is a draft). Every new or changed test passed 3 times in a row on 8x MI350X with the CI pins (AITER v0.1.26, FlyDSL 0.4.0), but an external job held about 215 GB of every GPU throughout, so:

• no step has run 10 times in a loop in its final form;
• test_dsv41_mono_layers.py has not run at its committed gpu_memory_utilization=0.5;
• the MI300 steps have never run (no gfx942 here);
• the DeepSeek-V4.1 and Kimi-K3 gsm8k thresholds are not calibrated against mono-off runs yet (each config says so); MiniMax-M3's were, on and off, at gpu_memory_utilization=0.27.

AI assistance (Claude Code) was used to draft this change. Every line was reviewed by the submitter.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @AndreasKaratzas.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork