[AMD][DI][CI] Pin Kimi-K2.6 disagg to known-good image (mitigate non-MTP GSM8K regression)

[michaelzhang-ai]

Summary

✅ Verified green: scheduled-style dispatch on this branch passed at GSM8K 0.939 (> 0.92 gate) — run 29282616011.

The Nightly Test (AMD MI355X 2N 1P1D Disagg) Kimi-K2.6 non-MTP GSM8K regressed to ~0.88 (< 0.92 gate); MTP stays ~0.95 and single-node is 0.947.

A code-vs-image bisect (diagnostic PR #30433) isolated the regression to the docker image, not sglang code:

python	image	GSM8K
main	resolved-latest	0.887
srt @67361ff	resolved-latest	0.879
full @67361ff	Jun-23 (v0.5.13.post1-...-20260623)	0.941

So the same code on the Jun-23 image = 0.941, but on newer images = ~0.88. The drop is disagg-specific (single-node unaffected) and lives in the image (aiter / ROCm / sgl-kernel).

Why a launcher change is needed

The disagg workflow's setup job resolves the latest MI35x image and passes it as IMAGE_OVERRIDE, which the launcher applies over each recipe's runtime.image. So pinning the recipe image alone does nothing for the scheduled nightly — the override always wins.

Modifications

• launch_mi355x.sh: add opt-in runtime.pin_image. When true, the recipe's pinned image wins over the resolved-latest IMAGE_OVERRIDE. An explicit workflow_dispatch image input still force-wins (via IMAGE_INPUT_EXPLICIT), so humans can still test any image.
• nightly-amd-mi355x-disagg.yml: pass IMAGE_INPUT_EXPLICIT (1 only when a human supplied image).
• Kimi-K2.6 1p1d.yaml + 1p1d-mtp.yaml: set pin_image: true, pinned to the Jun-23 known-good image.

Override truth table (verified): pin=0 → latest (DSV4 etc. unchanged); pin=1 + scheduled → recipe Jun-23 (mitigation); pin=1 + explicit input → human image wins.

Scope / follow-up

• Stopgap only — revert once the image regression is fixed upstream.
• DSV4 recipes are unaffected (no pin_image; still track latest).
• Image-tag bisect (Jun-23 good → latest bad) to find the first bad build is in progress; the v0.5.13.post1 → v0.5.14 bump at Jun-26 is the prime suspect.

Note: pin_image also disables checkout-runtime

The disagg CI reinstalls the checked-out (main) python into the container via checkout-runtime. A pinned old image's kernels (sgl-kernel/aiter) don't match main python, so the reinstall ABI-crashes the server at init (first verify attempt hit exactly this: server exited early before bench). Since the regression is in the image's baked kernels — and those are what newer python requires — the only coherent known-good is the old image + its own baked python (the Jun-23 build that scored 0.941, run with no checkout-runtime). So when pin_image wins, the launcher now also forces SGLANG_USE_CHECKOUT_RUNTIME=0 to run that image's coherent baked sglang.

Trade-off: the pinned Kimi legs test the Jun-23 baked build, not current main — an acceptable stopgap to keep the nightly green until the image regression is fixed; DSV4 legs still track latest + checkout-runtime unchanged.

Test plan

• Launcher override truth table verified locally for all pin_image/explicit combinations.
• Recipe + workflow YAML parse.
• Dispatched scheduled-style (no image input) on this branch: pin_image held Kimi non-MTP on the Jun-23 image (checkout-runtime auto-off) and passed at GSM8K 0.939 (run 29282616011). Nightly is green again.

---

CI States

Latest PR Test (Base): ✅ Run #29282593870
Latest PR Test (Extra): ❌ Run #29282593600

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[functionstackx]

how does this work? shouldn't ur nightly DI CI be reflective on the HEAD@main for that night instead of pinning it to an commit? @HaiShaw

[HaiShaw]

@functionstackx pin to specific commit isn't the call, debug phase I believe. @michaelzhang-ai @Lzy17

[michaelzhang-ai]

@functionstackx you're right that the nightly should normally track HEAD@main — this is a temporary debug/stopgap, not a permanent pin. Debug is for @Lzy17 give a fix.

Debug Info: the regression is in the docker image (baked aiter/sgl-kernel), not sglang code. Bisect (see #30433): the same code gives GSM8K 0.941 on the Jun-23 image but ~0.88 on the latest image (first bad = Jun-26, the v0.5.13.post1 → v0.5.14 bump); single-node is unaffected (0.947). So the DI gate goes red every night for a reason that's independent of that night's main code — pinning the last known-good image restores a meaningful signal on the recipe/launcher path in the meantime.

Two clarifications on scope:

• It pins the image (and, since an old image's kernels are ABI-incompatible with the checked-out newer python, also runs that image's coherent baked sglang for the pinned leg). Trade-off during the debug phase: the Kimi leg won't reflect that night's main Kimi code until the pin is lifted — that's the acknowledged cost.
• Kimi recipes only; DSV4 legs still track the resolved-latest image + checkout-runtime, unchanged. An explicit workflow_dispatch image input still force-wins.

[functionstackx]

hi @michaelzhang-ai

Thank you for helping on AMD DI CI! we really appreipcate it! 🥳 we shouted you to senior AMD leadership and they are quite happy with this workstream!

Happy to instead gate this behind a "known-failing/continue-on-error" mark if the team prefers keeping HEAD@main image

Thanks for the suggestion, until it is fixed, "known-failing/continue-on-error" mark probably makes more sense than just running on an old image since that way to view the nightly DI CI is that it should capture the HEAD@main Dockerfile for that night too such that it is an full e2e test of the sglang code, and the dependencies in the Dockerfile (i.e. AITER, sgl-kernel, etc) instead of just an random image.