[Rockdu]

What

• Build Miles' CUDA 13 image on SGLang v0.5.23 (lmsysorg/sglang:v0.5.23) with the rebased patch stack from sglang-miles-v0.5.23. The ROCm image keeps its v0.5.22 ROCm base image and checks out the same sglang-miles stack. ROCm CI did not run on it.
• Adapt miles to the SGLang v0.5.23 APIs and behaviour changes listed below. Without them miles does not import on the new stack, or it silently changes rollouts.

Why

SGLang v0.5.23 keeps torch 2.14.1 and transformers 5.19.0, so the cu130-torch214 wheels and the Transformer Engine 2.17.0+miles triplet carry over. The base image moves flashinfer 0.7.1 -> 0.7.2 and sglang-kernel 0.4.9 -> 0.4.10.

These v0.5.23 changes reach miles:

Upstream change	Effect on miles	Fix
sgl-project/sglang#45315 deletes encoding_dsv4.py / encoding_dsv41.py and renders DeepSeek-V4/V4.1 through deepseek-recipe	chat_template_utils/deepseek.py fails to import, and so does all of miles	Vendor both encoders unchanged (as encoding_dsv32 already is), and move the three V4.1 helpers into the bridge
sgl-project/sglang#46133: Function keeps the client's key order and drops an unset strict	Every tool-bearing DeepSeek-V4 prompt changes, even with the vendored encoder	dsv4_tool_payload rebuilds the v0.5.22 tool dict for V4
sgl-project/sglang#44403: enable_decoder_swa_bounded_replay becomes Optional and resolves to on for DeepSeek-V4.1	DSv4.1 rollouts stop being numerically equivalent to full prefill, so logprobs drift from the trainer and replayed routing for the tail layers misaligns	Pass False when the user leaves it unset. An explicit flag or a per-group override still wins
sgl-project/sglang#43556: should_deepgemm_weight_requant_ue8m0(..., grouped_moe=)	On SM120, expert FP8 scales are sent as fp32 while SGLang holds them in UE8M0	Pass grouped_moe for expert weights.
sgl-project/sglang#45112 renames and moves the DSv4 KV-cache kernels	The DeepSeek-V4.1 plugin import fails	Import from the new modules (signatures unchanged)
sgl-project/sglang#45791 adds a QSA prefill indexer kernel that T.Pipelined-loops over a runtime trip count	miles' image runs TileLang 0.1.14 (SGLang pins 0.1.12), which only pipelines constant trip counts. The kernel fails to compile (Downcast from tirx.Sub to ir.IntImm failed), and Qwen3-Next rollouts hang on 503s	Fold into [28/76], which already makes the same T.serial change for the older sibling kernel in that file
sgl-project/sglang#45893 moves SGLang's FlashMLA ops to the sgl-flash-mla==1.0.0 wheel, which installs as flash_mla	The image installs the wheels release's FlashMLA nv_dev under the same name, so every SGLang DSA path fails at warmup (no attribute 'get_mla_decoding_metadata', unexpected keyword argument 'use_legacy'). Dropping nv_dev instead breaks the trainer: Megatron's CSA forward passes indexer_topk, and miles' DSA kernel runs the sparse prefill on SM90, which sgl-flash-mla only has for SM100	Ship both builds. nv_dev stays flash_mla, so the trainer is unchanged from main. The base image's package moves to sgl_flash_mla, which SGLang imports first ([76/76])

There are also test-only adaptations to new OpenAIServingChat attributes, a new no-CLI ServerArgs field, the Anthropic codec's tool spelling, and the recorded ServerArgs defaults in the launch and chart snapshots.

One known behaviour change is not a defect: tool JSON for HF chat templates and DeepSeek-V3.2 now keeps the client's key order and no injected "strict": false. The v0.5.23 server renders it the same way, and miles stays aligned with the server.

The PR was validated with SGLANG_BRANCH and a ci-sglang-pr: pin on sglang-miles-v0.5.23. Right before merge, sglang-miles moved to that same commit, both images' SGLANG_BRANCH point at sglang-miles, and the pin was removed.

SGLang stack (sglang-miles-v0.5.23 = b0ed761650)

• The 74 commits on sglang-miles were cherry-picked onto the v0.5.23 tag (297e54fd10). Five were dropped because the tag already contains them. That leaves [1/76]-[69/76] ([28/76] also carries the TileLang 0.1.14 change for the new QSA kernel), followed by:
  • [70/76]: the stack's ServerArgs fields go after upstream's pinned positional slots.
  • [71/76]: DeepSeek-V4.1 with LoRA stays off the FlashMLA MegaAttention backend, which has no LoRA path.
  • [72/76]: MegaGate records hash-layer routed experts for R3.
  • [73/76]: an unset diffusion --scheduler-port binds port 0 and reports the bound port.
  • [74/76]: a diffusion shutdown test fake gets scheduler_endpoint.
  • [75/76]: RL: Read byte ranges of output store fields into caller memory sgl-project/sglang#47320, which landed on sglang-miles after the freeze, cherry-picked unchanged.
  • [76/76]: SGLang's FlashMLA interface imports sgl_flash_mla when it exists and falls back to flash_mla, so miles' image can ship both FlashMLA builds.
• The previous head is frozen as sglang-miles-v0.5.22-final (a5701d958b). sglang-miles moved from 29b34bc6f7 to b0ed761650 right before merge. The rollback point is 29b34bc6f7, not -final, which predates RL: Read byte ranges of output store fields into caller memory sgl-project/sglang#47320.
• pre-commit run --all-files passes on every commit.

Validation

• SGLang unit tests ran on an 8x H200 devbox (lmsysorg/sglang:v0.5.23) with the stack checked out: 172 files. Every bump-caused failure was fixed in the commit that caused it. The remaining failures also fail on the v0.5.23 tag or on the old stack.
• Static audit of miles against SGLang (old stack 29b34bc6f7 vs new 865cb6486d) found the fixes above. It covered:
  • imports, call signatures, module and enum attributes, lazy and string imports, and script flags;
  • ServerArgs fields (only speculative_skip_dp_mlp_sync was removed; miles does not use it);
  • the 30 HTTP endpoints miles calls, the request structs and the meta_info fields miles reads;
  • runtime and source patches (all 8 dumper anchors match once);
  • model load_weights mappings, parser names, env vars and dependencies;
  • every package the Dockerfile installs over the base image, and the callers of each in SGLang, Megatron and miles. FlashMLA is the only conflict;
• CPU suites (stage-a-cpu, stage-b-cpu) were replayed on a devbox the way the hosted job runs them: Python 3.12 with requirements.txt + tests/ci/requirements-ci-cpu.txt, and SGLang/Megatron on PYTHONPATH. The runs were compared test by test with main on the old stack. No test fails on this branch that passes on the baseline: stage-a-cpu 20407 passed / 11 failed / 235 errors and stage-b-cpu 770 passed / 1 failed / 158 errors, with every failure and error also present on the baseline (devbox environment only: Ray tests need the GPUs hidden, and the SGLang Anthropic contract needs sgl_kernel once a GPU is visible). SGLang's own Anthropic tests, which that contract runs, pass on the new stack on their own (100 passed). Helm-gated chart tests ran with helm 4.2.3 and pass.
• DeepSeek prompts, byte-compared between main on the old stack and this branch on the new stack:
  • V4 and V4.1: 108 cases each are identical (tool spellings, system and tool turns, thinking on/off, generation prompt on/off).
  • V3.2 differs only in tool-bearing cases, by the upstream tool-JSON change above.
• QSA kernel on TileLang 0.1.14 (8x H200 devbox): SGLang's QSA kernel and indexer tests pass on TileLang 0.1.12 before and after the change (330 passed). On 0.1.14 with miles' patch, they fail with the CI error before the change and pass after it. Benchmarked at the Qwen3.8-Flash-Next indexer shapes (H200): logits are bit-identical to upstream's pipelined kernel and to the old packed path, in bf16 and fp8. In bf16 (the default) the serial loop matches upstream (0.94-1.06x). The opt-in fp8 indexer is 1.22-1.42x slower than upstream's pipelined loop. Both are 1.5-9x faster than the packed path miles ran before this bump. B200 was not measured.
• FlashMLA, both builds in one image (8x H200 devbox, same layout as the Dockerfile): SGLang's FlashMLA kernel tests give the same result as the stock layout (15 passed, 75 skipped by arch, both ways). In the same process, nv_dev's sparse forward runs with Megatron's and miles' call shapes at head dims 512 and 576, and its LSE matches an fp32 reference (max error 1e-6). With sgl-flash-mla alone, both of those calls fail on SM90.
• GPU CI (pinned): run 41219496557 passed all 59 NVIDIA jobs (attempt 2 re-ran the three 2-GPU shards that marking the PR ready had cancelled). It tested miles 1afbc6328 (tree d7098cd47) with sglang-miles-v0.5.23 = b0ed761650 (tree 83d2da6956). The finalize commit only points both images' SGLANG_BRANCH at sglang-miles and drops the ROCm-only grouped_moe compat wrapper (a pass-through on v0.5.23, whose function takes grouped_moe), so the merged code can be checked against these trees. PR Test (ROCm) never ran: no MI350 runner picked up its jobs, and the run was cancelled after 3 h. The ROCm image (v0.5.22 ROCm base with the v0.5.23 stack) is untested.

Files

• docker/Dockerfile: SGLANG_IMAGE_TAG=v0.5.23; both FlashMLA builds, with a build-time check that the moved package no longer imports flash_mla. docs/developer/versions.md, requirements.txt: version notes.
• miles/utils/chat_template_utils/templates/encoding_dsv4.py, encoding_dsv41.py: vendored, unchanged apart from a provenance line. deepseek.py: the vendored encoders, the V4.1 helpers and dsv4_tool_payload.
• miles/backends/sglang_utils/sglang_engine.py: keeps full prefill for DeepSeek-V4.1 unless the user opts in.
• miles/backends/megatron_utils/megatron_to_hf/processors/quantizer_fp8.py: grouped_moe for expert scales.
• miles_plugins/models/deepseek_v4_1/ops/kvnorm.py: kernel imports.
• tests/:
  • regression tests for the DSv4.1 default, the SM120 scale format and the V4 tool payload;
  • the test_template stub, the no-CLI field registry and the Anthropic parity spelling;
  • the regenerated launch and chart snapshots.
• .pre-commit-config.yaml: excludes the vendored encoders, like encoding_dsv32.

Checklist

• pre-commit passes (Miles: changed files; SGLang stack: --all-files per commit)
• Added/updated tests for new behaviour (regression tests above; each fails before its fix and passes after)
• pytest -x is green (pinned GPU CI run 41219496557: all 59 NVIDIA jobs; CPU suites also replayed, see Validation)
• If launch flags changed, python3 train.py --help still parses (no miles flags changed)
• If a public flag was added, it appears in the CLI reference docs (none added)
• If an example was added, it has a real walkthrough (none added)

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[Rockdu]

@claude review

[claude]

🔴 If DeepSeek-V4.1 training runs on the ROCm image, it can now fail with ImportError in the KV-cache roundtrip, where the base imports worked. The ROCm image stays on sglang-miles-v0.5.22-final, which has only the old names dequantize_k_cache_paged and dsv4.elementwise.fused_k_norm_rope_flashmla. Fix: resolve the kernels with a v0.5.23-first fallback to the v0.5.22 locations, the same way _accept_grouped_moe handles the ROCm stack, so both images import. That covers the 2 sites (kvnorm.py:22-23, kvnorm.py:57).

Why this was flagged

Docker/Dockerfile.rocm now pins SGLANG_BRANCH=sglang-miles-v0.5.22-final. The base code imported dequantize_k_cache_paged from sglang.kernels.ops.attention.dsv4.dequant_k_cache and fused_k_norm_rope_flashmla from sglang.kernels.ops.attention.dsv4.elementwise. That shows those were the v0.5.22 names, and the PR says sgl-project/sglang#45112 renamed and moved them in v0.5.23. kvnorm.py:22-23 and kvnorm.py:57 now import only the v0.5.23 names dequantize_k_cache_paged_dsv4 and dsv4.norm_rope. These imports are lazy, inside kv_cache_roundtrip and compressed_kv_cache_roundtrip. They are reached on every DSv4.1 forward through kv_norm_rope_fp8, called at deepseek_v4_1.py:516, and through the compressed path. On a ROCm build that runs DSv4.1 with these kernels, every forward raises ImportError, where the base imported successfully. The PR adds a compat shim for grouped_moe but none for these renamed kernels. The trigger depends on these kernels actually running on ROCm, which cannot be confirmed from this checkout.

Verification: docker/Dockerfile.rocm now pins ARG SGLANG_BRANCH=sglang-miles-v0.5.22-final. kvnorm.py line 22, line 23 and line 57 now import only the v0.5.23 names, and nothing falls back to the old names. deepseek_v4_1.py:516 calls kv_norm_rope_fp8(...) unconditionally, so on the ROCm stack every DSv4.1 forward would raise ImportError. _accept_grouped_moe covers this ROCm skew but not this rename.