Skip to content

[AMD] Fix DeepSeek block-FP8 loading on gfx94x - #38446

Merged
HaiShaw merged 3 commits into
sgl-project:mainfrom
bingxche:codex/fix-amd-ue8m0-attr
Sep 11, 2026
Merged

HaiShaw merged 3 commits into
sgl-project:mainfrom
bingxche:codex/fix-amd-ue8m0-attr

Conversation

@bingxche

@bingxche bingxche commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Motivation

On MI300/MI325, SGLang converts checkpoint FP8 weights to the FP8 form used by the GPU. That conversion creates a new scale tensor and drops the small label that says whether the scale holds normal numbers or raw UE8M0 bytes. Later code reads the missing label and crashes before DeepSeek can start.

Seen in https://github.com/sgl-project/sglang/actions/runs/34070685413/job/101587367851
image
.

Modifications

  • Remember whether the original scale contains raw UE8M0 bytes before conversion.
  • Keep the existing conversion unchanged for normal block-FP8 scales.
  • Do not treat raw UE8M0 bytes as normal numbers; let the existing requantization handle them.

This keeps non-FNUZ paths unchanged and preserves the UE8M0 support added in #36805.

Accuracy Tests

image

Speed Tests and Profiling

Not applicable; this is a load-time condition fix.

Checklist

  • Code is formatted.
  • Documentation is not needed for this internal loader fix.
  • Accuracy validation is listed above.
  • The change follows the existing code style.

CI States

Latest PR Test (Base): ❌ Run #34242452181
Latest PR Test (Extra): ❌ Run #34242451829
Latest PR Test (AMD ROCm 10): ❌ Run #34242452279

@bingxche bingxche changed the title [ROCm] Fix DeepSeek block-FP8 loading on gfx94x [AMD] Fix DeepSeek block-FP8 loading on gfx94x Sep 8, 2026
Signed-off-by: Bingxu Chen <bingxche@amd.com>
@bingxche
bingxche force-pushed the codex/fix-amd-ue8m0-attr branch from 71ec480 to 4788c2c Compare September 8, 2026 07:39
@bingxche bingxche closed this Sep 8, 2026
@bingxche bingxche reopened this Sep 8, 2026
@bingxche
bingxche marked this pull request as ready for review September 8, 2026 14:24
@bingxche bingxche added the run-ci label Sep 8, 2026
@HaiShaw
HaiShaw merged commit 2465ee3 into sgl-project:main Sep 11, 2026
243 of 286 checks passed
mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Sep 15, 2026
Signed-off-by: Bingxu Chen <bingxche@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants