Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
@tanth47 glm-5.2-fp8 with redhat-ai dspark have some problem ,resault is that , |
|
Thanks for the heads up. That is exactly the failure mode I want to pin down before claiming accuracy/perf. This PR is only the GLM-5.2 DSpark config loader. I am splitting the runtime fixes separately; the first stacked one is the Could you share the setup you used? Target/draft model, SGLang commit or image, launch command/env, TP/GPU/quantization, prompt set, sampling params/chat template, and how you computed AR/AL would be enough. If you have a small accept-length dump, even better. I will try to reproduce the |
|
Accuracy update from a clean one-lane MI350 TP4 run. This is not a claim that this config-only PR includes all runtime fixes; those remain split into follow-up PRs. I am adding the numbers here because this thread raised the Setup:
Results: Artifacts: Important caveat:
So far I cannot reproduce the “most requests <30% AR” behavior on this 256-prompt ultrachat slice. If you can share your exact model/image/launch command/prompt set/sampling params and AR calculation, I will try to reproduce that case directly. |
|
run command acceptance log: benchmark command: summary: |
|
more detail ablout glm-5.2-dspark refernce: |
|
Thanks for the details. This helps. One important difference: your run is vLLM On the SGLang side, I reran a clean MI350 TP4 gate with the RedHat draft: So I still cannot reproduce "almost all requests accept 0%" on the SGLang TP4 setup. Your log is interesting though: it has both very low windows ( I will keep the SGLang runtime fixes split out from this config PR. If you can share a small request sample or the exact benchmark prompt template, I can run that same traffic through SGLang and compare AR/AL directly. |
|
you can try Radom dataset?
发自我的iPhone
…------------------ Original ------------------
From: Tan Trinh ***@***.***>
Date: Tue,Jul 14,2026 0:50 PM
To: sgl-project/sglang ***@***.***>
Cc: yiminghub2024 ***@***.***>, Comment ***@***.***>
Subject: Re: [sgl-project/sglang] Support GLM-5.2 DSpark draft config (PR #31047)
tanth47 left a comment (sgl-project/sglang#31047)
Thanks for the details. This helps.
One important difference: your run is vLLM 0.25.0 with TP8 + expert parallel + long context, not the SGLang path in this PR. So I do not want to compare the numbers 1:1 yet.
On the SGLang side, I reran a clean MI350 TP4 gate with the RedHat draft:
target: zai-org/GLM-5.2-FP8 draft: RedHatAI/GLM-5.2-speculator.dspark dataset: mgoin/GLM-5.2-FP8-magpie-ultrachat 256 prompts x 128 tokens DSpark compact, no CUDA graph 256/256 OK aggregate AL: 4.5676 aggregate AR: 51.61% mean AL: 4.8148 mean AR: 55.23%
So I still cannot reproduce "almost all requests accept 0%" on the SGLang TP4 setup.
Your log is interesting though: it has both very low windows (mean acceptance length ~1.35) and later better windows (~3.8-3.9). That makes me suspect workload/template/runtime mode differences rather than just the RedHat draft being bad.
I will keep the SGLang runtime fixes split out from this config PR. If you can share a small request sample or the exact benchmark prompt template, I can run that same traffic through SGLang and compare AR/AL directly.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you commented.Message ID: ***@***.***>
|
@tanth47 I have similar test result with @yiminghub2024
So as per my test, the current redhat dspark model in vllm tp8+ep path, the ar/al in long prompt is very low, so probrably you need to test against 8K or more input token to verify its al/ar |
cb58083 to
0d3903e
Compare
|
Looking forward to it! |
|
Hello, I deployed it on H800 using your commit. The startup command is as follows: python3 -m sglang.launch_server Then, several requests were tested, such as: But it turns out the acceptance rates are very low: |
|
we also see low accept rate in glm-5.2-fp8.less than 1% using dspark（with vllm），around 15% using eager（with sglang） |
0d3903e to
fc09f91
Compare
tanth47
changed the title
|
Cleaned this PR up and rebased it on current main. It now includes the generic correctness pieces needed by this checkpoint format:
It still does not register MI350 gate at The AR caveat from the discussion still stands: the current RedHat checkpoint is workload-sensitive, so this is not a universal speedup claim. |
|
@yiminghub2024 @lisuizhe @dongyibo @enternal111 Thanks all, these reports make sense. The current RedHat checkpoint is clearly workload-sensitive: UltraChat looks reasonable, but random, long-context, code, and very short prompts can be much worse. I've scoped this PR to the config and runtime contract only. It does not enable GLM DSpark or claim general speedup; better checkpoints can plug into the same path later. |
|
@tanth47 hello~ |
5 participants
What this PR does
DSparkDraftModel, nestedtransformer_layer_config, auxiliary hidden-state layers, and proposal gamma.gammaproposed tokens.Commit order
Support GLM-5.2 DSpark draft configAlign DSpark draft block with anchor slotFix DFlash DSpark verify length metadataFix DSpark compact anchor strideEach commit has its focused regression tests.
Validation
Validated at
fc09f91be, rebased on upstreamf5155d960, using one MI350 GPU:Also passed
git diff --checkandpy_compilefor every changed Python file.Scope
DSparkDraftModelentry class.Acceptance rate
The current RedHat checkpoint is workload-sensitive. UltraChat-like prompts work reasonably well, while random, long-context, and code traffic can be much worse.
This PR validates the config/runtime contract so improved GLM speculators can be plugged in later. It does not claim workload-independent speedup from the current checkpoint.
CI States
Latest PR Test (Base): ❌ Run #30148542704
Latest PR Test (Extra): ❌ Run #30148542580