Repository navigation
[GLM 5.3 Perf] Align draft model kv cache dtype align with target, 50% GPU Memory saved for each draft model - #65198
Conversation
Signed-off-by: yewentao256 <zhyanwentao@126.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
yewentao256
changed the title
|
/ci run |
|
✅ Triggered Buildkite CI #99264 for commit |
|
Sorry what use case is this for? Running GLM 5.3 with a small Qwen as draft model doesn't seem realistic A valuable improvement in this code would be expanding the hardware support for default fp8 kv cache for this model beyond sm10x imo |
There was a problem hiding this comment.
@mgoin This PR should be more considered as a cleaning up for the code that won't be achieved and make it easier to read. Basically I don't think people will specifically change the default to eagle or draft_model for GLM 5.3 (no good choice from my understanding now)
A valuable improvement in this code would be expanding the hardware support for default fp8 kv cache for this model beyond sm10x imo
Nice suggestion, I tried on Hopper but seems performance loss, so hold it temporally
Overview
Align draft model kv cache dtype align with target
Currently only
eagle、eagle3、draft_modelmethods left,ngram、suffixdoesn't require a dtype.We don't have eagle for GLM 5.3 draft, for
draft_model, really depends on the user case and we can't test all on that. Case like Qwen will certainly work for fp8 kv cache