Skip to content

[Optimize] Support WINT8 and group scale for Machete #3905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Sep 15, 2025
Merged

[Optimize] Support WINT8 and group scale for Machete #3905

merged 12 commits into from
Sep 15, 2025

Conversation

@Sunny-bot1
Copy link
Collaborator

@Sunny-bot1 Sunny-bot1 commented Sep 4, 2025 •
edited
Loading

支持Machete WINT8，对PR：#3561 的补充

性能测试

模型

Qwen3-30B-A3B WINT8 TPS提升7% 🚀

backend 解码速度 TPS
paddle 35.86 6886.35
machete 38.85 7362.53

kernel

k, n = 7168, 1536

m paddle machete
32 23.999 19.654
64 24.262 23.046
128 24.601 31.570
256 32.015 26.281
512 51.989 28.038
1024 73.391 48.761
2048 137.550 76.463
4096 208.295 143.307

k, n = 2048, 5120

m paddle machete
32 9.218 11.532
64 11.439 13.187
128 17.359 14.134
256 24.262 17.138
512 39.237 24.786
1024 65.093 39.154
2048 106.475 68.299
4096 205.088 118.632

@paddle-bot
Copy link

paddle-bot bot commented Sep 4, 2025

Thanks for your contribution!

Sunny-bot1 and others added 11 commits September 8, 2025 15:50
@Sunny-bot1
Merge branch 'develop' of https://github.com/PaddlePaddle/FastDeploy …
a3c3c18 
…into machete_wint8
@Sunny-bot1
Merge branch 'develop' of https://github.com/PaddlePaddle/FastDeploy …
6903297 
…into machete_wint8
@Sunny-bot1
Merge branch 'develop' of https://github.com/PaddlePaddle/FastDeploy …
c37f669 
…into machete_wint8
@Sunny-bot1
Merge branch 'develop' of https://github.com/PaddlePaddle/FastDeploy …
d81ef5b 
…into machete_wint8
@Sunny-bot1
support machete wint8 model
00fb14e
@Sunny-bot1
Merge branch 'develop' of https://github.com/PaddlePaddle/FastDeploy …
40dd91a 
…into machete_wint8
@Sunny-bot1
Merge branch 'develop' of https://github.com/PaddlePaddle/FastDeploy …
4075f68 
…into machete_wint8
@Sunny-bot1
fix bias
7048df6
@Sunny-bot1
Merge branch 'develop' into machete_wint8
992c5d8
@Sunny-bot1
fix
f6b9b4d
@Sunny-bot1
Merge branch 'develop' into machete_wint8
88a5566
@zhoutianzi666 zhoutianzi666 merged commit b1a5b75 into PaddlePaddle:develop Sep 15, 2025
34 of 39 checks passed
Sunny-bot1 added a commit to Sunny-bot1/FastDeploy that referenced this pull request Sep 18, 2025
@Sunny-bot1
[Optimize] Support WINT8 and group scale for Machete (PaddlePaddle#3905)
bdd8ddb
@Sunny-bot1 Sunny-bot1 mentioned this pull request Sep 19, 2025
Merged
Jiang-Jia-Jun pushed a commit that referenced this pull request Sep 19, 2025
@Sunny-bot1
[CP2.2] Machete support group scale & wint8 & v1 loader (#4166)
4f460db 
* support v1 loader for machete (#3999)

* [Optimize] Support WINT8 and group scale for Machete (#3905)

* [Optimize] Machete using group scale default (#4121)
@Sunny-bot1 Sunny-bot1 mentioned this pull request Oct 20, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhoutianzi666 zhoutianzi666 zhoutianzi666 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Sunny-bot1 @zhoutianzi666