Skip to content

Make Marlin incompatible AWQ models work#3759

Closed
bjmsong wants to merge 3 commits intosgl-project:mainfrom
bjmsong:marlin_uncompatible
Closed

Make Marlin incompatible AWQ models work#3759
bjmsong wants to merge 3 commits intosgl-project:mainfrom
bjmsong:marlin_uncompatible

Conversation

@bjmsong
Copy link
Contributor

@bjmsong bjmsong commented Feb 21, 2025

Motivation

Relate to #3571, some AWQ models are incompatible with marlin kernels.

Modifications

Use unoptimized kernel if the models are incompatible with marlin kernels.

test script

python examples/runtime/engine/offline_batch_inference.py --model=${DeepSeek-V2-Lite-Chat-AWQ} --trust-remote-code

refer to this PR

Checklist

@merrymercy merrymercy requested a review from HaiShaw as a code owner March 3, 2025 08:12
@github-actions github-actions bot closed this May 30, 2025
@github-actions
Copy link
Contributor

This pull request has been automatically closed due to inactivity. Please feel free to reopen it if needed.

@phuc2k3bn-jpg
Copy link

At present, this bug is fixed, isn't it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants