Skip to content

[Migration] Migrate GGUF quantization support to plugin#39612

Open
Isotr0py wants to merge 34 commits into
vllm-project:mainfrom
Isotr0py:remove-gguf
Open

[Migration] Migrate GGUF quantization support to plugin#39612
Isotr0py wants to merge 34 commits into
vllm-project:mainfrom
Isotr0py:remove-gguf

Conversation

@Isotr0py
Copy link
Copy Markdown
Member

@Isotr0py Isotr0py commented Apr 12, 2026 •
edited
Loading

Purpose

After this PR, GGUF support will be migrated to https://github.com/vllm-project/vllm-gguf-plugin, you can still use GGUF models normally after plugin installation!

uv pip install vllm-gguf-plugin

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@Isotr0py
draft gguf remove
937eae3 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Apr 12, 2026

Documentation preview: https://vllm--39612.org.readthedocs.build/en/39612/

@mergify mergify Bot added documentation Improvements or additions to documentation ci/build llama Related to Llama models multi-modality Related to multi-modality (#4194) rocm Related to AMD ROCm v1 labels Apr 12, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Apr 12, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Apr 12, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes hardcoded GGUF support from the core vLLM codebase and replaces it with a more extensible ModelFormatHandler architecture. The changes involve deleting GGUF-specific CUDA kernels, documentation, and tests, while refactoring model loaders and layers (Linear, MoE, Embedding) to use generic quantization configuration hooks. I have no feedback to provide.

Isotr0py added 4 commits April 12, 2026 14:44
@Isotr0py
remove gguf materialization
642e1ec 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
clean
1b53ba7 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
remove gguf materialization
6332347 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
clean
a270d85 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
This was referenced Apr 12, 2026
Merged
Open
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Apr 13, 2026

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Apr 13, 2026
@Isotr0py
sync main
f4e25f6 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@mergify mergify Bot removed the needs-rebase label May 5, 2026
@Isotr0py
workaround tie words embedding
4d940cb 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented May 7, 2026

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label May 7, 2026
Isotr0py added 6 commits May 8, 2026 23:25
@Isotr0py
remove siglip maybe_swap_ffn_param
83fe14d 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
pass quant_config
df565fa 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
clean
ca76814 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
clean openpangu
4b1e2d0 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
clean dead code
070ce48 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
clean MoE weight loader
de99b2b 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
Merge remote-tracking branch 'upstream/main' into remove-gguf
df4fa7e 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@mergify mergify Bot removed the needs-rebase label May 20, 2026
Isotr0py added 2 commits May 20, 2026 15:20
@Isotr0py
Merge branch 'main' into remove-gguf
783875e 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
add GGUF doc back
c321339 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Harry-Chen
Copy link
Copy Markdown
Member

Harry-Chen commented May 20, 2026

Just FYI now gguf kernels are moved to csrc/libtorch_stable after #42663 merged. Do you want to include the removal of these files in the PR? @Isotr0py

@Isotr0py
remove gguf kernels again
3467f25 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
Copy link
Copy Markdown
Member Author

Isotr0py commented May 20, 2026

Do you want to include the removal of these files in the PR?

Let's include them in this PR

Isotr0py added 3 commits May 20, 2026 22:25
@Isotr0py
remove ggml bindings
ae0324c 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
fix build
c33cf71 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py
Merge branch 'main' into remove-gguf
ff1750e
@Isotr0py Isotr0py requested a review from dllehr-amd as a code owner May 21, 2026 09:44
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented May 21, 2026

Hi @Isotr0py, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?
mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally: 
# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

@Isotr0py
Merge branch 'main' into remove-gguf
ca5acbf 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented May 23, 2026

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label May 23, 2026
@Isotr0py
Merge branch 'main' into remove-gguf
1abb112 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@mergify mergify Bot removed the needs-rebase label May 23, 2026
@Isotr0py
Copy link
Copy Markdown
Member Author

Isotr0py commented May 25, 2026

@mgoin Can you stamp this PR? Then we can push bnb migration ahead as well: #43529

@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented May 28, 2026

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label May 28, 2026
@Isotr0py
Merge branch 'main' into remove-gguf
51092be 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py Isotr0py requested a review from AndreasKaratzas as a code owner May 28, 2026 03:11
@mergify mergify Bot removed the needs-rebase label May 28, 2026
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented May 29, 2026

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label May 29, 2026
@Isotr0py Isotr0py mentioned this pull request Jun 3, 2026
Open
4 tasks
Sunt-ing added a commit to Sunt-ing/vllm-gguf-plugin that referenced this pull request Jun 3, 2026
@Sunt-ing
[Bugfix][GGUF] Accept file-type-only quant types (IQ2_M, IQ3_XS, ...)…
1c938ce 
… in remote GGUF model IDs

The quant_type in a repo_id:quant_type reference is a GGUF file type
(LlamaFileType), not a tensor type (GGMLQuantizationType). File types
such as IQ2_M, IQ3_M, IQ3_XS and MXFP4_MOE have no GGMLQuantizationType
member, so is_valid_gguf_quant_type() rejected them and the whole
reference was treated as a plain repo id, failing with "Repo id must use
alphanumeric chars...". Accept either enum (LlamaFileType members are
prefixed MOSTLY_) so these file-type-only quants are recognized; the
existing extended-suffix handling (e.g. Q4_K_M -> Q4_K) is preserved.

Ports the fix approved at vllm-project/vllm#44218 to the plugin, as
requested by the maintainer since GGUF support is migrating here
(vllm-project/vllm#39612).

Fixes vllm-project/vllm#42734

Signed-off-by: Ting Sun <suntcrick@gmail.com>
Isotr0py pushed a commit to vllm-project/vllm-gguf-plugin that referenced this pull request Jun 3, 2026
@Sunt-ing
[Bugfix][GGUF] Accept file-type-only quant types (IQ2_M, IQ3_XS, ...)…
69dae43 
… in remote GGUF model IDs (#19)

The quant_type in a repo_id:quant_type reference is a GGUF file type
(LlamaFileType), not a tensor type (GGMLQuantizationType). File types
such as IQ2_M, IQ3_M, IQ3_XS and MXFP4_MOE have no GGMLQuantizationType
member, so is_valid_gguf_quant_type() rejected them and the whole
reference was treated as a plain repo id, failing with "Repo id must use
alphanumeric chars...". Accept either enum (LlamaFileType members are
prefixed MOSTLY_) so these file-type-only quants are recognized; the
existing extended-suffix handling (e.g. Q4_K_M -> Q4_K) is preserved.

Ports the fix approved at vllm-project/vllm#44218 to the plugin, as
requested by the maintainer since GGUF support is migrating here
(vllm-project/vllm#39612).

Fixes vllm-project/vllm#42734

Signed-off-by: Ting Sun <suntcrick@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Harry-Chen Harry-Chen Harry-Chen left review comments

@claude claude[bot] claude[bot] left review comments

@tjtanaa tjtanaa Awaiting requested review from tjtanaa tjtanaa is a code owner

@hmellor hmellor Awaiting requested review from hmellor hmellor is a code owner

@markmc markmc Awaiting requested review from markmc markmc is a code owner

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

@zyongye zyongye Awaiting requested review from zyongye zyongye is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@houseroad houseroad Awaiting requested review from houseroad houseroad is a code owner

@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg ProExpertProg is a code owner

@LucasWilkinson LucasWilkinson Awaiting requested review from LucasWilkinson LucasWilkinson is a code owner

@22quinn 22quinn Awaiting requested review from 22quinn 22quinn is a code owner

@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety is a code owner

@dllehr-amd dllehr-amd Awaiting requested review from dllehr-amd dllehr-amd is a code owner

@AndreasKaratzas AndreasKaratzas Awaiting requested review from AndreasKaratzas AndreasKaratzas is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ci/build documentation Improvements or additions to documentation llama Related to Llama models multi-modality Related to multi-modality (#4194) needs-rebase ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm v1

Projects

AMD
Status: Todo

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Isotr0py @Harry-Chen