[Bugfix] Fix quantization skip modules logic

[jeejeelee]

Motivation

Some models, such as QWEN25-VL, have modified their layer hierarchy compared to their original transformers implementation. This change causes quantization's skip modules to become ineffective, leading to incorrect initialization of linear methods.

Reproduce code

import vllm
llm = vllm.LLM(
    "unsloth/Qwen2.5-VL-72B-Instruct-unsloth-bnb-4bit",
    max_model_len=3200,
    quantization="bitsandbytes",
    load_format="bitsandbytes",
    trust_remote_code=True,
)

TODO

• Investigate other quantization method (e.g. AWQ)

• Optimize the implementation logic

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mgoin]

Maybe we should introduce a common ignored_modules or ignored_prefixes to QuantizationConfig like packed_modules_mapping https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/base_config.py#L60-L66

Then each quant config can convert their specific llm_int8_skip_modules, modules_to_not_convert, etc in a canonical format in ignored_modules. This will also allow us to generalize the is_layer_skipped function

[kylesayrs]

I'd support an implementation like this as well. This current implementation could fail to properly map module names in nested models.

modules_to_not_convert = ["SubModel.A"]
SubModel.hf_to_vllm_mapper = Mapper(orig_to_new_prefix={"A": "B"})

Note that "SubModel.A" will not match because "SubModel.A" does not start with "A"

This is a fairly minor issue, but something to keep in mind.

Another implementation could look like this:

1. Add a mutable ignored_modules attribute to QuantizationConfig
2. At construction-time, using the method-specific constructor to populate the ignored_modules attribute from disk
3. At initialize-time, within SupportsQuant, use the given model prefix and mapper to update the ignored_modules list with the proper model-specific mapping
   a. ignored_modules = [prefix + hf_to_vllm_mapper[module - prefix] for module in ignored_modules]

This has the advantage of further standardizing around the QuantizationConfig base, as well as supporting mapping with nested models

[kylesayrs]

@jeejeelee Here's a WIP of what that might look like: #14635

[jeejeelee]

@kylesayrs Can you provide an example?

[mgoin]

Why is this needed after we added SupportsQuant (#13104), I thought getting the packed_modules_mapping from the model to the quant config was the main purpose of that. cc @kylesayrs

[kylesayrs]

The _configure_packed_modules_mapping function needs to remain in place until SupportsQuant has been added to all applicable models

[jeejeelee]

Close due to #14635