[Bugfix] Expand quantization method support in perf metrics

[thillai-c]

Purpose

The MFU (Model Flops Utilization) metrics module uses AttentionQuantizationConfigParser and FfnQuantizationConfigParser to determine weight_byte_size for flops/memory estimation. Currently, only 3 quantization methods are supported (fp8, fbgemm_fp8, mxfp4). All other methods — including widely-used ones like GPTQ, AWQ, and BitsAndBytes — raise InvalidComponent, silently breaking MFU reporting for quantized models.

This PR resolves the multiple FIXME comments in vllm/v1/metrics/perf.py requesting broader quantization method support:

• FIXME: Add more parsing logic for different quant methods.
• FIXME: This is a hacky coarse-grained fp8 quantization detection.

Changes

• Added _QUANT_WEIGHT_BYTE_SIZE dict — A shared mapping of 22 quantization method names to their effective weight_byte_size (1 byte for FP8 variants, 0.5 bytes for INT4/FP4 variants), used by both parsers.
• Refactored both quantization parsers to use the shared dict lookup instead of separate if/elif/else chains. Unknown methods still raise InvalidComponent, now with a descriptive error message including the method name.
• Added 4 new tests: parametrized tests covering all INT4/FP4 and FP8/INT8 methods, unknown method error handling, and end-to-end ModelMetrics aggregation with a quantized config.

Newly supported methods

Byte size	Methods
1 (FP8/INT8)	fp8, fbgemm_fp8, ptpc_fp8, fp_quant, modelopt, modelopt_mxfp8, experts_int8
0.5 (FP4/INT4)	mxfp4, awq, awq_marlin, gptq, gptq_marlin, bitsandbytes, modelopt_fp4, petit_nvfp4, gguf, compressed-tensors, torchao, quark, moe_wna16, inc, cpu_awq

Note: Methods like GPTQ and BitsAndBytes support variable bit-widths (e.g., 4-bit and 8-bit). We default to the most common configuration (4-bit). The existing FIXME comments about per-layer "ignored layers" handling remain as a separate concern for a follow-up.

Test Plan

Run the perf metrics test suite:

pytest tests/v1/metrics/test_perf_metrics.py -v

New tests added

Test	Description
test_quantization_config_parser_int4_methods[<method>]	Parametrized across all 15 INT4/FP4 methods, asserts weight_byte_size == 0.5 for both Attention and FFN parsers
test_quantization_config_parser_fp8_methods[<method>]	Parametrized across all 7 FP8/INT8 methods, asserts weight_byte_size == 1 for both parsers
test_quantization_config_parser_unknown_method	Verifies that an unrecognized quant method correctly raises InvalidComponent
test_quantized_model_metrics_aggregation	End-to-end test that ModelMetrics produces valid, consistent flops breakdowns with a GPTQ-quantized model config

Test Results

All existing tests continue to pass. The 4 new tests (expanding to 22+ via parametrization) verify correctness for every supported quantization method.

tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[mxfp4]        PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[awq]          PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[awq_marlin]   PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[gptq]         PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[gptq_marlin]  PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[bitsandbytes] PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[modelopt_fp4] PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[petit_nvfp4]  PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[gguf]         PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[compressed-tensors] PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[torchao]      PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[quark]        PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[moe_wna16]    PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[inc]          PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_int4_methods[cpu_awq]      PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_fp8_methods[fp8]           PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_fp8_methods[fbgemm_fp8]    PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_fp8_methods[ptpc_fp8]      PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_fp8_methods[fp_quant]      PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_fp8_methods[modelopt]      PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_fp8_methods[modelopt_mxfp8] PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_fp8_methods[experts_int8]  PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantization_config_parser_unknown_method             PASSED
tests/v1/metrics/test_perf_metrics.py::test_quantized_model_metrics_aggregation                   PASSED

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request is a solid improvement that expands support for various quantization methods in the performance metrics module. The refactoring to use a shared dictionary for quantization method properties is a clean and maintainable approach. The accompanying tests are thorough and cover all the new additions. I have one suggestion to refactor the new tests to reduce code duplication and further improve maintainability.

[gemini-code-assist]

The two new test functions, test_quantization_config_parser_int4_methods and test_quantization_config_parser_fp8_methods, are nearly identical. To improve maintainability and reduce code duplication, they can be combined into a single, more general parametrized test that iterates over all items in _QUANT_WEIGHT_BYTE_SIZE. This will also make it easier to add tests for new quantization sizes in the future.

@pytest.mark.parametrize("quant_method, expected_byte_size",
                         list(_QUANT_WEIGHT_BYTE_SIZE.items()))
def test_quantization_config_parser(quant_method, expected_byte_size):
    """Test quantization parsers with all supported methods."""

    class MockQuantConfig:
        def get_name(self):
            return quant_method

    hf_config = Qwen3Config(
        hidden_size=2048,
        num_attention_heads=16,
        intermediate_size=8192,
        num_hidden_layers=1,
    )
    vllm_config = create_mock_vllm_config(
        hf_config, quant_config=MockQuantConfig()
    )

    attn_result = AttentionMetrics.get_parser().parse(vllm_config)
    assert attn_result.weight_byte_size == expected_byte_size, (
        f"Expected {expected_byte_size} for {quant_method}, "
        f"got {attn_result.weight_byte_size}"
    )

    ffn_result = FfnMetrics.get_parser().parse(vllm_config)
    assert ffn_result.weight_byte_size == expected_byte_size, (
        f"Expected {expected_byte_size} for {quant_method}, "
        f"got {ffn_result.weight_byte_size}"
    )

[mergify]

Hi @thillai-c, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[markmc]

lgtm, thanks

[thillai-c]

Hi @markmc, the CI pipeline takes ~2 hours to complete, and the PR repeatedly becomes out-of-date before I get a chance to merge.

Would you be able to enable auto-merge or help merge this once checks pass? That would help avoid rerunning the full pipeline again.