[Leamonz]

What does this PR do?

This PR fixes NVFP4 QAT training failures caused by a mismatch between the install script dependency stack and verl's QAT runtime requirements.

The official install script still installed vllm==0.11.0, while the NVFP4 QAT path needs compatibility with newer vLLM Marlin helper signatures (>=0.12.0) and compressed-tensors(>=0.14.x) API/import. This PR updates the vLLM install pin to 0.12.0 and adds compatibility shims so QAT weight loading works across the affected vLLM and compressed-tensors versions.

Duplicate check: the closest open PR found was #6455, but that PR targets TRT-LLM rollout W4A4 IPC / dynamic quantization behavior. This PR targets vLLM installation, vLLM QAT patch compatibility, and compressed-tensors quantizer compatibility, so it does not duplicate #6455.

AI assistance was used in preparing this PR. The submitting author is responsible for reviewing every changed line and validating the final results before requesting review.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+NVFP4+QAT
  • Also checked:
    • https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+NVFP4+QAT+vLLM+compressed-tensors
    • https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+vllm+compressed+tensors+qat
• Format the PR title as [{modules}] {type}: {description}

Test

python -m pytest tests/utils/test_qat_vllm_patch_on_cpu.py -q

Result: all passed.

pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always

Result: all passed.

NVFP4 QAT training is conducted and finished with no error regarding packages version dependency. This is to ensure the changes successfully solve the version mismatch problem. The training script is attached with PR: run_qwen3_0.6b_w4a16.sh.

API and Usage Example

No API or config changes are required. Existing NVFP4 QAT configs continue to use the same entrypoints. The install script now installs the compatible vLLM version:

bash scripts/install_vllm_sglang_mcore.sh

Design & Code Changes

• Update scripts/install_vllm_sglang_mcore.sh to install vllm==0.12.0.
• Add vLLM-version-aware wrappers for gptq_marlin_repack and marlin_permute_scales, only passing is_a_8bit when supported by the installed vLLM version.
• Support the MoE compressed-tensors method rename between vLLM versions by trying both patch targets and applying the first available one.
• Make the QAT quantizer compatible with compressed-tensors import-path changes for NVFP4PackedCompressor.
• Gate QuantizationArgs.scale_dtype by compressed-tensors version so older versions do not receive an unsupported argument.
• Add CPU tests for the vLLM Marlin helper compatibility behavior.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide (https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).

• Apply pre-commit checks (https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): pre-commit install && pre-commit run --all-files
  --show-diff-on-failure --color=always

• Add / Update the documentation (https://github.com/verl-project/verl/tree/main/docs). Not updated; this PR does not change the public API or user-facing QAT config.

• Add unit or end-to-end test(s) to the CI workflow (https://github.com/verl-project/verl/tree/main/.github/workflows) to cover all the code. Added
  tests/utils/test_qat_vllm_patch_on_cpu.py; no new workflow is required for this CPU pytest.

• Once your PR is ready for CI, send a message in the ci-request channel (https://verl-project.slack.com/archives/C091TCESWB1) in the verl Slack workspace
  (https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try the Feishu group (飞书群)
  (https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

• If this PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main. Not applicable; this PR does not change recipe.

[CLAassistant]

All committers have signed the CLA.

[gemini-code-assist]

Code Review

This pull request updates the vLLM dependency to version 0.12.0 and introduces compatibility layers to support multiple versions of both vLLM and compressed-tensors. Key changes include wrapping vLLM's repack and permute operations to conditionally pass arguments based on the active version, updating patch targets to try multiple candidate paths sequentially, and adding unit tests to verify compatibility. The reviewer feedback recommends caching the vLLM version check in _is_vllm_ge_012 to avoid repeated import and parsing overhead inside loops, and catching ImportError alongside AttributeError when applying patches to handle missing modules more robustly.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[sophiayyya]

The QAT part looks good to me. @wuxibin89 Could you please help check whether this script will affect other modules?
scripts/install_vllm_sglang_mcore.sh

[wuxibin89]

The QAT part looks good to me. @wuxibin89 Could you please help check whether this script will affect other modules? scripts/install_vllm_sglang_mcore.sh

@sophiayyya The scripts/install_vllm_sglang_mcore.sh is not used anymore, we recommend to use pre-built docker image.

[wuxibin89]

@Leamonz Did you verify this PR with NVFP4 QAT training?

[Leamonz]

@Leamonz Did you verify this PR with NVFP4 QAT training?

Yes, I conducted NVFP4 QAT training to verify the changes, and the experiment successfully launched with no errors regarding vllm_patch.py or quantizer.py. And this is the training script used: run_qwen3_0.6b_w4a16.sh.

[sophiayyya]

Hi @Leamonz In the qat experiment, we use vllm0.12 for megatron and 0.15 for fsdp. So I am not sure why you need to modify the code with vllm0.12. Let me verify with this script.

[Leamonz]

Hi @Leamonz In the qat experiment, we use vllm0.12 for megatron and 0.15 for fsdp. So I am not sure why you need to modify the code with vllm0.12. Let me verify with this script.

Thanks for your time! I wasn't able to use pre-built verl docker images in my environment, so I followed the Install from custom environment guide from the documentation and ran into the compatibility issue. My python environment for the script is as follows:

verl==0.8.0.dev0
torch==2.8.0+cu128
vllm==0.11.0
compressed_tensors==0.11.0

[wuxibin89]

@Leamonz The document is quite outdated, you can follow Dockerfile to install deps in you local environment.
https://github.com/verl-project/verl/blob/main/docker/Dockerfile.stable.vllm

Please test with newer vllm version, we don't maintain backward compatibility for vllm<0.12 anymore.

[sophiayyya]

@Leamonz The guide of QAT locates here. https://github.com/verl-project/verl-recipe/tree/main/qat
Have you referred to this guide? Here noticed the vllm version of Megatron and FSDP backend.

[Leamonz]

Hi @sophiayyya @wuxibin89 Thanks for your time! I think I overlooked these documents, I'll check the given installation guides for qat features.