fix qwen2vl grpo for vllm 0.9 and transformers 4.52 #1880

hiyouga · 2025-06-06T07:05:32Z

What does this PR do?

Fixes #1710

vLLM 0.9.0 does not support limit_mm_per_prompt=None; this parameter must be a dict.
Transformers 4.52.* changes the weight keys in the model state dict, causing mismatches with vLLM's weight loader.

See also:
huggingface/transformers#38385
vllm-project/vllm#19054
vllm-project/vllm#19151

Test

run bash examples/grpo_trainer/run_qwen2_5_vl-7b.sh

Checklist Before Submitting

Read the Contribute Guide.
Apply pre-commit checks.
Add [BREAKING] to the PR title if it breaks any API.
Update the documentation about your changes in the docs.
New CI unit test(s) are added to cover the code path.
Rely on existing unit tests on CI that covers the code path.

hiyouga · 2025-06-07T00:19:10Z

resolved conflicts

eric-haibin-lin · 2025-06-07T23:26:52Z

is this change compatible with older version of transformers/vllm?

hiyouga · 2025-06-09T02:27:54Z

@eric-haibin-lin Sure, the older version of transformers will skip the logic in this PR

### What does this PR do? Fixes volcengine#1710 ![image](https://github.com/user-attachments/assets/185d37b6-a4fe-4e89-8eed-72f4477937e8) 1. vLLM 0.9.0 does not support `limit_mm_per_prompt=None`; this parameter must be a `dict`. 2. Transformers 4.52.* changes the weight keys in the model state dict, causing mismatches with vLLM's weight loader. See also: huggingface/transformers#38385 vllm-project/vllm#19054 vllm-project/vllm#19151 ### Test run `bash examples/grpo_trainer/run_qwen2_5_vl-7b.sh` ![image](https://github.com/user-attachments/assets/b8137c87-f250-40d0-b9c3-c3f44f1a40a1) ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl?tab=readme-ov-file#contribution-guide). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl?tab=readme-ov-file#code-linting-and-formatting). - [ ] Add `[BREAKING]` to the PR title if it breaks any API. - [ ] Update the documentation about your changes in the [docs](https://github.com/volcengine/verl/tree/main/docs). - [ ] New CI unit test(s) are added to cover the code path. - [ ] Rely on existing unit tests on CI that covers the code path.

Fixes volcengine#1710 ![image](https://github.com/user-attachments/assets/185d37b6-a4fe-4e89-8eed-72f4477937e8) 1. vLLM 0.9.0 does not support `limit_mm_per_prompt=None`; this parameter must be a `dict`. 2. Transformers 4.52.* changes the weight keys in the model state dict, causing mismatches with vLLM's weight loader. See also: huggingface/transformers#38385 vllm-project/vllm#19054 vllm-project/vllm#19151 run `bash examples/grpo_trainer/run_qwen2_5_vl-7b.sh` ![image](https://github.com/user-attachments/assets/b8137c87-f250-40d0-b9c3-c3f44f1a40a1) - [x] Read the [Contribute Guide](https://github.com/volcengine/verl?tab=readme-ov-file#contribution-guide). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl?tab=readme-ov-file#code-linting-and-formatting). - [ ] Add `[BREAKING]` to the PR title if it breaks any API. - [ ] Update the documentation about your changes in the [docs](https://github.com/volcengine/verl/tree/main/docs). - [ ] New CI unit test(s) are added to cover the code path. - [ ] Rely on existing unit tests on CI that covers the code path.

hebiao064 · 2025-07-09T05:50:43Z

Hi @hiyouga, do you have plan to bump transformer to 4.52 and vllm to 0.9 in main branch of verl?

hiyouga · 2025-07-09T08:24:05Z

@hebiao064 no

### What does this PR do? Reapply #1880 A earlier PR: #2365 accidentally removed the weight key conversion: ##### Why it wasn't caught by CI? Because all CI are based on transformers 4.51, while the issue only happens for transformer 4.52 ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).

### What does this PR do? Reapply volcengine#1880 A earlier PR: volcengine#2365 accidentally removed the weight key conversion: ##### Why it wasn't caught by CI? Because all CI are based on transformers 4.51, while the issue only happens for transformer 4.52 ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).

### What does this PR do? Fixes volcengine#1710 ![image](https://github.com/user-attachments/assets/185d37b6-a4fe-4e89-8eed-72f4477937e8) 1. vLLM 0.9.0 does not support `limit_mm_per_prompt=None`; this parameter must be a `dict`. 2. Transformers 4.52.* changes the weight keys in the model state dict, causing mismatches with vLLM's weight loader. See also: huggingface/transformers#38385 vllm-project/vllm#19054 vllm-project/vllm#19151 ### Test run `bash examples/grpo_trainer/run_qwen2_5_vl-7b.sh` ![image](https://github.com/user-attachments/assets/b8137c87-f250-40d0-b9c3-c3f44f1a40a1) ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl?tab=readme-ov-file#contribution-guide). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl?tab=readme-ov-file#code-linting-and-formatting). - [ ] Add `[BREAKING]` to the PR title if it breaks any API. - [ ] Update the documentation about your changes in the [docs](https://github.com/volcengine/verl/tree/main/docs). - [ ] New CI unit test(s) are added to cover the code path. - [ ] Rely on existing unit tests on CI that covers the code path.

### What does this PR do? Reapply volcengine#1880 A earlier PR: volcengine#2365 accidentally removed the weight key conversion: ##### Why it wasn't caught by CI? Because all CI are based on transformers 4.51, while the issue only happens for transformer 4.52 ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).

### What does this PR do? Reapply volcengine/verl#1880 A earlier PR: volcengine/verl#2365 accidentally removed the weight key conversion: ##### Why it wasn't caught by CI? Because all CI are based on transformers 4.51, while the issue only happens for transformer 4.52 ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).

### What does this PR do? Fixes volcengine#1710 ![image](https://github.com/user-attachments/assets/185d37b6-a4fe-4e89-8eed-72f4477937e8) 1. vLLM 0.9.0 does not support `limit_mm_per_prompt=None`; this parameter must be a `dict`. 2. Transformers 4.52.* changes the weight keys in the model state dict, causing mismatches with vLLM's weight loader. See also: huggingface/transformers#38385 vllm-project/vllm#19054 vllm-project/vllm#19151 ### Test run `bash examples/grpo_trainer/run_qwen2_5_vl-7b.sh` ![image](https://github.com/user-attachments/assets/b8137c87-f250-40d0-b9c3-c3f44f1a40a1) ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl?tab=readme-ov-file#contribution-guide). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl?tab=readme-ov-file#code-linting-and-formatting). - [ ] Add `[BREAKING]` to the PR title if it breaks any API. - [ ] Update the documentation about your changes in the [docs](https://github.com/volcengine/verl/tree/main/docs). - [ ] New CI unit test(s) are added to cover the code path. - [ ] Rely on existing unit tests on CI that covers the code path.

### What does this PR do? Reapply volcengine#1880 A earlier PR: volcengine#2365 accidentally removed the weight key conversion: ##### Why it wasn't caught by CI? Because all CI are based on transformers 4.51, while the issue only happens for transformer 4.52 ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [ ] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).

fix qwen2vl train with transformers latest

96dcf92

hiyouga requested a review from vermouth1992 June 6, 2025 07:09

vermouth1992 previously approved these changes Jun 6, 2025

View reviewed changes

Merge branch 'main' into yaowei/fix_vllm090

f637fcf

hiyouga dismissed vermouth1992’s stale review via f637fcf June 7, 2025 00:18

vermouth1992 approved these changes Jun 7, 2025

View reviewed changes

vermouth1992 merged commit 5937953 into main Jun 7, 2025
39 checks passed

vermouth1992 deleted the yaowei/fix_vllm090 branch June 7, 2025 10:09

hebiao064 mentioned this pull request Jul 9, 2025

[sglang] fix: Fix qwen2vl weight keys issue #2434

Merged

7 tasks

ZhengliangDuanfang mentioned this pull request Jul 27, 2025

SGLang + LoRA: KeyError: 'base_model.model.model.layers.0.self_attn.qkv_proj.base_layer.weight' #2702

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix qwen2vl grpo for vllm 0.9 and transformers 4.52 #1880

fix qwen2vl grpo for vllm 0.9 and transformers 4.52 #1880

Uh oh!

hiyouga commented Jun 6, 2025 •

edited

Loading

Uh oh!

hiyouga commented Jun 7, 2025

Uh oh!

Uh oh!

eric-haibin-lin commented Jun 7, 2025

Uh oh!

hiyouga commented Jun 9, 2025

Uh oh!

hebiao064 commented Jul 9, 2025

Uh oh!

hiyouga commented Jul 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fix qwen2vl grpo for vllm 0.9 and transformers 4.52 #1880

fix qwen2vl grpo for vllm 0.9 and transformers 4.52 #1880

Uh oh!

Conversation

hiyouga commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Test

Checklist Before Submitting

Uh oh!

hiyouga commented Jun 7, 2025

Uh oh!

Uh oh!

eric-haibin-lin commented Jun 7, 2025

Uh oh!

hiyouga commented Jun 9, 2025

Uh oh!

hebiao064 commented Jul 9, 2025

Uh oh!

hiyouga commented Jul 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hiyouga commented Jun 6, 2025 •

edited

Loading