[misc] feat: support logging rollout prob vs. actor probs in **multi-turn** for debugging purpose, follow up of #1712 by TomQunChao · Pull Request #2808 · verl-project/verl

TomQunChao · 2025-07-30T03:25:27Z

What does this PR do?

This PR is a follow-up to #1712.

adds support for recording rollout log-probs in multi-turn conversations
moves the diff-computation code into a separate file.

Checklist Before Starting

Search for similar PRs. Paste at least one query link here: ...
Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

Read the Contribute Guide.
Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

CLAassistant · 2025-07-30T03:25:33Z

All committers have signed the CLA.

gemini-code-assist

Code Review

This is a large pull request that introduces a wide range of changes, including support for logging rollout probabilities in multi-turn scenarios, major refactoring of Dockerfiles for different CUDA and ROCm environments, addition of new recipes like one-step-off-policy training and agent-based learning, and numerous updates to documentation and examples.

My review focuses on critical and high-severity issues. I've identified a bug in a new Dockerfile where the CMD instruction is duplicated, and a significant breaking change in pyproject.toml where the minimum required Python version is raised from 3.8 to 3.10, which needs to be clearly communicated to users.

pyproject.toml

vermouth1992 · 2025-07-30T05:30:41Z

@wuxibin89 Could you please help review this PR?

wuxibin89 · 2025-07-31T06:45:17Z

verl/workers/rollout/sglang_rollout/sglang_rollout.py

            reward_scores.append(req.reward_scores)
-            multi_modal_inputs.append(req.multi_modal_inputs)
-            request_ids.append(req.request_id)
+        multi_modal_inputs.append(req.multi_modal_inputs)


#L1162-#L1167 should be under the for-loop?

thanks for the review.. you are right, and I will move them in...

merge main content, resolve 'config key missing' error when executing unit tests

…output data

wuxibin89 · 2025-07-31T08:02:07Z

verl/utils/debug/metrics.py

+def calculate_debug_metrics(data: DataProto) -> dict:
+    rollout_old_log_probs = data.batch["rollout_log_probs"]
+    actor_old_log_probs = data.batch["old_log_probs"]
+    if "loss_mask" in data.batch:


loss_mask has been replaced by response_mask

thanks for the review, I changed it.

…rollout_config

wuxibin89 · 2025-08-01T02:24:07Z

@TomQunChao CI has failed, please fix it.

TomQunChao · 2025-08-01T10:22:31Z

@wuxibin89 Hi Wu, just checking in - all CI checks have passed for this PR. Would you mind taking a look when you get a chance? I'd appreciate your review or any suggestions for improvement. Thanks for your time!

…turn** for debugging purpose, follow up of verl-project#1712 (verl-project#2808) ### What does this PR do? This PR is a follow-up to verl-project#1712. - adds support for recording rollout log-probs in multi-turn conversations - moves the diff-computation code into a separate file. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)

TomQunChao added 3 commits July 30, 2025 10:44

1. add actor-rollout logprob diff in multiturn for debugging purpose

35d160e

update main branch

0dc2a16

update license and lint

cf9f82d

TomQunChao requested review from PeterSH6, SwordFaith, ZihengJiang, chenhaiq, eric-haibin-lin, tongyx361, vermouth1992, wuxibin89, zhaochenyang20 and zw0610 as code owners July 30, 2025 03:25

gemini-code-assist bot reviewed Jul 30, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

Merge branch 'main' into feature/multi_turn_rollout_log_prob

aa4d726

repair pre-commit error

371d645

wuxibin89 reviewed Jul 31, 2025

View reviewed changes

TomQunChao added 3 commits July 31, 2025 15:12

Merge branch 'main' into feature/multi_turn_rollout_log_prob

0fa7e65

merge main content, resolve 'config key missing' error when executing unit tests

moved molti_modal inputs ids and rollout_logprobs inner loop, repair …

7611b28

…output data

remove 'verl' folder in tests to satisfy spec

7ba14d8

wuxibin89 reviewed Jul 31, 2025

View reviewed changes

TomQunChao added 3 commits July 31, 2025 16:37

1. add docstring in metrics 2. add calculate_log_probs config in get_…

123d744

…rollout_config

change loss_mask in metrics.py to adapt to new version

2f4524c

change log to provide detail info

814e186

TomQunChao requested a review from wuxibin89 August 1, 2025 01:49

TomQunChao added 2 commits August 1, 2025 10:53

make default logprob mask to avoid local var referenced before assined

450c0f8

Merge branch 'main' into feature/multi_turn_rollout_log_prob

f6f27bf

wuxibin89 approved these changes Aug 4, 2025

View reviewed changes

wuxibin89 merged commit 551d4cc into verl-project:main Aug 4, 2025
55 of 58 checks passed

aoshen524 mentioned this pull request Feb 7, 2026

[trainer] feat: add per-round logprob mismatch metrics for multi-turn training #5229

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[misc] feat: support logging rollout prob vs. actor probs in multi-turn for debugging purpose, follow up of #1712#2808

[misc] feat: support logging rollout prob vs. actor probs in multi-turn for debugging purpose, follow up of #1712#2808
wuxibin89 merged 13 commits intoverl-project:mainfrom
TomQunChao:feature/multi_turn_rollout_log_prob

TomQunChao commented Jul 30, 2025

Uh oh!

CLAassistant commented Jul 30, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

vermouth1992 commented Jul 30, 2025

Uh oh!

wuxibin89 Jul 31, 2025

Uh oh!

TomQunChao Jul 31, 2025 •

edited

Loading

Uh oh!

wuxibin89 Jul 31, 2025

Uh oh!

TomQunChao Jul 31, 2025

Uh oh!

wuxibin89 commented Aug 1, 2025

Uh oh!

TomQunChao commented Aug 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

TomQunChao commented Jul 30, 2025

What does this PR do?

Checklist Before Starting

Test

Design & Code Changes

Checklist Before Submitting

Uh oh!

CLAassistant commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

vermouth1992 commented Jul 30, 2025

Uh oh!

wuxibin89 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

TomQunChao Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wuxibin89 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

TomQunChao Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

wuxibin89 commented Aug 1, 2025

Uh oh!

TomQunChao commented Aug 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Jul 30, 2025 •

edited

Loading

TomQunChao Jul 31, 2025 •

edited

Loading