[rollout] chore: update reward loop file names

[yyDing1]

What does this PR do?

Previous PRs about reward loop may introduce some different names, which may cause foncusion to developers. I plan to rename them to clearer folder/file/class names:

For file/folder name:

• verl/experimental/reward -> verl/experimental/reward_loop (this PR)
• verl/experimental/reward_loop/reward_manager -> verl/experimental/reward_loop/reward_loop.py (same as naming logic of agent_loop)
• verl/experimental/reward/reward_loop -> verl/experimental/reward/reward_manager (the new name is the same as the legacy reward manager, and they actually do the same thing; the new reward manager will deprecate the legacy in the future)

For class name:

• The reward manager class name will be kept the same as the old ones, like DAPORewardManager, ...

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[gemini-code-assist]

Code Review

This pull request aims to clarify file and folder names related to the reward loop. It renames verl/experimental/reward/reward_manager.py to verl/experimental/reward/reward_loop.py and updates the corresponding import. However, this change introduces a critical issue: a name collision between the new reward_loop.py module and the existing reward_loop/ package directory at the same level. This will cause import errors and break the application. I've left a comment detailing the issue and suggesting a path to resolution, which likely involves completing the other renames mentioned in the PR description.

[yyDing1]

/gemini review

[gemini-code-assist]

Code Review

This pull request aims to clarify file names related to the reward loop, but the current changes introduce significant confusion by swapping the names 'reward_loop' and 'reward_manager'. Specifically, the file reward_manager.py has been renamed to reward_loop.py, while the directory reward_loop/ has been renamed to reward_manager/. This name swap is highly counter-intuitive and will likely make the codebase harder to understand and maintain. This confusion is also reflected in the PR description, where the described renames do not accurately match the changes in the patch.

I strongly recommend reconsidering this renaming strategy. A better approach would be to choose new, distinct names that clearly describe the role of each component. For example:

• Rename verl/experimental/reward/reward_loop.py (which contains RewardLoopManager and RewardLoopWorker) to something like reward_service.py.
• Rename the directory verl/experimental/reward/reward_manager/ (which contains different reward calculation strategies) to reward_calculators/ or reward_strategies/.
• Rename the classes inside, like RateLimitedRewardLoopManager, to RateLimitedRewardCalculator.

This would eliminate the ambiguity and make the structure much clearer.