[rollout, trainer] feat: extend agent loop for custom implementations

[JoyboyBrian]

What does this PR do?

This PR enhances the extensibility of the agent loop module by:

1. Supporting custom AgentLoopManager: Allow users to specify a custom AgentLoopManager class via configuration (agent_loop_manager_class), enabling external projects to implement their own rollout management logic.

2. Extracting gpt-oss tool response builder: Extract the gpt-oss specific tool response formatting logic into a reusable utility function build_gpt_oss_tool_response_text(), combining existing format_gpt_oss_tool_response_manually() and add_generation_prompt_for_gpt_oss() calls.

3. Adding extension point for custom configurations: Introduce a custom: Optional[dict] field in RolloutConfig to support arbitrary user-defined configurations.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: https://github.com/volcengine/verl/pulls?q=is%3Apr+AgentLoopManager
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)

Test

This PR is a refactoring and extension of existing functionality:

• The build_gpt_oss_tool_response_text() function maintains the same behavior as the original inline code in ToolAgentLoop
• The custom AgentLoopManager loading uses importlib which is a standard Python mechanism
• No behavioral changes to existing workflows when agent_loop_manager_class is not configured

API and Usage Example

1. Custom AgentLoopManager

Users can now specify a custom AgentLoopManager class in their config:

actor_rollout_ref:
  rollout:
    mode: async
    agent:
      agent_loop_manager_class: "myproject.custom_manager.MyAgentLoopManager"

Security Note: The agent_loop_manager_class configuration uses dynamic import via importlib, which will execute the specified module's code. Only use class paths from trusted sources. Ensure your configuration files are not writable by untrusted users or processes.

2. Reusable gpt-oss Tool Response Builder

External projects can now import and use the gpt-oss utility:

from verl.experimental.agent_loop.utils import build_gpt_oss_tool_response_text

# Build gpt-oss tool response text (combines formatting + generation prompt)
tool_response_text = build_gpt_oss_tool_response_text(messages, tool_call_names)

3. Custom Configuration Extension Point

actor_rollout_ref:
  rollout:
    custom:
      my_custom_key: "my_custom_value"
      another_setting: 123

Design & Code Changes

High-Level Design

┌─────────────────────────────────────────────────────────────────┐
│                      ray_trainer.py                             │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  if agent_loop_manager_class configured:                  │  │
│  │      dynamically import via importlib                     │  │
│  │  else:                                                    │  │
│  │      use default AgentLoopManager                         │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│         verl/experimental/agent_loop/utils.py (new export)      │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  build_gpt_oss_tool_response_text()                       │  │
│  │    - combines format_gpt_oss_tool_response_manually()     │  │
│  │    - and add_generation_prompt_for_gpt_oss()              │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Specific Changes

File	Changes
verl/trainer/ppo/ray_trainer.py	Added dynamic loading of custom AgentLoopManager via importlib
verl/workers/config/rollout.py	Added agent_loop_manager_class to AgentLoopConfig; Added custom field to RolloutConfig
verl/experimental/agent_loop/utils.py	Added build_gpt_oss_tool_response_text() combining existing gpt-oss formatting functions
verl/experimental/agent_loop/tool_agent_loop.py	Refactored to use build_gpt_oss_tool_response_text() for cleaner code

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation. N/A - This PR targets advanced users extending verl. The code is self-documenting with type hints and docstrings, and the PR description includes complete usage examples.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. N/A - This is a pure refactoring with no behavioral changes: (1) build_gpt_oss_tool_response_text() maintains identical behavior to the original inline code; (2) agent_loop_manager_class is optional and defaults to the existing AgentLoopManager; (3) existing agent loop tests already cover the default code path.
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[gemini-code-assist]

Code Review

This pull request enhances the extensibility of the agent loop by allowing a custom AgentLoopManager, extracting tokenization utilities for reusability, and adding a generic custom configuration field. The changes are well-structured and the refactoring of tokenization logic into utility functions is a good step towards better modularity. My review focuses on the new dynamic import mechanism for the custom AgentLoopManager. I have identified a robustness issue that could lead to a crash if the configuration is malformed, and a potential security vulnerability related to code injection. I've provided suggestions to make the implementation more robust and secure.

[wuxibin89]

Please move this to a util function

[JoyboyBrian]

Thanks for the suggestion!
I've refactored this by adding a load_class_from_fqn() utility function to verl/utils/import_utils.py. This function handles FQN parsing, dynamic import, and provides clear error messages. Commit: 3b78336
The code in ray_trainer.py is now simplified from ~20 lines to just 4 lines:

  manager_class_fqn = self.config.actor_rollout_ref.rollout.get("agent", {}).get("agent_loop_manager_class")
  if manager_class_fqn:
      AgentLoopManager = load_class_from_fqn(manager_class_fqn, "AgentLoopManager")
  else:
      from verl.experimental.agent_loop import AgentLoopManager

This also removes the now-unused import importlib from the file.

[wuxibin89]

This field seems not used?

[JoyboyBrian]

This field is designed as an extension point for downstream projects to pass custom configurations without modifying core config classes.

For example, a project implementing remote rollout could use:

actor_rollout_ref:
  rollout:
    custom:
      remote_rollout:
        server_url: "https://..."
        callback_url: "http://..."
        timeout_seconds: 300

Then access it via config.actor_rollout_ref.rollout.get("custom", {}) in their custom AgentLoopManager.

Happy to add a docstring clarifying this is an extension point!

[JoyboyBrian]

@wuxibin89 thanks for the review!

1. “Move this to a util function”: addressed — I extracted the dynamic import logic into a reusable utility (load_class_from_fqn) and simplified the ray_trainer.py path accordingly.
   Commit: 3b78336

2. “This field seems not used?”: clarified — RolloutConfig.custom is intentionally an extension point for downstream projects, and I added inline documentation to make that intent explicit.

Would you mind taking another look when you have a chance? 🙏

[wuxibin89]

It's a bit redundant here, we can safely pass tools=None to processor.apply_chat_template?

[JoyboyBrian]

Reverted!

[wuxibin89]

I don't quit understand why we need these util function, it's more straightforward to call processor and tokenizer function directly.

• apply_chat_template_with_processor
• apply_chat_template_with_tokenizer
• tokenize_with_processor

[JoyboyBrian]

I don't quit understand why we need these util function, it's more straightforward to call processor and tokenizer function directly.

• apply_chat_template_with_processor
• apply_chat_template_with_tokenizer
• tokenize_with_processor

You're right. I initially extracted these utilities for consistency with initialize_system_prompt which was already in chat_template.py, thinking it might help centralize chat template related operations. But looking at it again, these wrappers are too thin to justify the indirection. Reverted.