[rollout, perf] fix: support profiling for rollout via remote command in async mode

[bithighrr]

… in async mode

What does this PR do?

fix 4575

Fix the core issue that rollout processes cannot be profiled in async mode. Since rollout (e.g., sglang implementation) runs as an independent process in async mode, profiling cannot be triggered via local function calls. This PR adds remote command interfaces to send start/stop instructions to the rollout process, enabling proper control of profiling workflows and successful collection of performance data from rollout processes.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.
The newly added feature has been verified successfully in the sglang 0.5.5 environment. Core validation steps and results are as follows:

1. Add rollout profiling configurations when launching training

python3 -m verl.trainer.main_ppo \
...
    actor_rollout_ref.rollout.name=sglang \
    actor_rollout_ref.rollout.mode=async \
    actor_rollout_ref.rollout.profiler.enable=True \
    actor_rollout_ref.rollout.profiler.tool=torch \
    actor_rollout_ref.rollout.profiler.ranks="[0]" \
    actor_rollout_ref.rollout.profiler.save_path="/path/to/save" \
    actor_rollout_ref.rollout.profiler.tool_config.torch.profile_by_stage=False \
    actor_rollout_ref.rollout.profiler.tool_config.torch.merge_profiles=False \
    actor_rollout_ref.rollout.profiler.tool_config.torch.step_start=1 \
    actor_rollout_ref.rollout.profiler.tool_config.torch.step_end=5 \
    global_profiler.steps='[0, 1, 2]' \
...

2. Validation Results: The rollout process successfully starts profiling, collects performance data as expected, and outputs performance data files to /path/to/save.

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Added three interfaces (start_profile_auto_stop, start_profile, stop_profile) to the BaseRollout class, implemented these interfaces in ServerAdapter, and called them in ActorRolloutRefWorker.rollout_mode to control the rollout profiling workflow.
Added the file verl/utils/profiler/rollout_profile.py, which includes rollout_profile_args and _get_sglang_profile_tags interfaces to encapsulate user-specified parameters for the newly added APIs.

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)
• If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

[CLAassistant]

All committers have signed the CLA.

[gemini-code-assist]

Code Review

This pull request introduces remote command interfaces to enable profiling for rollout processes in async mode, specifically addressing issues with SGLang implementations. The changes include adding new profiling-related functions and methods across verl/utils/profiler/__init__.py, verl/utils/profiler/rollout_profile.py, verl/workers/megatron_workers.py, verl/workers/rollout/base.py, verl/workers/rollout/sglang_rollout/http_server_engine.py, and verl/workers/rollout/sglang_rollout/sglang_rollout.py. The new rollout_profile_args function generates profiling parameters, and abstract methods for profiling control are added to BaseRollout and implemented in ServerAdapter. The integration in ActorRolloutRefWorker allows for starting and stopping profiling based on configuration. Overall, the changes enhance the profiling capabilities for distributed rollout systems.

[wuxibin89]

Support in #4320