[perf] feat: verl profiler system support Agent Loop scenario and integrate torch.profiler

[mengchengTang]

What does this PR do?

Summary
This PR enhances the verl profiling system to support the Agent Loop architecture (decoupled inference and training processes). It integrates torch.profiler as a core backend within the DistProfiler framework, providing a unified performance tuning interface for both distributed training workers and asynchronous rollout servers.

Key Enhancements
Agent Loop Support: Implemented a coordinated control flow that allows the RayTrainer to explicitly trigger profiling sessions on remote inference servers (vLLM/SGLang) via AgentLoopManager.

• For vLLM, we leverage the AsyncLLM profiling interface.
• For SGLang, we utilize the TokenizerCommunicatorMixin profiling API.

Unified Torch Profiler: Integrated the native PyTorch Profiler into the verl ecosystem, supporting both continuous and discrete collection modes consistent with Nsight systems and NPU.

Profiling Workflow
The provided Sequence Diagram illustrates the decoupled profiling logic:
Rollout Phase: Profiling is managed through explicit RPC calls (start_profile/stop_profile) to the inference engine's server interface (AsyncLLM / TokenizerCommunicatorMixin), ensuring capture is synchronized with generation steps.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[CLAassistant]

All committers have signed the CLA.

[gemini-code-assist]

Code Review

This pull request introduces profiling capabilities for the agent loop by adding start_profile and stop_profile methods to AgentLoopManager, vLLMHttpServerBase, and vLLMReplica. The changes correctly propagate the profiling calls down to the workers. My review includes one high-severity suggestion to fix a blocking call (ray.get) within an asynchronous actor in vLLMHttpServerBase, which should be converted to a non-blocking asyncio.gather to prevent blocking the event loop and to maintain consistency with other asynchronous methods in the class.

[tardis-key]

It can be seen that the overall process control and the actual profiling startup are far apart, and the original solution requires passing the calling function many times, which is not elegant enough.

[tardis-key]

prof_step and profile_step are confusing. And new variables are unnecessary, the original role&profile_step can still meet the requirements.

[mengchengTang]

The corresponding variable(s) have been deprecated in the new solution.

[tardis-key]

@FightingZhen @wuxibin89 ready for review. This pr is bugfix for legacy_workers.

[wuxibin89]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and valuable enhancement to the verl profiling system. It successfully integrates torch.profiler and extends support to the Agent Loop architecture, providing a more unified and powerful performance tuning experience. The refactoring of the profiler implementations under a central DistProfiler is a solid architectural improvement. While the overall changes are excellent, I've identified a few critical issues related to null-pointer exceptions in the new profiler initialization logic for sglang and vllm rollout servers. These could lead to crashes if profiling is configured in specific ways. Addressing these will make the new profiling system much more robust.

[gemini-code-assist]

There's a potential AttributeError here. If self.config.profiler is not set in the configuration, profiler_config will be None. In that case, the else block is executed, and accessing profiler_config.tool will raise an exception. You should add a check to ensure profiler_config is not None before accessing its attributes.

Suggested change

	else:
	logger.warning(f"agent loop only support torch and npu profiler, got {profiler_config.tool}")
	self.tool_config = None
	if profiler_config is not None:
	logger.warning(f"agent loop only support torch and npu profiler, got {profiler_config.tool}")
	self.tool_config = None

[mengchengTang]

Fixed. I have added a check if profiler_config is not None: to ensure we only access .tool when the config exists. If profiler_config is None, we skip the logic and pass None to DistProfiler, which handles it gracefully.

[gemini-code-assist]

This condition can lead to a crash. If profiling is enabled with a tool other than torch or npu (e.g., nsys), self.tool_config will be None from the __init__ method. However, self.profiler_controller.check_enable() will return True, causing the code to enter this block and fail with an AttributeError on self.tool_config.contents. You should add a check to ensure self.tool_config is not None before proceeding.

Suggested change

	if (self.profiler_controller.check_enable() and self.profiler_controller.check_this_rank() and
	self.profiler_controller.is_discrete_mode()):
	if (self.tool_config and self.profiler_controller.check_enable() and self.profiler_controller.check_this_rank() and
	self.profiler_controller.is_discrete_mode()):

[mengchengTang]

I believe the crash scenario won't happen because when profiler_config.tool is not "torch" or "npu" (e.g. "nsys"), we explicitly set profiler_config = None in the __init__ method.

As a result, DistProfiler is initialized with config=None, which defaults to enable=False. Therefore, self.profiler_controller.check_enable() will return False, and the code block accessing tool_config.contents will not be executed.

[gemini-code-assist]

There's a potential AttributeError here. If self.config.profiler is not set in the configuration, profiler_config will be None. When the else block is executed, accessing profiler_config.tool will raise an exception. You should guard this access with a check to ensure profiler_config is not None.

Suggested change

	else:
	logger.warning(f"agent loop only support torch and npu profiler, got {profiler_config.tool}")
	tool_config = None
	else:
	if profiler_config is not None:
	logger.warning(f"agent loop only support torch and npu profiler, got {profiler_config.tool}")
	tool_config = None

[mengchengTang]

Fixed. I have added a check if profiler_config is not None: to ensure we only access .tool when the config exists. If profiler_config is None, we skip the logic and pass None to DistProfiler, which handles it gracefully.

[mengchengTang]

Please add the test results for the different backends of GPU and NPU.

Verification Results

Scene	Discrete Mode	With Stack	Record Shapes	Steps	Ranks	Status
vLLM + NPU	True	False	False	[2]	[1]	✅ Pass
vLLM + NPU	True	True	True	[2]	All	✅ Pass
vLLM + GPU	True	False	False	[2]	[1]	✅ Pass
vLLM + GPU	True	True	True	[2]	All	✅ Pass
SGLang + NPU	True	False	False	[2]	[1]	✅ Pass
SGLang + NPU	True	True	True	[2]	All	✅ Pass
SGLang + GPU	True	False	False	[2]	[1]	✅ Pass
SGLang + GPU	True	True	True	[2]	All	✅ Pass

Profiling Interface Compatibility Report

1. SGLang Compatibility
   We leverage TokenizerCommunicatorMixin (inherited by TokenizerManager) for profiling control.
   Stability: The start_profile and stop_profile interfaces, along with parameters like output_dir and with_stack, have been consistent across versions.
   Conclusion: Integration via self.tokenizer_manager.start_profile() is fully compatible with existing SGLang versions.
2. vLLM Compatibility
   We use AsyncLLM's stable start_profile() / stop_profile() interfaces.
   Configuration: Currently relies on environment variables (e.g., VLLM_TORCH_PROFILER_DIR), which works across versions.
   Future Plan: After PR [BREAKING][worker, rollout, vllm] feat: implement vLLM colocated training-inference rollout with process separation #4280 (process separation for colocated rollout) merges, we will upgrade to inject profiler_config and environment variables before the inference process launches, providing native support for vLLM >= 0.13.0 while keeping legacy compatibility.

[wuxibin89]

Nice work!