Skip to content

[hardware, profile] fix: improve existing pytorch profiling to support other pytorch supported device - #8151

Merged
wuxibin89 merged 4 commits into
verl-project:mainfrom
kahlun:feature/torch-profiler-device-activity
Oct 9, 2026
Merged

wuxibin89 merged 4 commits into
verl-project:mainfrom
kahlun:feature/torch-profiler-device-activity

Conversation

@kahlun

@kahlun kahlun commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

get_torch_profiler knew exactly one device activity, ProfilerActivity.CUDA, and
TorchProfilerToolConfig accepted exactly one device keyword, "cuda". On an accelerator that is
not CUDA there was then no way to record device kernels: the default contents=[] asks for CUDA,
torch logs CUDA is not available, disabling CUDA profiling and the trace holds CPU ops only, while
naming the real device in contents fails config validation.

torch exposes one ProfilerActivity member per backend it can profile, named after the device type
it profiles, and a single profiler session records exactly one of them. So the active platform's
device_name is both the activity to collect and the contents keyword that requests it. This PR
resolves it that way instead of hardcoding CUDA, and takes the accepted device keywords from
torch.profiler.ProfilerActivity rather than from a verl-maintained list.

No new PlatformBase API — device_name is already an abstract property.

No change for existing platforms

PlatformCUDA.device_name is "cuda", and PlatformROCm, PlatformMetaX and PlatformIluvatar
inherit it (torch exposes HIP as torch.cuda), so all four build exactly the same activities list
as before. Platforms with no matching member (npu, mlu, musa, gcu, tpu, supa) still get a
CPU-only trace, exactly as they do today when torch disables the unavailable CUDA activity — the
warning explaining it is now verl's, and it names the device.

xpu, hpu and mtia start working with no further verl change and no plugin code.

Out-of-tree backends whose activity is not named after the device type

DEVICE_ACTIVITIES is a module-level device type -> ProfilerActivity map, so a backend that needs
a different pairing can add one entry from its plugin:

from verl.utils.profiler.config import DEVICE_ACTIVITIES
DEVICE_ACTIVITIES["mlu"] = torch.profiler.ProfilerActivity.PrivateUse1

This is deliberately data rather than API: it keeps the mapping out of PlatformBase while leaving
privateuse1-based backends a one-line override. In practice those backends all select a vendor
profiler through global_profiler.tool instead, which this PR does not touch.

CPU and PrivateUse1 are excluded from the map and "privateuse1" is not an accepted contents
keyword: CPU activity is collected in every trace regardless of contents, and PrivateUse1 is
torch's slot for an out-of-tree backend rather than a device type any platform reports as its
device_name, so accepting it would validate a value that is then silently never recorded.
"cpu" remains accepted.

Why this is not a duplicate

  • [hardware, profile] add optional torch.profiler activity hook to Plat… #7988 proposed the same fix through two optional PlatformBase hooks,
    torch_profiler_activity() and torch_profiler_content_name(). This PR drops both: device_name
    already carries the information, which removes the paired-hook invariant between them, removes the
    requirement for every plugin to implement them, and leaves platform_base.py untouched.
  • gh pr list --repo verl-project/verl --state open --search "profiler activity" — nothing else open
    in this area.

Test

pytest tests/utils/test_torch_profile.py
# 50 passed, 18 subtests passed

VERL_PLATFORM=intel     pytest tests/utils/test_torch_profile.py   # 50 passed, 18 subtests
VERL_PLATFORM=cambricon pytest tests/utils/test_torch_profile.py   # 50 passed, 18 subtests

ruff check verl/utils/profiler/config.py verl/utils/profiler/torch_profile.py \
           tests/utils/test_torch_profile.py                        # clean
ruff format --check <same three files>                              # clean

New tests:

  • TestDeviceActivityFromPlatform — a non-CUDA device is recorded and CUDA is not added alongside
    it (torch checks CUDA before every other device, so leaving it in would disable device profiling
    altogether); CUDA platforms are unchanged; every entry in DEVICE_ACTIVITIES resolves to its own
    activity, so the lookup holds for whatever members a given torch build exposes; a device with no
    member warns; a CUDA-written contents run on another accelerator warns and records no device
    activity; contents naming only profiler options warns nothing; a CPU platform does not add CPU
    twice.

    The non-CUDA cases take their device from whatever torch.profiler.ProfilerActivity the running
    build exposes rather than hardcoding one accelerator, and skip if a build offers none.

  • TestDeviceContentsConfigValidation — every keyword in DEVICE_CONTENTS validates, profiler
    options still validate, an unknown keyword and "privateuse1" are both rejected.

Two pre-existing tests (test_get_torch_profiler, test_cpu_activity_added_for_device_only_contents)
assert the CUDA activity, which until now was hardcoded and so held on any machine. They now patch in
a CUDA platform, stating the assumption they always had — without it they fail when the suite runs on
non-CUDA hardware.

No new e2e CI test: CI has no non-CUDA device. Manually verified end to end on Intel Arc Pro B60
(device kernels present in the chrome trace, which was not the case before this change).

API and Usage Example

No API change. On a non-CUDA accelerator, the default config now records device kernels:

global_profiler:
  tool: torch
  global_tool_config:
    torch:
      contents: []          # records the active platform's device activity, e.g. xpu
      # contents: [xpu]     # equivalent, explicit
      # contents: [stack]   # CPU-only, as requested, no warning

Design & Code Changes

  • verl/utils/profiler/config.py — DEVICE_ACTIVITIES map built from
    torch.profiler.ProfilerActivity.__members__ (keyed by lowercased device type, so the lookup does
    not depend on how torch capitalizes a member); DEVICE_CONTENTS derived from it for config
    validation, replacing the hardcoded ["cuda", "cpu", "memory", "shapes", "stack"].
  • verl/utils/profiler/torch_profile.py — get_torch_profiler resolves the active platform's
    device_name through that map; warns when the platform has no activity, and when contents names
    a device activity belonging to a different backend.
  • docs/perf/torch_profiling.md — documents the device keyword per accelerator.
  • tests/utils/test_torch_profile.py — as above.

Checklist

  • Searched for similar PRs (see "Why this is not a duplicate").
  • Title follows [{modules}] {type}: {description}.
  • pre-commit / ruff clean on changed files.
  • Docs updated (docs/perf/torch_profiling.md).
  • New CI-runnable unit tests added.
  • Slack ci-request ping if CI needs a nudge.
  • Recipe submodule unchanged (n/a).

kahlun added 4 commits October 8, 2026 11:31
…ctive platform

### What does this PR do?

`get_torch_profiler` knew exactly one device activity, `ProfilerActivity.CUDA`, and
`TorchProfilerToolConfig` accepted exactly one device keyword, `"cuda"`. On an accelerator
that is not CUDA there was then no way to record device kernels: the default `contents=[]`
asks for CUDA, torch logs "CUDA is not available, disabling CUDA profiling" and the trace
holds CPU ops only, while naming the real device in `contents` fails config validation.

torch names each `ProfilerActivity` member after the device type it profiles, and a single
profiler session records exactly one of them. So the active platform's `device_name` is
both the activity to collect and the `contents` keyword that requests it. Resolve it that
way instead of hardcoding CUDA, and take the accepted device keywords from
`torch.profiler.ProfilerActivity` rather than from a verl-maintained list.

No new `PlatformBase` API: `device_name` is already an abstract property.

### No change for existing platforms

`PlatformCUDA.device_name` is `"cuda"`, and `PlatformROCm`, `PlatformMetaX` and
`PlatformIluvatar` inherit it, so all four keep building the same `activities` list.
Platforms with no matching member (`npu`, `mlu`, `musa`, `gcu`, `tpu`, `supa`) still get a
CPU-only trace, exactly as they do today when torch disables the unavailable CUDA
activity -- the warning explaining it is now verl's, and it names the device.

### Tests

`test_get_torch_profiler` and `test_cpu_activity_added_for_device_only_contents` assert the
CUDA activity, which until now was hardcoded and so held on any machine. They now patch in
a CUDA platform, stating the assumption they always had -- without it they fail when the
suite is run on non-CUDA hardware. Verified by running the file with `VERL_PLATFORM` forced
to a non-CUDA platform.
`getattr(torch.profiler.ProfilerActivity, device.upper(), None)` depends on torch
capitalizing every member as the uppercased device type, which `PrivateUse1`
already breaks, and `DEVICE_CONTENTS` lowercased every member name -- so
`contents: ["privateuse1"]` passed config validation and was then never recorded,
since no platform reports "privateuse1" as its `device_name`.

Resolve the activity through an explicit lowercased device-type -> activity map
instead, and build the accepted `contents` keywords from it. `CPU` and
`PrivateUse1` are left out of the map: neither is a device type a platform
reports, CPU activity is collected in every trace regardless of `contents`, and
"cpu" stays an accepted keyword.

An out-of-tree backend whose torch activity is not named after its device type
can add its own entry to the map, rather than verl carrying a hook for it.

Tests: every entry in the map resolves to its own activity (so the lookup holds
for whatever members a torch build exposes), and "privateuse1" is rejected by
config validation.
@kahlun kahlun changed the title Feature/torch profiler device activity [hardware, profile] fix: improve existing pytorch profiling to support other pytorch supported device Oct 8, 2026
@kahlun
kahlun marked this pull request as ready for review October 8, 2026 12:45
@wuxibin89
wuxibin89 merged commit 14758bd into verl-project:main Oct 9, 2026
52 of 64 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants