[XPU] feat(xpu): add VTune (ITT) profiler backend

docs/accelerators/xpu/README.md

@@ -1,26 +1,39 @@
 # Intel XPU User Guide
 
-Last updated: 06/04/2026.
+Last updated: 10/09/2026.
 
 ## Introduction
 
-This document describes how to use verl for reinforcement learning training on Intel XPU (Data Center GPU Max / Arc).
+This document describes how to use verl for reinforcement learning training on Intel XPU
+(Arc Pro / Data Center GPU Max). Intel XPU support ships as an external plugin package
+(`verl_hardware_plugin`), not baked into the verl source tree.
 
 ## Directory Structure
 
+```text
+verl_hardware_plugin/accelerators/xpu/
+├── platform_xpu.py                  # Platform metadata + profiler hook
+├── registration.py                  # Engine registration on import
+├── engines/fsdp_xpu.py              # FSDP/FSDP2 actor + critic engines
+├── engines/megatron_xpu.py          # Megatron engine registration (Work in Progress — not yet validated end-to-end)
+├── patches/reduce_avg_allreduce_patch.py  # xccl ReduceOp.AVG workaround
+├── profilers/itt_profile_xpu.py     # ITT range markers + VtuneProfiler
+└── profilers/register_vtune.py      # claims `profiler.tool: vtune`
 ```
+
+```text
 docs/accelerators/xpu/
 ├── README.md              # This file
 ├── install_guidance.md    # Installation guide
 ├── quick_start.md         # Quick start
-└── faq.md                 # FAQ and troubleshooting
+└── profiling.md           # Intel VTune (ITT) profiling
 ```
 
 ## Getting Started
 
-- [Installation Guide](./install_guidance.md)
-- [Quick Start](./quick_start.md)
-- [FAQ](./faq.md)
+- [Installation Guide](./install_guidance.md) — prerequisites and environment setup
+- [Quick Start](./quick_start.md) — run a GRPO training example and verify the platform
+- [Profiling](./profiling.md) — capture an Intel VTune (ITT) trace
 
 ## Platform Summary
 
@@ -35,7 +48,13 @@ docs/accelerators/xpu/
 
 ## Environment Variables
 
-```bash
-export VERL_PLATFORM=intel
-source /opt/intel/oneapi/setvars.sh
-```
+| Variable | Purpose |
+|----------|---------|
+| `VERL_PLATFORM` | Set to `intel` to force platform selection instead of relying on auto-detection |
+| `VERL_USE_EXTERNAL_PLUGINS` | `auto` (default) discovers this plugin via entry_points; `none` disables discovery |
+| `ZE_AFFINITY_MASK` | Selects physical device indices visible to this process, e.g. `0,1` |
+| `ONEAPI_DEVICE_SELECTOR` | Must be left unset alongside `ZE_AFFINITY_MASK` — see [quick_start.md](./quick_start.md) |
+
+## Related Documentation
+
+- [verl plugin system](../../development.md)

docs/accelerators/xpu/install_guidance.md

@@ -0,0 +1,72 @@
+# Installation Guide
+
+Intel XPU support ships as this external plugin package
+(`verl_hardware_plugin`), not baked into the verl source tree — see
+[`verl/plugin/platform/README.md`](https://github.com/verl-project/verl/blob/main/verl/plugin/platform/README.md)
+in verl-core for the plugin pattern this follows.
+
+## Prerequisites
+
+- PyTorch with XPU support (`torch.xpu.is_available() == True`)
+- vLLM with XPU kernels — either the prebuilt wheel from vLLM's XPU wheel
+  index (`pip install "vllm==<version>+xpu" --extra-index-url
+  https://wheels.vllm.ai/<version>/xpu`) or built from source with
+  `VLLM_TARGET_DEVICE=xpu`. The default PyPI `pip install vllm` wheel does
+  not ship XPU kernels.
+- oneCCL runtime for the `xccl` distributed backend
+
+**verl-core version note:** the one `PlatformBase` hook this plugin
+implements, `profiler_markers()`, was merged into `verl-project/verl:main` on
+2026-10-08. A checkout from before that date still installs and runs this
+plugin, but the hook silently no-ops on it: ITT ranges are not emitted, and
+the `tool_config` allowlist in `engine_workers.py` that the same change
+lifted drops a `tool_config.vtune` entry. `VtuneProfiler` itself is
+registered by a plugin-side monkeypatch and works either way. Attention
+padding needs nothing from this plugin: verl core uses
+its own pure-PyTorch `attention_padding_utils` whenever `flash_attn` is
+unavailable. The `xccl` reduce_avg workaround is likewise unaffected — it's a
+plugin-side monkeypatch over `torch.distributed.all_reduce`
+(`accelerators/xpu/patches/reduce_avg_allreduce_patch.py`), not a
+`PlatformBase` hook.
+
+## 1. Install verl and verl-hardware-plugin
+
+```bash
+# Install verl (needs the profiler_markers() hook, on main since 2026-10-08 — see note above)
+git clone https://github.com/verl-project/verl.git
+cd verl
+pip install -e .
+
+# Install verl-hardware-plugin
+git clone https://github.com/verl-project/verl-hardware-plugin.git
+cd verl-hardware-plugin
+pip install -e .
+```
+
+No environment variable is required beyond this. The plugin is
+auto-discovered by verl through the `verl.plugins` setuptools entry_points
+group declared in this repo's `pyproject.toml`
+(`[project.entry-points."verl.plugins"] hardware = "verl_hardware_plugin"`),
+which verl loads by default (`VERL_USE_EXTERNAL_PLUGINS=auto`). Set
+`VERL_USE_EXTERNAL_PLUGINS=none` to disable discovery, e.g. to isolate a bug
+to this plugin, or `VERL_PLATFORM=intel` to force platform selection instead
+of relying on auto-detection.
+
+## 2. Verify the Install
+
+```bash
+python3 -c "
+from verl.plugin.platform import get_platform
+p = get_platform()
+print('device:', p.device_name, '/ vendor:', p.vendor_name)
+"
+```
+
+Expected on Intel GPU: `device: xpu / vendor: intel`. If it instead falls
+back to `nvidia`, the plugin was not discovered — check that `pip install`
+completed without error and that no `VERL_USE_EXTERNAL_PLUGINS=none` is set
+in the environment.
+
+## Next Steps
+
+Follow the [Quick Start](./quick_start.md).

docs/accelerators/xpu/profiling.md

@@ -0,0 +1,111 @@
+# Intel VTune Profiling Guide
+
+Last updated: 10/09/2026.
+
+This guide describes Intel VTune (ITT) profiling support in `verl-hardware-plugin`
+for Intel XPU. There is no PyTorch-native profiler activity for Intel XPU, so
+VTune support is wired in two ways instead: the tracing markers come from the
+`PlatformXPU.profiler_markers()` plugin hook, which verl core discovers without
+needing to know about ITT or XPU itself, while `VtuneProfiler` is registered for
+`profiler.tool=vtune` by a plugin-side monkeypatch applied from
+`PlatformXPU.__init__` (verl core has no profiler registry).
+
+## Prerequisites
+
+- `profiler_markers()` was merged into verl-core `main` on 2026-10-08. Against
+  an older checkout, `PlatformBase` has no such method to override, so ITT
+  ranges are not emitted — it falls back to nvtx or the generic no-op markers
+  instead of erroring. That same change also lifted the `tool_config`
+  allowlist in `engine_workers.py`; without it a `tool_config.vtune` entry is
+  dropped. The `VtuneProfiler` registration itself is pure monkeypatch and
+  needs no core change either way.
+- **A VTune collector must be installed separately.** ITT is notify-only: the
+  markers this plugin emits are handed to whatever collector has attached to the
+  process, and are discarded when none has. The plugin side needs nothing beyond
+  PyTorch's ITT bindings, which Intel's XPU wheels ship
+  (`torch.profiler.itt.is_available()` is `True` on `torch 2.13.0+xpu`); the
+  collector is the `intel-vtune` oneAPI component, which a plain oneAPI
+  compiler/runtime install (ccl, compiler, mkl, mpi, pti, …) does **not**
+  include on its own. Capturing a trace therefore needs a host with VTune
+  installed, and enough sampling permission for the collection type you pick
+  (`hotspots` reads `perf_event_open`, so a container typically needs
+  `kernel.perf_event_paranoid <= 2` or `CAP_PERFMON`).
+- Install `verl` (`main` includes `profiler_markers()` as of 2026-10-08) and
+  `verl-hardware-plugin` in editable mode, and enable the plugin in Ray
+  runtime:
+
+  ```yaml
+  working_dir: ./
+  excludes: ["/.git/"]
+  env_vars:
+    VERL_USE_EXTERNAL_MODULES: "verl_hardware_plugin"
+  ```
+
+## Enabling VTune
+
+Selecting the tool is all that is required:
+
+```bash
+python -m verl.trainer.main_ppo \
+  ... \
+  actor_rollout_ref.actor.profiler.tool=vtune \
+  actor_rollout_ref.actor.profiler.enable=True \
+  actor_rollout_ref.actor.profiler.all_ranks=False \
+  actor_rollout_ref.actor.profiler.ranks=[0]
+```
+
+There is no dedicated `tool_config.vtune` schema entry in verl-core's generated
+config (only `nsys`/`npu`/`torch`/`torch_memory`/`precision_debugger` have one),
+and `vtune` does not need one: the only field `VtuneProfiler` reads is
+`tool_config.discrete`, which has no effect on XPU because
+`PlatformXPU.profiler_start`/`profiler_stop` are no-ops. It therefore defaults to
+`False` and no `+tool_config.vtune.*` override is needed.
+
+If you want the config to carry an explicit entry anyway, add it with Hydra's `+`
+— `VtuneProfiler` reuses `NsightToolConfig`'s shape:
+
+```bash
+  +actor_rollout_ref.actor.profiler.tool_config.vtune._target_=verl.utils.profiler.config.NsightToolConfig \
+  +actor_rollout_ref.actor.profiler.tool_config.vtune.discrete=False
+```
+
+This changes nothing functionally on XPU.
+
+## How this differs from Nsight/torch/NPU profiling
+
+Nsight, torch, and NPU profiling are process-level: `profiler.start()`/`stop()`
+tell the backend when to begin and end recording, and each writes a self-contained
+trace file under `global_profiler.save_path`. **VTune does not work this way.**
+`PlatformXPU.profiler_start()`/`profiler_stop()` are no-ops by design — VTune
+attaches externally as a collector and observes `range_push`/`range_pop` events
+rather than being started/stopped by the profiled process itself. The real
+signal is the ITT range markers (`mark_start_range`/`mark_end_range`, wrapping
+each named stage — `compute_log_prob`, `update_actor`, etc. — via
+`profiler_markers()`), which are only visible when the training process runs
+*under* an actual VTune collector, e.g.:
+
+```bash
+vtune -collect hotspots -result-dir ./vtune_results -- \
+  python -m verl.trainer.main_ppo ... actor_rollout_ref.actor.profiler.tool=vtune ...
+```
+
+No trace file appears under `global_profiler.save_path` for the `vtune` tool —
+open `./vtune_results` in the VTune GUI/CLI instead. Because `profiler_start`/
+`profiler_stop` are no-ops, `tool_config.discrete` has little practical effect
+for `vtune` specifically (unlike `torch`/`nsys`, where it materially changes
+what gets recorded) — the ITT ranges are emitted the same way either way.
+
+## Troubleshooting
+
+- If `profiler.tool=vtune` appears to do nothing, confirm the running verl-core
+  build actually includes `profiler_markers()` (merged 2026-10-08; see
+  Prerequisites) — on an older checkout this is a silent no-op, not an error.
+- If no ITT ranges show up in VTune, confirm the process was launched *under* a
+  VTune collector (`vtune -collect ... -- python ...`), not run standalone —
+  standalone runs execute the same code but nothing is listening for the events.
+- If `vtune: command not found`, the collector isn't installed (see
+  Prerequisites). Check with `ls /opt/intel/oneapi` for a `vtune` entry; a
+  standalone run is still harmless, it just records nothing. To tell a missing
+  collector apart from broken ITT bindings, check
+  `python -c "import torch.profiler.itt as i; print(i.is_available())"` — `True`
+  means the plugin's markers are live and only the collector is absent.

docs/accelerators/xpu/quick_start.md

@@ -0,0 +1,118 @@
+# Quick Start
+
+Training example for running verl on Intel GPU (Arc / Arc Pro) with this
+plugin installed (see [`install_guidance.md`](install_guidance.md) first).
+
+Current software and hardware scope:
+
+- Runtime mode: **Colocate** (FSDP actor + vLLM rollout on the same device).
+- Inference engine: **vLLM** validated. SGLang weight sync — Work in
+  Progress on Intel GPU.
+- Trainer backend: **FSDP**, **FSDP2**. Megatron — Work in Progress.
+- Algorithms: GRPO, PPO, SFT — validated on GSM8K / Qwen2.5.
+- Hardware targets:
+  - Intel Arc Pro B60 (Battlemage, 24 GB) — validated, 1-GPU and 2-GPU
+  - Intel Arc Pro B70 (2x GPU, 32 GB each)
+  - Multi-node — Work in Progress
+
+## 1. Verify Platform Resolution
+
+```bash
+python3 -c '
+from verl.plugin.platform import get_platform
+p = get_platform()
+print("device:  ", p.device_name)
+print("vendor:  ", p.vendor_name)
+print("backend: ", p.communication_backend_name())
+print("ray res: ", p.ray_resource_name())
+print("ipc:     ", p.is_ipc_supported())
+'
+```
+
+Expected output:
+
+```text
+device:   xpu
+vendor:   intel
+backend:  xccl
+ray res:  GPU
+ipc:      False
+```
+
+## 2. Prepare Data
+
+```bash
+python3 examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k
+```
+
+## 3. Launch Training
+
+```bash
+export ZE_AFFINITY_MASK=0,1            # select physical device indices
+unset ONEAPI_DEVICE_SELECTOR           # must NOT be set
+
+python3 -m verl.trainer.main_ppo \
+    algorithm.adv_estimator=grpo \
+    data.train_files=$HOME/data/gsm8k/train.parquet \
+    data.val_files=$HOME/data/gsm8k/test.parquet \
+    data.train_batch_size=16 \
+    data.max_prompt_length=512 \
+    data.max_response_length=128 \
+    actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
+    actor_rollout_ref.actor.optim.lr=5e-7 \
+    actor_rollout_ref.model.use_remove_padding=False \
+    +actor_rollout_ref.model.override_config.attn_implementation=eager \
+    actor_rollout_ref.actor.use_kl_loss=True \
+    actor_rollout_ref.actor.kl_loss_coef=0.001 \
+    actor_rollout_ref.actor.fsdp_config.param_offload=True \
+    actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
+    actor_rollout_ref.rollout.name=vllm \
+    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
+    actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
+    actor_rollout_ref.rollout.enforce_eager=True \
+    trainer.n_gpus_per_node=1 \
+    trainer.nnodes=1 \
+    trainer.total_epochs=1 \
+    +ray_kwargs.ray_init.num_gpus=1
+```
+
+Expected (Qwen2.5-0.5B-Instruct, GSM8K, batch 16, 1 step, 1x Arc Pro B60):
+
+```text
+timing_s/step: ~51   timing_per_token_ms/gen: ~0.35
+perf/throughput: ~148 tok/s
+```
+
+On 2 GPUs with the same batch size (throughput scales with larger batch —
+this keeps the comparison apples-to-apples rather than showing peak
+throughput):
+
+```text
+timing_s/step: ~93   perf/throughput: ~41 tok/s
+```
+
+## 4. Run the Plugin Test Suite
+
+```bash
+pytest tests/test_plugin_registration.py -k xpu -v
+```
+
+Expected: all XPU cases pass. These run on any host, with or without an
+Intel GPU attached.
+
+## Feature Support Matrix
+
+| Category | Status | Notes |
+|---|---|---|
+| Runtime mode | Colocate | FSDP/FSDP2 actor + vLLM rollout on same GPU(s) |
+| Inference engine | vLLM validated | SGLang: `update_weights` — Work in Progress |
+| Trainer backend | FSDP, FSDP2 | Megatron — Work in Progress |
+| Algorithms | GRPO, PPO, SFT | Validated on GSM8K / Qwen2.5 |
+| Hardware (1-GPU) | Validated | Arc Pro B60: 51.2 s/step, 148.2 tok/s (batch 16, Qwen2.5-0.5B) |
+| Hardware (2-GPU) | Validated | Arc Pro B60 x2: 93.0 s/step at same batch size |
+| Multi-node | Work in Progress | — |
+| Profiler | Intel VTune (ITT) — Validated | `profiler_markers()` hook merged into verl-core `main` on 2026-10-08; VTune collector capture itself still untested, see [profiling.md](./profiling.md) |
+
+## Next Steps
+
+See [Profiling](./profiling.md) to capture an Intel VTune (ITT) trace.