From 6ea972e9b0336b41d9836a180bdad255205754c5 Mon Sep 17 00:00:00 2001
From: kahlun <kah.lun.teoh@intel.com>
Date: Thu, 15 Oct 2026 11:50:01 +0000
Subject: [PATCH 1/5] [XPU] docs: validate install and quick start against verl
 main; add Docker / from-source options

Checked on 1x Arc Pro B60 (verl 5b550b0c, this plugin e5ab532, vLLM 0.29.0+xpu), in an image
and in a fresh venv, with system oneAPI present and hidden.

- install_guidance: system oneAPI is not required (torch/vLLM wheels bring their own SYCL runtime
  and oneCCL); document the Level Zero driver prerequisite; add Docker (docker/xpu in verl-core) and
  from-source options; warn that `pip install -e .` on verl can replace the XPU torch; VERL_PLATFORM
  is an override, auto-detection already selects intel once the plugin is installed.
- quick_start: the GRPO command failed on verl main (needs ppo_micro_batch_size_per_gpu and the
  log_prob micro batch sizes) and ran ~467 steps instead of the described single step; add the
  batch-size overrides and total_training_steps=1, align ZE_AFFINITY_MASK with the 1-GPU command,
  and replace the stale throughput numbers with measured ones (2-GPU number flagged as not re-measured).
- README: list patches/ray_device_pin_patch.py, link the Docker image.
- profiling: the plugin is auto-discovered in Ray workers; VERL_USE_EXTERNAL_MODULES is not needed.
---
 docs/accelerators/xpu/README.md           | 11 +++-
 docs/accelerators/xpu/install_guidance.md | 64 ++++++++++++++---------
 docs/accelerators/xpu/profiling.md        | 15 ++----
 docs/accelerators/xpu/quick_start.md      | 34 +++++++-----
 4 files changed, 75 insertions(+), 49 deletions(-)

diff --git a/docs/accelerators/xpu/README.md b/docs/accelerators/xpu/README.md
index 638b804c..c506626e 100644
--- a/docs/accelerators/xpu/README.md
+++ b/docs/accelerators/xpu/README.md
@@ -1,6 +1,6 @@
 # Intel XPU User Guide
 
-Last updated: 10/09/2026.
+Last updated: 10/15/2026.
 
 ## Introduction
 
@@ -17,6 +17,7 @@ verl_hardware_plugin/accelerators/xpu/
 ├── engines/fsdp_xpu.py              # FSDP/FSDP2 actor + critic engines
 ├── engines/megatron_xpu.py          # Megatron engine registration (Work in Progress — not yet validated end-to-end)
 ├── patches/reduce_avg_allreduce_patch.py  # xccl ReduceOp.AVG workaround
+├── patches/ray_device_pin_patch.py        # pins each Ray worker to its assigned device (otherwise every rank lands on xpu:0)
 ├── profilers/itt_profile_xpu.py     # ITT range markers + VtuneProfiler
 └── profilers/register_vtune.py      # claims `profiler.tool: vtune`
 ```
@@ -50,11 +51,17 @@ docs/accelerators/xpu/
 
 | Variable | Purpose |
 |----------|---------|
-| `VERL_PLATFORM` | Set to `intel` to force platform selection instead of relying on auto-detection |
+| `VERL_PLATFORM` | Optional. Auto-detection already selects `intel` once the plugin is installed and `torch.xpu` is available; set `intel` only to force the choice |
 | `VERL_USE_EXTERNAL_PLUGINS` | `auto` (default) discovers this plugin via entry_points; `none` disables discovery |
 | `ZE_AFFINITY_MASK` | Selects physical device indices visible to this process, e.g. `0,1` |
 | `ONEAPI_DEVICE_SELECTOR` | Must be left unset alongside `ZE_AFFINITY_MASK` — see [quick_start.md](./quick_start.md) |
 
+## Docker
+
+A ready-made Intel GPU image (verl + this plugin + vLLM) is maintained with verl-core in
+[`docker/xpu/`](https://github.com/verl-project/verl/tree/main/docker/xpu). A system oneAPI install is not
+needed at runtime; see the [installation guide](./install_guidance.md).
+
 ## Related Documentation
 
 - [verl plugin system](../../development.md)
diff --git a/docs/accelerators/xpu/install_guidance.md b/docs/accelerators/xpu/install_guidance.md
index c5ffd3c7..911f0447 100644
--- a/docs/accelerators/xpu/install_guidance.md
+++ b/docs/accelerators/xpu/install_guidance.md
@@ -7,13 +7,14 @@ in verl-core for the plugin pattern this follows.
 
 ## Prerequisites
 
-- PyTorch with XPU support (`torch.xpu.is_available() == True`)
-- vLLM with XPU kernels — either the prebuilt wheel from vLLM's XPU wheel
-  index (`pip install "vllm==<version>+xpu" --extra-index-url
-  https://wheels.vllm.ai/<version>/xpu`) or built from source with
-  `VLLM_TARGET_DEVICE=xpu`. The default PyPI `pip install vllm` wheel does
-  not ship XPU kernels.
-- oneCCL runtime for the `xccl` distributed backend
+- An Intel GPU with the Level Zero driver stack on the host (compute-runtime >= 26.18 and Level Zero).
+  **A system oneAPI install is not required**: PyTorch's XPU wheel and vLLM's XPU wheel bring their own SYCL
+  runtime and oneCCL (the `xccl` backend). This was checked by hiding `/opt/intel/oneapi` and the oneAPI
+  environment variables: platform detection, an `xccl` all_reduce and a GRPO training step all still worked.
+- PyTorch with XPU support (`torch.xpu.is_available() == True`).
+- vLLM with XPU kernels, from vLLM's XPU wheel index (`pip install "vllm==<version>+xpu"
+  --extra-index-url https://wheels.vllm.ai/<version>/xpu`) or built from source with
+  `VLLM_TARGET_DEVICE=xpu`. The default PyPI `pip install vllm` wheel does not ship XPU kernels.
 
 **verl-core version note:** the one `PlatformBase` hook this plugin
 implements, `profiler_markers()`, was merged into `verl-project/verl:main` on
@@ -31,26 +32,40 @@ plugin-side monkeypatch over `torch.distributed.all_reduce`
 
 ## 1. Install verl and verl-hardware-plugin
 
-```bash
-# Install verl (needs the profiler_markers() hook, on main since 2026-10-08 — see note above)
-git clone https://github.com/verl-project/verl.git
-cd verl
-pip install -e .
+Pick one. Both give the same environment (checked: torch 2.13.0+xpu, vLLM 0.29.0+xpu, transformers 5.17.0).
+
+### Option A: Docker (recommended)
+
+The Intel GPU image is maintained with verl-core, in
+[`docker/xpu/`](https://github.com/verl-project/verl/tree/main/docker/xpu). It installs verl and this plugin
+together, so there is nothing else to do. Build and run steps are in that directory's README.
+
+### Option B: From source
 
-# Install verl-hardware-plugin
+verl pins its own versions, and `pip install -e .` on a verl checkout can replace the XPU build of torch with a
+different one (it also downgrades `transformers` and `tensordict` to verl's pins). Install the XPU dependency
+set the way verl-core documents it, in
+[`docker/xpu/README.md` ("Installing Without Docker")](https://github.com/verl-project/verl/blob/main/docker/xpu/README.md),
+which uses verl's `requirements-xpu.txt`. Then install this plugin on top of that environment:
+
+```bash
 git clone https://github.com/verl-project/verl-hardware-plugin.git
 cd verl-hardware-plugin
-pip install -e .
+pip install --no-deps -e .
 ```
 
-No environment variable is required beyond this. The plugin is
-auto-discovered by verl through the `verl.plugins` setuptools entry_points
+`--no-deps` keeps the plugin install from touching the torch/verl already in place. The plugin needs no
+environment variable: it is auto-discovered by verl through the `verl.plugins` setuptools entry_points
 group declared in this repo's `pyproject.toml`
-(`[project.entry-points."verl.plugins"] hardware = "verl_hardware_plugin"`),
-which verl loads by default (`VERL_USE_EXTERNAL_PLUGINS=auto`). Set
-`VERL_USE_EXTERNAL_PLUGINS=none` to disable discovery, e.g. to isolate a bug
-to this plugin, or `VERL_PLATFORM=intel` to force platform selection instead
-of relying on auto-detection.
+(`[project.entry-points."verl.plugins"] hardware = "verl_hardware_plugin"`), which verl loads by default
+(`VERL_USE_EXTERNAL_PLUGINS=auto`).
+
+### Platform selection
+
+With the plugin installed, verl picks the Intel platform by itself on a host where `torch.xpu` is available.
+`VERL_PLATFORM=intel` is only an override, for forcing the choice (for example on a host with several kinds of
+accelerator). `VERL_USE_EXTERNAL_PLUGINS=none` disables discovery, e.g. to isolate a bug to this plugin; verl then
+falls back to `nvidia`.
 
 ## 2. Verify the Install
 
@@ -63,9 +78,10 @@ print('device:', p.device_name, '/ vendor:', p.vendor_name)
 ```
 
 Expected on Intel GPU: `device: xpu / vendor: intel`. If it instead falls
-back to `nvidia`, the plugin was not discovered — check that `pip install`
-completed without error and that no `VERL_USE_EXTERNAL_PLUGINS=none` is set
-in the environment.
+back to `nvidia`, either the plugin was not discovered (check that `pip install`
+completed without error and that no `VERL_USE_EXTERNAL_PLUGINS=none` is set) or
+torch lost its XPU build (check `python3 -c "import torch; print(torch.__version__, torch.xpu.is_available())"`;
+a plain version like `2.14.1` without `+xpu` means a dependency install replaced it).
 
 ## Next Steps
 
diff --git a/docs/accelerators/xpu/profiling.md b/docs/accelerators/xpu/profiling.md
index aa88d2fe..e5d22bce 100644
--- a/docs/accelerators/xpu/profiling.md
+++ b/docs/accelerators/xpu/profiling.md
@@ -1,6 +1,6 @@
 # Intel VTune Profiling Guide
 
-Last updated: 10/09/2026.
+Last updated: 10/15/2026.
 
 This guide describes Intel VTune (ITT) profiling support in `verl-hardware-plugin`
 for Intel XPU. There is no PyTorch-native profiler activity for Intel XPU, so
@@ -31,15 +31,10 @@ needing to know about ITT or XPU itself, while `VtuneProfiler` is registered for
   (`hotspots` reads `perf_event_open`, so a container typically needs
   `kernel.perf_event_paranoid <= 2` or `CAP_PERFMON`).
 - Install `verl` (`main` includes `profiler_markers()` as of 2026-10-08) and
-  `verl-hardware-plugin` in editable mode, and enable the plugin in Ray
-  runtime:
-
-  ```yaml
-  working_dir: ./
-  excludes: ["/.git/"]
-  env_vars:
-    VERL_USE_EXTERNAL_MODULES: "verl_hardware_plugin"
-  ```
+  `verl-hardware-plugin` (see [install_guidance.md](./install_guidance.md)). The plugin is auto-discovered in
+  every process, including Ray workers, through its `verl.plugins` entry point, so no Ray `runtime_env` setting is
+  needed. (`VERL_USE_EXTERNAL_MODULES: "verl_hardware_plugin"` is an equivalent explicit way to load it, and is
+  only useful if entry-point discovery is disabled.)
 
 ## Enabling VTune
 
diff --git a/docs/accelerators/xpu/quick_start.md b/docs/accelerators/xpu/quick_start.md
index f660cdfa..4181bcc7 100644
--- a/docs/accelerators/xpu/quick_start.md
+++ b/docs/accelerators/xpu/quick_start.md
@@ -48,7 +48,7 @@ python3 examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k
 ## 3. Launch Training
 
 ```bash
-export ZE_AFFINITY_MASK=0,1            # select physical device indices
+export ZE_AFFINITY_MASK=0              # select physical device indices (0,1 for the 2-GPU run below)
 unset ONEAPI_DEVICE_SELECTOR           # must NOT be set
 
 python3 -m verl.trainer.main_ppo \
@@ -72,25 +72,33 @@ python3 -m verl.trainer.main_ppo \
     actor_rollout_ref.rollout.enforce_eager=True \
     trainer.n_gpus_per_node=1 \
     trainer.nnodes=1 \
+    actor_rollout_ref.actor.ppo_mini_batch_size=16 \
+    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
+    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=4 \
+    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
     trainer.total_epochs=1 \
+    trainer.total_training_steps=1 \
+    trainer.test_freq=-1 \
+    trainer.save_freq=-1 \
     +ray_kwargs.ray_init.num_gpus=1
 ```
 
-Expected (Qwen2.5-0.5B-Instruct, GSM8K, batch 16, 1 step, 1x Arc Pro B60):
+This is a smoke test: `total_training_steps=1` stops after one step (without it, `total_epochs=1` over GSM8K at
+batch 16 is about 467 steps). Raise it for a longer run.
 
-```text
-timing_s/step: ~51   timing_per_token_ms/gen: ~0.35
-perf/throughput: ~148 tok/s
-```
-
-On 2 GPUs with the same batch size (throughput scales with larger batch —
-this keeps the comparison apples-to-apples rather than showing peak
-throughput):
+Expected (Qwen2.5-0.5B-Instruct, GSM8K, batch 16, 1x Arc Pro B60). The first step includes warm-up; a second
+step (`total_training_steps=2`) shows steady state:
 
 ```text
-timing_s/step: ~93   perf/throughput: ~41 tok/s
+step 1:  timing_s/step: ~18-21   perf/throughput: ~190-210 tok/s
+step 2:  timing_s/step: ~8        perf/throughput: ~430-460 tok/s
+timing_per_token_ms/gen: ~1.0    perf/max_memory_allocated_gb: ~9.2
 ```
 
+(Measured 2026-10-15 with verl `5b550b0c`, this plugin at `e5ab532`, vLLM 0.29.0+xpu, torch 2.13.0+xpu.
+Timings vary with the stack and the host.) The 2-GPU figure that used to be listed here (~93 s/step at the same
+batch size) has not been re-measured on this stack.
+
 ## 4. Run the Plugin Test Suite
 
 ```bash
@@ -108,8 +116,8 @@ Intel GPU attached.
 | Inference engine | vLLM validated | SGLang: `update_weights` — Work in Progress |
 | Trainer backend | FSDP, FSDP2 | Megatron — Work in Progress |
 | Algorithms | GRPO, PPO, SFT | Validated on GSM8K / Qwen2.5 |
-| Hardware (1-GPU) | Validated | Arc Pro B60: 51.2 s/step, 148.2 tok/s (batch 16, Qwen2.5-0.5B) |
-| Hardware (2-GPU) | Validated | Arc Pro B60 x2: 93.0 s/step at same batch size |
+| Hardware (1-GPU) | Validated | Arc Pro B60: ~8 s/step steady state, ~450 tok/s (batch 16, Qwen2.5-0.5B, vLLM 0.29.0) |
+| Hardware (2-GPU) | Validated | Arc Pro B60 x2 (not re-measured on the current stack) |
 | Multi-node | Work in Progress | — |
 | Profiler | Intel VTune (ITT) — Validated | `profiler_markers()` hook merged into verl-core `main` on 2026-10-08; VTune collector capture itself still untested, see [profiling.md](./profiling.md) |
 

From d80a44b74ad65beea3274871ccf269df68527048 Mon Sep 17 00:00:00 2001
From: kahlun <kah.lun.teoh@intel.com>
Date: Fri, 23 Oct 2026 05:02:07 +0000
Subject: [PATCH 2/5] [XPU] docs: make the install options self-contained;
 re-verify the quick start

- install_guidance.md: Option A shows the docker build/run commands; Option B lists the
  exact dependency install (the one the Docker image runs) instead of linking to verl-core
  files that do not exist yet, plus the host packages it needs (g++, libnuma-dev, libze-dev,
  ffmpeg). Mention pip >= 26.2.2, uv >= 0.13.4 and certifi >= 2026.10.19.
- quick_start.md: run two steps so the "step 1 / step 2" figures can be reproduced; replace
  the benchmark numbers with ones re-measured on the current stack (1 and 2 GPUs, batch 16);
  fix the "2-GPU run below" reference; note the multi-node status.
- README.md: drop the link to a verl-core directory that is not on main.
---
 docs/accelerators/xpu/README.md           |  5 +-
 docs/accelerators/xpu/install_guidance.md | 60 ++++++++++++++++++-----
 docs/accelerators/xpu/quick_start.md      | 37 +++++++++-----
 3 files changed, 75 insertions(+), 27 deletions(-)

diff --git a/docs/accelerators/xpu/README.md b/docs/accelerators/xpu/README.md
index c506626e..303150c7 100644
--- a/docs/accelerators/xpu/README.md
+++ b/docs/accelerators/xpu/README.md
@@ -58,9 +58,8 @@ docs/accelerators/xpu/
 
 ## Docker
 
-A ready-made Intel GPU image (verl + this plugin + vLLM) is maintained with verl-core in
-[`docker/xpu/`](https://github.com/verl-project/verl/tree/main/docker/xpu). A system oneAPI install is not
-needed at runtime; see the [installation guide](./install_guidance.md).
+An Intel GPU image (verl + this plugin + vLLM) is built from `docker/xpu/` in verl-core. A system oneAPI install is
+not needed at runtime. Build and run commands are in the [installation guide](./install_guidance.md).
 
 ## Related Documentation
 
diff --git a/docs/accelerators/xpu/install_guidance.md b/docs/accelerators/xpu/install_guidance.md
index 911f0447..967ddb78 100644
--- a/docs/accelerators/xpu/install_guidance.md
+++ b/docs/accelerators/xpu/install_guidance.md
@@ -32,29 +32,67 @@ plugin-side monkeypatch over `torch.distributed.all_reduce`
 
 ## 1. Install verl and verl-hardware-plugin
 
-Pick one. Both give the same environment (checked: torch 2.13.0+xpu, vLLM 0.29.0+xpu, transformers 5.17.0).
+Pick one. Both install the same set of packages (torch 2.13.0+xpu, vLLM 0.29.0+xpu, transformers 5.17.0): the Docker
+image runs the commands of Option B.
 
 ### Option A: Docker (recommended)
 
-The Intel GPU image is maintained with verl-core, in
-[`docker/xpu/`](https://github.com/verl-project/verl/tree/main/docker/xpu). It installs verl and this plugin
-together, so there is nothing else to do. Build and run steps are in that directory's README.
+The image is built from `docker/xpu/` in verl-core. It installs verl and this plugin together, so nothing else is
+needed. Pin the two checkouts with `--build-arg VERL_REF=<commit> --build-arg PLUGIN_REF=<commit>`; for release builds
+add `--no-cache` so the system certificates, `pip`, `certifi` and `uv` are fetched fresh.
+
+```bash
+git clone https://github.com/verl-project/verl.git && cd verl
+docker build -f docker/xpu/Dockerfile.stable.vllm -t verl:xpu-vllm029 .
+docker run --rm -it --device /dev/dri --ipc=host --privileged verl:xpu-vllm029
+```
 
 ### Option B: From source
 
 verl pins its own versions, and `pip install -e .` on a verl checkout can replace the XPU build of torch with a
-different one (it also downgrades `transformers` and `tensordict` to verl's pins). Install the XPU dependency
-set the way verl-core documents it, in
-[`docker/xpu/README.md` ("Installing Without Docker")](https://github.com/verl-project/verl/blob/main/docker/xpu/README.md),
-which uses verl's `requirements-xpu.txt`. Then install this plugin on top of that environment:
+different one (it also moves `transformers` and `tensordict` to verl's pins). Install the dependencies as below and
+add verl and the plugin with `--no-deps`.
+
+System packages on the host: `g++`/`gcc` and the Python 3.12 headers (Triton compiles its launcher stubs at first
+use), `libnuma-dev`, `libze-dev` (it provides the unversioned `libze_loader.so` that oneCCL opens; without it `xccl`
+fails with "ze_data was not initialized"), and `ffmpeg` (the FFmpeg shared libraries that `torchcodec` needs to decode
+video).
 
 ```bash
-git clone https://github.com/verl-project/verl-hardware-plugin.git
-cd verl-hardware-plugin
+# Use pip >= 26.2.2 and uv >= 0.13.4 (both carry the corrected CA bundle).
+uv venv --python 3.12 --seed /opt/venv && source /opt/venv/bin/activate
+export UV_INDEX_URL=https://download.pytorch.org/whl/xpu UV_EXTRA_INDEX_URL=https://pypi.org/simple \
+       UV_INDEX_STRATEGY=unsafe-best-match VLLM_TARGET_DEVICE=xpu
+
+# vLLM XPU wheel (the PyPI wheel has no XPU kernels). --extra-index-url on the command line replaces
+# UV_EXTRA_INDEX_URL, so PyPI is repeated.
+uv pip install "vllm==0.29.0+xpu" \
+    --extra-index-url https://wheels.vllm.ai/0.29.0/xpu --extra-index-url https://wheels.vllm.ai/xpu \
+    --extra-index-url https://pypi.org/simple
+
+git clone https://github.com/verl-project/verl.git && cd verl
+grep -v '^transformers' requirements.txt > /tmp/requirements-core.txt
+echo 'pyzes>=0.1.2' > /tmp/override-pyzes.txt   # torch pins pyzes==0.1.1; the device-memory query needs >= 0.1.2
+
+# --no-config: run inside a verl checkout, uv would otherwise read verl's pyproject [tool.uv] overrides
+# (transformers 5.12.1, kernels<0.15 for the CUDA stack) and they would beat the XPU pins below.
+uv --no-config pip install -r /tmp/requirements-core.txt \
+    "torch==2.13.0+xpu" "transformers==5.17.0" "kernels>=0.16.0,<0.17.0" \
+    dpctl "pyzes>=0.1.2" triton-xpu "pip>=26.2.2" "certifi>=2026.10.19" qwen_vl_utils mathruler \
+    --override /tmp/override-pyzes.txt
+uv --no-config pip install --no-deps -e .
+
+# liger-kernel pulls the CUDA triton, which shares triton/ with triton-xpu: keep triton-xpu only.
+TRITON_XPU="$(uv pip freeze | grep -i '^triton-xpu==')"
+uv pip uninstall triton
+uv pip install --reinstall --no-deps "${TRITON_XPU}" --extra-index-url https://download.pytorch.org/whl/xpu
+
+# This plugin, on top of that environment:
+cd .. && git clone https://github.com/verl-project/verl-hardware-plugin.git && cd verl-hardware-plugin
 pip install --no-deps -e .
 ```
 
-`--no-deps` keeps the plugin install from touching the torch/verl already in place. The plugin needs no
+`--no-deps` keeps the plugin install from touching the torch and verl already in place. The plugin needs no
 environment variable: it is auto-discovered by verl through the `verl.plugins` setuptools entry_points
 group declared in this repo's `pyproject.toml`
 (`[project.entry-points."verl.plugins"] hardware = "verl_hardware_plugin"`), which verl loads by default
diff --git a/docs/accelerators/xpu/quick_start.md b/docs/accelerators/xpu/quick_start.md
index 4181bcc7..95d7b915 100644
--- a/docs/accelerators/xpu/quick_start.md
+++ b/docs/accelerators/xpu/quick_start.md
@@ -48,7 +48,8 @@ python3 examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k
 ## 3. Launch Training
 
 ```bash
-export ZE_AFFINITY_MASK=0              # select physical device indices (0,1 for the 2-GPU run below)
+export ZE_AFFINITY_MASK=0              # select physical device indices; for two GPUs use 0,1 together with
+                                       # trainer.n_gpus_per_node=2 and +ray_kwargs.ray_init.num_gpus=2
 unset ONEAPI_DEVICE_SELECTOR           # must NOT be set
 
 python3 -m verl.trainer.main_ppo \
@@ -77,27 +78,37 @@ python3 -m verl.trainer.main_ppo \
     actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=4 \
     actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
     trainer.total_epochs=1 \
-    trainer.total_training_steps=1 \
+    trainer.total_training_steps=2 \
     trainer.test_freq=-1 \
     trainer.save_freq=-1 \
     +ray_kwargs.ray_init.num_gpus=1
 ```
 
-This is a smoke test: `total_training_steps=1` stops after one step (without it, `total_epochs=1` over GSM8K at
-batch 16 is about 467 steps). Raise it for a longer run.
+This is a smoke test: `total_training_steps=2` stops after two steps (without it, `total_epochs=1` over GSM8K at
+batch 16 is about 467 steps). Raise it for a longer run. A `DataLoader worker ... is killed by signal` traceback
+printed at exit is shutdown noise after training has finished.
 
-Expected (Qwen2.5-0.5B-Instruct, GSM8K, batch 16, 1x Arc Pro B60). The first step includes warm-up; a second
-step (`total_training_steps=2`) shows steady state:
+Expected (Qwen2.5-0.5B-Instruct, GSM8K, batch 16, 1x Arc Pro B60). The first step includes warm-up; the second shows
+steady state:
 
 ```text
-step 1:  timing_s/step: ~18-21   perf/throughput: ~190-210 tok/s
-step 2:  timing_s/step: ~8        perf/throughput: ~430-460 tok/s
+step 1:  timing_s/step: ~16-21   perf/throughput: ~190-235 tok/s
+step 2:  timing_s/step: ~8-8.5   perf/throughput: ~440-465 tok/s
 timing_per_token_ms/gen: ~1.0    perf/max_memory_allocated_gb: ~9.2
 ```
 
-(Measured 2026-10-15 with verl `5b550b0c`, this plugin at `e5ab532`, vLLM 0.29.0+xpu, torch 2.13.0+xpu.
-Timings vary with the stack and the host.) The 2-GPU figure that used to be listed here (~93 s/step at the same
-batch size) has not been re-measured on this stack.
+(Three runs on 2026-10-21 to 2026-10-23: vLLM 0.29.0+xpu, torch 2.13.0+xpu, this plugin at `e5ab532`, verl `main` at
+different commits. Timings vary with the stack and the host.)
+
+Two GPUs: set `ZE_AFFINITY_MASK=0,1`, `trainer.n_gpus_per_node=2` and `+ray_kwargs.ray_init.num_gpus=2`, and keep
+everything else the same. At this small batch the step time does not drop, because fixed per-step overhead dominates.
+`perf/throughput` is reported per GPU, so it is about half of the 1-GPU value; use a larger batch to see scaling.
+
+```text
+step 1:  timing_s/step: ~16   perf/throughput: ~125 tok/s per GPU
+step 2:  timing_s/step: ~8    perf/throughput: ~235 tok/s per GPU
+perf/max_memory_allocated_gb: ~5.4 per GPU
+```
 
 ## 4. Run the Plugin Test Suite
 
@@ -117,8 +128,8 @@ Intel GPU attached.
 | Trainer backend | FSDP, FSDP2 | Megatron — Work in Progress |
 | Algorithms | GRPO, PPO, SFT | Validated on GSM8K / Qwen2.5 |
 | Hardware (1-GPU) | Validated | Arc Pro B60: ~8 s/step steady state, ~450 tok/s (batch 16, Qwen2.5-0.5B, vLLM 0.29.0) |
-| Hardware (2-GPU) | Validated | Arc Pro B60 x2 (not re-measured on the current stack) |
-| Multi-node | Work in Progress | — |
+| Hardware (2-GPU) | Validated | Arc Pro B60 x2: ~8 s/step steady state at the same batch (~235 tok/s per GPU); FSDP2 and sleep mode also run |
+| Multi-node | Work in Progress | Two-node collectives work only with `CCL_ENABLE_SYCL_KERNELS=0` (then `ReduceOp.AVG` is unavailable); with oneCCL defaults a 64 MiB `all_reduce` across nodes hangs. A multi-node verl training run is not validated |
 | Profiler | Intel VTune (ITT) — Validated | `profiler_markers()` hook merged into verl-core `main` on 2026-10-08; VTune collector capture itself still untested, see [profiling.md](./profiling.md) |
 
 ## Next Steps

From 4468dd5c2eaa68799cae46b364a07b37e6634897 Mon Sep 17 00:00:00 2001
From: kahlun <kah.lun.teoh@intel.com>
Date: Fri, 30 Oct 2026 05:32:38 +0000
Subject: [PATCH 3/5] [XPU] docs: align install guide with the merged vLLM
 0.32.0 XPU Dockerfile (verl#8401)

Update versions (vLLM 0.32.0, torch 2.14.0, transformers from verl's window), image tag, driver stack and oneCCL TMP_BUF variables, and link the Dockerfile in verl-core.
---
 docs/accelerators/xpu/README.md           |  2 +-
 docs/accelerators/xpu/install_guidance.md | 41 ++++++++++++++++-------
 docs/accelerators/xpu/profiling.md        |  2 +-
 docs/accelerators/xpu/quick_start.md      |  4 +--
 4 files changed, 32 insertions(+), 17 deletions(-)

diff --git a/docs/accelerators/xpu/README.md b/docs/accelerators/xpu/README.md
index 303150c7..8512fffa 100644
--- a/docs/accelerators/xpu/README.md
+++ b/docs/accelerators/xpu/README.md
@@ -1,6 +1,6 @@
 # Intel XPU User Guide
 
-Last updated: 10/15/2026.
+Last updated: 10/30/2026.
 
 ## Introduction
 
diff --git a/docs/accelerators/xpu/install_guidance.md b/docs/accelerators/xpu/install_guidance.md
index 967ddb78..240ee879 100644
--- a/docs/accelerators/xpu/install_guidance.md
+++ b/docs/accelerators/xpu/install_guidance.md
@@ -32,26 +32,39 @@ plugin-side monkeypatch over `torch.distributed.all_reduce`
 
 ## 1. Install verl and verl-hardware-plugin
 
-Pick one. Both install the same set of packages (torch 2.13.0+xpu, vLLM 0.29.0+xpu, transformers 5.17.0): the Docker
+Pick one. Both install the same set of packages (torch 2.14.0+xpu, vLLM 0.32.0+xpu, transformers 5.16.1): the Docker
 image runs the commands of Option B.
 
 ### Option A: Docker (recommended)
 
-The image is built from `docker/xpu/` in verl-core. It installs verl and this plugin together, so nothing else is
-needed. Pin the two checkouts with `--build-arg VERL_REF=<commit> --build-arg PLUGIN_REF=<commit>`; for release builds
-add `--no-cache` so the system certificates, `pip`, `certifi` and `uv` are fetched fresh.
+The image is [`docker/xpu/Dockerfile.stable.vllm`](https://github.com/verl-project/verl/blob/main/docker/xpu/Dockerfile.stable.vllm)
+in verl-core. It installs verl and this plugin together, so nothing else is needed. Pin the two checkouts with
+`--build-arg VERL_REF=<commit> --build-arg PLUGIN_REF=<commit>`; for release builds add `--no-cache` so the system
+certificates, `pip`, `certifi` and `uv` are fetched fresh.
 
 ```bash
 git clone https://github.com/verl-project/verl.git && cd verl
-docker build -f docker/xpu/Dockerfile.stable.vllm -t verl:xpu-vllm029 .
-docker run --rm -it --device /dev/dri --ipc=host --privileged verl:xpu-vllm029
+docker build -f docker/xpu/Dockerfile.stable.vllm -t verl:xpu-vllm032 .
+docker run --rm -it --device /dev/dri --ipc=host --privileged verl:xpu-vllm032
 ```
 
+The image is built in stages (`drivers` -> `runtime-base` -> `builder` -> `runtime`); only the finished `/opt/venv` and
+the verl / plugin checkouts (`/opt/verl`, `/opt/verl-hardware-plugin`) are shipped. It contains:
+
+- the Intel GPU user-mode driver (compute-runtime 26.27, IGC 2.38.2, Level Zero 1.32), matching vLLM 0.32.0's XPU image;
+- gcc/g++ and the Python headers, `ffmpeg`, `libnuma-dev` and `libze-dev` (see the system packages under Option B);
+- `CCL_SYCL_ALLGATHERV_TMP_BUF`, `CCL_SYCL_ALLREDUCE_TMP_BUF`, `CCL_SYCL_REDUCE_SCATTER_TMP_BUF` and
+  `CCL_SYCL_BROADCAST_TMP_BUF`, all set to `1`: they stop oneCCL from caching IPC handles of freed buffers, which
+  leaked host memory on multi-GPU runs. Set them yourself when you install from source.
+
+The build needs a verl checkout whose `requirements.txt` allows transformers >= 5.16.1 (verl `main` since PR #8390).
+An older `VERL_REF` is rejected by a check at the end of the build.
+
 ### Option B: From source
 
 verl pins its own versions, and `pip install -e .` on a verl checkout can replace the XPU build of torch with a
 different one (it also moves `transformers` and `tensordict` to verl's pins). Install the dependencies as below and
-add verl and the plugin with `--no-deps`.
+add verl and the plugin with `--no-deps`. The Dockerfile above is the reference for these steps.
 
 System packages on the host: `g++`/`gcc` and the Python 3.12 headers (Triton compiles its launcher stubs at first
 use), `libnuma-dev`, `libze-dev` (it provides the unversioned `libze_loader.so` that oneCCL opens; without it `xccl`
@@ -63,21 +76,23 @@ video).
 uv venv --python 3.12 --seed /opt/venv && source /opt/venv/bin/activate
 export UV_INDEX_URL=https://download.pytorch.org/whl/xpu UV_EXTRA_INDEX_URL=https://pypi.org/simple \
        UV_INDEX_STRATEGY=unsafe-best-match VLLM_TARGET_DEVICE=xpu
+export CCL_SYCL_ALLGATHERV_TMP_BUF=1 CCL_SYCL_ALLREDUCE_TMP_BUF=1 \
+       CCL_SYCL_REDUCE_SCATTER_TMP_BUF=1 CCL_SYCL_BROADCAST_TMP_BUF=1
 
 # vLLM XPU wheel (the PyPI wheel has no XPU kernels). --extra-index-url on the command line replaces
 # UV_EXTRA_INDEX_URL, so PyPI is repeated.
-uv pip install "vllm==0.29.0+xpu" \
-    --extra-index-url https://wheels.vllm.ai/0.29.0/xpu --extra-index-url https://wheels.vllm.ai/xpu \
+uv pip install "vllm==0.32.0+xpu" \
+    --extra-index-url https://wheels.vllm.ai/0.32.0/xpu --extra-index-url https://wheels.vllm.ai/xpu \
     --extra-index-url https://pypi.org/simple
 
 git clone https://github.com/verl-project/verl.git && cd verl
-grep -v '^transformers' requirements.txt > /tmp/requirements-core.txt
 echo 'pyzes>=0.1.2' > /tmp/override-pyzes.txt   # torch pins pyzes==0.1.1; the device-memory query needs >= 0.1.2
 
 # --no-config: run inside a verl checkout, uv would otherwise read verl's pyproject [tool.uv] overrides
-# (transformers 5.12.1, kernels<0.15 for the CUDA stack) and they would beat the XPU pins below.
-uv --no-config pip install -r /tmp/requirements-core.txt \
-    "torch==2.13.0+xpu" "transformers==5.17.0" "kernels>=0.16.0,<0.17.0" \
+# (for the CUDA stack) and they would beat the XPU pins below. transformers is not pinned: it comes from verl's
+# requirements.txt window and vLLM's (>= 5.16.1), and the [kernels] extra brings the matching kernels release.
+uv --no-config pip install -r requirements.txt \
+    "torch==2.14.0+xpu" "transformers[kernels]" \
     dpctl "pyzes>=0.1.2" triton-xpu "pip>=26.2.2" "certifi>=2026.10.19" qwen_vl_utils mathruler \
     --override /tmp/override-pyzes.txt
 uv --no-config pip install --no-deps -e .
diff --git a/docs/accelerators/xpu/profiling.md b/docs/accelerators/xpu/profiling.md
index e5d22bce..2f90c741 100644
--- a/docs/accelerators/xpu/profiling.md
+++ b/docs/accelerators/xpu/profiling.md
@@ -1,6 +1,6 @@
 # Intel VTune Profiling Guide
 
-Last updated: 10/15/2026.
+Last updated: 10/30/2026.
 
 This guide describes Intel VTune (ITT) profiling support in `verl-hardware-plugin`
 for Intel XPU. There is no PyTorch-native profiler activity for Intel XPU, so
diff --git a/docs/accelerators/xpu/quick_start.md b/docs/accelerators/xpu/quick_start.md
index 95d7b915..814400e1 100644
--- a/docs/accelerators/xpu/quick_start.md
+++ b/docs/accelerators/xpu/quick_start.md
@@ -98,7 +98,7 @@ timing_per_token_ms/gen: ~1.0    perf/max_memory_allocated_gb: ~9.2
 ```
 
 (Three runs on 2026-10-21 to 2026-10-23: vLLM 0.29.0+xpu, torch 2.13.0+xpu, this plugin at `e5ab532`, verl `main` at
-different commits. Timings vary with the stack and the host.)
+different commits; a 1-GPU rerun on vLLM 0.32.0+xpu, torch 2.14.0+xpu gave the same step times. Timings vary with the stack and the host.)
 
 Two GPUs: set `ZE_AFFINITY_MASK=0,1`, `trainer.n_gpus_per_node=2` and `+ray_kwargs.ray_init.num_gpus=2`, and keep
 everything else the same. At this small batch the step time does not drop, because fixed per-step overhead dominates.
@@ -127,7 +127,7 @@ Intel GPU attached.
 | Inference engine | vLLM validated | SGLang: `update_weights` — Work in Progress |
 | Trainer backend | FSDP, FSDP2 | Megatron — Work in Progress |
 | Algorithms | GRPO, PPO, SFT | Validated on GSM8K / Qwen2.5 |
-| Hardware (1-GPU) | Validated | Arc Pro B60: ~8 s/step steady state, ~450 tok/s (batch 16, Qwen2.5-0.5B, vLLM 0.29.0) |
+| Hardware (1-GPU) | Validated | Arc Pro B60: ~8 s/step steady state, ~450 tok/s (batch 16, Qwen2.5-0.5B, vLLM 0.29.0 and 0.32.0) |
 | Hardware (2-GPU) | Validated | Arc Pro B60 x2: ~8 s/step steady state at the same batch (~235 tok/s per GPU); FSDP2 and sleep mode also run |
 | Multi-node | Work in Progress | Two-node collectives work only with `CCL_ENABLE_SYCL_KERNELS=0` (then `ReduceOp.AVG` is unavailable); with oneCCL defaults a 64 MiB `all_reduce` across nodes hangs. A multi-node verl training run is not validated |
 | Profiler | Intel VTune (ITT) — Validated | `profiler_markers()` hook merged into verl-core `main` on 2026-10-08; VTune collector capture itself still untested, see [profiling.md](./profiling.md) |

From e1a4bddc64de145df81e77f5dffccd920835ec06 Mon Sep 17 00:00:00 2001
From: kahlun <kah.lun.teoh@intel.com>
Date: Fri, 30 Oct 2026 06:30:20 +0000
Subject: [PATCH 4/5] [XPU] docs: Option B needs LD_LIBRARY_PATH for oneCCL;
 list the Sysman libraries

Checked on a bare ubuntu:24.04 job with one B60: the Option B commands install the same versions as the image and platform detection works, but an xccl all_reduce crashes in oneCCL plugin loading until LD_LIBRARY_PATH=/opt/venv/lib:/usr/local/lib is set.
---
 docs/accelerators/xpu/install_guidance.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/docs/accelerators/xpu/install_guidance.md b/docs/accelerators/xpu/install_guidance.md
index 240ee879..0e3da5c8 100644
--- a/docs/accelerators/xpu/install_guidance.md
+++ b/docs/accelerators/xpu/install_guidance.md
@@ -69,13 +69,15 @@ add verl and the plugin with `--no-deps`. The Dockerfile above is the reference
 System packages on the host: `g++`/`gcc` and the Python 3.12 headers (Triton compiles its launcher stubs at first
 use), `libnuma-dev`, `libze-dev` (it provides the unversioned `libze_loader.so` that oneCCL opens; without it `xccl`
 fails with "ze_data was not initialized"), and `ffmpeg` (the FFmpeg shared libraries that `torchcodec` needs to decode
-video).
+video). The Docker image also installs `libhwloc15`, `libpciaccess0`, `libigsc1` and `intel-metrics-library` (from
+Intel's `ppa:kobuk-team/intel-graphics`) for the Sysman PCI/NUMA query that pyzes makes; install them as well.
 
 ```bash
 # Use pip >= 26.2.2 and uv >= 0.13.4 (both carry the corrected CA bundle).
 uv venv --python 3.12 --seed /opt/venv && source /opt/venv/bin/activate
 export UV_INDEX_URL=https://download.pytorch.org/whl/xpu UV_EXTRA_INDEX_URL=https://pypi.org/simple \
        UV_INDEX_STRATEGY=unsafe-best-match VLLM_TARGET_DEVICE=xpu
+export LD_LIBRARY_PATH=/opt/venv/lib:/usr/local/lib   # without it oneCCL fails: "could not load any plugin"
 export CCL_SYCL_ALLGATHERV_TMP_BUF=1 CCL_SYCL_ALLREDUCE_TMP_BUF=1 \
        CCL_SYCL_REDUCE_SCATTER_TMP_BUF=1 CCL_SYCL_BROADCAST_TMP_BUF=1
 

From b48f49f9f6fb66b878b54ac3310a095edea1dfea Mon Sep 17 00:00:00 2001
From: kahlun <kah.lun.teoh@intel.com>
Date: Wed, 4 Nov 2026 08:04:59 +0000
Subject: [PATCH 5/5] [XPU] docs: drop measured run metrics from the quick
 start

Remove the step-time, throughput and memory numbers (and the run dates and stack versions they came from) from the expected output and the feature matrix. They depend on the GPU and the software stack and go stale; the launch commands and what a successful run looks like stay.
---
 docs/accelerators/xpu/quick_start.md | 26 +++++---------------------
 1 file changed, 5 insertions(+), 21 deletions(-)

diff --git a/docs/accelerators/xpu/quick_start.md b/docs/accelerators/xpu/quick_start.md
index 814400e1..3307e44d 100644
--- a/docs/accelerators/xpu/quick_start.md
+++ b/docs/accelerators/xpu/quick_start.md
@@ -88,27 +88,11 @@ This is a smoke test: `total_training_steps=2` stops after two steps (without it
 batch 16 is about 467 steps). Raise it for a longer run. A `DataLoader worker ... is killed by signal` traceback
 printed at exit is shutdown noise after training has finished.
 
-Expected (Qwen2.5-0.5B-Instruct, GSM8K, batch 16, 1x Arc Pro B60). The first step includes warm-up; the second shows
-steady state:
-
-```text
-step 1:  timing_s/step: ~16-21   perf/throughput: ~190-235 tok/s
-step 2:  timing_s/step: ~8-8.5   perf/throughput: ~440-465 tok/s
-timing_per_token_ms/gen: ~1.0    perf/max_memory_allocated_gb: ~9.2
-```
-
-(Three runs on 2026-10-21 to 2026-10-23: vLLM 0.29.0+xpu, torch 2.13.0+xpu, this plugin at `e5ab532`, verl `main` at
-different commits; a 1-GPU rerun on vLLM 0.32.0+xpu, torch 2.14.0+xpu gave the same step times. Timings vary with the stack and the host.)
+Expected: two training steps are logged, each with metrics such as `timing_s/step` and `perf/throughput`, and the
+process exits. Step times and throughput depend on the GPU and the software stack, so none are listed here.
 
 Two GPUs: set `ZE_AFFINITY_MASK=0,1`, `trainer.n_gpus_per_node=2` and `+ray_kwargs.ray_init.num_gpus=2`, and keep
-everything else the same. At this small batch the step time does not drop, because fixed per-step overhead dominates.
-`perf/throughput` is reported per GPU, so it is about half of the 1-GPU value; use a larger batch to see scaling.
-
-```text
-step 1:  timing_s/step: ~16   perf/throughput: ~125 tok/s per GPU
-step 2:  timing_s/step: ~8    perf/throughput: ~235 tok/s per GPU
-perf/max_memory_allocated_gb: ~5.4 per GPU
-```
+everything else the same. `perf/throughput` is reported per GPU.
 
 ## 4. Run the Plugin Test Suite
 
@@ -127,8 +111,8 @@ Intel GPU attached.
 | Inference engine | vLLM validated | SGLang: `update_weights` — Work in Progress |
 | Trainer backend | FSDP, FSDP2 | Megatron — Work in Progress |
 | Algorithms | GRPO, PPO, SFT | Validated on GSM8K / Qwen2.5 |
-| Hardware (1-GPU) | Validated | Arc Pro B60: ~8 s/step steady state, ~450 tok/s (batch 16, Qwen2.5-0.5B, vLLM 0.29.0 and 0.32.0) |
-| Hardware (2-GPU) | Validated | Arc Pro B60 x2: ~8 s/step steady state at the same batch (~235 tok/s per GPU); FSDP2 and sleep mode also run |
+| Hardware (1-GPU) | Validated | Arc Pro B60 (the launch command above) |
+| Hardware (2-GPU) | Validated | Arc Pro B60 x2; FSDP2 and sleep mode also run |
 | Multi-node | Work in Progress | Two-node collectives work only with `CCL_ENABLE_SYCL_KERNELS=0` (then `ReduceOp.AVG` is unavailable); with oneCCL defaults a 64 MiB `all_reduce` across nodes hangs. A multi-node verl training run is not validated |
 | Profiler | Intel VTune (ITT) — Validated | `profiler_markers()` hook merged into verl-core `main` on 2026-10-08; VTune collector capture itself still untested, see [profiling.md](./profiling.md) |