[Intel][XPU]Enable chunked prefill scnearios for XPU with UT

[AnuSajikumar6264]

Motivation

The chunked-prefill scripted-runtime suite under test/manual/chunked_prefill/ could not run at
all on Intel XPU: the harness killed every engine during startup. Three argument-resolution paths
also ended in an unhelpful failure rather than a clear rejection. This makes the suite runnable on
--device xpu and reports honestly where a path cannot work there.

Rebased on f1a512c51c. 23 files, +389/-125.

Modifications

Engine startup and runtime

• test/scripted_runtime/http_server.py — kv-canary JIT-compiles its plan/verify/write kernels
  from .cuh against the arch reported by torch.cuda, so install_canary raised inside
  ModelRunner.alloc_memory_pool; the server died before the dispatch-loop handshake and the test
  side waited out LISTENER_ACCEPT_TIMEOUT_S. The harness now launches with the canary off wherever
  its kernels cannot build, and passes nothing rather than restating the ServerArgs defaults.
• managers/scheduler.py — continue_generation read torch.cuda.memory_reserved(), which
  returns 0 when CUDA was never initialized, so the reclaim log asserted a call that never happened.
  It now reads the active device module, and when the device exposes no allocator hook it says so
  instead of reporting a 0 MB reclaim.
• utils/common.py — new device_memory_reserved, next to the existing empty_device_cache,
  returning 0 where the device module has no memory_reserved (torch.cpu and torch.mps do not).
• layers/attention/{aiter,wave}_backend.py — both read value buffer 0, but the pool indexes as
  layer_id - start_layer, so a non-first PP rank read a negative index. Both now index from
  start_layer, reusing the wording of the same fix already on main in triton_backend.py.
  wave_backend uses the self.token_to_kv_pool reference the constructor already captured;
  aiter_backend keeps a local because its own attribute is not assigned until later.

Argument resolution

• arg_groups/kv_cache_hook.py — a host KV pool resolves its page movers from
  sgl_kernel.kvcacheio, imported only under if _is_cuda or _is_hip with no else branch, then
  called unconditionally (pool_host/mha.py:278,331,475,863). So on XPU
  --enable-hierarchical-cache, --disaggregation-decode-enable-offload-kvcache and an explicit
  --disaggregation-decode-retraction-backup=host_pool each died minutes into startup on a bare
  NameError. All three are now rejected in handle_cache_compatibility, where the other host-pool
  combinations are already validated and which runs unconditionally.
• mem_cache/kv_cache_builder.py — a decode server that leaves the retraction backup unset
  degrades to cpu_tensor on XPU instead of being rejected for a pool it never asked for, matching
  what HIP already gets. Keyed on the resolved device, not a hardware probe.
• arg_groups/overrides.py — deterministic inference fell through to fa3, which asserts
  SM 8x/9x in its own factory, so --device xpu --enable-deterministic-inference could not start.
  XPU now resolves to triton, which unlike intel_xpu is in
  RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND and so does not silently disable the radix cache.
  Two lines, inserted after the SM branches; no other device's fallback changes.
• layers/quantization/mxfp4.py — mxfp4 is registered on cpu and xpu, where the weight-load
  tail called torch.cuda.empty_cache() and freed nothing.

Test-side fixes the above exposed

• scripted_runtime/background_http_poster.py, context/http_post.py — a rejected control POST
  surfaced only as an unrelated socket-arrival timeout. submit_coro now returns its future and the
  timeout handler interrogates that one future, so the server's own complaint is reported. This
  removes the process-global failure list, whose entries could be attributed to a later request, and
  restores _log_coro_exception as a staticmethod — which also unbreaks four pre-existing CI tests
  in test/registered/unit/scripted_runtime/test_background_http_poster.py.
• test_scripted_abort.py — a repeat abort is dropped at the TokenizerManager once
  abort_sent is set, so awaiting a second AbortReq could only time out; and reusing a rid needs
  the TokenizerManager to have released it, which no number of scheduler steps in the same yield
  can achieve.
• scripted_runtime_chunked_helpers.py — _drain_until_released had drifted into four copies,
  three byte-identical; it is now one drain_until_released helper that also waits on lock_refs,
  which drops an iteration after the KV pages. Call sites in test_scripted_abort.py,
  test_scripted_lifecycle.py, test_scripted_regression.py and test_scripted_pp.py are rewired to
  it (test_scripted_pp.py passes max_steps=16 to keep its longer drain). The rid settle constant
  moved up with the other module constants and states where its value came from and how a slower
  device fails; test_scripted_multi_req.py loses a comment that duplicated it.
• test_scripted_regression.py — the abort-release regression guards read through
  find_req_by_rid, which stops seeing a req whether it released or leaked, so they passed on a
  leaked row. They now compare pool free counts, with the per-req probes kept only on the branch
  where they can still fail.
• test_scripted_special_case.py — the flashinfer and HiCache classes skip on devices that
  cannot run them, instead of each burning a 300 s listener timeout.

Accuracy Tests

New CI unit tests, on CPU runners:

python -m pytest -q \
  test/registered/unit/server_args/ \
  test/registered/unit/scripted_runtime/ \
  test/registered/unit/utils/test_device_module_probes.py
# 331 passed, 99 subtests passed

Added as test/registered/unit/utils/test_device_module_probes.py (new file), plus cases appended to
test/registered/unit/scripted_runtime/test_http_server.py and
test/registered/unit/server_args/test_server_args.py. They cover the four runtime behaviours this
PR adds: the device-module probes
(device_memory_reserved / empty_device_cache with and without the hooks), the canary knobs per
device string, the host-pool rejection per opt-in path, and the deterministic attention fallback with
the platform pinned so the result holds on any runner.

The manual suite, on one Intel Arc Pro B60 (23.9 GiB):

source /opt/intel/oneapi/2026.1/oneapi-vars.sh --force
export PYTHONPATH=$PWD/python
ZE_AFFINITY_MASK=7 CUDA_VISIBLE_DEVICES=7 MASTER_PORT=29700 SGLANG_TEST_PORT_BASE=29720 \
  python -m pytest -v -ra --timeout=1200 test/manual/chunked_prefill/

302 collected. Latest full run on this branch: 301 passed, 11 skipped, 19 errors, 0 failures
(baseline before this PR on the same hardware: 252 passed, 23 skipped, 27 errors). No test-body
assertion failures in either run.

End-to-end output accuracy

The scripted tests verify bookkeeping (chunk counts, KV pages, lock refs, pool accounting); none of
them checks that the emitted tokens are correct, so a chunked KV write landing at the wrong position
would pass all 301. GSM8K covers that gap. The number that matters is chunked vs unchunked on the
same model and seed, not the absolute score.

Qwen/Qwen3-0.6B, --device xpu --attention-backend triton, 200 examples, 32 threads:

--chunked-prefill-size	GSM8K score	eval latency (s)	output tok/s
256	0.575	296.76	551.44
-1 (off)	0.605	320.56	536.68

Delta -0.030 against a standard error of 0.049 on the difference (0.61 sigma), i.e. indistinguishable
at this sample size; chunking is also 7 % faster end to end, consistent with the sweep above.

Two honest caveats on this figure. Only 11 of 200 requests actually spanned a chunk boundary, because
most GSM8K prompts are shorter than 256 tokens — so it is a weak probe of the chunked path. And
n=200 cannot resolve a difference smaller than about 0.10, so it rules out a gross corruption, not a
subtle one. The stronger measurement is the suite's own test_mixed_prefix_gsm8k_chunked, which uses
24-shot mixed-prefix prompts specifically so every request spans several chunks; it is currently
blocked on XPU by the Llama-3.2-1B-Instruct failure described below, not by anything in this PR.

Suite errors

The 19 remaining errors are environmental or out of scope, not chunked-prefill defects:

Cause	Count
openai/gpt-oss-20b not cached (mxfp4 is registered on XPU upstream, so the SWA classes attempt the load)	10
Multi-card classes on a single-card run (device index is out of range)	5
meta-llama/Llama-3.2-1B-Instruct fails on XPU on current main: RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x14336 and 2048x2048), raised from the plain model forward (eager_runner._execute_extend), with no LoRA in the stack. This PR touches no model or LoRA file, and it reproduces with these commits applied to the parent of #30345, so it is neither this PR's doing nor that commit's. It is the sole cause of the four lora_overlap errors and of every e2e GSM8K error	4

Speed Tests

This PR alters no kernel and no scheduling path, so it cannot move a number by construction. What is
worth showing for an enablement PR is that chunked prefill behaves sensibly on XPU once the suite can
run, so the sweep below is a characterisation of the feature on this hardware rather than a
before/after for this diff.

One Intel Arc Pro B60 (23.9 GiB), Qwen/Qwen3-0.6B, --device xpu --attention-backend triton,
--mem-fraction-static 0.7. 65 536 total input tokens in every row.

python -m sglang.benchmark.offline_throughput \
  --model-path Qwen/Qwen3-0.6B --device xpu --attention-backend triton \
  --dataset-name random --random-input 4096 --random-output 8 --random-range-ratio 1.0 \
  --num-prompts 16 --chunked-prefill-size <N> --mem-fraction-static 0.7

--random-input 4096, 16 prompts:

--chunked-prefill-size	duration (s)	input tok/s	vs chunking off
512	18.90	3467.58	+9.3 %
1024	18.66	3512.29	+10.7 %
2048	19.20	3412.92	+7.6 %
4096 (= input, one chunk)	20.80	3151.15	-0.6 %
-1 (off)	20.66	3171.58	baseline

--random-input 8192, 8 prompts:

--chunked-prefill-size	duration (s)	input tok/s	vs chunking off
1024	33.14	1977.25	+16.7 %
2048	33.94	1931.09	+13.9 %
8192 (= input, one chunk)	38.84	1687.22	-0.4 %
-1 (off)	38.67	1694.78	baseline

Chunked prefill is worth 10-17 % input throughput on XPU at these shapes, best around 1024, and the
benefit grows with prompt length. Setting the chunk size equal to the input length reproduces the
unchunked number to within 0.6 %, which is the expected degenerate case and a useful sanity check that
the sweep is measuring what it claims.

Note for anyone reproducing this: sglang.benchmark.one_batch is not usable here. It constructs a
ModelRunner directly and never enters the scheduler, so every --chunked-prefill-size yields an
identical number. offline_throughput drives a real Engine, and the scheduler log confirms chunking
engages (Prefill batch, #new-token: 512 ... #pending-token: 3583).

Notes

• test/manual/ is excluded from CI by test/README.md, so the runtime behaviours added here are
  covered by the test/registered/unit/ tests above rather than by the manual suite.
• A median/MAD latency-outlier filter for ChunkSizePredictor.fit was considered and dropped.
  Replayed against the only real 127-sample XPU profile in the tree, it moved the quadratic
  coefficient from -1.24e-05 to -3.65e-05 — further from the a > 0 acceptance test — by discarding
  the largest and most curvature-informative sample. On that sweep corr(seq_len, latency) is
  -0.108, so the quadratic term is unidentifiable and the problem is the sweep range, not outliers.
  TestPPDynamic therefore still cannot pass on XPU; a wider sweep or a linear fallback is the real
  fix and is not attempted here.

---

CI States

Latest PR Test (Base): ✅ Run #36384585722
Latest PR Test (Extra): ❌ Run #36384585473
Latest PR Test (AMD ROCm 10): ❌ Run #36384585714

[siju-samuel]

/tag-run-ci-label

[AnuSajikumar6264]

Thanks — all twelve are addressed. Rebased onto f1a512c51c (was 1357 commits behind at the
first pass, then 42, then 27), which by itself dropped the hunks you flagged as already landed.
Per-comment below; #10 has a longer reply in its own thread.

---

1. arg_groups/hicache_hook.py — wrong function, incomplete coverage. Agreed on both. Moved out
of the layout/IO resolver, but into handle_cache_compatibility (kv_cache_hook.py) rather than
validate_hicache_host_memory_mode: it already groups these three host-pool flags, already raises,
and runs unconditionally, so the veto can't be skipped by an unrelated gate. Broadened to all three
opt-ins including an explicit --disaggregation-decode-retraction-backup=host_pool, which your
proposal would still have let through since the selector only runs when the backup is None. The
auto-resolved case degrades to cpu_tensor in kv_cache_builder.py instead, keyed on
get_device().device rather than is_xpu() — the probe is hardware presence, so it would silently
degrade a --device cpu config on any XPU-capable host. hicache_hook.py now drops out of the diff
entirely. Left the try/except-stub idea for a follow-up; it is the better long-term shape but a much
larger change.

2. scheduler.py — false-success reporting one device over. Correct, and thank you: I had traded
an XPU bug for a CPU one. Now branches on empty_device_cache's return, exactly as you suggested,
and device_memory_reserved was added because torch.cpu has no memory_reserved either.

3. background_http_poster.py — return the future. Implemented as described; _failures,
_failures_lock, _record_failure and take_failures() are gone and _log_coro_exception is a
staticmethod again. Settle timeout is 1.0 s: loopback POST into the same process, so anything
unsettled a second past an already-elapsed 60 s deadline is stuck rather than racing.
This turned up something neither of us had flagged: the PR was breaking four pre-existing CI tests in
test/registered/unit/scripted_runtime/test_background_http_poster.py (9 pass on main, 4 failed on
the branch) because the file calls _log_coro_exception unbound and post() had started reading
resp.status. That directory is 39 passing now.
One correction to the diagnosis: neither failing abort test was caused by this. start_req posts
stream=True, so a duplicate-rid ValueError comes back inside a 200 SSE body and the future
carries no exception — resp.status >= 400 can never fire for /generate. The real causes were
abort_sent short-circuiting a repeat abort, and same-step rid reuse being impossible because the
release is cross-process. Both fixed, both passing.

4. http_server.py — return {} and the policy split. Took the return {}. On the policy: rather
than add a production rejection, I made the two harness paths share one predicate —
CANARY_SUPPORTED_DEVICES is now public and chunked_prefill_test_utils.py gates on it too. That
mattered more than it looked: use_kv_canary defaults True, so every e2e GSM8K server was
launching with --kv-canary raise and dying on XPU. Rejecting kv_canary != "none" in
validation_hook.py is the cleaner end state but changes production launch behaviour, which I did not
want to smuggle into this PR. Skipped (c) — a log line is enough now that a test asserts the kwargs.

5. test_case.py — one call site, shared base class. Agreed; moved to
@unittest.skipUnless(is_flashinfer_available(), ...) on the single class that sets that backend.
test_case.py is back to its upstream state and out of the diff. Commit messages rewritten — the
quantization claim was stale, and I removed the mxfp4 guard entirely since upstream now registers
mxfp4 for XPU, so it was dead code.

6. RID_RELEASE_SETTLE_STEPS — magic number, wrong placement. Moved up with the other module
constants, and it now records where 40 came from and that a slower device fails as a duplicate-rid
rejection rather than a timeout. I did not take the retry option: the rejection is invisible to the
harness for the reason in #3 (200 + SSE body), so a retry loop would have nothing to key on.

7. _drain_until_released — four copies. Hoisted to drain_until_released in
scripted_runtime_chunked_helpers.py with DRAIN_RELEASE_STEPS = 12; abort, lifecycle,
regression and pp all call it, pp passing max_steps=16. Comment reworded to state the
consequence rather than the mechanism.

8. Vacuous asserts, and >=. Both correct. The per-req probes are now inside
if req is not None:, and the pool comparisons use == — you were right that >= could only fire on
the impossible over-free side. A verifier reproduced the leak you described by no-op'ing
release_kv_cache, which is what convinced me the pool deltas had to carry the check.

9. wave_backend.py — shadowed reference, weaker comment. Both fixed. Uses the
self.token_to_kv_pool the constructor already captured, and both hunks now carry triton_backend.py's
wording verbatim so the three copies read alike. Kept the local in aiter_backend.py, since as you
noted its own attribute is not assigned until later.

10. Description drift. See the dedicated reply in that thread.

11. overrides.py — drifted mirror, and why triton. Took the substance, not the shared-resolver
call: routing through get_default_attn_backend breaks the arg-resolution unit tests on a non-CUDA
host (measured — it reddens test_cuda_host_keeps_its_fa3_default and the CPU case), so this stays a
two-line elif view.device == "xpu": backend = "triton" after the SM branches, scoped to XPU only.
Your guess about the reason was right and it is now in the comment: intel_xpu is absent from
RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND, so choosing it would silently trip the
disable_radix_cache fallback. ROCm/MPS/CPU deliberately unchanged — widening this in an XPU PR is
the drive-by you flagged elsewhere.

12. Nothing here is covered by CI. Fair, and the sharpest of the twelve. Added
test/registered/unit/utils/test_device_module_probes.py plus cases in test_http_server.py and
test_server_args.py, covering the device probes, the canary knobs per device string, the host-pool
rejection per opt-in path, and the deterministic fallback with the platform pinned so it holds on any
runner. 329 passing.
Two things I want to be straight about rather than let the green tick imply more than it should.
test/manual/ is excluded from CI and the CI runners are CUDA/CPU, so a green run on this PR carries
no XPU signal — the new tests are device-agnostic by design and would pass identically on a CUDA
box. The XPU evidence is a manual A/B on one Arc Pro B60: on pristine main,
TestChunkSizeDefault is 7 errors in 322 s (install_canary -> "HTTP server did not connect"); on
this branch it is 7 passed in 189 s, and the full 302 run is 301 passed / 0 failures / 3 h 28 m.
And register_xpu_ci exists (47 files use it), while this PR registers nothing for it — so the
enablement is not currently CI-enforced. The cheap fix is registering
test/registered/chunked_prefill/test_scripted_core_1gpu.py and
test/registered/scripted_runtime/test_scripted_runtime_core.py for stage-b-test-1-gpu-xpu; happy
to add it here or as a follow-up, but I cannot validate the XPU runners myself, so the first CI run
would be the real test.

---

Also worth flagging, unrelated to the review: meta-llama/Llama-3.2-1B-Instruct currently fails on
XPU on main with RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x14336 and 2048x2048),
raised from the plain model forward (eager_runner._execute_extend) with no LoRA in the stack. This
PR touches no model or LoRA file, and it reproduces with these commits applied to the parent of
#30345, so it is neither this PR's doing nor that commit's. It is the sole cause of the four
lora_overlap errors and of every e2e GSM8K error in the run above, and I will file it separately.

[siju-samuel]

/tag-run-ci-label

[AnuSajikumar6264]

  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1512, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1505, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1379, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 557, in uvloop.loop.Loop._run
  File "uvloop/loop.pyx", line 476, in uvloop.loop.Loop._on_idle
  File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/opt/venv/lib/python3.12/site-packages/sglang/srt/managers/tokenizer_manager.py", line 3701, in print_exception_wrapper
    await func()
  File "/opt/venv/lib/python3.12/site-packages/sglang/srt/managers/tokenizer_manager.py", line 3253, in sigterm_watchdog
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/starlette/routing.py", line 655, in lifespan
    await receive()
  File "/opt/venv/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive
    return await self.receive_queue.get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/queues.py", line 158, in get
    await getter
asyncio.exceptions.CancelledError

[2026-09-12 19:59:54] kill_process_tree called: parent_pid=15847, include_parent=False, pid=15847

======================================================================
ERROR: test_run (__main__.TestEncoderAttention.test_run)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/sglang/srt/utils/common.py", line 3539, in retry
    return fn()
           ^^^^
  File "/opt/venv/lib/python3.12/site-packages/sglang/test/test_utils.py", line 2195, in <lambda>
    lambda: super(CustomTestCase, self)._callTestMethod(method),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/sglang-checkout/test/registered/xpu/test_encoder_attention_backend.py", line 109, in test_run
    self.run_decode()
  File "/sglang-checkout/test/registered/xpu/test_encoder_attention_backend.py", line 100, in run_decode
    assert_one_item(ret)
  File "/sglang-checkout/test/registered/xpu/test_encoder_attention_backend.py", line 87, in assert_one_item
    if item["meta_info"]["finish_reason"]["type"] == "stop":
       ~~~~^^^^^^^^^^^^^
KeyError: 'meta_info'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/sglang/test/test_utils.py", line 2194, in _callTestMethod
    retry(
  File "/opt/venv/lib/python3.12/site-packages/sglang/srt/utils/common.py", line 3554, in retry
    raise Exception(f"retry() exceed maximum number of retries.")
Exception: retry() exceed maximum number of retries.

----------------------------------------------------------------------
Ran 1 test in 45.250s

FAILED (errors=1)
{
  "error": {
    "message": "Qwen-VL image artifacts do not match prompt placeholders"
  }
}
.
.
End (5/29):
filename='/sglang-checkout/test/registered/xpu/test_encoder_attention_backend.py', elapsed=52, estimated_time=360.0
.
.


✗ FAILED: /sglang-checkout/test/registered/xpu/test_encoder_attention_backend.py returned exit code 1

Fail. Time elapsed: 616.07s

The xpu failure is unrelated to my changes.

[mingfeima]

keep change smaller.

[mingfeima]

/rerun-failed-ci

[mingfeima]

keep changes minimal and focused.

[mingfeima]

LGTM now. XPU ci fail on main has been fixed, need rebase.