[XPU] Enable Gemma 4 E2B / E4B / 31B/ 26B-A4B on Intel XPU

[jmunetong]

[XPU] Enable Gemma 4 E2B / E4B / 31B on Intel XPU with SWA KV pool, sgl-kernel-xpu PR#191 alignment, and fused QKV RMSNorm

1. Motivation

Enable google/gemma-4-E2B-it, google/gemma-4-E4B-it, and google/gemma-4-31B-it on Intel XPU with both the triton and intel_xpu attention backends. Gemma 4 is a hybrid model that interleaves sliding-window attention (SWA, head_dim=256) with full attention (head_dim=512). Bringing it up on XPU surfaced four cross-layer correctness gaps and one performance gap that this PR closes:

1. RMSNorm.forward_xpu rejected the >2D tensor produced by Gemma 4's per-layer input projection.
2. The hybrid SWAKVPool page-table indices were not translated to the SWA sub-pool on the intel_xpu backend.
3. Gemma 4's cross-layer KV sharing (is_kv_shared_layer) was not declared on intel_xpu.
4. sgl-kernel-xpu PR Support prepopulated kv cache #191 silently changed the contract of cu_seqlens_k_new in flash_attn_with_kvcache; pre-PR-191 SGLang was passing total-K cumsums there, which post-PR-191 doubles the effective K length and corrupts chunked prefill.
5. The fused Triton kernel gemma_qkv_rmsnorm (Q/K/V RMSNorm before attention) was hard-gated to CUDA tensors despite being a device-agnostic Triton kernel that lowers cleanly to XPU via intel-xpu-backend-for-triton.

The PR also widens the Gemma4ForConditionalGeneration attention-backend whitelist to include intel_xpu (PR #25547 had restricted it to trtllm_mha / triton), and adds a 31B smoke test under test/srt/xpu/.

---

2. Modifications

#	Commit	File(s)	Change
1	b39ced740	python/sglang/srt/layers/layernorm.py	RMSNorm.forward_xpu reshapes >2D inputs to 2D before calling sgl_kernel.rmsnorm, mirroring the CUDA path.
2	b39ced740, da7f6e6b5, c52dd732f, 77d628607	python/sglang/srt/layers/attention/xpu_backend.py, test/srt/xpu/test_gemma_4_e2b.py, test/srt/xpu/gemma4_chat_template.jinja, test/srt/run_suite.py	Hybrid SWA detection in XPUAttentionBackend; full-pool → SWA-pool page-table translation for prefill, decode, and the page-size > 1 strided path. Adds the E2B XPU smoke test (test_simple_code_qa, test_sliding_window_long_context, test_sliding_window_3k_tokens) and registers it under the per-commit-xpu suite.
3	d1beb95a1	python/sglang/srt/layers/attention/xpu_backend.py	Cross-layer KV sharing: forward_extend / forward_decode now explicitly accept (k=None, v=None) for shared layers, mirroring triton_backend.py:907-916. Materialization is skipped; the paged kernel reads K/V via the upstream layer's pool because RadixAttention is initialized with layer_id=kv_shared_layer_index for shared layers.
4	2937e6e2d	python/sglang/srt/layers/attention/xpu_backend.py	sgl-kernel-xpu PR #191 alignment — 7 single-line edits at lines 565, 586, 696, 716, 841, 904, 925 setting cu_seqlens_k_new=None. cache_seqlens already encodes the full key length, so the kernel resolves to cu_seqlens_k = cache_seqlens and recovers the pre-PR-191 behavior. Mirrors the already-correct decode local-attn site at line 860.
5	ff13ca2a3	python/sglang/srt/layers/layernorm.py	Enable the sgl-kernel gemma_rmsnorm fused path on Gemma 4.
6	caf4d392f	python/sglang/srt/server_args.py, test/srt/xpu/test_gemma_4_31b.py, test/srt/xpu/gemma_4_{31b,e2b}_comparison.txt	Extend Gemma4ForConditionalGeneration accepted_backends to include intel_xpu. Add a 31B XPU smoke test stencil (single Q&A via OpenAI /v1).
7	c4335a52f	python/sglang/srt/layers/gemma4_fused_ops.py, python/sglang/srt/models/gemma4_causal.py	Admit XPU into the fused gemma_qkv_rmsnorm Triton path. Relax the Python-level q.is_cuda predicates (and the matched k/v assert) to q.is_cuda or q.is_xpu. The kernel itself (_gemma_qkv_rmsnorm_kernel) is device-agnostic and lowers via intel-xpu-backend-for-triton; only the eligibility checks needed widening. Smoke-tested bit-exact vs the eager pow / mean / rsqrt chain (max abs err 0.0).
8	f2c0d022f (cherry-pick of ckvermaAI/sglang PR #23757)	python/sglang/srt/layers/attention/xpu_backend.py	Stride-and-divide page-table normalization inside _init_local_attn_metadata so the page-granular block_table correctly indexes the token-granular req_to_token table when page_size > 1. Conflict-free cherry-pick on top of the SWA work above; gated on layer.use_irope, so it is dormant for Gemma 4 (no iRoPE) but kept to fix Llama-4-class iRoPE chunked attention on XPU.

The tensor-dump scaffolding that was used during cross-backend bring-up has been stripped (e36e51279); only production code ships in this PR.

---

3. sgl-kernel-xpu PR #191 — full integration

3.1 What changed in the kernel wrapper

sgl-kernel-xpu PR #191 ("Support prepopulated kv cache") rewrites flash_attn_with_kvcache in sgl_kernel/flash_attn.py:

	Pre-PR-191	Post-PR-191
cu_seqlens_k_new	accepted, made contiguous, silently discarded	interpreted as the cumulative count of new keys; k_new_lens = diff(cu_seqlens_k_new) is added to cache_seqlens to build the final cu_seqlens_k

xpu_backend.py always passed the cumsum of the total (cached + new) sequence length there — this is the same pattern every sister backend (CUDA flashattention_backend.py, MUSA flashattention_backend.py) follows, because the kwarg used to be a no-op. Post-PR-191 the kernel ends up with cu_seqlens_k ≈ 2 × total_len, attends past the valid KV region, and on Gemma 4 31B at chunked-prefill ON this surfaced as a 5.2 pp GSM8K accuracy drop and a 7.6× higher invalid rate.

3.2 Fix shape

Option A (chosen): pass cu_seqlens_k_new=None at every call site. The kernel then takes cu_seqlens_k = cache_seqlens directly, recovering pre-PR-191 behavior. 7 single-line edits in 1 file, all of the same shape:

                 cache_seqlens=cache_seqlens,
                 cu_seqlens_q=cu_seqlens_q,
-                cu_seqlens_k_new=cu_seqlens_k if not use_local_attn else None,
+                cu_seqlens_k_new=None,
                 max_seqlen_q=max_seqlen_q,
                 ...

Sites: 565 (extend non-MLA primary), 586 (extend non-MLA cascade-expand), 696 (extend MLA absorbed primary), 716 (extend MLA absorbed cascade-expand), 841 (decode cross-attention), 904 (decode self-attention primary), 925 (decode cascade-expand). Site at line 860 (decode local-attn) was already None.

Considered alternatives: (B) refactor metadata so cache_seqlens_int32 = extend_prefix_lens and add a true metadata.cu_seqlens_k_new = pad(cumsum(extend_seq_lens)) — much larger blast radius, kept as the long-term shape if we ever want to use the prepopulated-KV optimization PR #191 enables. (C) pin sgl-kernel-xpu to a pre-#191 commit — punts the problem.

3.3 Validation — Gemma 4 31B-it, GSM8K (200 questions, 5-shot, T=0, 4× Arc Pro B60, tp=4)

Pre-fix and post-fix on the same SGLang gemma-xpu branch; only difference is whether the 7 cu_seqlens_k_new=None edits are applied.

run	chunked prefill	radix cache	GSM8K acc	invalid	output thpt (tok/s)
intel_xpu pre-fix, chunked ON (default)	on	on	0.723	0.038	81.5
intel_xpu post-fix, chunked ON (default)	on	on	0.775	0.005	86.5
intel_xpu pre-fix, chunked OFF	off	off	0.753	0.005	39.0
intel_xpu post-fix, chunked OFF	off	off	0.750	0.005	48.8
triton on XPU (reference)	on	on	0.805	0.008	58.4
H200 cookbook (reference)	on	on	0.805	0.005	552.8

Headline (chunked ON, the production path): +5.2 pp accuracy, 7.6× lower invalid rate, +6 % throughput. Cross-validated on Gemma 4 26B-A4B-it (MoE) where chunked-ON ↔ chunked-OFF parity confirms no residual cu_seqlens_k_new corruption on the MoE attention path.

3.3.1 26B-A4B-it cross-validation (2026-05-26, 4× Arc Pro B60, tp=4, ZE_AFFINITY_MASK=0,1,2,3)

200 questions, 5-shot, T=0, parallel=16, --attention-backend intel_xpu, --page-size 64, post-PR-191 wrapper + cu_seqlens_k_new=None fix applied. gemma-xpu HEAD = f2c0d022f.

run	chunked prefill	GSM8K acc	invalid	latency (s)	output thpt (tok/s)
intel_xpu, chunked ON (default)	on	0.460	0.020	425.3	121.9
intel_xpu, chunked OFF (--chunked-prefill-size -1)	off	0.435	0.020	423.2	118.0

Δ = 2.5 pp (chunked-ON higher), well inside the Wilson 95 % CI at N=200 (≈±6.9 pp). Invalid rate identical at 0.020 on both runs. Confirms the MoE expert-routing path is not destabilised by chunked prefill on intel_xpu post-fix. Run dirs: model_enablement/gemma/results/26b_a4b_xpu_acc_20260526T{161718,162603}Z/.

Sources: model_enablement/gemma/pr191_fix_plan_REPORT.md §9, run dirs model_enablement/gemma-4-31b/results/31b_xpu_acc_20260515T{212706,213418}Z/, and the 26B-A4B run dirs above.

---

4. Latest GSM8K accuracy benchmarks (2026-05-19, post all fixes)

Re-ran GSM8K at the cookbook shape on gemma-xpu HEAD (f2c0d022f) with sgl-kernel-xpu post-PR-191 wrapper, on 4× Arc Pro B60. All runs: 200 questions, 5-shot, T=0, parallel=16, --attention-backend intel_xpu, --page-size 64, default chunked prefill ON, default radix cache ON, bare server (no chat template / parser overrides).

Variant	TP	GSM8K acc	Invalid	Latency (s)	Output thpt (tok/s)	H200 cookbook ref	Δ (pp)
gemma-4-E2B-it	1	0.180	0.005	124.3	255.6	0.170	+1.0
gemma-4-E4B-it	2	0.725	0.000	137.0	139.8	0.745	−2.0
gemma-4-E4B-it	1	0.730	0.000	94.5	205.4	0.745	−1.5
gemma-4-31B-it	4	0.810	0.005	250.0	94.8	0.805	+0.5
gemma-4-31B-it	4 (rerun)	0.840	0.005	244.9	98.0	0.805	+3.5
gemma-4-26B-A4B-it †	4	0.460	0.020	425.3	121.9	0.450	+1.0
gemma-4-26B-A4B-it † ‡	4	0.435	0.020	423.2	118.0	0.450	−1.5

† 26B-A4B-it runs at tp=4 with ZE_AFFINITY_MASK=0,1,2,3 on the 4× Arc Pro B60 host; date 2026-05-26. See §3.3.1 for the chunked-ON ↔ chunked-OFF cross-validation read-through.
‡ Chunked prefill OFF (--chunked-prefill-size -1); all other rows in this table run with default chunked prefill ON.

Run dirs (under model_enablement/gemma-4-31b/results/):

• E2B-it tp=1: e2b_xpu_acc_20260519T182533Z/ (0.180), e2b_xpu_acc_20260519T182534Z/ (0.180) — identical numbers from two parallel ports, confirms determinism at T=0.
• E4B-it tp=2: e4b_xpu_acc_20260519T182535Z/ (0.725), e4b_xpu_acc_20260519T183856Z/ (0.725), e4b_xpu_acc_20260519T190755Z/ (0.720), e4b_xpu_acc_20260519T190757Z/ (0.725).
• E4B-it tp=1: e4b_xpu_acc_20260519T191205Z/ (0.730), e4b_xpu_acc_20260519T191207Z/ (0.730).
• 31B-it tp=4: 31b_xpu_acc_20260519T175219Z/ (0.810), 31b_xpu_acc_20260519T175843Z/ (0.840).
• 26B-A4B-it tp=4 (under model_enablement/gemma/results/, 2026-05-26, 5-shot): 26b_a4b_xpu_acc_20260526T161718Z/ (0.460, chunked ON), 26b_a4b_xpu_acc_20260526T162603Z/ (0.435, chunked OFF).

4.1 How chunked-prefill accuracy got here

Stage	Stack	E2B (tp=1)	E4B (tp=2)	31B (tp=4)
Initial enablement (2026-04-20)	base models, chat-template, 5-shot	0.200 (E2B)	0.520 (E4B base)	—
-it rerun + bare server (2026-04-23 / 2026-04-30)	tp=2 fix on E4B; cookbook-aligned N=200	0.170	0.740	—
31B intel_xpu collapse (2026-05-07)	unfixed PR-191 contract	—	—	0.170 (invalid 0.675)
31B with triton backend workaround (2026-05-07)	swap attention backend only	—	—	0.805
Workaround flags (--chunked-prefill-size -1 --disable-radix-cache)	chunked-prefill OFF	—	—	0.740
Post-PR-191 cu_seqlens_k_new=None fix, chunked ON (2026-05-15)	this PR	—	—	0.775
Latest, all fixes including fused QKV RMSNorm (2026-05-19)	this PR's HEAD	0.180	0.725	0.810 (median 0.825)

Reading: the 31B-it model went from a hard collapse (acc 0.170, invalid 0.675, gibberish loops mid-generation) to cookbook-matching accuracy with chunked prefill on, without disabling radix cache. The PR-191 cu_seqlens_k_new fix (§3) is what bridges the workaround-OFF (0.740) and triton-backend (0.805) numbers to the production-default-flags 0.810. E2B and E4B were already correct after the SWA / KV-share / RMSNorm-shape fixes from the original PR; the recent work did not regress them.

---

5. Fused QKV RMSNorm (gemma_qkv_rmsnorm) — XPU enablement

5.1 Why this was needed

Gemma 4 applies an RMSNorm to Q, K, and V independently after the QKV projection and before attention. Three separate pow / mean / rsqrt chains per layer ran in eager mode on XPU — three kernel launches plus three element-wise scans per layer per decode step. The CUDA path already used a fused Triton kernel (_gemma_qkv_rmsnorm_kernel in python/sglang/srt/layers/gemma4_fused_ops.py) that does Q/K/V norm + Q/K weight scaling in one launch with strided views (no .contiguous() copy).

The kernel is device-agnostic Triton — it lowers to XPU via intel-xpu-backend-for-triton exactly as it does to CUDA via the upstream Triton compiler. The only thing rejecting XPU was the Python-level eligibility checks:

# Before
assert q.is_cuda
...
if has_kv:
    assert k.is_cuda and v.is_cuda

# After (commit c4335a52f)
assert q.is_cuda or q.is_xpu
...
if has_kv:
    assert (k.is_cuda and v.is_cuda) or (k.is_xpu and v.is_xpu)

Plus the matching can_fuse_qkv_norm predicate in gemma4_causal.py:321 flipped from q.is_cuda to q.is_cuda or q.is_xpu.

5.2 Bit-exact validation

Smoke-tested on Intel XPU (bf16, M=4, n_q=16, n_kv=4, head_dim=128): the fused kernel matches the eager pow / mean / rsqrt reference bit-exactly (max abs err 0.0). Result captured at commit message of c4335a52f.

5.3 Performance impact

The fused kernel collapses 3 RMSNorm launches per layer (Q, K, V) into 1, and folds the Q/K weight multiplication into the same pass. From the warm-pass profile (results/profile_e2b_tp1_20260519T201934Z/ etc.):

Variant	TP	Decode wall (May-12, eager Q/K/V norm)	Decode wall (May-19, fused QKV norm)	Δ
E2B-it	1	70.4 ms / step	51.8 ms / step	−26 %
E4B-it	2	113 ms / step	89.5 ms / step	−20.8 %
31B-it	4	145 ms / step	123.9 ms / step	−14.5 %

End-to-end (prefill + 1024 decode) on the 4× B60 host:

Variant	TP	tok/s (May-12)	tok/s (May-19, post-fused)	Δ
E2B-it	1	128.4	155.2	+21 %
E4B-it	2	88.0	88.2	flat (comm-bound)
31B-it	4	55.6	62.2	+12 %

E4B-it tp=2 is unchanged because its decode wall is dominated by AllReduce launch latency on the intra-host PCIe path (84 AllReduce calls / forward), not by the QKV-norm path. The fused kernel still collapses 3 launches to 1 there; the win is just absorbed by the comm bottleneck. E2B-it (single-rank, no AllReduce) and 31B-it (4-way TP, but with attention_k_eq_v=true halving QKV-norm work) both pick up the wall-time improvement.

Per-op kernel breakdown is unchanged from the May-12 baseline (FMHA head_dim=512 and the unfused GELU × gate MLP scan remain the dominant prefill bottlenecks). Sources: model_enablement/gemma/projection_vs_profile_{e2b,e4b}_README.md, model_enablement/gemma-4-31b/projection_vs_profile_31b_README.md.

---

6. PR #23757 cherry-pick — page-table normalization for _init_local_attn_metadata

f2c0d022f cherry-picks the single substantive commit from ckvermaAI/sglang#23757 ("Normalize page table values"). It adds an 11-line stride-and-divide block inside XPUAttentionBackend._init_local_attn_metadata so that make_local_attention_virtual_batches receives a page-granular block table (column = logical page number, value = physical page index) rather than the token-granular req_to_token directly. Required for page_size > 1 on the iRoPE chunked-attention path.

Two gates separate this code path from Gemma 4: (a) _init_local_attn_metadata is called only when self.attention_chunk_size is not None, and Gemma 4 sets attention_chunk_size = None; (b) the dispatcher requires layer.use_irope, which Gemma 4 layers do not set. So this cherry-pick is dormant on Gemma 4 but unblocks Llama-4-class iRoPE models on XPU. Verified harmless against the 31B baseline (all step-1 q/k/v/o cells within ULP-level run-to-run noise; see model_enablement/gemma-4-31b/REPORT_stage1_pr23757_results.md §5).

---

7. Tests

7.1 Smoke tests (test/srt/xpu/)

• test_gemma_4_e2b.py::test_simple_code_qa — basic text Q&A on E2B-it via OpenAI /v1.
• test_gemma_4_e2b.py::test_sliding_window_long_context — generates ≥500 tokens to exceed the 511-token SWA window; exercises out-of-window masking.
• test_gemma_4_e2b.py::test_sliding_window_3k_tokens — generates ~3000 tokens (≈6× the SWA window); stresses the decode kernel's local masking and KV cache page table management.
• test_gemma_4_31b.py::test_simple_code_qa — 31B XPU smoke test mirroring the e2b stencil.

All four pass against the current branch HEAD (f2c0d022f) on --attention-backend intel_xpu and --attention-backend triton.

7.2 Accuracy + perf benchmarks

Driven by model_enablement/gemma/run_accuracy_benchmark.py and run_performance_benchmark.py. Reproduce the GSM8K numbers in §4:

ZE_AFFINITY_MASK=0 \
  python3 run_accuracy_benchmark.py --variant e2b --tp 1 \
    --num-questions 200 --num-shots 5 --max-new-tokens 512 \
    --parallel 16 --mem-fraction 0.7 --max-running 32 --skip-mmlu

ZE_AFFINITY_MASK=0,1 \
  python3 run_accuracy_benchmark.py --variant e4b --tp 2 \
    --num-questions 200 --num-shots 5 --max-new-tokens 512 \
    --parallel 16 --mem-fraction 0.8 --max-running 32 --skip-mmlu

ZE_AFFINITY_MASK=0,1,2,3 \
  python3 run_accuracy_benchmark.py --variant 31b --tp 4 \
    --num-questions 200 --num-shots 5 --max-new-tokens 512 \
    --parallel 16 --mem-fraction 0.85 --max-running 32 \
    --launch-timeout 1800 --skip-mmlu

Per-op kernel profile (used in §5.3):

SGLANG_USE_SGL_XPU=1 ZE_AFFINITY_MASK=0,1,2,3 \
python -m sglang.bench_one_batch \
  --model-path google/gemma-4-31B-it \
  --attention-backend intel_xpu \
  --batch-size 4 --page-size 64 --input 1024 --output 1024 --tp 4 \
  --mem-fraction-static 0.85 --context-length 8192 \
  --disable-radix-cache --disable-overlap-schedule \
  --profile --profile-activities XPU

---

8. Checklist

• Format: pre-commit clean.
• Unit tests: 4 XPU smoke tests under test/srt/xpu/ (E2B simple Q&A, E2B SWA long-context, E2B SWA 3K tokens, 31B simple Q&A).
• Accuracy + speed benchmark results: §4 (GSM8K E2B/E4B/31B), §5.3 (decode wall-time + end-to-end tok/s on the fused QKV-norm path).
• Code style: matches xpu_backend.py / triton_backend.py conventions; no new abstractions introduced beyond what the bug fixes require.
• Documentation: not updated in this PR — the cookbook entry for Gemma 4 on XPU will follow once the MMLU-57 issue is closed.

---

9. References

• sgl-kernel-xpu PR #191 (Support prepopulated kv cache) — the upstream kernel-wrapper change this PR aligns with.
• ckvermaAI/sglang PR [Intel GPU] Fix incorrect KV-cache page table for local attention when page_size > 1 #23757 ([Intel GPU] Fix incorrect KV-cache page table for local attention when page_size > 1) — cherry-picked as f2c0d022f.
• Background analyses (in intel_workspace/):
  • model_enablement/sgl_kernel_xpu_pr191_analysis.md
  • model_enablement/sgl_kernel_xpu_pr191_gemma4_impact.md
  • model_enablement/gemma/pr191_fix_plan_REPORT.md (per-call-site analysis + post-fix results)
  • model_enablement/gemma-4-31b/REPORT_intel_xpu_vs_triton_attention.md (pre-fix XPU vs triton matrix)
  • model_enablement/gemma-4-31b/REPORT_stage1_pr23757_results.md (PR 23757 effect analysis on Gemma 4)
  • model_enablement/gemma/projection_vs_profile_{e2b,e4b}_README.md, model_enablement/gemma-4-31b/projection_vs_profile_31b_README.md (analytic vs profiled per-op breakdown)

---

10. Review and Merge Process (unchanged)

1. Ping Merge Oncalls. See the PR Merge Process.
2. Approvals from CODEOWNERS and other reviewers.
3. Trigger CI (/tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci).
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge.

---

CI States

Latest PR Test (Base): Not run yet
Latest PR Test (Extra): ⚠️ Not run on latest push -- push again to dispatch.

---

11. Kernel Fusion & FMHA Tile Optimizations (2026-05-26 — 2026-05-27)

Building on the fused QKV-RMSNorm in §5, this round adds five more kernel optimizations that collectively deliver +36% decode throughput on E2B, +20% on E4B, and +2.8% on 31B vs the pre-optimization baselines.

11.1 Summary of Changes

ID	Name	Vehicle	Impact (E2B)
A2	Residual + RMSNorm fusion (post-attention)	Triton gate relaxation	+5.4% incremental
A3	Gemma3RMSNorm → sgl_kernel on XPU	forward_xpu alias	Eliminates pow/mean/rsqrt chain
B	Fused RoPE + KV-cache write	New SYCL kernel (sgl-kernel-xpu)	−60 launches/10 steps (3 eligible layers)
B'	Fused store_cache (K+V)	New SYCL kernel (sgl-kernel-xpu)	−240 _index_put_impl_ calls
FMHA	Decode PV tile + Prefill Q-tile for head_dim=512	Template config (sgl-kernel-xpu)	+13% from host-dispatch improvement

11.2 New SYCL Kernels (in sgl-kernel-xpu branch fused-rope-kvcache)

apply_rope_inplace_with_kvcache_xpu — single kernel that:

• Rotates Q in-place (fp32 cos/sin precision)
• Rotates K in-place + writes to k_cache[slot]
• Writes V to v_cache[slot]

store_cache_xpu — single kernel replacing 2× aten::_index_put_impl_:

• Writes both K and V into the flat cache at given indices

Both pass bit-parity tests (test/srt/xpu/test_rope_kvcache_fused.py, 5/5).

11.3 Performance Results

Variant	TP	Baseline	Final	Δ
E2B	1	58.27 tok/s	79.26 tok/s	+36.0%
E4B	2	35.85 tok/s	43.05 tok/s	+20.1%
31B	4	30.62 tok/s	31.49 tok/s	+2.8%

11.4 KV-Cache Write Elimination

Variant	Baseline _index_put_impl_ / 10 steps	Final	Δ
E2B	~700	50	−92.9%
E4B	530	~50	~−90%
31B	1,250	~1,050	−16%

11.5 Accuracy Verification

GSM8K 5-shot (N=200, T=0): E2B = 0.180 (no regression vs 0.175 pre-optimization). E4B/31B pending full rerun but expected stable (kernel fusions are numerically identical to unfused paths).

11.6 Companion sgl-kernel-xpu PR

These optimizations require the fused-rope-kvcache branch on sgl-kernel-xpu which adds:

• src/sycl/Rope.cpp — fused RoPE+KVcache kernel (~120 lines)
• src/sycl/KVCache.cpp — fused store_cache kernel (~75 lines)
• src/FMHAPrefillXe20.cmake — TILED_Q=128 for head_dim=512
• src/sycl/xe_fmha_fwd_decode_kernel.cpp.in — PV inner tile=64 for head_dim≥512
• src/sycl/xe_fmha_fwd_split_decode_kernel.cpp.in — same

Full analysis: model_enablement/profiling/REPORT_kernel_optimizations_summary.md

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[kpham-sgl]

For Gemma 4 E2B and E4B, the attention backend needs to support KV cache retrieving
as well. Make sure you implement this for XPU

sglang/python/sglang/srt/layers/attention/triton_backend.py

Lines 877 to 881 in f63def8

	if k is None and v is None:
	pool = forward_batch.token_to_kv_pool
	cache_loc = forward_batch.out_cache_loc
	if isinstance(pool, SWAKVPool) and pool.layers_mapping[layer.layer_id][1]:
	cache_loc = pool.translate_loc_from_full_to_swa(cache_loc)

[airMeng]

This is a general modification, why cuda doesn't need this?

[jmunetong]

Lines 216-220 showed that it was also applied for forward_cuda. Is this correct?

[mingfeima]

i prefer that we modify fused_add_rmsnorm to make it accepts both 2D and 3D inputs, therefore we can skip:

• view - which creates new TenorImpl
• possible contiguous - memory copy that needed for the reshape.

[mingfeima]

don't call forward_cuda from xpu dispatch, this is going to be ambiguous.

call forward_native from xpu is OK.

you can copy everything from forward_cuda here and later try to reduce if-else by refatoring sycl kernel interface.

[mingfeima]

add xpu dispatch in store_cache at L107 is more decent (L106 cuda hip and xpu go the same path)

[mingfeima]

[XPU] Enable Gemma 4 E2B / E4B / 31B on Intel XPU with SWA KV pool, sgl-kernel-xpu PR#191 alignment, and fused QKV RMSNorm

1. Motivation

Enable google/gemma-4-E2B-it, google/gemma-4-E4B-it, and google/gemma-4-31B-it on Intel XPU with both the triton and intel_xpu attention backends. Gemma 4 is a hybrid model that interleaves sliding-window attention (SWA, head_dim=256) with full attention (head_dim=512). Bringing it up on XPU surfaced four cross-layer correctness gaps and one performance gap that this PR closes:
...

ai this is OK, but try to keep it simpler :) we still need human to review it.

grasping the big picture is usually more important at this era, usually the first step is enabling the workload, after that more importantly: analysis for hotspots, kernel efficiency evaluation (30% improvement not necessarily mean it is good enough), and following plans for optimization or feature enabling.