Repository navigation
[XPU] split Intel nightly into kernel-main and kernel-wheel jobs - #41857
Conversation
The existing nightly jobs now test sglang main + sgl-kernel-xpu main, and a
new job tests sglang main + the prebuilt kernel wheel with release coverage.
- Rename nightly-xpu-{1,2,4,8}-gpu jobs to main-nightly-xpu-{1,2,4,8}-gpu and
pull intel/sglang-dev:latest-kernel-main (published by
release-docker-intel-xpu-nightly.yml). Their suites are unchanged.
- Add release-nightly-xpu-1-gpu, which pulls intel/sglang-dev:latest and runs
the PR CI suites (stage-a-test-1-gpu-xpu + stage-b-test-1-gpu-xpu) followed
by the new release-nightly-xpu-1-gpu suite.
- Register the new suite in run_suite.py and add it to 9 1-GPU tests, which
keep their nightly-xpu-1-gpu registration.
… IST release-nightly-xpu-1-gpu runs on the 17:00 UTC cron and main-nightly-xpu-* on the 19:30 UTC cron; manual, push and workflow_call runs still run all jobs. The concurrency group now includes the cron, so one scheduled run doesn't cancel the other while it is still running.
Healthy stage-a/stage-b files finish in under ~8 min (max 487s across recent runs); hung files ran to the 30-min limit before retrying.
run_suite adds --retry-timeout-increase (600s default) to --timeout-per-file when retry is on, so 600 became 1200s; the old default was 1200+600 = 1800s, the 30-min hangs seen in CI.
sgl-project#40972 passes defer_expansion to select_decode_tokens and updated the CUDA test's _DispatchIndexer but not this XPU copy, so it raised TypeError in stage-a.
Add a kwarg-only `timeout=` to the register_*_ci markers. When set it overrides --timeout-per-file for that file, so a known-flaky hang fails quickly and the existing --enable-retry path reruns it. Across 156 recent XPU stage-a jobs, these files intermittently hang right after XPU device init with no output, burning the full 600s/1800s limit before a retry that passes in seconds. Cap each at ~3x its observed max: file normal hangs timeout xpu/test_xpu_basic.py 21-41s 7 180 kv_canary/test_self_unit_req_to_expected_...py 8-74s 3 120 kernels/ops/kvcache/test_hisparse.py 6-9s 11 120 kv_canary/test_self_unit_plan_input.py 7-11s 6 120 xpu/test_layernorm_xpu.py 7-11s 4 120 e2e/xpu/test_xpu_fp8_linear.py 8-72s 7 240 xpu/test_intel_xpu_backend.py 58-112s 7 360
Keep the hang-timeout change scoped to XPU so other vendors' CI is not touched: revert the register_*_ci `timeout=` kwarg and the per-test-file edits, and instead add a repeatable `run_suite.py --timeout-override PATH=SECONDS` (inert unless passed). pr-test-xpu.yml passes the same 7 overrides as before.
The overrides exist to cut silent hangs short; a retry that is merely slow should still get the normal --timeout-per-file budget (600s on XPU) rather than being killed by the tight first-attempt cap.
The pr-test-xpu.yml per-file cap and the --timeout-override work now live in their own PR, so this one only carries the nightly kernel-main/release split.
mingfeima
left a comment
mingfeima
left a comment
There was a problem hiding this comment.
I would avoid A/B for PR #41857. They describe kernel provenance, not test stages, so the names are opaque and easy to confuse with PR CI’s stage-a and stage-b.
Other backends use descriptive names: NVIDIA includes the nightly-test-nvidia.yml, NPU distinguishes nightly-test-npu.yml, and AMD has a nightly-test-amd.yml. The XPU image publisher already calls its variants main and wheel.
I recommend nightly-xpu-kernel-main-{1,2,4,8}-gpu and nightly-xpu-kernel-wheel-1-gpu, consistently across jobs, suites, and artifacts.
| docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir "lm-eval==0.4.9" | ||
| # Test deps of the PR CI suites (mirrors pr-test-xpu.yml, minus the sglang reinstall). | ||
| docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir pytest expecttest ray huggingface_hub "lmcache>=0.3.9" accelerate | ||
| # Diffusion extras (needed by test_xpu_zimage_turbo). | ||
| # Pins mirror python/pyproject_xpu.toml [project.optional-dependencies.diffusion]. | ||
| docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir "sglang[diffusion]" |
There was a problem hiding this comment.
nightly-B-xpu-1-gpu is supposed to test the image's sglang and the prebuilt kernel wheel. pip install "lm-eval==0.4.9" and pip install "sglang[diffusion]" can replace that torch/sglang with PyPI builds. Install the diffusion extra's dependencies directly, and keep the XPU torch index, so the image packages stay in place.
There was a problem hiding this comment.
Fixed in 141d8ba. The job now writes a constraints file pinning the image's installed torch/torchvision/torchaudio/sglang/sglang-kernel-xpu versions. It installs lm-eval, the PR CI test deps and the diffusion extra's deps with -c that file plus --extra-index-url https://download.pytorch.org/whl/xpu. sglang[diffusion] is gone: the diffusion deps are read from the checkout's pyproject_xpu.toml. If a dependency ever needs a different torch/sglang, pip now fails instead of silently replacing the image's builds.
…ages in wheel job
- Rename nightly-A-xpu-{1,2,4,8}-gpu -> nightly-xpu-kernel-main-{1,2,4,8}-gpu and
nightly-B-xpu-1-gpu -> nightly-xpu-kernel-wheel-1-gpu across jobs, suites,
test registrations and metrics artifacts, matching the image's main/wheel variants.
- In the kernel-wheel job, install test extras under a constraints file pinning the
image's torch/torchvision/torchaudio/sglang/sglang-kernel-xpu, and install the
diffusion extra's deps from pyproject_xpu.toml instead of sglang[diffusion] from
PyPI, so the job keeps testing the image's builds.
|
@mingfeima Thanks, renamed in 141d8ba: |
arathi-hlab
changed the title
mingfeima
left a comment
mingfeima
left a comment
There was a problem hiding this comment.
test/registered/xpu/test_fused_inplace_qknorm_rope_xpu.py this file is still registered to nightly-xpu-1-gpu, update it.
sgl-project#40190 landed on main registering test_fused_inplace_qknorm_rope_xpu.py to nightly-xpu-1-gpu, which this PR removes, so run_suite.py's validate_all_suites rejects it. The test imports sgl_kernel.fused_inplace_qknorm_rope, a new op not guaranteed in the released wheel, so register it to kernel-main only.
2 participants
The Intel XPU nightly now runs two setups:
nightly-xpu-kernel-main-{1,2,4,8}-gpu(renamed fromnightly-xpu-*)intel/sglang-dev:latest-kernel-mainnightly-xpu-kernel-main-{1,2,4,8}-gpu(renamed fromnightly-xpu-{1,2,4,8}-gpu)nightly-xpu-kernel-wheel-1-gpu(new)intel/sglang-dev:latestpyproject_xpu.tomlstage-a-test-1-gpu-xpu+stage-b-test-1-gpu-xpu(PR CI, 39 files), thennightly-xpu-kernel-wheel-1-gpu(9 files)Both images are published by
release-docker-intel-xpu-nightly.yml(#41650).Schedule (single
intel-bmg-nightlyrunner, so jobs run one after another):nightly-xpu-kernel-wheel-1-gpu: 22:30 IST (0 17 * * *UTC)nightly-xpu-kernel-main-*: 01:00 IST (30 19 * * *UTC)Each cron runs only its own jobs;
workflow_dispatch, push andworkflow_callruns still run all of them. The concurrency group includes the cron, so one scheduled run doesn't cancel the other.Changes
nightly-test-intel.yml:--image-tag latest-kernel-maintoxpu_ci_start_container.sh.nightly-xpu-kernel-wheel-1-gpu. It installs the PR CI test deps and the diffusion extra's deps (read frompyproject_xpu.toml) under a constraints file that pins the image's torch/sglang/kernel builds, so the image itself is tested. It runs the PR CI suites with the same 180-min budget and--enable-retry, then runsnightly-xpu-kernel-wheel-1-gpu --nightly. The second step runs even if the first fails. The job timeout is 600 min because the two suites together (~5h est.) exceed the 360-min default.check-all-jobs.nightly-xpu-kernel-wheel-1-gpu's PR CI step: pass--timeout-per-file 600 --retry-timeout-increase 0to the stage-a + stage-b run. The slowest passing file in recent runs took 487s; hung files (e.g.test_xpu_basic,kv_canary/test_self_unit_req_to_expected_token_ids_manager) ran the full 30 min before retrying. The same cap forpr-test-xpu.yml, plus fail-fast per-file overrides, is in [XPU CI] Optimize PR CI per-file timeouts and fail hanging tests fast #42048.nightly-xpu-{1,2,4,8}-gpusuites tonightly-xpu-kernel-main-{1,2,4,8}-gpu(test registrations,run_suite.py,--suiteargs, metrics artifact names). This includestest_fused_inplace_qknorm_rope_xpu(added on main by [XPU] Enable fused QK-norm + RoPE for Qwen3-MoE #40190), which goes tokernel-mainonly because it imports the newsgl_kernel.fused_inplace_qknorm_ropeop, which the released wheel may not have.xpu-ci-job-monitor.yml: point the per-model report at the renamednightly-xpu-kernel-main-*-metricsartifacts.run_suite.py: addnightly-xpu-kernel-wheel-1-gputo the XPU nightly suites.nightly-xpu-kernel-wheel-1-gpu. They keep theirnightly-xpu-kernel-main-1-gpuregistration and existingest_time:test_deepseek_ocr_triton,test_gemma_4_e2b,test_spec_eagle_parity,test_xpu_classification,test_xpu_embedding,test_xpu_rerank,test_xpu_reward,test_xpu_zimage_turbo,test_deepseek_ocr_2_olmbench.The workflow file name is unchanged, so
release-branch-cut.ymlandci_failures_analysis.py, which refer to it by file name, are unaffected.Tests
intel-bmg-nightlylabel andcancel-in-progresswarnings).run_suite.py's collector:validate_all_suitespasses.stage-a+stage-bselect 39 files (6737s est.),nightly-xpu-kernel-wheel-1-gpuselects the 9 files above (10500s est.), andnightly-xpu-kernel-main-1-gpu(renamed fromnightly-xpu-1-gpu) was unchanged at 13 before the latest main merge; it now also selectstest_fused_inplace_qknorm_rope_xpu.pre-commit run --all-files).workflow_dispatchrun will be the first real check.CI States
Latest PR Test (Base): ❌ Run #37750782757
Latest PR Test (Extra): ❌ Run #37750782172
Latest PR Test (AMD ROCm 10): ❌ Run #37750782599