Skip to content

docker(xpu): drop redundant setvars.sh from torch_memory_saver RUN - #38665

Merged
mingfeima merged 2 commits into
sgl-project:mainfrom
arathi-hlab:fix/xpu-drop-setvars-from-tms-run
Sep 10, 2026
Merged

mingfeima merged 2 commits into
sgl-project:mainfrom
arathi-hlab:fix/xpu-drop-setvars-from-tms-run

Conversation

@arathi-hlab

@arathi-hlab arathi-hlab commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #38617. That PR turned a silent exit 3 into a visible one, which surfaced the real bug: setvars.sh should not have been in this RUN at all. Removing it fixes the nightly and gets rid of the fragility for good.

Failing nightly run (post-#38617): https://github.com/sgl-project/sglang/actions/runs/34330707892

Root cause

The base image intel/deep-learning-essentials:2026.0.0-devel-ubuntu24.04 already runs setvars.sh in its own build layers and freezes the resulting env into image ENV. Every subsequent RUN starts with the SYCL toolchain fully wired up:

$ docker run --rm intel/deep-learning-essentials:2026.0.0-devel-ubuntu24.04 \
    env | grep -E '^(PATH|LIBRARY_PATH|CMPLR_ROOT|SETVARS_COMPLETED)'
PATH=/opt/intel/oneapi/mpi/…/bin:…/compiler/2026.0/bin:…
LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:…/compiler/2026.0/lib:…
CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.0
SETVARS_COMPLETED=1

Re-sourcing setvars.sh in the RUN was redundant. It was also actively breaking the build: setvars.sh sees SETVARS_COMPLETED=1, refuses to re-run, prints its usage, and exits 3. --force gets dropped by dash's . builtin (Docker's default sh -c on Ubuntu 24.04), so setvars.sh never actually sees it. The exit-in-a-sourced-script kills the parent shell before pip runs, defeating even the || echo … guard from #38617.

Fix

Drop the setvars.sh invocation from the RUN and let the base image's env do its job:

RUN TMS_PLATFORM=xpu pip install --no-cache-dir --no-build-isolation \
    git+https://github.com/fzyzcjy/torch_memory_saver.git@${TORCH_MEMORY_SAVER_REF}

Also simplify CMD for the same reason — the container's ENV already has everything set, so source setvars.sh --force && exec bash reduces to just bash.

Why CI didn't catch this

.github/workflows/release-docker-intel-xpu-nightly.yml only fires on:

on:
  workflow_dispatch:
  schedule:
    - cron: '0 12 * * *'

There is no pull_request trigger. Any PR that touches docker/xpu.Dockerfile or python/pyproject_xpu.toml can merge without ever building the affected image — the first time the change actually runs is the next nightly, hours after merge. That's exactly the pathway #29935 → the original exit 3, and #38617's diagnostic fix → this exit 3.

Recommended follow-up (kept out of this PR because updating workflow files requires the workflow OAuth scope, which the author's token doesn't hold — a maintainer or a second PR can land it):

on:
  workflow_dispatch:
  schedule:
    - cron: '0 12 * * *'
  pull_request:
    paths:
      - 'docker/xpu.Dockerfile'
      - 'python/pyproject_xpu.toml'
      - '.github/workflows/release-docker-intel-xpu-nightly.yml'

With that in place, any PR that could break the XPU image gets a docker build in CI and can't merge with a broken Dockerfile — the same protection the CUDA image already has.

Test plan

  • Trigger Release Docker Images Nightly (Intel XPU) on this branch → build reaches Push intel/sglang-dev and succeeds.
  • docker run --rm <built-image> python -c "import torch_memory_saver; print(torch_memory_saver.__file__)" succeeds.

CI States

Latest PR Test (Base): ✅ Run #34337535233
Latest PR Test (Extra): ✅ Run #34428340387
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

The base image intel/deep-learning-essentials:2026.0.0-devel-ubuntu24.04
already applies setvars.sh in its own layers and freezes the result into
image ENV. Every subsequent RUN therefore starts with SETVARS_COMPLETED=1,
icpx on PATH, and LIBRARY_PATH / CPATH / CMPLR_ROOT populated:

  $ docker run --rm intel/deep-learning-essentials:2026.0.0-devel-ubuntu24.04 \
      env | grep -E '^(PATH|LIBRARY_PATH|CMPLR_ROOT|SETVARS_COMPLETED)'
  PATH=/opt/intel/oneapi/mpi/…/bin:…/compiler/2026.0/bin:…
  LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:…/compiler/2026.0/lib:…
  CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.0
  SETVARS_COMPLETED=1

Re-sourcing setvars.sh in the RUN was therefore never necessary. It also
actively broke the build (nightly run 34330707892, follow-up to sgl-project#38617's
diagnostic fix): setvars.sh sees SETVARS_COMPLETED=1 and exits 3 without
honoring --force (dash's dot builtin dropping positional args to sourced
scripts). The exit 3 propagates out of the sourced script and kills the
parent shell before pip can run, even with '. setvars.sh || echo ...'.

Drop the setvars.sh call from the RUN. Also simplify CMD to just ['bash']
for the same reason: env is already applied, no need to re-source at
container start.

Nightly failing run: https://github.com/sgl-project/sglang/actions/runs/34330707892
@mingfeima mingfeima added intel xpu intel gpu with device `torch.xpu` labels Sep 10, 2026
@mingfeima
mingfeima merged commit 9a1b1d2 into sgl-project:main Sep 10, 2026
142 of 154 checks passed
pllimax added a commit to pllimax/sglang that referenced this pull request Sep 10, 2026
* origin/main: (27 commits)
  [Simulator] Give the OFFLINE/BLOCKING comparison tolerances real headroom (sgl-project#38732)
  [Config] msgspec.Struct for the config tier (sgl-project#38753)
  [AMD] ci: move the miles nightlies from rocm700 to rocm10 (sgl-project#37495)
  [Config] One writer for the declaration stash; no exception to the write seal (sgl-project#38752)
  docker(xpu): drop redundant setvars.sh from torch_memory_saver RUN (sgl-project#38665)
  [XPU][Fix] Pack device-pointer tables as uint64 to avoid 64-bit address overflow (sgl-project#35051)
  [CI] Temporarily disable GB300 tests (sgl-project#38770)
  [diffusion] feat: spill large tensors over shared memory like numpy arrays (sgl-project#38656)
  [diffusion] refactor: refactor utility ownership and document helper placement (sgl-project#38699)
  [NPU]Support GLM5.2 and FP8 DSA&Indexer kvcache for 950 (sgl-project#38250)
  [CI] Answer unrecognized slash commands instead of skipping silently (sgl-project#38736)
  [AMD] Parallelize aiter spec-decode KV index building over token blocks (sgl-project#37659)
  [DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103 (sgl-project#30805)
  Add Opt-In for GLM-5.3 Flash breakable prefill CUDA graphs (sgl-project#38522)
  [CI] Install helion 1.4.0 for the KDA Helion kernel tests (sgl-project#38688)
  [Rust] Gate health on startup warmup completion (sgl-project#37994)
  [HiCache] Replace skip_lock_node_ids with a segment lock protocol (sgl-project#36848)
  feat: add optimized Domino rollout to DFlash V2 (sgl-project#36899)
  [CI] Add /run-full-ci and /run-extra-ci slash commands (sgl-project#38734)
  [Model] Support GLM-5.3 Flash NVFP4 loading (sgl-project#38621)
  ...
mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Sep 15, 2026
…gl-project#38665)

Co-authored-by: arathi-hlab <arathi-hlab@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

intel run-ci xpu intel gpu with device `torch.xpu`

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants