docker(xpu): drop redundant setvars.sh from torch_memory_saver RUN - #38665
Merged
mingfeima merged 2 commits intoSep 10, 2026
Merged
Conversation
The base image intel/deep-learning-essentials:2026.0.0-devel-ubuntu24.04
already applies setvars.sh in its own layers and freezes the result into
image ENV. Every subsequent RUN therefore starts with SETVARS_COMPLETED=1,
icpx on PATH, and LIBRARY_PATH / CPATH / CMPLR_ROOT populated:
$ docker run --rm intel/deep-learning-essentials:2026.0.0-devel-ubuntu24.04 \
env | grep -E '^(PATH|LIBRARY_PATH|CMPLR_ROOT|SETVARS_COMPLETED)'
PATH=/opt/intel/oneapi/mpi/…/bin:…/compiler/2026.0/bin:…
LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:…/compiler/2026.0/lib:…
CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.0
SETVARS_COMPLETED=1
Re-sourcing setvars.sh in the RUN was therefore never necessary. It also
actively broke the build (nightly run 34330707892, follow-up to sgl-project#38617's
diagnostic fix): setvars.sh sees SETVARS_COMPLETED=1 and exits 3 without
honoring --force (dash's dot builtin dropping positional args to sourced
scripts). The exit 3 propagates out of the sourced script and kills the
parent shell before pip can run, even with '. setvars.sh || echo ...'.
Drop the setvars.sh call from the RUN. Also simplify CMD to just ['bash']
for the same reason: env is already applied, no need to re-source at
container start.
Nightly failing run: https://github.com/sgl-project/sglang/actions/runs/34330707892
arathi-hlab
requested review from
Fridge003,
HaiShaw,
ishandhanani,
ispobock,
sogalin and
yctseng0211
as code owners
September 9, 2026 09:48
mingfeima
approved these changes
Sep 10, 2026
pllimax
added a commit
to pllimax/sglang
that referenced
this pull request
Sep 10, 2026
* origin/main: (27 commits) [Simulator] Give the OFFLINE/BLOCKING comparison tolerances real headroom (sgl-project#38732) [Config] msgspec.Struct for the config tier (sgl-project#38753) [AMD] ci: move the miles nightlies from rocm700 to rocm10 (sgl-project#37495) [Config] One writer for the declaration stash; no exception to the write seal (sgl-project#38752) docker(xpu): drop redundant setvars.sh from torch_memory_saver RUN (sgl-project#38665) [XPU][Fix] Pack device-pointer tables as uint64 to avoid 64-bit address overflow (sgl-project#35051) [CI] Temporarily disable GB300 tests (sgl-project#38770) [diffusion] feat: spill large tensors over shared memory like numpy arrays (sgl-project#38656) [diffusion] refactor: refactor utility ownership and document helper placement (sgl-project#38699) [NPU]Support GLM5.2 and FP8 DSA&Indexer kvcache for 950 (sgl-project#38250) [CI] Answer unrecognized slash commands instead of skipping silently (sgl-project#38736) [AMD] Parallelize aiter spec-decode KV index building over token blocks (sgl-project#37659) [DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103 (sgl-project#30805) Add Opt-In for GLM-5.3 Flash breakable prefill CUDA graphs (sgl-project#38522) [CI] Install helion 1.4.0 for the KDA Helion kernel tests (sgl-project#38688) [Rust] Gate health on startup warmup completion (sgl-project#37994) [HiCache] Replace skip_lock_node_ids with a segment lock protocol (sgl-project#36848) feat: add optimized Domino rollout to DFlash V2 (sgl-project#36899) [CI] Add /run-full-ci and /run-extra-ci slash commands (sgl-project#38734) [Model] Support GLM-5.3 Flash NVFP4 loading (sgl-project#38621) ...
mqhc2020
pushed a commit
to mqhc2020/sglang
that referenced
this pull request
Sep 15, 2026
…gl-project#38665) Co-authored-by: arathi-hlab <arathi-hlab@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #38617. That PR turned a silent exit 3 into a visible one, which surfaced the real bug:
setvars.shshould not have been in this RUN at all. Removing it fixes the nightly and gets rid of the fragility for good.Failing nightly run (post-#38617): https://github.com/sgl-project/sglang/actions/runs/34330707892
Root cause
The base image
intel/deep-learning-essentials:2026.0.0-devel-ubuntu24.04already runssetvars.shin its own build layers and freezes the resulting env into image ENV. Every subsequent RUN starts with the SYCL toolchain fully wired up:Re-sourcing
setvars.shin the RUN was redundant. It was also actively breaking the build:setvars.shseesSETVARS_COMPLETED=1, refuses to re-run, prints its usage, and exits 3.--forcegets dropped by dash's.builtin (Docker's defaultsh -con Ubuntu 24.04), so setvars.sh never actually sees it. The exit-in-a-sourced-script kills the parent shell before pip runs, defeating even the|| echo …guard from #38617.Fix
Drop the
setvars.shinvocation from the RUN and let the base image's env do its job:RUN TMS_PLATFORM=xpu pip install --no-cache-dir --no-build-isolation \ git+https://github.com/fzyzcjy/torch_memory_saver.git@${TORCH_MEMORY_SAVER_REF}Also simplify
CMDfor the same reason — the container's ENV already has everything set, sosource setvars.sh --force && exec bashreduces to justbash.Why CI didn't catch this
.github/workflows/release-docker-intel-xpu-nightly.ymlonly fires on:There is no
pull_requesttrigger. Any PR that touchesdocker/xpu.Dockerfileorpython/pyproject_xpu.tomlcan merge without ever building the affected image — the first time the change actually runs is the next nightly, hours after merge. That's exactly the pathway #29935 → the original exit 3, and #38617's diagnostic fix → this exit 3.Recommended follow-up (kept out of this PR because updating workflow files requires the
workflowOAuth scope, which the author's token doesn't hold — a maintainer or a second PR can land it):With that in place, any PR that could break the XPU image gets a docker build in CI and can't merge with a broken Dockerfile — the same protection the CUDA image already has.
Test plan
Release Docker Images Nightly (Intel XPU)on this branch → build reachesPush intel/sglang-devand succeeds.docker run --rm <built-image> python -c "import torch_memory_saver; print(torch_memory_saver.__file__)"succeeds.CI States
Latest PR Test (Base): ✅ Run #34337535233
Latest PR Test (Extra): ✅ Run #34428340387
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.