Skip to content

docker: install transformer_engine_torch runtime deps - #1800

Merged
Zhichenzzz merged 1 commit into
mainfrom
fix-te-runtime-deps
Jul 27, 2026
Merged

docker: install transformer_engine_torch runtime deps#1800
Zhichenzzz merged 1 commit into
mainfrom
fix-te-runtime-deps

Conversation

@yueming-yuan

Copy link
Copy Markdown
Collaborator

Problem

#1796 switched TransformerEngine to the rolling wheels, installed with
--no-deps. Pinning the three TE dists to exactly those wheels is the point of
that flag, but it also drops transformer_engine_torch's own runtime
requirements:

torch>=2.1, einops, onnxscript, onnx, packaging, pydantic,
nvdlfw-inspect, transformer_engine_cu13==2.17.0

transformer_engine.pytorch imports onnxscript unconditionally on its core
import path:

transformer_engine/pytorch/__init__.py:21   -> module/__init__.py
  module/layernorm_linear.py:21 -> module/base.py:23 -> module/_common.py:15
    export.py:61 -> onnx_extensions.py:26 -> import onnxscript

so every image built since #1796 dies as soon as anything touches TE — which is
every GPU test:

File ".../transformer_engine/pytorch/onnx_extensions.py", line 26, in <module>
    import onnxscript
ModuleNotFoundError: No module named 'onnxscript'

Seen on #1795's stage-c-*-gpu-h200 jobs (run 30190570673), where it takes out
the run before any test body executes. It is not specific to that PR — it
reproduces on any image built from current main.

Why the build stayed green

verify_transformer_engine.py compared versions and metadata and then called

importlib.util.find_spec("transformer_engine")

find_spec resolves the package directory without importing it, so it succeeds
against an installation that cannot actually be imported. The verifier passed,
the image was pushed, and the failure surfaced hours later in GPU tests.

Change

  • Install the non-TE runtime deps after the pinned --no-deps triplet.
  • Have the verifier assert that every non-TE requirement of
    transformer_engine_torch resolves, so a future --no-deps gap fails at the
    layer that introduced it rather than shipping a green image.

Deliberately not dropping --no-deps: letting pip resolve the triplet is what
that flag exists to prevent.

Verification

Ran the new verifier inside a miles image that has the deps, then hid
onnxscript's dist-info to reproduce exactly the state #1796 produces:

--- deps present (expect PASS) ---
PASS
--- simulate the #1796 state: hide onnxscript ---
AssertionError: transformer_engine_torch runtime deps not installed: ['onnxscript']
--- restored ---
PASS again

#1796 switched TransformerEngine to the rolling wheels installed with
--no-deps. That pins the three TE dists to exactly those wheels, which is the
point, but it also drops transformer_engine_torch's own runtime requirements:
einops, onnx, onnxscript, pydantic, nvdlfw-inspect.

transformer_engine.pytorch imports onnxscript unconditionally on its core
import path (module/__init__ -> layernorm_linear -> base -> _common -> export
-> onnx_extensions), so every image built since #1796 fails at

    ModuleNotFoundError: No module named 'onnxscript'

as soon as anything touches TE -- which is every GPU test.

verify_transformer_engine.py did not catch it because it only compared
versions and metadata and called find_spec('transformer_engine'), which
resolves the package directory without importing it. Check that every non-TE
requirement of transformer_engine_torch is actually installed, so the build
fails at the layer that introduced the gap instead of shipping a green image.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@guapisolo guapisolo Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add this file to CI?

@guapisolo guapisolo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. wait for CI green and e2e pass then we can merge.

@Zhichenzzz

Copy link
Copy Markdown
Contributor

Looks great! should we merge it now?

@Zhichenzzz
Zhichenzzz merged commit 43d38ad into main Jul 27, 2026
40 checks passed
@Zhichenzzz
Zhichenzzz deleted the fix-te-runtime-deps branch July 27, 2026 01:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants