docker: install transformer_engine_torch runtime deps - #1800
Merged
Conversation
#1796 switched TransformerEngine to the rolling wheels installed with --no-deps. That pins the three TE dists to exactly those wheels, which is the point, but it also drops transformer_engine_torch's own runtime requirements: einops, onnx, onnxscript, pydantic, nvdlfw-inspect. transformer_engine.pytorch imports onnxscript unconditionally on its core import path (module/__init__ -> layernorm_linear -> base -> _common -> export -> onnx_extensions), so every image built since #1796 fails at ModuleNotFoundError: No module named 'onnxscript' as soon as anything touches TE -- which is every GPU test. verify_transformer_engine.py did not catch it because it only compared versions and metadata and called find_spec('transformer_engine'), which resolves the package directory without importing it. Check that every non-TE requirement of transformer_engine_torch is actually installed, so the build fails at the layer that introduced the gap instead of shipping a green image.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
guapisolo
reviewed
Jul 26, 2026
Collaborator
There was a problem hiding this comment.
Is it possible to add this file to CI?
guapisolo
approved these changes
Jul 26, 2026
guapisolo
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. wait for CI green and e2e pass then we can merge.
Contributor
|
Looks great! should we merge it now? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#1796 switched TransformerEngine to the rolling wheels, installed with
--no-deps. Pinning the three TE dists to exactly those wheels is the point ofthat flag, but it also drops
transformer_engine_torch's own runtimerequirements:
transformer_engine.pytorchimportsonnxscriptunconditionally on its coreimport path:
so every image built since #1796 dies as soon as anything touches TE — which is
every GPU test:
Seen on #1795's
stage-c-*-gpu-h200jobs (run 30190570673), where it takes outthe run before any test body executes. It is not specific to that PR — it
reproduces on any image built from current main.
Why the build stayed green
verify_transformer_engine.pycompared versions and metadata and then calledfind_specresolves the package directory without importing it, so it succeedsagainst an installation that cannot actually be imported. The verifier passed,
the image was pushed, and the failure surfaced hours later in GPU tests.
Change
--no-depstriplet.transformer_engine_torchresolves, so a future--no-depsgap fails at thelayer that introduced it rather than shipping a green image.
Deliberately not dropping
--no-deps: letting pip resolve the triplet is whatthat flag exists to prevent.
Verification
Ran the new verifier inside a miles image that has the deps, then hid
onnxscript's dist-info to reproduce exactly the state #1796 produces: