Add mHC support for HybridModel on dsv4 - #4568
Closed
Connor-XY wants to merge 11 commits into
Closed
Conversation
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three VPP+mHC guards in transformer_config.__post_init__, get_tensor_shapes, and forward_backward_pipelining_with_interleaving were added during defensive review iterations and ended up rejecting configurations that the existing test suite (test_pp_mhc_compatibility.py) expects to work. Drop them so the layer-count, shape-consistency, and forward-pass tests for VPP + mHC can run. Also resolves the AttributeError on ModelParallelConfig in test_schedules.py interleaved tests, since the guard reading config.enable_hyper_connections without getattr is now gone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ed schedule The previous guard removal also dropped the n-stream hidden_dim calculation in forward_backward_pipelining_with_interleaving, which was part of the original mHC PR (NVIDIA#2943). Restore it. Also remove the pp_group-None raise in get_tensor_shapes that was added defensively later — not present in the original PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/ok to test 9bced07 |
The test was added in the same defensive review pass that introduced a pp_group=None ValueError raise in get_tensor_shapes. The raise was later removed in 9bced07 to match the original PR's behavior, but this test was missed and continued to expect the raise. Drop it to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/ok to test d02e99a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related PR: #3430 adds mHC support for TransformerModel targeting main. This PR targets dsv4 and builds on that work by adding initial mHC support for HybridModel / HybridStack.
Summary
This PR stacks on the mHC TransformerLayer support port for
dsv4and adds initial mHC support forHybridModel/HybridStack.Changes include:
HyperConnectionHybridLayer, a layer-boundary mHC wrapper for HybridStack layers.enable_hyper_connections=True.Validation
python3 -m py_compile tests/unit_tests/models/test_hybrid_model.pygit diff --checktests/unit_tests/models/test_hybrid_model.py::TestHybridModel::test_dummy_hybrid_model_with_hyper_connections_forward_backward1 passed, 20 warnings in 14.65sNotes
This is intended as the follow-up HybridModel PR on top of the mHC TransformerLayer support for
dsv4. The Hybrid path currently uses a generic layer-boundary wrapper around HybridStack layers rather than a bespoke sublayer-level integration for every hybrid layer type.