🚨 Separate EP logic into its own file by 3outeille · Pull Request #49679 · huggingface/transformers

3outeille · 2026-10-16T17:26:40Z

Only moving, no behaviour change.
Referencing #49429 as it needs it

What changed

New distributed/expert_parallel.py: all MoE/EP styles (moe_tp_experts, moe_identity_expert, grouped_gemm, ep_router, ep_dispatch_experts, megamoe_router, megamoe_experts) and apply_expert_parallelism. The EP styles get an ExpertParallelLayer base class with the same hooks as TensorParallelLayer.
New distributed/parallel_utils.py: the DTensor helpers shared by both files.
tensor_parallel.py keeps only the dense styles. Still one registry, ALL_PARALLEL_STYLES; expert_parallel.py registers its styles into it.
New tests/test_expert_parallel_mixin.py: ExpertParallelTesterMixin with the test_ep_* tests and its own model allowlist, moved out of the TP mixin.

Breaking changes 🚨

EP styles subclass ExpertParallelLayer, not TensorParallelLayer / MoeExpertsParallel. Hooks are unchanged.
_use_local_dtensor_params, _AllReduceForward, _AllReduceBackward moved to parallel_utils.py (private, no shim).
TensorParallelTesterMixin no longer carries the EP tests; add ExpertParallelTesterMixin to get them.

Old imports from transformers.distributed.tensor_parallel keep working with a FutureWarning until v5.23.

HuggingFaceDocBuilderDev · 2026-10-16T20:07:18Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

3outeille · 2026-10-16T23:26:35Z

+)
+
+
+def __getattr__(name: str):


not so sure about this one. Wdyt ? @ArthurZucker @vasqu

dang yea kind of a wear workaround but its for cases where you import from the file right? Imo ok to keep as its for BC but we should add a small comment to explain this!

IlyasMoutawwakil

Thanks ! very much needed ! left a couple comments, my one concern is the double file dependency which is preferably avoidable imo, it'd be much better if we draw clear lines between the diff parallelisms in terms of integration as well, the registery can be somewhere central

IlyasMoutawwakil · 2026-10-17T07:32:44Z

+# Set to None to run distributed EP tests for every model with a plan.
+# Representative MoE model types covered by distributed EP tests.
+EP_DISTRIBUTED_TEST_MODEL_TYPES = {
+    "qwen4_exp_text",
+    "qwen3_moe",
+    "glm_moe_dsa",
+    "deepseek_v4",
+    "glm4_moe_lite",
+    "glm4_moe",
+    "olmoe",
+    "qwen2_moe",
+    "cohere2_moe",
+    # VLM
+    "glm4v_moe",
+}


in general these allow lists keeps a lot of problems under the radar, i'd much more prefer making the tests cheaper an enabling them on all MoEs, and VLMs, simple lm decoders are generally fine having one representative, but moes/vlms/alms/omnis can implement some things very differently (ernie text/vision experts, DSAs, attn sinks, ..)

they are already tiny models running. I once enabled them for a lot of models but @ydshieh was complaining that it was still taking a lot of time. Maybe I can distribute it on more machines ?

OR, we can keep it this way and wait for my transformer training CI dashboard coming up next. I am planning to run small training through HF jobs to check bitwise correctness on all models. Will post internally about the idea soon

yeh i was also surprised when i ran them all locally they weren't that slow but collectively they might be, the biggest slowdown might be in the multi-process spawning, i wonder if we can use threads to simulate a multi-proc setup for example, wdyt ?

or cache it once you are on a single machine and config does not change

I still dont see why we cant run all for nightly at least? Its fine to have a small list for the fast CI but nightly should run all no?

yes yes i agree, a once per day nightly is better ! same as we do for export.
but we need a way to @slow all models except a list of models that continue to run in fast ci

agree but this belongs to another PR tho

IlyasMoutawwakil · 2026-10-17T07:47:50Z

+        return output
+
+
+if is_torch_distributed_available():


no need for guarding ig, both ALL_PARALLEL_STYLES and the layers are defined

maybe we should move ALL_PARALLEL_STYLES to something more general and let tp / ep define their own list => update the ALL_PARALLEL_STYLES based on both of theirs

atm it's still very tp focused and needs a more centralized point imo

ArthurZucker

Yep let's be careful with BC but otherwise full gassss

ArthurZucker · 2026-10-19T04:27:26Z

    parallelism=6,
 )

+expert_parallel_ci_job = CircleCIJob(


not sure that needs to be separated!

imo better to separate, it was already weird enough imo that tp and ep were used in the same context of tests

yeah I prefer separating as well

vasqu

Some smaller comments but overall much needed

vasqu · 2026-10-19T16:49:56Z

    parallelism=6,
 )

+expert_parallel_ci_job = CircleCIJob(


imo better to separate, it was already weird enough imo that tp and ep were used in the same context of tests

vasqu · 2026-10-19T17:04:56Z

+)
+
+
+def __getattr__(name: str):


dang yea kind of a wear workaround but its for cases where you import from the file right? Imo ok to keep as its for BC but we should add a small comment to explain this!

vasqu · 2026-10-19T17:08:42Z

+        return output
+
+
+if is_torch_distributed_available():


maybe we should move ALL_PARALLEL_STYLES to something more general and let tp / ep define their own list => update the ALL_PARALLEL_STYLES based on both of theirs

atm it's still very tp focused and needs a more centralized point imo

vasqu · 2026-10-19T17:10:44Z

+# Set to None to run distributed EP tests for every model with a plan.
+# Representative MoE model types covered by distributed EP tests.
+EP_DISTRIBUTED_TEST_MODEL_TYPES = {
+    "qwen4_exp_text",
+    "qwen3_moe",
+    "glm_moe_dsa",
+    "deepseek_v4",
+    "glm4_moe_lite",
+    "glm4_moe",
+    "olmoe",
+    "qwen2_moe",
+    "cohere2_moe",
+    # VLM
+    "glm4v_moe",
+}


I still dont see why we cant run all for nightly at least? Its fine to have a small list for the fast CI but nightly should run all no?

The rename of `_hf_quantized_needs_local_tp` had no user here and collides with #49679. Scalar replication goes back from `resolve_parallel_plans` into `HfQuantizer.update_parallel_plan`, as Arthur asked on #49015, so every quantizer gets it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Consolidated the registration of parallel styles into the `ParallelInterface` class for better organization. - Updated the `expert_parallel.py` and `tensor_parallel.py` files to streamline the registration process of expert parallel strategies. - Enhanced documentation for clarity regarding the usage of tensor parallel strategies and their registration. - Removed redundant code related to `MoeExpertsParallel` and `MoeIdentityParallel` classes, which are now properly registered in the new structure.

3outeille · 2026-10-23T00:36:52Z

+    ALL_PARALLEL_STYLES.register("moe_tp_experts", MoeExpertsParallel())
+    ALL_PARALLEL_STYLES.register("moe_identity_expert", MoeIdentityParallel())
+    # Every rank keeps the full param, e.g. a per-tensor quantization scale
+    ALL_PARALLEL_STYLES.register("replicate", TensorParallelLayer())


why do we need this one @IlyasMoutawwakil ?

for non shardable scales f850c82
not ep specific tho

github-actions · 2026-10-27T00:44:18Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: deepseek_v3, diffusion_gemma, glm4v_moe, granitemoehybrid, jamba, nemotron_h, olmoe, openai_privacy_filter, qwen3_omni_moe

3outeille added this pull request to stack #49680 October 16, 2026 17:55

3outeille changed the title ~~Merge branch 'main' into cleanup-ep (on top of unify-resolve-plan)~~ 🚨 Separate EP logic into its own file Oct 16, 2026

3outeille force-pushed the cleanup-ep branch 2 times, most recently from cbb8a4d to 2948f5d Compare October 16, 2026 19:56

3outeille force-pushed the cleanup-ep branch 2 times, most recently from 4193ff1 to 4481fe3 Compare October 16, 2026 21:49

3outeille marked this pull request as ready for review October 16, 2026 23:22

3outeille requested review from ArthurZucker, IlyasMoutawwakil and vasqu and removed request for ArthurZucker, IlyasMoutawwakil and vasqu October 16, 2026 23:22

3outeille commented Oct 16, 2026

View reviewed changes

3outeille force-pushed the cleanup-ep branch 2 times, most recently from 6fc8d85 to 4909c46 Compare October 17, 2026 00:06

3outeille force-pushed the unify-resolve-plan branch from a4aaabd to 3e6c08c Compare October 17, 2026 00:09

3outeille force-pushed the cleanup-ep branch from 4909c46 to 3052e9e Compare October 17, 2026 00:09

3outeille force-pushed the unify-resolve-plan branch from 3e6c08c to 8c17784 Compare October 17, 2026 00:10

3outeille force-pushed the cleanup-ep branch from 3052e9e to 78b4ade Compare October 17, 2026 00:10

3outeille force-pushed the unify-resolve-plan branch from 8c17784 to b68ebc4 Compare October 17, 2026 00:11

3outeille force-pushed the cleanup-ep branch from 78b4ade to 622ebf6 Compare October 17, 2026 00:11

3outeille force-pushed the unify-resolve-plan branch from b68ebc4 to 9ef53bb Compare October 17, 2026 00:17

3outeille force-pushed the cleanup-ep branch from 622ebf6 to 9ed3f10 Compare October 17, 2026 00:17

IlyasMoutawwakil reviewed Oct 17, 2026

View reviewed changes

ArthurZucker approved these changes Oct 19, 2026

View reviewed changes

vasqu approved these changes Oct 19, 2026

View reviewed changes

IlyasMoutawwakil mentioned this pull request Oct 21, 2026

Experts and TP: per-instance experts flags, load-time-only experts, scalars and per-expert scales in TP and EP #49775

Open

3outeille removed this pull request from stack #49680 October 23, 2026 00:00

3outeille changed the base branch from unify-resolve-plan to main October 23, 2026 00:00

3outeille added 2 commits October 23, 2026 00:20

the branch is now pointing to main and not to unify-resolve-plan

b5eb882

3outeille force-pushed the cleanup-ep branch from 70c11aa to 0de3659 Compare October 23, 2026 00:31

3outeille commented Oct 23, 2026

View reviewed changes

3outeille added 2 commits October 23, 2026 00:54

comment

bc4b6e0

some models missed the tests EP mixin

c47e5dc

3outeille mentioned this pull request Oct 23, 2026

Add tests_expert_parallel_ci job to PR CI huggingface/transformers-ci#353

Merged

3outeille and others added 2 commits October 23, 2026 01:19

fix copyright

f88b616

Merge branch 'main' into cleanup-ep

d7591e1

3outeille added this pull request to the merge queue Oct 27, 2026

3outeille removed this pull request from the merge queue due to a manual request Oct 27, 2026

3outeille enabled auto-merge October 27, 2026 01:36

ArthurZucker disabled auto-merge October 27, 2026 08:14

ArthurZucker merged commit 575b59f into main Oct 27, 2026
447 of 455 checks passed

ArthurZucker deleted the cleanup-ep branch October 27, 2026 08:15

hmellor mentioned this pull request Oct 29, 2026

[CI] Bump Transformers version to 5.22.0 vllm-project/vllm#64529

Merged

6 tasks done

Conversation

3outeille commented Oct 16, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Breaking changes 🚨

Uh oh!

HuggingFaceDocBuilderDev commented Oct 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IlyasMoutawwakil left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3outeille commented Oct 16, 2026 •

edited by github-actions Bot

Loading