feat(moe): add hash-based token routing

[FDecaYed]

What

Add model-neutral Hash MoE routing:

• persistent token-to-expert lookup tables for the leading MoE layers;
• softmax, sigmoid, and sqrt-softplus combination scores;
• forced benchmark routing override and inference fallback;
• input_ids plumbing through TransformerBlock, TransformerLayer, recompute, partial CUDA graphs, and MoELayer;
• validation for vocabulary size, PP placement, MTP exclusion, and unsupported EP overlap;
• focused router and real TransformerLayer/MoE coverage.

This is an independent root. It intentionally stops before HybridModel/HybridStack, training CLI, attention, Clamp/SwiGLU, mHC, and recipe wiring.

Stack

• feat(hybrid): pass token ids to hash-routed layers #5945 passes token IDs through HybridModel to hash-routed TransformerLayer instances.
• A separate small training/config bridge supplies the padded vocabulary size without adding a duplicate CLI.

Provenance

• Reconstructs the Hash routing portion of [dev] [DeepSeek-v4] Part 2: Hash MoE and SwiGLU clamp #4481.
• Includes the forced-routing correction from [dev] [DeepSeek-v4] Add ClampedSwiGLU to MoE mlp_op_fuser and add force balance to hash routing #5130.
• Refactors the same work captured in [Tracking] DeepSeek-v4 HybridModel merge plan #5795 (commits 156bb6c and 2194442).
• Original implementation by the [dev] [DeepSeek-v4] Part 2: Hash MoE and SwiGLU clamp #4481/[dev] [DeepSeek-v4] Add ClampedSwiGLU to MoE mlp_op_fuser and add force balance to hash routing #5130 authors; DSv4 adaptation by @hxbai.
• Reconstructed against current main; feature cutoff: 2026-07-21.
• GPTModel integration is intentionally excluded.

Testing

• isort --check-only, ruff, Black --check, Python compilation, and git diff --check passed for the slice.
• The 8-rank focused pytest was attempted locally; collection is blocked on this macOS host by missing Triton/CUDA (CUDA device count is zero), not by a test assertion.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[FDecaYed]

/ok to test da1b6f4

[FDecaYed]

Closing this draft until Round 2. Its logical successor is P4: complete HybridModel-only Hash MoE, rebuilt after P2 and absorbing #5945. GPTModel/TransformerBlock-only support is intentionally excluded. Tracking PR: #5795.