[Tracking] DeepSeek-v4 HybridModel merge plan

[hxbai]

DeepSeek-v4 HybridModel merge plan

Summary

This PR is the merge guide for the DSv4 feature set frozen at the current
development cutoff. The implementation has been reconstructed on frozen
main bb5647a9bdd044af85ca323b1aef5ede02e96a37 as reviewable functional
units; historical PRs are behavioral provenance, not a commit sequence to
replay. This recut will not continuously refresh from main.

The final supported path is HybridModel only. This effort does not continuously
absorb later development-branch features.

Reusable prerequisites remain separate from the nine DSv4-specific children.
We will publish by dependency rounds: a child is opened only after its direct
bases merge, and is rebased onto current main so its GitHub diff contains
only that slice. We are not using stacked PRs for parallel review.

Existing external prerequisites

• [Split 1/N of #3430] feat(mHC): basic pytorch implementation of manifold hyper connection #4531 — reviewed mHC and checkpoint-without-output primitives
  (current-CUDA-graph API follow-up: fix(mhc): update CUDA graph module API #5994)
• fix(mtp): use padded cu_seqlens in MTP roll for THD with CP #4495 — padded THD boundaries for MTP
• [main] add thd sequence packing dispatcher support for main #5008 — generic HybridEP variable-token padding primitive
• Support PackedSeqParams in TE CUDA graphs #5672 — generic PackedSeqParams flatten/rebuild for TE CUDA graphs

Fresh reusable prerequisites

• P1 — selective-FP32 runtime and native-FP32 optimizer checkpoint lifecycle (Selective-FP32 runtime and native-FP32 optimizer checkpoint lifecycle #5929)
• P2 — fused/FP32 mHC with complete HybridModel and MTP integration ([PR TBD])
• P3 — mHC pipeline and virtual-pipeline transport ([PR TBD])
• P4 — complete HybridModel Hash MoE ([PR TBD])
• P5 — complete clamped SwiGLU across expert paths (Complete clamped SwiGLU across expert paths #5940)
• P6 — explicit safe MLA RoPE primitives and consumers (Explicit safe MLA RoPE primitives and consumers #5944)
• P7 — fixed dp-balanced THD packing scheduler (Fixed dp-balanced THD packing scheduler #5993)
• P8 — Hybrid-only raw-text and mock variable-length dataset ([PR TBD])
• P9 — contiguous-CP packed MTP roll ([PR TBD])
• P10 — variable-token HybridEP correctness ([PR TBD])
• P11 — static-shape THD Transformer Engine CUDA graphs ([PR TBD])

DSv4-specific series

• D1 — unfused SBHD compressed sparse attention ([PR TBD])
• D2 — SBHD DSv4 orchestration and indexer-loss scaling ([PR TBD])
• D3 — fused SBHD CSA/DSA backend ([PR TBD])
• D4 — HybridModel DSv4 symbols, specs, and construction ([PR TBD])
• D5 — packed THD CSA/DSA algorithms ([PR TBD])
• D6 — packed THD DSv4 attention orchestration ([PR TBD])
• D7 — static contiguous-CP layout and boundary primitives ([PR TBD])
• D8 — static-CP DSv4 attention integration ([PR TBD])
• D9 — Hybrid-only FLOP/MoE accounting, functional E2E, and GB200 64K recipe ([PR TBD]; H100 golden required before publication)

Dependency order

Reusable prerequisite edges:
P1 -> P2 -> {P3, P4}, P7 -> P8,
{#5008, P4, P7} -> P10, and {#5672, P4, P8} -> P11.
P9 depends on #4495. P4 and P10 are logically distinct features despite their
router-code dependency.

DSv4 edges: {P1, P6} -> D1, {D1, P6} -> D2,
D2 -> D3, {D2, P2} -> D4, {D3, P6} -> D5,
{D2, D5} -> D6, D5 -> D7, {D6, D7, P9, P11} -> D8,
and {D4, D8, all recipe prerequisites} -> D9.

The braces indicate work that may be reviewed in the same publication round
after its shared base has merged; they are not stacked review branches.

Detached work

The following work is independent of this frozen DSv4 merge and is linked for
provenance only:

Item	Relationship
#4452	Generic Hybrid MLA plus/QK-norm support; not consumed by the frozen recipe
#5412	Legacy fused-MLA query-alias compatibility gate; not a dependency
#5931 / #5485	Nested full-iteration CUDA-graph buffers; separate execution mode
#5933 + #5934 / #2987	Optimizer-state offload; pursue as one independent end-to-end PR if desired

Explicit non-goals

• GPTModel support
• Dynamic context parallelism
• SFT-specific packing extensions
• Optimizer-state offload
• Full-iteration CUDA-graph support
• Features developed after the frozen cutoff

Original authors and contributors remain credited in each reconstructed PR
through links to the relevant source PRs.

Each published slice is recreated as one signed, signed-off commit on its
publication base. Round-0 links above are live; later links are added only
after their direct dependencies merge.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.