Skip to content

Count DSv4 attention FLOPs in HybridModel training metrics - #7972

Draft
FDecaYed wants to merge 1 commit into
NVIDIA:mainfrom
FDecaYed:codex/dsv4-flops-main
Draft

FDecaYed wants to merge 1 commit into
NVIDIA:mainfrom
FDecaYed:codex/dsv4-flops-main

Conversation

@FDecaYed

@FDecaYed FDecaYed commented Oct 8, 2026

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

The current HybridModel FLOPs calculation omits C/H/W attention layers. Count their query/KV/output projections, window attention, compression, ratio-4 indexer and sparse attention, and ratio-128 attention. Read layer counts and MTP depth from the actual hybrid pattern, and use DSv4's head dimension rather than an uninferred MLA argument default.

Reuse main's real-token and squared-sequence-length statistics for packed batches. Packed Top-K saturation remains an estimate based on token-weighted mean sequence length; the existing statistics do not retain individual document lengths. Keep the existing forward/backward FLOPs convention and include MTP normalization/projection work.

Related to #5795. Adapted from the DSv4 accounting in #5142, #5358, and #5762. Independent of #7336 and #7469. This changes reported FLOPs, not model execution speed.

Validation

  • 48 CPU cases passed: the 40 existing calculator cases plus 8 focused DSv4 cases covering W/C/H, packed statistics, repeated MTP, and dense CSA without an indexer.
  • The CPU harness executes the production calculator and pattern parser with only unrelated package imports isolated. The new W case fails against the original calculator as expected.
  • Required autoformat checks, targeted Ruff, syntax compilation, and diff whitespace checks passed. The separately inspected training.py Pylint output has no diagnostics on changed lines; that file has existing diagnostics outside this change. Advisory mypy lacks local dependency stubs.
  • No GPU is required to validate these arithmetic changes; full training CI has not run for this draft.

Account for C/H/W attention using the actual hybrid pattern and real-token
statistics, including compressors, indexer work, grouped output projection,
and MTP overhead. Preserve the existing non-DSv4 calculations.

Related to NVIDIA#5795.
Adapted from NVIDIA#5142, NVIDIA#5358, and NVIDIA#5762.

Signed-off-by: Deyu Fu <deyuf@nvidia.com>
@FDecaYed FDecaYed added the deepseekv4 DeepSeek V4 PRs label Oct 8, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@FDecaYed

FDecaYed commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4734000

This branch was successfully deployed

2 active deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseekv4 DeepSeek V4 PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant