Skip to content

test: remove HF reference suites and organize native model coverage - #3527

Draft
S1ro1 wants to merge 15 commits into
mainfrom
chore/native-model-test-layout
Draft

S1ro1 wants to merge 15 commits into
mainfrom
chore/native-model-test-layout

Conversation

@S1ro1

@S1ro1 S1ro1 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #3429 (feat/qwen3-5-native). Remove HF reference-model test scaffolding and organize retained model tests by family so native behavioral coverage can be expanded consistently.

Created and tracked with gh stack. The PR is based directly on Qwen3.5. GitHub stack-group linking is separate: the existing group already contains #3419, #3429, and #3434, and the CLI only permits appending above #3434. Its base was not substituted for Qwen3.5, and the existing PR bases were left unchanged.

Exactly two commits:

  1. Remove HF model reference coverage. Delete the Llama, GLM4-MoE, Qwen3-MoE, and skipped AFMoE parity cases, AFMoE's vendored HF config/model, three HF-only LM-head integration cases, and the HF-backed performance-counter test. Preserve Qwen3-MoE router replay using Prime's public model factory and automatic attention resolver. Keep checkpoint-format conversion tests.
  2. Organize retained tests by model. Move family-specific tests under tests/unit/train/models/<model_name>/, splitting the generic Qwen3/GLM4/Llama file into those families. Keep shared fusion, checkpointing, MoE, and multi-family conversion tests shared. Qwen3.5 already has the target layout.

No production changes. DeepSeek V4 and its associated kernel tests are unchanged. Existing non-HF test limitations, including skipped checks and weak replay assertions, are retained for the coverage follow-up. Model families whose only tests were deleted do not get empty test files.

Proposed native coverage

This is the proposed follow-up, not coverage implemented by this cleanup. Use small local architecture configs and public Prime model construction with the normal attention resolver. No HF forward reference models, copied HF initialization, or per-model FlashAttention backend matrices.

The common model suite should cover:

  • Native initialization, including architecture-specific norm/buffer values where meaningful.
  • Forward and backward with finite outputs and gradients through the paths exercised. Do not require every routed expert to receive a nonzero gradient from a small random batch.
  • Packed versus separate documents, including unequal lengths; compare outputs and, where practical, accumulated gradients. Use identical valid expert selections when isolating packing from routing.
  • Checkpoint conversion with explicit expected names/layouts and tensor values, plus roundtrip checks. These can generally run on CPU and do not need a reference model.
Family Proposed shared coverage and model-specific checks
Llama Native forward/backward and packing; checkpoint layout; share the fused/vanilla LM-head behavioral check rather than HF model wrappers.
Qwen3 Same native baseline using a local config; Q/K norm initialization and checkpoint layout. Replace Hub-backed shape-only and permanently skipped packing checks.
Qwen3-MoE Same baseline, including mixed dense/MoE layers and checkpoint expert conversion. Strengthen replay to prove the supplied expert selections are used.
GLM4-MoE Same baseline; initial dense layers, shared expert contribution, routing selection bias, partial rotary positions, and expert conversion.
AFMoE Restore running native baseline coverage; packed sliding/full attention, shared experts, and routing bias/scale behavior. Reuse retained conversion coverage.
GPT-OSS Add a full-model forward/backward and packing baseline. Retain learned-sink/sliding-window attention checks and checkpoint conversion; run ring/Ulysses numerical checks in a real two-rank job.
Nemotron-H Parameterize native model checks over latent/non-latent MoE and representative hybrid layers. Retain Mamba boundary output/gradient isolation and checkpoint conversion. Replace the model-heavy KL file with appropriate shared loss/head tensor checks; add real distributed checks only where supported.
Laguna Native baseline covering sliding/full attention and shared/routed experts; retain checkpoint conversion and selection-bias preservation coverage.
MiniMax M2 Native baseline and replay behavior; retain expert conversion and verify routing-bias state survives loading.
GLM-MoE-DSA Native baseline at supported sparse-attention shapes, document isolation, and index reuse where configured. Strengthen existing vLLM weight conversion with expected tensor values and quantization checks.
Qwen3.5 Keep the shared dense/MoE baseline; parameterize VLM behavior over dense and MoE, strengthen replay/vision-gradient assertions, and exercise packed image forwards as well as CPU MRoPE positions. Keep compiled image+CP backward explicitly unresolved until verified.

Shared follow-up work:

  • Replace the removed performance test with deterministic local-config arithmetic checks.
  • Keep tensor-level fused-head, frozen-weight, partial-chunk, dispatcher, fusion, and checkpoint-recomputation tests.
  • Move genuinely model-independent KL checks out of Nemotron; remove tests that only restate arithmetic or assume random perturbation monotonically increases KL.
  • Add an explicit two-rank test invocation: plain pytest -m gpu skips the current distributed model checks.
  • Keep renderer/serialization tests separate from model E2E validation. Use math for actual trainer/inference KL runs.

Transformers remains a production dependency of the current Prime factory/base/configs. This PR removes HF model-reference testing; it does not claim to remove that dependency.

Validation

  • Full unit-suite collection: 731 tests collected successfully.
  • All 90 existing model test function bodies and decorators are unchanged by the relocation commit (AST comparison).
  • Retained model and shared LM-head suite on H200: 74 passed, 7 skipped, Slurm 409, exit 0. Skips: GPT-OSS two-rank CP (2), Qwen3.5 two-rank CP (1), existing disabled Qwen3 packing (2), gated Llama (1), and full-model fused-head comparison (1). DeepSeek V4 and associated kernel tests were excluded.
  • Ruff check and formatting pass.
  • DeepSeek V4 and associated kernel files have no diff against the parent branch.

Stack created with GitHub Stacks CLIGive Feedback 💬

@S1ro1 S1ro1 changed the title chore/native model test layout test: remove HF reference suites and organize native model coverage Sep 10, 2026
Base automatically changed from feat/qwen3-5-native to main September 12, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant