Perf Config for 1 node GB200 DSV3

[gautham-kollu]

What does this PR do ?

Perf Config for 1 node GB200 DSV3 with all the perf knobs.

This is a squashed re-submission of #3545 with only the commits authored by @gautham-kollu (merge/rebase commits from main excluded).

Changelog

• Add GB200 DSV3 perf functional test:
  • tests/functional_tests/launch_scripts/gb200/active/L0_Launch_recipes_deepseek_perf.sh
  • tests/functional_tests/test_groups/recipes/test_deepseek_recipes_pretrain_perf.py
  • tests/functional_tests/test_groups/recipes/test_deepseek_recipes_pretrain_perf_gb200.json
• Wire reuse_grad_buf_for_mxfp8_param_ag for FSDP and handle None mixed_precision in training/config.py
• Drop unused branch in training/mixed_precision.py
• Update unit tests covering the above
• Extend scripts/performance argument parser and evaluate utilities

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

Additional Information

• Supersedes Perf Config for 1 node GB200 DSV3 #3545

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[gautham-kollu]

/ok to test 4e5ffbe

[claude]

Light Code Review

Issues Found

Bugs:

1. Shebang on wrong line (L0_Launch_recipes_deepseek_perf.sh): GPU_COUNT=x4 comment is on line 1, pushing the shebang to line 2. The OS will not recognize the script as bash when executed directly. Swap lines 1 and 2.

2. Broken fallback import (scripts/performance/utils/evaluate.py:29): The relative fallback from .scripts.performance.argument_parser import _testing_args resolves to a non-existent path (scripts/performance/utils/scripts/...). This is dead code that will always fail. Either fix the path or remove the try/except and keep only the direct import.

Missing test coverage:

3. The new main() function in evaluate.py (lines 851-942) has no unit tests. It wires up argument parsing and calls calc_convergence_and_performance -- at minimum, test that it exits non-zero on failure and zero on success with mocked inputs.

Other observations:

4. 4-GPU functional test vs 2-GPU limit: The launch script requests 4 GPUs (GPU_COUNT=x4, CUDA_VISIBLE_DEVICES="0,1,2,3", expert_model_parallel_size=4). CONTRIBUTING.md states functional tests must use at most 2 GPUs. If this is an intentional exception for GB200 perf tests, please document it.

5. Copyright year: test_deepseek_recipes_pretrain_perf.py uses 2025; project guidelines require 2026.

6. Commented-out code: About 30 lines of commented-out evaluation logic at the end of test_deepseek_recipes_pretrain_perf.py. Either remove it or add a TODO explaining when it should be enabled.

7. Missing trailing newline: test_deepseek_recipes_pretrain_perf_gb200.json is missing a trailing newline.

8. Missing CI_TIMEOUT header: The launch script does not set CI_TIMEOUT=<minutes>. The CI defaults to 30 minutes -- confirm this is sufficient for 50 training iterations of DSV3 on 4 GPUs.

Suggested test cases

No perf tests impacted.

[gautham-kollu]

/ok to test 22c5840

[gautham-kollu]

/ok to test b4f75f1

[gautham-kollu]

/ok to test d71ef48

[gautham-kollu]

/ok to test 8f6cad7

[gautham-kollu]

/ok to test 9009710

[gautham-kollu]

/ok to test 528a439