Fix: perf script ddp nccl-ub

[youngeunkwon0405]

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Changelog

• Add specific line by line info of high level changes in this PR.

GitHub Actions CI

See the CI sectionin the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

• Related to # (issue)

Summary by CodeRabbit

• Performance

  • Enhanced NCCL configuration settings for distributed training optimization.

• Refactor

  • Improved internal organization of configuration management.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

The pull request refactors NCCL UB (Undefined Behavior) override handling by extracting it from the Megatron FSDP configuration function into a dedicated helper function, and updates call sites accordingly. Additionally, NCCL environment variable configuration is expanded to include NCCL_CTA_POLICY="1" alongside the existing NCCL_NVLS_ENABLE="1" setting.

Changes

Cohort / File(s)	Summary
NCCL Configuration scripts/performance/setup_experiment.py	Enhanced NCCL environment variable configuration to set both NCCL_NVLS_ENABLE="1" and NCCL_CTA_POLICY="1" when nccl_ub is enabled, replacing the previous single-variable assignment.
Override Helper Refactoring scripts/performance/utils/overrides.py	Extracted nccl_ub parameter from _set_megatron_fsdp_overrides() signature and created new _set_nccl_ub_overrides() helper function to manage NCCL UB-related configuration. Updated two call sites (set_workload_base_configs and set_user_overrides) to invoke both functions separately.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Test Results For Major Changes	⚠️ Warning	PR modifies critical NCCL settings and refactors override functions affecting distributed training performance and convergence, but contains no test results or validation evidence.	Add test results and before-and-after performance metrics from training runs with modified NCCL settings to validate no convergence or performance regression occurred.
Title check	❓ Inconclusive	The title 'Fix: perf script ddp nccl-ub' is vague and uses abbreviated terminology (ddp, nccl-ub) without clearly explaining what issue is being fixed or what changes are made.	Expand the title to be more descriptive, such as 'Fix NCCL UB configuration in performance script by separating FSDP and NCCL override handling' to clearly communicate the primary change.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix-perf-script-ddp-nccl-ub

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[youngeunkwon0405]

/ok to test 3b98db1

[youngeunkwon0405]

/ok to test acc60cf

[youngeunkwon0405]

Hi, @malay-nagda, @erhoo82, and @ko3n1g, can I ask your approval for the merge?

[erhoo82]

Can we add a comment saying that TODO: need to remove this condition upon the NCCL support