Skip to content

Add settings/validation for deterministic mode - #968

Merged
ananthsub merged 2 commits into
NVIDIA-NeMo:mainfrom
ananthsub:deterministic-mode
Oct 27, 2025
Merged

Add settings/validation for deterministic mode#968
ananthsub merged 2 commits into
NVIDIA-NeMo:mainfrom
ananthsub:deterministic-mode

Conversation

@ananthsub

Copy link
Copy Markdown
Contributor

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub
ananthsub requested a review from BoxiangW October 15, 2025 23:45
@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2025

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ananthsub

Copy link
Copy Markdown
Contributor Author

/ok to test aa5f690

@maanug-nv maanug-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had looked into some other settings affecting determinism earlier:

  • Check NVTE_ALLOW_NONDETERMINISTIC_ALGO is 0
  • Set torch.backends.cudnn.benchmark = False , according to pytorch docs

@ananthsub ananthsub added the r0.2.0 Cherry-pick label for r0.2.0 release branch label Oct 21, 2025
@ananthsub
ananthsub merged commit f003cd8 into NVIDIA-NeMo:main Oct 27, 2025
31 checks passed
@ananthsub
ananthsub deleted the deterministic-mode branch October 27, 2025 20:59
chtruong814 pushed a commit that referenced this pull request Oct 27, 2025
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
paul-gibbons pushed a commit to paul-gibbons/Megatron-Bridge that referenced this pull request Oct 29, 2025
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Paul Gibbons <pgibbons@nvidia.com>
paul-gibbons pushed a commit to paul-gibbons/Megatron-Bridge that referenced this pull request Oct 29, 2025
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Paul Gibbons <pgibbons@nvidia.com>
nv-mollys pushed a commit that referenced this pull request Oct 31, 2025
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: mollys <mollys@mollys.nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

r0.2.0 Cherry-pick label for r0.2.0 release branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants