[Gaurav-Shah05]

This PR NCCL_GRAPH_REGISTER=0 by default. The problem was that with registration on, Cosmos3 at SP4 with batched captures returns corrupted images (#897).

• NCCL registers buffers inside CUDA graphs only when PyTorch uses expandable_segments:True, so other runs don't change.
• With it on, the output depends on where the exchange buffers sit in memory.
• Sequence parallel: one captured all-to-all for Ulysses; drop the all-gather exchange #897's first version breaks too once its buffers move.
• With it off, everything matches main exactly.

Testing

On H100s with expandable_segments:True, GPU pairs swapped between boots:

	main	This PR
Cosmos3 SP2 t2i, 320x192	0.77 s	0.75 s	1.03x
Cosmos3 SP2 t2i, 832x480	1.13 s	1.16 s	0.98x
Qwen3.5-9B TP2, c32	3,159 tok/s	3,252 tok/s	1.03x

All within run-to-run noise, and the Cosmos3 images match main.

[mstar-review]

An earlier review run didn't complete; a later run finished: #915 (comment)

[NSagan271]

/review

[NSagan271]

LGTM, waiting for the bot review to see if there's anything non-obvious

[mstar-review]

Automated review

🟡 No findings on 59a2abf, light review

Description

• The Testing table covers Cosmos3 SP2 and Qwen3.5-9B TP2 but not Cosmos3 SP4 with batched captures, which is the configuration the PR exists to fix, so nothing in the description shows SP4 images coming out correct with this change.
• "With it off, everything matches main exactly" and "the Cosmos3 images match main" don't say what was compared (bit-exact pixels or by eye), on which prompts or seeds, or at which SP degree.
• The mechanism bullets ("the output depends on where the exchange buffers sit in memory", "Sequence parallel: one captured all-to-all for Ulysses; drop the all-gather exchange #897's first version breaks too once its buffers move") are stated as fact with no evidence such as a minimal repro, NCCL debug output or a test that fails without the change; AGENTS.md asks for an unverified mechanism to be labeled a hypothesis.
• Missing: the torch, CUDA and NCCL versions used, the command or config that reproduces the SP4 corruption, and which CPU tests were run (for example test/modular/test_platforms.py) and their results.

Questions

• Is the corruption understood as an NCCL/PyTorch problem (for example, expandable segments remapping graph-pool memory after NCCL has registered it, possibly fixed in a known NCCL version), or could turning off registration be hiding an M* buffer-lifetime bug in the SP exchange path? If it's the latter, the bug could still corrupt output on other paths and this change would only be masking it.

What the review checked

• apply_process_env_defaults uses os.environ.setdefault and is called at the top of _worker_process_target and in Conductor._launch_workers before spawn, so NCCL_GRAPH_REGISTER=0 is in place before NCCL initializes, and a value the user sets still wins.
• _built_for() picks CudaPlatform from torch.version.cuda without a device call, and ROCm builds return RocmPlatform first, so the default applies only to NVIDIA builds and RCCL is unaffected.
• No shipped config enables expandable_segments (it appears only in benchmark/cosmos3/reproduce*.sh and test/wan22/launch_server_wan22.sh), which fits the claim that default runs are unchanged.
• The new env var has a row in docs/environment_variables.rst next to the other NCCL variables, as invariant 9 requires.
• Existing test/modular/test_platforms.py tests monkeypatch process_env_defaults, so the new CUDA default doesn't break them.

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}