fix(diffusion): resolve flux nightly CI failures - #2529
Merged
Conversation
Two startup failures in the diffusion nightly functional tests: - flux_t2i_flow_lora: the CI launcher unconditionally passed --fsdp.dp_size, which injects an 'fsdp' section and conflicts with the recipe's 'ddp' section (mutual-exclusion ValueError). The launcher now skips the override for DDP-based recipes. - flux_t2i_flow: _build_diffusion_parallel_manager_args called dict() on ConfigNode sections, which are not iterable. Normalize fsdp/ddp sections via to_dict() before use; this also fixes the identical latent bug in the DDP branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com>
Contributor
Author
|
/ok to test 084840c |
akoumpa
added
the
r0.5.0
akoumpa
approved these changes
Jun 12, 2026
thomasdhc
approved these changes
Jun 15, 2026
akoumpa
pushed a commit
that referenced
this pull request
Jun 15, 2026
…0.5.0` (#2567) fix(diffusion): resolve flux nightly CI failures (#2529) Two startup failures in the diffusion nightly functional tests: - flux_t2i_flow_lora: the CI launcher unconditionally passed --fsdp.dp_size, which injects an 'fsdp' section and conflicts with the recipe's 'ddp' section (mutual-exclusion ValueError). The launcher now skips the override for DDP-based recipes. - flux_t2i_flow: _build_diffusion_parallel_manager_args called dict() on ConfigNode sections, which are not iterable. Normalize fsdp/ddp sections via to_dict() before use; this also fixes the identical latent bug in the DDP branch. Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Pranav Thombre <pthombre@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Simar-malhotra09
pushed a commit
to Simar-malhotra09/Automodel
that referenced
this pull request
Jun 16, 2026
Two startup failures in the diffusion nightly functional tests: - flux_t2i_flow_lora: the CI launcher unconditionally passed --fsdp.dp_size, which injects an 'fsdp' section and conflicts with the recipe's 'ddp' section (mutual-exclusion ValueError). The launcher now skips the override for DDP-based recipes. - flux_t2i_flow: _build_diffusion_parallel_manager_args called dict() on ConfigNode sections, which are not iterable. Normalize fsdp/ddp sections via to_dict() before use; this also fixes the identical latent bug in the DDP branch. Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Fixes two startup failures in the diffusion nightly functional tests (
flux_t2i_flow_loraandflux_t2i_flow).Changelog
tests/ci_tests/scripts/diffusion_finetune_launcher.sh: the launcher unconditionally passed--fsdp.dp_sizeto every recipe. Since perf(diffusion): improve Flux training throughput #2251 switched the Flux LoRA config to addp:section, that CLI override injected a conflictingfsdpsection and the recipe raisedValueError: Cannot specify both 'fsdp' and 'ddp' configurations. The launcher now greps the recipe YAML for a top-levelddp:section and skips the override for DDP-based recipes (DDP replicates across all ranks and needs no dp_size). This also pre-empts the same failure forqwen_image_t2i_flow_loraif it is added to the nightly list.nemo_automodel/recipes/diffusion/train.py:_build_diffusion_parallel_manager_args(introduced in feat: make mesh accept meshcontext #2266) calleddict()on thefsdp/ddpconfig sections, but the recipe passesConfigNodeobjects which support.to_dict()but notdict()conversion, crashingflux_t2i_flowwithTypeError: 'ConfigNode' object is not iterable. The sections are now normalized via.to_dict()at the top of the function, fixing both the FSDP branch and the identical latent bug in the DDP branch.tests/unit_tests/recipes/test_diffusion_train_metrics.py: added unit tests that passConfigNodesections to_build_diffusion_parallel_manager_argsthe way the recipe does (the existing tests only covered plain dicts, which is why the regression slipped through to the nightly).Before your PR is "Ready for review"
Pre checks:
Additional Information
flux_t2i_flow_lora(Slurm job 5422723) andflux_t2i_flow(Slurm job 5422718) on pipeline 54319513.🤖 Generated with Claude Code