fix: Fix adv estimator configs

[yfw]

What does this PR do ?

Previously there were cases where the grpo configs and adv_estimator configs did not match.

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

Summary by CodeRabbit

• Chores
  • Externalized GRPO training configuration parameters (normalize_rewards and use_leave_one_out_baseline) from hardcoded values to template variables across example and template configurations, enabling flexible runtime configuration management.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

---

📝 Walkthrough

Walkthrough

The PR replaces hardcoded boolean values in the adv_estimator configuration across five YAML configuration files with template variable references, allowing these values to be resolved at runtime from the grpo configuration namespace instead of being statically defined.

Changes

Cohort / File(s)	Summary
GRPO Configuration Updates examples/configs/grpo_math_1B.yaml, examples/configs/vlm_grpo_3B.yaml, examples/configs/vlm_grpo_3B_megatron.yaml, examples/nemo_gym/grpo_workplace_assistant_nemotron_nano_v2_9b.yaml, research/template_project/configs/grpo_math_1B.yaml	Replaced hardcoded boolean values in adv_estimator with template variable references: normalize_rewards: true → ${grpo.normalize_rewards} and use_leave_one_out_baseline: false → ${grpo.use_leave_one_out_baseline}.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• feat: add config_cli.py and refactor configs + config pre-commit #1024: Touches the same GRPO adv_estimator boolean settings (normalize_rewards and use_leave_one_out_baseline), though the retrieved PR refactors/minimizes configs while this PR parameterizes those booleans.

Suggested labels

CI:L1

Suggested reviewers

• parthchadha

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Results For Major Changes	⚠️ Warning	PR modifies algorithm hyperparameters in GRPO configs (normalize_rewards, use_leave_one_out_baseline) without test results documentation.	Include test results demonstrating no training regressions; run benchmarks on affected configs and document loss curves, convergence metrics.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix: Fix adv estimator configs' directly matches the PR's core objective to fix adv estimator configuration mismatches, clearly summarizing the main change.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch yifu/fix_configs

Tip

Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}