feat: Enable simulated user for multi-turn GRPO

[jialei777]

What does this PR do ?

Add an simple example on multi-turn GRPO using ADK.

Issues

List issues that this PR closes (syntax):

Usage

uv run python examples/run_grpo_unique_numbers_w_adk.py

Training reward:

Validation acc:

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[terrykong]

Hi @jialei777 . This is a really cool example! This flew under the radar, but did you want to contribute this back into NeMo RL?

Your example is very succinct and it would probably be a great starting example for other folks interested in multi-turn. Let us know if you're interested

[jialei777]

@terrykong definitely interested in contributing. The current issue is the convocation format is incorrect during "policy_generation.generate()". I guess you guys are more familiar with that? Would you help with that? What is the best way to discuss?

[terrykong]

Awesome

@jialei777 I created an issue where we can discuss so we don't crowd your PR. Feel free to share what issues you seem to be running into

[jialei777]

@terrykong is that possible to have a meeting to discuss in detail? May be easier.

[jialei777]

hey @terrykong, I think the PR is mostly ready. But unfortunately i did not sign-off my initial commits, does that really matter for squash merge?

[terrykong]

Thanks @jialei777 for the contribution. Could you share an example run command and the train/reward curves for this in the PR description so others can try your example?

re: DCO

unfortunately the DCO check can't run on the main branch, do you mind resolving on your branch?

The DCO job gives these instructinos:

Ensure you have a local copy of your branch by checking out the pull request locally via command line.
In your local branch, run: git rebase HEAD~9 --signoff
Force push your changes to overwrite the branch: git push --force-with-lease origin jialei/simulated-user

---

Could you also add some tests to the code you added? You made some of the scripts executable, but those checks would be great to have in unit tests. Also for the environment, can you check out

RL/tests/unit/environments/test_math_environment.py

Line 173 in a607b41

def test_math_env_step_basic(math_env, basic_test_data):

and write similar tests

[terrykong]

@SahilJain314 can you review this logic?

also, I think whatever change happens here should be copied in run_multi_turn_rollout_async so they have feature parity

[jialei777]

already updated in run_sample_multi_turn_rollout, which is used in run_multi_turn_rollout_async. Please let me know if the logic is correct.

[terrykong]

Out of curiosity, how much larger does the google-adk make our images (when you include all the dependencies)?

Note that any change in pyproject.toml requires you to run uv lock to update the dependency lock file (that lock file is like a requirements.txt that pins the last known working state). A quick way to check is just building the docker image before and after and checking total image || layer size to see the delta.

At a glance, the direct dependencies of google adk are numerous so I think it would be better to add it as an "extra" under optional-dependencies like https://github.com/NVIDIA-NeMo/RL/blob/main/pyproject.toml#L51

and then adding a PY_EXECUTABLE for it and reference it in the environment:

RL/nemo_rl/distributed/virtual_cluster.py

Line 49 in a607b41

VLLM = "uv run --locked --extra vllm"

The benefit of slimming down our core dependencies when possible makes it easier/faster for certain CI jobs to run.

[jialei777]

do i also need to checkin the uv lock file?

[jialei777]

@terrykong thank you for your review. I will try to address those issue shortly.

I ran into a new issue after sync with main branch. Seems now "num_valid_samples" is bigger than batch_size (512). is that as expected, or it is a bug included recently? Here is a comparison of jobs before (red) and after (green) merge main

[jialei777]

Closing this PR and creating #732 since I messed up history with rebasing...