feat: add BaseMultiRewardVerifyResponse for shared multi-reward contract - #2050
Conversation
Define reward_components once in nemo_gym as the trainer-agnostic producer-side contract, and migrate example_tool_call_multireward to inherit it. Closes #1664 Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Making the reward_components field requiredIn the case of a MultiRewardVerifyResponse, I think we should keep the reward_components field required (non-optional). if you subclass MultiRewardVerifyResponse, you're saying this env is multi-reward so reward_components should always be there |
Not providing sum(reward_components.values() as the default rewardI don't think we should implicitly aggregate with sum. The aggregation strategy is part of the reward/task definition, and an unweighted sum isn't the common case in modern LLM post-training. Making sum the default bakes in an opinionated choice that will be wrong for many workloads. verify() in each env definition will specify how to aggregate rewards and if that's not enough, advanced trainer integrations can choose to do custom logic using the reward components. |
…d-verify-response
ananthsub
left a comment
There was a problem hiding this comment.
LGTM, but please also update the docs in fern/versions/latest/pages/api-reference/index.mdx and fern/versions/latest/pages/build-verifiers/multi-reward-verification.mdx
so that BaseMultiRewardVerifyResponse is documented for users too
…d-verify-response Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com> # Conflicts: # tests/unit_tests/test_base_resources_server.py
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
|
🌿 Preview your docs: https://nvidia-preview-feat-base-multi-reward-verify-response.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
|
Aggregate metrics (and reward profiler?) go off of top level numeric fields in BaseVerifyResponse. I think we want these updated to go into the reward_components dict when its present? @ananthsub lmk if that makes sense to you and I'll add that support in a follow on PR. |
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
|
@ananthsub just realized that if I make changes after a PR approval, it'll need approval again, so re-requesting :) |
…act (NVIDIA-NeMo#2050) Define reward_components once in nemo_gym as the trainer-agnostic producer-side contract, and migrate example_tool_call_multireward to inherit it. TODO: I'm updating docs to reflect these changes. Closes NVIDIA-NeMo#1664 --------- Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Define reward_components once in nemo_gym as the trainer-agnostic producer-side contract, and migrate example_tool_call_multireward to inherit it.
TODO: I'm updating docs to reflect these changes.
Closes #1664