feat: preference datasets

[jveronvialard]

What does this PR do ?

This PR adds a more generic preference dataset class that can be used for both RM and DPO training. It also aligns the RM and DPO training implementations more closely and adds support for multiple validation preference datasets.

Usage

You can specify multiple validation preference datasets in your RM or DPO training configuration:

data:
  dataset_name: PreferenceDataset
  train_data_path: <LocalPathToTrainingDataset>
  val_data_paths:
    <NameOfValidationDataset1>: <LocalPathToValidationDataset1>
    <NameOfValidationDataset2>: <LocalPathToValidationDataset2>
    ...

For example, when using local preference datasets based on HelpSteer2 and HelpSteer3, where ties have been filtered

Comparing RM convergence plots before and after this PR, using the default data.dataset_name: HelpSteer3

Comparing DPO convergence plots before and after this PR, using the default data.dataset_name: HelpSteer3

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

[odelalleau]

Thanks! At high level lgtm, just a bunch of mostly minor questions & polish suggestions, except for one point that requires potentially more work (extending this new dataset format to DPO as well).

[odelalleau]

A few minor remarks on latest changes

[github-actions]

⚠️ File Consistency Check

Check based on commit: 52a20bc (PR #673 from jveronvialard/preference-datasets)

⚠️ DTensor Policy Worker Synchronization Warning

The file nemo_rl/models/policy/dtensor_policy_worker.py was modified in this PR, but nemo_rl/models/policy/dtensor_policy_worker_v2.py was not updated.

Why this matters:
These files contain related DTensor policy worker implementations that should be kept synchronized to ensure consistency across different versions.

Action required:

• Please review if the changes in nemo_rl/models/policy/dtensor_policy_worker.py should also be applied to nemo_rl/models/policy/dtensor_policy_worker_v2.py
• Update nemo_rl/models/policy/dtensor_policy_worker_v2.py if necessary to maintain consistency
• If the files are intentionally different, please add a comment in the PR explaining why

Files to check:

• Modified: nemo_rl/models/policy/dtensor_policy_worker.py
• Not modified: nemo_rl/models/policy/dtensor_policy_worker_v2.py

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[terrykong]

@jveronvialard and i synced up offline. came to agreement on changes, after changes + convergence metric comparison we should be good to go