chore(customizer): restore DPO and update DPO config - #943
Merged
Conversation
anubhutivyas
force-pushed
the
aalgo-414-rl-image/anubhutiv
branch
2 times, most recently
from
July 29, 2026 21:58
97edbf0 to
51db768
Compare
anubhutivyas
marked this pull request as ready for review
July 30, 2026 06:34
anubhutivyas
force-pushed
the
aalgo-414-rl-image/anubhutiv
branch
from
July 30, 2026 19:09
08ae0fe to
be54770
Compare
anubhutivyas
force-pushed
the
anubhutiv/restore-dpo
branch
from
July 30, 2026 22:47
bba9a64 to
fe37f44
Compare
Contributor
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds the NeMo-RL DPO backend, RL customization API contracts, Kubernetes/Ray routing guidance, typed driver configuration, dependency updates, and an end-to-end HelpSteer3 tutorial with deployment and inference evaluation. ChangesNeMo-RL DPO integration
Sequence Diagram(s)sequenceDiagram
participant Notebook
participant Platform
participant RLBackend
participant RayWorkers
participant DeploymentGateway
Notebook->>Platform: Upload preference files and register base model
Notebook->>RLBackend: Submit RlJobInput
RLBackend->>RayWorkers: Run DPO training
Notebook->>RLBackend: Poll status and metrics
Notebook->>Platform: Create output-model deployment
Platform->>DeploymentGateway: Start deployment and gateway
Notebook->>DeploymentGateway: Send chat-completion request
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
plugins/nemo-customizer/README.md (1)
3-3: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftAdd the required documentation framing to these Markdown pages.
Each page starts directly with operational/reference content and lacks top-level prerequisites and a final Next Steps section with cross-links.
plugins/nemo-customizer/README.mdLine 3: add Prerequisites before the overview and Next Steps at the end.plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.mdLines 3-3: add both sections.plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.mdLine 58: add both sections.plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.mdLine 3: add both sections.plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.mdLine 3: add both sections.plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.mdLine 3: add both sections.As per coding guidelines: Markdown pages must list prerequisites at the top and include a Next Steps section at the end.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-customizer/README.md` at line 3, Add a top-level Prerequisites section before the existing content and a final Next Steps section with relevant cross-links in each affected page: plugins/nemo-customizer/README.md (line 3), plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md (line 3), hf-conversion.md (line 58), hyperparameters-rl.md (line 3), hyperparameters.md (line 3), and integrations-setup.md (line 3). Preserve the existing operational and reference content while ensuring every page follows the required Markdown structure.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md`:
- Line 72: Add the required blank line immediately before the “NeMo-RL (DPO) —
preference data” heading in dataset-formats.md so the heading is separated from
the preceding content and satisfies markdownlint MD022.
In `@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md`:
- Line 144: Make the max_steps/epochs guidance consistent: in
plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md
lines 144-144 and 170-170, replace the blanket prohibition with wording that
permits the combination only for explicit bounded smoke runs and requires
omitting max_steps in production; update
plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.md
lines 45-45 to preserve the same semantics.
In `@services/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.py`:
- Line 132: Update the hyperparameter logging call in the DPO driver to pass an
explicit allowlisted or redacted dictionary instead of config.model_dump().
Exclude W&B/MLflow tokens, tracking URIs, and other sensitive resolved
configuration fields while retaining only safe training hyperparameters.
---
Nitpick comments:
In `@plugins/nemo-customizer/README.md`:
- Line 3: Add a top-level Prerequisites section before the existing content and
a final Next Steps section with relevant cross-links in each affected page:
plugins/nemo-customizer/README.md (line 3),
plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md
(line 3), hf-conversion.md (line 58), hyperparameters-rl.md (line 3),
hyperparameters.md (line 3), and integrations-setup.md (line 3). Preserve the
existing operational and reference content while ensuring every page follows the
required Markdown structure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: cac4f9bf-b2df-4f15-9334-f64eef98674c
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (23)
docs/customizer/tutorials/dpo-customization-job.ipynbdocs/customizer/tutorials/dpo-customization-job.mdxdocs/fern/components/NotebookViewer.tsxdocs/fern/components/notebooks/dpo-customization-job.jsondocs/fern/components/notebooks/dpo-customization-job.tsdocs/fern/scripts/validate-notebook-viewer.mjsdocs/troubleshooting/customizer.mdxplugins/nemo-customizer/README.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/rl-kubernetes-runtime.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.jsonplugins/nemo-rl/pyproject.tomlpyproject.tomlpytest.iniservices/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.pythird_party/requirements-main.txt
💤 Files with no reviewable changes (1)
- pytest.ini
Contributor
|
albcui
approved these changes
Jul 31, 2026
albcui
left a comment
albcui
left a comment
Contributor
There was a problem hiding this comment.
make sure to also run make refresh-openapi to regenerate the combined Customizer OpenAPI spec.
anubhutivyas
force-pushed
the
anubhutiv/restore-dpo
branch
from
August 3, 2026 17:45
fe37f44 to
aaeab12
Compare
Signed-off-by: anubhutiv <anubhutiv@nvidia.com>
anubhutivyas
force-pushed
the
anubhutiv/restore-dpo
branch
from
August 3, 2026 17:50
aaeab12 to
19e9826
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md`:
- Line 57: Update the DPO input documentation to distinguish the canonical
{prompt, chosen, rejected} triple as one supported format rather than the only
format. In
plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md at
lines 57-57, 164-166, 201-204, and 334-337, state that RL accepts all schemas
documented in the reference and remove triple-only workflow language. In
plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md
at lines 74-76, identify BinaryPreference as one supported option and link the
shared contract; also allow the documented raw HelpSteer3 and Tulu3 preference
formats.
- Line 168: Update the job-tracking instructions in the Nemo customizer skill to
require capturing the unique identifier returned by rl submit or using a unique
server-side correlation; remove the guidance to select the newest rl-* job from
nemo jobs list. If submission provides no unique identifier, instruct the
workflow to stop rather than polling a potentially unrelated job.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 6a659291-60fc-434e-aefd-5e34f838af25
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (23)
docs/customizer/tutorials/dpo-customization-job.ipynbdocs/customizer/tutorials/dpo-customization-job.mdxdocs/fern/components/NotebookViewer.tsxdocs/fern/components/notebooks/dpo-customization-job.jsondocs/fern/components/notebooks/dpo-customization-job.tsdocs/fern/scripts/validate-notebook-viewer.mjsdocs/troubleshooting/customizer.mdxplugins/nemo-customizer/README.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/rl-kubernetes-runtime.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.jsonplugins/nemo-rl/pyproject.tomlpyproject.tomlpytest.iniservices/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.pythird_party/requirements-main.txt
💤 Files with no reviewable changes (1)
- pytest.ini
🚧 Files skipped from review as they are similar to previous changes (19)
- docs/fern/scripts/validate-notebook-viewer.mjs
- docs/customizer/tutorials/dpo-customization-job.mdx
- docs/troubleshooting/customizer.mdx
- docs/fern/components/NotebookViewer.tsx
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.md
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.md
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.md
- plugins/nemo-customizer/README.md
- pyproject.toml
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.md
- plugins/nemo-rl/pyproject.toml
- docs/fern/components/notebooks/dpo-customization-job.json
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.md
- docs/fern/components/notebooks/dpo-customization-job.ts
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.json
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/rl-kubernetes-runtime.md
- services/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.py
- docs/customizer/tutorials/dpo-customization-job.ipynb
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.md
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (2)
plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md (2)
168-168:⚠️ Potential issue | 🟠 MajorCapture the submitted RL job ID.
Selecting the newest
rl-*job fromnemo jobs listdoes not correlate the listing with this submission. Concurrent jobs or stale listings can make the workflow poll another job and report the wrong result. Use the ID returned byrl submitor a server-side correlation. Stop if no unique ID is available.Also applies to: 206-207, 368-374, 451-451
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md` at line 168, Update the RL submission workflow and its references around rl submit and poll_customization_job.sh to capture the job ID returned by rl submit, rather than selecting the newest rl-* entry from nemo jobs list. Require a unique submitted job ID, stop with an error when it is unavailable, and use that captured ID for polling and result reporting.
51-57:⚠️ Potential issue | 🟠 MajorUse the supported preference schemas in the RL instructions.
dataset-formats.mdsupports BinaryPreference, raw HelpSteer3, and Tulu3. These lines still define RL input as{prompt, chosen, rejected}. This can make the skill reject valid raw datasets. Describe the triple as one supported format and link the reference at each workflow entry.Also applies to: 115-115, 164-166, 203-204, 336-337
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md` around lines 51 - 57, Update the RL dataset guidance in SKILL.md so `{prompt, chosen, rejected}` is presented as one supported preference format rather than the sole schema. At each RL workflow entry, including the overview and referenced sections around lines 115, 164–166, 203–204, and 336–337, link to dataset-formats.md and mention BinaryPreference, raw HelpSteer3, and Tulu3 as supported inputs.
🧹 Nitpick comments (1)
plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md (1)
332-377: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftKeep the executable skill concise.
Move the full RL command walkthrough, JSON examples, defaults, and worked example to a linked reference. Keep routing, prerequisites, and a short checklist in
SKILL.md. This reduces prompt size and prevents duplicated instructions from drifting.Based on learnings,
SKILL.mdis an agent skill execution definition, not a rendered documentation page; detailed reference material belongs in linked files.Also applies to: 409-451
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md` around lines 332 - 377, Reduce the RL sections in SKILL.md, including the DPO fast path and the content around the additionally referenced lines, to routing, prerequisites, and a concise execution checklist. Move the full command walkthroughs, JSON payloads, defaults, polling details, and worked examples into an appropriate linked reference file, then link to it from SKILL.md while preserving links to existing hyperparameter and runtime references.Source: Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md`:
- Line 3: Add a short Prerequisites section before the dataset format details
and a Next Steps section after the notes in the dataset-formats documentation.
Include links to the related conversion, hyperparameter, and evaluation
references, preserving the existing backend-specific format guidance.
In `@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md`:
- Around line 201-204: The common upload checklist in SKILL.md must not require
train.jsonl for RL workflows. Update the checklist to make the filename
backend-specific or explicitly exempt the RL branch, ensuring RL uploads use
training.jsonl and validation.jsonl in the single fileset while preserving the
existing requirements for other workflows.
---
Duplicate comments:
In `@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md`:
- Line 168: Update the RL submission workflow and its references around rl
submit and poll_customization_job.sh to capture the job ID returned by rl
submit, rather than selecting the newest rl-* entry from nemo jobs list. Require
a unique submitted job ID, stop with an error when it is unavailable, and use
that captured ID for polling and result reporting.
- Around line 51-57: Update the RL dataset guidance in SKILL.md so `{prompt,
chosen, rejected}` is presented as one supported preference format rather than
the sole schema. At each RL workflow entry, including the overview and
referenced sections around lines 115, 164–166, 203–204, and 336–337, link to
dataset-formats.md and mention BinaryPreference, raw HelpSteer3, and Tulu3 as
supported inputs.
---
Nitpick comments:
In `@plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md`:
- Around line 332-377: Reduce the RL sections in SKILL.md, including the DPO
fast path and the content around the additionally referenced lines, to routing,
prerequisites, and a concise execution checklist. Move the full command
walkthroughs, JSON payloads, defaults, polling details, and worked examples into
an appropriate linked reference file, then link to it from SKILL.md while
preserving links to existing hyperparameter and runtime references.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 660b1550-e94b-4a75-8b0f-2dad147de67e
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (24)
docs/customizer/tutorials/dpo-customization-job.ipynbdocs/customizer/tutorials/dpo-customization-job.mdxdocs/fern/components/NotebookViewer.tsxdocs/fern/components/notebooks/dpo-customization-job.jsondocs/fern/components/notebooks/dpo-customization-job.tsdocs/fern/scripts/validate-notebook-viewer.mjsdocs/troubleshooting/customizer.mdxplugins/nemo-customizer/README.mdplugins/nemo-customizer/openapi/openapi.yamlplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/rl-kubernetes-runtime.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.mdplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.jsonplugins/nemo-rl/pyproject.tomlpyproject.tomlpytest.iniservices/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.pythird_party/requirements-main.txt
💤 Files with no reviewable changes (1)
- pytest.ini
🚧 Files skipped from review as they are similar to previous changes (20)
- pyproject.toml
- plugins/nemo-customizer/README.md
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.json
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/rl-kubernetes-runtime.md
- docs/fern/components/NotebookViewer.tsx
- docs/troubleshooting/customizer.mdx
- services/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.py
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.md
- docs/fern/scripts/validate-notebook-viewer.mjs
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.md
- plugins/nemo-rl/pyproject.toml
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.md
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.md
- docs/customizer/tutorials/dpo-customization-job.mdx
- third_party/requirements-main.txt
- docs/fern/components/notebooks/dpo-customization-job.ts
- docs/fern/components/notebooks/dpo-customization-job.json
- docs/customizer/tutorials/dpo-customization-job.ipynb
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.md
- plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.md
Signed-off-by: anubhutiv <anubhutiv@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Testing:
Verified on new image that training works
Summary by CodeRabbit
New Features
Documentation
Tests