diff --git a/docs/customizer/tutorials/dpo-customization-job.ipynb b/docs/customizer/tutorials/dpo-customization-job.ipynb new file mode 100644 index 0000000000..5c8437dfcb --- /dev/null +++ b/docs/customizer/tutorials/dpo-customization-job.ipynb @@ -0,0 +1,528 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "\n", + "# DPO Model Customization Job\n", + "\n", + "Learn how to use the NeMo Platform to align a model with **DPO** (Direct Preference Optimization) on a preference dataset. For each prompt, DPO trains on a *chosen* (preferred) and a *rejected* response so the model prefers the chosen style — no separate reward model required.\n", + "\n", + "This tutorial uses the `rl` customization backend (powered by [NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL)), which runs DPO on a **Ray** cluster. Unlike the [SFT](/documentation/customizer-reference/tutorials/sft-customization-job) and [LoRA](/documentation/customizer-reference/tutorials/lora-customization-job) tutorials (Docker GPU jobs), `rl` requires a **Kubernetes-backed** NeMo Platform. DPO here is **full-weight** (no LoRA/adapter); the output is a full model entity.\n", + "\n", + "**Time to complete:** approximately 45-60 minutes. Job duration increases with model and dataset size." + ], + "id": "40916acf" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "Before starting this tutorial, ensure you have:\n", + "\n", + "1. **Completed the [Quickstart](/documentation/get-started)** to install the NeMo Platform and Python SDK.\n", + "2. **Installed the Python SDK** (PyPI wrapper: `pip install \"nemo-platform[all]\"`; source checkout: run `make bootstrap` from the repository root).\n", + "3. **Installed the `datasets` package**: `pip install datasets`.\n", + "4. **A platform configured with `platform.runtime: kubernetes`.** The `rl` (DPO) backend provisions a Ray cluster and has **no local Docker fallback** — `submit` fails fast on a Docker-runtime platform. Multi-node jobs (`parallelism.num_nodes > 1`) additionally require the platform-side `NMP_RL_MULTINODE_SHARED_STORAGE_PATH`.\n", + "5. **A Hugging Face token** with access to the gated base model (this tutorial uses `meta-llama/Llama-3.2-1B-Instruct`). Export it as `HF_TOKEN`.\n", + "6. **At least one GPU with CUDA 13+** and a GPU execution profile (`nemo jobs list-execution-profiles`)." + ], + "id": "15435f3c" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Quick Start\n", + "\n", + "### 1. Initialize the SDK\n", + "\n", + "The SDK needs your NeMo Platform server URL. By default `http://localhost:8080` is used; set `NMP_BASE_URL` to override:\n", + "\n", + "```sh\n", + "export NMP_BASE_URL=\n", + "```" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "import json\n", + "import os\n", + "import time\n", + "import uuid\n", + "from pathlib import Path\n", + "from nemo_platform import NeMoPlatform, ConflictError\n", + "from nemo_platform.types.secrets import PlatformSecretResponse\n", + "from nemo_platform.types.files import HuggingfaceStorageConfigParam\n", + "from nemo_rl_plugin.schema import RlJobInput\n", + "\n", + "\n", + "def max_wait_time_checker(seconds: int, label: str = \"\"):\n", + " \"\"\"Return a check() that raises TimeoutError once `seconds` have elapsed.\"\"\"\n", + " start = time.time()\n", + "\n", + " def check():\n", + " if time.time() - start > seconds:\n", + " raise TimeoutError(f\"{label} took longer than {seconds} seconds\")\n", + "\n", + " return check\n", + "\n", + "\n", + "NMP_BASE_URL = os.environ.get(\"NMP_BASE_URL\", \"http://localhost:8080\")\n", + "sdk = NeMoPlatform(base_url=NMP_BASE_URL, workspace=\"default\")" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2. Prepare the Preference Dataset\n", + "\n", + "DPO trains on **preference data**. The `rl` backend takes a **single** dataset fileset that holds both `training.jsonl` and `validation.jsonl`, and auto-detects the row schema from the first line. Three preference formats are supported (see the platform's `BinaryPreferenceDatasetItemSchema` / `HelpSteer3DatasetItemSchema` / `Tulu3PreferenceDatasetItemSchema`):" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Binary Preference Format\n", + "\n", + "Simple `prompt` / `chosen` / `rejected` (the `prompt` may be a string or a list of chat messages):\n", + "\n", + "```json\n", + "{\"prompt\": \"What is the capital of France?\", \"chosen\": \"The capital of France is Paris.\", \"rejected\": \"I'm not sure.\"}\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### HelpSteer3 Format (used here)\n", + "\n", + "A conversation `context` (string or chat messages), two candidate `response1` / `response2`, and a signed `overall_preference` in -3..3 — **negative** means response 1 is preferred, **positive** means response 2, **0** is a tie. This is the **raw** schema of `nvidia/HelpSteer3`, so no conversion is needed:\n", + "\n", + "```json\n", + "{\"context\": [{\"role\": \"user\", \"content\": \"Explain how to use git rebase\"}], \"response1\": \"...\", \"response2\": \"...\", \"overall_preference\": -2}\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Tulu3 Preference Format\n", + "\n", + "Full chat conversations for both the chosen and rejected branches (each a list of messages ending with the assistant turn):\n", + "\n", + "```json\n", + "{\"chosen\": [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"preferred\"}], \"rejected\": [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"dispreferred\"}]}\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Download nvidia/HelpSteer3\n", + "\n", + "We use [nvidia/HelpSteer3](https://huggingface.co/datasets/nvidia/HelpSteer3) (the `preference` subset), NVIDIA's open preference dataset. It ships native `train` and `validation` splits and matches the HelpSteer3 schema above, so we upload the rows **as-is** — the platform's `HelpSteer3Dataset` loader handles the `overall_preference` semantics (including ties) at training time." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "from datasets import load_dataset, Dataset\n", + "\n", + "print(\"Loading dataset nvidia/HelpSteer3 (preference subset)\")\n", + "ds = load_dataset(\"nvidia/HelpSteer3\", \"preference\")\n", + "\n", + "# Small subsets keep the tutorial fast; larger sets train better but take longer.\n", + "training_size = 3000\n", + "validation_size = 300\n", + "DATASET_NAME = \"dpo-dataset\"\n", + "DATASET_PATH = Path(\"dpo-dataset\").absolute()\n", + "os.makedirs(DATASET_PATH, exist_ok=True)\n", + "\n", + "train_dataset = ds[\"train\"]\n", + "validation_dataset = ds[\"validation\"]\n", + "assert isinstance(train_dataset, Dataset) and isinstance(validation_dataset, Dataset)\n", + "\n", + "# Save raw HelpSteer3 rows directly — no conversion. The platform detects the\n", + "# HelpSteer3 schema from the row keys (context / response1 / response2 / overall_preference).\n", + "train_dataset.select(range(training_size)).to_json(f\"{DATASET_PATH}/training.jsonl\")\n", + "validation_dataset.select(range(validation_size)).to_json(f\"{DATASET_PATH}/validation.jsonl\")\n", + "\n", + "print(f\"Saved training.jsonl ({training_size} rows) and validation.jsonl ({validation_size} rows)\")\n", + "with open(f\"{DATASET_PATH}/training.jsonl\") as f:\n", + " sample = json.loads(f.readline())\n", + "print(\"Sample keys:\", sorted(sample.keys()))\n", + "print(\"overall_preference:\", sample[\"overall_preference\"])" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3. Create FileSet and Upload Preference Data\n", + "\n", + "Upload both JSONL files to a single FileSet so the DPO job can read them." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "try:\n", + " sdk.files.filesets.create(workspace=\"default\", name=DATASET_NAME, description=\"DPO preference data\")\n", + " print(f\"Created fileset: {DATASET_NAME}\")\n", + "except ConflictError:\n", + " print(f\"Fileset '{DATASET_NAME}' already exists, continuing...\")\n", + "\n", + "sdk.files.upload(local_path=DATASET_PATH, remote_path=\"\", fileset=DATASET_NAME, workspace=\"default\")\n", + "\n", + "print(\"Preference data:\")\n", + "print(json.dumps([f.model_dump() for f in sdk.files.list(fileset=DATASET_NAME, workspace=\"default\").data], indent=2, default=str))" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4. Secrets Setup\n", + "\n", + "The base model (`meta-llama/Llama-3.2-1B-Instruct`) is gated, so store your Hugging Face token as a platform secret named `hf-token` and reference it on the model fileset." + ], + "id": "7f8bde21" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "HF_TOKEN = os.getenv(\"HF_TOKEN\")\n", + "if not HF_TOKEN:\n", + " raise RuntimeError(\"Set HF_TOKEN before running this tutorial.\")\n", + "\n", + "def create_or_get_secret(name: str, value: str, label: str) -> PlatformSecretResponse:\n", + " try:\n", + " secret = sdk.secrets.create(name=name, workspace=\"default\", value=value)\n", + " print(f\"Created secret: {name}\")\n", + " return secret\n", + " except ConflictError:\n", + " print(f\"Secret '{name}' already exists, continuing...\")\n", + " return sdk.secrets.retrieve(name=name, workspace=\"default\")\n", + "\n", + "\n", + "hf_secret = create_or_get_secret(\"hf-token\", HF_TOKEN, \"HF_TOKEN\")" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5. Create Base Model FileSet and Model Entity\n", + "\n", + "DPO starts from an instruction-tuned base model. The model entity's spec is inferred asynchronously after creation." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "HF_REPO_ID = \"meta-llama/Llama-3.2-1B-Instruct\"\n", + "MODEL_NAME = \"llama-3-2-1b-instruct\"\n", + "\n", + "storage = HuggingfaceStorageConfigParam(\n", + " type=\"huggingface\",\n", + " repo_id=HF_REPO_ID,\n", + " repo_type=\"model\",\n", + " token_secret=hf_secret.name,\n", + ")\n", + "\n", + "try:\n", + " base_model_fs = sdk.files.filesets.create(\n", + " workspace=\"default\", name=MODEL_NAME, description=\"Llama 3.2 1B Instruct base model\", storage=storage\n", + " )\n", + " print(f\"Created base model fileset: {MODEL_NAME}\")\n", + "except ConflictError:\n", + " base_model_fs = sdk.files.filesets.retrieve(workspace=\"default\", name=MODEL_NAME)\n", + " print(\"Base model fileset already exists.\")\n", + "\n", + "try:\n", + " base_model = sdk.models.create(workspace=\"default\", name=MODEL_NAME, fileset=f\"default/{MODEL_NAME}\")\n", + "except ConflictError:\n", + " base_model = sdk.models.retrieve(workspace=\"default\", name=MODEL_NAME)\n", + "\n", + "print(f\"Base model fileset: fileset://default/{base_model.name}\")\n", + "\n", + "# Wait for the ModelSpec to be inferred from the checkpoint.\n", + "check = max_wait_time_checker(600, \"Model spec\")\n", + "while not base_model.spec:\n", + " check()\n", + " time.sleep(10)\n", + " base_model = sdk.models.retrieve(workspace=\"default\", name=MODEL_NAME)\n", + "print(\"Model spec ready\")" + ], + "execution_count": null, + "outputs": [], + "id": "1b798ede" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6. Create the DPO Customization Job\n", + "\n", + "Submit a DPO job to the `rl` backend with `RlJobInput`. Note the DPO-specific shape:\n", + "\n", + "- `model` is a string ref to the model entity; `dataset` is a **single** string ref to the preference fileset (holding both files).\n", + "- The training method is `{\"type\": \"dpo\", ...}` — full-weight, no `finetuning_type`/LoRA.\n", + "- `ref_policy_kl_penalty` is **β** (DPO paper): how strongly the policy stays tied to the reference model.\n", + "- `rl` auto-generates the job id (`rl-`); read it back from the response.\n", + "\n", + "Other configurable knobs: `optimizer_type`, `adam_eps`, `activation_checkpointing`, `keep_top_k`, `val_at_end`, `preference_loss_weight`, `sft_loss_weight`. Run `nemo customization rl explain` for the live schema." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "job_suffix = uuid.uuid4().hex[:8]\n", + "OUTPUT_NAME = f\"llama-3-2-1b-dpo-{job_suffix}\"\n", + "\n", + "spec = RlJobInput(\n", + " model=f\"default/{base_model.name}\",\n", + " dataset=f\"default/{DATASET_NAME}\",\n", + " training={\n", + " \"type\": \"dpo\",\n", + " \"epochs\": 1,\n", + " \"batch_size\": 16,\n", + " \"micro_batch_size\": 1,\n", + " \"learning_rate\": 5e-6,\n", + " \"max_seq_length\": 4096,\n", + " \"ref_policy_kl_penalty\": 0.1,\n", + " \"parallelism\": {\n", + " \"num_nodes\": 1,\n", + " \"num_gpus_per_node\": 1,\n", + " \"tensor_parallel_size\": 1,\n", + " \"pipeline_parallel_size\": 1,\n", + " },\n", + " },\n", + " output={\"name\": OUTPUT_NAME},\n", + ")\n", + "\n", + "# `rl` auto-generates the job id (rl-); do not pass name=.\n", + "job = sdk.customization.rl.jobs.create(spec=spec, workspace=\"default\")\n", + "print(f\"Job ID: {job.job.name}\")\n", + "print(f\"Output model: {OUTPUT_NAME}\")" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 7. Track Training Progress\n", + "\n", + "The DPO job runs four steps: download -> **dpo-training** (Ray) -> upload -> model-entity. We poll the top-level job status and surface the training step's progress." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "from IPython.display import clear_output\n", + "\n", + "check = max_wait_time_checker(7200, \"DPO job\")\n", + "while True:\n", + " check()\n", + " status = sdk.jobs.get_status(name=job.job.name, workspace=\"default\")\n", + " clear_output(wait=True)\n", + " print(f\"Job Status: {status.status}\")\n", + "\n", + " step = max_steps = phase = None\n", + " for job_step in status.steps or []:\n", + " if job_step.name == \"dpo-training\":\n", + " for task in job_step.tasks or []:\n", + " d = task.status_details or {}\n", + " step, max_steps, phase = d.get(\"step\"), d.get(\"max_steps\"), d.get(\"phase\")\n", + " break\n", + " break\n", + " if step is not None and max_steps:\n", + " print(f\"Training: Step {step}/{max_steps} ({100 * step / max_steps:.1f}%)\")\n", + " if phase:\n", + " print(f\"Phase: {phase}\")\n", + "\n", + " if status.status in (\"completed\", \"failed\", \"cancelled\", \"error\"):\n", + " print(f\"\\nJob finished: {status.status}\")\n", + " break\n", + " time.sleep(15)\n", + "\n", + "assert status.status == \"completed\"" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Interpreting DPO training metrics** (in `status_details.metrics`):\n", + "\n", + "- **`loss`** — the DPO loss; should trend down as the policy learns to separate chosen from rejected.\n", + "- **Reward margin** (chosen minus rejected reward) — should trend **up**: the model increasingly prefers chosen responses.\n", + "- **Validation `loss`** — watch for divergence from training loss (overfitting). Raise `ref_policy_kl_penalty` (β) or add `sft_loss_weight` if the policy drifts too far from the reference." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 8. Validate the Output Model\n", + "\n", + "DPO produces a **full-weight model entity** (not an adapter). Confirm it was registered." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "model_entity = sdk.models.retrieve(workspace=\"default\", name=OUTPUT_NAME)\n", + "print(model_entity.model_dump_json(indent=2))" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 9. Deploy and Evaluate (optional)\n", + "\n", + "The DPO output is a full model, so it deploys like any full-weight checkpoint (see the [Full SFT](/documentation/customizer-reference/tutorials/sft-customization-job) tutorial for details). We deploy with vLLM and send a chat completion." + ], + "id": "fbbce3cb" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "deploy_suffix = uuid.uuid4().hex[:8]\n", + "DEPLOYMENT_CONFIG_NAME = f\"dpo-deployment-cfg-{deploy_suffix}\"\n", + "DEPLOYMENT_NAME = f\"dpo-deployment-{deploy_suffix}\"\n", + "\n", + "deployment_config = sdk.inference.deployment_configs.create(\n", + " workspace=\"default\",\n", + " name=DEPLOYMENT_CONFIG_NAME,\n", + " engine=\"vllm\",\n", + " model_spec={\"model_namespace\": \"default\", \"model_name\": OUTPUT_NAME},\n", + " executor_config={\"gpu\": 1, \"image_name\": \"vllm/vllm-openai\", \"image_tag\": \"v0.22.1\"},\n", + ")\n", + "\n", + "deployment = sdk.inference.deployments.create(\n", + " workspace=\"default\", name=DEPLOYMENT_NAME, config=deployment_config.name\n", + ")\n", + "print(f\"Deployment name: {deployment.name}\")" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "check = max_wait_time_checker(1800, \"Deployment\")\n", + "while True:\n", + " check()\n", + " deployment_status = sdk.inference.deployments.retrieve(name=deployment.name, workspace=\"default\")\n", + " clear_output(wait=True)\n", + " print(f\"Deployment status: {deployment_status.status}\")\n", + " deployment_state = str(deployment_status.status).lower()\n", + " if deployment_state in (\"ready\", \"running\"):\n", + " if not sdk.models.wait_for_gateway(deployment.name, workspace=\"default\", timeout=60):\n", + " raise RuntimeError(\"Inference gateway did not become ready\")\n", + " break\n", + " if deployment_state in (\"failed\", \"error\", \"terminated\", \"lost\"):\n", + " raise RuntimeError(f\"Deployment failed with status: {deployment_status.status}\")\n", + " time.sleep(15)" + ], + "execution_count": null, + "outputs": [], + "id": "a5811863" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "messages = [\n", + " {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n", + " {\"role\": \"user\", \"content\": \"Write a short, friendly email to a colleague asking to reschedule our meeting to Thursday.\"},\n", + "]\n", + "\n", + "response = sdk.inference.gateway.provider.post(\n", + " \"v1/chat/completions\",\n", + " name=deployment.name,\n", + " workspace=\"default\",\n", + " body={\"model\": f\"default/{OUTPUT_NAME}\", \"messages\": messages, \"temperature\": 0.7, \"max_tokens\": 256},\n", + ")\n", + "print(\"Model output:\\n\")\n", + "print(response[\"choices\"][0][\"message\"][\"content\"])" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion\n", + "\n", + "You aligned a base model with **DPO** on the NeMo Platform using the `rl` backend:\n", + "\n", + "- Uploaded a HelpSteer3 preference dataset **as-is** (the platform detects the schema natively).\n", + "- Submitted a full-weight DPO job that ran on a Ray cluster via the Kubernetes executor.\n", + "- Registered the output as a full model entity and (optionally) deployed it for inference.\n", + "\n", + "**Next steps:** tune the alignment strength with `ref_policy_kl_penalty` (β), add `sft_loss_weight` to anchor the policy to the chosen responses, enable `activation_checkpointing` for memory headroom, or scale up with `parallelism`. See the [Training Configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration) reference for the full hyperparameter set." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.14" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/docs/customizer/tutorials/dpo-customization-job.mdx b/docs/customizer/tutorials/dpo-customization-job.mdx new file mode 100644 index 0000000000..a6732520c8 --- /dev/null +++ b/docs/customizer/tutorials/dpo-customization-job.mdx @@ -0,0 +1,9 @@ +--- +title: "DPO Customization" +description: "" +--- + + diff --git a/docs/fern/components/NotebookViewer.tsx b/docs/fern/components/NotebookViewer.tsx index cfac08e343..e05f48ec41 100644 --- a/docs/fern/components/NotebookViewer.tsx +++ b/docs/fern/components/NotebookViewer.tsx @@ -6,6 +6,7 @@ import type { ReactNode } from "react"; import distillationCustomizationJob from "./notebooks/distillation-customization-job"; +import dpoCustomizationJob from "./notebooks/dpo-customization-job"; import embeddingCustomizationJob from "./notebooks/embedding-customization-job"; import loraCustomizationJob from "./notebooks/lora-customization-job"; import optimizeThroughput from "./notebooks/optimize-throughput"; @@ -17,6 +18,7 @@ import toolCalling from "./notebooks/tool-calling"; // pages look it up by name: ``. const notebooks: Record = { "distillation-customization-job": distillationCustomizationJob, + "dpo-customization-job": dpoCustomizationJob, "embedding-customization-job": embeddingCustomizationJob, "lora-customization-job": loraCustomizationJob, "optimize-throughput": optimizeThroughput, diff --git a/docs/fern/components/notebooks/dpo-customization-job.json b/docs/fern/components/notebooks/dpo-customization-job.json new file mode 100644 index 0000000000..ab69dab155 --- /dev/null +++ b/docs/fern/components/notebooks/dpo-customization-job.json @@ -0,0 +1,155 @@ +{ + "cells": [ + { + "type": "markdown", + "source": "\n\n\n# DPO Model Customization Job\n\nLearn how to use the NeMo Platform to align a model with **DPO** (Direct Preference Optimization) on a preference dataset. For each prompt, DPO trains on a *chosen* (preferred) and a *rejected* response so the model prefers the chosen style — no separate reward model required.\n\nThis tutorial uses the `rl` customization backend (powered by [NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL)), which runs DPO on a **Ray** cluster. Unlike the [SFT](/documentation/customizer-reference/tutorials/sft-customization-job) and [LoRA](/documentation/customizer-reference/tutorials/lora-customization-job) tutorials (Docker GPU jobs), `rl` requires a **Kubernetes-backed** NeMo Platform. DPO here is **full-weight** (no LoRA/adapter); the output is a full model entity.\n\n**Time to complete:** approximately 45-60 minutes. Job duration increases with model and dataset size.", + "source_html": "\n\n

DPO Model Customization Job

\n

Learn how to use the NeMo Platform to align a model with DPO (Direct Preference Optimization) on a preference dataset. For each prompt, DPO trains on a chosen (preferred) and a rejected response so the model prefers the chosen style — no separate reward model required.

\n

This tutorial uses the rl customization backend (powered by NVIDIA NeMo-RL), which runs DPO on a Ray cluster. Unlike the SFT and LoRA tutorials (Docker GPU jobs), rl requires a Kubernetes-backed NeMo Platform. DPO here is full-weight (no LoRA/adapter); the output is a full model entity.

\n

Time to complete: approximately 45-60 minutes. Job duration increases with model and dataset size.

\n" + }, + { + "type": "markdown", + "source": "## Prerequisites\n\nBefore starting this tutorial, ensure you have:\n\n1. **Completed the [Quickstart](/documentation/get-started)** to install the NeMo Platform and Python SDK.\n2. **Installed the Python SDK** (PyPI wrapper: `pip install \"nemo-platform[all]\"`; source checkout: run `make bootstrap` from the repository root).\n3. **Installed the `datasets` package**: `pip install datasets`.\n4. **A platform configured with `platform.runtime: kubernetes`.** The `rl` (DPO) backend provisions a Ray cluster and has **no local Docker fallback** — `submit` fails fast on a Docker-runtime platform. Multi-node jobs (`parallelism.num_nodes > 1`) additionally require the platform-side `NMP_RL_MULTINODE_SHARED_STORAGE_PATH`.\n5. **A Hugging Face token** with access to the gated base model (this tutorial uses `meta-llama/Llama-3.2-1B-Instruct`). Export it as `HF_TOKEN`.\n6. **At least one GPU with CUDA 13+** and a GPU execution profile (`nemo jobs list-execution-profiles`).", + "source_html": "

Prerequisites

\n

Before starting this tutorial, ensure you have:

\n
    \n
  1. Completed the Quickstart to install the NeMo Platform and Python SDK.
    2. \n
  2. Installed the Python SDK (PyPI wrapper: pip install "nemo-platform[all]"; source checkout: run make bootstrap from the repository root).
    3. \n
  3. Installed the datasets package: pip install datasets.
    4. \n
  4. A platform configured with platform.runtime: kubernetes. The rl (DPO) backend provisions a Ray cluster and has no local Docker fallback — submit fails fast on a Docker-runtime platform. Multi-node jobs (parallelism.num_nodes > 1) additionally require the platform-side NMP_RL_MULTINODE_SHARED_STORAGE_PATH.
    5. \n
  5. A Hugging Face token with access to the gated base model (this tutorial uses meta-llama/Llama-3.2-1B-Instruct). Export it as HF_TOKEN.
    6. \n
  6. At least one GPU with CUDA 13+ and a GPU execution profile (nemo jobs list-execution-profiles).
    7. \n
\n" + }, + { + "type": "markdown", + "source": "## Quick Start\n\n### 1. Initialize the SDK\n\nThe SDK needs your NeMo Platform server URL. By default `http://localhost:8080` is used; set `NMP_BASE_URL` to override:\n\n```sh\nexport NMP_BASE_URL=\n```", + "source_html": "

Quick Start

\n

1. Initialize the SDK

\n

The SDK needs your NeMo Platform server URL. By default http://localhost:8080 is used; set NMP_BASE_URL to override:

\n
export NMP_BASE_URL=<YOUR_NMP_BASE_URL>\n
\n" + }, + { + "type": "code", + "source": "import json\nimport os\nimport time\nimport uuid\nfrom pathlib import Path\nfrom nemo_platform import NeMoPlatform, ConflictError\nfrom nemo_platform.types.secrets import PlatformSecretResponse\nfrom nemo_platform.types.files import HuggingfaceStorageConfigParam\nfrom nemo_rl_plugin.schema import RlJobInput\n\n\ndef max_wait_time_checker(seconds: int, label: str = \"\"):\n \"\"\"Return a check() that raises TimeoutError once `seconds` have elapsed.\"\"\"\n start = time.time()\n\n def check():\n if time.time() - start > seconds:\n raise TimeoutError(f\"{label} took longer than {seconds} seconds\")\n\n return check\n\n\nNMP_BASE_URL = os.environ.get(\"NMP_BASE_URL\", \"http://localhost:8080\")\nsdk = NeMoPlatform(base_url=NMP_BASE_URL, workspace=\"default\")", + "language": "python", + "source_html": "import json\nimport os\nimport time\nimport uuid\nfrom pathlib import Path\nfrom nemo_platform import NeMoPlatform, ConflictError\nfrom nemo_platform.types.secrets import PlatformSecretResponse\nfrom nemo_platform.types.files import HuggingfaceStorageConfigParam\nfrom nemo_rl_plugin.schema import RlJobInput\n\n\ndef max_wait_time_checker(seconds: int, label: str = ""):\n """Return a check() that raises TimeoutError once `seconds` have elapsed."""\n start = time.time()\n\n def check():\n if time.time() - start > seconds:\n raise TimeoutError(f"{label} took longer than {seconds} seconds")\n\n return check\n\n\nNMP_BASE_URL = os.environ.get("NMP_BASE_URL", "http://localhost:8080")\nsdk = NeMoPlatform(base_url=NMP_BASE_URL, workspace="default")\n" + }, + { + "type": "markdown", + "source": "### 2. Prepare the Preference Dataset\n\nDPO trains on **preference data**. The `rl` backend takes a **single** dataset fileset that holds both `training.jsonl` and `validation.jsonl`, and auto-detects the row schema from the first line. Three preference formats are supported (see the platform's `BinaryPreferenceDatasetItemSchema` / `HelpSteer3DatasetItemSchema` / `Tulu3PreferenceDatasetItemSchema`):", + "source_html": "

2. Prepare the Preference Dataset

\n

DPO trains on preference data. The rl backend takes a single dataset fileset that holds both training.jsonl and validation.jsonl, and auto-detects the row schema from the first line. Three preference formats are supported (see the platform's BinaryPreferenceDatasetItemSchema / HelpSteer3DatasetItemSchema / Tulu3PreferenceDatasetItemSchema):

\n" + }, + { + "type": "markdown", + "source": "#### Binary Preference Format\n\nSimple `prompt` / `chosen` / `rejected` (the `prompt` may be a string or a list of chat messages):\n\n```json\n{\"prompt\": \"What is the capital of France?\", \"chosen\": \"The capital of France is Paris.\", \"rejected\": \"I'm not sure.\"}\n```", + "source_html": "

Binary Preference Format

\n

Simple prompt / chosen / rejected (the prompt may be a string or a list of chat messages):

\n
{"prompt": "What is the capital of France?", "chosen": "The capital of France is Paris.", "rejected": "I'm not sure."}\n
\n" + }, + { + "type": "markdown", + "source": "#### HelpSteer3 Format (used here)\n\nA conversation `context` (string or chat messages), two candidate `response1` / `response2`, and a signed `overall_preference` in -3..3 — **negative** means response 1 is preferred, **positive** means response 2, **0** is a tie. This is the **raw** schema of `nvidia/HelpSteer3`, so no conversion is needed:\n\n```json\n{\"context\": [{\"role\": \"user\", \"content\": \"Explain how to use git rebase\"}], \"response1\": \"...\", \"response2\": \"...\", \"overall_preference\": -2}\n```", + "source_html": "

HelpSteer3 Format (used here)

\n

A conversation context (string or chat messages), two candidate response1 / response2, and a signed overall_preference in -3..3 — negative means response 1 is preferred, positive means response 2, 0 is a tie. This is the raw schema of nvidia/HelpSteer3, so no conversion is needed:

\n
{"context": [{"role": "user", "content": "Explain how to use git rebase"}], "response1": "...", "response2": "...", "overall_preference": -2}\n
\n" + }, + { + "type": "markdown", + "source": "#### Tulu3 Preference Format\n\nFull chat conversations for both the chosen and rejected branches (each a list of messages ending with the assistant turn):\n\n```json\n{\"chosen\": [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"preferred\"}], \"rejected\": [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"dispreferred\"}]}\n```", + "source_html": "

Tulu3 Preference Format

\n

Full chat conversations for both the chosen and rejected branches (each a list of messages ending with the assistant turn):

\n
{"chosen": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "preferred"}], "rejected": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "dispreferred"}]}\n
\n" + }, + { + "type": "markdown", + "source": "#### Download nvidia/HelpSteer3\n\nWe use [nvidia/HelpSteer3](https://huggingface.co/datasets/nvidia/HelpSteer3) (the `preference` subset), NVIDIA's open preference dataset. It ships native `train` and `validation` splits and matches the HelpSteer3 schema above, so we upload the rows **as-is** — the platform's `HelpSteer3Dataset` loader handles the `overall_preference` semantics (including ties) at training time.", + "source_html": "

Download nvidia/HelpSteer3

\n

We use nvidia/HelpSteer3 (the preference subset), NVIDIA's open preference dataset. It ships native train and validation splits and matches the HelpSteer3 schema above, so we upload the rows as-is — the platform's HelpSteer3Dataset loader handles the overall_preference semantics (including ties) at training time.

\n" + }, + { + "type": "code", + "source": "from datasets import load_dataset, Dataset\n\nprint(\"Loading dataset nvidia/HelpSteer3 (preference subset)\")\nds = load_dataset(\"nvidia/HelpSteer3\", \"preference\")\n\n# Small subsets keep the tutorial fast; larger sets train better but take longer.\ntraining_size = 3000\nvalidation_size = 300\nDATASET_NAME = \"dpo-dataset\"\nDATASET_PATH = Path(\"dpo-dataset\").absolute()\nos.makedirs(DATASET_PATH, exist_ok=True)\n\ntrain_dataset = ds[\"train\"]\nvalidation_dataset = ds[\"validation\"]\nassert isinstance(train_dataset, Dataset) and isinstance(validation_dataset, Dataset)\n\n# Save raw HelpSteer3 rows directly — no conversion. The platform detects the\n# HelpSteer3 schema from the row keys (context / response1 / response2 / overall_preference).\ntrain_dataset.select(range(training_size)).to_json(f\"{DATASET_PATH}/training.jsonl\")\nvalidation_dataset.select(range(validation_size)).to_json(f\"{DATASET_PATH}/validation.jsonl\")\n\nprint(f\"Saved training.jsonl ({training_size} rows) and validation.jsonl ({validation_size} rows)\")\nwith open(f\"{DATASET_PATH}/training.jsonl\") as f:\n sample = json.loads(f.readline())\nprint(\"Sample keys:\", sorted(sample.keys()))\nprint(\"overall_preference:\", sample[\"overall_preference\"])", + "language": "python", + "source_html": "from datasets import load_dataset, Dataset\n\nprint("Loading dataset nvidia/HelpSteer3 (preference subset)")\nds = load_dataset("nvidia/HelpSteer3", "preference")\n\n# Small subsets keep the tutorial fast; larger sets train better but take longer.\ntraining_size = 3000\nvalidation_size = 300\nDATASET_NAME = "dpo-dataset"\nDATASET_PATH = Path("dpo-dataset").absolute()\nos.makedirs(DATASET_PATH, exist_ok=True)\n\ntrain_dataset = ds["train"]\nvalidation_dataset = ds["validation"]\nassert isinstance(train_dataset, Dataset) and isinstance(validation_dataset, Dataset)\n\n# Save raw HelpSteer3 rows directly — no conversion. The platform detects the\n# HelpSteer3 schema from the row keys (context / response1 / response2 / overall_preference).\ntrain_dataset.select(range(training_size)).to_json(f"{DATASET_PATH}/training.jsonl")\nvalidation_dataset.select(range(validation_size)).to_json(f"{DATASET_PATH}/validation.jsonl")\n\nprint(f"Saved training.jsonl ({training_size} rows) and validation.jsonl ({validation_size} rows)")\nwith open(f"{DATASET_PATH}/training.jsonl") as f:\n sample = json.loads(f.readline())\nprint("Sample keys:", sorted(sample.keys()))\nprint("overall_preference:", sample["overall_preference"])\n" + }, + { + "type": "markdown", + "source": "### 3. Create FileSet and Upload Preference Data\n\nUpload both JSONL files to a single FileSet so the DPO job can read them.", + "source_html": "

3. Create FileSet and Upload Preference Data

\n

Upload both JSONL files to a single FileSet so the DPO job can read them.

\n" + }, + { + "type": "code", + "source": "try:\n sdk.files.filesets.create(workspace=\"default\", name=DATASET_NAME, description=\"DPO preference data\")\n print(f\"Created fileset: {DATASET_NAME}\")\nexcept ConflictError:\n print(f\"Fileset '{DATASET_NAME}' already exists, continuing...\")\n\nsdk.files.upload(local_path=DATASET_PATH, remote_path=\"\", fileset=DATASET_NAME, workspace=\"default\")\n\nprint(\"Preference data:\")\nprint(json.dumps([f.model_dump() for f in sdk.files.list(fileset=DATASET_NAME, workspace=\"default\").data], indent=2, default=str))", + "language": "python", + "source_html": "try:\n sdk.files.filesets.create(workspace="default", name=DATASET_NAME, description="DPO preference data")\n print(f"Created fileset: {DATASET_NAME}")\nexcept ConflictError:\n print(f"Fileset '{DATASET_NAME}' already exists, continuing...")\n\nsdk.files.upload(local_path=DATASET_PATH, remote_path="", fileset=DATASET_NAME, workspace="default")\n\nprint("Preference data:")\nprint(json.dumps([f.model_dump() for f in sdk.files.list(fileset=DATASET_NAME, workspace="default").data], indent=2, default=str))\n" + }, + { + "type": "markdown", + "source": "### 4. Secrets Setup\n\nThe base model (`meta-llama/Llama-3.2-1B-Instruct`) is gated, so store your Hugging Face token as a platform secret named `hf-token` and reference it on the model fileset.", + "source_html": "

4. Secrets Setup

\n

The base model (meta-llama/Llama-3.2-1B-Instruct) is gated, so store your Hugging Face token as a platform secret named hf-token and reference it on the model fileset.

\n" + }, + { + "type": "code", + "source": "HF_TOKEN = os.getenv(\"HF_TOKEN\")\nif not HF_TOKEN:\n raise RuntimeError(\"Set HF_TOKEN before running this tutorial.\")\n\ndef create_or_get_secret(name: str, value: str, label: str) -> PlatformSecretResponse:\n try:\n secret = sdk.secrets.create(name=name, workspace=\"default\", value=value)\n print(f\"Created secret: {name}\")\n return secret\n except ConflictError:\n print(f\"Secret '{name}' already exists, continuing...\")\n return sdk.secrets.retrieve(name=name, workspace=\"default\")\n\n\nhf_secret = create_or_get_secret(\"hf-token\", HF_TOKEN, \"HF_TOKEN\")", + "language": "python", + "source_html": "HF_TOKEN = os.getenv("HF_TOKEN")\nif not HF_TOKEN:\n raise RuntimeError("Set HF_TOKEN before running this tutorial.")\n\ndef create_or_get_secret(name: str, value: str, label: str) -> PlatformSecretResponse:\n try:\n secret = sdk.secrets.create(name=name, workspace="default", value=value)\n print(f"Created secret: {name}")\n return secret\n except ConflictError:\n print(f"Secret '{name}' already exists, continuing...")\n return sdk.secrets.retrieve(name=name, workspace="default")\n\n\nhf_secret = create_or_get_secret("hf-token", HF_TOKEN, "HF_TOKEN")\n" + }, + { + "type": "markdown", + "source": "### 5. Create Base Model FileSet and Model Entity\n\nDPO starts from an instruction-tuned base model. The model entity's spec is inferred asynchronously after creation.", + "source_html": "

5. Create Base Model FileSet and Model Entity

\n

DPO starts from an instruction-tuned base model. The model entity's spec is inferred asynchronously after creation.

\n" + }, + { + "type": "code", + "source": "HF_REPO_ID = \"meta-llama/Llama-3.2-1B-Instruct\"\nMODEL_NAME = \"llama-3-2-1b-instruct\"\n\nstorage = HuggingfaceStorageConfigParam(\n type=\"huggingface\",\n repo_id=HF_REPO_ID,\n repo_type=\"model\",\n token_secret=hf_secret.name,\n)\n\ntry:\n base_model_fs = sdk.files.filesets.create(\n workspace=\"default\", name=MODEL_NAME, description=\"Llama 3.2 1B Instruct base model\", storage=storage\n )\n print(f\"Created base model fileset: {MODEL_NAME}\")\nexcept ConflictError:\n base_model_fs = sdk.files.filesets.retrieve(workspace=\"default\", name=MODEL_NAME)\n print(\"Base model fileset already exists.\")\n\ntry:\n base_model = sdk.models.create(workspace=\"default\", name=MODEL_NAME, fileset=f\"default/{MODEL_NAME}\")\nexcept ConflictError:\n base_model = sdk.models.retrieve(workspace=\"default\", name=MODEL_NAME)\n\nprint(f\"Base model fileset: fileset://default/{base_model.name}\")\n\n# Wait for the ModelSpec to be inferred from the checkpoint.\ncheck = max_wait_time_checker(600, \"Model spec\")\nwhile not base_model.spec:\n check()\n time.sleep(10)\n base_model = sdk.models.retrieve(workspace=\"default\", name=MODEL_NAME)\nprint(\"Model spec ready\")", + "language": "python", + "source_html": "HF_REPO_ID = "meta-llama/Llama-3.2-1B-Instruct"\nMODEL_NAME = "llama-3-2-1b-instruct"\n\nstorage = HuggingfaceStorageConfigParam(\n type="huggingface",\n repo_id=HF_REPO_ID,\n repo_type="model",\n token_secret=hf_secret.name,\n)\n\ntry:\n base_model_fs = sdk.files.filesets.create(\n workspace="default", name=MODEL_NAME, description="Llama 3.2 1B Instruct base model", storage=storage\n )\n print(f"Created base model fileset: {MODEL_NAME}")\nexcept ConflictError:\n base_model_fs = sdk.files.filesets.retrieve(workspace="default", name=MODEL_NAME)\n print("Base model fileset already exists.")\n\ntry:\n base_model = sdk.models.create(workspace="default", name=MODEL_NAME, fileset=f"default/{MODEL_NAME}")\nexcept ConflictError:\n base_model = sdk.models.retrieve(workspace="default", name=MODEL_NAME)\n\nprint(f"Base model fileset: fileset://default/{base_model.name}")\n\n# Wait for the ModelSpec to be inferred from the checkpoint.\ncheck = max_wait_time_checker(600, "Model spec")\nwhile not base_model.spec:\n check()\n time.sleep(10)\n base_model = sdk.models.retrieve(workspace="default", name=MODEL_NAME)\nprint("Model spec ready")\n" + }, + { + "type": "markdown", + "source": "### 6. Create the DPO Customization Job\n\nSubmit a DPO job to the `rl` backend with `RlJobInput`. Note the DPO-specific shape:\n\n- `model` is a string ref to the model entity; `dataset` is a **single** string ref to the preference fileset (holding both files).\n- The training method is `{\"type\": \"dpo\", ...}` — full-weight, no `finetuning_type`/LoRA.\n- `ref_policy_kl_penalty` is **β** (DPO paper): how strongly the policy stays tied to the reference model.\n- `rl` auto-generates the job id (`rl-`); read it back from the response.\n\nOther configurable knobs: `optimizer_type`, `adam_eps`, `activation_checkpointing`, `keep_top_k`, `val_at_end`, `preference_loss_weight`, `sft_loss_weight`. Run `nemo customization rl explain` for the live schema.", + "source_html": "

6. Create the DPO Customization Job

\n

Submit a DPO job to the rl backend with RlJobInput. Note the DPO-specific shape:

\n
    \n
  • model is a string ref to the model entity; dataset is a single string ref to the preference fileset (holding both files).
    • \n
  • The training method is {"type": "dpo", ...} — full-weight, no finetuning_type/LoRA.
    • \n
  • ref_policy_kl_penalty is β (DPO paper): how strongly the policy stays tied to the reference model.
    • \n
  • rl auto-generates the job id (rl-<hex>); read it back from the response.
    • \n
\n

Other configurable knobs: optimizer_type, adam_eps, activation_checkpointing, keep_top_k, val_at_end, preference_loss_weight, sft_loss_weight. Run nemo customization rl explain for the live schema.

\n" + }, + { + "type": "code", + "source": "job_suffix = uuid.uuid4().hex[:8]\nOUTPUT_NAME = f\"llama-3-2-1b-dpo-{job_suffix}\"\n\nspec = RlJobInput(\n model=f\"default/{base_model.name}\",\n dataset=f\"default/{DATASET_NAME}\",\n training={\n \"type\": \"dpo\",\n \"epochs\": 1,\n \"batch_size\": 16,\n \"micro_batch_size\": 1,\n \"learning_rate\": 5e-6,\n \"max_seq_length\": 4096,\n \"ref_policy_kl_penalty\": 0.1,\n \"parallelism\": {\n \"num_nodes\": 1,\n \"num_gpus_per_node\": 1,\n \"tensor_parallel_size\": 1,\n \"pipeline_parallel_size\": 1,\n },\n },\n output={\"name\": OUTPUT_NAME},\n)\n\n# `rl` auto-generates the job id (rl-); do not pass name=.\njob = sdk.customization.rl.jobs.create(spec=spec, workspace=\"default\")\nprint(f\"Job ID: {job.job.name}\")\nprint(f\"Output model: {OUTPUT_NAME}\")", + "language": "python", + "source_html": "job_suffix = uuid.uuid4().hex[:8]\nOUTPUT_NAME = f"llama-3-2-1b-dpo-{job_suffix}"\n\nspec = RlJobInput(\n model=f"default/{base_model.name}",\n dataset=f"default/{DATASET_NAME}",\n training={\n "type": "dpo",\n "epochs": 1,\n "batch_size": 16,\n "micro_batch_size": 1,\n "learning_rate": 5e-6,\n "max_seq_length": 4096,\n "ref_policy_kl_penalty": 0.1,\n "parallelism": {\n "num_nodes": 1,\n "num_gpus_per_node": 1,\n "tensor_parallel_size": 1,\n "pipeline_parallel_size": 1,\n },\n },\n output={"name": OUTPUT_NAME},\n)\n\n# `rl` auto-generates the job id (rl-<hex>); do not pass name=.\njob = sdk.customization.rl.jobs.create(spec=spec, workspace="default")\nprint(f"Job ID: {job.job.name}")\nprint(f"Output model: {OUTPUT_NAME}")\n" + }, + { + "type": "markdown", + "source": "### 7. Track Training Progress\n\nThe DPO job runs four steps: download -> **dpo-training** (Ray) -> upload -> model-entity. We poll the top-level job status and surface the training step's progress.", + "source_html": "

7. Track Training Progress

\n

The DPO job runs four steps: download -> dpo-training (Ray) -> upload -> model-entity. We poll the top-level job status and surface the training step's progress.

\n" + }, + { + "type": "code", + "source": "from IPython.display import clear_output\n\ncheck = max_wait_time_checker(7200, \"DPO job\")\nwhile True:\n check()\n status = sdk.jobs.get_status(name=job.job.name, workspace=\"default\")\n clear_output(wait=True)\n print(f\"Job Status: {status.status}\")\n\n step = max_steps = phase = None\n for job_step in status.steps or []:\n if job_step.name == \"dpo-training\":\n for task in job_step.tasks or []:\n d = task.status_details or {}\n step, max_steps, phase = d.get(\"step\"), d.get(\"max_steps\"), d.get(\"phase\")\n break\n break\n if step is not None and max_steps:\n print(f\"Training: Step {step}/{max_steps} ({100 * step / max_steps:.1f}%)\")\n if phase:\n print(f\"Phase: {phase}\")\n\n if status.status in (\"completed\", \"failed\", \"cancelled\", \"error\"):\n print(f\"\\nJob finished: {status.status}\")\n break\n time.sleep(15)\n\nassert status.status == \"completed\"", + "language": "python", + "source_html": "from IPython.display import clear_output\n\ncheck = max_wait_time_checker(7200, "DPO job")\nwhile True:\n check()\n status = sdk.jobs.get_status(name=job.job.name, workspace="default")\n clear_output(wait=True)\n print(f"Job Status: {status.status}")\n\n step = max_steps = phase = None\n for job_step in status.steps or []:\n if job_step.name == "dpo-training":\n for task in job_step.tasks or []:\n d = task.status_details or {}\n step, max_steps, phase = d.get("step"), d.get("max_steps"), d.get("phase")\n break\n break\n if step is not None and max_steps:\n print(f"Training: Step {step}/{max_steps} ({100 * step / max_steps:.1f}%)")\n if phase:\n print(f"Phase: {phase}")\n\n if status.status in ("completed", "failed", "cancelled", "error"):\n print(f"\\nJob finished: {status.status}")\n break\n time.sleep(15)\n\nassert status.status == "completed"\n" + }, + { + "type": "markdown", + "source": "**Interpreting DPO training metrics** (in `status_details.metrics`):\n\n- **`loss`** — the DPO loss; should trend down as the policy learns to separate chosen from rejected.\n- **Reward margin** (chosen minus rejected reward) — should trend **up**: the model increasingly prefers chosen responses.\n- **Validation `loss`** — watch for divergence from training loss (overfitting). Raise `ref_policy_kl_penalty` (β) or add `sft_loss_weight` if the policy drifts too far from the reference.", + "source_html": "

Interpreting DPO training metrics (in status_details.metrics):

\n
    \n
  • loss — the DPO loss; should trend down as the policy learns to separate chosen from rejected.
    • \n
  • Reward margin (chosen minus rejected reward) — should trend up: the model increasingly prefers chosen responses.
    • \n
  • Validation loss — watch for divergence from training loss (overfitting). Raise ref_policy_kl_penalty (β) or add sft_loss_weight if the policy drifts too far from the reference.
    • \n
\n" + }, + { + "type": "markdown", + "source": "### 8. Validate the Output Model\n\nDPO produces a **full-weight model entity** (not an adapter). Confirm it was registered.", + "source_html": "

8. Validate the Output Model

\n

DPO produces a full-weight model entity (not an adapter). Confirm it was registered.

\n" + }, + { + "type": "code", + "source": "model_entity = sdk.models.retrieve(workspace=\"default\", name=OUTPUT_NAME)\nprint(model_entity.model_dump_json(indent=2))", + "language": "python", + "source_html": "model_entity = sdk.models.retrieve(workspace="default", name=OUTPUT_NAME)\nprint(model_entity.model_dump_json(indent=2))\n" + }, + { + "type": "markdown", + "source": "### 9. Deploy and Evaluate (optional)\n\nThe DPO output is a full model, so it deploys like any full-weight checkpoint (see the [Full SFT](/documentation/customizer-reference/tutorials/sft-customization-job) tutorial for details). We deploy with vLLM and send a chat completion.", + "source_html": "

9. Deploy and Evaluate (optional)

\n

The DPO output is a full model, so it deploys like any full-weight checkpoint (see the Full SFT tutorial for details). We deploy with vLLM and send a chat completion.

\n" + }, + { + "type": "code", + "source": "deploy_suffix = uuid.uuid4().hex[:8]\nDEPLOYMENT_CONFIG_NAME = f\"dpo-deployment-cfg-{deploy_suffix}\"\nDEPLOYMENT_NAME = f\"dpo-deployment-{deploy_suffix}\"\n\ndeployment_config = sdk.inference.deployment_configs.create(\n workspace=\"default\",\n name=DEPLOYMENT_CONFIG_NAME,\n engine=\"vllm\",\n model_spec={\"model_namespace\": \"default\", \"model_name\": OUTPUT_NAME},\n executor_config={\"gpu\": 1, \"image_name\": \"vllm/vllm-openai\", \"image_tag\": \"v0.22.1\"},\n)\n\ndeployment = sdk.inference.deployments.create(\n workspace=\"default\", name=DEPLOYMENT_NAME, config=deployment_config.name\n)\nprint(f\"Deployment name: {deployment.name}\")", + "language": "python", + "source_html": "deploy_suffix = uuid.uuid4().hex[:8]\nDEPLOYMENT_CONFIG_NAME = f"dpo-deployment-cfg-{deploy_suffix}"\nDEPLOYMENT_NAME = f"dpo-deployment-{deploy_suffix}"\n\ndeployment_config = sdk.inference.deployment_configs.create(\n workspace="default",\n name=DEPLOYMENT_CONFIG_NAME,\n engine="vllm",\n model_spec={"model_namespace": "default", "model_name": OUTPUT_NAME},\n executor_config={"gpu": 1, "image_name": "vllm/vllm-openai", "image_tag": "v0.22.1"},\n)\n\ndeployment = sdk.inference.deployments.create(\n workspace="default", name=DEPLOYMENT_NAME, config=deployment_config.name\n)\nprint(f"Deployment name: {deployment.name}")\n" + }, + { + "type": "code", + "source": "check = max_wait_time_checker(1800, \"Deployment\")\nwhile True:\n check()\n deployment_status = sdk.inference.deployments.retrieve(name=deployment.name, workspace=\"default\")\n clear_output(wait=True)\n print(f\"Deployment status: {deployment_status.status}\")\n deployment_state = str(deployment_status.status).lower()\n if deployment_state in (\"ready\", \"running\"):\n if not sdk.models.wait_for_gateway(deployment.name, workspace=\"default\", timeout=60):\n raise RuntimeError(\"Inference gateway did not become ready\")\n break\n if deployment_state in (\"failed\", \"error\", \"terminated\", \"lost\"):\n raise RuntimeError(f\"Deployment failed with status: {deployment_status.status}\")\n time.sleep(15)", + "language": "python", + "source_html": "check = max_wait_time_checker(1800, "Deployment")\nwhile True:\n check()\n deployment_status = sdk.inference.deployments.retrieve(name=deployment.name, workspace="default")\n clear_output(wait=True)\n print(f"Deployment status: {deployment_status.status}")\n deployment_state = str(deployment_status.status).lower()\n if deployment_state in ("ready", "running"):\n if not sdk.models.wait_for_gateway(deployment.name, workspace="default", timeout=60):\n raise RuntimeError("Inference gateway did not become ready")\n break\n if deployment_state in ("failed", "error", "terminated", "lost"):\n raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")\n time.sleep(15)\n" + }, + { + "type": "code", + "source": "messages = [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Write a short, friendly email to a colleague asking to reschedule our meeting to Thursday.\"},\n]\n\nresponse = sdk.inference.gateway.provider.post(\n \"v1/chat/completions\",\n name=deployment.name,\n workspace=\"default\",\n body={\"model\": f\"default/{OUTPUT_NAME}\", \"messages\": messages, \"temperature\": 0.7, \"max_tokens\": 256},\n)\nprint(\"Model output:\\n\")\nprint(response[\"choices\"][0][\"message\"][\"content\"])", + "language": "python", + "source_html": "messages = [\n {"role": "system", "content": "You are a helpful assistant."},\n {"role": "user", "content": "Write a short, friendly email to a colleague asking to reschedule our meeting to Thursday."},\n]\n\nresponse = sdk.inference.gateway.provider.post(\n "v1/chat/completions",\n name=deployment.name,\n workspace="default",\n body={"model": f"default/{OUTPUT_NAME}", "messages": messages, "temperature": 0.7, "max_tokens": 256},\n)\nprint("Model output:\\n")\nprint(response["choices"][0]["message"]["content"])\n" + }, + { + "type": "markdown", + "source": "## Conclusion\n\nYou aligned a base model with **DPO** on the NeMo Platform using the `rl` backend:\n\n- Uploaded a HelpSteer3 preference dataset **as-is** (the platform detects the schema natively).\n- Submitted a full-weight DPO job that ran on a Ray cluster via the Kubernetes executor.\n- Registered the output as a full model entity and (optionally) deployed it for inference.\n\n**Next steps:** tune the alignment strength with `ref_policy_kl_penalty` (β), add `sft_loss_weight` to anchor the policy to the chosen responses, enable `activation_checkpointing` for memory headroom, or scale up with `parallelism`. See the [Training Configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration) reference for the full hyperparameter set.", + "source_html": "

Conclusion

\n

You aligned a base model with DPO on the NeMo Platform using the rl backend:

\n
    \n
  • Uploaded a HelpSteer3 preference dataset as-is (the platform detects the schema natively).
    • \n
  • Submitted a full-weight DPO job that ran on a Ray cluster via the Kubernetes executor.
    • \n
  • Registered the output as a full model entity and (optionally) deployed it for inference.
    • \n
\n

Next steps: tune the alignment strength with ref_policy_kl_penalty (β), add sft_loss_weight to anchor the policy to the chosen responses, enable activation_checkpointing for memory headroom, or scale up with parallelism. See the Training Configuration reference for the full hyperparameter set.

\n" + } + ] +} \ No newline at end of file diff --git a/docs/fern/components/notebooks/dpo-customization-job.ts b/docs/fern/components/notebooks/dpo-customization-job.ts new file mode 100644 index 0000000000..fc8e558903 --- /dev/null +++ b/docs/fern/components/notebooks/dpo-customization-job.ts @@ -0,0 +1,159 @@ +/** + * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Auto-generated by ipynb-to-fern-json.py - do not edit manually. + */ +export default { cells: [ + { + "type": "markdown", + "source": "\n\n\n# DPO Model Customization Job\n\nLearn how to use the NeMo Platform to align a model with **DPO** (Direct Preference Optimization) on a preference dataset. For each prompt, DPO trains on a *chosen* (preferred) and a *rejected* response so the model prefers the chosen style — no separate reward model required.\n\nThis tutorial uses the `rl` customization backend (powered by [NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL)), which runs DPO on a **Ray** cluster. Unlike the [SFT](/documentation/customizer-reference/tutorials/sft-customization-job) and [LoRA](/documentation/customizer-reference/tutorials/lora-customization-job) tutorials (Docker GPU jobs), `rl` requires a **Kubernetes-backed** NeMo Platform. DPO here is **full-weight** (no LoRA/adapter); the output is a full model entity.\n\n**Time to complete:** approximately 45-60 minutes. Job duration increases with model and dataset size.", + "source_html": "\n\n

DPO Model Customization Job

\n

Learn how to use the NeMo Platform to align a model with DPO (Direct Preference Optimization) on a preference dataset. For each prompt, DPO trains on a chosen (preferred) and a rejected response so the model prefers the chosen style — no separate reward model required.

\n

This tutorial uses the rl customization backend (powered by NVIDIA NeMo-RL), which runs DPO on a Ray cluster. Unlike the SFT and LoRA tutorials (Docker GPU jobs), rl requires a Kubernetes-backed NeMo Platform. DPO here is full-weight (no LoRA/adapter); the output is a full model entity.

\n

Time to complete: approximately 45-60 minutes. Job duration increases with model and dataset size.

\n" + }, + { + "type": "markdown", + "source": "## Prerequisites\n\nBefore starting this tutorial, ensure you have:\n\n1. **Completed the [Quickstart](/documentation/get-started)** to install the NeMo Platform and Python SDK.\n2. **Installed the Python SDK** (PyPI wrapper: `pip install \"nemo-platform[all]\"`; source checkout: run `make bootstrap` from the repository root).\n3. **Installed the `datasets` package**: `pip install datasets`.\n4. **A platform configured with `platform.runtime: kubernetes`.** The `rl` (DPO) backend provisions a Ray cluster and has **no local Docker fallback** — `submit` fails fast on a Docker-runtime platform. Multi-node jobs (`parallelism.num_nodes > 1`) additionally require the platform-side `NMP_RL_MULTINODE_SHARED_STORAGE_PATH`.\n5. **A Hugging Face token** with access to the gated base model (this tutorial uses `meta-llama/Llama-3.2-1B-Instruct`). Export it as `HF_TOKEN`.\n6. **At least one GPU with CUDA 13+** and a GPU execution profile (`nemo jobs list-execution-profiles`).", + "source_html": "

Prerequisites

\n

Before starting this tutorial, ensure you have:

\n
    \n
  1. Completed the Quickstart to install the NeMo Platform and Python SDK.
    2. \n
  2. Installed the Python SDK (PyPI wrapper: pip install "nemo-platform[all]"; source checkout: run make bootstrap from the repository root).
    3. \n
  3. Installed the datasets package: pip install datasets.
    4. \n
  4. A platform configured with platform.runtime: kubernetes. The rl (DPO) backend provisions a Ray cluster and has no local Docker fallback — submit fails fast on a Docker-runtime platform. Multi-node jobs (parallelism.num_nodes > 1) additionally require the platform-side NMP_RL_MULTINODE_SHARED_STORAGE_PATH.
    5. \n
  5. A Hugging Face token with access to the gated base model (this tutorial uses meta-llama/Llama-3.2-1B-Instruct). Export it as HF_TOKEN.
    6. \n
  6. At least one GPU with CUDA 13+ and a GPU execution profile (nemo jobs list-execution-profiles).
    7. \n
\n" + }, + { + "type": "markdown", + "source": "## Quick Start\n\n### 1. Initialize the SDK\n\nThe SDK needs your NeMo Platform server URL. By default `http://localhost:8080` is used; set `NMP_BASE_URL` to override:\n\n```sh\nexport NMP_BASE_URL=\n```", + "source_html": "

Quick Start

\n

1. Initialize the SDK

\n

The SDK needs your NeMo Platform server URL. By default http://localhost:8080 is used; set NMP_BASE_URL to override:

\n
export NMP_BASE_URL=<YOUR_NMP_BASE_URL>\n
\n" + }, + { + "type": "code", + "source": "import json\nimport os\nimport time\nimport uuid\nfrom pathlib import Path\nfrom nemo_platform import NeMoPlatform, ConflictError\nfrom nemo_platform.types.secrets import PlatformSecretResponse\nfrom nemo_platform.types.files import HuggingfaceStorageConfigParam\nfrom nemo_rl_plugin.schema import RlJobInput\n\n\ndef max_wait_time_checker(seconds: int, label: str = \"\"):\n \"\"\"Return a check() that raises TimeoutError once `seconds` have elapsed.\"\"\"\n start = time.time()\n\n def check():\n if time.time() - start > seconds:\n raise TimeoutError(f\"{label} took longer than {seconds} seconds\")\n\n return check\n\n\nNMP_BASE_URL = os.environ.get(\"NMP_BASE_URL\", \"http://localhost:8080\")\nsdk = NeMoPlatform(base_url=NMP_BASE_URL, workspace=\"default\")", + "language": "python", + "source_html": "import json\nimport os\nimport time\nimport uuid\nfrom pathlib import Path\nfrom nemo_platform import NeMoPlatform, ConflictError\nfrom nemo_platform.types.secrets import PlatformSecretResponse\nfrom nemo_platform.types.files import HuggingfaceStorageConfigParam\nfrom nemo_rl_plugin.schema import RlJobInput\n\n\ndef max_wait_time_checker(seconds: int, label: str = ""):\n """Return a check() that raises TimeoutError once `seconds` have elapsed."""\n start = time.time()\n\n def check():\n if time.time() - start > seconds:\n raise TimeoutError(f"{label} took longer than {seconds} seconds")\n\n return check\n\n\nNMP_BASE_URL = os.environ.get("NMP_BASE_URL", "http://localhost:8080")\nsdk = NeMoPlatform(base_url=NMP_BASE_URL, workspace="default")\n" + }, + { + "type": "markdown", + "source": "### 2. Prepare the Preference Dataset\n\nDPO trains on **preference data**. The `rl` backend takes a **single** dataset fileset that holds both `training.jsonl` and `validation.jsonl`, and auto-detects the row schema from the first line. Three preference formats are supported (see the platform's `BinaryPreferenceDatasetItemSchema` / `HelpSteer3DatasetItemSchema` / `Tulu3PreferenceDatasetItemSchema`):", + "source_html": "

2. Prepare the Preference Dataset

\n

DPO trains on preference data. The rl backend takes a single dataset fileset that holds both training.jsonl and validation.jsonl, and auto-detects the row schema from the first line. Three preference formats are supported (see the platform's BinaryPreferenceDatasetItemSchema / HelpSteer3DatasetItemSchema / Tulu3PreferenceDatasetItemSchema):

\n" + }, + { + "type": "markdown", + "source": "#### Binary Preference Format\n\nSimple `prompt` / `chosen` / `rejected` (the `prompt` may be a string or a list of chat messages):\n\n```json\n{\"prompt\": \"What is the capital of France?\", \"chosen\": \"The capital of France is Paris.\", \"rejected\": \"I'm not sure.\"}\n```", + "source_html": "

Binary Preference Format

\n

Simple prompt / chosen / rejected (the prompt may be a string or a list of chat messages):

\n
{"prompt": "What is the capital of France?", "chosen": "The capital of France is Paris.", "rejected": "I'm not sure."}\n
\n" + }, + { + "type": "markdown", + "source": "#### HelpSteer3 Format (used here)\n\nA conversation `context` (string or chat messages), two candidate `response1` / `response2`, and a signed `overall_preference` in -3..3 — **negative** means response 1 is preferred, **positive** means response 2, **0** is a tie. This is the **raw** schema of `nvidia/HelpSteer3`, so no conversion is needed:\n\n```json\n{\"context\": [{\"role\": \"user\", \"content\": \"Explain how to use git rebase\"}], \"response1\": \"...\", \"response2\": \"...\", \"overall_preference\": -2}\n```", + "source_html": "

HelpSteer3 Format (used here)

\n

A conversation context (string or chat messages), two candidate response1 / response2, and a signed overall_preference in -3..3 — negative means response 1 is preferred, positive means response 2, 0 is a tie. This is the raw schema of nvidia/HelpSteer3, so no conversion is needed:

\n
{"context": [{"role": "user", "content": "Explain how to use git rebase"}], "response1": "...", "response2": "...", "overall_preference": -2}\n
\n" + }, + { + "type": "markdown", + "source": "#### Tulu3 Preference Format\n\nFull chat conversations for both the chosen and rejected branches (each a list of messages ending with the assistant turn):\n\n```json\n{\"chosen\": [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"preferred\"}], \"rejected\": [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"dispreferred\"}]}\n```", + "source_html": "

Tulu3 Preference Format

\n

Full chat conversations for both the chosen and rejected branches (each a list of messages ending with the assistant turn):

\n
{"chosen": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "preferred"}], "rejected": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "dispreferred"}]}\n
\n" + }, + { + "type": "markdown", + "source": "#### Download nvidia/HelpSteer3\n\nWe use [nvidia/HelpSteer3](https://huggingface.co/datasets/nvidia/HelpSteer3) (the `preference` subset), NVIDIA's open preference dataset. It ships native `train` and `validation` splits and matches the HelpSteer3 schema above, so we upload the rows **as-is** — the platform's `HelpSteer3Dataset` loader handles the `overall_preference` semantics (including ties) at training time.", + "source_html": "

Download nvidia/HelpSteer3

\n

We use nvidia/HelpSteer3 (the preference subset), NVIDIA's open preference dataset. It ships native train and validation splits and matches the HelpSteer3 schema above, so we upload the rows as-is — the platform's HelpSteer3Dataset loader handles the overall_preference semantics (including ties) at training time.

\n" + }, + { + "type": "code", + "source": "from datasets import load_dataset, Dataset\n\nprint(\"Loading dataset nvidia/HelpSteer3 (preference subset)\")\nds = load_dataset(\"nvidia/HelpSteer3\", \"preference\")\n\n# Small subsets keep the tutorial fast; larger sets train better but take longer.\ntraining_size = 3000\nvalidation_size = 300\nDATASET_NAME = \"dpo-dataset\"\nDATASET_PATH = Path(\"dpo-dataset\").absolute()\nos.makedirs(DATASET_PATH, exist_ok=True)\n\ntrain_dataset = ds[\"train\"]\nvalidation_dataset = ds[\"validation\"]\nassert isinstance(train_dataset, Dataset) and isinstance(validation_dataset, Dataset)\n\n# Save raw HelpSteer3 rows directly — no conversion. The platform detects the\n# HelpSteer3 schema from the row keys (context / response1 / response2 / overall_preference).\ntrain_dataset.select(range(training_size)).to_json(f\"{DATASET_PATH}/training.jsonl\")\nvalidation_dataset.select(range(validation_size)).to_json(f\"{DATASET_PATH}/validation.jsonl\")\n\nprint(f\"Saved training.jsonl ({training_size} rows) and validation.jsonl ({validation_size} rows)\")\nwith open(f\"{DATASET_PATH}/training.jsonl\") as f:\n sample = json.loads(f.readline())\nprint(\"Sample keys:\", sorted(sample.keys()))\nprint(\"overall_preference:\", sample[\"overall_preference\"])", + "language": "python", + "source_html": "from datasets import load_dataset, Dataset\n\nprint("Loading dataset nvidia/HelpSteer3 (preference subset)")\nds = load_dataset("nvidia/HelpSteer3", "preference")\n\n# Small subsets keep the tutorial fast; larger sets train better but take longer.\ntraining_size = 3000\nvalidation_size = 300\nDATASET_NAME = "dpo-dataset"\nDATASET_PATH = Path("dpo-dataset").absolute()\nos.makedirs(DATASET_PATH, exist_ok=True)\n\ntrain_dataset = ds["train"]\nvalidation_dataset = ds["validation"]\nassert isinstance(train_dataset, Dataset) and isinstance(validation_dataset, Dataset)\n\n# Save raw HelpSteer3 rows directly — no conversion. The platform detects the\n# HelpSteer3 schema from the row keys (context / response1 / response2 / overall_preference).\ntrain_dataset.select(range(training_size)).to_json(f"{DATASET_PATH}/training.jsonl")\nvalidation_dataset.select(range(validation_size)).to_json(f"{DATASET_PATH}/validation.jsonl")\n\nprint(f"Saved training.jsonl ({training_size} rows) and validation.jsonl ({validation_size} rows)")\nwith open(f"{DATASET_PATH}/training.jsonl") as f:\n sample = json.loads(f.readline())\nprint("Sample keys:", sorted(sample.keys()))\nprint("overall_preference:", sample["overall_preference"])\n" + }, + { + "type": "markdown", + "source": "### 3. Create FileSet and Upload Preference Data\n\nUpload both JSONL files to a single FileSet so the DPO job can read them.", + "source_html": "

3. Create FileSet and Upload Preference Data

\n

Upload both JSONL files to a single FileSet so the DPO job can read them.

\n" + }, + { + "type": "code", + "source": "try:\n sdk.files.filesets.create(workspace=\"default\", name=DATASET_NAME, description=\"DPO preference data\")\n print(f\"Created fileset: {DATASET_NAME}\")\nexcept ConflictError:\n print(f\"Fileset '{DATASET_NAME}' already exists, continuing...\")\n\nsdk.files.upload(local_path=DATASET_PATH, remote_path=\"\", fileset=DATASET_NAME, workspace=\"default\")\n\nprint(\"Preference data:\")\nprint(json.dumps([f.model_dump() for f in sdk.files.list(fileset=DATASET_NAME, workspace=\"default\").data], indent=2, default=str))", + "language": "python", + "source_html": "try:\n sdk.files.filesets.create(workspace="default", name=DATASET_NAME, description="DPO preference data")\n print(f"Created fileset: {DATASET_NAME}")\nexcept ConflictError:\n print(f"Fileset '{DATASET_NAME}' already exists, continuing...")\n\nsdk.files.upload(local_path=DATASET_PATH, remote_path="", fileset=DATASET_NAME, workspace="default")\n\nprint("Preference data:")\nprint(json.dumps([f.model_dump() for f in sdk.files.list(fileset=DATASET_NAME, workspace="default").data], indent=2, default=str))\n" + }, + { + "type": "markdown", + "source": "### 4. Secrets Setup\n\nThe base model (`meta-llama/Llama-3.2-1B-Instruct`) is gated, so store your Hugging Face token as a platform secret named `hf-token` and reference it on the model fileset.", + "source_html": "

4. Secrets Setup

\n

The base model (meta-llama/Llama-3.2-1B-Instruct) is gated, so store your Hugging Face token as a platform secret named hf-token and reference it on the model fileset.

\n" + }, + { + "type": "code", + "source": "HF_TOKEN = os.getenv(\"HF_TOKEN\")\nif not HF_TOKEN:\n raise RuntimeError(\"Set HF_TOKEN before running this tutorial.\")\n\ndef create_or_get_secret(name: str, value: str, label: str) -> PlatformSecretResponse:\n try:\n secret = sdk.secrets.create(name=name, workspace=\"default\", value=value)\n print(f\"Created secret: {name}\")\n return secret\n except ConflictError:\n print(f\"Secret '{name}' already exists, continuing...\")\n return sdk.secrets.retrieve(name=name, workspace=\"default\")\n\n\nhf_secret = create_or_get_secret(\"hf-token\", HF_TOKEN, \"HF_TOKEN\")", + "language": "python", + "source_html": "HF_TOKEN = os.getenv("HF_TOKEN")\nif not HF_TOKEN:\n raise RuntimeError("Set HF_TOKEN before running this tutorial.")\n\ndef create_or_get_secret(name: str, value: str, label: str) -> PlatformSecretResponse:\n try:\n secret = sdk.secrets.create(name=name, workspace="default", value=value)\n print(f"Created secret: {name}")\n return secret\n except ConflictError:\n print(f"Secret '{name}' already exists, continuing...")\n return sdk.secrets.retrieve(name=name, workspace="default")\n\n\nhf_secret = create_or_get_secret("hf-token", HF_TOKEN, "HF_TOKEN")\n" + }, + { + "type": "markdown", + "source": "### 5. Create Base Model FileSet and Model Entity\n\nDPO starts from an instruction-tuned base model. The model entity's spec is inferred asynchronously after creation.", + "source_html": "

5. Create Base Model FileSet and Model Entity

\n

DPO starts from an instruction-tuned base model. The model entity's spec is inferred asynchronously after creation.

\n" + }, + { + "type": "code", + "source": "HF_REPO_ID = \"meta-llama/Llama-3.2-1B-Instruct\"\nMODEL_NAME = \"llama-3-2-1b-instruct\"\n\nstorage = HuggingfaceStorageConfigParam(\n type=\"huggingface\",\n repo_id=HF_REPO_ID,\n repo_type=\"model\",\n token_secret=hf_secret.name,\n)\n\ntry:\n base_model_fs = sdk.files.filesets.create(\n workspace=\"default\", name=MODEL_NAME, description=\"Llama 3.2 1B Instruct base model\", storage=storage\n )\n print(f\"Created base model fileset: {MODEL_NAME}\")\nexcept ConflictError:\n base_model_fs = sdk.files.filesets.retrieve(workspace=\"default\", name=MODEL_NAME)\n print(\"Base model fileset already exists.\")\n\ntry:\n base_model = sdk.models.create(workspace=\"default\", name=MODEL_NAME, fileset=f\"default/{MODEL_NAME}\")\nexcept ConflictError:\n base_model = sdk.models.retrieve(workspace=\"default\", name=MODEL_NAME)\n\nprint(f\"Base model fileset: fileset://default/{base_model.name}\")\n\n# Wait for the ModelSpec to be inferred from the checkpoint.\ncheck = max_wait_time_checker(600, \"Model spec\")\nwhile not base_model.spec:\n check()\n time.sleep(10)\n base_model = sdk.models.retrieve(workspace=\"default\", name=MODEL_NAME)\nprint(\"Model spec ready\")", + "language": "python", + "source_html": "HF_REPO_ID = "meta-llama/Llama-3.2-1B-Instruct"\nMODEL_NAME = "llama-3-2-1b-instruct"\n\nstorage = HuggingfaceStorageConfigParam(\n type="huggingface",\n repo_id=HF_REPO_ID,\n repo_type="model",\n token_secret=hf_secret.name,\n)\n\ntry:\n base_model_fs = sdk.files.filesets.create(\n workspace="default", name=MODEL_NAME, description="Llama 3.2 1B Instruct base model", storage=storage\n )\n print(f"Created base model fileset: {MODEL_NAME}")\nexcept ConflictError:\n base_model_fs = sdk.files.filesets.retrieve(workspace="default", name=MODEL_NAME)\n print("Base model fileset already exists.")\n\ntry:\n base_model = sdk.models.create(workspace="default", name=MODEL_NAME, fileset=f"default/{MODEL_NAME}")\nexcept ConflictError:\n base_model = sdk.models.retrieve(workspace="default", name=MODEL_NAME)\n\nprint(f"Base model fileset: fileset://default/{base_model.name}")\n\n# Wait for the ModelSpec to be inferred from the checkpoint.\ncheck = max_wait_time_checker(600, "Model spec")\nwhile not base_model.spec:\n check()\n time.sleep(10)\n base_model = sdk.models.retrieve(workspace="default", name=MODEL_NAME)\nprint("Model spec ready")\n" + }, + { + "type": "markdown", + "source": "### 6. Create the DPO Customization Job\n\nSubmit a DPO job to the `rl` backend with `RlJobInput`. Note the DPO-specific shape:\n\n- `model` is a string ref to the model entity; `dataset` is a **single** string ref to the preference fileset (holding both files).\n- The training method is `{\"type\": \"dpo\", ...}` — full-weight, no `finetuning_type`/LoRA.\n- `ref_policy_kl_penalty` is **β** (DPO paper): how strongly the policy stays tied to the reference model.\n- `rl` auto-generates the job id (`rl-`); read it back from the response.\n\nOther configurable knobs: `optimizer_type`, `adam_eps`, `activation_checkpointing`, `keep_top_k`, `val_at_end`, `preference_loss_weight`, `sft_loss_weight`. Run `nemo customization rl explain` for the live schema.", + "source_html": "

6. Create the DPO Customization Job

\n

Submit a DPO job to the rl backend with RlJobInput. Note the DPO-specific shape:

\n
    \n
  • model is a string ref to the model entity; dataset is a single string ref to the preference fileset (holding both files).
    • \n
  • The training method is {"type": "dpo", ...} — full-weight, no finetuning_type/LoRA.
    • \n
  • ref_policy_kl_penalty is β (DPO paper): how strongly the policy stays tied to the reference model.
    • \n
  • rl auto-generates the job id (rl-<hex>); read it back from the response.
    • \n
\n

Other configurable knobs: optimizer_type, adam_eps, activation_checkpointing, keep_top_k, val_at_end, preference_loss_weight, sft_loss_weight. Run nemo customization rl explain for the live schema.

\n" + }, + { + "type": "code", + "source": "job_suffix = uuid.uuid4().hex[:8]\nOUTPUT_NAME = f\"llama-3-2-1b-dpo-{job_suffix}\"\n\nspec = RlJobInput(\n model=f\"default/{base_model.name}\",\n dataset=f\"default/{DATASET_NAME}\",\n training={\n \"type\": \"dpo\",\n \"epochs\": 1,\n \"batch_size\": 16,\n \"micro_batch_size\": 1,\n \"learning_rate\": 5e-6,\n \"max_seq_length\": 4096,\n \"ref_policy_kl_penalty\": 0.1,\n \"parallelism\": {\n \"num_nodes\": 1,\n \"num_gpus_per_node\": 1,\n \"tensor_parallel_size\": 1,\n \"pipeline_parallel_size\": 1,\n },\n },\n output={\"name\": OUTPUT_NAME},\n)\n\n# `rl` auto-generates the job id (rl-); do not pass name=.\njob = sdk.customization.rl.jobs.create(spec=spec, workspace=\"default\")\nprint(f\"Job ID: {job.job.name}\")\nprint(f\"Output model: {OUTPUT_NAME}\")", + "language": "python", + "source_html": "job_suffix = uuid.uuid4().hex[:8]\nOUTPUT_NAME = f"llama-3-2-1b-dpo-{job_suffix}"\n\nspec = RlJobInput(\n model=f"default/{base_model.name}",\n dataset=f"default/{DATASET_NAME}",\n training={\n "type": "dpo",\n "epochs": 1,\n "batch_size": 16,\n "micro_batch_size": 1,\n "learning_rate": 5e-6,\n "max_seq_length": 4096,\n "ref_policy_kl_penalty": 0.1,\n "parallelism": {\n "num_nodes": 1,\n "num_gpus_per_node": 1,\n "tensor_parallel_size": 1,\n "pipeline_parallel_size": 1,\n },\n },\n output={"name": OUTPUT_NAME},\n)\n\n# `rl` auto-generates the job id (rl-<hex>); do not pass name=.\njob = sdk.customization.rl.jobs.create(spec=spec, workspace="default")\nprint(f"Job ID: {job.job.name}")\nprint(f"Output model: {OUTPUT_NAME}")\n" + }, + { + "type": "markdown", + "source": "### 7. Track Training Progress\n\nThe DPO job runs four steps: download -> **dpo-training** (Ray) -> upload -> model-entity. We poll the top-level job status and surface the training step's progress.", + "source_html": "

7. Track Training Progress

\n

The DPO job runs four steps: download -> dpo-training (Ray) -> upload -> model-entity. We poll the top-level job status and surface the training step's progress.

\n" + }, + { + "type": "code", + "source": "from IPython.display import clear_output\n\ncheck = max_wait_time_checker(7200, \"DPO job\")\nwhile True:\n check()\n status = sdk.jobs.get_status(name=job.job.name, workspace=\"default\")\n clear_output(wait=True)\n print(f\"Job Status: {status.status}\")\n\n step = max_steps = phase = None\n for job_step in status.steps or []:\n if job_step.name == \"dpo-training\":\n for task in job_step.tasks or []:\n d = task.status_details or {}\n step, max_steps, phase = d.get(\"step\"), d.get(\"max_steps\"), d.get(\"phase\")\n break\n break\n if step is not None and max_steps:\n print(f\"Training: Step {step}/{max_steps} ({100 * step / max_steps:.1f}%)\")\n if phase:\n print(f\"Phase: {phase}\")\n\n if status.status in (\"completed\", \"failed\", \"cancelled\", \"error\"):\n print(f\"\\nJob finished: {status.status}\")\n break\n time.sleep(15)\n\nassert status.status == \"completed\"", + "language": "python", + "source_html": "from IPython.display import clear_output\n\ncheck = max_wait_time_checker(7200, "DPO job")\nwhile True:\n check()\n status = sdk.jobs.get_status(name=job.job.name, workspace="default")\n clear_output(wait=True)\n print(f"Job Status: {status.status}")\n\n step = max_steps = phase = None\n for job_step in status.steps or []:\n if job_step.name == "dpo-training":\n for task in job_step.tasks or []:\n d = task.status_details or {}\n step, max_steps, phase = d.get("step"), d.get("max_steps"), d.get("phase")\n break\n break\n if step is not None and max_steps:\n print(f"Training: Step {step}/{max_steps} ({100 * step / max_steps:.1f}%)")\n if phase:\n print(f"Phase: {phase}")\n\n if status.status in ("completed", "failed", "cancelled", "error"):\n print(f"\\nJob finished: {status.status}")\n break\n time.sleep(15)\n\nassert status.status == "completed"\n" + }, + { + "type": "markdown", + "source": "**Interpreting DPO training metrics** (in `status_details.metrics`):\n\n- **`loss`** — the DPO loss; should trend down as the policy learns to separate chosen from rejected.\n- **Reward margin** (chosen minus rejected reward) — should trend **up**: the model increasingly prefers chosen responses.\n- **Validation `loss`** — watch for divergence from training loss (overfitting). Raise `ref_policy_kl_penalty` (β) or add `sft_loss_weight` if the policy drifts too far from the reference.", + "source_html": "

Interpreting DPO training metrics (in status_details.metrics):

\n
    \n
  • loss — the DPO loss; should trend down as the policy learns to separate chosen from rejected.
    • \n
  • Reward margin (chosen minus rejected reward) — should trend up: the model increasingly prefers chosen responses.
    • \n
  • Validation loss — watch for divergence from training loss (overfitting). Raise ref_policy_kl_penalty (β) or add sft_loss_weight if the policy drifts too far from the reference.
    • \n
\n" + }, + { + "type": "markdown", + "source": "### 8. Validate the Output Model\n\nDPO produces a **full-weight model entity** (not an adapter). Confirm it was registered.", + "source_html": "

8. Validate the Output Model

\n

DPO produces a full-weight model entity (not an adapter). Confirm it was registered.

\n" + }, + { + "type": "code", + "source": "model_entity = sdk.models.retrieve(workspace=\"default\", name=OUTPUT_NAME)\nprint(model_entity.model_dump_json(indent=2))", + "language": "python", + "source_html": "model_entity = sdk.models.retrieve(workspace="default", name=OUTPUT_NAME)\nprint(model_entity.model_dump_json(indent=2))\n" + }, + { + "type": "markdown", + "source": "### 9. Deploy and Evaluate (optional)\n\nThe DPO output is a full model, so it deploys like any full-weight checkpoint (see the [Full SFT](/documentation/customizer-reference/tutorials/sft-customization-job) tutorial for details). We deploy with vLLM and send a chat completion.", + "source_html": "

9. Deploy and Evaluate (optional)

\n

The DPO output is a full model, so it deploys like any full-weight checkpoint (see the Full SFT tutorial for details). We deploy with vLLM and send a chat completion.

\n" + }, + { + "type": "code", + "source": "deploy_suffix = uuid.uuid4().hex[:8]\nDEPLOYMENT_CONFIG_NAME = f\"dpo-deployment-cfg-{deploy_suffix}\"\nDEPLOYMENT_NAME = f\"dpo-deployment-{deploy_suffix}\"\n\ndeployment_config = sdk.inference.deployment_configs.create(\n workspace=\"default\",\n name=DEPLOYMENT_CONFIG_NAME,\n engine=\"vllm\",\n model_spec={\"model_namespace\": \"default\", \"model_name\": OUTPUT_NAME},\n executor_config={\"gpu\": 1, \"image_name\": \"vllm/vllm-openai\", \"image_tag\": \"v0.22.1\"},\n)\n\ndeployment = sdk.inference.deployments.create(\n workspace=\"default\", name=DEPLOYMENT_NAME, config=deployment_config.name\n)\nprint(f\"Deployment name: {deployment.name}\")", + "language": "python", + "source_html": "deploy_suffix = uuid.uuid4().hex[:8]\nDEPLOYMENT_CONFIG_NAME = f"dpo-deployment-cfg-{deploy_suffix}"\nDEPLOYMENT_NAME = f"dpo-deployment-{deploy_suffix}"\n\ndeployment_config = sdk.inference.deployment_configs.create(\n workspace="default",\n name=DEPLOYMENT_CONFIG_NAME,\n engine="vllm",\n model_spec={"model_namespace": "default", "model_name": OUTPUT_NAME},\n executor_config={"gpu": 1, "image_name": "vllm/vllm-openai", "image_tag": "v0.22.1"},\n)\n\ndeployment = sdk.inference.deployments.create(\n workspace="default", name=DEPLOYMENT_NAME, config=deployment_config.name\n)\nprint(f"Deployment name: {deployment.name}")\n" + }, + { + "type": "code", + "source": "check = max_wait_time_checker(1800, \"Deployment\")\nwhile True:\n check()\n deployment_status = sdk.inference.deployments.retrieve(name=deployment.name, workspace=\"default\")\n clear_output(wait=True)\n print(f\"Deployment status: {deployment_status.status}\")\n deployment_state = str(deployment_status.status).lower()\n if deployment_state in (\"ready\", \"running\"):\n if not sdk.models.wait_for_gateway(deployment.name, workspace=\"default\", timeout=60):\n raise RuntimeError(\"Inference gateway did not become ready\")\n break\n if deployment_state in (\"failed\", \"error\", \"terminated\", \"lost\"):\n raise RuntimeError(f\"Deployment failed with status: {deployment_status.status}\")\n time.sleep(15)", + "language": "python", + "source_html": "check = max_wait_time_checker(1800, "Deployment")\nwhile True:\n check()\n deployment_status = sdk.inference.deployments.retrieve(name=deployment.name, workspace="default")\n clear_output(wait=True)\n print(f"Deployment status: {deployment_status.status}")\n deployment_state = str(deployment_status.status).lower()\n if deployment_state in ("ready", "running"):\n if not sdk.models.wait_for_gateway(deployment.name, workspace="default", timeout=60):\n raise RuntimeError("Inference gateway did not become ready")\n break\n if deployment_state in ("failed", "error", "terminated", "lost"):\n raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")\n time.sleep(15)\n" + }, + { + "type": "code", + "source": "messages = [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Write a short, friendly email to a colleague asking to reschedule our meeting to Thursday.\"},\n]\n\nresponse = sdk.inference.gateway.provider.post(\n \"v1/chat/completions\",\n name=deployment.name,\n workspace=\"default\",\n body={\"model\": f\"default/{OUTPUT_NAME}\", \"messages\": messages, \"temperature\": 0.7, \"max_tokens\": 256},\n)\nprint(\"Model output:\\n\")\nprint(response[\"choices\"][0][\"message\"][\"content\"])", + "language": "python", + "source_html": "messages = [\n {"role": "system", "content": "You are a helpful assistant."},\n {"role": "user", "content": "Write a short, friendly email to a colleague asking to reschedule our meeting to Thursday."},\n]\n\nresponse = sdk.inference.gateway.provider.post(\n "v1/chat/completions",\n name=deployment.name,\n workspace="default",\n body={"model": f"default/{OUTPUT_NAME}", "messages": messages, "temperature": 0.7, "max_tokens": 256},\n)\nprint("Model output:\\n")\nprint(response["choices"][0]["message"]["content"])\n" + }, + { + "type": "markdown", + "source": "## Conclusion\n\nYou aligned a base model with **DPO** on the NeMo Platform using the `rl` backend:\n\n- Uploaded a HelpSteer3 preference dataset **as-is** (the platform detects the schema natively).\n- Submitted a full-weight DPO job that ran on a Ray cluster via the Kubernetes executor.\n- Registered the output as a full model entity and (optionally) deployed it for inference.\n\n**Next steps:** tune the alignment strength with `ref_policy_kl_penalty` (β), add `sft_loss_weight` to anchor the policy to the chosen responses, enable `activation_checkpointing` for memory headroom, or scale up with `parallelism`. See the [Training Configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration) reference for the full hyperparameter set.", + "source_html": "

Conclusion

\n

You aligned a base model with DPO on the NeMo Platform using the rl backend:

\n
    \n
  • Uploaded a HelpSteer3 preference dataset as-is (the platform detects the schema natively).
    • \n
  • Submitted a full-weight DPO job that ran on a Ray cluster via the Kubernetes executor.
    • \n
  • Registered the output as a full model entity and (optionally) deployed it for inference.
    • \n
\n

Next steps: tune the alignment strength with ref_policy_kl_penalty (β), add sft_loss_weight to anchor the policy to the chosen responses, enable activation_checkpointing for memory headroom, or scale up with parallelism. See the Training Configuration reference for the full hyperparameter set.

\n" + } +] }; diff --git a/docs/fern/scripts/validate-notebook-viewer.mjs b/docs/fern/scripts/validate-notebook-viewer.mjs index 15828396de..82d65fc8ef 100644 --- a/docs/fern/scripts/validate-notebook-viewer.mjs +++ b/docs/fern/scripts/validate-notebook-viewer.mjs @@ -28,6 +28,10 @@ const SOURCE_NOTEBOOKS = { ROOT, "../../customizer/tutorials/distillation-customization-job.ipynb", ), + "dpo-customization-job": join( + ROOT, + "../../customizer/tutorials/dpo-customization-job.ipynb", + ), "embedding-customization-job": join( ROOT, "../../customizer/tutorials/embedding-customization-job.ipynb", diff --git a/docs/troubleshooting/customizer.mdx b/docs/troubleshooting/customizer.mdx index ff48a71b43..7ea0b28651 100644 --- a/docs/troubleshooting/customizer.mdx +++ b/docs/troubleshooting/customizer.mdx @@ -14,6 +14,7 @@ description: "" - The platform's shared persistent volume is likely full. Budget against the downloaded base checkpoint size: approximately 3× for Full SFT and 1.5× for LoRA. For example, a 70B BF16 checkpoint is approximately 140 GB, so a Full SFT job can require approximately 420 GB of free disk at peak. - These peak estimates include the base checkpoint and job artifacts; the final Full SFT output itself is one full checkpoint. If you also retain a deployment copy, include it separately in capacity planning. - Clean up completed job artifacts or increase the PVC size (default: 200Gi at `/var/run/scratch/job`). +- DPO/GRPO jobs also consume ephemeral node storage under `/tmp` via Ray workers — check node disk in addition to the PVC. - See [ft-tut-understand-models](/documentation/customizer-reference/tutorials/understanding-models-and-training) for full storage requirement details. **Job fails with OOM (Out of Memory):** diff --git a/plugins/nemo-customizer/README.md b/plugins/nemo-customizer/README.md index 4fdf50b100..927ceda3ce 100644 --- a/plugins/nemo-customizer/README.md +++ b/plugins/nemo-customizer/README.md @@ -1,6 +1,6 @@ # nemo-customizer -Router service for `/apis/customization`. Training backends (Automodel, Unsloth, …) register as **`nemo.customization.contributors`** entry points (discovered via `nemo_platform_plugin.discovery`). +Router service for `/apis/customization`. Training backends (Automodel, RL, Megatron, …) register as **`nemo.customization.contributors`** entry points (discovered via `nemo_platform_plugin.discovery`). Registers **`nemo.sdk`** → `customization` for `client.customization.*` (composes contributor SDK modules such as `client.customization.automodel.jobs`). diff --git a/plugins/nemo-customizer/openapi/openapi.yaml b/plugins/nemo-customizer/openapi/openapi.yaml index 3d203e8ac8..18be8dd9b6 100644 --- a/plugins/nemo-customizer/openapi/openapi.yaml +++ b/plugins/nemo-customizer/openapi/openapi.yaml @@ -392,12 +392,12 @@ paths: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' - /apis/customization/v2/workspaces/{workspace}/unsloth/jobs: + /apis/customization/v2/workspaces/{workspace}/rl/jobs: post: tags: - - Unsloth Jobs + - Rl Jobs summary: Create Job - operationId: create_job_apis_customization_v2_workspaces__workspace__unsloth_jobs_post + operationId: create_job_apis_customization_v2_workspaces__workspace__rl_jobs_post parameters: - name: workspace in: path @@ -410,14 +410,14 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/UnslothJobsJobRequest' + $ref: '#/components/schemas/RlJobsJobRequest' responses: '201': description: Successful Response content: application/json: schema: - $ref: '#/components/schemas/UnslothJobsJob' + $ref: '#/components/schemas/RlJobsJob' '422': description: Validation Error content: @@ -426,9 +426,9 @@ paths: $ref: '#/components/schemas/HTTPValidationError' get: tags: - - Unsloth Jobs + - Rl Jobs summary: List Jobs - operationId: list_jobs_apis_customization_v2_workspaces__workspace__unsloth_jobs_get + operationId: list_jobs_apis_customization_v2_workspaces__workspace__rl_jobs_get parameters: - name: workspace in: path @@ -461,7 +461,7 @@ paths: required: false schema: allOf: - - $ref: '#/components/schemas/UnslothJobsJobsSortField' + - $ref: '#/components/schemas/RlJobsJobsSortField' description: The field to sort by. To sort in decreasing order, use `-` in front of the field name. default: -created_at @@ -473,7 +473,7 @@ paths: required: false explode: true schema: - $ref: '#/components/schemas/UnslothJobsJobsListFilter' + $ref: '#/components/schemas/RlJobsJobsListFilter' description: Filter jobs on various criteria. responses: '200': @@ -481,19 +481,19 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/UnslothJobsJobsPage' + $ref: '#/components/schemas/RlJobsJobsPage' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' - /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{job}/results/{name}: + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{job}/results/{name}: get: tags: - - Unsloth Jobs + - Rl Jobs summary: Get Job Result - operationId: get_job_result_apis_customization_v2_workspaces__workspace__unsloth_jobs__job__results__name__get + operationId: get_job_result_apis_customization_v2_workspaces__workspace__rl_jobs__job__results__name__get parameters: - name: workspace in: path @@ -526,12 +526,12 @@ paths: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' - /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{job}/results/{name}/download: + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{job}/results/{name}/download: get: tags: - - Unsloth Jobs + - Rl Jobs summary: Download Job Result - operationId: download_job_result_apis_customization_v2_workspaces__workspace__unsloth_jobs__job__results__name__download_get + operationId: download_job_result_apis_customization_v2_workspaces__workspace__rl_jobs__job__results__name__download_get parameters: - name: workspace in: path @@ -567,12 +567,12 @@ paths: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' - /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}: + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}: get: tags: - - Unsloth Jobs + - Rl Jobs summary: Get Job - operationId: get_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__get + operationId: get_job_apis_customization_v2_workspaces__workspace__rl_jobs__name__get parameters: - name: workspace in: path @@ -592,7 +592,7 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/UnslothJobsJob' + $ref: '#/components/schemas/RlJobsJob' '422': description: Validation Error content: @@ -601,9 +601,9 @@ paths: $ref: '#/components/schemas/HTTPValidationError' delete: tags: - - Unsloth Jobs + - Rl Jobs summary: Delete Job - operationId: delete_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__delete + operationId: delete_job_apis_customization_v2_workspaces__workspace__rl_jobs__name__delete parameters: - name: workspace in: path @@ -626,12 +626,12 @@ paths: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' - /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/cancel: + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}/cancel: post: tags: - - Unsloth Jobs + - Rl Jobs summary: Cancel Job - operationId: cancel_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__cancel_post + operationId: cancel_job_apis_customization_v2_workspaces__workspace__rl_jobs__name__cancel_post parameters: - name: workspace in: path @@ -651,19 +651,19 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/UnslothJobsJob' + $ref: '#/components/schemas/RlJobsJob' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' - /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/logs: + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}/logs: get: tags: - - Unsloth Jobs + - Rl Jobs summary: Get Job Logs - operationId: get_job_logs_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__logs_get + operationId: get_job_logs_apis_customization_v2_workspaces__workspace__rl_jobs__name__logs_get parameters: - name: workspace in: path @@ -702,12 +702,12 @@ paths: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' - /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/results: + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}/results: get: tags: - - Unsloth Jobs + - Rl Jobs summary: List Job Results - operationId: list_job_results_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__results_get + operationId: list_job_results_apis_customization_v2_workspaces__workspace__rl_jobs__name__results_get parameters: - name: workspace in: path @@ -734,12 +734,12 @@ paths: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' - /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/status: + /apis/customization/v2/workspaces/{workspace}/rl/jobs/{name}/status: get: tags: - - Unsloth Jobs + - Rl Jobs summary: Get Job Status - operationId: get_job_status_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__status_get + operationId: get_job_status_apis_customization_v2_workspaces__workspace__rl_jobs__name__status_get parameters: - name: workspace in: path @@ -766,89 +766,463 @@ paths: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' -components: - schemas: - AutomodelBatchSpec: - properties: - global_batch_size: - type: integer - exclusiveMinimum: 0.0 - title: Global Batch Size - default: 8 - micro_batch_size: + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs: + post: + tags: + - Unsloth Jobs + summary: Create Job + operationId: create_job_apis_customization_v2_workspaces__workspace__unsloth_jobs_post + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJobRequest' + responses: + '201': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + get: + tags: + - Unsloth Jobs + summary: List Jobs + operationId: list_jobs_apis_customization_v2_workspaces__workspace__unsloth_jobs_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: page + in: query + required: false + schema: type: integer - exclusiveMinimum: 0.0 - title: Micro Batch Size + exclusiveMinimum: 0 + description: Page number. default: 1 - sequence_packing: - type: boolean - title: Sequence Packing - default: false - sequence_packing_max_samples: + title: Page + description: Page number. + - name: page_size + in: query + required: false + schema: type: integer - exclusiveMinimum: 0.0 - title: Sequence Packing Max Samples - description: Samples analyzed to estimate the optimal pack size when packing - is enabled. - default: 1000 - additionalProperties: false - type: object - title: AutomodelBatchSpec - AutomodelDatasetSpec: - properties: - training: + exclusiveMinimum: 0 + description: Page size. + default: 10 + title: Page Size + description: Page size. + - name: sort + in: query + required: false + schema: + allOf: + - $ref: '#/components/schemas/UnslothJobsJobsSortField' + description: The field to sort by. To sort in decreasing order, use `-` + in front of the field name. + default: -created_at + description: The field to sort by. To sort in decreasing order, use `-` in + front of the field name. + - in: query + name: filter + style: deepObject + required: false + explode: true + schema: + $ref: '#/components/schemas/UnslothJobsJobsListFilter' + description: Filter jobs on various criteria. + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJobsPage' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{job}/results/{name}: + get: + tags: + - Unsloth Jobs + summary: Get Job Result + operationId: get_job_result_apis_customization_v2_workspaces__workspace__unsloth_jobs__job__results__name__get + parameters: + - name: workspace + in: path + required: true + schema: type: string - title: Training - description: Training fileset as 'name' or 'workspace/name'. - validation: - title: Validation + title: Workspace + - name: job + in: path + required: true + schema: type: string - prompt_template: - title: Prompt Template + title: Job + - name: name + in: path + required: true + schema: type: string - additionalProperties: false - type: object - required: - - training - title: AutomodelDatasetSpec - AutomodelJobInput: - properties: - name: title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobResultResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{job}/results/{name}/download: + get: + tags: + - Unsloth Jobs + summary: Download Job Result + operationId: download_job_result_apis_customization_v2_workspaces__workspace__unsloth_jobs__job__results__name__download_get + parameters: + - name: workspace + in: path + required: true + schema: type: string - model: + title: Workspace + - name: job + in: path + required: true + schema: type: string - title: Model - dataset: - $ref: '#/components/schemas/AutomodelDatasetSpec' - training: - $ref: '#/components/schemas/AutomodelTrainingSpec' - schedule: - $ref: '#/components/schemas/AutomodelScheduleSpec' - batch: - $ref: '#/components/schemas/AutomodelBatchSpec' - optimizer: - $ref: '#/components/schemas/AutomodelOptimizerSpec' - parallelism: - $ref: '#/components/schemas/AutomodelParallelismSpec' - output: - $ref: '#/components/schemas/AutomodelOutputRequest' - integrations: - $ref: '#/components/schemas/IntegrationsSpecInput' - additionalProperties: false - type: object - required: - - model - - dataset - - training - title: AutomodelJobInput - description: POST body / CLI JSON. - AutomodelJobOutput: - properties: - name: - title: Name + title: Job + - name: name + in: path + required: true + schema: type: string - model: + title: Name + responses: + '200': + description: Successful Response + content: + application/octet-stream: + schema: + type: string + format: binary + '404': + description: Not Found + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}: + get: + tags: + - Unsloth Jobs + summary: Get Job + operationId: get_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + delete: + tags: + - Unsloth Jobs + summary: Delete Job + operationId: delete_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__delete + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '204': + description: Successful Response + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/cancel: + post: + tags: + - Unsloth Jobs + summary: Cancel Job + operationId: cancel_job_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__cancel_post + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/UnslothJobsJob' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/logs: + get: + tags: + - Unsloth Jobs + summary: Get Job Logs + operationId: get_job_logs_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__logs_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + - name: limit + in: query + required: false + schema: + title: Limit + type: integer + - name: page_cursor + in: query + required: false + schema: + title: Page Cursor + type: string + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobLogPage' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/results: + get: + tags: + - Unsloth Jobs + summary: List Job Results + operationId: list_job_results_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__results_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobListResultResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /apis/customization/v2/workspaces/{workspace}/unsloth/jobs/{name}/status: + get: + tags: + - Unsloth Jobs + summary: Get Job Status + operationId: get_job_status_apis_customization_v2_workspaces__workspace__unsloth_jobs__name__status_get + parameters: + - name: workspace + in: path + required: true + schema: + type: string + title: Workspace + - name: name + in: path + required: true + schema: + type: string + title: Name + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/PlatformJobStatusResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' +components: + schemas: + AutomodelBatchSpec: + properties: + global_batch_size: + type: integer + exclusiveMinimum: 0.0 + title: Global Batch Size + default: 8 + micro_batch_size: + type: integer + exclusiveMinimum: 0.0 + title: Micro Batch Size + default: 1 + sequence_packing: + type: boolean + title: Sequence Packing + default: false + sequence_packing_max_samples: + type: integer + exclusiveMinimum: 0.0 + title: Sequence Packing Max Samples + description: Samples analyzed to estimate the optimal pack size when packing + is enabled. + default: 1000 + additionalProperties: false + type: object + title: AutomodelBatchSpec + AutomodelDatasetSpec: + properties: + training: + type: string + title: Training + description: Training fileset as 'name' or 'workspace/name'. + validation: + title: Validation + type: string + prompt_template: + title: Prompt Template + type: string + additionalProperties: false + type: object + required: + - training + title: AutomodelDatasetSpec + AutomodelJobInput: + properties: + name: + title: Name + type: string + model: + type: string + title: Model + dataset: + $ref: '#/components/schemas/AutomodelDatasetSpec' + training: + $ref: '#/components/schemas/AutomodelTrainingSpec' + schedule: + $ref: '#/components/schemas/AutomodelScheduleSpec' + batch: + $ref: '#/components/schemas/AutomodelBatchSpec' + optimizer: + $ref: '#/components/schemas/AutomodelOptimizerSpec' + parallelism: + $ref: '#/components/schemas/AutomodelParallelismSpec' + output: + $ref: '#/components/schemas/AutomodelOutputRequest' + integrations: + $ref: '#/components/schemas/IntegrationsSpecInput' + additionalProperties: false + type: object + required: + - model + - dataset + - training + title: AutomodelJobInput + description: POST body / CLI JSON. + AutomodelJobOutput: + properties: + name: + title: Name + type: string + model: type: string title: Model dataset: @@ -1404,6 +1778,23 @@ components: To enable MLflow, provide a non-null ``mlflow`` object on :class:`IntegrationsSpec`.' + OptimizerType: + type: string + enum: + - adamw_with_cosine_annealing + - adam_with_cosine_annealing + - adamw_with_flat_lr + - adam_with_flat_lr + title: OptimizerType + description: Optimizer and scheduler combination types. + OutputNameType: + type: string + enum: + - adapter + - model + title: OutputNameType + description: "Output artifact type \u2014 adapter (LoRA only) or model (merged\ + \ / full)." PaginationData: properties: page: @@ -1450,63 +1841,212 @@ components: timestamp: type: string format: date-time - title: Timestamp - job: - type: string - title: Job - job_step: - type: string - title: Job Step - job_task: - type: string - title: Job Task - message: + title: Timestamp + job: + type: string + title: Job + job_step: + type: string + title: Job Step + job_task: + type: string + title: Job Task + message: + type: string + title: Message + type: object + required: + - timestamp + - job + - job_step + - job_task + - message + title: PlatformJobLog + PlatformJobLogPage: + properties: + data: + items: + $ref: '#/components/schemas/PlatformJobLog' + type: array + title: Data + total: + type: integer + title: Total + next_page: + title: Next Page + type: string + prev_page: + title: Prev Page + type: string + type: object + required: + - data + - total + - next_page + - prev_page + title: PlatformJobLogPage + PlatformJobResultResponse: + properties: + name: + type: string + title: Name + job: + type: string + title: Job + workspace: + type: string + title: Workspace + project: + title: Project + type: string + created_at: + type: string + format: date-time + title: Created At + updated_at: + type: string + format: date-time + title: Updated At + artifact_url: + type: string + title: Artifact Url + artifact_storage_type: + $ref: '#/components/schemas/FileStorageType' + download_url: + title: Download Url + type: string + type: object + required: + - name + - job + - workspace + - artifact_url + - artifact_storage_type + title: PlatformJobResultResponse + PlatformJobStatus: + type: string + enum: + - created + - pending + - active + - cancelled + - cancelling + - error + - completed + - paused + - pausing + - resuming + title: PlatformJobStatus + description: 'Enumeration of possible job statuses. + + + This enum represents the various states a job can be in during its lifecycle, + + from creation to a terminal state.' + PlatformJobStatusResponse: + properties: + id: + type: string + title: Id + name: + type: string + title: Name + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + additionalProperties: true + type: object + title: Status Details + error_details: + title: Error Details + additionalProperties: true + type: object + steps: + items: + $ref: '#/components/schemas/PlatformJobStepStatusResponse' + type: array + title: Steps + created_at: + type: string + format: date-time + title: Created At + updated_at: type: string - title: Message + format: date-time + title: Updated At type: object required: - - timestamp - - job - - job_step - - job_task - - message - title: PlatformJobLog - PlatformJobLogPage: + - id + - name + - status + - status_details + - error_details + - steps + - created_at + - updated_at + title: PlatformJobStatusResponse + PlatformJobStepStatusResponse: properties: - data: + id: + type: string + title: Id + name: + type: string + title: Name + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + additionalProperties: true + type: object + title: Status Details + error_details: + title: Error Details + additionalProperties: true + type: object + tasks: items: - $ref: '#/components/schemas/PlatformJobLog' + $ref: '#/components/schemas/PlatformJobTaskStatusResponse' type: array - title: Data - total: - type: integer - title: Total - next_page: - title: Next Page + title: Tasks + created_at: type: string - prev_page: - title: Prev Page + format: date-time + title: Created At + updated_at: type: string + format: date-time + title: Updated At type: object required: - - data - - total - - next_page - - prev_page - title: PlatformJobLogPage - PlatformJobResultResponse: + - id + - name + - status + - status_details + - error_details + - tasks + - created_at + - updated_at + title: PlatformJobStepStatusResponse + PlatformJobTaskStatusResponse: properties: + id: + type: string + title: Id name: type: string title: Name - job: - type: string - title: Job - workspace: - type: string - title: Workspace - project: - title: Project + status: + $ref: '#/components/schemas/PlatformJobStatus' + status_details: + additionalProperties: true + type: object + title: Status Details + error_details: + title: Error Details + additionalProperties: true + type: object + error_stack: + title: Error Stack type: string created_at: type: string @@ -1516,166 +2056,467 @@ components: type: string format: date-time title: Updated At - artifact_url: + type: object + required: + - id + - name + - status + - status_details + - error_details + - error_stack + - created_at + - updated_at + title: PlatformJobTaskStatusResponse + RlDPOTraining: + properties: + optimizer_type: + allOf: + - $ref: '#/components/schemas/OptimizerType' + description: "Optimizer + LR-scheduler combination (AdamW/Adam \xD7 cosine-annealing/flat-LR).\ + \ Defaults to AdamW with cosine annealing." + learning_rate: + type: number + title: Learning Rate + description: Peak learning rate. + default: 0.0001 + min_learning_rate: + title: Min Learning Rate + description: Minimum LR for cosine decay. + type: number + weight_decay: + type: number + title: Weight Decay + description: Weight decay coefficient. + default: 0.01 + adam_beta1: + type: number + title: Adam Beta1 + description: Adam beta1. + default: 0.9 + adam_beta2: + type: number + title: Adam Beta2 + description: Adam beta2. + default: 0.999 + adam_eps: + type: number + exclusiveMinimum: 0.0 + title: Adam Eps + description: Adam epsilon (numerical stability term). + default: 1.0e-05 + warmup_steps: + type: integer + minimum: 0.0 + title: Warmup Steps + description: Linear warmup steps. + default: 0 + epochs: + type: integer + exclusiveMinimum: 0.0 + title: Epochs + description: Number of passes through the dataset. + default: 1 + max_steps: + title: Max Steps + description: Max training steps (overrides epochs if set). + type: integer + exclusiveMinimum: 0.0 + val_check_interval: + title: Val Check Interval + description: Validation interval. Float <= 1.0 is fraction of epoch; > 1.0 + is step count. + type: number + val_at_end: + type: boolean + title: Val At End + description: Run a final validation pass after the last training step. Keep + enabled so the final checkpoint carries validation metrics and best-checkpoint + selection works; set False only to skip the extra eval. + default: true + keep_top_k: + type: integer + exclusiveMinimum: 0.0 + title: Keep Top K + description: Number of best checkpoints to retain (ranked by validation + loss). + default: 1 + batch_size: + type: integer + exclusiveMinimum: 0.0 + title: Batch Size + description: Global batch size across all GPUs. + default: 32 + micro_batch_size: + type: integer + exclusiveMinimum: 0.0 + title: Micro Batch Size + description: Per-GPU micro batch size. + default: 1 + activation_checkpointing: + type: boolean + title: Activation Checkpointing + description: Recompute activations during the backward pass to reduce memory + at the cost of compute. Enable to fit larger models or longer sequences. + default: false + max_seq_length: + type: integer + exclusiveMinimum: 0.0 + title: Max Seq Length + description: Maximum token sequence length for training. + default: 2048 + seed: + title: Seed + description: Random seed for reproducibility. + type: integer + parallelism: + $ref: '#/components/schemas/RlParallelismParams' + execution_profile: + title: Execution Profile + description: Execution profile for the GPU training step (operator-configured). + Falls back to the service default when omitted. type: string - title: Artifact Url - artifact_storage_type: - $ref: '#/components/schemas/FileStorageType' - download_url: - title: Download Url + minLength: 1 + type: + type: string + const: dpo + title: Type + default: dpo + ref_policy_kl_penalty: + type: number + minimum: 0.0 + title: Ref Policy Kl Penalty + description: KL penalty coefficient (beta in the DPO paper). + default: 0.05 + preference_average_log_probs: + type: boolean + title: Preference Average Log Probs + description: Average log probabilities for preference loss calculation. + default: false + sft_average_log_probs: + type: boolean + title: Sft Average Log Probs + description: Average log probabilities for SFT regularization loss. + default: false + preference_loss_weight: + type: number + minimum: 0.0 + title: Preference Loss Weight + description: Weight for the preference (DPO) loss term. + default: 1.0 + sft_loss_weight: + type: number + minimum: 0.0 + title: Sft Loss Weight + description: Weight for SFT regularization loss (0 = disabled). + default: 0.0 + max_grad_norm: + type: number + minimum: 0.0 + title: Max Grad Norm + description: Maximum gradient norm for clipping. + default: 1.0 + additionalProperties: false + type: object + title: RlDPOTraining + description: "Direct Preference Optimization (full-weight only \u2014 PEFT unsupported)." + RlJobInput: + properties: + name: + title: Name + type: string + model: + type: string + title: Model + description: Model entity reference ('name' or 'workspace/name'). + dataset: + type: string + title: Dataset + description: Preference dataset fileset reference. Must contain training.jsonl + + validation.jsonl. + training: + allOf: + - $ref: '#/components/schemas/RlDPOTraining' + description: DPO training method and hyperparameters. + integrations: + $ref: '#/components/schemas/IntegrationsSpecInput' + output: + $ref: '#/components/schemas/RlOutputRequest' + additionalProperties: false + type: object + required: + - model + - dataset + - training + title: RlJobInput + description: POST body / CLI JSON for ``nemo customization rl submit``. + RlJobOutput: + properties: + name: + title: Name + description: Optional job name; auto-generated when omitted. + type: string + model: + type: string + title: Model + description: Model entity reference ('name' or 'workspace/name'). + dataset: type: string + title: Dataset + description: Preference dataset fileset reference ('name' or 'workspace/name'). + training: + allOf: + - $ref: '#/components/schemas/RlDPOTraining' + description: Training method and hyperparameters (DPO). + integrations: + allOf: + - $ref: '#/components/schemas/IntegrationsSpecOutput' + description: W&B / MLflow integrations. + output: + allOf: + - $ref: '#/components/schemas/RlOutputResponse' + description: Output artifact created by this job. + additionalProperties: false type: object required: - - name - - job - - workspace - - artifact_url - - artifact_storage_type - title: PlatformJobResultResponse - PlatformJobStatus: - type: string - enum: - - created - - pending - - active - - cancelled - - cancelling - - error - - completed - - paused - - pausing - - resuming - title: PlatformJobStatus - description: 'Enumeration of possible job statuses. + - model + - dataset + - training + - output + title: RlJobOutput + description: 'Canonical NeMo-RL job spec (output of the plugin transform). - This enum represents the various states a job can be in during its lifecycle, + The ``dataset`` fileset must contain ``training.jsonl`` and ``validation.jsonl`` - from creation to a terminal state.' - PlatformJobStatusResponse: + (any of the four supported preference formats); the dataset-preparation step + + splits/normalizes them at runtime.' + RlJobsJob: properties: id: - type: string title: Id + type: string name: type: string title: Name + description: + title: Description + type: string + project: + title: Project + type: string + workspace: + title: Workspace + type: string + created_at: + title: Created At + type: string + format: date-time + updated_at: + title: Updated At + type: string + format: date-time + spec: + $ref: '#/components/schemas/RlJobOutput' status: $ref: '#/components/schemas/PlatformJobStatus' status_details: + title: Status Details additionalProperties: true type: object - title: Status Details error_details: title: Error Details additionalProperties: true type: object - steps: - items: - $ref: '#/components/schemas/PlatformJobStepStatusResponse' - type: array - title: Steps - created_at: - type: string - format: date-time - title: Created At - updated_at: - type: string - format: date-time - title: Updated At + ownership: + title: Ownership + additionalProperties: true + type: object + custom_fields: + title: Custom Fields + additionalProperties: true + type: object type: object required: - - id - name - - status - - status_details - - error_details - - steps - - created_at - - updated_at - title: PlatformJobStatusResponse - PlatformJobStepStatusResponse: + - spec + title: RlJobsJob + RlJobsJobRequest: properties: - id: - type: string - title: Id name: - type: string title: Name - status: - $ref: '#/components/schemas/PlatformJobStatus' - status_details: + type: string + description: + title: Description + type: string + project: + title: Project + type: string + spec: + $ref: '#/components/schemas/RlJobInput' + ownership: + title: Ownership additionalProperties: true type: object - title: Status Details - error_details: - title: Error Details + custom_fields: + title: Custom Fields additionalProperties: true type: object - tasks: - items: - $ref: '#/components/schemas/PlatformJobTaskStatusResponse' - type: array - title: Tasks + output_location: + title: Output Location + type: string + type: object + required: + - spec + title: RlJobsJobRequest + RlJobsJobsListFilter: + additionalProperties: false + properties: created_at: + allOf: + - $ref: '#/components/schemas/DatetimeFilter' + description: Jobs created at 'gte' datetime or 'lte' datetime. + name: + anyOf: + - $ref: '#/components/schemas/StringFilter' + - type: string + description: Name of the job. + title: Name + workspace: + description: Workspace of the job. + title: Workspace type: string - format: date-time - title: Created At + project: + description: Project containing the job. + title: Project + type: string + status: + allOf: + - $ref: '#/components/schemas/PlatformJobStatus' + description: The current status. updated_at: + allOf: + - $ref: '#/components/schemas/DatetimeFilter' + description: Jobs updated at 'gte' datetime or 'lte' datetime. + title: RlJobsJobsListFilter + type: object + RlJobsJobsPage: + properties: + data: + items: + $ref: '#/components/schemas/RlJobsJob' + type: array + title: Data + pagination: + allOf: + - $ref: '#/components/schemas/PaginationData' + description: Pagination information. + sort: + title: Sort + description: The field on which the results are sorted. type: string - format: date-time - title: Updated At + filter: + title: Filter + description: Filtering information. + additionalProperties: true + type: object type: object required: - - id - - name - - status - - status_details - - error_details - - tasks + - data + title: RlJobsJobsPage + RlJobsJobsSortField: + type: string + enum: - created_at + - -created_at - updated_at - title: PlatformJobStepStatusResponse - PlatformJobTaskStatusResponse: + - -updated_at + title: RlJobsJobsSortField + RlOutputRequest: properties: - id: - type: string - title: Id name: - type: string title: Name - status: - $ref: '#/components/schemas/PlatformJobStatus' - status_details: - additionalProperties: true - type: object - title: Status Details - error_details: - title: Error Details - additionalProperties: true - type: object - error_stack: - title: Error Stack type: string - created_at: + additionalProperties: false + type: object + title: RlOutputRequest + description: Submitter-facing output preferences. ``name`` is auto-derived if + omitted. + RlOutputResponse: + properties: + name: type: string - format: date-time - title: Created At - updated_at: + maxLength: 255 + title: Name + description: Name of the output artifact. Used to identify it during deployment + and inference. + examples: + - my-dpo-llama + type: + allOf: + - $ref: '#/components/schemas/OutputNameType' + description: Output artifact type. DPO is full-weight, so always `model`. + default: model + fileset: type: string - format: date-time - title: Updated At + maxLength: 255 + title: Fileset + description: FileSet name where output artifacts are stored. + additionalProperties: false type: object required: - - id - name - - status - - status_details - - error_details - - error_stack - - created_at - - updated_at - title: PlatformJobTaskStatusResponse + - fileset + title: RlOutputResponse + description: Resolved output artifact details. + RlParallelismParams: + properties: + num_gpus_per_node: + type: integer + exclusiveMinimum: 0.0 + title: Num Gpus Per Node + description: Number of GPUs per node. + default: 1 + num_nodes: + type: integer + exclusiveMinimum: 0.0 + title: Num Nodes + description: "Number of nodes (>1 \u2192 multi-node Ray cluster)." + default: 1 + tensor_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Tensor Parallel Size + description: Tensor parallel size. + default: 1 + pipeline_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Pipeline Parallel Size + description: Pipeline parallel size. + default: 1 + context_parallel_size: + type: integer + exclusiveMinimum: 0.0 + title: Context Parallel Size + description: Context parallel size. + default: 1 + sequence_parallel: + type: boolean + title: Sequence Parallel + description: Enable sequence parallelism. + default: false + additionalProperties: false + type: object + title: RlParallelismParams + description: 'Distributed training parallelism configuration. + + + Single-node multi-GPU uses ``num_nodes=1`` with ``num_gpus_per_node>1``; + + multi-node sets ``num_nodes>1`` and the compiler emits a distributed-GPU + + executor (see :mod:`nmp.rl.app.jobs.compiler`).' SecretRef: type: string pattern: ^[a-z0-9_-]+(/[a-z0-9_-]+)?$ diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md index 03c0f3c1c9..a1ced8a4f3 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md @@ -1,11 +1,12 @@ --- name: nemo-customizer description: >- - Fine-tune models on NeMo Platform with `automodel` or `unsloth` (both + Fine-tune models on NeMo Platform with `automodel`, `unsloth`, or `rl` (all `submit`-only): HF dataset conversion, filesets, model entities, and job JSON (hyperparameters, batch, schedule, optimizer) + job polling. `automodel`/`unsloth` - run SFT/LoRA as Docker GPU jobs. Use for train, fine-tune, customize, SFT, LoRA, - learning rate, epochs, or nemo customization. + run SFT/LoRA as Docker GPU jobs; `rl` runs DPO (preference optimization) on a Ray + cluster (Kubernetes). Use for train, fine-tune, customize, SFT, LoRA, DPO, + preference optimization, learning rate, epochs, or nemo customization. triggers: - nemo-customizer - nemo customizer @@ -16,14 +17,21 @@ triggers: - customize a model - sft - lora + - dpo + - direct preference optimization + - preference optimization + - preference tuning - automodel - unsloth + - nemo-rl + - nemo rl - nemo customization - nemo-customization - customizer - customization training - automodel submit - unsloth submit + - rl submit not-for: - nemo-build-agent (agent scaffold/deploy, not weight training) - nemo-explore (agent design only) @@ -40,16 +48,17 @@ allowed-tools: [Bash, Read, Grep] # NeMo Customizer -End-to-end **SFT + LoRA** (automodel/unsloth) on NeMo Platform. Two backend plugins ship in this repo — both are **`submit`-only** (local `run` is hard-disabled on each): +End-to-end **SFT + LoRA** (automodel/unsloth) and **DPO** (rl) on NeMo Platform. Three backend plugins ship in this repo — all are **`submit`-only** (local `run` is hard-disabled on each): | Backend | Verb | Trains | Where it runs | Pick when | |---------|------|--------|---------------|-----------| | **`automodel`** (default) | `submit` | SFT / LoRA | Platform **Docker GPU executor** (Jobs service schedules containers on the platform host's daemon) | General SFT/LoRA; multi-GPU (data/tensor parallel); distillation; full-weight SFT | | **`unsloth`** | `submit` | SFT / LoRA | Same — Docker GPU job with 4 steps (download → train → upload → model-entity) | User asks for Unsloth, or wants Unsloth's 4-bit LoRA path / optimizer defaults on a single GPU | +| **`rl`** | `submit` | **DPO** (preference) | Platform **Kubernetes executor** — provisions a **Ray** cluster; 4 steps (download → DPO train → upload → model-entity) | Preference optimization / DPO / RLHF-style alignment from a `{prompt, chosen, rejected}` dataset; full-weight only | -`nemo-customizer` is the router (`nemo customization …`); training backends are separate plugins (`nemo-automodel`, `nemo-unsloth`). `submit` posts to the platform API; the platform runs training in container steps — **not** in the CLI shell. Heavy ML deps live in container images only. +`nemo-customizer` is the router (`nemo customization …`); training backends are separate plugins (`nemo-automodel`, `nemo-unsloth`, `nemo-rl`). `submit` posts to the platform API; the platform runs training in container steps — **not** in the CLI shell. Heavy ML deps live in container images only. -**Runtime split:** `automodel`/`unsloth` need `platform.runtime: docker`. +**Runtime split:** `automodel`/`unsloth` need `platform.runtime: docker`; `rl` needs `platform.runtime: kubernetes` (no local Docker fallback — it schedules a Ray cluster on the remote cluster). A given platform is usually one or the other — confirm with execution profiles before picking `rl`. Decision rule below in **Plugin pick**. Batch shell work; reuse resources with `--exist-ok`; skip CLI `--help` unless a command fails. @@ -103,12 +112,15 @@ Full create/update commands, fileset `token_secret`, license acceptance, and dow ## Plugin pick 1. Run `nemo jobs list-execution-profiles -f json` (login first only if auth is enabled — see **Authentication**; see `references/troubleshooting.md` for parsing). -2. If the user explicitly asked for Unsloth → **`unsloth`**. -3. Else if the user explicitly asked for Automodel → **`automodel`**. -4. Else if any profile has `provider: gpu` or `gpu_distributed` → **`automodel`** (default, SFT/LoRA). -5. Else stop and tell the user GPU customization is unavailable (all backends need a GPU execution profile and `platform.runtime: docker`). +2. If the task is **DPO / preference optimization** (a `{prompt, chosen, rejected}` dataset, "align", "preference", "RLHF-style") **or** the user explicitly asked for NeMo-RL → **`rl`** (requires a GPU profile **and** `platform.runtime: kubernetes`). +3. Else if the user explicitly asked for Unsloth → **`unsloth`**. +4. Else if the user explicitly asked for Automodel → **`automodel`**. +5. Else if any profile has `provider: gpu` or `gpu_distributed` → **`automodel`** (default, SFT/LoRA). +6. Else stop and tell the user GPU customization is unavailable (all backends need a GPU execution profile; `automodel`/`unsloth` also need `platform.runtime: docker`, `rl` needs `platform.runtime: kubernetes`). -For **`automodel`/`unsloth`**, training never runs inside the `nemo` CLI process. After `submit`, the platform's **local Docker executor** launches GPU container steps on the daemon attached to that platform host (often the same machine as `http://127.0.0.1:8080`, but always query the platform — not the agent's shell GPU or a separate `docker info` on another box). +**`rl` runtime gate:** `rl submit` fails fast unless the platform runs `platform.runtime: kubernetes` (`require_distributed_runtime`). rl job steps execute as **Kubernetes pods via the `kubernetes_job` execution backend** — the **`docker` job backend cannot run rl**. Before submitting rl, confirm with `nemo jobs list-execution-profiles -f json` that the `cpu`/`gpu` profiles report `backend: kubernetes_job` (or `volcano_job`). If they report `backend: docker`/`subprocess`, the platform is **not** configured for rl: stop and tell the user DPO needs a Kubernetes-runtime platform — do **not** start/reuse a docker-runtime platform, and do **not** fall back to automodel/unsloth (those are SFT/LoRA, not DPO). To stand up or configure one, see `references/rl-kubernetes-runtime.md`. + +For **`automodel`/`unsloth`**, training never runs inside the `nemo` CLI process. After `submit`, the platform's **local Docker executor** launches GPU container steps on the daemon attached to that platform host (often the same machine as `http://127.0.0.1:8080`, but always query the platform — not the agent's shell GPU or a separate `docker info` on another box). **`rl` does not use the Docker executor** — its steps run on the Kubernetes cluster the platform is configured against. ## Gotchas @@ -127,9 +139,10 @@ For **`automodel`/`unsloth`**, training never runs inside the `nemo` CLI process ``` Poll until healthy (`curl -sf http://127.0.0.1:8080/health/ready` or retry `nemo jobs list-execution-profiles -f json`), then continue the workflow. Do not start services without asking. -- **All backends are `submit` only** — `nemo customization run …` hard-fails with a pointer to `submit` (automodel and unsloth each disable local `run`). Do not improvise verbs or pass `--venv`. -- **Never set `max_steps` together with `epochs`** (automodel + unsloth). `max_steps` is a global cap and stops mid-epoch. Test fixtures include `max_steps` for smoke tests — do not copy into production jobs. Unsloth's schema enforces this as a hard mutex; automodel allows both but the result is surprising. -- **Job done (all backends) = top-level `status`** in `completed` | `error` | `cancelled`. Steps can all be `completed` while the job is still `active` (upload, entity registration). `status_details.phase` may stay `training` with `progress_pct: 100` for a long time — keep polling. `poll_customization_job.sh` works for any job id (`automodel-…` or `unsloth-…`); it exits **1** on `error` or `cancelled`. + - ⚠️ **This default start is a DOCKER-runtime platform — valid for `automodel`/`unsloth` only.** It is **NOT** valid for **`rl`**: rl needs `platform.runtime: kubernetes` with a `kubernetes_job` execution backend. Starting this default and submitting rl will fail the runtime gate. For rl, configure/point at a Kubernetes-runtime platform instead — see `references/rl-kubernetes-runtime.md`. Never start or reuse a docker-runtime platform for rl. +- **All backends are `submit` only** — `nemo customization run …` hard-fails with a pointer to `submit` (automodel, unsloth, and rl each disable local `run`). Do not improvise verbs or pass `--venv`. +- **Never set `max_steps` together with `epochs`** (automodel + unsloth; rl has the same caveat — see **rl (DPO) gotchas**). `max_steps` is a global cap and stops mid-epoch. Test fixtures include `max_steps` for smoke tests — do not copy into production jobs. Unsloth's schema enforces this as a hard mutex; automodel allows both but the result is surprising. +- **Job done (all backends) = top-level `status`** in `completed` | `error` | `cancelled`. Steps can all be `completed` while the job is still `active` (upload, entity registration). `status_details.phase` may stay `training` with `progress_pct: 100` for a long time — keep polling. `poll_customization_job.sh` works for any job id (`automodel-…`, `unsloth-…`, or `rl-…`); it exits **1** on `error` or `cancelled`. - Model spec fills async: **submit without polling** `nemo models get` unless submit fails. - HF dataset id from the user → convert locally; do not ask for local paths first. - Dataset fileset name = HF dataset **name** only (`tau/commonsense_qa` → `commonsense_qa`), not the model name. @@ -140,12 +153,22 @@ For **`automodel`/`unsloth`**, training never runs inside the `nemo` CLI process - **Unsloth validation defaults** — when `dataset.validation_path` is set and `schedule.eval_steps` is omitted, the trainer runs validation once per effective epoch automatically. Report final `metrics.val_loss` from job status (see `references/reporting.md`). Set `eval_steps` explicitly to override cadence. - **Do not use local `docker info`** to pick automodel vs unsloth. Run `nemo jobs list-execution-profiles -f json` against the user's platform (login first only if auth is enabled — see **Authentication**; see `references/troubleshooting.md`). Default output is a table — **`-f json` is required** for scripting; parse **stdout only** (do not pipe `2>&1` into `json.load`). - **Do not merge stderr into stdout when parsing JSON** — `submit`, `explain`, and `-f json` commands write **JSON on stdout**; harmless warnings like `Configuration file not found, using defaults` go to **stderr**. Piping with **`2>&1`** before `json.load` raises `JSONDecodeError` even when submit **succeeded** — a common cause of **duplicate jobs** when the agent re-submits after a parse error. Parse stdout only; redirect stderr if needed (`2>/dev/null`). See `references/troubleshooting.md` § **Parsing CLI JSON**. -- For submit/image/plugin errors (both backends), read `references/troubleshooting.md`. Unsloth needs the `nmp-unsloth-training` container image on the **platform host's** Docker daemon (see `docker/unsloth/README.md`). +- For submit/image/plugin errors (all backends), read `references/troubleshooting.md`. Unsloth needs the `nmp-unsloth-training` container image on the **platform host's** Docker daemon (see `docker/unsloth/README.md`); rl needs the `nmp-customizer-tasks` / `nmp-rl-training` images on the Kubernetes cluster (see **rl (DPO) gotchas** and `references/rl-kubernetes-runtime.md`). - **Missing training image on a remote platform** — if the user gave a non-localhost `NMP_BASE_URL` and the job errors with `Failed to pull image`, `manifest unknown`, or missing `nmp-unsloth-training` / automodel training image: **do not** run `docker build`, `docker pull`, or `docker buildx bake` on the agent machine. Report with the template in `references/reporting.md` (use **Output adapter fileset (planned):** on error), then append on-target build steps from `references/troubleshooting.md` § **Missing training images**. - **Gated HuggingFace models** (Llama, Gemma, …) — confirm `hf-token` + fileset `token_secret` before submit; download fails with `Failed to access upstream storage` / 502 when missing. See **HuggingFace token (gated models)** and `references/troubleshooting.md` § **Gated HuggingFace models**. - **Post-training eval format** — use the same CHAT `messages` JSONL as training. **Do not** flatten rows to `prompt`/`expected` for the evaluator. Send `messages[:-1]` at inference (exclude final assistant label); score against `messages[-1].content`. See `references/post-training-eval.md` and `references/eval_helpers.py`. - **LoRA adapters load automatically for eval** — when a LoRA job completes (`save_method: lora`), the adapter is registered on the base model entity and hot-reloaded on any **READY** deployment with `lora_enabled: true`. **Do not** create or update deployments before LoRA eval. **Full SFT** (`finetuning_type: all_weights`) and **merged checkpoints** (`merged_16bit` / `merged_4bit`) register a new **model** entity at `output.name` — **deploy that entity for inference** before chat or eval; full weights are not hot-reloaded onto the base deployment. For LoRA eval, route through the **provider** gateway (`/provider//-/v1` with `model: default--`); the model-entity path (`/model//-/v1`) always hits the base model. See `references/post-training-eval.md` § **Request routing (base vs LoRA)**. +### rl (DPO) gotchas + +- **rl is DPO, not SFT** — it trains on **preference pairs** `{prompt, chosen, rejected}`, full-weight (no LoRA/adapter; `finetuning_type` is not user-set). Don't route SFT/LoRA work here, and don't route DPO to automodel/unsloth. +- **One preference fileset, two files** — `dataset` is a **single string** ref to a fileset that holds **both** `training.jsonl` and `validation.jsonl` (uploaded with `--remote-path`). Unlike automodel (`dataset.training`/`dataset.validation`) and unsloth (`dataset.path`/`validation_path`), there is no separate validation ref. See `references/dataset-formats.md` § NeMo-RL. +- **String refs** — `model` and `dataset` are plain strings (`"workspace/name"`), not objects. The training method goes under `training` with `type: "dpo"`. +- **Kubernetes job backend, not Docker** — rl steps run as Kubernetes pods via the `kubernetes_job` backend; the docker job backend cannot run rl. `rl submit` fails fast on a docker-runtime platform. The target cluster must have the **job-step images** (`nmp-customizer-tasks`, `nmp-rl-training`), the **jobs-launcher** image (the per-step init container), and a **job-storage PVC**. Verify the platform with `nemo jobs list-execution-profiles -f json` (expect `backend: kubernetes_job`); to configure one, see `references/rl-kubernetes-runtime.md`. Multi-node (`parallelism.num_nodes > 1`) also needs the platform-side `NMP_RL_MULTINODE_SHARED_STORAGE_PATH` (shared FS for Ray coordination) or compile fails fast. +- **Job id prefix is `rl-`** and the platform auto-generates it — `rl submit` has **no `--name` flag** (the job JSON `name` is the *output* name, not the job id). Read the job id from the `"name"` field in **submit stdout** (JSON), same as automodel/unsloth; `poll_customization_job.sh rl-` works. **Do not** pick the newest `rl-*` from `nemo jobs list` — a concurrent job or an earlier failed submit selects the wrong one. If submit stdout could not be parsed, stop and re-check rather than guessing a job id. +- **DPO main knob is `ref_policy_kl_penalty`** (β). For OOM, enable `activation_checkpointing: true` first. Full DPO field reference: `references/hyperparameters-rl.md`. +- **`max_steps` + `epochs`** — same caveat as the other backends: `max_steps` caps mid-epoch; it's in the smoke fixture (`plugins/nemo-rl/tests/fixtures/minimal_dpo.json`) — omit for real runs. + ## Workflow Common steps then **branch by plugin pick**: @@ -157,7 +180,7 @@ Common steps then **branch by plugin pick**: - [ ] nemo jobs list-execution-profiles -f json — apply Plugin pick rules above (retry login on 401/403) - [ ] On connection error: default URL → ask to start platform (see Platform unreachable); custom URL → report unreachable and stop - [ ] Convert HF dataset → /tmp/train-data/*.jsonl (see references/hf-conversion.md) -- [ ] Create dataset fileset (--exist-ok), upload train.jsonl (+ validation.jsonl), nemo files list to verify +- [ ] Create dataset fileset (--exist-ok), upload the JSONL files, nemo files list to verify — automodel/unsloth: train.jsonl (+ validation.jsonl); rl: training.jsonl + validation.jsonl (see rl branch) - [ ] Gated HF base model? → confirm `hf-token` exists; ask user and stop if missing (see HuggingFace token + troubleshooting § Gated HuggingFace models) - [ ] Create HF weights fileset + model entity if missing (--exist-ok; gated repos need `token_secret` on fileset — see troubleshooting) @@ -174,6 +197,15 @@ Common steps then **branch by plugin pick**: - [ ] Poll until top-level terminal (`poll_customization_job.sh unsloth-`; default 15s interval) - [ ] Report using the template in `references/reporting.md` - [ ] Optional: compare base vs adapter on validation — `references/eval_helpers.py …` (LoRA only; CHAT format; adapters hot-reload automatically; see `references/post-training-eval.md`) + +# rl branch (DPO; submit → Kubernetes/Ray job) — requires platform.runtime: kubernetes +- [ ] Verify execution backend: `nemo jobs list-execution-profiles -f json` shows cpu/gpu at `backend: kubernetes_job` (NOT docker/subprocess). If not → stop; do not start a docker platform; configure per references/rl-kubernetes-runtime.md +- [ ] Dataset is PREFERENCE data: upload training.jsonl + validation.jsonl ({prompt,chosen,rejected}) to ONE fileset +- [ ] Write /tmp/job.json using the RlJobInput shape (see Fast path — rl (DPO)) +- [ ] nemo customization rl submit /tmp/job.json --workspace default [--profile ] +- [ ] Read job id from the "name" field in submit stdout (JSON) — submit has no --name flag; do NOT pick the newest rl-* from `nemo jobs list` +- [ ] Poll until top-level terminal (`poll_customization_job.sh rl-`; default 15s interval) +- [ ] Report using the template in `references/reporting.md` ``` ## Fast path — automodel @@ -297,6 +329,54 @@ Read `` from the `"name"` field in submit stdout (JSON). **Do not use `2 If you try `nemo customization unsloth run …`, the CLI hard-fails with a pointer to `submit`. +## Fast path — rl (DPO) + +DPO on a Ray cluster — **Kubernetes runtime only**, full-weight. **Before anything else**, confirm the platform dispatches jobs to Kubernetes: `nemo jobs list-execution-profiles -f json` must show `cpu`/`gpu` at `backend: kubernetes_job` (not `docker`/`subprocess`). If it doesn't, stop — do not start/use a docker-runtime platform; configure a Kubernetes-runtime one per `references/rl-kubernetes-runtime.md`. Model-entity setup (step 2) is identical to automodel; the dataset is **preference data** and the job JSON is the `RlJobInput` shape. + +**1. Preference dataset** — rows are `{prompt, chosen, rejected}` (see `references/dataset-formats.md` § NeMo-RL). Upload **both** files to **one** fileset: + +```bash +DATASET= # e.g. dpo-data +nemo files filesets create "$DATASET" --workspace default --purpose dataset --exist-ok +nemo files upload /tmp/dpo-train.jsonl "$DATASET" --workspace default --remote-path training.jsonl +nemo files upload /tmp/dpo-val.jsonl "$DATASET" --workspace default --remote-path validation.jsonl +nemo files list "$DATASET" --workspace default +``` + +**2. Model** — same as automodel Fast path step 2 (HF weights fileset + model entity; gated repos need `token_secret`). + +**3. Job JSON** — write `/tmp/job.json`. `model` and `dataset` are **strings**; the method is under `training` with `type: "dpo"`. Full field reference: `references/hyperparameters-rl.md`. + +```json +{ + "model": "default/", + "dataset": "default/", + "training": { + "type": "dpo", + "epochs": 1, + "learning_rate": 5e-6, + "max_seq_length": 1024, + "batch_size": 32, + "micro_batch_size": 1, + "ref_policy_kl_penalty": 0.05, + "parallelism": { "num_nodes": 1, "num_gpus_per_node": 1 } + }, + "output": { "name": "" } +} +``` + +**4. Submit and poll** — `rl submit` has **no `--name` flag** (the platform auto-generates the `rl-` job id), so read it from submit stdout: + +```bash +nemo customization rl submit /tmp/job.json --workspace default > /tmp/rl-submit.json # add --profile if the default gpu profile is wrong +JOB=$(python3 -c "import json;print(json.load(open('/tmp/rl-submit.json'))['name'])") +bash plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/scripts/poll_customization_job.sh "$JOB" +``` + +**Do not** derive the job id by picking the newest `rl-*` from `nemo jobs list` — a concurrent job or an earlier failed submit selects the wrong one. If `/tmp/rl-submit.json` does not parse, the submit result is unknown: stop and inspect it (re-submitting risks a duplicate job). + +**Do not use `2>&1`** before `json.load` — warnings on stderr break parsing; see Gotchas. Or poll manually: `nemo jobs get-status rl-` every 30–60s. If submit fails on an unknown profile, re-list execution profiles and pass `--profile `. `nemo customization rl run …` is disabled (no local execution — it provisions a Ray cluster); `nemo customization rl explain` prints the live schema. + ## Defaults Shared: @@ -328,9 +408,28 @@ Unsloth-specific: | Output | `save_method: "lora"` (adapter-only) unless user asks for merged checkpoint | | Gradient checkpointing | `training.use_gradient_checkpointing: "unsloth"` | +rl-specific (DPO): + +| Field | Value | +|-------|-------| +| Training | DPO, full-weight (`type: "dpo"`; no LoRA) | +| Model (if user gives none) | `Qwen/Qwen3-0.6B` | +| Dataset (if user gives none) | `nvidia/HelpSteer3` (preference subset; uploaded raw — see `references/dataset-formats.md` § NeMo-RL) | +| Schedule (if user gives none) | small demo run: `max_steps` 20 (completes fast, proves the pipeline). For a real run, set `epochs` and **omit** `max_steps`. | +| Parallelism | 1 node, 1 GPU (`parallelism.num_nodes`/`num_gpus_per_node`) | +| Batch | `batch_size` 32, `micro_batch_size` 1 | +| Optimizer | `learning_rate` 5e-6 (DPO uses a low LR), AdamW + cosine | +| DPO | `ref_policy_kl_penalty` (β) 0.05, `sft_loss_weight` 0.0 | +| Max sequence length | 1024 | +| Output | full-weight model entity (`output.name`); no adapter | + +When the user asks for a DPO job **without specifics**, default to the above: a +20-step run of `Qwen/Qwen3-0.6B` on `nvidia/HelpSteer3` — small enough to finish +quickly and confirm the pipeline end-to-end. + ## Batch sizing -`micro_batch_size` / `global_batch_size` (automodel) and `per_device_train_batch_size` × `gradient_accumulation_steps` (unsloth) on **≥48 GB GPUs**, multi-GPU (data vs tensor parallel), and OOM / throughput tuning live in **`references/batch-sizing.md`**. On unknown VRAM the **Defaults** above are safe — read batch-sizing before raising batch on a known ≥48 GB card. +`micro_batch_size` / `global_batch_size` (automodel) and `per_device_train_batch_size` × `gradient_accumulation_steps` (unsloth) on **≥48 GB GPUs**, multi-GPU (data vs tensor parallel), and OOM / throughput tuning live in **`references/batch-sizing.md`**. On unknown VRAM the **Defaults** above are safe — read batch-sizing before raising batch on a known ≥48 GB card. rl (DPO) batch knobs (`batch_size` / `micro_batch_size`) are in `references/hyperparameters-rl.md`. ## Worked example @@ -338,29 +437,46 @@ Unsloth-specific: **Unsloth:** same model + dataset + entity + fileset, but `nemo customization unsloth submit /tmp/job.json -w default`. Job JSON ≤4B row: `batch.per_device_train_batch_size` 8, `batch.gradient_accumulation_steps` 16 (effective 128), `learning_rate` `1e-4`, `hardware.gpus` `"0"`, `output.save_method` `"lora"`. Poll `unsloth-` to completion. Reference fixture: `plugins/nemo-unsloth/tests/fixtures/minimal_unsloth_sft.json` (ignore `max_steps` for real runs). +**rl (DPO):** the no-details default — `Qwen/Qwen3-0.6B` + `nvidia/HelpSteer3` (preference subset, uploaded raw), output `qwen3-0.6b-dpo`. First confirm `kubernetes_job` backend (see **Plugin pick** → rl runtime gate). Upload `training.jsonl` + `validation.jsonl` to one fileset, register the model entity, then submit a **small 20-step demo** job: + +```json +{ + "model": "default/qwen3-0.6b", + "dataset": "default/helpsteer3-dpo", + "training": { "type": "dpo", "max_steps": 20, "batch_size": 32, "micro_batch_size": 1, + "learning_rate": 5e-6, "max_seq_length": 1024, "ref_policy_kl_penalty": 0.05, + "parallelism": { "num_nodes": 1, "num_gpus_per_node": 1 } }, + "output": { "name": "qwen3-0.6b-dpo" } +} +``` + +`nemo customization rl submit /tmp/job.json -w default`, derive the `rl-` id (submit has no `--name`), poll to completion. Reference fixture: `plugins/nemo-rl/tests/fixtures/minimal_dpo.json`. For a real run, replace `max_steps: 20` with `epochs`. + ## Report to user -After polling reaches a **terminal** status (`completed`, `error`, or `cancelled`), report using the template in **`references/reporting.md`** — one format for all backends. It covers the **Fine-tune result** header, the **Training configuration** table (with per-backend examples: automodel, unsloth), and **Using the adapter** (automodel/unsloth LoRA) vs **Using the fine-tuned model** (full SFT / merged), plus metrics extraction, notes by status, `/tmp` report saving, and error follow-ups. +After polling reaches a **terminal** status (`completed`, `error`, or `cancelled`), report using the template in **`references/reporting.md`** — one format for all backends. It covers the **Fine-tune result** header, the **Training configuration** table (with per-backend examples: automodel, unsloth, rl/DPO), and **Using the adapter** (automodel/unsloth LoRA) vs **Using the fine-tuned model** (full SFT / merged / rl DPO), plus metrics extraction, notes by status, `/tmp` report saving, and error follow-ups. ## Reference files | When | Read | |------|------| | HF conversion or MCQA shaping | `references/hf-conversion.md` | -| CHAT vs SFT vs CUSTOM (automodel); text vs messages (unsloth) | `references/dataset-formats.md` | -| Field glossary, full JSON template, distillation/KD, live-schema pointers (index routes per backend) | `references/hyperparameters.md` → `hyperparameters-automodel.md` / `hyperparameters-unsloth.md` | +| CHAT vs SFT vs CUSTOM (automodel); text vs messages (unsloth); preference triples (rl/DPO) | `references/dataset-formats.md` | +| Field glossary, full JSON template, distillation/KD, live-schema pointers (index routes per backend) | `references/hyperparameters.md` → `hyperparameters-automodel.md` / `hyperparameters-unsloth.md` / `hyperparameters-rl.md` | | Batch sizing (≥48 GB), OOM / throughput (automodel + unsloth) | `references/batch-sizing.md` | | Multi-GPU same node | `references/batch-sizing.md` § **Multi-GPU (same node)** (unsloth is single-GPU) | | Reporting: result template, Training configuration, Using the adapter / fine-tuned model | `references/reporting.md` | | Backend choice, execution profiles, submit failure, container images, missing image on remote platform, gated HF auth / download 502, CLI, connection errors | `references/troubleshooting.md` (§ **Parsing CLI JSON** for `2>&1` / `json.load`; § **Gated HuggingFace models** for `hf-token`) | -| Live JSON schema | `uv run nemo customization automodel explain` / `uv run nemo customization unsloth explain` | +| rl (DPO) needs Kubernetes job execution — verifying / configuring `runtime: kubernetes` + `kubernetes_job` executors (local platform → remote cluster, launcher image, PVC, loopback) | `references/rl-kubernetes-runtime.md` | +| Live JSON schema | `uv run nemo customization automodel explain` / `uv run nemo customization unsloth explain` / `uv run nemo customization rl explain` | | Job JSON fixture (automodel, minimal) | `plugins/nemo-automodel/tests/fixtures/qwen3_0.6b_sft_lora.json` (ignore `max_steps` for real runs) | | Job JSON fixture (unsloth, minimal) | `plugins/nemo-unsloth/tests/fixtures/minimal_unsloth_sft.json` (ignore `max_steps` for real runs) | -| Job JSON fixture — integrations (W&B / MLflow) | automodel: `plugins/nemo-automodel/tests/fixtures/integrations_wandb_mlflow.json` · unsloth: `plugins/nemo-unsloth/tests/fixtures/integrations_wandb_mlflow.json` | +| Job JSON fixture (rl / DPO, minimal) | `plugins/nemo-rl/tests/fixtures/minimal_dpo.json` (ignore `max_steps` for real runs) | +| Job JSON fixture — integrations (W&B / MLflow) | automodel: `plugins/nemo-automodel/tests/fixtures/integrations_wandb_mlflow.json` · unsloth: `plugins/nemo-unsloth/tests/fixtures/integrations_wandb_mlflow.json` · rl: `plugins/nemo-rl/tests/fixtures/integrations_wandb_mlflow.json` | | Automodel compile-path contract configs | `services/automodel/tests/contract/input_configs/` → YAML in `output_configs/` (legacy `TrainingStepConfig` shape, not submit JSON) | | W&B / MLflow field reference (all backends) | `references/hyperparameters.md` § **Integrations (all backends)** | | W&B secret + MLflow local server + jobs-launcher | `references/integrations-setup.md` | | Gated HF model auth (`hf-token`, fileset `token_secret`) | `references/troubleshooting.md` § **Gated HuggingFace models** | | Post-training eval (base vs LoRA, CHAT format parity) | `references/post-training-eval.md`, `references/eval_helpers.py` | -Related: `plugins/nemo-automodel/README.md`, `plugins/nemo-unsloth/README.md`, `plugins/nemo-customizer/docs/CUSTOMIZATION.md`, skills **`nemo-files`**, **`nemo-status`**, **`nemo-secrets`**. +Related: `plugins/nemo-automodel/README.md`, `plugins/nemo-unsloth/README.md`, `plugins/nemo-rl/README.md`, `docs/customizer/nemo-rl-dpo-plugin-design.md`, `plugins/nemo-customizer/docs/CUSTOMIZATION.md`, skills **`nemo-files`**, **`nemo-status`**, **`nemo-secrets`**. diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md index 4beef5b0a5..87cd83a4b6 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/dataset-formats.md @@ -1,10 +1,11 @@ # Dataset formats -All backends read JSONL from a platform fileset, but the **row shape and the job-JSON dataset block differ**. Pick the section that matches your plugin. +All three backends read JSONL from a platform fileset, but the **row shape and the job-JSON dataset block differ**. Pick the section that matches your plugin (automodel, unsloth, or rl/DPO). -Upload the JSONL files at the **fileset root**, then reference the fileset from the job JSON `dataset` block: +Upload the JSONL files at the **fileset root**, then reference the fileset from the job JSON `dataset` block. The filenames differ by backend, so keep the two contracts separate: - **SFT (automodel, unsloth):** upload `train.jsonl` and optional `validation.jsonl`. Automodel points `dataset.training` / `dataset.validation` at the fileset; unsloth uses `dataset.path` (and `dataset.validation_path`). +- **rl (DPO):** upload both `training.jsonl` **and** `validation.jsonl` to a single fileset, referenced by one `dataset` string (no separate validation ref) — see § NeMo-RL. ## Automodel @@ -68,3 +69,51 @@ Eval rows must use the **same CHAT `messages` shape** as training. Do not flatte LoRA inference and eval use the **provider** gateway on the **base** entity (`/provider//-/v1`, `model: default--`). Base model uses the model-entity path. Full SFT / merged checkpoints use the **output** model entity's model-entity URL — deploy first. See `post-training-eval.md` and the **Using the adapter** / **Using the fine-tuned model** sections in `reporting.md`. Shared helpers and compare CLI: `references/eval_helpers.py`. Full workflow: `references/post-training-eval.md`. + +## NeMo-RL (DPO) — preference data + +DPO trains on **preference pairs**, not prompt→completion examples. The `rl` backend takes a **single** dataset fileset that must contain **both** `training.jsonl` **and** `validation.jsonl` at the fileset root (unlike automodel/unsloth, the dataset block in the job JSON is a single ref — there is no separate validation ref). + +The dataset-preparation step **auto-detects the row schema from the first line** and selects the matching NeMo-RL loader. **Three preference formats are supported** (platform schemas `BinaryPreferenceDatasetItemSchema` / `HelpSteer3DatasetItemSchema` / `Tulu3PreferenceDatasetItemSchema`): + +### Binary preference (`BinaryPreferenceDataset`) + +Simple `prompt` / `chosen` / `rejected` — the `prompt` may be a plain string **or** a list of chat messages: + +```json +{"prompt": "What is the capital of France?", "chosen": "The capital of France is Paris.", "rejected": "I'm not sure."} +``` + +| Key | Meaning | +|-----|---------| +| `prompt` | The input/context shown to the model (string or list of chat messages). | +| `chosen` | The preferred (higher-reward) response. | +| `rejected` | The dispreferred response. | + +### HelpSteer3 (`HelpSteer3`) + +A conversation `context` (string or chat messages), two candidate responses, and a signed `overall_preference` in **-3..3** — **negative** means `response1` is preferred, **positive** means `response2`, **0** is a tie. This is the **raw** schema of `nvidia/HelpSteer3` (the `preference` subset), so no conversion is needed: + +```json +{"context": [{"role": "user", "content": "Explain how to use git rebase"}], "response1": "...", "response2": "...", "overall_preference": -2} +``` + +### Tulu3 preference (`Tulu3Preference`) + +Full chat conversations for both branches — `chosen` and `rejected` are each a **list of messages** ending with the assistant turn: + +```json +{"chosen": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "preferred"}], "rejected": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "dispreferred"}]} +``` + +Whichever format you upload, the job JSON dataset block is just the single fileset ref: + +```json +"dataset": "default/" +``` + +**Notes** +- Upload both files to the **same** fileset (`--remote-path training.jsonl` and `--remote-path validation.jsonl`). +- `prompt` may be a plain string; the model's chat template is applied at training time (override with `training.chat_template` only when needed). +- DPO is **full-weight** — there is no LoRA/adapter dataset variant. The output is a full model checkpoint. + diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.md index 27778a232d..e9f045aa60 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.md +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hf-conversion.md @@ -55,7 +55,7 @@ Then upload (see main skill). Validate with `nemo files list --wo ## Mapping to job JSON -Only the **chat-template / `messages` (`to_chat`) output is backend-agnostic** — that JSONL feeds both SFT backends (automodel + unsloth). The `to_sft` (`prompt` / `completion`) shape is **automodel-only**; unsloth needs a `to_text` rendering instead (see below). Either way, the **dataset block in job JSON is shaped per backend**. +Only the **chat-template / `messages` (`to_chat`) output is backend-agnostic** — that JSONL feeds both SFT backends (automodel + unsloth). The `to_sft` (`prompt` / `completion`) shape is **automodel-only**; unsloth needs a `to_text` rendering instead (see below). Either way, the **dataset block in job JSON is shaped per backend**. (rl/DPO uses preference data, not this SFT output — see `dataset-formats.md` § NeMo-RL.) | Backend | Row format used | Dataset block in job JSON | |---------|----------------|---------------------------| diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.md new file mode 100644 index 0000000000..958b79694c --- /dev/null +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters-rl.md @@ -0,0 +1,83 @@ + + +# NeMo-RL (DPO) job JSON + +The `rl` backend (`nemo customization rl submit`) runs **DPO** (Direct Preference Optimization) on a Ray cluster — **Kubernetes runtime only**, full-weight (no LoRA). Schema: `RlJobInput` in `plugins/nemo-rl/src/nemo_rl_plugin/schema.py`. Run `nemo customization rl explain` for the live schema. + +## Job JSON layout + +```json +{ + "model": "default/", + "dataset": "default/", + "training": { + "type": "dpo", + "epochs": 1, + "learning_rate": 5e-6, + "max_seq_length": 1024, + "batch_size": 32, + "micro_batch_size": 1, + "ref_policy_kl_penalty": 0.05, + "parallelism": { "num_nodes": 1, "num_gpus_per_node": 1 } + }, + "output": { "name": "" } +} +``` + +- `model` is a **string** ref to a registered model entity (`"name"` or `"workspace/name"`) — not an object (unsloth) and not the HF id. +- `dataset` is a **single string** ref to a preference fileset containing `training.jsonl` + `validation.jsonl` (see `references/dataset-formats.md` § NeMo-RL). There is no separate validation ref. +- `output.name` is the full-weight model entity the job registers. DPO output type is always `model` (no adapter). + +## Field reference — `training` (DPOTraining) + +### General (shared training knobs) + +| Field | Default | Notes | +|-------|---------|-------| +| `learning_rate` | `1e-4` | Peak LR. DPO typically uses a **low** LR (e.g. `5e-6`–`1e-5`). | +| `min_learning_rate` | `null` | Floor for cosine decay. | +| `weight_decay` | `0.01` | | +| `adam_beta1` / `adam_beta2` | `0.9` / `0.999` | Adam betas. | +| `adam_eps` | `1e-5` | Adam epsilon (numerical stability). | +| `warmup_steps` | `0` | Linear warmup steps. | +| `optimizer_type` | `null` → `adamw_with_cosine_annealing` | One of `adamw_with_cosine_annealing`, `adam_with_cosine_annealing`, `adamw_with_flat_lr`, `adam_with_flat_lr` (optimizer × LR-scheduler). | +| `epochs` | `1` | Passes over the dataset. | +| `max_steps` | `null` | Global step cap. Caps the run at `min(max_steps, epochs × steps_per_epoch)`, so it's safe to combine with `epochs` to stop smoke jobs mid-epoch — omit for real runs. | +| `val_check_interval` | `null` | Float ≤ 1.0 = fraction of epoch; > 1.0 = step count. | +| `val_at_end` | `true` | Run a final validation pass after the last step. Keep enabled: it makes the final checkpoint carry validation metrics so best-checkpoint selection (`metric_name`/`keep_top_k`) works — otherwise NeMo-RL warns and falls back to the latest checkpoint. Set `false` only to skip the extra eval. | +| `keep_top_k` | `1` | Number of best checkpoints to retain (ranked by validation loss). | +| `batch_size` | `32` | Global batch (preference **pairs**) across all GPUs. | +| `micro_batch_size` | `1` | Per-GPU micro batch. | +| `max_seq_length` | `2048` | Max token sequence length. | +| `activation_checkpointing` | `false` | Recompute activations in the backward pass to cut memory — the first knob to enable for OOM / larger models / longer sequences. | +| `seed` | `null` → `42` | | +| `execution_profile` | `null` | GPU execution profile; falls back to the service default. | + +### DPO-specific + +| Field | Default | Notes | +|-------|---------|-------| +| `ref_policy_kl_penalty` | `0.05` | **β** in the DPO paper — strength of the KL penalty tying the policy to the reference model. Higher = stay closer to the reference. The main DPO knob. | +| `preference_loss_weight` | `1.0` | Weight on the preference (DPO) loss term. | +| `sft_loss_weight` | `0.0` | Weight on an auxiliary SFT regularization loss (`0` = pure DPO). Raise (e.g. `0.1`) to anchor the policy to the chosen responses. | +| `preference_average_log_probs` | `false` | Normalize preference log-probs by sequence length. | +| `sft_average_log_probs` | `false` | Normalize SFT-loss log-probs by sequence length. | +| `max_grad_norm` | `1.0` | Gradient clipping norm. | + +### `parallelism` + +Same block as automodel (`num_nodes`, `num_gpus_per_node`, `tensor_parallel_size`, `pipeline_parallel_size`, `context_parallel_size`, `sequence_parallel`). Divisibility rule (enforced by `RlJobOutput.validate_for_training`): `total_gpus = num_nodes × num_gpus_per_node` must be divisible by `tensor_parallel_size × pipeline_parallel_size × context_parallel_size`, and `batch_size` by `micro_batch_size × data_parallel_size`. **Multi-node (`num_nodes > 1`)** additionally requires the platform to set `NMP_RL_MULTINODE_SHARED_STORAGE_PATH` (shared filesystem for Ray's cross-node coordination); the compiler fails fast otherwise. + +## Integrations (W&B / MLflow) + +rl supports **W&B and MLflow** through the top-level `integrations` object (`integrations.wandb` / `integrations.mlflow`) — the same object shape used across all backends; full field reference in `hyperparameters.md` § **Integrations (all backends)**. rl specifics: the run name defaults to the **job id**, tags are auto-prefixed (`service:rl`, `framework:…`), and because rl runs on Kubernetes / Ray the `tracking_uri` / self-hosted W&B `base_url` must be reachable **from the cluster** (the local `docker0` recipe in `integrations-setup.md` is Docker-runtime only). + +## DPO tuning guide + +| Symptom | Action | +|---------|--------| +| Policy degenerates / drifts too far | Raise `ref_policy_kl_penalty` (β), e.g. `0.05` → `0.1`–`0.5`. | +| Barely changes from the reference | Lower β, or raise `learning_rate` one step (still keep it low for DPO). | +| Forgets base capabilities | Add SFT regularization: `sft_loss_weight` `0.1`–`0.5`. | +| CUDA OOM | Set `activation_checkpointing: true`; then lower `micro_batch_size`; then `max_seq_length`. | + diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.md index 5a722beae4..3ffa7c3b42 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.md +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/hyperparameters.md @@ -1,15 +1,16 @@ # Hyperparameters -Two backend job schemas live in this skill. Each backend has its own field reference file — **pick by plugin**: +Three backend job schemas live in this skill. Each backend has its own field reference file — **pick by plugin**: | Plugin | Schema class | Schema dump | Field reference | |--------|--------------|-------------|-----------------| | `automodel` | `AutomodelJobInput` (`plugins/nemo-automodel/src/nemo_automodel_plugin/schema.py`) | `nemo customization automodel explain` | **`hyperparameters-automodel.md`** | | `unsloth` | `UnslothJobInput` (`plugins/nemo-unsloth/src/nemo_unsloth_plugin/schema.py`) | `nemo customization unsloth explain` | **`hyperparameters-unsloth.md`** | +| `rl` (DPO) | `RlJobInput` (`plugins/nemo-rl/src/nemo_rl_plugin/schema.py`) | `nemo customization rl explain` | **`hyperparameters-rl.md`** | -Both schemas use `extra="forbid"` — unknown keys raise validation errors. Field names are **not** interchangeable across backends (e.g. automodel uses `micro_batch_size` / `global_batch_size` / `parallelism`; unsloth uses `per_device_train_batch_size` / `gradient_accumulation_steps` / `hardware`). Use the right schema for the chosen plugin. +All three schemas use `extra="forbid"` — unknown keys raise validation errors. Field names are **not** interchangeable across backends (e.g. automodel uses `micro_batch_size` / `global_batch_size` / `parallelism`; unsloth uses `per_device_train_batch_size` / `gradient_accumulation_steps` / `hardware`; rl uses `batch_size` / `micro_batch_size` under `training` and takes `model` / `dataset` as plain strings). Use the right schema for the chosen plugin. -**Batch sizing, 48 GB VRAM tables, multi-GPU (data parallel vs tensor parallel), and throughput tuning** live in **`batch-sizing.md`** (automodel + unsloth). These per-backend files are the **field glossary**, full JSON template per backend, and distillation/KD (automodel) — not the place to pick batch sizes for production runs. +**Batch sizing, 48 GB VRAM tables, multi-GPU (data parallel vs tensor parallel), and throughput tuning** live in **`batch-sizing.md`** (automodel + unsloth). These per-backend files are the **field glossary**, full JSON template per backend, distillation/KD (automodel), and DPO knobs (rl) — not the place to pick batch sizes for production runs. ## Table of contents @@ -17,15 +18,16 @@ Both schemas use `extra="forbid"` — unknown keys raise validation errors. Fiel |----------------|-----| | **`hyperparameters-automodel.md`** | Automodel job JSON layout, full template, `training` / `schedule` / `batch` / `optimizer` / `parallelism` field reference, LR & LoRA-rank tuning, presets, distillation/KD | | **`hyperparameters-unsloth.md`** | Unsloth job JSON layout, full template, `model` / `dataset` / `training` / `schedule` / `batch` / `optimizer` / `hardware` / `output` field reference, LR & LoRA-rank tuning, save-method picker | +| **`hyperparameters-rl.md`** | NeMo-RL (DPO) job JSON layout, `training` (DPOTraining) field reference — shared knobs + DPO-specific (`ref_policy_kl_penalty` = β, `sft_loss_weight`), `parallelism`, DPO tuning guide | | **`batch-sizing.md`** | ≥48 GB VRAM batch tables, multi-GPU (data vs tensor parallel), OOM / throughput tuning (automodel + unsloth) | -| **Integrations** (below) | W&B / MLflow `integrations` object — both backends (automodel, unsloth) | +| **Integrations** (below) | W&B / MLflow `integrations` object — all three backends (automodel, unsloth, rl) | | **Source of truth** (below) | Schema source files, compiler mappings, fixtures per backend | --- ## Integrations (all backends) -**Both backends** (automodel, unsloth) accept the same `integrations` object on job JSON (`IntegrationsSpec` in `nemo_platform_plugin.integrations`) — **W&B** and **MLflow**. A non-null `wandb` / `mlflow` block **requests** that integration; the training runtime **activates** it only when credentials/URIs are available (W&B needs `WANDB_API_KEY`, MLflow needs a tracking URI). Omit the field or set a block to `null` to disable. There is no `enabled` flag and no `report_to` on input — `report_to` is derived at runtime from activated integrations. The compiler logs a warning when W&B is requested without `api_key_secret` or MLflow without `tracking_uri`. +**All three backends** (automodel, unsloth, rl) accept the same `integrations` object on job JSON (`IntegrationsSpec` in `nemo_platform_plugin.integrations`) — **W&B** and **MLflow**. A non-null `wandb` / `mlflow` block **requests** that integration; the training runtime **activates** it only when credentials/URIs are available (W&B needs `WANDB_API_KEY`, MLflow needs a tracking URI). Omit the field or set a block to `null` to disable. There is no `enabled` flag and no `report_to` on input — `report_to` is derived at runtime from activated integrations. The compiler logs a warning when W&B is requested without `api_key_secret` or MLflow without `tracking_uri`. ```json "integrations": { @@ -63,10 +65,12 @@ Both schemas use `extra="forbid"` — unknown keys raise validation errors. Fiel | `mlflow.name` | MLflow run name; defaults to job ID. Legacy `run_name` is accepted with a deprecation warning. | | `mlflow.tags` / `mlflow.description` | Optional run metadata. | -Set `"integrations": null` or omit the field when tracking is not needed. Fixtures per backend: automodel → `plugins/nemo-automodel/tests/fixtures/integrations_wandb_mlflow.json`; unsloth → `plugins/nemo-unsloth/tests/fixtures/integrations_wandb_mlflow.json`. +Set `"integrations": null` or omit the field when tracking is not needed. Fixtures per backend: automodel → `plugins/nemo-automodel/tests/fixtures/integrations_wandb_mlflow.json`; unsloth → `plugins/nemo-unsloth/tests/fixtures/integrations_wandb_mlflow.json`; rl → `plugins/nemo-rl/tests/fixtures/integrations_wandb_mlflow.json`. **Local setup (MLflow server, `docker0` tracking URI, jobs-launcher, W&B secret) — Docker-runtime (automodel / unsloth):** `references/integrations-setup.md`. +**rl (DPO) note:** rl supports **W&B and MLflow** through this object exactly like automodel and unsloth. Two rl specifics: the run name defaults to the **job id** (stable across pause/resume) and NeMo-RL auto-adds tags (`service:rl`, `framework:…`, plus workspace / job / task / model); and because rl runs on **Kubernetes / Ray** (not the Docker executor), point `tracking_uri` and any self-hosted W&B `base_url` at an endpoint **reachable from the cluster** — the `docker0` local-MLflow recipe above is Docker-runtime only. (NeMo-RL's TensorBoard / SwanLab logger slots aren't exposed via `integrations`, same as the other backends — `IntegrationsSpec` carries only `wandb` + `mlflow`.) + **Unsloth note:** HuggingFace `TrainingArguments.run_name` is shared by W&B and MLflow. When both backends are active, `wandb.name` wins if set; otherwise `mlflow.name` is used. If both names are set to different values, a runtime warning is logged and W&B's name is used. --- @@ -87,3 +91,9 @@ Set `"integrations": null` or omit the field when tracking is not needed. Fixtur | JSON example (unsloth) | `plugins/nemo-unsloth/tests/fixtures/minimal_unsloth_sft.json` | Smoke-test template (ignore `max_steps` for real runs) | | Full spec doc (automodel) | `plugins/nemo-automodel/SCOPE.md` (simplified JSON section) | Design notes | | Plugin README (unsloth) | `plugins/nemo-unsloth/README.md` | Submit-only CLI, 4-step container job, GPU selection | +| Submit schema (rl / DPO) | `plugins/nemo-rl/src/nemo_rl_plugin/schema.py` | Allowed JSON fields (`RlJobInput` / `DPOTraining`) | +| Canonical schema (rl / DPO) | `services/rl/src/nmp/rl/schemas.py` | Post-transform shape (`RlJobOutput`); divisibility validator | +| DPO config builder (rl) | `services/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_config.py` | Field → NeMo-RL YAML mapping | +| JSON fixture (rl / DPO) | `plugins/nemo-rl/tests/fixtures/minimal_dpo.json` | Minimal template (ignore `max_steps` for real runs) | +| Plugin README (rl / DPO) | `plugins/nemo-rl/README.md` | Submit-only CLI, Kubernetes/Ray runtime, constraints | +| Plugin design doc (rl / DPO) | `docs/customizer/nemo-rl-dpo-plugin-design.md` | Architecture, 4-step job, image split | diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.md index 6893365ab2..6e2cda6411 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.md +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/integrations-setup.md @@ -1,6 +1,6 @@ # Integrations setup (W&B + MLflow, local / Docker platform) -Use this when job JSON includes `integrations.wandb` and/or `integrations.mlflow` on a **local or single-node Docker** NeMo Platform (`platform.runtime: docker`) — i.e. **automodel / unsloth**. Field reference: `hyperparameters.md` § **Integrations (all backends)**. +Use this when job JSON includes `integrations.wandb` and/or `integrations.mlflow` on a **local or single-node Docker** NeMo Platform (`platform.runtime: docker`) — i.e. **automodel / unsloth**. Field reference: `hyperparameters.md` § **Integrations (all backends)**. rl (DPO) accepts the same `integrations` block but runs on **Kubernetes / Ray**: reuse the field reference, but point `tracking_uri` / self-hosted W&B `base_url` at an endpoint reachable from the cluster (the `docker0` recipe below is Docker-runtime only). ## MLflow — local tracking server diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.md index a4c26025e8..f9cf8a7816 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.md +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/reporting.md @@ -1,38 +1,40 @@ # Report to user -After polling reaches a **terminal** status (`completed`, `error`, or `cancelled`), report using this template for **both** backends (automodel, unsloth). Fill fields from the job JSON and `nemo jobs get-status`. +After polling reaches a **terminal** status (`completed`, `error`, or `cancelled`), report using this template for **all** backends (automodel, unsloth, rl). Fill fields from the job JSON and `nemo jobs get-status`. ## Result template ```markdown ## Fine-tune result -- **Job:** -- **Backend:** +- **Job:** +- **Backend:** - **Model entity:** default/ - **Dataset fileset:** default/ -- **Output fileset:** +- **Output fileset:** - **Status:** - **Final train loss:** - **Final validation loss:** - **Notes:** ``` +For **rl (DPO)** the output is a **full-weight model entity** (no adapter): label the line **Output model entity**, **skip Using the adapter**, and use **Using the fine-tuned model** (below) — confirm with `nemo models get --workspace default`. The DPO loss series lands in `status_details.metrics` like the other backends. + ## Field guidance | Field | Source | |-------|--------| -| **Job** | Job id from submit or poll (`automodel-…` / `unsloth-…`) | +| **Job** | Job id from submit or poll (`automodel-…` / `unsloth-…` / `rl-…`) | | **Backend** | Plugin used for submit | -| **Model entity** | `model` in job JSON (automodel: string ref; unsloth: `model.name`) | -| **Dataset fileset** | automodel: `dataset.training`; unsloth: `dataset.path` | +| **Model entity** | `model` in job JSON (automodel & rl: string ref; unsloth: `model.name`) | +| **Dataset fileset** | automodel: `dataset.training`; unsloth: `dataset.path`; rl: `dataset` (single preference-fileset string) | | **Output adapter fileset** | `output.name` from job JSON. Label **Output adapter fileset (planned):** when status is `error` or `cancelled` and no output was registered | | **Status** | Top-level `status` from `nemo jobs get-status` — not step-level status | | **Final train loss** | Last entry in `status_details.metrics.train_loss` (or nested under a step's `status_details.metrics`). Use the **last** `value` in the list — not `status_details.train_loss` alone (that is the most recent logged step, which may differ from epoch-average loss on some backends). Round to 3 decimal places. | | **Final validation loss** | Last entry in `status_details.metrics.val_loss`. If the list is empty, report `n/a (no validation run)` and note whether validation data was configured. Automodel validates once per epoch by default. Unsloth validates once per epoch when `dataset.validation_path` is set and `schedule.eval_steps` is omitted (platform default: `max(1, effective_steps - 1)`). | | **Notes** | See **Notes by status** below | -**Metrics extraction** — after polling, always run `nemo jobs get-status ` and read `status_details.metrics` (both backends accumulate `train_loss` and `val_loss` time series there). Include both final losses in the report even when status is `error` if training completed before the failure (e.g. entity registration failed after upload). +**Metrics extraction** — after polling, always run `nemo jobs get-status ` and read `status_details.metrics` (all backends accumulate `train_loss` and `val_loss` time series there). Include both final losses in the report even when status is `error` if training completed before the failure (e.g. entity registration failed after upload). ## Notes by status @@ -62,7 +64,7 @@ Append a `### Training configuration` table after the header block (before **Usi | GPU | `parallelism.num_gpus_per_node` (and `tensor_parallel_size` when >1) | `hardware.gpus` | | Output save method | `output.type` (e.g. `adapter`) | `output.save_method` (e.g. `lora`) | -The two examples below show the filled-in table per backend. +The three examples below show the filled-in table per backend. ## Automodel example @@ -109,6 +111,29 @@ The two examples below show the filled-in table per backend. | Output save method | lora | ``` +## RL (DPO) example + +Map rows from the `training` (DPOTraining) block; there is no LoRA/save-method. Add DPO-specific rows (`ref_policy_kl_penalty` = β, `sft_loss_weight`): + +```markdown +### Training configuration + +| Setting | Value | +|---------|-------| +| Training type | DPO (full-weight) | +| Reference KL penalty (β) | 0.05 | +| SFT loss weight | 0.0 | +| Max sequence length | 1024 | +| Epochs | 1 | +| Micro batch size | 1 | +| Global batch size | 32 | +| Learning rate | 5e-6 | +| Optimizer | AdamW + cosine annealing | +| Precision | bf16 | +| GPU | 1 node × 1 GPU | +| Output | full-weight model entity | +``` + ## Using the output (`completed` only) After **Training configuration**, branch on output type: @@ -116,11 +141,11 @@ After **Training configuration**, branch on output type: | Output | When | Report section | |--------|------|----------------| | LoRA adapter | `save_method: lora` (default) | **Using the adapter** — below | -| Full model | `finetuning_type: all_weights`, `save_method: merged_16bit` / `merged_4bit` | **Using the fine-tuned model** — below | +| Full model | `finetuning_type: all_weights`, `save_method: merged_16bit` / `merged_4bit`, or **rl (DPO)** (always full-weight) | **Using the fine-tuned model** — below | ### Using the adapter (LoRA / `save_method: lora`) -**Automodel / unsloth LoRA only.** Run these discovery commands (parse stdout only; do not pipe `2>&1` into JSON parsers): +**Automodel / unsloth LoRA only** — DPO (rl) never produces an adapter; for rl output use **Using the fine-tuned model** below. Run these discovery commands (parse stdout only; do not pipe `2>&1` into JSON parsers): 1. `nemo models get --workspace default` — confirm `` appears under `adapters` with `enabled: true`. 2. `nemo inference providers list --workspace default -f json` — pick a **READY** provider whose `served_models` includes `default/` (base entity). Record its `name` as `` (often matches the deployment name). @@ -197,9 +222,9 @@ uv run python plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer Uses CHAT `messages` rows unchanged from the training fileset (`messages[:-1]` at inference). Repeat `--adapter` for multi-adapter compare. `--provider` is optional when a READY provider is auto-discovered. Set `NMP_BASE_URL` (or pass `--base-url`) when the platform is not localhost. LoRA only — full SFT / merged outputs need a deployed model entity (see **Using the fine-tuned model**). ``` -### Using the fine-tuned model (full SFT / merged checkpoint) +### Using the fine-tuned model (full SFT / merged checkpoint / DPO) -When `finetuning_type: all_weights` or `save_method` is `merged_16bit` / `merged_4bit`, the job registers a **model** entity at `output.name` with full fine-tuned weights. **Deploy that entity before inference or eval** — full checkpoints are not hot-reloaded onto the base model's LoRA deployment. +When `finetuning_type: all_weights`, `save_method` is `merged_16bit` / `merged_4bit`, or the backend is **rl (DPO)**, the job registers a **model** entity at `output.name` with full fine-tuned weights. **Deploy that entity before inference or eval** — full checkpoints are not hot-reloaded onto the base model's LoRA deployment. 1. `nemo models get --workspace default` — confirm the fine-tuned model entity exists. 2. Create or update an inference deployment / provider that serves `default/` (same workflow as deploying any model entity). @@ -217,7 +242,7 @@ Fine-tuned weights are on model entity `default/`. Unlike LoRA adap Use the same chat settings as LoRA inference (`messages[:-1]`, `max_tokens`, `temperature`, `enable_thinking` as appropriate). Post-training eval: run generation eval against this model-entity URL (not `eval_helpers.py --adapter`, which is LoRA-specific). ``` -Use the user's platform URL in `NMP_BASE_URL` when they overrode it; omit the export line for default `http://127.0.0.1:8080`. Substitute ``, concrete URLs, and entity names with values from discovery — do not leave generic placeholders in the user-facing report. For **LoRA**, do **not** tell the user to update the deployment before calling the adapter — registration on the base model entity is sufficient. For **full SFT / merged**, tell the user they must deploy `` before inference. +Use the user's platform URL in `NMP_BASE_URL` when they overrode it; omit the export line for default `http://127.0.0.1:8080`. Substitute ``, concrete URLs, and entity names with values from discovery — do not leave generic placeholders in the user-facing report. For **LoRA**, do **not** tell the user to update the deployment before calling the adapter — registration on the base model entity is sufficient. For **full SFT / merged / DPO**, tell the user they must deploy `` before inference. **Save report to `/tmp`** — unless the user opts out, write the full Markdown report (header, **Training configuration**, **Using the adapter** or **Using the fine-tuned model** when `completed`, and **Resources created** when a slug or new filesets were used) to `/tmp/fine-tune-result-.md`. Use the random slug from the run when one was assigned; otherwise use the job id suffix (e.g. `a925b07ff678`). diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/rl-kubernetes-runtime.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/rl-kubernetes-runtime.md new file mode 100644 index 0000000000..8b328020e8 --- /dev/null +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/rl-kubernetes-runtime.md @@ -0,0 +1,57 @@ +# rl backend — Kubernetes job execution requirement + +The `rl` (DPO) backend runs **each job step as a Kubernetes pod** via the +`kubernetes_job` execution backend. This is different from `automodel` / `unsloth`, +which use the **docker** job backend. So the platform you submit against must be +deployed/configured for Kubernetes job execution — the docker job backend cannot +run rl, and `rl submit` fails fast (`require_distributed_runtime`) on a +docker-runtime platform. + +Deployment model is the same as automodel/unsloth: **run the platform locally** +(`nemo services run` — never anything else). The only difference for rl is the +**execution backend** the local platform dispatches jobs to: `kubernetes_job` +(pointing at a Kubernetes GPU cluster via its kubeconfig) instead of `docker`. +What matters is that the platform's jobs backend is `kubernetes_job` and the job +pods can reach the platform's APIs. + +## Step 1 — verify the connected platform qualifies (always do this first) + +```bash +nemo jobs list-execution-profiles -f json +``` + +- `cpu` and `gpu` profiles report `backend: kubernetes_job` (or `volcano_job`) → + the platform is ready for rl. Proceed to submit. +- They report `backend: docker` / `subprocess` → the platform is **not** + configured for rl. Do **not** reuse it and do **not** fall back to + automodel/unsloth (those are SFT/LoRA, not DPO). Instead, run the local + platform configured for the `kubernetes_job` backend pointed at a Kubernetes + GPU cluster (see **Configuring the local platform for rl** below). If **no** + Kubernetes cluster is available to point at, stop and tell the user rl needs + one. + +## Configuring the local platform for rl + +When you start the platform locally (`nemo services run`) for an rl job, it must +be configured with all of: + +1. `platform.runtime: kubernetes`. +2. `jobs` `kubernetes_job` executors registered for **both** providers the + customizer stamps — `cpu` (download / upload / model-entity steps) and `gpu` + (DPO training) — at the resolved profile. +3. `platform.loopback_address` set to a platform address the **job pods can reach** + (the platform rewrites the `NMP_*_URL` it injects into pods to this, so the + download/upload steps can call the files/jobs APIs). +4. The target GPU cluster has, available as pullable/loaded images: the job-step + images (`nmp-customizer-tasks`, `nmp-rl-training`), the **jobs-launcher** image (each + step runs a launcher init container), and a **job-storage PVC** the steps share. +5. Multi-node only (`parallelism.num_nodes > 1`): `NMP_RL_MULTINODE_SHARED_STORAGE_PATH` + (a shared filesystem for Ray's cross-node coordination). + +If a job pod shows `ErrImagePull` / `ImagePullBackOff` on the launcher init +container or a step image, that image isn't available in the cluster — surface it; +do not build/pull it as part of the customization workflow. + +Reference (local platform config): `docs/set-up/manage-jobs.mdx` (execution +backends — `kubernetes_job`), `docs/set-up/config-reference.mdx` +(`platform.runtime`, `loopback_address`, `kubernetes_job` executor config). diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.md b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.md index 08ba7c1e9c..ea63cf640b 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.md +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.md @@ -64,7 +64,7 @@ Each entry has `provider`, `profile` (name), and `backend` (e.g. `docker`, `kube | Response includes **`provider`: `gpu` or `gpu_distributed`** | **`automodel`** (default) | | No GPU profiles (only `subprocess` and/or CPU `provider`) | Report that GPU customization is unavailable | -Automodel and unsloth are **`submit`-only**. After submit, the platform's **Docker executor** runs GPU container steps on the daemon attached to the connected platform host (`platform.runtime: docker`). Training does not run in the CLI shell — query execution profiles on the platform (`NMP_BASE_URL`), not GPU availability in the agent's terminal. +Automodel and unsloth are **`submit`-only**. After submit, the platform's **Docker executor** runs GPU container steps on the daemon attached to the connected platform host (`platform.runtime: docker`). (rl is also submit-only but runs on Kubernetes/Ray — see `rl-kubernetes-runtime.md`.) Training does not run in the CLI shell — query execution profiles on the platform (`NMP_BASE_URL`), not GPU availability in the agent's terminal. ### Pick execution profile diff --git a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.json b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.json index 7663d036a4..49490836e7 100644 --- a/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.json +++ b/plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/tests.json @@ -51,6 +51,21 @@ "prompt": "I want a quick LoRA adapter on Qwen with Unsloth's optimizer defaults, single GPU container job.", "expected_skill": "nemo-customizer" }, + { + "type": "explicit", + "prompt": "Use nemo customization rl submit to run a DPO job on NeMo Platform.", + "expected_skill": "nemo-customizer" + }, + { + "type": "implicit", + "prompt": "Align Qwen3-0.6B with DPO on a {prompt, chosen, rejected} preference dataset via nemo customization.", + "expected_skill": "nemo-customizer" + }, + { + "type": "implicit", + "prompt": "I want to run direct preference optimization on the platform's Ray cluster and register the trained model.", + "expected_skill": "nemo-customizer" + }, { "type": "contextual", "prompt": "NeMo Platform is running. Before any customization training, help me explore what my support agent should do.", diff --git a/plugins/nemo-rl/pyproject.toml b/plugins/nemo-rl/pyproject.toml index e9f4064033..5ed5122379 100644 --- a/plugins/nemo-rl/pyproject.toml +++ b/plugins/nemo-rl/pyproject.toml @@ -18,18 +18,11 @@ dependencies = [ # image, not installed by the plugin. The plugin process only needs the # lightweight compile-side imports. -# RL customization backend temporarily disabled pending CVE remediation of the -# nmp-rl-training container image (20 Critical / 135 High as of the 2026-07 scan). -# The plugin code, the `nmp-rl` service, and the Dockerfiles are intentionally -# left intact so re-enabling is a one-step revert: uncomment the two entry-point -# blocks below and run `uv sync` (entry points are read from installed dist -# metadata, so a re-sync is required for the change to take effect). No other -# code changes are required. -# [project.entry-points."nemo.customization.contributors"] -# rl = "nemo_rl_plugin.contributor:RlContributor" -# -# [project.entry-points."nemo.jobs"] -# "customization.rl.jobs" = "nemo_rl_plugin.jobs.jobs:RlJob" +[project.entry-points."nemo.customization.contributors"] +rl = "nemo_rl_plugin.contributor:RlContributor" + +[project.entry-points."nemo.jobs"] +"customization.rl.jobs" = "nemo_rl_plugin.jobs.jobs:RlJob" [build-system] requires = ["hatchling"] diff --git a/pyproject.toml b/pyproject.toml index 8159982842..a20d518b6e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -187,6 +187,7 @@ enabled-plugins = [ "nemo-customizer-plugin", "nemo-automodel-plugin", "nemo-unsloth-plugin", + "nemo-rl-plugin", ] # Legacy runtime needed specifically for task images that still invoke @@ -580,12 +581,6 @@ extra-paths = [ # on sys.path in tests/conftest.py. "tests/agentic-use", "tests/agentic-use/shared", - # nemo-rl is a workspace member but is temporarily excluded from enabled-plugins - # (RL/DPO backend disabled pending nmp-rl-training CVE remediation), so it is not - # installed in the default env. Add its src + the services/rl src so ty can still - # resolve nemo_rl_plugin / nmp.rl imports when checking the RL source and tests. - "plugins/nemo-rl/src", - "services/rl/src", ] [tool.ty.src] diff --git a/pytest.ini b/pytest.ini index a56efda6ee..0568982260 100644 --- a/pytest.ini +++ b/pytest.ini @@ -13,8 +13,6 @@ pythonpath = plugins/nemo-deployments/tests/unit plugins/nemo-deployments/tests/integration plugins/nemo-safe-synthesizer/src - plugins/nemo-rl/src - services/rl/src services/core/jobs/tests/controllers services/core/models/tests/unit diff --git a/services/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.py b/services/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.py index 148ea5c26d..897a14a10b 100644 --- a/services/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.py +++ b/services/rl/src/nmp/rl/tasks/training/backends/nemo_rl/dpo_driver.py @@ -16,7 +16,7 @@ import argparse import logging -from typing import cast +from typing import Any, cast from nemo_rl.algorithms.dpo import MasterConfig, dpo_train, setup from nemo_rl.algorithms.utils import get_tokenizer @@ -56,23 +56,29 @@ def main(): print(f"Overrides: {overrides}") cfg = parse_hydra_overrides(cfg, overrides) - config = cast(MasterConfig, OmegaConf.to_container(cfg, resolve=True)) + # NeMo-RL's MasterConfig is a Pydantic BaseModel; setup()/dpo_train() read it + # by attribute (e.g. master_config.dpo.seed). OmegaConf.to_container() returns + # a plain dict, so build the model here — mirroring NeMo-RL's own run_dpo.py — + # which validates the config up front and matches how the algorithm consumes + # it. Only the top level is a model: the sub-configs (policy/logger/... ) stay + # TypedDict dicts, so they are still accessed by subscript (config.policy["x"]). + config = MasterConfig(**cast(dict[str, Any], OmegaConf.to_container(cfg, resolve=True))) print("Applied CLI overrides") # Log only the top-level config section names. The resolved config carries # integration secrets (W&B / MLflow tokens, tracking URIs), so never dump the - # full structure to stdout. - print(f"Config sections loaded: {sorted(config.keys())}") + # full structure to stdout. model_fields is names-only — no values materialized. + print(f"Config sections loaded: {sorted(type(config).model_fields)}") - config["logger"]["log_dir"] = get_next_experiment_dir(config["logger"]["log_dir"]) - print(f"📊 Using log directory: {config['logger']['log_dir']}") - if config["checkpointing"]["enabled"]: - print(f"📊 Using checkpoint directory: {config['checkpointing']['checkpoint_dir']}") + config.logger["log_dir"] = get_next_experiment_dir(config.logger["log_dir"]) + print(f"📊 Using log directory: {config.logger['log_dir']}") + if config.checkpointing["enabled"]: + print(f"📊 Using checkpoint directory: {config.checkpointing['checkpoint_dir']}") init_ray() # setup tokenizer - tokenizer = get_tokenizer(config["policy"]["tokenizer"]) + tokenizer = get_tokenizer(config.policy["tokenizer"]) # Register our local-file-capable HelpSteer3 / Tulu3 datasets into NeMo-RL's # DATASET_REGISTRY before building data. Without this, setup_preference_data @@ -84,7 +90,7 @@ def main(): # dataset specs). The compiler emits one of BinaryPreferenceDataset / # PreferenceDataset / HelpSteer3 / Tulu3Preference per detected schema, each # pointing at the prepared local training.jsonl / validation.jsonl. - dataset, val_dataset = setup_preference_data(tokenizer, config["data"]) + dataset, val_dataset = setup_preference_data(tokenizer, config.data) ( policy, cluster, @@ -104,10 +110,10 @@ def main(): print(f"Job context loaded (job_id={job_ctx.job_id})") if job_ctx.jobs_url: # Extract training parameters for progress reporting - max_steps = config["dpo"].get("max_num_steps", 0) - num_epochs = config["dpo"].get("max_num_epochs", 1) - steps_per_epoch = config["dpo"]["steps_per_epoch"] # type: ignore - we need to pass this additional parameter to the logger - log_interval = (config["dpo"]["val_period"] // 10) + 1 + max_steps = config.dpo.max_num_steps + num_epochs = config.dpo.max_num_epochs + steps_per_epoch = config.dpo.steps_per_epoch # type: ignore[attr-defined] - extra (undeclared) DPOConfig field, allowed via extra="allow" + log_interval = (config.dpo.val_period // 10) + 1 customizer_logger = NemoRLLogger( steps_per_epoch=steps_per_epoch, @@ -123,7 +129,7 @@ def main(): else: print("WARNING: logger has no `.loggers`; NeMo Platform progress reporting disabled.") - logger.log_hyperparams(config) + logger.log_hyperparams(config.model_dump()) dpo_train( policy, diff --git a/third_party/requirements-main.txt b/third_party/requirements-main.txt index 8627de2ade..0e62949991 100644 --- a/third_party/requirements-main.txt +++ b/third_party/requirements-main.txt @@ -30,6 +30,7 @@ # nemo-experimentalist-plugin # nemo-guardrails-plugin # nemo-insights-plugin + # nemo-rl-plugin # nemo-safe-synthesizer-plugin # nemo-unsloth-plugin # nemoplatform @@ -52,6 +53,7 @@ # nemo-platform # nemo-platform-ext # nemo-platform-sdk + # nemo-rl-plugin # nemo-safe-synthesizer-plugin # nemo-switchyard # nemo-unsloth-plugin @@ -62,6 +64,7 @@ # nmp-models # nmp-platform # nmp-platform-runner + # nmp-rl # nmp-unsloth -e ./packages/nmp_common ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') # via @@ -82,14 +85,17 @@ # nmp-platform # nmp-platform-runner # nmp-platform-seed + # nmp-rl # nmp-secrets # nmp-studio # nmp-unsloth -e ./packages/nmp_customization_common ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') # via # nemo-automodel-plugin + # nemo-rl-plugin # nemo-unsloth-plugin # nmp-automodel + # nmp-rl # nmp-unsloth -e ./packages/nmp_platform ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') # via nemoplatform @@ -120,6 +126,7 @@ # via # nemo-eval-author-plugin # nemo-experimentalist-plugin +-e ./plugins/nemo-rl ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') -e ./plugins/nemo-safe-synthesizer ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') -e ./plugins/nemo-switchyard ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') -e ./plugins/nemo-switchyard/vendor/switchyard ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') @@ -138,6 +145,7 @@ # nmp-core-mcp # nmp-customization-common # nmp-entities + # nmp-rl # nmp-unsloth -e ./services/automodel ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') # via nemo-automodel-plugin @@ -183,6 +191,8 @@ # via nemoplatform -e ./services/platform-seed ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') # via nemoplatform +-e ./services/rl ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') + # via nemo-rl-plugin -e ./services/studio ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') # via nemoplatform -e ./services/unsloth ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') @@ -1068,6 +1078,7 @@ httpx==0.28.1 ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (p # nmp-automodel # nmp-customization-common # nmp-guardrails + # nmp-rl # nmp-unsloth # nooa # nvidia-nat-core @@ -2431,6 +2442,7 @@ pydantic==2.12.5 ; (platform_machine == 'arm64' and sys_platform == 'darwin') or # nemo-platform-ext # nemo-platform-plugin # nemo-platform-sdk + # nemo-rl-plugin # nemo-safe-synthesizer # nemo-safe-synthesizer-plugin # nemo-unsloth-plugin @@ -2446,6 +2458,7 @@ pydantic==2.12.5 ; (platform_machine == 'arm64' and sys_platform == 'darwin') or # nmp-intake # nmp-jobs # nmp-models + # nmp-rl # nmp-unsloth # nooa # nvidia-nat-atif @@ -2539,6 +2552,7 @@ pydantic-settings==2.14.2 ; (platform_machine == 'arm64' and sys_platform == 'da # nemo-anonymizer # nemo-automodel-plugin # nemo-platform-plugin + # nemo-rl-plugin # nemo-safe-synthesizer # nemo-safe-synthesizer-plugin # nemo-unsloth-plugin @@ -2553,6 +2567,7 @@ pydantic-settings==2.14.2 ; (platform_machine == 'arm64' and sys_platform == 'da # nmp-intake # nmp-jobs # nmp-models + # nmp-rl # nmp-unsloth pygments==2.20.0 ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') \ --hash=sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f \ @@ -3145,6 +3160,7 @@ tenacity==9.1.4 ; (platform_machine == 'arm64' and sys_platform == 'darwin') or # nmp-automodel # nmp-customization-common # nmp-models + # nmp-rl # nmp-unsloth tiktoken==0.12.0 ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') \ --hash=sha256:01d99484dc93b129cd0964f9d34eee953f2737301f18b3c7257bf368d7615baa \ @@ -3252,6 +3268,7 @@ typer==0.24.1 ; (platform_machine == 'arm64' and sys_platform == 'darwin') or (p # nemo-platform-ext # nemo-platform-plugin # nemo-platform-sdk + # nemo-rl-plugin # nemo-safe-synthesizer-plugin # nemo-unsloth-plugin # nemoguardrails diff --git a/uv.lock b/uv.lock index c6791baae3..9ce01d41dd 100644 --- a/uv.lock +++ b/uv.lock @@ -6325,6 +6325,7 @@ core-services = [ { name = "nemo-insights-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-platform", extra = ["services"], marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-platform-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, + { name = "nemo-rl-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-safe-synthesizer-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-switchyard", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-unsloth-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, @@ -6420,6 +6421,7 @@ enabled-plugins = [ { name = "nemo-experimentalist-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-guardrails-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-insights-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, + { name = "nemo-rl-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-safe-synthesizer-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-switchyard", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-unsloth-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, @@ -6443,6 +6445,7 @@ functional-services = [ { name = "nemo-insights-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-platform", extra = ["services"], marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-platform-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, + { name = "nemo-rl-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-safe-synthesizer-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-switchyard", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, { name = "nemo-unsloth-plugin", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, @@ -6540,6 +6543,7 @@ core-services = [ { name = "nemo-platform", editable = "packages/nemo_platform" }, { name = "nemo-platform", extras = ["services"], editable = "packages/nemo_platform" }, { name = "nemo-platform-plugin", editable = "packages/nemo_platform_plugin" }, + { name = "nemo-rl-plugin", editable = "plugins/nemo-rl" }, { name = "nemo-safe-synthesizer-plugin", editable = "plugins/nemo-safe-synthesizer" }, { name = "nemo-switchyard", editable = "plugins/nemo-switchyard" }, { name = "nemo-unsloth-plugin", editable = "plugins/nemo-unsloth" }, @@ -6637,6 +6641,7 @@ enabled-plugins = [ { name = "nemo-experimentalist-plugin", marker = "python_full_version < '3.14'", editable = "plugins/nemo-experimentalist" }, { name = "nemo-guardrails-plugin", editable = "plugins/nemo-guardrails" }, { name = "nemo-insights-plugin", editable = "plugins/nemo-insights" }, + { name = "nemo-rl-plugin", editable = "plugins/nemo-rl" }, { name = "nemo-safe-synthesizer-plugin", editable = "plugins/nemo-safe-synthesizer" }, { name = "nemo-switchyard", editable = "plugins/nemo-switchyard" }, { name = "nemo-unsloth-plugin", editable = "plugins/nemo-unsloth" }, @@ -6661,6 +6666,7 @@ functional-services = [ { name = "nemo-platform", editable = "packages/nemo_platform" }, { name = "nemo-platform", extras = ["services"], editable = "packages/nemo_platform" }, { name = "nemo-platform-plugin", editable = "packages/nemo_platform_plugin" }, + { name = "nemo-rl-plugin", editable = "plugins/nemo-rl" }, { name = "nemo-safe-synthesizer-plugin", editable = "plugins/nemo-safe-synthesizer" }, { name = "nemo-switchyard", editable = "plugins/nemo-switchyard" }, { name = "nemo-unsloth-plugin", editable = "plugins/nemo-unsloth" },