From 80f2fe5f2910fff9bb7a40daf2c224e423f507bc Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Thu, 4 Dec 2025 14:39:24 -0500 Subject: [PATCH 01/11] docs(get-started): move cluster, installation, and quick-start to get-started directory Signed-off-by: Lawrence Lane --- docs/{ => get-started}/cluster.md | 0 docs/{about/quick-start.md => get-started/index.md} | 0 docs/{about => get-started}/installation.md | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename docs/{ => get-started}/cluster.md (100%) rename docs/{about/quick-start.md => get-started/index.md} (100%) rename docs/{about => get-started}/installation.md (100%) diff --git a/docs/cluster.md b/docs/get-started/cluster.md similarity index 100% rename from docs/cluster.md rename to docs/get-started/cluster.md diff --git a/docs/about/quick-start.md b/docs/get-started/index.md similarity index 100% rename from docs/about/quick-start.md rename to docs/get-started/index.md diff --git a/docs/about/installation.md b/docs/get-started/installation.md similarity index 100% rename from docs/about/installation.md rename to docs/get-started/installation.md From 87ab66f9e35bc1b8aa13b1690e7d691cef3d54bb Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Thu, 4 Dec 2025 14:39:29 -0500 Subject: [PATCH 02/11] docs(get-started): restructure content, add quickstart guides, update links - Remove redundant cluster redirect and local-workstation files - Add algorithm quickstart guides for DPO, GRPO, and SFT - Restructure and enhance cluster, installation, and quickstart content - Update index, debugging, and guide links for new structure - Restore missing sbatch flags, env vars, and auth setup Signed-off-by: Lawrence Lane --- docs/about/clusters.md | 4 - docs/debugging.md | 2 +- docs/get-started/cluster.md | 315 ++++++++++++++++------------ docs/get-started/dpo.md | 156 ++++++++++++++ docs/get-started/grpo.md | 146 +++++++++++++ docs/get-started/index.md | 258 ++++++++++++++++++++--- docs/get-started/installation.md | 227 +++++++++++++++----- docs/get-started/sft.md | 153 ++++++++++++++ docs/guides/dpo.md | 2 +- docs/guides/grpo.md | 2 +- docs/guides/rm.md | 2 +- docs/guides/sft.md | 2 +- docs/index.md | 348 ++++++++++++++++--------------- docs/local-workstation.md | 35 ---- 14 files changed, 1226 insertions(+), 426 deletions(-) delete mode 100644 docs/about/clusters.md create mode 100644 docs/get-started/dpo.md create mode 100644 docs/get-started/grpo.md create mode 100644 docs/get-started/sft.md delete mode 100644 docs/local-workstation.md diff --git a/docs/about/clusters.md b/docs/about/clusters.md deleted file mode 100644 index cfb6041d87..0000000000 --- a/docs/about/clusters.md +++ /dev/null @@ -1,4 +0,0 @@ -# Installation: Set Up Clusters - -For detailed instructions on how to set up and launch NeMo RL on Slurm or Kubernetes clusters, please refer to the dedicated [Cluster Start](../cluster.md) documentation. - diff --git a/docs/debugging.md b/docs/debugging.md index cd3b55d354..51d013ab08 100644 --- a/docs/debugging.md +++ b/docs/debugging.md @@ -9,7 +9,7 @@ Since Ray programs can spawn multiple workers and actors, using the Ray Distribu ### Prerequisites * Install the [Ray Debugger VS Code/Cursor extension](https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html). -* Launch the [interactive cluster](./cluster.md#interactive-launching) with `ray.sub`. +* Launch the [interactive cluster](get-started/cluster.md#2-submit-a-job) with `ray.sub`. * Launch VS Code/Cursor on the SLURM login node (where `squeue`/`sbatch` is available). * Add `breakpoint()` in your code under actors & tasks (i.e. classes or functions decorated with `@ray.remote`). * **Ensure** `RAY_DEBUG=legacy` is not set since this debugging requires the default distributed debugger. diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index 73e2225a1b..3d488f5555 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -1,176 +1,213 @@ -# Set Up Clusters +--- +description: "Step-by-step guide to running NeMo RL on multi-node Slurm clusters" +categories: ["getting-started"] +tags: ["cluster", "slurm", "kubernetes", "multi-node", "ray"] +personas: ["mle-focused", "cluster-administrator-focused"] +difficulty: "intermediate" +content_type: "tutorial" +--- -This guide explains how to run NeMo RL with Ray on Slurm or Kubernetes. +(gs-cluster)= -## Use Slurm for Batched and Interactive Jobs +# Set Up a Training Cluster - The following code provides instructions on how to use Slurm to run batched job submissions and run jobs interactively. +Scaling from a single GPU to a multi-node cluster allows you to train larger models (70B+) and process data much faster. -### Batched Job Submission +NeMo RL uses **Ray** to manage distributed computing. -```sh +:::{card} +**Goal**: Submit a multi-node training job (e.g., GRPO) to a Slurm cluster. + +^^^ + +**Steps**: + +1. **Understand the Architecture**: How Ray and Slurm work together. +2. **Submit a Job**: Choose between Interactive (Debug) or Batch (Production) modes. +3. **Verify**: Check logs and the Ray Dashboard. +::: + +:::{button-ref} index +:color: secondary +:outline: +:ref-type: doc + +← Previous: Quickstart Guide +::: + +--- + +## 1. Understand the Architecture + +When you submit a job, two layers of orchestration happen: + +1. **Slurm (The Hardware Layer)**: Allocates physical nodes (e.g., 4 nodes with 8 GPUs each). +2. **Ray (The Application Layer)**: Connects those nodes into a unified cluster. + * **Head Node**: Runs the driver script (e.g., `run_grpo.py`) and manages the cluster state. + * **Worker Nodes**: Execute the heavy lifting (model training, generation). + +You don't need to manually configure Ray. NeMo RL provides a helper script, `ray.sub`, that handles the bootstrapping for you. + +--- + +## 2. Submit a Job + +The submission process is identical for SFT, DPO, Reward Model (RM), and GRPO. You swap the Python script in the `COMMAND` variable. + +### Command Cheatsheet + +Copy the command for your training type: + +| Type | Command | +| :--- | :--- | +| **SFT** | `uv run examples/run_sft.py` | +| **DPO** | `uv run examples/run_dpo.py` | +| **Reward Model** | `uv run examples/run_rm.py` | +| **GRPO** | `uv run examples/run_grpo_math.py` | + +::::{tab-set} + +:::{tab-item} Interactive Mode (Recommended for Debugging) +Interactive mode launches the cluster and gives you a shell on the **Head Node**. This is perfect for debugging because you can run scripts, check files, and kill/restart jobs without re-queueing. + +**1. Submit the Request** +Ask for the resources you need (e.g., 1 node, 8 GPUs). + +```bash # Run from the root of NeMo RL repo NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) -COMMAND="uv run ./examples/run_grpo_math.py" \ -CONTAINER=YOUR_CONTAINER \ +CONTAINER=nvcr.io/nvidia/nemo:latest \ MOUNTS="$PWD:$PWD" \ sbatch \ --nodes=${NUM_ACTOR_NODES} \ --account=YOUR_ACCOUNT \ - --job-name=YOUR_JOBNAME \ --partition=YOUR_PARTITION \ - --time=1:0:0 \ - --gres=gpu:8 \ + --gpus-per-node=8 \ + --time=04:00:00 \ + --job-name=nemo-rl-interactive \ ray.sub ``` -> [!TIP] -> Depending on your Slurm cluster configuration, you may or may not need to include the `--gres=gpu:8` option in the `sbatch` command. +:::{tip} +Replace `YOUR_ACCOUNT` and `YOUR_PARTITION` with values from your cluster. Run `sacctmgr show associations user=$USER` to find your account. +::: -Upon successful submission, Slurm will print the `SLURM_JOB_ID`: -```text -Submitted batch job 1980204 +**2. Attach to the Cluster** +Once the job starts, Slurm creates an attach script (e.g., `12345-attach.sh`). Run it: + +```bash +bash -attach.sh ``` -Make a note of the job submission number. Once the job begins, you can track its process in the driver logs which you can `tail`: -```sh -tail -f 1980204-logs/ray-driver.log + +**3. Run Your Training** +You are now inside the container on the head node. Run your command (see Cheatsheet above): + +```bash +uv run examples/run_sft.py ``` -### Interactive Launching +::: + +:::{tab-item} Batch Mode (Production) +Batch mode is "fire and forget." You specify the command upfront, and the cluster shuts down automatically when it finishes. -> [!TIP] -> A key advantage of running interactively on the head node is the ability to execute multiple multi-node jobs without needing to requeue in the Slurm job queue. This means that during debugging sessions, you can avoid submitting a new `sbatch` command each time. Instead, you can debug and re-submit your NeMo RL job directly from the interactive session. +**1. Submit the Job** +Include the `COMMAND` variable in your submission. Replace the command below with the one from the Cheatsheet. -To run interactively, launch the same command as [Batched Job Submission](#batched-job-submission), but omit the `COMMAND` line: -```sh +```bash # Run from the root of NeMo RL repo NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) -CONTAINER=YOUR_CONTAINER \ +COMMAND="uv run examples/run_sft.py" \ +CONTAINER=nvcr.io/nvidia/nemo:latest \ MOUNTS="$PWD:$PWD" \ sbatch \ --nodes=${NUM_ACTOR_NODES} \ --account=YOUR_ACCOUNT \ - --job-name=YOUR_JOBNAME \ --partition=YOUR_PARTITION \ - --time=1:0:0 \ - --gres=gpu:8 \ + --gpus-per-node=8 \ + --time=24:00:00 \ + --job-name=nemo-rl-prod \ ray.sub ``` -Upon successful submission, Slurm will print the `SLURM_JOB_ID`: -```text -Submitted batch job 1980204 -``` -Once the Ray cluster is up, a script will be created to attach to the Ray head node. Run this script to launch experiments: -```sh -bash 1980204-attach.sh -``` -Now that you are on the head node, you can launch the command as follows: -```sh -uv run ./examples/run_grpo_math.py + +**2. Check Status** +Slurm will write the output to a log file (e.g., `12345-logs/ray-driver.log`). +::: + +:::: + +--- + +## 3. Verify Your Cluster + +Once your job is running, you have two main ways to see what's happening. + +### 1. Slurm Logs + +The `ray.sub` script creates a log directory named after your Job ID (e.g., `1980204-logs/`). + +* **`ray-driver.log`**: The stdout/stderr of your Python script. Check this for training progress (loss values). +* **`ray-worker-*.log`**: Logs for individual worker nodes (useful for debugging specific node failures). +* **`dashboard.log`**: Debug info for the Ray dashboard. + +```bash +tail -f 1980204-logs/ray-driver.log ``` -### Slurm Environment Variables +### 2. Ray Dashboard -All Slurm environment variables described below can be added to the `sbatch` -invocation of `ray.sub`. For example, `GPUS_PER_NODE=8` can be specified as follows: +Ray provides a visual dashboard to see GPU usage, memory usage, and actor status. -```sh -GPUS_PER_NODE=8 \ -... \ -sbatch ray.sub \ - ... +1. Find the dashboard port in the logs (printed at startup). +2. Forward the port to your local machine: + +```bash +ssh -L 8265:localhost:8265 user@cluster-login-node ``` -#### Common Environment Configuration -``````{list-table} -:header-rows: 1 - -* - Environment Variable - - Explanation -* - `CONTAINER` - - (Required) Specifies the container image to be used for the Ray cluster. - Use either a docker image from a registry or a squashfs (if using enroot/pyxis). -* - `MOUNTS` - - (Required) Defines paths to mount into the container. Examples: - ```md - * `MOUNTS="$PWD:$PWD"` (mount in current working directory (CWD)) - * `MOUNTS="$PWD:$PWD,/nfs:/nfs:ro"` (mounts the current working directory and `/nfs`, with `/nfs` mounted as read-only) - ``` -* - `COMMAND` - - Command to execute after the Ray cluster starts. If empty, the cluster idles and enters interactive mode (see the [Slurm interactive instructions](#interactive-launching)). -* - `HF_HOME` - - Sets the cache directory for huggingface-hub assets (e.g., models/tokenizers). -* - `WANDB_API_KEY` - - Setting this allows you to use the wandb logger without having to run `wandb login`. -* - `HF_TOKEN` - - Setting the token used by huggingface-hub. Avoids having to run the `huggingface-cli login` -* - `HF_DATASETS_CACHE` - - Sets the cache dir for downloaded Huggingface datasets. -`````` - -> [!TIP] -> When `HF_TOKEN`, `WANDB_API_KEY`, `HF_HOME`, and `HF_DATASETS_CACHE` are set in your shell environment using `export`, they are automatically passed to `ray.sub`. For instance, if you set: -> -> ```sh -> export HF_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -> ``` -> this token will be available to your NeMo RL run. Consider adding these exports to your shell configuration file, such as `~/.bashrc`. - -#### Advanced Environment Configuration -``````{list-table} -:header-rows: 1 - -* - Environment Variable - (and default) - - Explanation -* - `UV_CACHE_DIR_OVERRIDE` - - By default, this variable does not need to be set. If unset, `ray.sub` uses the - `UV_CACHE_DIR` defined within the container (defaulting to `/root/.cache/uv`). - `ray.sub` intentionally avoids using the `UV_CACHE_DIR` from the user's host - environment to prevent the host's cache from interfering with the container's cache. - Set `UV_CACHE_DIR_OVERRIDE` if you have a customized `uv` environment (e.g., - with pre-downloaded packages or specific configurations) that you want to persist - and reuse across container runs. This variable should point to a path on a shared - filesystem accessible by all nodes (head and workers). This path will be mounted - into the container and will override the container's default `UV_CACHE_DIR`. -* - `CPUS_PER_WORKER=128` - - CPUs each Ray worker node claims. Default is `16 * GPUS_PER_NODE`. -* - `GPUS_PER_NODE=8` - - Number of GPUs each Ray worker node claims. To determine this, run `nvidia-smi` on a worker node. -* - `BASE_LOG_DIR=$SLURM_SUBMIT_DIR` - - Base directory for storing Ray logs. Defaults to the Slurm submission directory ([SLURM_SUBMIT_DIR](https://slurm.schedmd.com/sbatch.html#OPT_SLURM_SUBMIT_DIR)). -* - `NODE_MANAGER_PORT=53001` - - Port for the Ray node manager on worker nodes. -* - `OBJECT_MANAGER_PORT=53003` - - Port for the Ray object manager on worker nodes. -* - `RUNTIME_ENV_AGENT_PORT=53005` - - Port for the Ray runtime environment agent on worker nodes. -* - `DASHBOARD_AGENT_GRPC_PORT=53007` - - gRPC port for the Ray dashboard agent on worker nodes. -* - `METRICS_EXPORT_PORT=53009` - - Port for exporting metrics from worker nodes. -* - `PORT=6379` - - Main port for the Ray head node. -* - `RAY_CLIENT_SERVER_PORT=10001` - - Port for the Ray client server on the head node. -* - `DASHBOARD_GRPC_PORT=52367` - - gRPC port for the Ray dashboard on the head node. -* - `DASHBOARD_PORT=8265` - - Port for the Ray dashboard UI on the head node. This is also the port - used by the Ray distributed debugger. -* - `DASHBOARD_AGENT_LISTEN_PORT=52365` - - Listening port for the dashboard agent on the head node. -* - `MIN_WORKER_PORT=54001` - - Minimum port in the range for Ray worker processes. -* - `MAX_WORKER_PORT=54257` - - Maximum port in the range for Ray worker processes. -`````` - -> [!NOTE] -> For the most part, you will not need to change ports unless these -> are already taken by some other service backgrounded on your cluster. - -## Kubernetes - -TBD + +3. Open `http://localhost:8265` in your browser. + +--- + +## Common Environment Variables + +You can pass these variables to `sbatch` to configure the environment: + +| Variable | Description | +| :--- | :--- | +| **`CONTAINER`** | Docker image to use (required). | +| **`MOUNTS`** | Paths to mount (e.g., `"$PWD:$PWD,/data:/data"`). | +| **`HF_TOKEN`** | Hugging Face token (for downloading gated models). | +| **`HF_HOME`** | Cache directory for Hugging Face models and tokenizers. | +| **`HF_DATASETS_CACHE`** | Cache directory for Hugging Face datasets. | +| **`WANDB_API_KEY`** | Weights & Biases key (for logging). | +| **`GPUS_PER_NODE`** | Number of GPUs per node (default: 8). | + +:::{tip} +Export secrets like `HF_TOKEN` in your shell profile (`~/.bashrc`) so you don't have to type them every time. +::: + +:::{dropdown} Advanced Environment Configuration +The following variables allow for deeper customization of the Ray cluster. Most users will not need to change these defaults. + +| Variable (and default) | Explanation | +| :--- | :--- | +| `UV_CACHE_DIR_OVERRIDE` | Override the UV cache directory to a shared filesystem location. Essential for persisting package caches across jobs. | +| `CPUS_PER_WORKER=128` | CPUs each Ray worker node claims. Default is `16 * GPUS_PER_NODE`. | +| `GPUS_PER_NODE=8` | Number of GPUs each Ray worker node claims. | +| `BASE_LOG_DIR=$SLURM_SUBMIT_DIR` | Base directory for storing Ray logs. | +| `NODE_MANAGER_PORT=53001` | Port for the Ray node manager on worker nodes. | +| `OBJECT_MANAGER_PORT=53003` | Port for the Ray object manager on worker nodes. | +| `RUNTIME_ENV_AGENT_PORT=53005` | Port for the Ray runtime environment agent on worker nodes. | +| `DASHBOARD_AGENT_GRPC_PORT=53007` | gRPC port for the Ray dashboard agent on worker nodes. | +| `METRICS_EXPORT_PORT=53009` | Port for exporting metrics from worker nodes. | +| `PORT=54514` | Main port for the Ray head node (default: 54514; Ray's standard default is 6379, but this script uses 54514 for multi-node compatibility). | +| `RAY_CLIENT_SERVER_PORT=10001` | Port for the Ray client server on the head node. | +| `DASHBOARD_GRPC_PORT=52367` | gRPC port for the Ray dashboard on the head node. | +| `DASHBOARD_PORT=8265` | Port for the Ray dashboard UI on the head node. | +| `DASHBOARD_AGENT_LISTEN_PORT=52365` | Listening port for the dashboard agent on the head node. | +| `MIN_WORKER_PORT=54001` | Minimum port in the range for Ray worker processes. | +| `MAX_WORKER_PORT=54257` | Maximum port in the range for Ray worker processes. | +::: diff --git a/docs/get-started/dpo.md b/docs/get-started/dpo.md new file mode 100644 index 0000000000..5b271bcbbb --- /dev/null +++ b/docs/get-started/dpo.md @@ -0,0 +1,156 @@ +--- +description: "Step-by-step guide to running Direct Preference Optimization (DPO) with NeMo RL" +categories: ["getting-started"] +tags: ["dpo", "quickstart", "preference-learning", "python-api"] +personas: ["data-scientist-focused", "mle-focused"] +difficulty: "beginner" +content_type: "tutorial" +--- + +(gs-dpo)= + +# Get Started with DPO + +**Direct Preference Optimization (DPO)** aligns models to human preferences without needing a separate reward model. It learns directly from "A is better than B" data. + +:::{card} +**Goal**: Align a fine-tuned model to preferences using chosen/rejected pairs. + +^^^ + +**Steps**: + +1. **Prepare Data**: Format your preference pairs. +2. **Configure**: Set the reference model and hyperparameters. +3. **Train**: Run the `run_dpo.py` script. +4. **Verify**: Monitor the preference loss. +::: + +:::{button-ref} index +:color: secondary +:outline: +:ref-type: doc + +← Previous: Quickstart Guide +::: + +--- + +## Prerequisites + +Before running DPO, you typically need: + +* A **Supervised Fine-Tuned (SFT)** model checkpoint (or a base instruct model) to use as the starting policy. +* NeMo RL installed via `uv`. + +--- + +## 1. Prepare Your Data + +DPO requires a dataset of "Preference Triplets". Each example contains a prompt and two potential responses: one "chosen" (preferred) and one "rejected". + +### The Data Format + +NeMo RL expects a JSONL file with the following structure: + +```json +{ + "prompt": "What is the capital of France?", + "chosen": "The capital of France is Paris.", + "rejected": "I think it's London." +} +``` + +* **prompt**: The instruction given to the model. +* **chosen**: The better response. +* **rejected**: The worse response. + +--- + +## 2. Configure the Job + +The default configuration is at `examples/configs/dpo.yaml`. + +Key parameters to tune: + +* **`policy.model_name`**: Your starting model (e.g., the SFT checkpoint you created). +* **`data.dataset_name`**: Set to `BinaryPreferenceDataset` to use the custom JSONL format described above. +* **`data.train_data_path`**: Path to your preference `.jsonl` file. +* **`dpo.reference_policy_kl_penalty`**: Controls how much the model stays close to the original behavior (preventing "reward hacking"). +* **`dpo.preference_loss_weight`**: The strength of the preference signal. + +:::{tip} +For a local test, stick to small models like `meta-llama/Llama-3.2-1B-Instruct` to avoid OOM errors. +::: + +--- + +## 3. Run the Training + +Run the `examples/run_dpo.py` script. + +::::{tab-set} + +:::{tab-item} Native PyTorch (DTensor) +```bash +uv run python examples/run_dpo.py \ + dpo.max_num_epochs=1 \ + dpo.reference_policy_kl_penalty=0.1 \ + policy.train_global_batch_size=32 \ + data.dataset_name=BinaryPreferenceDataset \ + data.train_data_path=path/to/your/data.jsonl +``` +::: + +:::{tab-item} Megatron Core +```bash +uv run python examples/run_dpo.py \ + dpo.max_num_epochs=1 \ + dpo.reference_policy_kl_penalty=0.1 \ + policy.train_global_batch_size=32 \ + data.dataset_name=BinaryPreferenceDataset \ + data.train_data_path=path/to/your/data.jsonl \ + policy.megatron_cfg.enabled=true \ + policy.dtensor_cfg.enabled=false +``` +::: + +:::: + +**What's happening?** + +* The script loads the **Reference Policy** (frozen) and the **Active Policy** (trainable). +* It calculates the likelihood of "chosen" vs "rejected" responses for both models. +* It updates the Active Policy to increase the margin between chosen and rejected likelihoods. + +--- + +## 4. Monitor and Verify + +Watch the logs for these metrics: + +1. **`preference_loss`**: Should decrease, indicating the model is learning to rank "chosen" higher than "rejected". +2. **`chosen_reward` vs `rejected_reward`**: Ideally, the "reward" (implicit) for chosen responses should go up, and rejected should go down. + +### Output Artifacts + +Results are saved to `results/dpo`: +* **Checkpoints**: The aligned model weights. + +--- + +## Scaling Up + +### Custom Loss Functions + +NeMo RL supports variations of preference loss. You can change weights to balance SFT loss (maintaining instruction following) and Preference loss: + +```bash +uv run python examples/run_dpo.py \ + dpo.preference_loss_weight=1.0 \ + dpo.sft_loss_weight=0.1 +``` + +### Multi-Node Training + +For large-scale alignment, enable the Megatron backend in `examples/configs/dpo.yaml` by setting `policy.megatron_cfg.enabled=true` and configuring the distributed parameters, or use a specific Megatron-compatible configuration file if available. diff --git a/docs/get-started/grpo.md b/docs/get-started/grpo.md new file mode 100644 index 0000000000..373654c0cd --- /dev/null +++ b/docs/get-started/grpo.md @@ -0,0 +1,146 @@ +--- +description: "Step-by-step guide to running Group Relative Policy Optimization (GRPO) with NeMo RL" +categories: ["getting-started"] +tags: ["grpo", "quickstart", "reinforcement-learning", "python-api", "reasoning"] +personas: ["data-scientist-focused", "mle-focused"] +difficulty: "intermediate" +content_type: "tutorial" +--- + +(gs-grpo)= + +# Get Started with GRPO + +**Group Relative Policy Optimization (GRPO)** is a highly efficient algorithm for reasoning tasks (like Math or Coding). Unlike PPO, it removes the need for a Critic model by using group-based baselines. + +:::{card} +**Goal**: Train a model to solve math problems by generating multiple solutions and reinforcing the correct ones. + +^^^ + +**Steps**: + +1. **Prepare Data**: Understand the prompt-only format. +2. **Configure**: Set group size and generation parameters. +3. **Train**: Run the `run_grpo_math.py` script. +4. **Verify**: Monitor the group rewards. +::: + +:::{button-ref} index +:color: secondary +:outline: +:ref-type: doc + +← Previous: Quickstart Guide +::: + +--- + +## Prerequisites + +* NeMo RL installed via `uv`. +* A base model capable of some reasoning (e.g., `Qwen/Qwen2.5-1.5B-Instruct`). + +--- + +## 1. Prepare Your Data + +GRPO is unique because it primarily needs **Prompts** (questions) and a way to verify the answer (Ground Truth). + +NeMo RL's math example uses the **OpenMathInstruct-2** dataset format. + +### The Data Format + +```json +{ + "problem": "What is 2 + 2?", + "generated_solution": "...", + "expected_answer": "4" +} +``` + +* **problem**: The input prompt given to the model. +* **expected_answer**: Used by the reward function (rule-based verifier) to score the model's output. + +You don't need a pre-built preference dataset like DPO; the algorithm generates its own data during training. + +--- + +## 2. Configure the Job + +The configuration is located at `examples/configs/grpo_math_1B.yaml`. + +Key parameters for GRPO: + +* **`grpo.num_generations_per_prompt`**: The group size ($G$). The model generates this many outputs for *each* prompt (e.g., 16). +* **`grpo.num_prompts_per_step`**: How many unique prompts to process in one batch. +* **`policy.model_name`**: The model being trained. +* **`policy.generation.temperature`**: Controls diversity. GRPO needs diverse outputs to find the correct answer, so `1.0` is common. + +--- + +## 3. Run the Training + +Run the `examples/run_grpo_math.py` script. This example uses a deterministic "Math Verifier" to reward correct answers. + +::::{tab-set} + +:::{tab-item} Native PyTorch (DTensor) +```bash +uv run python examples/run_grpo_math.py \ + grpo.max_num_steps=100 \ + grpo.num_generations_per_prompt=4 +``` +::: + +:::{tab-item} Megatron Core +```bash +uv run python examples/run_grpo_math.py \ + --config examples/configs/grpo_math_1B_megatron.yaml \ + grpo.max_num_steps=100 \ + grpo.num_generations_per_prompt=4 +``` +::: + +:::: + +**What's happening?** + +1. **Rollout**: The model generates 4 solutions for each math problem. +2. **Evaluation**: The system checks each solution against the correct answer. Correct = Reward 1.0, Incorrect = Reward 0.0. +3. **Update**: The model is updated to make the correct solutions more likely compared to the group average. + +--- + +## 4. Monitor and Verify + +GRPO training logs specific metrics that tell you if "Reasoning" is emerging: + +1. **`reward`**: The average accuracy of the group (mean reward). This should steadily increase. +2. **`policy_kl_error`**: How far the model has drifted from the original behavior (KL divergence). +3. **`advantages/mean`**: The average relative score of an output compared to its group peers. + +### Output Artifacts + +Results are saved to `results/grpo`: +* **Checkpoints**: The model weights, optimized for reasoning. + +--- + +## Scaling Up + +### Multi-GPU Training + +GRPO involves heavy generation (inference) and training. For larger models (7B+), you almost certainly need multiple GPUs. + +Use the Megatron backend for efficient scaling: + +```bash +uv run python examples/run_grpo_math.py \ + --config examples/configs/grpo_math_8B_megatron.yaml \ + cluster.gpus_per_node=8 +``` + +### Async Training + +For maximum throughput, you can decouple generation and training. See the [GRPO Guide](../guides/grpo.md) for advanced `async_grpo` configurations. diff --git a/docs/get-started/index.md b/docs/get-started/index.md index 2cc0849006..3f8d55a037 100644 --- a/docs/get-started/index.md +++ b/docs/get-started/index.md @@ -1,40 +1,246 @@ -# Quick Start +--- +description: "Quickstart guide to installing and running your first NeMo RL training job" +categories: ["getting-started"] +tags: ["quickstart", "installation", "tutorial"] +personas: ["data-scientist-focused", "mle-focused"] +difficulty: "beginner" +content_type: "tutorial" +--- -Use this quick start to get going with either the native PyTorch DTensor or Megatron Core training backends. +(gs-overview)= -> [!NOTE] -> Both training backends are independent — you can install and use either one on its own. +# Quickstart Guide -For more examples and setup details, continue to the [Prerequisites](installation.md) section. +Welcome to NeMo RL! -## Quick Start Options +:::{card} -| Native PyTorch (DTensor) | Megatron Core | -|--------------------------|---------------| -| **Clone and create the environment** | | +**Goal**: Install NeMo RL and run your first local training job. -```sh -git clone git@github.com:NVIDIA-NeMo/RL.git nemo-rl -cd nemo-rl -git submodule update --init --recursive -uv venv +^^^ + +**Steps**: + +1. Install `uv` and system prerequisites. +2. Clone the repository and initialize the environment. +3. Run a sample GRPO training job to verify installation. + +::: + +## Prerequisites + +* **OS**: Linux (Ubuntu 22.04/20.04 recommended) +* **Hardware**: + * NVIDIA GPU (Volta/Compute Capability 7.0+ required) + * Sufficient VRAM for the model and batch sizes configured in the example (memory requirements vary by configuration; reduce batch sizes if you encounter out-of-memory errors) +* **Software**: + * Python 3.12+ + * CUDA 12+ + * Git + +## 1. Installation + +:::{seealso} +For detailed system requirements, bare-metal setup (non-container), and troubleshooting, refer to the [Comprehensive Installation Guide](installation.md). +::: + +We use `uv` for fast, reliable package management. + +1. **Install `uv`** (if not installed): + + ```bash + curl -LsSf https://astral.sh/uv/install.sh | sh + source $HOME/.local/bin/env + ``` + +2. **Clone NeMo RL**: + Clone the repository with submodules to include all dependencies. + + ```bash + git clone git@github.com:NVIDIA-NeMo/RL.git nemo-rl --recursive + cd nemo-rl + + # If you cloned without recursive, run: + # git submodule update --init --recursive + ``` + + :::{warning} + If you cloned without the `--recursive` flag, you may need to rebuild virtual environments: + `NRL_FORCE_REBUILD_VENVS=true uv sync` + ::: + +3. **Initialize Environment**: + Create the virtual environment. + + ```bash + uv venv + ``` + + ```{note} + Do not use `-p/--python`. `uv` will automatically read the correct Python version from `.python-version`. + ``` + +## 2. Run Your First Job (Local) + +Let's verify your installation by running a **Group Relative Policy Optimization (GRPO)** training job. This example fine-tunes a small model on a math dataset. + +1. **Set Environment Variables**: + + ```bash + export HF_HOME=/path/to/your/hf_cache + export HF_DATASETS_CACHE=/path/to/your/hf_datasets_cache + # Optional: For logging + # export WANDB_API_KEY=your_key + ``` + +2. **Run the Training Script**: + Use `uv run` to execute the script. This automatically handles dependencies. + + ::::{tab-set} + + :::{tab-item} Native PyTorch (DTensor) + ```bash + uv run python examples/run_grpo_math.py + ``` + ::: + + :::{tab-item} Megatron Core + ```bash + uv run examples/run_grpo_math.py \ + --config examples/configs/grpo_math_1B_megatron.yaml + ``` + ::: + + :::: + + **What to expect**: + * NeMo RL will automatically start a local Ray cluster on your machine. + * It will download a small model (`Qwen/Qwen2.5-1.5B-Instruct` or similar) and dataset. + * You should see training logs indicating "Training started" and loss metrics streaming. + * The Ray dashboard URL will appear in the logs (typically `http://127.0.0.1:8265`). + + **Example output**: + ``` + Initializing Ray cluster... + Ray dashboard available at http://127.0.0.1:8265 + Loading model: Qwen/Qwen2.5-1.5B-Instruct + Training started... + Step 1: reward=0.25, policy_kl_error=0.001 + Step 2: reward=0.31, policy_kl_error=0.002 + ... + ``` + +### Local Development Tips + +Use these tips to manage your local resources and troubleshoot. + +:::{dropdown} 💡 How to Control GPU Usage & Run Concurrent Jobs + +**Controlling GPU Usage** + +By default, Ray detects and uses all available GPUs. To restrict a job to specific GPUs, use `CUDA_VISIBLE_DEVICES`: + +```bash +# Only use GPU 0 and 3 +CUDA_VISIBLE_DEVICES=0,3 uv run examples/run_grpo_math.py ``` -> [!NOTE] -> If you previously ran without checking out the submodules, you may need to rebuild virtual environments by setting `NRL_FORCE_REBUILD_VENVS=true`. See [Tips and Tricks](tips-and-tricks.md). +**Running Concurrent Jobs** -| Native PyTorch (DTensor) | Megatron Core | -|--------------------------|---------------| -| **Run GRPO (DTensor)** | **Run GRPO (Megatron)** | +You can run independent training jobs on the same machine by isolating them to different GPUs. Each job spins up its own isolated Ray instance. -```sh -# DTensor -uv run python examples/run_grpo_math.py +**Terminal 1 (Job A)**: + +```bash +CUDA_VISIBLE_DEVICES=0 uv run examples/run_grpo_math.py ``` -```sh -# Megatron -uv run examples/run_grpo_math.py \ - --config examples/configs/grpo_math_1B_megatron.yaml +**Terminal 2 (Job B)**: + +```bash +CUDA_VISIBLE_DEVICES=1 uv run examples/run_sft.py ``` +::: + +:::{dropdown} 🔍 Monitoring & Logs + +**Ray Dashboard** + +When a job starts, Ray provides a dashboard URL (`http://127.0.0.1:8265`) in the logs. Open this URL in your browser to view actor status, logs, and resource usage. + +**Weights & Biases** + +If you set `WANDB_API_KEY`, metrics stream to W&B. This is the recommended way to track training curves (loss, reward, KL divergence). +::: + +:::{dropdown} ℹ️ How NeMo RL manages the local cluster + +When you execute a training script (for example, `uv run ...`), NeMo RL: + +1. Checks for an existing Ray cluster. +2. If no cluster exists, it automatically starts a local Ray instance using your available resources. +3. It shuts down the cluster when the script finishes (unless connected to a persistent Ray server). + +You generally do **not** need to start Ray manually. +::: + +:::{dropdown} 🛠️ Troubleshooting + +* **"Resources not available"**: If a job hangs, check if another Ray instance is holding the GPUs. You may need to manually stop stray Ray processes: + + ```bash + ray stop + ``` + +* **OOM Errors**: If you run out of memory, try reducing the batch size or model size in the configuration YAML. +::: + +### How It Works + +NeMo RL uses a distributed architecture built on **Ray** to coordinate multiple components (RL Actors) during training: + +* **Policy Model**: The model being trained (e.g., Qwen, Llama) +* **Generation Backend**: Fast inference engine (vLLM) that generates responses +* **Environment**: Reward evaluator (e.g., Math verifier) that scores outputs +* **Training Backend**: PyTorch DTensor or Megatron Core for efficient distributed training + +Ray manages resource allocation, process isolation, and communication between these components, allowing NeMo RL to scale seamlessly from a single GPU to multi-node clusters. + +:::{seealso} +For more details on the architecture, design philosophy, and how RL Actors coordinate, refer to: +* [NeMo RL Overview](../about/overview.md) - High-level introduction and capabilities +* [Design and Philosophy](../design-docs/design-and-philosophy.md) - Deep dive into the architecture +* [Training Backends](../about/backends.md) - PyTorch DTensor vs Megatron Core comparison +::: + +## 3. Choose Your Path + +Now that you have a working setup, choose the workflow that matches your goal. + +::::{grid} 1 1 1 3 +:gutter: 2 + +:::{grid-item-card} {octicon}`mortar-board;1.5em;sd-mr-1` Fine-Tune (SFT) +:link: gs-sft +:link-type: ref +**Start here** if you have a base model and want to teach it instructions. SFT is the standard first step in aligning language models using supervised learning on instruction-response pairs. +::: + +:::{grid-item-card} {octicon}`graph;1.5em;sd-mr-1` Align (DPO) +:link: gs-dpo +:link-type: ref +**Start here** if you have preference data (chosen vs rejected pairs) and want to align your model to human preferences. DPO learns directly from preference comparisons without needing a separate reward model. +::: + +:::{grid-item-card} {octicon}`rocket;1.5em;sd-mr-1` Reinforce (GRPO) +:link: gs-grpo +:link-type: ref +**Start here** for reasoning tasks (math, coding) where you can verify correctness programmatically. GRPO is efficient for on-policy RL without requiring a separate critic model—perfect for tasks with deterministic rewards. +::: + +:::: + +## Advanced Setup + +* **Cluster Setup**: Ready to scale? Set up multi-node training on [Slurm or Kubernetes](cluster.md). diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index 4b7c9ba89b..dc36488476 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -1,92 +1,213 @@ +--- +description: "Detailed step-by-step guide for installing NeMo RL, including bare-metal dependencies and troubleshooting" +categories: ["getting-started", "setup"] +tags: ["installation", "dependencies", "setup", "system-requirements"] +personas: ["data-scientist-focused", "mle-focused", "cluster-administrator-focused"] +difficulty: "beginner" +content_type: "tutorial" +--- + # Installation and Prerequisites -## Clone the Repository +Use this guide if you are setting up on a bare-metal system, need specific backend dependencies (like Megatron or vLLM), or are troubleshooting your environment. -Clone **NeMo RL** with submodules: +:::{card} +**Goal**: Fully configure your system, install NeMo RL dependencies, and prepare the virtual environment. -```sh -git clone git@github.com:NVIDIA-NeMo/RL.git nemo-rl --recursive -cd nemo-rl +^^^ -# If you are already cloned without the recursive option, you can initialize the submodules recursively -git submodule update --init --recursive +**Steps**: -# Different branches of the repo can have different pinned versions of these third-party submodules. Ensure -# submodules are automatically updated after switching branches or pulling updates by configuring git with: -# git config submodule.recurse true +1. **System Dependencies**: Install backend-specific libraries (cuDNN, libibverbs). +2. **Clone**: Get the source code with submodules. +3. **Package Manager**: Install `uv`. +4. **Virtual Env**: Create and verify your Python environment. +::: -# **NOTE**: this setting will not download **new** or remove **old** submodules with the branch's changes. -# You will have to run the full `git submodule update --init --recursive` command in these situations. -``` +:::{button-ref} index +:color: secondary +:outline: +:ref-type: doc + +← Back to Quickstart +::: + +--- + +## 1. Install System Dependencies -## Install System Dependencies +Before installing the Python package, ensure your operating system has the required libraries for your chosen backend. -### cuDNN (For Megatron Backend) +:::{note} +If you are using a pre-built NVIDIA container (e.g., from NGC), most of these dependencies are likely pre-installed. These steps are critical for **bare-metal** installations (e.g., a fresh Ubuntu server). +::: -If you are using the Megatron backend on bare metal (outside of a container), you may need to install the cuDNN headers. Here is how you check and install them: +::::{tab-set} +:::{tab-item} Megatron Backend (Bare Metal) +If you plan to use the **Megatron Core** backend, you must have the cuDNN headers installed. + +**Check for existing installation:** ```sh -# Check if you have libcudnn installed dpkg -l | grep cudnn.*cuda - -# Find the version you need here: https://developer.nvidia.com/cudnn-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=deb_network -# As an example, these are the "Linux Ubuntu 20.04 x86_64" instructions -wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.1-1_all.deb -sudo dpkg -i cuda-keyring_1.1-1_all.deb -sudo apt update -sudo apt install cudnn # Will install cuDNN meta packages which points to the latest versions -# sudo apt install cudnn9-cuda-12 # Will install cuDNN version 9.x.x compiled for cuda 12.x -# sudo apt install cudnn9-cuda-12-8 # Will install cuDNN version 9.x.x compiled for cuda 12.8 ``` -### libibverbs (For vLLM Dependencies) +**Install cuDNN (Ubuntu 20.04/22.04 example):** + +Find the correct version for your system at the [NVIDIA cuDNN Downloads page](https://developer.nvidia.com/cudnn-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=deb_network). + +1. Add the NVIDIA repo key: + ```sh + wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.1-1_all.deb + sudo dpkg -i cuda-keyring_1.1-1_all.deb + sudo apt update + ``` + +2. Install the libraries: + ```sh + # Installs meta-packages pointing to the latest versions + sudo apt install cudnn -If you encounter problems when installing vllm's dependency `deepspeed` on bare-metal (outside of a container), you may need to install `libibverbs-dev`: + # OR install specific versions (adjust version numbers as needed) + # sudo apt install cudnn9-cuda-12 + ``` +::: +:::{tab-item} vLLM / DeepSpeed +For the **vLLM** inference backend (often used with DeepSpeed), `libibverbs-dev` is required on bare metal to avoid build errors. + +**Install libibverbs:** ```sh sudo apt-get update sudo apt-get install libibverbs-dev ``` +::: -## Install UV Package Manager +:::: -For faster setup and environment isolation, we use [uv](https://docs.astral.sh/uv/). +--- -Follow [these instructions](https://docs.astral.sh/uv/getting-started/installation/) to install uv. +## 2. Clone the Repository -Quick install: -```sh -curl -LsSf https://astral.sh/uv/install.sh | sh -``` +NeMo RL relies on several third-party libraries included as git submodules. You must clone the repository recursively. -## Create Virtual Environment +1. **Clone with recursion**: + ```sh + git clone git@github.com:NVIDIA-NeMo/RL.git nemo-rl --recursive + cd nemo-rl + ``` -Initialize the NeMo RL project virtual environment: +2. **(Optional) Initialize existing clone**: + If you already cloned without the `--recursive` flag, fix it by running: + ```sh + git submodule update --init --recursive + ``` + +:::{tip} Keep Submodules in Sync +Different branches may pin different versions of submodules. To ensure they update automatically when you switch branches or pull, configure git: ```sh -uv venv +git config submodule.recurse true ``` +*Note: This will not remove old submodules or download new ones if the directory structure changes significantly; in those cases, run the full update command above.* +::: + +--- + +## 3. Install UV Package Manager + +We use `uv` for fast, reliable, and isolated Python package management. + +1. **Install `uv`**: + Follow the official [uv installation guide](https://docs.astral.sh/uv/getting-started/installation/) or use the quick script: + ```sh + curl -LsSf https://astral.sh/uv/install.sh | sh + ``` + +2. **Verify installation**: + ```sh + uv --version + ``` + +--- -> [!NOTE] -> Please do not use `-p/--python` and instead allow `uv venv` to read it from `.python-version`. -> This ensures that the version of python used is always what we prescribe. +## 4. Create Virtual Environment -## Using UV to Run Commands +Initialize the project-specific virtual environment. NeMo RL uses a `.python-version` file to pin the supported Python version automatically. -Use `uv run` to launch all commands. It handles pip installing implicitly and ensures your environment is up to date with our lock file. +1. **Create the venv**: + ```sh + uv venv + ``` + + :::{important} + Do **not** specify a python version manually (e.g., `-p python3.10`). Let `uv` read the correct version from the configuration file to ensure compatibility. + ::: + +2. **(Optional) Rebuilding Environments**: + If you change branches or modify `pyproject.toml` significantly, you may need to force a rebuild of the environment variables and dependencies: + ```sh + NRL_FORCE_REBUILD_VENVS=true uv sync + ``` + +--- + +## 5. Using UV to Run Commands + +In NeMo RL, we recommend using `uv run` to execute scripts rather than manually activating the virtual environment. This ensures you are always using the locked dependencies and correct environment variables. + +**Examples**: + +* **Run a Python script**: + ```sh + uv run python examples/run_grpo_math.py + ``` + +* **Run with arguments**: + ```sh + uv run python examples/run_grpo_math.py --config examples/configs/grpo_math_1B_megatron.yaml + ``` + +--- + +## 6. Configure Access Tokens + +Some models and datasets require authentication. + +**Hugging Face Token** (for gated models like Llama): ```sh -# Example: Run GRPO with DTensor backend -uv run python examples/run_grpo_math.py +# Set the token (avoids huggingface-cli login prompts) +export HF_TOKEN=your_token_here + +# OR login interactively (required for Llama and other gated models) +huggingface-cli login +``` + +**Weights & Biases** (for experiment tracking): -# Example: Run GRPO with Megatron backend -uv run python examples/run_grpo_math.py --config examples/configs/grpo_math_1B_megatron.yaml +```sh +export WANDB_API_KEY=your_key_here ``` -> [!NOTE] -> - It is not recommended to activate the `venv`, and you should use `uv run ` instead to execute scripts within the managed environment. -> This ensures consistent environment usage across different shells and sessions. -> - Ensure your system has the appropriate CUDA drivers installed, and that your PyTorch version is compatible with both your CUDA setup and hardware. -> - If you update your environment in `pyproject.toml`, it is necessary to force a rebuild of the virtual environments by setting `NRL_FORCE_REBUILD_VENVS=true` next time you launch a run. -> - **Reminder**: Don't forget to set your `HF_HOME`, `WANDB_API_KEY`, and `HF_DATASETS_CACHE` (if needed). You'll need to do a `huggingface-cli login` as well for Llama models. +:::{tip} +Add these exports to your `~/.bashrc` or `~/.zshrc` so they persist across sessions. +::: + +--- + +## Troubleshooting + +:::{dropdown} CUDA Compatibility +Ensure your system has the appropriate CUDA drivers installed and that your PyTorch version is compatible with both your CUDA setup and hardware. Run `nvidia-smi` to check your driver version. +::: + +:::{dropdown} Gated Model Access Errors +If you see authentication errors when downloading models like Llama: +1. Accept the model license on Hugging Face (visit the model page) +2. Run `huggingface-cli login` or set `HF_TOKEN` +::: +:::{seealso} +Ready to run your first job? Go back to the [Quickstart Guide](index.md) or jump to [Supervised Fine-Tuning (SFT)](sft.md). +::: diff --git a/docs/get-started/sft.md b/docs/get-started/sft.md new file mode 100644 index 0000000000..cfa2941d6d --- /dev/null +++ b/docs/get-started/sft.md @@ -0,0 +1,153 @@ +--- +description: "Step-by-step guide to running Supervised Fine-Tuning (SFT) with NeMo RL" +categories: ["getting-started"] +tags: ["sft", "quickstart", "supervised-fine-tuning", "python-api"] +personas: ["data-scientist-focused", "mle-focused"] +difficulty: "beginner" +content_type: "tutorial" +--- + +(gs-sft)= + +# Get Started with SFT + +**Supervised Fine-Tuning (SFT)** is the standard first step in aligning language models. + +:::{card} +**Goal**: Train a basic model (Llama-3.2-1B) on a sample dataset using your local machine. + +^^^ + +**Steps**: + +1. **Prepare Data**: Understand the input format. +2. **Configure**: Set hyperparameters in the YAML configuration. +3. **Train**: Run the `run_sft.py` script. +4. **Verify**: Check logs and model artifacts. +::: + +:::{button-ref} index +:color: secondary +:outline: +:ref-type: doc + +← Previous: Quickstart Guide +::: + +--- + +## Prerequisites + +Ensure you have completed the [Quickstart Installation](index.md). You should have: + +* NeMo RL installed via `uv` +* Environment variables set (`HF_HOME`, etc.) + +--- + +## 1. Prepare Your Data + +SFT requires a dataset of "Instruction" and "Response" pairs. The goal is to teach the model to generate the target response given the instruction. + +By default, the example script uses the **SQuAD** dataset, but you can use your own data. + +### The Data Format + +NeMo RL supports standard JSONL files where each line is a training example. + +```json +{"input": "What is the capital of France?", "output": "The capital of France is Paris."} +{"input": "Write a python function to add two numbers.", "output": "def add(a, b):\n return a + b"} +``` + +You can specify your data in the configuration file (see Step 2). + +--- + +## 2. Configure the Job + +NeMo RL uses **Hydra** for configuration, allowing you to manage parameters in structured YAML files. The default configuration is at `examples/configs/sft.yaml`. + +Key parameters you might want to change: + +* **`policy.model_name`**: The base model to start from (e.g., `meta-llama/Llama-3.2-1B`). +* **`data.train_data_path`**: Path to your training `.jsonl` file. +* **`sft.max_num_epochs`**: How many times to iterate over the dataset. +* **`policy.optimizer.kwargs.lr`**: The step size for the optimizer. + +:::{tip} +You don't need to edit the YAML file directly. You can override any parameter from the command line (shown in Step 3). +::: + +--- + +## 3. Run the Training + +We will use the `examples/run_sft.py` script to start the training. + +::::{tab-set} + +:::{tab-item} Native PyTorch (DTensor) +```bash +uv run python examples/run_sft.py \ + sft.max_num_epochs=1 \ + sft.max_num_steps=100 \ + policy.model_name="meta-llama/Llama-3.2-1B" +``` +::: + +:::{tab-item} Megatron Core +```bash +uv run python examples/run_sft.py \ + sft.max_num_epochs=1 \ + sft.max_num_steps=100 \ + policy.model_name="meta-llama/Llama-3.2-1B" \ + policy.megatron_cfg.enabled=true \ + policy.dtensor_cfg.enabled=false +``` +::: + +:::: + +**What's happening?** + +* `uv run`: Ensures the script runs in the managed environment. +* `sft.max_num_epochs=1`: Overrides the configuration to run for just 1 epoch. +* `sft.max_num_steps=100`: Limits the run to 100 steps (good for a quick test). + +--- + +## 4. Monitor and Verify + +Once the script starts, it will print logs to your console. Watch for these key indicators: + +1. **Data Loading**: Look for "Training and validation datasets loaded". +2. **Loss**: You should see the loss value printed periodically. Ideally, this number should decrease over time. + +### Output Artifacts + +By default, NeMo RL saves results to the `results/sft` directory (defined in `checkpointing.checkpoint_dir`). + +* **Checkpoints**: Saved model weights (e.g., `results/sft/checkpoints/`). +* **Logs**: Training metrics (e.g., `logs/`). + +To verify your training was successful, check that a checkpoint file exists in the results directory. + +--- + +## Scaling Up + +Once you are comfortable with the basic workflow, you can scale up to larger models and multi-GPU training. + +### Using Megatron Core + +For high-performance training on large models, switch to a Megatron-compatible configuration: + +```bash +uv run python examples/run_sft.py \ + --config examples/configs/sft_openmathinstruct2_megatron.yaml +``` + +### Multi-Node Training + +To run on a cluster (Slurm or Kubernetes), refer to the [Cluster Setup](cluster.md) guide. diff --git a/docs/guides/dpo.md b/docs/guides/dpo.md index f00dde9f12..6229aa5d57 100644 --- a/docs/guides/dpo.md +++ b/docs/guides/dpo.md @@ -6,7 +6,7 @@ to increase the probability of the chosen response and decrease the probability ## Launch a DPO Run -The script [examples/run_dpo.py](../../examples/run_dpo.py) can be used to launch a DPO experiment. This script can either be launched locally or via Slurm. For details on how to set up Ray and launch a job using Slurm, refer to the [cluster documentation](../cluster.md). +The script [examples/run_dpo.py](../../examples/run_dpo.py) can be used to launch a DPO experiment. This script can either be launched locally or via Slurm. For details on how to set up Ray and launch a job using Slurm, refer to the [cluster documentation](../get-started/cluster.md). Be sure to launch the job using `uv`. The command to launch a DPO job is as follows: ```bash diff --git a/docs/guides/grpo.md b/docs/guides/grpo.md index e396e66cd6..73fe0349b6 100755 --- a/docs/guides/grpo.md +++ b/docs/guides/grpo.md @@ -4,7 +4,7 @@ This guide details the Group Relative Policy Optimization (GRPO) implementation ## Quickstart: Launch a GRPO Run -To get started quickly, use the script [examples/run_grpo_math.py](../../examples/run_grpo_math.py), which demonstrates how to train a model on math problems using GRPO. You can launch this script locally or via Slurm. For detailed instructions on setting up Ray and launching a job with Slurm, refer to the [cluster documentation](../cluster.md). +To get started quickly, use the script [examples/run_grpo_math.py](../../examples/run_grpo_math.py), which demonstrates how to train a model on math problems using GRPO. You can launch this script locally or via Slurm. For detailed instructions on setting up Ray and launching a job with Slurm, refer to the [cluster documentation](../get-started/cluster.md). We recommend launching the job using `uv`: diff --git a/docs/guides/rm.md b/docs/guides/rm.md index f5deb05f0d..ee53222786 100644 --- a/docs/guides/rm.md +++ b/docs/guides/rm.md @@ -4,7 +4,7 @@ This document explains how to train reward models (RM) within NeMo RL. Currently ## Launch a Training Job -The script, [examples/run_rm.py](../../examples/run_rm.py), is used to train a Bradley-Terry reward model. This script can be launched either locally or via Slurm. For details on how to set up Ray and launch a job using Slurm, refer to the [cluster documentation](../cluster.md). +The script, [examples/run_rm.py](../../examples/run_rm.py), is used to train a Bradley-Terry reward model. This script can be launched either locally or via Slurm. For details on how to set up Ray and launch a job using Slurm, refer to the [cluster documentation](../get-started/cluster.md). Be sure to launch the job using `uv`. The command to launch a training job is as follows: diff --git a/docs/guides/sft.md b/docs/guides/sft.md index 982052f074..9cb16af655 100644 --- a/docs/guides/sft.md +++ b/docs/guides/sft.md @@ -4,7 +4,7 @@ This document explains how to perform SFT within NeMo RL. It outlines key operat ## Launch an SFT Run -The script, [examples/run_sft.py](../../examples/run_sft.py), can be used to launch an experiment. This script can be launched either locally or via Slurm. For details on how to set up Ray and launch a job using Slurm, refer to the [cluster documentation](../cluster.md). +The script, [examples/run_sft.py](../../examples/run_sft.py), can be used to launch an experiment. This script can be launched either locally or via Slurm. For details on how to set up Ray and launch a job using Slurm, refer to the [cluster documentation](../get-started/cluster.md). Be sure to launch the job using `uv`. The command to launch an SFT job is as follows: diff --git a/docs/index.md b/docs/index.md index 954518c4e1..73a4c1fd0d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,263 +1,283 @@ +--- +description: "NeMo RL is an open-source post-training library for scaling reinforcement learning methods for multimodal models (LLMs, VLMs, etc.)" +categories: + - documentation + - home +tags: + - reinforcement-learning + - post-training + - scalable + - distributed + - llm-training +personas: + - Data Scientists + - Machine Learning Engineers + - Cluster Administrators +difficulty: beginner +content_type: index +--- + +(rl-home)= + # NeMo RL Documentation -Welcome to the NeMo RL documentation. NeMo RL is an open-source post-training library developed by NVIDIA, designed to streamline and scale reinforcement learning methods for multimodal models (LLMs, VLMs, etc.). +**NeMo RL** is an open-source post-training library within the [NeMo Framework](https://github.com/NVIDIA-NeMo), designed to streamline and scale reinforcement learning methods for multimodal models (LLMs, VLMs, etc.). Designed for flexibility, reproducibility, and scale, NeMo RL enables both small-scale experiments and massive multi-GPU, multi-node deployments for fast experimentation in research and production environments. -This documentation provides comprehensive guides, examples, and references to help you get started with NeMo RL and build powerful post-training pipelines for your models. +## Introduction to NeMo RL -## Getting Started +Learn about NeMo RL, how it works at a high-level, and the key features. -::::{grid} 1 1 2 2 -:gutter: 3 +::::{grid} 1 2 2 2 +:gutter: 1 1 1 2 -:::{grid-item-card} {octicon}`book` Overview +:::{grid-item-card} {octicon}`book;1.5em;sd-mr-1` About NeMo RL :link: about/overview :link-type: doc - -Learn about NeMo RL's architecture, design philosophy, and key features that make it ideal for scalable reinforcement learning. -::: - -:::{grid-item-card} {octicon}`rocket` Quick Start -:link: about/quick-start -:link-type: doc - -Get up and running quickly with examples for both DTensor and Megatron Core training backends. -::: - -:::{grid-item-card} {octicon}`download` Installation -:link: about/installation -:link-type: doc - -Step-by-step instructions for installing NeMo RL, including prerequisites, system dependencies, and environment setup. +Overview of NeMo RL and its capabilities. ++++ +{bdg-secondary}`architecture` {bdg-secondary}`design-philosophy` {bdg-secondary}`scalable-rl` ::: -:::{grid-item-card} {octicon}`star` Features +:::{grid-item-card} {octicon}`star;1.5em;sd-mr-1` Key Features :link: about/features :link-type: doc - -Explore the current features and upcoming enhancements in NeMo RL, including distributed training, advanced parallelism, and more. +Discover the main features of NeMo RL for post-training. ++++ +{bdg-secondary}`algorithms` {bdg-secondary}`backends` {bdg-secondary}`distributed-training` ::: -:::{grid-item-card} {octicon}`light-bulb` Tips and Tricks -:link: about/tips-and-tricks +:::{grid-item-card} {octicon}`cpu;1.5em;sd-mr-1` Training Backends +:link: about/backends :link-type: doc - -Troubleshooting common issues including missing submodules, Ray dashboard access, and debugging techniques. +Explore PyTorch DTensor and Megatron Core training backends and how to choose the right one. ++++ +{bdg-secondary}`dtensor` {bdg-secondary}`megatron-core` {bdg-secondary}`parallelism` ::: :::: -## Training and Generation +## Get Started -::::{grid} 1 1 2 2 -:gutter: 3 +Start here to install NeMo RL and run your first training job. -:::{grid-item-card} {octicon}`cpu` Training Backends -:link: about/backends -:link-type: doc +::::{grid} 1 2 2 2 +:gutter: 1 1 1 2 -Learn about DTensor and Megatron Core training backends, their capabilities, and how to choose the right one for your use case. +:::{grid-item-card} {octicon}`play;1.5em;sd-mr-1` Installation & Quickstart +:link: get-started/index +:link-type: doc +**Start Here**: Install NeMo RL and run your first local training job in minutes. ++++ +{bdg-primary}`installation` {bdg-secondary}`first-run` ::: -:::{grid-item-card} {octicon}`workflow` Algorithms -:link: about/algorithms/index +:::{grid-item-card} {octicon}`rocket;1.5em;sd-mr-1` GRPO Worfklow +:link: get-started/grpo :link-type: doc - -Discover supported algorithms including GRPO, SFT, DPO, RM, and on-policy distillation with detailed guides and examples. +Run Group Relative Policy Optimization (GRPO) training. ++++ +{bdg-secondary}`on-policy` {bdg-secondary}`reinforcement-learning` ::: -:::{grid-item-card} {octicon}`graph` Evaluation -:link: about/evaluation +:::{grid-item-card} {octicon}`mortar-board;1.5em;sd-mr-1` SFT Worfklow +:link: get-started/sft :link-type: doc - -Learn how to evaluate your models using built-in evaluation datasets and custom evaluation pipelines. +Run supervised fine-tuning (SFT) on instruction datasets. ++++ +{bdg-secondary}`fine-tuning` {bdg-secondary}`instruction-following` ::: -:::{grid-item-card} {octicon}`server` Cluster Setup -:link: about/clusters +:::{grid-item-card} {octicon}`graph;1.5em;sd-mr-1` DPO Worfklow +:link: get-started/dpo :link-type: doc - -Configure and deploy NeMo RL on multi-node Slurm or Kubernetes clusters for distributed computing. +Run Direct Preference Optimization (DPO) training. ++++ +{bdg-secondary}`preference-learning` {bdg-secondary}`alignment` ::: :::: -## Guides and Examples +## Training Algorithms -::::{grid} 1 1 2 2 -:gutter: 3 +Explore how you can use NeMo RL with different training algorithms. -:::{grid-item-card} {octicon}`mortar-board` GRPO DeepscaleR -:link: guides/grpo-deepscaler -:link-type: doc +::::{grid} 1 2 2 2 +:gutter: 1 1 1 2 -Reproduce DeepscaleR results with NeMo RL using GRPO on mathematical reasoning tasks. +:::{grid-item-card} {octicon}`rocket;1.5em;sd-mr-1` GRPO +:link: guides/grpo +:link-type: doc +Group Relative Policy Optimization for efficient on-policy reinforcement learning. ++++ +{bdg-secondary}`on-policy-rl` {bdg-secondary}`reward-optimization` {bdg-secondary}`multi-turn` ::: -:::{grid-item-card} {octicon}`number` SFT on OpenMathInstruct2 -:link: guides/sft-openmathinstruct2 +:::{grid-item-card} {octicon}`mortar-board;1.5em;sd-mr-1` Supervised Fine-Tuning +:link: guides/sft :link-type: doc - -Step-by-step guide for supervised fine-tuning on the OpenMathInstruct2 dataset. +Fine-tune models on instruction-following datasets with supervised learning. ++++ +{bdg-secondary}`instruction-tuning` {bdg-secondary}`supervised-learning` ::: -:::{grid-item-card} {octicon}`stack` Environments -:link: guides/environments +:::{grid-item-card} {octicon}`graph;1.5em;sd-mr-1` DPO +:link: guides/dpo :link-type: doc - -Create custom reward environments and integrate them with NeMo RL training pipelines. +Direct Preference Optimization for preference-based training without reward models. ++++ +{bdg-secondary}`preference-learning` {bdg-secondary}`alignment` ::: -:::{grid-item-card} {octicon}`plus-circle` Adding New Models -:link: adding-new-models +:::{grid-item-card} {octicon}`trophy;1.5em;sd-mr-1` Reward Modeling +:link: guides/rm :link-type: doc - -Learn how to add support for new model architectures in NeMo RL. +Train reward models for preference learning and evaluation. ++++ +{bdg-secondary}`reward-models` {bdg-secondary}`preference-learning` ::: :::: -## Advanced Topics +## Tutorial Highlights -::::{grid} 1 1 2 2 -:gutter: 3 +Check out tutorials to get a quick start on using NeMo RL. -:::{grid-item-card} {octicon}`telescope` Design and Philosophy -:link: design-docs/design-and-philosophy -:link-type: doc - -Deep dive into NeMo RL's architecture, APIs, and design decisions for scalable RL. -::: +::::{grid} 1 2 2 2 +:gutter: 1 1 1 2 -:::{grid-item-card} {octicon}`bug` Debugging -:link: debugging +:::{grid-item-card} {octicon}`mortar-board;1.5em;sd-mr-1` GRPO DeepscaleR +:link: guides/grpo-deepscaler :link-type: doc - -Tools and techniques for debugging distributed Ray applications and RL training runs. +Reproduce DeepscaleR results with NeMo RL using GRPO on mathematical reasoning tasks. ++++ +{bdg-secondary}`mathematical-reasoning` {bdg-secondary}`reproduction` ::: -:::{grid-item-card} {octicon}`zap` FP8 Quantization -:link: fp8 +:::{grid-item-card} {octicon}`number;1.5em;sd-mr-1` SFT on OpenMathInstruct2 +:link: guides/sft-openmathinstruct2 :link-type: doc - -Optimize large language models with FP8 quantization for faster training and inference. +Step-by-step guide for supervised fine-tuning on the OpenMathInstruct2 dataset. ++++ +{bdg-secondary}`math-datasets` {bdg-secondary}`instruction-tuning` ::: -:::{grid-item-card} {octicon}`container` Docker Containers -:link: docker +:::{grid-item-card} {octicon}`stack;1.5em;sd-mr-1` Custom Environments +:link: guides/environments :link-type: doc - -Build and use Docker containers for reproducible NeMo RL environments. +Create custom reward environments and integrate them with NeMo RL training pipelines. ++++ +{bdg-secondary}`custom-rewards` {bdg-secondary}`environment-integration` ::: -:::: - -## API Reference - -::::{grid} 1 1 1 1 -:gutter: 3 - -:::{grid-item-card} {octicon}`code` Complete API Documentation -:link: apidocs/index +:::{grid-item-card} {octicon}`plus-circle;1.5em;sd-mr-1` Adding New Models +:link: adding-new-models :link-type: doc - -Comprehensive reference for all NeMo RL modules, classes, functions, and methods. Browse the complete Python API with detailed docstrings and usage examples. +Learn how to add support for new model architectures in NeMo RL. ++++ +{bdg-secondary}`model-integration` {bdg-secondary}`custom-models` ::: :::: -```{toctree} -:caption: About -:hidden: - -about/overview -about/performance-summary -about/features -about/backends -about/quick-start -about/installation -about/algorithms/index -about/evaluation -about/clusters -about/tips-and-tricks -``` - - +--- -```{toctree} -:caption: Environment Start +::::{toctree} :hidden: +Home +:::: -local-workstation.md -cluster.md - -``` - -```{toctree} -:caption: E2E Examples +::::{toctree} :hidden: +:caption: About NeMo RL +:maxdepth: 1 +about/overview.md +about/features.md +about/backends.md +about/algorithms/index.md +about/evaluation.md +about/tips-and-tricks.md +about/performance-summary.md +:::: -guides/sft-openmathinstruct2.md -``` +::::{toctree} +:hidden: +:caption: Get Started +:maxdepth: 2 + +Quickstart +Installation +SFT +DPO +GRPO +Cluster Setup +:::: -```{toctree} -:caption: Guides +::::{toctree} :hidden: +:caption: Training Algorithms +:maxdepth: 2 -adding-new-models.md +guides/grpo.md +guides/dapo.md guides/sft.md guides/dpo.md -guides/dapo.md -guides/grpo.md -guides/grpo-deepscaler.md -guides/grpo-sliding-puzzle.md guides/rm.md guides/environments.md guides/eval.md -guides/deepseek.md -model-quirks.md -guides/async-grpo.md -``` +:::: -```{toctree} -:caption: Containers +::::{toctree} :hidden: +:caption: Examples & Tutorials +:maxdepth: 2 -docker.md -``` +guides/grpo-deepscaler.md +guides/sft-openmathinstruct2.md +guides/grpo-sliding-puzzle.md +guides/deepseek.md +guides/async-grpo.md +adding-new-models.md +model-quirks.md +:::: -```{toctree} -:caption: Development +::::{toctree} :hidden: +:caption: Setup & Deployment +:maxdepth: 2 -testing.md -documentation.md -debugging.md -nsys-profiling.md -fp8.md -guides/use-custom-vllm.md -``` +get-started/cluster.md +docker.md +:::: -```{toctree} -:caption: Design Docs +::::{toctree} :hidden: +:caption: Advanced Topics +:maxdepth: 2 design-docs/design-and-philosophy.md -design-docs/padding.md -design-docs/logger.md -design-docs/uv.md -design-docs/chat-datasets.md +design-docs/training-backends.md design-docs/generation.md design-docs/checkpointing.md design-docs/loss-functions.md -design-docs/fsdp2-parallel-plan.md -design-docs/training-backends.md design-docs/sequence-packing-and-dynamic-batching.md +design-docs/fsdp2-parallel-plan.md +design-docs/padding.md +design-docs/logger.md +design-docs/chat-datasets.md +design-docs/uv.md design-docs/env-vars.md -``` +debugging.md +fp8.md +nsys-profiling.md +testing.md +documentation.md +guides/use-custom-vllm.md +:::: -```{toctree} -:caption: API Reference +::::{toctree} :hidden: +:caption: Reference +:maxdepth: 2 -apidocs/index -``` +apidocs/index.rst +:::: diff --git a/docs/local-workstation.md b/docs/local-workstation.md deleted file mode 100644 index b99da39ca9..0000000000 --- a/docs/local-workstation.md +++ /dev/null @@ -1,35 +0,0 @@ -# Run on Your Local Workstation - -When launching examples locally with `uv`, {py:class}`init_ray() ` will first attempt to connect to an existing cluster. If none is found, it will start a local one and connect to it using all available GPU and CPU resources on your node. - -To launch a job outside of a container, simply run: - -```sh -uv run examples/run_grpo_math.py -``` - -In the logs, you will see that Ray has started a local cluster instance, along with details on the resources made available to it: -``` -2025-03-17 13:37:45,360 INFO worker.py:1841 -- Started a local Ray instance. -... -INFO:nemo_rl.distributed.virtual_cluster:Started local cluster with: {'node:__internal_head__': 1.0, 'CPU': 24.0, 'object_store_memory': 80448493977.0, 'accelerator_type:RTX': 1.0, 'memory': 177713152615.0, 'GPU': 1.0, 'node:10.0.0.1': 1.0} -``` - -To have more precise control over the GPUs Ray uses locally, please use `CUDA_VISIBLE_DEVICES`: - -```sh -# Use the 0th and 3rd indexed GPU (for a total of 2 GPUs) -CUDA_VISIBLE_DEVICES=0,3 uv run examples/run_grpo_math.py -``` - -We also allow multiple colocated local clusters, which are uniquely identified by the values in -`CUDA_VISIBLE_DEVICES`. Concretely: - -```sh -# (1) Start a fresh cluster on GPU=0 -CUDA_VISIBLE_DEVICES=0 uv run examples/run_grpo_math.py - -# (2) While (1) is running, this will start a new cluster using GPUs 1 and 2 without interferring with (1) -# Ensure that the CUDA_VISIBLE_DEVICES do not overlap already running jobs. -CUDA_VISIBLE_DEVICES=1,2 uv run examples/run_grpo_math.py -``` From 51225453dd21c2d29dbc073706b774707992d068 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Thu, 4 Dec 2025 15:03:31 -0500 Subject: [PATCH 03/11] docs(get-started): revert to original sbatch syntax (--gres, --time); add GPU flag note Signed-off-by: Lawrence Lane --- docs/get-started/cluster.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index 3d488f5555..1a7ab5e53e 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -83,14 +83,15 @@ sbatch \ --nodes=${NUM_ACTOR_NODES} \ --account=YOUR_ACCOUNT \ --partition=YOUR_PARTITION \ - --gpus-per-node=8 \ - --time=04:00:00 \ + --gres=gpu:8 \ + --time=1:0:0 \ --job-name=nemo-rl-interactive \ ray.sub ``` :::{tip} -Replace `YOUR_ACCOUNT` and `YOUR_PARTITION` with values from your cluster. Run `sacctmgr show associations user=$USER` to find your account. +- Replace `YOUR_ACCOUNT` and `YOUR_PARTITION` with values from your cluster. Run `sacctmgr show associations user=$USER` to find your account. +- Depending on your Slurm cluster configuration, you may need `--gres=gpu:8` or `--gpus-per-node=8`. Check with your cluster admin if jobs don't receive GPUs. ::: **2. Attach to the Cluster** @@ -126,8 +127,8 @@ sbatch \ --nodes=${NUM_ACTOR_NODES} \ --account=YOUR_ACCOUNT \ --partition=YOUR_PARTITION \ - --gpus-per-node=8 \ - --time=24:00:00 \ + --gres=gpu:8 \ + --time=1:0:0 \ --job-name=nemo-rl-prod \ ray.sub ``` From 955c4a3be51e312a730dca2271518feaca38375c Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Thu, 4 Dec 2025 15:08:53 -0500 Subject: [PATCH 04/11] docs(get-started): revert job-name to YOUR_JOBNAME placeholder for consistency Signed-off-by: Lawrence Lane --- docs/get-started/cluster.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index 1a7ab5e53e..74fbde1373 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -85,7 +85,7 @@ sbatch \ --partition=YOUR_PARTITION \ --gres=gpu:8 \ --time=1:0:0 \ - --job-name=nemo-rl-interactive \ + --job-name=YOUR_JOBNAME \ ray.sub ``` @@ -129,7 +129,7 @@ sbatch \ --partition=YOUR_PARTITION \ --gres=gpu:8 \ --time=1:0:0 \ - --job-name=nemo-rl-prod \ + --job-name=YOUR_JOBNAME \ ray.sub ``` From b0834190493b806bf836e6a24474ad2d729ca1b9 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Thu, 4 Dec 2025 15:18:16 -0500 Subject: [PATCH 05/11] fix tabs Signed-off-by: Lawrence Lane --- docs/get-started/cluster.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index 74fbde1373..b1a7c13dda 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -65,9 +65,9 @@ Copy the command for your training type: | **Reward Model** | `uv run examples/run_rm.py` | | **GRPO** | `uv run examples/run_grpo_math.py` | -::::{tab-set} +:::::{tab-set} -:::{tab-item} Interactive Mode (Recommended for Debugging) +::::{tab-item} Interactive Mode (Recommended for Debugging) Interactive mode launches the cluster and gives you a shell on the **Head Node**. This is perfect for debugging because you can run scripts, check files, and kill/restart jobs without re-queueing. **1. Submit the Request** @@ -108,9 +108,9 @@ You are now inside the container on the head node. Run your command (see Cheatsh uv run examples/run_sft.py ``` -::: +:::: -:::{tab-item} Batch Mode (Production) +::::{tab-item} Batch Mode (Production) Batch mode is "fire and forget." You specify the command upfront, and the cluster shuts down automatically when it finishes. **1. Submit the Job** @@ -135,10 +135,10 @@ sbatch \ **2. Check Status** Slurm will write the output to a log file (e.g., `12345-logs/ray-driver.log`). -::: - :::: +::::: + --- ## 3. Verify Your Cluster From 4aa6f50afabc231c414cd4facbfd24ed1de9a3e1 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Thu, 4 Dec 2025 15:36:42 -0500 Subject: [PATCH 06/11] minor updates Signed-off-by: Lawrence Lane --- docs/get-started/cluster.md | 5 ++--- docs/get-started/installation.md | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index b1a7c13dda..6b77f879d6 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -151,7 +151,7 @@ The `ray.sub` script creates a log directory named after your Job ID (e.g., `198 * **`ray-driver.log`**: The stdout/stderr of your Python script. Check this for training progress (loss values). * **`ray-worker-*.log`**: Logs for individual worker nodes (useful for debugging specific node failures). -* **`dashboard.log`**: Debug info for the Ray dashboard. +* **`ray-head.log`**: Output from the Ray head node (includes cluster initialization and dashboard startup info). ```bash tail -f 1980204-logs/ray-driver.log @@ -206,9 +206,8 @@ The following variables allow for deeper customization of the Ray cluster. Most | `METRICS_EXPORT_PORT=53009` | Port for exporting metrics from worker nodes. | | `PORT=54514` | Main port for the Ray head node (default: 54514; Ray's standard default is 6379, but this script uses 54514 for multi-node compatibility). | | `RAY_CLIENT_SERVER_PORT=10001` | Port for the Ray client server on the head node. | -| `DASHBOARD_GRPC_PORT=52367` | gRPC port for the Ray dashboard on the head node. | | `DASHBOARD_PORT=8265` | Port for the Ray dashboard UI on the head node. | | `DASHBOARD_AGENT_LISTEN_PORT=52365` | Listening port for the dashboard agent on the head node. | | `MIN_WORKER_PORT=54001` | Minimum port in the range for Ray worker processes. | -| `MAX_WORKER_PORT=54257` | Maximum port in the range for Ray worker processes. | +| `MAX_WORKER_PORT=54513` | Maximum port in the range for Ray worker processes. | ::: diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index dc36488476..17a4fb7d79 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -141,7 +141,7 @@ Initialize the project-specific virtual environment. NeMo RL uses a `.python-ver ``` :::{important} - Do **not** specify a python version manually (e.g., `-p python3.10`). Let `uv` read the correct version from the configuration file to ensure compatibility. + Do **not** specify a python version manually (e.g., `-p python3.12`). Let `uv` read the correct version from the configuration file to ensure compatibility. ::: 2. **(Optional) Rebuilding Environments**: From 4c8b6762a5b81bca2f216571800295d8ee3100b0 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Thu, 4 Dec 2025 15:43:59 -0500 Subject: [PATCH 07/11] run grpo math fix Signed-off-by: Lawrence Lane --- docs/get-started/cluster.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index 6b77f879d6..ccaf293acc 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -105,7 +105,7 @@ bash -attach.sh You are now inside the container on the head node. Run your command (see Cheatsheet above): ```bash -uv run examples/run_sft.py +uv run examples/run_grpo_math.py ``` :::: @@ -120,7 +120,7 @@ Include the `COMMAND` variable in your submission. Replace the command below wit # Run from the root of NeMo RL repo NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) -COMMAND="uv run examples/run_sft.py" \ +COMMAND="uv run examples/run_grpo_math.py" \ CONTAINER=nvcr.io/nvidia/nemo:latest \ MOUNTS="$PWD:$PWD" \ sbatch \ From 5c89c8437a6a6ca8221fec5a28e7acb590999b23 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Thu, 4 Dec 2025 15:50:02 -0500 Subject: [PATCH 08/11] cleanup of tab lists Signed-off-by: Lawrence Lane --- docs/get-started/cluster.md | 107 ++++++++++++++++--------------- docs/get-started/installation.md | 2 +- 2 files changed, 57 insertions(+), 52 deletions(-) diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index ccaf293acc..e30ab7c5b5 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -70,71 +70,76 @@ Copy the command for your training type: ::::{tab-item} Interactive Mode (Recommended for Debugging) Interactive mode launches the cluster and gives you a shell on the **Head Node**. This is perfect for debugging because you can run scripts, check files, and kill/restart jobs without re-queueing. -**1. Submit the Request** -Ask for the resources you need (e.g., 1 node, 8 GPUs). +1. Submit the Request. -```bash -# Run from the root of NeMo RL repo -NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) - -CONTAINER=nvcr.io/nvidia/nemo:latest \ -MOUNTS="$PWD:$PWD" \ -sbatch \ - --nodes=${NUM_ACTOR_NODES} \ - --account=YOUR_ACCOUNT \ - --partition=YOUR_PARTITION \ - --gres=gpu:8 \ - --time=1:0:0 \ - --job-name=YOUR_JOBNAME \ - ray.sub -``` + Ask for the resources you need (e.g., 1 node, 8 GPUs). -:::{tip} -- Replace `YOUR_ACCOUNT` and `YOUR_PARTITION` with values from your cluster. Run `sacctmgr show associations user=$USER` to find your account. -- Depending on your Slurm cluster configuration, you may need `--gres=gpu:8` or `--gpus-per-node=8`. Check with your cluster admin if jobs don't receive GPUs. -::: + ```bash + # Run from the root of NeMo RL repo + NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) -**2. Attach to the Cluster** -Once the job starts, Slurm creates an attach script (e.g., `12345-attach.sh`). Run it: + CONTAINER=nvcr.io/nvidia/nemo:latest \ + MOUNTS="$PWD:$PWD" \ + sbatch \ + --nodes=${NUM_ACTOR_NODES} \ + --account=YOUR_ACCOUNT \ + --partition=YOUR_PARTITION \ + --gres=gpu:8 \ + --time=1:0:0 \ + --job-name=YOUR_JOBNAME \ + ray.sub + ``` -```bash -bash -attach.sh -``` + :::{tip} + - Replace `YOUR_ACCOUNT` and `YOUR_PARTITION` with values from your cluster. Run `sacctmgr show associations user=$USER` to find your account. + - Depending on your Slurm cluster configuration, you may need `--gres=gpu:8` or `--gpus-per-node=8`. Check with your cluster admin if jobs don't receive GPUs. + ::: -**3. Run Your Training** -You are now inside the container on the head node. Run your command (see Cheatsheet above): +2. Attach to the Cluster. -```bash -uv run examples/run_grpo_math.py -``` + Once the job starts, Slurm creates an attach script (e.g., `12345-attach.sh`). Run it: + + ```bash + bash -attach.sh + ``` + +3. Run Your Training. + + You are now inside the container on the head node. Run your command (see Cheatsheet above): + + ```bash + uv run examples/run_grpo_math.py + ``` :::: ::::{tab-item} Batch Mode (Production) Batch mode is "fire and forget." You specify the command upfront, and the cluster shuts down automatically when it finishes. -**1. Submit the Job** -Include the `COMMAND` variable in your submission. Replace the command below with the one from the Cheatsheet. +1. Submit the Job. -```bash -# Run from the root of NeMo RL repo -NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) - -COMMAND="uv run examples/run_grpo_math.py" \ -CONTAINER=nvcr.io/nvidia/nemo:latest \ -MOUNTS="$PWD:$PWD" \ -sbatch \ - --nodes=${NUM_ACTOR_NODES} \ - --account=YOUR_ACCOUNT \ - --partition=YOUR_PARTITION \ - --gres=gpu:8 \ - --time=1:0:0 \ - --job-name=YOUR_JOBNAME \ - ray.sub -``` + Include the `COMMAND` variable in your submission. Replace the command below with the one from the Cheatsheet. + + ```bash + # Run from the root of NeMo RL repo + NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) + + COMMAND="uv run examples/run_grpo_math.py" \ + CONTAINER=nvcr.io/nvidia/nemo:latest \ + MOUNTS="$PWD:$PWD" \ + sbatch \ + --nodes=${NUM_ACTOR_NODES} \ + --account=YOUR_ACCOUNT \ + --partition=YOUR_PARTITION \ + --gres=gpu:8 \ + --time=1:0:0 \ + --job-name=YOUR_JOBNAME \ + ray.sub + ``` + +2. Check Status. -**2. Check Status** -Slurm will write the output to a log file (e.g., `12345-logs/ray-driver.log`). + Slurm will write the output to a log file (e.g., `12345-logs/ray-driver.log`). :::: ::::: diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index 17a4fb7d79..dc36488476 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -141,7 +141,7 @@ Initialize the project-specific virtual environment. NeMo RL uses a `.python-ver ``` :::{important} - Do **not** specify a python version manually (e.g., `-p python3.12`). Let `uv` read the correct version from the configuration file to ensure compatibility. + Do **not** specify a python version manually (e.g., `-p python3.10`). Let `uv` read the correct version from the configuration file to ensure compatibility. ::: 2. **(Optional) Rebuilding Environments**: From c0bfad5168bb6aea2672f15917d23efc2045fbe9 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Tue, 9 Dec 2025 09:37:28 -0500 Subject: [PATCH 09/11] docs fix toc Signed-off-by: Lawrence Lane --- docs/index.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/index.md b/docs/index.md index 42b7542b7c..d92e5529f1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -256,11 +256,7 @@ docker.md design-docs/design-and-philosophy.md design-docs/training-backends.md -design-docs/padding.md -design-docs/logger.md -design-docs/uv.md design-docs/dependency-management.md -design-docs/chat-datasets.md design-docs/generation.md design-docs/checkpointing.md design-docs/loss-functions.md From 4ba711d6661edbf394fc33246dd91dcd673ed09d Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 17 Dec 2025 12:19:43 -0500 Subject: [PATCH 10/11] feedback sweep Signed-off-by: Lawrence Lane --- docs/get-started/cluster.md | 12 ++++++-- docs/get-started/grpo.md | 9 ++---- docs/get-started/index.md | 48 +++++++++++++++++--------------- docs/get-started/installation.md | 10 +++++-- docs/get-started/sft.md | 2 +- 5 files changed, 46 insertions(+), 35 deletions(-) diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index e30ab7c5b5..a8ca621f3f 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -76,7 +76,7 @@ Interactive mode launches the cluster and gives you a shell on the **Head Node** ```bash # Run from the root of NeMo RL repo - NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) + NUM_ACTOR_NODES=1 CONTAINER=nvcr.io/nvidia/nemo:latest \ MOUNTS="$PWD:$PWD" \ @@ -90,6 +90,9 @@ Interactive mode launches the cluster and gives you a shell on the **Head Node** ray.sub ``` + > [!TIP] + > The `nvcr.io/nvidia/nemo:latest` image may not always be up to date. If you encounter issues, see the [Docker build instructions](../docker.md) to build a fresh container from source. + :::{tip} - Replace `YOUR_ACCOUNT` and `YOUR_PARTITION` with values from your cluster. Run `sacctmgr show associations user=$USER` to find your account. - Depending on your Slurm cluster configuration, you may need `--gres=gpu:8` or `--gpus-per-node=8`. Check with your cluster admin if jobs don't receive GPUs. @@ -105,7 +108,7 @@ Interactive mode launches the cluster and gives you a shell on the **Head Node** 3. Run Your Training. - You are now inside the container on the head node. Run your command (see Cheatsheet above): + You are now inside the container on the head node. Run your command (see [Command Cheatsheet](#command-cheatsheet) above): ```bash uv run examples/run_grpo_math.py @@ -122,7 +125,7 @@ Batch mode is "fire and forget." You specify the command upfront, and the cluste ```bash # Run from the root of NeMo RL repo - NUM_ACTOR_NODES=1 # Total nodes requested (head is colocated on ray-worker-0) + NUM_ACTOR_NODES=1 COMMAND="uv run examples/run_grpo_math.py" \ CONTAINER=nvcr.io/nvidia/nemo:latest \ @@ -137,6 +140,9 @@ Batch mode is "fire and forget." You specify the command upfront, and the cluste ray.sub ``` + > [!TIP] + > The `nvcr.io/nvidia/nemo:latest` image may not always be up to date. If you encounter issues, see the [Docker build instructions](../docker.md) to build a fresh container from source. + 2. Check Status. Slurm will write the output to a log file (e.g., `12345-logs/ray-driver.log`). diff --git a/docs/get-started/grpo.md b/docs/get-started/grpo.md index 373654c0cd..1cf4e6294a 100644 --- a/docs/get-started/grpo.md +++ b/docs/get-started/grpo.md @@ -75,7 +75,6 @@ Key parameters for GRPO: * **`grpo.num_generations_per_prompt`**: The group size ($G$). The model generates this many outputs for *each* prompt (e.g., 16). * **`grpo.num_prompts_per_step`**: How many unique prompts to process in one batch. * **`policy.model_name`**: The model being trained. -* **`policy.generation.temperature`**: Controls diversity. GRPO needs diverse outputs to find the correct answer, so `1.0` is common. --- @@ -114,11 +113,9 @@ uv run python examples/run_grpo_math.py \ ## 4. Monitor and Verify -GRPO training logs specific metrics that tell you if "Reasoning" is emerging: +Monitor the `reward` metric to see if the model is improving. Note that some models may plateau early if they have already been fine-tuned or RL-trained for the task. -1. **`reward`**: The average accuracy of the group (mean reward). This should steadily increase. -2. **`policy_kl_error`**: How far the model has drifted from the original behavior (KL divergence). -3. **`advantages/mean`**: The average relative score of an output compared to its group peers. +Training can destabilize over time, so also monitor `policy_kl_error` and `token_mult_prob_error` to detect instability. For detailed information about these metrics, see the [GRPO Guide](../guides/grpo.md#metrics). ### Output Artifacts @@ -131,7 +128,7 @@ Results are saved to `results/grpo`: ### Multi-GPU Training -GRPO involves heavy generation (inference) and training. For larger models (7B+), you almost certainly need multiple GPUs. +GRPO involves heavy generation (inference) and training. For larger models (8B+), you almost certainly need multiple GPUs. Use the Megatron backend for efficient scaling: diff --git a/docs/get-started/index.md b/docs/get-started/index.md index 3f8d55a037..422cfb4482 100644 --- a/docs/get-started/index.md +++ b/docs/get-started/index.md @@ -29,13 +29,13 @@ Welcome to NeMo RL! ## Prerequisites -* **OS**: Linux (Ubuntu 22.04/20.04 recommended) +* **OS**: Linux (Ubuntu 24.04 recommended) * **Hardware**: * NVIDIA GPU (Volta/Compute Capability 7.0+ required) * Sufficient VRAM for the model and batch sizes configured in the example (memory requirements vary by configuration; reduce batch sizes if you encounter out-of-memory errors) * **Software**: - * Python 3.12+ - * CUDA 12+ + * Python 3.12 (see [`.python-version`](https://github.com/NVIDIA-NeMo/RL/blob/main/.python-version) for the prescribed version) + * CUDA 12.9 (see [`docker/Dockerfile`](https://github.com/NVIDIA-NeMo/RL/blob/main/docker/Dockerfile#L8) for the currently prescribed version) * Git ## 1. Installation @@ -50,7 +50,6 @@ We use `uv` for fast, reliable package management. ```bash curl -LsSf https://astral.sh/uv/install.sh | sh - source $HOME/.local/bin/env ``` 2. **Clone NeMo RL**: @@ -64,10 +63,9 @@ We use `uv` for fast, reliable package management. # git submodule update --init --recursive ``` - :::{warning} - If you cloned without the `--recursive` flag, you may need to rebuild virtual environments: - `NRL_FORCE_REBUILD_VENVS=true uv sync` - ::: + > [!WARNING] + > If you cloned without the `--recursive` flag, you may need to rebuild virtual environments: + > `uv run nemo_rl/utils/prefetch_venvs.py` 3. **Initialize Environment**: Create the virtual environment. @@ -76,9 +74,8 @@ We use `uv` for fast, reliable package management. uv venv ``` - ```{note} - Do not use `-p/--python`. `uv` will automatically read the correct Python version from `.python-version`. - ``` + > [!NOTE] + > Do not use `-p/--python`. `uv` will automatically read the correct Python version from `.python-version`. ## 2. Run Your First Job (Local) @@ -116,18 +113,25 @@ Let's verify your installation by running a **Group Relative Policy Optimization **What to expect**: * NeMo RL will automatically start a local Ray cluster on your machine. * It will download a small model (`Qwen/Qwen2.5-1.5B-Instruct` or similar) and dataset. - * You should see training logs indicating "Training started" and loss metrics streaming. + * You should see training logs with step metrics including loss, reward, and timing information. * The Ray dashboard URL will appear in the logs (typically `http://127.0.0.1:8265`). **Example output**: ``` - Initializing Ray cluster... - Ray dashboard available at http://127.0.0.1:8265 - Loading model: Qwen/Qwen2.5-1.5B-Instruct - Training started... - Step 1: reward=0.25, policy_kl_error=0.001 - Step 2: reward=0.31, policy_kl_error=0.002 - ... + ========================= Step 1/29687 ========================= + ▶ Preparing batch... + ▶ Generating responses for batch of size 512... + + 📊 Training Results: + • Loss: 1.03 + • Avg Reward: 0.0000 + • Mean Generation Length: 368.7812 + + ⏱️ Timing: + • Total step time: 132.26s + • policy_training: 74.31s (56.2%) + • generation: 18.28s (13.8%) + ... ``` ### Local Development Tips @@ -201,9 +205,9 @@ You generally do **not** need to start Ray manually. NeMo RL uses a distributed architecture built on **Ray** to coordinate multiple components (RL Actors) during training: * **Policy Model**: The model being trained (e.g., Qwen, Llama) + * **Training Backend**: PyTorch DTensor or Megatron Core for efficient distributed training * **Generation Backend**: Fast inference engine (vLLM) that generates responses * **Environment**: Reward evaluator (e.g., Math verifier) that scores outputs -* **Training Backend**: PyTorch DTensor or Megatron Core for efficient distributed training Ray manages resource allocation, process isolation, and communication between these components, allowing NeMo RL to scale seamlessly from a single GPU to multi-node clusters. @@ -233,10 +237,10 @@ Now that you have a working setup, choose the workflow that matches your goal. **Start here** if you have preference data (chosen vs rejected pairs) and want to align your model to human preferences. DPO learns directly from preference comparisons without needing a separate reward model. ::: -:::{grid-item-card} {octicon}`rocket;1.5em;sd-mr-1` Reinforce (GRPO) +:::{grid-item-card} {octicon}`rocket;1.5em;sd-mr-1` Reinforcement Learning (GRPO) :link: gs-grpo :link-type: ref -**Start here** for reasoning tasks (math, coding) where you can verify correctness programmatically. GRPO is efficient for on-policy RL without requiring a separate critic model—perfect for tasks with deterministic rewards. +**Start here** for reasoning tasks (math, coding) where you can verify correctness programmatically. GRPO is efficient for on-policy RL without requiring a separate critic model—perfect for tasks with verifiable rewards. ::: :::: diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index dc36488476..31a2d86d09 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -73,8 +73,8 @@ Find the correct version for your system at the [NVIDIA cuDNN Downloads page](ht ``` ::: -:::{tab-item} vLLM / DeepSpeed -For the **vLLM** inference backend (often used with DeepSpeed), `libibverbs-dev` is required on bare metal to avoid build errors. +:::{tab-item} vLLM +For the **vLLM** inference backend, `libibverbs-dev` is required on bare metal to avoid build errors (needed for [deep_ep](https://github.com/NVIDIA-NeMo/RL/blob/363165a87fcf9dc42accc590ff4e92ca4da8a505/pyproject.toml#L69C1-L71C40)). **Install libibverbs:** ```sh @@ -147,7 +147,7 @@ Initialize the project-specific virtual environment. NeMo RL uses a `.python-ver 2. **(Optional) Rebuilding Environments**: If you change branches or modify `pyproject.toml` significantly, you may need to force a rebuild of the environment variables and dependencies: ```sh - NRL_FORCE_REBUILD_VENVS=true uv sync + uv run nemo_rl/utils/prefetch_venvs.py ``` --- @@ -168,6 +168,10 @@ In NeMo RL, we recommend using `uv run` to execute scripts rather than manually uv run python examples/run_grpo_math.py --config examples/configs/grpo_math_1B_megatron.yaml ``` +:::{tip} +If you prefer to run without `uv`, you can use [frozen environments](https://github.com/NVIDIA-NeMo/RL/blob/main/docs/design-docs/dependency-management.md#frozen-environments) to manage dependencies manually. +::: + --- ## 6. Configure Access Tokens diff --git a/docs/get-started/sft.md b/docs/get-started/sft.md index cfa2941d6d..f192a96a67 100644 --- a/docs/get-started/sft.md +++ b/docs/get-started/sft.md @@ -11,7 +11,7 @@ content_type: "tutorial" # Get Started with SFT -**Supervised Fine-Tuning (SFT)** is the standard first step in aligning language models. +**Supervised Fine-Tuning (SFT)** is the standard first step in aligning language models. It is also used to teach models how to call tools, which is a prerequisite to agentic RL. :::{card} **Goal**: Train a basic model (Llama-3.2-1B) on a sample dataset using your local machine. From 0b7b06ea81a357a4a12490e5f8e45e4f2f13368f Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 17 Dec 2025 12:50:13 -0500 Subject: [PATCH 11/11] admonitions Signed-off-by: Lawrence Lane --- docs/get-started/cluster.md | 12 +++++------ docs/get-started/dpo.md | 5 ++--- docs/get-started/installation.md | 36 ++++++++++++++------------------ docs/get-started/sft.md | 5 ++--- 4 files changed, 25 insertions(+), 33 deletions(-) diff --git a/docs/get-started/cluster.md b/docs/get-started/cluster.md index a8ca621f3f..c7e874750f 100644 --- a/docs/get-started/cluster.md +++ b/docs/get-started/cluster.md @@ -93,10 +93,9 @@ Interactive mode launches the cluster and gives you a shell on the **Head Node** > [!TIP] > The `nvcr.io/nvidia/nemo:latest` image may not always be up to date. If you encounter issues, see the [Docker build instructions](../docker.md) to build a fresh container from source. - :::{tip} - - Replace `YOUR_ACCOUNT` and `YOUR_PARTITION` with values from your cluster. Run `sacctmgr show associations user=$USER` to find your account. - - Depending on your Slurm cluster configuration, you may need `--gres=gpu:8` or `--gpus-per-node=8`. Check with your cluster admin if jobs don't receive GPUs. - ::: + > [!TIP] + > - Replace `YOUR_ACCOUNT` and `YOUR_PARTITION` with values from your cluster. Run `sacctmgr show associations user=$USER` to find your account. + > - Depending on your Slurm cluster configuration, you may need `--gres=gpu:8` or `--gpus-per-node=8`. Check with your cluster admin if jobs don't receive GPUs. 2. Attach to the Cluster. @@ -197,9 +196,8 @@ You can pass these variables to `sbatch` to configure the environment: | **`WANDB_API_KEY`** | Weights & Biases key (for logging). | | **`GPUS_PER_NODE`** | Number of GPUs per node (default: 8). | -:::{tip} -Export secrets like `HF_TOKEN` in your shell profile (`~/.bashrc`) so you don't have to type them every time. -::: +> [!TIP] +> Export secrets like `HF_TOKEN` in your shell profile (`~/.bashrc`) so you don't have to type them every time. :::{dropdown} Advanced Environment Configuration The following variables allow for deeper customization of the Ray cluster. Most users will not need to change these defaults. diff --git a/docs/get-started/dpo.md b/docs/get-started/dpo.md index 5b271bcbbb..ba4bd6a065 100644 --- a/docs/get-started/dpo.md +++ b/docs/get-started/dpo.md @@ -79,9 +79,8 @@ Key parameters to tune: * **`dpo.reference_policy_kl_penalty`**: Controls how much the model stays close to the original behavior (preventing "reward hacking"). * **`dpo.preference_loss_weight`**: The strength of the preference signal. -:::{tip} -For a local test, stick to small models like `meta-llama/Llama-3.2-1B-Instruct` to avoid OOM errors. -::: +> [!TIP] +> For a local test, stick to small models like `meta-llama/Llama-3.2-1B-Instruct` to avoid OOM errors. --- diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index 31a2d86d09..ac97def05c 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -38,9 +38,8 @@ Use this guide if you are setting up on a bare-metal system, need specific backe Before installing the Python package, ensure your operating system has the required libraries for your chosen backend. -:::{note} -If you are using a pre-built NVIDIA container (e.g., from NGC), most of these dependencies are likely pre-installed. These steps are critical for **bare-metal** installations (e.g., a fresh Ubuntu server). -::: +> [!NOTE] +> If you are using a pre-built NVIDIA container (e.g., from NGC), most of these dependencies are likely pre-installed. These steps are critical for **bare-metal** installations (e.g., a fresh Ubuntu server). ::::{tab-set} @@ -103,14 +102,14 @@ NeMo RL relies on several third-party libraries included as git submodules. You git submodule update --init --recursive ``` -:::{tip} Keep Submodules in Sync -Different branches may pin different versions of submodules. To ensure they update automatically when you switch branches or pull, configure git: - -```sh -git config submodule.recurse true -``` -*Note: This will not remove old submodules or download new ones if the directory structure changes significantly; in those cases, run the full update command above.* -::: +> [!TIP] +> **Keep Submodules in Sync** +> Different branches may pin different versions of submodules. To ensure they update automatically when you switch branches or pull, configure git: +> +> ```sh +> git config submodule.recurse true +> ``` +> *Note: This will not remove old submodules or download new ones if the directory structure changes significantly; in those cases, run the full update command above.* --- @@ -140,9 +139,8 @@ Initialize the project-specific virtual environment. NeMo RL uses a `.python-ver uv venv ``` - :::{important} - Do **not** specify a python version manually (e.g., `-p python3.10`). Let `uv` read the correct version from the configuration file to ensure compatibility. - ::: + > [!IMPORTANT] + > Do **not** specify a python version manually (e.g., `-p python3.10`). Let `uv` read the correct version from the configuration file to ensure compatibility. 2. **(Optional) Rebuilding Environments**: If you change branches or modify `pyproject.toml` significantly, you may need to force a rebuild of the environment variables and dependencies: @@ -168,9 +166,8 @@ In NeMo RL, we recommend using `uv run` to execute scripts rather than manually uv run python examples/run_grpo_math.py --config examples/configs/grpo_math_1B_megatron.yaml ``` -:::{tip} -If you prefer to run without `uv`, you can use [frozen environments](https://github.com/NVIDIA-NeMo/RL/blob/main/docs/design-docs/dependency-management.md#frozen-environments) to manage dependencies manually. -::: +> [!TIP] +> If you prefer to run without `uv`, you can use [frozen environments](https://github.com/NVIDIA-NeMo/RL/blob/main/docs/design-docs/dependency-management.md#frozen-environments) to manage dependencies manually. --- @@ -194,9 +191,8 @@ huggingface-cli login export WANDB_API_KEY=your_key_here ``` -:::{tip} -Add these exports to your `~/.bashrc` or `~/.zshrc` so they persist across sessions. -::: +> [!TIP] +> Add these exports to your `~/.bashrc` or `~/.zshrc` so they persist across sessions. --- diff --git a/docs/get-started/sft.md b/docs/get-started/sft.md index f192a96a67..370806588c 100644 --- a/docs/get-started/sft.md +++ b/docs/get-started/sft.md @@ -75,9 +75,8 @@ Key parameters you might want to change: * **`sft.max_num_epochs`**: How many times to iterate over the dataset. * **`policy.optimizer.kwargs.lr`**: The step size for the optimizer. -:::{tip} -You don't need to edit the YAML file directly. You can override any parameter from the command line (shown in Step 3). -::: +> [!TIP] +> You don't need to edit the YAML file directly. You can override any parameter from the command line (shown in Step 3). ---