Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
15a7d0b
docs(customizer): align guides with current APIs and workflows
soluwalana Jul 21, 2026
285850e
Apply suggestions from code review
soluwalana Jul 21, 2026
729f384
Code Rabbit resolutions
soluwalana Jul 21, 2026
6f0c052
Fix code rabbit comments
soluwalana Jul 22, 2026
c8f2249
test(auditor): E2E tests for Auditor (#794)
parkanzky Jul 21, 2026
e95985e
Add local services lifecycle SDK (#734)
ironcommit Jul 21, 2026
7b02be8
feat(studio): Dedicated Agent Details Page (#806)
steramae-nvidia Jul 21, 2026
b4c8115
fix(studio): Use RHF for render performance (#746)
steramae-nvidia Jul 21, 2026
611b510
chore: enable volume extra test (#754)
mckornfield Jul 21, 2026
7a0756a
feat(experiments): Allow an evaluation to belong to multiple experime…
shanaiabuggy Jul 22, 2026
b542376
fix(studio): Fix RadioCard interactions (#730)
htolentino-nvidia Jul 22, 2026
b049b3d
feat(studio): Use LogViewer on Customizer Job Details (#728)
htolentino-nvidia Jul 22, 2026
15604f6
fix(guardrails): Fix type errors in plugin (#825)
JashG Jul 22, 2026
0637cbf
fix(guardrails): Propagate mutated text to proxied request/response (…
JashG Jul 22, 2026
d436f13
fix: sanitize Jobs and Auditor API error responses (#826)
anastasia-nesterenko Jul 22, 2026
a6b9902
fix(models): redirect LoRA sidecar XDG_DATA_HOME to a writable volume…
albcui Jul 22, 2026
5759f23
chore: fix heading mistake for agents page (#840)
mckornfield Jul 22, 2026
5d0c79c
fix(guardrails): Ensure input rail logs are not dropped from final re…
JashG Jul 22, 2026
fd185d2
feat(nemo-agents): one shot local invocation support for fabric backe…
mmogallapalli Jul 22, 2026
32cb038
feat(studio): add experiment trace comparison view [ASE-186] (#641)
walston Jul 22, 2026
968f6d1
feat(studio): Replace hand rolled types in Customizer (#807)
htolentino-nvidia Jul 22, 2026
1bc79f0
feat(studio): add opt-in dev-server proxy for platform /apis (#836)
aray12 Jul 22, 2026
66e3cf7
Merge branch 'main' into solu/doc-fixes
soluwalana Jul 22, 2026
5e1ba24
code rabbit again
soluwalana Jul 22, 2026
a261efb
code rabbit again
soluwalana Jul 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Run these from the repo root (they wrap `cd docs/fern && npm run …`):
| `make docs-deps` | Install docs tooling (first run on a machine) |
| `make docs` | Local dev server (live preview) |
| `make docs-watch` | Local dev server plus repo-level watcher for `docs/**` changes outside `docs/fern/` |
| `make docs-check` | `fern check` + MDX validation + gated-link check (what CI runs) |
| `make docs-check` | `fern check` + MDX validation + NotebookViewer artifact validation + gated-link check (what CI runs) |
| `make docs-check-python-snippets DOCS_PATH=...` | Syntax-check and type-check Python fenced snippets in one doc |
| `make docs-run-notebook DOCS_PATH=...` | Execute the source notebook for one Fern `.mdx`/`.ipynb` doc using `nemo-nb` markers |
| `make docs-broken-links` | Report broken links |
Expand All @@ -24,7 +24,7 @@ Use `make docs` when you are only editing `docs/fern/` config. Use `make docs-wa
## Rules that bite if you miss them

- **Navigation is the build.** Fern only builds pages listed in `docs/fern/versions/latest.yml`. A `.mdx` not in the nav is **not built** (404, not indexed) — that is how unready features are gated. Do **not** use `hidden: true` for gating (it still builds/serves the page).
- **Gated (unready) features** stay in the repo but out of the nav: `auth/`, `customizer/`, `safe-synthesizer/`, `evaluator/benchmarks/`, and a few individual pages. Ready-to-paste nav blocks for re-publishing are in `docs/fern/gated-nav.yml`. To publish one: move its block into `latest.yml`, re-add inbound links, run `make docs-check && make docs-broken-links`.
- **Publication state is nav-derived.** Do not maintain or rely on a hard-coded list of gated directories. Check `docs/fern/versions/latest.yml`: listed pages are published, and omitted pages are gated. `docs/fern/gated-nav.yml` contains reference blocks for some gated features. To publish one: move its block into `latest.yml`, re-add inbound links, run `make docs-check && make docs-broken-links`.
- **Don't link into gated pages.** A link from a published page into a gated page is a dead link. `make docs-check` fails on it; `make docs-fix-links` delinks it to plain text. (Replaces the old MkDocs `hide_unready_docs` auto-delinking.)
- **Internal links** use canonical nav URLs like `/documentation/get-started/core-concepts/workspaces`, not relative `.md`/source paths. `make docs-broken-links` is the check.
- **No `{{variable}}` substitutions.** Fern has no substitution step; product names are inlined as literal text. (Prompt-template tokens like `` `{{input}}` `` inside backticks are real content — leave them.)
Expand Down
82 changes: 49 additions & 33 deletions docs/customizer/about.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
title: "Customization Concepts"
description: ""
---

<a id="nemo-ms-about-concepts-customization"></a>

This page provides an overview of the customization concepts for the NeMo Platform.
Expand All @@ -14,30 +15,30 @@ Supervised fine-tuning (SFT) is a traditional technique for customizing a pre-tr

Full SFT models require a NIM deployment to serve inference. The Deployment Management Service supports two deployment modes:

| Deployment Mode | Image Type | Weight Loading | Best For |
|-----------------|------------|----------------|----------|
| **Multi-LLM** (Default) | Generic multi-model NIM | On-the-fly download via Files service | Any HF model, custom fine-tuned models, development |
| **Model-Specific NIM** | Dedicated model image | Pre-download via model puller | Production, optimized performance and latency |
| Deployment Mode | Image Type | Weight Loading | Best For |
| ----------------------- | ----------------------- | ------------------------------------- | --------------------------------------------------------------------------- |
| **Multi-LLM** (Default) | Generic multi-model NIM | On-the-fly download via Files service | Supported Hugging Face architectures, custom fine-tuned models, development |
| **Model-Specific NIM** | Dedicated model image | Pre-download via model puller | Production, optimized performance and latency |

- **Multi-LLM Image**: Can deploy any HuggingFace-compatible model, providing maximum flexibility for custom fine-tuned models. Does not guarantee optimized inference performance.
- **Multi-LLM Image**: Can deploy Hugging Face checkpoints whose architectures are supported by the image's inference engine, providing flexibility for custom fine-tuned models. Importing a checkpoint does not guarantee training or deployment compatibility; for example, Automodel LoRA does not support Conv1D-based architectures. It also does not guarantee optimized inference performance.

- **Model-Specific NIM**: Provides optimized inference performance and latency through model-specific optimizations. Recommended for production deployments where performance is critical.

## Parameter-Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient model customization by training a small number of parameters while keeping the base model frozen. For example, when customizing LLaMa 3.3 70B:
Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient model customization by training a small number of parameters while keeping the base model frozen. For example, when customizing Llama 3.3 70B:

- **Traditional SFT**: Trains and stores ~40 GB per task.
- **PEFT**: Trains and stores only a few MB per task while maintaining comparable performance.
- **Traditional SFT**: Produces a full BF16 checkpoint of approximately 140 GB per task. During training, budget free disk space separately for the base checkpoint, intermediate checkpoint, and final output—approximately 3× the downloaded base checkpoint size.
- **PEFT**: Produces an adapter that is approximately 100–500 MB per task while maintaining comparable performance. During training, budget approximately 1.5× the downloaded base checkpoint size.

```mermaid
---
caption: Traditional Fine-Tuning
---
flowchart TD
T1[Task 1] --> M1[LLaMa 3.3 - 70B]
T2[Task 2] --> M2[LLaMa 3.3 - 70B]
T3[Task 3] --> M3[LLaMa 3.3 - 70B]
T1[Task 1] --> M1[Llama 3.3 - 70B]
T2[Task 2] --> M2[Llama 3.3 - 70B]
T3[Task 3] --> M3[Llama 3.3 - 70B]

style M1 fill:#B8D5F2
style M2 fill:#B8D5F2
Expand All @@ -47,13 +48,13 @@ flowchart TD

```mermaid
---
caption: Parameter-Efficient Fine Tuning
caption: Parameter-Efficient Fine-Tuning
---
flowchart TD
P1[Task 1] --> A1[LoRA 113M]
P2[Task 2] --> A2[LoRA 20M]
P3[Task 3] --> A3[LoRA 10M]
A1 & A2 & A3 --> M[LLaMa 3.3 - 70B]
A1 & A2 & A3 --> M[Llama 3.3 - 70B]

style A1 fill:#B8D5F2
style A2 fill:#B8D5F2
Expand Down Expand Up @@ -121,6 +122,7 @@ Additional Resources:
## Training with Your Own Data

Use NeMo Customizer to train custom models on your own data. The workflow can be carried out as follows:

- Upload a dataset
- Train a custom model
- Perform inference with the trained model
Expand Down Expand Up @@ -205,17 +207,17 @@ Hyperparameters are configuration settings used to control the training process.

Common hyperparameters you'll tune include:

| Hyperparameter | Description |
|----------------|-------------|
| Epochs | Number of complete passes through the training dataset |
| Batch size | Number of samples processed before updating model weights |
| Learning rate | Step size for weight updates during training |
| LoRA rank | Low-rank dimension of the adapter (lower = fewer parameters, higher = more expressive) |
| LoRA alpha | LoRA scaling factor |
| Hyperparameter | Description |
| -------------- | -------------------------------------------------------------------------------------- |
| Epochs | Number of complete passes through the training dataset |
| Batch size | Number of samples processed before updating model weights |
| Learning rate | Step size for weight updates during training |
| LoRA rank | Low-rank dimension of the adapter (lower = fewer parameters, higher = more expressive) |
| LoRA alpha | LoRA scaling factor |

<Note>

NeMo Customizer offers **two training backends** — Automodel (multi-GPU) and Unsloth (single-GPU, quantized) — and each accepts its own job configuration. The exact field names, defaults, and available knobs differ between them. For the full per-backend hyperparameter reference, see [Training Configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration).
NeMo Customizer offers **two training backends** — Automodel (multi-GPU) and Unsloth (single-GPU, with optional quantized loading for LoRA) — and each accepts its own job configuration. Unsloth full-weight training requires unquantized model loading. The exact field names, defaults, and available knobs differ between them. For the full per-backend hyperparameter reference, see [Training Configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration).

</Note>

Expand All @@ -234,24 +236,31 @@ TP can be configured via `parallelism.tensor_parallel_size` in the [training con
As of release 25.10.0, AutoModel engines including Phi-4, Qwen, and Gemma support tensor parallelism greater than 1 through the multi-GPU LoRA patch. Previous releases only supported `TP=1` for these models.

</Note>

#### Tensor Parallelism Configuration

**Constraints**

- TP must be less than or equal to the total number of GPUs available.
- TP should divide the total GPU count evenly.

**Multi-node considerations**

TP can span nodes, but doing so increases network communication overhead. For multi-node setups, keep TP within a single node when possible. High-bandwidth inter-node connections such as InfiniBand are important when TP must span nodes.

For example, with 2 nodes and 4 GPUs per node, start with `TP=4` to keep tensor-parallel operations within each node. If the model still requires more memory, increase to `TP=8` to distribute tensor operations across both nodes.

**Performance**

- Smaller TP values generally have less communication overhead.
- Larger TP values provide more memory savings but increase communication costs.

### Pipeline Parallelism

[Pipeline Parallelism](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/features/parallelisms.html#pipeline-parallelism) (PP) distributes the layers of a neural network across GPUs. The GPUs then process the different layers sequentially.

PP can be configured via `parallelism.pipeline_parallel_size` in the [training configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration).

Comment thread
coderabbitai[bot] marked this conversation as resolved.
#### Configuration

- Constraints
- TP must be less than or equal to the total number of GPUs available. It should be a factor of the total GPU count (divisible evenly).
- Multi-node considerations
- TP can span across nodes, but this introduces network communication overhead. For multi-node setups, it's often recommended to keep TP within a single node when possible. If using TP across nodes, high-bandwidth inter-node connections (like InfiniBand) become critical.

Example: if you have 2 nodes with 4 GPUs each, start with TP=4 first. This keeps all tensor parallel operations within a single node. If your model still uses too much GPU memory with this setting, increase to TP=8, which will distribute tensor operations across both nodes.
- Performance
- Smaller TP values generally have less communication overhead.
- Larger TP values provide more memory savings but increase communication costs.

### Context Parallelism

[Context Parallelism](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/features/parallelisms.html#context-parallelism) (CP) distributes activation memory along the sequence dimension across GPUs, which is particularly useful when training on datasets with very long sequences.
Expand Down Expand Up @@ -287,3 +296,10 @@ If sequence packing is enabled for a model that does not support it, fine-tuning
</Note>

Learn how to create a LoRA customization job with sequence packing by following the [Optimizing for Tokens/GPU](/documentation/customizer-reference/tutorials/optimize-throughput) tutorial.

## Next Steps

- Review all backend-specific options in [Training Configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration).
- [Create a LoRA customization job](/documentation/customizer-reference/tutorials/lora-customization-job).
- [Create a Full SFT customization job](/documentation/customizer-reference/tutorials/sft-customization-job).
- [Optimize training throughput](/documentation/customizer-reference/tutorials/optimize-throughput) with sequence packing.
10 changes: 5 additions & 5 deletions docs/customizer/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ View the available Phi models from Microsoft, designed for strong reasoning capa
</Card>
<Card title="GPT-OSS Models" href="/documentation/customizer-reference/models/gpt-oss">

View the available GPT-OSS models supported for Full SFT customization.
View the available GPT-OSS models supported for Full SFT and LoRA customization.

</Card>
<Card title="Embedding Models" href="/documentation/customizer-reference/models/embedding">
Expand Down Expand Up @@ -118,21 +118,21 @@ Learn how to format datasets for different model types.
<small><span class="md-tag">datasets</span> <span class="md-tag">chat-models</span> <span class="md-tag">completion-models</span></small>

</Card>
<Card title="Start a LoRA Customization Job" href="tutorials/lora-customization-job.ipynb">
<Card title="Start a LoRA Customization Job" href="/documentation/customizer-reference/tutorials/lora-customization-job">

Learn how to start a LoRA customization job using a custom dataset.

<small><span class="md-tag">nemo-customizer</span></small>

</Card>
<Card title="Start a Full SFT Customization Job" href="tutorials/sft-customization-job.ipynb">
<Card title="Start a Full SFT Customization Job" href="/documentation/customizer-reference/tutorials/sft-customization-job">

Learn how to start a SFT customization job using a custom dataset.

<small><span class="md-tag">nemo-customizer</span></small>

</Card>
<Card title="Distill a Model with Knowledge Distillation" href="tutorials/distillation-customization-job.ipynb">
<Card title="Distill a Model with Knowledge Distillation" href="/documentation/customizer-reference/tutorials/distillation-customization-job">

Learn how to compress a larger teacher model into a smaller student model.

Expand All @@ -146,7 +146,7 @@ Learn how to check job metrics using MLFlow or Weights & Biases.
<small><span class="md-tag">nemo-customizer</span> <span class="md-tag">mlflow</span> <span class="md-tag">wandb</span></small>

</Card>
<Card title="Optimize Tokens per GPU" href="tutorials/optimize-throughput.ipynb">
<Card title="Optimize Tokens per GPU" href="/documentation/customizer-reference/tutorials/optimize-throughput">

Learn how to optimize the token-per-GPU throughput for a LoRA optimization job.

Expand Down
10 changes: 6 additions & 4 deletions docs/customizer/manage-customization-jobs/create-job.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Customization jobs are submitted to one of two backends. Choose the backend that
| Backend | Best for | Methods |
|---------|----------|---------|
| **Automodel** (default) | Production fine-tuning, larger models, multi-GPU scaling | SFT, distillation; LoRA, merged-LoRA, or full-weight |
| **Unsloth** | Memory-constrained single-GPU LoRA | SFT; LoRA or full-weight, with 4-bit / 8-bit loading |
| **Unsloth** | Memory-constrained single-GPU training | SFT; LoRA with optional 4-bit / 8-bit loading, or unquantized full-weight |

## Prerequisites

Expand All @@ -19,7 +19,7 @@ Before you can create a customization job, make sure that you have:
- Created a [FileSet and Model Entity](/documentation/customizer-reference/manage-model-entities/overview) for your base model.
- [Uploaded a dataset](/documentation/get-started/core-concepts/manage-files) as a FileSet.
- Determined the [training configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration) you want to use for the customization job.
- Verified that the platform has sufficient storage for the job. Full SFT jobs require approximately 3× the base model size in free disk space; LoRA jobs require approximately 1.5×. See [ft-tut-understand-models](/documentation/customizer-reference/tutorials/understanding-models-and-training) for details. If you are also deploying the model from a base checkpoint fileset, plan for ~2.5× model size overall for LoRA.
- Verified that the platform has sufficient storage for the job. Budget against the downloaded base checkpoint size: Full SFT jobs require approximately 3× in free disk space, and LoRA jobs require approximately 1.5×. See [ft-tut-understand-models](/documentation/customizer-reference/tutorials/understanding-models-and-training) for details. Include any retained deployment copies separately.
- Set the `NMP_BASE_URL` environment variable to your NeMo Platform endpoint.

```bash
Expand Down Expand Up @@ -67,13 +67,15 @@ print(f"Submitted job: {job.job.name}")
print(f"Job status: {job.job.status}")
```

The response preserves the explicit `name`. If you omit `name`, the platform generates a backend-prefixed job name.

<Accordion title="Example Response">

:open:

```json
{
"name": "automodel-a1b2c3d4e5f6",
"name": "my-lora-job",
"workspace": "default",
"id": "platform-job-2k8i3i1HqJHHPVB5M6Bk9Z",
"status": "queued",
Expand Down Expand Up @@ -104,7 +106,7 @@ print(f"Job status: {job.job.status}")

## Submit an Unsloth Job

The Unsloth backend runs on a single GPU and supports 4-bit / 8-bit quantized loading. Build a `UnslothJobInput` spec and submit it to the `unsloth` backend. Note that Unsloth uses its own field names (`model.name`, `dataset.path`, `batch.per_device_train_batch_size`).
The Unsloth backend runs on a single GPU and supports 4-bit / 8-bit quantized loading for LoRA. Full-weight training requires `model.load_in_4bit=false` and `model.load_in_8bit=false`. Build a `UnslothJobInput` spec and submit it to the `unsloth` backend. Note that Unsloth uses its own field names (`model.name`, `dataset.path`, `batch.per_device_train_batch_size`).

```python
import os
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Get detailed execution status for a customization job, including step-by-step pr
This endpoint provides granular execution details including:

- **Step-level status**: `model-and-dataset-download` → `training` → `model-upload` → `model-entity-creation`
- **Training metrics**: `step`, `epoch`, `loss`, `lr` (learning rate), `grad_norm`, `val_loss`
- **Training metrics**: `step`, `epoch`, `train_loss`, `lr` (learning rate), `grad_norm`, `val_loss`
- **Progress tracking**: `downloaded_files`, `uploaded_bytes`, `progress_pct`

To list jobs or get job definitions (model entity, hyperparameters, spec), use [List Active Jobs](/documentation/customizer-reference/manage-customization-jobs/list-active-jobs) instead.
Expand Down Expand Up @@ -146,7 +146,7 @@ curl -X GET \
"num_epochs": 2,
"step": 8,
"epoch": 1,
"loss": 2.8918895721435547,
"train_loss": 2.8918895721435547,
"lr": 4.9101714686276044e-05,
"grad_norm": 26.0
}
Expand Down Expand Up @@ -222,7 +222,7 @@ curl -X GET \
"num_epochs": 2,
"step": 94,
"epoch": 2,
"loss": 0.3437718152999878,
"train_loss": 0.3437718152999878,
"lr": 5.000000000000001e-07,
"grad_norm": 20.125,
"val_loss": 0.5527229905128479,
Expand Down
Loading
Loading