Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions api-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,6 @@ class Resources:
cpus: float = 1.0 # Number of CPU cores
gpu_memory_gb: float = 0.0 # Number of GPU memory in GB (Only for single GPU)
gpus: float = 0.0 # Number of GPUs (Only for multi-GPU)
nvdecs: int = 0 # Number of NVDEC decoders
nvencs: int = 0 # Number of NVENC encoders
entire_gpu: bool = False # Whether to use the entire GPU
```

Expand Down
5 changes: 2 additions & 3 deletions docs/about/concepts/video/abstractions.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ A stage represents a single step in your data curation workflow. Video stages ar
Each processing stage:

1. Inherits from `ProcessingStage`
2. Declares a stable `name` and `resources: Resources` (CPU cores, GPU memory, optional NVDEC/NVENC, or more than one GPU)
2. Declares a stable `name` and `resources: Resources` (CPU cores, GPU memory, entire GPU flag, or multiple GPUs)
3. Defines `inputs()`/`outputs()` to document required attributes and produced attributes on tasks
4. Implements `setup(worker_metadata)` for model initialization and `process(task)` to transform tasks

Expand All @@ -75,9 +75,8 @@ Refer to the stage base and resources definitions in Curator for full details.
`Resources` support both fractional and whole‑GPU semantics:

- `gpu_memory_gb`: Request a fraction of a single GPU by memory; Curator rounds to a fractional GPU share and enforces that `gpu_memory_gb` stays within one device.
- `entire_gpu`: Request an entire GPU regardless of memory (also implies access to NVDEC/NVENC on that device).
- `entire_gpu`: Request an entire GPU regardless of memory (also implies access to hardware decoders and encoders on that device).
- `gpus`: Request more than one GPU for a stage that is multi‑GPU aware.
- `nvdecs` / `nvencs`: Request hardware decode/encode units when needed.

Choose one of `gpu_memory_gb` (single‑GPU fractional) or `gpus` (multi‑GPU). Combining both is not allowed.

Expand Down
294 changes: 110 additions & 184 deletions docs/about/release-notes/index.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/about/release-notes/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ modality: "universal"
This guide explains how to transition existing Dask-based NeMo Curator workflows to the new Ray-based pipeline architecture.

```{seealso}
For broader NeMo Framework migration topics, refer to the [NeMo Framework 2.0 Migration Guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemo-2.0/migration/index.html).
For broader NeMo Framework migration topics, refer to the [NeMo Framework 2.0 Migration Guide](https://docs.nvidia.com/nemo-framework/user-guide/25.11/nemo-2.0/migration/index.html).
```

## Overview
Expand Down
27 changes: 22 additions & 5 deletions docs/admin/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,15 @@ This guide covers installing NeMo Curator with support for **all modalities** an

### System Requirements

For comprehensive system requirements and production deployment specifications, see [Production Deployment Requirements](deployment/requirements.md).
For comprehensive system requirements and production deployment specifications, refer to [Production Deployment Requirements](deployment/requirements.md).

**Quick Start Requirements:**

- **OS**: Ubuntu 24.04/22.04/20.04 (recommended)
- **Python**: 3.10, 3.11, or 3.12
- **Memory**: 16GB+ RAM for basic text processing
- **GPU** (optional): NVIDIA GPU with 16GB+ VRAM for acceleration
- **CUDA 12** (required for `audio_cuda12`, `video_cuda12`, `image_cuda12`, and `text_cuda12` extras)

### Development vs Production

Expand All @@ -41,9 +42,13 @@ For comprehensive system requirements and production deployment specifications,

Choose one of the following installation methods based on your needs:

:::{tip}
**Docker is the recommended installation method** for video and audio workflows. The NeMo Curator container includes FFmpeg (with NVENC support) pre-configured, avoiding manual dependency setup. Refer to the [Container Installation](#container-installation) tab below.
:::

::::{tab-set}

:::{tab-item} PyPI Installation (Recommended)
:::{tab-item} PyPI Installation

Install NeMo Curator from the Python Package Index using `uv` for proper dependency resolution.

Expand Down Expand Up @@ -89,9 +94,9 @@ uv sync --all-extras --all-groups

:::

:::{tab-item} Container Installation
:::{tab-item} Container Installation (Recommended for Video/Audio)

NeMo Curator is available as a standalone container on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator. The container includes NeMo Curator with all dependencies pre-installed. You can run it with:
NeMo Curator is available as a standalone container on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator. The container includes NeMo Curator with all dependencies pre-installed, including FFmpeg with NVENC support.

```bash
# Pull the container from NGC
Expand All @@ -101,6 +106,14 @@ docker pull nvcr.io/nvidia/nemo-curator:{{ container_version }}
docker run --gpus all -it --rm nvcr.io/nvidia/nemo-curator:{{ container_version }}
```

```{important}
After entering the container, activate the virtual environment before running any NeMo Curator commands:

source /opt/venv/env.sh

The container uses a virtual environment at `/opt/venv`. If you see `No module named nemo_curator`, the environment has not been activated.
```

Alternatively, you can build the NeMo Curator container locally using the provided Dockerfile:

```bash
Expand All @@ -115,7 +128,7 @@ docker run --gpus all -it --rm nemo-curator:latest

**Benefits:**

- Pre-configured environment with all dependencies
- Pre-configured environment with all dependencies (FFmpeg, CUDA libraries)
- Consistent runtime across different systems
- Ideal for production deployments

Expand Down Expand Up @@ -157,6 +170,10 @@ If encoders are missing, reinstall `FFmpeg` with the required options or use the
:::
::::

```{note}
**FFmpeg build requires CUDA toolkit (nvcc):** If you encounter `ERROR: failed checking for nvcc` during FFmpeg installation, ensure that the CUDA toolkit is installed and `nvcc` is available on your `PATH`. You can verify with `nvcc --version`. If using the NeMo Curator container, FFmpeg is pre-installed with NVENC support.
```

---

## Package Extras
Expand Down
1 change: 1 addition & 0 deletions docs/broken_links_false_positives.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"filename": "get-started/text.md", "lineno": 119, "status": "broken", "code": 0, "uri": "https://huggingface.co/settings/tokens", "info": "unauthorized"}
4 changes: 2 additions & 2 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
project = "NeMo-Curator"
project_copyright = "2025, NVIDIA Corporation"
author = "NVIDIA Corporation"
release = "25.09"
release = "26.02"

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
Expand Down Expand Up @@ -122,7 +122,7 @@
"min_python_version": "3.8",
"recommended_cuda": "12.0+",
"current_release": release,
"container_version": "25.09",
"container_version": "26.02",
}

# Enable figure numbering
Expand Down
4 changes: 2 additions & 2 deletions docs/curate-audio/tutorials/beginner.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ cd NeMo-Curator/tutorials/audio/fleurs/

## Prerequisites

* NeMo Curator installed (see [Installation Guide](docs/admin/installation.md))
* NeMo Curator installed (see [Installation Guide](admin-installation))
* NVIDIA GPU (required for ASR inference, minimum 16GB VRAM recommended)
* Internet connection for dataset download
* Basic Python knowledge
Expand Down Expand Up @@ -373,7 +373,7 @@ After completing this tutorial, explore:

## Related Topics

- **[Audio Curation Quickstart](docs/get-started/audio.md)**: Quick introduction to audio curation
- **[Audio Curation Quickstart](gs-audio)**: Quick introduction to audio curation
- **[FLEURS Dataset](../load-data/fleurs-dataset.md)**: Detailed FLEURS dataset documentation
- **[Quality Assessment](../process-data/quality-assessment/index.md)**: Comprehensive quality metrics guide
- **[Save & Export](../save-export.md)**: Advanced export options and formats
11 changes: 11 additions & 0 deletions docs/curate-text/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,17 @@ Domain-specific processing for code and advanced curation tasks
{bdg-secondary}`code-processing`
:::

:::{grid-item-card} {octicon}`sparkles;1.5em;sd-mr-1` Synthetic Data Generation
:link: synthetic/index
:link-type: doc
Generate and augment training data using LLMs
+++
{bdg-secondary}`llm`
{bdg-secondary}`augmentation`
{bdg-secondary}`multilingual`
{bdg-secondary}`nemotron-cc`
:::

::::


Expand Down
4 changes: 4 additions & 0 deletions docs/curate-text/process-data/deduplication/fuzzy.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ Ideal for detecting documents with minor differences such as formatting changes,
- Ray cluster with GPU support (required for distributed processing)
- Stable document identifiers for removal (either existing IDs or IDs generated by the workflow and removal stages)

```{note}
**Running in Docker**: When running fuzzy deduplication inside the NeMo Curator container, ensure the container is started with `--gpus all` so that Ray workers can access the GPU. Without GPU access, you may see `CUDARuntimeError` or `AttributeError: 'CUDARuntimeError' object has no attribute 'msg'`. Also activate the virtual environment with `source /opt/venv/env.sh` after entering the container.
```

## Quick Start

Get started with fuzzy deduplication using the following example of identifying duplicates, then remove them:
Expand Down
20 changes: 10 additions & 10 deletions docs/curate-text/process-data/deduplication/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ NeMo Curator provides three deduplication approaches, each optimized for differe

::::{tab-item} Exact

**Method**: MD5 hashing
**Detects**: Character-for-character identical documents
**Method**: MD5 hashing
**Detects**: Character-for-character identical documents
**Speed**: Fastest

Computes MD5 hashes for each document's text content and groups documents with identical hashes. Best for removing exact copies.
Expand Down Expand Up @@ -58,8 +58,8 @@ For removal, use `TextDuplicatesRemovalWorkflow` with the generated duplicate ID

::::{tab-item} Fuzzy

**Method**: MinHash + Locality Sensitive Hashing (LSH)
**Detects**: Near-duplicates with minor edits (~80% similarity)
**Method**: MinHash + Locality Sensitive Hashing (LSH)
**Detects**: Near-duplicates with minor edits (~80% similarity)
**Speed**: Fast

Generates MinHash signatures and uses LSH to find similar documents. Best for detecting documents with small formatting differences or typos.
Expand Down Expand Up @@ -96,8 +96,8 @@ For removal, use `TextDuplicatesRemovalWorkflow` with the generated duplicate ID

::::{tab-item} Semantic

**Method**: Embeddings + clustering + pairwise similarity
**Detects**: Semantically similar content (paraphrases, translations)
**Method**: Embeddings + clustering + pairwise similarity
**Detects**: Semantically similar content (paraphrases, translations)
**Speed**: Moderate

Generates embeddings using transformer models, clusters them, and computes pairwise similarities. Best for meaning-based deduplication.
Expand All @@ -110,7 +110,7 @@ from nemo_curator.stages.text.deduplication.semantic import TextSemanticDeduplic

text_workflow = TextSemanticDeduplicationWorkflow(
input_path="/path/to/input/data",
output_path="/path/to/output",
output_path="/path/to/output",
cache_path="/path/to/cache",
text_field="text",
model_identifier="sentence-transformers/all-MiniLM-L6-v2",
Expand All @@ -126,7 +126,7 @@ text_workflow.run()
- `TextSemanticDeduplicationWorkflow`: For raw text with automatic embedding generation
- `SemanticDeduplicationWorkflow`: For pre-computed embeddings

See {ref}`Semantic Deduplication <text-process-data-dedup-semdedup>` for details.
See {ref}`Semantic Deduplication <text-process-data-format-sem-dedup>` for details.
:::

:::{dropdown} Advanced: Step-by-Step Semantic Deduplication
Expand Down Expand Up @@ -378,7 +378,7 @@ For detailed implementation guides, see:

- {ref}`Exact Duplicate Removal <text-process-data-dedup-exact>`
- {ref}`Fuzzy Duplicate Removal <text-process-data-dedup-fuzzy>`
- {ref}`Semantic Deduplication <text-process-data-dedup-semdedup>`
- {ref}`Semantic Deduplication <text-process-data-format-sem-dedup>`

:::{dropdown} Performance Considerations
:icon: zap
Expand Down Expand Up @@ -455,7 +455,7 @@ The ID Generator ensures consistent IDs across workflow stages.

- **New to deduplication**: Start with {ref}`Exact Duplicate Removal <text-process-data-dedup-exact>` for the fastest approach
- **Need near-duplicate detection**: See {ref}`Fuzzy Duplicate Removal <text-process-data-dedup-fuzzy>` for MinHash-based matching
- **Require semantic matching**: Explore {ref}`Semantic Deduplication <text-process-data-dedup-semdedup>` for meaning-based deduplication
- **Require semantic matching**: Explore {ref}`Semantic Deduplication <text-process-data-format-sem-dedup>` for meaning-based deduplication

**For hands-on guidance**: See {ref}`Text Curation Tutorials <text-tutorials>` for step-by-step examples.

Expand Down
4 changes: 4 additions & 0 deletions docs/curate-text/process-data/deduplication/semdedup.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ Based on [SemDeDup: Data-efficient learning at web-scale through semantic dedupl
- GPU acceleration (required for embedding generation and clustering)
- Stable document identifiers for removal (either existing IDs or IDs managed by the workflow and removal stages)

```{note}
**Running in Docker**: When running semantic deduplication inside the NeMo Curator container, ensure the container is started with `--gpus all` so that CUDA GPUs are available. Without this flag, you will see `RuntimeError: No CUDA GPUs are available`. Also activate the virtual environment with `source /opt/venv/env.sh` after entering the container.
```

## Quick Start

Get started with semantic deduplication using the following example of identifying duplicates, then remove them in one step:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The distributed data classification in NeMo Curator works by:

1. **Parallel Processing**: Chunking datasets across multiple computing nodes and GPUs to accelerate classification
2. **Pre-trained Models**: Using specialized models for different classification tasks
3. **Batched Inference**: Optimizing throughput with intelligent batching via CrossFit integration
3. **Batched Inference**: Optimizing throughput with intelligent batching
4. **Consistent API**: Providing a unified interface through the `DistributedDataClassifier` base class

The `DistributedDataClassifier` is designed to run on GPU clusters with minimal code changes regardless of which specific classifier you're using. All classifiers support filtering based on classification results and storing prediction scores as metadata.
Expand All @@ -29,6 +29,16 @@ The `DistributedDataClassifier` is designed to run on GPU clusters with minimal
Distributed classification requires GPU acceleration and is not supported for CPU-only processing. As long as GPU resources are available and NeMo Curator is correctly installed, GPU acceleration is handled automatically.
:::

```{tip}
**Running the tutorial notebooks**: The classification tutorial notebooks require the `text_cuda12` or `all` installation extra to include all relevant dependencies. If you encounter `ModuleNotFoundError`, reinstall with the appropriate extra:

uv pip install "nemo-curator[text_cuda12]"

When using classifiers that download from Hugging Face (such as Aegis and InstructionDataGuard), set your `HF_TOKEN` environment variable to avoid rate limiting:

export HF_TOKEN="your_token_here"
```

---

## Usage
Expand All @@ -39,16 +49,16 @@ NVIDIA NeMo Curator provides a base class `DistributedDataClassifier` that can b

| Classifier | Purpose | Model Location | Key Parameters | Requirements |
|---|---|---|---|---|
| DomainClassifier | Categorize English text by domain | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) | `filter_by`, `text_field` | None |
| MultilingualDomainClassifier | Categorize text in 52 languages by domain | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) | `filter_by`, `text_field` | None |
| QualityClassifier | Assess document quality | [nvidia/quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | `filter_by`, `text_field` | None |
| AegisClassifier | Detect unsafe content | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) | `aegis_variant`, `filter_by` | HuggingFace token |
| InstructionDataGuardClassifier | Detect poisoning attacks | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) | `text_field`, `label_field` | HuggingFace token |
| FineWebEduClassifier | Score educational value | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | `label_field`, `int_field` | None |
| FineWebMixtralEduClassifier | Score educational value (Mixtral annotations) | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None |
| FineWebNemotronEduClassifier | Score educational value (Nemotron annotations) | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None |
| ContentTypeClassifier | Categorize by speech type | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) | `filter_by`, `text_field` | None |
| PromptTaskComplexityClassifier | Classify prompt tasks and complexity | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `text_field` | None |
| DomainClassifier | Assigns one of 26 domain labels (such as "Sports," "Science," "News") to English text | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) | `filter_by`, `text_field` | None |
| MultilingualDomainClassifier | Assigns domain labels to text in 52 languages; same labels as DomainClassifier | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) | `filter_by`, `text_field` | None |
| QualityClassifier | Rates document quality as "Low," "Medium," or "High" using a DeBERTa model | [nvidia/quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | `filter_by`, `text_field` | None |
| AegisClassifier | Detects unsafe content across 13 risk categories (violence, hate speech, and others) using LlamaGuard | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) | `aegis_variant`, `filter_by` | HuggingFace token |
| InstructionDataGuardClassifier | Identifies LLM poisoning attacks in instruction-response pairs | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) | `text_field`, `label_field` | HuggingFace token |
| FineWebEduClassifier | Scores educational value from 0 to 5 (0=spam, 5=scholarly) for training data selection | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | `label_field`, `int_field` | None |
| FineWebMixtralEduClassifier | Scores educational value from 0 to 5 using Mixtral 8x22B annotation data | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None |
| FineWebNemotronEduClassifier | Scores educational value from 0 to 5 using Nemotron-4-340B annotation data | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None |
| ContentTypeClassifier | Categorizes text into 11 speech types (such as "Blogs," "News," "Academic") | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) | `filter_by`, `text_field` | None |
| PromptTaskComplexityClassifier | Labels prompts by task type (such as QA and summarization) and complexity dimensions | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `text_field` | None |

### Domain Classifier

Expand Down Expand Up @@ -365,6 +375,10 @@ pipeline.add_stage(writer)
results = pipeline.run() # Uses XennaExecutor by default
```

## Custom Model Integration

You can integrate your own classification models by extending `DistributedDataClassifier`. Refer to the [Text Classifiers README](https://github.com/NVIDIA-NeMo/Curator/tree/main/nemo_curator/stages/text/classifiers#text-classifiers) for implementation details and examples.

## Performance Optimization

NVIDIA NeMo Curator's distributed classifiers are optimized for high-throughput processing through several key features:
Expand Down
Loading