From 7b8c122cb59b62a56126343ffbb079fe424e61b0 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Fri, 6 Feb 2026 17:08:32 -0500 Subject: [PATCH 01/13] docs: vdr feedback Signed-off-by: Lawrence Lane --- docs/admin/installation.md | 29 ++++++++--- .../process-data/deduplication/fuzzy.md | 4 ++ .../process-data/deduplication/semdedup.md | 4 ++ .../distributed-classifier.md | 52 +++++++++++++++---- .../process-data/captions-preview.md | 2 +- docs/curate-video/process-data/clipping.md | 4 +- docs/curate-video/process-data/embeddings.md | 4 +- docs/curate-video/process-data/filtering.md | 4 +- .../process-data/frame-extraction.md | 4 +- docs/curate-video/tutorials/beginner.md | 4 +- docs/curate-video/tutorials/split-dedup.md | 4 +- docs/get-started/audio.md | 2 +- docs/get-started/text.md | 10 +++- docs/get-started/video.md | 27 ++++++++-- .../infrastructure/container-environments.md | 2 +- tutorials/quickstart.py | 6 +-- .../llama-nemotron-data-curation/README.md | 29 +++++++---- 17 files changed, 139 insertions(+), 52 deletions(-) diff --git a/docs/admin/installation.md b/docs/admin/installation.md index 10066ce064..fbf9df44a9 100644 --- a/docs/admin/installation.md +++ b/docs/admin/installation.md @@ -18,7 +18,7 @@ This guide covers installing NeMo Curator with support for **all modalities** an ### System Requirements -For comprehensive system requirements and production deployment specifications, see [Production Deployment Requirements](deployment/requirements.md). +For comprehensive system requirements and production deployment specifications, refer to [Production Deployment Requirements](deployment/requirements.md). **Quick Start Requirements:** @@ -26,6 +26,7 @@ For comprehensive system requirements and production deployment specifications, - **Python**: 3.10, 3.11, or 3.12 - **Memory**: 16GB+ RAM for basic text processing - **GPU** (optional): NVIDIA GPU with 16GB+ VRAM for acceleration +- **CUDA 12** (required for `audio_cuda12`, `video_cuda12`, `image_cuda12`, and `text_cuda12` extras) ### Development vs Production @@ -41,9 +42,13 @@ For comprehensive system requirements and production deployment specifications, Choose one of the following installation methods based on your needs: +:::{tip} +**Docker is the recommended installation method** for video and audio workflows. The NeMo Curator container includes FFmpeg (with NVENC support) and InternVideo2 pre-configured, avoiding manual dependency setup. Refer to the [Container Installation](#container-installation) tab below. +::: + ::::{tab-set} -:::{tab-item} PyPI Installation (Recommended) +:::{tab-item} PyPI Installation Install NeMo Curator from the Python Package Index using `uv` for proper dependency resolution. @@ -96,9 +101,9 @@ uv add InternVideo/InternVideo2/multi_modality ::: -:::{tab-item} Container Installation +:::{tab-item} Container Installation (Recommended for Video/Audio) -NeMo Curator is available as a standalone container on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator. The container includes NeMo Curator with all dependencies pre-installed. You can run it with: +NeMo Curator is available as a standalone container on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator. The container includes NeMo Curator with all dependencies pre-installed, including FFmpeg with NVENC support and InternVideo2. ```bash # Pull the container from NGC @@ -108,6 +113,14 @@ docker pull nvcr.io/nvidia/nemo-curator:{{ container_version }} docker run --gpus all -it --rm nvcr.io/nvidia/nemo-curator:{{ container_version }} ``` +```{important} +After entering the container, activate the virtual environment before running any NeMo Curator commands: + + source /opt/venv/env.sh + +The container uses a virtual environment at `/opt/venv`. If you see `No module named nemo_curator`, the environment has not been activated. +``` + Alternatively, you can build the NeMo Curator container locally using the provided Dockerfile: ```bash @@ -122,7 +135,7 @@ docker run --gpus all -it --rm nemo-curator:latest **Benefits:** -- Pre-configured environment with all dependencies +- Pre-configured environment with all dependencies (FFmpeg, InternVideo2, CUDA libraries) - Consistent runtime across different systems - Ideal for production deployments @@ -164,9 +177,13 @@ If encoders are missing, reinstall `FFmpeg` with the required options or use the ::: :::: +```{note} +**FFmpeg build requires CUDA toolkit (nvcc):** If you encounter `ERROR: failed checking for nvcc` during FFmpeg installation, ensure that the CUDA toolkit is installed and `nvcc` is available on your `PATH`. You can verify with `nvcc --version`. If using the NeMo Curator container, FFmpeg is pre-installed with NVENC support. +``` + ### InternVideo2 Support (Optional for Video) -Video processing includes optional support for InternVideo2. To install InternVideo2, run these commands before installing NeMo Curator based on whether you install via PyPI or from source: +Video processing includes optional support for InternVideo2. To install InternVideo2, run these commands before installing NeMo Curator based on whether you install through PyPI or from source: ::::{tab-set} diff --git a/docs/curate-text/process-data/deduplication/fuzzy.md b/docs/curate-text/process-data/deduplication/fuzzy.md index fb5e84c451..90e645832f 100644 --- a/docs/curate-text/process-data/deduplication/fuzzy.md +++ b/docs/curate-text/process-data/deduplication/fuzzy.md @@ -34,6 +34,10 @@ Ideal for detecting documents with minor differences such as formatting changes, - Ray cluster with GPU support (required for distributed processing) - Stable document identifiers for removal (either existing IDs or IDs generated by the workflow and removal stages) +```{note} +**Running in Docker**: When running fuzzy deduplication inside the NeMo Curator container, ensure the container is started with `--gpus all` so that Ray workers can access the GPU. Without GPU access, you may see `CUDARuntimeError` or `AttributeError: 'CUDARuntimeError' object has no attribute 'msg'`. Also activate the virtual environment with `source /opt/venv/env.sh` after entering the container. +``` + ## Quick Start Get started with fuzzy deduplication using the following example of identifying duplicates, then remove them: diff --git a/docs/curate-text/process-data/deduplication/semdedup.md b/docs/curate-text/process-data/deduplication/semdedup.md index 5d14ea1c1c..cbcb581f16 100644 --- a/docs/curate-text/process-data/deduplication/semdedup.md +++ b/docs/curate-text/process-data/deduplication/semdedup.md @@ -42,6 +42,10 @@ Based on [SemDeDup: Data-efficient learning at web-scale through semantic dedupl - GPU acceleration (required for embedding generation and clustering) - Stable document identifiers for removal (either existing IDs or IDs managed by the workflow and removal stages) +```{note} +**Running in Docker**: When running semantic deduplication inside the NeMo Curator container, ensure the container is started with `--gpus all` so that CUDA GPUs are available. Without this flag, you will see `RuntimeError: No CUDA GPUs are available`. Also activate the virtual environment with `source /opt/venv/env.sh` after entering the container. +``` + ## Quick Start Get started with semantic deduplication using the following example of identifying duplicates, then remove them in one step: diff --git a/docs/curate-text/process-data/quality-assessment/distributed-classifier.md b/docs/curate-text/process-data/quality-assessment/distributed-classifier.md index d9f92280e8..a57d54790c 100644 --- a/docs/curate-text/process-data/quality-assessment/distributed-classifier.md +++ b/docs/curate-text/process-data/quality-assessment/distributed-classifier.md @@ -20,7 +20,7 @@ The distributed data classification in NeMo Curator works by: 1. **Parallel Processing**: Chunking datasets across multiple computing nodes and GPUs to accelerate classification 2. **Pre-trained Models**: Using specialized models for different classification tasks -3. **Batched Inference**: Optimizing throughput with intelligent batching via CrossFit integration +3. **Batched Inference**: Optimizing throughput with intelligent batching through CrossFit integration 4. **Consistent API**: Providing a unified interface through the `DistributedDataClassifier` base class The `DistributedDataClassifier` is designed to run on GPU clusters with minimal code changes regardless of which specific classifier you're using. All classifiers support filtering based on classification results and storing prediction scores as metadata. @@ -29,6 +29,16 @@ The `DistributedDataClassifier` is designed to run on GPU clusters with minimal Distributed classification requires GPU acceleration and is not supported for CPU-only processing. As long as GPU resources are available and NeMo Curator is correctly installed, GPU acceleration is handled automatically. ::: +```{tip} +**Running the tutorial notebooks**: The classification tutorial notebooks in `tutorials/text/distributed-data-classification/` require additional packages (such as `pandas`) that may not be included in all installation configurations. If you encounter `ModuleNotFoundError`, install the missing packages: + + uv pip install pandas + +When using classifiers that download from Hugging Face (such as Aegis and InstructionDataGuard), set your `HF_TOKEN` environment variable to avoid rate limiting: + + export HF_TOKEN="your_token_here" +``` + --- ## Usage @@ -39,16 +49,16 @@ NVIDIA NeMo Curator provides a base class `DistributedDataClassifier` that can b | Classifier | Purpose | Model Location | Key Parameters | Requirements | |---|---|---|---|---| -| DomainClassifier | Categorize English text by domain | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) | `filter_by`, `text_field` | None | -| MultilingualDomainClassifier | Categorize text in 52 languages by domain | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) | `filter_by`, `text_field` | None | -| QualityClassifier | Assess document quality | [nvidia/quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | `filter_by`, `text_field` | None | -| AegisClassifier | Detect unsafe content | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) | `aegis_variant`, `filter_by` | HuggingFace token | -| InstructionDataGuardClassifier | Detect poisoning attacks | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) | `text_field`, `label_field` | HuggingFace token | -| FineWebEduClassifier | Score educational value | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | `label_field`, `int_field` | None | -| FineWebMixtralEduClassifier | Score educational value (Mixtral annotations) | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None | -| FineWebNemotronEduClassifier | Score educational value (Nemotron annotations) | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None | -| ContentTypeClassifier | Categorize by speech type | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) | `filter_by`, `text_field` | None | -| PromptTaskComplexityClassifier | Classify prompt tasks and complexity | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `text_field` | None | +| DomainClassifier | Assigns one of 26 domain labels (such as "Sports," "Science," "News") to English text | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) | `filter_by`, `text_field` | None | +| MultilingualDomainClassifier | Assigns domain labels to text in 52 languages; same labels as DomainClassifier | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) | `filter_by`, `text_field` | None | +| QualityClassifier | Rates document quality as "Low," "Medium," or "High" using a DeBERTa model | [nvidia/quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | `filter_by`, `text_field` | None | +| AegisClassifier | Detects unsafe content across 13 risk categories (violence, hate speech, and others) using LlamaGuard | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) | `aegis_variant`, `filter_by` | HuggingFace token | +| InstructionDataGuardClassifier | Identifies LLM poisoning attacks in instruction-response pairs | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) | `text_field`, `label_field` | HuggingFace token | +| FineWebEduClassifier | Scores educational value from 0 to 5 (0=spam, 5=scholarly) for training data selection | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | `label_field`, `int_field` | None | +| FineWebMixtralEduClassifier | Scores educational value from 0 to 5 using Mixtral 8x22B annotation data | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None | +| FineWebNemotronEduClassifier | Scores educational value from 0 to 5 using Nemotron-4-340B annotation data | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None | +| ContentTypeClassifier | Categorizes text into 11 speech types (such as "Blogs," "News," "Academic") | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) | `filter_by`, `text_field` | None | +| PromptTaskComplexityClassifier | Labels prompts by task type (such as QA and summarization) and complexity dimensions | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `text_field` | None | ### Domain Classifier @@ -365,6 +375,26 @@ pipeline.add_stage(writer) results = pipeline.run() # Uses XennaExecutor by default ``` +## Custom Model Integration + +You can integrate your own classification models by extending `DistributedDataClassifier`. The only requirement is that the model can fit on a single GPU. Create a subclass that implements the classification logic: + +```python +from nemo_curator.stages.text.classifiers.base import DistributedDataClassifier + +class MyCustomClassifier(DistributedDataClassifier): + """Custom classifier using your own model.""" + + def __init__(self, model_path: str, filter_by: list[str] | None = None, **kwargs): + super().__init__( + model_path=model_path, + filter_by=filter_by, + **kwargs + ) +``` + +All built-in classifiers follow this same pattern. Refer to an existing classifier implementation (such as `DomainClassifier`) as a template. + ## Performance Optimization NVIDIA NeMo Curator's distributed classifiers are optimized for high-throughput processing through several key features: diff --git a/docs/curate-video/process-data/captions-preview.md b/docs/curate-video/process-data/captions-preview.md index 0f32b03460..27674236c1 100644 --- a/docs/curate-video/process-data/captions-preview.md +++ b/docs/curate-video/process-data/captions-preview.md @@ -68,7 +68,7 @@ pipe.run() :::{tab-item} Script Flags ```bash -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --generate-captions \ --captioning-algorithm qwen \ diff --git a/docs/curate-video/process-data/clipping.md b/docs/curate-video/process-data/clipping.md index 86adceec77..d34045778b 100644 --- a/docs/curate-video/process-data/clipping.md +++ b/docs/curate-video/process-data/clipping.md @@ -84,7 +84,7 @@ pipe.run() ```bash # Fixed stride -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --splitting-algorithm fixed_stride \ --fixed-stride-split-duration 10.0 \ @@ -92,7 +92,7 @@ python -m nemo_curator.examples.video.video_split_clip_example \ --limit-clips 0 # TransNetV2 -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --splitting-algorithm transnetv2 \ --transnetv2-frame-decoder-mode pynvc \ diff --git a/docs/curate-video/process-data/embeddings.md b/docs/curate-video/process-data/embeddings.md index b7b30591f6..35dcd78448 100644 --- a/docs/curate-video/process-data/embeddings.md +++ b/docs/curate-video/process-data/embeddings.md @@ -65,14 +65,14 @@ pipe.run() ```bash # InternVideo2 -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --generate-embeddings \ --embedding-algorithm internvideo2 \ --embedding-gpu-memory-gb 20.0 # Cosmos-Embed1 (224p) -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --generate-embeddings \ --embedding-algorithm cosmos-embed1-224p \ diff --git a/docs/curate-video/process-data/filtering.md b/docs/curate-video/process-data/filtering.md index 8219171079..1c4b567f03 100644 --- a/docs/curate-video/process-data/filtering.md +++ b/docs/curate-video/process-data/filtering.md @@ -83,7 +83,7 @@ pipe.run() ```bash # Motion filtering -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --motion-filter enable \ --motion-decode-target-fps 2.0 \ @@ -95,7 +95,7 @@ python -m nemo_curator.examples.video.video_split_clip_example \ --motion-score-gpus-per-worker 0.5 # Aesthetic filtering -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --aesthetic-threshold 3.5 \ --aesthetic-reduction min \ diff --git a/docs/curate-video/process-data/frame-extraction.md b/docs/curate-video/process-data/frame-extraction.md index fab2c7b844..e7b3dacbbc 100644 --- a/docs/curate-video/process-data/frame-extraction.md +++ b/docs/curate-video/process-data/frame-extraction.md @@ -66,13 +66,13 @@ pipe.run() ```bash # Clip frames implicitly when generating embeddings or aesthetics -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --generate-embeddings \ --clip-extraction-target-res -1 # Full-video frames for TransNetV2 scene change -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --splitting-algorithm transnetv2 \ --transnetv2-frame-decoder-mode pynvc diff --git a/docs/curate-video/tutorials/beginner.md b/docs/curate-video/tutorials/beginner.md index 950b78dc84..965148ba95 100644 --- a/docs/curate-video/tutorials/beginner.md +++ b/docs/curate-video/tutorials/beginner.md @@ -230,10 +230,10 @@ pipeline.add_stage( When using the example pipeline module, configure the writer-related flags: ```bash -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ --video-dir "$VIDEO_DIR" \ --model-dir "$MODEL_DIR" \ - --output-clip-path "$OUT_DIR" \ + --output-path "$OUT_DIR" \ --no-upload-clips # optional: do not write mp4s --dry-run # optional: write nothing, validate only --generate-embeddings # optional: enable embedding outputs diff --git a/docs/curate-video/tutorials/split-dedup.md b/docs/curate-video/tutorials/split-dedup.md index 978a55dfa9..6e5e4f76dc 100644 --- a/docs/curate-video/tutorials/split-dedup.md +++ b/docs/curate-video/tutorials/split-dedup.md @@ -31,10 +31,10 @@ Learn how to run the splitting pipeline to generate clips and embeddings, then r Run the splitting example. Set `VIDEO_DIR`, `OUT_DIR`, and `MODEL_DIR` first. ```bash -python -m nemo_curator.examples.video.video_split_clip_example \ +python tutorials/video/getting-started/video_split_clip_example.py \ --video-dir "$VIDEO_DIR" \ --model-dir "$MODEL_DIR" \ - --output-clip-path "$OUT_DIR" \ + --output-path "$OUT_DIR" \ --splitting-algorithm fixed_stride \ --fixed-stride-split-duration 10.0 \ --embedding-algorithm internvideo2 \ diff --git a/docs/get-started/audio.md b/docs/get-started/audio.md index c90a32e3b9..95daec81b2 100644 --- a/docs/get-started/audio.md +++ b/docs/get-started/audio.md @@ -182,7 +182,7 @@ You can also run the pipeline using the downloaded configuration: ```bash cd ~/nemo_curator -python -m nemo_curator.examples.audio.fleurs.run \ +python tutorials/audio/fleurs/run.py \ --config-path ~/nemo_curator/configs \ --config-name fleurs_pipeline.yaml \ raw_data_dir=~/nemo_curator/audio_data diff --git a/docs/get-started/text.md b/docs/get-started/text.md index c168c1d9a5..84455d31ee 100644 --- a/docs/get-started/text.md +++ b/docs/get-started/text.md @@ -108,7 +108,15 @@ mkdir -p ~/nemo_curator/data/curated ``` ```{note} -For this example, you'll need sample JSONL files in `~/nemo_curator/data/sample/`. Each line should be a JSON object with at least `text` and `id` fields. You can create test data or refer to {ref}`Read Existing Data ` and {ref}`Data Loading ` for information on downloading data. +For this example, you need sample JSONL files in `~/nemo_curator/data/sample/`. Each line should be a JSON object with at least `text` and `id` fields. You can create test data or refer to {ref}`Read Existing Data ` and {ref}`Data Loading ` for information on downloading data. +``` + +```{tip} +**Set your HuggingFace token** to avoid rate limiting when downloading models or datasets: + + export HF_TOKEN="your_token_here" + +Without a token, repeated downloads from Hugging Face may result in `429 Client Error` (rate limiting). Get a free token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). ``` ## Basic Text Curation Example diff --git a/docs/get-started/video.md b/docs/get-started/video.md index e9c6388970..b9242b9474 100644 --- a/docs/get-started/video.md +++ b/docs/get-started/video.md @@ -26,7 +26,7 @@ This quickstart guide demonstrates how to: 4. **Process videos** through a complete splitting and embedding pipeline 5. **Generate outputs** ready for duplicate removal, captioning, and model training -**What you'll build:** A video processing pipeline that: +**What you build:** A video processing pipeline that: - Splits videos into 10-second clips using fixed stride or scene detection - Generates clip-level embeddings for similarity search and deduplication - Optionally creates captions and preview images @@ -63,7 +63,7 @@ To use NeMo Curator's video curation capabilities, ensure your system meets thes - CPU encoders: `libopenh264` or `libx264` (fallback options) :::{tip} -If you don't have `uv` installed, refer to the [Installation Guide](../admin/installation.md) for setup instructions, or install it quickly with: +If `uv` is not installed, refer to the [Installation Guide](../admin/installation.md) for setup instructions, or install it quickly with: ```bash curl -LsSf https://astral.sh/uv/0.8.22/install.sh | sh @@ -228,7 +228,11 @@ NeMo Curator supports two embedding model families: Open model that requires the IV2 checkpoint and BERT model files to be available locally; higher VRAM usage. - [InternVideo Official Github Page](https://github.com/OpenGVLab/InternVideo) -For this quickstart, we're going to set up support for **Cosmos-Embed1-224p**. +```{important} +If you plan to use InternVideo2 embeddings, you must install InternVideo2 **before** running the pipeline. Refer to the [InternVideo2 installation instructions](../admin/installation.md#internvideo2-support-optional-for-video) for setup steps. The NeMo Curator container includes InternVideo2 pre-installed. +``` + +For this quickstart, the following steps set up support for **Cosmos-Embed1-224p**. ### Prepare Model Weights @@ -256,7 +260,7 @@ Organize input videos and output locations before running the pipeline. MODEL_DIR=/path/to/models ``` -- **S3**: For cloud storage (AWS S3, MinIO, etc.). Configure credentials in `~/.aws/credentials` and use `s3://` paths for `--video-dir` and `--output-clip-path`. +- **S3**: For cloud storage (AWS S3, MinIO, etc.). Configure credentials in `~/.aws/credentials` and use `s3://` paths for `--video-dir` and `--output-path`. **S3 usage notes:** - Input videos can be read from S3 paths @@ -272,7 +276,7 @@ Use the example script from https://github.com/NVIDIA-NeMo/Curator/tree/main/tut python tutorials/video/getting-started/video_split_clip_example.py \ --video-dir "$DATA_DIR" \ --model-dir "$MODEL_DIR" \ - --output-clip-path "$OUT_DIR" \ + --output-path "$OUT_DIR" \ --splitting-algorithm fixed_stride \ --fixed-stride-split-duration 10.0 \ --embedding-algorithm cosmos-embed1-224p \ @@ -287,6 +291,19 @@ python tutorials/video/getting-started/video_split_clip_example.py \ 4. Encodes clips using libopenh264 codec 5. Writes output clips and metadata to `$OUT_DIR` +```{tip} +**Using a config file**: The example script accepts many command-line arguments. For complex configurations, you can store arguments in a file and pass them with the `@` prefix: + + echo '--video-dir /data/videos + --output-path /data/output + --splitting-algorithm fixed_stride + --fixed-stride-split-duration 10.0 + --embedding-algorithm cosmos-embed1-224p + --transcode-encoder libopenh264' > my_config.txt + + python tutorials/video/getting-started/video_split_clip_example.py @my_config.txt +``` + ### Configuration Options Reference | Option | Values | Description | diff --git a/docs/reference/infrastructure/container-environments.md b/docs/reference/infrastructure/container-environments.md index 0ce46be76e..3075215eef 100644 --- a/docs/reference/infrastructure/container-environments.md +++ b/docs/reference/infrastructure/container-environments.md @@ -76,7 +76,7 @@ The primary container includes comprehensive support for all curation modalities * - Installation - NeMo Curator installed with all optional dependencies (`[all]` extras) using uv with NVIDIA index * - Environment Path - - Virtual environment activated by default: `/opt/venv/bin:$PATH` + - Virtual environment at `/opt/venv`. Activate with `source /opt/venv/env.sh` after entering the container. ``` --- diff --git a/tutorials/quickstart.py b/tutorials/quickstart.py index aa6b7fe271..5cf2ade2fa 100644 --- a/tutorials/quickstart.py +++ b/tutorials/quickstart.py @@ -39,9 +39,9 @@ from nemo_curator.tasks import Task, _EmptyTask SAMPLE_SENTENCES = [ - "I love this product", - "I hate this product", - "I'm neutral about this product", + "I love this product, it works great", + "I hate this product, it broke immediately", + "This product is okay but nothing special", ] diff --git a/tutorials/text/llama-nemotron-data-curation/README.md b/tutorials/text/llama-nemotron-data-curation/README.md index f70f128767..c4fb9a271b 100644 --- a/tutorials/text/llama-nemotron-data-curation/README.md +++ b/tutorials/text/llama-nemotron-data-curation/README.md @@ -4,7 +4,7 @@ The [Llama Nemotron Post-Training Dataset](https://huggingface.co/datasets/nvidi Organized into distinct subsets for supervised fine-tuning (SFT) or reinforcement learning (RL), it encompasses samples from various problem domains. All samples are in JSON lines (JSONL) format and contain metadata such as license type, source model, as well as the [Llama Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/llama-nemotron/) model(s) trained with that sample. -Each sample consists of a prompt, along with an expected response with detailed chain-of-thought (CoT) reasoning traces followed by responses (i.e., "reasoning on"), as well as samples with direct responses (i.e., "reasoning off"). +Each sample consists of a prompt, along with an expected response with detailed chain-of-thought (CoT) reasoning traces followed by responses (that is, "reasoning on"), as well as samples with direct responses (that is, "reasoning off"). Here is an example of what a sample from the dataset may look like: ```bash @@ -42,7 +42,7 @@ Setup requirements: - Hardware: This tutorial can be run entirely on CPU workers - Recommended environment: This tutorial was developed and tested with a Conda environment -Please refer to NeMo Curator's [documentation](https://docs.nvidia.com/nemo/curator/latest/) for instructions on how to download NeMo Curator via PyPI, source, or Docker. +Refer to the NeMo Curator [documentation](https://docs.nvidia.com/nemo/curator/latest/) for instructions on how to download NeMo Curator through PyPI, source, or Docker. ## Prerequisites @@ -58,7 +58,7 @@ git lfs install git clone https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset ``` -Alternatively, the dataset can be downloaded via Python: +Alternatively, the dataset can be downloaded using Python: ```python from huggingface_hub import snapshot_download @@ -71,7 +71,7 @@ snapshot_download( ) ``` -Please ensure that the above dataset was downloaded correctly. You can check: +Ensure that the dataset was downloaded correctly. You can verify with the following commands: ```bash $ ls /path/to/Llama-Nemotron-Post-Training-Dataset/SFT @@ -89,7 +89,7 @@ The tokenizer used by this tutorial is called [meta-llama/Llama-3.1-8B-Instruct] 1. Visit https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct 2. Click "Access request" 3. Fill out the form and wait for approval -4. Once approved, log in to your Hugging Face account via the Hugging Face CLI. In the terminal, this can be done via `huggingface-cli login` +4. After approval, log in to your Hugging Face account using the Hugging Face CLI. In the terminal, run `huggingface-cli login` ### Download FastText language identification model @@ -130,20 +130,27 @@ The above script applies basic filtering to the input dataset: - Only take samples used for Nemotron Nano training - Remove empty and malformed samples - Remove non-English samples -- Remove samples with total length (system prompt, input, and output responses) longer than 16k tokens (with chat template applied via the tokenizer) -- Remove samples with output responses longer than 8k tokens (with chat template applied via the tokenizer) +- Remove samples with total length (system prompt, input, and output responses) longer than 16k tokens (with chat template applied using the tokenizer) +- Remove samples with output responses longer than 8k tokens (with chat template applied using the tokenizer) - Only keep columns specified by the `--keep-columns` parameter. We recommend keeping the "input", "output", and "completion_token_count" columns (the "completion_token_count" column always needs to be kept, so that we can sort the samples) -After filtering, it sorts all samples by completion (output response) length, then "interleaves" thinking ON/thinking OFF for curriculum learning. The idea here is to sort the samples in increasing order of difficulty, using the completion token count as a measure of sample difficulty. By default, we interleave one record at a time (i.e., 1 thinking ON sample, then 1 thinking OFF sample, then 1 thinking ON sample, etc.). The user may pass `--chunk-size` followed by an integer to interleave 10 records at a time (10 thinking ON samples, then 10 thinking OFF samples, then 10 thinking ON samples, and so on), 100 records at a time, etc. as desired. We interleave samples from the "reasoning on" and "reasoning off" buckets to gradually introduce complexity. +After filtering, it sorts all samples by completion (output response) length, then "interleaves" thinking ON/thinking OFF for curriculum learning. The idea here is to sort the samples in increasing order of difficulty, using the completion token count as a measure of sample difficulty. By default, we interleave one record at a time (that is, one thinking ON sample, then one thinking OFF sample, then one thinking ON sample, and so on). The user may pass `--chunk-size` followed by an integer to interleave 10 records at a time (10 thinking ON samples, then 10 thinking OFF samples, then 10 thinking ON samples, and so on), 100 records at a time, and so on as desired. We interleave samples from the "reasoning on" and "reasoning off" buckets to gradually introduce complexity. + +## System Requirements + +- **Memory**: This tutorial can be CPU-only but is memory-intensive. We recommend **128 GB+ RAM** for processing the full dataset. For smaller memory systems, use `--filename-filter` to select a subset of the data. +- **CPU allocation**: The `--num-cpus` parameter controls parallelism. Each CPU worker processes data in parallel, so more CPUs means more memory usage. **Start with a conservative value** (such as `--num-cpus 8`) and increase gradually. ## Debugging Out of Memory Errors -If you are running into out of memory (OOM) errors, there are a couple of approaches you can try. One is to avoid very large partitions of data. By default, the JSONL data is read with a blocksize of 100 MB per partition. To customize the file reading logic, the user may specify `--json-blocksize "100mb"` with any string representation for the partition size in MB (e.g., "100mb", "256mb"). +If you encounter out-of-memory (OOM) errors: -It can be useful to play around with the `--num-cpus` parameter as well. The goal is to maximize it for improved performance without oversubscribing your available hardware. +1. **Reduce partition size**: Lower the blocksize to reduce per-partition memory. Set `--json-blocksize "50mb"` (default is "100mb"). +2. **Reduce CPU count**: Lower `--num-cpus` to reduce parallel memory pressure. For example, if you have 128 GB RAM, start with `--num-cpus 8` rather than using all available cores. +3. **Subset the data**: Use `--filename-filter` to process only specific subsets (such as `--filename-filter "chat"`). ## Next Steps -To see how to train a reasoning model with the resulting dataset, please refer to this NeMo tutorial: [Train Your Own Reasoning Model in 48 Hours on a Single GPU](https://github.com/NVIDIA/NeMo/tree/main/tutorials/llm/reasoning). +To see how to train a reasoning model with the resulting dataset, refer to this NeMo tutorial: [Train Your Own Reasoning Model in 48 Hours on a Single GPU](https://github.com/NVIDIA/NeMo/tree/main/tutorials/llm/reasoning). The NeMo tutorial expects the `/path/to/curated-data/training.jsonl` file generated by this tutorial as input. From f874e77141b820922cbf2314b5cd12526aad36ba Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Mon, 9 Feb 2026 14:53:38 -0500 Subject: [PATCH 02/13] Update docs/admin/installation.md Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Signed-off-by: Lawrence Lane --- docs/admin/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/installation.md b/docs/admin/installation.md index b8a0fd8d1d..3cd6a39295 100644 --- a/docs/admin/installation.md +++ b/docs/admin/installation.md @@ -43,7 +43,7 @@ For comprehensive system requirements and production deployment specifications, Choose one of the following installation methods based on your needs: :::{tip} -**Docker is the recommended installation method** for video and audio workflows. The NeMo Curator container includes FFmpeg (with NVENC support) and InternVideo2 pre-configured, avoiding manual dependency setup. Refer to the [Container Installation](#container-installation) tab below. +**Docker is the recommended installation method** for video and audio workflows. The NeMo Curator container includes FFmpeg (with NVENC support) pre-configured, avoiding manual dependency setup. Refer to the [Container Installation](#container-installation) tab below. ::: ::::{tab-set} From a98e5efa0e991092e0a981dae0a320615debdfac Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Mon, 9 Feb 2026 14:53:49 -0500 Subject: [PATCH 03/13] Update docs/admin/installation.md Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Signed-off-by: Lawrence Lane --- docs/admin/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/installation.md b/docs/admin/installation.md index 3cd6a39295..7dc3192772 100644 --- a/docs/admin/installation.md +++ b/docs/admin/installation.md @@ -96,7 +96,7 @@ uv sync --all-extras --all-groups :::{tab-item} Container Installation (Recommended for Video/Audio) -NeMo Curator is available as a standalone container on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator. The container includes NeMo Curator with all dependencies pre-installed, including FFmpeg with NVENC support and InternVideo2. +NeMo Curator is available as a standalone container on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator. The container includes NeMo Curator with all dependencies pre-installed, including FFmpeg with NVENC support. ```bash # Pull the container from NGC From 7bc31f8a026453280cc80dc6f3f8fdd84298c5c7 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Mon, 9 Feb 2026 14:54:01 -0500 Subject: [PATCH 04/13] Update docs/admin/installation.md Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Signed-off-by: Lawrence Lane --- docs/admin/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/installation.md b/docs/admin/installation.md index 7dc3192772..2e333369a3 100644 --- a/docs/admin/installation.md +++ b/docs/admin/installation.md @@ -128,7 +128,7 @@ docker run --gpus all -it --rm nemo-curator:latest **Benefits:** -- Pre-configured environment with all dependencies (FFmpeg, InternVideo2, CUDA libraries) +- Pre-configured environment with all dependencies (FFmpeg, CUDA libraries) - Consistent runtime across different systems - Ideal for production deployments From 76db4e5478fa4d8331d9471c37835f34b9b081ec Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Mon, 9 Feb 2026 14:54:48 -0500 Subject: [PATCH 05/13] Update docs/curate-text/process-data/quality-assessment/distributed-classifier.md Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Signed-off-by: Lawrence Lane --- .../process-data/quality-assessment/distributed-classifier.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/curate-text/process-data/quality-assessment/distributed-classifier.md b/docs/curate-text/process-data/quality-assessment/distributed-classifier.md index a57d54790c..2442db0c65 100644 --- a/docs/curate-text/process-data/quality-assessment/distributed-classifier.md +++ b/docs/curate-text/process-data/quality-assessment/distributed-classifier.md @@ -20,7 +20,7 @@ The distributed data classification in NeMo Curator works by: 1. **Parallel Processing**: Chunking datasets across multiple computing nodes and GPUs to accelerate classification 2. **Pre-trained Models**: Using specialized models for different classification tasks -3. **Batched Inference**: Optimizing throughput with intelligent batching through CrossFit integration +3. **Batched Inference**: Optimizing throughput with intelligent batching 4. **Consistent API**: Providing a unified interface through the `DistributedDataClassifier` base class The `DistributedDataClassifier` is designed to run on GPU clusters with minimal code changes regardless of which specific classifier you're using. All classifiers support filtering based on classification results and storing prediction scores as metadata. From 00da06b8204cff9f5c21915e22fc3dbd2a04676f Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Mon, 9 Feb 2026 14:56:38 -0500 Subject: [PATCH 06/13] feedback Signed-off-by: Lawrence Lane --- docs/admin/installation.md | 4 ++++ .../distributed-classifier.md | 22 +++---------------- docs/get-started/video.md | 2 +- .../llama-nemotron-data-curation/README.md | 6 ++--- 4 files changed, 11 insertions(+), 23 deletions(-) diff --git a/docs/admin/installation.md b/docs/admin/installation.md index 2e333369a3..b779a6af67 100644 --- a/docs/admin/installation.md +++ b/docs/admin/installation.md @@ -170,6 +170,10 @@ If encoders are missing, reinstall `FFmpeg` with the required options or use the ::: :::: +```{note} +**FFmpeg build requires CUDA toolkit (nvcc):** If you encounter `ERROR: failed checking for nvcc` during FFmpeg installation, ensure that the CUDA toolkit is installed and `nvcc` is available on your `PATH`. You can verify with `nvcc --version`. If using the NeMo Curator container, FFmpeg is pre-installed with NVENC support. +``` + --- ## Package Extras diff --git a/docs/curate-text/process-data/quality-assessment/distributed-classifier.md b/docs/curate-text/process-data/quality-assessment/distributed-classifier.md index 2442db0c65..f3e4abcde1 100644 --- a/docs/curate-text/process-data/quality-assessment/distributed-classifier.md +++ b/docs/curate-text/process-data/quality-assessment/distributed-classifier.md @@ -30,9 +30,9 @@ Distributed classification requires GPU acceleration and is not supported for CP ::: ```{tip} -**Running the tutorial notebooks**: The classification tutorial notebooks in `tutorials/text/distributed-data-classification/` require additional packages (such as `pandas`) that may not be included in all installation configurations. If you encounter `ModuleNotFoundError`, install the missing packages: +**Running the tutorial notebooks**: The classification tutorial notebooks require the `text_cuda12` or `all` installation extra to include all relevant dependencies. If you encounter `ModuleNotFoundError`, reinstall with the appropriate extra: - uv pip install pandas + uv pip install "nemo-curator[text_cuda12]" When using classifiers that download from Hugging Face (such as Aegis and InstructionDataGuard), set your `HF_TOKEN` environment variable to avoid rate limiting: @@ -377,23 +377,7 @@ results = pipeline.run() # Uses XennaExecutor by default ## Custom Model Integration -You can integrate your own classification models by extending `DistributedDataClassifier`. The only requirement is that the model can fit on a single GPU. Create a subclass that implements the classification logic: - -```python -from nemo_curator.stages.text.classifiers.base import DistributedDataClassifier - -class MyCustomClassifier(DistributedDataClassifier): - """Custom classifier using your own model.""" - - def __init__(self, model_path: str, filter_by: list[str] | None = None, **kwargs): - super().__init__( - model_path=model_path, - filter_by=filter_by, - **kwargs - ) -``` - -All built-in classifiers follow this same pattern. Refer to an existing classifier implementation (such as `DomainClassifier`) as a template. +You can integrate your own classification models by extending `DistributedDataClassifier`. Refer to the [Text Classifiers README](https://github.com/NVIDIA-NeMo/Curator/tree/main/nemo_curator/stages/text/classifiers#text-classifiers) for implementation details and examples. ## Performance Optimization diff --git a/docs/get-started/video.md b/docs/get-started/video.md index 5309848462..74cac57df4 100644 --- a/docs/get-started/video.md +++ b/docs/get-started/video.md @@ -183,7 +183,7 @@ NeMo Curator supports two embedding model families: - [cosmos-embed1-336p on Hugging Face](https://huggingface.co/nvidia/cosmos-embed1-336p) - [cosmos-embed1-448p on Hugging Face](https://huggingface.co/nvidia/cosmos-embed1-448p) -For this quickstart, we're going to set up support for **Cosmos-Embed1-224p**. +For this quickstart, the following steps set up support for **Cosmos-Embed1-224p**. ### Prepare Model Weights diff --git a/tutorials/text/llama-nemotron-data-curation/README.md b/tutorials/text/llama-nemotron-data-curation/README.md index c4fb9a271b..28b8d5a2da 100644 --- a/tutorials/text/llama-nemotron-data-curation/README.md +++ b/tutorials/text/llama-nemotron-data-curation/README.md @@ -138,15 +138,15 @@ After filtering, it sorts all samples by completion (output response) length, th ## System Requirements -- **Memory**: This tutorial can be CPU-only but is memory-intensive. We recommend **128 GB+ RAM** for processing the full dataset. For smaller memory systems, use `--filename-filter` to select a subset of the data. -- **CPU allocation**: The `--num-cpus` parameter controls parallelism. Each CPU worker processes data in parallel, so more CPUs means more memory usage. **Start with a conservative value** (such as `--num-cpus 8`) and increase gradually. +- **Memory**: This tutorial can be CPU-only but is memory-intensive. For smaller memory systems, use `--filename-filter` to select a subset of the data. +- **CPU allocation**: The `--num-cpus` parameter controls parallelism. Each CPU worker processes data in parallel, so more CPUs means more memory usage. Start with a conservative value and increase gradually. ## Debugging Out of Memory Errors If you encounter out-of-memory (OOM) errors: 1. **Reduce partition size**: Lower the blocksize to reduce per-partition memory. Set `--json-blocksize "50mb"` (default is "100mb"). -2. **Reduce CPU count**: Lower `--num-cpus` to reduce parallel memory pressure. For example, if you have 128 GB RAM, start with `--num-cpus 8` rather than using all available cores. +2. **Reduce CPU count**: Lower `--num-cpus` to reduce parallel memory pressure rather than using all available cores. 3. **Subset the data**: Use `--filename-filter` to process only specific subsets (such as `--filename-filter "chat"`). ## Next Steps From dc1109e0abedbc4ac7fc92b26ed3d8117267e1aa Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 11 Feb 2026 11:15:35 -0500 Subject: [PATCH 07/13] feedback Signed-off-by: Lawrence Lane --- docs/broken_links_false_positives.json | 1 + docs/get-started/image.md | 14 ++++++++++++-- docs/project.json | 2 +- .../infrastructure/container-environments.md | 2 +- 4 files changed, 15 insertions(+), 4 deletions(-) create mode 100644 docs/broken_links_false_positives.json diff --git a/docs/broken_links_false_positives.json b/docs/broken_links_false_positives.json new file mode 100644 index 0000000000..8567d468c0 --- /dev/null +++ b/docs/broken_links_false_positives.json @@ -0,0 +1 @@ +{"filename": "get-started/text.md", "lineno": 119, "status": "broken", "code": 0, "uri": "https://huggingface.co/settings/tokens", "info": "unauthorized"} diff --git a/docs/get-started/image.md b/docs/get-started/image.md index 588028621b..6f21a506c9 100644 --- a/docs/get-started/image.md +++ b/docs/get-started/image.md @@ -114,6 +114,16 @@ For this example, you'll need: Here's a simple example to get started with NeMo Curator's image curation pipeline: +:::{note} +**CPU Memory Considerations** + +Image loading and decoding happens in CPU memory before GPU processing. If you encounter out-of-memory errors during the `ImageReaderStage`, reduce: +- `batch_size`: Number of images per batch (reduce to 32-50 for systems with limited RAM) +- `num_threads`: Parallel decoding threads (reduce to 4 for systems with limited RAM) + +The example below uses conservative defaults suitable for most systems. For high-memory systems, you can increase these values for better performance. +::: + ```python from nemo_curator.pipeline import Pipeline from nemo_curator.backends.xenna import XennaExecutor @@ -136,9 +146,9 @@ pipeline.add_stage(FilePartitioningStage( # Stage 2: Read images from tar files using DALI pipeline.add_stage(ImageReaderStage( - batch_size=100, + batch_size=50, verbose=True, - num_threads=8, + num_threads=4, num_gpus_per_worker=0.25, )) diff --git a/docs/project.json b/docs/project.json index f719e487e2..496251b8c2 100644 --- a/docs/project.json +++ b/docs/project.json @@ -1 +1 @@ -{"name": "nemo-curator", "version": "25.09"} \ No newline at end of file +{"name": "nemo-curator", "version": "26.02"} \ No newline at end of file diff --git a/docs/reference/infrastructure/container-environments.md b/docs/reference/infrastructure/container-environments.md index fd60b4e4e7..bb05df1613 100644 --- a/docs/reference/infrastructure/container-environments.md +++ b/docs/reference/infrastructure/container-environments.md @@ -37,7 +37,7 @@ NeMo Curator provides official Docker containers with all dependencies pre-insta The primary container includes comprehensive support for all curation modalities: -**Container registry:** `nvcr.io/nvidia/nemo-curator:25.09` +**Container registry:** `nvcr.io/nvidia/nemo-curator:26.02` **Supported modalities:** - ✅ Text curation (CPU/GPU) From ddb5c6e9d2786f4fd315f6371ac25c39816aeec7 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 11 Feb 2026 12:07:20 -0500 Subject: [PATCH 08/13] re order sidebar Signed-off-by: Lawrence Lane --- docs/index.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/index.md b/docs/index.md index fc4390ebf3..62f4581f16 100644 --- a/docs/index.md +++ b/docs/index.md @@ -215,6 +215,14 @@ Video Curation Quickstart Audio Curation Quickstart :::: +::::{toctree} +:hidden: +:caption: Setup & Deployment +:maxdepth: 2 +admin/index.md +Install Curator +:::: + ::::{toctree} :hidden: :caption: Curate Text @@ -261,14 +269,6 @@ Process Data Save & Export :::: -::::{toctree} -:hidden: -:caption: Setup & Deployment -:maxdepth: 2 -admin/index.md -Install Curator -:::: - ::::{toctree} :hidden: :caption: Reference From a2a340dcb4affcd1ee14544936e322d364e0cbdd Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 11 Feb 2026 12:42:21 -0500 Subject: [PATCH 09/13] release notes draft Signed-off-by: Lawrence Lane --- docs/about/release-notes/index.md | 308 ++++++++++++------------------ 1 file changed, 123 insertions(+), 185 deletions(-) diff --git a/docs/about/release-notes/index.md b/docs/about/release-notes/index.md index 2e2ded5f08..58562d9838 100644 --- a/docs/about/release-notes/index.md +++ b/docs/about/release-notes/index.md @@ -12,215 +12,153 @@ modality: "universal" # NeMo Curator Release Notes: {{ current_release }} -## Synthetic Data Generation +## What's New in 26.02 -New Ray-based synthetic data generation capabilities for creating and augmenting training data using LLMs: - -- **LLM Client Infrastructure**: OpenAI-compatible async/sync clients with automatic rate limiting, retry logic, and exponential backoff -- **Multilingual Q&A Generation**: Generate synthetic Q&A pairs across multiple languages using customizable prompts -- **Nemotron-CC Pipelines**: Advanced text transformation and knowledge extraction workflows: - - **Wikipedia Paraphrasing**: Improve low-quality text by rewriting in Wikipedia-style prose - - **Diverse QA**: Generate diverse question-answer pairs for reading comprehension training - - **Distill**: Create condensed, information-dense paraphrases preserving key concepts - - **Extract Knowledge**: Extract factual content as textbook-style passages - - **Knowledge List**: Extract structured fact lists from documents - -Learn more in the [Synthetic Data Generation documentation](../../curate-text/synthetic/index.md). - - ```{list-table} Available Installation Extras - :header-rows: 1 - :widths: 25 35 40 - - * - Extra - - Installation Command - - Description - * - **All Modalities** - - `nemo-curator[all]` - - Complete installation with all modalities and GPU support - * - **Text Curation** - - `nemo-curator[text_cuda12]` - - GPU-accelerated text processing with RAPIDS - * - **Image Curation** - - `nemo-curator[image_cuda12]` - - Image processing with NVIDIA DALI - * - **Audio Curation** - - `nemo-curator[audio_cuda12]` - - Speech recognition with NeMo ASR models - * - **Video Curation** - - `nemo-curator[video_cuda12]` - - Video processing with GPU acceleration - * - **Basic GPU** - - `nemo-curator[cuda12]` - - CUDA utilities without modality-specific dependencies - ``` - - All GPU installations require the NVIDIA PyPI index: - ```bash - uv pip install https://pypi.nvidia.com nemo-curator[EXTRA] - ``` - -## New Modalities - -### Video - -NeMo Curator now supports comprehensive [video data curation](../../curate-video/index.md) with distributed processing capabilities: - -- **Video splitting**: [Fixed-stride](../../curate-video/process-data/clipping.md) and [scene-change detection (TransNetV2)](../../curate-video/process-data/clipping.md) for clip extraction -- **Semantic deduplication**: [K-means clustering and pairwise similarity](../../curate-video/process-data/dedup.md) for near-duplicate clip removal -- **Content filtering**: [Motion-based filtering](../../curate-video/process-data/filtering.md) and [aesthetic filtering](../../curate-video/process-data/filtering.md) for quality improvement -- **Embedding generation**: Cosmos-Embed1 models for clip-level embeddings -- **Enhanced captioning**: [VL-based caption generation with optional LLM-based rewriting](../../curate-video/process-data/captions-preview.md) (Qwen-VL and Qwen-LM supported) for detailed video descriptions -- **Ray-based distributed architecture**: Scalable video processing with [autoscaling support](../concepts/video/architecture.md) - -### Audio - -New [audio curation capabilities](../../curate-audio/index.md) for speech data processing: - -- **ASR inference**: [Automatic speech recognition](../../curate-audio/process-data/asr-inference/index.md) using NeMo Framework pretrained models -- **Quality assessment**: [Word Error Rate (WER) and Character Error Rate (CER)](../../curate-audio/process-data/quality-assessment/index.md) calculation -- **Speech metrics**: [Duration analysis and speech rate metrics](../../curate-audio/process-data/audio-analysis/index.md) (words/characters per second) -- **Text integration**: Seamless integration with [text curation workflows](../../curate-audio/process-data/text-integration/index.md) via `AudioToDocumentStage` -- **Manifest support**: JSONL manifest format for audio file management - -## Modality Refactors - -### Text - -- **Ray backend migration**: Complete transition from Dask to Ray for distributed [text processing](../../curate-text/index.md) -- **Improved model-based classifier throughput**: Better overlapping of compute between tokenization and inference through [length-based sequence sorting](../../curate-text/process-data/quality-assessment/distributed-classifier.md) for optimal GPU memory utilization -- **Task-centric architecture**: New `Task`-based processing model for finer-grained control -- **Pipeline redesign**: Updated `ProcessingStage` and `Pipeline` architecture with resource specification - -### Image - -- **Pipeline-based architecture**: Transitioned from legacy `ImageTextPairDataset` to modern [stage-based processing](../../curate-images/index.md) with `ImageReaderStage`, `ImageEmbeddingStage`, and filter stages -- **DALI-based image loading**: New `ImageReaderStage` uses NVIDIA DALI for high-performance WebDataset tar shard processing with GPU/CPU fallback -- **Modular processing stages**: Separate stages for [embedding generation](../../curate-images/process-data/embeddings/index.md), [aesthetic filtering](../../curate-images/process-data/filters/aesthetic.md), and [NSFW filtering](../../curate-images/process-data/filters/nsfw.md) -- **Task-based data flow**: Images processed as `ImageBatch` tasks containing `ImageObject` instances with metadata, embeddings, and classification scores - -Learn more about [image curation](../../curate-images/index.md). - -## Deduplication Improvements - -Enhanced deduplication capabilities across all modalities with improved performance and flexibility: - -- **Exact and Fuzzy deduplication**: Updated [rapidsmpf-based shuffle backend](../../reference/infrastructure/gpu-processing.md) for more efficient GPU-to-GPU data transfer and better spilling capabilities -- **Semantic deduplication**: Support for deduplicating [text](../../curate-text/process-data/deduplication/semdedup.md) and [video](../../curate-video/process-data/dedup.md) datasets using unified embedding-based workflows -- **New ranking strategies**: Added `RankingStrategy` which allows you to rank elements within cluster centers to decide which point to prioritize during duplicate removal, supporting [metadata-based ranking](../../curate-text/process-data/deduplication/semdedup.md) to prioritize specific datasets or inputs - -## Core Refactors - -The architecture refactor introduces a layered system with unified interfaces and multiple execution backends: - -```{mermaid} -graph LR - subgraph "User Layer" - P[Pipeline] - S1[ProcessingStage X→Y] - S2[ProcessingStage Y→Z] - S3[ProcessingStage Z→W] - R[Resources
CPU/GPU/NVDEC/NVENC] - end - - subgraph "Orchestration Layer" - BE[BaseExecutor Interface] - end - - subgraph "Backend Layer" - XE[XennaExecutor] - RAP[RayActorPoolExecutor] - RDE[RayDataExecutor] - end - - subgraph "Adaptation Layer" - XA[Xenna Adapter] - RAPA[Ray Actor Adapter] - RDA[Ray Data Adapter] - end - - subgraph "Execution Layer" - X[Cosmos-Xenna
Streaming/Batch] - RAY1[Ray Actor Pool
Load Balancing] - RAY2[Ray Data API
Dataset Processing] - end - - P --> S1 - P --> S2 - P --> S3 - S1 -.-> R - S2 -.-> R - S3 -.-> R - - P --> BE - BE --> XE - BE --> RAP - BE --> RDE - - XE --> XA - RAP --> RAPA - RDE --> RDA - - XA --> X - RAPA --> RAY1 - RDA --> RAY2 - - style P fill:#E6F3FF - style BE fill:#F0F8FF +### Benchmarking Infrastructure + +New comprehensive benchmarking framework for performance monitoring and optimization: + +- **End-to-End Pipeline Benchmarking**: Automated benchmarks for all curation modalities (text, image, video, audio) +- **Performance Tracking**: Integration with MLflow for metrics tracking and Slack for notifications +- **Nightly Benchmarks**: Continuous performance monitoring across: + - Text pipelines: exact deduplication, fuzzy deduplication, semantic deduplication, score filters, modifiers + - Image curation workflows with DALI-based processing + - Video processing pipelines with scene detection and semantic deduplication + - Audio ASR inference and quality assessment +- **Grafana Dashboards**: Real-time monitoring of pipeline performance and resource utilization + +### Ray Actor Pool Executor (Experimental) + +New execution backend offering an alternative to Xenna for distributed processing: + +- **RayActorPoolExecutor**: Experimental executor with load balancing and progress tracking +- **Progress Bars**: Visual feedback for long-running actor pool operations +- **Flexible Resource Allocation**: Better control over worker distribution and task scheduling + +Learn more in the [Execution Backends documentation](../../reference/infrastructure/execution-backends.md). + +### Enhanced Embedding Generation + +Expanded embedding support with new model integrations: + +- **vLLM Integration**: High-performance LLM-based embedding generation with automatic batching +- **Sentence Transformers**: Support for popular sentence embedding models +- **Unified API**: Consistent embedding interface across text, image, and video modalities + +### YAML Configuration Support + +Declarative pipeline configuration for text curation workflows: + +- **YAML-Based Pipelines**: Define entire curation pipelines in YAML configuration files +- **Pre-Built Configurations**: Ready-to-use configs for common workflows: + - Code filtering, exact/fuzzy/semantic deduplication + - Heuristic filtering (English and non-English) + - FastText language identification +- **Reproducible Workflows**: Version-controlled pipeline definitions for consistent results + +Example: +```bash +python -m nemo_curator.config.run --config_file heuristic_filter_english_pipeline.yaml ``` -### Pipelines +### Workflow Results API + +New API for tracking and analyzing pipeline execution: + +- **WorkflowRunResult**: Structured results object capturing execution metrics +- **Performance Metrics**: Automatic tracking of processing time, throughput, and resource usage +- **Better Debugging**: Detailed logs and error reporting for failed stages + +## Improvements from 25.09 + +### Video Curation -- **New Pipeline API**: Ray-based pipeline execution with `BaseExecutor` interface -- **Multiple backends**: Support for [Xenna, Ray Actor Pool, and Ray Data execution backends](../../reference/infrastructure/execution-backends.md) -- **Resource specification**: Configurable CPU and GPU memory requirements per stage -- **Stage composition**: Improved stage validation and execution orchestration +- **Model Updates**: Removed InternVideo2 dependency; updated to more performant alternatives +- **vLLM 0.14.1**: Upgraded for better video captioning compatibility and performance +- **FFmpeg 8.0.1**: Latest FFmpeg with improved codec support and performance +- **Enhanced Tutorials**: Improved video processing examples with real-world scenarios -### Stages +### Audio Curation -- **ProcessingStage redesign**: Generic `ProcessingStage[X, Y]` base class with type safety -- **Resource requirements**: Built-in resource specification for CPU and GPU memory -- **Backend adapters**: Stage adaptation layer for different Ray orchestration systems -- **Input/output validation**: Enhanced type checking and data validation +- **Enhanced Documentation**: Comprehensive ASR inference and quality assessment guides +- **Improved WER Filtering**: Better guidance for Word Error Rate filtering thresholds +- **Manifest Handling**: More robust JSONL manifest processing for large audio datasets -## Tutorials +### Image Curation -- **Text tutorials**: Updated all [text curation tutorials](https://github.com/NVIDIA-NeMo/Curator/tree/main/tutorials/text) to use new Ray-based API -- **Image tutorials**: Migrated [image processing tutorials](https://github.com/NVIDIA-NeMo/Curator/tree/main/tutorials/image) to unified backend -- **Audio tutorials**: New [audio curation tutorials](https://github.com/NVIDIA-NeMo/Curator/tree/main/tutorials/audio) -- **Video tutorials**: New [video processing tutorials](https://github.com/NVIDIA-NeMo/Curator/tree/main/tutorials/video) +- **Optimized Batch Sizes**: Reduced default batch sizes for better CPU memory usage (batch_size=50, num_threads=4) +- **Memory Guidance**: Added troubleshooting documentation for out-of-memory errors +- **Tutorial Improvements**: Updated examples optimized for typical GPU configurations -For all tutorial content, refer to the [tutorials directory](https://github.com/NVIDIA-NeMo/Curator/tree/main/tutorials) in the NeMo Curator GitHub repository. +### Text Curation -## Known Limitations +- **ID Field Standardization**: Unified ID naming conventions across all deduplication workflows +- **Performance Optimizations**: Fused document iterate and extract stages for reduced overhead +- **Better Memory Management**: Improved handling of large-scale semantic deduplication +- **Small Cluster Warnings**: Automatic warnings when n_clusters is too small for effective deduplication +- **FilePartitioning Improvements**: One worker per partition for better parallelization -> (Pending Refactor in Future Release) +### Deduplication Enhancements -### Generation +- **Cloud Storage Support**: Fixed ParquetReader/Writer and pairwise I/O for S3, GCS, and Azure Blob +- **Non-Blocking ID Generation**: Improved ID generator performance for large datasets +- **Empty Batch Handling**: Better error handling for filters processing empty data batches -- **Synthetic data generation**: Synthetic text generation features are being refactored for Ray compatibility -- **Hard negative mining**: Retrieval-based data generation workflows under development +## Dependency Updates -### PII +- **Transformers**: Pinned to 4.55.2 for stability and compatibility +- **vLLM**: Updated to 0.14.1 with video pipeline compatibility fixes +- **FFmpeg**: Upgraded to 8.0.1 for enhanced multimedia processing +- **Security Patches**: + - Addressed CVEs in aiohttp, urllib3, python-multipart, setuptools + - Removed vulnerable thirdparty aiohttp file from Ray + - Updated to secure dependency versions -- **PII processing**: Personal Identifiable Information removal tools are being updated for Ray backend -- **Privacy workflows**: Enhanced privacy-preserving data curation capabilities in development +## Bug Fixes -### Blending & Shuffling +- Fixed fasttext predict call compatibility with numpy>2 +- Fixed broken NeMo Framework documentation links +- Fixed MegatronTokenizerWriter to download only necessary tokenizer files +- Fixed ID generator blocking issues for large-scale processing +- Fixed vLLM API compatibility with video captioning pipeline +- Fixed Gliner tutorial examples and SDG workflow bugs +- Improved semantic deduplication unit test reliability -- **Data blending**: Multi-source dataset blending functionality being refactored -- **Dataset shuffling**: Large-scale data shuffling operations under development +## Infrastructure & Developer Experience -## Docs Refactor +- **Secrets Detection**: Automated secret scanning in CI/CD workflows +- **Dependabot Integration**: Automatic dependency update pull requests +- **Enhanced Install Tests**: Comprehensive installation validation across environments +- **AWS Runner Support**: CI/CD execution on AWS infrastructure +- **Docker Optimization**: Improved layer caching and build times with uv +- **Code Linting**: Standardized code quality checks with markdownlint and pre-commit hooks +- **Cursor Rules**: Development guidelines and patterns for IDE assistance + +## Breaking Changes + +- **InternVideo2 Removed**: Video pipelines must use alternative embedding models (Cosmos-Embed1) +- **ID Field Standardization**: Custom deduplication workflows may need updates to use standardized ID field names + +## Documentation Improvements + +- **Heuristic Filter Guide**: Comprehensive documentation for language-specific filtering strategies +- **Distributed Classifier**: Enhanced GPU memory optimization guidance with length-based sequence sorting +- **Installation Guide**: Clearer instructions with troubleshooting for common issues +- **Memory Management**: New guidance for handling CPU/GPU memory constraints +- **AWS Integration**: Updated tutorials with correct AWS credentials setup -- **Local preview capability**: Improved documentation build system with local preview support -- **Modality-specific guides**: Comprehensive documentation for each supported modality ([text](../../curate-text/index.md), [image](../../curate-images/index.md), [audio](../../curate-audio/index.md), [video](../../curate-video/index.md)) -- **API reference**: Complete [API documentation](../../apidocs/index.rst) with type annotations and examples --- ## What's Next -The next release will focus on code curation and math curation. +Future releases will focus on: + +- **Code Curation**: Specialized pipelines for curating code datasets +- **Math Curation**: Mathematical reasoning and problem-solving data curation +- **Generation Features**: Completing the Ray refactor for synthetic data generation +- **PII Processing**: Enhanced privacy-preserving data curation with Ray backend +- **Blending & Shuffling**: Large-scale multi-source dataset blending and shuffling operations ```{toctree} :hidden: From 5798cd076f2d19d07f47a47ae854a91df153e1a7 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 11 Feb 2026 12:49:33 -0500 Subject: [PATCH 10/13] feedback Signed-off-by: Lawrence Lane --- docs/about/release-notes/index.md | 3 +-- docs/curate-video/process-data/dedup.md | 2 +- docs/get-started/image.md | 8 ++++++++ docs/reference/infrastructure/container-environments.md | 2 +- 4 files changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/about/release-notes/index.md b/docs/about/release-notes/index.md index 58562d9838..03d38b51a9 100644 --- a/docs/about/release-notes/index.md +++ b/docs/about/release-notes/index.md @@ -116,7 +116,7 @@ New API for tracking and analyzing pipeline execution: ## Bug Fixes -- Fixed fasttext predict call compatibility with numpy>2 +- Fixed fasttext predict call compatibility with numpy>2 - Fixed broken NeMo Framework documentation links - Fixed MegatronTokenizerWriter to download only necessary tokenizer files - Fixed ID generator blocking issues for large-scale processing @@ -147,7 +147,6 @@ New API for tracking and analyzing pipeline execution: - **Memory Management**: New guidance for handling CPU/GPU memory constraints - **AWS Integration**: Updated tutorials with correct AWS credentials setup - --- ## What's Next diff --git a/docs/curate-video/process-data/dedup.md b/docs/curate-video/process-data/dedup.md index 6cba46e520..d2d0507476 100644 --- a/docs/curate-video/process-data/dedup.md +++ b/docs/curate-video/process-data/dedup.md @@ -56,7 +56,7 @@ workflow = SemanticDeduplicationWorkflow( n_clusters=1000, id_field="id", embedding_field="embedding", - embedding_dim=512, # 512 for InternVideo2, varies for Cosmos-Embed1 + embedding_dim=768, # Embedding dimension (768 for Cosmos-Embed1, varies by model) input_filetype="parquet", eps=0.1, # Similarity threshold: cosine_sim >= 1.0 - eps identifies duplicates ranking_strategy=RankingStrategy.metadata_based( diff --git a/docs/get-started/image.md b/docs/get-started/image.md index 6f21a506c9..a739479e25 100644 --- a/docs/get-started/image.md +++ b/docs/get-started/image.md @@ -120,8 +120,16 @@ Here's a simple example to get started with NeMo Curator's image curation pipeli Image loading and decoding happens in CPU memory before GPU processing. If you encounter out-of-memory errors during the `ImageReaderStage`, reduce: - `batch_size`: Number of images per batch (reduce to 32-50 for systems with limited RAM) - `num_threads`: Parallel decoding threads (reduce to 4 for systems with limited RAM) +- `num_cpus`: Ray Client CPU allocation (reduce to 8-16 for systems with limited RAM) The example below uses conservative defaults suitable for most systems. For high-memory systems, you can increase these values for better performance. + +To configure Ray with limited CPU resources: +```python +from nemo_curator.core.client import RayClient +ray_client = RayClient(num_cpus=8) # Adjust based on available CPU cores +ray_client.start() +``` ::: ```python diff --git a/docs/reference/infrastructure/container-environments.md b/docs/reference/infrastructure/container-environments.md index bb05df1613..a13e7fccaf 100644 --- a/docs/reference/infrastructure/container-environments.md +++ b/docs/reference/infrastructure/container-environments.md @@ -37,7 +37,7 @@ NeMo Curator provides official Docker containers with all dependencies pre-insta The primary container includes comprehensive support for all curation modalities: -**Container registry:** `nvcr.io/nvidia/nemo-curator:26.02` +**Container registry:** `nvcr.io/nvidia/nemo-curator:{{ container_version }}` **Supported modalities:** - ✅ Text curation (CPU/GPU) From e5ca35d380044e8f32078f29b233cb32bbd5ea3f Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 11 Feb 2026 12:51:57 -0500 Subject: [PATCH 11/13] remove more internvid content Signed-off-by: Lawrence Lane --- docs/curate-video/process-data/dedup.md | 6 +++--- docs/curate-video/process-data/index.md | 2 +- docs/curate-video/save-export.md | 9 ++++----- docs/curate-video/tutorials/split-dedup.md | 12 ++++++------ 4 files changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/curate-video/process-data/dedup.md b/docs/curate-video/process-data/dedup.md index d2d0507476..f343425cb9 100644 --- a/docs/curate-video/process-data/dedup.md +++ b/docs/curate-video/process-data/dedup.md @@ -15,7 +15,7 @@ modality: "video-only" Use clip-level embeddings to identify near-duplicate video clips so your dataset remains compact, diverse, and efficient to train on. ## Before You Start -- Make sure you have embeddings which are written by the [`ClipWriterStage`](video-save-export) under `iv2_embd_parquet/` or `ce1_embd_parquet/`. For a runnable workflow, refer to the [Split and Remove Duplicates Workflow](video-tutorials-split-dedup). The embeddings must be in parquet files containing the columns `id` and `embedding`. +- Make sure you have embeddings which are written by the [`ClipWriterStage`](video-save-export) under `ce1_embd_parquet/`. For a runnable workflow, refer to the [Split and Remove Duplicates Workflow](video-tutorials-split-dedup). The embeddings must be in parquet files containing the columns `id` and `embedding`. - Verify local paths or configure S3-compatible credentials. Provide `storage_options` in read/write keyword arguments when reading or writing cloud paths. @@ -24,7 +24,7 @@ Use clip-level embeddings to identify near-duplicate video clips so your dataset Duplicate identification operates on clip-level embeddings produced during processing: 1. **Inputs** - - Parquet batches from `ClipWriterStage` under `iv2_embd_parquet/` or `ce1_embd_parquet/` + - Parquet batches from `ClipWriterStage` under `ce1_embd_parquet/` - Columns: `id`, `embedding` 2. **Outputs** @@ -50,7 +50,7 @@ from nemo_curator.stages.deduplication.semantic.ranking import RankingStrategy from nemo_curator.backends.xenna import XennaExecutor workflow = SemanticDeduplicationWorkflow( - input_path="/path/to/embeddings/", # e.g., iv2_embd_parquet/ or ce1_embd_parquet/ + input_path="/path/to/embeddings/", # e.g., ce1_embd_parquet/ output_path="/path/to/duplicates/", cache_path="/path/to/cache/", # Optional: defaults to output_path n_clusters=1000, diff --git a/docs/curate-video/process-data/index.md b/docs/curate-video/process-data/index.md index 35f3844129..6e49781525 100644 --- a/docs/curate-video/process-data/index.md +++ b/docs/curate-video/process-data/index.md @@ -124,7 +124,7 @@ pipeline.add_stage( ) ``` -Path helpers are available to resolve common locations (such as `clips/`, `filtered_clips/`, `previews/`, `metas/v0/`, and `iv2_embd_parquet/`). +Path helpers are available to resolve common locations (such as `clips/`, `filtered_clips/`, `previews/`, `metas/v0/`, and `ce1_embd_parquet/`). ```{toctree} :maxdepth: 2 diff --git a/docs/curate-video/save-export.md b/docs/curate-video/save-export.md index ac0d72ef69..f120055400 100644 --- a/docs/curate-video/save-export.md +++ b/docs/curate-video/save-export.md @@ -95,8 +95,8 @@ The writer produces these directories under `output_path`: - `filtered_clips/`: Media for filtered-out clips. - `previews/`: Preview images (`.webp`). - `metas/v0/`: Per-clip metadata (`.json`). -- `iv2_embd/`, `ce1_embd/`: Per-clip embeddings (`.pickle`). -- `iv2_embd_parquet/`, `ce1_embd_parquet/`: Parquet batches with columns `id` and `embedding`. +- `ce1_embd/`: Per-clip embeddings (`.pickle`). +- `ce1_embd_parquet/`: Parquet batches with columns `id` and `embedding`. - `processed_videos/`, `processed_clip_chunks/`: Video-level metadata and per-chunk statistics. ### Per-Clip Metadata @@ -132,8 +132,8 @@ Each clip writes a JSON file under `metas/v0/` with clip- and window-level field ### Embeddings and Parquet outputs -- When embeddings exist, the stage writes per-clip `.pickle` files under `iv2_embd/` or `ce1_embd/`. -- The stage also batches embeddings per clip chunk into Parquet files under `iv2_embd_parquet/` or `ce1_embd_parquet/` with columns `id` and `embedding` and writes those files to disk. +- When embeddings exist, the stage writes per-clip `.pickle` files under `ce1_embd/`. +- The stage also batches embeddings per clip chunk into Parquet files under `ce1_embd_parquet/` with columns `id` and `embedding` and writes those files to disk. ## Helpers @@ -150,7 +150,6 @@ clips_dir = ClipWriterStage.get_output_path_clips(OUT) filtered_clips_dir = ClipWriterStage.get_output_path_clips(OUT, filtered=True) previews_dir = ClipWriterStage.get_output_path_previews(OUT) metas_dir = ClipWriterStage.get_output_path_metas(OUT, "v0") -iv2_parquet_dir = ClipWriterStage.get_output_path_iv2_embd_parquet(OUT) ce1_parquet_dir = ClipWriterStage.get_output_path_ce1_embd_parquet(OUT) processed_videos_dir = ClipWriterStage.get_output_path_processed_videos(OUT) processed_chunks_dir = ClipWriterStage.get_output_path_processed_clip_chunks(OUT) diff --git a/docs/curate-video/tutorials/split-dedup.md b/docs/curate-video/tutorials/split-dedup.md index 8def49bf36..580043a290 100644 --- a/docs/curate-video/tutorials/split-dedup.md +++ b/docs/curate-video/tutorials/split-dedup.md @@ -49,7 +49,7 @@ Writer-related flags you can add: --dry-run # Write nothing; validate only ``` -The pipeline writes embeddings under `$OUT_DIR/iv2_embd_parquet/` (or `ce1_embd_parquet/` if you use Cosmos-Embed1). +The pipeline writes embeddings under `$OUT_DIR/ce1_embd_parquet/` when using Cosmos-Embed1. ### Embedding Format Example @@ -64,7 +64,7 @@ The pipeline writes embeddings to Parquet with two columns: ```text $OUT_DIR/ - iv2_embd_parquet/ + ce1_embd_parquet/ 1a2b3c4d-....parquet 5e6f7g8h-....parquet ``` @@ -93,7 +93,7 @@ embedding: list # length = 768 for Cosmos-Embed1 ```python import pyarrow.parquet as pq -table = pq.read_table(f"{OUT_DIR}/iv2_embd_parquet") +table = pq.read_table(f"{OUT_DIR}/ce1_embd_parquet") df = table.to_pandas() print(df.head()) # columns: id, embedding (list[float]) ``` @@ -113,7 +113,7 @@ from nemo_curator.pipeline import Pipeline from nemo_curator.stages.deduplication.semantic.kmeans import KMeansStage from nemo_curator.stages.deduplication.semantic.pairwise import PairwiseStage -INPUT_PARQUET = f"{OUT_DIR}/iv2_embd_parquet" # or s3://... +INPUT_PARQUET = f"{OUT_DIR}/ce1_embd_parquet" # or s3://... OUTPUT_DIR = f"{OUT_DIR}/semantic_dedup" pipe = Pipeline(name="video_semantic_dedup", description="K-means + pairwise duplicate removal") @@ -175,7 +175,7 @@ Video-specific pointers: - Use `ClipWriterStage` path helpers to locate outputs: `nemo_curator/stages/video/io/clip_writer.py`. - Processed videos: `get_output_path_processed_videos(OUT_DIR)` - Clip chunks and previews: `get_output_path_processed_clip_chunks(OUT_DIR)`, `get_output_path_previews(OUT_DIR)` - - Embeddings parquet: `${OUT_DIR}/iv2_embd_parquet` (or `${OUT_DIR}/ce1_embd_parquet`) + - Embeddings parquet: `${OUT_DIR}/ce1_embd_parquet` (or `${OUT_DIR}/ce1_embd_parquet`) ### Example Export @@ -188,7 +188,7 @@ from glob import glob OUT_DIR = os.environ["OUT_DIR"] clips_dir = os.path.join(OUT_DIR, "clips") # adjust if filtering path used -meta_parquet = os.path.join(OUT_DIR, "iv2_embd_parquet") +meta_parquet = os.path.join(OUT_DIR, "ce1_embd_parquet") def iter_clips(path): for p in glob(os.path.join(path, "**", "*.mp4"), recursive=True): From 64d0424380f01ad38ae9c0411227c93ed5fa5b3b Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 11 Feb 2026 13:22:08 -0500 Subject: [PATCH 12/13] release note fix Signed-off-by: Lawrence Lane --- docs/about/release-notes/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/about/release-notes/index.md b/docs/about/release-notes/index.md index 03d38b51a9..a660ad933f 100644 --- a/docs/about/release-notes/index.md +++ b/docs/about/release-notes/index.md @@ -27,13 +27,13 @@ New comprehensive benchmarking framework for performance monitoring and optimiza - Audio ASR inference and quality assessment - **Grafana Dashboards**: Real-time monitoring of pipeline performance and resource utilization -### Ray Actor Pool Executor (Experimental) +### Ray Actor Pool Executor Improvements -New execution backend offering an alternative to Xenna for distributed processing: +Enhanced features for the experimental Ray Actor Pool execution backend: -- **RayActorPoolExecutor**: Experimental executor with load balancing and progress tracking -- **Progress Bars**: Visual feedback for long-running actor pool operations -- **Flexible Resource Allocation**: Better control over worker distribution and task scheduling +- **Progress Bars**: New visual feedback for long-running actor pool operations, making it easier to monitor pipeline execution +- **Improved Load Balancing**: Better worker distribution and task scheduling +- **Enhanced Stability**: Continued refinements to the experimental executor Learn more in the [Execution Backends documentation](../../reference/infrastructure/execution-backends.md). From 923e5687069af6ddb167c921fde4032c56e38b7f Mon Sep 17 00:00:00 2001 From: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Date: Wed, 11 Feb 2026 10:33:03 -0800 Subject: [PATCH 13/13] Update docs/curate-video/tutorials/split-dedup.md Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> --- docs/curate-video/tutorials/split-dedup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/curate-video/tutorials/split-dedup.md b/docs/curate-video/tutorials/split-dedup.md index 580043a290..2147d0bddd 100644 --- a/docs/curate-video/tutorials/split-dedup.md +++ b/docs/curate-video/tutorials/split-dedup.md @@ -175,7 +175,7 @@ Video-specific pointers: - Use `ClipWriterStage` path helpers to locate outputs: `nemo_curator/stages/video/io/clip_writer.py`. - Processed videos: `get_output_path_processed_videos(OUT_DIR)` - Clip chunks and previews: `get_output_path_processed_clip_chunks(OUT_DIR)`, `get_output_path_previews(OUT_DIR)` - - Embeddings parquet: `${OUT_DIR}/ce1_embd_parquet` (or `${OUT_DIR}/ce1_embd_parquet`) + - Embeddings parquet: `${OUT_DIR}/ce1_embd_parquet` ### Example Export