docs: vdr feedback

docs/admin/installation.md

@@ -18,14 +18,15 @@ This guide covers installing NeMo Curator with support for **all modalities** an
 
 ### System Requirements
 
-For comprehensive system requirements and production deployment specifications, see [Production Deployment Requirements](deployment/requirements.md).
+For comprehensive system requirements and production deployment specifications, refer to [Production Deployment Requirements](deployment/requirements.md).
 
 **Quick Start Requirements:**
 
 - **OS**: Ubuntu 24.04/22.04/20.04 (recommended)
 - **Python**: 3.10, 3.11, or 3.12
 - **Memory**: 16GB+ RAM for basic text processing
 - **GPU** (optional): NVIDIA GPU with 16GB+ VRAM for acceleration
+- **CUDA 12** (required for `audio_cuda12`, `video_cuda12`, `image_cuda12`, and `text_cuda12` extras)
 
 ### Development vs Production
 
@@ -41,9 +42,13 @@ For comprehensive system requirements and production deployment specifications,
 
 Choose one of the following installation methods based on your needs:
 
+:::{tip}
+**Docker is the recommended installation method** for video and audio workflows. The NeMo Curator container includes FFmpeg (with NVENC support) pre-configured, avoiding manual dependency setup. Refer to the [Container Installation](#container-installation) tab below.
+:::
+
 ::::{tab-set}
 
-:::{tab-item} PyPI Installation (Recommended)
+:::{tab-item} PyPI Installation
 
 Install NeMo Curator from the Python Package Index using `uv` for proper dependency resolution.
 
@@ -89,9 +94,9 @@ uv sync --all-extras --all-groups
 
 :::
 
-:::{tab-item} Container Installation
+:::{tab-item} Container Installation (Recommended for Video/Audio)
 
-NeMo Curator is available as a standalone container on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator. The container includes NeMo Curator with all dependencies pre-installed. You can run it with:
+NeMo Curator is available as a standalone container on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator. The container includes NeMo Curator with all dependencies pre-installed, including FFmpeg with NVENC support.
 
 ```bash
 # Pull the container from NGC
@@ -101,6 +106,14 @@ docker pull nvcr.io/nvidia/nemo-curator:{{ container_version }}
 docker run --gpus all -it --rm nvcr.io/nvidia/nemo-curator:{{ container_version }}
 ```
 
+```{important}
+After entering the container, activate the virtual environment before running any NeMo Curator commands:
+
+    source /opt/venv/env.sh
+
+The container uses a virtual environment at `/opt/venv`. If you see `No module named nemo_curator`, the environment has not been activated.
+```
+
 Alternatively, you can build the NeMo Curator container locally using the provided Dockerfile:
 
 ```bash
@@ -115,7 +128,7 @@ docker run --gpus all -it --rm nemo-curator:latest
 
 **Benefits:**
 
-- Pre-configured environment with all dependencies
+- Pre-configured environment with all dependencies (FFmpeg, CUDA libraries)
 - Consistent runtime across different systems
 - Ideal for production deployments
 
@@ -157,6 +170,10 @@ If encoders are missing, reinstall `FFmpeg` with the required options or use the
 :::
 ::::
 
+```{note}
+**FFmpeg build requires CUDA toolkit (nvcc):** If you encounter `ERROR: failed checking for nvcc` during FFmpeg installation, ensure that the CUDA toolkit is installed and `nvcc` is available on your `PATH`. You can verify with `nvcc --version`. If using the NeMo Curator container, FFmpeg is pre-installed with NVENC support.
+```
+
 ---
 
 ## Package Extras

docs/broken_links_false_positives.json

@@ -0,0 +1 @@
+{"filename": "get-started/text.md", "lineno": 119, "status": "broken", "code": 0, "uri": "https://huggingface.co/settings/tokens", "info": "unauthorized"}

docs/curate-text/process-data/deduplication/fuzzy.md

@@ -34,6 +34,10 @@ Ideal for detecting documents with minor differences such as formatting changes,
 - Ray cluster with GPU support (required for distributed processing)
 - Stable document identifiers for removal (either existing IDs or IDs generated by the workflow and removal stages)
 
+```{note}
+**Running in Docker**: When running fuzzy deduplication inside the NeMo Curator container, ensure the container is started with `--gpus all` so that Ray workers can access the GPU. Without GPU access, you may see `CUDARuntimeError` or `AttributeError: 'CUDARuntimeError' object has no attribute 'msg'`. Also activate the virtual environment with `source /opt/venv/env.sh` after entering the container.
+```
+
 ## Quick Start
 
 Get started with fuzzy deduplication using the following example of identifying duplicates, then remove them:

docs/curate-text/process-data/deduplication/semdedup.md

@@ -42,6 +42,10 @@ Based on [SemDeDup: Data-efficient learning at web-scale through semantic dedupl
 - GPU acceleration (required for embedding generation and clustering)
 - Stable document identifiers for removal (either existing IDs or IDs managed by the workflow and removal stages)
 
+```{note}
+**Running in Docker**: When running semantic deduplication inside the NeMo Curator container, ensure the container is started with `--gpus all` so that CUDA GPUs are available. Without this flag, you will see `RuntimeError: No CUDA GPUs are available`. Also activate the virtual environment with `source /opt/venv/env.sh` after entering the container.
+```
+
 ## Quick Start
 
 Get started with semantic deduplication using the following example of identifying duplicates, then remove them in one step:

docs/curate-text/process-data/quality-assessment/distributed-classifier.md

@@ -20,7 +20,7 @@ The distributed data classification in NeMo Curator works by:
 
 1. **Parallel Processing**: Chunking datasets across multiple computing nodes and GPUs to accelerate classification
 2. **Pre-trained Models**: Using specialized models for different classification tasks
-3. **Batched Inference**: Optimizing throughput with intelligent batching via CrossFit integration
+3. **Batched Inference**: Optimizing throughput with intelligent batching
 4. **Consistent API**: Providing a unified interface through the `DistributedDataClassifier` base class
 
 The `DistributedDataClassifier` is designed to run on GPU clusters with minimal code changes regardless of which specific classifier you're using. All classifiers support filtering based on classification results and storing prediction scores as metadata.
@@ -29,6 +29,16 @@ The `DistributedDataClassifier` is designed to run on GPU clusters with minimal
 Distributed classification requires GPU acceleration and is not supported for CPU-only processing. As long as GPU resources are available and NeMo Curator is correctly installed, GPU acceleration is handled automatically.
 :::
 
+```{tip}
+**Running the tutorial notebooks**: The classification tutorial notebooks require the `text_cuda12` or `all` installation extra to include all relevant dependencies. If you encounter `ModuleNotFoundError`, reinstall with the appropriate extra:
+
+    uv pip install "nemo-curator[text_cuda12]"
+
+When using classifiers that download from Hugging Face (such as Aegis and InstructionDataGuard), set your `HF_TOKEN` environment variable to avoid rate limiting:
+
+    export HF_TOKEN="your_token_here"
+```
+
 ---
 
 ## Usage
@@ -39,16 +49,16 @@ NVIDIA NeMo Curator provides a base class `DistributedDataClassifier` that can b
 
 | Classifier | Purpose | Model Location | Key Parameters | Requirements |
 |---|---|---|---|---|
-| DomainClassifier | Categorize English text by domain | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) | `filter_by`, `text_field` | None |
-| MultilingualDomainClassifier | Categorize text in 52 languages by domain | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) | `filter_by`, `text_field` | None |
-| QualityClassifier | Assess document quality | [nvidia/quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | `filter_by`, `text_field` | None |
-| AegisClassifier | Detect unsafe content | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) | `aegis_variant`, `filter_by` | HuggingFace token |
-| InstructionDataGuardClassifier | Detect poisoning attacks | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) | `text_field`, `label_field` | HuggingFace token |
-| FineWebEduClassifier | Score educational value | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | `label_field`, `int_field` | None |
-| FineWebMixtralEduClassifier | Score educational value (Mixtral annotations) | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None |
-| FineWebNemotronEduClassifier | Score educational value (Nemotron annotations) | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None |
-| ContentTypeClassifier | Categorize by speech type | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) | `filter_by`, `text_field` | None |
-| PromptTaskComplexityClassifier | Classify prompt tasks and complexity | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `text_field` | None |
+| DomainClassifier | Assigns one of 26 domain labels (such as "Sports," "Science," "News") to English text | [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier) | `filter_by`, `text_field` | None |
+| MultilingualDomainClassifier | Assigns domain labels to text in 52 languages; same labels as DomainClassifier | [nvidia/multilingual-domain-classifier](https://huggingface.co/nvidia/multilingual-domain-classifier) | `filter_by`, `text_field` | None |
+| QualityClassifier | Rates document quality as "Low," "Medium," or "High" using a DeBERTa model | [nvidia/quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | `filter_by`, `text_field` | None |
+| AegisClassifier | Detects unsafe content across 13 risk categories (violence, hate speech, and others) using LlamaGuard | [nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0](https://huggingface.co/nvidia/Aegis-AI-Content-Safety-LlamaGuard-Defensive-1.0) | `aegis_variant`, `filter_by` | HuggingFace token |
+| InstructionDataGuardClassifier | Identifies LLM poisoning attacks in instruction-response pairs | [nvidia/instruction-data-guard](https://huggingface.co/nvidia/instruction-data-guard) | `text_field`, `label_field` | HuggingFace token |
+| FineWebEduClassifier | Scores educational value from 0 to 5 (0=spam, 5=scholarly) for training data selection | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | `label_field`, `int_field` | None |
+| FineWebMixtralEduClassifier | Scores educational value from 0 to 5 using Mixtral 8x22B annotation data | [nvidia/nemocurator-fineweb-mixtral-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-mixtral-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None |
+| FineWebNemotronEduClassifier | Scores educational value from 0 to 5 using Nemotron-4-340B annotation data | [nvidia/nemocurator-fineweb-nemotron-4-edu-classifier](https://huggingface.co/nvidia/nemocurator-fineweb-nemotron-4-edu-classifier) | `label_field`, `int_field`, `model_inference_batch_size=1024` | None |
+| ContentTypeClassifier | Categorizes text into 11 speech types (such as "Blogs," "News," "Academic") | [nvidia/content-type-classifier-deberta](https://huggingface.co/nvidia/content-type-classifier-deberta) | `filter_by`, `text_field` | None |
+| PromptTaskComplexityClassifier | Labels prompts by task type (such as QA and summarization) and complexity dimensions | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `text_field` | None |
 
 ### Domain Classifier
 
@@ -365,6 +375,10 @@ pipeline.add_stage(writer)
 results = pipeline.run()  # Uses XennaExecutor by default
 ```
 
+## Custom Model Integration
+
+You can integrate your own classification models by extending `DistributedDataClassifier`. Refer to the [Text Classifiers README](https://github.com/NVIDIA-NeMo/Curator/tree/main/nemo_curator/stages/text/classifiers#text-classifiers) for implementation details and examples.
+
 ## Performance Optimization
 
 NVIDIA NeMo Curator's distributed classifiers are optimized for high-throughput processing through several key features:

docs/curate-video/process-data/captions-preview.md

@@ -68,7 +68,7 @@ pipe.run()
 :::{tab-item} Script Flags
 
 ```bash
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   ... \
   --generate-captions \
   --captioning-algorithm qwen \

docs/curate-video/process-data/clipping.md

@@ -84,15 +84,15 @@ pipe.run()
 
 ```bash
 # Fixed stride
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   ... \
   --splitting-algorithm fixed_stride \
   --fixed-stride-split-duration 10.0 \
   --fixed-stride-min-clip-length-s 2.0 \
   --limit-clips 0
 
 # TransNetV2
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   ... \
   --splitting-algorithm transnetv2 \
   --transnetv2-frame-decoder-mode pynvc \

docs/curate-video/process-data/dedup.md

@@ -15,7 +15,7 @@ modality: "video-only"
 Use clip-level embeddings to identify near-duplicate video clips so your dataset remains compact, diverse, and efficient to train on.
 
 ## Before You Start
-- Make sure you have embeddings which are written by the [`ClipWriterStage`](video-save-export) under `iv2_embd_parquet/` or `ce1_embd_parquet/`. For a runnable workflow, refer to the [Split and Remove Duplicates Workflow](video-tutorials-split-dedup). The embeddings must be in parquet files containing the columns `id` and `embedding`.
+- Make sure you have embeddings which are written by the [`ClipWriterStage`](video-save-export) under `ce1_embd_parquet/`. For a runnable workflow, refer to the [Split and Remove Duplicates Workflow](video-tutorials-split-dedup). The embeddings must be in parquet files containing the columns `id` and `embedding`.
 - Verify local paths or configure S3-compatible credentials. Provide `storage_options` in read/write keyword arguments when reading or writing cloud paths.
 
 
@@ -24,7 +24,7 @@ Use clip-level embeddings to identify near-duplicate video clips so your dataset
 Duplicate identification operates on clip-level embeddings produced during processing:
 
 1. **Inputs**
-   - Parquet batches from `ClipWriterStage` under `iv2_embd_parquet/` or `ce1_embd_parquet/`
+   - Parquet batches from `ClipWriterStage` under `ce1_embd_parquet/`
    - Columns: `id`, `embedding`
 
 2. **Outputs**
@@ -50,13 +50,13 @@ from nemo_curator.stages.deduplication.semantic.ranking import RankingStrategy
 from nemo_curator.backends.xenna import XennaExecutor
 
 workflow = SemanticDeduplicationWorkflow(
-    input_path="/path/to/embeddings/",  # e.g., iv2_embd_parquet/ or ce1_embd_parquet/
+    input_path="/path/to/embeddings/",  # e.g., ce1_embd_parquet/
     output_path="/path/to/duplicates/",
     cache_path="/path/to/cache/",  # Optional: defaults to output_path
     n_clusters=1000,
     id_field="id",
     embedding_field="embedding",
-    embedding_dim=512,  # 512 for InternVideo2, varies for Cosmos-Embed1
+    embedding_dim=768,  # Embedding dimension (768 for Cosmos-Embed1, varies by model)
     input_filetype="parquet",
     eps=0.1,  # Similarity threshold: cosine_sim >= 1.0 - eps identifies duplicates
     ranking_strategy=RankingStrategy.metadata_based(

docs/curate-video/process-data/embeddings.md

@@ -65,7 +65,7 @@ pipe.run()
 
 ```bash
 # Cosmos-Embed1 (224p)
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   ... \
   --generate-embeddings \
   --embedding-algorithm cosmos-embed1-224p \

docs/curate-video/process-data/filtering.md

@@ -83,7 +83,7 @@ pipe.run()
 
 ```bash
 # Motion filtering
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   ... \
   --motion-filter enable \
   --motion-decode-target-fps 2.0 \
@@ -95,7 +95,7 @@ python -m nemo_curator.examples.video.video_split_clip_example \
   --motion-score-gpus-per-worker 0.5
 
 # Aesthetic filtering
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   ... \
   --aesthetic-threshold 3.5 \
   --aesthetic-reduction min \

docs/curate-video/process-data/frame-extraction.md

@@ -66,13 +66,13 @@ pipe.run()
 
 ```bash
 # Clip frames implicitly when generating embeddings or aesthetics
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   ... \
   --generate-embeddings \
   --clip-extraction-target-res -1
 
 # Full-video frames for TransNetV2 scene change
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   ... \
   --splitting-algorithm transnetv2 \
   --transnetv2-frame-decoder-mode pynvc

docs/curate-video/process-data/index.md

@@ -124,7 +124,7 @@ pipeline.add_stage(
 )
 ```
 
-Path helpers are available to resolve common locations (such as `clips/`, `filtered_clips/`, `previews/`, `metas/v0/`, and `iv2_embd_parquet/`).
+Path helpers are available to resolve common locations (such as `clips/`, `filtered_clips/`, `previews/`, `metas/v0/`, and `ce1_embd_parquet/`).
 
 ```{toctree}
 :maxdepth: 2

docs/curate-video/save-export.md

@@ -95,8 +95,8 @@ The writer produces these directories under `output_path`:
 - `filtered_clips/`: Media for filtered-out clips.
 - `previews/`: Preview images (`.webp`).
 - `metas/v0/`: Per-clip metadata (`.json`).
-- `iv2_embd/`, `ce1_embd/`: Per-clip embeddings (`.pickle`).
-- `iv2_embd_parquet/`, `ce1_embd_parquet/`: Parquet batches with columns `id` and `embedding`.
+- `ce1_embd/`: Per-clip embeddings (`.pickle`).
+- `ce1_embd_parquet/`: Parquet batches with columns `id` and `embedding`.
 - `processed_videos/`, `processed_clip_chunks/`: Video-level metadata and per-chunk statistics.
 
 ### Per-Clip Metadata
@@ -132,8 +132,8 @@ Each clip writes a JSON file under `metas/v0/` with clip- and window-level field
 
 ### Embeddings and Parquet outputs
 
-- When embeddings exist, the stage writes per-clip `.pickle` files under `iv2_embd/` or `ce1_embd/`.
-- The stage also batches embeddings per clip chunk into Parquet files under `iv2_embd_parquet/` or `ce1_embd_parquet/` with columns `id` and `embedding` and writes those files to disk.
+- When embeddings exist, the stage writes per-clip `.pickle` files under `ce1_embd/`.
+- The stage also batches embeddings per clip chunk into Parquet files under `ce1_embd_parquet/` with columns `id` and `embedding` and writes those files to disk.
 
 ## Helpers
 
@@ -150,7 +150,6 @@ clips_dir = ClipWriterStage.get_output_path_clips(OUT)
 filtered_clips_dir = ClipWriterStage.get_output_path_clips(OUT, filtered=True)
 previews_dir = ClipWriterStage.get_output_path_previews(OUT)
 metas_dir = ClipWriterStage.get_output_path_metas(OUT, "v0")
-iv2_parquet_dir = ClipWriterStage.get_output_path_iv2_embd_parquet(OUT)
 ce1_parquet_dir = ClipWriterStage.get_output_path_ce1_embd_parquet(OUT)
 processed_videos_dir = ClipWriterStage.get_output_path_processed_videos(OUT)
 processed_chunks_dir = ClipWriterStage.get_output_path_processed_clip_chunks(OUT)

docs/curate-video/tutorials/beginner.md

@@ -230,10 +230,10 @@ pipeline.add_stage(
 When using the example pipeline module, configure the writer-related flags:
 
 ```bash
-python -m nemo_curator.examples.video.video_split_clip_example \
+python tutorials/video/getting-started/video_split_clip_example.py \
   --video-dir "$VIDEO_DIR" \
   --model-dir "$MODEL_DIR" \
-  --output-clip-path "$OUT_DIR" \
+  --output-path "$OUT_DIR" \
   --no-upload-clips          # optional: do not write mp4s
   --dry-run                   # optional: write nothing, validate only
   --generate-embeddings      # optional: enable embedding outputs