From 02f85c09cfb7c428e7bd36df78110ee696ce8b6e Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Fri, 3 Apr 2026 11:20:11 -0400 Subject: [PATCH] docs: add fern docs for 26.04 PRs (#1575, #1576, #1160, #1603, #1612, #1652) Add audio preprocessing docs (MonoConversion, SegmentConcatenation, TimestampMapper), band filtering docs, Nemotron VLM video captioning updates, fuzzy dedup LSH memory tuning params, interleaved IO schema changes, and release note entries for all new features, dependency updates, and breaking changes. Signed-off-by: Lawrence Lane Co-Authored-By: Claude Opus 4.6 Signed-off-by: Lawrence Lane --- fern/versions/v26.04.yml | 9 ++ .../pages/about/release-notes/index.mdx | 57 ++++++++ .../preprocessing/band-filtering.mdx | 97 +++++++++++++ .../process-data/preprocessing/index.mdx | 127 ++++++++++++++++++ .../process-data/deduplication/fuzzy.mdx | 23 ++++ .../process-data/captions-preview.mdx | 64 +++++++-- 6 files changed, 369 insertions(+), 8 deletions(-) create mode 100644 fern/versions/v26.04/pages/curate-audio/process-data/preprocessing/band-filtering.mdx create mode 100644 fern/versions/v26.04/pages/curate-audio/process-data/preprocessing/index.mdx diff --git a/fern/versions/v26.04.yml b/fern/versions/v26.04.yml index eb4ea19bd3..757a12e5b1 100644 --- a/fern/versions/v26.04.yml +++ b/fern/versions/v26.04.yml @@ -459,6 +459,15 @@ navigation: - page: Format Validation path: ./v26.04/pages/curate-audio/process-data/audio-analysis/format-validation.mdx slug: format-validation + - section: Preprocessing + slug: preprocessing + contents: + - page: Overview + path: ./v26.04/pages/curate-audio/process-data/preprocessing/index.mdx + slug: "" + - page: Band Filtering + path: ./v26.04/pages/curate-audio/process-data/preprocessing/band-filtering.mdx + slug: band-filtering - page: Text Integration path: ./v26.04/pages/curate-audio/process-data/text-integration/index.mdx slug: text-integration diff --git a/fern/versions/v26.04/pages/about/release-notes/index.mdx b/fern/versions/v26.04/pages/about/release-notes/index.mdx index 527e9d888e..dfb3a80a74 100644 --- a/fern/versions/v26.04/pages/about/release-notes/index.mdx +++ b/fern/versions/v26.04/pages/about/release-notes/index.mdx @@ -62,6 +62,54 @@ Standardized return type for all deduplication workflows: - **Structured metadata**: Access per-stage timing (`total_time`, `identification_time`, and others), duplicate counts (`num_duplicates`, `num_duplicates_removed`), and output paths through `result.metadata` - **`TaskPerfUtils` compatibility**: `collect_stage_metrics()` and `aggregate_task_metrics()` now accept `WorkflowRunResult` directly +### Nemotron VLM for Video Captioning (PR #1160) + +Added NemotronH hybrid Mamba-Attention VLM as an alternative to Qwen-VL for video captioning: + +- **`NemotronHVL` model class**: Supports `nemotron` (BF16), `nemotron-fp8`, and `nemotron-nvfp4` quantization variants from HuggingFace (`nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-*`). +- **`CaptionGenerationStage` and `CaptionPreparationStage`**: Updated to accept `model_variant="nemotron"` alongside the existing `"qwen"` option. +- **Automatic model download**: Models download from HuggingFace on first use with pinned revision hashes for reproducibility. + +Learn more in the [Captions and Preview](/curate-video/process-data/captions-preview) documentation. + +### Audio Preprocessing Stages (PR #1575) + +New stages for preparing raw audio before ASR and quality assessment: + +- **`MonoConversionStage`**: Converts multi-channel audio to mono by averaging channels and verifies sample rate compliance. +- **`SegmentConcatenationStage`**: Assembles VAD-segmented audio (from `VADSegmentationStage(nested=True)`) into a single waveform per source file with configurable silence between segments. +- **`TimestampMapperStage`**: Resolves segment positions in the concatenated waveform back to original file timestamps, producing metadata-only output. + +Learn more in the [Audio Preprocessing](/curate-audio/process-data/preprocessing) documentation. + +### Audio Band Filtering (PR #1576) + +New bandwidth classification stage for filtering audio by spectral quality: + +- **`BandFilterStage`**: Classifies audio as `"full_band"` or `"narrow_band"` using a pre-trained model from HuggingFace (`nvidia/nemocurator-speech-bandwidth-filter`), then filters based on the specified band type. +- **GPU support**: Configurable via `Resources(gpus=N)` for GPU-accelerated inference. + +Learn more in the [Band Filtering](/curate-audio/process-data/preprocessing/band-filtering) documentation. + +### Fuzzy Dedup LSH Memory Tuning (PR #1603) + +Added GPU memory management parameters to `FuzzyDeduplicationWorkflow` for the LSH stage: + +- **`lsh_num_output_partitions`**: Control the number of shuffle output partitions (default: auto-selected as closest power of 2 <= input tasks). +- **`lsh_rmm_pool_size`**: Configure the RMM GPU memory pool size (`"auto"` = 90% of free memory, `None` = 50% expandable). +- **`lsh_spill_memory_limit`**: Set the device memory spill threshold (`"auto"` = 80% of RMM pool, `None` = disabled). + +Learn more in the [Fuzzy Deduplication](/curate-text/process-data/deduplication/fuzzy) documentation. + +### Interleaved IO Improvements (PR #1652) + +Schema utilities and base-class enhancements for multimodal readers and writers: + +- **`schema` and `schema_overrides` parameters**: New parameters on `BaseInterleavedReader` for strict schema alignment or partial type overrides on output tables. +- **`WebdatasetReaderStage` renamed to `InterleavedWebdatasetReaderStage`**: Clarifies naming for the interleaved reader stage. +- **`source_id_field` removed**: The `source_id_field` parameter has been removed from `WebdatasetReader` and `InterleavedWebdatasetReaderStage`. +- **Schema utility functions**: `reconcile_schema`, `resolve_schema`, and `align_table` handle type reconciliation for reserved and passthrough columns. + ### Actor Pool Progress Bars (PR #1457) Added tqdm progress bars to `RayActorPoolExecutor` for real-time visibility into task completion during stage processing and shuffle inserts. Progress bars are enabled by default and can be configured with `show_progress` and `progress_interval` parameters. This is particularly useful for long-running deduplication jobs where progress is not otherwise apparent. @@ -72,6 +120,11 @@ Added tqdm progress bars to `RayActorPoolExecutor` for real-time visibility into - **Ray**: Updated to 2.54 - **cryptography**: Bumped from >=46.0.5 to >=46.0.6 to address CVE GHSA-m959-cc7f-wv43 (PR #1682) - **uv**: Added minimum required version (>=0.7.0) to prevent lockfile revision drift +- **nemo-toolkit[asr]**: Bumped to >=2.7.2 to fix CVE-2025-33245 and CVE-2025-33253 (RCE via unsafe deserialization) (PR #1612) +- **xgrammar**: Bumped to >=0.1.32 to fix GHSA-7rgv-gqhr-fxg3 (DoS via uncontrolled recursion) (PR #1612) +- **pyopenssl**: Bumped from 22.0.0 to 26.0.0 (PR #1669) +- **onnx**: Bumped from 1.20.0 to 1.21.0 (PR #1713) +- **flash-attn**: Pinned to >=2.8.3 for NemotronH VLM compatibility (PR #1160) ## Bug Fixes @@ -95,3 +148,7 @@ Fixed a race condition in `CaptionGenerationStage` and `CaptionEnhancementStage` - **`SemanticDeduplicationWorkflow.run()` and `TextSemanticDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `dict` - **`TextDuplicatesRemovalWorkflow.run()`** now returns `WorkflowRunResult` instead of `list[FileGroupTask] | None` - Code that previously ignored the return value is unaffected. Code that consumed the old `dict` return from semantic workflows must migrate to `result.metadata` and `result.pipeline_tasks`. +- **`WebdatasetReaderStage` renamed to `InterleavedWebdatasetReaderStage`** (PR #1652): Code that imports `WebdatasetReaderStage` directly must update to `InterleavedWebdatasetReaderStage`. The `WebdatasetReader` composite stage is unchanged. +- **`source_id_field` removed** (PR #1652): The `source_id_field` parameter has been removed from `WebdatasetReader` and the underlying reader stage. Remove this parameter from existing code. +- **`ray_dist.jar` removed from Docker image** (PR #1612): The `ray_dist.jar` file has been removed from the Dockerfile to address GHSA-72hv-8253-57qq. This does not affect Python-based workflows. +- **`FuzzyDeduplicationWorkflow` LSH defaults changed** (PR #1603): The `rmm_pool_size` and `spill_memory_limit` parameters passed to the internal LSH stage now default to `"auto"` (previously hardcoded). This is exposed through the new `lsh_rmm_pool_size` and `lsh_spill_memory_limit` workflow parameters. diff --git a/fern/versions/v26.04/pages/curate-audio/process-data/preprocessing/band-filtering.mdx b/fern/versions/v26.04/pages/curate-audio/process-data/preprocessing/band-filtering.mdx new file mode 100644 index 0000000000..3eff4e7d74 --- /dev/null +++ b/fern/versions/v26.04/pages/curate-audio/process-data/preprocessing/band-filtering.mdx @@ -0,0 +1,97 @@ +--- +description: "Classify and filter audio by bandwidth using a pre-trained model to remove low-quality narrow-band recordings" +categories: ["audio-processing"] +tags: ["audio-filtering", "bandwidth-classification", "band-filter", "quality-filtering"] +personas: ["data-scientist-focused", "mle-focused"] +difficulty: "beginner" +content_type: "how-to" +modality: "audio-only" +--- + +# Band Filtering + +Classify audio as full-band or narrow-band based on spectral characteristics, then filter to keep only the desired bandwidth type. This is useful for removing low-quality telephone or compressed audio from training datasets. + +## How It Works + +`BandFilterStage` uses a pre-trained classifier model to predict whether each audio file is full-band or narrow-band: + +- **Full-band**: Broad frequency range typical of high-quality recordings +- **Narrow-band**: Limited frequency range typical of telephone audio or heavy compression + +The model downloads automatically from HuggingFace (`nvidia/nemocurator-speech-bandwidth-filter`) on first use. + +## Quick Start + +```python +from nemo_curator.pipeline import Pipeline +from nemo_curator.stages.audio.filtering import BandFilterStage + +pipeline = Pipeline(name="band_filter") + +# Keep only full-band audio +pipeline.add_stage(BandFilterStage(band_value="full_band")) +``` + +## Configuration + +| Parameter | Type | Default | Description | +| --- | --- | --- | --- | +| `model_path` | str \| None | None | Local path to `.joblib` model file. Auto-downloads from HuggingFace if not provided. | +| `cache_dir` | str \| None | None | Directory to cache downloaded models | +| `band_value` | "full_band" \| "narrow_band" | "full_band" | Which bandwidth type to pass through the filter | + +**Output field**: `band_prediction` (str: "full_band" or "narrow_band") + +## GPU Support + +`BandFilterStage` runs on CPU by default. To use GPU acceleration, configure the stage resources: + +```python +from nemo_curator.stages.audio.filtering import BandFilterStage +from nemo_curator.stages.resources import Resources + +stage = BandFilterStage( + band_value="full_band", + resources=Resources(gpus=1), +) +``` + +## Using a Local Model + +If you have a pre-downloaded model or a custom-trained classifier, provide the path directly: + +```python +stage = BandFilterStage( + model_path="/path/to/band_classifier_model.joblib", + band_value="full_band", +) +``` + +## Pipeline Example + +Combine band filtering with other audio processing stages: + +```python +from nemo_curator.pipeline import Pipeline +from nemo_curator.stages.audio.filtering import BandFilterStage +from nemo_curator.stages.audio.common import GetAudioDurationStage, PreserveByValueStage + +pipeline = Pipeline(name="audio_quality") + +# 1. Calculate duration +pipeline.add_stage(GetAudioDurationStage( + audio_filepath_key="audio_filepath", + duration_key="duration", +)) + +# 2. Filter short/long audio +pipeline.add_stage(PreserveByValueStage( + input_value_key="duration", + target_value=1.0, + operator="ge", +)) + +# 3. Keep only full-band audio +pipeline.add_stage(BandFilterStage(band_value="full_band")) +``` diff --git a/fern/versions/v26.04/pages/curate-audio/process-data/preprocessing/index.mdx b/fern/versions/v26.04/pages/curate-audio/process-data/preprocessing/index.mdx new file mode 100644 index 0000000000..427d36631c --- /dev/null +++ b/fern/versions/v26.04/pages/curate-audio/process-data/preprocessing/index.mdx @@ -0,0 +1,127 @@ +--- +description: "Convert multi-channel audio to mono, concatenate VAD segments, and map timestamps back to original files" +categories: ["audio-processing"] +tags: ["audio-preprocessing", "mono-conversion", "segment-concatenation", "timestamp-mapping", "vad"] +personas: ["data-scientist-focused", "mle-focused"] +difficulty: "intermediate" +content_type: "how-to" +modality: "audio-only" +--- + +# Audio Preprocessing + +Prepare raw audio files for downstream processing with stages that handle channel conversion, segment assembly, and timestamp resolution. These stages typically run early in an audio pipeline before ASR inference and quality assessment. + +## Overview + +Audio preprocessing in NeMo Curator provides three stages that work together: + +1. **MonoConversionStage**: Converts multi-channel audio to mono and verifies sample rate +2. **SegmentConcatenationStage**: Assembles VAD-segmented audio into a single waveform per source file +3. **TimestampMapperStage**: Resolves segment positions in the concatenated waveform back to the original file + +A typical preprocessing pipeline chains these stages with VAD segmentation: + +``` +MonoConversion → VADSegmentation(nested=True) → SegmentConcatenation → [processing] → TimestampMapper +``` + +## Mono Conversion + +Convert multi-channel audio to mono by averaging channels and verify the sample rate matches expectations. + +```python +from nemo_curator.stages.audio.preprocessing import MonoConversionStage + +stage = MonoConversionStage( + output_sample_rate=48000, + audio_filepath_key="audio_filepath", + strict_sample_rate=True, +) +``` + +| Parameter | Type | Default | Description | +| --- | --- | --- | --- | +| `output_sample_rate` | int | 48000 | Expected sample rate in Hz | +| `audio_filepath_key` | str | "audio_filepath" | Key in data dict for audio file path | +| `strict_sample_rate` | bool | True | Reject audio with mismatched sample rate | + +**Output fields**: `waveform`, `sample_rate`, `is_mono`, `duration`, `num_samples` + +When `strict_sample_rate` is `True`, audio files with a sample rate different from `output_sample_rate` are filtered out. Set to `False` to accept any sample rate. + +## Segment Concatenation + +Concatenate VAD segments from `VADSegmentationStage(nested=True)` into a single waveform per source file, with configurable silence between segments. + +```python +from nemo_curator.stages.audio.preprocessing import SegmentConcatenationStage + +stage = SegmentConcatenationStage( + silence_duration_sec=0.5, +) +``` + +| Parameter | Type | Default | Description | +| --- | --- | --- | --- | +| `silence_duration_sec` | float | 0.5 | Duration of silence inserted between segments (seconds) | + +**Input**: Expects `task.data["segments"]` list from `VADSegmentationStage(nested=True)` + +**Output fields**: `waveform`, `sample_rate`, `num_segments`, `total_duration_sec`, `original_file` + +Segments are sorted by `segment_num` before concatenation. Gaps from filtered-out segments are preserved in the correct order. Segment-to-original mappings are stored in `task._metadata["segment_mappings"]` for use by `TimestampMapperStage`. + +## Timestamp Mapping + +Resolve segment positions in the concatenated waveform back to positions in the original audio file. Strips waveform data from the output, producing metadata-only results. + +```python +from nemo_curator.stages.audio.postprocessing import TimestampMapperStage + +stage = TimestampMapperStage( + passthrough_keys=["speaker_id", "quality_score"], +) +``` + +| Parameter | Type | Default | Description | +| --- | --- | --- | --- | +| `passthrough_keys` | list[str] \| None | None | Additional data keys to preserve in output | + +**Input**: Requires `task._metadata["segment_mappings"]` from `SegmentConcatenationStage` + +**Output fields**: `original_file`, `original_start_ms`, `original_end_ms`, `duration_ms`, `duration_sec` + +## Complete Pipeline + +Here is a complete pipeline combining preprocessing stages with VAD and ASR: + +```python +from nemo_curator.pipeline import Pipeline +from nemo_curator.stages.audio.preprocessing import MonoConversionStage, SegmentConcatenationStage +from nemo_curator.stages.audio.postprocessing import TimestampMapperStage + +pipeline = Pipeline(name="audio_preprocessing") + +# 1. Convert to mono and verify sample rate +pipeline.add_stage(MonoConversionStage( + output_sample_rate=48000, + strict_sample_rate=True, +)) + +# 2. Run VAD segmentation (nested mode groups segments per file) +# pipeline.add_stage(VADSegmentationStage(nested=True)) + +# 3. Concatenate segments into single waveform +pipeline.add_stage(SegmentConcatenationStage( + silence_duration_sec=0.5, +)) + +# 4. Run downstream processing (ASR, quality assessment, etc.) +# ... + +# 5. Map timestamps back to original file positions +pipeline.add_stage(TimestampMapperStage( + passthrough_keys=["transcript", "quality_score"], +)) +``` diff --git a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx index 8a53338fab..bf44bd81ed 100644 --- a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx +++ b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx @@ -95,6 +95,29 @@ Configure fuzzy deduplication using these key parameters: | `input_filetype` | str | "parquet" | Input file format ("parquet" or "jsonl") | | `input_blocksize` | str \| int | "1GiB" | Size of input blocks for processing | | `perform_removal` | bool | False | Reserved; must remain `False`. Fuzzy removal is performed with `TextDuplicatesRemovalWorkflow`. | +| `lsh_num_output_partitions` | int \| None | None | Total partitions during LSH shuffle. If None, auto-selects closest power of 2 <= input tasks. | +| `lsh_rmm_pool_size` | int \| "auto" \| None | "auto" | RMM GPU memory pool size in bytes. `"auto"` = 90% of free GPU memory; `None` = 50% expandable. | +| `lsh_spill_memory_limit` | int \| "auto" \| None | "auto" | Device memory spill limit in bytes. `"auto"` = 80% of RMM pool; `None` = spilling disabled. | + +### LSH GPU Memory Tuning + +The `lsh_*` parameters control GPU memory allocation during the LSH stage, which is the most memory-intensive phase: + +- **`lsh_rmm_pool_size`**: Controls the RAPIDS Memory Manager pool. `"auto"` (default) allocates 90% of free GPU memory as a fixed pool. Set to `None` for a 50% expandable pool that grows on demand. +- **`lsh_spill_memory_limit`**: Controls when data spills from GPU to host memory. `"auto"` (default) spills at 80% of the RMM pool. Set to `None` to disable spilling entirely (risks OOM on large datasets). +- **`lsh_num_output_partitions`**: Controls shuffle parallelism. Higher values reduce per-partition memory at the cost of more shuffle overhead. Leave as `None` for automatic selection. + +```python +# Example: manual GPU memory configuration for a 40GB GPU +fuzzy_workflow = FuzzyDeduplicationWorkflow( + input_path="input_data/", + cache_path="./cache", + output_path="./results", + lsh_rmm_pool_size=34_000_000_000, # 34GB fixed pool + lsh_spill_memory_limit=27_000_000_000, # Spill at 27GB + lsh_num_output_partitions=64, # 64 output partitions +) +``` ### Similarity Threshold diff --git a/fern/versions/v26.04/pages/curate-video/process-data/captions-preview.mdx b/fern/versions/v26.04/pages/curate-video/process-data/captions-preview.mdx index 3e0146f8ba..5ac1ef5e39 100644 --- a/fern/versions/v26.04/pages/curate-video/process-data/captions-preview.mdx +++ b/fern/versions/v26.04/pages/curate-video/process-data/captions-preview.mdx @@ -1,7 +1,7 @@ --- -description: "Generate clip captions with Qwen and optional preview images" +description: "Generate clip captions with Qwen or Nemotron VLM and optional preview images" categories: ["video-curation"] -tags: ["captions", "qwen", "preview", "video"] +tags: ["captions", "qwen", "nemotron", "preview", "video", "vlm"] personas: ["data-scientist-focused", "mle-focused"] difficulty: "intermediate" content_type: "howto" @@ -10,7 +10,7 @@ modality: "video-only" # Captions and Preview -Prepare inputs, generate captions, optionally enhance them, and produce preview images. +Prepare inputs, generate captions, optionally enhance them, and produce preview images. NeMo Curator supports two vision-language models for video captioning: **Qwen-VL** and **Nemotron VLM**. --- @@ -19,7 +19,7 @@ Prepare inputs, generate captions, optionally enhance them, and produce preview Use the pipeline stages or the example script flags to prepare captions and preview images. - + ```python from nemo_curator.pipeline import Pipeline @@ -62,6 +62,40 @@ pipe.run() + + +```python +from nemo_curator.pipeline import Pipeline +from nemo_curator.stages.video.caption.caption_preparation import CaptionPreparationStage +from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage +from nemo_curator.stages.video.preview.preview import PreviewStage + +pipe = Pipeline(name="nemotron_captions") +pipe.add_stage( + CaptionPreparationStage( + model_variant="nemotron", + prompt_variant="default", + sampling_fps=2.0, + window_size=256, + remainder_threshold=128, + generate_previews=True, + verbose=True, + ) +) +pipe.add_stage(PreviewStage(target_fps=1.0, target_height=240, verbose=True)) +pipe.add_stage( + CaptionGenerationStage( + model_dir="/models", + model_variant="nemotron", + caption_batch_size=8, + max_output_tokens=512, + ) +) +pipe.run() +``` + + + ```bash @@ -122,7 +156,7 @@ python tutorials/video/getting-started/video_split_clip_example.py \ | Parameter | Type | Default | Description | | --- | --- | --- | --- | -| `model_variant` | str | `"qwen"` | Vision‑language model used to format inputs for captioning (currently `qwen`). | +| `model_variant` | str | `"qwen"` | Vision‑language model variant: `"qwen"` or `"nemotron"`. | | `prompt_variant` | {"default", "av", "av-surveillance"} | `"default"` | Built‑in prompt to steer caption content when `prompt_text` is not provided. | | `prompt_text` | str \| None | `None` | Custom prompt text. When set, overrides `prompt_variant`. | | `sampling_fps` | float | 2.0 | Source sampling rate for creating per‑window inputs. | @@ -151,7 +185,7 @@ python tutorials/video/getting-started/video_split_clip_example.py \ ## Caption generation and enhancement -1. Generate window‑level captions with a vision‑language model (Qwen‑VL). This stage reads `clip.windows[*].qwen_llm_input` created earlier and writes `window.caption["qwen"]`. +1. Generate window‑level captions with a vision‑language model. Supported models are **Qwen-VL** and **Nemotron VLM** (NemotronH hybrid Mamba-Attention). This stage reads the model-specific LLM input created by `CaptionPreparationStage` and writes captions per window. ```python from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage @@ -194,7 +228,7 @@ python tutorials/video/getting-started/video_split_clip_example.py \ | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `model_dir` | str | `"models/qwen"` | Directory for model weights; downloaded on each node if missing. | -| `model_variant` | {"qwen"} | `"qwen"` | Vision‑language model variant. | +| `model_variant` | {"qwen", "nemotron", "nemotron-bf16", "nemotron-fp8", "nemotron-nvfp4"} | `"qwen"` | Vision‑language model variant. See [Nemotron VLM variants](#nemotron-vlm-variants) for quantization options. | | `caption_batch_size` | int | 16 | Batch size for caption generation. | | `fp8` | bool | `False` | Use FP8 weights when available. | | `max_output_tokens` | int | 512 | Maximum number of tokens to generate per caption. | @@ -211,7 +245,7 @@ python tutorials/video/getting-started/video_split_clip_example.py \ | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `model_dir` | str | `"models/qwen"` | Directory for language‑model weights; downloaded per node if missing. | -| `model_variant` | {"qwen"} | `"qwen"` | Language‑model variant. | +| `model_variant` | {"qwen", "nemotron"} | `"qwen"` | Language‑model variant. | | `prompt_variant` | {"default", "av-surveillance"} | `"default"` | Built‑in enhancement prompt when `prompt_text` is not provided. | | `prompt_text` | str \| None | `None` | Custom enhancement prompt. When set, overrides `prompt_variant`. | | `model_batch_size` | int | 128 | Batch size for enhancement generation. | @@ -266,6 +300,20 @@ previews_dir = ClipWriterStage.get_output_path_previews("/outputs") Refer to Save & Export for directory structure and file locations: [Save & Export](/curate-video/save-export). +## Nemotron VLM Variants + +The Nemotron VLM (NemotronH hybrid Mamba-Attention) is available in multiple quantization variants from HuggingFace. Models download automatically on first use. + +| Variant | HuggingFace Model ID | Precision | Notes | +| --- | --- | --- | --- | +| `nemotron` / `nemotron-bf16` | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16` | BF16 | Default variant, highest quality | +| `nemotron-fp8` | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8` | FP8 | Reduced memory, slight quality trade-off | +| `nemotron-nvfp4` | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD` | NVFP4 | Smallest memory footprint | + + +Nemotron VLM requires vLLM with V1 engine support and `flash-attn>=2.8.3`. + + ### Requirements and Troubleshooting - `ffmpeg` with WebP (`libwebp`) support must be available in the environment.