Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion fern/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,26 @@ redirects:
- source: "/nemo/curator/:path*.html"
destination: "/nemo/curator/:path*"
# Audio concepts: legacy nested nav URLs → flat slugs (aligned with Sphinx filenames)
- source: "/about/concepts/audio/curation/pipeline"
destination: "/about/concepts/audio/curation-pipeline"
- source: "/about/concepts/audio/audio/batch"
destination: "/about/concepts/audio/audio-task"
- source: "/about/concepts/audio/audio-batch"
destination: "/about/concepts/audio/audio-task"
- source: "/about/concepts/audio/asr/pipeline"
destination: "/about/concepts/audio/asr-pipeline"
- source: "/about/concepts/audio/quality/metrics"
destination: "/about/concepts/audio/quality-metrics"
- source: "/about/concepts/audio/manifests/ingest"
destination: "/about/concepts/audio/manifests-ingest"
- source: "/about/concepts/audio/text/integration"
destination: "/about/concepts/audio/text-integration"
- source: "/nemo/curator/about/concepts/audio/curation/pipeline"
destination: "/nemo/curator/about/concepts/audio/curation-pipeline"
- source: "/nemo/curator/about/concepts/audio/audio/batch"
destination: "/nemo/curator/about/concepts/audio/audio-batch"
destination: "/nemo/curator/about/concepts/audio/audio-task"
- source: "/nemo/curator/about/concepts/audio/audio-batch"
destination: "/nemo/curator/about/concepts/audio/audio-task"
- source: "/nemo/curator/about/concepts/audio/asr/pipeline"
destination: "/nemo/curator/about/concepts/audio/asr-pipeline"
- source: "/nemo/curator/about/concepts/audio/quality/metrics"
Expand Down
12 changes: 6 additions & 6 deletions fern/versions/v26.04.yml
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,9 @@ navigation:
- page: Curation Pipeline
path: ./v26.04/pages/about/concepts/audio/curation-pipeline.mdx
slug: curation-pipeline
- page: Audio Batch
path: ./v26.04/pages/about/concepts/audio/audio-batch.mdx
slug: audio-batch
- page: Audio Task
path: ./v26.04/pages/about/concepts/audio/audio-task.mdx
slug: audio-task
- page: ASR Pipeline
path: ./v26.04/pages/about/concepts/audio/asr-pipeline.mdx
slug: asr-pipeline
Expand Down Expand Up @@ -567,9 +567,9 @@ navigation:
- page: VideoTask
path: ./v26.04/pages/api-reference/tasks/video-task.mdx
slug: video-task
- page: AudioBatch
path: ./v26.04/pages/api-reference/tasks/audio-batch.mdx
slug: audio-batch
- page: AudioTask
path: ./v26.04/pages/api-reference/tasks/audio-task.mdx
slug: audio-task
- section: Executors
slug: executors
contents:
Expand Down
28 changes: 14 additions & 14 deletions fern/versions/v26.04/pages/about/concepts/audio/asr-pipeline.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The ASR pipeline in NeMo Curator follows a systematic approach to speech process

```mermaid
graph TD
A[Audio Files] --> B[AudioBatch Creation]
A[Audio Files] --> B[AudioTask Creation]
B --> C[ASR Model Loading]
C --> D[Batch Inference]
D --> E[Transcription Output]
Expand Down Expand Up @@ -46,15 +46,15 @@ graph TD

### 1. Audio Input Management

**AudioBatch Structure**: The foundation for audio processing
**AudioTask Structure**: The foundation for audio processing

- Contains audio file paths and associated metadata
- Validates file existence and accessibility automatically
- Supports efficient batch processing for scalability

**Input Validation**: Ensures data integrity before processing

- File path existence checks using `AudioBatch.validate()` and `validate_item()`
- File path existence checks using `AudioTask.validate()` and `validate_item()`
- Optional metadata validation added by downstream stages (such as duration and format checks)

### 2. ASR Model Integration
Expand Down Expand Up @@ -104,7 +104,7 @@ manifest_reader = JsonlReader(
input_file_path="/path/to/manifest.jsonl"
)

# Stages automatically create AudioBatch objects from loaded data
# Stages automatically create AudioTask objects from loaded data
```

### Stage 2: ASR Model Setup
Expand All @@ -125,12 +125,12 @@ asr_stage.setup() # Downloads and loads model
### Stage 3: Transcription Generation

```python
# ASR stage processes AudioBatch objects automatically
# The stage extracts file paths and calls transcribe() internally
processed_batch = asr_stage.process(audio_batch)
# Don't call process() directly — the Pipeline/Executor handles dispatch:
pipeline.add_stage(asr_stage)
results = pipeline.run(executor)

# Output: AudioBatch with added "pred_text" field
# Each item now contains both original data and predictions
# Output: AudioTask objects with added "pred_text" field
# Each task now contains both original data and predictions
```

### Stage 4: Quality Assessment Integration
Expand All @@ -156,7 +156,7 @@ duration_stage = GetAudioDurationStage(

1. **Audio Files** → File system
2. **Manifest Files** → JSONL format with metadata
3. **AudioBatch Objects** → Validated, structured data containers
3. **AudioTask Objects** → Validated, structured data containers

### Processing Data Flow

Expand Down Expand Up @@ -217,10 +217,10 @@ asr_stage = InferenceAsrNemoStage(

```python
# Validate and filter invalid file paths
audio_batch = AudioBatch(data=audio_data, filepath_key="audio_filepath")
audio_task = AudioTask(data=audio_data, filepath_key="audio_filepath")

# Filter out entries that do not exist on disk
valid_samples = [item for item in audio_batch.data if audio_batch.validate_item(item)]
# Validate the audio file exists on disk
is_valid = audio_task.validate()
```

### Pipeline Recovery
Expand All @@ -237,7 +237,7 @@ The ASR pipeline seamlessly integrates with text processing workflows:
# Audio → Text pipeline
audio_to_text = [
InferenceAsrNemoStage(), # Audio → Transcriptions
AudioToDocumentStage(), # AudioBatch → DocumentBatch
AudioToDocumentStage(), # AudioTask → DocumentBatch
# Continue with text processing stages...
]
```
249 changes: 0 additions & 249 deletions fern/versions/v26.04/pages/about/concepts/audio/audio-batch.mdx

This file was deleted.

Loading
Loading