Skip to content

docs: rename AudioBatch to AudioTask in audio curation docs - #1694

Merged
lbliii merged 5 commits into
26.04-stagingfrom
lbliii/docs-cosmos-xenna-update
Apr 14, 2026
Merged

docs: rename AudioBatch to AudioTask in audio curation docs#1694
lbliii merged 5 commits into
26.04-stagingfrom
lbliii/docs-cosmos-xenna-update

Conversation

@lbliii

@lbliii lbliii commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Description

Renames the AudioBatch class and concept to AudioTask throughout the 26.04 audio curation documentation to reflect the upstream API change. Updates navigation, URL redirects, concept pages, API reference, tutorials, and adds a release note entry for the rename.

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

@lbliii
lbliii requested a review from a team as a code owner March 31, 2026 18:20
@lbliii
lbliii requested review from praateekmahajan and removed request for a team March 31, 2026 18:20
Rename AudioBatch class/concept to AudioTask throughout the 26.04
documentation to reflect the upstream API rename. Updates navigation,
redirects, concepts, API reference, tutorials, and release notes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
@greptile-apps

greptile-apps Bot commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR renames AudioBatch to AudioTask across the v26.04 audio curation documentation, reflecting the upstream API change to a single-entry task model. All prior review concerns (broken list-comprehension over dict keys, passing a list to AudioTask, process_batch() contradiction between pages, and misleading "batch-level validation" wording) have been resolved in the follow-up commit.

Two minor documentation inconsistencies remain in the newly added files: audio-task.mdx lists AudioToDocumentStage in the Batched pattern table with a return type of list[AudioTask], while the rest of the docs consistently describe this stage as converting to DocumentBatch; and api-reference/tasks/audio-task.mdx (echoed in the release notes) states "all audio stages subclass ProcessingStage[AudioTask, AudioTask]", which does not hold for AudioToDocumentStage.

Confidence Score: 5/5

  • Documentation-only PR; all prior blocking concerns addressed. Safe to merge with optional follow-up on the AudioToDocumentStage return-type wording.
  • All P0/P1 findings from previous review rounds have been fixed. The two remaining findings are P2 documentation precision issues that do not affect runtime behavior or correctness of any code example.
  • fern/versions/v26.04/pages/about/concepts/audio/audio-task.mdx and fern/versions/v26.04/pages/api-reference/tasks/audio-task.mdx — both contain the AudioToDocumentStage return-type inconsistency.

Important Files Changed

Filename Overview
fern/versions/v26.04/pages/about/concepts/audio/audio-task.mdx New concept page for AudioTask; well-structured, but the Processing Patterns table incorrectly lists AudioToDocumentStage with return type list[AudioTask] when it converts to DocumentBatch.
fern/versions/v26.04/pages/api-reference/tasks/audio-task.mdx New API reference replacing audio-batch.mdx; accurate for AudioTask itself, but the blanket claim "All audio stages subclass ProcessingStage[AudioTask, AudioTask]" is incorrect for AudioToDocumentStage.
fern/versions/v26.04/pages/about/concepts/audio/manifests-ingest.mdx Correctly updated: validation example now iterates over manifest entries one-at-a-time using a for-loop, and "batch-level validation" phrasing replaced with accurate single-entry wording.
fern/versions/v26.04/pages/about/concepts/audio/asr-pipeline.mdx All AudioBatch references replaced; validate/filter example fixed to use audio_task.validate() instead of the broken list-comprehension over dict keys.
fern/versions/v26.04/pages/curate-audio/process-data/asr-inference/index.mdx Note now correctly says the ASR stage defines process_batch() (consistent with audio-task.mdx), and "file paths" plural corrected to singular "audio file path".
fern/versions/v26.04/pages/curate-audio/tutorials/beginner.mdx No AudioBatch references remain; pipeline code and stage explanations look accurate for the single-entry AudioTask model.
fern/versions/v26.04/pages/about/release-notes/index.mdx Audio Task Redesign entry accurately describes the AudioBatch → AudioTask rename and single-entry model; the "all stages subclass ProcessingStage[AudioTask, AudioTask]" wording is a minor imprecision shared with the API reference.
fern/docs.yml URL redirects from /audio-batch to /audio-task added correctly for both root and /nemo/curator path prefixes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["JSONL Manifest\n(audio_filepath, text, …)"] --> B["AudioTask\n(single manifest entry)"]
    B --> C["CreateInitialManifestFleursStage\nprocess_batch → list[AudioTask]"]
    C --> D["InferenceAsrNemoStage\nprocess_batch → list[AudioTask]\n+ pred_text field"]
    D --> E["GetPairwiseWerStage\nprocess → AudioTask\n+ wer field"]
    E --> F["GetAudioDurationStage\nprocess → AudioTask\n+ duration field"]
    F --> G["PreserveByValueStage\nprocess_batch → list[AudioTask]\n(filtering)"]
    G --> H["AudioToDocumentStage\nprocess_batch → list[DocumentBatch]"]
    H --> I["JsonlWriter\nOutput JSONL"]

    style A fill:#e1f5fe
    style B fill:#f3e5f5
    style H fill:#fff3e0
    style I fill:#fce4ec
Loading

Reviews (4): Last reviewed commit: "merge: resolve conflicts with 26.04-stag..." | Re-trigger Greptile


# Filter out entries that do not exist on disk
valid_samples = [item for item in audio_batch.data if audio_batch.validate_item(item)]
valid_samples = [item for item in audio_task.data if audio_task.validate_item(item)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Iterating over AudioTask.data yields dict keys, not items

AudioTask.data is a single dict (or _AttrDict), so for item in audio_task.data iterates over the dict's keys (strings like "audio_filepath", "text", etc.) rather than over audio sample entries. The original AudioBatch.data was a list[dict], which is why the list comprehension made sense before.

Since AudioTask holds exactly one manifest entry, the validation should be called on that single entry directly — for example, audio_task.validate_item(audio_task.data). If the intent is to show bulk filtering across many tasks, each task should be constructed individually with one dict, then filtered in a loop.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

audio_task.data is a dict (specifically _AttrDict). Iterating over a dict yields its keys — strings like "audio_filepath", "text", "duration" — not manifest entries. This is holdover from AudioBatch.data which was list[dict].

The comprehension would produce something like ["audio_filepath", "text"] (keys that happen to pass validate_item), which is nonsensical.

Since AudioTask is single-entry, validation is simply:

is_valid = audio_task.validate()

Comment on lines 107 to 118
```python
from nemo_curator.tasks import AudioBatch
from nemo_curator.tasks import AudioTask

# Create AudioBatch with validation
audio_batch = AudioBatch(
# Create AudioTask with validation
audio_task = AudioTask(
data=manifest_data,
filepath_key="audio_filepath"
)

# Validate file existence
is_valid = audio_batch.validate()
is_valid = audio_task.validate()
print(f"Batch validation: {is_valid}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 AudioTask only accepts a single dict, but the example passes a list

manifest_data is defined earlier in this same file (lines 84–96) as a list of dicts. AudioTask only accepts data: dict — a single manifest entry — whereas the old AudioBatch accepted dict | list[dict].

Passing a list here would either raise a type error or produce silently wrong behaviour at runtime. The code example should construct AudioTask from a single entry (e.g., manifest_data[0]), or demonstrate creating one AudioTask per entry in a loop, to match the new single-entry model.

Comment on lines +143 to +144
| **Per-task** | CPU stages (`GetAudioDurationStage`, `GetPairwiseWerStage`) | `process(task) → AudioTask` — mutates `task.data` in-place |
| **Batched** | GPU stages (`InferenceAsrNemoStage`), IO stages (`AudioToDocumentStage`), filtering (`PreserveByValueStage`) | `process_batch(tasks) → list[AudioTask]` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Contradicts asr-inference/index.mdx on whether InferenceAsrNemoStage defines process_batch()

This table lists InferenceAsrNemoStage under the Batched pattern and states its method is process_batch(tasks) → list[AudioTask].

However, fern/versions/v26.04/pages/curate-audio/process-data/asr-inference/index.mdx line 105 explicitly states the opposite:

"The ASR stage does not define process_batch(); the executor batches tasks."

One of these two statements is incorrect. Please align both pages. If the ASR stage does not define process_batch(), InferenceAsrNemoStage should be moved to the Per-task row, or a clarifying note added explaining that the executor groups tasks before calling process().

`batch_size` controls the number of tasks the executor groups per call. The ASR stage does not define `process_batch()`; the executor batches tasks.

Within a single `AudioBatch`, `process()` transcribes the file paths together.
Within a single `AudioTask`, `process()` transcribes the file paths together.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 "file paths" (plural) is inaccurate for a single-entry AudioTask

The original phrasing made sense for AudioBatch, which could hold a list of entries. Now that AudioTask holds exactly one manifest entry (one file path), the plural "file paths" is misleading. Consider updating to: "Within a single AudioTask, process() transcribes the audio file path."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1


- `AudioBatch` automatically validates file paths during creation
- `AudioTask` automatically validates file paths during creation
- Use `validate()` for batch-level validation

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 "batch-level validation" is misleading after the rename

AudioTask is a single-entry model; there is no longer a "batch level" to validate. This bullet retained the old AudioBatch framing. Consider updating to: "Use validate() to check whether the audio file for this task exists on disk".

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Replace latinisms (via, etc.) with plain English equivalents and fix
code formatting spacing in text-integration how-to.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
@lbliii

lbliii commented Apr 6, 2026

Copy link
Copy Markdown
Contributor Author

@mohammadaaftabv pls review

Keep both PR's AudioBatch→AudioTask rename changes and upstream's new
release note sections (metrics, workflows, CVE fixes, bug fixes).
For docs.yml redirects, add /nemo/curator/ prefixed versions with
audio-task destination applied.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@copy-pr-bot

copy-pr-bot Bot commented Apr 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.


# Access duration information
duration = result_batch[0].data[0]["duration"]
duration = result_batch[0].data["duration"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetAudioDurationStage.process() returns a single AudioTask, not a list. Three code blocks index the result as result_batch[0] or iterate with for result in result_batches:, both of which would raise TypeError at runtime.

Fix:
GetAudioDurationStage.process() returns a single AudioTask, not a list. Three code blocks index the result as result_batch[0] or iterate with for result in result_batches:, both of which would raise TypeError at runtime.

Fix also occurrences in
Occurrence 2 (lines 104-109):

WRONG: same issue

result_batch = duration_stage.process(audio_task)
processed_sample = result_batch[0].data # TypeError
Fix:

result = duration_stage.process(audio_task)
processed_sample = result.data
Occurrence 3 (lines 164-168):

WRONG: iterating over AudioTask yields nothing useful

result_batches = duration_stage.process(audio_task)
for result in result_batches: # iterates over AudioTask attributes, not tasks
Fix:

result = duration_stage.process(audio_task)
print(f"File: {result.data['audio_filepath']}")
print(f"Duration: {result.data['duration']:.3f} seconds")

import json

# Create simple manifest
manifest_data = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs code on lines 84-97 defines manifest_data as a list of 2 dicts, then lines 111-113 pass the whole list to AudioTask(data=manifest_data). Any user following this will get the ValueError above.

Fix: Show per-entry construction:

for entry in manifest_data:
audio_task = AudioTask(data=entry, filepath_key="audio_filepath")
is_valid = audio_task.validate()
print(f"Task validation: {is_valid}")

# ASR stage processes AudioTask objects automatically
# The stage extracts file paths and calls transcribe() internally
processed_batch = asr_stage.process(audio_batch)
processed_batch = asr_stage.process(audio_task)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InferenceAsrNemoStage.process() raises NotImplementedError.
Users should never call process() directly on ASR stages — the pipeline executor handles dispatch via process_batch().

Fix — either show process_batch() directly:

results = asr_stage.process_batch([audio_task])
Or (better) show the executor-driven pattern:

Don't call process() directly — the Pipeline/Executor handles dispatch:

pipeline.add_stage(asr_stage)
results = pipeline.run(executor)

@@ -104,17 +104,17 @@ asr_stage = asr_stage.with_(
<Note>
`batch_size` controls the number of tasks the executor groups per call. The ASR stage does not define `process_batch()`; the executor batches tasks.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InferenceAsrNemoStage does define process_batch() as its canonical method (line 111 of asr_nemo.py)
process() explicitly raises NotImplementedError
Fix:
batch_size controls the number of tasks the executor groups per call. The ASR stage defines process_batch() as its canonical method — the executor groups tasks by batch_size before calling it.


- `AudioBatch` automatically validates file paths during creation
- `AudioTask` automatically validates file paths during creation
- Use `validate()` for batch-level validation

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

`batch_size` controls the number of tasks the executor groups per call. The ASR stage does not define `process_batch()`; the executor batches tasks.

Within a single `AudioBatch`, `process()` transcribes the file paths together.
Within a single `AudioTask`, `process()` transcribes the file paths together.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1


# Filter out entries that do not exist on disk
valid_samples = [item for item in audio_batch.data if audio_batch.validate_item(item)]
valid_samples = [item for item in audio_task.data if audio_task.validate_item(item)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1


# Filter out entries that do not exist on disk
valid_samples = [item for item in audio_batch.data if audio_batch.validate_item(item)]
valid_samples = [item for item in audio_task.data if audio_task.validate_item(item)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

audio_task.data is a dict (specifically _AttrDict). Iterating over a dict yields its keys — strings like "audio_filepath", "text", "duration" — not manifest entries. This is holdover from AudioBatch.data which was list[dict].

The comprehension would produce something like ["audio_filepath", "text"] (keys that happen to pass validate_item), which is nonsensical.

Since AudioTask is single-entry, validation is simply:

is_valid = audio_task.validate()

lbliii and others added 2 commits April 14, 2026 11:27
Address review comments from #1694:
- Fix process() return type in duration-calculation.mdx (returns single
  AudioTask, not a list)
- Fix AudioTask construction in manifests-ingest.mdx (single dict per
  task, not a list)
- Update "batch-level validation" wording to match single-entry model
- Correct asr-inference/index.mdx to state ASR stage defines
  process_batch() as its canonical method
- Fix "file paths" plural to singular for single-entry AudioTask
- Replace invalid list comprehension over AudioTask.data dict keys in
  asr-pipeline.mdx with validate() call
- Replace direct process() call on ASR stage with pipeline-driven pattern

Signed-off-by: Lawrence Lane <llane@nvidia.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep both sides in release-notes/index.mdx:
- Dependency Updates: use upstream's more detailed Ray description,
  keep pynvml from our branch, add sentence-transformers and vllm
- Breaking Changes: keep AudioBatch removal note from our branch,
  add RayDataExecutor, DocumentExtractStage, DocumentIterateStage,
  and Three-Stage Pipeline entries from upstream

Signed-off-by: Lawrence Lane <llane@nvidia.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants