docs: rename AudioBatch to AudioTask in audio curation docs - #1694
Conversation
Rename AudioBatch class/concept to AudioTask throughout the 26.04 documentation to reflect the upstream API rename. Updates navigation, redirects, concepts, API reference, tutorials, and release notes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
Greptile SummaryThis PR renames Two minor documentation inconsistencies remain in the newly added files: Confidence Score: 5/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["JSONL Manifest\n(audio_filepath, text, …)"] --> B["AudioTask\n(single manifest entry)"]
B --> C["CreateInitialManifestFleursStage\nprocess_batch → list[AudioTask]"]
C --> D["InferenceAsrNemoStage\nprocess_batch → list[AudioTask]\n+ pred_text field"]
D --> E["GetPairwiseWerStage\nprocess → AudioTask\n+ wer field"]
E --> F["GetAudioDurationStage\nprocess → AudioTask\n+ duration field"]
F --> G["PreserveByValueStage\nprocess_batch → list[AudioTask]\n(filtering)"]
G --> H["AudioToDocumentStage\nprocess_batch → list[DocumentBatch]"]
H --> I["JsonlWriter\nOutput JSONL"]
style A fill:#e1f5fe
style B fill:#f3e5f5
style H fill:#fff3e0
style I fill:#fce4ec
Reviews (4): Last reviewed commit: "merge: resolve conflicts with 26.04-stag..." | Re-trigger Greptile |
|
|
||
| # Filter out entries that do not exist on disk | ||
| valid_samples = [item for item in audio_batch.data if audio_batch.validate_item(item)] | ||
| valid_samples = [item for item in audio_task.data if audio_task.validate_item(item)] |
There was a problem hiding this comment.
Iterating over
AudioTask.data yields dict keys, not items
AudioTask.data is a single dict (or _AttrDict), so for item in audio_task.data iterates over the dict's keys (strings like "audio_filepath", "text", etc.) rather than over audio sample entries. The original AudioBatch.data was a list[dict], which is why the list comprehension made sense before.
Since AudioTask holds exactly one manifest entry, the validation should be called on that single entry directly — for example, audio_task.validate_item(audio_task.data). If the intent is to show bulk filtering across many tasks, each task should be constructed individually with one dict, then filtered in a loop.
There was a problem hiding this comment.
audio_task.data is a dict (specifically _AttrDict). Iterating over a dict yields its keys — strings like "audio_filepath", "text", "duration" — not manifest entries. This is holdover from AudioBatch.data which was list[dict].
The comprehension would produce something like ["audio_filepath", "text"] (keys that happen to pass validate_item), which is nonsensical.
Since AudioTask is single-entry, validation is simply:
is_valid = audio_task.validate()
| ```python | ||
| from nemo_curator.tasks import AudioBatch | ||
| from nemo_curator.tasks import AudioTask | ||
|
|
||
| # Create AudioBatch with validation | ||
| audio_batch = AudioBatch( | ||
| # Create AudioTask with validation | ||
| audio_task = AudioTask( | ||
| data=manifest_data, | ||
| filepath_key="audio_filepath" | ||
| ) | ||
|
|
||
| # Validate file existence | ||
| is_valid = audio_batch.validate() | ||
| is_valid = audio_task.validate() | ||
| print(f"Batch validation: {is_valid}") |
There was a problem hiding this comment.
AudioTask only accepts a single dict, but the example passes a list
manifest_data is defined earlier in this same file (lines 84–96) as a list of dicts. AudioTask only accepts data: dict — a single manifest entry — whereas the old AudioBatch accepted dict | list[dict].
Passing a list here would either raise a type error or produce silently wrong behaviour at runtime. The code example should construct AudioTask from a single entry (e.g., manifest_data[0]), or demonstrate creating one AudioTask per entry in a loop, to match the new single-entry model.
| | **Per-task** | CPU stages (`GetAudioDurationStage`, `GetPairwiseWerStage`) | `process(task) → AudioTask` — mutates `task.data` in-place | | ||
| | **Batched** | GPU stages (`InferenceAsrNemoStage`), IO stages (`AudioToDocumentStage`), filtering (`PreserveByValueStage`) | `process_batch(tasks) → list[AudioTask]` | |
There was a problem hiding this comment.
Contradicts
asr-inference/index.mdx on whether InferenceAsrNemoStage defines process_batch()
This table lists InferenceAsrNemoStage under the Batched pattern and states its method is process_batch(tasks) → list[AudioTask].
However, fern/versions/v26.04/pages/curate-audio/process-data/asr-inference/index.mdx line 105 explicitly states the opposite:
"The ASR stage does not define
process_batch(); the executor batches tasks."
One of these two statements is incorrect. Please align both pages. If the ASR stage does not define process_batch(), InferenceAsrNemoStage should be moved to the Per-task row, or a clarifying note added explaining that the executor groups tasks before calling process().
| `batch_size` controls the number of tasks the executor groups per call. The ASR stage does not define `process_batch()`; the executor batches tasks. | ||
|
|
||
| Within a single `AudioBatch`, `process()` transcribes the file paths together. | ||
| Within a single `AudioTask`, `process()` transcribes the file paths together. |
There was a problem hiding this comment.
"file paths" (plural) is inaccurate for a single-entry
AudioTask
The original phrasing made sense for AudioBatch, which could hold a list of entries. Now that AudioTask holds exactly one manifest entry (one file path), the plural "file paths" is misleading. Consider updating to: "Within a single AudioTask, process() transcribes the audio file path."
|
|
||
| - `AudioBatch` automatically validates file paths during creation | ||
| - `AudioTask` automatically validates file paths during creation | ||
| - Use `validate()` for batch-level validation |
There was a problem hiding this comment.
Replace latinisms (via, etc.) with plain English equivalents and fix code formatting spacing in text-integration how-to. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
|
@mohammadaaftabv pls review |
Keep both PR's AudioBatch→AudioTask rename changes and upstream's new release note sections (metrics, workflows, CVE fixes, bug fixes). For docs.yml redirects, add /nemo/curator/ prefixed versions with audio-task destination applied. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
|
||
| # Access duration information | ||
| duration = result_batch[0].data[0]["duration"] | ||
| duration = result_batch[0].data["duration"] |
There was a problem hiding this comment.
GetAudioDurationStage.process() returns a single AudioTask, not a list. Three code blocks index the result as result_batch[0] or iterate with for result in result_batches:, both of which would raise TypeError at runtime.
Fix:
GetAudioDurationStage.process() returns a single AudioTask, not a list. Three code blocks index the result as result_batch[0] or iterate with for result in result_batches:, both of which would raise TypeError at runtime.
Fix also occurrences in
Occurrence 2 (lines 104-109):
WRONG: same issue
result_batch = duration_stage.process(audio_task)
processed_sample = result_batch[0].data # TypeError
Fix:
result = duration_stage.process(audio_task)
processed_sample = result.data
Occurrence 3 (lines 164-168):
WRONG: iterating over AudioTask yields nothing useful
result_batches = duration_stage.process(audio_task)
for result in result_batches: # iterates over AudioTask attributes, not tasks
Fix:
result = duration_stage.process(audio_task)
print(f"File: {result.data['audio_filepath']}")
print(f"Duration: {result.data['duration']:.3f} seconds")
| import json | ||
|
|
||
| # Create simple manifest | ||
| manifest_data = [ |
There was a problem hiding this comment.
The docs code on lines 84-97 defines manifest_data as a list of 2 dicts, then lines 111-113 pass the whole list to AudioTask(data=manifest_data). Any user following this will get the ValueError above.
Fix: Show per-entry construction:
for entry in manifest_data:
audio_task = AudioTask(data=entry, filepath_key="audio_filepath")
is_valid = audio_task.validate()
print(f"Task validation: {is_valid}")
| # ASR stage processes AudioTask objects automatically | ||
| # The stage extracts file paths and calls transcribe() internally | ||
| processed_batch = asr_stage.process(audio_batch) | ||
| processed_batch = asr_stage.process(audio_task) |
There was a problem hiding this comment.
InferenceAsrNemoStage.process() raises NotImplementedError.
Users should never call process() directly on ASR stages — the pipeline executor handles dispatch via process_batch().
Fix — either show process_batch() directly:
results = asr_stage.process_batch([audio_task])
Or (better) show the executor-driven pattern:
Don't call process() directly — the Pipeline/Executor handles dispatch:
pipeline.add_stage(asr_stage)
results = pipeline.run(executor)
| @@ -104,17 +104,17 @@ asr_stage = asr_stage.with_( | |||
| <Note> | |||
| `batch_size` controls the number of tasks the executor groups per call. The ASR stage does not define `process_batch()`; the executor batches tasks. | |||
There was a problem hiding this comment.
InferenceAsrNemoStage does define process_batch() as its canonical method (line 111 of asr_nemo.py)
process() explicitly raises NotImplementedError
Fix:
batch_size controls the number of tasks the executor groups per call. The ASR stage defines process_batch() as its canonical method — the executor groups tasks by batch_size before calling it.
|
|
||
| - `AudioBatch` automatically validates file paths during creation | ||
| - `AudioTask` automatically validates file paths during creation | ||
| - Use `validate()` for batch-level validation |
| `batch_size` controls the number of tasks the executor groups per call. The ASR stage does not define `process_batch()`; the executor batches tasks. | ||
|
|
||
| Within a single `AudioBatch`, `process()` transcribes the file paths together. | ||
| Within a single `AudioTask`, `process()` transcribes the file paths together. |
|
|
||
| # Filter out entries that do not exist on disk | ||
| valid_samples = [item for item in audio_batch.data if audio_batch.validate_item(item)] | ||
| valid_samples = [item for item in audio_task.data if audio_task.validate_item(item)] |
|
|
||
| # Filter out entries that do not exist on disk | ||
| valid_samples = [item for item in audio_batch.data if audio_batch.validate_item(item)] | ||
| valid_samples = [item for item in audio_task.data if audio_task.validate_item(item)] |
There was a problem hiding this comment.
audio_task.data is a dict (specifically _AttrDict). Iterating over a dict yields its keys — strings like "audio_filepath", "text", "duration" — not manifest entries. This is holdover from AudioBatch.data which was list[dict].
The comprehension would produce something like ["audio_filepath", "text"] (keys that happen to pass validate_item), which is nonsensical.
Since AudioTask is single-entry, validation is simply:
is_valid = audio_task.validate()
Address review comments from #1694: - Fix process() return type in duration-calculation.mdx (returns single AudioTask, not a list) - Fix AudioTask construction in manifests-ingest.mdx (single dict per task, not a list) - Update "batch-level validation" wording to match single-entry model - Correct asr-inference/index.mdx to state ASR stage defines process_batch() as its canonical method - Fix "file paths" plural to singular for single-entry AudioTask - Replace invalid list comprehension over AudioTask.data dict keys in asr-pipeline.mdx with validate() call - Replace direct process() call on ASR stage with pipeline-driven pattern Signed-off-by: Lawrence Lane <llane@nvidia.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep both sides in release-notes/index.mdx: - Dependency Updates: use upstream's more detailed Ray description, keep pynvml from our branch, add sentence-transformers and vllm - Breaking Changes: keep AudioBatch removal note from our branch, add RayDataExecutor, DocumentExtractStage, DocumentIterateStage, and Three-Stage Pipeline entries from upstream Signed-off-by: Lawrence Lane <llane@nvidia.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Description
Renames the
AudioBatchclass and concept toAudioTaskthroughout the 26.04 audio curation documentation to reflect the upstream API change. Updates navigation, URL redirects, concept pages, API reference, tutorials, and adds a release note entry for the rename.Checklist