Skip to content

docs: fix ImageReaderStage batch_size parameter name in v26.04 docs #1837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lbliii merged 2 commits into from
Apr 24, 2026
Merged

docs: fix ImageReaderStage batch_size parameter name in v26.04 docs #1837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7e78865
docs: fix ImageReaderStage batch_size parameter name in v26.04 docs
lbliii Apr 20, 2026
796b0e4
Merge branch '26.04-staging' into lbliii/qa-docs-fixes
lbliii Apr 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion fern/versions/v26.04/pages/about/concepts/image/data-loading-concepts.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ pipeline.add_stage(FilePartitioningStage(

# Load JPEG images from tar files using DALI
pipeline.add_stage(ImageReaderStage(
batch_size=100, # Number of images per batch
dali_batch_size=100, # Number of images per batch
verbose=True,
num_threads=8, # Number of threads for I/O operations
num_gpus_per_worker=0.25, # Allocate 1/4 GPU per worker
Expand Down
2 changes: 1 addition & 1 deletion fern/versions/v26.04/pages/about/release-notes/migration-guide.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ In the new version, data loading is encapsulated in a dedicated pipeline stage (
```python
# New: Read images from webdataset tar files
read_stage = ImageReaderStage(
batch_size=args.batch_size,
dali_batch_size=args.batch_size,
num_threads=16,
num_gpus_per_worker=0.25,
)
Expand Down
6 changes: 3 additions & 3 deletions fern/versions/v26.04/pages/curate-images/load-data/tar-archives.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ pipeline.add_stage(FilePartitioningStage(

# Stage 2: Read JPEG images from tar files using DALI
pipeline.add_stage(ImageReaderStage(
batch_size=100,
dali_batch_size=100,
verbose=True,
num_threads=8,
num_gpus_per_worker=0.25,
Expand All @@ -79,7 +79,7 @@ results = pipeline.run()

- `file_paths`: Path to directory containing tar files
- `files_per_partition`: Number of tar files to process per partition (controls parallelism)
- `batch_size`: Number of images per ImageBatch for processing
- `dali_batch_size`: Number of images per ImageBatch for processing

---

Expand Down Expand Up @@ -116,7 +116,7 @@ The `ImageReaderStage` is the core component that handles tar archive loading wi

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `batch_size` | int | 100 | Number of images per ImageBatch for processing |
| `dali_batch_size` | int | 100 | Number of images per ImageBatch for processing |
| `verbose` | bool | True | Enable verbose logging for debugging |
| `num_threads` | int | 8 | Number of threads for DALI operations |
| `num_gpus_per_worker` | float | 0.25 | GPU allocation per worker (0.25 = 1/4 GPU) |
Expand Down
2 changes: 1 addition & 1 deletion fern/versions/v26.04/pages/curate-images/process-data/embeddings/clip-embedder.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ pipeline.add_stage(FilePartitioningStage(

# Stage 2: Read images
pipeline.add_stage(ImageReaderStage(
batch_size=100,
dali_batch_size=100,
num_threads=8,
num_gpus_per_worker=0.25,
))
Expand Down
2 changes: 1 addition & 1 deletion fern/versions/v26.04/pages/curate-images/process-data/filters/aesthetic.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ pipeline.add_stage(FilePartitioningStage(

# Stage 2: Read images
pipeline.add_stage(ImageReaderStage(
batch_size=100,
dali_batch_size=100,
num_gpus_per_worker=0.25,
))

Expand Down
2 changes: 1 addition & 1 deletion fern/versions/v26.04/pages/curate-images/process-data/filters/nsfw.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ pipeline.add_stage(FilePartitioningStage(

# Stage 2: Read images
pipeline.add_stage(ImageReaderStage(
batch_size=100,
dali_batch_size=100,
num_gpus_per_worker=0.25,
))

Expand Down
4 changes: 2 additions & 2 deletions fern/versions/v26.04/pages/curate-images/tutorials/beginner.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Load images from tar archives and extract metadata.

```python
pipeline.add_stage(ImageReaderStage(
batch_size=100, # Images per batch
dali_batch_size=100, # Images per batch
verbose=True,
num_threads=16, # I/O threads
num_gpus_per_worker=0.25,
Expand Down Expand Up @@ -215,7 +215,7 @@ def create_image_curation_pipeline():
))

pipeline.add_stage(ImageReaderStage(
batch_size=100,
dali_batch_size=100,
verbose=True,
num_threads=16,
num_gpus_per_worker=0.25,
Expand Down
8 changes: 4 additions & 4 deletions fern/versions/v26.04/pages/curate-images/tutorials/dedup-workflow.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ def create_image_embedding_pipeline(input_dir, embeddings_dir, model_dir):

# Read images from tar archives
pipeline.add_stage(ImageReaderStage(
batch_size=100,
dali_batch_size=100,
verbose=True,
num_threads=16,
num_gpus_per_worker=0.25,
Expand Down Expand Up @@ -207,7 +207,7 @@ def create_image_removal_pipeline(input_dir, removal_dir, output_dir):

# Read original images
pipeline.add_stage(ImageReaderStage(
batch_size=100,
dali_batch_size=100,
verbose=True,
num_threads=16,
num_gpus_per_worker=0.25,
Expand Down Expand Up @@ -324,7 +324,7 @@ def run_image_deduplication_workflow():
file_paths=INPUT_TAR_DIR, files_per_partition=1, file_extensions=[".tar"]
))
embedding_pipeline.add_stage(ImageReaderStage(
batch_size=100, verbose=True, num_threads=16, num_gpus_per_worker=0.25
dali_batch_size=100, verbose=True, num_threads=16, num_gpus_per_worker=0.25
))
embedding_pipeline.add_stage(ImageEmbeddingStage(
model_dir=MODEL_DIR, num_gpus_per_worker=0.25,
Expand Down Expand Up @@ -360,7 +360,7 @@ def run_image_deduplication_workflow():
file_paths=INPUT_TAR_DIR, files_per_partition=1, file_extensions=[".tar"]
))
removal_pipeline.add_stage(ImageReaderStage(
batch_size=100, verbose=True, num_threads=16, num_gpus_per_worker=0.25
dali_batch_size=100, verbose=True, num_threads=16, num_gpus_per_worker=0.25
))
removal_pipeline.add_stage(ImageDuplicatesRemovalStage(
removal_parquets_dir=REMOVAL_DIR + "/duplicates",
Expand Down
4 changes: 2 additions & 2 deletions fern/versions/v26.04/pages/get-started/image.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ Here's a simple example to get started with NeMo Curator's image curation pipeli
**CPU Memory Considerations**

Image loading and decoding happens in CPU memory before GPU processing. If you encounter out-of-memory errors during the `ImageReaderStage`, reduce:
- `batch_size`: Number of images per batch (reduce to 32-50 for systems with limited RAM)
- `dali_batch_size`: Number of images per batch (reduce to 32-50 for systems with limited RAM)
- `num_threads`: Parallel decoding threads (reduce to 4 for systems with limited RAM)
- `num_cpus`: Ray Client CPU allocation (reduce to 8-16 for systems with limited RAM)

Expand Down Expand Up @@ -152,7 +152,7 @@ pipeline.add_stage(FilePartitioningStage(

# Stage 2: Read images from tar files using DALI
pipeline.add_stage(ImageReaderStage(
batch_size=50,
dali_batch_size=50,
verbose=True,
num_threads=4,
num_gpus_per_worker=0.25,
Expand Down
Loading