docs: update docs and release notes for fused iterate-extract (PR #1458)

fern/versions/v26.04/pages/about/concepts/text/data-acquisition-concepts.mdx

@@ -14,22 +14,21 @@ This guide covers the core concepts for acquiring and processing text data from
 
 ## Overview
 
-Data acquisition in NeMo Curator follows a four-stage architecture:
+Data acquisition in NeMo Curator follows a three-stage architecture:
 
-1. **Generate URLs**: Discover and generate download URLs from minimal input
-2. **Download**: Retrieve raw data files from remote sources
-3. **Iterate**: Extract individual records from downloaded containers
-4. **Extract**: Convert raw content to clean, structured text
+1. **Generate URLs**: Discover and generate download URLs from minimal input.
+2. **Download**: Retrieve raw data files from remote sources.
+3. **Iterate** and **Extract**: Extract individual records from downloaded containers and convert raw content to clean, structured text.
 
 This process transforms diverse remote data sources into a standardized `DocumentBatch` that can be used throughout the text curation pipeline.
 
 ## Core Components
 
-The data acquisition framework consists of four abstract base classes that define the acquisition workflow:
+The data acquisition framework consists of abstract base classes that define the acquisition workflow:
 
 ### URLGenerator
 
-Generates URLs for downloading from minimal input configuration. You need to override `generate_urls` which generates a bunch of URLs that user wants to download.
+Generates URLs for downloading from minimal input configuration. Override the `generate_urls` method, which returns a list of URLs for the user to download.
 
 **Example Implementation**:
 
@@ -48,7 +47,7 @@ class CustomURLGenerator(URLGenerator):
 
 ### DocumentDownloader
 
-Connects to and downloads data from remote repositories. You must override `_get_output_filename` and `_download_to_path` which are called by an underlying function called `download` which tries to be idempotent.
+Connects to and downloads data from remote repositories. Override `_get_output_filename` and `_download_to_path`, which are called by the underlying `download` function, designed to be idempotent.
 
 **Example Implementation**:
 
@@ -75,7 +74,7 @@ class CustomDownloader(DocumentDownloader):
 
 ### DocumentIterator
 
-Extracts individual records from downloaded containers. You should only override `iterate` and `output_columns` where `iterate` must have logic to load the local file path and return bunch of documents. The `list[dict]` is finally considered to a Pandas DataFrame which is passed to Extractor.
+Extracts individual records from downloaded containers. Override `iterate` to load the file at the given path and yield records, and override `output_columns` to declare the output schema. Records are passed directly to the extractor (if provided) inline during iteration.
 
 **Example Implementation**:
 
@@ -100,7 +99,7 @@ class CustomIterator(DocumentIterator):
 
 ### DocumentExtractor (Optional)
 
-DocumentExtractor works on a Pandas DataFrame and is optional.
+DocumentExtractor transforms individual records and is optional. When provided to `DocumentIterateExtractStage`, it processes each record inline during iteration rather than as a separate stage.
 
 **Example Implementation**:
 
@@ -238,17 +237,17 @@ ray_client.stop()
 
 Data acquisition leverages distributed computing frameworks for scalable processing:
 
-- **Parallel Downloads**: Each URL in the generated list downloads through separate workers
-- **Concurrent Extraction**: Files process in parallel across workers
-- **Memory Management**: Streaming processing for large files
+- **Parallel Downloads**: Each URL in the generated list downloads through separate workers.
+- **Concurrent Extraction**: Files process in parallel across workers.
+- **Memory Management**: Streaming processing for large files.
 
 ## Integration with Data Loading
 
-Data acquisition produces a standardized output that integrates seamlessly with Curator's [Data Loading Concepts ](/about/concepts/text/data/loading):
+Data acquisition produces a standardized output that integrates seamlessly with Curator's [Data Loading Concepts](/about/concepts/text/data/loading):
 
-<Note>
+:::{note}
 Data acquisition includes basic content-level deduplication during extraction (such as removing duplicate HTML content within individual web pages). This is separate from the main deduplication pipeline stages (exact, fuzzy, and semantic deduplication) that operate on the full dataset after acquisition.
-</Note>
+:::
 
 ```python
 from nemo_curator.stages.text.io.writer import ParquetWriter
@@ -274,7 +273,7 @@ load_pipeline.add_stage(reader)
 
 This enables you to:
 
-- **Separate acquisition from processing** for better workflow management
-- **Cache acquired data** to avoid re-downloading
-- **Mix acquired and local data** in the same processing pipeline
-- **Use standard loading patterns** regardless of data origin
+- **Separate acquisition from processing** for better workflow management.
+- **Cache acquired data** to avoid re-downloading.
+- **Mix acquired and local data** in the same processing pipeline.
+- **Use standard loading patterns** regardless of data origin.

fern/versions/v26.04/pages/about/release-notes/index.mdx

@@ -57,6 +57,14 @@ Upgraded Cosmos-Xenna from 0.1.2 to 0.2.0 with a simplified resource model and i
 - **Xenna-managed CUDA devices**: Xenna now manages CUDA device visibility directly, replacing the previous Ray-managed approach.
 - **Ray 2.54**: Updated Ray dependency to version 2.54 for compatibility with Cosmos-Xenna 0.2.0.
 
+### Fused Document Iterate and Extract Stages (PR #1458)
+
+The data acquisition pipeline now uses a three-stage architecture instead of four, fusing the iterate and extract steps into a single `DocumentIterateExtractStage`. This reduces memory overhead and improves pipeline performance:
+
+- **Fused `DocumentIterateExtractStage`**: Combines `DocumentIterateStage` and `DocumentExtractStage` into a single stage that iterates through downloaded files and extracts structured content in one pass.
+- **Improved Memory Efficiency**: The fused stage processes records inline instead of materializing intermediate DataFrames, reducing peak memory usage. With limited RAM (200 GB), the Common Crawl pipeline succeeds with 32 CPUs where the unfused pipeline ran out of memory even at 16 CPUs.
+- **Better Performance**: Benchmarks show faster runtimes across both the Ray Data and Xenna executors (e.g., ~6% faster with Ray Data, ~18% faster with Xenna).
+
 ### Pipeline Stage Metrics (PR #1385)
 
 Pipeline stages now track document-level metrics through `StagePerfStats.num_items_processed`, so you can see how each stage affects your dataset. After calling `pipeline.run()`, the returned task objects expose per-stage document counts that you can use to monitor filtering behavior and tune thresholds.
@@ -141,6 +149,9 @@ Fixed a race condition in `CaptionGenerationStage` and `CaptionEnhancementStage`
 
 - **`TextSemanticDeduplicationWorkflow` embedding backend**: The default embedding backend changed from SentenceTransformers to vLLM. The default model changed from `sentence-transformers/all-MiniLM-L6-v2` to `google/embeddinggemma-300m`. The parameters `embedding_model_inference_batch_size`, `embedding_pooling`, `embedding_padding_side`, and `embedding_max_seq_length` have been removed. Use `embedding_vllm_init_kwargs` to pass configuration to the vLLM backend instead.
 - **`Resources` API**: The `nvdecs`, `nvencs`, and `entire_gpu` fields have been removed from `Resources`. Stages that previously used `entire_gpu=True` should use `gpus=1` instead. Stages that used `nvdecs` or `nvencs` should use `gpus` for GPU allocation.
+- **`DocumentExtractStage` Removed**: The standalone `DocumentExtractStage` class has been removed. Use `DocumentIterateExtractStage` with an optional `extractor` parameter instead. The `DocumentExtractor` abstract base class is unchanged.
+- **`DocumentIterateStage` Renamed**: `DocumentIterateStage` has been replaced by `DocumentIterateExtractStage`. Update imports from `nemo_curator.stages.text.download.base.iterator`.
+- **Three-Stage Pipeline**: The data acquisition pipeline is now a three-step pattern (URL generation → download → iterate-extract) instead of four steps.
 - **`ExactDeduplicationWorkflow.run()` and `FuzzyDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `None`
 - **`SemanticDeduplicationWorkflow.run()` and `TextSemanticDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `dict`
 - **`TextDuplicatesRemovalWorkflow.run()`** now returns `WorkflowRunResult` instead of `list[FileGroupTask] | None`

fern/versions/v26.04/pages/curate-text/load-data/custom.mdx

@@ -14,15 +14,11 @@ Create custom data loading pipelines using Curator. This guide shows how to buil
 
 ## How It Works
 
-Curator uses the same **4-step pipeline pattern** described in [Data Acquisition Concepts ](/about/concepts/text/data/acquisition) for custom data loading. Each step uses an abstract base class with corresponding processing stages that compose into pipelines.
-
----
+Curator uses the same **3-step pipeline pattern** described in [Data Acquisition Concepts](/about/concepts/text/data/acquisition) for custom data loading. Each step maps to one or more abstract base classes with corresponding processing stages that compose into pipelines.
 
 ## Architecture Overview
 
-For detailed information about the core components and data flow, see [Data Acquisition Concepts ](/about/concepts/text/data/acquisition) and [Data Loading Concepts ](/about/concepts/text/data/loading).
-
----
+For detailed information about the core components and data flow, see [Data Acquisition Concepts](/about/concepts/text/data/acquisition) and [Data Loading Concepts](/about/concepts/text/data/loading).
 
 ## Implementation Guide
 
@@ -196,8 +192,6 @@ class CustomDataStage(DocumentDownloadExtractStage):
         return "Custom data"
 ```
 
----
-
 ## Usage Examples
 
 ### Basic Pipeline
@@ -240,8 +234,6 @@ if __name__ == "__main__":
 
 For executor options and configuration, refer to [Execution Backends](/reference/infra/execution-backends).
 
----
-
 ## Parameters Reference
 
 | Parameter | Type | Description | Default |
@@ -254,8 +246,6 @@ For executor options and configuration, refer to [Execution Backends](/reference
 | `record_limit` | int \| None | Maximum records per file | None |
 | `add_filename_column` | bool \| str | Add filename column to output; if str, uses it as the column name (default name: "file_name") | True |
 
----
-
 ## Output Format
 
 Processed data flows through the pipeline as `DocumentBatch` tasks containing Pandas DataFrames or PyArrow Tables: