docs: address 9 open documentation issues

README.md

@@ -15,6 +15,11 @@
 
 > *Part of the [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/products/nemo/) software suite for managing the AI agent lifecycle.*
 
+## Updates
+
+- 2026-04: NeMo Curator 26.04 released with Cosmos-Xenna 0.2.0 upgrade, simplified `Resources` API, and Ray 2.54. See the [release notes](https://docs.nvidia.com/nemo/curator/latest/about/release-notes).
+- 2026-02: NeMo Curator 26.02 released with Ray-based pipeline architecture for all modalities — text, image, video, and audio.
+
 ## What You Can Do
 
 | Modality | Key Capabilities | Get Started |
@@ -38,6 +43,16 @@ python tutorials/quickstart.py
 
 ---
 
+## Architecture
+
+NeMo Curator uses a modular, Ray-based pipeline architecture. Data flows through composable processing stages — each stage handles a discrete curation task (loading, filtering, deduplication, etc.) and can be configured with independent resource requirements.
+
+<p align="center">
+  <img src="./fern/assets/images/architecture-diagram.png" alt="NeMo Curator architecture diagram showing modular pipeline stages" width="700"/>
+</p>
+
+---
+
 ## Features by Modality
 
 ### Text Curation
@@ -92,6 +107,33 @@ Prepare high-quality speech datasets for automatic speech recognition (ASR) and
 
 ---
 
+## Why Data Curation?
+
+High-quality training data is the single most important factor in building performant AI models. Raw datasets contain noise, duplicates, low-quality content, and potentially harmful material that degrade model performance and increase training costs.
+
+<p align="center">
+  <img src="./fern/assets/images/data-curation-challenges.png" alt="Common data curation challenges: quality, deduplication, filtering, and scale" width="700"/>
+</p>
+
+At scale, data curation is a **throughput maximization problem**. A typical pipeline chains stages with very different compute profiles — lightweight CPU tokenization, small GPU classifiers, large GPU inference models — and a naive sequential approach leaves most hardware idle most of the time.
+
+**Example:** Consider a pipeline with language identification (0.5B model, 1 GB VRAM, 2s/sample), tokenization (CPU-only, 1s/sample), and a 5B answer model (10 GB VRAM, 10s/sample) processing 1,000 questions on a single 102 GB GPU:
+
+| Approach | How it works | Total runtime |
+|----------|-------------|---------------|
+| **Sequential** | Process each sample through all stages, one at a time | ~13,000 seconds |
+| **NeMo Curator** | Stream batches, auto-scale replicas per stage, overlap CPU/GPU work | ~1,000 seconds |
+
+NeMo Curator achieves this by streaming data through the pipeline so all stages run concurrently, auto-balancing replicas to match each stage's throughput (2× language ID, 1× tokenizer, 10× answer model), and keeping GPU workers busy over 99% of the time after an initial warm-up period. See the [scaling concepts](https://docs.nvidia.com/nemo/curator/latest/about/concepts/scaling) for details.
+
+---
+
+## Proven at Scale: Nemotron
+
+NeMo Curator powers the data pipelines behind [NVIDIA Nemotron](https://developer.nvidia.com/nemotron) models. For example, the [Nemotron-4 pre-training dataset](https://arxiv.org/abs/2402.16819) was curated using NeMo Curator's text processing pipeline across 8+ trillion tokens of multilingual web data, applying quality filtering, deduplication, and domain classification at scale.
+
+---
+
 ## Why NeMo Curator?
 
 ### Performance at Scale
@@ -104,15 +146,15 @@ NeMo Curator leverages NVIDIA RAPIDS™ libraries such as cuDF, cuML, and cuGrap
 - **Near-linear scaling** from one to four H100 80 GB nodes (2.05 hrs → 0.50 hrs)
 
 <p align="center">
-  <img src="./docs/_images/text-benchmarks.png" alt="Performance benchmarks showing 16x speed improvement, 40% cost savings, and near-linear scaling" width="700"/>
+  <img src="./fern/assets/images/text-benchmarks.png" alt="Performance benchmarks showing 16x speed improvement, 40% cost savings, and near-linear scaling" width="700"/>
 </p>
 
 ### Quality Improvements
 
 Data curation modules measurably improve model performance. In ablation studies using a 357M-parameter GPT model trained on curated Common Crawl data:
 
 <p align="center">
-  <img src="./docs/_images/ablation.png" alt="Model accuracy improvements across curation pipeline stages" width="700"/>
+  <img src="./fern/assets/images/ablation.png" alt="Model accuracy improvements across curation pipeline stages" width="700"/>
 </p>
 
 **Results:** Progressive improvements in zero-shot downstream task performance through text cleaning, deduplication, and quality filtering stages.
@@ -133,3 +175,29 @@ Data curation modules measurably improve model performance. In ablation studies
 ## Contribute
 
 We welcome community contributions! Please refer to [CONTRIBUTING.md](https://github.com/NVIDIA/NeMo/blob/stable/CONTRIBUTING.md) for guidelines.
+
+---
+
+## Citation
+
+If you find NeMo Curator useful in your research, please cite:
+
+```bibtex
+@misc{nemo_curator,
+  title = {NeMo Curator: GPU-Accelerated Data Curation for Training AI Models},
+  author = {NVIDIA},
+  year = {2024},
+  url = {https://github.com/NVIDIA-NeMo/Curator}
+}
+```
+
+For the data curation pipeline behind Nemotron models, please also cite:
+
+```bibtex
+@article{parmar2024nemotron4,
+  title = {Nemotron-4 15B Technical Report},
+  author = {Parmar, Jupinder and Satheesh, Shrimai and others},
+  journal = {arXiv preprint arXiv:2402.16819},
+  year = {2024}
+}
+```

fern/versions/v26.04.yml

@@ -33,6 +33,21 @@ navigation:
               - page: Deduplication
                 path: ./v26.04/pages/about/concepts/deduplication.mdx
                 slug: deduplication
+              - section: Scaling & Performance
+                slug: scaling
+                contents:
+                  - page: Resource Allocation
+                    path: ./v26.04/pages/about/concepts/resource-allocation.mdx
+                    slug: resource-allocation
+                  - page: Streaming
+                    path: ./v26.04/pages/about/concepts/streaming.mdx
+                    slug: streaming
+                  - page: Auto-Balancing
+                    path: ./v26.04/pages/about/concepts/auto-balancing.mdx
+                    slug: auto-balancing
+                  - page: Throughput
+                    path: ./v26.04/pages/about/concepts/throughput.mdx
+                    slug: throughput
               - section: Text Concepts
                 slug: text
                 contents:

fern/versions/v26.04/pages/about/concepts/auto-balancing.mdx

@@ -0,0 +1,45 @@
+---
+description: "How NeMo Curator automatically balances resources across pipeline stages"
+categories: ["architecture"]
+tags: ["deep-dive", "auto-balancing", "scheduling", "performance"]
+personas: ["data-scientist-focused", "mle-focused"]
+difficulty: "intermediate"
+content_type: "concept"
+modality: "universal"
+---
+
+# Auto-Balancing Heterogeneous Models
+
+NeMo Curator auto-balances resources at the application level across pipeline stages to maximize throughput. This means you can focus on defining your curation logic rather than manually tuning parallelism.
+
+## The Problem: Unbalanced Pipelines
+
+In a typical curation pipeline, stages have very different processing speeds. Consider a video pipeline with a fast stage, a slow stage, and a medium stage — all sharing a fixed GPU budget:
+
+**Without auto-balancing (3 GPUs, 1 worker per stage):**
+- The **fast stage** emits 4 tasks/s, but the **slow stage** can only handle 1 task/s. Jobs back up in the queue — 3 tasks accumulate per second, eventually causing memory pressure.
+- The **medium stage** produces 2 tasks/s but is limited by the slow stage upstream, so it's starved for work. In practice only ~1 task/s is realized.
+- 4 queued tasks per second could have been processed, but the final stage is starved.
+
+**With auto-balancing (7 GPUs, scaled workers):**
+- The executor detects the bottleneck and scales the **slow stage to 4× workers** and the **medium stage to 2× workers**.
+- Now every stage sustains **4 tasks/s** throughput. Queues stay relatively clear — new jobs are picked up promptly.
+- **Result: 4× throughput improvement** by intelligently redistributing the same GPU budget.
+
+## How Auto-Balancing Works
+
+The executor monitors the throughput and queue depth of each stage at runtime and uses this information to:
+
+1. **Monitor throughput of different stages** and rebalance resources at regular intervals, shifting GPU/CPU allocations toward bottleneck stages.
+2. **Apply backpressure.** When a downstream stage can't keep up, upstream stages slow their output rate rather than buffering unbounded data in memory. This reduces memory pressure and prevents spilling to disk.
+3. **Scale workers dynamically.** If a stage is falling behind, the executor allocates additional workers to that stage (within the available resource budget).
+
+## What This Means for You
+
+- **No manual parallelism tuning.** You don't need to calculate the optimal number of workers per stage — the executor adapts at runtime.
+- **Predictable memory usage.** Backpressure prevents unbounded buffering, so memory usage stays stable even with unbalanced stages.
+- **Efficient hardware utilization.** Resources shift toward the current bottleneck instead of being statically allocated.
+
+## Monitoring Stage Balance
+
+Use the Ray Dashboard to monitor how the executor is balancing your pipeline. If you notice a persistent bottleneck that auto-balancing can't resolve (for example, a stage that needs more GPU memory than is available), consider splitting the pipeline or scaling your cluster.

fern/versions/v26.04/pages/about/concepts/index.mdx

@@ -45,8 +45,29 @@ Core concepts that apply across all modalities in NeMo Curator.
 
 <Cards>
 
-<Card title="Deduplication Concepts" href="/about/concepts/deduplication">
+<Card title="Deduplication" href="/about/concepts/deduplication">
 
 Comprehensive overview of deduplication techniques across text, image, and video modalities including exact, fuzzy, and semantic approaches.
 </Card>
+
+<Card title="Resource Allocation" href="/about/concepts/scaling/resource-allocation">
+
+How NeMo Curator allocates CPUs, GPUs, and memory across pipeline stages for optimal hardware utilization.
+</Card>
+
+<Card title="Streaming" href="/about/concepts/scaling/streaming">
+
+How streaming execution processes data in batches for constant memory usage and higher GPU utilization.
+</Card>
+
+<Card title="Auto-Balancing" href="/about/concepts/scaling/auto-balancing">
+
+How the executor automatically balances resources across pipeline stages to eliminate bottlenecks.
+</Card>
+
+<Card title="Throughput" href="/about/concepts/scaling/throughput">
+
+Scale from a single GPU to multi-node clusters with near-linear scaling.
+</Card>
+
 </Cards>

fern/versions/v26.04/pages/about/concepts/resource-allocation.mdx

@@ -0,0 +1,94 @@
+---
+description: "How NeMo Curator allocates CPUs, GPUs, and memory across pipeline stages"
+categories: ["architecture"]
+tags: ["deep-dive", "resources", "gpu", "cpu", "allocation"]
+personas: ["data-scientist-focused", "mle-focused"]
+difficulty: "intermediate"
+content_type: "concept"
+modality: "universal"
+---
+
+# Scaling Up with Ray Resource Allocation
+
+NeMo Curator makes resource allocation across pipeline stages straightforward — both CPUs and GPUs. Each stage declares its own resource requirements, and the executor schedules work accordingly.
+
+This design improves performance on CPU-only stages in pipelines that also use GPUs, because CPU stages no longer block GPU resources.
+
+## How It Works
+
+Every `ProcessingStage` can specify a `Resources` object that declares its CPU and GPU needs:
+
+```python
+from nemo_curator.stages.core import ProcessingStage
+from nemo_curator.stages.function_definitions import processing_stage
+from nemo_curator.stages.resources import Resources
+from nemo_curator.tasks import DocumentBatch
+
+
+class TokenizerStage(ProcessingStage[DocumentBatch, DocumentBatch]):
+    name: str = "TokenizerStage"
+    resources: Resources = Resources(cpus=1.0)  # CPU-only — no GPU needed
+
+    def __init__(self):
+        super().__init__()
+        # ... stage logic ...
+
+
+class ModelStage(ProcessingStage[DocumentBatch, DocumentBatch]):
+    name: str = "ModelStage"
+
+    def __init__(self, model_path: str):
+        super().__init__()
+        # ... stage logic ...
+        pass
+
+
+@processing_stage(name="custom_filter", resources=Resources(cpus=1))
+def custom_filter_stage(task: DocumentBatch) -> DocumentBatch:
+    # ... filter logic ...
+    pass
+
+
+model_stage = ModelStage(model_path="path/to/model").with_(resources=Resources(gpus=1))
+```
+
+When a pipeline runs, the executor reads each stage's resource declaration and schedules tasks to satisfy those constraints. Stages that need GPUs are placed on GPU-equipped nodes; CPU-only stages can run on any available worker.
+
+## Key Concepts
+
+### CPU-Only vs. GPU Stages
+
+The most impactful optimization is correctly separating CPU and GPU work. In a mixed pipeline, CPU-only stages (tokenization, text parsing, filtering) should not request GPU resources — this frees GPUs for inference stages that actually need them:
+
+```python
+# CPU-only: tokenization, filtering, I/O
+# Runs on any worker, doesn't block GPU resources
+@processing_stage(name="tokenizer", resources=Resources(cpus=1))
+def tokenizer_stage(task: DocumentBatch) -> DocumentBatch:
+    pass
+
+# GPU: model inference, embeddings
+# Scheduled only on GPU-equipped nodes
+model_stage = ModelStage(model_path="path/to/model").with_(resources=Resources(gpus=1))
+```
+
+### Fractional GPU Allocation
+
+Some GPU stages don't need an entire GPU. You can use fractional allocation via `Resources(gpus=0.25)` or reserve a specific amount of GPU memory with `Resources(gpu_memory_gb=10)`:
+
+```python
+# 4 workers share one GPU via fractional allocation
+model_stage = ModelStage(model_path="path/to/model").with_(resources=Resources(gpus=0.25))
+
+# Or reserve a specific amount of GPU memory
+model_stage = ModelStage(model_path="path/to/model").with_(resources=Resources(gpu_memory_gb=10))
+```
+
+This is useful for inference stages where the model fits in a fraction of GPU memory, allowing you to increase parallelism without requiring more hardware. Note that `gpu_memory_gb` sets GPU memory for a single GPU.
+
+## Best Practices
+
+- **Start with defaults.** Most stages have sensible default resource declarations. Override only when you observe resource contention or underutilization.
+- **Separate CPU and GPU stages.** This is the single highest-impact optimization — it allows the executor to parallelize across heterogeneous hardware.
+- **Profile before tuning.** Use Ray Dashboard or stage performance stats to identify bottlenecks before adjusting allocations.
+- **Match hardware to workload.** If your pipeline is mostly CPU-bound (text filtering), you may not need GPU nodes at all.

fern/versions/v26.04/pages/about/concepts/streaming.mdx

@@ -0,0 +1,69 @@
+---
+description: "How NeMo Curator's streaming architecture processes data in constant memory"
+categories: ["architecture"]
+tags: ["deep-dive", "streaming", "batching", "memory", "data-flow"]
+personas: ["data-scientist-focused", "mle-focused"]
+difficulty: "intermediate"
+content_type: "concept"
+modality: "universal"
+---
+
+# Streaming vs. Batch Processing
+
+Different inference stages have different compute requirements. NeMo Curator uses Ray streaming to increase GPU utilization and processing speed compared to traditional batch-all-at-once approaches.
+
+## Batch Mode vs. Streaming Mode
+
+### Batch Mode
+
+In batch mode, each stage processes the **entire dataset** before the next stage begins. Stages with different compute requirements (CPU-only tokenization, single-GPU classifiers, multi-GPU encoders) all run sequentially:
+
+| Aspect | Batch Mode |
+|--------|-----------|
+| **Execution** | One stage at a time across the full dataset |
+| **Memory usage** | Proportional to dataset size |
+| **GPU utilization** | Low — GPUs idle while CPU stages run, and vice versa |
+| **Time to first output** | After the entire pipeline finishes |
+
+### Streaming Mode
+
+In streaming mode, data flows through the pipeline as discrete batches. Each stage processes its current batch and immediately passes it downstream, so **all stages run concurrently** on different batches:
+
+| Aspect | Streaming Mode |
+|--------|---------------|
+| **Execution** | All stages active simultaneously on different batches |
+| **Memory usage** | Constant (proportional to batch size, not dataset size) |
+| **GPU utilization** | High — stages with different hardware needs overlap |
+| **Time to first output** | After the first batch completes the pipeline |
+
+## Why Streaming Is Faster
+
+Streaming with heterogeneous compute allows NeMo Curator to overlap stages that use different resources. For example, while a GPU inference stage processes batch N, a CPU tokenization stage can process batch N+1 simultaneously — neither blocks the other.
+
+This overlap improves throughput in pipelines that mix CPU and GPU work, because both happen in parallel rather than taking turns.
+
+Combined with auto-balancing, streaming enables Curator to rearrange resources so that GPU stage workers are **kept busy over 99% of the time** after an initial warm-up period.
+
+## Heterogeneous Executors
+
+NeMo Curator supports streaming with multiple executors — Cosmos Xenna, Ray Data, and others — each optimized for different workload patterns. The executor handles scheduling, backpressure, and resource allocation so that streaming "just works" regardless of how many stages your pipeline has.
+
+## Configuring Batch Size
+
+Batch size controls the trade-off between memory usage and throughput:
+
+```python
+from nemo_curator.stages.resources import Resources
+
+# Configure batch size on a stage
+word_count_stage = WordCountStage().with_(batch_size=128)
+```
+
+- **Smaller batches**: Lower memory usage per batch. Ray may handle smaller batches more efficiently in some workloads.
+- **Larger batches**: More in-memory data per batch, which can reduce I/O overhead but uses more memory.
+
+## When Streaming Matters Most
+
+- **Datasets exceed memory.** A 10 TB Common Crawl snapshot cannot fit in RAM, but it can be streamed in manageable chunks.
+- **Pipeline stages have different hardware needs.** CPU-only and GPU stages overlap instead of taking turns.
+- **You need early results.** Inspect output from the first batch while the rest of the dataset is still processing.