docs: address 9 open documentation issues - #1722
Merged
Merged
Conversation
…o#1326, NVIDIA-NeMo#1546, NVIDIA-NeMo#1547, NVIDIA-NeMo#1548, NVIDIA-NeMo#1549, NVIDIA-NeMo#1550, NVIDIA-NeMo#1551, NVIDIA-NeMo#1552) - Fix broken Installation and Configuration links in tutorials/README.md (NVIDIA-NeMo#1264) - Document pip install dependency conflict and recommend uv (NVIDIA-NeMo#1326) - Clarify that Curator uses Ray (not Dask) in migration guide and about page (NVIDIA-NeMo#1546) - Add architecture diagram section to README (NVIDIA-NeMo#1547) - Add Nemotron dataset usage section to README (NVIDIA-NeMo#1548) - Add data curation importance section to README (NVIDIA-NeMo#1549) - Add deep-dives section to fern docs (resource allocation, streaming, auto-balancing, throughput) (NVIDIA-NeMo#1550) - Add citation section to README (NVIDIA-NeMo#1551) - Add Updates/News section to README (NVIDIA-NeMo#1552) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
Images now live in fern/assets/images/ and README references updated. Reverted docs/about/release-notes/migration-guide.md and docs/admin/installation.md since docs/ is deprecated in favor of fern/. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
Update resource allocation, streaming, and auto-balancing deep-dives with concrete details from internal slides: ~5x CPU stage speedup, 20% streaming overlap improvement, before/after auto-balancing example (1 vs 4 videos/s), and code examples matching actual API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
Add illustrative pipeline example from UCB lecture (lang ID → tokenization → 5B model, 13,000s naive → ~1,000s with Curator) to the "Why Data Curation?" section and throughput deep-dive. Update auto-balancing with accurate tasks/s numbers and streaming with 99% GPU utilization stat. Framed as illustrative, not a benchmark claim. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
These are concept articles, not a new content type. Moved resource allocation, streaming, auto-balancing, and throughput pages under About > Concepts > Scaling & Performance. Removed the separate Deep Dives nav section and index page. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
sarahyurick
reviewed
Apr 14, 2026
sarahyurick
left a comment
sarahyurick
left a comment
Contributor
There was a problem hiding this comment.
This is great, thanks!
Contributor
There was a problem hiding this comment.
Can we get the image without the red underline on DocumentBatch?
Comment on lines
+28
to
+35
| class TokenizerStage(ProcessingStage[DocumentBatch, DocumentBatch]): | ||
| def __init__(self): | ||
| super().__init__() | ||
|
|
||
| @property | ||
| def resources(self) -> Resources: | ||
| return Resources(cpus=1) # CPU-only — no GPU needed | ||
| # ... stage logic ... |
Contributor
There was a problem hiding this comment.
Nit but we don't want users setting the resources property themselves. The snippet should probably look more like:
class TokenizerStage(ProcessingStage[DocumentBatch, DocumentBatch]):
name: str = "TokenizerStage"
resources: Resources = Resources(cpus=1.0)
def __init__(self):
...
Comment on lines
+64
to
+74
| ### 1. Add More Nodes | ||
|
|
||
| The most straightforward way to increase throughput. When you add nodes to your Ray cluster, the executor automatically distributes pipeline stages across the expanded cluster. | ||
|
|
||
| ```bash | ||
| # Start head node | ||
| ray start --head --port=6379 | ||
|
|
||
| # Add worker nodes (run on each additional machine) | ||
| ray start --address=<head-node-ip>:6379 | ||
| ``` |
Contributor
There was a problem hiding this comment.
Is this how we should always do it? Maybe for now, we just reference RayClient and RaySlurmClient?
Comment on lines
+96
to
+99
| # CPU-heavy: no GPU needed | ||
| filter_stage = ScoreFilter( | ||
| filter_obj=WordCountFilter(min_words=80), | ||
| ) |
Contributor
There was a problem hiding this comment.
Maybe add a comment above that ScoreFilter by default uses Resources(cpus=1).
|
|
||
| Use Ray Dashboard and stage performance stats to identify bottlenecks. Common issues: | ||
|
|
||
| - **I/O bound reader**: Increase reader parallelism or use faster storage (NVMe, parallel file system). |
Contributor
There was a problem hiding this comment.
Is there a specific example you had in mind here? Not sure I instead the suggestion here.
| Use Ray Dashboard and stage performance stats to identify bottlenecks. Common issues: | ||
|
|
||
| - **I/O bound reader**: Increase reader parallelism or use faster storage (NVMe, parallel file system). | ||
| - **Single slow stage**: Check if the stage can use more GPU memory or workers. |
Contributor
There was a problem hiding this comment.
To be clear, Curator will auto balance to assign more workers to slower stages. But agree it is good for the user to assign more GPU resources to a stage right off the bat if it needs it.
- resource-allocation: Use class-level resources attr, fix .with_() API, replace fractional GPU example with Resources(gpus=0.25), remove unsourced 5x claim, remove deduplication from CPU-bound example - streaming: Remove unsourced 20% claim, use quickstart-style example, soften batch size trade-off language - throughput: Clarify GPU memory math in example, emphasize streaming, use RayClient/RaySlurmClient, add ScoreFilter default note, note auto-balancing behavior - auto-balancing: Remove untested stage_stats snippet, reference Ray Dashboard instead - tutorials/README: Add .html extensions to doc URLs for consistency Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @@ -133,3 +175,29 @@ Data curation modules measurably improve model performance. In ablation studies | |||
| ## Contribute | |||
|
|
|||
| We welcome community contributions! Please refer to [CONTRIBUTING.md](https://github.com/NVIDIA/NeMo/blob/stable/CONTRIBUTING.md) for guidelines. | |||
Contributor
There was a problem hiding this comment.
This link goes to NVIDIA/NeMo (the NeMo framework repo), not NVIDIA-NeMo/Curator. Contributors following it will land on the NeMo framework's contribution guide, which has different requirements. The correct URL is used in the PR checklist itself.
Suggested change
| We welcome community contributions! Please refer to [CONTRIBUTING.md](https://github.com/NVIDIA/NeMo/blob/stable/CONTRIBUTING.md) for guidelines. | |
| We welcome community contributions! Please refer to [CONTRIBUTING.md](https://github.com/NVIDIA-NeMo/Curator/blob/main/CONTRIBUTING.md) for guidelines. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Addresses all open documentation issues assigned to @lbliii in a single PR. Adds new README sections (Updates, Architecture, Why Data Curation with throughput example from Berkeley lecture, Nemotron usage, Citation), fixes broken tutorial links, clarifies Dask→Ray migration, documents pip install limitations, and adds four new Scaling & Performance concept pages (resource allocation, streaming, auto-balancing, throughput) to the fern docs.
Closes #1264, #1326, #1546, #1547, #1548, #1549, #1550, #1551, #1552
Usage
# N/A — documentation-only changes.Checklist