docs: update docs and release notes for fused iterate-extract (PR #1458) by lbliii · Pull Request #1684 · NVIDIA-NeMo/Curator

lbliii · 2026-03-30T19:05:40Z

Description

Updates v26.04 fern documentation and release notes to reflect the fused document iterate and extract stages from #1458. Replaces the 26.02 fern release notes with a 26.04 skeleton including the fused stages entry and breaking changes. Updates the data acquisition concepts page from four-stage to three-stage architecture, updates the custom data loading guide from 4-step to 3-step, and adds a 26.04 section to the non-fern release notes with matching content.

Usage

from nemo_curator.stages.text.download.base.iterator import DocumentIterateExtractStage

# Before (26.02): separate iterate and extract stages
# iterate_stage = DocumentIterateStage(iterator=my_iterator)
# extract_stage = DocumentExtractStage(extractor=my_extractor)

# After (26.04): fused into a single stage
iterate_extract_stage = DocumentIterateExtractStage(
    iterator=my_iterator,
    extractor=my_extractor,  # optional
    record_limit=100,
    add_filename_column=True,
)

Checklist

I am familiar with the Contributing Guide.
New or Existing tests cover these changes.
The documentation is up to date with these changes.

jgerh

Completed tech pubs review and provided a few copyedits

jgerh · 2026-03-30T23:10:41Z

Delete horizontal divider in the rendered output

…(PR #1458) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Revert 26.02 release notes line to avoid referencing 26.04 class name in wrong version section. Update custom.mdx wording to reflect that the fused step maps to multiple abstract base classes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Apply tech pubs copyedits: add periods to list items, fix link spacing, backtick code references, correct product names (FastText, GLiNER), replace OOM'd with "ran out of memory", rewrite component descriptions for clarity, convert <Note> to :::{note} directive, remove horizontal dividers, and align fern/non-fern breaking changes (add Three-Stage Pipeline entry to non-fern). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Release notes live in fern only; remove docs/ changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

lbliii · 2026-04-01T14:03:41Z

@claude review

lbliii · 2026-04-01T15:58:54Z

/claude review

claude · 2026-04-01T16:00:25Z

+:::{note}
 Data acquisition includes basic content-level deduplication during extraction (such as removing duplicate HTML content within individual web pages). This is separate from the main deduplication pipeline stages (exact, fuzzy, and semantic deduplication) that operate on the full dataset after acquisition.
-</Note>
+:::


Bug: The rest of the v26.04 fern docs consistently use <Note>/</Note> syntax (e.g., installation.mdx, dedup.mdx, index.mdx). This :::{note} / ::: syntax is inconsistent and may not render correctly in fern.

Suggested change

:::{note}

Data acquisition includes basic content-level deduplication during extraction (such as removing duplicate HTML content within individual web pages). This is separate from the main deduplication pipeline stages (exact, fuzzy, and semantic deduplication) that operate on the full dataset after acquisition.

</Note>

:::

<Note>

Data acquisition includes basic content-level deduplication during extraction (such as removing duplicate HTML content within individual web pages). This is separate from the main deduplication pipeline stages (exact, fuzzy, and semantic deduplication) that operate on the full dataset after acquisition.

</Note>

sarahyurick · 2026-04-01T16:09:14Z

- **InternVideo2 Removed**: Video pipelines must use alternative embedding models (Cosmos-Embed1)
- **ID Field Standardization**: Custom deduplication workflows may need updates to use standardized ID field names
+- **InternVideo2 Removed**: Video pipelines must use alternative embedding models (Cosmos-Embed1).
+- **ID Field Standardization**: Custom deduplication workflows may need updates to use standardized ID field names.


fyi this one was for 26.04, not 26.02 (#1390). Can't tell which one it's supposed to be associated with here.

Remove ID Field Standardization entries from non-fern 26.02 release notes since PR #1390 is a 26.04 feature. Remove fused iterate-extract entry from same section (26.04 via PR #1458). Remove remaining horizontal divider in custom.mdx. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

copy-pr-bot · 2026-04-06T15:22:05Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Restore docs/about/release-notes/index.md to staging state; this PR should not modify the non-fern release notes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Keep both fused iterate-extract stages (PR #1458) and new staging content (inference server, workflow results, CVE fixes, etc.) in the fern release notes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Keep fused iterate-extract entry (PR #1458) and add pipeline stage metrics entry (PR #1385) from upstream. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

lbliii requested review from a team as code owners March 30, 2026 19:05

lbliii requested review from VibhuJawa and removed request for a team March 30, 2026 19:05

copy-pr-bot Bot temporarily deployed to test March 30, 2026 19:06 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci March 30, 2026 19:06 Error

copy-pr-bot Bot temporarily deployed to nemo-ci March 30, 2026 19:06 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci March 30, 2026 19:25 Inactive

copy-pr-bot Bot temporarily deployed to test March 30, 2026 19:46 Inactive

jgerh reviewed Mar 30, 2026

View reviewed changes

lbliii and others added 10 commits March 31, 2026 09:40

docs: update docs and release notes for fused iterate-extract stages …

7bacc9f

…(PR #1458) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

docs: revert non-fern release notes to staging state

4ede4e9

Release notes live in fern only; remove docs/ changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Update docs/about/release-notes/index.md

255e7f1

Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Update docs/about/release-notes/index.md

c212791

Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Update docs/about/release-notes/index.md

cefd845

Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Update docs/about/release-notes/index.md

fae01f7

Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Update docs/about/release-notes/index.md

fa9f945

Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

Apply suggestion from @jgerh

4ce9309

Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

claude Bot reviewed Apr 1, 2026

View reviewed changes

sarahyurick reviewed Apr 1, 2026

View reviewed changes

lbliii and others added 2 commits April 6, 2026 11:26

docs: revert all changes to non-fern release notes

fabd6c8

Restore docs/about/release-notes/index.md to staging state; this PR should not modify the non-fern release notes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

sarahyurick approved these changes Apr 6, 2026

View reviewed changes

Merge 26.04-staging: resolve release notes conflict

105b916

Keep fused iterate-extract entry (PR #1458) and add pipeline stage metrics entry (PR #1385) from upstream. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Navigation Menu

Uh oh!

docs: update docs and release notes for fused iterate-extract (PR #1458) - #1684

docs: update docs and release notes for fused iterate-extract (PR #1458)#1684
lbliii merged 14 commits into
26.04-stagingfrom
lbliii/classifier-docs-1458

lbliii commented Mar 30, 2026

Uh oh!

jgerh left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgerh Mar 30, 2026

Uh oh!

Uh oh!

lbliii commented Apr 1, 2026

Uh oh!

lbliii commented Apr 1, 2026

Uh oh!

claude Bot Apr 1, 2026

Uh oh!

sarahyurick Apr 1, 2026

Uh oh!

copy-pr-bot Bot commented Apr 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

lbliii commented Mar 30, 2026

Description

Usage

Checklist

Uh oh!

jgerh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgerh Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lbliii commented Apr 1, 2026

Uh oh!

lbliii commented Apr 1, 2026

Uh oh!

claude Bot Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

sarahyurick Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

copy-pr-bot Bot commented Apr 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants