docs: add release notes and docs for math benchmarking support (PR #1604)

[lbliii]

Description

Adds fern documentation for PR #1604 (math modality benchmarking support). Updates the Common Crawl page with CommonCrawlWARCReader usage, S3 transport parameters, and environment variable reference. Adds release note entries for S3 transport, serialization bug fixes for MathContentExtractor and CommonCrawlWARCReader, and the boto3 dependency addition.

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[greptile-apps]

Greptile Summary

This documentation-only PR adds a CommonCrawlWARCReader S3 transport section to the Common Crawl how-to page and corresponding 26.04 release note entries for S3 transport, MathContentExtractor/CommonCrawlWARCReader serialization fixes, and the boto3 dependency.

• The "How it Works" section in common-crawl.mdx still describes a four-stage pipeline, directly contradicting the PR Fuse document iterate and extract stages #1458 release note in the same file that reduced it to three stages (DocumentIterateExtractStage).
• release-notes/index.mdx contains a duplicated ## Improvements heading and a duplicated "Video vLLM Setup Race Condition (PR fix: vLLM setup race condition inside video captioning stages #1590)" bug-fix entry.

Confidence Score: 4/5

Safe to merge after fixing the three-stage/four-stage pipeline contradiction and removing the duplicate release note entries.

One P1 finding (four-stage vs three-stage pipeline description directly contradicts another entry in the same 26.04 release notes) and two P2 structural issues (duplicate section heading, duplicated bug-fix entry) keep the score at 4.

Both files need attention: common-crawl.mdx for the stage-count inconsistency, and release-notes/index.mdx for the duplicate heading and duplicated bug-fix entry.

Vulnerabilities

No security concerns identified. This PR is documentation-only and introduces no executable code.

Important Files Changed

Filename	Overview
fern/versions/v26.04/pages/about/release-notes/index.mdx	Adds S3 transport, MathContentExtractor, and CommonCrawlWARCReader serialization entries; has a duplicate ## Improvements heading and a duplicated Video vLLM race-condition bug-fix entry.
fern/versions/v26.04/pages/curate-text/load-data/common-crawl.mdx	Adds CommonCrawlWARCReader S3 transport section and parameter table; "How it Works" still describes a four-stage pipeline that was reduced to three stages by PR #1458 in the same 26.04 release.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[CommonCrawlWARCReader] --> B{Transport mode}
    B -->|use_s3=False / default| C[HTTPS via requests library]
    B -->|use_s3=True or CC_USE_S3=1| D[S3 via boto3]
    B -->|use_s3=None| E{Read CC_USE_S3 env var}
    E -->|set| D
    E -->|not set| C
    C --> F[data.commoncrawl.org byte-range request]
    D --> G{Resolve S3 bucket}
    G -->|s3_bucket param| H[use param value]
    G -->|CC_S3_BUCKET env| H
    G -->|fallback| I[commoncrawl]
    H --> J[S3 byte-range GetObject]
    I --> J
    F --> K[binary_content column]
    J --> K

Loading

_{Reviews (2): Last reviewed commit: "docs: add release notes and docs for mat..." | Re-trigger Greptile}

[greptile-apps]

S3 parameters not present in current implementation

The documented use_s3, s3_bucket, and s3_key_prefix constructor parameters do not exist in the current CommonCrawlWARCReader implementation at nemo_curator/stages/text/download/common_crawl/download.py. The actual __init__ signature only accepts:

def __init__(
    self,
    warc_filename_col: str = "warc_filename",
    warc_record_offset_col: str = "warc_record_offset",
    warc_record_length_col: str = "warc_record_length",
    binary_content_col: str = "binary_content",
    drop_failed: bool = True,
    max_workers: int = 16,
    timeout: int = 30,
    max_retries: int = 3,
):

Passing use_s3=True (as shown in the S3 code example at line 232) will raise TypeError: __init__() got an unexpected keyword argument 'use_s3' until the code from PR #1604 is merged into 26.04-staging. The documentation also references __getstate__/__setstate__ serialization methods and boto3 integration in the release notes, none of which are present in the current codebase.

This docs PR should either be held until the corresponding code PR (#1604) is merged into 26.04-staging, or clearly mark these sections as upcoming/unreleased.

[greptile-apps]

Serialization fixes not yet in codebase

The MathContentExtractor at nemo_curator/stages/math/download/extract.py does not have the documented __getstate__/__setstate__ methods, and CommonCrawlWARCReader at nemo_curator/stages/text/download/common_crawl/download.py likewise has no pickle-compatibility methods nor a threading.Lock or S3 client attribute. Grepping for __getstate__, __setstate__, _lock, and use_s3 across both files returns no matches.

These release note entries accurately describe the intended code changes from PR #1604, but those code changes are not yet present on the 26.04-staging branch. Publishing these release notes before the code lands may mislead users who try to reproduce the described behaviour.

[lbliii]

The Greptile comments are expected — PR #1604 is already merged to main and will be cherry-picked to 26.04-staging before release. This docs PR follows the same pattern as the other docs PRs on this branch (PRs #1698, #1729, #1704, #1706, #1705), all of which document code that landed on main first.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.