Fixes #1232 reorganize document filter document modifier dir

[KunalSachdev2005]

Description

Fixes: #1232

Importing DocumentFilter eagerly imports many filters, some of which have heavy dependencies - TokenCountFilter (transformers; HuggingFace deps), HistogramFilter (downloads; cache), and FastTextLangId / FastTextQualityFilter.

We want to make the DocumentFilter import lightweight. Users must opt in for heavier filters explicitly.

Additionally, DocumentFilter and specific filters are defined under nemo_curator/stages/text/filters while Filter/Score/ScoreFilter are under nemo_curator/stages/text/modules.

DocumentModifier and other modifiers are under nemo_curator/stages/text/modifiers while Modify is under nemo_curator/stages/text/modules.

This is confusing from a user perspective.

New Directory structures in this PR:

nemo_curator/stages/text/filters/
├── __init__.py                  # Exposes DocumentFilter, Filter, Score, ScoreFilter
├── doc_filter.py                # DocumentFilter (lightweight, no heavy deps)
├── score_filter.py              # Filter, Score, ScoreFilter
│
├── heuristic/
│   ├── __init__.py              # Exposes string-based heuristic filters directly at the root because they are lightweight
│   ├── string.py                # regex / string-based filters
│   │
│   ├── code/
│   │   ├── __init__.py          # Exposes code-based heuristic filters (logical grouping under heuristic filters)            
│   │   └── code.py              # code-specific heuristic filters
│   │
│   └── repetition/
│       ├── __init__.py          # Exposes repetition detection heuristic filters (logical grouping under heuristic filters)
│       └── repetition.py        # repeated lines / paragraphs / n-grams
│
├── token/
│   ├── __init__.py
│   └── token_count.py           # TokenCountFilter (transformers; HuggingFace deps)
│
├── histogram/
│   ├── __init__.py
│   └── histogram.py             # HistogramFilter (downloads; cache)
│
└── fasttext/
    ├── __init__.py
    └── fasttext_filters.py      # FastTextLangId / FastTextQualityFilter

nemo_curator/stages/text/modifiers/
├── __init__.py                 # DocumentModifier, Modify, and lightweight heuristic filters like LineRemover, MarkdownRemover, etc.
├── modifier.py                 # Modify
├── doc_modifier.py             # DocumentModifier
│
├── heuristic/
│   ├── __init__.py
│   ├── c4.py                   # C4-style text normalization
│   ├── line_remover.py         # Line-based removal heuristics
│   ├── markdown_remover.py     # Markdown cleanup / stripping
│   ├── newline_normalizer.py   # Normalize newlines
│   ├── quotation_remover.py    # Remove quoted text
│   ├── slicer.py               # Slice / truncate documents
│   └── url_remover.py          # URL removal heuristics
│
├── fasttext/
│   ├── __init__.py
│   └── label.py                # FastTextLabelModifier
│
└── unicode/
    ├── __init__.py
    └── reformatter.py          # UnicodeReformatter

nemo_curator/stages/text/modules/ (removed modifier.py and score_filter.py)
├── __init__.py
├── add_id.py
├── joiner.py
└── splitter.py

Usage

Filters

from nemo_curator.stages.text.filters import DocumentFilter, Score, Filter, ScoreFilter # lightweight. Only exposes these classes

from nemo_curator.stages.text.filters.heuristic import BoilerPlateStringFilter, LongWordFilter # string-based filters are lightweight. So they are exposed at the root
from nemo_curator.stages.text.filters.heuristic.code import GeneralCommentToCodeFilter # code-based filters
from nemo_curator.stages.text.filters.heuristic.repetition import RepeatedLinesFilter # filters detecting repetition

# users must opt in for these filters with heavy dependencies. they won't be available directly from text.filters anymore
from nemo_curator.stages.text.filters.histogram import HistogramFilter
from nemo_curator.stages.text.filters.token import TokenCountFilter
from nemo_curator.stages.text.filters.fasttext import FastTextLangId, FastTextQualityFilter

Modifiers

from nemo_curator.stages.text.modifiers import DocumentModifier, Modify

from nemo_curator.stages.text.modifiers import LineRemover, MarkdownRemover # lightweight. So they are exposed at the root for ease of access
from nemo_curator.stages.text.modifiers.heuristic import LineRemover, MarkdownRemover # can be accessed from here too

# users must opt in for these filters they won't be available directly from text.modifiers anymore
from nemo_curator.stages.text.modifiers.fasttext import FastTextLabelModifier
from nemo_curator.stages.text.modifiers.unicode import UnicodeReformatter

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
  • I tested these: tests/stages/common/, tests/stages/text/modules/, and tests/pipelines/. All of these pass :)
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR successfully reorganizes the directory structure for DocumentFilter and DocumentModifier to make imports lightweight and more user-friendly.

Key Changes

• Lightweight base imports: from nemo_curator.stages.text.filters import DocumentFilter, Filter, Score, ScoreFilter now imports only base classes with no heavy dependencies
• Opt-in heavy filters: Users must explicitly import TokenCountFilter, HistogramFilter, and FastText filters from their respective subpackages (filters.token, filters.histogram, filters.fasttext)
• Lazy loading implementation: All heavy-dependency filters and modifiers use __getattr__ pattern with TYPE_CHECKING to defer imports until actual usage
• Logical grouping: Heuristic filters organized into heuristic/ (lightweight strings), heuristic/code/, and heuristic/repetition/ subdirectories
• Fixed circular imports: Both score_filter.py and modifier.py now use relative imports from defining modules instead of package-level imports
• Comprehensive updates: All 70 files updated including documentation, config files, tutorials, benchmarks, and tests

Issues Addressed from Previous Review

All previously identified issues have been resolved:

• Circular import issues fixed in score_filter.py and modifier.py
• Lazy loading implemented for TokenCountFilter, HistogramFilter, FastTextLangId, FastTextQualityFilter, FastTextLabelModifier, and UnicodeReformatter
• Documentation import statements corrected
• Module path typos fixed

The refactoring maintains backward compatibility through comprehensive updates across the codebase while achieving the primary goal of making core filter/modifier imports lightweight.

Confidence Score: 5/5

• This PR is safe to merge - it's a well-executed refactoring with comprehensive updates across all affected files
• The refactoring successfully achieves its goals with no remaining issues. All circular imports are resolved, lazy loading is properly implemented, documentation and tests are updated, and previous review comments have been addressed. The PR author explicitly tested the affected test suites and confirmed they pass.
• No files require special attention - the refactoring is comprehensive and well-executed

Important Files Changed

Filename	Overview
nemo_curator/stages/text/filters/init.py	Lightweight filter exports - successfully exposes only base classes (DocumentFilter, Filter, Score, ScoreFilter) with no heavy dependencies
nemo_curator/stages/text/filters/score_filter.py	Fixed circular import by importing DocumentFilter directly from .doc_filter instead of package-level import
nemo_curator/stages/text/filters/token/init.py	Implements lazy loading with __getattr__ - heavy dependencies (transformers, huggingface_hub) only load when TokenCountFilter is accessed
nemo_curator/stages/text/filters/histogram/init.py	Implements lazy loading with __getattr__ for HistogramFilter to defer loading of requests and platformdirs dependencies
nemo_curator/stages/text/filters/fasttext/init.py	Implements lazy loading with __getattr__ for FastText filters to defer loading of fasttext dependency
nemo_curator/stages/text/modifiers/init.py	Exports DocumentModifier, Modify, and lightweight string modifiers at root for easy access
nemo_curator/stages/text/modifiers/modifier.py	Fixed circular import by importing DocumentModifier directly from .doc_modifier instead of package-level import
nemo_curator/stages/text/modifiers/fasttext/init.py	Implements lazy loading with __getattr__ for FastTextLabelModifier to defer loading of fasttext dependency
nemo_curator/stages/text/modules/init.py	Successfully removed modifier.py and score_filter.py - now only exports AddId, DocumentJoiner, DocumentSplitter

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User imports filters] --> B{Import Type?}
    B -->|Base Classes| C[filters.__init__]
    B -->|Heuristic| D[filters.heuristic]
    B -->|Heavy Deps| E[Opt-in Subpackages]
    
    C -->|Lightweight| C1[DocumentFilter<br/>Filter<br/>Score<br/>ScoreFilter]
    
    D -->|Lightweight| D1[String Filters]
    D -->|Specialized| D2[heuristic.code]
    D -->|Specialized| D3[heuristic.repetition]
    
    E -->|Lazy Load| E1[filters.token<br/>TokenCountFilter]
    E -->|Lazy Load| E2[filters.histogram<br/>HistogramFilter]
    E -->|Lazy Load| E3[filters.fasttext<br/>FastText*Filter]
    
    E1 -.->|__getattr__| F1[Heavy: transformers<br/>huggingface_hub]
    E2 -.->|__getattr__| F2[Heavy: requests<br/>platformdirs]
    E3 -.->|__getattr__| F3[Heavy: fasttext]
    
    style C1 fill:#90EE90
    style D1 fill:#90EE90
    style F1 fill:#FFB6C6
    style F2 fill:#FFB6C6
    style F3 fill:#FFB6C6
    style E1 fill:#87CEEB
    style E2 fill:#87CEEB
    style E3 fill:#87CEEB

Loading

_{Last reviewed commit: f6aeeb6}

[greptile-apps]

_{70 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (2)

tests/config/test_run.py
Broken Hydra _target_ paths

test_pipeline_with_hydra_instantiated_stage will fail at runtime because both _target_ strings point to modules that no longer exist after the refactor: nemo_curator.stages.text.modules.score_filter.ScoreFilter (module removed) and nemo_curator.stages.text.filters.heuristic_filter.NonAlphaNumericFilter (moved under filters.heuristic). These should be updated to the new import locations (e.g., nemo_curator.stages.text.filters.score_filter.ScoreFilter and nemo_curator.stages.text.filters.heuristic.NonAlphaNumericFilter).

---

tutorials/text/llama-nemotron-data-curation/main.py
Invalid ScoreFilter import

This script imports ScoreFilter from nemo_curator.stages.text.modules, but nemo_curator/stages/text/modules/__init__.py only exports AddId, DocumentJoiner, and DocumentSplitter, so this will raise ImportError on import. ScoreFilter should be imported from nemo_curator.stages.text.filters (or nemo_curator.stages.text.filters.score_filter).

[greptile-apps]

_{5 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{6 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{13 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

nemo_curator/stages/text/modifiers/modifier.py
outputs() may return None

outputs() does sorted(set(self._output_fields)), but self._output_fields can contain None when output_fields is a list and includes None (see _normalize_output_fields returning list[str] but actually allows None entries). set([... , None, ...]) will include None, and sorted(...) will then raise TypeError (cannot compare NoneType/str) or will propagate None as an output column name, which will break downstream stage wiring. This happens for e.g. Modify(..., output_fields=[None, "ab"], input_fields=[["a"], ["a","b"]]).

Also appears in: nemo_curator/stages/text/modifiers/modifier.py:71 (same line).

[sarahyurick]

Tysm @KunalSachdev2005 ! I left a couple suggestions, let me know what you think.

[greptile-apps]

_{6 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{6 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{8 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{5 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

nemo_curator/stages/text/filters/score_filter.py
Single-element list replication bug

In _format_field_list, when _field is a list of length 1, the code does _field = [_field] * filter_count, which produces a nested list (e.g., ['text'] becomes [['text'], ['text'], ...]) rather than replicating the element (['text', 'text', ...]). This will break downstream logic for multi-filter stages (e.g., Score/ScoreFilter zips will see text_field_i as a list, and df[text_field_i] will error). Consider using element replication (_field = _field * filter_count or _field = [_field[0]] * filter_count).

[sarahyurick]

/ok to test 2a9a601

[greptile-apps]

_{69 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[KunalSachdev2005]

Hi @sarahyurick, I see that two checks are failing:

1. Secrets detector: I clicked into the logs and can see it's probably a false positive caused by line number shifts from my changes. I'll run detect-secrets scan locally to regenerate the baseline file and commit the updated .secrets.baseline.

2. Codecov: The patch coverage is at 56.84% against target of 80%. Could you point me to which files/areas need more test coverage or let me know what tests are expected for my specific changes? I can add them once I know what's needed.

[sarahyurick]

Hi @sarahyurick, I see that two checks are failing:

1. Secrets detector: I clicked into the logs and can see it's probably a false positive caused by line number shifts from my changes. I'll run detect-secrets scan locally to regenerate the baseline file and commit the updated .secrets.baseline.
2. Codecov: The patch coverage is at 56.84% against target of 80%. Could you point me to which files/areas need more test coverage or let me know what tests are expected for my specific changes? I can add them once I know what's needed.

Hi @KunalSachdev2005 sorry for the confusion about 1. Thanks.

For 2, you don't have to worry about it. The Codecov check can be flaky. I can still merge it even if it continues to fail.

[greptile-apps]

Additional Comments (1)

nemo_curator/config/text/heuristic_filter_english_pipeline.yaml, line 52
Config uses heuristic.string.NonAlphaNumericFilter but heuristic/__init__.py now exports string filters directly - could simplify to heuristic.NonAlphaNumericFilter for consistency with the design intent

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[KunalSachdev2005]

Hi @sarahyurick, I've fixed the secrets detector issue - just to be safe, I replaced the nvapi-... placeholder strings with non-secret placeholders <your-nvapi-key-here> in doc/curate-text/synthetic/llm-client.md and doc/curate-text/synthetic/multilingual-qa.md and updated .secrets.baseline. The check should pass now.

[sarahyurick]

/ok to test f6aeeb6

[KunalSachdev2005]

@sarahyurick The Secrets detector test is passing now!

However, I see that a few other tests are failing. Looking at the logs:

1. Unit_Test_stages-audio_CPU_python-3.10 - Failing with 403 Forbidden while downloading nvidia/parakeet-tdt-0.6b-v2 from Hugging Face. Doesn't look like something introduced by my changes.
2. Unit_Test_stages-deduplication_CPU_python-3.12 - Fails due to Ray actor SIGSEGV. Seems like an infra/runtime issue unrelated to my changes.
3. L0_Unit_Test_GPU - The Docker build is failing when installing torch due to network timeout (UV_HTTP_TIMEOUT=30s) when downloading nvidia-dali-cuda120==1.52.0 via nemo-curator[all] - unrelated to my PR changes.

Please let me know if you'd like me to do anything else at this time, but from the logs the errors seem infra-related rather than caused by my refactor/changes.

[sarahyurick]

@sarahyurick The Secrets detector test is passing now!

However, I see that a few other tests are failing. Looking at the logs:

1. Unit_Test_stages-audio_CPU_python-3.10 - Failing with 403 Forbidden while downloading nvidia/parakeet-tdt-0.6b-v2 from Hugging Face. Doesn't look like something introduced by my changes.
2. Unit_Test_stages-deduplication_CPU_python-3.12 - Fails due to Ray actor SIGSEGV. Seems like an infra/runtime issue unrelated to my changes.
3. L0_Unit_Test_GPU - The Docker build is failing when installing torch due to network timeout (UV_HTTP_TIMEOUT=30s) when downloading nvidia-dali-cuda120==1.52.0 via nemo-curator[all] - unrelated to my PR changes.

Please let me know if you'd like me to do anything else at this time, but from the logs the errors seem infra-related rather than caused by my refactor/changes.

No worries, it looks like it is just GitHub CI weirdness. I am rerunning them now. Nothing needed from your side.

Thank you for the contribution @KunalSachdev2005 !