Skip to content

docs: add fern docs and release notes for LSH memory config (PR #1603) #1732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lbliii merged 2 commits into from
Apr 7, 2026
Merged

docs: add fern docs and release notes for LSH memory config (PR #1603) #1732

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
98dba51
docs: add release notes and docs for LSH memory config (PR #1603)
lbliii Apr 3, 2026
244a8df
fix: address review feedback and resolve merge conflict
lbliii Apr 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions fern/versions/v26.04/pages/about/release-notes/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,16 @@ The exact deduplication identification stage now supports batched insertion into
- **New `identification_batchsize` parameter** on `ExactDeduplicationWorkflow`: Controls how many input blocks are concatenated and inserted together. For example, an `input_blocksize` of `256MiB` with `identification_batchsize=4` processes ~1 GB of data per insertion call.
- **Batch-aware shuffle adapter**: The Ray actor pool shuffle adapter now automatically uses `read_and_insert_batch` when a stage provides it, falling back to single-task processing otherwise.

### LSH Memory Configuration for Fuzzy Deduplication (PR #1603)

`FuzzyDeduplicationWorkflow` now exposes three parameters for controlling GPU memory and shuffle behavior during the LSH stage:

- **`lsh_num_output_partitions`**: Sets the total number of output partitions for the LSH shuffle. When `None` (default), the partition count is chosen automatically.
- **`lsh_rmm_pool_size`**: Controls the RMM GPU memory pool size in bytes. Defaults to `"auto"` (90% of free GPU memory).
- **`lsh_spill_memory_limit`**: Controls the device memory limit for spilling to host. Defaults to `"auto"` (80% of the RMM pool size). Set to `None` to disable spilling.

These parameters were previously hardcoded in the LSH stage and are now configurable at the workflow level, enabling finer-grained GPU memory tuning for large-scale fuzzy deduplication jobs.
Comment on lines +82 to +90

@greptile-apps greptile-apps Bot Apr 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Release note describes unmerged code changes

This entry states that PR #1603 has landed ("now exposes three parameters … previously hardcoded"), but the FuzzyDeduplicationWorkflow constructor in the current HEAD of 26.04-staging still has no lsh_* parameters and _create_lsh_pipeline still hard-codes rmm_pool_size="auto" and spill_memory_limit="auto".

Publishing a release note for a change that isn't yet in the branch will be confusing to users who read the changelog and try to use these parameters against an installed 26.04 build. This PR should be blocked on (or merged simultaneously with) PR #1603.


## Security Fixes

### CVE Fixes for Audio and Inference Dependencies (PR #1612)
Expand Down
5 changes: 5 additions & 0 deletions fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,9 @@ Configure fuzzy deduplication using these key parameters:
| `seed` | int | 42 | Random seed for MinHash permutations |
| `input_filetype` | str | "parquet" | Input file format ("parquet" or "jsonl") |
| `input_blocksize` | str \| int | "1GiB" | Size of input blocks for processing |
| `lsh_num_output_partitions` | int \| None | None | Total number of partitions to write during the LSH shuffle. If `None`, the partition count is chosen automatically as the closest power of 2 <= the number of input tasks. |
| `lsh_rmm_pool_size` | int \| "auto" \| None | "auto" | Size of the RMM GPU memory pool in bytes for the LSH stage. `"auto"` sets the pool to 90% of free GPU memory. `None` sets the pool to 50% of free GPU memory and allows expansion. |
| `lsh_spill_memory_limit` | int \| "auto" \| None | "auto" | Device memory limit in bytes for spilling to host during the LSH stage. `"auto"` sets the limit to 80% of the RMM pool size. `None` disables spilling. |
Comment on lines +97 to +99

@greptile-apps greptile-apps Bot Apr 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Documented parameters don't exist in FuzzyDeduplicationWorkflow yet

These three table rows document lsh_num_output_partitions, lsh_rmm_pool_size, and lsh_spill_memory_limit as accepted parameters of FuzzyDeduplicationWorkflow, but the current codebase shows they have not been added to the workflow's __init__ signature or wired up.

In nemo_curator/stages/deduplication/fuzzy/workflow.py, the FuzzyDeduplicationWorkflow.__init__ (lines 62–84) has no lsh_* parameters, and _create_lsh_pipeline (lines 209–234) still hard-codes the values that PR #1603 was meant to expose:

LSHStage(
    ...
    rmm_pool_size="auto",        # hard-coded, not self.lsh_rmm_pool_size
    spill_memory_limit="auto",   # hard-coded, not self.lsh_spill_memory_limit
    # total_nparts not passed at all
),

If a user tries to pass any of these keyword arguments to FuzzyDeduplicationWorkflow after reading this table, they will get a TypeError: __init__() got an unexpected keyword argument. This PR should either be held until PR #1603 is merged into 26.04-staging, or the new rows should be removed/marked as upcoming.

| `perform_removal` | bool | False | Reserved; must remain `False`. Fuzzy removal is performed with `TextDuplicatesRemovalWorkflow`. |

### Similarity Threshold
Expand Down Expand Up @@ -206,6 +209,8 @@ The workflow produces these output files:
**Performance tuning**:

- **Memory**: Adjust `bands_per_iteration` (lower = less memory, more iterations)
- **GPU memory (LSH)**: Use `lsh_rmm_pool_size` to control GPU memory allocation and `lsh_spill_memory_limit` to tune host-spilling behavior during the LSH stage. Reducing the pool size or lowering the spill threshold can prevent out-of-memory errors on smaller GPUs.
- **Shuffle partitions**: Set `lsh_num_output_partitions` to control the number of output partitions during the LSH shuffle. More partitions reduce per-partition memory but increase I/O overhead.
- **Accuracy**: Use `char_ngrams >= 20` to reduce false positives
- **Best practices**: Clear cache between runs, use `input_blocksize="1GiB"`

Expand Down
Loading