From 98dba51edf24bb983b3c7d3eb237914d1c5fd428 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Fri, 3 Apr 2026 11:43:46 -0400 Subject: [PATCH] docs: add release notes and docs for LSH memory config (PR #1603) Co-Authored-By: Claude Opus 4.6 Signed-off-by: Lawrence Lane --- .../v26.04/pages/about/release-notes/index.mdx | 10 ++++++++++ .../curate-text/process-data/deduplication/fuzzy.mdx | 5 +++++ 2 files changed, 15 insertions(+) diff --git a/fern/versions/v26.04/pages/about/release-notes/index.mdx b/fern/versions/v26.04/pages/about/release-notes/index.mdx index 9d29d003db..ae6b82fc9a 100644 --- a/fern/versions/v26.04/pages/about/release-notes/index.mdx +++ b/fern/versions/v26.04/pages/about/release-notes/index.mdx @@ -75,6 +75,16 @@ The exact deduplication identification stage now supports batched insertion into - **New `identification_batchsize` parameter** on `ExactDeduplicationWorkflow`: Controls how many input blocks are concatenated and inserted together. For example, an `input_blocksize` of `256MiB` with `identification_batchsize=4` processes ~1 GB of data per insertion call. - **Batch-aware shuffle adapter**: The Ray actor pool shuffle adapter now automatically uses `read_and_insert_batch` when a stage provides it, falling back to single-task processing otherwise. +### LSH Memory Configuration for Fuzzy Deduplication (PR #1603) + +`FuzzyDeduplicationWorkflow` now exposes three parameters for controlling GPU memory and shuffle behavior during the LSH stage: + +- **`lsh_num_output_partitions`**: Sets the total number of output partitions for the LSH shuffle. When `None` (default), the partition count is chosen automatically. +- **`lsh_rmm_pool_size`**: Controls the RMM GPU memory pool size in bytes. Defaults to `"auto"` (90% of free GPU memory). +- **`lsh_spill_memory_limit`**: Controls the device memory limit for spilling to host. Defaults to `"auto"` (80% of the RMM pool size). Set to `None` to disable spilling. + +These parameters were previously hardcoded in the LSH stage and are now configurable at the workflow level, enabling finer-grained GPU memory tuning for large-scale fuzzy deduplication jobs. + ## Dependency Updates - **Cosmos-Xenna**: Updated from 0.1.2 to 0.2.0 with simplified resource model diff --git a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx index 8a53338fab..64e67caf5d 100644 --- a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx +++ b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx @@ -94,6 +94,9 @@ Configure fuzzy deduplication using these key parameters: | `seed` | int | 42 | Random seed for MinHash permutations | | `input_filetype` | str | "parquet" | Input file format ("parquet" or "jsonl") | | `input_blocksize` | str \| int | "1GiB" | Size of input blocks for processing | +| `lsh_num_output_partitions` | int \| None | None | Total number of partitions to write during the LSH shuffle. If `None`, the partition count is chosen automatically as the closest power of 2 <= the number of input tasks. | +| `lsh_rmm_pool_size` | int \| "auto" \| None | "auto" | Size of the RMM GPU memory pool in bytes for the LSH stage. `"auto"` sets the pool to 90% of free GPU memory. `None` sets the pool to 50% of free GPU memory and allows expansion. | +| `lsh_spill_memory_limit` | int \| "auto" \| None | "auto" | Device memory limit in bytes for spilling to host during the LSH stage. `"auto"` sets the limit to 80% of the RMM pool size. `None` disables spilling. | | `perform_removal` | bool | False | Reserved; must remain `False`. Fuzzy removal is performed with `TextDuplicatesRemovalWorkflow`. | ### Similarity Threshold @@ -206,6 +209,8 @@ The workflow produces these output files: **Performance tuning**: - **Memory**: Adjust `bands_per_iteration` (lower = less memory, more iterations) +- **GPU memory (LSH)**: Use `lsh_rmm_pool_size` and `lsh_spill_memory_limit` to control GPU memory allocation and host spilling during the LSH stage. Reducing the pool size or enabling spilling can prevent out-of-memory errors on smaller GPUs. +- **Shuffle partitions**: Set `lsh_num_output_partitions` to control the number of output partitions during the LSH shuffle. More partitions reduce per-partition memory but increase I/O overhead. - **Accuracy**: Use `char_ngrams >= 20` to reduce false positives - **Best practices**: Clear cache between runs, use `input_blocksize="1GiB"`