diff --git a/fern/versions/v26.04/pages/about/release-notes/index.mdx b/fern/versions/v26.04/pages/about/release-notes/index.mdx index d5da5ff80c..4156933eed 100644 --- a/fern/versions/v26.04/pages/about/release-notes/index.mdx +++ b/fern/versions/v26.04/pages/about/release-notes/index.mdx @@ -79,6 +79,16 @@ The exact deduplication identification stage now supports batched insertion into - **New `identification_batchsize` parameter** on `ExactDeduplicationWorkflow`: Controls how many input blocks are concatenated and inserted together. For example, an `input_blocksize` of `256MiB` with `identification_batchsize=4` processes ~1 GB of data per insertion call. - **Batch-aware shuffle adapter**: The Ray actor pool shuffle adapter now automatically uses `read_and_insert_batch` when a stage provides it, falling back to single-task processing otherwise. +### LSH Memory Configuration for Fuzzy Deduplication (PR #1603) + +`FuzzyDeduplicationWorkflow` now exposes three parameters for controlling GPU memory and shuffle behavior during the LSH stage: + +- **`lsh_num_output_partitions`**: Sets the total number of output partitions for the LSH shuffle. When `None` (default), the partition count is chosen automatically. +- **`lsh_rmm_pool_size`**: Controls the RMM GPU memory pool size in bytes. Defaults to `"auto"` (90% of free GPU memory). +- **`lsh_spill_memory_limit`**: Controls the device memory limit for spilling to host. Defaults to `"auto"` (80% of the RMM pool size). Set to `None` to disable spilling. + +These parameters were previously hardcoded in the LSH stage and are now configurable at the workflow level, enabling finer-grained GPU memory tuning for large-scale fuzzy deduplication jobs. + ## Security Fixes ### CVE Fixes for Audio and Inference Dependencies (PR #1612) diff --git a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx index 8a53338fab..8647e61390 100644 --- a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx +++ b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/fuzzy.mdx @@ -94,6 +94,9 @@ Configure fuzzy deduplication using these key parameters: | `seed` | int | 42 | Random seed for MinHash permutations | | `input_filetype` | str | "parquet" | Input file format ("parquet" or "jsonl") | | `input_blocksize` | str \| int | "1GiB" | Size of input blocks for processing | +| `lsh_num_output_partitions` | int \| None | None | Total number of partitions to write during the LSH shuffle. If `None`, the partition count is chosen automatically as the closest power of 2 <= the number of input tasks. | +| `lsh_rmm_pool_size` | int \| "auto" \| None | "auto" | Size of the RMM GPU memory pool in bytes for the LSH stage. `"auto"` sets the pool to 90% of free GPU memory. `None` sets the pool to 50% of free GPU memory and allows expansion. | +| `lsh_spill_memory_limit` | int \| "auto" \| None | "auto" | Device memory limit in bytes for spilling to host during the LSH stage. `"auto"` sets the limit to 80% of the RMM pool size. `None` disables spilling. | | `perform_removal` | bool | False | Reserved; must remain `False`. Fuzzy removal is performed with `TextDuplicatesRemovalWorkflow`. | ### Similarity Threshold @@ -206,6 +209,8 @@ The workflow produces these output files: **Performance tuning**: - **Memory**: Adjust `bands_per_iteration` (lower = less memory, more iterations) +- **GPU memory (LSH)**: Use `lsh_rmm_pool_size` to control GPU memory allocation and `lsh_spill_memory_limit` to tune host-spilling behavior during the LSH stage. Reducing the pool size or lowering the spill threshold can prevent out-of-memory errors on smaller GPUs. +- **Shuffle partitions**: Set `lsh_num_output_partitions` to control the number of output partitions during the LSH shuffle. More partitions reduce per-partition memory but increase I/O overhead. - **Accuracy**: Use `char_ngrams >= 20` to reduce false positives - **Best practices**: Clear cache between runs, use `input_blocksize="1GiB"`