From 6a4934697a0c6eca3cc40009516b6281896a12f4 Mon Sep 17 00:00:00 2001 From: Lawrence Lane Date: Wed, 1 Apr 2026 11:15:12 -0400 Subject: [PATCH] docs: update fern docs for vLLM default in semantic dedup (PR #1606) Update semantic deduplication docs and release notes for the switch from SentenceTransformers to vLLM as the default embedding backend in TextSemanticDeduplicationWorkflow. Signed-off-by: Lawrence Lane Co-Authored-By: Claude Opus 4.6 Signed-off-by: Lawrence Lane --- .../pages/about/release-notes/index.mdx | 19 ++++---- .../process-data/deduplication/semdedup.mdx | 45 ++++++++++--------- 2 files changed, 35 insertions(+), 29 deletions(-) diff --git a/fern/versions/v26.04/pages/about/release-notes/index.mdx b/fern/versions/v26.04/pages/about/release-notes/index.mdx index e01437b78b..527e9d888e 100644 --- a/fern/versions/v26.04/pages/about/release-notes/index.mdx +++ b/fern/versions/v26.04/pages/about/release-notes/index.mdx @@ -24,14 +24,14 @@ Built-in LLM serving alongside curation pipelines using Ray Serve and vLLM: Learn more in the [Inference Server](/curate-text/synthetic/inference-server) documentation. -### vLLM Default for Semantic Deduplication Embeddings +### vLLM Default for Semantic Deduplication Embeddings (PR #1606) -The default embedding backend for `TextSemanticDeduplicationWorkflow` switched from SentenceTransformers to vLLM: +Switched the default embedding backend in `TextSemanticDeduplicationWorkflow` from SentenceTransformers to vLLM, with `google/embeddinggemma-300m` as the new default model: -- **Default model**: `google/embeddinggemma-300m` (previously `sentence-transformers/all-MiniLM-L6-v2`) -- **Backend**: `VLLMEmbeddingModelStage` replaces `EmbeddingCreatorStage` for embedding generation in the semantic dedup workflow -- **New parameters**: `embedding_pretokenize`, `embedding_vllm_init_kwargs`, `model_cache_dir` replace the previous SentenceTransformers-specific parameters -- **Bug fix**: Resolved CUDA fork error when running vLLM stages with `RayDataExecutor` by setting `VLLM_WORKER_MULTIPROC_METHOD=spawn` in Ray remote tasks +- **vLLM embedding backend**: `TextSemanticDeduplicationWorkflow` now uses `VLLMEmbeddingModelStage` for embedding generation, replacing `EmbeddingCreatorStage` (SentenceTransformers). +- **New default model**: Changed from `sentence-transformers/all-MiniLM-L6-v2` to `google/embeddinggemma-300m`. +- **New parameters**: Added `embedding_pretokenize`, `embedding_vllm_init_kwargs`, and `model_cache_dir` for vLLM configuration. +- **Removed parameters**: The SentenceTransformers-specific parameters `embedding_model_inference_batch_size`, `embedding_pooling`, `embedding_padding_side`, and `embedding_max_seq_length` are no longer available. ### Multi-User Metrics Isolation @@ -75,6 +75,10 @@ Added tqdm progress bars to `RayActorPoolExecutor` for real-time visibility into ## Bug Fixes +### CUDA Fork Error with vLLM and RayDataExecutor (PR #1606) + +Fixed a `RuntimeError: Cannot re-initialize CUDA in forked subprocess` error that occurred when running vLLM stages with `RayDataExecutor`. The vLLM auto-detection for `spawn` versus `fork` multiprocessing only triggers inside Ray actors, not Ray tasks. The `RayDataExecutor.execute_setup_on_node` method dispatches `setup_on_node` as a remote task, so vLLM previously defaulted to `fork` and caused a CUDA reinitialization error. Fixed by setting `VLLM_WORKER_MULTIPROC_METHOD=spawn` in the remote task. + ### Audio Stage Name Propagation (PR #1470) Fixed audio pipeline stage names not propagating in `StagePerfStats`, making benchmark output unable to identify per-stage timing. All audio stages (`GetAudioDurationStage`, `PreserveByValueStage`, `AudioToDocumentStage`, `GetPairwiseWerStage`) now correctly report their names, and stage performance history persists when stages create new task objects. @@ -85,8 +89,7 @@ Fixed a race condition in `CaptionGenerationStage` and `CaptionEnhancementStage` ## Breaking Changes -- **`TextSemanticDeduplicationWorkflow` parameter changes**: `embedding_max_seq_length`, `embedding_padding_side`, `embedding_pooling`, and `embedding_model_inference_batch_size` are removed. Use `embedding_pretokenize`, `embedding_vllm_init_kwargs`, and `model_cache_dir` instead. -- **Default embedding model changed**: `TextSemanticDeduplicationWorkflow` now defaults to `google/embeddinggemma-300m` with vLLM instead of `sentence-transformers/all-MiniLM-L6-v2` with SentenceTransformers. +- **`TextSemanticDeduplicationWorkflow` embedding backend**: The default embedding backend changed from SentenceTransformers to vLLM. The default model changed from `sentence-transformers/all-MiniLM-L6-v2` to `google/embeddinggemma-300m`. The parameters `embedding_model_inference_batch_size`, `embedding_pooling`, `embedding_padding_side`, and `embedding_max_seq_length` have been removed. Use `embedding_vllm_init_kwargs` to pass configuration to the vLLM backend instead. - **`Resources` API**: The `nvdecs`, `nvencs`, and `entire_gpu` fields have been removed from `Resources`. Stages that previously used `entire_gpu=True` should use `gpus=1` instead. Stages that used `nvdecs` or `nvencs` should use `gpus` for GPU allocation. - **`ExactDeduplicationWorkflow.run()` and `FuzzyDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `None` - **`SemanticDeduplicationWorkflow.run()` and `TextSemanticDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `dict` diff --git a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx index 4885569a14..95d9886282 100644 --- a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx +++ b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx @@ -51,7 +51,6 @@ Get started with semantic deduplication using the following example of identifyi ```python from nemo_curator.stages.text.deduplication.semantic import TextSemanticDeduplicationWorkflow -from nemo_curator.backends.experimental.ray_data import RayDataExecutor # Default: uses vLLM with google/embeddinggemma-300m workflow = TextSemanticDeduplicationWorkflow( @@ -61,12 +60,11 @@ workflow = TextSemanticDeduplicationWorkflow( n_clusters=100, eps=0.07, # Similarity threshold id_field="doc_id", - perform_removal=True # Complete deduplication + perform_removal=True, # Complete deduplication ) -executor = RayDataExecutor() -result = workflow.run(executor) -# result.metadata contains: total_time, num_duplicates, num_duplicates_removed, embedding_time, identification_time, removal_time, final_output_path +results = workflow.run() +# Clean dataset saved to ./results/deduplicated/ ``` ## Configuration @@ -80,7 +78,7 @@ For fine-grained control, break semantic deduplication into separate stages: ```python from nemo_curator.stages.deduplication.id_generator import create_id_generator_actor from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage -from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow, IdentifyDuplicatesStage +from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow from nemo_curator.stages.text.deduplication.removal_workflow import TextDuplicatesRemovalWorkflow from nemo_curator.pipeline import Pipeline from nemo_curator.stages.text.io.reader import ParquetReader @@ -94,10 +92,12 @@ embedding_pipeline = Pipeline( name="embedding_pipeline", stages=[ ParquetReader(file_paths=input_path, files_per_partition=1, fields=["text"], _generate_ids=True), + # VLLMEmbeddingModelStage uses shorter parameter names than the workflow wrapper: + # pretokenize (not embedding_pretokenize), vllm_init_kwargs (not embedding_vllm_init_kwargs), + # max_chars (not embedding_max_chars), cache_dir (not model_cache_dir) VLLMEmbeddingModelStage( model_identifier="google/embeddinggemma-300m", text_field="text", - embedding_field="embeddings", ), ParquetWriter(path=embedding_output_path, fields=["_curator_dedup_id", "embeddings"]), ], @@ -138,14 +138,15 @@ Compare semantic deduplication with other methods: | Parameter | Type | Default | Description | | --- | --- | --- | --- | -| `model_identifier` | str | `"google/embeddinggemma-300m"` | Pre-trained model for embedding generation (uses vLLM backend) | +| `model_identifier` | str | `"google/embeddinggemma-300m"` | Pre-trained model for embedding generation (vLLM backend) | | `embedding_pretokenize` | bool | `False` | Whether to pre-tokenize input before passing to vLLM | -| `embedding_vllm_init_kwargs` | dict | `None` | Additional kwargs passed to vLLM's `LLM` initializer (for example, `{"enforce_eager": True, "max_model_len": 2048}`) | -| `model_cache_dir` | str | `None` | Directory to cache model weights (uses default HuggingFace cache if not set) | +| `embedding_vllm_init_kwargs` | dict | `None` | Additional keyword arguments passed to the vLLM `LLM` initializer | +| `embedding_max_chars` | int | `None` | Maximum number of characters for text truncation | +| `model_cache_dir` | str | `None` | Directory to cache model weights | | `n_clusters` | int | `100` | Number of clusters for k-means clustering | | `kmeans_max_iter` | int | `300` | Maximum iterations for clustering | | `eps` | float | `0.01` | Threshold for deduplication (higher = more aggressive) | -| `which_to_keep` | str | `"hard"` | Strategy for keeping duplicates ("hard"/"easy"/"random") | +| `which_to_keep` | str | `"hard"` | Strategy for keeping duplicates ("hard", "easy", or "random") | | `pairwise_batch_size` | int | `1024` | Batch size for similarity computation | | `distance_metric` | str | `"cosine"` | Distance metric for similarity ("cosine" or "l2") | | `perform_removal` | bool | `True` | Whether to perform duplicate removal | @@ -163,35 +164,37 @@ Experiment with different values to balance data reduction and dataset diversity -The default embedding backend uses **vLLM** with `google/embeddinggemma-300m`. You can use any HuggingFace embedding model that vLLM supports. +Embedding generation uses vLLM as the inference backend. The default model is `google/embeddinggemma-300m`. **Default (vLLM)**: ```python workflow = TextSemanticDeduplicationWorkflow( - model_identifier="google/embeddinggemma-300m", - # ... other parameters + # Uses google/embeddinggemma-300m by default + input_path="input_data/", + output_path="./results", + cache_path="./sem_cache", ) ``` -**Custom vLLM configuration**: +**Custom model with vLLM options**: ```python workflow = TextSemanticDeduplicationWorkflow( model_identifier="google/embeddinggemma-300m", embedding_pretokenize=True, embedding_vllm_init_kwargs={"enforce_eager": True, "max_model_len": 2048}, - model_cache_dir="/path/to/model/cache", # ... other parameters ) ``` **When choosing a model**: -- Use models trained for sentence embeddings, such as EmbeddingGemma, E5, BGE, or SBERT -- Avoid generic decoder-only LLMs such as OPT or GPT for embeddings +- Use models that support vLLM pooling (embedding) mode - Choose models appropriate for your language or domain -- Use `embedding_vllm_init_kwargs` to tune vLLM behavior (for example, `max_model_len`, `enforce_eager`) +- Prefer models trained for sentence embeddings (for example, EmbeddingGemma, E5, BGE, or SBERT) +- Use `embedding_pretokenize=True` for models that benefit from explicit tokenization control +- Pass additional vLLM configuration through `embedding_vllm_init_kwargs` @@ -207,7 +210,7 @@ workflow = TextSemanticDeduplicationWorkflow( text_field="text", model_identifier="google/embeddinggemma-300m", embedding_pretokenize=False, - embedding_vllm_init_kwargs={"enforce_eager": True, "max_model_len": 2048}, + embedding_max_chars=None, model_cache_dir=None, # Deduplication @@ -221,7 +224,7 @@ workflow = TextSemanticDeduplicationWorkflow( kmeans_tol=1e-4, pairwise_batch_size=1024, - perform_removal=True + perform_removal=True, ) ```