diff --git a/fern/versions/v26.04/pages/about/release-notes/index.mdx b/fern/versions/v26.04/pages/about/release-notes/index.mdx
index e01437b78b..527e9d888e 100644
--- a/fern/versions/v26.04/pages/about/release-notes/index.mdx
+++ b/fern/versions/v26.04/pages/about/release-notes/index.mdx
@@ -24,14 +24,14 @@ Built-in LLM serving alongside curation pipelines using Ray Serve and vLLM:
Learn more in the [Inference Server](/curate-text/synthetic/inference-server) documentation.
-### vLLM Default for Semantic Deduplication Embeddings
+### vLLM Default for Semantic Deduplication Embeddings (PR #1606)
-The default embedding backend for `TextSemanticDeduplicationWorkflow` switched from SentenceTransformers to vLLM:
+Switched the default embedding backend in `TextSemanticDeduplicationWorkflow` from SentenceTransformers to vLLM, with `google/embeddinggemma-300m` as the new default model:
-- **Default model**: `google/embeddinggemma-300m` (previously `sentence-transformers/all-MiniLM-L6-v2`)
-- **Backend**: `VLLMEmbeddingModelStage` replaces `EmbeddingCreatorStage` for embedding generation in the semantic dedup workflow
-- **New parameters**: `embedding_pretokenize`, `embedding_vllm_init_kwargs`, `model_cache_dir` replace the previous SentenceTransformers-specific parameters
-- **Bug fix**: Resolved CUDA fork error when running vLLM stages with `RayDataExecutor` by setting `VLLM_WORKER_MULTIPROC_METHOD=spawn` in Ray remote tasks
+- **vLLM embedding backend**: `TextSemanticDeduplicationWorkflow` now uses `VLLMEmbeddingModelStage` for embedding generation, replacing `EmbeddingCreatorStage` (SentenceTransformers).
+- **New default model**: Changed from `sentence-transformers/all-MiniLM-L6-v2` to `google/embeddinggemma-300m`.
+- **New parameters**: Added `embedding_pretokenize`, `embedding_vllm_init_kwargs`, and `model_cache_dir` for vLLM configuration.
+- **Removed parameters**: The SentenceTransformers-specific parameters `embedding_model_inference_batch_size`, `embedding_pooling`, `embedding_padding_side`, and `embedding_max_seq_length` are no longer available.
### Multi-User Metrics Isolation
@@ -75,6 +75,10 @@ Added tqdm progress bars to `RayActorPoolExecutor` for real-time visibility into
## Bug Fixes
+### CUDA Fork Error with vLLM and RayDataExecutor (PR #1606)
+
+Fixed a `RuntimeError: Cannot re-initialize CUDA in forked subprocess` error that occurred when running vLLM stages with `RayDataExecutor`. The vLLM auto-detection for `spawn` versus `fork` multiprocessing only triggers inside Ray actors, not Ray tasks. The `RayDataExecutor.execute_setup_on_node` method dispatches `setup_on_node` as a remote task, so vLLM previously defaulted to `fork` and caused a CUDA reinitialization error. Fixed by setting `VLLM_WORKER_MULTIPROC_METHOD=spawn` in the remote task.
+
### Audio Stage Name Propagation (PR #1470)
Fixed audio pipeline stage names not propagating in `StagePerfStats`, making benchmark output unable to identify per-stage timing. All audio stages (`GetAudioDurationStage`, `PreserveByValueStage`, `AudioToDocumentStage`, `GetPairwiseWerStage`) now correctly report their names, and stage performance history persists when stages create new task objects.
@@ -85,8 +89,7 @@ Fixed a race condition in `CaptionGenerationStage` and `CaptionEnhancementStage`
## Breaking Changes
-- **`TextSemanticDeduplicationWorkflow` parameter changes**: `embedding_max_seq_length`, `embedding_padding_side`, `embedding_pooling`, and `embedding_model_inference_batch_size` are removed. Use `embedding_pretokenize`, `embedding_vllm_init_kwargs`, and `model_cache_dir` instead.
-- **Default embedding model changed**: `TextSemanticDeduplicationWorkflow` now defaults to `google/embeddinggemma-300m` with vLLM instead of `sentence-transformers/all-MiniLM-L6-v2` with SentenceTransformers.
+- **`TextSemanticDeduplicationWorkflow` embedding backend**: The default embedding backend changed from SentenceTransformers to vLLM. The default model changed from `sentence-transformers/all-MiniLM-L6-v2` to `google/embeddinggemma-300m`. The parameters `embedding_model_inference_batch_size`, `embedding_pooling`, `embedding_padding_side`, and `embedding_max_seq_length` have been removed. Use `embedding_vllm_init_kwargs` to pass configuration to the vLLM backend instead.
- **`Resources` API**: The `nvdecs`, `nvencs`, and `entire_gpu` fields have been removed from `Resources`. Stages that previously used `entire_gpu=True` should use `gpus=1` instead. Stages that used `nvdecs` or `nvencs` should use `gpus` for GPU allocation.
- **`ExactDeduplicationWorkflow.run()` and `FuzzyDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `None`
- **`SemanticDeduplicationWorkflow.run()` and `TextSemanticDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `dict`
diff --git a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx
index 4885569a14..95d9886282 100644
--- a/fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx
+++ b/fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx
@@ -51,7 +51,6 @@ Get started with semantic deduplication using the following example of identifyi
```python
from nemo_curator.stages.text.deduplication.semantic import TextSemanticDeduplicationWorkflow
-from nemo_curator.backends.experimental.ray_data import RayDataExecutor
# Default: uses vLLM with google/embeddinggemma-300m
workflow = TextSemanticDeduplicationWorkflow(
@@ -61,12 +60,11 @@ workflow = TextSemanticDeduplicationWorkflow(
n_clusters=100,
eps=0.07, # Similarity threshold
id_field="doc_id",
- perform_removal=True # Complete deduplication
+ perform_removal=True, # Complete deduplication
)
-executor = RayDataExecutor()
-result = workflow.run(executor)
-# result.metadata contains: total_time, num_duplicates, num_duplicates_removed, embedding_time, identification_time, removal_time, final_output_path
+results = workflow.run()
+# Clean dataset saved to ./results/deduplicated/
```
## Configuration
@@ -80,7 +78,7 @@ For fine-grained control, break semantic deduplication into separate stages:
```python
from nemo_curator.stages.deduplication.id_generator import create_id_generator_actor
from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage
-from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow, IdentifyDuplicatesStage
+from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow
from nemo_curator.stages.text.deduplication.removal_workflow import TextDuplicatesRemovalWorkflow
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import ParquetReader
@@ -94,10 +92,12 @@ embedding_pipeline = Pipeline(
name="embedding_pipeline",
stages=[
ParquetReader(file_paths=input_path, files_per_partition=1, fields=["text"], _generate_ids=True),
+ # VLLMEmbeddingModelStage uses shorter parameter names than the workflow wrapper:
+ # pretokenize (not embedding_pretokenize), vllm_init_kwargs (not embedding_vllm_init_kwargs),
+ # max_chars (not embedding_max_chars), cache_dir (not model_cache_dir)
VLLMEmbeddingModelStage(
model_identifier="google/embeddinggemma-300m",
text_field="text",
- embedding_field="embeddings",
),
ParquetWriter(path=embedding_output_path, fields=["_curator_dedup_id", "embeddings"]),
],
@@ -138,14 +138,15 @@ Compare semantic deduplication with other methods:
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
-| `model_identifier` | str | `"google/embeddinggemma-300m"` | Pre-trained model for embedding generation (uses vLLM backend) |
+| `model_identifier` | str | `"google/embeddinggemma-300m"` | Pre-trained model for embedding generation (vLLM backend) |
| `embedding_pretokenize` | bool | `False` | Whether to pre-tokenize input before passing to vLLM |
-| `embedding_vllm_init_kwargs` | dict | `None` | Additional kwargs passed to vLLM's `LLM` initializer (for example, `{"enforce_eager": True, "max_model_len": 2048}`) |
-| `model_cache_dir` | str | `None` | Directory to cache model weights (uses default HuggingFace cache if not set) |
+| `embedding_vllm_init_kwargs` | dict | `None` | Additional keyword arguments passed to the vLLM `LLM` initializer |
+| `embedding_max_chars` | int | `None` | Maximum number of characters for text truncation |
+| `model_cache_dir` | str | `None` | Directory to cache model weights |
| `n_clusters` | int | `100` | Number of clusters for k-means clustering |
| `kmeans_max_iter` | int | `300` | Maximum iterations for clustering |
| `eps` | float | `0.01` | Threshold for deduplication (higher = more aggressive) |
-| `which_to_keep` | str | `"hard"` | Strategy for keeping duplicates ("hard"/"easy"/"random") |
+| `which_to_keep` | str | `"hard"` | Strategy for keeping duplicates ("hard", "easy", or "random") |
| `pairwise_batch_size` | int | `1024` | Batch size for similarity computation |
| `distance_metric` | str | `"cosine"` | Distance metric for similarity ("cosine" or "l2") |
| `perform_removal` | bool | `True` | Whether to perform duplicate removal |
@@ -163,35 +164,37 @@ Experiment with different values to balance data reduction and dataset diversity
-The default embedding backend uses **vLLM** with `google/embeddinggemma-300m`. You can use any HuggingFace embedding model that vLLM supports.
+Embedding generation uses vLLM as the inference backend. The default model is `google/embeddinggemma-300m`.
**Default (vLLM)**:
```python
workflow = TextSemanticDeduplicationWorkflow(
- model_identifier="google/embeddinggemma-300m",
- # ... other parameters
+ # Uses google/embeddinggemma-300m by default
+ input_path="input_data/",
+ output_path="./results",
+ cache_path="./sem_cache",
)
```
-**Custom vLLM configuration**:
+**Custom model with vLLM options**:
```python
workflow = TextSemanticDeduplicationWorkflow(
model_identifier="google/embeddinggemma-300m",
embedding_pretokenize=True,
embedding_vllm_init_kwargs={"enforce_eager": True, "max_model_len": 2048},
- model_cache_dir="/path/to/model/cache",
# ... other parameters
)
```
**When choosing a model**:
-- Use models trained for sentence embeddings, such as EmbeddingGemma, E5, BGE, or SBERT
-- Avoid generic decoder-only LLMs such as OPT or GPT for embeddings
+- Use models that support vLLM pooling (embedding) mode
- Choose models appropriate for your language or domain
-- Use `embedding_vllm_init_kwargs` to tune vLLM behavior (for example, `max_model_len`, `enforce_eager`)
+- Prefer models trained for sentence embeddings (for example, EmbeddingGemma, E5, BGE, or SBERT)
+- Use `embedding_pretokenize=True` for models that benefit from explicit tokenization control
+- Pass additional vLLM configuration through `embedding_vllm_init_kwargs`
@@ -207,7 +210,7 @@ workflow = TextSemanticDeduplicationWorkflow(
text_field="text",
model_identifier="google/embeddinggemma-300m",
embedding_pretokenize=False,
- embedding_vllm_init_kwargs={"enforce_eager": True, "max_model_len": 2048},
+ embedding_max_chars=None,
model_cache_dir=None,
# Deduplication
@@ -221,7 +224,7 @@ workflow = TextSemanticDeduplicationWorkflow(
kmeans_tol=1e-4,
pairwise_batch_size=1024,
- perform_removal=True
+ perform_removal=True,
)
```