Skip to content

docs: update fern docs for vLLM default in semantic dedup (PR #1606) #1704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lbliii merged 1 commit into from
Apr 3, 2026
Merged

docs: update fern docs for vLLM default in semantic dedup (PR #1606) #1704

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions fern/versions/v26.04/pages/about/release-notes/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ Built-in LLM serving alongside curation pipelines using Ray Serve and vLLM:

Learn more in the [Inference Server](/curate-text/synthetic/inference-server) documentation.

### vLLM Default for Semantic Deduplication Embeddings
### vLLM Default for Semantic Deduplication Embeddings (PR #1606)

The default embedding backend for `TextSemanticDeduplicationWorkflow` switched from SentenceTransformers to vLLM:
Switched the default embedding backend in `TextSemanticDeduplicationWorkflow` from SentenceTransformers to vLLM, with `google/embeddinggemma-300m` as the new default model:

- **Default model**: `google/embeddinggemma-300m` (previously `sentence-transformers/all-MiniLM-L6-v2`)
- **Backend**: `VLLMEmbeddingModelStage` replaces `EmbeddingCreatorStage` for embedding generation in the semantic dedup workflow
- **New parameters**: `embedding_pretokenize`, `embedding_vllm_init_kwargs`, `model_cache_dir` replace the previous SentenceTransformers-specific parameters
- **Bug fix**: Resolved CUDA fork error when running vLLM stages with `RayDataExecutor` by setting `VLLM_WORKER_MULTIPROC_METHOD=spawn` in Ray remote tasks
- **vLLM embedding backend**: `TextSemanticDeduplicationWorkflow` now uses `VLLMEmbeddingModelStage` for embedding generation, replacing `EmbeddingCreatorStage` (SentenceTransformers).
- **New default model**: Changed from `sentence-transformers/all-MiniLM-L6-v2` to `google/embeddinggemma-300m`.
- **New parameters**: Added `embedding_pretokenize`, `embedding_vllm_init_kwargs`, and `model_cache_dir` for vLLM configuration.
- **Removed parameters**: The SentenceTransformers-specific parameters `embedding_model_inference_batch_size`, `embedding_pooling`, `embedding_padding_side`, and `embedding_max_seq_length` are no longer available.

### Multi-User Metrics Isolation

Expand Down Expand Up @@ -75,6 +75,10 @@ Added tqdm progress bars to `RayActorPoolExecutor` for real-time visibility into

## Bug Fixes

### CUDA Fork Error with vLLM and RayDataExecutor (PR #1606)

Fixed a `RuntimeError: Cannot re-initialize CUDA in forked subprocess` error that occurred when running vLLM stages with `RayDataExecutor`. The vLLM auto-detection for `spawn` versus `fork` multiprocessing only triggers inside Ray actors, not Ray tasks. The `RayDataExecutor.execute_setup_on_node` method dispatches `setup_on_node` as a remote task, so vLLM previously defaulted to `fork` and caused a CUDA reinitialization error. Fixed by setting `VLLM_WORKER_MULTIPROC_METHOD=spawn` in the remote task.

### Audio Stage Name Propagation (PR #1470)

Fixed audio pipeline stage names not propagating in `StagePerfStats`, making benchmark output unable to identify per-stage timing. All audio stages (`GetAudioDurationStage`, `PreserveByValueStage`, `AudioToDocumentStage`, `GetPairwiseWerStage`) now correctly report their names, and stage performance history persists when stages create new task objects.
Expand All @@ -85,8 +89,7 @@ Fixed a race condition in `CaptionGenerationStage` and `CaptionEnhancementStage`

## Breaking Changes

- **`TextSemanticDeduplicationWorkflow` parameter changes**: `embedding_max_seq_length`, `embedding_padding_side`, `embedding_pooling`, and `embedding_model_inference_batch_size` are removed. Use `embedding_pretokenize`, `embedding_vllm_init_kwargs`, and `model_cache_dir` instead.
- **Default embedding model changed**: `TextSemanticDeduplicationWorkflow` now defaults to `google/embeddinggemma-300m` with vLLM instead of `sentence-transformers/all-MiniLM-L6-v2` with SentenceTransformers.
- **`TextSemanticDeduplicationWorkflow` embedding backend**: The default embedding backend changed from SentenceTransformers to vLLM. The default model changed from `sentence-transformers/all-MiniLM-L6-v2` to `google/embeddinggemma-300m`. The parameters `embedding_model_inference_batch_size`, `embedding_pooling`, `embedding_padding_side`, and `embedding_max_seq_length` have been removed. Use `embedding_vllm_init_kwargs` to pass configuration to the vLLM backend instead.
- **`Resources` API**: The `nvdecs`, `nvencs`, and `entire_gpu` fields have been removed from `Resources`. Stages that previously used `entire_gpu=True` should use `gpus=1` instead. Stages that used `nvdecs` or `nvencs` should use `gpus` for GPU allocation.
- **`ExactDeduplicationWorkflow.run()` and `FuzzyDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `None`
- **`SemanticDeduplicationWorkflow.run()` and `TextSemanticDeduplicationWorkflow.run()`** now return `WorkflowRunResult` instead of `dict`
Expand Down
45 changes: 24 additions & 21 deletions fern/versions/v26.04/pages/curate-text/process-data/deduplication/semdedup.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ Get started with semantic deduplication using the following example of identifyi

```python
from nemo_curator.stages.text.deduplication.semantic import TextSemanticDeduplicationWorkflow
from nemo_curator.backends.experimental.ray_data import RayDataExecutor

# Default: uses vLLM with google/embeddinggemma-300m
workflow = TextSemanticDeduplicationWorkflow(
Expand All @@ -61,12 +60,11 @@ workflow = TextSemanticDeduplicationWorkflow(
n_clusters=100,
eps=0.07, # Similarity threshold
id_field="doc_id",
perform_removal=True # Complete deduplication
perform_removal=True, # Complete deduplication
)

executor = RayDataExecutor()
result = workflow.run(executor)
# result.metadata contains: total_time, num_duplicates, num_duplicates_removed, embedding_time, identification_time, removal_time, final_output_path
results = workflow.run()
# Clean dataset saved to ./results/deduplicated/
```

## Configuration
Expand All @@ -80,7 +78,7 @@ For fine-grained control, break semantic deduplication into separate stages:
```python
from nemo_curator.stages.deduplication.id_generator import create_id_generator_actor
from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage
from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow, IdentifyDuplicatesStage
from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow
from nemo_curator.stages.text.deduplication.removal_workflow import TextDuplicatesRemovalWorkflow
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import ParquetReader
Expand All @@ -94,10 +92,12 @@ embedding_pipeline = Pipeline(
name="embedding_pipeline",
stages=[
ParquetReader(file_paths=input_path, files_per_partition=1, fields=["text"], _generate_ids=True),
# VLLMEmbeddingModelStage uses shorter parameter names than the workflow wrapper:
# pretokenize (not embedding_pretokenize), vllm_init_kwargs (not embedding_vllm_init_kwargs),
# max_chars (not embedding_max_chars), cache_dir (not model_cache_dir)
VLLMEmbeddingModelStage(
model_identifier="google/embeddinggemma-300m",
text_field="text",
embedding_field="embeddings",
),
Comment on lines 80 to 101

@greptile-apps greptile-apps Bot Apr 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Direct VLLMEmbeddingModelStage parameter names differ from workflow wrapper names

The step-by-step example instantiates VLLMEmbeddingModelStage directly, but the Key Parameters table lists the TextSemanticDeduplicationWorkflow wrapper names (embedding_pretokenize, embedding_vllm_init_kwargs, embedding_max_chars, model_cache_dir). A user reading the step-by-step section and wanting to customise the stage will find that these wrapper names are not valid constructor parameters on VLLMEmbeddingModelStage itself.

Looking at nemo_curator/stages/text/embedders/vllm.py, the actual constructor parameters are:

  • pretokenize (not embedding_pretokenize)
  • vllm_init_kwargs (not embedding_vllm_init_kwargs)
  • max_chars (not embedding_max_chars)
  • cache_dir (not model_cache_dir)

Consider adding a brief inline note to the Step-by-Step Workflow section clarifying that VLLMEmbeddingModelStage uses the shorter names, while TextSemanticDeduplicationWorkflow prefixes them with embedding_ / model_.

ParquetWriter(path=embedding_output_path, fields=["_curator_dedup_id", "embeddings"]),
],
Expand Down Expand Up @@ -138,14 +138,15 @@ Compare semantic deduplication with other methods:

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `model_identifier` | str | `"google/embeddinggemma-300m"` | Pre-trained model for embedding generation (uses vLLM backend) |
| `model_identifier` | str | `"google/embeddinggemma-300m"` | Pre-trained model for embedding generation (vLLM backend) |
| `embedding_pretokenize` | bool | `False` | Whether to pre-tokenize input before passing to vLLM |
| `embedding_vllm_init_kwargs` | dict | `None` | Additional kwargs passed to vLLM's `LLM` initializer (for example, `{"enforce_eager": True, "max_model_len": 2048}`) |
| `model_cache_dir` | str | `None` | Directory to cache model weights (uses default HuggingFace cache if not set) |
| `embedding_vllm_init_kwargs` | dict | `None` | Additional keyword arguments passed to the vLLM `LLM` initializer |
| `embedding_max_chars` | int | `None` | Maximum number of characters for text truncation |
| `model_cache_dir` | str | `None` | Directory to cache model weights |
| `n_clusters` | int | `100` | Number of clusters for k-means clustering |
| `kmeans_max_iter` | int | `300` | Maximum iterations for clustering |
| `eps` | float | `0.01` | Threshold for deduplication (higher = more aggressive) |
| `which_to_keep` | str | `"hard"` | Strategy for keeping duplicates ("hard"/"easy"/"random") |
| `which_to_keep` | str | `"hard"` | Strategy for keeping duplicates ("hard", "easy", or "random") |
| `pairwise_batch_size` | int | `1024` | Batch size for similarity computation |
| `distance_metric` | str | `"cosine"` | Distance metric for similarity ("cosine" or "l2") |
| `perform_removal` | bool | `True` | Whether to perform duplicate removal |
Expand All @@ -163,35 +164,37 @@ Experiment with different values to balance data reduction and dataset diversity

<Accordion title="Embedding Models">

The default embedding backend uses **vLLM** with `google/embeddinggemma-300m`. You can use any HuggingFace embedding model that vLLM supports.
Embedding generation uses vLLM as the inference backend. The default model is `google/embeddinggemma-300m`.

**Default (vLLM)**:

```python
workflow = TextSemanticDeduplicationWorkflow(
model_identifier="google/embeddinggemma-300m",
# ... other parameters
# Uses google/embeddinggemma-300m by default
input_path="input_data/",
output_path="./results",
cache_path="./sem_cache",
)
```

**Custom vLLM configuration**:
**Custom model with vLLM options**:

```python
workflow = TextSemanticDeduplicationWorkflow(
model_identifier="google/embeddinggemma-300m",
embedding_pretokenize=True,
embedding_vllm_init_kwargs={"enforce_eager": True, "max_model_len": 2048},
model_cache_dir="/path/to/model/cache",
# ... other parameters
)
```

**When choosing a model**:

- Use models trained for sentence embeddings, such as EmbeddingGemma, E5, BGE, or SBERT
- Avoid generic decoder-only LLMs such as OPT or GPT for embeddings
- Use models that support vLLM pooling (embedding) mode
- Choose models appropriate for your language or domain
- Use `embedding_vllm_init_kwargs` to tune vLLM behavior (for example, `max_model_len`, `enforce_eager`)
- Prefer models trained for sentence embeddings (for example, EmbeddingGemma, E5, BGE, or SBERT)
- Use `embedding_pretokenize=True` for models that benefit from explicit tokenization control
- Pass additional vLLM configuration through `embedding_vllm_init_kwargs`
</Accordion>

<Accordion title="Advanced Configuration">
Expand All @@ -207,7 +210,7 @@ workflow = TextSemanticDeduplicationWorkflow(
text_field="text",
model_identifier="google/embeddinggemma-300m",
embedding_pretokenize=False,
embedding_vllm_init_kwargs={"enforce_eager": True, "max_model_len": 2048},
embedding_max_chars=None,
model_cache_dir=None,

# Deduplication
Expand All @@ -221,7 +224,7 @@ workflow = TextSemanticDeduplicationWorkflow(
kmeans_tol=1e-4,
pairwise_batch_size=1024,

perform_removal=True
perform_removal=True,
)
```
</Accordion>
Expand Down
Loading