Repository navigation
docs: add text embeddings guide and release notes for PR #1346 #1687
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
d51b750
5090134
05d3ee4
023d668
1a19dd8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -12,6 +12,17 @@ modality: "universal" | |||||
|
|
||||||
| ## What's New in 26.04 | ||||||
|
|
||||||
| ### vLLM and Sentence Transformers Embedding Support (PR #1346) | ||||||
|
|
||||||
| Added two new embedding backends for text curation, giving users flexibility to choose the best engine for their model size and throughput needs: | ||||||
|
|
||||||
| - **`VLLMEmbeddingModelStage`**: A new standalone embedding stage powered by [vLLM](https://docs.vllm.ai/) for high-throughput GPU-accelerated inference. Supports optional pretokenization (`pretokenize=True`) for best per-task throughput. Ideal for large embedding models where vLLM's batching and memory management outperform Sentence Transformers. | ||||||
| - **`SentenceTransformerEmbeddingModelStage`**: A new embedding stage using the `sentence-transformers` library directly, providing native support for models from the Sentence Transformers ecosystem. | ||||||
| - **`EmbeddingCreatorStage` enhancements**: Added `use_sentence_transformer` flag (defaults to `True`) to select between Sentence Transformers and HuggingFace `AutoModel` backends. Added `cache_dir` parameter for controlling model download location. | ||||||
| - **New `vllm` optional dependency**: Install with `pip install nemo_curator[vllm]` (x86_64 Linux only). The `sentence-transformers` package is now included in the `text_cpu` extra. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Users should never install Curator with only the |
||||||
|
|
||||||
| For usage details, see [Text Embeddings](/curate-text/process-data/embeddings) and [vLLM Embedder](/curate-text/process-data/embeddings/vllm-embedder). | ||||||
|
|
||||||
| ### Cosmos-Xenna 0.2.0 (PR #1571) | ||||||
|
|
||||||
| Upgraded Cosmos-Xenna from 0.1.2 to 0.2.0 with a simplified resource model and improved GPU management: | ||||||
|
|
@@ -24,6 +35,8 @@ Upgraded Cosmos-Xenna from 0.1.2 to 0.2.0 with a simplified resource model and i | |||||
|
|
||||||
| - **Cosmos-Xenna**: Updated from 0.1.2 to 0.2.0 with simplified resource model | ||||||
| - **Ray**: Updated to 2.54 | ||||||
| - **sentence-transformers**: Added to the `text_cpu` optional dependency group | ||||||
| - **vllm**: New `vllm` optional dependency group (`pip install nemo_curator[vllm]`, x86_64 Linux only) | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| ## Breaking Changes | ||||||
|
|
||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -180,12 +180,55 @@ workflow = TextSemanticDeduplicationWorkflow( | |
| ) | ||
| ``` | ||
|
|
||
| **vLLM Embedder** (recommended for large models): | ||
|
|
||
| For large embedding models, you can generate embeddings separately using `VLLMEmbeddingModelStage` before running the deduplication workflow. This provides better GPU utilization and throughput for models with 500M+ parameters. See [vLLM Embedder](/curate-text/process-data/embeddings/vllm-embedder) for details. | ||
|
|
||
| ```python | ||
| from nemo_curator.backends.xenna import XennaExecutor | ||
| from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage | ||
| from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow | ||
| from nemo_curator.pipeline import Pipeline | ||
| from nemo_curator.stages.text.io.reader import ParquetReader | ||
| from nemo_curator.stages.text.io.writer import ParquetWriter | ||
|
|
||
| executor = XennaExecutor() | ||
|
|
||
| # Step 1: Generate embeddings with vLLM | ||
| embedding_pipeline = Pipeline( | ||
| name="vllm_embedding_pipeline", | ||
| stages=[ | ||
| ParquetReader(file_paths=input_path, files_per_partition=1, fields=["text"], _generate_ids=True), | ||
| VLLMEmbeddingModelStage( | ||
| model_identifier="large-embedding-model", | ||
| text_field="text", | ||
| embedding_field="embeddings", | ||
| pretokenize=True, | ||
| ), | ||
| ParquetWriter(path=embedding_output_path, fields=["_curator_dedup_id", "embeddings"]), | ||
| ], | ||
| ) | ||
| embedding_pipeline.run(executor) | ||
|
|
||
| # Step 2: Run deduplication on pre-computed embeddings | ||
| semantic_workflow = SemanticDeduplicationWorkflow( | ||
| input_path=embedding_output_path, | ||
| output_path=output_path, | ||
| n_clusters=100, | ||
| eps=0.07, | ||
| id_field="_curator_dedup_id", | ||
| embedding_field="embeddings", | ||
| ) | ||
| semantic_workflow.run() | ||
|
Comment on lines
+197
to
+209
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The new vLLM embedding code block is missing several imports and uses an undefined variable
A standalone, self-contained snippet should include all its imports: from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage
from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow
from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import ParquetReader
from nemo_curator.stages.text.io.writer import ParquetWriter
executor = XennaExecutor()
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The existing Step-by-Step Workflow accordion on this same page acknowledges this with Consider adding a placeholder comment (or a link to semantic_workflow.run()
# Step 3: Filter original text dataset using the IDs to remove
# See TextDuplicatesRemovalWorkflow for the removal step |
||
| ``` | ||
|
|
||
| **When choosing a model**: | ||
|
|
||
| - Ensure compatibility with your data type | ||
| - Adjust `embedding_model_inference_batch_size` for memory requirements | ||
| - Choose models appropriate for your language or domain | ||
| - Avoid generic decoder-only LLMs (e.g., OPT/GPT) for embeddings; prefer models trained for sentence embeddings (e.g., E5/BGE/SBERT) | ||
| - For large models (>500M params), consider using [VLLMEmbeddingModelStage](/curate-text/process-data/embeddings/vllm-embedder) for better throughput | ||
| </Accordion> | ||
|
|
||
| <Accordion title="Advanced Configuration"> | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,107 @@ | ||||||
| --- | ||||||
| description: "Generate text embeddings using vLLM, Sentence Transformers, or HuggingFace models for deduplication, similarity search, and downstream tasks" | ||||||
| categories: ["how-to-guides"] | ||||||
| tags: ["embeddings", "vllm", "sentence-transformers", "gpu-accelerated", "similarity-search"] | ||||||
| personas: ["data-scientist-focused", "mle-focused"] | ||||||
| difficulty: "intermediate" | ||||||
| content_type: "how-to" | ||||||
| modality: "text-only" | ||||||
| --- | ||||||
|
|
||||||
| # Text Embedding | ||||||
|
|
||||||
| Generate text embeddings for large-scale datasets using NeMo Curator's built-in embedding stages. Text embeddings enable downstream tasks such as semantic deduplication, similarity search, and clustering. | ||||||
|
|
||||||
| ## How It Works | ||||||
|
|
||||||
| NeMo Curator provides three embedding backends for text data, each suited to different model sizes and throughput requirements: | ||||||
|
|
||||||
| 1. **`EmbeddingCreatorStage`** — A composite stage that handles tokenization and embedding in sequence. Supports both Sentence Transformers and HuggingFace `AutoModel` backends via the `use_sentence_transformer` flag. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| 2. **`VLLMEmbeddingModelStage`** — A standalone stage that uses vLLM for GPU-accelerated embedding generation with optional pretokenization. Best for large embedding models where vLLM's batching and GPU utilization provide significant throughput gains. | ||||||
| 3. **`SentenceTransformerEmbeddingModelStage`** — A model stage that uses the `sentence-transformers` library directly. Used internally by `EmbeddingCreatorStage` when `use_sentence_transformer=True`. | ||||||
|
|
||||||
| ## Choosing an Embedding Backend | ||||||
|
|
||||||
| | Backend | Best For | GPU Utilization | Setup | | ||||||
| | --- | --- | --- | --- | | ||||||
| | `EmbeddingCreatorStage` (Sentence Transformers) | Small to medium models (e.g., all-MiniLM-L6-v2) | Good | Minimal — included in `text_cpu` extra | | ||||||
| | `VLLMEmbeddingModelStage` | Large models (e.g., Gemma-based embedders) | Excellent | Requires `vllm` extra (`pip install nemo_curator[vllm]`) | | ||||||
| | `EmbeddingCreatorStage` (AutoModel) | Custom pooling strategies | Good | Set `use_sentence_transformer=False` | | ||||||
|
sarahyurick marked this conversation as resolved.
Resolved
|
||||||
|
|
||||||
| <Note> | ||||||
| Benchmarks on 5 GB of Common Crawl data show that vLLM outperforms Sentence Transformers for larger embedding models, while Sentence Transformers is faster for smaller models. The vLLM `pretokenize` mode provides the best per-task throughput across both model sizes when amortized over many tasks. | ||||||
| </Note> | ||||||
|
|
||||||
| ## Quick Start | ||||||
|
|
||||||
| ### EmbeddingCreatorStage (Recommended for Most Use Cases) | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The new semantic deduplication default is vLLM, so that should be the recommended case. |
||||||
|
|
||||||
| ```python | ||||||
| from nemo_curator.backends.xenna import XennaExecutor | ||||||
| from nemo_curator.stages.text.embedders import EmbeddingCreatorStage | ||||||
| from nemo_curator.pipeline import Pipeline | ||||||
| from nemo_curator.stages.text.io.reader import ParquetReader | ||||||
| from nemo_curator.stages.text.io.writer import ParquetWriter | ||||||
|
|
||||||
| pipeline = Pipeline( | ||||||
| name="text_embeddings", | ||||||
| stages=[ | ||||||
| ParquetReader(file_paths="input_data/", files_per_partition=1, fields=["text"]), | ||||||
| EmbeddingCreatorStage( | ||||||
| model_identifier="sentence-transformers/all-MiniLM-L6-v2", | ||||||
|
sarahyurick marked this conversation as resolved.
Resolved
|
||||||
| text_field="text", | ||||||
| embedding_field="embeddings", | ||||||
| model_inference_batch_size=256, | ||||||
| ), | ||||||
| ParquetWriter(path="output/", fields=["text", "embeddings"]), | ||||||
| ], | ||||||
| ) | ||||||
|
|
||||||
| executor = XennaExecutor() | ||||||
| pipeline.run(executor) | ||||||
|
Comment on lines
+60
to
+61
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This applies to both the from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.embedders import EmbeddingCreatorStage
from nemo_curator.stages.text.io.reader import ParquetReader
from nemo_curator.stages.text.io.writer import ParquetWriterThe same missing import also affects |
||||||
| ``` | ||||||
|
|
||||||
| ### VLLMEmbeddingModelStage (For Large Models) | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Embedding Gemma is the new default for semantic deduplication. |
||||||
|
|
||||||
| ```python | ||||||
| from nemo_curator.backends.xenna import XennaExecutor | ||||||
| from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage | ||||||
| from nemo_curator.pipeline import Pipeline | ||||||
| from nemo_curator.stages.text.io.reader import ParquetReader | ||||||
| from nemo_curator.stages.text.io.writer import ParquetWriter | ||||||
|
|
||||||
| pipeline = Pipeline( | ||||||
| name="vllm_embeddings", | ||||||
| stages=[ | ||||||
| ParquetReader(file_paths="input_data/", files_per_partition=1, fields=["text"]), | ||||||
| VLLMEmbeddingModelStage( | ||||||
| model_identifier="google/gemma-embedding-model", | ||||||
| text_field="text", | ||||||
| embedding_field="embeddings", | ||||||
| pretokenize=True, # Recommended for best throughput | ||||||
| ), | ||||||
| ParquetWriter(path="output/", fields=["text", "embeddings"]), | ||||||
| ], | ||||||
| ) | ||||||
|
|
||||||
| executor = XennaExecutor() | ||||||
| pipeline.run(executor) | ||||||
| ``` | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Available Embedding Tools | ||||||
|
|
||||||
| <Cards> | ||||||
|
|
||||||
| <Card title="vLLM Embedder" href="/curate-text/process-data/embeddings/vllm-embedder"> | ||||||
| Generate embeddings using vLLM for high-throughput GPU-accelerated inference with large embedding models. | ||||||
| </Card> | ||||||
|
|
||||||
| </Cards> | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Integration with Semantic Deduplication | ||||||
|
|
||||||
| Text embeddings are a key input for [semantic deduplication](/curate-text/process-data/deduplication/semdedup). The `TextSemanticDeduplicationWorkflow` uses `EmbeddingCreatorStage` internally, but you can also generate embeddings separately and feed them into the deduplication workflow for more control over the embedding process. | ||||||
|
lbliii marked this conversation as resolved.
Resolved
|
||||||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,138 @@ | ||||||
| --- | ||||||
| description: "Generate text embeddings using vLLM for high-throughput GPU-accelerated inference with large embedding models" | ||||||
| categories: ["how-to-guides"] | ||||||
| tags: ["embeddings", "vllm", "gpu-accelerated", "large-models"] | ||||||
| personas: ["data-scientist-focused", "mle-focused"] | ||||||
| difficulty: "intermediate" | ||||||
| content_type: "how-to" | ||||||
| modality: "text-only" | ||||||
| --- | ||||||
|
|
||||||
| # vLLM Embedder | ||||||
|
|
||||||
| Generate text embeddings using vLLM's optimized inference engine. The `VLLMEmbeddingModelStage` provides high-throughput embedding generation, particularly for large embedding models where vLLM's batching and GPU memory management provide significant performance advantages over Sentence Transformers. | ||||||
|
|
||||||
| <Note> | ||||||
| **Installation**: The vLLM embedder requires the `vllm` optional dependency. Install it with: | ||||||
|
|
||||||
| ```bash | ||||||
| pip install nemo_curator[vllm] | ||||||
| ``` | ||||||
|
|
||||||
| vLLM is only available on x86_64 Linux systems. | ||||||
| </Note> | ||||||
|
sarahyurick marked this conversation as resolved.
Resolved
|
||||||
|
|
||||||
| ## How It Works | ||||||
|
|
||||||
| `VLLMEmbeddingModelStage` is a single-stage embedder that handles both tokenization and embedding generation within one stage. Unlike `EmbeddingCreatorStage` (which splits tokenization and model inference into separate stages), the vLLM embedder delegates all GPU operations to vLLM's inference engine. | ||||||
|
|
||||||
| Key features: | ||||||
|
|
||||||
| - **Optional pretokenization**: When `pretokenize=True`, the stage tokenizes text on CPU before passing tokens to vLLM, reducing GPU idle time and improving throughput | ||||||
| - **vLLM-managed batching**: Leverages vLLM's built-in request scheduling for optimal GPU utilization | ||||||
| - **Model download caching**: Automatically downloads and caches models from HuggingFace Hub | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| - **Character truncation**: Optional `max_chars` parameter to limit input length before tokenization | ||||||
|
|
||||||
| ## Quick Start | ||||||
|
|
||||||
| ```python | ||||||
| from nemo_curator.backends.xenna import XennaExecutor | ||||||
| from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage | ||||||
| from nemo_curator.pipeline import Pipeline | ||||||
| from nemo_curator.stages.text.io.reader import ParquetReader | ||||||
| from nemo_curator.stages.text.io.writer import ParquetWriter | ||||||
|
|
||||||
| pipeline = Pipeline( | ||||||
| name="vllm_embeddings", | ||||||
| stages=[ | ||||||
| ParquetReader(file_paths="input_data/", files_per_partition=1, fields=["text"]), | ||||||
| VLLMEmbeddingModelStage( | ||||||
| model_identifier="google/gemma-embedding-model", | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is not the correct identifier. |
||||||
| text_field="text", | ||||||
| embedding_field="embeddings", | ||||||
| pretokenize=True, | ||||||
| ), | ||||||
| ParquetWriter(path="output/", fields=["text", "embeddings"]), | ||||||
|
Comment on lines
+48
to
+54
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The Quick Start example uses Consider replacing the model identifier with a large embedding model or a descriptive placeholder like |
||||||
| ], | ||||||
| ) | ||||||
|
|
||||||
| executor = XennaExecutor() | ||||||
| pipeline.run(executor) | ||||||
| ``` | ||||||
|
|
||||||
| ## Configuration | ||||||
|
|
||||||
| ### Parameters | ||||||
|
|
||||||
| | Parameter | Type | Default | Description | | ||||||
| | --- | --- | --- | --- | | ||||||
| | `model_identifier` | `str` | Required | HuggingFace model name or path for the embedding model | | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| | `vllm_init_kwargs` | `dict` | `None` | Additional keyword arguments passed to `vllm.LLM()` for engine configuration | | ||||||
| | `text_field` | `str` | `"text"` | Name of the input text column in the data | | ||||||
| | `pretokenize` | `bool` | `False` | Tokenize text on CPU before passing to vLLM. Recommended for best throughput | | ||||||
| | `embedding_field` | `str` | `"embeddings"` | Name of the output embedding column | | ||||||
| | `max_chars` | `int` | `None` | Maximum characters per document (truncates before tokenization) | | ||||||
| | `cache_dir` | `str` | `None` | Directory for caching downloaded model files | | ||||||
| | `hf_token` | `str` | `None` | HuggingFace token for accessing gated models | | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| | `verbose` | `bool` | `False` | Enable verbose logging and progress bars | | ||||||
|
|
||||||
| ### vLLM Engine Options | ||||||
|
|
||||||
| Pass additional vLLM configuration through `vllm_init_kwargs`: | ||||||
|
|
||||||
| ```python | ||||||
| VLLMEmbeddingModelStage( | ||||||
| model_identifier="large-embedding-model", | ||||||
| pretokenize=True, | ||||||
| vllm_init_kwargs={ | ||||||
| "enforce_eager": True, # Disable CUDA graph for debugging | ||||||
| "tensor_parallel_size": 2, # Distribute across 2 GPUs | ||||||
| "gpu_memory_utilization": 0.9, | ||||||
| "max_model_len": 512, | ||||||
| }, | ||||||
| ) | ||||||
| ``` | ||||||
|
|
||||||
| Default vLLM settings applied by the stage (can be overridden): | ||||||
|
|
||||||
| - `enforce_eager=False` — Uses CUDA graphs for faster inference | ||||||
| - `runner="pooling"` — Configures vLLM for embedding (pooling) tasks | ||||||
| - `model_impl="vllm"` — Uses vLLM's native model implementation | ||||||
| - `disable_log_stats=True` — Suppresses stats logging when `verbose=False` | ||||||
|
|
||||||
| ### Pretokenization | ||||||
|
|
||||||
| When `pretokenize=True`, the stage: | ||||||
|
|
||||||
| 1. Loads a HuggingFace `AutoTokenizer` for the specified model | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| 2. Tokenizes the input text batch on CPU with truncation to `max_model_len` | ||||||
| 3. Passes token IDs directly to vLLM using `TokensPrompt` | ||||||
|
|
||||||
| This mode is recommended for production workloads. Benchmarks show it provides the best per-task throughput across both small and large embedding models by reducing GPU idle time during tokenization. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||||||
|
|
||||||
| ```python | ||||||
| # Pretokenize mode (recommended) | ||||||
| VLLMEmbeddingModelStage( | ||||||
| model_identifier="google/gemma-embedding-model", | ||||||
| pretokenize=True, # Tokenize on CPU, embed on GPU | ||||||
| ) | ||||||
|
|
||||||
| # Direct text mode (simpler, slightly less throughput) | ||||||
| VLLMEmbeddingModelStage( | ||||||
| model_identifier="google/gemma-embedding-model", | ||||||
| pretokenize=False, # vLLM handles tokenization internally | ||||||
| ) | ||||||
| ``` | ||||||
|
|
||||||
| ## Resources | ||||||
|
|
||||||
| The `VLLMEmbeddingModelStage` requests 1 CPU and 1 GPU per worker by default. For multi-GPU models, configure `tensor_parallel_size` in `vllm_init_kwargs`. | ||||||
|
|
||||||
| ## When to Use vLLM vs. Sentence Transformers | ||||||
|
|
||||||
| | Scenario | Recommendation | | ||||||
| | --- | --- | | ||||||
| | Large embedding models (>500M params) | vLLM — better GPU utilization and memory management | | ||||||
| | Small embedding models (<100M params) | Sentence Transformers — lower overhead, faster startup | | ||||||
| | High-throughput production pipelines | vLLM with `pretokenize=True` — best amortized throughput | | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| | Quick prototyping | `EmbeddingCreatorStage` — simpler setup, no extra dependency | | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.