Skip to content

docs: update semdedup docs for vLLM default backend (PR #1606) - #1659

Merged
lbliii merged 4 commits into
NVIDIA-NeMo:26.04-stagingfrom
lbliii:lbliii/pr1606-doc-plan
Apr 2, 2026
Merged

docs: update semdedup docs for vLLM default backend (PR #1606)#1659
lbliii merged 4 commits into
NVIDIA-NeMo:26.04-stagingfrom
lbliii:lbliii/pr1606-doc-plan

Conversation

@lbliii

@lbliii lbliii commented Mar 24, 2026

Copy link
Copy Markdown
Contributor

Description

Updates v26.04 fern documentation to reflect the vLLM default embedding backend change from #1606. Replaces all EmbeddingCreatorStage references with VLLMEmbeddingModelStage, updates the default model from sentence-transformers/all-MiniLM-L6-v2 to google/embeddinggemma-300m, removes stale parameters (embedding_max_seq_length, embedding_padding_side, embedding_pooling, embedding_model_inference_batch_size), and adds new vLLM-specific parameters (embedding_pretokenize, embedding_vllm_init_kwargs, model_cache_dir). Replaces the 26.02 release notes with a 26.04 skeleton including the vLLM default entry and breaking changes.

Usage

from nemo_curator.stages.text.deduplication.semantic import TextSemanticDeduplicationWorkflow

# Default: uses vLLM with google/embeddinggemma-300m
workflow = TextSemanticDeduplicationWorkflow(
    input_path="input_data/",
    output_path="./results",
    cache_path="./sem_cache",
    n_clusters=100,
    eps=0.07,
    perform_removal=True
)
result = workflow.run()

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

Combine vLLM default backend release notes and breaking changes from
this PR with Cosmos-Xenna, Workflow Results API, bug fixes, and other
breaking changes from upstream. Use upstream's more detailed
result.metadata comments in semdedup code examples.

Signed-off-by: Logan Lane <llane@nvidia.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
@lbliii
lbliii requested a review from a team as a code owner April 2, 2026 17:49
@lbliii
lbliii requested review from oyilmaz-nvidia and removed request for a team April 2, 2026 17:49
Comment on lines +117 to +120
<<<<<<< HEAD
=======
# result.metadata contains: total_time, num_duplicates, kmeans_time, pairwise_time
>>>>>>> upstream/26.04-staging

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0 Unresolved git merge conflict markers

Lines 117–120 contain raw <<<<<<< HEAD, =======, and >>>>>>> upstream/26.04-staging markers. These were not resolved before committing and will cause the Fern documentation build to fail or render the conflict markers verbatim to users.

The upstream side of the conflict contains a useful comment that should be retained:

Suggested change
<<<<<<< HEAD
=======
# result.metadata contains: total_time, num_duplicates, kmeans_time, pairwise_time
>>>>>>> upstream/26.04-staging
# result.metadata contains: total_time, num_duplicates, kmeans_time, pairwise_time

lbliii and others added 2 commits April 2, 2026 13:54
Resolve leftover conflict markers that were missed in the previous
merge resolution commit.

Signed-off-by: Logan Lane <llane@nvidia.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Include both the vLLM default backend entry from this PR and the
multi-user metrics isolation entry that landed via PR NVIDIA-NeMo#1656.

Signed-off-by: Logan Lane <llane@nvidia.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants