Skip to content

[Rust Frontend][gRPC] Preserve skip_special_tokens decoding option - #52384

Merged
njhill merged 2 commits into
vllm-project:mainfrom
biswapanda:feat/grpc-skip-special-tokens
Aug 15, 2026
Merged

njhill merged 2 commits into
vllm-project:mainfrom
biswapanda:feat/grpc-skip-special-tokens

Conversation

@biswapanda

@biswapanda biswapanda commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

Align the native Generate gRPC API with vLLM's existing Python serving APIs by carrying the per-request skip_special_tokens decoding option through ResponseOptions.

The Python chat completions, completions, responses, and token-native APIs default this option to true, but the gRPC request did not expose it. Consequently, a gRPC caller could not explicitly preserve tokenizer-defined special markers required by some reasoning and tool parsers.

The protobuf field is optional so the server can distinguish omission from an explicit false. Omitted values retain the established Python default of true; explicit false values are preserved. This affects decoded text only and does not change sampling, generated token IDs, or EOS behavior.

  • Python API behavior
Endpoint Field location Default
/v1/chat/completions Top-level request field true
/v1/completions Top-level request field true
/v1/responses Top-level request field true
/inference/v1/generate sampling_params.skip_special_tokens true

Test Plan

  • Verify Rust formatting.
  • Run the existing gRPC conversion test module after mapping the new response option into TextDecodeOptions.

Test Result

$ cargo fmt --all -- --check

$ cargo test -p vllm-server grpc::convert::tests
running 13 tests
test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 328 filtered out

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. No documentation change is required for this parity fix.

Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@mergify mergify Bot added the rust label Aug 14, 2026
@biswapanda
biswapanda force-pushed the feat/grpc-skip-special-tokens branch from 78b6aba to cf16afb Compare August 14, 2026 19:22
@biswapanda
biswapanda marked this pull request as ready for review August 14, 2026 19:26

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Comment thread rust/proto/inference.proto Outdated
Comment thread rust/src/server/src/grpc/convert.rs Outdated
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
biswapanda force-pushed the feat/grpc-skip-special-tokens branch from bb8d02b to c98c11b Compare August 14, 2026 20:10

@njhill njhill left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @biswapanda

@njhill njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 14, 2026
@njhill

njhill commented Aug 14, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

@biswapanda, CI is now available for this PR.

  • /ci run starts a CI build.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /ci cancel cancels scheduled or running CI builds for this PR branch.

@njhill
njhill enabled auto-merge (squash) August 14, 2026 23:59
@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #83993 for commit c98c11b154af.

@njhill
njhill merged commit 4215646 into vllm-project:main Aug 15, 2026
32 of 33 checks passed
zufangzhu pushed a commit to zufangzhu/vllm that referenced this pull request Aug 24, 2026
…llm-project#52384)

Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed rust

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants