feat(frontend): forward vllm_xargs to vLLM SamplingParams.extra_args

[agupta01]

Overview:

vLLM's OpenAI server accepts vllm_xargs and passes it to SamplingParams.extra_args. Custom extensions and model-specific request options read that field. One example is DiffusionGemma's structured "read" requests, added in vllm-project/vllm#57250. They send diffusion_seed_canvas, diffusion_max_steps, diffusion_read_only and similar fields. Dynamo's OpenAI frontend rejects vllm_xargs with Unsupported parameter(s). With DYN_IGNORE_OPENAI_FE_UNSUPPORTED_FIELDS set, it silently drops the field instead. Either way, these workloads can't run behind the Dynamo frontend and KV router.

This PR forwards vllm_xargs the same way logprob_token_ids and the other extra-body sampling fields are already forwarded. It also raises the /v1/completions logprobs cap from 5 to 20, the chat top_logprobs limit. Those structured reads also call /v1/completions with token-id prompts and ask for the top 20 candidates.

Details:

Commit 1: feat(frontend): forward vllm_xargs to vLLM SamplingParams.extra_args

• lib/llm/src/protocols/openai/validate.rs: vllm_xargs is added to PASSTHROUGH_EXTRA_FIELDS. It is validated against vLLM's schema: an object whose values are strings, numbers or booleans, or arrays of those. Booleans are accepted because vLLM's pydantic model coerces them.
  • kv_transfer_params and ec_transfer_params are rejected. vLLM fills those keys from dedicated request fields, and Dynamo only forwards router-generated connector inputs.
• lib/llm/src/preprocessor.rs: vllm_xargs is forwarded in extra_args.sampling_options, next to logprob_token_ids.
• components/src/dynamo/vllm/handlers.py: build_sampling_params pops vllm_xargs before the generic setattr loop. It then merges the field over any default extra_args. The worker drops the two connector keys as well, as a second line of defense, so the router's kv_hint handling is unchanged.
• Backend gating: token-input vLLM workers advertise a vllm_xargs runtime capability (VLLM_XARGS_CAPABILITY in runtime_config.rs; published by dynamo.vllm for base and LoRA model cards). Requests carrying vllm_xargs reach only workers that advertise it (see the per-worker enforcement commit below). When no worker serving the model advertises it (SGLang, TensorRT-LLM, text-input or older vLLM workers), the new ModelManagerError::CapabilityUnsupported becomes a 400 instead of the field being dropped silently. The text-input path is covered too, since it forwards the raw request and drops passthrough fields.

Commit 2: feat(frontend): allow completions logprobs up to 20

• MAX_LOGPROBS changes from 5 to 20, matching MAX_TOP_LOGPROBS and vLLM's default --max-logprobs.
• Backends already receive up to 20 through chat top_logprobs, which maps to the same output_options.logprobs. So this doesn't add any new backend-facing range.

Commit 3: fix(frontend): require vllm_xargs support on every destination worker

The capability is not part of the MDC checksum, so upgraded and older workers share a WorkerSet during a rolling upgrade. Checking the WorkerSet's representative card is therefore not enough, so support is enforced per worker:

• MemberCapabilities (lib/llm/src/discovery/worker_set.rs) records which committed members of a WorkerSet advertise each request capability. ModelManager::{commit,replace}_discovery_group update it. The controller calls these before it admits a newcomer and after it withdraws a departed worker, so every admitted worker is described.
• Engine selection (Model::get_{chat,completions}_engine_with_parsing) admits a vllm_xargs request only when every member advertises it. For a decode WorkerSet, every prefill WorkerSet in its namespace must as well. Partial support, as during a rolling upgrade, is a 503; no support anywhere is a 400. Ordinary requests keep using mixed-version WorkerSets.
• Dispatch: decode and prefill RoutingHosts check the final destination worker on every path (KV, builtin modes, Direct, LoRA, transport fallback). A worker that is not described as capable is refused with Unavailable, which covers a worker admitted between engine selection and dispatch.

Requests without vllm_xargs are unaffected on every backend.

Where should the reviewer start?

• validate_vllm_xargs in lib/llm/src/protocols/openai/validate.rs: the accepted value shapes and the reserved keys.
• MemberCapabilities in lib/llm/src/discovery/worker_set.rs and describe_members in model_manager.rs: per-member capability tracking and when it is updated relative to admission.
• CapabilityFilter / capability_engine_error in lib/llm/src/discovery/model.rs: WorkerSet selection, including prefill peers, and the 400 vs 503 decision.
• RoutingHost::ensure_worker_capability and its call sites in routing_host/builtin.rs and routing_host/kv.rs: the per-destination check at dispatch.
• build_sampling_params in components/src/dynamo/vllm/handlers.py: how the merge interacts with default_sampling_params["extra_args"] and with the later kv_hint / kv_transfer_params handling.
• Commit 2 is a separate policy change: it moves away from OpenAI's legacy completions cap. I can split it into its own PR if you'd prefer.

Validation

Unit and lint checks ran on this branch, on top of upstream main, inside vllm/vllm-openai:nightly-e9757321527ca1ecd514c07c1418dd2c53da3d19 (vLLM 0.29.1rc1.dev573, which includes vllm-project/vllm#57250):

cargo fmt --all -- --check                                   # clean
cargo clippy --locked -p dynamo-llm --all-targets -- -D warnings   # clean
cargo test --locked -p dynamo-llm --lib -- validate:: \
  preprocessor::tests::test_backend_extra_args \
  discovery::model::tests::capability_requirement openai::completions \
  http::service::openai::tests                               # 227 passed, 0 failed
pytest components/src/dynamo/vllm/tests/test_vllm_tito_parity.py \
  components/src/dynamo/vllm/tests/test_vllm_unit.py \
  -k "Xargs or LogprobTokenIds or kv_hint or build_sampling"   # 28 passed
pre-commit run --files <changed files>                       # passed

Commit 3 was validated on macOS (--no-default-features, per the repo guidance for dynamo-llm):

cargo fmt --all                                               # clean
cargo clippy -p dynamo-llm --no-default-features --all-targets  # no dynamo-llm warnings
cargo test -p dynamo-llm --no-default-features --lib -- vllm_xargs \
  dispatch_requires_worker_capability capability_requirement adapter_view \
  validate:: preprocessor::tests::test_backend_extra_args     # 33 passed
cargo test -p dynamo-llm --no-default-features --lib -- discovery:: kv_router:: \
  entrypoint:: http::service::openai::tests openai::completions
  # 842 passed; 5 environment-sensitive tests pass when run serially, and
  # grpc::tests::metrics_bind_failure_shuts_down_* fails identically on the parent commit
pre-commit run --files <changed files>                        # passed

New regression tests: vllm_xargs_requires_every_worker_set_member (mixed members in both registration orders, worker removal and rejoin, 503 vs 400), vllm_xargs_requires_capable_prefill_peers (upgraded decode with older prefill), and {kv,builtin}_dispatch_requires_worker_capability (dispatch to an admitted worker without the capability is refused while ordinary requests still reach it). All four fail with the enforcement disabled.

End-to-end, the same change backported to v1.5.0 has been serving DiffusionGemma NVFP4 behind dynamo.frontend --router-mode kv --router-kv-events. The setup is 8 dynamo.vllm workers, each on its own RTX PRO 6000 Blackwell GPU, on Kubernetes through the Dynamo operator:

• Chat and completions reads with vllm_xargs, logprob_token_ids and return_tokens_as_token_ids returned correct per-slot label distributions. The prompt Dynamo rendered from the chat template had the same token count as Hugging Face's rendering (206 tokens).
• A public structured-decision benchmark of 231 tasks completed with 0 failures. A 60 s load run with 32 clients returned 5,163 of 5,163 requests successfully.

Two other things came up during that integration. They are not changed in this PR, and I can open issues for them if useful:

1. vLLM rejects the default vLLM health-check canary (temperature: 0.0) for diffusion models, whose sampler runs a fixed schedule. Workers never become healthy unless DYN_HEALTH_CHECK_PAYLOAD is overridden.
2. vLLM's read-only diffusion path sets ignore_eos internally, but Dynamo's own EOS stop handling still truncates the emitted canvas. Clients have to send ignore_eos: true explicitly.

Related Issues

• Relates to [Roadmap]: Track Dynamo 2.0 diffusion work #13935 (text diffusion workflows)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi agupta01! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1f90569e-4423-4b5c-8c60-ac46ad4b52c9

📥 Commits

Reviewing files that changed from the base of the PR and between 82d10b6 and 52e41e7.

📒 Files selected for processing (5)

• components/src/dynamo/vllm/handlers.py
• components/src/dynamo/vllm/tests/test_vllm_tito_parity.py
• lib/llm/src/http/service/openai.rs
• lib/llm/src/preprocessor.rs
• lib/llm/src/protocols/openai/validate.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The request path now validates and forwards vllm_xargs as vLLM sampling extra arguments. The maximum completion-request logprobs value increases from 5 to 20.

Changes

vllm_xargs forwarding

Layer / File(s)	Summary
Validate vllm_xargs lib/llm/src/protocols/openai/validate.rs	The validator accepts scalar values and arrays of scalars in vllm_xargs. It rejects non-object values, nested values, and the reserved kv_transfer_params and ec_transfer_params keys.
Forward vllm_xargs to sampling parameters lib/llm/src/preprocessor.rs, components/src/dynamo/vllm/handlers.py, components/src/dynamo/vllm/tests/test_vllm_tito_parity.py, lib/llm/src/protocols/openai/validate.rs	The preprocessor copies vllm_xargs into sampling_options. The handler merges non-reserved entries into SamplingParams.extra_args, preserving existing unmatched entries. Tests cover forwarding, overrides, and filtering reserved keys.

Completion logprobs limit

Layer / File(s)	Summary
Set and test the completion logprobs limit lib/llm/src/protocols/openai/validate.rs, lib/llm/src/http/service/openai.rs	The completion MAX_LOGPROBS limit changes from 5 to 20. Documentation and HTTP validation expectations use the 0–20 range and reject 21.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 52e41

No actionable issue is established for the changed request options or logprobs limit; the PR is mergeable after normal checks.

🚥 Pre-merge checks | ✅ 4 | ❓ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	❓ Inconclusive	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 4 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely identifies the main change: forwarding vllm_xargs to vLLM SamplingParams.extra_args.
Description check	✅ Passed	The description follows the required template, explains the implementation and validation, identifies review starting points, and includes the required related issue reference.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 4 files. (1 skipped: 1 too large.)

---

_{Comment @coderabbitai help to get the list of available commands.}

[biswapanda]

Enforce capability support on every destination worker

Summary: This checks the WorkerSet representative card, not every routable instance. vllm_xargs is excluded from the MDC checksum/cohort fingerprint, so upgraded and older replicas can share a WorkerSet during a rolling upgrade. A capable representative can therefore admit a request that reaches an older worker and silently loses its options; an older representative can reject requests despite available capable replicas. The selected decode set's prefill destinations are not capability-filtered either.

Example: Register an upgraded replica advertising vllm_xargs and an older replica at the same endpoint/model/role. A completion with vllm_xargs: {"diffusion_read_only": true} passes this check when the upgraded replica is representative, but can route to the older replica, whose sampling builder ignores the field.

Actionable fix: Carry the requirement through per-instance routing/admission, including prefill, or conservatively reject these requests until all eligible destinations support it. Preserve ordinary requests during mixed-version operation rather than treating the capability difference as an incompatible model revision. Add discovery/routing coverage for mixed capabilities inside one WorkerSet, both registration orders, worker removal, and upgraded decode with older prefill.

[biswapanda]

/ok to test de05f4c

[devin-ai-integration]

⏹️ Dynamo PR CI cancelled — run 37683231183 (attempt 1) on 7c6ad15dc9

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Other	Jobs
planner	⏹️ 2 ✅ 3

Failure details

No jobs failed. 2 jobs were cancelled: planner / Compliance cpu, arm64 and planner / Compliance cpu, amd64 both hit the 15m job timeout in step Compliance scan (annotation: "The job has exceeded the maximum execution time of 15m0s"), so no compliance CSV was uploaded. These were not fail-fast or manual/concurrency cancels. The planner build and CPU tests passed and all gate checks are green. Rerunning the failed jobs is likely enough.

For agents

{"pr": 15281, "run_id": 37683231183, "run_attempt": 1, "head_sha": "7c6ad15dc9a838d7d8489c214e125a46b9f39c09",
 "failures": [],
 "cancelled": [{"job": "planner / Compliance cpu, arm64", "job_id": 113010744938, "cancelled_step": "Compliance scan", "signature": "job exceeded maximum execution time of 15m0s", "log_cmd": "gh run view --job 113010744938 -R ai-dynamo/dynamo --log"},
               {"job": "planner / Compliance cpu, amd64", "job_id": 113010745080, "cancelled_step": "Compliance scan", "signature": "job exceeded maximum execution time of 15m0s", "log_cmd": "gh run view --job 113010745080 -R ai-dynamo/dynamo --log"}]}

_{Posted automatically by Devin for run 37683231183. Updated on every full-CI run of this PR.}

[rmccorm4]

/ok to test 7c6ad15