fix(llm): return chat prompt logprobs

[yankay]

Overview

Return requested prompt logprobs at the root of non-streaming OpenAI chat completion responses.

Summary

Dynamo computed prompt_logprobs but only exposed them through nvext. Preserve them through unary aggregation and return them at the response root while keeping existing nvext behavior.

Details

• Keep the internal transport field out of streaming responses.
• Preserve explicit prompt_logprobs: 0.
• Share request-sized PromptLogprobs payloads with Arc across parser template and fan-out paths to avoid deep clones.

Where should the reviewer start?

lib/llm/src/protocols/openai/chat_completions/delta.rs

Validation

• Targeted prompt-logprobs tests passed, including prompt_logprobs: 0.
• Request-trace conversion test passed.
• cargo check -p dynamo-llm --lib --no-default-features
• cargo clippy -p dynamo-llm --tests --no-default-features -- -D warnings
• cargo fmt --all -- --check
• git diff --check
• Real Dynamo E2E with vLLM 0.27.1 and Qwen/Qwen3-0.6B, using a
  non-streaming request with max_tokens: 1, temperature: 0, and
  prompt_logprobs: 2:
  • Before (fc956aad0b): HTTP 200, but response-root prompt_logprobs
    was absent. It was only available through
    nvext.extra_fields: ["prompt_logprobs"].
  • After (43adb22444): response-root prompt_logprobs contained 13
    entries, matching usage.prompt_tokens; the first entry was null,
    and no nvext request extension was needed.

Patched response excerpt (first two prompt-logprob entries shown):

{
  "model": "Qwen/Qwen3-0.6B",
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 1,
    "total_tokens": 14
  },
  "prompt_logprobs": [
    null,
    {
      "14582": {
        "logprob": -5.97626,
        "rank": 1
      },
      "38297": {
        "logprob": -6.47626,
        "rank": 2
      }
    }
  ]
}

• Native vLLM returned prompt_logprobs at the response root for the same
  request.

Related Issues

• Closes [BUG]: Chat prompt_logprobs only returned on nvext, not on the OpenAI choice #13941

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi yankay! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e67766af-a4ee-4cf9-ab61-12f875c71a47

📥 Commits

Reviewing files that changed from the base of the PR and between 481c3d2 and 4c9b5de.

📒 Files selected for processing (25)

• lib/llm/src/backend.rs
• lib/llm/src/http/service/metrics.rs
• lib/llm/src/http/service/openai.rs
• lib/llm/src/perf/logprobs.rs
• lib/llm/src/preprocessor.rs
• lib/llm/src/protocols/anthropic/stream_converter.rs
• lib/llm/src/protocols/anthropic/types.rs
• lib/llm/src/protocols/openai/chat_completions.rs
• lib/llm/src/protocols/openai/chat_completions/aggregator.rs
• lib/llm/src/protocols/openai/chat_completions/delta.rs
• lib/llm/src/protocols/openai/chat_completions/tool_parser_v2.rs
• lib/llm/src/protocols/openai/chat_completions/unified_parser.rs
• lib/llm/src/protocols/openai/completions/delta.rs
• lib/llm/src/protocols/openai/delta_common.rs
• lib/llm/src/protocols/openai/responses/mod.rs
• lib/llm/src/protocols/openai/responses/stream_converter.rs
• lib/llm/src/request_trace/agent_context.rs
• lib/llm/src/request_trace/payload_stream.rs
• lib/llm/tests/aggregators.rs
• lib/llm/tests/logprob_analysis_integration.rs
• lib/llm/tests/postprocessor_parsing_stream.rs
• lib/llm/tests/test_reasoning_parser.rs
• lib/llm/tests/test_streaming_tool_parsers.rs
• lib/llm/tests/tool_choice.rs
• lib/llm/tests/tool_choice_finish_reasons.rs

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

Changes

The change adds optional prompt logprobs to OpenAI chat response types. It routes values to the response root and optional nvext field, preserves them during aggregation and stream conversion, and carries them through the tool-calling jail. Tests update fixtures and verify propagation.

Prompt logprobs response contracts
lib/llm/src/protocols/openai/chat_completions.rs, lib/llm/src/protocols/openai/delta_common.rs, lib/llm/src/backend.rs|Chat response types and DeltaGeneratorOptions now support prompt logprobs.|
|Chat generation and aggregation
lib/llm/src/protocols/openai/chat_completions/delta.rs, lib/llm/src/protocols/openai/chat_completions/aggregator.rs|Generation honors request configuration, final chunks route prompt logprobs, and aggregation includes them in unary responses.|
|Stream conversion and tool-jail metadata
lib/llm/src/request_trace/payload_stream.rs, lib/llm/src/preprocessor.rs|Stream conversion and tool-jail processing preserve prompt logprobs across chunks, EOF metadata, and terminal errors.|
|Fixture and regression coverage
lib/llm/src/http/service/*, lib/llm/src/protocols/anthropic/*, lib/llm/src/protocols/openai/responses/*, lib/llm/tests/*|Existing fixtures initialize the new field, while tests verify scrubbing, routing, and preservation.|

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to 4c9b5

This change exposes requested prompt logprobs in non-streaming chat responses while preserving existing behavior; no actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 67.42% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 89 functions across 25 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address issue #13941 by exposing requested prompt logprobs at the non-streaming response root, preserving unary aggregation, retaining optional nvext output, and keeping streaming transpor…
Out of Scope Changes check	✅ Passed	The changes are related to prompt logprob propagation, response conversion, metadata preservation, and supporting test fixtures. No unrelated code changes are evident.
Title check	✅ Passed	The title clearly and concisely identifies the main change: returning chat prompt logprobs.
Description check	✅ Passed	The description includes the required overview, details, reviewer starting point, and related issue section. It also documents validation and behavioral requirements.

• Fix all pre-merge checks with AI

---

_{Comment @coderabbitai help to get the list of available commands.}

[dynamo-ops]

/ok to test be62e16

[rmccorm4]

/ok to test 0ebdf81

[dynamo-ops]

/ok to test 0ebdf81

[Copilot]

Copilot review overview

🟡 Changes recommended

Multi-choice responses repeatedly deserialize the request-sized prompt-logprobs payload after each terminal choice.

Review effort: Balanced
Findings: 1

Open (1)

• Avoid reserializing prompt logprobs for every output choice · New

What changed in this PR

Returns requested prompt log probabilities at the root of non-streaming OpenAI chat responses while preserving existing nvext behavior.

Changes:

• Propagates prompt logprobs through delta generation, parsing, aggregation, and request tracing.
• Uses Arc for large internal payloads and excludes them from SSE serialization.
• Adds regression coverage and updates response fixtures.

File	Description
lib/​llm/​tests/​tool_choice.rs	Updates stream-response fixtures.
lib/​llm/​tests/​tool_choice_finish_reasons.rs	Updates finish-reason fixture.
lib/​llm/​tests/​test_streaming_tool_parsers.rs	Updates parser fixtures.
lib/​llm/​tests/​test_reasoning_parser.rs	Updates reasoning fixture.
lib/​llm/​tests/​postprocessor_parsing_stream.rs	Updates postprocessor fixtures.
lib/​llm/​tests/​logprob_analysis_integration.rs	Updates logprob fixtures.
lib/​llm/​tests/​aggregators.rs	Updates aggregation fixtures.
lib/​llm/​src/​request_trace/​payload_stream.rs	Preserves prompt logprobs through trace conversion.
lib/​llm/​src/​request_trace/​agent_context.rs	Updates agent-context fixtures.
lib/​llm/​src/​protocols/​openai/​responses/​stream_converter.rs	Updates response-converter fixtures.
lib/​llm/​src/​protocols/​openai/​responses/​mod.rs	Updates unary-response fixtures.
lib/​llm/​src/​protocols/​openai/​completions/​delta.rs	Validates requested backend prompt-logprob payloads.
lib/​llm/​src/​protocols/​openai/​chat_completions/​unified_parser.rs	Updates unified-parser fixtures.
lib/​llm/​src/​protocols/​openai/​chat_completions/​tool_parser_v2.rs	Updates tool-parser fixture.
lib/​llm/​src/​protocols/​openai/​chat_completions/​delta.rs	Captures and emits requested prompt logprobs.
lib/​llm/​src/​protocols/​openai/​chat_completions/​aggregator.rs	Aggregates prompt logprobs into unary responses.
lib/​llm/​src/​protocols/​openai/​chat_completions.rs	Adds root and internal transport fields.
lib/​llm/​src/​protocols/​common/​llm_backend.rs	Makes payload decoding errors explicit.
lib/​llm/​src/​protocols/​anthropic/​types.rs	Updates Anthropic conversion fixtures.
lib/​llm/​src/​protocols/​anthropic/​stream_converter.rs	Updates Anthropic stream fixtures.
lib/​llm/​src/​preprocessor/​speculative_prefill.rs	Updates speculative-prefill fixture.
lib/​llm/​src/​preprocessor.rs	Preserves metadata through legacy parser processing.
lib/​llm/​src/​perf/​logprobs.rs	Updates performance-analysis fixtures.
lib/​llm/​src/​http/​service/​openai.rs	Updates OpenAI service fixtures.
lib/​llm/​src/​http/​service/​metrics.rs	Updates metrics fixtures.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dynamo-ops]

/ok to test a18df8c

[dynamo-ops]

/ok to test 924e495

[GuanLuo]

/ok to test 924e495

[devin-ai-integration]

⏹️ Dynamo PR CI cancelled — run 37412746204 (attempt 1) on fa1f35e9c1

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Other	Jobs
dynamo-runtime	⏹️ 1
frontend	⏹️ 1

Failure details

No jobs failed. The 2 cancelled jobs are not fail-fast or manual cancels: both image builds hit their job timeout while the Docker build was still running. The gate checks fail only because they mirror these. The logs show no error before the timeout, so rerunning the failed jobs is likely enough.

• ⏹️ dynamo-runtime / image / Build multi-arch cuda13.0 · Step: Build and Push Test Image. Hit the 1h0m0s job timeout (The job has exceeded the maximum execution time of 1h0m0s).
• ⏹️ frontend / Build multi-arch cpu · Step: Build and Push Image. Hit the 1h30m0s job timeout (The job has exceeded the maximum execution time of 1h30m0s).

For agents

{"pr": 13957, "run_id": 37412746204, "run_attempt": 1, "head_sha": "fa1f35e9c15d7728c5219869e97d2af9aea1debd",
 "failures": [{"job": "dynamo-runtime / image / Build multi-arch cuda13.0", "job_id": 112104717833, "failed_step": "Build and Push Test Image", "signature": "job timeout (1h0m0s) during image build", "tests": [], "log_cmd": "gh api repos/ai-dynamo/dynamo/actions/jobs/112104717833/logs"},
              {"job": "frontend / Build multi-arch cpu", "job_id": 112104715473, "failed_step": "Build and Push Image", "signature": "job timeout (1h30m0s) during image build", "tests": [], "log_cmd": "gh api repos/ai-dynamo/dynamo/actions/jobs/112104715473/logs"}]}

_{Posted automatically by Devin for run 37412746204. Updated on every full-CI run of this PR.}

[dynamo-ops]

/ok to test feac185

[GuanLuo]

@yankay can you take a pass to the opening conversation and address / mark them resolved?

[houshengbo]

@yankay This PR worked fine with default --dyn-chat-processor.

We tested it with --dyn-chat-processor set to vllm, it did not work well: non-streaming /v1/chat/completions with prompt_logprobs: 3 on a vLLM-processor deployment returns 200 and no prompt_logprobs.

--dyn-chat-processor vllm uses vllm_processor.py, which never copies engine_data.prompt_logprobs onto the chat chunk, so unary aggregation still omits the field.

--dyn-chat-processor sglang PROBABLY misses prompt_logprobs too, need further confirmation.

This needs follow-up changes.

[rmccorm4]

/ok to test 80cdb61

[dynamo-ops]

/ok to test 80cdb61

[rmccorm4]

/ok to test fa1f35e

[dynamo-ops]

/ok to test fa1f35e