fix(frontend): return SGLang chat logprobs

[jain-ria]

Overview:

Fix SGLang chat-completion responses returning logprobs.content: null when logprobs were requested, including requests with logprobs=true and top_logprobs=0.

The SGLang backend calculated and returned token logprobs, but the frontend discarded them before postprocessing. The postprocessor also hardcoded the response’s logprobs field to None.

Details:

• Preserve log_probs and top_logprobs while batching generated tokens in the SGLang frontend.
• Forward those values to the SGLang postprocessor.
• Convert backend logprobs into the OpenAI-compatible per-token format:
  • token
  • logprob
  • bytes
  • top_logprobs
• Return populated logprobs instead of hardcoded null.
• Preserve the existing behavior when logprobs are not requested.

Where should the reviewer start?

• components/src/dynamo/frontend/sglang_processor.py
  • Preserves and forwards backend logprob data while batching tokens.
• components/src/dynamo/frontend/sglang_prepost.py
  • Converts backend logprob data into the OpenAI response contract.

Related Issues

• Closes DIS-2624

---

Summary by CodeRabbit

• New Features

  • Added OpenAI-compatible log probability details to streamed responses.
  • Streaming output can include per-token scores, top candidate tokens, and UTF-8 byte values.
  • Log probability data is now preserved across streamed chunks and attached to text, completion, and parsed-output responses.

• Bug Fixes

  • Prevented incomplete or mismatched token probability data from being emitted.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[devin-ai-integration]

Devin Review found 4 potential issues.

[coderabbitai]

Walkthrough

Streaming SGLang responses now carry per-token log probabilities and optional top candidates. The frontend converts these values into OpenAI-compatible content entries, forwards them during chunk processing, and resets the associated buffers after each emitted chunk.

Changes

Streaming log probability support

Layer / File(s)	Summary
OpenAI logprob conversion components/src/dynamo/frontend/sglang_prepost.py	process_output converts token scores and candidate alternatives into OpenAI-compatible logprob entries. The payload is included in plain-text, finish, and parsed responses.
Stream probability accumulation components/src/dynamo/frontend/sglang_processor.py	Streaming processing accumulates log_probs and top_logprobs, forwards them during chunk flushes, and clears both buffers after emission.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely identifies the fix for missing SGLang chat logprobs.
Description check	✅ Passed	The description includes the required overview, details, reviewer guidance, and linked issue information.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

components/src/dynamo/frontend/sglang_prepost.py (1)

1146-1160: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Keep probability arrays aligned after EOS removal.

Line 1149 removes trailing EOS token IDs. It does not remove the matching log_probs and top_logprobs entries. _build_openai_logprobs then detects a length mismatch and returns None.

Trim both probability arrays to the retained token count after validating their original alignment. Add a terminal-chunk test with generated tokens followed by EOS.

Proposed fix

         if finish_reason is not None:
+            raw_token_count = len(token_ids)
             token_ids = self._strip_trailing_eos_token_ids(list(token_ids))
+            if log_probs is not None and len(log_probs) == raw_token_count:
+                log_probs = log_probs[: len(token_ids)]
+            if top_logprobs and len(top_logprobs) == raw_token_count:
+                top_logprobs = top_logprobs[: len(token_ids)]

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/sglang_prepost.py` around lines 1146 - 1160,
Update the EOS handling in the response-processing flow around
_strip_trailing_eos_token_ids to validate that log_probs and top_logprobs are
aligned with the original token_ids, then trim both arrays to the retained token
count after EOS removal. Keep _build_openai_logprobs receiving matching token
and probability lengths, and add a terminal-chunk test covering generated tokens
followed by EOS.

🧹 Nitpick comments (1)

components/src/dynamo/frontend/sglang_prepost.py (1)

1057-1057: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Set an explicit zip() strictness value.

The project requires Python 3.10 or later, and Ruff reports B905 here. Use strict=True.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/sglang_prepost.py` at line 1057, Update the
zip call in the token iteration loop to pass the explicit strict=True argument,
resolving Ruff B905 while preserving the existing enumerate behavior.

Sources: Coding guidelines, Path instructions, Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/frontend/sglang_prepost.py`:
- Around line 1156-1177: Update the incremental decoding flow around
`_incremental_decode` and `_pending_decode_ids` to retain logprob entries when
decoding returns an empty string for a deferred byte-fallback sequence. Queue
the probability data with the pending token IDs, then emit it only when those
matching IDs produce decoded text, rather than allowing `sglang_processor.py` to
clear it prematurely. Add a test covering one UTF-8 byte-fallback sequence split
across stream chunks and verify its text and logprobs are preserved.

---

Outside diff comments:
In `@components/src/dynamo/frontend/sglang_prepost.py`:
- Around line 1146-1160: Update the EOS handling in the response-processing flow
around _strip_trailing_eos_token_ids to validate that log_probs and top_logprobs
are aligned with the original token_ids, then trim both arrays to the retained
token count after EOS removal. Keep _build_openai_logprobs receiving matching
token and probability lengths, and add a terminal-chunk test covering generated
tokens followed by EOS.

---

Nitpick comments:
In `@components/src/dynamo/frontend/sglang_prepost.py`:
- Line 1057: Update the zip call in the token iteration loop to pass the
explicit strict=True argument, resolving Ruff B905 while preserving the existing
enumerate behavior.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8458afde-f517-4bba-8d85-c456e71b19b2

📥 Commits

Reviewing files that changed from the base of the PR and between ee62866 and 5723620.

📒 Files selected for processing (2)

• components/src/dynamo/frontend/sglang_prepost.py
• components/src/dynamo/frontend/sglang_processor.py

[rmccorm4]

Thanks @jain-ria ! Please address the devin comments 🙏

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 14.71%
• Overall Coverage: 45.78% (-13.10%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ec3b64b | Docs | Datadog PR Page | Give us feedback!}

[rmccorm4]

/ok to test 31f8c33

[jain-ria]

/ok to test ec3b64b

[rmccorm4]

Some of the more problematic looking findings from local review:

[P1] return_tokens_as_token_ids=true remains ignored for selected tokens at sglang_prepost.py (line 1061). Exact-head repro still returns selected "A" beside candidate "token_id:65".

[P1] Contextual reconstruction corrupts per-token Unicode attribution at sglang_prepost.py (line 1106). Qwen3-0.6B encodes 𐍈 as IDs [126190, 230], representing bytes [240,144,141] and [136]. The PR reports:
token 126190: token="", bytes=null
token 230: token="𐍈", bytes=[240,144,141,136]
This moves the first token’s bytes onto the second token. A valid U+FFFD token is also reported as token="", bytes=null, despite emitted content being �.

[P1] Non-finite values remain unsanitized at sglang_prepost.py (line 1080). The repro emits bare -Infinity; the Python/Rust typed transport rewrites this incompatibly and can discard the requested logprobs payload.

However, I don't think any of these issues are blocking to the release, so these can be followed up in separate PRs to main as you see fit.

[rmccorm4]

Some comments for future follow-up, not blocking to this PR: #12820 (comment)

LGTM to unblock the release fix.