[rmccorm4]

Summary

Preserve Anthropic tool.strict through request deserialization and conversion to FunctionObject.strict, including explicit false and omission. Keep the tool input schema intact.

Use the published frontend crates released after ai-dynamo/frontend-crates#322 and ai-dynamo/frontend-crates#375

Adapt the Responses API types, enum variants, and usage initializers to the new protocol API. Preserve existing error codes and the legacy prompt-cache retention wire field; reject the new ultrafast tier because Chat Completions cannot represent it.

Finalize the tokenizer at stream termination and stop-token boundaries so tokenizer 3's buffered byte-fallback suffix is emitted through the existing stop-string filter. The regression check covers length completion, bare EOF, predecoded terminal text, hidden stop tokens, and visible stop tokens, including overlapping visible/hidden IDs.

Closes ai-dynamo/frontend-crates#320.

Internal tracking: DIS-3038.

Validation

Using the committed registry dependencies and lockfiles, with no local Cargo override:

• 1,457 targeted dynamo-llm library tests passed; 12 ignored. Covers protocols, preprocessing, HTTP handlers, backend decoding, migration, and mocker behavior.
• 86 integration tests passed across frontend_protocol_validation_http, responses_http_replay, test_stop_behavior, test_streaming_usage, and tokenizers. The 22 tokenizer tests include the HuggingFace prefill/multibyte cases and now finalize their decode streams.

Summary by CodeRabbit

• Bug Fixes
  • Improved streaming text completion so buffered text is properly delivered at stream endings and stop boundaries.
  • Corrected error reporting for Responses API requests, including clearer handling of rate limits and invalid prompts.
• New Features
  • Anthropic tool requests now preserve each tool’s strictness setting.
  • Responses API streaming now recognizes additional audio and shell-call events, and handles service tiers according to their supported options.

[rmccorm4]

/ok to test eafa074

[rmccorm4]

/ok to test e6f6901

[rmccorm4]

/ok to test 7bd5f17

[coderabbitai]

Walkthrough

The pull request updates pinned dependencies, changes decoder finalization at terminal boundaries, forwards Anthropic tool strictness, and adapts OpenAI Responses conversions to updated types. It also updates usage-detail initializers and related tests.

Changes

LLM decoding and protocol updates

Layer / File(s)	Summary
Dependency pins and usage-detail initializers Cargo.toml, lib/bindings/python/Cargo.toml, lib/llm/src/http/service/generate.rs, lib/llm/src/http/service/openai.rs, lib/llm/src/mocker.rs, lib/llm/src/protocols/anthropic/stream_converter.rs, lib/llm/tests/test_streaming_usage.rs, lib/sidecar/sglang/src/protocol.rs, lib/sidecar/trtllm/src/convert.rs, lib/sidecar/trtllm/src/tests/convert_response.rs	The pinned dynamo-* dependency versions change. Usage-detail initializers set unspecified fields from defaults. The OpenAI service also uses typed Responses error codes.
Decoder finalization and terminal output lib/llm/src/backend.rs, lib/llm/tests/tokenizers.rs	The backend finalizes tokenizer output at terminal chunks and EOF, applies stop-string filtering, and queues terminal outputs for finalized choices. Tokenizer tests append the final chunk returned by finish().
Anthropic tool and text conversion lib/llm/src/protocols/anthropic/types.rs	Anthropic text parts initialize added fields with defaults. Tool conversion forwards strict; tests cover true, false, and omitted values.
Responses request and response conversion lib/llm/src/protocols/openai/responses/mod.rs	Responses conversion uses Responses-specific service tiers and tool-choice variants. It returns an error for the unsupported ultrafast tier and updates response and input initializers for added fields.
Responses streaming conversion lib/llm/src/protocols/openai/responses/stream_converter.rs	The stream converter uses typed errors and Responses service tiers, initializes added response and function-call fields, and maps additional audio and shell-call event types.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Severity of issue fixed: Medium

Merge Risk: 🔵 Low · up to 7bd5f

A response can include text that its stop string should exclude at a visible stop-token boundary. This is a narrow output defect to fix before merging or explicitly accept.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check		Issue #320 is limited to Anthropic tool strictness propagation. The PR also changes tokenizer stream finalization and stop-token handling, Responses API service-tier and event behavior, typed error co…	Split the unrelated tokenizer and additional Responses API behavior into separate pull requests, or link active issues that require those changes and document the required connection.
Docstring Coverage		Docstring coverage is 79.76% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 84 functions across 13 files. (2 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check		Issue #320 requires Anthropic tool strict to reach ChatCompletionTool.function.strict. lib/llm/src/protocols/anthropic/types.rs now forwards tool.strict. Tests cover Some(true), `Some(false)…
Title check		The title clearly identifies the main changes: preserving Anthropic tool strictness and updating frontend dependencies. It is concise and related to the changeset.
Description check		The description provides detailed scope, implementation context, issue references, and validation results. It does not use the template headings or include a dedicated reviewer-start section, but it i…

Full details: Out of Scope Changes check

Explanation

Issue #320 is limited to Anthropic tool strictness propagation. The PR also changes tokenizer stream finalization and stop-token handling, Responses API service-tier and event behavior, typed error conversion, and related migration fixtures. Dependency updates and required API adaptations can support the strictness fix, but the tokenizer behavior and additional Responses API behavior have no stated connection to #320.

Full details: Docstring Coverage

Explanation

Docstring coverage is 79.76% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 84 functions across 13 files. (2 skipped: 2 unsupported.)

• Fix all pre-merge checks with AI

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[nv-yna]

Reviewed d7fc3cafab1a11cb3d487bdbedc38da873ec58c8 with Codex only.

Two verified findings are attached: High — byte-fallback output is lost across migration; Medium — buffered text is attributed to the wrong token's logprob. These findings remain actionable despite this approval.

Validation: 19 backend, 900 protocol, 162 HTTP/OpenAI, and 35 existing migration test executions passed; 10 protocol tests ignored. Two added review-only regression probes reproduced the attached defects. No GPU inference or HTTP end-to-end test was run. The existing visible-stop-token discussion was checked and not duplicated.

[dmitry-tokarev-nv]

No P0 or P1 found in the dependency files. One P2 is open. Read at d7fc3cafab1a11cb3d487bdbedc38da873ec58c8.

This review covers Cargo.toml, lib/bindings/python/Cargo.toml and the three Cargo.lock files.

[P2] On a migration retry to another worker, the tokenizer pin =3.1.0 at Cargo.toml:68 loses a buffered byte-fallback character. This review reproduced the open thread from @nv-yna on that line. The measurement is in the reply on that thread. Please answer there.

@GuanLuo, yes. The head pins dynamo-protocols = { version = "=8.1.0" } at Cargo.toml:77. lib/llm, lib/backend-common and deploy/inference-gateway/ext-proc take it with workspace = true. The root, lib/bindings/python and lib/bindings/kvbm lockfiles all resolve 8.1.0 with sha256 aca812fe988abff6….

Open items from other reviewers that this review did not test:

• @nv-yna [Medium], selected-token text and logprob, lib/llm/src/backend.rs:898.
• CodeRabbit, a visible stop token and a stop string, lib/llm/src/backend.rs:912.

The six crate moves are consistent and match their sources, with controls.

• Each lockfile changes six packages and nothing else: root 1052 packages, lib/bindings/python 945, lib/bindings/kvbm 884. The dependency list of each moved crate did not change.
• Each new .crate from static.crates.io has the sha256 that all three lockfiles record (18 of 18).
• The five dynamo-* crates came from crates.io trusted publishing in ai-dynamo/frontend-crates. Each packaged source file is byte-identical to that repository at the commit in .cargo_vcs_info.json, and each commit is on main. Renderer 8.0.1, protocols 8.1.0, parsers 10.0.1 and parsers-v2 0.7.20 contain ai-dynamo/frontend-crates#322 and ai-dynamo/frontend-crates#375.
• async-openai 0.42.1 has the same sole owner (64bit) and repository as 0.41.1. dynamo-protocols turns on only *-types features, so it resolves with 5 dependencies and no HTTP client.
• None of the twelve old and new crate versions has a build script, a proc-macro or a links key. The new process spawn in dynamo-parsers 10.0.1 is in a #[cfg(test)] module.
• cargo tree --locked --offline exits 0 in the root, lib/bindings/python and lib/bindings/kvbm. With the base lockfiles under the head manifests, it exits 101 in all three.
• cargo test --locked --offline -p dynamo-llm --no-default-features at the head on macOS: lib 3027 passed, 3 failed, 13 ignored. The base tree has the same failure count. The gRPC metrics-bind test fails on both trees, and the other failures change between runs and pass alone. These targets pass: backend 1, test_stop_behavior 16, test_streaming_usage 14, tokenizers 22, responses_http_replay 24, frontend_protocol_validation_http 10, anthropic_http_replay 9.
• On the head, Pre Merge passed, including rust-tests and rust-clippy for the root and both bindings. At 18:40Z the PR run on pull-request/15547 still had 4 jobs running, and no job had failed.
• Merged with main at 6c541e70e3, which brings #15979 into the same Anthropic files, the tree builds. The Anthropic lib tests (102) and anthropic_http_replay (10) pass on that merge.

[dmitry-tokarev-nv]

Approving from DevOps side.

[rmccorm4]

@GuanLuo @luckyq - I addressed the crate version bumps

@GuanLuo - please help address @nv-yna 's findings on tokenizer + migration, feel free to push directly to this PR

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rmccorm4]

/ok to test 4f3cb58