[jthomson04]

Overview:

Selection requests carry a whole prompt's token_ids. Parsing them was the largest cost in both the standalone selection service and the embedded SelectionService that Ray Serve LLM's KVAwareRouter calls. This PR removes the serde flatten buffering on the prompt, extracts token_ids with a typed pyo3 conversion in the binding, and parses HTTP bodies without serde_path_to_error.

A/B on one GB200 Grace node, same allocation. Setup: 1,024 mocker workers publishing real ZMQ KV events, 8,192 concurrent AgentX sessions with a 16k shared prompt (mean ISL 99k tokens), service pinned to 16 cores, 0 errors in every arm.

	HTTP before	HTTP after	Embedded (Ray-style) before	Embedded after
Requests/s	1,323	1,327 (engine-bound)	120	221 (+85%)
Service cores	15.1	12.0 (−21%)	1.5 (GIL thread 1.0)	2.0 (GIL thread 1.0)
CPU ms per request	11.4	9.1	12.4	9.1
Selection latency p50 / p99	6.5 / 25 ms	1.8 / 4.3 ms	65 s (queued on GIL)	34 s
Parse/convert, CPU ms per request	4.4	1.3	4.3	1.2
Share of CPU in serde Content buffering	38%	2.5%	40%	2%

Parse microbench, 100k-token /select body, local x86: 3.8 ms before, 1.55 ms without flatten, 0.82 ms with both changes.

Details:

• services/selection/types.rs. SelectRequest, SelectAndReserveRequest, ReservationRequest, PotentialLoadsRequest and OverlapScoresRequest deserialize through flat wire structs (#[serde(from)], generated by inline_prompt_wire!) instead of #[serde(flatten)] prompt. The public structs and the wire fields are unchanged. flatten buffered the body into serde Content; under pythonize it also forced pyo3's slow 128-bit int extraction, about 48 ns per token. is_eagle still accepts only a boolean or null: a plain typed bool would let pythonize coerce truthy values such as "false".
• lib/bindings/python/rust/llm/kv.rs. select, select_and_reserve, create_reservation, overlap_scores and potential_loads extract token_ids as a typed Vec<u32>, then depythonize the rest of the dict. Invalid tokens still raise ValueError.
• services/selection/server.rs. The five prompt-carrying routes use a PromptJson<T> extractor: serde_json::from_slice, without axum Json's per-element path tracking. It keeps Json's rejections: 415 for a missing or non-JSON Content-Type, 400 for malformed JSON, 422 for a body of the wrong shape, and the body limit.

Where should the reviewer start?

services/selection/types.rs (inline_prompt_wire!), then PromptJson in server.rs.

Validation

• cargo clippy -p dynamo-kv-router --features standalone-selection --all-targets -- -D warnings
• cargo clippy --features select-service -- -D warnings in lib/bindings/python
• cargo test -p dynamo-kv-router --features standalone-selection --lib services::: 251 passed, including the new prompt_carrying_requests_read_top_level_prompt_fields and prompt_routes_keep_json_rejection_statuses
• pytest lib/bindings/python/tests/test_kv_bindings.py -k selection_service: 16 passed; the malformed-payload test now also covers "is_eagle": "false" (fails before the fix)

🤖 Generated with Claude Code

[jthomson04]

/ok to test b3ace9a

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: ai-dynamo/dynamo
• Branch: main
• File: .coderabbit.yaml
• Commit: 3032e8a
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: df4150ef-0e36-4d55-b3c7-85f3e95a86c3

Commits

Reviewing files that changed from the base of the PR and between 3032e8a and b3ace9a.

Files selected for processing: 4

• lib/bindings/python/rust/llm/kv.rs
• lib/kv-router/src/services/selection/server.rs
• lib/kv-router/src/services/selection/tests.rs
• lib/kv-router/src/services/selection/types.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Selection request deserialization now maps top-level prompt fields into nested prompts. Router endpoints use a custom JSON extractor with explicit content-type and parse-error statuses. Python bindings use a helper to process dictionary token_ids.

Changes

Selection prompt requests

Layer / File(s)	Summary
Flat prompt request deserialization lib/kv-router/src/services/selection/types.rs, lib/kv-router/src/services/selection/tests.rs	Wire structs deserialize top-level prompt fields into nested prompts and apply request defaults. Tests check prompt fields, defaults, and selected request values.
Router JSON extraction and rejection responses lib/kv-router/src/services/selection/server.rs, lib/kv-router/src/services/selection/tests.rs	Selection endpoints use PromptJson. It accepts JSON and +json content types, returns 415 for missing or unsupported types, preserves body-extraction rejection details, maps data errors to 422, and maps other parse errors to 400. Tests check selected rejection responses.
Python prompt request deserialization lib/bindings/python/rust/llm/kv.rs	A helper extracts non-None token_ids from dictionary inputs before request deserialization, then assigns them to the parsed prompt. Five request methods use the helper.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: Minimal · up to b3ace

No merge-blocking behavior change was identified; the selection request parsing changes are ready for normal validation.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 30.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 23 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the performance change: parsing selection requests without serde flattening or path tracking.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description is thorough and mostly complete. It has an Overview with the goal (removing serde flatten buffering, typed token_ids extraction, and HTTP parsing without serde_path_to_error), a Detail…

Full details: Description check

Explanation

The description is thorough and mostly complete. It has an Overview with the goal (removing serde flatten buffering, typed token_ids extraction, and HTTP parsing without serde_path_to_error), a Details section that lists the changed files and the reasoning, a Where should the reviewer start section, and a Validation section with commands and results. Benchmark data supports the claims. The one required gap is the Related Issues section. It still holds the template placeholder 'Closes #XXXX' and does not name a real issue, although the objectives cite PR #17238 and the template marks this section as required. The check passes on overall completeness, but the Related Issues section needs a real reference before merge.

Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

✅ Dynamo PR CI passed — run 40131064575 (attempt 1) on 654d716824

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 40131064575. Updated on every full-CI run of this PR.}

[jthomson04]

/ok to test 654d716