[JustinTong0323]

Follow-up to #42316: brings the router's /v1/messages in line with the engine's own Messages adapter (python/sglang/srt/entrypoints/anthropic/). Where the router behaves differently on purpose, the reason is the Anthropic Messages API contract (listed below).

Fixes

1. Streaming error envelope. Every failure path (in-band upstream error, malformed chunk, upstream cut, body error) now emits message_start (if not yet sent), content_block_stop for the open block, then error, then message_stop, matching _flush_on_error. Strict SDK stream parsers reject a stream that leaves a block open.
2. Error message scrubbing. 5xx messages become "Internal server error". 4xx messages lose Traceback/File "/…" lines and are capped at 500 characters (_scrub_error_message). This applies to buffered errors and to in-band stream errors. An Anthropic-shaped upstream envelope keeps its error.type for 4xx only.
3. Token cap. Unknown top-level fields are no longer forwarded. The only fields forwarded as-is are temperature, top_p, top_k, chat_template_kwargs and continue_final_message. As a result max_completion_tokens can no longer override max_tokens.
4. thinking validation. enabled requires budget_tokens >= 1024. disabled rejects budget_tokens and display. adaptive rejects budget_tokens. display must be summarized or omitted. All of these return 400.
5. redacted_thinking in assistant history returns 400 instead of being dropped.
6. Empty signatures. Thinking blocks no longer carry "signature": "", in both buffered replies and content_block_start.
7. SSE robustness.
   • A data: payload that is not valid JSON ends the stream through the envelope in (1) with Stream processing error.
   • [DONE] completes the message right away; the router no longer waits for EOF.
   • Tool-call argument deltas that arrive with no open tool_use block are dropped with a warning. Before, they opened an unnamed block.
8. tool_result. The legacy id field is accepted when tool_use_id is absent. is_error: true content is forwarded unchanged; the router no longer adds an Error: prefix.
9. Deferred tooling.
   • tool_reference blocks inside tool_result become {"type":"tool_reference","name":…} parts.
   • Each run of references and each run of other parts becomes its own tool message with the same tool_call_id, because templates expand references only at the start of a tool message.
   • Tool definitions keep defer_loading.
10. Smaller parity items.
    • input_schema without type gets "type": "object".
    • A tool_choice: {type: "tool"} name that is not a forwarded custom tool returns 400.
    • output_config.effort accepts minimal, and xhigh is sent as max.
    • message_delta.usage carries only output_tokens.
    • The buffered reply's model echoes the engine's response.
    • A blank system prompt is skipped.

New tests:

• stream.rs:
  • the failure envelope for each error source;
  • [DONE] finalization without EOF;
  • dropping argument deltas with no open call.
• request.rs: thinking shapes, tool_result/tool_reference grouping, tool definitions.
• mod.rs: scrubbing.
• tests/proxy/messages_api.rs: an end-to-end malformed-chunk stream through the proxy.

Existing tests that asserted the old behavior were updated: the Error: prefix, the blanket pass-through, the empty signature, and message_delta input usage.

Intentional differences from the engine adapter

• stop_reason.

  • A matched stop_sequences entry gives stop_reason: "stop_sequence" and echoes the sequence in stop_sequence.
  • content_filter maps to "refusal".
  • Both are documented Messages API stop reasons. The engine adapter reports both as end_turn.

• Error types. The router follows the documented status mapping: 402 billing_error, 413 request_too_large, 504 timeout_error, 529 overloaded_error. The engine adapter maps 504 to api_error.

• Stricter validation. These all return 400, where the engine adapter accepts or drops them:

  • role: "system" turns (the Messages API has only user/assistant; system is top-level);
  • non-text system blocks;
  • empty content arrays;
  • image/tool_result/search_result blocks in assistant turns;
  • a tool_result with neither tool_use_id nor id;
  • a tool_reference without a name;
  • an image source without data;
  • unknown block types inside tool_result.

  A bare-string stop_sequences is still accepted.

• Assistant text. Several text blocks are joined with \n into one string, and content is null when the turn has only tool calls. The engine adapter sends a content-part array. A plain string is the representation every chat template renders for assistant turns.

• Turn merging and prefill. Consecutive same-role turns are merged, and a final assistant turn is sent with continue_final_message, as the Messages API documents.

• Forwarded extensions. chat_template_kwargs and continue_final_message pass through because the router README documents them as the way to reach template knobs.

• Undocumented chat fields. Other top-level chat-extension fields (for example a top-level response_format, seed, or min_p) are no longer passed to the chat backend. Only the allowlist above is forwarded; structured output goes through output_config.format.

• Truncated streams. A stream that reaches [DONE] or EOF without a finish_reason ends with an api_error event. The engine adapter reports end_turn in this case if any content was produced.

• Stream errors without a code. An in-band stream error with no code is treated as 5xx and scrubbed.

Known gap

The router sends prior-turn thinking as reasoning_content, and sets thinking/enable_thinking template kwargs for the thinking parameter. The engine adapter does two things the router can't:

• it re-wraps reasoning history in the reasoning parser's delimiters for encoders without native reasoning history;
• it applies parser-specific enable/disable.

Both depend on the engine's reasoning parser, which the router doesn't know. The README already points such models at output_config.effort / chat_template_kwargs.

Verification

Run on an EC2 CPU devbox (c7i.4xlarge, Rust 1.92 from rust-toolchain.toml) at head 6eae98943b40eb119025f8604b4e5bde7b9c35b4, from experimental/sgl-router, with the same commands as pr-test-sgl-router.yml:

• cargo +nightly fmt --all -- --check: pass
• cargo fmt --all -- --check: pass
• cargo clippy --locked --workspace --all-targets -- -D warnings: pass, no warnings
• cargo test --locked --workspace -- --skip parity_matrix: all suites pass
  • sgl_router lib: 1059 passed
  • component: 127 passed
  • proxy: 186 passed, including all 7 messages_api tests
  • kv-indexer crates: 125 passed
• pre-commit on the changed files: pass

---

CI States

Latest PR Test (Base): ❌ Run #38061741758
Latest PR Test (Extra): ❌ Run #38061741628
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

[ShangmingCai]

LGTM