docs: /v1/messages/count_tokens now works on gRPC and ZMQ workers; the docs say they answer 501

[github-actions]

What this PR does

Does POST /v1/messages/count_tokens work when my workers are gRPC- or ZMQ-connected (including gRPC prefill/decode and EPD deployments), or does it still answer 501 Not Implemented as the docs say — who counts the tokens, which request fields take part, and what are the refusals for an unknown model, a tokenizer still loading, or a prompt that does not render?

Why we need it

Source change: smg-project/smg@322b8a3

Commit 173 (#3071, 322b8a33, 2026-10-10, 'count_tokens on every worker type, GET /v1/models/{id}, and the models list in the caller's API shape'); its supplied diff was read in full and every claim re-verified on the current default-branch snapshot. Behavior on current main: model_gateway/src/routers/grpc/router.rs:708-734 route_messages_count_tokens_impl — '/v1/messages/count_tokens, answered from the gateway's own tokenizer: the request is rendered through the model's chat template exactly as /v1/messages renders it and the prompt's token count is returned. No worker takes part, so every worker type is covered'; count_prompt_tokens (:736-798) resolves the model alias, passes the request's tools through message_utils::template_tools and resolves its media as the preparation stage does (resolve_messages_media), and answers {"input_tokens": N}. The RouterTrait impl (:1173-1180) is mode-independent, so the gRPC PD and EPD routers answer too, and ZMQ workers ride the same router (routers/factory.rs:63, ConnectionMode::Grpc | Zmq). Refusals, each pinned by a router.rs test: 404 model_not_found for a model no worker serves, even when a removed worker left a tokenizer in the registry (count_tokens_refuses_a_model_no_worker_serves_whatever_the_tokenizer_registry_holds, :1376); 503 tokenizer_not_ready with Retry-After: 1 while a served model's tokenizer is still loading (count_tokens_answers_not_ready_while_a_served_models_tokenizer_loads, :1402 — this loading-window code comes from the later commit 64, #3280, 79ab46b0, whose diff was also read; 500 tokenizer_not_found remains only under --disable-tokenizer-autoload); 400 process_messages_failed for a prompt that does not render; 500 tokenization_failed when encoding fails (count_tokens_is_answered_from_the_tokenizer, :1307-1370, pins the OK shape). IGW: routers/gateway.rs:514-553 — counting now qualifies every self-hosted pool ('a disaggregated pool qualifies by its prefill leg alone, whichever gRPC family is mounted'); a model with an external worker still goes to that provider's backend, and a model no pool serves answers model_not_found. The same commit switched the endpoint's body extractor to ValidatedJson (model_gateway/src/server.rs:467 on current main; crates/protocols/src/validated.rs:48-373 — rejections answer the public JSON envelope, e.g. invalid_json for a syntax error, test rejections_carry_their_code_as_an_extension), so the old plain-text parse answers are gone. What stays true (verified, not to be touched): the Anthropic API backend and the other external providers still answer 501 (routers/mod.rs:279-291 default impl; messages.md:30/:34/:353), and the HTTP-worker and HTTP-PD forwarding rows (messages.md:271-272) are unchanged. Stale docs, all re-read tonight at current line numbers: reference/api/messages.md:26-27 — the Supported Backends table's count_tokens column reads '501 Not Implemented' for gRPC workers and ZMQ workers — false on main; :231 — 'SMG does not count the tokens itself: it forwards the request to a worker that implements the endpoint and relays the worker's response (smg-project/smg#2638)' — false for gRPC/ZMQ workers; :244 — 'Any other field ... is forwarded to the worker unchanged' — true only for HTTP workers, needs rescoping (on the gRPC path extra fields are accepted and take no part in the count); :273 — Backend Support row 'gRPC or ZMQ workers | 501 Not Implemented' — false, and the gRPC-served refusal contract (404/503 tokenizer_not_ready with Retry-After/400) is undocumented; :275 — the IGW row ('SMG chooses between the model's HTTP and HTTP PD workers') is under-inclusive; :454 — the count_tokens plain-text parse-answer exception (400/415/422) is stale, body refusals now leave in the standard envelope. concepts/routing/pd-disaggregation.md:238 — the Supported APIs table says gRPC PD answers ':material-close: 501' for Messages token counting — false on main (the gRPC PD router answers from the gateway's tokenizer, no leg takes part); :242 'SMG sends it to a single prefill worker' needs HTTP-PD scoping. getting-started/pd-disaggregation.md:410 — '/v1/messages/count_tokens works with HTTP workers and goes to a single prefill worker' — under-inclusive on main. Ownership: which worker transports serve the endpoint is gRPC worker serving (this scan); the same commit's GET /v1/models/{id} route and Anthropic-shaped models-list halves are public-API-shape concerns left to the api-protocols scan and are not documented here, and the Messages response-field semantics are untouched. Version framing: #3071 (ID 173, 2026-10-10) postdates the v1.12.0 bump (ID 364, #2888, 2026-10-08) on first-parent order, so present gateway-side counting as current-main behavior newer than v1.12.0 and keep 501-on-gRPC/ZMQ as the v1.12.0-and-earlier caveat, the style these pages already use; #3280 (ID 64, 2026-10-12) likewise postdates the bump, so the 503 tokenizer_not_ready loading-window refusal is also current-main-only. One independent concern: one commit made one endpoint served by the gRPC router family, corrected on the three pages that state that endpoint's backend support. Dedup: a repo-wide grep of the PR index shows no open or closed PR mentions #3071, count_tokens, or these 501 rows; closed-unmerged #93 (Messages stream content blocks) and #110 (stop_reason precedence) were different Messages concerns a maintainer declined and are not recreated; no open PR (#144-#161) holds any of the three files; reference/api/extensions.md:57 (held by open PR #153) is a pointer row with no backend claim and needs no edit. Edits are row- and sentence-level across three pages, far under 1,000 lines.

Scope: grpc-pipeline / messages-count-tokens-served-from-gateway-tokenizer. Other concerns are deferred.

How to test

• Passed the documentation path and size guard (under 1000 added plus deleted lines; no file-count limit).
• Passed an independent accuracy and single-concern review.
• Passed git diff --check and pnpm check and the production pnpm build.

Checklist

• Tests added/updated (if applicable)
• Docs updated (if applicable)
• Commit includes a DCO sign-off

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: d7b5dde6-f559-4697-95c8-e072fd37b9e0

---

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}