[wgordon17]

Summary

• Route Claude models on Vertex AI through the native Anthropic rawPredict endpoint instead of the generic OpenAI-compatible partner endpoint (which 404s/errors for Claude on at least some projects), stripping the body-level model field rawPredict rejects and injecting the required anthropic_version.
• Since rawPredict only ever returns a complete JSON body (never real SSE framing), streaming requests now get a genuine Anthropic-format SSE stream synthesized from that JSON so the existing claude-to-openai response translator can parse it correctly.
• Fix two response-format resolution bugs that silently dropped a custom model's DB-stored targetFormat override whenever the model id also existed in the static provider registry (as claude-sonnet-4-6/claude-opus-4-7 do under vertex).

⚠️ base-red inherited: #9985 — the ESLint parsing error in @omniroute/opencode-plugin/src/index.ts, the no-new-func finding in tests/unit/cli-env-inline-comment-10100.test.ts, the 5 window.location.href warnings, and 4 fast-path unit test failures (combo-quota-exhaustion-only-fallback, free-tier-providers-wave1-a, combo-499-abort, combo-stream-readiness-fallback) are confirmed identical to release/v3.8.50 tip (zero diff) and independently red on the release branch's own CI — not introduced by this PR.

[diegosouzapw]

The PR remains in the release drain with needs-vps/hold-vps: the Vertex rawPredict streaming behavior requires the documented live validation before merge. It remains open and will not be bypassed with an admin merge.

[diegosouzapw]

The PR remains in the release drain with needs-vps/hold-vps: the Vertex rawPredict streaming behavior requires the documented live validation before merge. It remains open and will not be bypassed with an admin merge.

[wgordon17]

Live validation complete (Hard Rule #18 — the network-dependent rawPredict/streaming behavior TDD can't fully prove). Redacted below: GCP project id → <project>, service-account email → <service-account>, internal connection/account ids → <conn-id>, egress IP → <ip>. Nothing else altered.

Environment: isolated OmniRoute instance built from this branch (fresh throwaway DATA_DIR, no production data touched), wired to a real GCP service account with Anthropic Claude access enabled on Vertex AI. All calls below are real network round-trips to the live rawPredict endpoint — no mocks, no stubbed fetch.

---

1. Raw upstream proof — rawPredict itself, bypassing OmniRoute entirely

Direct fetch to https://aiplatform.googleapis.com/v1/projects/<project>/locations/{region}/publishers/anthropic/models/claude-haiku-4-5@20251001:rawPredict with a minted OAuth token, same body shape the executor sends (anthropic_version, max_tokens, messages, no model field):

=== us-east5 → 200 ===
{"model":"claude-haiku-4-5-20251001","id":"msg_vrtx_011Ce8XWP3ndv9gYvCxtjnHT","type":"message","role":"assistant","content":[{"type":"text","text":"Hey! 👋 How's it going"}],"stop_reason":"max_tokens","stop_sequence":null,"usage":{"input_tokens":8,"output_tokens":10}}

=== europe-west1 → 200 ===
{"model":"claude-haiku-4-5-20251001","id":"msg_vrtx_011Ce8XWTKmFMHbrc6Y9MWgR","type":"message","role":"assistant","content":[{"type":"text","text":"# Hello! 👋\n\nHow can I"}],"stop_reason":"max_tokens","stop_sequence":null,"usage":{"input_tokens":8,"output_tokens":10}}

=== global → 200 ===
{"model":"claude-haiku-4-5-20251001","id":"msg_vrtx_011Ce8XWfMokRu2jzX5XFuJU","type":"message","role":"assistant","content":[{"type":"text","text":"Hello! 👋 How can I help"}],"stop_reason":"max_tokens","stop_sequence":null,"usage":{"input_tokens":8,"output_tokens":10}}

Confirms rawPredict genuinely works and returns real Anthropic-message-shaped JSON — when the request is servable in that region. us-central1 (this executor's hardcoded region default when no providerSpecificData.region is set) is not a valid region for this model on this project; global/us-east5/europe-west1 are. Not this PR's bug, but worth documenting here since it'll otherwise look like a "model not found" code issue to the next person debugging this path.

2. Full pipeline, non-streaming — /v1/chat/completions → routing → VertexExecutor → live upstream

Request:

{"model":"vertex/claude-haiku-4-5@20251001","stream":false,"max_tokens":30,"temperature":0,"messages":[{"role":"user","content":"Reply with exactly the token: RAWPREDICT_OK"}]}

Relevant log lines (format resolution + request):

[sse] Provider: vertex, Model: claude-haiku-4-5@20251001                 tag=ROUTING
[sse] Using vertex account: <conn-id>                                    tag=AUTH
[sse] openai → claude | stream=false                                     tag=FORMAT
[sse] VERTEX | claude-haiku-4-5@20251001 | 1 msgs                        tag=REQUEST

Response — 200, full body:

{
  "id": "chatcmpl-msg_vrtx_011Ce8Y2SBe1Yr3ub1dv2fSA",
  "object": "chat.completion",
  "created": 1786979518,
  "model": "claude-haiku-4-5@20251001",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": { "role": "assistant", "content": "RAWPREDICT_OK" }
    }
  ],
  "usage": { "prompt_tokens": 19, "completion_tokens": 9, "total_tokens": 28 }
}

The completion is the exact literal token requested — not a cached/mocked value, a real generation.

3. Full pipeline, streaming — synthesized SSE consumed by the downstream translator

Request:

{"model":"vertex/claude-haiku-4-5@20251001","stream":true,"max_tokens":30,"temperature":0,"messages":[{"role":"user","content":"Reply with exactly the token: STREAM_OK"}]}

Relevant log lines:

[sse] openai → claude | stream=true                                      tag=FORMAT
[sse] VERTEX | claude-haiku-4-5@20251001 | 1 msgs                        tag=REQUEST
[ProxyEgress] vertex/<conn-id> in=? out=<ip> proxy=direct status=success
[sse] Stream readiness confirmed in 8ms (vertex/claude-haiku-4-5@20251001)  tag=STREAM
[sse] Translation mode: claude → openai                                  tag=STREAM
[USAGE] VERTEX | in=17 | out=7 | account=<conn-id>
[STREAM] VERTEX | claude-haiku-4-5@20251001 | 1273ms | complete

Response — 200, content-type: text/event-stream, full raw body as received by the client:

data: {"id":"chatcmpl-msg_vrtx_011Ce8Y2WzN2ApV6JzhmS5iM","object":"chat.completion.chunk","created":1786979519,"model":"claude-haiku-4-5-20251001","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-msg_vrtx_011Ce8Y2WzN2ApV6JzhmS5iM","object":"chat.completion.chunk","created":1786979519,"model":"claude-haiku-4-5-20251001","choices":[{"index":0,"delta":{"content":"STREAM_OK"},"finish_reason":null}]}

data: {"id":"chatcmpl-msg_vrtx_011Ce8Y2WzN2ApV6JzhmS5iM","object":"chat.completion.chunk","created":1786979519,"model":"claude-haiku-4-5-20251001","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":17,"completion_tokens":7,"total_tokens":24}}

: x-omniroute-cache-hit=false
: x-omniroute-latency-ms=153
: x-omniroute-tokens-in=17
: x-omniroute-tokens-out=7
: x-omniroute-model=claude-haiku-4-5@20251001
: x-omniroute-provider=vertex
data: [DONE]

This is the chain the PR actually adds: rawPredict returns one complete JSON body (confirmed in §1) → synthesizeClaudeSse() turns it into a real Anthropic message_start/content_block_delta/message_stop event sequence internally → the existing claude→openai stream translator parses that correctly (Translation mode: claude → openai above) and re-emits valid OpenAI chunk-shaped SSE with the actual streamed content, correct finish_reason, correct token usage, clean [DONE].

4. Baseline without a format override (context, not a regression)

Same model, same connection, but without registering it as a custom model with targetFormat: "claude" — i.e. relying only on the provider's base format:

[sse] openai → gemini | stream=false                                     tag=FORMAT
...
[MALFORMED-200] mode=nonstream provider=vertex model=claude-haiku-4-5@20251001 reason=empty_choices recvBytes=457
status: 502
{"error":{"message":"[vertex/claude-haiku-4-5@20251001] upstream returned an empty response without usable output", ...}}

Expected: without targetFormat: "claude" set, format resolution defaults to the provider's base "gemini" and the response parser can't read Claude's response shape → empty. Confirms the override in §2/§3 is load-bearing, not incidental — this is the exact mechanism bullet 3 of the PR description is about.

---

No code changes from this validation — ready to merge on this evidence.

[diegosouzapw]

Hi @wgordon17 — thanks for the detailed diagnosis here. Looking at the current release branch, the rawPredict routing + real-streaming-synthesis fix you describe is already live in open-sse/executors/vertex.ts (merged via a separate fix for issue #8994). The remaining diff on this PR is just whitespace/formatting noise with no functional change left.

We're evaluating this as a likely-subsumed PR — really appreciate you tracking down the root cause, and we want to make sure you get credit for the diagnosis before deciding how to close this out. Will follow up here once that's settled.

[diegosouzapw]

Thanks for the diagnosis and fix — verified the substance already landed on the release tip: open-sse/executors/vertex.ts already routes Claude models through the native Anthropic rawPredict endpoint (with the anthropic_version injection and stripped model field) and already synthesizes a genuine SSE stream from the complete JSON body rawPredict returns.

Auditing this PR's actual current diff against the release tip: it's now down to config/quality/eslint-suppressions.json (trailing-newline only), and pure Prettier reformatting in open-sse/utils/stream.ts and tests/unit/executor-vertex-extended.test.ts (line wrapping, no behavior change) — the functional Vertex/Claude routing change itself isn't in this diff anymore, presumably because it already merged separately and this branch never got a chance to rebase past it.

Closing as subsumed — thanks again for finding and root-causing the original bug.