feat(vertex): Llama + OpenAI-compatible MaaS family via the openapi shim (#302 Phase E) - #446
Conversation
…him (#302 Phase E, D5.4) Vertex Model Garden serves Meta Llama, DeepSeek, Qwen, gpt-oss, MiniMax, Moonshot, and Z.ai through a single OpenAI chat-completions shim at `endpoints/openapi/chat/completions` (the model id rides in the request body, NOT a `publishers/<vendor>/...:rawPredict` URL). This wires that one rail, so the whole family dispatches through it. - `VertexPublisher`: the old (unimplemented) `Meta` variant becomes `OpenAiCompat`, and `from_upstream_id` resolves the documented MaaS prefixes (`meta/` | `llama` | `deepseek` | `qwen` | `openai/gpt-oss` | `minimaxai/` | `moonshotai/` | `zai-org/`) to it. Mistral / AI21 stay separate (they use `:rawPredict`, still deferred). - `chat_openai_shim` (non-stream) + `chat_openai_shim_stream` (streaming): POST the OpenAI chat body to the openapi shim with the GCP OAuth2 Bearer (same token path as Gemini/Claude). Reuses the OpenAI request serializer + response/stream-chunk decoders verbatim (new dep on aisix-provider-openai) so the wire matches direct OpenAI. Streaming reuses the shared SseDecoder ([DONE]-terminated OpenAI SSE). - The model id is KEPT in the body (the shim keys off it) — opposite of the Gemini/Anthropic publisher paths which strip it from the body and carry it in the URL. Reference impl confirms the shim endpoint + the OpenAI-handler family grouping + the OpenAI body/response wire; Google's Vertex Llama docs confirm the `endpoints/openapi/chat/completions` path. Tests: resolver maps the whole MaaS family to OpenAiCompat (+ Mistral/ AI21 stay on their own arms); wiremock dispatch test pins the openapi URL + Bearer + model-in-body; the stream-not-implemented test repointed off Llama (now wired) to the still-deferred rawPredict publishers. 77 unit tests pass; clippy + workspace check clean. Live e2e (mock-vertex openapi shim + spec) lands in the AISIX-Cloud companion PR. Streaming for Claude-on-Vertex (`:streamRawPredict`, D5.3) + Mistral/AI21 (D5.4) remain.
|
Warning Review limit reached
More reviews will be available in 14 minutes and 43 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
Note 🎁 Summarized by CodeRabbit FreeYour organization has reached its limit of developer seats under the Pro Plan. For new users, CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please add seats to your subscription by visiting https://app.coderabbit.ai/login.If you believe this is a mistake and have available seats, please assign one to the pull request author through the subscription management page using the link above. Comment |
…stream) The non-stream openapi-shim dispatch had a wire test pinning the URL, Bearer, and model-in-body; the streaming handler had none. Add a streaming wire test (capturing responder + OpenAI chat.completion.chunk SSE + [DONE]) that pins the openapi URL, Bearer auth, `stream:true` in the body, the model id staying in the body (never the URL), and that the SSE frames decode into ChatChunks whose aggregated content and terminal finish_reason match the upstream stream.
1 participant
Summary
Wires the OpenAI chat-completions shim rail on Vertex. Model Garden serves Meta Llama, DeepSeek, Qwen, gpt-oss, MiniMax, Moonshot, and Z.ai through a single endpoint
endpoints/openapi/chat/completions(model id in the body, not apublishers/<vendor>/...:rawPredictURL). One handler ⇒ the whole MaaS family.VertexPublisher: old unimplementedMeta→OpenAiCompat;from_upstream_idresolvesmeta/|llama|deepseek|qwen|openai/gpt-oss|minimaxai/|moonshotai/|zai-org/to it. Mistral/AI21 stay separate (:rawPredict, deferred).chat_openai_shim(non-stream) +chat_openai_shim_stream(streaming): POST the OpenAI body to the shim with the GCP OAuth2 Bearer (same token path as Gemini/Claude). Reuses the OpenAI request serializer + response/stream decoders verbatim (new dep onaisix-provider-openai) so the wire matches direct OpenAI. Streaming reuses the sharedSseDecoder([DONE]-terminated OpenAI SSE).Reference-impl + upstream
endpoints/openapi/chat/completionsURL; Mistral/AI21/Claude userawPredict(kept separate here).Tests
cargo test -p aisix-provider-vertex— 77 pass (resolver maps the full MaaS family toOpenAiCompat; wiremock dispatch test pins openapi URL + Bearer + model-in-body; stream-not-implemented test repointed off Llama to the still-deferred rawPredict publishers).cargo clippy --all-targets -- -D warnings+cargo check --workspaceclean.Scope / follow-ups
:streamRawPredict, D5.3) + Mistral/AI21 (D5.4) remain.🤖 Generated with Claude Code