Repository navigation
docs: add Messages API, external providers, and mesh HA documentation - #786
Conversation
What changed: - docs/reference/api/messages.md: new page documenting Anthropic Messages API support (/v1/messages), covering streaming, gRPC backend translation, and connection modes - docs/getting-started/external-providers.md: new guide for using SMG with external LLM providers (OpenAI, Anthropic, xAI, Gemini), covering worker registration, model discovery fan-out, BYOK, and multi-provider routing - docs/reference/configuration.md: add --remove-unhealthy-workers, --disable-tokenizer-autoload, and Mesh Server section (--enable-mesh, --mesh-server-name, --mesh-host, --mesh-port, --mesh-peer-urls) - docs/concepts/architecture/high-availability.md: add quick start CLI examples, Python bindings support, cache-aware state sync endpoints, --router-selector for k8s pod discovery - docs/concepts/routing/cache-aware.md: add event-driven block size learning from KV events, mesh state synchronization section - mkdocs.yml: add External Providers and Anthropic Messages API to nav Why: 69 features shipped since last docs update (~Feb 10) with zero documentation. This covers the highest-impact user-facing features. Signed-off-by: Simo Lin <linsimo.mark@gmail.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly updates the documentation to reflect features shipped since February 10th. It introduces comprehensive guides for integrating with external LLM providers and utilizing the Anthropic Messages API. Additionally, it enhances existing documentation for high-availability architecture, cache-aware routing, and CLI configuration options, ensuring the documentation aligns with the latest capabilities of the system. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
📝 WalkthroughWalkthroughThis pull request adds comprehensive documentation for external LLM provider gateway support (focusing on Anthropic integration), expands high-availability and cache-aware routing documentation with mesh networking details, introduces a new Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Comment |
github-actions
Bot
added
the
documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds extensive documentation for several new features, including the Anthropic Messages API, external provider integration, and high-availability mesh networking. The changes are well-structured and add valuable information. I've found a few issues in the examples provided, primarily an incorrect model name used in the Anthropic API examples which will cause them to fail, and a potentially confusing IP address in a quick-start guide. Addressing these will improve the accuracy and user-friendliness of the new documentation.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/getting-started/external-providers.md`:
- Around line 9-16: The "Before you begin" heading inside the prerequisites div
currently uses an h4 markdown token (#### Before you begin), skipping h2/h3 and
violating heading hierarchy; update the heading to a proper h2/h3 level (e.g.,
replace '#### Before you begin' with '## Before you begin') or alternatively
retain the semantic heading level and apply CSS styling to the .prerequisites
container to preserve visual appearance—modify the markdown token in the <div
class="prerequisites"> block accordingly (look for the '#### Before you begin'
text) and ensure the new heading level follows h1 without skipping levels.
In `@docs/reference/api/messages.md`:
- Around line 15-17: The fenced code block showing the endpoint "POST
/v1/messages" is missing a language specifier; update that fenced block to
include a language (e.g., use ```http) so the block becomes a language-tagged
code fence, which resolves markdownlint MD040 and enables proper syntax
highlighting for the "POST /v1/messages" example.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: efb9a21f-c5a5-4548-844c-8d88d4b1edfe
📒 Files selected for processing (6)
docs/concepts/architecture/high-availability.mddocs/concepts/routing/cache-aware.mddocs/getting-started/external-providers.mddocs/reference/api/messages.mddocs/reference/configuration.mdmkdocs.yml
| ``` | ||
| POST /v1/messages | ||
| ``` |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Add language specifier to fenced code block.
The code block for the endpoint lacks a language specifier. This triggers markdownlint MD040 and may affect syntax highlighting in the rendered documentation.
📝 Proposed fix
-```
+```http
POST /v1/messages</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.21.0)
[warning] 15-15: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/reference/api/messages.md` around lines 15 - 17, The fenced code block
showing the endpoint "POST /v1/messages" is missing a language specifier; update
that fenced block to include a language (e.g., use ```http) so the block becomes
a language-tagged code fence, which resolves markdownlint MD040 and enables
proper syntax highlighting for the "POST /v1/messages" example.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b80f0ce5b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| 1. Fans out the request to all healthy external workers concurrently | ||
| 2. Forwards the caller's token to each upstream provider | ||
| 3. Aggregates and returns the discovered models from all providers |
There was a problem hiding this comment.
The model discovery section says the gateway aggregates results from all external providers, but the implementation in model_gateway/src/routers/router_manager.rs (fetch_upstream_models) fans out and then returns on the first non-empty upstream response, not a merged union. In multi-provider BYOK setups this means users will only see one provider’s models, so this new documentation is materially inaccurate for GET /v1/models behavior.
Useful? React with 👍 / 👎.
| - `GET /mesh/cluster/status` — Cluster membership | ||
| - `GET /mesh/workers/state` — Worker state sync | ||
| - `GET /mesh/policies/state` — Policy state sync |
There was a problem hiding this comment.
This new endpoint list documents /mesh/cluster/status, /mesh/workers/state, and /mesh/policies/state, but the server only registers /ha/status, /ha/workers, and /ha/policies in model_gateway/src/server.rs. Operators following these docs will get 404s for all listed paths, which breaks HA observability and troubleshooting workflows.
Useful? React with 👍 / 👎.
…smg-project#786) Signed-off-by: Simo Lin <linsimo.mark@gmail.com>
…smg-project#786) Signed-off-by: Simo Lin <linsimo.mark@gmail.com>
1 participant
Summary
Catches up documentation with 69 features shipped since the last docs update (~Feb 10).
New Pages
reference/api/messages.md— Anthropic Messages API (/v1/messages) with streaming, gRPC backend translation, and connection modesgetting-started/external-providers.md— Guide for external providers (OpenAI, Anthropic, xAI, Gemini): worker registration, model discovery fan-out, BYOK, multi-provider routingUpdated Pages
reference/configuration.md— New CLI flags:--remove-unhealthy-workers,--disable-tokenizer-autoload, mesh server flags (--enable-mesh,--mesh-host,--mesh-port,--mesh-peer-urls,--mesh-server-name)concepts/architecture/high-availability.md— Quick start CLI examples, Python bindingssmg launch --enable-mesh, cache-aware state sync endpoints,--router-selectorfor k8s pod discoveryconcepts/routing/cache-aware.md— Event-driven KV block size learning, mesh state synchronizationmkdocs.yml— Navigation entries for new pagesNot Documented (Not Ready)
Test plan
mkdocs serverenders all pages without errorsSummary by CodeRabbit