Repository navigation
docs: document experimental Switchyard integration #400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
rapids-bot
merged 2 commits into
NVIDIA:main
from
bbednarski9:docs/switchyard-experimental
Jul 15, 2026
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| --- | ||
| title: "Switchyard (Experimental)" | ||
| sidebar-title: "Switchyard (Experimental)" | ||
| description: "Route Relay LLM requests through the experimental Switchyard Decision API integration." | ||
| position: 5 | ||
| --- | ||
| {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| SPDX-License-Identifier: Apache-2.0 */} | ||
|
|
||
| > **Experimental:** The Switchyard integration is an early-access feature. It | ||
| > is not enabled in default Relay builds, its configuration and contracts can | ||
| > change, and the current deployment requires a separately running Switchyard | ||
| > Decision API service. | ||
|
|
||
| The `nemo-relay-switchyard` plugin uses the | ||
| [Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) Decision API to select | ||
| backends for supported LLM requests. Relay validates each decision, translates | ||
| requests and responses in process through Switchyard's | ||
| `switchyard-translation` library, and dispatches directly to the selected | ||
| Relay-owned target. Relay also owns provider credentials, retries, trusted | ||
| fallbacks, and observability. The separately running Switchyard service owns | ||
| routing decisions. | ||
|
|
||
| ## Before You Start | ||
|
bbednarski9 marked this conversation as resolved.
Outdated
|
||
|
|
||
| The current integration has two runtime pieces: | ||
|
|
||
| 1. A Relay build with the optional `switchyard` feature. | ||
| 2. A Switchyard service built from the pinned | ||
| [`topic/nemo-relay-integration` revision](https://github.com/NVIDIA-NeMo/Switchyard/tree/topic/nemo-relay-integration). | ||
|
|
||
| The current Relay examples pin Switchyard commit | ||
| [`8f9db9a6`](https://github.com/NVIDIA-NeMo/Switchyard/commit/8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8). | ||
| Treat that revision as part of this experimental compatibility boundary; update | ||
| it deliberately when testing a different Switchyard topic commit. | ||
|
|
||
| The plugin does not start or supervise the Switchyard service. Relay does not | ||
| require the service to be healthy at startup; unavailable Decision API calls | ||
| fail open to the configured same-protocol fallback. Profiles that depend on | ||
| ATOF history, including StageRouter, must use | ||
| `context_mode = "atof_required"` and name an enabled Relay ATOF HTTP endpoint | ||
| with `atof_endpoint_name`. Relay fails startup validation when that endpoint is | ||
| missing or invalid. Relay pushes ATOF events to the endpoint; local ATOF JSONL | ||
| output alone does not populate Switchyard's accumulator. | ||
|
|
||
| Build the opt-in CLI feature with: | ||
|
|
||
| ```bash | ||
| cargo build -p nemo-relay-cli --features switchyard | ||
| ``` | ||
|
|
||
| The default CLI build does not include the plugin or the Switchyard translation | ||
| library. | ||
|
|
||
| ## Capabilities | ||
|
|
||
| - Enforce or observe-only routing decisions. | ||
| - OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages inbound | ||
| profiles. | ||
| - Buffered and streaming protocol translation through Switchyard's | ||
| `switchyard-translation` library. | ||
| - Exact Relay-owned backend bindings and per-protocol trusted fallbacks. | ||
| - Bounded retries before the first streaming item. | ||
| - Canonical routing marks and shared LLM optimization accounting. | ||
|
|
||
| ## Pages | ||
|
|
||
| - [Switchyard Configuration](/configure-plugins/switchyard/configuration) | ||
| is the complete option and deployment reference. Review its | ||
| [experimental limitations](/configure-plugins/switchyard/configuration#experimental-limitations) | ||
| before adopting the integration. | ||
| - [Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/main/examples/switchyard) | ||
| documents the pinned service worktree and manual compatibility smoke tests. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.