Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions crates/switchyard/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This crate is intentionally not published to crates.io yet. Build it from the
NeMo Relay source checkout with the optional CLI feature while the Switchyard
Decision API contract and service/library boundary are still evolving.

## Why use it?
## Why Use It?
Comment thread
bbednarski9 marked this conversation as resolved.
Outdated

- **Route through Switchyard decisions**: Select an exact Relay-owned target
using a versioned Decision API contract.
Expand All @@ -31,7 +31,7 @@ Decision API contract and service/library boundary are still evolving.
- **Support staged rollout**: Run in enforce or observe-only mode with
explicit target bindings and protocol defaults.

## What you get
## What You Get
Comment thread
bbednarski9 marked this conversation as resolved.
Outdated

- `SwitchyardConfig`: the typed plugin configuration contract.
- `SwitchyardRuntime`: buffered and streaming routing intercepts.
Expand All @@ -43,7 +43,7 @@ Decision API contract and service/library boundary are still evolving.
- Switchyard-owned protocol translation through the pinned
`switchyard-translation` dependency.

## Source build
## Source Build

The crate is a private workspace member and must be built from source:

Expand All @@ -56,7 +56,7 @@ The resulting CLI includes the Switchyard component only when the `switchyard`
feature is enabled. A default Relay build does not include this experimental
integration.

## Runtime boundary
## Runtime Boundary

The current integration calls Switchyard's HTTP Decision API at runtime. Relay
does not start or supervise the Switchyard service. For ATOF-backed profiles,
Expand All @@ -70,10 +70,10 @@ the pinned topic-branch commit, local configuration, compatibility smoke test,
and trajectory workflow.

Translation is already in-process through Switchyard's Rust translation
library. A future in-process DecisionProvider may replace the HTTP Decision API
library. A future in-process DecisionProvider can replace the HTTP Decision API
call without changing the Relay-owned dispatch and observability boundary.

## Configuration and registration
## Configuration and Registration

The CLI registers the component when built with `--features switchyard` and
accepts a `[[components]]` entry with `kind = "switchyard"`. A minimal
Expand All @@ -98,9 +98,11 @@ anthropic_messages = "my-anthropic-target"
```

For ATOF-backed profiles, configure an enabled Relay ATOF HTTP exporter that
targets the Switchyard `/v1/atof/events` endpoint and use environment-referenced
authentication headers. Keep provider and Decision API credentials outside
tracked configuration files.
has a unique `name`, targets the Switchyard ingestion URL, and uses
environment-referenced authentication headers. Set `atof_endpoint_name` in the
Switchyard component to that name. Local ATOF JSONL output alone does not
populate the Switchyard accumulator. Keep provider and Decision API credentials
outside tracked configuration files.

## Documentation

Expand Down
3 changes: 3 additions & 0 deletions docs/configure-plugins/about.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ entries in `plugins.toml`.
Guardrails-backed policy checks.
- [PII Redaction](/configure-plugins/pii-redaction/about) sanitizes sensitive
data in observability payloads.
- [Switchyard (Experimental)](/configure-plugins/switchyard/about) routes
requests using decisions from a separately running Switchyard Decision API
service.
- [Model Pricing](/configure-plugins/model-pricing) configures catalog sources
for cost estimates on managed LLM responses.

Expand Down
9 changes: 9 additions & 0 deletions docs/configure-plugins/observability/atof.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ filename = "events.jsonl"
mode = "overwrite"

[[components.config.atof.endpoints]]
name = "archive"
Comment thread
coderabbitai[bot] marked this conversation as resolved.
url = "http://localhost:8080/events"
transport = "http_post"
timeout_millis = 3000
Expand Down Expand Up @@ -69,12 +70,18 @@ The following table describes each streaming endpoint:

| Field | Default | Notes |
|---|---|---|
| `name` | None | Optional unique name that another component can use to reference this endpoint. Names must be non-empty and cannot have leading or trailing whitespace. |
| `url` | Required | Endpoint URL. |
| `transport` | `http_post` | `http_post`, `websocket`, or `ndjson`. |
| `headers` | `{}` | String-to-string headers for requests or handshakes. |
| `header_env` | `{}` | Map of header names to environment-variable names. Referenced values must be present and non-blank. |
| `timeout_millis` | `3000` | Per-endpoint timeout. Must be greater than `0`. |
| `field_name_policy` | `preserve` | Field-name handling before Relay sends endpoint events. Accepted values are `preserve` and `replace_dots`. |

Do not define the same header in both `headers` and `header_env`. Use
`header_env` for secrets so configuration stores the environment-variable name
instead of the secret value.

`preserve` sends canonical ATOF field names unchanged. `replace_dots` replaces
dots in JSON object keys with underscores recursively. If replacement produces
a collision, Relay keeps both values and deterministically appends `_2`, `_3`,
Expand Down Expand Up @@ -226,9 +233,11 @@ let component = ComponentSpec::new(ObservabilityConfig {
filename: Some("events.jsonl".into()),
mode: "overwrite".into(),
endpoints: vec![AtofEndpointSectionConfig {
name: Some("archive".into()),
url: "http://localhost:8080/events".into(),
transport: "http_post".into(),
headers: Default::default(),
header_env: Default::default(),
timeout_millis: 3000,
field_name_policy: "replace_dots".into(),
}],
Expand Down
3 changes: 3 additions & 0 deletions docs/configure-plugins/observability/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ filename = "events.jsonl"
mode = "overwrite"

[[components.config.atof.endpoints]]
name = "archive"
url = "http://localhost:8080/events"
transport = "http_post"
timeout_millis = 3000
Expand Down Expand Up @@ -320,9 +321,11 @@ let component = ComponentSpec::new(ObservabilityConfig {
filename: Some("events.jsonl".into()),
mode: "overwrite".into(),
endpoints: vec![AtofEndpointSectionConfig {
name: Some("archive".into()),
url: "http://localhost:8080/events".into(),
transport: "http_post".into(),
headers: [("authorization".into(), "Bearer <token>".into())].into(),
header_env: Default::default(),
timeout_millis: 3000,
field_name_policy: "preserve".into(),
}],
Expand Down
73 changes: 73 additions & 0 deletions docs/configure-plugins/switchyard/about.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
title: "Switchyard (Experimental)"
sidebar-title: "Switchyard (Experimental)"
description: "Route Relay LLM requests through the experimental Switchyard Decision API integration."
position: 5
---
{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0 */}

> **Experimental:** The Switchyard integration is an early-access feature. It
> is not enabled in default Relay builds, its configuration and contracts can
> change, and the current deployment requires a separately running Switchyard
> Decision API service.

The `nemo-relay-switchyard` plugin uses the
[Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) Decision API to select
backends for supported LLM requests. Relay validates each decision, translates
requests and responses in process through Switchyard's
`switchyard-translation` library, and dispatches directly to the selected
Relay-owned target. Relay also owns provider credentials, retries, trusted
fallbacks, and observability. The separately running Switchyard service owns
routing decisions.

## Before You Start
Comment thread
bbednarski9 marked this conversation as resolved.
Outdated

The current integration has two runtime pieces:

1. A Relay build with the optional `switchyard` feature.
2. A Switchyard service built from the pinned
[`topic/nemo-relay-integration` revision](https://github.com/NVIDIA-NeMo/Switchyard/tree/topic/nemo-relay-integration).

The current Relay examples pin Switchyard commit
[`8f9db9a6`](https://github.com/NVIDIA-NeMo/Switchyard/commit/8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8).
Treat that revision as part of this experimental compatibility boundary; update
it deliberately when testing a different Switchyard topic commit.

The plugin does not start or supervise the Switchyard service. Relay does not
require the service to be healthy at startup; unavailable Decision API calls
fail open to the configured same-protocol fallback. Profiles that depend on
ATOF history, including StageRouter, must use
`context_mode = "atof_required"` and name an enabled Relay ATOF HTTP endpoint
with `atof_endpoint_name`. Relay fails startup validation when that endpoint is
missing or invalid. Relay pushes ATOF events to the endpoint; local ATOF JSONL
output alone does not populate Switchyard's accumulator.

Build the opt-in CLI feature with:

```bash
cargo build -p nemo-relay-cli --features switchyard
```

The default CLI build does not include the plugin or the Switchyard translation
library.

## Capabilities

- Enforce or observe-only routing decisions.
- OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages inbound
profiles.
- Buffered and streaming protocol translation through Switchyard's
`switchyard-translation` library.
- Exact Relay-owned backend bindings and per-protocol trusted fallbacks.
- Bounded retries before the first streaming item.
- Canonical routing marks and shared LLM optimization accounting.

## Pages

- [Switchyard Configuration](/configure-plugins/switchyard/configuration)
is the complete option and deployment reference. Review its
[experimental limitations](/configure-plugins/switchyard/configuration#experimental-limitations)
before adopting the integration.
- [Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/main/examples/switchyard)
documents the pinned service worktree and manual compatibility smoke tests.
Loading