Skip to content

docs: document experimental Switchyard integration #400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 2 commits into from
Jul 15, 2026
Merged

docs: document experimental Switchyard integration #400

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9b07e6f
docs: document experimental Switchyard integration
bbednarski9 Jul 14, 2026
779054b
docs: address Switchyard review feedback
bbednarski9 Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 19 additions & 13 deletions crates/switchyard/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Install it from crates.io, or build it from the NeMo Relay source checkout with
the optional CLI feature while the Switchyard Decision API contract and
service/library boundary are still evolving.

## Why use it?
Use the plugin to:

- **Route through Switchyard decisions**: Select an exact Relay-owned target
using a versioned Decision API contract.
Expand All @@ -31,19 +31,21 @@ service/library boundary are still evolving.
- **Support staged rollout**: Run in enforce or observe-only mode with
explicit target bindings and protocol defaults.

## What you get
## Implementation and Runtime Behavior

- `SwitchyardConfig`: the typed plugin configuration contract.
- `SwitchyardRuntime`: buffered and streaming routing intercepts.
The plugin includes the following implementation and runtime behavior:

- `SwitchyardConfig`: The typed plugin configuration contract.
- `SwitchyardRuntime`: Buffered and streaming routing intercepts.
- Decision and target validation for exact backend, model, protocol, and
endpoint bindings.
- ATOF-backed or payload-only routing context modes.
- Routing marks and model-routing optimization contributions for Relay's
cumulative accounting pipeline.
- Switchyard-owned protocol translation through the crates.io
- Switchyard-owned protocol translation through the
`switchyard-translation` dependency.

## Installation and source build
## Installation and Source Build

Add the crate from crates.io:

Expand All @@ -62,7 +64,7 @@ The resulting CLI includes the Switchyard component only when the `switchyard`
feature is enabled. A default Relay build does not include this experimental
integration.

## Runtime boundary
## Runtime Boundary

The current integration calls Switchyard's HTTP Decision API at runtime. Relay
does not start or supervise the Switchyard service. For ATOF-backed profiles,
Expand All @@ -78,10 +80,10 @@ the pinned topic-branch commit, local configuration, compatibility smoke test,
and trajectory workflow.

Translation is already in-process through Switchyard's Rust translation
library. A future in-process DecisionProvider may replace the HTTP Decision API
library. A future in-process `DecisionProvider` can replace the HTTP Decision API
call without changing the Relay-owned dispatch and observability boundary.

## Configuration and registration
## Configuration and Registration

The CLI registers the component when built with `--features switchyard` and
accepts a `[[components]]` entry with `kind = "switchyard"`. A minimal
Expand All @@ -105,13 +107,17 @@ openai_responses = "my-responses-target"
anthropic_messages = "my-anthropic-target"
```

For ATOF-backed profiles, configure an enabled Relay ATOF HTTP exporter that
targets the Switchyard `/v1/atof/events` endpoint and use environment-referenced
authentication headers. Keep provider and Decision API credentials outside
tracked configuration files.
For ATOF-backed profiles, configure an enabled Relay ATOF HTTP stream sink that
has a unique `name`, targets the Switchyard ingestion URL, and uses
environment-referenced authentication headers. Set `atof_endpoint_name` in the
Switchyard component to that name. Local ATOF JSONL output alone does not
populate the Switchyard accumulator. Keep provider and Decision API credentials
outside tracked configuration files.

## Documentation

For more information, refer to the following resources:

- [Switchyard integration examples](../../examples/switchyard/README.md)
- [NeMo Relay documentation](https://docs.nvidia.com/nemo/relay)
- [Switchyard repository](https://github.com/NVIDIA-NeMo/Switchyard)
3 changes: 3 additions & 0 deletions docs/configure-plugins/about.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ entries in `plugins.toml`.
Guardrails-backed policy checks.
- [PII Redaction](/configure-plugins/pii-redaction/about) sanitizes sensitive
data in observability payloads.
- [Switchyard (Experimental)](/configure-plugins/switchyard/about) routes
requests using decisions from a separately running Switchyard Decision API
service.
- [Model Pricing](/configure-plugins/model-pricing) configures catalog sources
for cost estimates on managed LLM responses.

Expand Down
120 changes: 78 additions & 42 deletions docs/configure-plugins/observability/atof.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,58 +28,78 @@ kind = "observability"
enabled = true

[components.config]
version = 1
version = 2

[components.config.atof]
enabled = true

[[components.config.atof.sinks]]
type = "file"
output_directory = "logs"
filename = "events.jsonl"
mode = "overwrite"

[[components.config.atof.endpoints]]
[[components.config.atof.sinks]]
type = "stream"
name = "archive"
Comment thread
coderabbitai[bot] marked this conversation as resolved.
url = "http://localhost:8080/events"
transport = "http_post"
timeout_millis = 3000
field_name_policy = "replace_dots"
```

This configuration registers the plugin-managed ATOF exporter and writes one
JSON object per lifecycle event to `logs/events.jsonl`. It also sends each raw
ATOF event to the configured endpoint.
This configuration registers the plugin-managed ATOF exporter, writes one JSON
object per lifecycle event to `logs/events.jsonl`, and sends each raw event to
the configured stream sink.

## Fields

The following table describes the top-level ATOF settings:

| Field | Default | Notes |
|---|---|---|
| `enabled` | `false` | Must be `true` to write events. |
| `enabled` | `false` | Must be `true` to export events. |
| `sinks` | `[]` | File and stream destinations. An enabled ATOF section requires at least one sink. |

## File Sinks

The following table describes each file sink:

| Field | Default | Notes |
|---|---|---|
| `type` | Required | Must be `file`. |
| `output_directory` | Current working directory | Directory containing the JSONL file. |
| `filename` | Timestamped `nemo-relay-events-*.jsonl` | Explicit output filename. |
| `filename` | Timestamped `nemo-relay-events-*.jsonl` | Output filename. |
| `mode` | `append` | `append` or `overwrite`. |
| `endpoints` | `[]` | Optional streaming destinations. File output remains active when endpoints are configured. |

## Streaming Endpoints
## Streaming Sinks

Each endpoint receives the same raw ATOF JSON object that the file exporter
writes as one JSONL line. Endpoints are independent: a failed endpoint is
skipped or retried without blocking file output or other endpoints.
Each stream sink receives the same raw ATOF JSON object that file sinks write as
one JSONL line. Sinks are independent: a failed stream sink is skipped or
retried without blocking file output or other stream sinks.

The following table describes each streaming endpoint:
The following table describes each stream sink:

| Field | Default | Notes |
|---|---|---|
| `type` | Required | Must be `stream`. |
| `name` | None | Optional unique name that another component can use to reference this stream sink. Names must be non-empty and cannot have leading or trailing whitespace. |
| `url` | Required | Endpoint URL. |
| `transport` | `http_post` | `http_post`, `websocket`, or `ndjson`. |
| `headers` | `{}` | String-to-string headers for requests or handshakes. |
| `timeout_millis` | `3000` | Per-endpoint timeout. Must be greater than `0`. |
| `field_name_policy` | `preserve` | Field-name handling before Relay sends endpoint events. Accepted values are `preserve` and `replace_dots`. |
| `header_env` | `{}` | Map of header names to environment-variable names. Referenced values must be present and non-blank. |
| `timeout_millis` | `3000` | Per-sink timeout. Must be greater than `0`. |
| `field_name_policy` | `preserve` | Field-name handling before Relay sends stream sink events. Accepted values are `preserve` and `replace_dots`. |

Do not define the same header in both `headers` and `header_env`. Use
`header_env` for secrets so configuration stores the environment-variable name
instead of the secret value.

`preserve` sends canonical ATOF field names unchanged. `replace_dots` replaces
dots in JSON object keys with underscores recursively. If replacement produces
a collision, Relay keeps both values and deterministically appends `_2`, `_3`,
and later suffixes as needed. Validation rejects any other
`endpoints[i].field_name_policy` value.
`sinks[i].field_name_policy` value.

- `http_post` sends each event as one JSONL record in an HTTP `POST` request
with `Content-Type: application/x-ndjson`. Any `2xx` response is treated as
Expand All @@ -89,7 +109,7 @@ and later suffixes as needed. Validation rejects any other
- `ndjson` opens one long-lived HTTP upload and writes each event as one
newline-delimited JSON record.

`force_flush()` flushes file output and drains queued endpoint events without
`force_flush()` flushes file output and drains queued stream sink events without
closing streaming connections. `shutdown()` is terminal: it flushes pending
work, closes streaming connections, and makes later events no-op.

Expand Down Expand Up @@ -131,7 +151,8 @@ import asyncio
from nemo_relay import plugin
from nemo_relay.observability import (
AtofConfig,
AtofEndpointConfig,
AtofFileSinkConfig,
AtofStreamSinkConfig,
ComponentSpec,
ObservabilityConfig,
)
Expand All @@ -142,11 +163,13 @@ config = plugin.PluginConfig(
ObservabilityConfig(
atof=AtofConfig(
enabled=True,
output_directory="logs",
filename="events.jsonl",
mode="overwrite",
endpoints=[
AtofEndpointConfig(
sinks=[
AtofFileSinkConfig(
output_directory="logs",
filename="events.jsonl",
mode="overwrite",
),
AtofStreamSinkConfig(
url="http://localhost:8080/events",
transport="http_post",
)
Expand Down Expand Up @@ -184,14 +207,18 @@ void (async () => {
version: 1,
components: [
observability.ComponentSpec({
version: 1,
version: 2,
atof: observability.atofConfig({
enabled: true,
output_directory: "logs",
filename: "events.jsonl",
mode: "overwrite",
endpoints: [
sinks: [
{
type: "file",
output_directory: "logs",
filename: "events.jsonl",
mode: "overwrite",
},
{
type: "stream",
url: "http://localhost:8080/events",
transport: "http_post",
},
Expand Down Expand Up @@ -219,7 +246,8 @@ void (async () => {
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
use nemo_relay::observability::plugin_component::{
AtofEndpointSectionConfig, AtofSectionConfig, ComponentSpec, ObservabilityConfig,
AtofFileSinkSectionConfig, AtofSectionConfig, AtofSinkSectionConfig,
AtofStreamSinkSectionConfig, ComponentSpec, ObservabilityConfig,
};
use nemo_relay::plugin::{
clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig,
Expand All @@ -228,16 +256,22 @@ use nemo_relay::plugin::{
let component = ComponentSpec::new(ObservabilityConfig {
atof: Some(AtofSectionConfig {
enabled: true,
output_directory: Some("logs".into()),
filename: Some("events.jsonl".into()),
mode: "overwrite".into(),
endpoints: vec![AtofEndpointSectionConfig {
url: "http://localhost:8080/events".into(),
transport: "http_post".into(),
headers: Default::default(),
timeout_millis: 3000,
field_name_policy: "replace_dots".into(),
}],
sinks: vec![
AtofSinkSectionConfig::File(AtofFileSinkSectionConfig {
output_directory: Some("logs".into()),
filename: Some("events.jsonl".into()),
mode: "overwrite".into(),
}),
AtofSinkSectionConfig::Stream(AtofStreamSinkSectionConfig {
url: "http://localhost:8080/events".into(),
transport: "http_post".into(),
headers: Default::default(),
header_env: Default::default(),
timeout_millis: 3000,
field_name_policy: "replace_dots".into(),
name: Some("archive".into()),
}),
],
}),
..ObservabilityConfig::default()
});
Expand Down Expand Up @@ -351,10 +385,12 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {

## Common Configuration and Runtime Issues

- ATOF is enabled without at least one plugin-configured sink.
- `mode` is not `append` or `overwrite`.
- `endpoints[i].url` is empty, `endpoints[i].transport` is not supported, or
`endpoints[i].timeout_millis` is `0`.
- A stream sink URL is empty, its transport is not supported, or its timeout is
`0`.
- The output directory is not writable at runtime.
- `nemo-relay doctor` cannot deliver its synthetic ATOF mark probe to a
configured endpoint.
- ATOF is enabled in a target that cannot access the native filesystem.
- A file sink is configured in a target that cannot access the native
filesystem.
Loading
Loading