Skip to content
Merged
4 changes: 3 additions & 1 deletion crates/aisix-obs/src/usage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,9 @@ pub struct UsageEvent {
/// it can use —
///
/// - non-streaming: the complete response is written;
/// - streaming: the first token is forwarded downstream.
/// - streaming: the first token is forwarded downstream — for a
/// passthrough route, which relays an opaque byte stream and has no
/// token to recognise, the first relayed frame handed to the client.
///
/// Request-scoped (unlike the two `upstream_*` fields above), so it
/// spans request parsing, guardrail scans, every failed attempt,
Expand Down
23 changes: 23 additions & 0 deletions crates/aisix-proxy/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,29 @@ writes the legacy `aisix_requests_total` **and** the detailed `aisix_proxy_*` /
directly silently produces a request that exists in one family and not the
others — the bug AISIX-Cloud#1234 fixed across ten endpoints.

## A passthrough route that detects an envelope must observe what the typed endpoint does

`passthrough_route.rs` relays bytes verbatim, but it *detects* the request
envelope and extracts from it. Whatever it extracts is the whole observation —
there is no bridge behind it to fill anything in. So a new observation added to
a typed endpoint has a second home: a new `UsageEvent` token dimension needs
reading in `usage_of`, a new guardrail scan input needs adding to
`message_scan_text` / `request_guardrail_text`, a new attribution field needs
setting on `RouteTelemetry`. Miss it and the route keeps answering 200 while
metering and enforcement silently weaken — the shape of #988, where the cache
and reasoning counters existed everywhere except here.

Two rules bound the extraction:

- An **opaque** (`Raw`) body is never mined for meaning. Buffered opaque
responses are not probed for usage at all, and an opaque stream's flat token
fields count only on a frame the server itself labelled one
(`event: token_usage`) — a caller-shaped body must not be able to mint tokens.
- Usage accumulates **field-wise max** across frames, never last-wins: one
stream reports it in pieces (Anthropic's `message_start` carries the input
and cache counters, its `message_delta` only the output ones), so an
assignment truncates whatever arrived earlier.

## A new proxy route has to be declared in three places

Adding a `.route(…)` in `build_router` is not enough, and nothing fails loudly
Expand Down
14 changes: 13 additions & 1 deletion crates/aisix-proxy/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,12 @@ fn inbound_protocol_for_endpoint(endpoint: &str) -> &'static str {
"a2a"
} else if endpoint == "/v1/realtime" {
"realtime"
} else if endpoint == "/passthrough_route" {
// Whatever API a route relays, it is not the gateway's own OpenAI
// surface — and the usage event already tags these rows
// `passthrough`, so labelling the metric `openai` put the two
// halves of one request on different protocols.
"passthrough"
} else {
"openai"
}
Expand Down Expand Up @@ -2415,7 +2421,13 @@ mod tests {
"the caller's path must not reach the label: {sample}"
);
assert!(sample.contains(r#"outcome="completed""#), "{sample}");
assert!(sample.contains(r#"inbound_protocol="openai""#), "{sample}");
// A relayed API is not the gateway's own OpenAI surface, and the
// usage event already tags these rows `passthrough` — the two must
// not put one request on different protocols.
assert!(
sample.contains(r#"inbound_protocol="passthrough""#),
"{sample}"
);
}

/// The chunked path reaches the handler, which rejects at its body
Expand Down
Loading