Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions cmd/waired-agent/inference.go
Original file line number Diff line number Diff line change
Expand Up @@ -962,11 +962,12 @@ func startInferenceSubsystem(ctx context.Context, wg *sync.WaitGroup, logger *sl
// overlay set it is mesh-capable — the intercept is a LOCAL surface
// (loopback from Claude Code on this device), so a remote dispatch
// here is one hop and the receiving peer's overlay stays local-only.
// The directiveSelector applies the operator's per-class node policy
// (main / sub → local | pinned peer) per request; ClassifyModel
// derives the class from the managed-settings subagent label; the
// resolver maps unresolvable Anthropic ids to the class target
// node's model (#600 extended per-class).
// The directiveSelector picks WHICH Waired node serves each request —
// this device or a mesh peer, following the operator's `waired worker`
// preference unless the /model id names a node; classifyClaudeClass
// derives the class from the X-Claude-Code-Agent-Id header
// (waired-agent#1186); the resolver maps unresolvable Anthropic ids to
// the serving node's model (#600).
claudeDeps := baseGatewayDeps()
claudeDeps.Selector = &directiveSelector{p: provider}
// AllowOpenAI stays false: the intercept surface speaks Anthropic
Expand All @@ -983,8 +984,9 @@ func startInferenceSubsystem(ctx context.Context, wg *sync.WaitGroup, logger *sl
// directives under the same flag.
claudeDeps.ClaudeModelDirectives = cfg.ClaudeModelRouteDirectives
// #757: bound the pre-first-byte window on a PEER leg per traffic class so a
// stalled-but-reachable serving peer reroutes (auto mode only — see the
// intercept's X-Waired-Fallback-Allowed gate) instead of hanging the turn.
// stalled-but-reachable serving peer ends the turn with a 4xx that names it
// instead of hanging the turn (every peer leg, the pinned one included — the
// auto-only gate went with the auto route, waired-agent#1184).
// Subagents get the tighter budget; 0 disables. The gateway arms this only
// for remote:* selections, so a locally-served turn is never affected.
claudeDeps.TTFBBudget = func(class string) time.Duration {
Expand Down Expand Up @@ -3154,8 +3156,8 @@ func (p *agentInferenceProvider) EngineReady() (bool, string) {
// ActiveSelection. It backs the Claude-intercept model mapping (#600):
// unlike EngineReady it does NOT gate on ready/parked state — a mid-pull
// or loading model must still resolve so the router can answer with the
// precise ErrModelNotReady (503 + Retry-After, which auto mode falls back
// on) rather than a blanket "no local model". Only a missing selection
// precise ErrModelNotReady (503 + Retry-After, which the client waits
// out) rather than a blanket "no local model". Only a missing selection
// reports false.
//
// The setup report reads it too, through setupActiveModelID: on a host
Expand Down
26 changes: 15 additions & 11 deletions internal/agentconfig/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -172,14 +172,16 @@ type InferenceConfig struct {
// ClaudeTTFBBudgetMainMs / ClaudeTTFBBudgetSubMs bound the pre-first-byte
// window (milliseconds) for a MAIN / SUBAGENT Claude request routed to a
// mesh PEER (#757). If the peer returns no response headers within the
// budget the leg is aborted BEFORE the response commits, so an auto-routed
// turn reroutes to the Anthropic API instead of hanging on a
// stalled-but-reachable peer; a pinned (route=waired) leg is never
// affected. These are generous infinite-hang BACKSTOPS, not snappy reroute
// thresholds: /healthz readiness does not imply the model is loaded, so a
// cold model load legitimately sits inside this window. Subagents get the
// tighter budget (a stalled subagent is cheap to reroute and reads to the
// user as a hang). 0 disables the deadline for that class.
// budget the leg is aborted BEFORE the response commits, so the person
// reads a 4xx that names the peer instead of watching a hang; nothing
// reroutes the turn to the Anthropic API
// (docs/decisions/20260903/0333-no-automatic-crossing-to-or-from-anthropic.md),
// and the pinned peer's leg is covered like any other. These are generous
// infinite-hang BACKSTOPS, not snappy thresholds: /healthz readiness does
// not imply the model is loaded, so a cold model load legitimately sits
// inside this window. With a ClaudePeerWaitCeiling*Ms set for the class
// the budget is a grace period rather than a deadline (below). 0
// disables the deadline for that class.
ClaudeTTFBBudgetMainMs int `json:"claude_ttfb_budget_main_ms"`
ClaudeTTFBBudgetSubMs int `json:"claude_ttfb_budget_sub_ms"`

Expand Down Expand Up @@ -224,9 +226,11 @@ type InferenceConfig struct {
// real turns for.
//
// The cost of the larger figure is that such a turn is silent for longer:
// a leg the intercept may reroute cannot be held open with an SSE
// keepalive (docs/decisions/20260821/2142), so nothing is written until
// the first byte either way.
// a peer leg is not held open with an SSE keepalive
// (docs/decisions/20260821/2142 — its non-2xx can be an over-window 400
// that a committed keepalive could no longer relay, see
// internal/gateway/anthropic.go), so nothing is written until the first
// byte either way.
//
// 0, or any value not longer than the main budget, leaves the flat
// deadline in place.
Expand Down
23 changes: 10 additions & 13 deletions internal/agentgrade/probe.go
Original file line number Diff line number Diff line change
Expand Up @@ -362,20 +362,18 @@ func (p Probe) one(ctx context.Context, model string, c Case, offered map[string
// upstreamError carries a non-2xx from the gateway with its body, so
// the caller can tell an engine that is down from an engine rejecting
// the model's output.
//
// It used to carry the X-Waired-Fallback marker as well, which told "the
// request left local routing entirely" apart from "the model answered
// badly" (waired-agent#29). That header went with the auto route
// (waired-agent#1184), and the local gateway this probe drives has no
// passthrough to leave through.
type upstreamError struct {
Status int
Body string
Fallback string
Status int
Body string
}

func (e *upstreamError) Error() string {
if e.Fallback != "" {
// Without this marker, "the model answered badly" and "the
// request left local routing entirely" look identical from
// here, and that ambiguity cost a week once (waired-agent#29).
return fmt.Sprintf("HTTP %d (X-Waired-Fallback: %s — the request did not stay local): %s",
e.Status, e.Fallback, truncate(e.Body))
}
return fmt.Sprintf("HTTP %d: %s", e.Status, truncate(e.Body))
}

Expand Down Expand Up @@ -421,9 +419,8 @@ func (p Probe) post(ctx context.Context, req gateway.AnthropicRequest) (gateway.
}
if httpResp.StatusCode < 200 || httpResp.StatusCode >= 300 {
return gateway.AnthropicResponse{}, &upstreamError{
Status: httpResp.StatusCode,
Body: string(raw),
Fallback: httpResp.Header.Get("X-Waired-Fallback"),
Status: httpResp.StatusCode,
Body: string(raw),
}
}

Expand Down
22 changes: 0 additions & 22 deletions internal/agentgrade/probe_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,28 +89,6 @@ func TestRun_engineDownIsNotAVerdict(t *testing.T) {
}
}

// A fail-open — the request escaping local routing to the real upstream
// — must be visible in the error, not silently graded. Without the
// marker, "the model answered badly" and "this answer came from
// somewhere else entirely" are indistinguishable (waired-agent#29).
func TestRun_failOpenMarkerSurfaces(t *testing.T) {
p := probeAgainst(t, func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("X-Waired-Fallback", "local_status_404")
w.WriteHeader(http.StatusBadGateway)
_, _ = w.Write([]byte(`{"error":"upstream"}`))
})
rep, err := p.Run(context.Background(), "subject")
if err != nil {
t.Fatalf("Run: %v", err)
}
if rep.Grade != GradeUnknown {
t.Fatalf("grade = %q, want %q", rep.Grade, GradeUnknown)
}
if !strings.Contains(rep.Results[0].Detail, "did not stay local") {
t.Errorf("detail must name the fail-open, got %q", rep.Results[0].Detail)
}
}

func TestRun_passingModel(t *testing.T) {
p := probeAgainst(t, func(w http.ResponseWriter, r *http.Request) {
var req struct {
Expand Down
3 changes: 1 addition & 2 deletions internal/e2e/integration/budget.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,7 @@ func engineDeadReason(body []byte) string {
}

// retryableStatus reports whether a bare status code is plausibly transient.
// Used both for a direct response and for the local status recovered from the
// fallback header (where no local body survives).
// Used for the status of a direct response.
func retryableStatus(status int) bool {
switch status {
case http.StatusTooEarly, // 425 — engine/model not ready yet
Expand Down
12 changes: 7 additions & 5 deletions internal/e2e/integration/harness.go
Original file line number Diff line number Diff line change
Expand Up @@ -192,11 +192,13 @@ func routeRecorded(leg Leg) bool {
// --- HTTP drives ---

// driveResponse is one drive attempt's response. It carries the HEADERS the
// old (status, body) pair threw away: on a Claude-leg fail-open the local
// error is discarded before the upstream replay, and X-Waired-Fallback is the
// only evidence of it left on the wire. Dropping that header is exactly why a
// dead model runner read as "waired proxy could not reach the upstream API"
// for a week (waired-agent#29).
// old (status, body) pair threw away: the X-Waired-* headers are where the
// gateway says what served the turn and why a leg failed. Dropping them is
// exactly why a dead model runner once read as "waired proxy could not reach
// the upstream API" for a week (waired-agent#29) — back then a Claude-leg
// fail-open replayed the turn upstream and X-Waired-Fallback was the only
// evidence left on the wire. That replay is retired (waired-agent#1184); the
// sentinel keeps proving it stays retired.
type driveResponse struct {
Status int
Header http.Header
Expand Down
4 changes: 2 additions & 2 deletions internal/e2e/integration/legs.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ func claudeRealAnthropicIDLeg() Leg {

// claudeUnresolvableIDLeg keeps the #600 mapping covered. The Anthropic ids
// Claude Code sends name no catalog model, so an alias miss must resolve to
// something servable instead of 404ing into the auto-fallback (the
// local_status_404 class).
// something servable instead of 404ing (once the local_status_404 fallback
// class, now a 404 the person reads).
//
// It exists because rewriting claudeRealAnthropicIDLeg against #1091 would
// otherwise take that coverage to ZERO: the tiny alias resolves in the
Expand Down
35 changes: 18 additions & 17 deletions internal/gateway/anthropic.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,9 @@ func writeAnthropicError(w http.ResponseWriter, status int, errType, message str
// becomes a 400. A 500 tells a well-behaved client "transient, try
// again" — Claude Code retried one 11 times over 182 s against a
// rejection that would have failed identically on attempt 12. Saying 400
// is both the accurate statement and the one that stops the storm; auto
// mode still reroutes, because the intercept's fallback window is any
// status >= 400.
// is both the accurate statement and the one that stops the storm: the
// person reads one error instead of eleven retries, and nothing reroutes
// the turn elsewhere (waired-agent#1184).
//
// Matched on the marker regardless of which status the engine picked:
// the classification is about why the request failed, not about the
Expand Down Expand Up @@ -214,16 +214,17 @@ func (h *HandlerSet) handleAnthropicMessagesImpl(w http.ResponseWriter, r *http.
)

// #623 context-window guard: reject a prompt that overruns the served
// model's effective window with the exact Anthropic 400 that triggers
// Claude Code's auto-compaction, instead of forwarding it to the engine
// (Ollama would silently truncate the prompt head — the root cause of
// local-model tool spam / instruction drift). Placed before the engine
// is looked up / started so an over-window request never loads a model.
// The staged HeaderLocalError marks this 400 as "surface, don't fall
// back" for the intercept's auto mode (a fallback to the real Anthropic
// API would abandon local serving instead of compacting). The guard is
// active only where Deps.ContextWindowFor is wired (the Claude-intercept
// HandlerSet); a 0 window means "unknown" and fails open.
// model's effective window with the 400 Claude Code compacts on — its
// message is the documented `capability_rejected: prompt_too_long`
// token (contextOverflowToken, waired-agent#1187) — instead of
// forwarding it to the engine (Ollama would silently truncate the
// prompt head — the root cause of local-model tool spam / instruction
// drift). Placed before the engine is looked up / started so an
// over-window request never loads a model. The staged HeaderLocalError
// names the reason for the journal and for a relaying waired node
// (relayPeerContextOverflow). The guard is active only where
// Deps.ContextWindowFor is wired (the Claude-intercept HandlerSet); a 0
// window means "unknown" and lets the request through.
//
// The window belongs to whoever ANSWERS. Deps.ContextWindowFor knows
// only this device — its manifests, its applied tuning — so on a mesh
Expand Down Expand Up @@ -793,7 +794,8 @@ func (h *HandlerSet) proxyAnthropicStream(ctx context.Context, client *http.Clie
// #757: bound only the PRE-first-byte window. reqCtx governs the peer
// request; the watch below cancels it when the leg may not go on
// waiting, so postToEngine errors BEFORE the stream commits and the
// intercept's auto fallback reroutes. The watch is disarmed the instant
// client reads a 4xx that names the peer instead of a half-written
// stream (nothing reroutes since waired-agent#1184). The watch is disarmed the instant
// postToEngine returns (headers received), so a slow-but-progressing
// completion is never cut mid-stream (mid-stream cancellation is #651).
reqCtx, cancel := context.WithCancel(ctx)
Expand Down Expand Up @@ -1770,9 +1772,8 @@ func stageContextOverflow(w http.ResponseWriter, promptTokens, window int) {
// which kind of 400 it sent, and re-emitting the canonical envelope is
// what makes a mesh leg behave like a local one (waired-agent#436).
//
// The staged header also marks it "surface, don't fall back" for the
// intercept's auto mode: falling back to the real Anthropic API here
// would abandon local serving for a turn that only needed compacting.
// The staged header also carries the reason to the journal, the same way
// a local over-window 400 does (stageContextOverflow).
func relayPeerContextOverflow(w http.ResponseWriter, resp *http.Response, body []byte, rr *requestRec) bool {
if resp.StatusCode != http.StatusBadRequest ||
resp.Header.Get(HeaderLocalError) != LocalErrorContextOverflow {
Expand Down
14 changes: 8 additions & 6 deletions internal/gateway/anthropic_models.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,14 @@ import (
)

// anthropicModel is the Anthropic Models API object, extended with
// max_input_tokens — the field Claude Code's gateway model discovery
// (CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1) reads to size its
// auto-compaction threshold (#623). We advertise the effective LOCAL
// window (min native / host-sustainable, from Deps.ContextWindowFor) so
// Claude Code compacts before it overruns the model and Ollama truncates
// the prompt head. Omitted (0) when the window is unknown.
// max_input_tokens (#623) — the field Claude Code's gateway model
// discovery reads when that discovery is switched on. waired no longer
// switches it on (the /model rows come from the modelPicker setting,
// waired-agent#1185), so for Claude Code the window travels in
// CLAUDE_CODE_MAX_CONTEXT_TOKENS and the over-window 400 does the
// guarding; the field still advertises the effective LOCAL window (min
// native / host-sustainable, from Deps.ContextWindowFor) to whatever
// client lists models here. Omitted (0) when the window is unknown.
type anthropicModel struct {
Type string `json:"type"`
ID string `json:"id"`
Expand Down
2 changes: 1 addition & 1 deletion internal/gateway/peerwait.go
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ func classifyPeerWork(res router.ProbeResult, engineLive, engineKnown bool) peer
// a replayed nonce, a missing identity (see router.ProbeOutcome).
// That is this device failing to ask, not the peer failing to work,
// and calling it "the peer vanished" would name the wrong machine in
// the reroute notice.
// the error the person reads.
return peerUnknowable
default:
return peerSilent
Expand Down
Loading
Loading