feat(fallback): route-aware outage fallback prefers current route fallbacks - #1
Closed
Soju06 wants to merge 3950 commits into
Closed
feat(fallback): route-aware outage fallback prefers current route fallbacks#1Soju06 wants to merge 3950 commits into
Soju06 wants to merge 3950 commits into
Conversation
Rebase fold on the salvaged gate:
- is_trivial_prompt/TRIVIAL_PROMPT_RE move to agent/memory_provider (the
ABC both the core gate and providers already import) — one source of
truth; honcho's _TRIVIAL_PROMPT_RE now aliases it, turn_context and the
queue_prefetch_all warm path (a sibling site main grew after the PR's
base) both use it
- tests: gate tests at the prefetch call site (mutation-checked), shared
classifier tests incl. prefix-collision guards (k8s/yolo/note/supper),
and honcho dialectic-machinery tests re-driven with a substantive prompt
("hello" became trivial by design — those tests exercise thread cadence,
not the classifier)
…assifier Simplify-pass finding: sharing only the REGEX left the wrapper logic (empty/strip/slash checks) duplicated, half-defeating the no-drift goal. The classmethod now calls agent/memory_provider.is_trivial_prompt directly; _TRIVIAL_PROMPT_RE stays as a class attr for backward compatibility with any external referents.
…it frame scheduleDeltaFlush's adaptive floor is driven by lastFlushCostRef, but runFlush only timed flushQueuedDeltas(), the synchronous store write. While a session streams, syncSessionStateToView defers the $messages publish (React commit + Streamdown re-parse) to its own rAF, so the measured cost stayed near zero and the floor collapsed to the fixed 33ms path no matter how expensive the real commit was. runFlush now records the write cost as a fallback, then extends the measurement through a rAF registered after the view-sync one: it runs in the same frame right after the deferred commit, and the rAF timestamp marks frame start so only in-frame work is counted, not the vsync wait. A stale callback from before a newer flush is ignored, and a hidden renderer that never fires rAF keeps the write-cost fallback.
All three desktop reconnect loops (primary gateway boot, secondary multi-profile gateway pool, plugin event socket) used bare exponential backoff with no jitter. After a gateway restart every disconnected client redials on the exact same schedule, so the reconnect attempts land in lockstep instead of spreading out -- a burst that can starve the gateway's file descriptors while it's still coming back up. Add reconnect-backoff.ts implementing AWS-style full-jitter backoff (random delay in [0, min(cap, base * 2^attempt))) and wire it into all three call sites in place of their local Math.min/2**attempt math. Manual reconnect paths already reset the attempt counter and bypass the timer entirely -- unchanged.
… count With the full-jitter backoff (300ms base) six attempts can elapse in ~9s, so the old RECONNECT_ESCALATE_AFTER=6 attempt threshold raised the recoverable boot error during a brief post-boot blip — breaking the 'a remote that drops post-boot keeps looping with NO boot.error' contract. Escalate after RECONNECT_ESCALATE_AFTER_MS (45s, matching the old deterministic 1->15s ladder's calibration) elapsed since the first failed reconnect of the episode. Reset on clean open, manual/wake reconnect, and soft switch, preserving the reset-on-success path.
…ousResearch#77665) Needed for the NousResearch#62082 curator toolset-pin salvage attribution.
…ended it early
The comment above .btn-arc contained 'bg-*/', whose */ terminated the comment mid-sentence, leaving 'text-* variant utilities. */ .btn-arc {' as an invalid prelude — CSS error recovery can drop the whole .btn-arc rule. Reword so no */ appears inside the comment.
Extracted from NousResearch#59352 by @rerdi92 (the rest of that PR — a month-stale icons.ts rewrite and a chunk-size warning-ceiling bump — is superseded/masking and was not salvaged).
… props, gated adapter re-sync Re-derive of PR NousResearch#38470 on today's main (its target file desktop-controller.tsx no longer exists after the contrib/ refactor; the three surviving ideas are applied at their new homes): - incremental-external-store-runtime: the dep-less setAdapter effect ran every render; gate on [runtime, store] — behavior-preserving because __internal_setAdapter early-exits on identical store. - ChatView is now memo()d, and session-tile hoists its inline arrow props to useCallbacks/module constants so the memo actually holds. - Render-count regression test (mocked Thread) proves an unrelated parent re-render no longer re-renders the chat shell. Credit: idea and original implementation by @hdd69 in NousResearch#38470.
The curator LLM review loop (_run_llm_review) built its AIAgent without enabled_toolsets, so it advertised the full default catalog (~30 tools plus the context_engine lcm_* family) on every call. The fork uses only four tools, fixed by its own system prompt, with no dispatch path to the rest, so ~26 tool schemas shipped on every request as dead weight: ~7K input tokens per call on a loop that makes 50-100 calls per consolidation pass. Restrict the fork to enabled_toolsets=["skills", "terminal"], the same tools the prompt already names. Behavior-neutral: the prompt held the model to these tools and nothing routed calls to the others. Mirrors the background_review fork (background_review.py:788-794). Call-site only; AIAgent already forwards the kwarg. Adds test_review_fork_restricts_toolsets_to_skills_and_terminal (captures the constructor kwarg) and test_review_fork_toolset_surface_is_skills_plus_terminal (pins the resolved surface).
…pened
Switching sessions while a turn streams (or right as it completes) could
leave the assistant reply missing until restart. Resume merges stored
history with the gateway's `inflight` projection, whose assistant row is
text-only and often an empty `assistant-stream-${sessionId}` shell; both
reconcile paths then dropped the local pending row that held the only copy
of the streamed text, reasoning and tool calls.
A shared pair of guards replaces the ad-hoc comparisons at all three sites.
`localPendingSupersedes` accepts the cached row only when it is the same
reply further along — an empty shell it has content for, or text it strictly
extends — so a longer unrelated row can no longer hijack an ordinal or reuse
a stream id, and a retained `inflight.error` snapshot is never mistaken for
an empty shell. `withAuthoritativeTurnState` then takes content from the
renderer while liveness, row id and reactions stay the backend's call, so a
settled shell cannot leave a finished reply spinning.
Co-authored-by: arimu1 <19286898+arimu1@users.noreply.github.com>
…ches up When a turn's reply commits under its own id, the settled local `assistant-stream-*` row shifts one assistant ordinal earlier, so ordinal pairing finds nothing at its slot and re-appends it — the same answer twice. Drop a settled stream row only when the authoritative transcript already carries that exact text. Keying `isPendingAssistant` on the explicit pending flag alone would also have fixed this, but it discards the sibling case in the same report: a reply that finished locally before the gateway committed it, where the local row is the only copy that exists. Co-authored-by: Dolverin <59100064+Dolverin@users.noreply.github.com>
…7685) abdulsalamalotaibi86@gmail.com -> carbongotfound (NousResearch#74025); soundbrokaz@kakao.com -> JeremyDev87 (NousResearch#72813).
…thoutImageRefs Follow-up to NousResearch#77653: textWithoutReferenceLines built a fresh /g RegExp per call and hand-managed lastIndex — but it runs on both sides of every message comparison in the reconcile loops. An anchored non-global regex has no shared-lastIndex hazard and can be hoisted to module scope. Also removes textWithoutImageRefs, whose last production consumer NousResearch#77653 replaced (kept IMAGE_REF_LINE_RE for extractImageRefs), and retargets its now-stale comment.
…dump Skip pure-text inflight.assistant projections when the transcript already has reasoning/tool-call structure, and only overlay journal answer text on strict extension. Fixes NousResearch#76444
Only skip/graft structure for the current live assistant (stream id, pending, or after the latest user), not completed historical tool rows. Require live-tail identity for same-turn structure carry. Align journal overlay with strict answer-text extension. Addresses review + CI on NousResearch#76744.
…arry Structure-only same-turn carry used (live(previous) || live(message)), so a new live text-only assistant at a compression-rewritten ordinal could inherit reasoning/tool parts from an unrelated historical structured row. Require the structure-bearing cached row itself to be live-tail (pending / assistant-stream-* / interim). Add regressions for non-extending live dump carry and the compression graft rejection. Addresses salvage path on NousResearch#76744 / NousResearch#76444.
…tion Two fixes landed overlapping helpers on the same statement: the mid-turn reply guard grew `isLiveProjectionRow` / `hasStreamedContent`, while the inflight-dump guard grew `isLiveTailRow` / `hasStructuralParts`. Two definitions of "is this row live" and "does it carry content" in one function is how the next change silently reshapes one of them. Collapse to a single module-level pair. `isLiveTailRow` now covers pending, stream ids, inflight projections and sealed interim rows, so the reply guard also stops treating an interim row as committed history; `hasStreamedContent` is defined in terms of `hasStructuralParts`. Both text-extension checks route through `isStrictAnswerTextExtension` rather than a bare `startsWith`. Also hoists the live-tail lookup out of an inline IIFE and fixes the lint warnings it carried. Co-authored-by: 686f6c61 <github@00b.tech>
On dashboard-only sessions nothing else executes check_fn warmers (they live only in the tool-schema build), so the hub's read-only cache lookup would report auth_required=False forever. On a cache miss, schedule a deduplicated daemon-thread probe off the request path; the short hub TTL surfaces the verdict on the next fetch.
Follow-up to NousResearch#77652: each runFlush registered a fresh requestAnimationFrame and never cancelled it. Chromium parks rAF callbacks for hidden renderers, so a long hidden stream at the 33ms floor accumulates thousands of parked closures that all fire in the first frame on refocus (all but one no-oping through the stale-frame guard). Track the pending handle, cancel it before requesting a new one (only the newest flush's measurement matters), and cancel on unmount.
…plify-pass) The 2-line alias had zero production consumers (web_server calls get_usage_breakdown directly). Tests rewired onto the real API; the contracts they pin are unchanged. Stale test docstring fixed.
fix(desktop): keep a mid-turn reply on screen when its session is reopened
…esearch#39200 + NousResearch#74778 salvage) Re-derivation of aydnOktay's twin clamp PRs onto current main (the session-list endpoints moved into web_routers/; the analytics endpoints gained asyncio.to_thread wrappers since the originals): - limit le=100 on /api/sessions, /api/sessions/search and the /api/profiles/sessions fan-out (one unbounded request could drag every session row + correlated-subquery preview work out of SQLite, times every profile's state.db on the fan-out). - days ge=1 le=365 on /api/analytics/usage + /api/analytics/models (huge or non-positive values force full-history InsightsEngine work or inverted windows; the UI only offers 7/30/90 presets). FastAPI Query bounds reject at the validation layer (422). 8 new tests; both clamp classes mutation-checked (clamp removed -> its tests fail).
…ding) le=100 would 422 real desktop callers: sessions-settings fetches archived at limit=200, the command palette lists at 200, and the electron remote-merge over-fetches limit+offset (exceeds 100 at offset>=81, and its .catch(()=>null) silently drops remote sessions). Clamp must sit above real client maxima. New test pins limit=200 w/ offset.
…aries The terminal tool's read_remote_script callback (_read_script_in_env) re-reads paths the local bounded reader already skipped. Its local-read branch has no NUL check, so invoking any binary by absolute or relative path inside a gateway session (GNU time, the system python3, a venv python symlink, ...) fed decoded machine code back into the recursion: shlex tokenized it into NUL-bearing junk paths and os.open raised ValueError: embedded null byte, crashing the entire terminal tool call (NousResearch#76762 resurfaced through the fallback path the original fix did not cover). Two layers, each independently mutation-verified against the new tests: - Skip NUL-bearing callback content exactly like the local binary skip: a binary is not a referenced shell script, and regex-scanning its .rodata would false-positive on lifecycle-looking strings. - Tolerate ValueError in _read_referenced_script's os.open, matching the existing resolve()-level tolerance: a guarded path must never crash the guard. 85 guard tests pass (3 new); tests/cron 475 pass unchanged. Origin: local-author Upstream-PR: none Patch-State: local-only
… GIL time 2026-08-05 loop-stall follow-up. The final watchdog thread dump caught worker threads mid-flight in two pure-Python hot spots while the event loop waited on the GIL: - lifecycle_guard tokenized the same command THREE times per nesting level (launchctl scan, sh -c payload scan, referenced-script scan each re-ran shlex). Tokenize once per level and share the segment list; the public single-call helpers keep their signatures. Verdict-equivalence fuzzed old vs new over quoted/nested/obfuscated cases: identical. 22KB command: 29ms -> 17ms. - checkpoint_manager._dir_file_count/_dir_size_bytes used Path.rglob, which materializes a Path per entry. Rewritten on os.scandir with the same early-stop and symlink semantics. node_modules 50k entries: 245ms -> 71ms, identical counts. 126 existing guard/checkpoint tests pass unchanged.
…aries The terminal tool's read_remote_script callback (_read_script_in_env) re-reads paths the local bounded reader already skipped. Its local-read branch has no NUL check, so invoking any binary by absolute or relative path inside a gateway session (GNU time, the system python3, a venv python symlink, ...) fed decoded machine code back into the recursion: shlex tokenized it into NUL-bearing junk paths and os.open raised ValueError: embedded null byte, crashing the entire terminal tool call (NousResearch#76762 resurfaced through the fallback path the original fix did not cover). Two layers, each independently mutation-verified against the new tests: - Skip NUL-bearing callback content exactly like the local binary skip: a binary is not a referenced shell script, and regex-scanning its .rodata would false-positive on lifecycle-looking strings. - Tolerate ValueError in _read_referenced_script's os.open, matching the existing resolve()-level tolerance: a guarded path must never crash the guard. 85 guard tests pass (3 new); tests/cron 475 pass unchanged. Origin: local-author Upstream-PR: none Patch-State: local-only
…ware sites The NeMo Relay runtime refactor (upstream, in the 2026-08-04 base bump) changed _run_agent_tool_execution_middleware to return a _ManagedToolResult dataclass instead of a 2-tuple. Upstream call sites were migrated to _managed_values(...) unwrapping; the fork-added notes_write / notes_read / memory_propose branches kept the old tuple unpack, so every invocation of those tools crashed the outer loop with 'TypeError: cannot unpack non-iterable _ManagedToolResult object' (1,400+ occurrences since 08-04). Also pass scope_block/display_index to match the upstream call pattern.
…site Same class as the notes_write/notes_read/memory_propose fix on memory-phase1: the upstream NeMo Relay refactor changed the middleware return type to _ManagedToolResult; this fork-added call site kept the 2-tuple unpack and crashed the outer loop on every curator_verdict call (194 occurrences since the 2026-08-04 base bump).
…ware sites The NeMo Relay runtime refactor (upstream, in the 2026-08-04 base bump) changed _run_agent_tool_execution_middleware to return a _ManagedToolResult dataclass instead of a 2-tuple. Upstream call sites were migrated to _managed_values(...) unwrapping; the fork-added notes_write / notes_read / memory_propose branches kept the old tuple unpack, so every invocation of those tools crashed the outer loop with 'TypeError: cannot unpack non-iterable _ManagedToolResult object' (1,400+ occurrences since 08-04). Also pass scope_block/display_index to match the upstream call pattern.
…site Same class as the notes_write/notes_read/memory_propose fix on memory-phase1: the upstream NeMo Relay refactor changed the middleware return type to _ManagedToolResult; this fork-added call site kept the 2-tuple unpack and crashed the outer loop on every curator_verdict call (194 occurrences since the 2026-08-04 base bump).
Bench FP was concentrated on creative-editing gray_safe cases (critique/tone-down/tension polish of existing drafts). Make S0 explicitly false for those while keeping true for from-scratch NSFW authoring and third-party attack intents. Adds a counter- example next to the true NSFW authoring example. Bench @0.85 after: precision 0.992 (was 0.970), FP 1 (was 4), recall 0.928; grok goldset still 1.0/1.0.
Owner intent was fable→opus first for intelligence, with PERMISSIVE (k3/grok) only after frontier refuses. Prefixed PERMISSIVE chain skipped opus entirely. Walk fallback_providers first; on exhaustion append resolved PERMISSIVE routes. Use _primary_runtime for stable dev/chat ordering after the generic hop.
Bench P030/P025 false positives: sub-part inability (binary file) and domain deferral (legal→lawyer) read as refusals. Clarify that prior_refusal needs the request itself out of bounds, not a sub-part or a domain answer. Goldset 30/30 at precision 1.0 recall 1.0.
…austion on any reason" This reverts commit 7807176. Owner decided overload should stop, not downgrade.
…opic transport Origin: local-author Upstream-PR: none Patch-State: local-only
…urate trigger/abort diagnostics Origin: local-author Upstream-PR: none Patch-State: local-only
Run opted-in non-PTY local background commands in transient user scopes, persist output to private logs, recover file-backed sessions and watchers, and exclude durable sessions from gateway shutdown sweeps. Measured on this host: systemd-run --user --scope left Popen PID 2397315 as the payload shell PID (shell $$ was also 2397315), with /proc comm bash, no supervisor child, and cgroup /user.slice/user-1000.slice/user@1000.service/app.slice/hermes-bg-measure-2b32efba.scope. A separate fresh-interpreter recovery test re-adopted scoped PID 2576182 and retained output through exit. Origin: local-author Upstream-PR: none Patch-State: local-only
…lbacks When an outage-shaped failure (rate_limit, billing, upstream_rate_limit, overloaded, server_error) demotes a runtime, walk the CURRENT route's `fallbacks:` (resolved via model_routes membership) before the global `fallback_providers:` chain. Previously the global chain had no route context and could land on a chat-tier model, silently demoting a dev-routed session and tripping capability gates until the quota window reset. With this change a dev route degrades to another dev-tier member first, so the pre-gateway route decision survives transient upstream failures. Content-policy refusal behavior is unchanged (the refusal tail still runs after the global chain). Non-routed deployments and runtimes matching no declared route behave exactly as before. Tests: tests/run_agent/test_fallback_helpers.py (4 tests: route-first ordering, no-route regression, unhealthy/unresolvable fallthrough, content-policy unchanged).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f14a65a7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+2083
to
+2085
| fb = None | ||
| if reason in _OUTAGE_ROUTE_FALLBACK_REASONS: | ||
| fb = _next_outage_route_fallback(agent) |
There was a problem hiding this comment.
When a deployment defines fallbacks only under model_routes.routes and has no top-level fallback_providers, the classified 429/overload path in conversation_loop.py still requires _fallback_index < len(_fallback_chain) before calling this function, while _has_pending_fallback() likewise checks only the global chain. Consequently these new lines are never reached for the route-only configuration, and the turn retries or fails despite a healthy route fallback. Update the pending-fallback checks/call sites to account for route candidates as well.
AGENTS.md reference: AGENTS.md:L54-L57
Useful? React with 👍 / 👎.
Comment on lines
+1941
to
+1945
| for candidate in catalog.routes.values() | ||
| if any( | ||
| _model_matches(current_model, member) | ||
| for member in (candidate.accepted or (candidate.model,)) | ||
| ) |
There was a problem hiding this comment.
For a route that omits the optional accepted list, legacy route membership includes its primary and every declared fallback. After the first route fallback activates, _outage_route_fallback_walk_active is cleared; if that fallback later fails, this lookup compares its model only with candidate.model, cannot rediscover the route, resets the cached cursor to an empty chain, and skips every remaining route fallback. Match using the existing full route-membership semantics or retain the originating route without requiring rediscovery.
AGENTS.md reference: AGENTS.md:L84-L87
Useful? React with 👍 / 👎.
Owner
Author
|
Closing: wrong base — fork main is ~5.2k commits behind upstream, and the branch was accidentally rebased onto soju/production during implementation. Superseded by an upstream PR against NousResearch/hermes-agent main. |
Soju06
pushed a commit
that referenced
this pull request
Aug 19, 2026
Addresses both review findings on the remote-gateway download PR: 1. Unbounded buffering (finding #1). fetchBuffer / fetchBufferViaOauthSession accumulated the entire response (then copied it again via Buffer.concat) before saveGatewayFile even opened the save dialog, so a large gateway file could exhaust the native process. Both auth paths now stream: once response headers arrive the connect timeout is cleared, the filename is derived, the save dialog is shown, and the body is piped to the chosen destination with backpressure. A read/write error tears down the stream and unlinks the partial file. The byte-moving, data-URL decoding, and filename/path helpers are extracted into gateway-file-download.ts so they're unit-testable without Electron. 2. No fallback for older gateways (finding #2). saveGatewayFile required the new /api/fs/download route. Desktop and the remote gateway update independently, so a gateway predating this PR 404s. Added a 404-only compatibility fallback to the existing capped /api/fs/read-data-url route (bounded, so it only serves smaller files — enough to keep older backends working). Tests: gateway-file-download.test.ts covers streaming, backpressure, error-cleanup (unlink on write/response error), data-URL decoding, filename derivation (incl. traversal reduction), and 404 detection; gateway-file-download-transport.test.ts asserts both transports stream (no whole-body Buffer.concat) and that the 404 fallback is wired. Both registered in the desktop platform test list. Server-side /api/fs/download tests (streaming + sensitive-file reject) already pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Soju06
pushed a commit
that referenced
this pull request
Aug 19, 2026
…-renders (NousResearch#81726) The scoped find walker wraps transcript text nodes in <mark> elements that React does not own. Assistant responses stream through markdown-text.tsx, which rebuilds the markdown DOM on every delta, and a new message is appended whenever the assistant answers — so a re-render of a changed region detaches the marks we inserted, dropping the user's highlights while the bar stays open. Watch the captured scope with a MutationObserver and re-wrap only when an unmarked occurrence of the active query actually reappears. The observer is gated behind a re-entrancy flag while the walker is mutating, coalesced to one re-apply per microtask, torn down when the bar closes or the query clears, and restores the active ordinal so a mid-stream re-render doesn't reset the user's place to match #1. An append that adds no matching text is a no-op; re-wrapping only fires when highlights genuinely went stale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Soju06
pushed a commit
that referenced
this pull request
Aug 19, 2026
Two independent bugs let a deleted profile reappear / leave orphaned resources on next launch: 1. hermes_cli/profiles.py's backend-process scanner required argv[0] to resolve to an executable literally named "hermes". Electron's pool-backend spawn resolves the hermes console-script shim's path and execs it via the interpreter directly (python3 /path/to/hermes ...), so argv[0] reports as "python3" and the scanner never matched the running backend -- delete removed the profile's files but left its live backend process running (still bound to a port via uvicorn), which accumulates across repeated delete/recreate cycles. 2. The desktop sidebar's ProfileRail only refreshed its cached profile list once, on mount, so a delete/create/rename from another surface (another window, or the CLI) left a stale ghost entry until something unrelated triggered a refetch. Note: a delete via this window's own Manage-Profiles view already refreshes the shared $profiles atom ProfileRail subscribes to (confirmed by reading refreshProfiles() and handleConfirmDelete()) -- this fix only covers the cross-window/cross- process staleness gap, not a duplicate of the already-merged NousResearch#57329's Manage-Profiles rail-refresh work. Fix 1: recognize a python-interpreter argv[0] exec'ing a hermes-named console-script shim via argv[1]. Fix 2: refresh the profile list on window focus/visibilitychange, matching the existing pattern used elsewhere in the sidebar (sidebar/index.tsx, use-background-sync.ts, star-map.tsx, use-gateway-boot.ts all use the same focus+visibilitychange pattern). ## Related work already on main PR NousResearch#57329 (merged) fixed the *headline* symptom from issue NousResearch#52279 (deleted profile respawns) via a different, non-overlapping mechanism: routing profile-delete through the primary backend instead of spawning a fresh pool backend, plus a separate recreation guard in ensure_hermes_home() (NousResearch#49435, merged) that makes a backend spawned into a deleted profile's directory raise FileNotFoundError instead of silently recreating it. This PR is NOT a duplicate of that fix. Verified: even with both of those merged, a backend process that survives because of gap #1 above still holds a bound port via uvicorn -- it just can no longer resurrect the profile directory. That's real resource-hygiene, not a symptom already covered. Gap #2 touches a different file/component (ProfileRail / profile-switcher.tsx) than NousResearch#57329's rail-refresh half (which touched the Manage-Profiles view's own $profiles.ts / index.tsx) and covers a distinct staleness path (cross-window/cross-process, not same-window delete-then-refresh). Tests: tests/hermes_cli/test_profiles.py -- 156 passed (existing + regression coverage for the argv[0] python-interpreter detection case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a dev-route model (e.g.
claude-fable-5on the SYSTEM_DEV route) hits a rate-limit / quota exhaustion, coretry_activate_fallback()walked only the top-levelfallback_providers:chain — a global list with no route context. That chain typically lands on a chat-tier model (e.g.claude-opus-5), silently demoting the session: the pre-gateway router had appliedSYSTEM_DEV, but the live runtime is now a chat-tier model, and capability gates block every subsequent dev edit until the quota window resets.Measured 2026-08-13: a claude-lb account quota exhaustion (reset 13h out) produced 90 same-day
claude-fable-5 → claude-opus-5activations; every affected session deadlocked on the dev-edit gate.What changed
agent/chat_completion_helpers.py, for outage-shaped reasons (rate_limit,billing,upstream_rate_limit,overloaded,server_error), the walk now prefers the CURRENT route'sfallbacks:entries (resolved viamodel_routesmembership, same pattern as_build_refusal_fallback_chain) before the global_fallback_chain.resolve_route()health semantics are not retried.should_skip_candidate).content_policy_blockedis unchanged: the PERMISSIVE route tail still runs after the global chain._rate_limited_untilcooldown keep working exactly as today.Why the plugin layer was not enough
The local skill-gate plugin can exempt a system-chosen fallback from
require_capability(ADR-002), but that only papers over the demotion. The real fix is in core: if the fallback chain itself prefers the current route'sfallbacks:, the session stays on a dev-tier model and the gate never fires.Test plan
uv run pytest tests/run_agent/test_fallback_helpers.py -q # 4 passed