From 36399135e4de5e61d2e6396d3f096206e1fd15a9 Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Fri, 16 Oct 2026 23:15:50 -0500 Subject: [PATCH 1/4] fix(omp): chain Sol to direct DeepSeek Flash on a usage limit The session overlay emptied the Sol fallback chain, so when both Codex accounts hit their weekly limit omp had no credential to rotate to and no chain to walk, and it stopped at its wait cap (retry.maxDelayMs) instead. Chain openai-codex/gpt-6.1-sol to deepseek/deepseek-v4-flash:high and set retry.fallbackOn to usage-limit, so only quota-class errors move a session and a provider error burst still stops on Sol. fallbackRevertPolicy stays cooldown-expiry. The live fallback-chain guard now asserts the move to Flash and fails on the old overlay. --- .omp/fm-session-overlay.yml | 28 +++++---- bin/fm-spawn.sh | 10 +-- docs/verification/runtime-backends.md | 32 ++++++---- tests/fm-omp-fallback-chain-live-e2e.test.sh | 65 +++++++++++++++----- 4 files changed, 93 insertions(+), 42 deletions(-) diff --git a/.omp/fm-session-overlay.yml b/.omp/fm-session-overlay.yml index 8b0e1fe0b18..f8d95b9c221 100644 --- a/.omp/fm-session-overlay.yml +++ b/.omp/fm-session-overlay.yml @@ -8,7 +8,7 @@ # unattended worker on an interactive prompt, change its pinned model under it, # or make its composer unreadable to bin/fm-composer-lib.sh. Verified against # omp 18.1.11's own settings schema (`omp config list`); the fallback keys against -# omp 18.8.7 by tests/fm-omp-fallback-chain-live-e2e.test.sh. +# omp 18.11.0 by tests/fm-omp-fallback-chain-live-e2e.test.sh. composer: # Pins the bare `❯` (U+276F) row for predictable worker rendering. # omp live-reloads this overlay, so removing the pin also changes sessions @@ -27,17 +27,23 @@ retry: # `confirm` pops a dialog when a usage window runs low; `auto` lets omp fall # through the fallback chain without a keystroke. usageReservePolicy: auto - # A session on the strongest class never falls to a weaker model, and fleet - # agents never run on OpenRouter, so the Sol chain is empty: when the - # recorded model fails, the run stops with the provider's error instead of - # moving on. The global chain for this model once continued to a weaker one, - # which stranded a lane there after a provider error burst. omp merges - # overlays key by key and replaces an array whole, so this entry replaces that - # one chain and leaves every other chain as the operator set it. The - # session's live-model record (bin/fm-omp-live-model.ts) carries the stop to - # the supervisor, and a stand-in is chosen at dispatch or relaunch. + # A Sol session whose usage window is out moves at once to direct DeepSeek + # Flash, the approved stand-in, and never to an OpenRouter route. With both + # Codex accounts out there is no credential to rotate to, and without a + # chain omp stops at its wait cap instead (a usage window of several days + # exceeds retry.maxDelayMs) and the lane sits dead. omp merges overlays key by + # key and replaces an array whole, so this entry replaces the global Sol + # chain and leaves every other chain as the operator set it. A per-task + # overlay still replaces it for a worker. `usage-limit` limits every switch + # to quota-class errors, so an error burst from a provider that still has + # quota retries on Sol and then stops with the provider's error; the + # session's live-model record (bin/fm-omp-live-model.ts) carries that stop + # to the supervisor. omp reads this key for every chain, so another model's + # global chain also moves only on a quota-class error. + fallbackOn: usage-limit fallbackChains: - openai-codex/gpt-6.1-sol: [] + openai-codex/gpt-6.1-sol: + - deepseek/deepseek-v4-flash:high # omp's own way back to the recorded model, for every chain that remains: the # next prompt after the primary's suppression window ends restores it. # `never` would leave a session on its fallback until someone changes the diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 0b4305453cc..caa13c71043 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -310,10 +310,12 @@ # .omp/fm-session-overlay.yml through --config. That overlay pins composer # shape, plan mode off, prewalk off, and the non-interactive usage-reserve # policy for the one session only (--auto-approve alone owns approval, -# forcing tools.approvalMode: yolo for the session). It also empties the Sol -# model's fallback chain, so a failing Sol run stops and reports through the -# live-model record instead of moving to a weaker model, and it pins omp's own -# return to the recorded model once the primary serves again. +# forcing tools.approvalMode: yolo for the session). It also chains the Sol +# model to direct DeepSeek Flash and limits fallback to usage-limit class +# errors, so a Sol session whose usage window is out moves to the stand-in at +# once while any other failing Sol run stops and reports through the +# live-model record, and it pins omp's own return to the recorded model once +# the primary serves again. # Each session publishes the model it is serving through # bin/fm-omp-live-model.ts (a crewmate or scout from its per-task extension, a # secondmate from the home's tracked extension), and bin/fm-crew-state.sh diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 73d8100303d..d902f4748e8 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -1730,19 +1730,24 @@ Additional isolated live validation on 2026-10-09, with omp 18.8.1 on Herdr 0.9. After the bounded wait, the extension submitted the wake alone, omp consumed the queued follow-up, and the editor retained exactly the unsent draft. The before-change extension also reproduced the stranded older-wiring wake: it remained above the draft until the live check timed out; the changed extension delivered it and left exactly the draft. -### 2026-10-09 omp fallback chain +### 2026-10-16 omp fallback chain -Verified on 2026-10-09 on macOS arm64 against omp 18.8.7, with the built-in `openai-codex` and `deepseek` providers pointed at scripted local servers through an isolated agent directory's `models.yml`, so no model tokens were spent. -The agent directory's `config.yml` carries the operator's Sol chain as it stands, `openai-codex/gpt-6.1-sol` to `deepseek/deepseek-v4-pro`, and one other chain, `deepseek/deepseek-v4-pro` to a lab-only route. +Verified on 2026-10-16 on macOS arm64 against omp 18.11.0, with the built-in `openai-codex` and `deepseek` providers pointed at scripted local servers through an isolated agent directory's `models.yml`, so no model tokens were spent. +The agent directory's `config.yml` carries the operator's Sol chain as it stood on 2026-10-09, `openai-codex/gpt-6.1-sol` to `deepseek/deepseek-v4-pro`, and one other chain, `deepseek/deepseek-v4-pro` to a lab-only route. The guard drives `omp -p` with the tracked session overlay `.omp/fm-session-overlay.yml`, the overlay every Firstmate-launched omp session carries. - **The incident reproduces without the overlay.** With the primary refusing, the global chain moved the session on to the weak model, which answered. - That is how a lane recorded on Sol ended on `deepseek-v4-flash` on 2026-10-09: the primary hit its usage limit, then the equal refused for credits. -- **The overlay empties the Sol chain.** - With the overlay, the weak model received no request, the run exited non-zero with the primary's rate-limit error, and the live-model record named the recorded model and carried the error. - omp merges `--config` overlays key by key and replaces an array whole, so the overlay's empty Sol entry replaced the global Sol chain and left every other chain alone: with the overlay, a session on the other chain's model still moved to that chain's route when its model refused. - Fleet agents run on no OpenRouter route, so the equal is not a chain hop; a stand-in is chosen at dispatch or relaunch. + That is how a lane recorded on Sol ended on `deepseek-v4-flash` through OpenRouter on 2026-10-09: the primary hit its usage limit, then the equal refused for credits. +- **The overlay chains Sol to direct DeepSeek Flash.** + With a scripted `usage_limit_reached` answer and a retry-after of one hour, far above `retry.maxDelayMs`, the session moved to `deepseek/deepseek-v4-flash`, the run succeeded, and the live-model record followed the session to the stand-in. + An empty Sol chain stopped the same run at the wait cap with the provider's usage-limit error, which is how the lanes stopped on 2026-10-16 when both Codex accounts were out. + omp merges `--config` overlays key by key and replaces an array whole, so the overlay's Sol entry replaced the global Sol chain: the global chain's weak model received no request, and a session on the other chain's model still moved to that chain's route when its model refused with a usage limit. + A per-task overlay keyed by the exact model still replaces the Sol entry for a worker. +- **`retry.fallbackOn: usage-limit` scopes the switch to quota-class errors.** + omp classes a `usage_limit_reached` body and a 402 credit refusal as quota-class, so both moved the session to the stand-in. + A 500 from a provider that still has quota was retried on Sol and then stopped with the provider's error and the record carried it, where the same chain without the key moved the session to the stand-in. + The key is global to the session, not per chain, so the other chain's model also moves only on a quota-class error. - **omp returns to the recorded model by itself.** `retry.fallbackRevertPolicy: cooldown-expiry` restores the original selector at the next prompt once its suppression window ends; it never steps back to an intermediate route, and a primary that is out of quota for days stays suppressed for days. The guard layers a lab-only chain after the tracked overlay (a later `--config` wins) so a fallback can happen at all. @@ -1759,11 +1764,12 @@ FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh ``` ```text -ok - omp (omp/18.8.7): control - the global Sol chain alone still falls to the weak model -ok - omp (omp/18.8.7): the overlay empties the Sol chain, the run stops with the provider error, and the record carries it -ok - omp (omp/18.8.7): the overlay leaves another model's global chain in place -ok - omp (omp/18.8.7): with the overlay a fallen-back session returns to its recorded model at the next prompt even under a global fallbackRevertPolicy: never -ok - omp (omp/18.8.7): control - without the overlay a global fallbackRevertPolicy: never keeps the session on the fallback +ok - omp (omp/18.11.0): control - the global Sol chain alone still falls to the weak model +ok - omp (omp/18.11.0): the overlay moves a Sol session whose usage window is out to DeepSeek Flash, and the record follows +ok - omp (omp/18.11.0): the overlay keeps a Sol session on its recorded model on a non-quota error, and the record carries the error +ok - omp (omp/18.11.0): the overlay leaves another model's global chain in place +ok - omp (omp/18.11.0): with the overlay a fallen-back session returns to its recorded model at the next prompt even under a global fallbackRevertPolicy: never +ok - omp (omp/18.11.0): control - without the overlay a global fallbackRevertPolicy: never keeps the session on the fallback ``` ### 2026-10-10 agy composer through Herdr diff --git a/tests/fm-omp-fallback-chain-live-e2e.test.sh b/tests/fm-omp-fallback-chain-live-e2e.test.sh index 20c1532cd3b..2f8bb296e75 100755 --- a/tests/fm-omp-fallback-chain-live-e2e.test.sh +++ b/tests/fm-omp-fallback-chain-live-e2e.test.sh @@ -6,19 +6,23 @@ # openrouter/deepseek/deepseek-v4-flash. The operator's global omp config chained # Sol to its equal and then on to DeepSeek, so when the equal answered 402 and # 401, omp moved the lane to a model far weaker than the one it was recorded on, -# and it stayed there. Fleet agents also run on no OpenRouter route, so the -# tracked overlay now empties the Sol chain. This guard drives the INSTALLED omp +# and it stayed there. The tracked overlay then emptied the Sol chain, and on +# 2026-10-16 both Codex accounts hit their weekly limit: with no credential to +# rotate to and no chain, omp stopped at its wait cap and the lanes sat dead. +# The tracked overlay now chains Sol to direct DeepSeek Flash and limits every +# switch to usage-limit class errors. This guard drives the INSTALLED omp # against scripted local providers that stand in for the real routes (the # built-in openai-codex and deepseek providers are pointed at them through the # isolated agent directory's models.yml), so it spends no model tokens, and # requires through the tracked session overlay that Firstmate launches omp with: # 1. control: the global chain alone still reaches the weak model, so the lab # reproduces the incident and the next checks cannot pass vacuously; -# 2. the overlay stops the Sol chain: when the primary fails, the weak model -# receives no request, the run ends with the provider's error, and the -# live-model record carries that error and the recorded model -# (bin/fm-omp-live-model.ts); another model's global chain still falls back -# under the overlay; +# 2. the overlay moves a Sol session whose usage window is out to DeepSeek +# Flash and to nothing else: the global chain's weak model receives no +# request, the run succeeds, and the live-model record follows the session +# to the stand-in (bin/fm-omp-live-model.ts); a 500 burst from a provider +# that still has quota stays on Sol and stops with the provider's error; +# another model's global chain still falls back under the overlay; # 3. return: for a chain that remains (the lab layers one on after the tracked # overlay), omp's own revert restores the recorded model at the next prompt # after the primary's suppression window ends, even when the global config @@ -54,6 +58,10 @@ trap cleanup EXIT # ok [delay] answers "ack" after delay seconds # credits always 402 # limit always 429 with that retry-after +# usage always 429 with a usage_limit_reached body, the shape the +# ChatGPT backend sends when a weekly window is out, and +# that retry-after +# error always 500 # flaky the first request refused as a concurrency limit, which omp # suppresses for 5 seconds (a provider's own retry-after that # short is waited out inside the provider call, so no @@ -80,6 +88,13 @@ class Handler(BaseHTTPRequestHandler): body = json.loads(self.rfile.read(int(self.headers.get("content-length", 0))) or b"{}") seen["n"] += 1 with open(log_path, "a") as log: log.write(str(body.get("model")) + "\n") + if behavior == "usage": + data = json.dumps({"error": {"type": "usage_limit_reached", "message": "The usage limit has been reached", "plan_type": "pro", "resets_in_seconds": int(param)}}).encode() + self.send_response(429); self.send_header("retry-after", str(int(param))) + self.send_header("content-type", "application/json"); self.send_header("content-length", str(len(data))); self.end_headers(); self.wfile.write(data) + return + if behavior == "error": + return fail(self, 500, "The server had an error while processing your request.") if behavior == "limit": return fail(self, 429, "You have hit your usage limit. Try again later.", param) if behavior == "flaky" and seen["n"] == 1: @@ -193,24 +208,46 @@ run_omp "$AGENT" none "say hi" || fail "$SUBJECT: without the overlay the global chain no longer reaches the weak model, so the lab does not reproduce the incident (weak saw: $(tr '\n' ' ' < "$LAB/weak.log")); output: $OUT" pass "$SUBJECT: control - the global Sol chain alone still falls to the weak model" -: > "$LAB/codex.log"; : > "$LAB/weak.log"; : > "$LAB/equal.log" +# The Sol usage window is out for days, the way the ChatGPT backend reports it. +# The overlay's chain replaces the global one, so the stand-in is Flash and the +# global chain's Pro receives nothing. +serve codex usage 3600 +AGENT=$(agent_dir chain never 300000) +: > "$LAB/weak.log"; : > "$LAB/equal.log" run_omp "$AGENT" "$OVERLAY" "say hi" [ "$(requests codex)" -ge 1 ] \ || fail "$SUBJECT: with the overlay the Sol session never tried its recorded model; output: $OUT" -[ "$(requests weak)" -eq 0 ] \ - || fail "$SUBJECT: with the overlay the Sol session reached the weak model after its recorded model failed: $(tr '\n' ' ' < "$LAB/weak.log")" +[ "$(grep -c '^deepseek-v4-flash$' "$LAB/weak.log")" -ge 1 ] \ + || fail "$SUBJECT: with the overlay a Sol session whose usage window is out never reached DeepSeek Flash (weak saw: $(tr '\n' ' ' < "$LAB/weak.log")); output: $OUT" +[ "$(grep -c '^deepseek-v4-pro$' "$LAB/weak.log")" -eq 0 ] \ + || fail "$SUBJECT: with the overlay the Sol session reached the global chain's weak model: $(tr '\n' ' ' < "$LAB/weak.log")" [ "$(requests equal)" -eq 0 ] || fail "$SUBJECT: with the overlay the Sol session left its recorded model for another route" +[ "$RC" -eq 0 ] || fail "$SUBJECT: with the stand-in answering the run must succeed, but it exited $RC: $OUT" +record_has "model=deepseek/deepseek-v4-flash" \ + || fail "$SUBJECT: the live-model record did not follow the session to the stand-in: $(cat "$LAB/record" 2>/dev/null)" +pass "$SUBJECT: the overlay moves a Sol session whose usage window is out to DeepSeek Flash, and the record follows" + +# An error burst from a provider that still has quota is not a quota error: the +# session stays on Sol, retries it, and stops with the provider's error. +serve codex error +AGENT=$(agent_dir chain never 300000) +: > "$LAB/weak.log" +run_omp "$AGENT" "$OVERLAY" "say hi" +[ "$(requests codex)" -ge 1 ] \ + || fail "$SUBJECT: with the overlay the Sol session never tried its recorded model; output: $OUT" +[ "$(requests weak)" -eq 0 ] \ + || fail "$SUBJECT: with the overlay a Sol session left its recorded model on a non-quota error: $(tr '\n' ' ' < "$LAB/weak.log")" [ "$RC" -ne 0 ] || fail "$SUBJECT: with the recorded model failing the run must stop with an error, but it exited 0: $OUT" -case "$OUT" in *"rate limit exceeded"*) ;; *) fail "$SUBJECT: the stopped run did not report the provider's error: $OUT" ;; esac +case "$OUT" in *"error while processing your request"*) ;; *) fail "$SUBJECT: the stopped run did not report the provider's error: $OUT" ;; esac record_has "model=openai-codex/gpt-6.1-sol" \ || fail "$SUBJECT: the live-model record does not name the recorded model that stopped: $(cat "$LAB/record" 2>/dev/null)" record_has "error=" \ || fail "$SUBJECT: the live-model record does not carry the unrecovered run error: $(cat "$LAB/record" 2>/dev/null)" -pass "$SUBJECT: the overlay empties the Sol chain, the run stops with the provider error, and the record carries it" +pass "$SUBJECT: the overlay keeps a Sol session on its recorded model on a non-quota error, and the record carries the error" # The overlay replaces only the Sol chain: the global chain of another model -# still reaches its fallback under it. -serve weak limit 3600 +# still reaches its fallback under it, on a quota-class error. +serve weak usage 3600 AGENT=$(agent_dir other never 300000) MODEL=deepseek/deepseek-v4-pro run_omp "$AGENT" "$OVERLAY" "say hi" [ "$(requests weak)" -ge 1 ] || fail "$SUBJECT: the other model's session never tried its recorded model; output: $OUT" From 13ecc3ba929485fae1264e3fbbd741fc0f2908e6 Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Fri, 16 Oct 2026 23:52:45 -0500 Subject: [PATCH 2/4] no-mistakes(review): Drop fallbackOn; keep plain Sol-to-Flash chain --- .../references/harness/omp.md | 3 +- .omp/fm-session-overlay.yml | 31 +++--- bin/fm-spawn.sh | 10 +- docs/configuration.md | 4 +- docs/verification/runtime-backends.md | 14 ++- tests/fm-omp-fallback-chain-live-e2e.test.sh | 94 +++++++++---------- 6 files changed, 73 insertions(+), 83 deletions(-) diff --git a/.agents/skills/harness-adapters/references/harness/omp.md b/.agents/skills/harness-adapters/references/harness/omp.md index 5c4484ae303..b5d7fede109 100644 --- a/.agents/skills/harness-adapters/references/harness/omp.md +++ b/.agents/skills/harness-adapters/references/harness/omp.md @@ -39,7 +39,8 @@ The optional claude-bridge extension runs a nested executable literally named `c [`fm-spawn.sh --help`](../../../bin/fm-spawn.sh) owns session posture, worker-only memory scope, and the invariant that the captain's own configuration is never written. OMP 18.6.1 natively rotates pooled credentials on `usage_limit_reached`; native rotation also applies to review processes launched with extensions disabled. -Workers enable usage-aware selection without a generic percentage reserve or reset wait, and a later per-task overlay supplies only the selected rule's exact model-and-effort fallback chain. +Workers enable usage-aware selection without a generic percentage reserve or reset wait, and a later per-task overlay, `state/.omp-fallback.yml`, supplies only the selected rule's exact model-and-effort fallback chain. +The tracked session overlay chains `openai-codex/gpt-6.1-sol` to direct `deepseek/deepseek-v4-flash:high` for every launch, so a Sol session moves to the stand-in on any eligible provider error, a quota-out included, and `retry.fallbackRevertPolicy: cooldown-expiry` returns it to Sol once the suppression window ends; a worker's per-task chain replaces that entry for its own model. Inspect pooled headroom through `bin/fm-dispatch-capacity.sh`, not quota-axi's single Codex account; the schema and supported cross-harness stand-ins belong to `docs/configuration.md`. ## Extension loading diff --git a/.omp/fm-session-overlay.yml b/.omp/fm-session-overlay.yml index f8d95b9c221..b0efebcd90e 100644 --- a/.omp/fm-session-overlay.yml +++ b/.omp/fm-session-overlay.yml @@ -27,25 +27,24 @@ retry: # `confirm` pops a dialog when a usage window runs low; `auto` lets omp fall # through the fallback chain without a keystroke. usageReservePolicy: auto - # A Sol session whose usage window is out moves at once to direct DeepSeek - # Flash, the approved stand-in, and never to an OpenRouter route. With both - # Codex accounts out there is no credential to rotate to, and without a - # chain omp stops at its wait cap instead (a usage window of several days - # exceeds retry.maxDelayMs) and the lane sits dead. omp merges overlays key by - # key and replaces an array whole, so this entry replaces the global Sol - # chain and leaves every other chain as the operator set it. A per-task - # overlay still replaces it for a worker. `usage-limit` limits every switch - # to quota-class errors, so an error burst from a provider that still has - # quota retries on Sol and then stops with the provider's error; the - # session's live-model record (bin/fm-omp-live-model.ts) carries that stop - # to the supervisor. omp reads this key for every chain, so another model's - # global chain also moves only on a quota-class error. - fallbackOn: usage-limit + # A Sol session whose provider fails moves at once to direct DeepSeek Flash, + # the approved stand-in, and never to an OpenRouter route. With both Codex + # accounts out there is no credential to rotate to, and without a chain omp + # stops at its wait cap instead (a usage window of several days exceeds + # retry.maxDelayMs) and the lane sits dead. The switch happens on any eligible + # provider error, not only on a quota-out. omp merges overlays key by key and + # replaces an array whole, so this entry replaces the global Sol chain and + # leaves every other chain as the operator set it. A worker's per-task + # overlay (state/.omp-fallback.yml, bin/fm-spawn.sh) loads after this one + # and replaces the entry for the worker's own model. The session's live-model + # record (bin/fm-omp-live-model.ts) follows the session to the stand-in. fallbackChains: openai-codex/gpt-6.1-sol: - deepseek/deepseek-v4-flash:high - # omp's own way back to the recorded model, for every chain that remains: the - # next prompt after the primary's suppression window ends restores it. + # omp's own way back to the recorded model, for every chain including the Sol + # one above: the next prompt after the primary's suppression window ends + # restores it, so a Sol session that moved to Flash returns to Sol once its + # window is over. # `never` would leave a session on its fallback until someone changes the # model by hand. fallbackRevertPolicy: cooldown-expiry diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index caa13c71043..3664aecee6e 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -311,11 +311,11 @@ # shape, plan mode off, prewalk off, and the non-interactive usage-reserve # policy for the one session only (--auto-approve alone owns approval, # forcing tools.approvalMode: yolo for the session). It also chains the Sol -# model to direct DeepSeek Flash and limits fallback to usage-limit class -# errors, so a Sol session whose usage window is out moves to the stand-in at -# once while any other failing Sol run stops and reports through the -# live-model record, and it pins omp's own return to the recorded model once -# the primary serves again. +# model to direct DeepSeek Flash, so a Sol session whose provider fails, a +# usage window that is out included, moves to the stand-in at once (a worker's +# per-task state/.omp-fallback.yml replaces that entry for its own model), +# and it pins omp's own return to the recorded model once the primary serves +# again. # Each session publishes the model it is serving through # bin/fm-omp-live-model.ts (a crewmate or scout from its per-task extension, a # secondmate from the home's tracked extension), and bin/fm-crew-state.sh diff --git a/docs/configuration.md b/docs/configuration.md index 6e2c40d5978..8cf15427546 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1687,8 +1687,10 @@ A recorded rule still contains the task's profile when typed resolution raised i A recorded identifier that no longer exists, or no longer contains the task's profile, counts as absent. A task that switched to a stand-in stays on it until it is launched again from the matrix. Without an identifier, identical matching lists apply, and differing matching lists permit no fallback. -OMP workers receive a later per-task configuration overlay with exact model-and-effort chains, preventing ambient chains from weakening the task. +OMP workers receive a later per-task configuration overlay, `state/.omp-fallback.yml`, with exact model-and-effort chains, preventing ambient chains from weakening the task. Those chains contain only the order-preserving eligible native prefix; the first supported cross-harness destination ends the prefix and leaves the transition to recovery. +The tracked session overlay `.omp/fm-session-overlay.yml` chains `openai-codex/gpt-6.1-sol` to direct `deepseek/deepseek-v4-flash:high` for every omp launch, so a Sol session moves to the stand-in on any eligible provider error, a quota-out included, and `retry.fallbackRevertPolicy: cooldown-expiry` returns it to Sol once the suppression window ends. +A worker's per-task chain replaces that entry for its own model, and is empty when the rule names no eligible native stand-in. Native OMP account rotation precedes model fallback, and successful native model switches publish the serving selector. An idle terminal quota error after native rotation is handled by the existing bounded recovery scan through `fm-control.sh relaunch`, retaining its pause, captain-call, generation, and work-preservation guards. Launch and successful replacement append the selected route to task status. diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index d902f4748e8..c08b1c11be8 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -1742,15 +1742,13 @@ The guard drives `omp -p` with the tracked session overlay `.omp/fm-session-over - **The overlay chains Sol to direct DeepSeek Flash.** With a scripted `usage_limit_reached` answer and a retry-after of one hour, far above `retry.maxDelayMs`, the session moved to `deepseek/deepseek-v4-flash`, the run succeeded, and the live-model record followed the session to the stand-in. An empty Sol chain stopped the same run at the wait cap with the provider's usage-limit error, which is how the lanes stopped on 2026-10-16 when both Codex accounts were out. - omp merges `--config` overlays key by key and replaces an array whole, so the overlay's Sol entry replaced the global Sol chain: the global chain's weak model received no request, and a session on the other chain's model still moved to that chain's route when its model refused with a usage limit. - A per-task overlay keyed by the exact model still replaces the Sol entry for a worker. -- **`retry.fallbackOn: usage-limit` scopes the switch to quota-class errors.** - omp classes a `usage_limit_reached` body and a 402 credit refusal as quota-class, so both moved the session to the stand-in. - A 500 from a provider that still has quota was retried on Sol and then stopped with the provider's error and the record carried it, where the same chain without the key moved the session to the stand-in. - The key is global to the session, not per chain, so the other chain's model also moves only on a quota-class error. + omp merges `--config` overlays key by key and replaces an array whole, so the overlay's Sol entry replaced the global Sol chain: the global chain's weak model received no request, and a session on the other chain's model still moved to that chain's route when its model refused. + A worker's per-task overlay, `state/.omp-fallback.yml`, loads after the tracked overlay and replaces the Sol entry for the worker's own model. +- **A 500 moves the session too.** + A 500 from a provider that still has quota was retried on Sol, then the session moved to the stand-in and the record followed it, so the chain applies on any eligible provider error, not only on a quota-out. - **omp returns to the recorded model by itself.** `retry.fallbackRevertPolicy: cooldown-expiry` restores the original selector at the next prompt once its suppression window ends; it never steps back to an intermediate route, and a primary that is out of quota for days stays suppressed for days. - The guard layers a lab-only chain after the tracked overlay (a later `--config` wins) so a fallback can happen at all. + The guard layers a lab-only chain after the tracked overlay (a later `--config` wins) so the fallback route answers slowly enough for the primary's window to end. A refusal classed as a concurrency limit is suppressed for 5 seconds, so the guard's second prompt ran on the primary again and the record followed. The overlay pins the policy, because a global `never` left the same session on its fallback in the control run. - **A provider's own short `retry-after` never reaches the chain.** @@ -1766,7 +1764,7 @@ FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh ```text ok - omp (omp/18.11.0): control - the global Sol chain alone still falls to the weak model ok - omp (omp/18.11.0): the overlay moves a Sol session whose usage window is out to DeepSeek Flash, and the record follows -ok - omp (omp/18.11.0): the overlay keeps a Sol session on its recorded model on a non-quota error, and the record carries the error +ok - omp (omp/18.11.0): the overlay moves a Sol session whose provider answers 500 to DeepSeek Flash, and the record follows ok - omp (omp/18.11.0): the overlay leaves another model's global chain in place ok - omp (omp/18.11.0): with the overlay a fallen-back session returns to its recorded model at the next prompt even under a global fallbackRevertPolicy: never ok - omp (omp/18.11.0): control - without the overlay a global fallbackRevertPolicy: never keeps the session on the fallback diff --git a/tests/fm-omp-fallback-chain-live-e2e.test.sh b/tests/fm-omp-fallback-chain-live-e2e.test.sh index 2f8bb296e75..6739c5f5e75 100755 --- a/tests/fm-omp-fallback-chain-live-e2e.test.sh +++ b/tests/fm-omp-fallback-chain-live-e2e.test.sh @@ -9,24 +9,24 @@ # and it stayed there. The tracked overlay then emptied the Sol chain, and on # 2026-10-16 both Codex accounts hit their weekly limit: with no credential to # rotate to and no chain, omp stopped at its wait cap and the lanes sat dead. -# The tracked overlay now chains Sol to direct DeepSeek Flash and limits every -# switch to usage-limit class errors. This guard drives the INSTALLED omp -# against scripted local providers that stand in for the real routes (the -# built-in openai-codex and deepseek providers are pointed at them through the -# isolated agent directory's models.yml), so it spends no model tokens, and -# requires through the tracked session overlay that Firstmate launches omp with: +# The tracked overlay now chains Sol to direct DeepSeek Flash. This guard drives +# the INSTALLED omp against scripted local providers that stand in for the real +# routes (the built-in openai-codex and deepseek providers are pointed at them +# through the isolated agent directory's models.yml), so it spends no model +# tokens, and requires through the tracked session overlay that Firstmate +# launches omp with: # 1. control: the global chain alone still reaches the weak model, so the lab # reproduces the incident and the next checks cannot pass vacuously; -# 2. the overlay moves a Sol session whose usage window is out to DeepSeek -# Flash and to nothing else: the global chain's weak model receives no -# request, the run succeeds, and the live-model record follows the session -# to the stand-in (bin/fm-omp-live-model.ts); a 500 burst from a provider -# that still has quota stays on Sol and stops with the provider's error; -# another model's global chain still falls back under the overlay; -# 3. return: for a chain that remains (the lab layers one on after the tracked -# overlay), omp's own revert restores the recorded model at the next prompt -# after the primary's suppression window ends, even when the global config -# says `fallbackRevertPolicy: never`, and the record follows; +# 2. the overlay moves a Sol session to DeepSeek Flash and to nothing else, +# both when its usage window is out and when its provider answers 500: the +# global chain's weak model receives no request, the run succeeds, and the +# live-model record follows the session to the stand-in +# (bin/fm-omp-live-model.ts); another model's global chain still falls back +# under the overlay; +# 3. return: for a lab-only chain layered on after the tracked overlay, omp's +# own revert restores the recorded model at the next prompt after the +# primary's suppression window ends, even when the global config says +# `fallbackRevertPolicy: never`, and the record follows; # 4. control: without the overlay, `never` leaves the session on the fallback. # It fails naming omp and `omp --version`. Refresh docs/verification/runtime-backends.md # ("omp fallback chain") from its output after any omp upgrade. @@ -170,8 +170,8 @@ YML printf '%s\n' "$dir" } -# A chain that remains after the tracked overlay, layered on after it the way a -# later --config wins: the primary falls to a lab-only equal route. +# A lab-only chain, layered on after the tracked overlay the way a later --config +# wins: the primary falls to a lab-only equal route. cat > "$LAB/lab-chain.yml" < + AGENT=$(agent_dir chain never 300000) + : > "$LAB/weak.log"; : > "$LAB/equal.log" + run_omp "$AGENT" "$OVERLAY" "say hi" + [ "$(requests codex)" -ge 1 ] \ + || fail "$SUBJECT: with the overlay the Sol session never tried its recorded model; output: $OUT" + [ "$(grep -c '^deepseek-v4-flash$' "$LAB/weak.log")" -ge 1 ] \ + || fail "$SUBJECT: with the overlay a Sol session whose $1 never reached DeepSeek Flash (weak saw: $(tr '\n' ' ' < "$LAB/weak.log")); output: $OUT" + [ "$(grep -c '^deepseek-v4-pro$' "$LAB/weak.log")" -eq 0 ] \ + || fail "$SUBJECT: with the overlay the Sol session reached the global chain's weak model: $(tr '\n' ' ' < "$LAB/weak.log")" + [ "$(requests equal)" -eq 0 ] || fail "$SUBJECT: with the overlay the Sol session left its recorded model for another route" + [ "$RC" -eq 0 ] || fail "$SUBJECT: with the stand-in answering the run must succeed, but it exited $RC: $OUT" + record_has "model=deepseek/deepseek-v4-flash" \ + || fail "$SUBJECT: the live-model record did not follow the session to the stand-in: $(cat "$LAB/record" 2>/dev/null)" + pass "$SUBJECT: the overlay moves a Sol session whose $1 to DeepSeek Flash, and the record follows" +} + +# The Sol usage window is out for days, the way the ChatGPT backend reports it. serve codex usage 3600 -AGENT=$(agent_dir chain never 300000) -: > "$LAB/weak.log"; : > "$LAB/equal.log" -run_omp "$AGENT" "$OVERLAY" "say hi" -[ "$(requests codex)" -ge 1 ] \ - || fail "$SUBJECT: with the overlay the Sol session never tried its recorded model; output: $OUT" -[ "$(grep -c '^deepseek-v4-flash$' "$LAB/weak.log")" -ge 1 ] \ - || fail "$SUBJECT: with the overlay a Sol session whose usage window is out never reached DeepSeek Flash (weak saw: $(tr '\n' ' ' < "$LAB/weak.log")); output: $OUT" -[ "$(grep -c '^deepseek-v4-pro$' "$LAB/weak.log")" -eq 0 ] \ - || fail "$SUBJECT: with the overlay the Sol session reached the global chain's weak model: $(tr '\n' ' ' < "$LAB/weak.log")" -[ "$(requests equal)" -eq 0 ] || fail "$SUBJECT: with the overlay the Sol session left its recorded model for another route" -[ "$RC" -eq 0 ] || fail "$SUBJECT: with the stand-in answering the run must succeed, but it exited $RC: $OUT" -record_has "model=deepseek/deepseek-v4-flash" \ - || fail "$SUBJECT: the live-model record did not follow the session to the stand-in: $(cat "$LAB/record" 2>/dev/null)" -pass "$SUBJECT: the overlay moves a Sol session whose usage window is out to DeepSeek Flash, and the record follows" +expect_stand_in "usage window is out" -# An error burst from a provider that still has quota is not a quota error: the -# session stays on Sol, retries it, and stops with the provider's error. +# An error burst from a provider that still has quota moves the session too. serve codex error -AGENT=$(agent_dir chain never 300000) -: > "$LAB/weak.log" -run_omp "$AGENT" "$OVERLAY" "say hi" -[ "$(requests codex)" -ge 1 ] \ - || fail "$SUBJECT: with the overlay the Sol session never tried its recorded model; output: $OUT" -[ "$(requests weak)" -eq 0 ] \ - || fail "$SUBJECT: with the overlay a Sol session left its recorded model on a non-quota error: $(tr '\n' ' ' < "$LAB/weak.log")" -[ "$RC" -ne 0 ] || fail "$SUBJECT: with the recorded model failing the run must stop with an error, but it exited 0: $OUT" -case "$OUT" in *"error while processing your request"*) ;; *) fail "$SUBJECT: the stopped run did not report the provider's error: $OUT" ;; esac -record_has "model=openai-codex/gpt-6.1-sol" \ - || fail "$SUBJECT: the live-model record does not name the recorded model that stopped: $(cat "$LAB/record" 2>/dev/null)" -record_has "error=" \ - || fail "$SUBJECT: the live-model record does not carry the unrecovered run error: $(cat "$LAB/record" 2>/dev/null)" -pass "$SUBJECT: the overlay keeps a Sol session on its recorded model on a non-quota error, and the record carries the error" +expect_stand_in "provider answers 500" # The overlay replaces only the Sol chain: the global chain of another model -# still reaches its fallback under it, on a quota-class error. -serve weak usage 3600 +# still reaches its fallback under it. +serve weak limit 3600 AGENT=$(agent_dir other never 300000) MODEL=deepseek/deepseek-v4-pro run_omp "$AGENT" "$OVERLAY" "say hi" [ "$(requests weak)" -ge 1 ] || fail "$SUBJECT: the other model's session never tried its recorded model; output: $OUT" From f702970f9b972451d01771f30d4ad1494664cfe3 Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Sat, 17 Oct 2026 00:07:33 -0500 Subject: [PATCH 3/4] no-mistakes(document): Fix stale omp Sol-chain doc wording --- .agents/skills/harness-adapters/references/harness/omp.md | 3 +-- bin/fm-crew-state.sh | 2 +- docs/verification/runtime-backends.md | 6 +++--- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/.agents/skills/harness-adapters/references/harness/omp.md b/.agents/skills/harness-adapters/references/harness/omp.md index b5d7fede109..27350632906 100644 --- a/.agents/skills/harness-adapters/references/harness/omp.md +++ b/.agents/skills/harness-adapters/references/harness/omp.md @@ -40,8 +40,7 @@ The optional claude-bridge extension runs a nested executable literally named `c [`fm-spawn.sh --help`](../../../bin/fm-spawn.sh) owns session posture, worker-only memory scope, and the invariant that the captain's own configuration is never written. OMP 18.6.1 natively rotates pooled credentials on `usage_limit_reached`; native rotation also applies to review processes launched with extensions disabled. Workers enable usage-aware selection without a generic percentage reserve or reset wait, and a later per-task overlay, `state/.omp-fallback.yml`, supplies only the selected rule's exact model-and-effort fallback chain. -The tracked session overlay chains `openai-codex/gpt-6.1-sol` to direct `deepseek/deepseek-v4-flash:high` for every launch, so a Sol session moves to the stand-in on any eligible provider error, a quota-out included, and `retry.fallbackRevertPolicy: cooldown-expiry` returns it to Sol once the suppression window ends; a worker's per-task chain replaces that entry for its own model. -Inspect pooled headroom through `bin/fm-dispatch-capacity.sh`, not quota-axi's single Codex account; the schema and supported cross-harness stand-ins belong to `docs/configuration.md`. +Inspect pooled headroom through `bin/fm-dispatch-capacity.sh`, not quota-axi's single Codex account; the schema, the tracked session overlay's Sol-to-DeepSeek-Flash chain, and supported cross-harness stand-ins belong to `docs/configuration.md`. ## Extension loading diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index 767b2ec0e28..4cbe0ce9415 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -34,7 +34,7 @@ # compared only when the launch recorded an explicit provider/id. # run-error: # the session's last run ended in an error that no retry or fallback -# recovered, so a session that stopped rather than downgrade says so. +# recovered, so a session that stopped on an error says so. # model-stale: on is older than in the catalog # any harness: the launch was told, at spawn, that a newer model of the # same family was in its catalog (bin/fm-model-freshness.sh owns the rule). diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index c08b1c11be8..f1e8e8446b6 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -1741,11 +1741,11 @@ The guard drives `omp -p` with the tracked session overlay `.omp/fm-session-over That is how a lane recorded on Sol ended on `deepseek-v4-flash` through OpenRouter on 2026-10-09: the primary hit its usage limit, then the equal refused for credits. - **The overlay chains Sol to direct DeepSeek Flash.** With a scripted `usage_limit_reached` answer and a retry-after of one hour, far above `retry.maxDelayMs`, the session moved to `deepseek/deepseek-v4-flash`, the run succeeded, and the live-model record followed the session to the stand-in. - An empty Sol chain stopped the same run at the wait cap with the provider's usage-limit error, which is how the lanes stopped on 2026-10-16 when both Codex accounts were out. + The guard requires a request to reach Flash, so an empty Sol chain fails it. + The stop an empty chain caused in production on 2026-10-16, when both Codex accounts were out, is not exercised by the guard: omp stopped at its wait cap, `retry.maxDelayMs`, with the provider's usage-limit error. omp merges `--config` overlays key by key and replaces an array whole, so the overlay's Sol entry replaced the global Sol chain: the global chain's weak model received no request, and a session on the other chain's model still moved to that chain's route when its model refused. - A worker's per-task overlay, `state/.omp-fallback.yml`, loads after the tracked overlay and replaces the Sol entry for the worker's own model. - **A 500 moves the session too.** - A 500 from a provider that still has quota was retried on Sol, then the session moved to the stand-in and the record followed it, so the chain applies on any eligible provider error, not only on a quota-out. + A 500 from a provider that still has quota also moved the session to the stand-in and the record followed it, so the chain applies on any eligible provider error, not only on a quota-out. - **omp returns to the recorded model by itself.** `retry.fallbackRevertPolicy: cooldown-expiry` restores the original selector at the next prompt once its suppression window ends; it never steps back to an intermediate route, and a primary that is out of quota for days stays suppressed for days. The guard layers a lab-only chain after the tracked overlay (a later `--config` wins) so the fallback route answers slowly enough for the primary's window to end. From e970f4d59163ddcb239b32f1eeb2a865febc0d0b Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Sat, 17 Oct 2026 00:23:55 -0500 Subject: [PATCH 4/4] no-mistakes(document): Document omp Sol-to-Flash run-time fallback exception --- docs/configuration.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 8cf15427546..59a8ee11524 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1691,6 +1691,12 @@ OMP workers receive a later per-task configuration overlay, `state/.omp-fall Those chains contain only the order-preserving eligible native prefix; the first supported cross-harness destination ends the prefix and leaves the transition to recovery. The tracked session overlay `.omp/fm-session-overlay.yml` chains `openai-codex/gpt-6.1-sol` to direct `deepseek/deepseek-v4-flash:high` for every omp launch, so a Sol session moves to the stand-in on any eligible provider error, a quota-out included, and `retry.fallbackRevertPolicy: cooldown-expiry` returns it to Sol once the suppression window ends. A worker's per-task chain replaces that entry for its own model, and is empty when the rule names no eligible native stand-in. +Dispatch-time selection and the overlay chain are separate mechanisms. +Dispatch-time selection keeps the strongest-reasoning class, and a tight choice with no supported equal-class stand-in stops and is reported. +The overlay chain is a run-time provider-error fallback inside one session, not a dispatch decision, and it is a standing exception to that rule for omp Sol sessions. +Its destination is the direct `deepseek` provider only, never an API-billed router such as OpenRouter. +A lane that fell back runs on a lower reasoning class until omp returns it to Sol when the suppression window ends, or until the lane is launched again. +Narrowing the exception means editing the one chain key, `retry.fallbackChains."openai-codex/gpt-6.1-sol"`, in `.omp/fm-session-overlay.yml`. Native OMP account rotation precedes model fallback, and successful native model switches publish the serving selector. An idle terminal quota error after native rotation is handled by the existing bounded recovery scan through `fm-control.sh relaunch`, retaining its pause, captain-call, generation, and work-preservation guards. Launch and successful replacement append the selected route to task status.