fix(omp): chain Sol to direct DeepSeek Flash on provider failure

[MrGTV-love]

Intent

Both Codex accounts reached their weekly limit on 2026-10-16 at about 21:50 CDT and stay out until about 2026-10-22. Fleet lanes that ran on omp with the model openai-codex/gpt-6.1-sol stopped with: "You have hit your ChatGPT usage limit" and "Retry failed after 1 attempts: Provider requested 444099752ms wait, exceeds retry.maxDelayMs (300000ms)". One lane stayed dead for about an hour until the supervisor noticed. The captain's standing rules: never hold work for quota; when a quota is out, switch at once to the dispatch rule's named stand-in; the approved stand-in is direct DeepSeek Flash (omp provider deepseek, model deepseek/deepseek-v4-flash, never through OpenRouter). Measured cause (read-only investigation, report at ~/.claude/jobs/eb3cf121/tmp/omp-fallback-report.md): the tracked overlay .omp/fm-session-overlay.yml (lines 30-40, added by PR #89) sets retry.fallbackChains for openai-codex/gpt-6.1-sol to an empty list on purpose, so with both accounts out and no credential to rotate to, omp had no fallback candidate and the 444099752 ms wait hit the 300000 ms cap. omp 18.11.0 tries credential rotation first, then walks the chain, and only then applies the cap; it watches every --config overlay file and reads the chains fresh at each failure, so an overlay edit reaches live sessions without a restart. The global ~/.omp/agent/config.yml chain (OpenRouter mimo then haiku) is approved and must not change.

What Changed

• .omp/fm-session-overlay.yml now sets retry.fallbackChains for openai-codex/gpt-6.1-sol to deepseek/deepseek-v4-flash:high instead of an empty list. A Sol session whose provider fails, including a usage window that is out, now moves to direct DeepSeek Flash instead of stopping at retry.maxDelayMs. The chain applies on any eligible provider error, not only a quota-out. retry.fallbackRevertPolicy: cooldown-expiry returns the session to Sol once the suppression window ends. The global chain in ~/.omp/agent/config.yml is untouched.
• tests/fm-omp-fallback-chain-live-e2e.test.sh replaces the "empty Sol chain stops the run" check with two checks against the installed omp: one scripted usage_limit_reached answer with a one-hour retry-after, and one 500 answer. Each requires a request to reach deepseek-v4-flash, none to the global chain's deepseek-v4-pro, a successful run, and a live-model record that follows the session to the stand-in. The scripted provider gains usage and error behaviors, and the guard header now cites omp 18.11.0.
• Docs and comments match the new behavior: docs/configuration.md describes the overlay chain as a run-time exception to dispatch-time strongest-class selection (direct deepseek only, never OpenRouter) and names state/<id>.omp-fallback.yml as the per-task overlay. docs/verification/runtime-backends.md records the omp 18.11.0 run, bin/fm-spawn.sh, bin/fm-crew-state.sh, and the omp harness reference are reworded to drop the "stops rather than downgrades" wording.

Risk Assessment

✅ Low: The change is one YAML value (Sol chained to direct DeepSeek Flash) that matches the investigation report, the round-1 fallbackOn removal is complete (no fallbackOn remains in the tree), and the live guard would fail on the old empty chain; the only issue is one doc sentence that claims a lab result the guard does not run.

Testing

I ran the live omp fallback-chain guard against the real installed omp 18.11.0 with local fake providers. No model tokens were spent. All 6 checks passed on the change. The same guard fails on the old empty-chain overlay, so it does catch the incident. I also drove incident-size waits, per-task worker overlays, a Flash outage, and a real chat-completions Flash answer through real omp. All passed. The changes in bin, docs and skill files are comments and docs only. I saw no problems. Scratch files are removed and the worktree is clean. This change has no screenshot surface, so there are CLI transcripts only.

• Live validation: ✅ go - 10 of 10 scenarios driven live against the product

Scenario	Result	Live	Evidence
A Sol session whose usage window is out (429 usage_limit_reached, retry-after 3600 s) moves to direct DeepSeek Flash under the tracked overlay, the run succeeds, and the live-model record follows to d…	✅ pass	live	live-e2e-new-overlay.txt
The 2026-10-16 incident at real size (retry-after 444100 s, maxDelayMs 300000): the new overlay moves to Flash and exits 0; the old empty-chain overlay stops with the ChatGPT usage-limit error	✅ pass	live	scenarios-incident-size-and-worker-overlay.txt (S1, S1-control)
After the move, Flash answers for real: omp prints the Flash chat-completions answer and exits 0	✅ pass	live	scenarios-flash-outage-and-real-answer.txt (S5)
A Sol provider that answers 500 also moves the session to Flash (plain chain, no quota-only gate), and the record follows	✅ pass	live	live-e2e-new-overlay.txt
Regression: the test fails on the old empty-chain overlay and passes on the new plain chain	✅ pass	live	live-e2e-old-empty-chain-overlay.txt vs live-e2e-new-overlay.txt
Adversarial: the global chain (Pro, then the lab equal route) is replaced, so Sol never reaches Pro or the equal route, even when Flash is also out	✅ pass	live	live-e2e-new-overlay.txt; scenarios-flash-outage-and-real-answer.txt (S4)
Another model's global chain still falls back under the overlay (the overlay replaces only the Sol entry; the approved global chain is untouched)	✅ pass	live	live-e2e-new-overlay.txt
Return: with a lab chain layered later, omp returns to the recorded model at the next prompt after the suppression window, even under a global fallbackRevertPolicy: never; the control without the over…	✅ pass	live	live-e2e-new-overlay.txt
Per-task worker overlay: a Sol worker with a Flash stand-in moves to Flash; a Sol worker with no stand-in has the tracked entry replaced by an empty chain and stops, as the docs say	✅ pass	live	scenarios-incident-size-and-worker-overlay.txt (S2, S3)
The bin/fm-spawn.sh header (via --help) describes the plain chain and the per-task replacement, and no fallbackOn text remains anywhere	✅ pass	live	fm-spawn-help-excerpt.txt; grep for fallbackOn found nothing

Evidence: Live e2e on the new overlay (6 ok)

Source: Live e2e on the new overlay (6 ok)

ok - omp (omp/18.11.0): control - the global Sol chain alone still falls to the weak model
ok - omp (omp/18.11.0): the overlay moves a Sol session whose usage window is out to DeepSeek Flash, and the record follows
ok - omp (omp/18.11.0): the overlay moves a Sol session whose provider answers 500 to DeepSeek Flash, and the record follows
ok - omp (omp/18.11.0): the overlay leaves another model's global chain in place
ok - omp (omp/18.11.0): with the overlay a fallen-back session returns to its recorded model at the next prompt even under a global fallbackRevertPolicy: never
ok - omp (omp/18.11.0): control - without the overlay a global fallbackRevertPolicy: never keeps the session on the fallback
env FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh  7.84s user 5.10s system 21% cpu 59.105 total

Evidence: Live e2e on the old empty-chain overlay (fails as the incident did)

Source: Live e2e on the old empty-chain overlay (fails as the incident did)

ok - omp (omp/18.11.0): control - the global Sol chain alone still falls to the weak model
not ok - omp (omp/18.11.0): with the overlay a Sol session whose usage window is out never reached DeepSeek Flash (weak saw: ); output: Working...
You have hit your ChatGPT usage limit (pro plan). retry-after-ms=3600000

Evidence: Incident-size wait and worker-overlay scenarios

Source: Incident-size wait and worker-overlay scenarios

S1 NEW overlay, retry-after 444100 s (incident size), maxDelayMs 300000
    codex requests: 1 | flash requests: 12 | pro requests: 0 | equal requests: 0 | rc=0
    record: model=deepseek/deepseek-v4-flash 
    output: Working...
S1 RESULT: PASS
S1-control OLD empty-chain overlay, same incident
    codex requests: 2 | flash requests: 0 | pro requests: 0 | equal requests: 0 | rc=1
    record: model=openai-codex/gpt-6.1-sol error=You have hit your ChatGPT usage limit (pro plan). retry-after-ms=444100000 
    output: Working... You have hit your ChatGPT usage limit (pro plan). retry-after-ms=444100000
S1-control RESULT: REPRODUCES INCIDENT (no stand-in, run stops)
S2 worker on Sol WITH a dispatch stand-in [deepseek/deepseek-v4-flash:high] (per-task overlay after the tracked one), incident-size wait
{"default":[],"openai-codex/gpt-6.1-sol":["deepseek/deepseek-v4-flash:high"],"openai-codex/gpt-6.1-sol:high":["deepseek/deepseek-v4-flash:high"],"deepseek/deepseek-v4-flash":[],"deepseek/deepseek-v4-flash:high":[]}
    codex requests: 3 | flash requests: 12 | pro requests: 0 | equal requests: 0 | rc=0
    record: model=deepseek/deepseek-v4-flash 
    output: Working...
S2 RESULT: PASS
S3 worker on Sol with NO eligible stand-in (per-task chain empty, replaces the tracked Sol entry)
{"default":[],"openai-codex/gpt-6.1-sol":[],"openai-codex/gpt-6.1-sol:high":[]}
    codex requests: 4 | flash requests: 0 | pro requests: 0 | equal requests: 0 | rc=1
    record: model=openai-codex/gpt-6.1-sol error=You have hit your ChatGPT usage limit (pro plan). retry-after-ms=444100000 
    output: Working... You have hit your ChatGPT usage limit (pro plan). retry-after-ms=444100000
S3 RESULT: per-task empty chain replaces the tracked entry, run stops (matches the docs)
S4 adversarial: a Sol session must never reach an OpenRouter route or the global chain's Pro even when Flash itself is out
    codex requests: 5 | flash requests: 0 | pro requests: 0 | equal requests: 0 | rc=1
    record: model=deepseek/deepseek-v4-flash error=Retry budget exhausted after 2 retries: ECONNREFUSED: Unable to connect. Is the computer able to access the url? 
    output: Working... Retry budget exhausted after 2 retries: ECONNREFUSED: Unable to connect. Is the computer able to access the url?
S4 RESULT: PASS (no hop to Pro or the equal route)

Evidence: Flash outage guard and real Flash answer

Source: Flash outage guard and real Flash answer

S4 adversarial: Flash itself is also out; the Sol session must not hop to the global chain's Pro or the lab equal route
    codex requests: 1 | flash requests: 1 | pro requests: 0 | equal requests: 0 | rc=1
    record: model=deepseek/deepseek-v4-flash error=429 The usage limit has been reached retry-after-ms=444100000 The usage limit has been reached (type=usage_limit_reached) 
    output: Working... 429 The usage limit has been reached retry-after-ms=444100000 The usage limit has been reached (type=usage_limit_reached)
S4 RESULT: PASS (reached Flash, stopped there, no hop to Pro or equal)
S5 Flash serves a real chat-completions answer after Sol's window is out; the session finishes the prompt on Flash
    flash chat server saw: /v1/chat/completions deepseek-v4-flash,
    rc=0 record: model=deepseek/deepseek-v4-flash 
    output: Working... FLASH-ANSWER
S5 RESULT: PASS (answer text came from Flash)

Evidence: fm-spawn --help excerpt with the new header

Source: fm-spawn --help excerpt with the new header

310-  shape, plan mode off, prewalk off, and the non-interactive usage-reserve
311-  policy for the one session only (--auto-approve alone owns approval,
312:  forcing tools.approvalMode: yolo for the session). It also chains the Sol
313-  model to direct DeepSeek Flash, so a Sol session whose provider fails, a
314-  usage window that is out included, moves to the stand-in at once (a worker's
315-  per-task state/<id>.omp-fallback.yml replaces that entry for its own model),
316-  and it pins omp's own return to the recorded model once the primary serves
317-  again.
318-  Each session publishes the model it is serving through
319-  bin/fm-omp-live-model.ts (a crewmate or scout from its per-task extension, a
320-  secondmate from the home's tracked extension), and bin/fm-crew-state.sh

- Outcome: ⚠️ 1 info across 1 run (5m37s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

• ⚠️ .omp/fm-session-overlay.yml:43 - Unrequired component: retry.fallbackOn: usage-limit. The intent asks for one thing: chain openai-codex/gpt-6.1-sol to direct deepseek/deepseek-v4-flash so a quota-out Sol lane switches at once (the investigation report's 'Minimal change' is only the fallbackChains value). It does not ask to restrict fallback to quota-class errors. omp reads fallbackOn for every chain in the session (omp 18.11.0: oB(settings.fallbackOn, isUsageLimit) gates both isHardErrorFallbackEligible and the retry-handler chain walk), and nothing later in the --config stack overrides it. Concrete effects beyond the intent: (1) The intent says the approved global chain (OpenRouter mimo then haiku) must not change; under the overlay it now fires only on a quota-class error, where before it fired on any eligible error. (2) Every worker lane gets the session overlay first, then .omp/fm-worker-overlay.yml, then the per-task state/<id>.omp-fallback.yml (bin/fm-spawn.sh:6091-6097, 6565), and none of them sets fallbackOn. So a worker on Sol with a declared dispatch-matrix stand-in ([deepseek/deepseek-v4-flash:high]) used to move to it on a 5xx or overload burst, and now stops with the provider's error and holds work. The docs for that per-task chain (.agents/skills/harness-adapters/references/harness/omp.md:42, docs/configuration.md ~1686) and the overlay comment name only the global chain, not the worker chains. Smallest honest remedy: remove the fallbackOn line and its supporting text, which leaves the plain chain from the report. If the author wants the earlier PR fix(bin): stop omp Sol sessions falling to weaker models and surface model drift #89 protection against a weak-model strand after an error burst, that is a product decision and needs the captain's approval. The same sites must move together: .omp/fm-session-overlay.yml:36-42 (comment), bin/fm-spawn.sh:313-318 (header text), docs/verification/runtime-backends.md:1747-1750 (bullet; its 402 claim and its 'same chain without the key moved the session' claim are not exercised by the guard, which has no 402 case and no control without the key), tests/fm-omp-fallback-chain-live-e2e.test.sh:13 and :20-22 (header), :96-97 (error server behavior), :230-246 (500 scenario), :249-250 (serve weak usage 3600 for the other-model case).

🔧 Fix applied.
1 info still open:

• ℹ️ docs/verification/runtime-backends.md:1744 - Unexercised lab claim in a "Verified on 2026-10-16 ... omp 18.11.0" entry. docs/verification/runtime-backends.md:1744 says "An empty Sol chain stopped the same run at the wait cap with the provider's usage-limit error". The guard has no empty-chain run. Its six checks are: control with no overlay and a limit server, overlay with usage, overlay with error, the other-model case, the return case, and the no-overlay never control. So the sentence reads as a lab result that no test reproduces. This is the same class the round-1 decision asked to drop or label as stated, not tested, and the round-1 fix left a new instance of it behind. Smallest fix: reword the sentence to cite the 2026-10-16 incident and the investigation report as the source, or label it as not exercised by the guard. Adding an empty-chain control to the guard would extend the change, so it is not needed. Other sites checked: the .omp/fm-session-overlay.yml:32-34 comment and the tests/fm-omp-fallback-chain-live-e2e.test.sh:9-11 header describe the incident, not a lab result, so they are fine.

⚠️ **Test** - 1 info

• ℹ️ bin/fm-spawn.sh:6075 - A worker on Sol whose dispatch rule names no eligible native stand-in still stops on a quota-out. Its per-task overlay, state/<id>.omp-fallback.yml, loads after the tracked overlay and sets the Sol chain to []. This is the documented design (docs/configuration.md says the chain is empty when the rule names no stand-in), and it matches the recorded decision. The new tracked chain covers every launch with no per-task Sol entry. A Sol worker whose rule names no stand-in is covered only if the operator adds direct DeepSeek Flash to that dispatch rule.
• Live validation: ✅ go - 10 of 10 scenarios driven live against the product

Scenario	Result	Live	Evidence
A Sol session whose usage window is out (429 usage_limit_reached, retry-after 3600 s) moves to direct DeepSeek Flash under the tracked overlay, the run succeeds, and the live-model record follows to d…	✅ pass	live	live-e2e-new-overlay.txt
The 2026-10-16 incident at real size (retry-after 444100 s, maxDelayMs 300000): the new overlay moves to Flash and exits 0; the old empty-chain overlay stops with the ChatGPT usage-limit error	✅ pass	live	scenarios-incident-size-and-worker-overlay.txt (S1, S1-control)
After the move, Flash answers for real: omp prints the Flash chat-completions answer and exits 0	✅ pass	live	scenarios-flash-outage-and-real-answer.txt (S5)
A Sol provider that answers 500 also moves the session to Flash (plain chain, no quota-only gate), and the record follows	✅ pass	live	live-e2e-new-overlay.txt
Regression: the test fails on the old empty-chain overlay and passes on the new plain chain	✅ pass	live	live-e2e-old-empty-chain-overlay.txt vs live-e2e-new-overlay.txt
Adversarial: the global chain (Pro, then the lab equal route) is replaced, so Sol never reaches Pro or the equal route, even when Flash is also out	✅ pass	live	live-e2e-new-overlay.txt; scenarios-flash-outage-and-real-answer.txt (S4)
Another model's global chain still falls back under the overlay (the overlay replaces only the Sol entry; the approved global chain is untouched)	✅ pass	live	live-e2e-new-overlay.txt
Return: with a lab chain layered later, omp returns to the recorded model at the next prompt after the suppression window, even under a global fallbackRevertPolicy: never; the control without the over…	✅ pass	live	live-e2e-new-overlay.txt
Per-task worker overlay: a Sol worker with a Flash stand-in moves to Flash; a Sol worker with no stand-in has the tracked entry replaced by an empty chain and stops, as the docs say	✅ pass	live	scenarios-incident-size-and-worker-overlay.txt (S2, S3)
The bin/fm-spawn.sh header (via --help) describes the plain chain and the per-task replacement, and no fallbackOn text remains anywhere	✅ pass	live	fm-spawn-help-excerpt.txt; grep for fallbackOn found nothing

• FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh on the change: all 6 checks pass with real omp 18.11.0 and fake local providers (rc=0)
• The same live test with the old base-commit overlay (empty Sol chain) put back for one run: fails at the usage-window check, then the worktree was restored (git status clean)
• A scratch driver with an incident-size wait (retry-after 444100 s, wait cap 300000 ms): new overlay moves to Flash; old overlay stops with the usage-limit error
• A scratch driver with a per-task worker overlay built by the same jq program as bin/fm-spawn.sh: Sol with Flash stand-in moves to Flash; Sol with no stand-in stops
• A scratch driver where Flash is also out: the session stops at Flash and never reaches the global chain's Pro or the OpenRouter-style equal route
• A scratch driver with a real chat-completions Flash server: omp prints the Flash answer text and exits 0
• bin/fm-spawn.sh --help shows the new header text
• grep -rn fallbackOn . finds no references

⚠️ **Document** - 1 info

• ⚠️ docs/configuration.md:1692 - Docs now disagree about downgrades. The tracked overlay moves every Sol session to DeepSeek Flash on any eligible provider error. This is the new text at docs/configuration.md:1692. Other docs still say strongest-class work stops and reports instead of downgrading: docs/configuration.md:1674-1675, .agents/skills/quota-array-dispatch/SKILL.md:65 and :116, and AGENTS.md:105. The overlay behavior matches the captain's rule to switch to direct DeepSeek Flash at once, so I left all of it as written. AGENTS.md is agent memory, and I may only fix facts that are wrong there. Captain decision needed: either say in the dispatch docs that the overlay's Sol-to-Flash chain is a standing exception to the strongest-class rule, or narrow the overlay. Pick the first or second option and apply it in the one owner doc.

🔧 Fix applied.
1 info still open:

• ℹ️ docs/configuration.md:1698 - The instruction said a lane that fell back stays on the lower class 'until relaunched'. The same doc (line 1692) and the overlay say cooldown-expiry returns the session to Sol once the suppression window ends. I wrote both exits: 'until omp returns it to Sol when the suppression window ends, or until the lane is launched again'. This follows the code and the recorded review decision to state the cooldown-expiry return.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.