Repository navigation
fix(bin): stop omp Sol sessions falling to weaker models and surface model drift - #89
Merged
Merged
Conversation
…rift An omp session recorded on gpt-6.1-sol fell down a global fallback chain to a weaker model and stayed there. The tracked session overlay now empties the Sol chain so a failure stops with the provider error, and pins omp's own return to the recorded model. Sessions publish the model they serve, and fm-crew-state reports model-drift and run-error.
This was referenced Oct 11, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Owner words 2026-10-05: "we have a model matrix, so why not switch models accordingly to the fallback??"
Rule (shared preference record, reconfirmed 2026-10-06): strongest-class work drops only to its named equal, never to a weaker model. On 2026-10-06 the owner ruled that Sonnet 5.5 and gpt-6.1-sol are equals, that Claude runs only through TeamClaude (tc run), and that DeepSeek V4 Flash is the stand-in for Sonnet-class easy work only. Also 2026-10-06: every agent role selects its model dynamically from live quota; a static pin is a defect.
Incident 2026-10-09, 03:23-03:34 local: a burst of "401 User not found" made the vernant-d7-steward lane (recorded openai-codex/gpt-6.1-sol) fall to openrouter/deepseek/deepseek-v4-flash. It stayed there after the burst, and the lane then needed about ten manual wake nudges. The 401 source (a dead OpenRouter key) was already removed.
Needed: (1) a Sol session's fallback ends at its equal and never at DeepSeek V4 Flash; when every equal fails, stop and report. (2) A fallen-back session returns to its recorded model once the primary serves again, or the supervisor gets a visible signal that it did not.
Decisions already made in this change: the tracked session overlay empties the Sol fallback chain (stop and report, no weaker model, no OpenRouter route since the fleet rule forbids openrouter/ in omp chains) and pins fallbackRevertPolicy cooldown-expiry; omp sessions publish their live model to a record that fm-crew-state shows as model-drift and run-error. The owner's global omp config is deliberately NOT edited here (owner decision pending; a proposal file exists outside the repo). Related: PR 68 (firstmate code never writes the global omp config; workers get settings through the tracked --config overlay).
What Changed
.omp/fm-session-overlay.ymlnow setsretry.fallbackChainsforopenai-codex/gpt-6.1-solto an empty list. omp replaces an array whole and merges overlays key by key, so this one entry replaces only the global Sol chain. A failing Sol run now stops with the provider error. It does not move to a weaker model. The overlay also pinsretry.fallbackRevertPolicy: cooldown-expiry. With this setting, a session on any other chain goes back to its recorded model at the next prompt after the primary's suppression window ends.bin/fm-omp-live-model.tswrites a live-model record (model=and, after an unrecovered run error,error=). It writes the record atomically onsession_start,agent_start,retry_fallback_applied, andagent_end. Crewmate and scout per-task extensions fromfm-spawn.shinstall it and write tostate/<id>.live-model. The secondmate turn-end guard installs it too and writes tostate/.omp-live-model, but only when that session owns the lock.fm-spawn.shdeletes a stale record at each launch.fm-teardown.shdeletes the record with the other per-task files.bin/fm-crew-state.shreads that record for local omp tasks. It addsmodel-drift: <live> live (recorded <model>)when the live model is not the recordedprovider/id, without the thinking suffix. It addsrun-error: <message>when the last run ended in an error that nothing recovered. This change also adds a token-free live e2e guard against omp 18.8.7 (tests/fm-omp-fallback-chain-live-e2e.test.sh, registered infm-test-run.sh), new unit cases infm-crew-stateandfm-omp-harness, and a verification entry indocs/verification/runtime-backends.md.Risk Assessment
✅ Low: The change is well-bounded and matches the stated decisions. The overlay empties only the Sol chain and pins cooldown-expiry. The publisher writes the record atomically, and only the lock owner writes in a home. The appended crew-state components have the separator stripped, and every parser checked (crew_absorb_class, crew_gate_awaits_human_decision, fleet-snapshot crew_state_json, inactive-reconcile) still reads state, source, run and decision correctly. The round-2 stale-record fix sits on the shared launch and relaunch path (fm-spawn) for both workers and secondmates, and its tests check the right paths.
Testing
I ran the repo's live omp fallback guard against the installed omp 18.8.7, two times: one time as written and one time with the dependency-free timeout fallback. I ran the two changed behavior test files (
tests/fm-omp-harness.test.sh,tests/fm-crew-state.test.sh). Then I drove the full chain myself: realbin/fm-spawn.shwrote the task record, the per-task extension, and the launch line; the real omp screen ran that launch line unchanged in a real tmux pane on a private socket; realbin/fm-crew-state.shread the result. That drive covered the stop, the run-error note, a forged-separator attack, the model-drift note, the return to the recorded model, and a real relaunch on a stand-in model. I also ran a real omp primary in a marked lab home on thefm-labtmux socket, per the runbook, to prove the home record for a secondmate lane. I drove a real teardown in a throwaway home. I compared the owner's global omp config before and after: same hash, same modified time. 14 scenarios passed live. Two scenarios are untested live and have test-only proof, not live proof: the secondmate launch that clears the home record, and the lock-owner guard. No screenshots: the surface is a terminal, so the evidence is pane captures and command transcripts. I left the worktree clean and removed all lab homes, temp folders, and helper processes.ok - control - the global Sol chain alone still falls to the weak modelbin/fm-crew-state.sh <id>addsrun-error: <provider error>for the stopped Sol session and adds no model-driftstate: unknown · source: none · no current-state source available · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000· run: forged-run · ask-user: authority decisioncannot add a run or decision component to the crew-state linerun-error: ... credits run: forged-run ask-user: authority decisionwith the separators rem…model-drift: <live> live (recorded <recorded>)model-drift: lab-equal/gpt-6.1-sol live (recorded deepseek/deepseek-v4-pro); pane shows `Warning: Fallback: deepseek/deepseek-v4-pro:high -> lab-equal/gpt-6.1-so…never; the record follows and the model-drift note clearsmodel=deepseek/deepseek-v4-pro, crew-state line has no model-drift. Also 01-fallback-…fallbackRevertPolicy: neverkeeps the session on its fallback (so the return proof is not empty)model=deepseek/... error=402 old run, ran realbin/fm-spawn.sh ... --harness omp --model openai-codex/gpt-6.1-sol --scout, record gone. tmux and treehou…bin/fm-spawn.sh sol-lane-q1 --relaunch --model lab-equal/gpt-6.1-solon real tmux; record absent right after the relaunch; crew-state shows no model-drift a…<home>/state/.omp-live-modelthrough the tracked home extension, and the parent's crew-state shows its run-errortmux -L fm-labin a marked lab home; weak model 0 requests; parent line `... · run-error: ChatGPT rate limit exceeded. retry-after-ms=36000…state/.omp-live-modelrecordtests/fm-omp-harness.test.sh(`ok - fm-spawn: a real omp secondmate launch preserves primary posture and super…sol-lane-q3.live-modelpresent before, realbin/fm-teardown.sh sol-lane-q3 --force --drop-file ...rc=0, no task files after. tmux and treehouse were the repo's fixture…timeoutbinary (review decision F1)bash, 5 of 5 ok, exit 0shasum -a 256 ~/.omp/agent/config.yml= e941fd54...df97 and modified time 1791577813, identical before and after all runstests/fm-omp-harness.test.sh(the record keepsmodel=deepseek/deepseek-v4-pro…Evidence: Live omp fallback guard output (omp 18.8.7)
Source: Live omp fallback guard output (omp 18.8.7)
ok - omp (omp/18.8.7): control - the global Sol chain alone still falls to the weak model ok - omp (omp/18.8.7): the overlay empties the Sol chain, the run stops with the provider error, and the record carries it ok - omp (omp/18.8.7): the overlay leaves another model's global chain in place ok - omp (omp/18.8.7): with the overlay a fallen-back session returns to its recorded model at the next prompt even under a global fallbackRevertPolicy: never ok - omp (omp/18.8.7): control - without the overlay a global fallbackRevertPolicy: never keeps the session on the fallbackEvidence: Full chain transcript: real fm-spawn -> real omp TUI in tmux -> real fm-crew-state (steps A to G)
Source: Full chain transcript: real fm-spawn -> real omp TUI in tmux -> real fm-crew-state (steps A to G)
B. codex: gpt-6.1-sol refused-429-usage-limit | weak: 0 requests | equal: 0 requests record: model=openai-codex/gpt-6.1-sol / error=ChatGPT rate limit exceeded. retry-after-ms=3600000 C. state: unknown · source: none · no current-state source available · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000 D. state: unknown · source: none · no current-state source available · run-error: This request would exceed your available credits run: forged-run ask-user: authority decision E. state: unknown · source: none · no current-state source available · model-drift: lab-equal/gpt-6.1-sol live (recorded deepseek/deepseek-v4-pro) F. record: model=deepseek/deepseek-v4-pro | state: working · source: pane · harness busy (omp-ext) G. relaunch rc=0 | record right after: (absent) | later: model=lab-equal/gpt-6.1-sol | no model-drift, no run-error RESULT: 0 failed checksEvidence: Driver script for the full chain transcript
Source: Driver script for the full chain transcript
Evidence: Scripted local model provider used by the drivers (no tokens spent)
Source: Scripted local model provider used by the drivers (no tokens spent)
Evidence: Lab-home primary transcript: home record and the parent's crew-state line (steps H1, H2)
Source: Lab-home primary transcript: home record and the parent's crew-state line (steps H1, H2)
home record <lab-home>/state/.omp-live-model: model=openai-codex/gpt-6.1-sol error=ChatGPT rate limit exceeded. retry-after-ms=3600000 weak (deepseek, the global chain target): 0 requests $ bin/fm-crew-state.sh mate state: unknown · source: none · backend target gone: fm:fm-mate · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000 RESULT: 0 failed checksEvidence: Teardown transcript: the live-model record is removed
Source: Teardown transcript: the live-model record is removed
Evidence: Live guard with the dependency-free timeout fallback (no timeout binary)
Source: Live guard with the dependency-free timeout fallback (no timeout binary)
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 7 issues found → auto-fixed (2) ✅
tests/fm-omp-fallback-chain-live-e2e.test.sh:171- run_omp wraps omp in baretimeout 240, but the gate only requiresomp python3(line 32:fm_live_gate default-on FM_OMP_FALLBACK_LIVE omp python3). Stock macOS has notimeout; Homebrew coreutils only installsgtimeoutunless gnubin is on PATH. The test is default-on, so a Mac that has omp and python3 but notimeoutruns it. Each run_omp call then exits 127, and scenario 1 fails with a misleading message: 'the global chain no longer reaches the weak model ... the lab does not reproduce the incident'. It does not skip. The docs also say it 'runs wherever omp and python3 are installed' (docs/verification/runtime-backends.md:1447). Fix: source bin/fm-timeout-lib.sh and use its portablefm_run_timed 240 omp ..., as other bin callers do. Or addtimeoutto the fm_live_gate tool list so a missing tool skips cleanly.bin/fm-spawn.sh:268- The new spawn header comment says the overlay 'ends the Sol model's fallback chain at its equal, the same model on another route'. The overlay actually empties the chain (openai-codex/gpt-6.1-sol: []), so the session stops with no other route. The overlay's own comment (.omp/fm-session-overlay.yml:30-38) and the docs (runtime-backends.md:1437, 'the equal is not a chain hop') both say this. The comment describes the rejected design, which keeps an equal-route hop that the fleet rule forbids. Fix: reword it to say the overlay empties the Sol chain, so a failing Sol run stops and reports through the live-model record.docs/verification/runtime-backends.md:1427- The verified omp version does not match across the change. The docs section and its sample output say omp 18.8.7. The overlay comment (.omp/fm-session-overlay.yml:11, 'against omp 18.8.1'), the publisher header (bin/fm-omp-live-model.ts:13, 'verified on omp 18.8.1'), and the harness test comment (tests/fm-omp-harness.test.sh:830, 'verified on omp 18.8.1') name 18.8.1. Fix: make all the citations name the version the guard actually ran against.docs/verification/runtime-backends.md:1436- The docs list this as verified: 'the overlay's empty Sol entry replaced the global Sol chain and left every other chain alone'. The overlay comment says the same (.omp/fm-session-overlay.yml:34-36). The guard never checks the second half. No scenario puts a non-Sol chain in the global config.yml and confirms it still falls back with the overlay. Suppose omp replaced the wholefallbackChainsmap instead of merging it key by key. Then every non-Sol omp session would lose its fallback and stop. That change is outside the intent, and the guard would still print all four 'ok' lines. Fix: add a non-Sol chain to agent_dir's config.yml and one assertion that it still reaches its fallback under the overlay. Otherwise, drop the 'left every other chain alone' claim from the verified list.bin/fm-omp-live-model.ts:35- The run-error text is free-form provider output. It is put into one line, but the crew-state component separator ' · ' is not escaped. bin/fm-crew-state.sh:285 appends it asrun-error: <message>to every emitted line. crew_gate_awaits_human_decision (bin/fm-classify-lib.sh:918-930) splits the line on ' · '. It also takes the LASTrun: <id>component and checks for an exactask-user: authority decisioncomponent. Its comment says note text must never be able to mint that component. A provider error that contains ' · run: X' would replace the gate's run id. Fix: in unrecoveredError, replace '·' (U+00B7) in the message before publishing. Alternatively, strip it in crew-state where LIVE_NOTE is built (bin/fm-crew-state.sh:285).bin/fm-omp-live-model.ts:53- Simplification: the record publishes asince=<epoch>field, and the publisher tracks it in its own state (lines 45 and 67-69). bin/fm-crew-state.sh reads onlymodelanderror. No consumer, test assertion, or intent requirement usessince. Intent items (1) and (2) need only the live model and an unrecovered error. Recommended remedy: remove thesincefield and its tracking. If a consumer is planned, keep it on purpose.bin/fm-crew-state.sh:255- Simplification: the new record_value_to is a line-for-line copy of meta_value_to (bin/fm-crew-state.sh:236-244). The only difference is that it takes the file as a parameter instead of reading $META. Fix: keep one reader (record_value_to) and make meta_value_to callrecord_value_to "$1" "$META" "$2". This keeps one definition of the last-value-wins key=value rule.🔧 Fix applied.
2 infos still open:
bin/fm-spawn.sh:268- The new spawn header comment says the overlay 'ends the Sol model's fallback chain at its equal, the same model on another route'. The overlay actually empties the chain (openai-codex/gpt-6.1-sol: []), so the session stops with no other route. The overlay's own comment (.omp/fm-session-overlay.yml:30-38) and the docs (runtime-backends.md:1437, 'the equal is not a chain hop') both say this. The comment describes the rejected design, which keeps an equal-route hop that the fleet rule forbids. Fix: reword it to say the overlay empties the Sol chain, so a failing Sol run stops and reports through the live-model record.bin/fm-spawn.sh:5532- A relaunch does not clear the old live-model record. Only teardown removes it (bin/fm-teardown.sh:3774 and :4272). Sequence: a Sol session stops with a run error, so the record holdsmodel=openai-codex/gpt-6.1-solanderror=.... The supervisor relaunches the task on a stand-in model, which is the planned recovery. The meta now records the stand-in. Until the new omp session publishes its first event with a model, bin/fm-crew-state.sh:279-282 prints a falsemodel-drift: openai-codex/gpt-6.1-sol live (recorded <stand-in>)and the oldrun-error. If the new omp never starts, the false note stays. Sibling site: the secondmate record<home>/state/.omp-live-model(read at bin/fm-crew-state.sh:264) is also not cleared at a secondmate launch or relaunch. Fix: remove$STATE/$ID.live-modelwhere spawn writes the per-task omp extension, and remove<home>/state/.omp-live-modelat a secondmate omp launch.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
ok - control - the global Sol chain alone still falls to the weak modelbin/fm-crew-state.sh <id>addsrun-error: <provider error>for the stopped Sol session and adds no model-driftstate: unknown · source: none · no current-state source available · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000· run: forged-run · ask-user: authority decisioncannot add a run or decision component to the crew-state linerun-error: ... credits run: forged-run ask-user: authority decisionwith the separators rem…model-drift: <live> live (recorded <recorded>)model-drift: lab-equal/gpt-6.1-sol live (recorded deepseek/deepseek-v4-pro); pane shows `Warning: Fallback: deepseek/deepseek-v4-pro:high -> lab-equal/gpt-6.1-so…never; the record follows and the model-drift note clearsmodel=deepseek/deepseek-v4-pro, crew-state line has no model-drift. Also 01-fallback-…fallbackRevertPolicy: neverkeeps the session on its fallback (so the return proof is not empty)model=deepseek/... error=402 old run, ran realbin/fm-spawn.sh ... --harness omp --model openai-codex/gpt-6.1-sol --scout, record gone. tmux and treehou…bin/fm-spawn.sh sol-lane-q1 --relaunch --model lab-equal/gpt-6.1-solon real tmux; record absent right after the relaunch; crew-state shows no model-drift a…<home>/state/.omp-live-modelthrough the tracked home extension, and the parent's crew-state shows its run-errortmux -L fm-labin a marked lab home; weak model 0 requests; parent line `... · run-error: ChatGPT rate limit exceeded. retry-after-ms=36000…state/.omp-live-modelrecordtests/fm-omp-harness.test.sh(`ok - fm-spawn: a real omp secondmate launch preserves primary posture and super…sol-lane-q3.live-modelpresent before, realbin/fm-teardown.sh sol-lane-q3 --force --drop-file ...rc=0, no task files after. tmux and treehouse were the repo's fixture…timeoutbinary (review decision F1)bash, 5 of 5 ok, exit 0shasum -a 256 ~/.omp/agent/config.yml= e941fd54...df97 and modified time 1791577813, identical before and after all runstests/fm-omp-harness.test.sh(the record keepsmodel=deepseek/deepseek-v4-pro…FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh(real omp 18.8.7, scripted providers): 5 of 5 okFM_TIMEOUT_MECHANISM_OVERRIDE=bash FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh(no timeout or gtimeout binary used): 5 of 5 oktests/fm-omp-harness.test.sh: exit 0, includes the spawn stale-record step, the secondmate launch stale-record step, the publisher lifecycle, and the lock-owner guardtests/fm-crew-state.test.sh: exit 0, includesomp live-model record adds model-drift and run-error only on positive evidencebash 02-live-chain-driver.sh(evidence dir): realbin/fm-spawn.sh-> launch line run unchanged in a real tmux pane with the real omp TUI -> realbin/fm-crew-state.sh; steps A to G, 0 failed checksbash 03-lab-primary-driver.sh(evidence dir):bin/fm-lab-home.sh create, real omp as the session command ontmux -L fm-lab, tracked.omp/extensionsdiscovered by omp, thenbin/fm-crew-state.sh matefrom a parent home; 0 failed checksbash 04-teardown-driver.sh --force --drop-file <fixture file>(evidence dir): realbin/fm-spawn.shthen realbin/fm-teardown.shin a throwaway home; record removedshasum -a 256 ~/.omp/agent/config.ymlandstat -f %mbefore and after all runs: identical (read-only check)git status --shortafter all runs: clean at 03042cfc.agents/skills/stuck-crewmate-recovery/SKILL.md:35- No agent skill tells the supervisor what to do whenbin/fm-crew-state.shprintsmodel-driftorrun-error. The script header in bin/fm-crew-state.sh owns the meaning of the two components, and that is accurate. The gap is the action: relaunch on a stand-in model after a Sol run stops. I did not add this, because new agent guidance is a human decision and nothing existing is wrong. Recommendation (confidence: medium): add one short pointer in the stuck-crewmate-recovery skill in a follow-up change.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.