Skip to content

fix(bin): stop omp Sol sessions falling to weaker models and surface model drift - #89

Merged
MrGTV-love merged 5 commits into
mainfrom
fm/fm-omp-sol-fallback-downgrade-r2
Oct 10, 2026
Merged

MrGTV-love merged 5 commits into
mainfrom
fm/fm-omp-sol-fallback-downgrade-r2

Conversation

@MrGTV-love

Copy link
Copy Markdown
Owner

Intent

Owner words 2026-10-05: "we have a model matrix, so why not switch models accordingly to the fallback??"

Rule (shared preference record, reconfirmed 2026-10-06): strongest-class work drops only to its named equal, never to a weaker model. On 2026-10-06 the owner ruled that Sonnet 5.5 and gpt-6.1-sol are equals, that Claude runs only through TeamClaude (tc run), and that DeepSeek V4 Flash is the stand-in for Sonnet-class easy work only. Also 2026-10-06: every agent role selects its model dynamically from live quota; a static pin is a defect.

Incident 2026-10-09, 03:23-03:34 local: a burst of "401 User not found" made the vernant-d7-steward lane (recorded openai-codex/gpt-6.1-sol) fall to openrouter/deepseek/deepseek-v4-flash. It stayed there after the burst, and the lane then needed about ten manual wake nudges. The 401 source (a dead OpenRouter key) was already removed.

Needed: (1) a Sol session's fallback ends at its equal and never at DeepSeek V4 Flash; when every equal fails, stop and report. (2) A fallen-back session returns to its recorded model once the primary serves again, or the supervisor gets a visible signal that it did not.

Decisions already made in this change: the tracked session overlay empties the Sol fallback chain (stop and report, no weaker model, no OpenRouter route since the fleet rule forbids openrouter/ in omp chains) and pins fallbackRevertPolicy cooldown-expiry; omp sessions publish their live model to a record that fm-crew-state shows as model-drift and run-error. The owner's global omp config is deliberately NOT edited here (owner decision pending; a proposal file exists outside the repo). Related: PR 68 (firstmate code never writes the global omp config; workers get settings through the tracked --config overlay).

What Changed

  • .omp/fm-session-overlay.yml now sets retry.fallbackChains for openai-codex/gpt-6.1-sol to an empty list. omp replaces an array whole and merges overlays key by key, so this one entry replaces only the global Sol chain. A failing Sol run now stops with the provider error. It does not move to a weaker model. The overlay also pins retry.fallbackRevertPolicy: cooldown-expiry. With this setting, a session on any other chain goes back to its recorded model at the next prompt after the primary's suppression window ends.
  • The new bin/fm-omp-live-model.ts writes a live-model record (model= and, after an unrecovered run error, error=). It writes the record atomically on session_start, agent_start, retry_fallback_applied, and agent_end. Crewmate and scout per-task extensions from fm-spawn.sh install it and write to state/<id>.live-model. The secondmate turn-end guard installs it too and writes to state/.omp-live-model, but only when that session owns the lock. fm-spawn.sh deletes a stale record at each launch. fm-teardown.sh deletes the record with the other per-task files.
  • bin/fm-crew-state.sh reads that record for local omp tasks. It adds model-drift: <live> live (recorded <model>) when the live model is not the recorded provider/id, without the thinking suffix. It adds run-error: <message> when the last run ended in an error that nothing recovered. This change also adds a token-free live e2e guard against omp 18.8.7 (tests/fm-omp-fallback-chain-live-e2e.test.sh, registered in fm-test-run.sh), new unit cases in fm-crew-state and fm-omp-harness, and a verification entry in docs/verification/runtime-backends.md.

Risk Assessment

✅ Low: The change is well-bounded and matches the stated decisions. The overlay empties only the Sol chain and pins cooldown-expiry. The publisher writes the record atomically, and only the lock owner writes in a home. The appended crew-state components have the separator stripped, and every parser checked (crew_absorb_class, crew_gate_awaits_human_decision, fleet-snapshot crew_state_json, inactive-reconcile) still reads state, source, run and decision correctly. The round-2 stale-record fix sits on the shared launch and relaunch path (fm-spawn) for both workers and secondmates, and its tests check the right paths.

Testing

I ran the repo's live omp fallback guard against the installed omp 18.8.7, two times: one time as written and one time with the dependency-free timeout fallback. I ran the two changed behavior test files (tests/fm-omp-harness.test.sh, tests/fm-crew-state.test.sh). Then I drove the full chain myself: real bin/fm-spawn.sh wrote the task record, the per-task extension, and the launch line; the real omp screen ran that launch line unchanged in a real tmux pane on a private socket; real bin/fm-crew-state.sh read the result. That drive covered the stop, the run-error note, a forged-separator attack, the model-drift note, the return to the recorded model, and a real relaunch on a stand-in model. I also ran a real omp primary in a marked lab home on the fm-lab tmux socket, per the runbook, to prove the home record for a secondmate lane. I drove a real teardown in a throwaway home. I compared the owner's global omp config before and after: same hash, same modified time. 14 scenarios passed live. Two scenarios are untested live and have test-only proof, not live proof: the secondmate launch that clears the home record, and the lock-owner guard. No screenshots: the surface is a terminal, so the evidence is pane captures and command transcripts. I left the worktree clean and removed all lab homes, temp folders, and helper processes.

  • Live validation: ✅ go - 14 of 16 scenarios driven live against the product
Scenario Result Live Evidence
Control: with the operator's old Sol chain and no overlay, a Sol session whose model refuses lands on the weaker model (the incident reproduces) ✅ pass live 01-fallback-chain-live-e2e.txt, line 1: ok - control - the global Sol chain alone still falls to the weak model
A Sol session launched by fm-spawn stops when its model refuses: the weaker model gets zero requests, no other route gets a request, and the omp screen shows the provider error ✅ pass live 02-live-chain-transcript.txt step B (real omp TUI in a real tmux pane, launch line from real fm-spawn): codex 1 request refused, weak 0 requests, equal 0 requests, record `model=openai-codex/gpt-6.1-s…
The supervisor sees the stop: bin/fm-crew-state.sh &lt;id&gt; adds run-error: &lt;provider error&gt; for the stopped Sol session and adds no model-drift ✅ pass live 02-live-chain-transcript.txt step C: state: unknown · source: none · no current-state source available · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000
Attack: a provider error that contains · run: forged-run · ask-user: authority decision cannot add a run or decision component to the crew-state line ✅ pass live 02-live-chain-transcript.txt step D: real omp put the forged text in the record; the crew-state line shows run-error: ... credits run: forged-run ask-user: authority decision with the separators rem…
The overlay empties only the Sol chain: a session on another model still uses its own global chain when its model refuses ✅ pass live 01-fallback-chain-live-e2e.txt line 3; 02-live-chain-transcript.txt step E (weak refused once, the chain route answered)
A session that moved to another model is visible: crew-state adds model-drift: &lt;live&gt; live (recorded &lt;recorded&gt;) ✅ pass live 02-live-chain-transcript.txt step E: model-drift: lab-equal/gpt-6.1-sol live (recorded deepseek/deepseek-v4-pro); pane shows `Warning: Fallback: deepseek/deepseek-v4-pro:high -> lab-equal/gpt-6.1-so…
A fallen-back session returns to its recorded model at the next prompt, even when the global policy says never; the record follows and the model-drift note clears ✅ pass live 02-live-chain-transcript.txt step F: second prompt typed into the real omp pane, recorded route answered, record model=deepseek/deepseek-v4-pro, crew-state line has no model-drift. Also 01-fallback-…
Control: without the overlay, a global fallbackRevertPolicy: never keeps the session on its fallback (so the return proof is not empty) ✅ pass live 01-fallback-chain-live-e2e.txt line 5
A fresh launch removes a stale live-model record for that task ✅ pass live 02-live-chain-transcript.txt step A: seeded model=deepseek/... error=402 old run, ran real bin/fm-spawn.sh ... --harness omp --model openai-codex/gpt-6.1-sol --scout, record gone. tmux and treehou…
A relaunch on a stand-in model does not show the old model or the old error ✅ pass live 02-live-chain-transcript.txt step G: real bin/fm-spawn.sh sol-lane-q1 --relaunch --model lab-equal/gpt-6.1-sol on real tmux; record absent right after the relaunch; crew-state shows no model-drift a…
A secondmate-style omp primary in a lab home publishes &lt;home&gt;/state/.omp-live-model through the tracked home extension, and the parent's crew-state shows its run-error ✅ pass live 03-lab-primary-transcript.txt steps H1 and H2: real omp on tmux -L fm-lab in a marked lab home; weak model 0 requests; parent line `... · run-error: ChatGPT rate limit exceeded. retry-after-ms=36000…
A secondmate omp launch removes the home's stale state/.omp-live-model record ⏸️ untested no The prior payload did not establish a live result for this scenario. Its only proof was tests/fm-omp-harness.test.sh (`ok - fm-spawn: a real omp secondmate launch preserves primary posture and super…
Teardown removes the task's live-model record ✅ pass live 04-teardown-transcript.txt: sol-lane-q3.live-model present before, real bin/fm-teardown.sh sol-lane-q3 --force --drop-file ... rc=0, no task files after. tmux and treehouse were the repo's fixture…
The live guard runs on a Mac that has no timeout binary (review decision F1) ✅ pass live 05-fallback-guard-no-timeout-binary.txt: mechanism bash, 5 of 5 ok, exit 0
The owner's global omp config is not edited by any of these runs ✅ pass live shasum -a 256 ~/.omp/agent/config.yml = e941fd54...df97 and modified time 1791577813, identical before and after all runs
A descendant omp process that does not own the session lock cannot overwrite the home's live-model record ⏸️ untested no The prior payload did not establish a live result for this scenario. Its only proof was the turn-end extension case in tests/fm-omp-harness.test.sh (the record keeps model=deepseek/deepseek-v4-pro…
Evidence: Live omp fallback guard output (omp 18.8.7)

Source: Live omp fallback guard output (omp 18.8.7)

ok - omp (omp/18.8.7): control - the global Sol chain alone still falls to the weak model ok - omp (omp/18.8.7): the overlay empties the Sol chain, the run stops with the provider error, and the record carries it ok - omp (omp/18.8.7): the overlay leaves another model's global chain in place ok - omp (omp/18.8.7): with the overlay a fallen-back session returns to its recorded model at the next prompt even under a global fallbackRevertPolicy: never ok - omp (omp/18.8.7): control - without the overlay a global fallbackRevertPolicy: never keeps the session on the fallback

ok - omp (omp/18.8.7): control - the global Sol chain alone still falls to the weak model
ok - omp (omp/18.8.7): the overlay empties the Sol chain, the run stops with the provider error, and the record carries it
ok - omp (omp/18.8.7): the overlay leaves another model's global chain in place
ok - omp (omp/18.8.7): with the overlay a fallen-back session returns to its recorded model at the next prompt even under a global fallbackRevertPolicy: never
ok - omp (omp/18.8.7): control - without the overlay a global fallbackRevertPolicy: never keeps the session on the fallback
Evidence: Full chain transcript: real fm-spawn -> real omp TUI in tmux -> real fm-crew-state (steps A to G)

Source: Full chain transcript: real fm-spawn -> real omp TUI in tmux -> real fm-crew-state (steps A to G)

B. codex: gpt-6.1-sol refused-429-usage-limit | weak: 0 requests | equal: 0 requests record: model=openai-codex/gpt-6.1-sol / error=ChatGPT rate limit exceeded. retry-after-ms=3600000 C. state: unknown · source: none · no current-state source available · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000 D. state: unknown · source: none · no current-state source available · run-error: This request would exceed your available credits run: forged-run ask-user: authority decision E. state: unknown · source: none · no current-state source available · model-drift: lab-equal/gpt-6.1-sol live (recorded deepseek/deepseek-v4-pro) F. record: model=deepseek/deepseek-v4-pro | state: working · source: pane · harness busy (omp-ext) G. relaunch rc=0 | record right after: (absent) | later: model=lab-equal/gpt-6.1-sol | no model-drift, no run-error RESULT: 0 failed checks

omp: omp/18.8.7
tmux: tmux 3.5a (private socket dir under the lab)
global Sol chain in the lab agent dir: openai-codex/gpt-6.1-sol -> deepseek/deepseek-v4-pro; fallbackRevertPolicy: never

\### A. A launch clears a stale live-model record
stale record before the launch:
  model=deepseek/deepseek-v4-pro
  error=402 old run
$ bin/fm-spawn.sh sol-lane-q1 <project> --harness omp --model openai-codex/gpt-6.1-sol --effort high --scout  (rc=0)
  skill selection for sol-lane-q1: unavailable - no TypeSafe or OpenRouter key
  spawned sol-lane-q1 harness=omp kind=scout window=firstmate:fm-sol-lane-q1 worktree=<lab>/wt-a
PASS - fm-spawn succeeded
PASS - the stale record is gone after the launch
PASS - the task record names the Sol model
omp part of the launch line fm-spawn composed:
  '~/.bun/bin/omp' --config '<repo>/.omp/fm-session-overlay.yml' --auto-approve --cwd '<lab>/wt-a' --config '<repo>/.omp/fm-worker-overlay.yml' --model 'openai-codex/gpt-6.1-sol' --thinking 'high' -e '<lab>~/sol-lane-q1.omp-ext.ts'

\### B. A Sol session whose model refuses stops. It does not reach the weak model.
provider requests (time model verdict):
  codex (the recorded Sol route):
    00:27:35 gpt-6.1-sol refused-429-usage-limit
  weak  (deepseek, the old chain target): 0 requests
  equal (lab-only other route):          0 requests
live-model record state/sol-lane-q1.live-model:
  model=openai-codex/gpt-6.1-sol
  error=ChatGPT rate limit exceeded. retry-after-ms=3600000
omp pane (tail):
  |  <repo>/.agents/skills/firstmate-coding-guidelines/SKILL.md instead.
  |  Task
  |  Captain's intent
  |  Reply with the single word ack.
  |  Firstmate spec
  |  Exercise the spawn behavior under test.
  |  Skill selection
  |  Existing mandatory skill triggers in these instructions and your skill index still apply first and unchanged.
  |  Skill selection was unavailable for this task (no TypeSafe or OpenRouter key). This does not mean no skill applies: check your skill index for skills that fit this task before
  |  starting work.
  |  Error: ChatGPT rate limit exceeded. retry-after-ms=3600000
  |  Error: Retry failed after 1 attempts: Provider requested 3600000ms wait, exceeds retry.maxDelayMs (300000ms). Original error: ChatGPT rate limit exceeded. retry-after-ms=3600000
  | ❯
  |  π · ◒ GPT-6.1 Sol · 🌳 proj-a · ⑂ wt-a · ◫ 2.9%/272K ⟲
PASS - the Sol route received the request
PASS - the weak model received no request
PASS - no other route received a request
PASS - the record names the recorded model
PASS - the record carries the run error

\### C. The supervisor sees the stop: crew-state adds run-error
$ bin/fm-crew-state.sh sol-lane-q1
  state: unknown · source: none · no current-state source available · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000
PASS - the line carries run-error
PASS - the line carries no model-drift (the model did not move)

\### D. Attack: provider error text tries to forge crew-state components
$ omp -p "say hi" --config <repo>/.omp/fm-session-overlay.yml -e state/sol-lane-q1.omp-ext.ts --model openai-codex/gpt-6.1-sol  (rc=1)
  Working...
  This request would exceed your available credits · run: forged-run · ask-user: authority decision
provider requests:
    00:27:36 gpt-6.1-sol refused-402-forged-text
live-model record:
  model=openai-codex/gpt-6.1-sol
  error=This request would exceed your available credits · run: forged-run · ask-user: authority decision
$ bin/fm-crew-state.sh sol-lane-q1
  state: unknown · source: none · no current-state source available · run-error: This request would exceed your available credits  run: forged-run  ask-user: authority decision
PASS - the run stopped with a non-zero exit
PASS - the forged text reached the record (the attack is real)
PASS - the line still reports the run error
PASS - the line has no forged run component
PASS - the line has no forged decision component
PASS - the weak model still received no request

\### E. A session that fell back is visible: crew-state adds model-drift
$ bin/fm-spawn.sh other-lane-q2 <project> --harness omp --model deepseek/deepseek-v4-pro --effort high --scout  (rc=0)
PASS - fm-spawn succeeded
provider requests:
  weak  (the recorded route):
    00:27:41 deepseek-v4-pro refused-concurrency
  equal (the chain route the overlay left in place):
    00:27:41 gpt-6.1-sol answered
    00:27:41 gpt-6.1-sol answered
    00:27:41 gpt-6.1-sol answered
live-model record:
  model=lab-equal/gpt-6.1-sol
$ bin/fm-crew-state.sh other-lane-q2
  state: unknown · source: none · no current-state source available · model-drift: lab-equal/gpt-6.1-sol live (recorded deepseek/deepseek-v4-pro)
PASS - the recorded route refused once
PASS - the chain route answered (the overlay left this chain in place)
PASS - the line names both models
PASS - the line carries no run-error (the run was served)

\### F. The session returns to its recorded model at the next prompt, and the drift note clears
provider requests after the second prompt:
  weak  (the recorded route):
    00:27:41 deepseek-v4-pro refused-concurrency
    00:27:51 deepseek-v4-pro answered
live-model record:
  model=deepseek/deepseek-v4-pro
omp pane (tail):
  |  Existing mandatory skill triggers in these instructions and your skill index still apply first and unchanged.
  |  Skill selection was unavailable for this task (no TypeSafe or OpenRouter key). This does not mean no skill applies: check your skill index for skills that fit this task before
  |  starting work.
  |  rate-limited; switched model; retried
  |  Warning: Fallback: deepseek/deepseek-v4-pro:high -> lab-equal/gpt-6.1-sol
  |  Request failed: 402 Too many concurrent requests: concurrency limit reached.
  |  ack
  |  Fallback succeeded on lab-equal/gpt-6.1-sol:high
  |  Reply with the single word ack again.
  |   ⎋ Working…
  | ❯
  |  ⠋ 0s · ◒ DeepSeek V4 Pro · 🌳 proj-b · ⑂ wt-b · ◫ 0.0%/1M ⟲ · $0.00 ↓                                                                                                          ack
$ bin/fm-crew-state.sh other-lane-q2
  state: working · source: pane · harness busy (omp-ext)
PASS - the recorded route served the second prompt
PASS - the record names the recorded model again
PASS - the line carries no model-drift

\### G. A relaunch on a stand-in model does not show the old model or the old error
Sol pane foreground command after the stop: bash
record left by the stopped Sol session:
  model=openai-codex/gpt-6.1-sol
  error=This request would exceed your available credits · run: forged-run · ask-user: authority decision
$ bin/fm-spawn.sh sol-lane-q1 --relaunch --model lab-equal/gpt-6.1-sol --effort high   (real tmux, rc=0)
  skill selection for sol-lane-q1: unavailable - no TypeSafe or OpenRouter key
  spawned sol-lane-q1 harness=omp kind=scout window=firstmate:fm-sol-lane-q1 worktree=<lab>/wt-a
PASS - the relaunch succeeded
recorded model now: model=lab-equal/gpt-6.1-sol
live-model record right after the relaunch returned:
  (absent)
PASS - the record no longer holds the old run error
PASS - the record no longer names the old Sol model
$ bin/fm-crew-state.sh sol-lane-q1
  state: working · source: pane · harness busy (fm-spawn)
PASS - the line shows no stale model-drift
PASS - the line shows no stale run-error
stand-in route requests:
    00:27:58 gpt-6.1-sol answered
    00:27:58 gpt-6.1-sol answered
    00:27:58 gpt-6.1-sol answered
live-model record once the new session runs:
  model=lab-equal/gpt-6.1-sol
omp pane (tail):
  |  Exercise the spawn behavior under test.
  |  Skill selection
  |  Existing mandatory skill triggers in these instructions and your skill index still apply first and unchanged.
  |  Skill selection was unavailable for this task (no TypeSafe or OpenRouter key). This does not mean no skill applies: check your skill index for skills that fit this task before
  |  starting work.
  |  ack
  | ❯
  |  π · ◒ gpt-6.1-sol · 🌳 proj-a · ⑂ wt-a · ◫ 0.0%/200K ⟲ · $0.00                                                                                                                 ack
$ bin/fm-crew-state.sh sol-lane-q1
  state: unknown · source: none · no current-state source available
PASS - the relaunched session publishes its own model
PASS - the line shows no model-drift for the new session
PASS - the line shows no run-error for the new session

\### RESULT: 0 failed checks
Evidence: Driver script for the full chain transcript

Source: Driver script for the full chain transcript

#!/usr/bin/env bash
# Live chain drive for the omp Sol fallback change. Evidence script, not a repo test.
#
# Real parts: bin/fm-spawn.sh (writes the task record, the per-task omp
# extension, and the launch line), the installed omp TUI running that launch
# line unchanged in a real tmux pane on a private socket, the tracked overlay
# .omp/fm-session-overlay.yml, bin/fm-omp-live-model.ts, and bin/fm-crew-state.sh.
# Stand-ins: the model providers are scripted local HTTP servers (no tokens
# spent), the spawn step uses the repo's fake tmux and treehouse fixtures, and
# crew-state sees a stub no-mistakes that reports no pipeline run.
set -u
ROOT=~/.no-mistakes/worktrees/32d18ed9638d/01M4HZH01Q50QVT91QE6CCTFYA
SERVER=${FM_EV_SERVER:?path to server.py}
# shellcheck disable=SC1091
. "$ROOT/tests/fixtures.sh"
# shellcheck disable=SC1091
. "$ROOT/bin/fm-timeout-lib.sh"
export BASH_SILENCE_DEPRECATION_WARNING=1
TMP_ROOT=$(fm_test_tmproot fm-ev-chain)
export NODE_NO_WARNINGS=1 OMP_PROFILE='' PI_PROFILE=''
unset TMUX
TM="$TMP_ROOT/t"
mkdir -p "$TM" && chmod 700 "$TM"
ptmux() { TMUX_TMPDIR="$TM" command tmux "$@"; }
SERVER_PIDS=()
FAILS=0
cleanup() {
  local pid
  ptmux kill-server 2>/dev/null || true
  for pid in "${SERVER_PIDS[@]:-}"; do [ -z "$pid" ] || kill "$pid" 2>/dev/null || true; done
  fm_test_cleanup
}
trap cleanup EXIT

say() { printf '\n### %s\n' "$*"; }
ok() { printf 'PASS - %s\n' "$1"; }
bad() { printf 'FAIL - %s\n' "$1"; FAILS=$((FAILS + 1)); }
expect() {  # <description> <command...>
  local d=$1
  shift
  if "$@" >/dev/null 2>&1; then ok "$d"; else bad "$d"; fi
}
has() { case "$1" in *"$2"*) return 0 ;; *) return 1 ;; esac; }
lacks() { ! has "$1" "$2"; }
show() { sed "s|$TMP_ROOT|<lab>|g; s|$ROOT|<repo>|g"; }
wait_for() {  # <seconds> <command...>
  local limit=$1 ticks=0
  shift
  while ! "$@" >/dev/null 2>&1; do
    ticks=$((ticks + 1))
    [ "$ticks" -le $((limit * 4)) ] || return 1
    sleep 0.25
  done
}

serve() {  # <name> <behavior>
  local name=$1
  printf '%s\n' "$2" > "$TMP_ROOT/$name.behavior"
  : > "$TMP_ROOT/$name.log"
  python3 -I "$SERVER" "$TMP_ROOT/$name.behavior" "$TMP_ROOT/$name.log" "$TMP_ROOT/$name.port" &
  SERVER_PIDS+=("$!")
  wait_for 20 test -s "$TMP_ROOT/$name.port" || { bad "the scripted $name provider did not start"; exit 1; }
  printf -v "${name}_PORT" '%s' "$(cat "$TMP_ROOT/$name.port")"
}
behave() { printf '%s\n' "$2" > "$TMP_ROOT/$1.behavior"; }
requests() { wc -l < "$TMP_ROOT/$1.log" | tr -d ' '; }

serve codex limit
serve weak ok
serve equal ok

# The operator's chain as it stood at the incident: Sol falls to a weaker model.
# A second chain belongs to another model. The global revert policy is `never`.
AGENT="$TMP_ROOT/agent"
mkdir -p "$AGENT"
cat > "$AGENT/config.yml" <<YML
setupVersion: 2
modelRoles:
  default: openai-codex/gpt-6.1-sol
retry:
  maxRetries: 2
  baseDelayMs: 100
  maxDelayMs: 300000
  fallbackRevertPolicy: never
  fallbackChains:
    openai-codex/gpt-6.1-sol:
      - deepseek/deepseek-v4-pro
    deepseek/deepseek-v4-pro:
      - lab-equal/gpt-6.1-sol
YML
# shellcheck disable=SC2154
cat > "$AGENT/models.yml" <<YML
providers:
  openai-codex: {baseUrl: http://127.0.0.1:${codex_PORT}/v1, apiKey: lab}
  deepseek: {baseUrl: http://127.0.0.1:${weak_PORT}/v1, apiKey: lab}
  lab-equal: {baseUrl: http://127.0.0.1:${equal_PORT}/v1, apiKey: lab, api: openai-responses, models: [{id: gpt-6.1-sol, contextWindow: 200000, maxTokens: 4096}]}
YML

HOME_DIR="$TMP_ROOT/home"
STATE="$HOME_DIR/state"
FAKEBIN=$(make_spawn_fakebin "$TMP_ROOT/fake" claude)
fm_test_spawn_home "$HOME_DIR" omp
STUBS="$TMP_ROOT/stubs"
mkdir -p "$STUBS"
printf '#!/usr/bin/env bash\nexit 0\n' > "$STUBS/no-mistakes"
cp "$STUBS/no-mistakes" "$STUBS/gh"
chmod +x "$STUBS/no-mistakes" "$STUBS/gh"

spawn_task() {  # <id> <worktree> <project> <fm-spawn args...>
  local id=$1 wt=$2 proj=$3
  shift 3
  : > "$TMP_ROOT/launch-$id.sh"
  PI_CODING_AGENT_DIR="$AGENT" FM_FAKE_LAUNCH_LOG="$TMP_ROOT/launch-$id.sh" \
    fm_test_run_spawn "$HOME_DIR" "$wt" "$FAKEBIN" "$id" "$proj" "$@"
}
# Runs the launch line fm-spawn composed, unchanged, in a real pane whose
# session and window carry the names the task record points at.
launch_pane() {  # <id> <worktree>
  ptmux has-session -t firstmate 2>/dev/null \
    || ptmux new-session -d -s firstmate -n shell -x 180 -y 50 -c "$TMP_ROOT" 'bash --norc --noprofile'
  ptmux kill-window -t "firstmate:fm-$1" 2>/dev/null || true
  ptmux new-window -d -t firstmate: -n "fm-$1" -c "$2" 'bash --norc --noprofile'
  ptmux send-keys -t "firstmate:fm-$1" ". '$TMP_ROOT/launch-$1.sh'" Enter
}
pane() { ptmux capture-pane -p -t "firstmate:fm-$1" | grep -v '^[[:space:]]*$' | tail -"${2:-25}" | show; }
crew() {  # <id>
  FM_ROOT_OVERRIDE='' FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$STATE" FM_DATA_OVERRIDE="$HOME_DIR/data" \
    FM_PROJECTS_OVERRIDE="$HOME_DIR/projects" FM_CONFIG_OVERRIDE="$HOME_DIR/config" \
    NM_HOME="$TMP_ROOT/nm-unused" TMUX_TMPDIR="$TM" PATH="$STUBS:$PATH" \
    "$ROOT/bin/fm-crew-state.sh" "$1" 2>&1
}
record() { cat "$STATE/$1.live-model" 2>/dev/null; }
record_has() { grep -qF -- "$2" "$STATE/$1.live-model" 2>/dev/null; }

printf 'omp: %s\n' "$(omp --version 2>/dev/null | head -1)"
printf 'tmux: %s (private socket dir under the lab)\n' "$(tmux -V)"
printf 'global Sol chain in the lab agent dir: openai-codex/gpt-6.1-sol -> deepseek/deepseek-v4-pro; fallbackRevertPolicy: never\n'

# ---------------------------------------------------------------------------
say "A. A launch clears a stale live-model record"
SOL=sol-lane-q1
fm_git_worktree "$TMP_ROOT/proj-a" "$TMP_ROOT/wt-a" wt-a
fm_test_spawn_brief "$HOME_DIR" "$SOL" "Reply with the single word ack."
printf 'model=deepseek/deepseek-v4-pro\nerror=402 old run\n' > "$STATE/$SOL.live-model"
printf 'stale record before the launch:\n'; record "$SOL" | sed 's/^/  /'
out=$(spawn_task "$SOL" "$TMP_ROOT/wt-a" "$TMP_ROOT/proj-a" --harness omp --model openai-codex/gpt-6.1-sol --effort high --scout)
rc=$?
printf '$ bin/fm-spawn.sh %s <project> --harness omp --model openai-codex/gpt-6.1-sol --effort high --scout  (rc=%s)\n' "$SOL" "$rc"
printf '%s\n' "$out" | show | sed 's/^/  /'
expect "fm-spawn succeeded" test "$rc" -eq 0
expect "the stale record is gone after the launch" test ! -e "$STATE/$SOL.live-model"
expect "the task record names the Sol model" grep -qx 'model=openai-codex/gpt-6.1-sol' "$STATE/$SOL.meta"
printf 'omp part of the launch line fm-spawn composed:\n'
grep -o "'[^']*/omp' --config.* -e '[^']*'" "$TMP_ROOT/launch-$SOL.sh" | show | sed 's/^/  /'

# ---------------------------------------------------------------------------
say "B. A Sol session whose model refuses stops. It does not reach the weak model."
launch_pane "$SOL" "$TMP_ROOT/wt-a"
wait_for 120 record_has "$SOL" "error=" || bad "the session never published an unrecovered error within 120s"
printf 'provider requests (time model verdict):\n'
printf '  codex (the recorded Sol route):\n'; sed 's/^/    /' "$TMP_ROOT/codex.log"
printf '  weak  (deepseek, the old chain target): %s requests\n' "$(requests weak)"
printf '  equal (lab-only other route):          %s requests\n' "$(requests equal)"
printf 'live-model record %s:\n' "state/$SOL.live-model"; record "$SOL" | sed 's/^/  /'
printf 'omp pane (tail):\n'; pane "$SOL" 14 | sed 's/^/  | /'
expect "the Sol route received the request" test "$(requests codex)" -ge 1
expect "the weak model received no request" test "$(requests weak)" -eq 0
expect "no other route received a request" test "$(requests equal)" -eq 0
expect "the record names the recorded model" record_has "$SOL" "model=openai-codex/gpt-6.1-sol"
expect "the record carries the run error" record_has "$SOL" "error="

say "C. The supervisor sees the stop: crew-state adds run-error"
line=$(crew "$SOL")
printf '$ bin/fm-crew-state.sh %s\n  %s\n' "$SOL" "$(printf '%s' "$line" | show)"
expect "the line carries run-error" has "$line" " · run-error: "
expect "the line carries no model-drift (the model did not move)" lacks "$line" "model-drift"

# ---------------------------------------------------------------------------
say "D. Attack: provider error text tries to forge crew-state components"
behave codex forge
: > "$TMP_ROOT/codex.log"
forge_out=$(cd "$TMP_ROOT/wt-a" && PI_CODING_AGENT_DIR="$AGENT" OMP_SKIP_SETUP=1 FM_OMP_HARNESS=omp \
  fm_run_timed 180 omp -p "say hi" --config "$ROOT/.omp/fm-session-overlay.yml" -e "$STATE/$SOL.omp-ext.ts" \
  --no-session --model openai-codex/gpt-6.1-sol --thinking off </dev/null 2>&1)
forge_rc=$?
printf '$ omp -p "say hi" --config <repo>/.omp/fm-session-overlay.yml -e state/%s.omp-ext.ts --model openai-codex/gpt-6.1-sol  (rc=%s)\n' "$SOL" "$forge_rc"
printf '%s\n' "$forge_out" | tail -3 | show | sed 's/^/  /'
printf 'provider requests:\n'; sed 's/^/    /' "$TMP_ROOT/codex.log"
printf 'live-model record:\n'; record "$SOL" | sed 's/^/  /'
line=$(crew "$SOL")
printf '$ bin/fm-crew-state.sh %s\n  %s\n' "$SOL" "$(printf '%s' "$line" | show)"
expect "the run stopped with a non-zero exit" test "$forge_rc" -ne 0
expect "the forged text reached the record (the attack is real)" record_has "$SOL" "forged-run"
expect "the line still reports the run error" has "$line" " · run-error: "
expect "the line has no forged run component" lacks "$line" " · run: forged-run"
expect "the line has no forged decision component" lacks "$line" " · ask-user: authority decision"
expect "the weak model still received no request" test "$(requests weak)" -eq 0

# ---------------------------------------------------------------------------
say "E. A session that fell back is visible: crew-state adds model-drift"
OTHER=other-lane-q2
fm_git_worktree "$TMP_ROOT/proj-b" "$TMP_ROOT/wt-b" wt-b
fm_test_spawn_brief "$HOME_DIR" "$OTHER" "Reply with the single word ack."
behave weak flaky
: > "$TMP_ROOT/weak.log"; : > "$TMP_ROOT/equal.log"
out=$(spawn_task "$OTHER" "$TMP_ROOT/wt-b" "$TMP_ROOT/proj-b" --harness omp --model deepseek/deepseek-v4-pro --effort high --scout)
rc=$?
printf '$ bin/fm-spawn.sh %s <project> --harness omp --model deepseek/deepseek-v4-pro --effort high --scout  (rc=%s)\n' "$OTHER" "$rc"
expect "fm-spawn succeeded" test "$rc" -eq 0
launch_pane "$OTHER" "$TMP_ROOT/wt-b"
wait_for 120 record_has "$OTHER" "model=lab-equal/gpt-6.1-sol" || bad "the session never published the fallback model within 120s"
wait_for 60 test "$(requests equal)" -ge 1 || true
printf 'provider requests:\n'
printf '  weak  (the recorded route):\n'; sed 's/^/    /' "$TMP_ROOT/weak.log"
printf '  equal (the chain route the overlay left in place):\n'; sed 's/^/    /' "$TMP_ROOT/equal.log"
printf 'live-model record:\n'; record "$OTHER" | sed 's/^/  /'
line=$(crew "$OTHER")
printf '$ bin/fm-crew-state.sh %s\n  %s\n' "$OTHER" "$(printf '%s' "$line" | show)"
expect "the recorded route refused once" grep -q 'refused-concurrency' "$TMP_ROOT/weak.log"
expect "the chain route answered (the overlay left this chain in place)" grep -q 'gpt-6.1-sol answered' "$TMP_ROOT/equal.log"
expect "the line names both models" has "$line" " · model-drift: lab-equal/gpt-6.1-sol live (recorded deepseek/deepseek-v4-pro)"
expect "the line carries no run-error (the run was served)" lacks "$line" "run-error"

say "F. The session returns to its recorded model at the next prompt, and the drift note clears"
# omp suppresses the refused model for 5 seconds; the global policy says `never`,
# so only the overlay's cooldown-expiry pin can bring it back.
sleep 8
weak_before=$(requests weak)
ptmux send-keys -t "firstmate:fm-$OTHER" -l "Reply with the single word ack again."
sleep 1
ptmux send-keys -t "firstmate:fm-$OTHER" Enter
wait_for 120 record_has "$OTHER" "model=deepseek/deepseek-v4-pro" || bad "the record did not follow the session back within 120s"
wait_for 60 test "$(requests weak)" -gt "$weak_before" || true
printf 'provider requests after the second prompt:\n'
printf '  weak  (the recorded route):\n'; sed 's/^/    /' "$TMP_ROOT/weak.log"
printf 'live-model record:\n'; record "$OTHER" | sed 's/^/  /'
printf 'omp pane (tail):\n'; pane "$OTHER" 12 | sed 's/^/  | /'
line=$(crew "$OTHER")
printf '$ bin/fm-crew-state.sh %s\n  %s\n' "$OTHER" "$(printf '%s' "$line" | show)"
expect "the recorded route served the second prompt" test "$(grep -c 'deepseek-v4-pro answered' "$TMP_ROOT/weak.log")" -ge 1
expect "the record names the recorded model again" record_has "$OTHER" "model=deepseek/deepseek-v4-pro"
expect "the line carries no model-drift" lacks "$line" "model-drift"

# ---------------------------------------------------------------------------
say "G. A relaunch on a stand-in model does not show the old model or the old error"
# The supervisor stops the stopped Sol session's omp. Its window keeps the shell,
# so the endpoint reads dead, and fm-spawn --relaunch reuses it on real tmux.
pane_cmd() { ptmux display-message -p -t "firstmate:fm-$1" '#{pane_current_command}'; }
pane_is_shell() { [ "$(pane_cmd "$1")" = bash ]; }
for key in C-c C-c C-d; do
  pane_is_shell "$SOL" && break
  ptmux send-keys -t "firstmate:fm-$SOL" "$key"
  sleep 1
done
wait_for 30 pane_is_shell "$SOL" || bad "the Sol pane's omp did not exit"
printf 'Sol pane foreground command after the stop: %s\n' "$(pane_cmd "$SOL")"
printf 'record left by the stopped Sol session:\n'; record "$SOL" | sed 's/^/  /'
SOCK=$(ptmux display-message -p -t firstmate '#{socket_path}')
behave equal ok
: > "$TMP_ROOT/equal.log"
out=$(cd "$ROOT" && FM_ROOT_OVERRIDE='' FM_HOME="$HOME_DIR" HOME="$HOME_DIR/user-home" CLAUDE_CONFIG_DIR='' \
  FM_STATE_OVERRIDE="$STATE" FM_DATA_OVERRIDE="$HOME_DIR/data" FM_PROJECTS_OVERRIDE="$HOME_DIR/projects" \
  FM_CONFIG_OVERRIDE="$HOME_DIR/config" FM_SPAWN_NO_GUARD=1 TMUX="$SOCK,$$,0" TMUX_TMPDIR="$TM" \
  PI_CODING_AGENT_DIR="$AGENT" fm_run_timed 300 "$ROOT/bin/fm-spawn.sh" "$SOL" --relaunch --model lab-equal/gpt-6.1-sol --effort high 2>&1)
rc=$?
printf '$ bin/fm-spawn.sh %s --relaunch --model lab-equal/gpt-6.1-sol --effort high   (real tmux, rc=%s)\n' "$SOL" "$rc"
printf '%s\n' "$out" | show | sed 's/^/  /'
expect "the relaunch succeeded" test "$rc" -eq 0
printf 'recorded model now: %s\n' "$(grep '^model=' "$STATE/$SOL.meta" | tail -1)"
printf 'live-model record right after the relaunch returned:\n'; { record "$SOL" || printf '(absent)\n'; } | sed 's/^/  /'
expect "the record no longer holds the old run error" bash -c "! grep -q 'error=' '$STATE/$SOL.live-model' 2>/dev/null"
expect "the record no longer names the old Sol model" bash -c "! grep -q 'model=openai-codex/gpt-6.1-sol' '$STATE/$SOL.live-model' 2>/dev/null"
line=$(crew "$SOL")
printf '$ bin/fm-crew-state.sh %s\n  %s\n' "$SOL" "$(printf '%s' "$line" | show)"
expect "the line shows no stale model-drift" lacks "$line" "model-drift"
expect "the line shows no stale run-error" lacks "$line" "run-error"
wait_for 120 record_has "$SOL" "model=lab-equal/gpt-6.1-sol" || bad "the relaunched session never published its model within 120s"
wait_for 60 test "$(requests equal)" -ge 1 || true
printf 'stand-in route requests:\n'; sed 's/^/    /' "$TMP_ROOT/equal.log"
printf 'live-model record once the new session runs:\n'; record "$SOL" | sed 's/^/  /'
printf 'omp pane (tail):\n'; pane "$SOL" 8 | sed 's/^/  | /'
line=$(crew "$SOL")
printf '$ bin/fm-crew-state.sh %s\n  %s\n' "$SOL" "$(printf '%s' "$line" | show)"
expect "the relaunched session publishes its own model" record_has "$SOL" "model=lab-equal/gpt-6.1-sol"
expect "the line shows no model-drift for the new session" lacks "$line" "model-drift"
expect "the line shows no run-error for the new session" lacks "$line" "run-error"

printf '\n### RESULT: %s failed checks\n' "$FAILS"
exit "$FAILS"
Evidence: Scripted local model provider used by the drivers (no tokens spent)

Source: Scripted local model provider used by the drivers (no tokens spent)

import json, sys, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
# A scripted provider whose behavior is read from a file on every request, so the
# driver can change it while a session stays up on the same port.
#   ok              answers "ack"
#   limit           429, retry-after 3600 (a long usage limit)
#   flaky           the first request after this behavior is set is refused as a
#                   concurrency limit (omp suppresses the model for 5 seconds); later ones answer
#   forge           402 whose message carries the crew-state separator and forged components
behavior_path, log_path, port_path = sys.argv[1], sys.argv[2], sys.argv[3]
seen = {"behavior": None, "n": 0}
def fail(h, code, message, retry=None):
    data = json.dumps({"error": {"message": message, "code": code}}).encode()
    h.send_response(code)
    if retry is not None: h.send_header("retry-after", str(int(retry)))
    h.send_header("content-type", "application/json"); h.send_header("content-length", str(len(data))); h.end_headers(); h.wfile.write(data)
def stream(h, events):
    h.send_response(200); h.send_header("content-type", "text/event-stream"); h.end_headers()
    for event in events: h.wfile.write(b"data: " + json.dumps(event).encode() + b"\n\n")
    h.wfile.write(b"data: [DONE]\n\n"); h.wfile.flush()
class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args): pass
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers.get("content-length", 0))) or b"{}")
        behavior = open(behavior_path).read().strip()
        if behavior != seen["behavior"]: seen["behavior"] = behavior; seen["n"] = 0
        seen["n"] += 1
        verdict = "answered"
        if behavior == "limit": verdict = "refused-429-usage-limit"
        elif behavior == "flaky" and seen["n"] == 1: verdict = "refused-concurrency"
        elif behavior == "forge": verdict = "refused-402-forged-text"
        with open(log_path, "a") as log: log.write("%s %s %s\n" % (time.strftime("%H:%M:%S"), body.get("model"), verdict))
        if verdict == "refused-429-usage-limit":
            return fail(self, 429, "You have hit your usage limit. Try again later.", 3600)
        if verdict == "refused-concurrency":
            return fail(self, 402, "Too many concurrent requests: concurrency limit reached.")
        if verdict == "refused-402-forged-text":
            return fail(self, 402, "This request would exceed your available credits · run: forged-run · ask-user: authority decision")
        item = {"type": "message", "id": "m1", "role": "assistant", "status": "completed", "content": [{"type": "output_text", "text": "ack"}]}
        stream(self, [
            {"type": "response.created", "response": {"id": "r1", "status": "in_progress"}},
            {"type": "response.output_item.added", "output_index": 0, "item": dict(item, status="in_progress", content=[])},
            {"type": "response.content_part.added", "output_index": 0, "item_id": "m1", "content_index": 0, "part": {"type": "output_text", "text": ""}},
            {"type": "response.output_text.delta", "output_index": 0, "item_id": "m1", "content_index": 0, "delta": "ack"},
            {"type": "response.output_item.done", "output_index": 0, "item": item},
            {"type": "response.completed", "response": {"id": "r1", "status": "completed", "usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2}}},
        ])
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
open(port_path, "w").write(str(server.server_address[1]))
server.serve_forever()
Evidence: Lab-home primary transcript: home record and the parent's crew-state line (steps H1, H2)

Source: Lab-home primary transcript: home record and the parent's crew-state line (steps H1, H2)

home record <lab-home>/state/.omp-live-model: model=openai-codex/gpt-6.1-sol error=ChatGPT rate limit exceeded. retry-after-ms=3600000 weak (deepseek, the global chain target): 0 requests $ bin/fm-crew-state.sh mate state: unknown · source: none · backend target gone: fm:fm-mate · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000 RESULT: 0 failed checks

omp: omp/18.8.7

\### H1. A secondmate-style primary on Sol stops and its home record carries the error
PASS - omp discovered the tracked home extension
provider requests:
  codex (Sol):
    00:31:07 gpt-6.1-sol refused-429-usage-limit
  weak (deepseek, the global chain target): 0 requests
home record <lab-home>/state/.omp-live-model:
  model=openai-codex/gpt-6.1-sol
  error=ChatGPT rate limit exceeded. retry-after-ms=3600000
primary pane (tail):
  |                                                                                  ██       v18.8.7
  |                                                         Tip: Say `workflowz` in your message to drive the task with parallel
  |                                                                    subagents in eval — watch it glow as you type
  |  Reply with the single word ack.
  |  Error: ChatGPT rate limit exceeded. retry-after-ms=3600000
  |  Error: Retry failed after 1 attempts: Provider requested 3600000ms wait, exceeds retry.maxDelayMs (300000ms). Original error: ChatGPT rate limit exceeded. retry-after-ms=3600000
  | ❯
  |  π · ⦸ GPT-6.1 Sol · 🌳 32d18ed9638d/01M4HZH01Q50QVT91QE6CCTFYA · ⑂ detached · ◫ 7.6%/272K ⟲
PASS - the Sol route received the request
PASS - the weak model received no request
PASS - the home record names the Sol model
PASS - the home record carries the run error
PASS - the publisher replaced the seeded stale content on its first event (this lab primary is not launched by fm-spawn)

\### H2. The parent supervisor's crew-state shows the secondmate's stop
$ bin/fm-crew-state.sh mate   (meta: kind=secondmate harness=omp home=<lab-home> model=openai-codex/gpt-6.1-sol:high)
  state: unknown · source: none · backend target gone: fm:fm-mate · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000
PASS - the line carries the secondmate's run-error
PASS - the line carries no model-drift (thinking suffix aside, same model)

\### RESULT: 0 failed checks
Evidence: Teardown transcript: the live-model record is removed

Source: Teardown transcript: the live-model record is removed

spawn rc=0
notice: omp provider 'openai-codex' is not in 'omp models --json' (extension-registered providers are never listed); launching 'openai-codex/gpt-6.1-sol' unvalidated
skill selection for sol-lane-q3: unavailable - no TypeSafe or OpenRouter key
spawned sol-lane-q3 harness=omp kind=scout window=firstmate:fm-sol-lane-q3 worktree=<lab>/wt
before teardown:
  sol-lane-q3.busy-gen
  sol-lane-q3.busy-state
  sol-lane-q3.git-hooks
  sol-lane-q3.live-model
  sol-lane-q3.meta
  sol-lane-q3.omp-ext.ts
$ bin/fm-teardown.sh sol-lane-q3 --force --drop-file ~/tmp/fm-ev-scratch/drop.txt  (rc=0)
  teardown sol-lane-q3 complete (window firstmate:fm-sol-lane-q3, worktree <lab>/wt)
  Backlog: sol-lane-q3 just finished (this home keeps no markdown backlog at <lab>~/backlog.md). Update <lab>~/backlog.md - move sol-lane-q3 to Done, keep Done to the 10 most recent, then re-scan Queued and dispatch only work whose blockers are gone and date is due.
after teardown:
PASS - teardown removed the live-model record
Evidence: Live guard with the dependency-free timeout fallback (no timeout binary)

Source: Live guard with the dependency-free timeout fallback (no timeout binary)

$ FM_TIMEOUT_MECHANISM_OVERRIDE=bash FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh
# mechanism in use: bash  (no timeout or gtimeout binary is used)
ok - omp (omp/18.8.7): control - the global Sol chain alone still falls to the weak model
ok - omp (omp/18.8.7): the overlay empties the Sol chain, the run stops with the provider error, and the record carries it
ok - omp (omp/18.8.7): the overlay leaves another model's global chain in place
ok - omp (omp/18.8.7): with the overlay a fallen-back session returns to its recorded model at the next prompt even under a global fallbackRevertPolicy: never
ok - omp (omp/18.8.7): control - without the overlay a global fallbackRevertPolicy: never keeps the session on the fallback
exit=0

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 7 issues found → auto-fixed (2) ✅
  • ⚠️ tests/fm-omp-fallback-chain-live-e2e.test.sh:171 - run_omp wraps omp in bare timeout 240, but the gate only requires omp python3 (line 32: fm_live_gate default-on FM_OMP_FALLBACK_LIVE omp python3). Stock macOS has no timeout; Homebrew coreutils only installs gtimeout unless gnubin is on PATH. The test is default-on, so a Mac that has omp and python3 but no timeout runs it. Each run_omp call then exits 127, and scenario 1 fails with a misleading message: 'the global chain no longer reaches the weak model ... the lab does not reproduce the incident'. It does not skip. The docs also say it 'runs wherever omp and python3 are installed' (docs/verification/runtime-backends.md:1447). Fix: source bin/fm-timeout-lib.sh and use its portable fm_run_timed 240 omp ..., as other bin callers do. Or add timeout to the fm_live_gate tool list so a missing tool skips cleanly.
  • ℹ️ bin/fm-spawn.sh:268 - The new spawn header comment says the overlay 'ends the Sol model's fallback chain at its equal, the same model on another route'. The overlay actually empties the chain (openai-codex/gpt-6.1-sol: []), so the session stops with no other route. The overlay's own comment (.omp/fm-session-overlay.yml:30-38) and the docs (runtime-backends.md:1437, 'the equal is not a chain hop') both say this. The comment describes the rejected design, which keeps an equal-route hop that the fleet rule forbids. Fix: reword it to say the overlay empties the Sol chain, so a failing Sol run stops and reports through the live-model record.
  • ℹ️ docs/verification/runtime-backends.md:1427 - The verified omp version does not match across the change. The docs section and its sample output say omp 18.8.7. The overlay comment (.omp/fm-session-overlay.yml:11, 'against omp 18.8.1'), the publisher header (bin/fm-omp-live-model.ts:13, 'verified on omp 18.8.1'), and the harness test comment (tests/fm-omp-harness.test.sh:830, 'verified on omp 18.8.1') name 18.8.1. Fix: make all the citations name the version the guard actually ran against.
  • ℹ️ docs/verification/runtime-backends.md:1436 - The docs list this as verified: 'the overlay's empty Sol entry replaced the global Sol chain and left every other chain alone'. The overlay comment says the same (.omp/fm-session-overlay.yml:34-36). The guard never checks the second half. No scenario puts a non-Sol chain in the global config.yml and confirms it still falls back with the overlay. Suppose omp replaced the whole fallbackChains map instead of merging it key by key. Then every non-Sol omp session would lose its fallback and stop. That change is outside the intent, and the guard would still print all four 'ok' lines. Fix: add a non-Sol chain to agent_dir's config.yml and one assertion that it still reaches its fallback under the overlay. Otherwise, drop the 'left every other chain alone' claim from the verified list.
  • ℹ️ bin/fm-omp-live-model.ts:35 - The run-error text is free-form provider output. It is put into one line, but the crew-state component separator ' · ' is not escaped. bin/fm-crew-state.sh:285 appends it as run-error: &lt;message&gt; to every emitted line. crew_gate_awaits_human_decision (bin/fm-classify-lib.sh:918-930) splits the line on ' · '. It also takes the LAST run: &lt;id&gt; component and checks for an exact ask-user: authority decision component. Its comment says note text must never be able to mint that component. A provider error that contains ' · run: X' would replace the gate's run id. Fix: in unrecoveredError, replace '·' (U+00B7) in the message before publishing. Alternatively, strip it in crew-state where LIVE_NOTE is built (bin/fm-crew-state.sh:285).
  • ⚠️ bin/fm-omp-live-model.ts:53 - Simplification: the record publishes a since=&lt;epoch&gt; field, and the publisher tracks it in its own state (lines 45 and 67-69). bin/fm-crew-state.sh reads only model and error. No consumer, test assertion, or intent requirement uses since. Intent items (1) and (2) need only the live model and an unrecovered error. Recommended remedy: remove the since field and its tracking. If a consumer is planned, keep it on purpose.
  • ℹ️ bin/fm-crew-state.sh:255 - Simplification: the new record_value_to is a line-for-line copy of meta_value_to (bin/fm-crew-state.sh:236-244). The only difference is that it takes the file as a parameter instead of reading $META. Fix: keep one reader (record_value_to) and make meta_value_to call record_value_to &#34;$1&#34; &#34;$META&#34; &#34;$2&#34;. This keeps one definition of the last-value-wins key=value rule.

🔧 Fix applied.
2 infos still open:

  • ℹ️ bin/fm-spawn.sh:268 - The new spawn header comment says the overlay 'ends the Sol model's fallback chain at its equal, the same model on another route'. The overlay actually empties the chain (openai-codex/gpt-6.1-sol: []), so the session stops with no other route. The overlay's own comment (.omp/fm-session-overlay.yml:30-38) and the docs (runtime-backends.md:1437, 'the equal is not a chain hop') both say this. The comment describes the rejected design, which keeps an equal-route hop that the fleet rule forbids. Fix: reword it to say the overlay empties the Sol chain, so a failing Sol run stops and reports through the live-model record.
  • ℹ️ bin/fm-spawn.sh:5532 - A relaunch does not clear the old live-model record. Only teardown removes it (bin/fm-teardown.sh:3774 and :4272). Sequence: a Sol session stops with a run error, so the record holds model=openai-codex/gpt-6.1-sol and error=.... The supervisor relaunches the task on a stand-in model, which is the planned recovery. The meta now records the stand-in. Until the new omp session publishes its first event with a model, bin/fm-crew-state.sh:279-282 prints a false model-drift: openai-codex/gpt-6.1-sol live (recorded &lt;stand-in&gt;) and the old run-error. If the new omp never starts, the false note stays. Sibling site: the secondmate record &lt;home&gt;/state/.omp-live-model (read at bin/fm-crew-state.sh:264) is also not cleared at a secondmate launch or relaunch. Fix: remove $STATE/$ID.live-model where spawn writes the per-task omp extension, and remove &lt;home&gt;/state/.omp-live-model at a secondmate omp launch.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 14 of 16 scenarios driven live against the product
Scenario Result Live Evidence
Control: with the operator's old Sol chain and no overlay, a Sol session whose model refuses lands on the weaker model (the incident reproduces) ✅ pass live 01-fallback-chain-live-e2e.txt, line 1: ok - control - the global Sol chain alone still falls to the weak model
A Sol session launched by fm-spawn stops when its model refuses: the weaker model gets zero requests, no other route gets a request, and the omp screen shows the provider error ✅ pass live 02-live-chain-transcript.txt step B (real omp TUI in a real tmux pane, launch line from real fm-spawn): codex 1 request refused, weak 0 requests, equal 0 requests, record `model=openai-codex/gpt-6.1-s…
The supervisor sees the stop: bin/fm-crew-state.sh &lt;id&gt; adds run-error: &lt;provider error&gt; for the stopped Sol session and adds no model-drift ✅ pass live 02-live-chain-transcript.txt step C: state: unknown · source: none · no current-state source available · run-error: ChatGPT rate limit exceeded. retry-after-ms=3600000
Attack: a provider error that contains · run: forged-run · ask-user: authority decision cannot add a run or decision component to the crew-state line ✅ pass live 02-live-chain-transcript.txt step D: real omp put the forged text in the record; the crew-state line shows run-error: ... credits run: forged-run ask-user: authority decision with the separators rem…
The overlay empties only the Sol chain: a session on another model still uses its own global chain when its model refuses ✅ pass live 01-fallback-chain-live-e2e.txt line 3; 02-live-chain-transcript.txt step E (weak refused once, the chain route answered)
A session that moved to another model is visible: crew-state adds model-drift: &lt;live&gt; live (recorded &lt;recorded&gt;) ✅ pass live 02-live-chain-transcript.txt step E: model-drift: lab-equal/gpt-6.1-sol live (recorded deepseek/deepseek-v4-pro); pane shows `Warning: Fallback: deepseek/deepseek-v4-pro:high -> lab-equal/gpt-6.1-so…
A fallen-back session returns to its recorded model at the next prompt, even when the global policy says never; the record follows and the model-drift note clears ✅ pass live 02-live-chain-transcript.txt step F: second prompt typed into the real omp pane, recorded route answered, record model=deepseek/deepseek-v4-pro, crew-state line has no model-drift. Also 01-fallback-…
Control: without the overlay, a global fallbackRevertPolicy: never keeps the session on its fallback (so the return proof is not empty) ✅ pass live 01-fallback-chain-live-e2e.txt line 5
A fresh launch removes a stale live-model record for that task ✅ pass live 02-live-chain-transcript.txt step A: seeded model=deepseek/... error=402 old run, ran real bin/fm-spawn.sh ... --harness omp --model openai-codex/gpt-6.1-sol --scout, record gone. tmux and treehou…
A relaunch on a stand-in model does not show the old model or the old error ✅ pass live 02-live-chain-transcript.txt step G: real bin/fm-spawn.sh sol-lane-q1 --relaunch --model lab-equal/gpt-6.1-sol on real tmux; record absent right after the relaunch; crew-state shows no model-drift a…
A secondmate-style omp primary in a lab home publishes &lt;home&gt;/state/.omp-live-model through the tracked home extension, and the parent's crew-state shows its run-error ✅ pass live 03-lab-primary-transcript.txt steps H1 and H2: real omp on tmux -L fm-lab in a marked lab home; weak model 0 requests; parent line `... · run-error: ChatGPT rate limit exceeded. retry-after-ms=36000…
A secondmate omp launch removes the home's stale state/.omp-live-model record ⏸️ untested no The prior payload did not establish a live result for this scenario. Its only proof was tests/fm-omp-harness.test.sh (`ok - fm-spawn: a real omp secondmate launch preserves primary posture and super…
Teardown removes the task's live-model record ✅ pass live 04-teardown-transcript.txt: sol-lane-q3.live-model present before, real bin/fm-teardown.sh sol-lane-q3 --force --drop-file ... rc=0, no task files after. tmux and treehouse were the repo's fixture…
The live guard runs on a Mac that has no timeout binary (review decision F1) ✅ pass live 05-fallback-guard-no-timeout-binary.txt: mechanism bash, 5 of 5 ok, exit 0
The owner's global omp config is not edited by any of these runs ✅ pass live shasum -a 256 ~/.omp/agent/config.yml = e941fd54...df97 and modified time 1791577813, identical before and after all runs
A descendant omp process that does not own the session lock cannot overwrite the home's live-model record ⏸️ untested no The prior payload did not establish a live result for this scenario. Its only proof was the turn-end extension case in tests/fm-omp-harness.test.sh (the record keeps model=deepseek/deepseek-v4-pro…
  • FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh (real omp 18.8.7, scripted providers): 5 of 5 ok
  • FM_TIMEOUT_MECHANISM_OVERRIDE=bash FM_OMP_FALLBACK_LIVE=1 tests/fm-omp-fallback-chain-live-e2e.test.sh (no timeout or gtimeout binary used): 5 of 5 ok
  • tests/fm-omp-harness.test.sh: exit 0, includes the spawn stale-record step, the secondmate launch stale-record step, the publisher lifecycle, and the lock-owner guard
  • tests/fm-crew-state.test.sh: exit 0, includes omp live-model record adds model-drift and run-error only on positive evidence
  • bash 02-live-chain-driver.sh (evidence dir): real bin/fm-spawn.sh -> launch line run unchanged in a real tmux pane with the real omp TUI -> real bin/fm-crew-state.sh; steps A to G, 0 failed checks
  • bash 03-lab-primary-driver.sh (evidence dir): bin/fm-lab-home.sh create, real omp as the session command on tmux -L fm-lab, tracked .omp/extensions discovered by omp, then bin/fm-crew-state.sh mate from a parent home; 0 failed checks
  • bash 04-teardown-driver.sh --force --drop-file &lt;fixture file&gt; (evidence dir): real bin/fm-spawn.sh then real bin/fm-teardown.sh in a throwaway home; record removed
  • shasum -a 256 ~/.omp/agent/config.yml and stat -f %m before and after all runs: identical (read-only check)
  • git status --short after all runs: clean at 03042cfc
⚠️ **Document** - 1 info
  • ℹ️ .agents/skills/stuck-crewmate-recovery/SKILL.md:35 - No agent skill tells the supervisor what to do when bin/fm-crew-state.sh prints model-drift or run-error. The script header in bin/fm-crew-state.sh owns the meaning of the two components, and that is accurate. The gap is the action: relaunch on a stand-in model after a Sol run stops. I did not add this, because new agent guidance is a human decision and nothing existing is wrong. Recommendation (confidence: medium): add one short pointer in the stuck-crewmate-recovery skill in a follow-up change.
✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

…rift

An omp session recorded on gpt-6.1-sol fell down a global fallback chain to a weaker model and stayed there. The tracked session overlay now empties the Sol chain so a failure stops with the provider error, and pins omp's own return to the recorded model. Sessions publish the model they serve, and fm-crew-state reports model-drift and run-error.
@MrGTV-love
MrGTV-love merged commit ca6c277 into main Oct 10, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant