Skip to content

feat: add pooled Codex capacity checks and declared model failover - #39

Closed
MrGTV-love wants to merge 48 commits into
mainfrom
fm/fm-omp-codex-account-failover
Closed

MrGTV-love wants to merge 48 commits into
mainfrom
fm/fm-omp-codex-account-failover

Conversation

@MrGTV-love

@MrGTV-love MrGTV-love commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Intent

if codex is exhausted, why not switch to a different model like Deepseek flash v4, or sonnet 5.5 via tc run?

vs hold up work??!! we have a model matrix, so why not switch models accordingly to the fallback??

also confused. we have two codex credentials. the other one is completely unutilized

why did you hold up the work vs switching models?

okay. paid calls for development work do not. update so that deepseek flash replaces glm 5.3

deepseek/deepseek-v4-flash is what I think it is

Context: on 2026-10-05 the fleet sat about 13 hours paused because quota-axi reported the Codex weekly window exhausted. quota-axi reads only one Codex account (greentree promax). omp pools two openai-codex accounts (greentree promax and gmail prolite; omp usage lists both, and the gmail account also shows a banked saved reset). Workers and every no-mistakes review agent run on omp openai-codex models, so all work and all PR pipelines stopped together. config/crew-dispatch.json (the model matrix) already names stand-ins per rule: z-ai GLM 5.3 for Sonnet-class work, GLM 5.3 Flash for Luna-class work, and gpt-6.1-sol and Opus 5.5 (Claude only through the TeamClaude proxy) as equals for the strongest class. None were applied. Earlier, on 2026-09-30, a worker stalled about 9 hours on usage_limit_reached on one account while the other account had quota; a relaunch recovered it.

no attempt limits

What Changed

  • Assess pooled Codex capacity from destination-scoped OMP usage, accounting for model entitlements, independent Chat/Spark limits, and stale reset windows instead of relying on quota-axi’s single-account reading.
  • Add ordered, rule-specific OMP and TeamClaude stand-ins for dispatch, fresh spawn, and relaunch; preserve rule identity and enforce model-catalog, launch-policy, and credential boundaries.
  • Enable usage-aware worker retries and guarded quota-event recovery, honoring explicit exits, holds, and pending closes while scanning past failed tasks; prevent discovery probes from overwriting primary extension ownership markers.

Risk Assessment

⚠️ Medium: The change spans authentication-sensitive routing and transactional worker lifecycle behavior, but no material source defect or unnecessary introduced component was substantiated; the restart-only, one-attempt recovery limits match the explicitly authorized containment.

Testing

Eight targeted suites completed successfully after bounded redrives, and eight live routing, authentication, fallback, and recovery scenarios passed with CLI transcripts and captured-terminal HTML retained. Initial suite timeouts and print-mode stdin setup were resolved; OAuth-dependent proofs and watcher capacity inspection remain untested, so the overall verdict is inconclusive. All disposable setups were removed.

  • Live validation: ⚠️ inconclusive - 8 of 12 scenarios driven live against the product
Scenario Result Live Evidence
Fresh spawn launches the rule's permitted stand-in when the primary route is exhausted ✅ pass live current-live-lifecycle.json and current-fresh-worker.html: real spawn selected the declared local OMP stand-in, published rule_1 and its concrete profile, and the worker returned LIVE_WORKER_COMPLETED…
OMP continues through a destination-owned fallback chain after provider exhaustion ✅ pass live current-omp-native-fallback-final.json: real OMP requested the blocked local model, switched to the available model, returned LIVE_STANDIN_COMPLETED, and exited zero.
Typed intake keeps unestablished native capacity unranked and respects floors and approval ✅ pass live current-product-transcript.txt, Typed routing: the real resolver retained unknown pool capacity, spendPriority and runway; disclosed an unverifiable floor; and required explicit approval.
Alternate Claude authentication cannot borrow native default-account headroom ✅ pass live current-live-authentication.json: neutral authentication selected Claude; helper, gateway, alternate-store, base-URL, and custom-header selectors selected the next eligible candidate. The helper was n…
An accidental Anthropic API key refuses launch before task publication ✅ pass live current-live-authentication.json: real fm-spawn refused the API-key launch without --allow-api-key, and no task metadata was created.
A caller-only model catalog cannot authorize a fallback in another destination ✅ pass live current-destination-catalog.json: the same primary could launch the stand-in in its configured project, but a catalog-empty destination refused with no supported permitted fallback and published no ta…
Unsafe recovery refuses before changing instructions or stopping the worker ✅ pass live current-live-lifecycle.json: automatic quota control refused unknown adopted authentication; an invalid dispatch rule also refused. Brief bytes and uncommitted local work remained unchanged.
Manual replacement continues on an explicit profile while preserving local work and context ✅ pass live current-live-lifecycle.json and current-recovered-worker.html: real control relaunched in the same endpoint and worktree, retained PRESERVE_LOCAL_WORK, published the replacement profile, and completed…
An exhausted Codex OAuth account rotates to its usable sibling without pausing work ⏸️ untested no Tried isolated native usage and real OMP with disposable local providers. Those providers can exercise model fallback but cannot supply native OAuth usage-aware rotation. This requires two authorized…
Whole Codex-pool exhaustion launches the declared DeepSeek or authenticated TeamClaude stand-in ⏸️ untested no The isolated catalog contained only Apple on-device, native Codex accounts were absent, and TeamClaude was not on PATH; isolated launcher readiness refused. Local providers proved the routing mechanis…
Stale, missing, saved-reset, shared-meter, and wrong-tier native reports cannot falsely exhaust a live Codex pool ⏸️ untested no Deterministic report fixtures exercised the classifier but are not live OAuth evidence. A live proof requires authorized accounts and controlled native report conditions. Fabricated credentials, force…
The watcher drives quota capacity inspection and queues the intended recovery escalation ⏸️ untested no Tried the real watcher under a marked lab home and private socket. Its control invocation injects FM_STATE_OVERRIDE even for the stock home state directory, defeating the lab allowance. Direct automat…
Evidence: Current product command transcript

Source: Current product command transcript

## Typed routing
$ quota-axi --json
exit=0
{
  "generatedAt": "2026-10-08T12:44:53.594Z",
  "schemaVersion": 5,
  "providers": [
    {
      "provider": "claude",
      "windows": [
        {
          "id": "seven_day",
          "label": "week",
          "kind": "weekly",
          "percentRemaining": 75,
          "resetsAt": "2026-10-15T12:44:53.430071+00:00",
          "pace": {
            "status": "ahead",
            "reservePercentPoints": -25,
            "burnMultiple": 921951.2195
          }
        }
      ],
      "state": {
        "status": "fresh",
        "stale": false,
        "refreshedAt": "2026-10-08T12:44:53.430071+00:00",
        "reused": true
      },
      "quotaSemantics": {
        "status": "known",
        "effectiveAvailability": [
          {
            "scope": "all_models",
            "status": "known",
            "effectivePercentRemaining": 75,
            "boundedBy": [
              "seven_day"
            ],
            "limitingWindowIds": [
              "seven_day"
            ],
            "pace": {
              "status": "ahead",
              "aheadWindowIds": [
                "seven_day"
              ],
              "worstReservePercentPoints": -25,
              "worstReserveWindowId": "seven_day"
            },
            "runway": {
              "status": "projected_exhaustion",
              "usableRunwaySeconds": 0,
              "projectedExhaustedAt": "2026-10-08T12:44:54.086Z",
              "limitingWindowId": "seven_day",
              "projectionConfidence": "early"
            },
            "selection": {
              "status": "known",
              "spendPriority": -100
            }
          }
        ]
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "codex",
      "windows": [
        {
          "id": "weekly",
          "label": "week",
          "kind": "weekly",
          "percentRemaining": 0,
          "resetsAt": "2026-10-15T12:44:53.430071+00:00",
          "pace": {
            "status": "ahead",
            "reservePercentPoints": -100,
            "burnMultiple": 3687804.878
          }
        }
      ],
      "state": {
        "status": "fresh",
        "stale": false,
        "refreshedAt": "2026-10-08T12:44:53.430071+00:00",
        "reused": true
      },
      "quotaSemantics": {
        "status": "known",
        "effectiveAvailability": [
          {
            "scope": "all_models",
            "status": "known",
            "effectivePercentRemaining": 0,
            "boundedBy": [
              "weekly"
            ],
            "limitingWindowIds": [
              "weekly"
            ],
            "pace": {
              "status": "ahead",
              "aheadWindowIds": [
                "weekly"
              ],
              "worstReservePercentPoints": -100,
              "worstReserveWindowId": "weekly"
            },
            "runway": {
              "status": "exhausted_now",
              "usableRunwaySeconds": 0,
              "limitingWindowId": "weekly",
              "projectedExhaustedAt": "2026-10-08T12:44:53.594Z"
            },
            "selection": {
              "status": "known",
              "spendPriority": -100
            }
          }
        ]
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "cursor",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "copilot",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": [],
        "unresolvedWindowIds": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "grok",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "kimi",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "zai",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "agy",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "alibaba",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "opencode-go",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": [],
        "unresolvedWindowIds": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "commandcode",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "minimax",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "mimo",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": [],
        "unresolvedWindowIds": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "deepseek",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": [],
        "unresolvedWindowIds": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "openrouter",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": [],
        "unresolvedWindowIds": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "elevenlabs",
      "windows": [],
      "state": {
        "status": "unavailable",
        "stale": false,
        "error": "not_in_snapshot"
      },
      "quotaSemantics": {
        "status": "unknown",
        "effectiveAvailability": []
      },
      "accountKeys": [
        "default"
      ]
    },
    {
      "provider": "devin",
      "windows": []

... [11224 bytes truncated] ...

ved_brief": true, "automatic_refusal_preserved_local_work": true}
$ bash ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-control.sh task relaunch --dispatch-rule rule_99 --note Invalid rule must not stop the worker.
exit=1

fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l
error: invalid dispatch fallback configuration: jq: error (at <stdin>:19): dispatch rule does not contain the requested profile
{"invalid_rule_preserved_brief": true}
$ bash ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-control.sh task relaunch --harness omp --model fm-lab-secondary/available --effort high --note Deliberately continue on the permitted local profile, preserving work.
exit=0
relaunched task harness=omp from=omp model=fm-lab-secondary/available effort=high backend=tmux endpoint=primary:fm-task worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/worker
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l
{"replacement_metadata": "window=primary:fm-task\nendpoint_task_id=task\nworktree=~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/worker\nproject=~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/project\nharness=omp\nkind=scout\ntasktmp=/tmp/fm-task\nmodel=fm-lab-secondary/available\neffort=high\nbusy_gen=g1791463618.99930.5579\nspawn_gen=s1791463618.96831.19470\ndispatch_rule=rule_1\ncontrol_relaunch_tx=85658.20261008T124654Z.83\n", "replacement_preserved_local_work": "PRESERVE_LOCAL_WORK\n"}
Local provider requests: [{"path": "/v1/chat/completions", "model": "available"}, {"path": "/v1/chat/completions", "model": "available"}, {"path": "/v1/chat/completions", "model": "available"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "gpt-6.1-sol"}, {"path": "/v1/chat/completions", "model": "available"}, {"path": "/v1/chat/completions", "model": "available"}, {"path": "/v1/chat/completions", "model": "available"}]
Cleanup: Private tmux server stopped and lab removed

## Destination catalog containment
$ bash ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-lab-home.sh create ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l
exit=0
~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l

$ bash ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-brief.sh fresh ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/project --scout
exit=0
scaffolded: ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/data/fresh/brief.md (scout; replace {TASK} and {FIRSTMATE_SPEC})

$ bash ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-spawn.sh fresh ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/project --scout --harness claude --model sonnet --effort high --dispatch-rule rule_1
exit=0
model-matrix fallback launched omp fm-lab-secondary/available for rule_1
spawned fresh harness=omp kind=scout window=primary:fm-fresh worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/slots/.treehouse/project-2887b3/1/project
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l
{"fresh_spawn_metadata": "window=primary:fm-fresh\nendpoint_task_id=fresh\nworktree=~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/slots/.treehouse/project-2887b3/1/project\nproject=~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/project\nharness=omp\nkind=scout\ntasktmp=/tmp/fm-fresh\nmodel=fm-lab-secondary/available\neffort=high\nbusy_gen=g1791464045.85208.9063\nspawn_gen=s1791464045.64539.16129\ndispatch_rule=rule_1\n"}
$ bash ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-brief.sh catalog-guard ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/no-catalog --scout
exit=0
scaffolded: ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/data/catalog-guard/brief.md (scout; replace {TASK} and {FIRSTMATE_SPEC})

$ bash ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-spawn.sh catalog-guard ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/no-catalog --scout --harness claude --model sonnet --effort high --dispatch-rule rule_1
exit=1

fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l
error: dispatch rule_1 has exhausted capacity and no supported permitted fallback
{"caller_only_catalog_guard": {"exit": 1, "task_metadata_published": false}}
Local provider requests: [{"path": "/v1/chat/completions", "model": "available"}, {"path": "/v1/chat/completions", "model": "available"}, {"path": "/v1/chat/completions", "model": "available"}]
Cleanup: Private tmux server stopped and lab removed

## Native fallback
$ omp --version
exit=0
omp/18.8.1

$ omp usage --provider openai-codex --json
exit=0
{
  "generatedAt": 1791463911109,
  "reports": [],
  "accountsWithoutUsage": [],
  "disabledCredentials": [],
  "capacity": {}
}

$ omp --cwd ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/.no-mistakes/live-0fvpvst3/project --no-title --no-session --no-tools --no-lsp --no-skills --no-rules --no-extensions -e ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/.no-mistakes/live-0fvpvst3/project/.omp/extensions/local.ts --config ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/.no-mistakes/live-0fvpvst3/fallback.yml --config ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/.omp/fm-worker-overlay.yml --model fm-lab-primary/blocked --thinking off --max-time 45 -p Return the completion marker.
exit=0
LIVE_STANDIN_COMPLETED
Working...
Local provider requests: [{"path": "/v1/chat/completions", "model": "blocked"}, {"path": "/v1/chat/completions", "model": "blocked"}, {"path": "/v1/chat/completions", "model": "blocked"}, {"path": "/v1/chat/completions", "model": "blocked"}, {"path": "/v1/chat/completions", "model": "blocked"}, {"path": "/v1/chat/completions", "model": "blocked"}, {"path": "/v1/chat/completions", "model": "available"}]
Cleanup: Disposable lab removed

## Provider availability
$ omp usage --provider openai-codex --json
exit=0
{
  "generatedAt": 1791464097585,
  "reports": [],
  "accountsWithoutUsage": [],
  "disabledCredentials": [],
  "capacity": {}
}

$ bash ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-teamclaude-launch.sh --check
exit=1

error: TeamClaude is not installed: install an executable teamclaude on PATH or under one Node version in ~/.nvm/versions/node; refusing to launch Claude without it
Cleanup: Disposable home removed
Evidence: Captured fresh worker terminal

Source: Captured fresh worker terminal

<!doctype html><meta charset="utf-8"><title>fresh-worker</title><body style="background:#101418;color:#dce4e8"><pre style="font:14px monospace;line-height:1.3"> Never inspect or change any other home&#x27;s endpoint namespace; this authorization is limited to the exact task paths
 named by this brief.
 When this task works on Firstmate itself, the repository root AGENTS.md (also imported by CLAUDE.md) is project
 content and the supervisor contract for the firstmate managing you: follow this brief instead of that supervisor
 contract.
 Project instructions still govern the work wherever they do not conflict with this worker identity, including
 CONTRIBUTING.md and firstmate-coding-guidelines for Firstmate changes.

 You are a crewmate: an autonomous worker agent managed by firstmate. Work on your own; do not wait for a human.

 Task

 Captain&#x27;s intent

 Exercise only a local completion.

 Firstmate spec

 Do not call tools or edit files. Return only the completion marker.

 Herdr lifecycle declaration - NOT ENABLED

 HARD SAFETY GATE: this scaffold cannot inspect the task text filled in above.
 If the task will start, stop, delete, restart, profile, or otherwise drive Herdr lifecycle behavior, stop and
 regenerate the brief with --herdr-lab before dispatch.
 Do not add Herdr lifecycle commands to this unguarded brief by hand.

 Setup

 You are in a disposable git worktree of
 ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/project, at a detached HEAD on
 a clean default branch.
 This is a SCOUT task: the deliverable is a written report, not a PR.
 The worktree is your laboratory - install, run, edit, and make scratch commits freely; all of it is discarded at
 teardown.
 The report is the only thing that survives, so anything worth keeping must be in it.

 Rules

 1. Never push to any remote and never open a PR.
 2. Stay inside this worktree; the only files you may write outside it are the report and the status file below.
 3. Use gh-axi for GitHub operations and chrome-devtools-axi for browser operations.
 4. Report status by appending one line:
    echo &quot;{state} [at=&lt;epoch&gt;]: {one short line}&quot; &gt;&gt;
    &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/state/fresh.status&#x27; &amp;&amp; { [
    ! -e &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/config/fleet-ledger&#x27;
    ] || &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-fleet-ledger.sh&#x27;
    appended &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/config&#x27;
    &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/state/fresh.status&#x27;
    &gt;/dev/null 2&gt;&amp;1 || true; }
    States: working, needs-decision, blocked, paused, done, failed.
    Substitute &lt;epoch&gt; with the current Unix time in seconds - run date +%s and write the number it printed; a stamp
    that is not plain digits records no time at all.
    Each append wakes firstmate, so report sparingly: only phase changes a supervisor
    would act on and the needs-decision/blocked/paused/done/failed states. No step-by-step
    FYI progress lines; firstmate reads your pane for that.
    Whenever you mention a PR anywhere - a status line, your terminal, a summary - write its full
    https:// URL exactly as the forge printed it, never a bare number such as &quot;PR 108&quot;; firstmate
    copies that URL from your line rather than assembling one.
    Use paused: {why} - distinct from blocked: - when deliberately waiting for work or an external condition expected
    to clear on its own, including your own validation round.
    Before ending your turn with your own background shell or monitor still running, or before waiting on your own
    pipeline run or a long foreground command, append paused [at=&lt;epoch&gt;]: {job and completion condition} to the status
    file.
    Name what you are waiting for and what will let you resume; do not repeat the declaration on every poll.
    Do not declare active implementation or reasoning as a wait.
    Firstmate may still raise one first-sight alert; the declared wait then uses the existing long recheck cadence
    instead of repeated possible-wedge alarms.
    When you know when the wait clears, include until &lt;YYYY-MM-DDTHH:MMZ&gt; (UTC) for a recheck at that time.
    Follow the resolution rule below when the wait clears, then resume the task.
    Use blocked: when you are stuck and need help.
 5. If you hit the same obstacle twice, append blocked [at=&lt;epoch&gt;]: {why} and stop; firstmate will help.
 6. If a decision belongs to a human (product choices, destructive actions),
    append needs-decision [at=&lt;epoch&gt;]: {summary of options} and stop. Firstmate will reply with the decision.
    A decision or blocker you opened stays open until a resolved line carrying its exact key lands; a later done: or
    working: line never closes it, even when the answer is what started that work.
    Firstmate&#x27;s reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a
    firstmate reply, append resolved [at=&lt;epoch&gt;]: {how it cleared} yourself (same [key=&lt;slug&gt;] if you opened it with
    one) as you resume.
 7. Never administer infrastructure that every lane shares. Two things are shared:
   - The no-mistakes daemon - one instance serving every lane/home, so stopping, restarting, or
     updating it kills other lanes&#x27; in-flight pipeline runs; only firstmate manages the daemon.
     Before you append blocked: about the pipeline, run no-mistakes daemon status and
     no-mistakes axi status. If the daemon socket refuses connections or is missing, append
     blocked [at=&lt;epoch&gt;]: {the daemon error} and stop even when the local run record still says running or
     fixing, because that record can be stale after the daemon exits. A run record failed with a
     daemon error is also a real block.
     Only after ruling out socket refusal, if the run is still running or fixing, reattach and keep
     going. A drive-call error, timeout, slow read, or generic unreachability is NOT a daemon error:
     the daemon accepts respond immediately and runs the round in the background, so a killed or
     timed-out call was only waiting for a read while the run kept working.
   - The worktree pool your own worktree came from, and the repository every lane&#x27;s worktree
     shares. Never create, remove, return, prune, move, or reassign a worktree or pool slot, and
     never write into a sibling slot&#x27;s directory. Rule 2 does not cover this: removing a worktree
     is administration rather than an edit outside your directory, and it lands on lanes that are
     running right now. The act is the rule and commands are only examples of it - treehouse
     get/return/remove/prune, the equivalent operations on any other worktree provider or runtime
     backend, and git worktree add|remove|move|prune. A slot that looks unused is not evidence
     that it is free, and returning your own worktree is firstmate&#x27;s job at cleanup, not yours.
    If you genuinely need a second checkout, another slot, or the daemon touched, append
    blocked [at=&lt;epoch&gt;]: {what you need} and stop; firstmate arranges it.

 Firstmate instruction inbox

 Firstmate steers you through durable message files in
 &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/state/fresh.inbox&#x27;.
 When a terminal message says an instruction is waiting there - and at any natural checkpoint when you are unsure -
 list
 &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/state/fresh.inbox&#x27;/*.msg,
 read and act on each message in numeric order, then acknowledge each handled message by moving it: mv
 &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/state/fresh.inbox&#x27;/NNN.msg
 &#x27;~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/state/fresh.inbox&#x27;/handled/.
 The move IS the acknowledgement: without it firstmate rings again and eventually treats you as stuck. An empty or
 absent inbox needs no action.

 Definition of done

 Write your findings to
 ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/data/fresh/report.md.
 The report must stand alone: what you did, what you found, the evidence (commands run, output, file:line references),
 and what you recommend.
 If your deliverable is a visual artifact the captain will review and iterate on, use the lavish-axi rule: arm your
 board with bin/fm-procevent-lavish.sh arm &lt;artifact.html&gt; --for &lt;task-id&gt;; never run lavish-axi poll yourself. Re-arm
 with the reply after each nonterminal round to acknowledge it, route the board feedback through your steering inbox,
 write needs-decision [key=board-review] with the live board URL when the captain owes a decision, and stop at
 session_ended or an empty End without re-arming - acknowledge that final round with bin/fm-procevent.sh handled
 &lt;source-id&gt; &lt;sequence&gt; to conclude and retire your board.
 Before reporting done, read and follow
 ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/.agents/skills/captain-hold-life
 cycle/SKILL.md and pass its shared completion gate for the report and any visual review.
 When the report is complete, append done [at=&lt;epoch&gt;]: {one-line conclusion} to the status file and stop.
 If your findings reveal work that should ship (e.g. you reproduced a bug and the fix is clear), say so in the report;
 firstmate may promote this task in place, and you would then receive mode-specific ship instructions as a follow-up
 message.


 LIVE_WORKER_COMPLETED

❯
 π · ⬢ available · 🌳 project/project · ⑂ detached · ◫ 29.7%/33K ⟲                                LIVE_WORKER_COMPLETED
</pre></body>
Evidence: Captured exhausted worker terminal

Source: Captured exhausted worker terminal

<!doctype html><meta charset="utf-8"><title>quota-worker</title><body style="background:#101418;color:#dce4e8"><pre style="font:14px monospace;line-height:1.3">
                                            ████████████
                                               ██  ██       ▄▀▀▄ █▀▄▀▄ █▀▀▄
                                               ██  ██       ▀▄▄▀ █ █ █ █▄▄▀
                                               ▒▒  ██                  █
                                                   ██       v18.8.1

                          Tip: Say `workflowz` in your message to drive the task with parallel
                                     subagents in eval — watch it glow as you type


────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Update Available
 New version 18.8.4 is available. Run: omp update
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────


 ⁣FIRSTMATE_OP: v1 launch-brief: # Current worker role contract
 You are a crewmate: an autonomous worker agent managed by firstmate.
 This section establishes your current identity before every project or task instruction below and supersedes any
 conflicting role identity in those instructions.
 Do the assigned work yourself and report only to firstmate; do not adopt a firstmate or secondmate supervisor
 identity, delegate the task, run fleet supervision, or address the captain.
 Your steering inbox is
 ~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/l/state/task.inbox; this exact
 path belongs to your current task even when it is outside the worktree or under the supervising firstmate home, so
 read and acknowledge its messages and do not reject it as another home&#x27;s state.
 Never inspect or change any other home&#x27;s endpoint namespace; this authorization is limited to the exact task paths
 named by this brief.
 When this task works on Firstmate itself, the repository root AGENTS.md (also imported by CLAUDE.md) is project
 content and the supervisor contract for the firstmate managing you: follow this brief instead of that supervisor
 contract.
 Project instructions still govern the work wherever they do not conflict with this worker identity, including
 CONTRIBUTING.md and firstmate-coding-guidelines for Firstmate changes.

 Task

 Captain&#x27;s intent

 Verify that local work is preserved during quota recovery.

 Firstmate spec

 Do not edit or call tools. Return only the completion marker.


 Error: 429 usage_limit_reached: isolated provider quota exhausted
   usage_limit_reached: isolated provider quota exhausted (type=usage_limit_reached param=usage_limit_reached)

 Error: Retry failed after 1 attempts: Provider requested 1800000ms wait, exceeds retry.maxDelayMs (300000ms). Original
 error: 429 usage_limit_reached: isolated provider quota exhausted
 usage_limit_reached: isolated provider quota exhausted (type=usage_limit_reached param=usage_limit_reached)

❯
 π · ⬢ gpt-6.1-sol · 🌳 project/worker · ⑂ lab-task · ◫ 23.2%/33K ⟲
</pre></body>
Evidence: Isolated provider prerequisites

Source: Isolated provider prerequisites

{
  "observations": [
    {
      "command": [
        "omp",
        "usage",
        "--provider",
        "openai-codex",
        "--json"
      ],
      "exit": 0,
      "stdout": "{\n  \"generatedAt\": 1791464097585,\n  \"reports\": [],\n  \"accountsWithoutUsage\": [],\n  \"disabledCredentials\": [],\n  \"capacity\": {}\n}\n",
      "stderr": ""
    },
    {
      "command": [
        "omp",
        "models",
        "--json"
      ],
      "exit": 0,
      "stdout": "{\"models\":[{\"provider\":\"apple\",\"kind\":\"chat\",\"id\":\"on-device\",\"selector\":\"apple/on-device\",\"name\":\"Apple AFM 3 Core Advanced\",\"contextWindow\":8192,\"maxTokens\":8192,\"reasoning\":false,\"thinking\":null,\"input\":[\"text\",\"image\"],\"cost\":{\"input\":0,\"output\":0,\"cacheRead\":0,\"cacheWrite\":0},\"pricingStatus\":\"unknown\"}]}\n",
      "stderr": ""
    },
    {
      "command": [
        "bash",
        "~/.no-mistakes/worktrees/32d18ed9638d/01M49AAWM08X3K86CZWW2AQZ0P/bin/fm-teamclaude-launch.sh",
        "--check"
      ],
      "exit": 1,
      "stdout": "",
      "stderr": "error: TeamClaude is not installed: install an executable teamclaude on PATH or under one Node version in ~/.nvm/versions/node; refusing to launch Claude without it\n"
    }
  ],
  "isolation": "Disposable HOME, agent directory, XDG configuration; normal credentials never imported",
  "cleanup": "Disposable home removed"
}

Pipeline

Updates from git push no-mistakes

... (79 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)

🔧 **Review** - 8 issues found → auto-fixed (34) → no changes applied (2) ✅

✅ No issues found.

⚠️ **Test** - 1 warning
  • ⚠️ The live checks demonstrate native fallback against disposable local providers, authentication guards, destination-scoped discovery, and recovery containment, but do not establish the central two-account Codex rotation followed by an actual declared Deepseek or TeamClaude launch. Isolated native usage returned no authenticated accounts; local API-key providers cannot exercise OAuth usage-aware rotation. Completing that proof requires authorized disposable Codex OAuth accounts with controlled capacity and authenticated stand-in access, without modifying production credentials or shared configuration.
  • ⚠️ live validation verdict: inconclusive (7 of 10 scenarios were driven live against the product); untested: An exhausted Codex account rotates to the usable sibling instead of pausing work, Whole-pool exhaustion launches the declared Deepseek or TeamClaude stand-in, Stale, unmeasured, saved-reset, shared-meter, and wrong-tier evidence cannot falsely exhaust the pool
  • Live validation: ⚠️ inconclusive - 7 of 10 scenarios driven live against the product
Scenario Result Live Evidence
An exhausted Codex account rotates to the usable sibling instead of pausing work ⏸️ untested no Production assessment correctly classified an exhausted account plus a usable sibling as usable, but those were constructed protocol inputs. Real isolated OMP usage returned no authenticated accounts.…
Whole-pool exhaustion launches the declared Deepseek or TeamClaude stand-in ⏸️ untested no The real matrix consumer retained the exact Deepseek declaration, and native OMP successfully executed a local-provider fallback chain. That does not prove a fresh launch from genuinely exhausted Code…
Stale, unmeasured, saved-reset, shared-meter, and wrong-tier evidence cannot falsely exhaust the pool ⏸️ untested no All constructed protocol cases produced the expected results, including uppercase Spark and paid-model entitlement checks. Native live usage supplied no accounts or windows, so these provider-state bo…
Unknown native capacity does not authorize a paid fallback or borrow positive single-account quota ✅ pass live live-cli-scenarios.json; product-observations.json
The worker overlay enables native recovery while preserving destination-owned fallback chains ✅ pass live native-retry-smoke.json
Prospective tmux credential imports require billing opt-in, while removals and established sessions retain correct precedence ✅ pass live live-cli-scenarios.json; product-observations.json
Catalog discovery uses the destination project and accepts a relative-PATH OMP executable ✅ pass live live-cli-scenarios.json
Only explicit rule-scoped fallback permissions are accepted ✅ pass live live-cli-scenarios.json; product-observations.json
Automatic recovery refuses unknown adopted authentication before changing or stopping the worker ✅ pass live recovery-live-smoke.json
A quota-origin generation receives one recovery attempt and remains contained across later events and endpoint death ✅ pass live recovery-live-smoke.json
  • bin/fm-test-run.sh --json <evidence>/targeted-suite-timing.json tests/fm-dispatch-capacity.test.sh tests/fm-dispatch-resolve.test.sh tests/fm-quota-choose.test.sh tests/fm-omp-harness.test.sh tests/fm-session-end-relaunch.test.sh tests/fm-control-relaunch.test.sh tests/fm-spawn-claude-api-key-guard.test.sh tests/fm-spawn-dispatch-profile.test.sh

  • bin/fm-test-run.sh --json &lt;evidence&gt;/targeted-setup-redrive-timing.json tests/fm-omp-harness.test.sh tests/fm-control-relaunch.test.sh tests/fm-spawn-dispatch-profile.test.sh with normal toolchain TMPDIR; corrected the initial secondmate fixtures being placed inside the protected repository

  • Real isolated omp usage --provider openai-codex --json --no-extensions, omp models --json --no-extensions, and bin/fm-dispatch-capacity.sh --harness omp --model &lt;model&gt; --json

  • Production Bash fm_omp_codex_capacity with freshly constructed usage protocol inputs covering usable siblings, exhausted pools, saved resets, missing/stale siblings, negative shared flags, uppercase Spark, and paid-model entitlement

  • Real private tmux sessions: compared production environment previews with actual imports, removal markers, and established-session precedence

  • Real fm-spawn.sh preflight with caller credential sentinels, explicit API opt-in, launch allowlist filtering, invalid fallback declarations, and conflicting rule permissions

  • Production destination-scoped OMP catalog queries from a provider-disabled caller project, including a relative-PATH installation pointing to the real OMP executable

  • bin/fm-quota-choose.sh --snapshot &lt;disposable-positive-legacy-snapshot&gt; --candidate omp:openai-codex/gpt-6.1-sol; empty native pool returned none

  • Real OMP v18.7.0 RPC sessions consumed the tracked worker overlay and a disposable native fallback-chain overlay; a loopback endpoint returned HTTP 429 insufficient_quota, after which native OMP switched providers and completed with LOCAL_NATIVE_FALLBACK_OK

  • Real native OMP idle worker, direct fm-control.sh, production watcher, and durable wake queue exercised adopted-authentication refusal and generation-bound recovery containment using seeded task records

  • command -v teamclaude and isolated bin/fm-teamclaude-launch.sh --check; the supported launcher reported TeamClaude unavailable

  • Stopped owned endpoints and servers, removed disposable worktree fixtures and inactive sockets, and confirmed no owned disposable processes remained

  • ⚠️ The Test agent did not finish within its invocation budget. Reported: agent run tests timed out after 4h0m0s: agent last produced output 5s ago (2375 observed); agent reported: acpx parse events: context deadline exceeded. This is a budget or provider-slowness cut, not a code failure. Re-running the same request costs another full budget, so no further attempt is made automatically. If this repository's targeted tests or evidence gathering routinely approach the default 30m0s, raise test_agent_timeout in global config. Respond with fix to spend another budget: a repair turn runs only for selected findings other than this budget cut, then validation re-runs. Or abort and retry after raising the budget.

🔧 No changes applied.
1 warning still open:

  • ⚠️ live validation verdict: inconclusive (7 of 10 scenarios were driven live against the product); untested: An exhausted Codex OAuth account rotates to the usable sibling without pausing work, Whole Codex-pool exhaustion launches the declared DeepSeek or authenticated TeamClaude stand-in, Stale, missing, saved-reset, shared-meter, and wrong-tier native evidence cannot falsely exhaust a live Codex pool
  • Live validation: ⚠️ inconclusive - 7 of 10 scenarios driven live against the product
Scenario Result Live Evidence
OMP preserves a configured fallback chain and completes after a local provider returns usage_limit_reached ✅ pass live Native OMP local-provider fallback: real OMP requested the blocked provider, then the available provider, and returned LIVE_STANDIN_COMPLETED.
Fresh spawn launches a declared supported stand-in when bound native Claude capacity is exhausted ✅ pass live Routing and recovery transcript; Fresh declared stand-in terminal. Real spawn reported the matrix fallback, persisted its selected profile and rule, and the worker returned LIVE_WORKER_COMPLETED.
Typed intake does not turn single-account exhaustion, unknown pooled capacity, floors, or approval into fallback authority ✅ pass live Live typed-intake decisions: unknown pool, unknown economics, unverifiable floor, and approval outcomes all escalated without emitting a profile.
A worker using alternate Claude authentication cannot borrow default-subscription quota ✅ pass live Authentication boundary transcript: helper, gateway, base-URL, custom-header, and alternate-store cases selected the next measured candidate; the helper was not executed.
Claude spawn refuses ambient API billing without explicit opt-in before publishing a worker ✅ pass live Authentication boundary transcript: spawn refused ANTHROPIC_API_KEY and created no task metadata.
An actual OMP quota error reaches task state and automatic adopted-scope recovery preserves instructions and work ✅ pass live Actual OMP quota-error terminal; Routing and recovery transcript. OMP emitted an idle quota-exhausted event, control refused unknown adopted capacity, and instructions and uncommitted work remained un…
An operator can relaunch onto an explicit profile while preserving work, and an invalid rule is rejected before instruction mutation ✅ pass live Routing and recovery transcript; Recovered worker terminal. Invalid rule selection was refused, explicit relaunch published the replacement profile, preserved uncommitted.txt, delivered the progress n…
An exhausted Codex OAuth account rotates to the usable sibling without pausing work ⏸️ untested no Tried isolated native usage and real OMP against disposable local providers. Neither supplies two valid first-party Codex OAuth identities or controlled account capacity. Completing this requires auth…
Whole Codex-pool exhaustion launches the declared DeepSeek or authenticated TeamClaude stand-in ⏸️ untested no The local-provider and native quota-snapshot routes demonstrated selection and launch mechanics, but cannot prove Codex whole-pool exhaustion or execution by DeepSeek/TeamClaude. A workspace-local Tea…
Stale, missing, saved-reset, shared-meter, and wrong-tier native evidence cannot falsely exhaust a live Codex pool ⏸️ untested no Tried native usage in an isolated authentication home. Disposable HTTP providers cannot generate authentic Codex OAuth usage reports with these account and tier states, and fabricating credentials or…
  • Initial .no-mistakes/round2-bash/bash-5.3/bash bin/fm-test-run.sh --per-script-timeout-secs 900 invocation selected tests/fm-dispatch-capacity.test.sh, tests/fm-dispatch-resolve.test.sh, tests/fm-quota-choose.test.sh, tests/fm-omp-harness.test.sh, tests/fm-session-end-relaunch.test.sh, tests/fm-spawn-dispatch-profile.test.sh, tests/fm-spawn-claude-api-key-guard.test.sh, and tests/fm-control-relaunch.test.sh. The first five completed successfully; the invocation was subsequently cancelled to correct child-shell PATH.

  • Initial runner/timeout validation used bin/fm-test-run.sh --per-script-timeout-secs 600 tests/fm-test-run.test.sh tests/fm-timeout-lib.test.sh; it exposed an insufficient deadline and child commands resolving macOS Bash 3.2.

  • Re-ran with repository-local Bash 5.3 first on PATH: bash bin/fm-test-run.sh --per-script-timeout-secs 900 tests/fm-spawn-dispatch-profile.test.sh tests/fm-spawn-claude-api-key-guard.test.sh tests/fm-control-relaunch.test.sh tests/fm-test-run.test.sh tests/fm-timeout-lib.test.sh. Runner, timeout, and API-key guard completed successfully; the other two reached the imposed deadline.

  • Completed the remaining suites with the corrected PATH and larger bounded deadline: bash bin/fm-test-run.sh --per-script-timeout-secs 3000 tests/fm-spawn-dispatch-profile.test.sh tests/fm-control-relaunch.test.sh; both exited 0.

  • python3 .no-mistakes/live-validation.py: ran installed OMP 18.8.1 with the tracked worker overlay, a configured native fallback chain, and disposable local HTTP providers; observed completion through the secondary provider.

  • python3 .no-mistakes/live-boundaries.py: exercised the real quota chooser and spawn CLI with isolated HOME, disposable quota input, alternate Claude authentication selectors, a nonexecuted API-key helper, and an ambient API-billing refusal.

  • python3 .no-mistakes/live-resolver.py: drove the real resolver through real curl against a disposable local TLS endpoint and real quota-axi in its documented snapshot mode; exercised unknown pooled capacity, unverifiable floors, approval, and privacy refusal.

  • python3 .no-mistakes/live-lifecycle.py: launched real OMP in a private 120×40 tmux session; exercised fresh declared fallback, a genuine terminal quota event, automatic recovery containment, invalid-rule rejection, and explicit replacement preserving uncommitted work. Socket-path and endpoint-fixture setup errors were corrected before the successful run.

  • omp usage --provider openai-codex --json under isolated authentication returned no accounts; command -v teamclaude found no executable on PATH.

  • Stopped the private tmux server and local endpoints, removed disposable labs and drivers, archived reproduction drivers in evidence, and confirmed no matching disposable artifacts remained in the worktree.

  • ⚠️ live validation verdict: inconclusive (8 of 12 scenarios were driven live against the product); untested: An exhausted Codex OAuth account rotates to its usable sibling without pausing work, Whole Codex-pool exhaustion launches the declared DeepSeek or authenticated TeamClaude stand-in, Stale, missing, saved-reset, shared-meter, and wrong-tier native reports cannot falsely exhaust a live Codex pool, The watcher drives quota capacity inspection and queues the intended recovery escalation

  • Live validation: ⚠️ inconclusive - 8 of 12 scenarios driven live against the product

Scenario Result Live Evidence
Fresh spawn launches the rule's permitted stand-in when the primary route is exhausted ✅ pass live current-live-lifecycle.json and current-fresh-worker.html: real spawn selected the declared local OMP stand-in, published rule_1 and its concrete profile, and the worker returned LIVE_WORKER_COMPLETED…
OMP continues through a destination-owned fallback chain after provider exhaustion ✅ pass live current-omp-native-fallback-final.json: real OMP requested the blocked local model, switched to the available model, returned LIVE_STANDIN_COMPLETED, and exited zero.
Typed intake keeps unestablished native capacity unranked and respects floors and approval ✅ pass live current-product-transcript.txt, Typed routing: the real resolver retained unknown pool capacity, spendPriority and runway; disclosed an unverifiable floor; and required explicit approval.
Alternate Claude authentication cannot borrow native default-account headroom ✅ pass live current-live-authentication.json: neutral authentication selected Claude; helper, gateway, alternate-store, base-URL, and custom-header selectors selected the next eligible candidate. The helper was n…
An accidental Anthropic API key refuses launch before task publication ✅ pass live current-live-authentication.json: real fm-spawn refused the API-key launch without --allow-api-key, and no task metadata was created.
A caller-only model catalog cannot authorize a fallback in another destination ✅ pass live current-destination-catalog.json: the same primary could launch the stand-in in its configured project, but a catalog-empty destination refused with no supported permitted fallback and published no ta…
Unsafe recovery refuses before changing instructions or stopping the worker ✅ pass live current-live-lifecycle.json: automatic quota control refused unknown adopted authentication; an invalid dispatch rule also refused. Brief bytes and uncommitted local work remained unchanged.
Manual replacement continues on an explicit profile while preserving local work and context ✅ pass live current-live-lifecycle.json and current-recovered-worker.html: real control relaunched in the same endpoint and worktree, retained PRESERVE_LOCAL_WORK, published the replacement profile, and completed…
An exhausted Codex OAuth account rotates to its usable sibling without pausing work ⏸️ untested no Tried isolated native usage and real OMP with disposable local providers. Those providers can exercise model fallback but cannot supply native OAuth usage-aware rotation. This requires two authorized…
Whole Codex-pool exhaustion launches the declared DeepSeek or authenticated TeamClaude stand-in ⏸️ untested no The isolated catalog contained only Apple on-device, native Codex accounts were absent, and TeamClaude was not on PATH; isolated launcher readiness refused. Local providers proved the routing mechanis…
Stale, missing, saved-reset, shared-meter, and wrong-tier native reports cannot falsely exhaust a live Codex pool ⏸️ untested no Deterministic report fixtures exercised the classifier but are not live OAuth evidence. A live proof requires authorized accounts and controlled native report conditions. Fabricated credentials, force…
The watcher drives quota capacity inspection and queues the intended recovery escalation ⏸️ untested no Tried the real watcher under a marked lab home and private socket. Its control invocation injects FM_STATE_OVERRIDE even for the stock home state directory, defeating the lab allowance. Direct automat…
  • bin/fm-test-run.sh --per-script-timeout-secs 1500 --json &lt;evidence&gt;/targeted-timing.json tests/fm-dispatch-capacity.test.sh tests/fm-dispatch-resolve.test.sh tests/fm-quota-choose.test.sh tests/fm-omp-harness.test.sh tests/fm-session-end-relaunch.test.sh tests/fm-control-relaunch.test.sh: five suites passed; control reached the aggregate per-suite bound.
  • Ran fm-spawn-dispatch-profile.test.sh and fm-spawn-claude-api-key-guard.test.sh through the repository runner with worktree-local Bash 5.3; the initial invocation reached its bounds before completing both suites.
  • Diagnosed the control suite's next case using a disposable driver for test_herdr_relaunch_resumes_only_the_registered_pi_session; it passed with both closed and open stdin.
  • Re-ran the complete control suite through bin/fm-test-run.sh --per-script-timeout-secs 2400 with worktree-local Bash 5.3; it passed.
  • Re-ran the complete spawn-profile and Claude API-key-guard suites through the repository runner with worktree-local Bash 5.3, explicit stdin EOF, and a 2400-second per-suite bound; both passed.
  • Drove real fm-dispatch-resolve.sh through a disposable local TLS endpoint and real quota-axi snapshot mode, exercising unknown pooled capacity, unverifiable floors, approval requirements, and privacy refusal.
  • Drove real fm-quota-choose.sh and fm-spawn.sh in disposable homes with Claude helpers, gateway settings, alternate stores, base URLs, custom headers, and an accidental API-key launch.
  • Launched real OMP primaries and workers on private 120×40 tmux sessions; exercised declared fresh-spawn fallback, destination-catalog rejection, quota-event publication, automatic recovery refusal, invalid-rule refusal, and manual work-preserving replacement.
  • Ran real OMP print mode against disposable local HTTP providers with a destination-owned fallback chain and the tracked worker overlay; supplied stdin EOF after diagnosing initial startup waits.
  • Ran real fm-watch.sh against an actual worker quota event and retained its gate-refused recovery escalation.
  • Ran isolated omp usage --provider openai-codex --json, omp models --json, and bin/fm-teamclaude-launch.sh --check; observed no native Codex accounts, no authenticated named stand-in catalog, and unavailable TeamClaude.
  • Stopped all private tmux servers and local HTTP/TLS services, removed disposable labs and diagnostic scripts, and confirmed their removal.
⚠️ **Document** - 0 issues

✅ No issues found.

  • ⚠️ bin/fm-doc-audience-check.sh:95 - The documentation checker resolves links in skill reference files from the skill root rather than the containing document. It therefore rejects the user-approved ../../../../../ ownership links in the OMP reference as escaping the repository, although both resolve correctly from that document. Preserved the approved links; reconciling the checker's resolution convention requires executable changes outside this documentation-only phase.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed (3) ✅
  • ⚠️ linter found issues (exit code 1)

🔧 Fix applied.
✅ Re-checked - no issues remain.

  • ⚠️ linter found issues (exit code 1)

🔧 Fix applied.
✅ Re-checked - no issues remain.

  • ⚠️ linter found issues (exit code 1)

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.

✅ No issues found.

✅ No issues found.

@MrGTV-love
MrGTV-love force-pushed the fm/fm-omp-codex-account-failover branch from 6942f70 to 446e3a8 Compare October 8, 2026 10:52
@MrGTV-love MrGTV-love changed the title feat: add pooled Codex capacity checks and model-matrix failover fix: use pooled Codex capacity and declared model fallbacks Oct 8, 2026
@MrGTV-love
MrGTV-love force-pushed the fm/fm-omp-codex-account-failover branch from 446e3a8 to 9790a52 Compare October 8, 2026 16:19
@MrGTV-love MrGTV-love changed the title fix: use pooled Codex capacity and declared model fallbacks feat: add pooled Codex capacity checks and declared model failover Oct 8, 2026
@MrGTV-love

Copy link
Copy Markdown
Owner Author

Superseded by #91 (fm/fm-omp-codex-account-failover-v2-r3), which merged as b2e06c4 and carries this work.

@MrGTV-love MrGTV-love closed this Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant