Repository navigation
feat: add pooled Codex capacity checks and declared model failover - #39
Closed
MrGTV-love wants to merge 48 commits into
Closed
MrGTV-love wants to merge 48 commits into
MrGTV-love wants to merge 48 commits into
Conversation
MrGTV-love
force-pushed
the
fm/fm-omp-codex-account-failover
branch
from
October 8, 2026 10:52
6942f70 to
446e3a8
Compare
…llback invariants
…ck profile handling
…eligibility reporting
…and worker declarations
…d fallback validation
…e destination environment
…rktree regressions
…le control fallback reporting
… control relaunch
…allback selection
…rage, correct relaunch guidance
… lifecycle assertions
…nd routing policy
…-settings fixtures
…ght documentation
MrGTV-love
force-pushed
the
fm/fm-omp-codex-account-failover
branch
from
October 8, 2026 16:19
446e3a8 to
9790a52
Compare
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
if codex is exhausted, why not switch to a different model like Deepseek flash v4, or sonnet 5.5 via tc run?
vs hold up work??!! we have a model matrix, so why not switch models accordingly to the fallback??
also confused. we have two codex credentials. the other one is completely unutilized
why did you hold up the work vs switching models?
okay. paid calls for development work do not. update so that deepseek flash replaces glm 5.3
deepseek/deepseek-v4-flash is what I think it is
Context: on 2026-10-05 the fleet sat about 13 hours paused because quota-axi reported the Codex weekly window exhausted. quota-axi reads only one Codex account (greentree promax). omp pools two openai-codex accounts (greentree promax and gmail prolite;
omp usagelists both, and the gmail account also shows a banked saved reset). Workers and every no-mistakes review agent run on omp openai-codex models, so all work and all PR pipelines stopped together. config/crew-dispatch.json (the model matrix) already names stand-ins per rule: z-ai GLM 5.3 for Sonnet-class work, GLM 5.3 Flash for Luna-class work, and gpt-6.1-sol and Opus 5.5 (Claude only through the TeamClaude proxy) as equals for the strongest class. None were applied. Earlier, on 2026-09-30, a worker stalled about 9 hours on usage_limit_reached on one account while the other account had quota; a relaunch recovered it.no attempt limits
What Changed
Risk Assessment
Testing
Eight targeted suites completed successfully after bounded redrives, and eight live routing, authentication, fallback, and recovery scenarios passed with CLI transcripts and captured-terminal HTML retained. Initial suite timeouts and print-mode stdin setup were resolved; OAuth-dependent proofs and watcher capacity inspection remain untested, so the overall verdict is inconclusive. All disposable setups were removed.
Evidence: Current product command transcript
Source: Current product command transcript
Evidence: Captured fresh worker terminal
Source: Captured fresh worker terminal
Evidence: Captured exhausted worker terminal
Source: Captured exhausted worker terminal
Evidence: Isolated provider prerequisites
Source: Isolated provider prerequisites
Pipeline
Updates from git push no-mistakes
... (79 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)
🔧 **Review** - 8 issues found → auto-fixed (34) → no changes applied (2) ✅
✅ No issues found.
bin/fm-test-run.sh --json <evidence>/targeted-suite-timing.json tests/fm-dispatch-capacity.test.sh tests/fm-dispatch-resolve.test.sh tests/fm-quota-choose.test.sh tests/fm-omp-harness.test.sh tests/fm-session-end-relaunch.test.sh tests/fm-control-relaunch.test.sh tests/fm-spawn-claude-api-key-guard.test.sh tests/fm-spawn-dispatch-profile.test.shbin/fm-test-run.sh --json <evidence>/targeted-setup-redrive-timing.json tests/fm-omp-harness.test.sh tests/fm-control-relaunch.test.sh tests/fm-spawn-dispatch-profile.test.shwith normal toolchain TMPDIR; corrected the initial secondmate fixtures being placed inside the protected repositoryReal isolatedomp usage --provider openai-codex --json --no-extensions,omp models --json --no-extensions, andbin/fm-dispatch-capacity.sh --harness omp --model <model> --jsonProduction Bashfm_omp_codex_capacitywith freshly constructed usage protocol inputs covering usable siblings, exhausted pools, saved resets, missing/stale siblings, negative shared flags, uppercase Spark, and paid-model entitlementReal private tmux sessions: compared production environment previews with actual imports, removal markers, and established-session precedenceRealfm-spawn.shpreflight with caller credential sentinels, explicit API opt-in, launch allowlist filtering, invalid fallback declarations, and conflicting rule permissionsProduction destination-scoped OMP catalog queries from a provider-disabled caller project, including a relative-PATH installation pointing to the real OMP executablebin/fm-quota-choose.sh --snapshot <disposable-positive-legacy-snapshot> --candidate omp:openai-codex/gpt-6.1-sol; empty native pool returnednoneReal OMP v18.7.0 RPC sessions consumed the tracked worker overlay and a disposable native fallback-chain overlay; a loopback endpoint returned HTTP 429 insufficient_quota, after which native OMP switched providers and completed withLOCAL_NATIVE_FALLBACK_OKReal native OMP idle worker, directfm-control.sh, production watcher, and durable wake queue exercised adopted-authentication refusal and generation-bound recovery containment using seeded task recordscommand -v teamclaudeand isolatedbin/fm-teamclaude-launch.sh --check; the supported launcher reported TeamClaude unavailableStopped owned endpoints and servers, removed disposable worktree fixtures and inactive sockets, and confirmed no owned disposable processes remained🔧 No changes applied.
1 warning still open:
Initial.no-mistakes/round2-bash/bash-5.3/bash bin/fm-test-run.sh --per-script-timeout-secs 900invocation selectedtests/fm-dispatch-capacity.test.sh,tests/fm-dispatch-resolve.test.sh,tests/fm-quota-choose.test.sh,tests/fm-omp-harness.test.sh,tests/fm-session-end-relaunch.test.sh,tests/fm-spawn-dispatch-profile.test.sh,tests/fm-spawn-claude-api-key-guard.test.sh, andtests/fm-control-relaunch.test.sh. The first five completed successfully; the invocation was subsequently cancelled to correct child-shell PATH.Initial runner/timeout validation usedbin/fm-test-run.sh --per-script-timeout-secs 600 tests/fm-test-run.test.sh tests/fm-timeout-lib.test.sh; it exposed an insufficient deadline and child commands resolving macOS Bash 3.2.Re-ran with repository-local Bash 5.3 first on PATH:bash bin/fm-test-run.sh --per-script-timeout-secs 900 tests/fm-spawn-dispatch-profile.test.sh tests/fm-spawn-claude-api-key-guard.test.sh tests/fm-control-relaunch.test.sh tests/fm-test-run.test.sh tests/fm-timeout-lib.test.sh. Runner, timeout, and API-key guard completed successfully; the other two reached the imposed deadline.Completed the remaining suites with the corrected PATH and larger bounded deadline:bash bin/fm-test-run.sh --per-script-timeout-secs 3000 tests/fm-spawn-dispatch-profile.test.sh tests/fm-control-relaunch.test.sh; both exited 0.python3 .no-mistakes/live-validation.py: ran installed OMP 18.8.1 with the tracked worker overlay, a configured native fallback chain, and disposable local HTTP providers; observed completion through the secondary provider.python3 .no-mistakes/live-boundaries.py: exercised the real quota chooser and spawn CLI with isolated HOME, disposable quota input, alternate Claude authentication selectors, a nonexecuted API-key helper, and an ambient API-billing refusal.python3 .no-mistakes/live-resolver.py: drove the real resolver through real curl against a disposable local TLS endpoint and real quota-axi in its documented snapshot mode; exercised unknown pooled capacity, unverifiable floors, approval, and privacy refusal.python3 .no-mistakes/live-lifecycle.py: launched real OMP in a private 120×40 tmux session; exercised fresh declared fallback, a genuine terminal quota event, automatic recovery containment, invalid-rule rejection, and explicit replacement preserving uncommitted work. Socket-path and endpoint-fixture setup errors were corrected before the successful run.omp usage --provider openai-codex --jsonunder isolated authentication returned no accounts;command -v teamclaudefound no executable on PATH.Stopped the private tmux server and local endpoints, removed disposable labs and drivers, archived reproduction drivers in evidence, and confirmed no matching disposable artifacts remained in the worktree.Live validation:⚠️ inconclusive - 8 of 12 scenarios driven live against the product
bin/fm-test-run.sh --per-script-timeout-secs 1500 --json <evidence>/targeted-timing.json tests/fm-dispatch-capacity.test.sh tests/fm-dispatch-resolve.test.sh tests/fm-quota-choose.test.sh tests/fm-omp-harness.test.sh tests/fm-session-end-relaunch.test.sh tests/fm-control-relaunch.test.sh: five suites passed; control reached the aggregate per-suite bound.Ranfm-spawn-dispatch-profile.test.shandfm-spawn-claude-api-key-guard.test.shthrough the repository runner with worktree-local Bash 5.3; the initial invocation reached its bounds before completing both suites.Diagnosed the control suite's next case using a disposable driver fortest_herdr_relaunch_resumes_only_the_registered_pi_session; it passed with both closed and open stdin.Re-ran the complete control suite throughbin/fm-test-run.sh --per-script-timeout-secs 2400with worktree-local Bash 5.3; it passed.Re-ran the complete spawn-profile and Claude API-key-guard suites through the repository runner with worktree-local Bash 5.3, explicit stdin EOF, and a 2400-second per-suite bound; both passed.Drove realfm-dispatch-resolve.shthrough a disposable local TLS endpoint and real quota-axi snapshot mode, exercising unknown pooled capacity, unverifiable floors, approval requirements, and privacy refusal.Drove realfm-quota-choose.shandfm-spawn.shin disposable homes with Claude helpers, gateway settings, alternate stores, base URLs, custom headers, and an accidental API-key launch.Launched real OMP primaries and workers on private 120×40 tmux sessions; exercised declared fresh-spawn fallback, destination-catalog rejection, quota-event publication, automatic recovery refusal, invalid-rule refusal, and manual work-preserving replacement.Ran real OMP print mode against disposable local HTTP providers with a destination-owned fallback chain and the tracked worker overlay; supplied stdin EOF after diagnosing initial startup waits.Ran realfm-watch.shagainst an actual worker quota event and retained its gate-refused recovery escalation.Ran isolatedomp usage --provider openai-codex --json,omp models --json, andbin/fm-teamclaude-launch.sh --check; observed no native Codex accounts, no authenticated named stand-in catalog, and unavailable TeamClaude.Stopped all private tmux servers and local HTTP/TLS services, removed disposable labs and diagnostic scripts, and confirmed their removal.✅ No issues found.
bin/fm-doc-audience-check.sh:95- The documentation checker resolves links in skill reference files from the skill root rather than the containing document. It therefore rejects the user-approved ../../../../../ ownership links in the OMP reference as escaping the repository, although both resolve correctly from that document. Preserved the approved links; reconciling the checker's resolution convention requires executable changes outside this documentation-only phase.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ No issues found.
🔧 **Lint** - 1 issue found → auto-fixed (3) ✅
🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.