Skip to content

feat(bin): federated fleet add-on with multi-account spawn and quota-aware routing - #1217

Closed
adibirzu wants to merge 26 commits into
kunchenguid:mainfrom
adibirzu:fed-lib-split
Closed

adibirzu wants to merge 26 commits into
kunchenguid:mainfrom
adibirzu:fed-lib-split

Conversation

@adibirzu

Copy link
Copy Markdown

Intent

Two follow-ups a closing review flagged on the fleet quota work (PR #1187). First: tests/federation/test_fleet_ops.sh consulted the live quota-axi in its register/route lifecycle cases, so it went red whenever the operator's real subscription window was exhausted (observed 4/16 failing, all green after the window reset) — now hermetic via a stubbed quota-axi, with lifecycle semantics still genuinely exercised and one bounded case deliberately retained on the real quota_now path so live coverage is not silently deleted. Second: bin/fm-fleet-lib.sh had grown to 816 lines, past the repo's 800-line convention — split at a verified one-directional seam (surface->KB crossings: zero) into a new leaf bin/fm-fleet-quota-lib.sh, leaving 393 and 460 lines. Behaviour-preserving: proven by the committed golden fixtures being compared, never regenerated.

What Changed

  • Federated multi-operator fleet — new bin/fm-fleet.sh CLI over bin/fm-fleet-lib.sh, coordinating several OS operators through a shared, cross-uid-safe, git-backed KB: init/register/heartbeat/leave, queue/claim/handoff/reap under a lock, scope routing with heartbeat-freshness staleness (with a non-GNU date fallback), and status/view, plus fm-fleet-wait.sh and fm-fleet-join.sh. Documented in docs/fleet-addon.md, docs/fleet-quickstart.md, docs/fleet-token-economy.md, and the federation/multi-account agent skills.
  • Per-spawn --account axis — account registry, resolution and validation across three isolation methods (bin/fm-accounts-lib.sh, fm-account-env.sh, fm-account-exec.sh, fm-spawn-acct.sh), riding fm-spawn's existing raw-launch escape hatch so no core FirstMate script is modified and no secret lands on argv. Adds a user-scoped on-demand prereq installer (fm-accounts-prereq.sh) and one reviewable idempotent root step (scripts/fleet-root-prereq.sh).
  • Quota/pace surface layer — authed Copilot and Cursor usage readers (bin/quota-copilot-usage.sh, bin/quota-cursor-usage.sh, bin/quota-sources/*) feeding per-surface headroom, budget gating, and model-family → surface failover picking; that layer was split out of fm-fleet-lib.sh into a new leaf bin/fm-fleet-quota-lib.sh at a one-directional seam, bringing both files back under the repo's 800-line convention. Covered by tests/federation/*.sh with committed golden fixtures; test_fleet_ops.sh stubs quota-axi so lifecycle cases no longer depend on the operator's real subscription window, with one bounded case deliberately left on the live path.

Risk Assessment

✅ Low: All three previously-flagged warnings are verified durably fixed with their failure sequences reconstructed and no remaining reachable path, the intent-scoped hermetic test and verbatim library split are untouched and re-verified (442 + 460 lines, goldens unchanged, budget golden output and exit code preserved), and the sole remaining finding is a single test fixture with no product impact.

Testing

Ran the four federation suites relevant to the two changed areas (fleet_ops, quota_surfaces, fleet, fleet_guards — all green), then went beyond pass/fail to demonstrate each claim end-to-end. For hermeticity I recreated the operator's exhausted-subscription condition with a 0%-remaining quota-axi stub and showed the pre-hermetic test failing exactly the reported 4 of 16 assertions while the fixed suite passes all 24 under that same stub, with quota-axi absent, and with a quota-axi returning garbage; a sentinel value confirmed the lifecycle cases are stubbed while the retained case genuinely rides the real path, and a 600s-hang stub confirmed its 10s bound. For the split I reconstructed the pre-split 816-line library and diffed the actual operator CLI output surface by surface against HEAD's 442+460 layout under one shared stub environment — byte-identical everywhere including budget's exit code — and separately confirmed the committed goldens still match their pre-split commit and were not regenerated. The change is a bash CLI/library refactor with no UI, HTML, or rendered surface, so the reviewer-visible evidence is CLI transcripts of the actual fm-fleet.sh operator commands rather than screenshots; no visual surface exists to capture. Overall result: both intents satisfied, no defects found, worktree left clean.

Evidence: BEFORE: pre-hermetic test goes red under an exhausted quota-axi (reproduces the reported 4/16)

### BEFORE (commit 03ba378, pre-hermetic test) under an EXHAUSTED quota-axi (0% remaining) $ PATH=<exhausted-quota-axi>:$PATH bash tests/federation/test_fleet_ops.sh ... FAIL: route fresh register (got '') FAIL: route stale->overflow (got '') FAIL: heartbeat refresh (got '') FAIL: leave offline (got '') ----- 4 FAILURE(S) exit=1

### BEFORE (commit 03ba378, pre-hermetic test) under an EXHAUSTED quota-axi (0% remaining)
$ PATH=<exhausted-quota-axi>:$PATH bash tests/federation/test_fleet_ops.sh
PASS: register writes an operator row
FAIL: route fresh register (got '')
PASS: register is idempotent (one row)
FAIL: route stale->overflow (got '')
FAIL: heartbeat refresh (got '')
FAIL: leave offline (got '')
PASS: route: owner below quota floor -> overflow
PASS: route: owner above quota floor -> owner
PASS: budget_ok: above floor passes
PASS: budget_ok: below floor fails
PASS: register refuses a foreign home
PASS: wait --once: fresh claim wakes (exit 0 + id)
PASS: wait --once: no claim -> exit 1 (LLM stays idle)
PASS: wait --once: in-flight item is not a fresh wake
PASS: join: writes config/fleet-dir + registers self
PASS: join: idempotent (one row on rejoin)
-----
4 FAILURE(S)
exit=1
Evidence: AFTER: hermetic suite is green under the identical exhausted quota-axi

### AFTER (HEAD 007cc36, hermetic test) under the SAME EXHAUSTED quota-axi (0% remaining) $ PATH=<exhausted-quota-axi>:$PATH bash tests/federation/test_fleet_ops.sh PASS: register writes the stubbed quota value into the quota column PASS: route finds a freshly-registered operator PASS: route: stale-heartbeat operator treated offline -> overflow PASS: heartbeat refreshes seen -> operator online again PASS: leave -> offline -> overflow PASS: register->quota->route: low-headroom re-register falls through to overflow PASS: register->quota->route: high-headroom re-register restores the owner PASS: quota_now (LIVE, real quota-axi): returns a percentage or '-', never garbage ----- ALL PASS exit=0

### AFTER (HEAD 007cc36, hermetic test) under the SAME EXHAUSTED quota-axi (0% remaining)
$ PATH=<exhausted-quota-axi>:$PATH bash tests/federation/test_fleet_ops.sh
PASS: register writes an operator row
PASS: register writes the stubbed quota value into the quota column
PASS: route finds a freshly-registered operator
PASS: register is idempotent (one row)
PASS: stale-forcing sed hit the seen column (robustness check)
PASS: route: stale-heartbeat operator treated offline -> overflow
PASS: heartbeat refreshes seen: advances past the stale value and parses as a timestamp
PASS: heartbeat refreshes seen -> operator online again
PASS: leave -> offline -> overflow
PASS: route: owner below quota floor -> overflow
PASS: route: owner above quota floor -> owner
PASS: re-register with low stub writes quota=2
PASS: register->quota->route: low-headroom re-register falls through to overflow
PASS: re-register with high stub writes quota=80
PASS: register->quota->route: high-headroom re-register restores the owner
PASS: budget_ok: above floor passes
PASS: budget_ok: below floor fails
PASS: register refuses a foreign home
PASS: wait --once: fresh claim wakes (exit 0 + id)
PASS: wait --once: no claim -> exit 1 (LLM stays idle)
PASS: wait --once: in-flight item is not a fresh wake
PASS: join: writes config/fleet-dir + registers self
PASS: join: idempotent (one row on rejoin)
PASS: quota_now (LIVE, real quota-axi): returns a percentage or '-', never garbage
-----
ALL PASS
exit=0
Evidence: Stub isolation: real quota-axi returns sentinel 7.5 while lifecycle cases read the stub

-- what the REAL quota-axi path reports (this is what case 10 rides) -- $ PATH=<sentinel>:$PATH bash -c '. bin/fm-fleet-lib.sh; fm_fleet_quota_now' 7.5 -- suite run with that same sentinel on the outer PATH -- PASS: register writes the stubbed quota value into the quota column <- 42 from SUITE_STUB, not 7.5 PASS: re-register with low stub writes quota=2 PASS: re-register with high stub writes quota=80 PASS: quota_now (LIVE, real quota-axi): returns a percentage or '-', never garbage ALL PASS

### Stub-isolation proof: outer (real) quota-axi returns a SENTINEL 7.5, suite stub returns 42

-- what the REAL quota-axi path reports (this is what case 10 rides) --
$ PATH=<sentinel>:$PATH bash -c '. bin/fm-fleet-lib.sh; fm_fleet_quota_now'
7.5

-- suite run with that same sentinel on the outer PATH --
PASS: register writes an operator row
PASS: register writes the stubbed quota value into the quota column
PASS: route finds a freshly-registered operator
PASS: register is idempotent (one row)
PASS: stale-forcing sed hit the seen column (robustness check)
PASS: route: stale-heartbeat operator treated offline -> overflow
PASS: heartbeat refreshes seen: advances past the stale value and parses as a timestamp
PASS: heartbeat refreshes seen -> operator online again
PASS: leave -> offline -> overflow
PASS: route: owner below quota floor -> overflow
PASS: route: owner above quota floor -> owner
PASS: re-register with low stub writes quota=2
PASS: register->quota->route: low-headroom re-register falls through to overflow
PASS: re-register with high stub writes quota=80
PASS: register->quota->route: high-headroom re-register restores the owner
PASS: budget_ok: above floor passes
PASS: budget_ok: below floor fails
PASS: register refuses a foreign home
PASS: wait --once: fresh claim wakes (exit 0 + id)
PASS: wait --once: no claim -> exit 1 (LLM stays idle)
PASS: wait --once: in-flight item is not a fresh wake
PASS: join: writes config/fleet-dir + registers self
PASS: join: idempotent (one row on rejoin)
PASS: quota_now (LIVE, real quota-axi): returns a percentage or '-', never garbage
-----
ALL PASS
exit=0
Evidence: Operator CLI: pre-split vs post-split byte-identical on every quota surface

=== A/B DIFF: pre-split (1850451, one 816-line lib) vs post-split (HEAD, 442+460) === IDENTICAL fm-fleet.sh quota IDENTICAL fm-fleet.sh models IDENTICAL fm-fleet.sh pick grok IDENTICAL fm-fleet.sh pick claude IDENTICAL fm-fleet.sh pick kimi IDENTICAL fm-fleet.sh pick bogus IDENTICAL fm-fleet.sh budget (message + exit code) RESULT: every operator-facing quota surface is byte-identical across the split. === And post-split output vs the COMMITTED pre-split goldens (never regenerated) === IDENTICAL models == golden (committed 59ccbc9, pre-split) IDENTICAL pick grok / pick claude / pick kimi / pick bogus / budget == golden

=== A/B DIFF: pre-split (1850451, one 816-line lib) vs post-split (HEAD, 442+460) ===
    identical hermetic stub env; a byte diff on ANY surface would print below.

  IDENTICAL  fm-fleet.sh quota
  IDENTICAL  fm-fleet.sh models
  IDENTICAL  fm-fleet.sh pick grok
  IDENTICAL  fm-fleet.sh pick claude
  IDENTICAL  fm-fleet.sh pick kimi
  IDENTICAL  fm-fleet.sh pick bogus
  IDENTICAL  fm-fleet.sh budget (message + exit code)

RESULT: every operator-facing quota surface is byte-identical across the split.

=== And post-split output vs the COMMITTED pre-split goldens (never regenerated) ===
  IDENTICAL  models  == golden (committed 59ccbc9, pre-split)
  IDENTICAL  pick grok   == golden
  IDENTICAL  pick claude == golden
  IDENTICAL  pick kimi   == golden
  IDENTICAL  pick bogus  == golden
  IDENTICAL  budget      == golden
Evidence: Rendered operator surfaces after the split (the actual end-user CLI output)

$ fm-fleet.sh quota [POST-SPLIT / HEAD] SURFACE HEADROOM PACE RESERVE STATUS SOURCE NOTE claude 1% — — fresh oauth observable codex 90% — — fresh cli-rpc observable copilot 88% — — logged_in custom test cursor 55% — — logged_in custom test $ fm-fleet.sh models [POST-SPLIT / HEAD] MODEL SURFACES (pool: status headroom) grok grok: unconfigured — | cursor: logged_in 55% kimi kimi: unconfigured — claude claude: fresh 1% | copilot: logged_in 88% | cursor: logged_in 55% gpt codex: fresh 90% | copilot: logged_in 88% | cursor: logged_in 55% $ fm-fleet.sh pick grok -> cursor $ fm-fleet.sh pick claude -> copilot $ fm-fleet.sh budget -> below floor (< 5%) exit=1

======================================================================
 OPERATOR-FACING CLI, PRE-SPLIT (816-line single lib, commit 1850451)
   vs POST-SPLIT (442 + 460 leaf, HEAD 007cc36) — identical stub env
======================================================================

$ fm-fleet.sh quota   [POST-SPLIT / HEAD]
SURFACE  HEADROOM  PACE  RESERVE  STATUS     SOURCE   NOTE
claude   1%        —     —        fresh      oauth    observable
codex    90%       —     —        fresh      cli-rpc  observable
copilot  88%       —     —        logged_in  custom   test
cursor   55%       —     —        logged_in  custom   test

$ fm-fleet.sh models   [POST-SPLIT / HEAD]
MODEL   SURFACES (pool: status headroom)
grok    grok: unconfigured —  |  cursor: logged_in 55%
kimi    kimi: unconfigured —
claude  claude: fresh 1%  |  copilot: logged_in 88%  |  cursor: logged_in 55%
gpt     codex: fresh 90%  |  copilot: logged_in 88%  |  cursor: logged_in 55%

$ fm-fleet.sh pick grok   [POST-SPLIT / HEAD]
cursor

$ fm-fleet.sh pick claude   [POST-SPLIT / HEAD]
copilot

$ fm-fleet.sh pick kimi   [POST-SPLIT / HEAD]
kimi

$ fm-fleet.sh pick bogus   [POST-SPLIT / HEAD]
unknown model family: bogus

$ fm-fleet.sh budget   [POST-SPLIT / HEAD]
below floor (< 5%)
exit=1
Evidence: Leaf library is independently sourceable; consumer sourcing contract unchanged

=== LEAF check: fm-fleet-quota-lib.sh sourced ALONE (fm-fleet-lib.sh never sourced) === quota fns exported: 12 KB fns leaked in? : 0 (want 0) quota_now -> 1 budget_ok rc=1 pick claude -> copilot === Consumer contract unchanged: sourcing ONLY fm-fleet-lib.sh still yields the quota fns === available via fm-fleet-lib.sh: fm_fleet_quota_now / fm_fleet_budget_ok / fm_fleet_pick_surface / fm_fleet_pace_rows / fm_fleet_models_report

=== LEAF check: fm-fleet-quota-lib.sh sourced ALONE (fm-fleet-lib.sh never sourced) ===
$ bash -c '. bin/fm-fleet-quota-lib.sh; declare -F | grep fm_fleet_ | wc -l; fm_fleet_budget_ok; echo rc=$?'
quota fns exported: 12
KB fns leaked in?  : 0  (want 0)
quota_now -> 1
budget_ok rc=1
pick claude -> copilot

rc=0

=== Consumer contract unchanged: sourcing ONLY fm-fleet-lib.sh still yields the quota fns ===
  available via fm-fleet-lib.sh: fm_fleet_quota_now
  available via fm-fleet-lib.sh: fm_fleet_budget_ok
  available via fm-fleet-lib.sh: fm_fleet_pick_surface
  available via fm-fleet-lib.sh: fm_fleet_pace_rows
  available via fm-fleet-lib.sh: fm_fleet_models_report
Evidence: Goldens byte-identical to their pre-split commit and not regenerated by the run

=== goldens untouched by the test run? (git) === (empty = unchanged) === golden bytes still at the pre-split commit 59ccbc9? === IDENTICAL to pre-split commit: tests/federation/golden/v2-budget.golden IDENTICAL to pre-split commit: tests/federation/golden/v2-models.golden IDENTICAL to pre-split commit: tests/federation/golden/v2-pick-bogus.golden IDENTICAL to pre-split commit: tests/federation/golden/v2-pick-claude.golden IDENTICAL to pre-split commit: tests/federation/golden/v2-pick-grok.golden IDENTICAL to pre-split commit: tests/federation/golden/v2-pick-kimi.golden IDENTICAL to pre-split commit: tests/federation/golden/v2-quota.golden

IDENTICAL to pre-split commit: tests/federation/golden/v2-budget.golden
IDENTICAL to pre-split commit: tests/federation/golden/v2-models.golden
IDENTICAL to pre-split commit: tests/federation/golden/v2-pick-bogus.golden
IDENTICAL to pre-split commit: tests/federation/golden/v2-pick-claude.golden
IDENTICAL to pre-split commit: tests/federation/golden/v2-pick-grok.golden
IDENTICAL to pre-split commit: tests/federation/golden/v2-pick-kimi.golden
IDENTICAL to pre-split commit: tests/federation/golden/v2-quota.golden
Evidence: Hermeticity under quota-axi absent + retained live case boundedness

-- (a) real quota-axi EXITS NONZERO with garbage on stderr -- PASS: quota_now (LIVE, real quota-axi): returns a percentage or '-', never garbage ALL PASS elapsed=0s -- (b) real quota-axi HANGS (sleep 600) — suite must stay bounded, not wedge -- FAIL: quota_now LIVE (rc=124 got '') 1 FAILURE(S) elapsed=11s <- bound honoured: 11s, not 600s (separate file 04: quota-axi absent from PATH entirely -> ALL PASS)

=== Bounded-ness of the retained LIVE case: hostile real quota-axi variants ===

-- (a) real quota-axi EXITS NONZERO with garbage on stderr --
PASS: quota_now (LIVE, real quota-axi): returns a percentage or '-', never garbage
-----
ALL PASS
elapsed=0s

-- (b) real quota-axi HANGS (sleep 600) — suite must stay bounded, not wedge --
PASS: join: idempotent (one row on rejoin)
FAIL: quota_now LIVE (rc=124 got '')
-----
1 FAILURE(S)
elapsed=11s
Evidence: Lifecycle coverage 16 -> 24 assertions, nothing deleted
=== Lifecycle coverage: assertions BEFORE vs AFTER the hermeticity change ===
  pre-hermetic (03ba378): 16 assertions
  hermetic (HEAD)       : 24 assertions   -> coverage grew, nothing deleted

  assertions present ONLY in the hermetic version:
    + heartbeat refreshes seen -> operator online again
    + heartbeat refreshes seen: advances past the stale value and parses as a timestamp
    + leave -> offline -> overflow
    + quota_now (LIVE, real quota-axi): returns a percentage or '-', never garbage
    + re-register with high stub writes quota=80
    + re-register with low stub writes quota=2
    + register writes the stubbed quota value into the quota column
    + register->quota->route: high-headroom re-register restores the owner
    + register->quota->route: low-headroom re-register falls through to overflow
    + route finds a freshly-registered operator
    + route: stale-heartbeat operator treated offline -> overflow
    + stale-forcing sed hit the seen column (robustness check)

  lifecycle assertions retained from before (register/heartbeat/leave/route/join/wait):
    = budget_ok: above floor passes
    = budget_ok: below floor fails
    = join: idempotent (one row on rejoin)
    = join: writes config/fleet-dir + registers self
    = register is idempotent (one row)
    = register refuses a foreign home
    = register writes an operator row
    = route: owner above quota floor -> owner
    = route: owner below quota floor -> overflow
    = wait --once: fresh claim wakes (exit 0 + id)
    = wait --once: in-flight item is not a fresh wake
    = wait --once: no claim -> exit 1 (LLM stays idle)

NOTE on reading the lists above: four entries listed as "+ only in hermetic"
(route fresh register, route stale->overflow, heartbeat refresh, leave offline)
DID exist before — they appear here only because in the BEFORE run they emitted
their FAIL text instead of their PASS text. Accurate accounting:
  * 4 pre-existing lifecycle cases: were live-quota-dependent (red under an
    exhausted window), now deterministic.
  * 8 genuinely NEW assertions: register-writes-quota-column, stale-sed
    robustness guard, heartbeat seen-advance, the 4-step register->quota->route
    chain (low then high), and the retained LIVE quota_now shape check.
  * 12 assertions carried over untouched.
  16 -> 24 total. No lifecycle semantics were deleted to buy hermeticity.
Evidence: Golden/surface suite full run (62 assertions)
$ bash tests/federation/test_quota_surfaces.sh   # FM_TEST_REGEN_GOLDEN deliberately UNSET
PASS: pick grok fails over to cursor when grok drained
PASS: pick claude -> claude (has headroom)
PASS: pick kimi -> kimi (only surface; cline unmonitored)
PASS: pick unknown family is flagged
PASS: quota view includes cursor (custom source)
PASS: models view: grok reachable via cursor
PASS: cline is not a monitored surface
PASS: authed override: cursor headroom reads 55% from its reader
PASS: copilot custom source supersedes quota-axi auth_required row (88%)
PASS: models view: claude reachable via copilot
PASS: models view: gpt reachable via copilot
PASS: pick claude fails over to copilot when the claude pool is drained
PASS: pace_rows: A (v3, ahead) -> pace=ahead
PASS: pace_rows: A reserve=-18
PASS: pace_rows: B (v3, behind) -> pace=behind
PASS: pace_rows: B reserve=+20 (unsigned in row, caller signs it)
PASS: pace_rows: C -> pace=mixed
PASS: pace_rows: C worstReserve=-15 (prefers effective summary over per-window min)
PASS: pace_rows: E -> pace=unknown (explicit, not absent)
PASS: pace_rows: E reserve is absent (no worstReservePercentPoints)
PASS: pace_rows: F -> pace absent (v3, no pace fields) — never 'on_pace'
PASS: pace_rows: F reserve absent
PASS: pace_rows: v2 payload -> pace absent (never fabricated)
PASS: pace_rows: v2 payload -> reserve absent
PASS: pressured: A (ahead) -> pressured
PASS: pressured: C (mixed + aheadWindowIds non-empty) -> pressured
PASS: pressured: B (behind) -> not pressured
PASS: pressured: F (absent pace) -> not pressured
PASS: quota header: SURFACE HEADROOM PACE RESERVE STATUS SOURCE NOTE
PASS: quota: A shows pace=ahead reserve=-18 (signed)
PASS: quota: B shows pace=behind reserve=+20 (explicit +)
PASS: quota: C shows pace=mixed reserve=-15
PASS: quota: E shows pace=unknown reserve=— (unknown != absent)
PASS: quota: F (v3, no pace) shows —/— (absent, never fabricated)
PASS: quota: stale H still DISPLAYS its pace (R1: visible, but STATUS marks it stale)
PASS: quota: H STATUS column marks it stale (R1 visual marker)
PASS: quota: custom source masking a paced native row gets the advisory NOTE
PASS: quota: custom-source row itself renders —/— for PACE/RESERVE (never 'unknown')
PASS: FM_FLEET_RESERVE_MIN is unset in the test shell (library default -25 applies)
PASS: budget: not pressured -> ok, names headroom+pace (ok (headroom 70% >= 5%, pace behind, reserve +20))
PASS: budget: pressured but reserve -10 >= -25 -> ok (ok (headroom 45% >= 5%, pace ahead, reserve -10))
PASS: budget: pressured, reserve -31 < -25 -> below pace floor (below pace floor (headroom 45% >= 5% but pace mixed, reserve -31 < -25))
PASS: budget: FM_FLEET_RESERVE_MIN=-100 escape hatch disables the pace floor (ok (headroom 45% >= 5%, pace mixed, reserve -31))
PASS: budget: pressured but reserve unmeasurable -> ok, fail-open (ok (headroom 50% >= 5%, pace pressured but reserve unmeasurable, fail-open))
PASS: budget: stale provider's ahead pace (R1) is ignored, not a refusal (ok (headroom 60% >= 5%, no conservation pressure))
PASS: pick: higher-headroom-ahead (A,80%,-18) loses to lower-headroom-behind (B,55%,+20)
PASS: pick: mixed+aheadWindowIds (C) treated as pressured, not healthy, vs B
PASS: pick: among pressured, -4 (D) beats -18 (A)
PASS: pick: known-sustainable (B) preferred over explicit unknown (E)
PASS: pick: surface below FM_FLEET_QUOTA_MIN (G, 3%) excluded before any pace ordering
PASS: pick: R1 — stale surface's pace (H) not trusted as 1a-sustainable, fresh B wins
PASS: pick: code comment states operator-facing diagnostic, never called from dispatch
PASS: pick: code comment states map order is a documented operator preference
PASS: G5: v2 quota — other 5 columns byte-identical to T3 golden, PACE/RESERVE all —
PASS: G5: v2 models output byte-identical to T3 golden
PASS: G5: v2 pick grok byte-identical to T3 golden
PASS: G5: v2 pick claude byte-identical to T3 golden
PASS: G5: v2 pick kimi byte-identical to T3 golden
PASS: G5: v2 pick bogus byte-identical to T3 golden
PASS: G5: v2 budget (message+exit) byte-identical to T3 golden
PASS: T11: quota exits 0 with an empty bin/quota-sources/ dir
PASS: T11: quota still renders the native claude row with quota-sources/ empty
-----
ALL PASS (62)
exit=0
Evidence: KB and guard suites
$ bash tests/federation/test_fleet.sh
PASS: init creates KB
PASS: atomic claim: exactly one winner, one record
PASS: reap requeues stale claim
PASS: reap leaves in-flight alone
PASS: reap leaves an unreadable stamp claimed (fail-safe)
PASS: route: backend->adi, web->royce
PASS: route: offline owner -> overflow
PASS: handoff reassigns + logs event
PASS: handoff of a queued item -> claimed-by + status:claimed
PASS: handoff of a queued item moves it into ## Claimed
PASS: handoff of a queued item wakes the recipient
PASS: handed-off item is no longer claimable
PASS: handoff of an in-flight item keeps status:in-flight
PASS: queue refuses a duplicate id
PASS: the original item survives a rejected duplicate
PASS: view renders events
PASS: status renders header
PASS: safety: foreign home refused
PASS: safety: /opt shared dir allowed
-----
ALL PASS
exit=0

$ bash tests/federation/test_fleet_guards.sh
PASS: uninitialized fleet exits non-zero (was 0)
PASS: uninitialized fleet names the problem
PASS: no raw awk error
PASS: diagnostic says HOW the dir was chosen
PASS: default -> foreign fleet: 'status' refuses
PASS: default -> foreign fleet: 'status' leaks nothing
PASS: default -> foreign fleet: 'view' refuses
PASS: default -> foreign fleet: 'view' leaks nothing
PASS: route discloses no foreign operator
PASS: explicit FM_FLEET_DIR is honoured (no ownership check)
PASS: FM_FLEET_ACCEPT_DEFAULT=1 acknowledges the default
PASS: default is allowed when you ARE an operator in it
PASS: models is not blocked by a missing fleet
PASS: budget is not blocked by a missing fleet
PASS: wait refuses an uninitialized fleet loudly
PASS: default -> foreign fleet: wait refuses
PASS: default -> foreign fleet: wait leaks nothing
-----
ALL PASS (17)
exit=0
- Outcome: ⚠️ 2 infos across 1 run (8m29s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 warning
  • ⚠️ bin/fm-fleet-lib.sh:220 - fm_fleet_handoff on an item that is still queued produces a half-state that reports success but strands the work. Reassigning a never-claimed item takes the else branch and stamps claimed-by:&lt;to&gt;@&lt;ts&gt; while leaving status:queued. fm-fleet-wait.sh's wake grep is claimed-by:$op@[^ ]+ status:claimed, so the recipient never wakes, yet fm-fleet.sh handoff exits 0 and writes a handoff event. The line also stays claimable: a later fm_fleet_claim matches status:queued and prepends a second stamp, yielding claimed-by:royce@t1 claimed-by:adi@t2 status:claimed, after which fm_fleet_status's claimed-by:$op@.*status:claimed grep credits royce with adi's claim. Either reject a handoff of a non-claimed item, or have it move the item to Claimed like fm_fleet_claim does.
  • ⚠️ bin/fm-fleet-lib.sh:197 - fm_fleet_claim silently deletes a backlog line when two queued items share an id. fm_fleet_queue has no uniqueness guard, so fm-fleet.sh queue W-1 ... twice writes two [id:W-1] ... status:queued lines. The claim awk rule matches both, overwrites held with the second, and nexts past both without printing — so the first line disappears from backlog.md entirely, with the commit message still reading fleet: claim W-1. Suggest rejecting a duplicate id in fm_fleet_queue under the lock (it already holds it).
  • ⚠️ bin/fm-fleet-lib.sh:285 - fm_fleet_route's heartbeat-freshness rule silently never fires without GNU date. epoch() shells out to date -u -d &#34;&lt;iso&gt;&#34; +%s; BSD date (macOS, which README.md advertises as a supported platform) treats -d as the DST flag and does not parse the ISO string, so epoch returns 0 or the current time and if(ep&gt;0 &amp;&amp; (now-ep)&gt;ttl) can never trip. Every operator then routes as fresh, defeating the self-healing property docs/fleet-quickstart.md:175 promises ("if a firstmate crashes ... routing skips them"). The write verbs fail loudly on macOS because flock(1) is absent, but route takes no lock and fails silently. The add-on also relies on realpath -m, groupadd, and usermod with no documented platform restriction and no preflight check — either declare it Linux-only or refuse up front.
  • ℹ️ bin/fm-fleet.sh:58 - The budget verb echoes $fm_fleet_budget_reason identically in both branches. Because fm-fleet.sh runs under set -e, the collapsed form needs the || guard: rc=0; fm_fleet_budget_ok || rc=1; echo &#34;$fm_fleet_budget_reason&#34;; exit &#34;$rc&#34;. Same output, same exit codes, one echo.
  • ℹ️ bin/fm-accounts-lib.sh:136 - The config-dir-flag branch of _fm_account_headroom runs quota-axi with no isolation applied at all — it drops $flag/$cdir on the floor — so every flag-isolated account of the same harness would report identical headroom rather than its own. It is unreachable today only because fm_account_quota_provider returns non-zero for cline, the sole flag harness, and fm_account_pick bails before this call. The moment a flag-isolated harness gains quota coverage, fm_account_pick starts comparing the same number against itself. Make the branch return no output explicitly (matching the *) default) so it degrades to the documented first-registered fallback.
  • ℹ️ bin/quota-cursor-usage.sh:32 - The Cursor bearer token is written into the 0600 header file before the cleanup trap is installed, so an interrupt or kill between the write and line 32 leaves the access token sitting in a temp file. bin/quota-copilot-usage.sh gets the ordering right (trap at line 126, write at 127). Move trap &#39;rm -f &#34;$hdr&#34;&#39; EXIT to immediately after hdr=$(mktemp); chmod 600 &#34;$hdr&#34; on line 27.
  • ℹ️ bin/fm-accounts-prereq.sh:50 - The remote-script confirmation prompt is gated on the install command ending with | bash or | sh. Today's only such entry (curl https://cursor.com/install -fsS | bash) matches, so the guard works — but any future variant with trailing arguments (| bash -s -- --prefix ~/.local, | sh -) would silently skip the review prompt and pipe a remote script into a shell via the eval on line 57. Match the pipe-into-shell shape anywhere in the string, or mark the entries needing confirmation explicitly in fm_prereq_cmd.

🔧 Fix: fix fleet handoff, duplicate ids, and non-GNU date staleness
1 warning still open:

  • ⚠️ tests/federation/test_fleet.sh:50 - The new reap fail-safe regression test is vacuous — it passes identically on pre-fix code. @not-a-timestamp never reaches epoch(): reap's candidate guard is $0 ~ /claimed-by:[^@]+@[0-9TZ:-]+/ (bin/fm-fleet-lib.sh:292), and n is not in [0-9TZ:-], so the line is skipped before the new ep &gt; 0 check on line 297 is ever evaluated. That outer guard is unchanged by this commit, so the assertion would have passed before the fix too, leaving the deliberate behavior change (unreadable stamp → leave the claim alone, where GNU date previously parsed it and reaped) unlocked. Use a stamp that is inside the [0-9TZ:-] charset but off-shape — e.g. s/\(FL-11.*\)@[0-9TZ:-]\{1,\}/\1@2000-01-01/ — which passes the candidate guard, fails the new ^....-..-..T..:..:..Z$ shape check, and would have been reaped by the old date -u -d &#34;2000-01-01&#34; path.
⚠️ **Test** - 2 infos
  • ℹ️ tests/federation/test_fleet_ops.sh:180 - The deliberately-retained live case (quota_now (LIVE, real quota-axi)) still fails when the real quota-axi HANGS rather than answers: FAIL: quota_now LIVE (rc=124 got &#39;&#39;). The timeout 10 bound works correctly (suite finished in 11s, not 600s), and this is distinct from the exhausted-window false red the change fixes — a hang is a genuinely broken environment, and going red on it is what "live coverage not silently deleted" means. Recording it only so the one remaining live-red mode is known; no change requested.
  • ℹ️ bin/fm-fleet-lib.sh:1 - The intent states the split left bin/fm-fleet-lib.sh at 393 lines. That was exact at the split commit 282908c, but the later review commit 007cc36 added the non-GNU date fallback and other fixes, so at the target commit the file is 442 lines (quota lib unchanged at 460). Still well under the 800-line cap, so the stated goal holds — only the specific number would be stale if it is carried verbatim into the PR description.
  • bash tests/federation/test_fleet_ops.sh — normal environment, real quota-axi present (ALL PASS, 24 assertions)
  • PATH=&lt;exhausted-quota-axi&gt;:$PATH bash tests/federation/.pre-hermetic-fleet-ops.tmp.sh — pre-hermetic test from git show 03ba378:tests/federation/test_fleet_ops.sh under a stub reporting 0% remaining, reproducing the reported 4/16 failure
  • PATH=&lt;exhausted-quota-axi&gt;:$PATH bash tests/federation/test_fleet_ops.sh — fixed suite under the identical exhausted stub (ALL PASS)
  • PATH=&lt;PATH-minus-~/.local/bin&gt; bash tests/federation/test_fleet_ops.sh — quota-axi absent entirely (ALL PASS); same PATH against the pre-hermetic copy for contrast
  • PATH=&lt;sentinel-7.5-quota-axi&gt;:$PATH bash tests/federation/test_fleet_ops.sh — stub-isolation proof: lifecycle cases read 42/2/80 from SUITE_STUB while fm_fleet_quota_now on the real path returns 7.5
  • PATH=&lt;hanging-quota-axi&gt;:$PATH bash tests/federation/test_fleet_ops.sh and PATH=&lt;nonzero-exit-quota-axi&gt;:$PATH ... — boundedness of the retained live case (11s elapsed under a 600s hang; garbage/nonzero exit passes as '-')
  • bash tests/federation/test_quota_surfaces.sh — 62 assertions including the seven G5 golden byte-comparisons, with FM_TEST_REGEN_GOLDEN deliberately unset
  • bash tests/federation/test_fleet.sh and bash tests/federation/test_fleet_guards.sh — KB/route/claim and guard suites against the post-split lib
  • Manual A/B: reconstructed the pre-split tree via git show 1850451:bin/fm-fleet.sh + git show 1850451:bin/fm-fleet-lib.sh and diffed fm-fleet.sh quota|models|pick grok|pick claude|pick kimi|pick bogus|budget pre-split vs HEAD under one shared stub env
  • Manual check: diff &lt;(git show 59ccbc9:tests/federation/golden/v2-*.golden) tests/federation/golden/v2-*.golden plus git status --porcelain tests/federation/golden/ after the suite run, confirming goldens were never regenerated
  • Manual leaf check: bash -c &#39;. bin/fm-fleet-quota-lib.sh; declare -F | grep -c fm_fleet_; fm_fleet_quota_now; fm_fleet_budget_ok; fm_fleet_pick_surface claude&#39; with fm-fleet-lib.sh never sourced, and the reverse consumer-contract check
  • wc -l on git show 282908c:bin/fm-fleet-lib.sh (393), git show 1850451:bin/fm-fleet-lib.sh (816), and HEAD's 442 + 460
⚠️ **Document** - 1 info
  • ℹ️ .agents/skills/federation/SKILL.md:37 - Out-of-scope consolidation worth a follow-up: the fleet KB contract is stated in full twice — docs/fleet-addon.md "Shared KB" (maintainer-architecture) and .agents/skills/federation/SKILL.md "KB files" (agent-runtime) both restate the operators.md columns, the backlog.md sections, the item-line format, and the events.log TSV shape. The one-owner rule forbids two full copies, and this branch already shows the predicted drift: the corrected handoff semantics (a queued item is moved to ## Claimed as status:claimed) landed only in the skill, so handoff's state machine now lives in an agent-runtime surface while the sibling claim protocol lives in the maintainer-architecture surface. Reducing the skill's KB-files section to a pointer at docs/fleet-addon.md (keeping only the routing-eligibility rule agents genuinely need inline) would fix this, but it rewrites prose this change did not make stale, so I left it out of scope rather than multiplying edits here.
✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

adibirzu added 25 commits July 28, 2026 12:24
…edit); secure api-key exec shim; fix subshell-export bug (live-verified isolation)
…under isolation; tie/absent/no-provider guards)
…detect default; pi=system-managed detect-only; curl|sh gated)
…en economy

Makes federated mode actually usable per-operator, in-sync, and token-cheap.

- fm-fleet-lib.sh: register/heartbeat/leave (self-onboard, upsert, own-home-only);
  operators.md gains seen+quota columns; route is now freshness-aware (stale
  heartbeat => offline, TTL FM_FLEET_HEARTBEAT_TTL) AND quota-aware (skip operators
  below FM_FLEET_QUOTA_MIN, headroom self-published on heartbeat — no cross-user
  auth); fm_fleet_quota_now + fm_fleet_budget_ok. Heartbeat is a bash file-write,
  never git-committed (no audit-log bloat).
- fm-fleet.sh: register|heartbeat|leave|budget verbs.
- fm-fleet-wait.sh: bash wake-on-claim watcher (0 LLM tokens) — a primary blocks
  here and its LLM wakes only when it has a fresh status:claimed item; heartbeats
  while waiting.
- fm-fleet-join.sh: idempotent self-onboarding (verify shared access -> point
  config/fleet-dir -> register); cross-uid-safe.
- docs/fleet-token-economy.md + federation SKILL: the event-driven, bash-coordinated
  model and its knobs.
- tests/federation/test_fleet_ops.sh: 16 checks (all green); existing test_fleet.sh
  stays green (schema/route change is backward-compatible with 5-column rows).
…eged step)

scripts/fleet-root-prereq.sh creates the agents group + group-writable
/opt/agents/fleet (2775 setgid) + operator group membership. Idempotent, additive,
with reverse steps. Run once: sudo bash scripts/fleet-root-prereq.sh.
… authed override hook

- fm-fleet.sh quota: every LLM/CLI/app pool + observability (quota-axi + pluggable bin/quota-sources/*)
- fm-fleet.sh models: model family -> serving surfaces with live status
- fm-fleet.sh pick <family>: failover selector (grok from grok|cursor; kimi3 via cline)
- config/model-surfaces.json: model -> ordered pools
- bin/quota-sources/{cline,cursor}.sh: surfaces quota-axi can't see; authed numbers via
  config/quota-overrides.json escape hatch (operator-supplied reader supersedes blind row)
- cursor usage is browser-session-gated (CLI bearer can't read it): documented, blind fail-open default
- tests/federation/test_quota_surfaces.sh: 8 hermetic checks
…room)

Reads a STABLE CLINE_API_KEY (preferred) or cline's own ~1h WorkOS session token from
providers.json and queries GET api.cline.bot/api/v1/users/<uid>/balance; maps credit
balance -> routability headroom (credits>0 => 100). Blind when the token is stale (we do
NOT refresh out-of-band: the refresh token is single-use and would break cline's login).
Token via 0600 header file, never argv. Wire via config/quota-overrides.json .cline.
…e -> headroom)

Cursor usage IS readable with the CLI's OWN access token (no browser cookie) via the
native Connect RPC POST api2.cursor.sh/aiserver.v1.DashboardService/GetCurrentPeriodUsage
(Content-Type: application/json, Connect-Protocol-Version: 1, x-cursor-client-* headers).
headroom = 100 - planUsage.totalPercentUsed. Verified live: 71%. Token via 0600 header
file, never argv. Wire via config/quota-overrides.json .cursor.
… reader note fix

cline balance is locked behind an internal WS-hub credential (WorkOS token 401s on every
REST/header variant), so it can't be monitored. Remove its quota-source + reader + model-
surfaces entries; cline stays a usable crewmate harness (fm-spawn adapter untouched) but is
out of quota routing. Cursor stays live (71%). Tests updated (cline-free), all green.
Adds GitHub Copilot CLI as a live-monitored quota surface and repairs the
cursor reader, which had gone blind.

copilot:
- quota-axi ships a native `copilot` provider but probes only the OLD IDE
  credential path (~/.config/github-copilot/apps.json), so a standalone
  Copilot CLI login is invisible to it and the row sits at auth_required
  forever. bin/quota-sources/copilot.sh supersedes that row.
- bin/quota-copilot-usage.sh reads the CLI's own token from
  ~/.copilot/config.json (.copilotTokens; JSON-with-//-comments) and calls
  GET api.github.com/copilot_internal/user, taking the MIN percent_remaining
  across metered quota_snapshots buckets (chat/completions report
  unlimited=true; premium_interactions is the metered one). No browser
  cookie and no second login required — unlike cursor and cline.
- Token via 0600 header file, never argv/stdout; any failure exits 0 with no
  output => blind / fail-open, matching the cursor contract.
- model-surfaces: claude -> [claude, copilot, cursor], gpt -> [codex,
  copilot, cursor], so both families can now fail over into Copilot.

cursor fix:
- `cursor-agent about` emits ANSI SGR codes around the version string. The
  unstripped ESC[22m landed in x-cursor-client-version, and the usage RPC
  answered 400 Bad Request -> reader printed nothing -> surface blind. Strip
  CSI sequences and restrict to version-safe characters. Live again.

tests: test_quota_surfaces 8 -> 12 checks (custom source supersedes the
auth_required row; claude/gpt reach copilot in the models view; pick claude
fails over to copilot when the claude pool is drained). Federation
regression 49/49 green.

Verified live: copilot 71%, cursor 69% (restored).

(cherry picked from commit 9e4ba3a)
Ran the actual fresh-clone experience instead of reading the code. Four defects,
one of them a data leak.

1. A clone that configured NOTHING silently attached to the built-in default fleet
   dir. Where that path already exists (a conventional, group-writable, world-
   readable location), `status`/`route`/`view` listed another team's operators and
   dumped their entire event log — task ids, scopes, who claimed what. Membership
   is now the opt-in signal: reaching a fleet you are not an operator of *via the
   default* is refused. An explicit FM_FLEET_DIR or config/fleet-dir is always
   honoured, and FM_FLEET_ACCEPT_DEFAULT=1 acknowledges the default. register/
   heartbeat/leave are exempt (they are how you become an operator, and already
   need group write access).

2. An uninitialized dir surfaced as `awk: fatal: cannot open .../operators.md`
   with exit 0 — a raw internal error that also looked like success to a caller.
   Now a diagnostic naming the dir, HOW it was chosen, and the fix; exit 1.

3. scripts/fleet-root-prereq.sh defaulted to the maintainer's own usernames, so
   running it on any other host tried to enrol accounts that mean nothing there.
   Defaults to ${SUDO_USER} and refuses when it cannot infer one.

4. config/accounts.json.example hardcoded /home/adi in all 7 entries; paths are
   used literally (no ~ or $HOME expansion), so a copied template silently pointed
   into a foreign home. Now /home/YOUR-USER with that constraint documented.

Also: actionable messages when quota-axi is absent vs present-but-unauthenticated,
and the federation skill no longer presents one maintainer's home name as canonical.

docs/fleet-quickstart.md is the missing entry point, organised by use case:
  A  see remaining budget across every AI subscription + model->surface failover
     (no root, no fleet, works in a bare clone)
  B  several accounts for one person
  C  several people on one host
with requirements per tier, fleet-dir resolution, and troubleshooting for the
traps that cost real time here (stale group credentials in a long-lived tmux or
systemd --user manager; XDG_RUNTIME_DIR; heartbeats being mandatory for routing).
Linked from README so the feature is discoverable at all — previously nothing in
README or AGENTS mentioned it.

tests/federation/test_fleet_guards.sh: 13 checks over dir resolution, both guards,
and non-disclosure. Note the guards were briefly inert: `DIR=$(fm_fleet_dir)` runs
in a subshell, so a global set inside it never reached the caller — the provenance
is a function now, and the tests pin it.

Federation + adapter suites: 9/9 green.

(cherry picked from commit 4279c65)
…add-on

SC2034 in fm-account-env.sh: FM_ACCT_ARGV_SUFFIX is read by fm-account-exec.sh
after sourcing — annotated with the repo's existing disable convention, placed
before the case statement (a directive on an individual case branch is SC1124).
SC2015 A-&&-B-||-C in the two column|cat table fallbacks and the budget verb —
rewritten as explicit if/else.

Found by running bin/fm-lint.sh locally with the pinned shellcheck; these files
predate CI running on the fork PRs (first-time-fork approval), so upstream lint
had never seen them.

(cherry picked from commit 867e6e2)
The Repo invariants CI job requires zero tracked files under config/.
Move the three shipped fleet files to the docs/examples/ precedent
(accounts.json, quota-overrides.json, model-surfaces.json), teach
fm-fleet-lib.sh to fall back from the gitignored personal
config/model-surfaces.json to the shipped default so a bare clone still
routes, gitignore the personal override, and update the docs, skills,
tests, and documentation-audiences inventory references.

Verified locally: invariant ls-files empty, symlinks intact, fm-lint,
fm-doc-audience-check, coverage guard, and all 7 federation tests pass.
…T_DIR

The ownership-guard cases must resolve the fleet dir through the built-in
DEFAULT, because that is the only resolution source the guard applies to. An
exported FM_FLEET_DIR — which every real fleet operator has in their shell —
flipped the source to `env`, where the guard correctly skips, so three checks
reported failures that said nothing about the code.

Scrub the variable in the fleet() helper and in the two default-resolution
cases that call env directly rather than through it. Cases that deliberately
exercise the env source still set FM_FLEET_DIR explicitly and are unchanged.

Verified identical now in three environments: FM_FLEET_DIR unset, set to the
live fleet, and set to a nonexistent path — 17/17 in each. Other federation
suites and fm-lint unaffected.
Upgrade the shared ~/.local/bin/quota-axi 0.1.13 -> 0.1.15 (schemaVersion
2 -> 3) and teach the fleet's operator-facing quota surfaces to read the
new quota-window pace signals (reservePercentPoints = percentRemaining -
timeRemainingPercent), so an operator can tell 69%-healthy from
69%-burning at a glance instead of reading a bare percentage.

- fm_fleet_pace_rows / fm_fleet_pressured / fm_fleet_reserve_cmp /
  fm_fleet_render_reserve: new pace-extraction primitives in
  bin/fm-fleet-lib.sh (§5.1/§5.2), verbatim adoption of upstream
  quota-array-dispatch's conservation-pressure predicate.
- `fm-fleet.sh quota` gains PACE + RESERVE columns (reserve always
  signed); a bare-int custom source masking a paced native row gets a
  "custom int masks native pace" advisory NOTE.
- `fm_fleet_budget_ok` refuses on conservation pressure, not just raw
  headroom: new FM_FLEET_RESERVE_MIN floor (default -25, -100 disables
  it), restricted to FRESH providers only (R1 — a stale window's reserve
  only ages toward looking MORE ahead-of-pace). Prints inspectable facts
  (headroom/pace/reserve), never a bare verdict.
  Caller audit (T7): bin/fm-fleet.sh `budget` verb (same 0/1 exit
  convention); tests/federation/test_fleet_ops.sh (checks exit code
  only, v2-shaped stub -> legacy path, unaffected); docs/
  fleet-token-economy.md + .agents/skills/federation/SKILL.md (prose,
  updated here).
- `fm_fleet_pick_surface` splits pass 1 into 1a (known sustainable) / 1b
  (unknown or absent pace) / 1c (pressured, least-negative reserve
  wins); still purely an operator-facing diagnostic, never wired into
  fm-spawn or any dispatch path (anti-goal §4.1) — dispatch's pace-aware
  selection stays owned solely by upstream's quota-array-dispatch skill.
- Graceful degradation (G5) is exact: against a schemaVersion 2 payload
  or any provider missing pace, quota/models/pick/budget stay
  byte-identical to pre-change output (T3 golden fixtures, diffed by
  new tests) — pace never fabricated, `unknown` never conflated with
  absent `-`.
- docs/fleet-addon.md + .agents/skills/federation/SKILL.md: pace surface
  docs, the operator-facing-diagnostic boundary statement (verbatim,
  rebase-safe), and the custom-source retirement path (gates G1-G4,
  deletion list, survival list) — documented only, zero files deleted.
  AGENTS.md, docs/configuration.md, bin/fm-bootstrap.sh untouched.
- tests/federation/test_quota_surfaces.sh: 12 original checks unmodified
  (zero lines removed, pure appends) + 50 new pace/degradation checks =
  62 total, all passing. New empty-bin/quota-sources/-dir regression
  test (T11) for the Phase-B retirement's loader-tolerance requirement.

Verification: bin/fm-lint.sh clean; all tests/federation/*.sh green
(test_fleet_guards.sh 17/17); fm-instruction-owners.test.sh and
fm-documentation-audiences.test.sh green; live fleet proven unaffected
by the quota-axi upgrade (fm_fleet_quota_now still numeric from the
unchanged primary checkout, `fm-fleet.sh quota`/`route backend` render
correctly, /opt/agents/fleet heartbeat unbroken).
…tate

The register/heartbeat/route lifecycle cases shelled out to the real
quota-axi, so the suite's result depended on the operator's own paid
five-hour window (observed: 4/16 failures at 0% headroom, all green after
the window reset). Stub quota-axi on PATH (exported, since cases drive
bin/fm-fleet.sh and bin/fm-fleet-join.sh as subprocesses) with a rewritable
payload, mirroring the existing hermetic pattern in test_quota_surfaces.sh.

Hermeticity is used to sharpen the assertions it unblocks, not to weaken
them:
- register now asserts the stubbed value literally lands in the quota
  column, not just that a row exists.
- the stale-heartbeat sed gets a post-condition proving it hit the seen
  column, not quota.
- heartbeat now asserts seen actually advances and reparses as a
  timestamp, not just that routing changed its mind.
- a new case re-registers through the real register->quota-column->route
  chain at a low then high stub value, turning today's accidental
  live-quota failure into a deliberate, deterministic assertion in both
  directions.
- one case restores the real PATH and exercises fm_fleet_quota_now live,
  timeout-bounded and shape-only (percentage or '-'), so the suite keeps
  its only genuine coverage that quota-axi --json | jq still parses --
  hermeticity must not silently delete that.

Verified ALL PASS (24 cases, ~1s vs ~5.3s baseline) under three hostile
quota states: 0% stub, 100% stub, and quota-axi absent from PATH entirely
(PATH=/usr/bin:/bin) -- the exact reproduction of the prior 4-failure run.
Negative-control: inverting one new assertion made exactly that case fail;
reverted.
bin/fm-fleet-lib.sh had grown to 816 lines (over the 800-line ceiling)
during the pace-aware quota work. A verified call-graph analysis found a
clean seam: the quota/pace surface layer (fm_fleet_quota_now through
fm_fleet_budget_ok) is one contiguous block with exactly two KB->surface
crossings (register/heartbeat calling fm_fleet_quota_now) and zero
surface->KB dependencies.

Move that block (431 lines, verbatim -- comments, shellcheck directives,
and the three BASH_SOURCE-derived FM_HOME fallbacks included, unedited)
into a new leaf library, bin/fm-fleet-quota-lib.sh, independently
sourceable with zero fleet-KB dependencies. bin/fm-fleet-lib.sh sources it
via the repo's established source= idiom (copied from fm-marker-lib.sh),
so every existing consumer (fm-fleet.sh, fm-fleet-join.sh, fm-fleet-wait.sh,
tests/federation/*.sh) keeps sourcing only bin/fm-fleet-lib.sh with no
change to its sourcing contract. The relocated operator-lifecycle section
header now sits directly above fm_fleet_assert_own_home instead of
dangling above a hole.

Repoint the two path-coupled assertions in test_quota_surfaces.sh that
grepped fm-fleet-lib.sh by path for the anti-goal comments guarding
fm_fleet_pick_surface -- the only breakage from the move.

Zero functional change: all seven G5 golden assertions in
test_quota_surfaces.sh stay byte-identical (FM_TEST_REGEN_GOLDEN never
set), and both files land well under the 800-line ceiling (393 + 460).
fm_fleet_pick_surface remains unreachable from bin/fm-spawn.sh (anti-goal,
unchanged).

Update docs/scripts.md and docs/fleet-addon.md to name the new file.
@adibirzu

Copy link
Copy Markdown
Author

This PR supersedes both #1187 and #1103 — please review only this one

Sorry for the noise: I stacked three consecutive pieces of work on each other's branches instead of branching each from main, so the pipeline opened a PR per piece. They form a strict chain:

PR files relationship
#1103 35 fully contained in #1187 and #1217
#1187 43 fully contained in #1217
#1217 44 everything, + bin/fm-fleet-quota-lib.sh

Verified by file-set comparison: 0 files in #1103 or #1187 are absent from #1217. Merging #1217 alone loses nothing. I'm happy to close the other two — say the word, or close them yourself.

What the chain contains, oldest to newest

  1. Fleet add-on (feat(fleet): federated multi-operator coordination with per-surface quota routing #1103) — federated multi-operator coordination + per-surface subscription-quota visibility, hardened so it is usable from a bare clone (ownership/initialization guards, generic root prereq, use-case quickstart).
  2. Pace alignment (feat(fleet): federated multi-operator add-on with pace-aware quota surfaces #1187) — consumes quota-axi 0.1.15 schemaVersion 3 pace signals: PACE/RESERVE columns, a conservation-pressure floor on budget, and pick preferring sustainable surfaces. Deliberately not wired into dispatch — feat: route crew dispatch using quota-window pace #1172's quota-array-dispatch skill owns pace-aware selection and forbids routing wrappers, so this is observability only. Graceful degradation to schemaVersion 2 is gated by committed goldens.
  3. This PR — two follow-ups from a review pass:
    • tests/federation/test_fleet_ops.sh was not hermetic: its register/route lifecycle cases shelled out to the live quota-axi, so it went red whenever my real subscription window was exhausted (observed 4/16 failing, then all green after the window reset). Now stubbed, with lifecycle semantics still genuinely exercised and one bounded case deliberately retained on the real quota_now path so live coverage is not silently deleted.
    • bin/fm-fleet-lib.sh had reached 816 lines, past the 800-line convention. Split at a verified one-directional seam (surface→KB crossings: zero; KB→surface: exactly two, both fm_fleet_quota_now) into a leaf bin/fm-fleet-quota-lib.sh. Result: 393 + 460. Behaviour-preserving — proven by comparing the committed golden fixtures, never regenerating them.

Live sanity from this tip, showing the three pace states rendering distinctly and the reserve floor doing real work:

SURFACE  HEADROOM  PACE    RESERVE   STATUS
claude   15%       mixed   -28.5374  fresh     <- past the default -25pp floor
codex    99%       behind  +7.7687   fresh
grok     —         —       —         auth_required

One caveat worth stating plainly: tests/federation/*.sh sit outside the coverage partition, so none of these assertions run in CI — they are local-gate only.

@kunchenguid

kunchenguid commented Jul 30, 2026 •

Copy link
Copy Markdown
Owner

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#1217 at d87fbe3a.

@kunchenguid kunchenguid removed the wheelhouse:pending-contributor-action Managed by Wheelhouse label Jul 30, 2026
@kunchenguid

Copy link
Copy Markdown
Owner

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#1217 at a2b1ce0c.

@kunchenguid

Copy link
Copy Markdown
Owner

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#1217 at ce2f9073.

@kunchenguid

Copy link
Copy Markdown
Owner

Automated reminder: this PR still looks blocked on a rebase or merge conflict fix.

If you are still interested, please rebase onto the current base branch, resolve the conflict, and push.

If I do not hear back, I may close this as inactive.

@kunchenguid kunchenguid added wheelhouse:pending-contributor-action Managed by Wheelhouse and removed wheelhouse:pending-contributor-action Managed by Wheelhouse labels Aug 10, 2026
@adibirzu

Copy link
Copy Markdown
Author

Superseded by #1338 (consolidated federated-fleet + crew-routing PR): older federated-fleet branch; its content is superseded by the #1338 consolidated lineage. Closing in favor of the single consolidated PR. This can be reopened if needed.

@adibirzu adibirzu closed this Aug 11, 2026
MemoHealth added a commit to MemoHealth/firstmate that referenced this pull request Sep 2, 2026
The encoding gate runs over the PR title and body, not only the tree, for
the same reason the trailer rule does: they become master's commit
message. Two arrows in one line of a description cost a red required
check on kunchenguid#1217. Also note that two consecutive failures on the same head
are a real rejection, not the footer being written away.

The `[in progress]` title marker is now forbidden outright by Munra's own
`.claude/rules/issues.md`, which is the binding contract and outranks this
skill. Claim an issue by assigning it instead. Checking the open-PR list
alone does not prevent a collision: on 2026-09-02 the list was clean at
intake and a parallel session opened its PR for the same issue seven
minutes before our commit, wasting a full build and review round.

Reproducing a layout failure on master LOCALLY proves nothing about
master, because the font artifact reproduces there too. That reasoning
nearly earned a bogus issue against a healthy branch. CI on the same base
settles it, and `DEBUG=pw:webserver` names the artifact directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants