Conversation
|
Updated: the |
Update — usable from a bare clone, plus a use-case quickstartI tested this the way a new user would: cloned the branch into a scratch dir with a 1. A clone that configured nothing silently attached to another team's fleet. Membership is now the opt-in signal. Reaching a fleet you are not an operator of 2. An uninitialized dir surfaced as a raw internal error that looked like success. 3. 4. docs/fleet-quickstart.mdThe add-on had three reference docs but no entry point, and nothing in
Tier A needs no fleet, no root and no shared directory — it works in a bare clone. Troubleshooting covers the traps that actually cost time here: stale group Note on scopeThis PR now touches
Federation + adapter suites: 9/9 green. Rebased on current |
…ting, handoff, view, cross-uid safety
…, cross-uid-safe paths)
…edit); secure api-key exec shim; fix subshell-export bug (live-verified isolation)
…under isolation; tie/absent/no-provider guards)
…detect default; pi=system-managed detect-only; curl|sh gated)
…en economy Makes federated mode actually usable per-operator, in-sync, and token-cheap. - fm-fleet-lib.sh: register/heartbeat/leave (self-onboard, upsert, own-home-only); operators.md gains seen+quota columns; route is now freshness-aware (stale heartbeat => offline, TTL FM_FLEET_HEARTBEAT_TTL) AND quota-aware (skip operators below FM_FLEET_QUOTA_MIN, headroom self-published on heartbeat — no cross-user auth); fm_fleet_quota_now + fm_fleet_budget_ok. Heartbeat is a bash file-write, never git-committed (no audit-log bloat). - fm-fleet.sh: register|heartbeat|leave|budget verbs. - fm-fleet-wait.sh: bash wake-on-claim watcher (0 LLM tokens) — a primary blocks here and its LLM wakes only when it has a fresh status:claimed item; heartbeats while waiting. - fm-fleet-join.sh: idempotent self-onboarding (verify shared access -> point config/fleet-dir -> register); cross-uid-safe. - docs/fleet-token-economy.md + federation SKILL: the event-driven, bash-coordinated model and its knobs. - tests/federation/test_fleet_ops.sh: 16 checks (all green); existing test_fleet.sh stays green (schema/route change is backward-compatible with 5-column rows).
…eged step) scripts/fleet-root-prereq.sh creates the agents group + group-writable /opt/agents/fleet (2775 setgid) + operator group membership. Idempotent, additive, with reverse steps. Run once: sudo bash scripts/fleet-root-prereq.sh.
… authed override hook
- fm-fleet.sh quota: every LLM/CLI/app pool + observability (quota-axi + pluggable bin/quota-sources/*)
- fm-fleet.sh models: model family -> serving surfaces with live status
- fm-fleet.sh pick <family>: failover selector (grok from grok|cursor; kimi3 via cline)
- config/model-surfaces.json: model -> ordered pools
- bin/quota-sources/{cline,cursor}.sh: surfaces quota-axi can't see; authed numbers via
config/quota-overrides.json escape hatch (operator-supplied reader supersedes blind row)
- cursor usage is browser-session-gated (CLI bearer can't read it): documented, blind fail-open default
- tests/federation/test_quota_surfaces.sh: 8 hermetic checks
…room) Reads a STABLE CLINE_API_KEY (preferred) or cline's own ~1h WorkOS session token from providers.json and queries GET api.cline.bot/api/v1/users/<uid>/balance; maps credit balance -> routability headroom (credits>0 => 100). Blind when the token is stale (we do NOT refresh out-of-band: the refresh token is single-use and would break cline's login). Token via 0600 header file, never argv. Wire via config/quota-overrides.json .cline.
…e -> headroom) Cursor usage IS readable with the CLI's OWN access token (no browser cookie) via the native Connect RPC POST api2.cursor.sh/aiserver.v1.DashboardService/GetCurrentPeriodUsage (Content-Type: application/json, Connect-Protocol-Version: 1, x-cursor-client-* headers). headroom = 100 - planUsage.totalPercentUsed. Verified live: 71%. Token via 0600 header file, never argv. Wire via config/quota-overrides.json .cursor.
… reader note fix cline balance is locked behind an internal WS-hub credential (WorkOS token 401s on every REST/header variant), so it can't be monitored. Remove its quota-source + reader + model- surfaces entries; cline stays a usable crewmate harness (fm-spawn adapter untouched) but is out of quota routing. Cursor stays live (71%). Tests updated (cline-free), all green.
Adds GitHub Copilot CLI as a live-monitored quota surface and repairs the cursor reader, which had gone blind. copilot: - quota-axi ships a native `copilot` provider but probes only the OLD IDE credential path (~/.config/github-copilot/apps.json), so a standalone Copilot CLI login is invisible to it and the row sits at auth_required forever. bin/quota-sources/copilot.sh supersedes that row. - bin/quota-copilot-usage.sh reads the CLI's own token from ~/.copilot/config.json (.copilotTokens; JSON-with-//-comments) and calls GET api.github.com/copilot_internal/user, taking the MIN percent_remaining across metered quota_snapshots buckets (chat/completions report unlimited=true; premium_interactions is the metered one). No browser cookie and no second login required — unlike cursor and cline. - Token via 0600 header file, never argv/stdout; any failure exits 0 with no output => blind / fail-open, matching the cursor contract. - model-surfaces: claude -> [claude, copilot, cursor], gpt -> [codex, copilot, cursor], so both families can now fail over into Copilot. cursor fix: - `cursor-agent about` emits ANSI SGR codes around the version string. The unstripped ESC[22m landed in x-cursor-client-version, and the usage RPC answered 400 Bad Request -> reader printed nothing -> surface blind. Strip CSI sequences and restrict to version-safe characters. Live again. tests: test_quota_surfaces 8 -> 12 checks (custom source supersedes the auth_required row; claude/gpt reach copilot in the models view; pick claude fails over to copilot when the claude pool is drained). Federation regression 49/49 green. Verified live: copilot 71%, cursor 69% (restored). (cherry picked from commit 9e4ba3a)
Ran the actual fresh-clone experience instead of reading the code. Four defects,
one of them a data leak.
1. A clone that configured NOTHING silently attached to the built-in default fleet
dir. Where that path already exists (a conventional, group-writable, world-
readable location), `status`/`route`/`view` listed another team's operators and
dumped their entire event log — task ids, scopes, who claimed what. Membership
is now the opt-in signal: reaching a fleet you are not an operator of *via the
default* is refused. An explicit FM_FLEET_DIR or config/fleet-dir is always
honoured, and FM_FLEET_ACCEPT_DEFAULT=1 acknowledges the default. register/
heartbeat/leave are exempt (they are how you become an operator, and already
need group write access).
2. An uninitialized dir surfaced as `awk: fatal: cannot open .../operators.md`
with exit 0 — a raw internal error that also looked like success to a caller.
Now a diagnostic naming the dir, HOW it was chosen, and the fix; exit 1.
3. scripts/fleet-root-prereq.sh defaulted to the maintainer's own usernames, so
running it on any other host tried to enrol accounts that mean nothing there.
Defaults to ${SUDO_USER} and refuses when it cannot infer one.
4. config/accounts.json.example hardcoded /home/adi in all 7 entries; paths are
used literally (no ~ or $HOME expansion), so a copied template silently pointed
into a foreign home. Now /home/YOUR-USER with that constraint documented.
Also: actionable messages when quota-axi is absent vs present-but-unauthenticated,
and the federation skill no longer presents one maintainer's home name as canonical.
docs/fleet-quickstart.md is the missing entry point, organised by use case:
A see remaining budget across every AI subscription + model->surface failover
(no root, no fleet, works in a bare clone)
B several accounts for one person
C several people on one host
with requirements per tier, fleet-dir resolution, and troubleshooting for the
traps that cost real time here (stale group credentials in a long-lived tmux or
systemd --user manager; XDG_RUNTIME_DIR; heartbeats being mandatory for routing).
Linked from README so the feature is discoverable at all — previously nothing in
README or AGENTS mentioned it.
tests/federation/test_fleet_guards.sh: 13 checks over dir resolution, both guards,
and non-disclosure. Note the guards were briefly inert: `DIR=$(fm_fleet_dir)` runs
in a subshell, so a global set inside it never reached the caller — the provenance
is a function now, and the tests pin it.
Federation + adapter suites: 9/9 green.
(cherry picked from commit 4279c65)
…add-on SC2034 in fm-account-env.sh: FM_ACCT_ARGV_SUFFIX is read by fm-account-exec.sh after sourcing — annotated with the repo's existing disable convention, placed before the case statement (a directive on an individual case branch is SC1124). SC2015 A-&&-B-||-C in the two column|cat table fallbacks and the budget verb — rewritten as explicit if/else. Found by running bin/fm-lint.sh locally with the pinned shellcheck; these files predate CI running on the fork PRs (first-time-fork approval), so upstream lint had never seen them. (cherry picked from commit 867e6e2)
…budget exemption, prereq chmod
adibirzu
changed the title
The Repo invariants CI job requires zero tracked files under config/. Move the three shipped fleet files to the docs/examples/ precedent (accounts.json, quota-overrides.json, model-surfaces.json), teach fm-fleet-lib.sh to fall back from the gitignored personal config/model-surfaces.json to the shipped default so a bare clone still routes, gitignore the personal override, and update the docs, skills, tests, and documentation-audiences inventory references. Verified locally: invariant ls-files empty, symlinks intact, fm-lint, fm-doc-audience-check, coverage guard, and all 7 federation tests pass.
|
Note for reviewers: #1187 supersedes this PR. That branch was stacked on this one, so it contains all 35 files here plus the quota-pace work (8 further files). Merging #1187 alone loses nothing from this PR; alternatively merge this one first and the CI monitor will rebase #1187 down to just its delta. Sorry for the duplication — the stacking was my error. |
1 participant
Intent
Fleet add-on (PR #1103): federated multi-operator coordination + per-surface subscription-quota visibility with model->surface failover, made safe and usable from a bare clone (ownership + initialization guards, generic root prereq, use-case quickstart docs). Goal: pass the Require-no-mistakes gate and ship the branch through the sanctioned pipeline so PR #1103 carries the pipeline signature.
What Changed
fm-fleet-wait.shclaim-wake loop (bin/fm-fleet.sh,bin/fm-fleet-lib.sh,scripts/fleet-root-prereq.sh).quota/models/pick/budgetverbs, with an authed override hook and float-safe headroom comparisons (bin/quota-sources/,config/model-surfaces.json).--accountspawn axis via a raw-launch shim with fail-closed API-key handling, and on-demand user-scoped prereq installers — all covered by seven test suites undertests/federation/and documented in the fleet add-on, quickstart, and token-economy docs plus federation/multi-account operator skills.Risk Assessment
✅ Low: All four actionable review findings from round 1 are correctly fixed at the recommended shared boundary with new guard tests covering each fix, no new defects were introduced by the fix commit, and the only remaining items are previously surfaced informational no-ops.
Testing
Ran all 7 hermetic federation test suites added by this change (78/78 assertions pass, exit 0 each), then manually exercised the product CLI end-to-end: the bare-clone first-run guard refuses with quickstart guidance, a two-operator fleet completes the queue/route/claim/handoff/view workflow, and the quota surface shows per-surface headroom with correct model->surface failover (including float compares and the budget floor exiting non-zero); the root prereq script was syntax-checked only since it needs sudo, and the worktree was left clean.Evidence: Federation test suites output (7 suites, 78 assertions, all pass)
Evidence: Fleet CLI e2e transcript: bare-clone guard + multi-operator workflow
$ bin/fm-fleet.sh status # bare clone, nothing configured fm-fleet: no initialized fleet at .../nonexistent (nothing configured, so the built-in default ... was used) the directory does not exist. Pick one: solo / trying it out FM_FLEET_DIR=~/.firstmate-fleet bin/fm-fleet.sh init shared, multi-operator sudo bash scripts/fleet-root-prereq.sh already have one export FM_FLEET_DIR=/path/to/fleet (exit 1) $ bin/fm-fleet.sh init && register adi + royce && queue FL-42 backend $ bin/fm-fleet.sh route backend adi $ bin/fm-fleet.sh claim FL-42 adi && bin/fm-fleet.sh handoff FL-42 royce $ bin/fm-fleet.sh status operator claimed in-flight last-event adi 0 0 2026-07-28T07:25:48Z royce 1 0 2026-07-28T07:25:48ZEvidence: Quota surface + model->surface failover transcript
$ bin/fm-fleet.sh quota SURFACE HEADROOM STATUS SOURCE NOTE grok 2% fresh oauth observable claude 3.5% fresh oauth observable codex 90% fresh cli-rpc observable kimi 60% fresh oauth observable copilot 88% logged_in custom gh api reader (stub) cursor 80% logged_in custom authed Connect reader (stub) $ bin/fm-fleet.sh models grok grok: fresh 2% | cursor: logged_in 80% claude claude: fresh 3.5% | copilot: logged_in 88% | cursor: logged_in 80% $ bin/fm-fleet.sh pick grok # drained -> failover cursor $ bin/fm-fleet.sh pick claude # 3.5% float below floor -> failover copilot $ bin/fm-fleet.sh budget below floor (< 5%) (exit 1)Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 6 issues found → auto-fixed ✅
bin/fm-fleet-wait.sh:23- fm-fleet-wait.sh bypasses the bare-clone guards that commit 6150d6a introduced as the durable fix: it resolves DIR via fm_fleet_dir but never calls fm_fleet_assert_initialized or fm_fleet_assert_owned. Reachable failures: (a) bare clone with nothing configured -> fresh_claims greps a nonexistent backlog.md with 2>/dev/null and the script silently waits forever (or --once exits 1 indistinguishable from 'no claim'), the exact silent-failure mode the guards were written to eliminate; (b) on a shared host where the built-in default /opt/agents/fleet is another team's group-writable fleet, a bare clone running fm-fleet-wait.sh reads that team's backlog and, via the heartbeat path (fm_fleet_heartbeat -> fm_fleet_lock), creates/locks files inside their dir without the membership opt-in that fm_fleet_assert_owned enforces for fm-fleet.sh verbs. Recommend enforcing the guard at the shared boundary (e.g. a helper called right after fm_fleet_dir resolution, used by fm-fleet.sh, fm-fleet-wait.sh, and future entry points) instead of only in fm-fleet.sh's case statement.bin/fm-fleet-lib.sh:472- Headroom comparisons use integer-only shell tests and break on fractional percentages. fm_fleet_budget_ok does [ "$q" -ge "$floor" ] 2>/dev/null on the jq min of quota-axi percentRemaining; if quota-axi reports a float (e.g. 90.5), the test errors (suppressed) and the function returns failure — the operator is held back despite ample headroom, inverting the documented fail-open contract. Same pattern in fm_fleet_pick_surface ([ "$h" -ge "$floor" ], surface silently treated as unobservable) and fm_account_pick in fm-accounts-lib.sh:164 ([ "$hr" -gt "$best_hr" ], unsuppressed bash error on stderr and quota-aware pick silently degrades to first-registered). The awk paths (fm_fleet_route's (q+0)<floor) already handle floats; convert these three shell tests to awk numeric comparisons for consistency.bin/fm-fleet.sh:33- Thebudgetverb is misclassified in the guard dispatch: fm_fleet_budget_ok only calls quota-axi and never reads the fleet dir, making it surface-local exactly like quota/models/pick, but it falls into the*branch and therefore requires an initialized AND owned fleet. On a bare clone (Tier A usage, no fleet)fm-fleet.sh budgetfails with 'no initialized fleet' even though nothing it does needs one. Addbudgetto the surface-local exemption list on line 29. Note the guard tests exercise status/view/route/models but not budget, so this gap is untested.scripts/fleet-root-prereq.sh:46-chmod -R 2775 "$DIR"applies the setgid and execute bits to regular files as well as directories. On a re-run over an existing KB (the script advertises idempotent re-runs), operators.md, backlog.md, and events.log become mode 2775 — setgid executables, which on Linux can also flag mandatory locking semantics. Usefind "$DIR" -type d -exec chmod 2775 {} +for directories andchmod -R g+rw(no exec/setgid) for files.bin/fm-fleet-lib.sh:40- fm_fleet_dir and fm_fleet_dir_source duplicate the resolution logic and can disagree: if $FM_HOME/config/fleet-dir exists but its first line is empty, fm_fleet_dir falls through to the built-in default while fm_fleet_dir_source still reports 'config', so fm_fleet_assert_owned returns early and the ownership guard is silently skipped for what is actually the default shared dir. Low likelihood (join always writes a non-empty path), but the two functions must stay in lockstep; deriving the source alongside the dir in one place (e.g. emitting 'dir<TAB>source') would remove the divergence risk.bin/fm-spawn-acct.sh:32- Argument parsing in fm-spawn-acct.sh assigns any unrecognized token to the positional slots while fewer than two are filled, so a passthrough flag appearing before the two positionals (e.g.fm-spawn-acct.sh --scout T-1 /proj --account a) is silently captured as task-id and the real project-dir is demoted to a passthrough arg, producing a garbled fm-spawn invocation. The documented usage puts flags last, so this only bites out-of-order invocations; rejecting an unknown--*token before both positionals are filled would fail loudly instead.🔧 Fix: fix wait guard bypass, float headroom compares, budget exemption, prereq chmod
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
bash tests/federation/test_account_quota.sh— quota-aware account pick (headroom, tie, absent-provider guards): all passbash tests/federation/test_accounts.sh— account registry resolution/validation, foreign-home refusal: all passbash tests/federation/test_fleet.sh— init, atomic no-overlap claim race, TTL reap, scope routing, handoff, view/status, cross-uid safety: all passbash tests/federation/test_fleet_guards.sh— bare-clone resolution guards: uninitialized-fleet refusal, foreign-fleet non-disclosure (status/view/route/wait), explicit-dir and membership acceptance, budget/models exemption: all 17 passbash tests/federation/test_fleet_ops.sh— operator lifecycle (register/heartbeat/leave/join), quota-floor routing, wait --once wake semantics: all passbash tests/federation/test_quota_surfaces.sh— per-surface quota view, model->surface failover, authed override hook, copilot custom source superseding auth_required: all 12 passbash tests/federation/test_spawn_account.sh— --account spawn composition, api-key fail-closed shim: all passManual e2e transcript: bare-clonefm-fleet.sh statusrefuses with actionable diagnostic; then init → register x2 → queue → route → claim → handoff → status/view against a temp fleet dirManual e2e transcript:fm-fleet.sh quota,models,pick grok|claude|kimi(drained-primary failover incl. float 3.5% headroom),budgetexiting 1 below the 5% floor, with stubbed quota-axi and custom cursor/copilot readersbash -n scripts/fleet-root-prereq.shplus source inspection confirming the review-fix chmod (setgid 2775 dirs, g+rw files); not executed because it requires root and would modify system state outside the worktreedocs/fleet-addon.md:74- Out-of-scope consolidation follow-up: the CLI auth-isolation matrix is stated in full in both docs/fleet-addon.md (Part B) and docs/fleet-quickstart.md (Tier B), and fleet-addon.md's 'Install (drop-in overlay)' and 'Packaging options' sections are pre-merge delivery narrative now that the add-on ships in-repo via PR feat(fleet): federated multi-operator coordination with per-surface quota routing #1103. A follow-up should make one surface the matrix owner (reduce the other to a pointer) and trim the overlay/packaging prose from the maintainer reference.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.