feat(fleet): federated multi-operator coordination with per-surface quota routing

.agents/skills/federation/SKILL.md

@@ -0,0 +1,125 @@
+---
+name: federation
+description: >-
+  Procedure for federated multi-operator coordination. Use when more than one OS
+  operator (each their own first mate, own accounts) shares work through the fleet
+  KB. Owns fleet init, routing, claim/lock, handoff, TTL reap, and cross-uid safety.
+metadata:
+  internal: true
+---
+
+# federation
+
+Multiple operators (e.g. `alice`/`bob`/`carol` — any OS users on the host), each
+running their **own** first mate as themselves, whose firstmate homes may be named
+and located differently, coordinate through one **shared, group-writable,
+git-backed KB** so
+work is routed by domain, never overlaps, and is visible in realtime. This skill
+owns that procedure. The CLI is `bin/fm-fleet.sh` (lib: `bin/fm-fleet-lib.sh`).
+
+## Cross-uid safety (non-negotiable)
+
+Operators share **only** the fleet dir. **Never** read or write another operator's
+private home (`~/.claude`, credentials, their own firstmate home). Every
+mutating fleet function calls `fm_fleet_assert_shared`, which refuses any path
+resolving into a foreign `/home/<other>`. Credentials stay `0700`, read only by
+their owner's own processes. This replaces FirstMate's single-uid filesystem-copy
+propagation, which cannot work across uids.
+
+## Fleet dir resolution
+
+`FM_FLEET_DIR` → `$FM_HOME/config/fleet-dir` → `/opt/agents/fleet`.
+The real shared dir needs the one-time root prereq (`scripts/fleet-root-prereq.sh`): an
+`agents` group + `/opt/agents/fleet` mode `2775` (setgid) + each operator's
+`umask 002`. Until then it runs against a local dev dir (single-uid), which
+exercises every code path.
+
+## KB files (at `$FLEET`)
+
+- `operators.md` — `| operator | scope | home | accounts | status | seen | quota |`;
+  `scope` is a comma list; `status` is `online`/`offline`; `seen` is an ISO8601 UTC
+  heartbeat; `quota` is that operator's self-published `quota-axi` min headroom %
+  (or `-`). An operator counts as available for routing only when `status:online`
+  **and** `seen` is within `FM_FLEET_HEARTBEAT_TTL` (90s) **and** `quota` ≥
+  `FM_FLEET_QUOTA_MIN` (5). Legacy 5-column rows still route (freshness/quota skipped).
+- `backlog.md` — `## Queued / ## Claimed / ## In-flight / ## Done`; item line:
+  `- [id:<ID>] scope:<S> | <desc> | [claimed-by:<op>@<ISO8601>] status:<st>`.
+- `events.log` — append-only TSV `<ISO8601>\t<op>\t<event>\t<id>\t<detail>`.
+- `locks/backlog.lock` — the `flock` target for atomic claims.
+
+## Procedure
+
+0. **Onboard (once per operator, run AS YOURSELF):** `fm-fleet-join.sh <you> <scopes>
+   [accounts]` — verifies shared-dir access, points `config/fleet-dir` at the shared
+   KB, and registers you (`register` upserts your row: `status:online`, `seen:now`,
+   `quota:now`). Idempotent. Refuses a home outside your own `$HOME`.
+1. **Session start:** `fm-fleet.sh reap [ttl]` to requeue stale never-started
+   claims from offline operators (default ttl 86400s; only `status:claimed`,
+   never `status:in-flight`).
+2. **Intake a task:** resolve owner by domain — `fm-fleet.sh route <scope>`
+   (scope-primary: the online operator whose `scope` contains it; on
+   miss/offline/quota-saturation → the `overflow` operator; a human `--operator`
+   override always wins).
+3. **Take work meant for you:** `fm-fleet.sh claim <id> <you>` — atomic under
+   `flock`; returns non-zero if already claimed, so two operators can never grab
+   the same item.
+4. **Give work to its owner:** `fm-fleet.sh handoff <id> <owner>` — reassigns; the
+   owner's first mate then `claim`s it.
+5. **Dispatch:** run the crewmate (fm-spawn) in your own Treehouse worktree under
+   your own account; mark the item in-flight (integration point) and `done` on land.
+6. **Visibility:** `fm-fleet.sh status` (per-operator counts) and
+   `fm-fleet.sh view [--follow]` (the live cross-operator event stream).
+7. **Stay cheap (token economy):** do NOT poll for work with the LLM. Block on
+   `fm-fleet-wait.sh <you>` — bash, 0 tokens — which also heartbeats while waiting
+   and returns only when you have a fresh `status:claimed` item, so the LLM primary
+   wakes only for real work. Details: `docs/fleet-token-economy.md`.
+
+## Notes
+
+- Every KB mutation is git-committed in the shared dir → durable "who did what
+  when" audit; optionally mirror to a private GitHub repo for offsite/cross-box.
+  Heartbeats are the one exception — a transient file write, never committed, so
+  liveness does not bloat the log.
+- Quota-secondary routing is implemented: each operator self-publishes its
+  `quota-axi` min headroom into its `operators.md` `quota` column on heartbeat, and
+  `route` skips any operator below `FM_FLEET_QUOTA_MIN` (no cross-user auth needed).
+  `fm-fleet.sh budget` / `fm_fleet_budget_ok` gates a claim on local headroom.
+  Missing `quota-axi` is fail-open (`quota:-`), so routing falls back to scope alone.
+
+## Per-surface token visibility & model→surface failover
+
+Each CLI/app subscription is its OWN token pool, and one model can be reachable from
+several pools (grok via the `grok` CLI AND via Cursor; kimi3/open models via `cline`).
+Three read-only, 0-token verbs expose and route on this:
+
+- `fm-fleet.sh quota` — every surface's headroom + observability status. Base rows come
+  from `quota-axi` (claude/codex/cursor/copilot/grok/kimi); pluggable scripts in
+  `bin/quota-sources/<surface>.sh` add surfaces quota-axi can't see or attach an authed
+  reader (e.g. `cursor`), each emitting a normalized `{surface,status,headroom,unit,models,note}` object.
+- `fm-fleet.sh models` — for each model family in the model map (gitignored
+  `config/model-surfaces.json` if present, else the shipped default
+  `docs/examples/model-surfaces.json`), the
+  ordered surfaces that can serve it, each with live status + headroom.
+- `fm-fleet.sh pick <family>` — the failover selector (`fm_fleet_pick_surface`): first
+  surface with observable headroom ≥ floor → else a configured-but-unobservable surface
+  (fail-open) → else the first listed. This is "grok from whichever pool has tokens".
+
+Authed readers (server-side usage): some surfaces don't report through quota-axi. The
+mechanism is an **operator-supplied escape hatch**: `config/quota-overrides.json` maps
+`<surface> → a shell command that prints one int 0-100` (percent headroom); the
+`bin/quota-sources/*.sh` scripts run it and emit that as the surface's headroom, which
+**supersedes** the quota-axi / blind row. The command owns all secret handling (read the
+token from a 0600 file, never argv). Empty/missing = blind fail-open (default). Template:
+`docs/examples/quota-overrides.json` (real file gitignored).
+
+- **cursor** — SOLVED with a shipped reader `bin/quota-cursor-usage.sh`: Cursor's native
+  Connect RPC `POST api2.cursor.sh/aiserver.v1.DashboardService/GetCurrentPeriodUsage`
+  (Content-Type: application/json, Connect-Protocol-Version: 1, x-cursor-client-* headers)
+  accepts the CLI's OWN access token from `~/.config/cursor/auth.json` — no browser cookie.
+  headroom = 100 − `planUsage.totalPercentUsed`. Wire: `"cursor": "<repo>/bin/quota-cursor-usage.sh"`.
+- **cline** — intentionally NOT monitored. Its balance is served via an internal local
+  WS Hub (`ws://127.0.0.1:25463/hub`) using a server-derived credential, not the stored
+  WorkOS token (every REST/header variant returns 401). cline stays a usable crewmate
+  harness but is out of quota routing; we let it hit its wall and route open work elsewhere.
+
+Tests: `tests/federation/test_quota_surfaces.sh`.

.agents/skills/multi-account/SKILL.md

@@ -0,0 +1,54 @@
+---
+name: multi-account
+description: >-
+  Procedure for launching crewmates under a chosen provider account with isolated
+  auth, and for rotating across an operator's accounts by quota. Use when one
+  operator holds several accounts of a provider and work must be spread across
+  them without auth bleed.
+metadata:
+  internal: true
+---
+
+# multi-account
+
+One operator, several accounts per provider (each its own auth), selected per
+spawn so quota is spread and credentials never bleed. Registry:
+`config/accounts.json` (gitignored). Libs: `bin/fm-accounts-lib.sh` (registry) +
+`bin/fm-account-env.sh` (isolation). Full reference: `docs/fleet-addon.md`.
+
+## Isolation is verified, never guessed
+Each account declares an `isolation` method that MUST match its harness per
+the matrix in `docs/fleet-addon.md` (enforced by `bin/fm-accounts-lib.sh`):
+- `config-dir-env` — `CLAUDE_CONFIG_DIR` / `CODEX_HOME` / `PI_CODING_AGENT_DIR`
+- `config-dir-flag` — cline `--config <dir>`
+- `api-key-env` — grok `GROK_API_KEY` / cursor `CURSOR_API_KEY`
+
+## Secrets never touch argv or the registry
+api-key accounts store a `key_file` path (a `0600` file in the operator's OWN
+home). The key is read into the child's environment at launch — never onto the
+command line, never into a log, never into git. `config_dir`/`key_file` must live
+under the operator's own home (a foreign `/home/<other>` is refused).
+
+## Procedure
+1. **Prereq (once, on-demand):** `bin/fm-accounts-prereq.sh` to check the CLIs are
+   installed (user-scoped; `install` to add missing ones). Then the operator logs
+   in each account into its own config dir / key_file (auth is user-only).
+2. **Register:** copy `docs/examples/accounts.json` → `config/accounts.json`;
+   one entry per account. Validate: `fm_account_validate <name>`.
+3. **Pick by quota (optional):** `fm_account_pick <harness>` returns the account
+   with the most `quota-axi` headroom (runs quota-axi under each account's
+   isolation; ties → first registered; no quota data → first registered).
+4. **Supervised spawn (config-dir accounts):**
+   `bin/fm-spawn-acct.sh <id> <dir> --account <name> [--model M] [--effort E]`.
+   It composes an isolated launch command and hands it to `fm-spawn`'s raw-launch
+   hatch (no `fm-spawn` edit; no secret on argv).
+5. **Direct isolated launch (api-key accounts, or non-supervised):**
+   `bin/fm-account-exec.sh <name> <cli> [args]` — reads the key into the child's
+   env, then execs.
+
+## Limits (see docs/fleet-addon.md)
+- cursor OAuth mode is not per-spawn isolatable → use API-key mode.
+- api-key accounts can't go through the supervised `--account` path (would put the
+  key on argv) → use `fm-account-exec.sh`.
+- `quota-axi` is per-provider (current auth); real two-account quota
+  discrimination needs each account separately authed with creds quota-axi reads.

.gitignore

@@ -18,3 +18,7 @@ config/x-mode.env
 config/cmux-socket-password
 config/wedge-alarm
 config/herdr-presentation-spaces
+config/fleet-dir
+config/accounts.json
+config/quota-overrides.json
+config/model-surfaces.json

AGENTS.md

@@ -481,6 +481,8 @@ These skills are not captain-invocable; load them only at their precise triggers
 - `fmx-respond` - load on an `x-mention <request_id>` `check:` wake to handle the mention, on an `x-mode-error ...` `check:` wake to report the X-mode configuration blocker, and on any milestone or terminal wake for an X-mode-linked task before posting its completion follow-up; relevant only when X mode is on.
 - `firstmate-codexapp` - load before coordinating a visible Codex Desktop thread, evaluating a Codex App backend request, or reconciling Codex Desktop host-tool smoke evidence for Firstmate work.
 - `firstmate-coding-guidelines` - load before changing firstmate's shared, tracked material, as defined by section 1's list, whether editing directly or briefing a crewmate for a firstmate-repo task.
+- `federation` - load before reading or mutating a shared fleet KB (`bin/fm-fleet.sh` verbs, claim/handoff/routing) when this home is joined to a fleet with other operators.
+- `multi-account` - load before launching a crewmate under a chosen provider account (`bin/fm-spawn-acct.sh` / `bin/fm-account-exec.sh`) or selecting an account by quota headroom.
 
 ## 14. X mode
 

README.md

@@ -190,6 +190,8 @@ Firstmate's skills live in two separate places with different audiences:
 ## Documentation
 
 - [docs/architecture.md](docs/architecture.md) - maintainer architecture for the crew, supervision, worktrees, secondmates, and project modes.
+- [docs/fleet-quickstart.md](docs/fleet-quickstart.md) - **start here for the fleet add-on**: see remaining budget across every AI subscription you own and fail work over to whichever pool still has headroom (no setup), run several accounts for one person, or federate several people on one host.
+- [docs/fleet-addon.md](docs/fleet-addon.md) - reference for the fleet add-on: shared KB layout, claim protocol, routing rules, and the one-time root prerequisite.
 - [docs/configuration.md](docs/configuration.md) - environment variables, `FM_HOME`, runtime backend selection, optional X mode, the files you set, and harness support.
 - [docs/calm.md](docs/calm.md) - current Pi `/calm` behavior and supported presentation limits.
 - [docs/wedge-alarm.md](docs/wedge-alarm.md) - configure the active alert for an away-mode escalation delivery that gets stuck.

bin/fm-account-env.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# fm-account-env.sh — apply a registered account's auth isolation (Phase 4).
+#
+# Consumes fm-accounts-lib.sh. Two consumers, two mechanisms:
+#
+#   * fm-spawn-acct.sh (SUPERVISED paned spawns) -> fm_account_compose_launch
+#     Builds a launch command for fm-spawn's raw-launch escape hatch. Isolation
+#     rides IN the command string (env prefix, or --config flag) so it survives
+#     the tmux/Herdr pane boundary. config-dir methods only: an env PREFIX for a
+#     config dir is not a secret, but an api-key on argv WOULD be — so api-key
+#     accounts are refused here and routed to the exec shim.
+#
+#   * fm-account-exec.sh / direct launches -> fm_account_apply_env
+#     Exports the isolation env in THIS process (incl. api-key read from key_file),
+#     then the caller execs the CLI. The secret lands only in the child's env,
+#     never on argv, never in a log.
+
+_fm_acct_lib() { # source fm-accounts-lib.sh once
+  [ -n "${_FM_ACCT_LIB_LOADED:-}" ] && return 0
+  local d; d="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+  # shellcheck source=bin/fm-accounts-lib.sh disable=SC1091
+  . "$d/fm-accounts-lib.sh"
+  _FM_ACCT_LIB_LOADED=1
+}
+
+# Compose a launch command for fm-spawn's raw-launch escape hatch.
+# args: name [model] [effort]  -> echoes the launch command; nonzero on refusal.
+fm_account_compose_launch() { # name [model] [effort]
+  _fm_acct_lib
+  local name=$1 model=${2:-} effort=${3:-}
+  local line harness iso env flag cdir kfile out
+  fm_account_validate "$name" || return 1
+  line=$(fm_account_resolve "$name")
+  IFS=$'\t' read -r harness iso env flag cdir kfile <<<"$line"
+  case "$iso" in
+    config-dir-env)  out="$env=$cdir $harness" ;;
+    config-dir-flag) out="$harness $flag $cdir" ;;
+    api-key-env)
+      echo "fm-account: '$name' uses api-key isolation; a supervised spawn would put the key on argv. Use fm-account-exec.sh for a direct launch." >&2
+      return 2 ;;
+    *) echo "fm-account: $name unknown isolation '$iso'" >&2; return 1 ;;
+  esac
+  [ -n "$model" ] && out="$out --model $model"
+  if [ -n "$effort" ]; then
+    case "$harness" in
+      claude|codex|pi|cursor-agent) out="$out --effort $effort" ;;
+      grok) out="$out --reasoning-effort $effort" ;;
+      *) echo "fm-account: harness '$harness' has no known effort flag; ignoring --effort $effort" >&2 ;;
+    esac
+  fi
+  printf '%s\n' "$out"
+}
+
+# Apply isolation to THIS process's environment (direct/exec launches).
+# MUST be called directly (NOT inside $(...)) so the exports land in the caller's
+# shell rather than a dead subshell. config-dir-env / api-key-env export the env;
+# config-dir-flag sets FM_ACCT_ARGV_SUFFIX to the "FLAG dir" pair for the caller
+# to splice into argv (a flag cannot be an env). api-key reads key_file (0600,
+# own home); the key is exported, never echoed.
+fm_account_apply_env() { # name  -> exports env; sets FM_ACCT_ARGV_SUFFIX
+  _fm_acct_lib
+  local name=$1 line harness iso env flag cdir kfile key
+  FM_ACCT_ARGV_SUFFIX=""
+  fm_account_validate "$name" || return 1
+  line=$(fm_account_resolve "$name")
+  IFS=$'\t' read -r harness iso env flag cdir kfile <<<"$line"
+  # shellcheck disable=SC2034 # FM_ACCT_ARGV_SUFFIX is read by callers (fm-account-exec.sh) after sourcing.
+  case "$iso" in
+    config-dir-env)  export "$env=$cdir" ;;
+    config-dir-flag) FM_ACCT_ARGV_SUFFIX="$flag $cdir" ;;
+    api-key-env)
+      [ -r "$kfile" ] || { echo "fm-account: key_file unreadable: $kfile" >&2; return 1; }
+      key=$(head -n1 "$kfile"); [ -n "$key" ] || { echo "fm-account: key_file empty: $kfile" >&2; return 1; }
+      export "$env=$key" ;;
+    *) echo "fm-account: $name unknown isolation '$iso'" >&2; return 1 ;;
+  esac
+}

bin/fm-account-exec.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# fm-account-exec.sh — direct (non-supervised) account-isolated launch (Phase 4).
+#
+# Applies an account's auth isolation to THIS process, then execs the CLI. This
+# is the secure path for api-key harnesses (grok/cursor): the key is read from
+# the account's key_file into the child's ENVIRONMENT — never onto argv, never
+# into a log. Also usable for any direct config-dir launch outside a supervised
+# fm-spawn pane.
+#
+# Usage: fm-account-exec.sh <account> <cli> [args...]
+#   e.g. fm-account-exec.sh grok-personal grok -p "summarise this repo"
+#        fm-account-exec.sh claude-alt   claude --model opus
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FM_HOME="${FM_HOME:-$(cd "$SCRIPT_DIR/.." && pwd)}"; export FM_HOME
+# shellcheck source=bin/fm-account-env.sh disable=SC1091
+. "$SCRIPT_DIR/fm-account-env.sh"
+
+acct=${1:?usage: fm-account-exec.sh <account> <cli> [args...]}; shift
+cli=${1:?usage: fm-account-exec.sh <account> <cli> [args...]}; shift || true
+
+# apply_env exports the isolation env (incl. api-key from key_file) into THIS
+# shell — it must NOT run in $(...) or the exports die in the subshell. For
+# config-dir-flag harnesses it sets FM_ACCT_ARGV_SUFFIX ("FLAG dir") to splice
+# into argv.
+fm_account_apply_env "$acct" || exit 1
+# shellcheck disable=SC2086
+exec "$cli" $FM_ACCT_ARGV_SUFFIX "$@"