diff --git a/.agents/skills/federation/SKILL.md b/.agents/skills/federation/SKILL.md new file mode 100644 index 00000000000..8fcaf577b97 --- /dev/null +++ b/.agents/skills/federation/SKILL.md @@ -0,0 +1,125 @@ +--- +name: federation +description: >- + Procedure for federated multi-operator coordination. Use when more than one OS + operator (each their own first mate, own accounts) shares work through the fleet + KB. Owns fleet init, routing, claim/lock, handoff, TTL reap, and cross-uid safety. +metadata: + internal: true +--- + +# federation + +Multiple operators (e.g. `alice`/`bob`/`carol` — any OS users on the host), each +running their **own** first mate as themselves, whose firstmate homes may be named +and located differently, coordinate through one **shared, group-writable, +git-backed KB** so +work is routed by domain, never overlaps, and is visible in realtime. This skill +owns that procedure. The CLI is `bin/fm-fleet.sh` (lib: `bin/fm-fleet-lib.sh`). + +## Cross-uid safety (non-negotiable) + +Operators share **only** the fleet dir. **Never** read or write another operator's +private home (`~/.claude`, credentials, their own firstmate home). Every +mutating fleet function calls `fm_fleet_assert_shared`, which refuses any path +resolving into a foreign `/home/`. Credentials stay `0700`, read only by +their owner's own processes. This replaces FirstMate's single-uid filesystem-copy +propagation, which cannot work across uids. + +## Fleet dir resolution + +`FM_FLEET_DIR` → `$FM_HOME/config/fleet-dir` → `/opt/agents/fleet`. +The real shared dir needs the one-time root prereq (`scripts/fleet-root-prereq.sh`): an +`agents` group + `/opt/agents/fleet` mode `2775` (setgid) + each operator's +`umask 002`. Until then it runs against a local dev dir (single-uid), which +exercises every code path. + +## KB files (at `$FLEET`) + +- `operators.md` — `| operator | scope | home | accounts | status | seen | quota |`; + `scope` is a comma list; `status` is `online`/`offline`; `seen` is an ISO8601 UTC + heartbeat; `quota` is that operator's self-published `quota-axi` min headroom % + (or `-`). An operator counts as available for routing only when `status:online` + **and** `seen` is within `FM_FLEET_HEARTBEAT_TTL` (90s) **and** `quota` ≥ + `FM_FLEET_QUOTA_MIN` (5). Legacy 5-column rows still route (freshness/quota skipped). +- `backlog.md` — `## Queued / ## Claimed / ## In-flight / ## Done`; item line: + `- [id:] scope: | | [claimed-by:@] status:`. +- `events.log` — append-only TSV `\t\t\t\t`. +- `locks/backlog.lock` — the `flock` target for atomic claims. + +## Procedure + +0. **Onboard (once per operator, run AS YOURSELF):** `fm-fleet-join.sh + [accounts]` — verifies shared-dir access, points `config/fleet-dir` at the shared + KB, and registers you (`register` upserts your row: `status:online`, `seen:now`, + `quota:now`). Idempotent. Refuses a home outside your own `$HOME`. +1. **Session start:** `fm-fleet.sh reap [ttl]` to requeue stale never-started + claims from offline operators (default ttl 86400s; only `status:claimed`, + never `status:in-flight`). +2. **Intake a task:** resolve owner by domain — `fm-fleet.sh route ` + (scope-primary: the online operator whose `scope` contains it; on + miss/offline/quota-saturation → the `overflow` operator; a human `--operator` + override always wins). +3. **Take work meant for you:** `fm-fleet.sh claim ` — atomic under + `flock`; returns non-zero if already claimed, so two operators can never grab + the same item. +4. **Give work to its owner:** `fm-fleet.sh handoff ` — reassigns; the + owner's first mate then `claim`s it. +5. **Dispatch:** run the crewmate (fm-spawn) in your own Treehouse worktree under + your own account; mark the item in-flight (integration point) and `done` on land. +6. **Visibility:** `fm-fleet.sh status` (per-operator counts) and + `fm-fleet.sh view [--follow]` (the live cross-operator event stream). +7. **Stay cheap (token economy):** do NOT poll for work with the LLM. Block on + `fm-fleet-wait.sh ` — bash, 0 tokens — which also heartbeats while waiting + and returns only when you have a fresh `status:claimed` item, so the LLM primary + wakes only for real work. Details: `docs/fleet-token-economy.md`. + +## Notes + +- Every KB mutation is git-committed in the shared dir → durable "who did what + when" audit; optionally mirror to a private GitHub repo for offsite/cross-box. + Heartbeats are the one exception — a transient file write, never committed, so + liveness does not bloat the log. +- Quota-secondary routing is implemented: each operator self-publishes its + `quota-axi` min headroom into its `operators.md` `quota` column on heartbeat, and + `route` skips any operator below `FM_FLEET_QUOTA_MIN` (no cross-user auth needed). + `fm-fleet.sh budget` / `fm_fleet_budget_ok` gates a claim on local headroom. + Missing `quota-axi` is fail-open (`quota:-`), so routing falls back to scope alone. + +## Per-surface token visibility & model→surface failover + +Each CLI/app subscription is its OWN token pool, and one model can be reachable from +several pools (grok via the `grok` CLI AND via Cursor; kimi3/open models via `cline`). +Three read-only, 0-token verbs expose and route on this: + +- `fm-fleet.sh quota` — every surface's headroom + observability status. Base rows come + from `quota-axi` (claude/codex/cursor/copilot/grok/kimi); pluggable scripts in + `bin/quota-sources/.sh` add surfaces quota-axi can't see or attach an authed + reader (e.g. `cursor`), each emitting a normalized `{surface,status,headroom,unit,models,note}` object. +- `fm-fleet.sh models` — for each model family in the model map (gitignored + `config/model-surfaces.json` if present, else the shipped default + `docs/examples/model-surfaces.json`), the + ordered surfaces that can serve it, each with live status + headroom. +- `fm-fleet.sh pick ` — the failover selector (`fm_fleet_pick_surface`): first + surface with observable headroom ≥ floor → else a configured-but-unobservable surface + (fail-open) → else the first listed. This is "grok from whichever pool has tokens". + +Authed readers (server-side usage): some surfaces don't report through quota-axi. The +mechanism is an **operator-supplied escape hatch**: `config/quota-overrides.json` maps +` → a shell command that prints one int 0-100` (percent headroom); the +`bin/quota-sources/*.sh` scripts run it and emit that as the surface's headroom, which +**supersedes** the quota-axi / blind row. The command owns all secret handling (read the +token from a 0600 file, never argv). Empty/missing = blind fail-open (default). Template: +`docs/examples/quota-overrides.json` (real file gitignored). + +- **cursor** — SOLVED with a shipped reader `bin/quota-cursor-usage.sh`: Cursor's native + Connect RPC `POST api2.cursor.sh/aiserver.v1.DashboardService/GetCurrentPeriodUsage` + (Content-Type: application/json, Connect-Protocol-Version: 1, x-cursor-client-* headers) + accepts the CLI's OWN access token from `~/.config/cursor/auth.json` — no browser cookie. + headroom = 100 − `planUsage.totalPercentUsed`. Wire: `"cursor": "/bin/quota-cursor-usage.sh"`. +- **cline** — intentionally NOT monitored. Its balance is served via an internal local + WS Hub (`ws://127.0.0.1:25463/hub`) using a server-derived credential, not the stored + WorkOS token (every REST/header variant returns 401). cline stays a usable crewmate + harness but is out of quota routing; we let it hit its wall and route open work elsewhere. + +Tests: `tests/federation/test_quota_surfaces.sh`. diff --git a/.agents/skills/multi-account/SKILL.md b/.agents/skills/multi-account/SKILL.md new file mode 100644 index 00000000000..8e406ce5f89 --- /dev/null +++ b/.agents/skills/multi-account/SKILL.md @@ -0,0 +1,54 @@ +--- +name: multi-account +description: >- + Procedure for launching crewmates under a chosen provider account with isolated + auth, and for rotating across an operator's accounts by quota. Use when one + operator holds several accounts of a provider and work must be spread across + them without auth bleed. +metadata: + internal: true +--- + +# multi-account + +One operator, several accounts per provider (each its own auth), selected per +spawn so quota is spread and credentials never bleed. Registry: +`config/accounts.json` (gitignored). Libs: `bin/fm-accounts-lib.sh` (registry) + +`bin/fm-account-env.sh` (isolation). Full reference: `docs/fleet-addon.md`. + +## Isolation is verified, never guessed +Each account declares an `isolation` method that MUST match its harness per +the matrix in `docs/fleet-addon.md` (enforced by `bin/fm-accounts-lib.sh`): +- `config-dir-env` — `CLAUDE_CONFIG_DIR` / `CODEX_HOME` / `PI_CODING_AGENT_DIR` +- `config-dir-flag` — cline `--config ` +- `api-key-env` — grok `GROK_API_KEY` / cursor `CURSOR_API_KEY` + +## Secrets never touch argv or the registry +api-key accounts store a `key_file` path (a `0600` file in the operator's OWN +home). The key is read into the child's environment at launch — never onto the +command line, never into a log, never into git. `config_dir`/`key_file` must live +under the operator's own home (a foreign `/home/` is refused). + +## Procedure +1. **Prereq (once, on-demand):** `bin/fm-accounts-prereq.sh` to check the CLIs are + installed (user-scoped; `install` to add missing ones). Then the operator logs + in each account into its own config dir / key_file (auth is user-only). +2. **Register:** copy `docs/examples/accounts.json` → `config/accounts.json`; + one entry per account. Validate: `fm_account_validate `. +3. **Pick by quota (optional):** `fm_account_pick ` returns the account + with the most `quota-axi` headroom (runs quota-axi under each account's + isolation; ties → first registered; no quota data → first registered). +4. **Supervised spawn (config-dir accounts):** + `bin/fm-spawn-acct.sh --account [--model M] [--effort E]`. + It composes an isolated launch command and hands it to `fm-spawn`'s raw-launch + hatch (no `fm-spawn` edit; no secret on argv). +5. **Direct isolated launch (api-key accounts, or non-supervised):** + `bin/fm-account-exec.sh [args]` — reads the key into the child's + env, then execs. + +## Limits (see docs/fleet-addon.md) +- cursor OAuth mode is not per-spawn isolatable → use API-key mode. +- api-key accounts can't go through the supervised `--account` path (would put the + key on argv) → use `fm-account-exec.sh`. +- `quota-axi` is per-provider (current auth); real two-account quota + discrimination needs each account separately authed with creds quota-axi reads. diff --git a/.gitignore b/.gitignore index 372af4735f3..1d7a09a177e 100644 --- a/.gitignore +++ b/.gitignore @@ -18,3 +18,7 @@ config/x-mode.env config/cmux-socket-password config/wedge-alarm config/herdr-presentation-spaces +config/fleet-dir +config/accounts.json +config/quota-overrides.json +config/model-surfaces.json diff --git a/AGENTS.md b/AGENTS.md index d85e90b8eaa..509014e4854 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -481,6 +481,8 @@ These skills are not captain-invocable; load them only at their precise triggers - `fmx-respond` - load on an `x-mention ` `check:` wake to handle the mention, on an `x-mode-error ...` `check:` wake to report the X-mode configuration blocker, and on any milestone or terminal wake for an X-mode-linked task before posting its completion follow-up; relevant only when X mode is on. - `firstmate-codexapp` - load before coordinating a visible Codex Desktop thread, evaluating a Codex App backend request, or reconciling Codex Desktop host-tool smoke evidence for Firstmate work. - `firstmate-coding-guidelines` - load before changing firstmate's shared, tracked material, as defined by section 1's list, whether editing directly or briefing a crewmate for a firstmate-repo task. +- `federation` - load before reading or mutating a shared fleet KB (`bin/fm-fleet.sh` verbs, claim/handoff/routing) when this home is joined to a fleet with other operators. +- `multi-account` - load before launching a crewmate under a chosen provider account (`bin/fm-spawn-acct.sh` / `bin/fm-account-exec.sh`) or selecting an account by quota headroom. ## 14. X mode diff --git a/README.md b/README.md index a7f69e39c23..eda6992aa08 100644 --- a/README.md +++ b/README.md @@ -190,6 +190,8 @@ Firstmate's skills live in two separate places with different audiences: ## Documentation - [docs/architecture.md](docs/architecture.md) - maintainer architecture for the crew, supervision, worktrees, secondmates, and project modes. +- [docs/fleet-quickstart.md](docs/fleet-quickstart.md) - **start here for the fleet add-on**: see remaining budget across every AI subscription you own and fail work over to whichever pool still has headroom (no setup), run several accounts for one person, or federate several people on one host. +- [docs/fleet-addon.md](docs/fleet-addon.md) - reference for the fleet add-on: shared KB layout, claim protocol, routing rules, and the one-time root prerequisite. - [docs/configuration.md](docs/configuration.md) - environment variables, `FM_HOME`, runtime backend selection, optional X mode, the files you set, and harness support. - [docs/calm.md](docs/calm.md) - current Pi `/calm` behavior and supported presentation limits. - [docs/wedge-alarm.md](docs/wedge-alarm.md) - configure the active alert for an away-mode escalation delivery that gets stuck. diff --git a/bin/fm-account-env.sh b/bin/fm-account-env.sh new file mode 100755 index 00000000000..d56ee57d677 --- /dev/null +++ b/bin/fm-account-env.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# fm-account-env.sh — apply a registered account's auth isolation (Phase 4). +# +# Consumes fm-accounts-lib.sh. Two consumers, two mechanisms: +# +# * fm-spawn-acct.sh (SUPERVISED paned spawns) -> fm_account_compose_launch +# Builds a launch command for fm-spawn's raw-launch escape hatch. Isolation +# rides IN the command string (env prefix, or --config flag) so it survives +# the tmux/Herdr pane boundary. config-dir methods only: an env PREFIX for a +# config dir is not a secret, but an api-key on argv WOULD be — so api-key +# accounts are refused here and routed to the exec shim. +# +# * fm-account-exec.sh / direct launches -> fm_account_apply_env +# Exports the isolation env in THIS process (incl. api-key read from key_file), +# then the caller execs the CLI. The secret lands only in the child's env, +# never on argv, never in a log. + +_fm_acct_lib() { # source fm-accounts-lib.sh once + [ -n "${_FM_ACCT_LIB_LOADED:-}" ] && return 0 + local d; d="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + # shellcheck source=bin/fm-accounts-lib.sh disable=SC1091 + . "$d/fm-accounts-lib.sh" + _FM_ACCT_LIB_LOADED=1 +} + +# Compose a launch command for fm-spawn's raw-launch escape hatch. +# args: name [model] [effort] -> echoes the launch command; nonzero on refusal. +fm_account_compose_launch() { # name [model] [effort] + _fm_acct_lib + local name=$1 model=${2:-} effort=${3:-} + local line harness iso env flag cdir kfile out + fm_account_validate "$name" || return 1 + line=$(fm_account_resolve "$name") + IFS=$'\t' read -r harness iso env flag cdir kfile <<<"$line" + case "$iso" in + config-dir-env) out="$env=$cdir $harness" ;; + config-dir-flag) out="$harness $flag $cdir" ;; + api-key-env) + echo "fm-account: '$name' uses api-key isolation; a supervised spawn would put the key on argv. Use fm-account-exec.sh for a direct launch." >&2 + return 2 ;; + *) echo "fm-account: $name unknown isolation '$iso'" >&2; return 1 ;; + esac + [ -n "$model" ] && out="$out --model $model" + if [ -n "$effort" ]; then + case "$harness" in + claude|codex|pi|cursor-agent) out="$out --effort $effort" ;; + grok) out="$out --reasoning-effort $effort" ;; + *) echo "fm-account: harness '$harness' has no known effort flag; ignoring --effort $effort" >&2 ;; + esac + fi + printf '%s\n' "$out" +} + +# Apply isolation to THIS process's environment (direct/exec launches). +# MUST be called directly (NOT inside $(...)) so the exports land in the caller's +# shell rather than a dead subshell. config-dir-env / api-key-env export the env; +# config-dir-flag sets FM_ACCT_ARGV_SUFFIX to the "FLAG dir" pair for the caller +# to splice into argv (a flag cannot be an env). api-key reads key_file (0600, +# own home); the key is exported, never echoed. +fm_account_apply_env() { # name -> exports env; sets FM_ACCT_ARGV_SUFFIX + _fm_acct_lib + local name=$1 line harness iso env flag cdir kfile key + FM_ACCT_ARGV_SUFFIX="" + fm_account_validate "$name" || return 1 + line=$(fm_account_resolve "$name") + IFS=$'\t' read -r harness iso env flag cdir kfile <<<"$line" + # shellcheck disable=SC2034 # FM_ACCT_ARGV_SUFFIX is read by callers (fm-account-exec.sh) after sourcing. + case "$iso" in + config-dir-env) export "$env=$cdir" ;; + config-dir-flag) FM_ACCT_ARGV_SUFFIX="$flag $cdir" ;; + api-key-env) + [ -r "$kfile" ] || { echo "fm-account: key_file unreadable: $kfile" >&2; return 1; } + key=$(head -n1 "$kfile"); [ -n "$key" ] || { echo "fm-account: key_file empty: $kfile" >&2; return 1; } + export "$env=$key" ;; + *) echo "fm-account: $name unknown isolation '$iso'" >&2; return 1 ;; + esac +} diff --git a/bin/fm-account-exec.sh b/bin/fm-account-exec.sh new file mode 100755 index 00000000000..a293f871fab --- /dev/null +++ b/bin/fm-account-exec.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# fm-account-exec.sh — direct (non-supervised) account-isolated launch (Phase 4). +# +# Applies an account's auth isolation to THIS process, then execs the CLI. This +# is the secure path for api-key harnesses (grok/cursor): the key is read from +# the account's key_file into the child's ENVIRONMENT — never onto argv, never +# into a log. Also usable for any direct config-dir launch outside a supervised +# fm-spawn pane. +# +# Usage: fm-account-exec.sh [args...] +# e.g. fm-account-exec.sh grok-personal grok -p "summarise this repo" +# fm-account-exec.sh claude-alt claude --model opus +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_HOME="${FM_HOME:-$(cd "$SCRIPT_DIR/.." && pwd)}"; export FM_HOME +# shellcheck source=bin/fm-account-env.sh disable=SC1091 +. "$SCRIPT_DIR/fm-account-env.sh" + +acct=${1:?usage: fm-account-exec.sh [args...]}; shift +cli=${1:?usage: fm-account-exec.sh [args...]}; shift || true + +# apply_env exports the isolation env (incl. api-key from key_file) into THIS +# shell — it must NOT run in $(...) or the exports die in the subshell. For +# config-dir-flag harnesses it sets FM_ACCT_ARGV_SUFFIX ("FLAG dir") to splice +# into argv. +fm_account_apply_env "$acct" || exit 1 +# shellcheck disable=SC2086 +exec "$cli" $FM_ACCT_ARGV_SUFFIX "$@" diff --git a/bin/fm-accounts-lib.sh b/bin/fm-accounts-lib.sh new file mode 100644 index 00000000000..858a2622d80 --- /dev/null +++ b/bin/fm-accounts-lib.sh @@ -0,0 +1,167 @@ +#!/usr/bin/env bash +# fm-accounts-lib.sh — per-operator multi-account registry (Phase 4). +# +# Reads config/accounts.json; resolves + validates accounts against the verified +# per-CLI config-dir / auth-isolation matrix (adapters/config-dir-matrix.md). +# +# Three isolation methods (verified 2026-07-26): +# config-dir-env set = for the child only (claude/codex/pi) +# config-dir-flag pass in argv (cline) +# api-key-env set = for the child only (grok/cursor-agent) +# +# Secrets never live here. api-key accounts store a key_file path (0600, in the +# operator's OWN home); fm-spawn reads the key at launch — it is never printed, +# never committed, never placed on argv. +# +# accounts.json schema (one entry per account name): +# { "": { "provider":"...", "harness":"...", "isolation":"...", +# "env":""|null, "flag":""|null, +# "config_dir":""|null, "key_file":""|null, +# "scopes":["..."] } } + +# Verified harness -> expected "\t" (matrix source of truth). +fm_account_expect() { # harness + case "$1" in + claude) printf 'config-dir-env\tCLAUDE_CONFIG_DIR\n' ;; + codex) printf 'config-dir-env\tCODEX_HOME\n' ;; + pi) printf 'config-dir-env\tPI_CODING_AGENT_DIR\n' ;; + cline) printf 'config-dir-flag\t--config\n' ;; + grok) printf 'api-key-env\tGROK_API_KEY\n' ;; + cursor-agent) printf 'api-key-env\tCURSOR_API_KEY\n' ;; + *) return 1 ;; + esac +} + +fm_accounts_file() { + printf '%s\n' "${FM_ACCOUNTS_FILE:-${FM_HOME:-.}/config/accounts.json}" +} + +# Refuse a path resolving into ANOTHER operator's home (cross-uid safety). Own +# home, /tmp, /opt are allowed. Empty path => allowed (method may not use one). +fm_account_assert_safe_path() { # path + local p rp owner me; p=${1:-} + [ -n "$p" ] && [ "$p" != "-" ] || return 0 + rp=$(realpath -m "$p"); me=$(id -un) + case "$rp" in + /home/*) owner=${rp#/home/}; owner=${owner%%/*} + if [ "$owner" != "$me" ]; then + echo "fm-accounts: refusing foreign-home path: $rp" >&2; return 1 + fi ;; + esac + return 0 +} + +# resolve: print TSV harnessisolationenvflagconfig_dirkey_file +# missing fields -> "-". Exit 1 if the registry or account is absent. +fm_account_resolve() { # name + local name=$1 f; f=$(fm_accounts_file) + [ -f "$f" ] || { echo "fm-accounts: no registry at $f" >&2; return 1; } + jq -e --arg n "$name" 'has($n)' "$f" >/dev/null 2>&1 \ + || { echo "fm-accounts: unknown account: $name" >&2; return 1; } + jq -r --arg n "$name" ' + .[$n] | [ + (.harness // "-"), (.isolation // "-"), + (.env // "-"), (.flag // "-"), + (.config_dir // "-"), (.key_file // "-") + ] | @tsv' "$f" +} + +# validate: harness known; isolation matches the harness's expected method + +# env/flag; required method fields present; paths cross-uid-safe. Exit 0/1. +fm_account_validate() { # name + local name=$1 line harness iso env flag cdir kfile exp exp_iso exp_ef + line=$(fm_account_resolve "$name") || return 1 + IFS=$'\t' read -r harness iso env flag cdir kfile <<<"$line" + exp=$(fm_account_expect "$harness") \ + || { echo "fm-accounts: unknown harness: $harness" >&2; return 1; } + exp_iso=$(printf '%s' "$exp" | cut -f1) + exp_ef=$(printf '%s' "$exp" | cut -f2) + [ "$iso" = "$exp_iso" ] \ + || { echo "fm-accounts: $name isolation '$iso' != expected '$exp_iso' for $harness" >&2; return 1; } + case "$iso" in + config-dir-env) + [ "$env" = "$exp_ef" ] || { echo "fm-accounts: $name env '$env' != '$exp_ef'" >&2; return 1; } + [ "$cdir" != "-" ] || { echo "fm-accounts: $name missing config_dir" >&2; return 1; } + fm_account_assert_safe_path "$cdir" || return 1 ;; + config-dir-flag) + [ "$flag" = "$exp_ef" ] || { echo "fm-accounts: $name flag '$flag' != '$exp_ef'" >&2; return 1; } + [ "$cdir" != "-" ] || { echo "fm-accounts: $name missing config_dir" >&2; return 1; } + fm_account_assert_safe_path "$cdir" || return 1 ;; + api-key-env) + [ "$env" = "$exp_ef" ] || { echo "fm-accounts: $name env '$env' != '$exp_ef'" >&2; return 1; } + [ "$kfile" != "-" ] || { echo "fm-accounts: $name missing key_file" >&2; return 1; } + fm_account_assert_safe_path "$kfile" || return 1 ;; + *) echo "fm-accounts: $name unknown isolation '$iso'" >&2; return 1 ;; + esac + return 0 +} + +fm_account_list() { # -> account names, one per line + local f; f=$(fm_accounts_file); [ -f "$f" ] || return 0 + jq -r 'keys[]' "$f" +} + +fm_account_list_by_harness() { # harness -> matching account names (file order) + local h=$1 f; f=$(fm_accounts_file); [ -f "$f" ] || return 0 + jq -r --arg h "$h" 'to_entries[] | select(.value.harness==$h) | .key' "$f" +} + +# --- quota-aware account selection (Phase 4, Task 12) ------------------------- +# +# quota-axi reports headroom PER PROVIDER for the CURRENTLY-authenticated account +# (oauth/keychain) — it cannot see several accounts at once. So per-account +# headroom is obtained by running quota-axi UNDER each account's isolation (the +# same env/flag the spawn uses). The binding constraint for an account is the +# minimum percentRemaining across its windows. Pick the account with the most +# binding headroom; ties -> first registered. Guards: unsupported provider or +# quota-axi absent -> first registered (+ a note on stderr). + +# harness -> quota-axi --provider value; nonzero if quota-axi has no coverage. +fm_account_quota_provider() { # harness + case "$1" in + claude) echo claude ;; + codex) echo codex ;; + grok) echo grok ;; + cursor-agent) echo cursor ;; + kimi) echo kimi ;; + *) return 1 ;; # pi, cline: not covered by quota-axi + esac +} + +# Run quota-axi under one account's isolation; echo its min percentRemaining. +_fm_account_headroom() { # iso env cdir kfile qbin prov + local iso=$1 env=$2 cdir=$3 kfile=$4 qbin=$5 prov=$6 out key + case "$iso" in + config-dir-env) out=$(env "$env=$cdir" "$qbin" --provider "$prov" --json 2>/dev/null) ;; + config-dir-flag) out=$("$qbin" --provider "$prov" --json 2>/dev/null) ;; + api-key-env) + [ -r "$kfile" ] || return 0 + key=$(head -n1 "$kfile"); [ -n "$key" ] || return 0 + out=$(env "$env=$key" "$qbin" --provider "$prov" --json 2>/dev/null) ;; + *) return 0 ;; + esac + printf '%s' "$out" | jq -r --arg p "$prov" ' + [.providers[] | select(.provider==$p) | .windows[].percentRemaining] | min // empty' 2>/dev/null +} + +# fm_account_pick(harness) -> the registered account with the most headroom. +fm_account_pick() { # harness + local harness=$1 prov qbin best="" best_hr=-1 acct hr line _h iso env flag cdir kfile + local -a accts + mapfile -t accts < <(fm_account_list_by_harness "$harness") + [ "${#accts[@]}" -gt 0 ] || { echo "fm-account: no accounts for harness '$harness'" >&2; return 1; } + if [ "${#accts[@]}" -eq 1 ]; then printf '%s\n' "${accts[0]}"; return 0; fi + prov=$(fm_account_quota_provider "$harness") \ + || { printf '%s\n' "${accts[0]}"; echo "fm-account: no quota provider for '$harness'; picked first (${accts[0]})" >&2; return 0; } + qbin="${QUOTA_AXI_BIN:-quota-axi}" + command -v "$qbin" >/dev/null 2>&1 \ + || { printf '%s\n' "${accts[0]}"; echo "fm-account: quota-axi absent; picked first (${accts[0]})" >&2; return 0; } + for acct in "${accts[@]}"; do + line=$(fm_account_resolve "$acct") || continue + IFS=$'\t' read -r _h iso env flag cdir kfile <<<"$line" + hr=$(_fm_account_headroom "$iso" "$env" "$cdir" "$kfile" "$qbin" "$prov") + [ -n "$hr" ] || hr=-1 + if awk -v a="$hr" -v b="$best_hr" 'BEGIN{exit !((a+0)>(b+0))}'; then best_hr=$hr; best=$acct; fi + done + [ -n "$best" ] && printf '%s\n' "$best" || printf '%s\n' "${accts[0]}" +} diff --git a/bin/fm-accounts-prereq.sh b/bin/fm-accounts-prereq.sh new file mode 100755 index 00000000000..d21b216533b --- /dev/null +++ b/bin/fm-accounts-prereq.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# fm-accounts-prereq.sh — ensure the LLM CLIs multi-account needs are installed +# (Phase 4 add-on). USER-SCOPED, no sudo. Detect by default; install on request. +# +# fm-accounts-prereq.sh # detect: installed / MISSING + install cmd +# fm-accounts-prereq.sh install # install every MISSING harness +# fm-accounts-prereq.sh install cursor-agent # install specific harness(es) +# fm-accounts-prereq.sh install --yes cursor-agent # skip the curl|sh prompt (reviewed) +# +# pi is system-managed (/usr/bin) -> DETECT ONLY, never installed here. +# Install commands are user-scoped (npm prefix must be a user dir; cursor's +# installer writes to ~/.local). Review any remote-script install before running. +set -euo pipefail + +# harness -> install command (verified 2026-07-26). nonzero => not installable here. +fm_prereq_cmd() { # harness + case "$1" in + claude) echo 'npm install -g @anthropic-ai/claude-code' ;; + codex) echo 'npm install -g @openai/codex' ;; + grok) echo 'npm install -g @vibe-kit/grok-cli' ;; + cline) echo 'npm install -g cline' ;; + cursor-agent) echo 'curl https://cursor.com/install -fsS | bash' ;; + pi) return 1 ;; # system-managed; install via Pi, not here + *) return 1 ;; + esac +} + +HARNESSES="claude codex pi grok cline cursor-agent" + +detect() { + printf '%-14s %-10s %s\n' HARNESS STATUS DETAIL + local h + for h in $HARNESSES; do + if command -v "$h" >/dev/null 2>&1; then + printf '%-14s %-10s %s\n' "$h" "installed" "$(command -v "$h") ($("$h" --version 2>/dev/null | head -n1))" + elif [ "$h" = pi ]; then + printf '%-14s %-10s %s\n' "$h" "MISSING" "system-managed (install via Pi; not handled here)" + else + printf '%-14s %-10s %s\n' "$h" "MISSING" "install: $(fm_prereq_cmd "$h")" + fi + done +} + +install_one() { # harness yes + local h=$1 yes=$2 cmd ans + if command -v "$h" >/dev/null 2>&1; then echo "[skip] $h already installed"; return 0; fi + if [ "$h" = pi ]; then echo "[skip] pi is system-managed; install it via Pi, not here"; return 0; fi + cmd=$(fm_prereq_cmd "$h") || { echo "[err] no install command for '$h'" >&2; return 1; } + case "$cmd" in + *'| bash'|*'| sh') + if [ "$yes" != 1 ]; then + echo "[review] $h installs via a remote script:"; echo " $cmd" + printf " run it? [y/N] "; read -r ans || ans=n + [ "$ans" = y ] || [ "$ans" = Y ] || { echo "[skip] $h (declined)"; return 0; } + fi ;; + esac + echo "[install] $h: $cmd"; eval "$cmd" +} + +case "${1:-detect}" in + detect|"") detect ;; + install) + shift; yes=0; targets=() + for a in "$@"; do case "$a" in --yes|-y) yes=1 ;; *) targets+=("$a") ;; esac; done + if [ "${#targets[@]}" -eq 0 ]; then + for h in $HARNESSES; do command -v "$h" >/dev/null 2>&1 || targets+=("$h"); done + fi + if [ "${#targets[@]}" -eq 0 ]; then echo "all harnesses present; nothing to install"; exit 0; fi + for h in "${targets[@]}"; do install_one "$h" "$yes"; done ;; + *) echo "usage: fm-accounts-prereq.sh [detect|install [--yes] [harness...]]" >&2; exit 1 ;; +esac diff --git a/bin/fm-fleet-join.sh b/bin/fm-fleet-join.sh new file mode 100755 index 00000000000..bc99c4cd92f --- /dev/null +++ b/bin/fm-fleet-join.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# fm-fleet-join.sh — self-onboard THIS operator into the shared fleet (run AS YOURSELF). +# +# Points this deployment's config at the shared KB, verifies cross-uid-safe access, +# and registers you as an operator. Idempotent. It only ever writes YOUR OWN +# $FM_HOME/config and the group-writable shared KB — never another operator's home. +# The one-time root prereq (group `agents` + a group-writable shared dir, see +# docs/federation.md / ROOT-PREREQ) must already be done and the fleet `init`'d. +# +# Usage: fm-fleet-join.sh [accounts-csv] +# e.g. fm-fleet-join.sh adi backend,infra,deploy claude-default,codex-default +set -uo pipefail +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_HOME="${FM_HOME:-$(cd "$SCRIPT_DIR/.." && pwd)}" +# shellcheck source=bin/fm-fleet-lib.sh disable=SC1091 +. "$SCRIPT_DIR/fm-fleet-lib.sh" + +op=${1:-}; scopes=${2:-}; accounts=${3:-} +[ -n "$op" ] && [ -n "$scopes" ] || { echo "usage: fm-fleet-join.sh [accounts-csv]" >&2; exit 3; } + +DIR=$(fm_fleet_dir) +fm_fleet_assert_shared "$DIR" || exit 1 +if [ ! -f "$DIR/operators.md" ]; then + echo "error: fleet not initialized at $DIR." >&2 + echo " Once the shared group-writable dir exists (root prereq), run: fm-fleet.sh init" >&2 + exit 1 +fi +# Writability probe: prove group membership + 2775 perms before registering. +probe="$DIR/.join-probe.$$" +if ! ( : > "$probe" ) 2>/dev/null; then + echo "error: $DIR is not writable by $(id -un). Check 'agents' group membership and mode 2775 (see the root prereq)." >&2 + exit 1 +fi +rm -f "$probe" 2>/dev/null || true + +# Point THIS deployment at the shared fleet (own home only). +mkdir -p "$FM_HOME/config" +printf '%s\n' "$DIR" > "$FM_HOME/config/fleet-dir" + +# Register self (home = own $FM_HOME; the lib refuses a foreign home). +fm_fleet_register "$DIR" "$op" "$scopes" "$FM_HOME" "$accounts" || exit 1 + +echo "joined fleet at $DIR as '$op' (scopes: $scopes${accounts:+, accounts: $accounts})" +echo "next:" +echo " • wait for work (bash, 0 LLM tokens): $SCRIPT_DIR/fm-fleet-wait.sh $op" +echo " • or keep alive on a timer: watch -n60 '$SCRIPT_DIR/fm-fleet.sh heartbeat $op'" +echo " • see the fleet: $SCRIPT_DIR/fm-fleet.sh status" diff --git a/bin/fm-fleet-lib.sh b/bin/fm-fleet-lib.sh new file mode 100755 index 00000000000..126fb4288da --- /dev/null +++ b/bin/fm-fleet-lib.sh @@ -0,0 +1,559 @@ +#!/usr/bin/env bash +# fm-fleet-lib.sh — FirstMate federated multi-operator KB library. +# +# Cross-uid-safe coordination through a SHARED group-writable git-backed dir only. +# Operators never write each other's private homes; they share this KB and use +# flock advisory locks on the backlog for atomic, no-overlap claims. +# +# KB layout ($dir): +# operators.md md table: | operator | scope | home | accounts | status | seen | quota | +# projects.md md table: | project | owner | path | +# backlog.md sections ## Queued / ## Claimed / ## In-flight / ## Done +# item line: - [id:] scope: | | [claimed-by:@] status: +# events.log append-only TSV: \t\t\t\t +# locks/ flock targets (backlog.lock) +# +# Every mutating function takes the backlog lock and asserts the target is a +# shared/own dir (never a foreign /home). + +fm_fleet_now() { date -u +%Y-%m-%dT%H:%M:%SZ; } + +# Built-in last-resort fleet dir. Only reached when neither FM_FLEET_DIR nor +# $FM_HOME/config/fleet-dir is set. It is a *convention*, not a guarantee: on a +# shared host it may already belong to another team, so fm_fleet_assert_initialized +# tells the operator exactly which dir was chosen and how it was chosen. +FM_FLEET_DEFAULT_DIR=${FM_FLEET_DEFAULT_DIR:-/opt/agents/fleet} + +fm_fleet_dir() { + local d="${FM_FLEET_DIR:-}" + if [ -z "$d" ] && [ -n "${FM_HOME:-}" ] && [ -f "$FM_HOME/config/fleet-dir" ]; then + d=$(head -n1 "$FM_HOME/config/fleet-dir") + fi + [ -n "$d" ] || d=$FM_FLEET_DEFAULT_DIR + printf '%s\n' "$d" +} + +# How the dir was chosen: env|config|default. Deliberately a FUNCTION, not a global +# set inside fm_fleet_dir: callers do `DIR=$(fm_fleet_dir)`, and a variable assigned +# inside command substitution dies with the subshell — a global here would silently +# read as empty and any guard keyed on it would never fire. +fm_fleet_dir_source() { + if [ -n "${FM_FLEET_DIR:-}" ]; then printf 'env\n' + elif [ -n "${FM_HOME:-}" ] && [ -f "$FM_HOME/config/fleet-dir" ]; then printf 'config\n' + else printf 'default\n'; fi +} + +# Guard for every verb that READS an existing fleet. Without this, an uninitialized +# or wrong dir surfaces as `awk: fatal: cannot open .../operators.md` with exit 0 — +# a raw internal error that also *looks* like success to a caller. Fail loudly with +# the dir, how it was chosen, and the one command that fixes it. +fm_fleet_assert_initialized() { # dir + local dir=$1 how; how=$(fm_fleet_dir_source) + if [ -d "$dir" ] && [ -f "$dir/operators.md" ]; then return 0; fi + + { + printf 'fm-fleet: no initialized fleet at %s\n' "$dir" + case "$how" in + env) printf ' (chosen by FM_FLEET_DIR)\n' ;; + config) printf ' (chosen by %s/config/fleet-dir)\n' "${FM_HOME:-\$FM_HOME}" ;; + default) printf ' (nothing configured, so the built-in default %s was used)\n' "$FM_FLEET_DEFAULT_DIR" ;; + esac + if [ ! -d "$dir" ]; then + printf ' the directory does not exist.\n' + else + printf ' the directory exists but has no operators.md, so it is not a fleet.\n' + fi + printf '\nPick one:\n' + printf ' solo / trying it out FM_FLEET_DIR=~/.firstmate-fleet bin/fm-fleet.sh init\n' + printf ' shared, multi-operator sudo bash scripts/fleet-root-prereq.sh # then: bin/fm-fleet.sh init\n' + printf ' already have one export FM_FLEET_DIR=/path/to/fleet (or write it to %s/config/fleet-dir)\n' "${FM_HOME:-\$FM_HOME}" + printf '\nSee docs/fleet-quickstart.md.\n' + } >&2 + return 1 +} + +# Refuse to silently attach to a fleet the operator never chose. +# +# The built-in default is a shared, conventional path. On a multi-tenant host it may +# already be a DIFFERENT team's fleet, and those dirs are group-writable/world-readable +# by design — so a bare clone that configured nothing could read another team's +# operator table and event log without ever asking. Membership is the opt-in signal: +# if you are already an operator in that fleet it is yours, otherwise say so +# explicitly. Only applies when the dir came from the built-in default; an operator +# who set FM_FLEET_DIR or config/fleet-dir has already chosen. +fm_fleet_assert_owned() { # dir + local dir=$1 me + [ "$(fm_fleet_dir_source)" = default ] || return 0 + [ -z "${FM_FLEET_ACCEPT_DEFAULT:-}" ] || return 0 + me=$(id -un) + grep -qE "^\| *${me} *\|" "$dir/operators.md" 2>/dev/null && return 0 + + { + printf 'fm-fleet: %s is an existing fleet, but you are not one of its operators\n' "$dir" + printf ' and you have not chosen this fleet — it is only the built-in default.\n\n' + printf ' On a shared host that path may belong to another team. Refusing to read it.\n\n' + printf 'If it IS yours:\n' + printf ' bin/fm-fleet-join.sh %s # become an operator\n' "$me" + printf ' export FM_FLEET_ACCEPT_DEFAULT=1 # or just acknowledge the default\n\n' + printf 'If it is NOT yours, choose your own:\n' + printf ' FM_FLEET_DIR=~/.firstmate-fleet bin/fm-fleet.sh init\n\n' + printf 'See docs/fleet-quickstart.md.\n' + } >&2 + return 1 +} + +# The one guard every entry point that CONSUMES an existing fleet must pass: +# initialized AND chosen/owned. fm-fleet.sh, fm-fleet-wait.sh, and any future +# reader call this right after resolving the dir, so no entry point can reach a +# fleet the operator never set up or never opted into. +fm_fleet_assert_usable() { # dir + fm_fleet_assert_initialized "$1" && fm_fleet_assert_owned "$1" +} + +# Refuse any fleet dir that resolves into ANOTHER operator's home. Own home (dev +# test dir) and /opt/... shared dirs are allowed. +fm_fleet_assert_shared() { + local dir rp owner me; dir=$1 + rp=$(realpath -m "$dir") + me=$(id -un) + case "$rp" in + /home/*) + owner=${rp#/home/}; owner=${owner%%/*} + if [ "$owner" != "$me" ]; then + echo "fm-fleet: refusing to touch another operator's home: $rp" >&2 + return 1 + fi + ;; + esac + return 0 +} + +fm_fleet_event() { # dir operator event id detail + local dir=$1 op=$2 ev=$3 id=$4 detail=${5:-} + printf '%s\t%s\t%s\t%s\t%s\n' "$(fm_fleet_now)" "$op" "$ev" "$id" "$detail" >> "$dir/events.log" +} + +fm_fleet_commit() { # dir message + local dir=$1 msg=$2 + git -C "$dir" add -A >/dev/null 2>&1 || return 0 + git -C "$dir" commit -q -m "$msg" >/dev/null 2>&1 || true +} + +fm_fleet_init() { + local dir=$1 + fm_fleet_assert_shared "$dir" || return 1 + mkdir -p "$dir/locks" + git -C "$dir" rev-parse --git-dir >/dev/null 2>&1 || git -C "$dir" init -q + [ -f "$dir/operators.md" ] || printf '# Fleet operators\n\n| operator | scope | home | accounts | status | seen | quota |\n|---|---|---|---|---|---|---|\n' > "$dir/operators.md" + [ -f "$dir/projects.md" ] || printf '# Fleet projects\n\n| project | owner | path |\n|---|---|---|\n' > "$dir/projects.md" + [ -f "$dir/backlog.md" ] || printf '# Fleet backlog\n\n## Queued\n\n## Claimed\n\n## In-flight\n\n## Done\n' > "$dir/backlog.md" + [ -f "$dir/events.log" ] || : > "$dir/events.log" + fm_fleet_commit "$dir" "fleet: init" +} + +# --- backlog mutation (all under flock) --------------------------------------- + +# Open fd 9 on the backlog lock and block until held. Caller runs fm_fleet_unlock +# when done. Returns non-zero if the dir is unsafe. +fm_fleet_lock() { # dir + local dir=$1 + fm_fleet_assert_shared "$dir" || return 1 + mkdir -p "$dir/locks" + exec 9>"$dir/locks/backlog.lock" || return 1 + flock 9 +} +fm_fleet_unlock() { flock -u 9 2>/dev/null || true; } + +fm_fleet_queue() { # dir id scope desc + local dir=$1 id=$2 scope=$3 desc=$4 + fm_fleet_lock "$dir" || return 1 + awk -v line="- [id:$id] scope:$scope | $desc | status:queued" ' + { print } + /^## Queued$/ { print ""; print line } + ' "$dir/backlog.md" > "$dir/backlog.md.tmp" && mv "$dir/backlog.md.tmp" "$dir/backlog.md" + fm_fleet_event "$dir" "-" queue "$id" "scope:$scope" + fm_fleet_commit "$dir" "fleet: queue $id" + fm_fleet_unlock +} + +# Move a queued item to Claimed, stamp claimed-by + status:claimed. Returns 0 on +# win, 1 if the item is not currently queued (already claimed / absent). +fm_fleet_claim() { # dir id operator + local dir=$1 id=$2 op=$3 ts rc=1 + ts=$(fm_fleet_now) + fm_fleet_lock "$dir" || return 1 + if grep -q "\[id:$id\].*status:queued" "$dir/backlog.md"; then + awk -v id="$id" -v op="$op" -v ts="$ts" ' + $0 ~ ("\\[id:" id "\\].*status:queued") { + sub(/status:queued/, "claimed-by:" op "@" ts " status:claimed"); held=$0; next + } + /^## Claimed$/ { print; if (held != "") { print ""; print held; held="" } ; next } + { print } + ' "$dir/backlog.md" > "$dir/backlog.md.tmp" && mv "$dir/backlog.md.tmp" "$dir/backlog.md" + fm_fleet_event "$dir" "$op" claim "$id" "" + fm_fleet_commit "$dir" "fleet: claim $id by $op" + rc=0 + fi + fm_fleet_unlock + return $rc +} + +# Reassign an item to another operator (handoff): stamp claimed-by:, keep it +# in Claimed. Returns 0 if the item exists. +fm_fleet_handoff() { # dir id to_operator + local dir=$1 id=$2 to=$3 ts rc=1 + ts=$(fm_fleet_now) + fm_fleet_lock "$dir" || return 1 + if grep -q "\[id:$id\]" "$dir/backlog.md"; then + awk -v id="$id" -v to="$to" -v ts="$ts" ' + $0 ~ ("\\[id:" id "\\]") { + if ($0 ~ /claimed-by:[^ ]+/) sub(/claimed-by:[^ ]+/, "claimed-by:" to "@" ts) + else sub(/status:/, "claimed-by:" to "@" ts " status:") + print; next + } + { print } + ' "$dir/backlog.md" > "$dir/backlog.md.tmp" && mv "$dir/backlog.md.tmp" "$dir/backlog.md" + fm_fleet_event "$dir" "$to" handoff "$id" "assigned" + fm_fleet_commit "$dir" "fleet: handoff $id to $to" + rc=0 + fi + fm_fleet_unlock + return $rc +} + +# Requeue stale claims: items still status:claimed whose claimed-by:@ is older +# than ttl seconds go back to Queued (never-started work from an offline operator). +# status:in-flight items are left alone. +fm_fleet_reap() { # dir ttl_seconds + local dir=$1 ttl=${2:-86400} now + now=$(date -u +%s) + fm_fleet_lock "$dir" || return 1 + # Buffered two-pass: collect stale claimed lines (removing them in place), + # then re-emit and insert the requeued copies under ## Queued. + awk -v ttl="$ttl" -v now="$now" ' + function epoch(iso, c,e) { c="date -u -d \"" iso "\" +%s 2>/dev/null"; c|getline e; close(c); return e } + { lines[NR]=$0 + if ($0 ~ /status:claimed/ && $0 ~ /claimed-by:[^@]+@[0-9TZ:-]+/) { + match($0, /@[0-9TZ:-]+/); iso=substr($0, RSTART+1, RLENGTH-1) + if (now - epoch(iso) > ttl) { + remove[NR]=1 + r=$0; gsub(/claimed-by:[^ ]+ /, "", r); sub(/status:claimed/, "status:queued", r) + rn++; req[rn]=r + } + } + } + END { + for (i=1;i<=NR;i++) { + if (i in remove) continue + print lines[i] + if (lines[i]=="## Queued") for (j=1;j<=rn;j++) { print ""; print req[j] } + } + } + ' "$dir/backlog.md" > "$dir/backlog.md.tmp" && mv "$dir/backlog.md.tmp" "$dir/backlog.md" + fm_fleet_event "$dir" "-" reap "-" "ttl=$ttl" + fm_fleet_commit "$dir" "fleet: reap stale claims (ttl=$ttl)" + fm_fleet_unlock +} + +# --- routing ------------------------------------------------------------------ + +# Echo the operator who should own a task of the given scope. +# scope-primary: the online operator whose scope column contains the scope. +# overflow: if none online, the operator whose scope contains "overflow". +# Echo the operator who should own a task of the given scope. +# An operator is ELIGIBLE only when all three hold: +# status:online AND heartbeat fresh (seen within FM_FLEET_HEARTBEAT_TTL, default 90s) +# AND published quota headroom >= FM_FLEET_QUOTA_MIN (default 5), unless quota is '-'. +# Freshness + quota are self-healing: a crashed firstmate stops heartbeating and a +# low-headroom operator publishes it, so routing skips both without cross-user auth. +# 5-column legacy rows (no seen/quota) skip the freshness/quota checks (back-compat). +# scope-primary first; else the overflow operator. One awk pass over operators.md. +fm_fleet_route() { # dir scope + local dir=$1 scope=$2 now ttl floor + now=$(date -u +%s); ttl=${FM_FLEET_HEARTBEAT_TTL:-90}; floor=${FM_FLEET_QUOTA_MIN:-5} + awk -F'|' -v s="$scope" -v now="$now" -v ttl="$ttl" -v floor="$floor" ' + function trim(x){ gsub(/^ +| +$/,"",x); return x } + function epoch(iso, c,e){ if(iso==""||iso=="-")return -1; c="date -u -d \"" iso "\" +%s 2>/dev/null"; c|getline e; close(c); return e+0 } + function eligible(st,seen,q, ep){ + if(st!="online") return 0 + ep=epoch(seen); if(ep>0 && (now-ep)>ttl) return 0 + if(q!="" && q!="-" && (q+0)/dev/null || true) + inflt=$(grep -c "claimed-by:$op@.*status:in-flight" "$dir/backlog.md" 2>/dev/null || true) + last=$(awk -F'\t' -v o="$op" '$2==o{t=$1} END{print t}' "$dir/events.log" 2>/dev/null) + printf "%-20s %-8s %-10s %s\n" "$op" "${c:-0}" "${inflt:-0}" "${last:--}" + done < <(awk -F'|' '/^\| *[a-zA-Z0-9_.-]+ *\|/{op=$2; gsub(/^ +| +$/,"",op); if(op!="operator" && op !~ /^-+$/) print op}' "$dir/operators.md") +} + +# --- operator lifecycle + token economy (each user runs these AS THEMSELVES) --- +# operators.md row: | op | scope | home | accounts | status | seen(iso) | quota(%|-) | +# seen + quota are self-published by that operator's own heartbeat, so routing can +# treat a crashed (stale) or low-headroom peer as unavailable WITHOUT reading that +# peer's home or auth. The recorded home is validated under the caller's own $HOME. + +# Min headroom % across providers via quota-axi (current shell's auth); '-' if +# unavailable. Bash-only, zero LLM tokens. +fm_fleet_quota_now() { + command -v quota-axi >/dev/null 2>&1 || { printf '%s' '-'; return 0; } + command -v jq >/dev/null 2>&1 || { printf '%s' '-'; return 0; } + local j min + j=$(quota-axi --json 2>/dev/null) || { printf '%s' '-'; return 0; } + min=$(printf '%s' "$j" | jq -r '[.providers[]?.windows[]?.percentRemaining] | min // "-"' 2>/dev/null) + case "$min" in ''|null) printf '%s' '-' ;; *) printf '%s' "$min" ;; esac +} + +# Human-readable per-surface headroom for EVERY llm/cli/app quota-axi knows about, +# with each surface's observability status. Read-only, bash+jq, 0 LLM tokens. +# Rationale: each CLI/app subscription is its OWN token pool, so a model reachable via +# more than one surface (e.g. grok via the grok CLI AND via a Cursor subscription) has +# one row per surface. A surface only contributes to routing when status is "fresh"; +# "auth_required"/"unavailable"/"error" surfaces are shown but flagged un-observable. +fm_fleet_quota_report() { + command -v quota-axi >/dev/null 2>&1 || { + { echo "fm-fleet: quota-axi is not on PATH — per-surface headroom is unavailable." + echo " quota-axi reports how much budget each provider has left; the fleet uses it to" + echo " route work away from drained accounts. Install it, or skip quota-aware routing." + echo " Everything else (queue/claim/route/handoff) works without it." + } >&2 + return 1 + } + command -v jq >/dev/null 2>&1 || { echo "jq not installed" >&2; return 1; } + local j; j=$(quota-axi --json 2>/dev/null) || { + { echo "fm-fleet: quota-axi ran but returned no usable data." + echo " Most often this means no provider is signed in yet in THIS shell's environment." + echo " Check with: quota-axi auth (shows each provider's credential source/status)" + } >&2 + return 1 + } + local base="${FM_HOME:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" + # Collect custom-source rows once. A source is authoritative for its surface and + # SUPERSEDES the quota-axi row of the same name (e.g. an authed `cursor` override + # replaces quota-axi's blind cursor row; `cline` is added since quota-axi lacks it). + local -a SRC=(); local f s + for f in "$base"/bin/quota-sources/*.sh; do + [ -f "$f" ] || continue; s=$(bash "$f" 2>/dev/null) || continue + [ -n "$s" ] && SRC+=("$s") + done + local ex_json='[]' + if [ "${#SRC[@]}" -gt 0 ]; then + ex_json=$(printf '%s\n' "${SRC[@]}" | jq -r '.surface // empty' 2>/dev/null | jq -R . | jq -s . 2>/dev/null) + [ -n "$ex_json" ] || ex_json='[]' + fi + { + printf 'SURFACE\tHEADROOM\tSTATUS\tSOURCE\tNOTE\n' + printf '%s\n' "$j" | jq -r --argjson ex "$ex_json" ' + .providers[] | select((.provider as $p | $ex | index($p)) | not) + | ((.quotaSemantics.effectiveAvailability // [] + | map(select(.scope=="all_models").effectivePercentRemaining) | .[0]) + // (.windows // [] | map(.percentRemaining) | min)) as $rem + | [ .provider, + (if $rem==null then "—" else ($rem|tostring)+"%" end), + (.state.status // "?"), (.source // "?"), + (if (.state.status // "")=="fresh" then "observable" + else (.state.error // "not reporting") end) + ] | @tsv' + local row + for row in "${SRC[@]:-}"; do + [ -n "$row" ] || continue + printf '%s\n' "$row" | jq -r ' + [ .surface, + (if .headroom==null then "—" else (.headroom|tostring)+"%" end), + (.status // "?"), "custom", (.note // "") ] | @tsv' + done + } | if command -v column >/dev/null 2>&1; then column -t -s "$(printf '\t')"; else cat; fi +} + +# Resolve the model->surfaces map: the operator's gitignored config/model-surfaces.json +# when one exists, else the tracked default shipped at docs/examples/model-surfaces.json +# (config/ must hold no tracked files — repo invariant), so a bare clone still routes. +fm_fleet_model_map() { # base + local m="$1/config/model-surfaces.json" + [ -f "$m" ] || m="$1/docs/examples/model-surfaces.json" + printf '%s\n' "$m" +} + +# model family -> surfaces (quota pools) with each surface's live status + headroom. +# Answers "for model X, which pools can serve it and which have tokens?" — the basis +# for grok/kimi failover across surfaces. Reads fm_fleet_model_map. 0 LLM tokens. +fm_fleet_models_report() { + command -v jq >/dev/null 2>&1 || { echo "jq not installed" >&2; return 1; } + local base="${FM_HOME:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" + local map; map=$(fm_fleet_model_map "$base") + [ -f "$map" ] || { echo "no model map at $map" >&2; return 1; } + local j; j=$(quota-axi --json 2>/dev/null || echo '{"providers":[]}') + declare -A ST HR + local p st hr + while IFS=$'\t' read -r p st hr; do ST[$p]=$st; HR[$p]=$hr; done < <( + printf '%s\n' "$j" | jq -r '.providers[] + | ((.quotaSemantics.effectiveAvailability // [] | map(select(.scope=="all_models").effectivePercentRemaining) | .[0]) + // (.windows//[]|map(.percentRemaining)|min)) as $r + | [.provider, (.state.status//"?"), (if $r==null then "—" else ($r|tostring)+"%" end)] | @tsv') + local f s surf + for f in "$base"/bin/quota-sources/*.sh; do + [ -f "$f" ] || continue; s=$(bash "$f" 2>/dev/null) || continue + surf=$(printf '%s' "$s" | jq -r '.surface // empty' 2>/dev/null); [ -n "$surf" ] || continue + ST[$surf]=$(printf '%s' "$s" | jq -r '.status//"?"') + HR[$surf]=$(printf '%s' "$s" | jq -r 'if .headroom==null then "—" else (.headroom|tostring)+"%" end') + done + { + printf 'MODEL\tSURFACES (pool: status headroom)\n' + local fam surfaces sfx out + while IFS= read -r fam; do + surfaces=$(jq -r --arg k "$fam" '.[$k][]?' "$map") + out="" + while IFS= read -r sfx; do + [ -n "$sfx" ] || continue + out+="${out:+ | }${sfx}: ${ST[$sfx]:-unconfigured} ${HR[$sfx]:-—}" + done <<< "$surfaces" + printf '%s\t%s\n' "$fam" "$out" + done < <(jq -r 'keys_unsorted[] | select(startswith("_")|not)' "$map") + } | if command -v column >/dev/null 2>&1; then column -t -s "$(printf '\t')"; else cat; fi +} + +# Failover selector: pick the best surface (quota pool) to serve a model family. +# pass 1: first surface with OBSERVABLE headroom >= FM_FLEET_QUOTA_MIN (has tokens) +# pass 2: else first surface configured/online but unobservable (fail-open target) +# pass 3: else the first listed surface (last resort) +# Echoes the surface name; non-zero (with message) if the family is unknown. 0 tokens. +# This is the "grok from whichever pool has tokens / kimi3 via cline" decision. +fm_fleet_pick_surface() { # model-family + command -v jq >/dev/null 2>&1 || return 2 + local fam=$1 base map floor=${FM_FLEET_QUOTA_MIN:-5} + base="${FM_HOME:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" + map=$(fm_fleet_model_map "$base") + [ -f "$map" ] || return 2 + local surfaces; surfaces=$(jq -r --arg k "$fam" '.[$k][]?' "$map") + [ -n "$surfaces" ] || { echo "unknown model family: $fam" >&2; return 1; } + local j; j=$(quota-axi --json 2>/dev/null || echo '{"providers":[]}') + declare -A ST HR + local p st hr + while IFS=$'\t' read -r p st hr; do ST[$p]=$st; HR[$p]=$hr; done < <( + printf '%s\n' "$j" | jq -r '.providers[] + | ((.quotaSemantics.effectiveAvailability//[]|map(select(.scope=="all_models").effectivePercentRemaining)|.[0]) + //(.windows//[]|map(.percentRemaining)|min)) as $r + | [.provider,(.state.status//"?"),(if $r==null then "" else ($r|tostring) end)] | @tsv') + local f s surf + for f in "$base"/bin/quota-sources/*.sh; do + [ -f "$f" ] || continue; s=$(bash "$f" 2>/dev/null) || continue + surf=$(printf '%s' "$s" | jq -r '.surface//empty' 2>/dev/null); [ -n "$surf" ] || continue + ST[$surf]=$(printf '%s' "$s" | jq -r '.status//"?"') + HR[$surf]=$(printf '%s' "$s" | jq -r 'if .headroom==null then "" else (.headroom|tostring) end') + done + local sfx h + while IFS= read -r sfx; do [ -n "$sfx" ] || continue + h=${HR[$sfx]:-} + if [ -n "$h" ] && fm_fleet_num_ge "$h" "$floor"; then echo "$sfx"; return 0; fi + done <<< "$surfaces" + while IFS= read -r sfx; do [ -n "$sfx" ] || continue + case "${ST[$sfx]:-}" in fresh|configured|online|logged_in) echo "$sfx"; return 0;; esac + done <<< "$surfaces" + printf '%s\n' "$surfaces" | head -1 +} + +# Float-safe >= (headroom percentages can be fractional, e.g. 90.5, which the +# integer-only [ -ge ] test cannot parse). +fm_fleet_num_ge() { # a b + awk -v a="$1" -v b="$2" 'BEGIN{exit !((a+0)>=(b+0))}' +} + +# 0 if this operator has enough headroom to take work (min% >= FM_FLEET_QUOTA_MIN, +# default 5). Fail-OPEN when quota is unmeasurable ('-') so a missing quota-axi never +# blocks work; the guard only holds back a MEASURABLY-drained account. +fm_fleet_budget_ok() { + local floor=${FM_FLEET_QUOTA_MIN:-5} q + q=$(fm_fleet_quota_now) + [ "$q" = '-' ] && return 0 + fm_fleet_num_ge "$q" "$floor" +} + +# Refuse a recorded home outside the caller's own $HOME (cross-uid safety). +fm_fleet_assert_own_home() { # home + local home=${1%/} + case "$home" in + "$HOME"|"$HOME"/*) return 0 ;; + *) echo "error: refusing to register a home outside your own \$HOME ($HOME): $1" >&2; return 1 ;; + esac +} + +# Upsert this operator's row (self-onboard / update). Idempotent: an existing row for +# is replaced, not duplicated. Stamps status:online, seen:now, quota:now. +fm_fleet_register() { # dir op scopes home [accounts] + local dir=$1 op=$2 scopes=$3 home=$4 accounts=${5:-} ts q + fm_fleet_assert_own_home "$home" || return 1 + ts=$(fm_fleet_now); q=$(fm_fleet_quota_now) + fm_fleet_lock "$dir" || return 1 + grep -vE "^\| *$op *\|" "$dir/operators.md" > "$dir/operators.md.tmp" && mv "$dir/operators.md.tmp" "$dir/operators.md" + printf '| %s | %s | %s | %s | online | %s | %s |\n' "$op" "$scopes" "$home" "$accounts" "$ts" "$q" >> "$dir/operators.md" + fm_fleet_event "$dir" "$op" register "-" "scope:$scopes" + fm_fleet_commit "$dir" "fleet: register $op" + fm_fleet_unlock +} + +# Refresh this operator's liveness: seen:now + quota:now + status:online. Bash-only, +# meant to run on a cheap timer/daemon (NOT the LLM) so being "online" costs 0 tokens. +fm_fleet_heartbeat() { # dir op + local dir=$1 op=$2 ts q rc=1 + ts=$(fm_fleet_now); q=$(fm_fleet_quota_now) + fm_fleet_lock "$dir" || return 1 + if grep -qE "^\| *$op *\|" "$dir/operators.md"; then + awk -F'|' -v OFS='|' -v op="$op" -v ts=" $ts " -v q=" $q " ' + function trim(x){ gsub(/^ +| +$/,"",x); return x } + trim($2)==op { $6=" online "; $7=ts; $8=q; NF=(NF<9?9:NF); print; next } + { print } + ' "$dir/operators.md" > "$dir/operators.md.tmp" && mv "$dir/operators.md.tmp" "$dir/operators.md" + # No git commit / event line: heartbeat is transient liveness, not audit history. + # Committing every beat would bloat the KB log and churn the lock. The seen + # column IS the liveness record; on a same-machine shared FS the file write is + # visible to every operator immediately. + rc=0 + fi + fm_fleet_unlock + return $rc +} + +# Mark this operator offline (clean shutdown). Routing skips it immediately. +fm_fleet_leave() { # dir op + local dir=$1 op=$2 rc=1 + fm_fleet_lock "$dir" || return 1 + if grep -qE "^\| *$op *\|" "$dir/operators.md"; then + awk -F'|' -v OFS='|' -v op="$op" ' + function trim(x){ gsub(/^ +| +$/,"",x); return x } + trim($2)==op { $6=" offline "; print; next } + { print } + ' "$dir/operators.md" > "$dir/operators.md.tmp" && mv "$dir/operators.md.tmp" "$dir/operators.md" + fm_fleet_event "$dir" "$op" leave "-" "" + fm_fleet_commit "$dir" "fleet: leave $op" + rc=0 + fi + fm_fleet_unlock + return $rc +} diff --git a/bin/fm-fleet-wait.sh b/bin/fm-fleet-wait.sh new file mode 100755 index 00000000000..fd85e1dfb3b --- /dev/null +++ b/bin/fm-fleet-wait.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# fm-fleet-wait.sh — the token-economy core of federated mode. +# +# Blocks (BASH ONLY — zero LLM tokens) until THIS operator has a FRESH claim +# (an item stamped `claimed-by:@ status:claimed`, i.e. assigned but not yet +# started), then prints the claimed id(s) and exits 0. A firstmate primary (or its +# supervision daemon) runs this and stays idle until it returns, so an operator's +# LLM is invoked ONLY when there is real work — not on a polling loop of its own. +# While waiting it also heartbeats (cheap file write, no git), so "being online" +# costs nothing. +# +# Usage: +# fm-fleet-wait.sh [--interval N] [--timeout S] [--once] [--no-heartbeat] +# --interval N poll seconds (default FM_FLEET_WAIT_INTERVAL or 15) +# --timeout S give up after S seconds and exit 2 (default 0 = wait forever) +# --once check exactly once: exit 0 if a fresh claim exists, else 1 +# --no-heartbeat do not refresh liveness while waiting +# Exits 3 on a usage error or when the resolved fleet dir is not an +# initialized, chosen fleet (fm_fleet_assert_usable). +set -uo pipefail +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_HOME="${FM_HOME:-$(cd "$SCRIPT_DIR/.." && pwd)}" +# shellcheck source=bin/fm-fleet-lib.sh disable=SC1091 +. "$SCRIPT_DIR/fm-fleet-lib.sh" +DIR=$(fm_fleet_dir) + +op=${1:-}; [ -n "$op" ] || { echo "usage: fm-fleet-wait.sh [--interval N] [--timeout S] [--once] [--no-heartbeat]" >&2; exit 3; } +shift +interval=${FM_FLEET_WAIT_INTERVAL:-15}; timeout=0; once=0; heartbeat=1 +while [ $# -gt 0 ]; do + case "$1" in + --interval) interval=$2; shift 2 ;; + --timeout) timeout=$2; shift 2 ;; + --once) once=1; shift ;; + --no-heartbeat) heartbeat=0; shift ;; + *) echo "fm-fleet-wait.sh: unknown arg '$1'" >&2; exit 3 ;; + esac +done + +fm_fleet_assert_usable "$DIR" || exit 3 + +# A fresh claim for this operator: claimed-by:@ with status:claimed +# (NOT status:in-flight — once the firstmate starts an item it is no longer a wake). +fresh_claims() { + grep -E "claimed-by:$op@[^ ]+ status:claimed" "$DIR/backlog.md" 2>/dev/null | grep -oE '\[id:[^]]+\]' +} + +start=$(date -u +%s) +while :; do + ids=$(fresh_claims) + if [ -n "$ids" ]; then + printf '%s\n' "$ids" + exit 0 + fi + [ "$once" = 1 ] && exit 1 + [ "$heartbeat" = 1 ] && fm_fleet_heartbeat "$DIR" "$op" >/dev/null 2>&1 || true + if [ "$timeout" -gt 0 ] 2>/dev/null; then + now=$(date -u +%s) + [ $((now - start)) -ge "$timeout" ] && { echo "fm-fleet-wait: timed out after ${timeout}s with no claim for $op" >&2; exit 2; } + fi + sleep "$interval" +done diff --git a/bin/fm-fleet.sh b/bin/fm-fleet.sh new file mode 100755 index 00000000000..dfcb9b0ba05 --- /dev/null +++ b/bin/fm-fleet.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# fm-fleet.sh — FirstMate federation CLI. Coordinates multiple operators through a +# shared, cross-uid-safe, git-backed KB. See docs/fleet-quickstart.md, +# docs/fleet-addon.md, and .agents/skills/federation/SKILL.md. +# +# Usage: +# fm-fleet.sh init +# fm-fleet.sh register [accounts] +# fm-fleet.sh heartbeat +# fm-fleet.sh leave +# fm-fleet.sh queue +# fm-fleet.sh claim +# fm-fleet.sh handoff +# fm-fleet.sh reap [ttl-seconds] (default 86400) +# fm-fleet.sh route (echoes owning operator) +# fm-fleet.sh budget (exit 0 iff local headroom >= FM_FLEET_QUOTA_MIN) +# fm-fleet.sh quota (per-surface headroom report) +# fm-fleet.sh models (model family -> surfaces table) +# fm-fleet.sh pick (first surface with headroom) +# fm-fleet.sh status +# fm-fleet.sh view [--follow] +# +# Fleet dir resolves from: FM_FLEET_DIR -> $FM_HOME/config/fleet-dir -> /opt/agents/fleet +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_HOME="${FM_HOME:-$(cd "$SCRIPT_DIR/.." && pwd)}" +# shellcheck source=bin/fm-fleet-lib.sh disable=SC1091 +. "$SCRIPT_DIR/fm-fleet-lib.sh" +DIR=$(fm_fleet_dir) + +cmd=${1:-}; shift || true + +# Every verb that touches an existing fleet must find one first. `init` creates it; +# quota/models/pick are surface-local and need no fleet at all. +case "$cmd" in + init|budget|quota|models|pick|'') : ;; + # register/heartbeat/leave are how you BECOME an operator, and they already need + # write access to the shared dir (POSIX group), so they skip the ownership check. + register|heartbeat|leave) fm_fleet_assert_initialized "$DIR" || exit 1 ;; + *) fm_fleet_assert_usable "$DIR" || exit 1 ;; +esac + +case "$cmd" in + init) fm_fleet_init "$DIR"; echo "fleet initialized at $DIR" ;; + queue) id=$1; scope=$2; shift 2; fm_fleet_queue "$DIR" "$id" "$scope" "$*" ;; + claim) fm_fleet_claim "$DIR" "$1" "$2" ;; + handoff) fm_fleet_handoff "$DIR" "$1" "$2" ;; + reap) fm_fleet_reap "$DIR" "${1:-86400}" ;; + route) fm_fleet_route "$DIR" "$1" ;; + status) fm_fleet_status "$DIR" ;; + view) fm_fleet_view "$DIR" "${1:-}" ;; + register) op=$1; scopes=$2; home=$3; shift 3; fm_fleet_register "$DIR" "$op" "$scopes" "$home" "${1:-}" ;; + heartbeat) fm_fleet_heartbeat "$DIR" "$1" ;; + leave) fm_fleet_leave "$DIR" "$1" ;; + budget) if fm_fleet_budget_ok; then echo "ok (min headroom >= ${FM_FLEET_QUOTA_MIN:-5}%)"; else echo "below floor (< ${FM_FLEET_QUOTA_MIN:-5}%)"; exit 1; fi ;; + quota) fm_fleet_quota_report ;; + models) fm_fleet_models_report ;; + pick) fm_fleet_pick_surface "${1:?usage: fm-fleet.sh pick }" ;; + *) echo "usage: fm-fleet.sh init|register|heartbeat|leave|queue|claim|handoff|reap|route|budget|quota|models|pick|status|view" >&2; exit 1 ;; +esac diff --git a/bin/fm-spawn-acct.sh b/bin/fm-spawn-acct.sh new file mode 100755 index 00000000000..b04e0371c8d --- /dev/null +++ b/bin/fm-spawn-acct.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# fm-spawn-acct.sh — multi-account wrapper around fm-spawn.sh (Phase 4 add-on). +# +# Adds a per-spawn --account axis WITHOUT modifying fm-spawn.sh: it composes an +# account-isolated launch command (fm_account_compose_launch) and hands it to +# fm-spawn's raw-launch escape hatch. Isolation rides in the command string, so +# it survives the Herdr/tmux pane boundary; no secret is placed on argv. +# +# Scope: config-dir accounts (claude/codex/pi/cline). api-key accounts +# (grok/cursor) are refused here (a key would land on argv) — use +# fm-account-exec.sh for a direct, non-supervised isolated launch instead. +# +# Usage: +# fm-spawn-acct.sh --account [--model M] [--effort E] [passthrough flags...] +# +# Testable: set FM_SPAWN_BIN to a stub to capture the composed launch command. +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_HOME="${FM_HOME:-$(cd "$SCRIPT_DIR/.." && pwd)}"; export FM_HOME +# shellcheck source=bin/fm-account-env.sh disable=SC1091 +. "$SCRIPT_DIR/fm-account-env.sh" + +ACCOUNT=""; MODEL=""; EFFORT=""; POS=(); PASS=() +while [ "$#" -gt 0 ]; do + case "$1" in + --account) ACCOUNT=${2:-}; shift 2 ;; + --account=*) ACCOUNT=${1#--account=}; shift ;; + --model) MODEL=${2:-}; shift 2 ;; + --model=*) MODEL=${1#--model=}; shift ;; + --effort) EFFORT=${2:-}; shift 2 ;; + --effort=*) EFFORT=${1#--effort=}; shift ;; + *) if [ "${#POS[@]}" -lt 2 ]; then POS+=("$1"); else PASS+=("$1"); fi; shift ;; + esac +done + +[ -n "$ACCOUNT" ] || { echo "usage: fm-spawn-acct.sh --account [--model M] [--effort E] [flags...]" >&2; exit 1; } +[ "${#POS[@]}" -ge 1 ] || { echo "error: task-id (and usually project-dir) required" >&2; exit 1; } + +LAUNCH=$(fm_account_compose_launch "$ACCOUNT" "$MODEL" "$EFFORT") || exit $? + +FM_SPAWN_BIN="${FM_SPAWN_BIN:-$SCRIPT_DIR/fm-spawn.sh}" +# fm-spawn signature: [|] [flags...] +exec "$FM_SPAWN_BIN" "${POS[@]}" "$LAUNCH" ${PASS[@]+"${PASS[@]}"} diff --git a/bin/quota-copilot-usage.sh b/bin/quota-copilot-usage.sh new file mode 100755 index 00000000000..26afdef20a3 --- /dev/null +++ b/bin/quota-copilot-usage.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Authed usage reader for the `copilot` surface -> prints ONE integer 0-100 (headroom) or +# NOTHING (blind). Safe to reference from config/quota-overrides.json (.copilot). +# +# Why this exists: quota-axi ships a native `copilot` provider, but it probes only +# ~/.config/github-copilot/apps.json (the OLD IDE-plugin credential location). The +# standalone GitHub Copilot CLI (>=1.0.x) stores its OAuth token in ~/.copilot/config.json +# under .copilotTokens, so the native provider reports auth_required forever. This reader +# uses the CLI's OWN token — no browser cookie, no separate login: +# GET https://api.github.com/copilot_internal/user (Authorization: token ) +# Response .quota_snapshots..percent_remaining is the routable number. +# +# Bucket choice: `premium_interactions` is the only metered bucket on a subscriber plan +# (chat/completions report unlimited=true, percent_remaining=100). We report the minimum +# percent_remaining across all metered (unlimited=false) buckets, so a future plan that +# meters more than one bucket is bounded by its tightest limit rather than silently +# reporting the roomiest one. +# +# Token is passed via a 0600 header file (never argv, never stdout). Any failure -> exit 0 +# with no output, which the fleet treats as blind / fail-open. +set -uo pipefail + +cfg="$HOME/.copilot/config.json" +command -v curl >/dev/null 2>&1 || exit 0 +command -v python3 >/dev/null 2>&1 || exit 0 +[ -f "$cfg" ] || exit 0 + +# ~/.copilot/config.json is JSON-with-//-comments; strip comment lines before parsing. +tok=$(python3 - "$cfg" <<'PY' 2>/dev/null +import json, re, sys +try: + raw = re.sub(r'^\s*//.*$', '', open(sys.argv[1]).read(), flags=re.M) + toks = json.loads(raw).get("copilotTokens") or {} + print(next(iter(toks.values())) if toks else "") +except Exception: + print("") +PY +) +[ -n "$tok" ] || exit 0 + +hdr=$(mktemp); chmod 600 "$hdr" +out=$(mktemp); chmod 600 "$out" +trap 'rm -f "$hdr" "$out"' EXIT +{ printf 'Authorization: token %s\n' "$tok" + printf 'Accept: application/json\n' + printf 'User-Agent: GitHubCopilotCLI\n'; } > "$hdr" + +code=$(curl -sS -m 15 -o "$out" -w '%{http_code}' -H @"$hdr" \ + "https://api.github.com/copilot_internal/user" 2>/dev/null) || exit 0 +[ "$code" = "200" ] || exit 0 # 401/403/5xx -> blind + +python3 - "$out" <<'PY' 2>/dev/null +import json, sys +try: + snaps = (json.load(open(sys.argv[1])).get("quota_snapshots") or {}) +except Exception: + sys.exit(0) +metered = [q.get("percent_remaining") for q in snaps.values() + if isinstance(q, dict) and not q.get("unlimited") + and isinstance(q.get("percent_remaining"), (int, float))] +if not metered: + # every bucket unlimited -> full headroom (only when we actually saw buckets) + if snaps: + print(100) + sys.exit(0) +h = min(metered) +print(int(max(0, min(100, h)) + 0.5)) +PY diff --git a/bin/quota-cursor-usage.sh b/bin/quota-cursor-usage.sh new file mode 100755 index 00000000000..6864f0851e7 --- /dev/null +++ b/bin/quota-cursor-usage.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Authed usage reader for the `cursor` surface -> prints ONE integer 0-100 (headroom) or +# NOTHING (blind). Safe to reference from config/quota-overrides.json (.cursor). +# +# Uses Cursor's native Connect usage RPC with the CLI's OWN access token (the one cursor- +# agent already stores) — no browser cookie needed: +# POST https://api2.cursor.sh/aiserver.v1.DashboardService/GetCurrentPeriodUsage +# Content-Type: application/json + Connect-Protocol-Version: 1 + Bearer +# Response .planUsage.totalPercentUsed is "percent of included total used"; headroom = +# 100 - that. Token via a 0600 header file (never argv); nothing secret is printed. +set -uo pipefail +auth="$HOME/.config/cursor/auth.json" +command -v jq >/dev/null 2>&1 || exit 0 +command -v curl >/dev/null 2>&1 || exit 0 +[ -f "$auth" ] || exit 0 +tok=$(jq -r '.accessToken // empty' "$auth" 2>/dev/null) +[ -n "$tok" ] || exit 0 +# NOTE: `cursor-agent about` emits ANSI SGR codes (e.g. ESC[22m) around the version. An +# unstripped code in the header value makes the RPC return 400 Bad Request, so strip CSI +# sequences and keep only version-safe characters before using it. +ver=$(cursor-agent about 2>/dev/null \ + | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' \ + | awk -F' +' '/CLI Version/{print $2}' \ + | tr -cd '0-9A-Za-z.\-') +: "${ver:=2026.07.23}" + +hdr=$(mktemp); chmod 600 "$hdr" +{ printf 'Authorization: Bearer %s\n' "$tok" + printf 'Connect-Protocol-Version: 1\n' + printf 'x-cursor-client-version: %s\n' "$ver" + printf 'x-cursor-client-type: cli\n'; } > "$hdr" +trap 'rm -f "$hdr"' EXIT + +resp=$(curl -sS -m 15 -X POST -H @"$hdr" -H 'Content-Type: application/json' --data '{}' \ + "https://api2.cursor.sh/aiserver.v1.DashboardService/GetCurrentPeriodUsage" 2>/dev/null) || exit 0 +used=$(printf '%s' "$resp" | jq -r '.planUsage.totalPercentUsed // empty' 2>/dev/null) +[ -n "$used" ] || exit 0 # non-200 / unexpected shape -> blind +awk -v u="$used" 'BEGIN{ h=100-u; if(h<0)h=0; if(h>100)h=100; printf "%d\n", int(h+0.5) }' diff --git a/bin/quota-sources/copilot.sh b/bin/quota-sources/copilot.sh new file mode 100755 index 00000000000..158744d68df --- /dev/null +++ b/bin/quota-sources/copilot.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# quota source: copilot surface. quota-axi HAS a native `copilot` provider, but it probes +# only ~/.config/github-copilot/apps.json (the old IDE-plugin credential path), so a +# standalone GitHub Copilot CLI login is invisible to it and the row sits at +# auth_required. This source supersedes that row using the CLI's own token store. +# +# Unlike cursor/cline, copilot usage IS locally obtainable, so the default is a REAL +# number via bin/quota-copilot-usage.sh (wire it in config/quota-overrides.json .copilot). +# Without the override we still report accurate login state, headroom blind / fail-open. +# Read-only; no secret to stdout. +set -uo pipefail +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; root="$(cd "$here/../.." && pwd)" +ov="$root/config/quota-overrides.json"; hr=null + +override() { # surface -> echoes int 0-100 or nothing + command -v jq >/dev/null 2>&1 || return 0 + [ -f "$ov" ] || return 0 + local cmd; cmd=$(jq -r --arg s "$1" '.[$s] // ""' "$ov" 2>/dev/null) + [ -n "$cmd" ] || return 0 + local out; out=$(bash -c "$cmd" 2>/dev/null | tr -dc '0-9'); [ -n "$out" ] || return 0 + [ "$out" -ge 0 ] 2>/dev/null && [ "$out" -le 100 ] 2>/dev/null && printf '%s' "$out" +} + +# Login state = a stored token in the CLI's own config (presence only; value never read out). +copilot_logged_in() { + local cfg="$HOME/.copilot/config.json" + [ -f "$cfg" ] || return 1 + python3 - "$cfg" <<'PY' 2>/dev/null +import json, re, sys +try: + raw = re.sub(r'^\s*//.*$', '', open(sys.argv[1]).read(), flags=re.M) + d = json.loads(raw) +except Exception: + sys.exit(1) +sys.exit(0 if (d.get("copilotTokens") or d.get("loggedInUsers")) else 1) +PY +} + +if command -v copilot >/dev/null 2>&1; then + o=$(override copilot); [ -n "$o" ] && hr=$o + if copilot_logged_in; then status=logged_in; else status=auth_required; fi +else + status=unavailable +fi + +if [ "$hr" != null ]; then + note="live headroom via authed usage reader (copilot_internal/user quota_snapshots)" +elif [ "$status" = "logged_in" ]; then + note="blind: set config/quota-overrides.json .copilot to bin/quota-copilot-usage.sh for a live number" +else + note="GitHub Copilot CLI sign-in required (run: copilot login)" +fi +printf '{"surface":"copilot","status":"%s","headroom":%s,"unit":"premium interactions","models":["claude","gpt"],"note":"%s"}\n' "$status" "$hr" "$note" diff --git a/bin/quota-sources/cursor.sh b/bin/quota-sources/cursor.sh new file mode 100755 index 00000000000..8798cf6b05d --- /dev/null +++ b/bin/quota-sources/cursor.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# quota source: cursor surface. Usage is server-side (cursor.com dashboard, browser- +# session-cookie auth) and NOT readable from the CLI's api2.cursor.sh bearer, so headroom +# is blind by default. Supply an authed reader via config/quota-overrides.json (.cursor = +# a command printing one int 0-100) to get a real number. Read-only; no secret to stdout. +set -uo pipefail +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; root="$(cd "$here/../.." && pwd)" +ov="$root/config/quota-overrides.json"; hr=null + +override() { # surface -> echoes int 0-100 or nothing + command -v jq >/dev/null 2>&1 || return 0 + [ -f "$ov" ] || return 0 + local cmd; cmd=$(jq -r --arg s "$1" '.[$s] // ""' "$ov" 2>/dev/null) + [ -n "$cmd" ] || return 0 + local out; out=$(bash -c "$cmd" 2>/dev/null | tr -dc '0-9'); [ -n "$out" ] || return 0 + [ "$out" -ge 0 ] 2>/dev/null && [ "$out" -le 100 ] 2>/dev/null && printf '%s' "$out" +} + +if command -v cursor-agent >/dev/null 2>&1; then + o=$(override cursor); [ -n "$o" ] && hr=$o + status=$(cursor-agent status 2>/dev/null | grep -qi 'logged in' && echo logged_in || echo auth_required) +else + status=unavailable +fi +if [ "$hr" != null ]; then + note="live headroom via authed usage reader (Connect GetCurrentPeriodUsage)" +else + note="blind: set config/quota-overrides.json .cursor to bin/quota-cursor-usage.sh for a live number" +fi +printf '{"surface":"cursor","status":"%s","headroom":%s,"unit":"requests","models":["grok","claude","gpt"],"note":"%s"}\n' "$status" "$hr" "$note" diff --git a/docs/configuration.md b/docs/configuration.md index fed683343ea..b3e74b728e4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -245,6 +245,11 @@ Malformed JSON, an empty or malformed rule/default array, an unverified harness, While the file remains present, no crewmate or scout spawn may proceed without an explicit resolved harness; malformed configuration must be reported and corrected rather than selected around. Secondmate homes inherit this file from the primary, so a secondmate's own crewmates apply the same dispatch profile behavior. +## Fleet add-on (config/fleet-dir / config/accounts.json / FM_FLEET_*) + +The optional fleet add-on keeps its own operator configuration surfaces rather than duplicating them here. +[`docs/fleet-quickstart.md`](fleet-quickstart.md) owns setup for the gitignored `config/fleet-dir`, `config/accounts.json`, `config/quota-overrides.json`, and `config/model-surfaces.json` files plus the tracked [`docs/examples/model-surfaces.json`](examples/model-surfaces.json) failover map default, and [`docs/fleet-token-economy.md`](fleet-token-economy.md) owns the `FM_FLEET_*` knobs and their defaults. + ## Toolchain On session start the first mate detects what its required toolchain is missing or too old and lists each problem with either an exact install command or manual instructions. diff --git a/docs/documentation-audiences.json b/docs/documentation-audiences.json index 60773b0d451..320df6576c5 100644 --- a/docs/documentation-audiences.json +++ b/docs/documentation-audiences.json @@ -131,6 +131,10 @@ "path": ".agents/skills/diagnostic-reasoning/SKILL.md", "audience": "agent-runtime" }, + { + "path": ".agents/skills/federation/SKILL.md", + "audience": "agent-runtime" + }, { "path": ".agents/skills/firstmate-codexapp/SKILL.md", "audience": "agent-runtime" @@ -151,6 +155,10 @@ "path": ".agents/skills/harness-adapters/SKILL.md", "audience": "agent-runtime" }, + { + "path": ".agents/skills/multi-account/SKILL.md", + "audience": "agent-runtime" + }, { "path": ".agents/skills/project-management/SKILL.md", "audience": "agent-runtime" @@ -227,14 +235,38 @@ "path": "docs/documentation-audiences.md", "audience": "maintainer-architecture" }, + { + "path": "docs/examples/accounts.json", + "audience": "operator-example" + }, { "path": "docs/examples/crew-dispatch.json", "audience": "operator-example" }, + { + "path": "docs/examples/model-surfaces.json", + "audience": "operator-example" + }, + { + "path": "docs/examples/quota-overrides.json", + "audience": "operator-example" + }, { "path": "docs/examples/wedge-alarm", "audience": "operator-example" }, + { + "path": "docs/fleet-addon.md", + "audience": "maintainer-architecture" + }, + { + "path": "docs/fleet-quickstart.md", + "audience": "operator-current" + }, + { + "path": "docs/fleet-token-economy.md", + "audience": "operator-current" + }, { "path": "docs/fm-test-isolation-proof.md", "audience": "maintainer-verification" diff --git a/docs/examples/accounts.json b/docs/examples/accounts.json new file mode 100644 index 00000000000..ca380659a13 --- /dev/null +++ b/docs/examples/accounts.json @@ -0,0 +1,60 @@ +{ + "_comment": "TEMPLATE — copy to config/accounts.json (gitignored) and replace every /home/YOUR-USER path with your own. Paths are used LITERALLY: ~ and $HOME are NOT expanded, so write absolute paths. Delete the accounts you do not use; an entry whose config_dir/key_file does not exist is only an error when that account is actually selected. One entry per account. 'isolation' MUST match the harness per adapters/config-dir-matrix.md. NEVER put a secret here: api-key accounts point at a key_file (a 0600 file in YOUR OWN home); fm-spawn reads it at launch. config_dir/key_file must live under your own home (foreign /home/ is refused).", + + "claude-personal": { + "provider": "anthropic", + "harness": "claude", + "isolation": "config-dir-env", + "env": "CLAUDE_CONFIG_DIR", + "config_dir": "/home/YOUR-USER/.claude", + "scopes": ["backend", "infra", "deploy"] + }, + "claude-work": { + "provider": "anthropic", + "harness": "claude", + "isolation": "config-dir-env", + "env": "CLAUDE_CONFIG_DIR", + "config_dir": "/home/YOUR-USER/.claude-accounts/work", + "scopes": ["backend"] + }, + "codex-personal": { + "provider": "openai", + "harness": "codex", + "isolation": "config-dir-env", + "env": "CODEX_HOME", + "config_dir": "/home/YOUR-USER/.codex", + "scopes": ["backend"] + }, + "pi-personal": { + "provider": "pi", + "harness": "pi", + "isolation": "config-dir-env", + "env": "PI_CODING_AGENT_DIR", + "config_dir": "/home/YOUR-USER/.pi/agent", + "scopes": ["long-autonomous"] + }, + "cline-personal": { + "provider": "anthropic", + "harness": "cline", + "isolation": "config-dir-flag", + "flag": "--config", + "config_dir": "/home/YOUR-USER/.cline", + "scopes": ["web"] + }, + "grok-personal": { + "provider": "xai", + "harness": "grok", + "isolation": "api-key-env", + "env": "GROK_API_KEY", + "key_file": "/home/YOUR-USER/.secrets/grok-personal.key", + "scopes": ["research"] + }, + "cursor-personal": { + "provider": "cursor", + "harness": "cursor-agent", + "isolation": "api-key-env", + "env": "CURSOR_API_KEY", + "key_file": "/home/YOUR-USER/.secrets/cursor-personal.key", + "scopes": ["web"] + } +} diff --git a/docs/examples/model-surfaces.json b/docs/examples/model-surfaces.json new file mode 100644 index 00000000000..259f39002e8 --- /dev/null +++ b/docs/examples/model-surfaces.json @@ -0,0 +1,20 @@ +{ + "_comment": "Canonical model family -> ORDERED surfaces (quota pools) that can serve it. A 'surface' is a quota-axi provider (claude/codex/grok/cursor/copilot/kimi) or a custom source in bin/quota-sources/.sh. Routing/failover walks the list left-to-right and dispatches on the first surface with observable headroom; a surface whose quota is unobservable is treated fail-open as a valid failover target. Same model, several pools: this is how 'grok from whichever pool (grok CLI or Cursor) has tokens' is expressed. NOTE: cline is intentionally NOT a monitored surface (its usage is locked behind an internal WS-hub credential); it stays a usable crewmate harness but is out of quota routing \u2014 we let it hit its wall and route open work to other LLMs. Extend freely.", + "grok": [ + "grok", + "cursor" + ], + "kimi": [ + "kimi" + ], + "claude": [ + "claude", + "copilot", + "cursor" + ], + "gpt": [ + "codex", + "copilot", + "cursor" + ] +} \ No newline at end of file diff --git a/docs/examples/quota-overrides.json b/docs/examples/quota-overrides.json new file mode 100644 index 00000000000..9a30b81f1fc --- /dev/null +++ b/docs/examples/quota-overrides.json @@ -0,0 +1,4 @@ +{ + "_comment": "OPTIONAL authed usage readers, per surface. Map a surface -> a shell command that prints exactly ONE integer 0-100 = percent headroom remaining. The fleet runs it and uses the result as that surface's headroom, overriding quota-axi / the blind default. Use it for surfaces whose usage is not observable via quota-axi. The command OWNS all secret handling (read the token from a 0600 file; never put it on argv). Copy this to config/quota-overrides.json (gitignored) and fill in. Empty string or missing key = blind / fail-open (the default). Cursor ships a working reader: set \"cursor\": \"/bin/quota-cursor-usage.sh\".", + "cursor": "" +} diff --git a/docs/fleet-addon.md b/docs/fleet-addon.md new file mode 100644 index 00000000000..584802e961c --- /dev/null +++ b/docs/fleet-addon.md @@ -0,0 +1,181 @@ +# FirstMate Fleet add-on — federated multi-operator + multi-account + +Two general, reusable capabilities FirstMate does not ship today, built as a +**drop-in add-on that requires ZERO edits to FirstMate core**: + +1. **Federated / multi-operator mode** — several OS operators (each their own + first mate, own accounts), coordinating through a shared, cross-uid-safe, + git-backed KB with atomic claim/lock, scope routing, cross-operator handoff, + TTL reap, and a realtime `fleet view`. +2. **Per-spawn multi-account** — a `--account` axis that launches a crewmate under + a chosen account with isolated auth, plus quota-aware account selection. + +Everything lives in new `bin/fm-fleet*.sh`, `bin/fm-account*.sh`, +`bin/fm-accounts*.sh`, `bin/quota-*`, `scripts/fleet-root-prereq.sh`, +`.agents/skills/{federation,multi-account}/`, and `tests/federation/*.sh`. +No existing FirstMate script is modified — the `--account` +axis rides on `fm-spawn`'s existing raw-launch escape hatch. That is what makes +this shippable as an additive PR (or a standalone overlay). + +--- + +## Part A — Federated multi-operator + +### Why a new model +FirstMate today propagates prefs by **filesystem copy from main into secondmate +homes**. That breaks across uids (it would require writing another user's home). +The add-on uses a **shared-dir + read/claim** model instead: operators share only +a group-writable, git-backed KB and **never write each other's private homes**. + +### Shared KB (`$FM_FLEET_DIR`, default `/opt/agents/fleet`) +- `operators.md` — `| operator | scope | home | accounts | status | seen | quota |` +- `projects.md` — `| project | owner | path |` +- `backlog.md` — `## Queued / ## Claimed / ## In-flight / ## Done`; item line: + `- [id:] scope: | | [claimed-by:@] status:` +- `events.log` — append-only TSV `\t\t\t\t` +- `locks/backlog.lock` — the `flock` target for atomic claims + +Fleet dir resolves from: `FM_FLEET_DIR` → `$FM_HOME/config/fleet-dir` → +`/opt/agents/fleet`. During development it points at a local dir so every code +path is exercised single-uid; `flock` semantics are identical across uids. + +### CLI (`bin/fm-fleet.sh`, lib `bin/fm-fleet-lib.sh`) +`init | register | heartbeat | leave | queue | claim | handoff | reap | route | +budget | quota | models | pick | status | view`, plus `bin/fm-fleet-join.sh` +(operator onboarding) and `bin/fm-fleet-wait.sh` (token-free wait-for-work). + +- **Atomic claim** — under `flock`, verify item is `queued`, stamp + `claimed-by:@ status:claimed`, move to `## Claimed`, log, commit. Two + operators can never grab the same item (proven by a concurrent race test). +- **Routing** — `route `: scope-primary (the online operator whose scope + contains it), overflow fallback (the `overflow`-scoped operator) if the owner + is offline; a human `--operator` override always wins. +- **Reap** — requeue stale `status:claimed` items older than a TTL (offline + operators' never-started work); `status:in-flight` is left alone. +- **Visibility** — `status` (per-operator counts) + `view [--follow]` (the live + cross-operator event stream). + +### Cross-uid safety (non-negotiable) +Every mutating fleet function calls `fm_fleet_assert_shared`, which refuses any +path resolving into a foreign `/home/`. Credentials stay `0700`, read only +by their owner's own processes. See `.agents/skills/federation/SKILL.md`. + +### One privileged step (root, once) +Run the reviewable, idempotent `scripts/fleet-root-prereq.sh` (walkthrough: +[fleet-quickstart.md](fleet-quickstart.md), Tier C). It creates the shared +group, enrols the operators, and creates the setgid fleet dir; each operator +then sets `umask 002`. Nothing else needs root. + +--- + +## Part B — Per-spawn multi-account + +### Three isolation methods (verified per CLI — never guessed) +The matrix below records how each CLI isolates auth, probed from its own +`--help` (claude confirmed empirically); `bin/fm-accounts-lib.sh` validates +every registered account against it: + +| harness | method | env / flag | +|---|---|---| +| claude | `config-dir-env` | `CLAUDE_CONFIG_DIR` | +| codex | `config-dir-env` | `CODEX_HOME` | +| pi | `config-dir-env` | `PI_CODING_AGENT_DIR` | +| cline | `config-dir-flag` | `--config ` | +| grok | `api-key-env` | `GROK_API_KEY` | +| cursor-agent | `api-key-env` | `CURSOR_API_KEY` (OAuth mode not per-spawn isolatable) | + +### Account registry (`config/accounts.json`, gitignored) +```json +{ + "": { + "provider": "...", "harness": "...", "isolation": "config-dir-env|config-dir-flag|api-key-env", + "env": "", "flag": "", "config_dir": "", "key_file": "", + "scopes": ["..."] + } +} +``` +`bin/fm-accounts-lib.sh` resolves + **validates** each account against the matrix +(harness known, isolation matches the harness's method + env/flag, required +fields present, and — reusing the federation guard — paths never in a foreign +home). Copy `docs/examples/accounts.json` to start. + +**Secrets never live in the registry.** api-key accounts store a `key_file` path +(a `0600` file in the operator's own home); the key is read at launch into the +child's environment — never onto argv, never into a log. + +### The `--account` axis (`bin/fm-spawn-acct.sh`) +Adds `--account ` **without editing `fm-spawn.sh`**. It composes an +account-isolated launch command and hands it to `fm-spawn`'s raw-launch escape +hatch (which skips leading `ENV=val` tokens when detecting the harness): + +- `config-dir-env` → `CLAUDE_CONFIG_DIR=/path claude [--model … --effort …]` +- `config-dir-flag` → `cline --config /path [--model …]` + +The env prefix / flag rides **in the command string**, so isolation survives the +Herdr/tmux pane boundary. Config-dir isolation puts **no secret on argv**. + +api-key accounts are **refused** here (a key on argv would leak) → use +`bin/fm-account-exec.sh [args]` for a direct, non-supervised +isolated launch (reads the key_file into the child's env). Live-verified: a claude +crewmate launched under an isolated account writes to its own config dir and sees +a different MCP set than the default account. + +### Quota-aware selection (`fm_account_pick `) +`quota-axi` reports headroom **per provider for the currently-authed account**, so +per-account headroom is obtained by running `quota-axi` **under each account's +isolation**; the binding constraint is `min(percentRemaining)` across windows. +Pick the account with the most headroom; ties → first registered. Guards: +unsupported provider (pi/cline) or `quota-axi` absent → first registered. + +### Prereq installer (`bin/fm-accounts-prereq.sh`) +On-demand, **user-scoped, no sudo**. `detect` (default) shows installed / MISSING ++ the install command; `install [--yes] [harness…]` installs missing CLIs +(`npm i -g @anthropic-ai/claude-code|@openai/codex|@vibe-kit/grok-cli|cline`, +`curl https://cursor.com/install`). `pi` is system-managed → detect-only. Run this +first on a box that is missing, e.g., cursor. + +--- + +## Install (drop-in overlay onto a FirstMate clone) +1. Copy `bin/fm-fleet*.sh`, `bin/fm-account*.sh`, `bin/fm-accounts*.sh`, + `bin/fm-spawn-acct.sh`, `bin/quota-*.sh`, `bin/quota-sources/`, + `scripts/fleet-root-prereq.sh`, `tests/federation/`, + `.agents/skills/{federation,multi-account}/`, + `docs/examples/{model-surfaces,accounts,quota-overrides}.json`, and + `docs/fleet-*.md`. +2. `bin/fm-accounts-prereq.sh` — install any missing CLIs; then log in per account. +3. `cp docs/examples/accounts.json config/accounts.json` and edit; gitignore it. +4. Federation only: run the root prereq, then `bin/fm-fleet.sh init`. + +## Tests +``` +bash tests/federation/test_fleet.sh # federation: claim race, reap, route, handoff, view, safety +bash tests/federation/test_fleet_ops.sh # operator lifecycle: register/heartbeat/leave, TTL, quota routing +bash tests/federation/test_fleet_guards.sh # init/ownership guards on every fleet-consuming entry point +bash tests/federation/test_quota_surfaces.sh # per-surface quota report, models table, failover pick +bash tests/federation/test_accounts.sh # registry resolve/validate (+ cross-uid path guard) +bash tests/federation/test_spawn_account.sh # --account compose + wrapper + api-key refusal + apply_env +bash tests/federation/test_account_quota.sh # quota pick (isolate-then-query; tie/absent/no-provider) +``` + +## Known limitations (honest) +- **cursor-agent OAuth is not per-spawn isolatable** (creds in `~/.cursor`, no + relocation env). Multi-account for cursor uses API-key mode only. +- **grok** is API-key isolatable but stays out of the Herdr crew rotation (no + `GROK_AGENT` autonomy marker + no Herdr integration). +- **quota-axi is per-provider, not per-account** — on this box both claude config + dirs reported identical headroom because `quota-axi --provider claude` reads a + shared credential source regardless of `CLAUDE_CONFIG_DIR`. Genuine two-account + discrimination requires each account separately authed with creds quota-axi + reads (verify on a real second account); codex quota (in `$CODEX_HOME/auth.json`) + is expected to discriminate. `pi`/`cline` have no quota-axi coverage. +- The raw-launch path bypasses `fm-spawn`'s per-harness model/effort mapping; the + wrapper folds `--model` and (for claude/codex/pi) `--effort` into the command. + +## Packaging options +1. **Upstream PR to `kunchenguid/firstmate` (recommended).** All additive files, + no core edits → small, reviewable diff. Consent-gated (outward-facing). +2. **Standalone add-on repo** overlaid onto a FirstMate clone (same files). +3. **axi-style tool** — possible but *not* simpler: the bash scripts would need + npm-bin repackaging + a SessionStart hook, and federation needs a shared + git-backed dir that doesn't fit the per-user axi model. Recommend #1/#2. diff --git a/docs/fleet-quickstart.md b/docs/fleet-quickstart.md new file mode 100644 index 00000000000..9dbd476ecdb --- /dev/null +++ b/docs/fleet-quickstart.md @@ -0,0 +1,249 @@ +# Fleet quickstart — pick your use case + +The fleet add-on does two separable things. **You do not need both**, and the first +one needs no setup at all: + +1. **Per-surface token visibility** — one table showing how much budget is left in + every AI subscription you own (Claude, Codex, Copilot, Cursor, Grok, Kimi), plus + automatic failover to whichever pool still has headroom. +2. **Federation** — several people, each running their own first mate on one host, + coordinating through a shared work queue so they never collide. + +Start at the tier you actually need. Each is independent and additive. + +| | Use case | Root needed? | Setup | +|---|---|---|---| +| **A** | *"Which of my subscriptions still has budget, and can work route itself there?"* | no | ~2 min | +| **B** | *"I have two Claude accounts / a work and a personal one."* | no | ~5 min | +| **C** | *"Three of us share this box and keep stepping on each other."* | once | ~15 min | + +--- + +## Tier A — token visibility and model→surface failover + +No fleet, no root, no shared directory. This works in a plain clone. + +**Requires:** [`quota-axi`](https://www.npmjs.com/package/quota-axi) on `PATH`, plus +`jq`, `curl`, `python3`. Whichever agent CLIs you use should already be signed in. + +```bash +bin/fm-fleet.sh quota # headroom per surface +bin/fm-fleet.sh models # which surfaces can serve each model family +bin/fm-fleet.sh pick gpt # -> the first surface with headroom +``` + +``` +SURFACE HEADROOM STATUS SOURCE NOTE +claude 50% fresh oauth observable +codex 100% fresh cli-rpc observable +copilot 71% logged_in custom live headroom via authed usage reader +cursor 69% logged_in custom live headroom via authed usage reader +grok — auth_required unavailable Grok sign-in required +``` + +**Why this exists.** The same model often reaches you through several paid pools — +Claude via an Anthropic subscription *and* via Copilot *and* via Cursor. When one +pool is drained the work should move, not stop. The shipped map +`docs/examples/model-surfaces.json` (override: copy it to the gitignored +`config/model-surfaces.json` and edit) maps +each model family to an ordered list of surfaces, and `pick` walks it left to right: + +```json +{ "claude": ["claude", "copilot", "cursor"], + "gpt": ["codex", "copilot", "cursor"] } +``` + +A surface whose usage cannot be observed is treated **fail-open** (a valid target), +so an unreadable provider never blocks routing. + +### Surfaces whose usage is not locally readable + +Some vendors keep usage behind a browser session. For those, supply your own +reader — a command printing a single integer `0-100` (percent headroom): + +```bash +cp docs/examples/quota-overrides.json config/quota-overrides.json # gitignored +``` + +```json +{ "cursor": "/abs/path/to/firstmate/bin/quota-cursor-usage.sh" } +``` + +Two readers ship working, both using the CLI's *own* stored token — no browser +cookie, no second login: + +- `bin/quota-copilot-usage.sh` — GitHub Copilot. Reads `~/.copilot/config.json`, + calls `copilot_internal/user`, and takes the **minimum** `percent_remaining` + across *metered* quota buckets, so a plan that meters several is bounded by its + tightest limit. +- `bin/quota-cursor-usage.sh` — Cursor. Uses the CLI access token against Cursor's + own usage RPC. + +Your command owns all secret handling: read the token from a `0600` file and never +put it on `argv`. A reader that fails prints nothing and the surface goes blind — +never an error, never a wrong number. + +--- + +## Tier B — several accounts for one person + +Same machine, same user, more than one subscription. Isolation is **per CLI** and +there are three different mechanisms, so the registry records which one applies: + +```bash +cp docs/examples/accounts.json config/accounts.json # gitignored +$EDITOR config/accounts.json # replace /home/YOUR-USER +bin/fm-spawn-acct.sh --account claude-work +``` + +| Harness | Mechanism | Key | +|---|---|---| +| `claude` | config-dir env | `CLAUDE_CONFIG_DIR` | +| `codex` | config-dir env | `CODEX_HOME` | +| `pi` | config-dir env | `PI_CODING_AGENT_DIR` | +| `cline` | argv flag | `--config` | +| `grok`, `cursor-agent` | api key env | `GROK_API_KEY` / `CURSOR_API_KEY` | + +Paths in `accounts.json` are used **literally** — `~` and `$HOME` are not expanded. +Secrets never go in the file: api-key accounts name a `key_file` (a `0600` file in +your own home) that is read at launch into the child's environment, never onto +`argv`. `config_dir` and `key_file` must live under your own home; a path resolving +into another user's `/home/` is refused. + +> **Known limit:** `quota-axi` reports per *provider*, not per *account*, so two +> Claude accounts show one shared number. Per-account discrimination works where +> the CLI keys off its config dir (e.g. `CODEX_HOME`). + +--- + +## Tier C — several people on one host + +Each operator runs their **own** first mate as themselves. Nobody reads anyone +else's home. The only shared surface is the fleet directory. + +**One-time, root, once per host** — review the script first, it is short and additive: + +```bash +sudo FM_FLEET_OPERATORS="alice bob carol" bash scripts/fleet-root-prereq.sh +``` + +It creates group `agents`, adds those OS users to it, and creates the fleet dir +`2775` (setgid, so new files inherit the group). Nothing else. With no +`FM_FLEET_OPERATORS` it enrols only whoever ran `sudo`. Override the location with +`FM_FLEET_ROOT_DIR=/srv/agents/fleet`. + +**Then each operator, as themselves, with no root:** + +```bash +# group membership only applies to a NEW login — see Troubleshooting +echo 'umask 002' >> ~/.bashrc +bin/fm-fleet.sh init # first operator only +bin/fm-fleet-join.sh alice web,frontend # everyone +``` + +Day to day: + +```bash +bin/fm-fleet.sh queue TASK-12 backend "fix the migration" +bin/fm-fleet.sh route backend # -> which operator owns this scope +bin/fm-fleet.sh claim TASK-12 alice # atomic; exactly one winner under a race +bin/fm-fleet.sh status +bin/fm-fleet.sh view --follow # live event stream +``` + +Routing is **scope-primary, quota-secondary**: a task goes to the operator owning +that scope, unless they are stale or below `FM_FLEET_QUOTA_MIN` (default 5%), in +which case it overflows to someone with headroom. + +### Heartbeats are mandatory + +An operator that stops heartbeating is treated as offline after +`FM_FLEET_HEARTBEAT_TTL` (default **90s**) and routing skips them. Registration is +**not** enough — without a heartbeat every operator goes stale ~90s after joining +and routing silently returns nothing. Run it on a timer: + +```ini +# ~/.config/systemd/user/fm-heartbeat.service +[Service] +Type=oneshot +ExecStart=/usr/bin/sg agents -c "umask 002; /path/to/firstmate/bin/fm-fleet.sh heartbeat alice" +``` + +```ini +# ~/.config/systemd/user/fm-heartbeat.timer +[Timer] +OnBootSec=30s +OnUnitActiveSec=45s +[Install] +WantedBy=timers.target +``` + +```bash +systemctl --user enable --now fm-heartbeat.timer +loginctl enable-linger "$USER" # survive logout/reboot +``` + +The `sg agents -c` wrapper is not cosmetic — see Troubleshooting. + +### Idle at zero tokens + +`bin/fm-fleet-wait.sh` blocks in **bash** until work is claimed for you, heartbeating +while it waits. The agent burns no tokens idling and wakes only on real work. See +[fleet-token-economy.md](fleet-token-economy.md). + +--- + +## How the fleet directory is chosen + +``` +FM_FLEET_DIR → $FM_HOME/config/fleet-dir → built-in default (/opt/agents/fleet) +``` + +The built-in default is a **convention, not a guarantee** — on a shared host it may +already belong to someone else. Any verb that reads a fleet fails loudly if the +resolved directory is not an initialized fleet, and tells you which directory it +picked and how. Set `FM_FLEET_DIR`, or write the path into +`$FM_HOME/config/fleet-dir`, to be explicit. + +--- + +## Troubleshooting + +**`no initialized fleet at …`** — expected on a fresh clone. The message names the +directory and how it was chosen; follow the option it prints. + +**`quota-axi is not on PATH`** — Tier A only. Queue, claim, route and handoff all +work without it; you lose quota-aware routing. + +**A surface shows `auth_required` although the CLI is signed in** — `quota-axi auth` +prints where it looked. Some CLIs moved their credential store; that is exactly why +`bin/quota-sources/.sh` exists to supersede a stale native probe. + +**`Permission denied` on the fleet dir although `id` shows the group** — `id` reads +`/etc/group`; a *running process* carries the group set from when it started. Check +the truth with `grep Groups /proc/$$/status`. A long-lived `tmux`/`screen` server +and the `systemd --user` **manager** both keep their original credentials, and a new +shell does not refresh the manager. Either reconnect properly, or wrap the command +in `sg agents -c "…"`, which re-reads group membership at exec and is immune to +process age and reboots. + +**`systemctl --user` says `Failed to connect to bus`** — non-login/background +session. `export XDG_RUNTIME_DIR=/run/user/$(id -u)`. + +**Operators registered but `route` returns nothing** — nobody is heartbeating; see +*Heartbeats are mandatory*. + +--- + +## Requirements summary + +| | Tier A | Tier B | Tier C | +|---|---|---|---| +| `bash`, `git`, `awk`, `flock` | ✓ | ✓ | ✓ | +| `jq`, `curl`, `python3` | ✓ | ✓ | ✓ | +| `quota-axi` | ✓ | optional | optional | +| root, once per host | — | — | ✓ | +| shared POSIX group | — | — | ✓ | + +Portable across any POSIX host with a shared filesystem. No daemon, no database, no +network service — coordination is `flock` plus a git-backed directory. diff --git a/docs/fleet-token-economy.md b/docs/fleet-token-economy.md new file mode 100644 index 00000000000..41f9b0cab7d --- /dev/null +++ b/docs/fleet-token-economy.md @@ -0,0 +1,66 @@ +# Fleet token economy + +The point of federated mode is to run several operators' first mates at once **without +paying for several idle LLMs**. The whole coordination layer is **bash — zero LLM +tokens** — and each operator's expensive LLM is invoked only when it has real work. + +## The one rule: the LLM is event-driven, never polling + +A first-mate primary is an LLM (claude/opus, etc.). The expensive failure mode is an +always-on primary that *thinks on a timer*. Federated mode forbids that: + +``` +join ──► fm-fleet-wait.sh # BASH. blocks. 0 tokens. + │ (heartbeats every interval, also bash) + ▼ + a fresh claim for appears # another op routed/handed work here + │ + ▼ + wait exits 0 ──► wake the LLM primary ──► it does the work ──► back to wait +``` + +`fm-fleet-wait.sh` returns **only** when this operator has an item stamped +`claimed-by: status:claimed`. Until then the primary is not running a turn, so it +costs nothing. This is the single biggest saving vs. N self-polling primaries. + +## Everything coordination-related is bash (0 tokens) + +| Concern | Mechanism | Tokens | +|---|---|---| +| Claim / route / handoff / reap | `fm-fleet.sh` (flock + awk) | 0 | +| Liveness | `fm-fleet.sh heartbeat` (file write, **no git commit**) | 0 | +| Wait-for-work | `fm-fleet-wait.sh` (poll/block) | 0 | +| Quota headroom | `quota-axi` published into `operators.md` | 0 | +| Sync between operators | shared group-writable FS (same box) — live, no bus | 0 | + +Heartbeats deliberately **do not** `git commit` — liveness is transient, so it never +bloats the KB audit log or churns the lock. + +## Don't hand work to a drained account + +Routing is quota-aware without any cross-user auth: each operator publishes its own +`quota-axi` min headroom into its `operators.md` row on heartbeat, and `fm-fleet.sh +route` skips any operator below the floor (falls to overflow). Before claiming, a +primary can gate on `fm-fleet.sh budget` (`fm_fleet_budget_ok`). This keeps a +near-limit account from being handed work it would fail partway through (wasted +tokens), routing it to a peer with headroom instead. + +## Cheapest capable model per task + +`config/crew-dispatch.json` already tiers crewmate dispatch (haiku for rote, sonnet +for web/product, opus/codex-high for hard backend). The coordinator picks the cheapest +tier that fits, so even when work IS running the spend matches the task. + +## Knobs (env, all bash-side) + +| Var | Default | Meaning | +|---|---|---| +| `FM_FLEET_WAIT_INTERVAL` | 15s | wake-watcher poll cadence | +| `FM_FLEET_HEARTBEAT_TTL` | 90s | after this with no heartbeat, routing treats an operator offline | +| `FM_FLEET_QUOTA_MIN` | 5 (%) | headroom floor below which routing/claim skips an operator | + +## Net effect + +Three operators can be "online" continuously. Their **LLMs** only spend tokens while +actually executing a claimed task; the rest of the time the fleet is coordinated +entirely in bash. Idle cost across the whole fleet ≈ 0. diff --git a/docs/scripts.md b/docs/scripts.md index 6a10d1310ae..2741023fc75 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -93,3 +93,16 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-x-dismiss.sh` | Dismiss a skipped X-mode mention at the relay without replying | | `fm-x-link.sh` | Link a spawned task to its originating X-mode mention in task meta | | `fm-x-followup.sh` | Detect, post, and cap completion follow-ups for an X-mode-linked task | +| `fm-fleet.sh` | Federated multi-operator coordination CLI over the shared fleet KB, plus per-surface quota, models, and failover pick verbs (docs/fleet-quickstart.md) | +| `fm-fleet-lib.sh` | Shared federation KB helpers: atomic claim/lock, routing, usability guards, and quota surfaces | +| `fm-fleet-join.sh` | One-command operator onboarding into a shared fleet | +| `fm-fleet-wait.sh` | Token-free bash block-until-claimed wait that heartbeats while idle (docs/fleet-token-economy.md) | +| `fm-accounts-lib.sh` | Multi-account registry resolution, validation, and quota-aware account pick | +| `fm-account-env.sh` | Apply a registered account's auth isolation as a launch command or process environment | +| `fm-account-exec.sh` | Direct account-isolated launch: read the key_file into the child's environment, then exec | +| `fm-spawn-acct.sh` | Per-spawn `--account` axis for `fm-spawn.sh` via its raw-launch escape hatch | +| `fm-accounts-prereq.sh` | User-scoped detect-or-install of the LLM CLIs the account registry needs | +| `quota-copilot-usage.sh` | Authed Copilot headroom reader: minimum percent remaining across metered quota buckets | +| `quota-cursor-usage.sh` | Authed Cursor headroom reader via the CLI's own stored access token | +| `quota-sources/copilot.sh` | Copilot surface row for `fm-fleet.sh quota`, superseding the stale native probe | +| `quota-sources/cursor.sh` | Cursor surface row for `fm-fleet.sh quota` from an operator-supplied authed reader | diff --git a/scripts/fleet-root-prereq.sh b/scripts/fleet-root-prereq.sh new file mode 100755 index 00000000000..341fd75b793 --- /dev/null +++ b/scripts/fleet-root-prereq.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# ONE-TIME root prerequisite for FirstMate federation (the ONLY privileged step). +# Creates the shared group + group-writable KB dir that operators coordinate through. +# Idempotent and additive — safe to re-run. Review it, then run as root: +# +# sudo bash scripts/fleet-root-prereq.sh +# +# It changes nothing outside: (1) the `agents` group, (2) group membership for the +# listed operators, (3) /opt/agents/fleet (mode 2775 setgid). Reverse steps at the end. +set -euo pipefail +GROUP=${FM_FLEET_GROUP:-agents} +DIR=${FM_FLEET_ROOT_DIR:-/opt/agents/fleet} +# Operators default to whoever invoked sudo — NEVER a baked-in list, or running this +# on someone else's host would try to enrol names that mean nothing there. Add the +# rest of your team explicitly: +# sudo FM_FLEET_OPERATORS="alice bob carol" bash scripts/fleet-root-prereq.sh +OPERATORS=${FM_FLEET_OPERATORS:-${SUDO_USER:-}} + +[ "$(id -u)" -eq 0 ] || { echo "must run as root: sudo bash $0" >&2; exit 1; } + +if [ -z "${OPERATORS// /}" ]; then + cat >&2 </dev/null 2>&1; then + usermod -aG "$GROUP" "$u"; echo "added $u to $GROUP" + else + echo "skip: OS user '$u' does not exist" + fi +done + +mkdir -p "$DIR/locks" +chgrp -R "$GROUP" "$DIR" +find "$DIR" -type d -exec chmod 2775 {} + # setgid dirs: files created here inherit the group +find "$DIR" -type f -exec chmod g+rw {} + + +echo "--- verify ---" +stat -c '%A %U:%G %n' "$DIR" +echo "expect: drwxrwsr-x root:$GROUP $DIR" +echo +echo "NEXT (per operator, NO root needed):" +echo " • group membership takes effect on your NEXT login (re-login or 'newgrp $GROUP')" +echo " • echo 'umask 002' >> ~/.bashrc # keep shared files group-writable" +echo " • cd && FM_FLEET_DIR=$DIR bin/fm-fleet.sh init # once, first operator only" +echo " • FM_FLEET_DIR=$DIR bin/fm-fleet-join.sh [accounts-csv]" +echo +echo "# Reverse (only if you ever want to undo):" +echo "# for u in $OPERATORS; do gpasswd -d \$u $GROUP 2>/dev/null; done; groupdel $GROUP; rm -rf $DIR" diff --git a/tests/federation/test_account_quota.sh b/tests/federation/test_account_quota.sh new file mode 100644 index 00000000000..66756da4540 --- /dev/null +++ b/tests/federation/test_account_quota.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# Quota-aware account selection test (Phase 4, Task 12). Run from ~/kun-agent-workspace: +# bash tests/federation/test_account_quota.sh +# A stub quota-axi returns headroom that DEPENDS on the isolation env it runs +# under (CLAUDE_CONFIG_DIR), so this exercises the full isolate-then-query chain: +# pick highest headroom; tie -> first registered; quota-axi absent -> first; +# harness with no quota coverage (pi) -> first. +set -uo pipefail +cd "$(dirname "$0")/../.." || exit 2 +FM_HOME="$(pwd)"; export FM_HOME +# shellcheck source=bin/fm-accounts-lib.sh disable=SC1091 +. bin/fm-accounts-lib.sh +fails=0 +ok(){ echo "PASS: $1"; } +bad(){ echo "FAIL: $1"; fails=$((fails+1)); } + +TMP=$(mktemp -d); export FM_ACCOUNTS_FILE="$TMP/accounts.json" + +# stub quota-axi: headroom encoded in CLAUDE_CONFIG_DIR (proves isolation is applied) +STUB="$TMP/quota-stub.sh" +cat > "$STUB" <<'S' +#!/usr/bin/env bash +hr=10 +case "${CLAUDE_CONFIG_DIR:-}" in + *high*) hr=90 ;; + *low*) hr=20 ;; +esac +printf '{"providers":[{"provider":"claude","windows":[{"percentRemaining":%s}]}]}\n' "$hr" +S +chmod +x "$STUB" +export QUOTA_AXI_BIN="$STUB" + +# Case A: highest headroom wins +cat > "$FM_ACCOUNTS_FILE" </dev/null) +[ "$p" = "claude-high" ] && ok "pick highest headroom (90 > 20)" || bad "pick highest (got '$p')" + +# Case A2: quota-axi absent -> first registered +p=$(QUOTA_AXI_BIN="/nonexistent/quota-axi" fm_account_pick claude 2>/dev/null) +first=$(fm_account_list_by_harness claude | head -1) +[ "$p" = "$first" ] && ok "quota-axi absent -> first registered ($first)" || bad "absent fallback (got '$p' want '$first')" + +# Case B: harness with no quota coverage (pi) -> first registered +cat > "$FM_ACCOUNTS_FILE" </dev/null) +[ "$p" = "pi-a" ] && ok "no quota provider -> first (pi-a)" || bad "no-provider fallback (got '$p')" + +# Case C: tie -> first registered +cat > "$FM_ACCOUNTS_FILE" </dev/null) +[ "$p" = "claude-eqa" ] && ok "tie -> first registered (claude-eqa)" || bad "tie-break (got '$p')" + +# Case D: single account -> that account (no quota call needed) +cat > "$FM_ACCOUNTS_FILE" </dev/null) +[ "$p" = "solo" ] && ok "single account picked directly" || bad "single (got '$p')" + +rm -rf "$TMP" +echo "-----"; [ "$fails" -eq 0 ] && { echo "ALL PASS"; exit 0; } || { echo "$fails FAILURE(S)"; exit 1; } diff --git a/tests/federation/test_accounts.sh b/tests/federation/test_accounts.sh new file mode 100644 index 00000000000..30c1bfdf6ff --- /dev/null +++ b/tests/federation/test_accounts.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# Account registry + resolution/validation test (Phase 4, Task 10). +# Run from ~/kun-agent-workspace: bash tests/federation/test_accounts.sh +# Exercises: resolve returns per-account config_dir; unknown fails; api-key +# exposes key_file; validate accepts a good account and rejects unknown harness, +# wrong env-for-harness, and a foreign-home path (cross-uid safety). +set -uo pipefail +cd "$(dirname "$0")/../.." || exit 2 +FM_HOME="$(pwd)"; export FM_HOME +# shellcheck source=bin/fm-accounts-lib.sh disable=SC1091 +. bin/fm-accounts-lib.sh +fails=0 +ok(){ echo "PASS: $1"; } +bad(){ echo "FAIL: $1"; fails=$((fails+1)); } + +TMP=$(mktemp -d); export FM_ACCOUNTS_FILE="$TMP/accounts.json" +P1="$TMP/p1"; P2="$TMP/p2"; mkdir -p "$P1" "$P2" + +cat > "$FM_ACCOUNTS_FILE" </dev/null 2>&1; then bad "unknown account resolved (should fail)"; else ok "unknown account fails"; fi + +# 3. api-key account exposes key_file (field 6) +rg=$(fm_account_resolve grok-personal); kf=$(printf '%s' "$rg" | cut -f6) +[ "$kf" = "$P1/grok.key" ] && ok "api-key resolve exposes key_file" || bad "key_file (got '$kf')" + +# 4. validate: good account passes +fm_account_validate claude-personal >/dev/null 2>&1 && ok "validate accepts good account" || bad "validate good account" + +# 5. validate: unknown harness rejected +cat > "$FM_ACCOUNTS_FILE" </dev/null 2>&1 && bad "unknown harness passed validate" || ok "unknown harness rejected" + +# 6. validate: wrong env-for-harness rejected +cat > "$FM_ACCOUNTS_FILE" </dev/null 2>&1 && bad "wrong env-for-harness passed validate" || ok "wrong env-for-harness rejected" + +# 7. validate: foreign-home path refused (cross-uid safety) +cat > "$FM_ACCOUNTS_FILE" </dev/null 2>&1 && bad "foreign-home path passed validate" || ok "foreign-home path refused" + +rm -rf "$TMP" +echo "-----"; [ "$fails" -eq 0 ] && { echo "ALL PASS"; exit 0; } || { echo "$fails FAILURE(S)"; exit 1; } diff --git a/tests/federation/test_fleet.sh b/tests/federation/test_fleet.sh new file mode 100755 index 00000000000..537d4f30c79 --- /dev/null +++ b/tests/federation/test_fleet.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +# Comprehensive federation test. Run from ~/kun-agent-workspace: +# bash tests/federation/test_fleet.sh +# Exercises: init, atomic no-overlap claim race, TTL reap, scope routing, +# cross-operator handoff, view/status, and the cross-uid safety guard. +set -uo pipefail +cd "$(dirname "$0")/../.." || exit 2 +FLEET_CLI="bin/fm-fleet.sh" +fails=0 +ok(){ echo "PASS: $1"; } +bad(){ echo "FAIL: $1"; fails=$((fails+1)); } + +# ---- 1. init ---- +D=$(mktemp -d); export FM_FLEET_DIR="$D/fleet" +"$FLEET_CLI" init >/dev/null +allok=1 +for f in operators.md projects.md backlog.md events.log locks; do + [ -e "$FM_FLEET_DIR/$f" ] || { allok=0; echo " missing $f"; } +done +grep -q '## Queued' "$FM_FLEET_DIR/backlog.md" && [ "$allok" = 1 ] && ok "init creates KB" || bad "init" + +# ---- 2. atomic claim race (the crux) ---- +"$FLEET_CLI" queue FL-1 backend "race item" >/dev/null +( "$FLEET_CLI" claim FL-1 adi >/dev/null 2>&1; echo $? >"$D/a.rc" ) & +( "$FLEET_CLI" claim FL-1 royce >/dev/null 2>&1; echo $? >"$D/b.rc" ) & +wait +wins=$(( $(cat "$D/a.rc")==0 ? 1 : 0 )) +wins=$(( wins + ($(cat "$D/b.rc")==0 ? 1 : 0) )) +claims=$(grep -c 'claimed-by:' "$FM_FLEET_DIR/backlog.md") +{ [ "$wins" -eq 1 ] && [ "$claims" -eq 1 ]; } && ok "atomic claim: exactly one winner, one record" || bad "atomic claim (winners=$wins claims=$claims)" + +# ---- 3. TTL reap ---- +D2=$(mktemp -d); export FM_FLEET_DIR="$D2/fleet" +"$FLEET_CLI" init >/dev/null +"$FLEET_CLI" queue FL-9 backend demo >/dev/null; "$FLEET_CLI" claim FL-9 royce >/dev/null +# stale claimed -> should requeue +sed -i 's/@[0-9TZ:-]\{1,\}/@2000-01-01T00:00:00Z/' "$FM_FLEET_DIR/backlog.md" +"$FLEET_CLI" reap 3600 >/dev/null +grep -q '\[id:FL-9\].*status:queued' "$FM_FLEET_DIR/backlog.md" && ok "reap requeues stale claim" || bad "reap requeue" +# in-flight with old ts must NOT be requeued +"$FLEET_CLI" queue FL-10 backend demo2 >/dev/null; "$FLEET_CLI" claim FL-10 royce >/dev/null +sed -i 's/\(FL-10.*\)status:claimed/\1status:in-flight/' "$FM_FLEET_DIR/backlog.md" +sed -i 's/@[0-9TZ:-]\{1,\}/@2000-01-01T00:00:00Z/' "$FM_FLEET_DIR/backlog.md" +"$FLEET_CLI" reap 3600 >/dev/null +grep -q '\[id:FL-10\].*status:in-flight' "$FM_FLEET_DIR/backlog.md" && ok "reap leaves in-flight alone" || bad "reap in-flight" + +# ---- 4. scope routing ---- +D3=$(mktemp -d); export FM_FLEET_DIR="$D3/fleet" +"$FLEET_CLI" init >/dev/null +cat >> "$FM_FLEET_DIR/operators.md" <<'OPS' +| adi | backend,infra,deploy | /home/adi/kun-agent-workspace | claude:default | online | +| royce | web,mobile,product | /home/royce/kun-agent-workspace | claude:default | online | +| barf-ai | overflow | /home/barf-ai/kun-agent-workspace | claude:default | online | +OPS +r_back=$("$FLEET_CLI" route backend); r_web=$("$FLEET_CLI" route web) +{ [ "$r_back" = adi ] && [ "$r_web" = royce ]; } && ok "route: backend->adi, web->royce" || bad "route primary (got '$r_back'/'$r_web')" +# adi offline -> backend falls to overflow (barf-ai) +sed -i 's/| adi \(.*\)| online |/| adi \1| offline |/' "$FM_FLEET_DIR/operators.md" +r_off=$("$FLEET_CLI" route backend) +[ "$r_off" = barf-ai ] && ok "route: offline owner -> overflow" || bad "route overflow (got '$r_off')" + +# ---- 5. handoff ---- +D4=$(mktemp -d); export FM_FLEET_DIR="$D4/fleet" +"$FLEET_CLI" init >/dev/null +"$FLEET_CLI" queue FL-2 web "handoff item" >/dev/null +"$FLEET_CLI" claim FL-2 adi >/dev/null +"$FLEET_CLI" handoff FL-2 royce >/dev/null +grep -q '\[id:FL-2\].*claimed-by:royce@' "$FM_FLEET_DIR/backlog.md" \ + && grep -q $'\thandoff\tFL-2' "$FM_FLEET_DIR/events.log" \ + && ok "handoff reassigns + logs event" || bad "handoff" + +# ---- 6. view + status ---- +vlines=$("$FLEET_CLI" view | grep -c 'FL-2' || true) +[ "$vlines" -ge 1 ] && ok "view renders events" || bad "view" +status_out=$("$FLEET_CLI" status); echo "$status_out" | grep -q 'operator' && ok "status renders header" || bad "status" + +# ---- 7. cross-uid safety guard ---- +# sourcing the lib and asserting a foreign home is refused +( . bin/fm-fleet-lib.sh; fm_fleet_assert_shared "/home/someoneelse/kun-agent-workspace" ) 2>/dev/null \ + && bad "safety: foreign home NOT refused" || ok "safety: foreign home refused" +( . bin/fm-fleet-lib.sh; fm_fleet_assert_shared "/opt/agents/fleet" ) 2>/dev/null \ + && ok "safety: /opt shared dir allowed" || bad "safety: /opt wrongly refused" + +echo "-----" +[ "$fails" -eq 0 ] && { echo "ALL PASS"; exit 0; } || { echo "$fails FAILURE(S)"; exit 1; } diff --git a/tests/federation/test_fleet_guards.sh b/tests/federation/test_fleet_guards.sh new file mode 100755 index 00000000000..029f960f2e4 --- /dev/null +++ b/tests/federation/test_fleet_guards.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# Guards around fleet-dir RESOLUTION — the first-run experience for someone who just +# cloned the repo and configured nothing. +# +# Two failure modes are covered, both found by actually running a fresh clone: +# 1. dir is not a fleet -> used to surface as `awk: fatal: cannot open ...` with +# exit 0: a raw internal error that also looked like success. +# 2. dir IS a fleet, but someone else's -> the built-in default is a conventional +# shared path; a bare clone silently listed another team's operators and dumped +# their whole event log. Membership is the opt-in signal. +# +# Hermetic: every fleet lives in a temp dir and FM_FLEET_DEFAULT_DIR is overridden, so +# the real /opt/agents/fleet is never consulted. +set -uo pipefail +REAL="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +pass=0; fail=0 +ok(){ echo "PASS: $1"; pass=$((pass+1)); } +no(){ echo "FAIL: $1"; fail=$((fail+1)); } + +ME=$(id -un) +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +# A fleet that belongs to somebody else entirely. +THEIRS="$TMP/theirs" +mkdir -p "$THEIRS" +FM_FLEET_DIR="$THEIRS" "$REAL/bin/fm-fleet.sh" init >/dev/null 2>&1 +printf '| alice | web | /home/alice/firstmate | claude-default | online | 2026-07-28T06:00:00Z | 80 |\n' \ + >> "$THEIRS/operators.md" + +# Run fm-fleet.sh with a controlled environment. FM_HOME points at a config-less dir so +# `config/fleet-dir` can never be picked up from the real repo. +mkdir -p "$TMP/fmhome/config" +fleet() { # var-assignments... -- verb args + env FM_HOME="$TMP/fmhome" FM_FLEET_DEFAULT_DIR="$THEIRS" "$@" 2>&1 +} + +# --- 1. not a fleet at all ---------------------------------------------------------- +out=$(env FM_HOME="$TMP/fmhome" FM_FLEET_DIR="$TMP/nope" "$REAL/bin/fm-fleet.sh" status 2>&1); rc=$? +[ "$rc" -ne 0 ] && ok "uninitialized fleet exits non-zero (was 0)" || no "uninitialized fleet exited $rc" +printf '%s' "$out" | grep -q 'no initialized fleet' \ + && ok "uninitialized fleet names the problem" || no "no diagnostic: $out" +printf '%s' "$out" | grep -qi 'awk' && no "raw awk error leaked to the user" || ok "no raw awk error" +printf '%s' "$out" | grep -q 'chosen by FM_FLEET_DIR' \ + && ok "diagnostic says HOW the dir was chosen" || no "missing provenance" + +# --- 2. an existing fleet that is not yours, reached via the DEFAULT ---------------- +for verb in status view; do + out=$(fleet "$REAL/bin/fm-fleet.sh" "$verb"); rc=$? + [ "$rc" -ne 0 ] \ + && ok "default -> foreign fleet: '$verb' refuses" \ + || no "'$verb' returned $rc on a foreign fleet" + printf '%s' "$out" | grep -q 'alice' \ + && no "'$verb' leaked the other team's data" \ + || ok "default -> foreign fleet: '$verb' leaks nothing" +done + +# route must not disclose the foreign operator either +out=$(fleet "$REAL/bin/fm-fleet.sh" route web) +printf '%s' "$out" | grep -q '^alice$' && no "route disclosed a foreign operator" \ + || ok "route discloses no foreign operator" + +# --- 3. explicit choice is always honoured ------------------------------------------ +out=$(env FM_HOME="$TMP/fmhome" FM_FLEET_DIR="$THEIRS" "$REAL/bin/fm-fleet.sh" status 2>&1); rc=$? +[ "$rc" -eq 0 ] && printf '%s' "$out" | grep -q alice \ + && ok "explicit FM_FLEET_DIR is honoured (no ownership check)" \ + || no "explicit FM_FLEET_DIR blocked (rc=$rc): $out" + +out=$(env FM_HOME="$TMP/fmhome" FM_FLEET_DEFAULT_DIR="$THEIRS" FM_FLEET_ACCEPT_DEFAULT=1 \ + "$REAL/bin/fm-fleet.sh" status 2>&1); rc=$? +[ "$rc" -eq 0 ] && ok "FM_FLEET_ACCEPT_DEFAULT=1 acknowledges the default" \ + || no "acknowledged default still blocked (rc=$rc)" + +# --- 4. being an operator IS the opt-in --------------------------------------------- +MINE="$TMP/mine"; mkdir -p "$MINE" +FM_FLEET_DIR="$MINE" "$REAL/bin/fm-fleet.sh" init >/dev/null 2>&1 +printf '| %s | backend | %s | - | online | 2026-07-28T06:00:00Z | 90 |\n' "$ME" "$TMP/fmhome" \ + >> "$MINE/operators.md" +out=$(env FM_HOME="$TMP/fmhome" FM_FLEET_DEFAULT_DIR="$MINE" "$REAL/bin/fm-fleet.sh" status 2>&1); rc=$? +[ "$rc" -eq 0 ] && ok "default is allowed when you ARE an operator in it" \ + || no "own fleet via default was blocked (rc=$rc): $out" + +# --- 5. surface-local verbs need no fleet at all ------------------------------------ +# FM_HOME must be the real repo here: `models` reads the shipped model map +# (docs/examples/model-surfaces.json, or a local config/model-surfaces.json) from it. +# The point under test is only that a bogus FM_FLEET_DIR does not block it. +out=$(env FM_HOME="$REAL" FM_FLEET_DIR="$TMP/nope" "$REAL/bin/fm-fleet.sh" models 2>&1) +printf '%s' "$out" | grep -q 'no initialized fleet' \ + && no "models was blocked by the fleet guard" \ + || ok "models is not blocked by a missing fleet" + +out=$(env FM_HOME="$REAL" FM_FLEET_DIR="$TMP/nope" "$REAL/bin/fm-fleet.sh" budget 2>&1) +printf '%s' "$out" | grep -q 'no initialized fleet' \ + && no "budget was blocked by the fleet guard" \ + || ok "budget is not blocked by a missing fleet" + +# --- 6. fm-fleet-wait.sh honours the same guards ------------------------------------ +out=$(env FM_HOME="$TMP/fmhome" FM_FLEET_DIR="$TMP/nope" "$REAL/bin/fm-fleet-wait.sh" "$ME" --once 2>&1); rc=$? +[ "$rc" -eq 3 ] && printf '%s' "$out" | grep -q 'no initialized fleet' \ + && ok "wait refuses an uninitialized fleet loudly" \ + || no "wait on uninitialized fleet: rc=$rc: $out" + +out=$(fleet "$REAL/bin/fm-fleet-wait.sh" "$ME" --once); rc=$? +[ "$rc" -eq 3 ] && ok "default -> foreign fleet: wait refuses" \ + || no "wait on foreign default fleet: rc=$rc: $out" +printf '%s' "$out" | grep -q 'alice' \ + && no "wait leaked the other team's data" \ + || ok "default -> foreign fleet: wait leaks nothing" + +echo "-----" +[ "$fail" -eq 0 ] && echo "ALL PASS ($pass)" || { echo "$fail FAILED"; exit 1; } diff --git a/tests/federation/test_fleet_ops.sh b/tests/federation/test_fleet_ops.sh new file mode 100644 index 00000000000..35cdbfe906c --- /dev/null +++ b/tests/federation/test_fleet_ops.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# Operator-lifecycle + token-economy tests (fleet-ops). Run from ~/kun-agent-workspace: +# bash tests/federation/test_fleet_ops.sh +# Exercises the per-operator lifecycle that makes each user's own firstmate joinable, +# in-sync, and token-cheap: +# register (self-onboard, upsert, own-home-only), heartbeat (refresh seen+quota), +# leave (offline), online = status:online AND heartbeat-fresh AND quota>=floor, +# quota-aware routing (published headroom, no cross-user auth), and fm_fleet_budget_ok. +# +# operators.md row schema (backward-compatible superset of the 5-col form): +# | | | | | | | | +set -uo pipefail +cd "$(dirname "$0")/../.." || exit 2 +CLI="bin/fm-fleet.sh" +fails=0 +ok(){ echo "PASS: $1"; } +bad(){ echo "FAIL: $1"; fails=$((fails+1)); } + +# shellcheck source=bin/fm-fleet-lib.sh disable=SC1091 +. bin/fm-fleet-lib.sh + +now_iso(){ date -u +%Y-%m-%dT%H:%M:%SZ; } +old_iso(){ echo "2000-01-01T00:00:00Z"; } + +# 1. register self-onboards a fresh online row that route finds +D=$(mktemp -d); export FM_FLEET_DIR="$D/fleet"; unset FM_FLEET_HEARTBEAT_TTL FM_FLEET_QUOTA_MIN +"$CLI" init >/dev/null +"$CLI" register adi backend,infra "$HOME/kun-agent-workspace" claude-default >/dev/null 2>&1 +grep -qE "^\| *adi *\|" "$FM_FLEET_DIR/operators.md" && ok "register writes an operator row" || bad "register writes row" +[ "$("$CLI" route backend)" = adi ] && ok "route finds a freshly-registered operator" || bad "route fresh register (got '$("$CLI" route backend)')" + +# 2. register is idempotent (upsert, not duplicate) +"$CLI" register adi backend,infra "$HOME/kun-agent-workspace" claude-default >/dev/null 2>&1 +n=$(grep -cE "^\| *adi *\|" "$FM_FLEET_DIR/operators.md") +[ "$n" -eq 1 ] && ok "register is idempotent (one row)" || bad "register duplicated (n=$n)" + +# 3. heartbeat refreshes seen; a stale operator routes as offline +"$CLI" register royce web,mobile "$HOME/kun-agent-workspace" claude-default >/dev/null 2>&1 +"$CLI" register barf-ai overflow "$HOME/kun-agent-workspace" claude-default >/dev/null 2>&1 +# force royce's seen stale (replace the seen column in royce's row with an old ts) +sed -i "/^| royce /s#| [0-9][0-9TZ:-]\{1,\} |#| $(old_iso) |#" "$FM_FLEET_DIR/operators.md" +export FM_FLEET_HEARTBEAT_TTL=90 +r=$("$CLI" route web) +[ "$r" = barf-ai ] && ok "route: stale-heartbeat operator treated offline -> overflow" || bad "route stale->overflow (got '$r')" +# heartbeat royce back to fresh -> route returns royce +"$CLI" heartbeat royce >/dev/null 2>&1 +r=$("$CLI" route web) +[ "$r" = royce ] && ok "heartbeat refreshes seen -> operator online again" || bad "heartbeat refresh (got '$r')" + +# 4. leave marks offline -> route skips to overflow +"$CLI" leave royce >/dev/null 2>&1 +r=$("$CLI" route web) +[ "$r" = barf-ai ] && ok "leave -> offline -> overflow" || bad "leave offline (got '$r')" + +# 5. quota-aware routing: publish low headroom for the scope owner -> skip to overflow +D2=$(mktemp -d); export FM_FLEET_DIR="$D2/fleet" +"$CLI" init >/dev/null +ts=$(now_iso) +cat >> "$FM_FLEET_DIR/operators.md" < overflow" || bad "route quota floor (got '$r')" +# raise adi's quota -> owner wins again +sed -i "/^| adi /s#| 3 |#| 50 |#" "$FM_FLEET_DIR/operators.md" +r=$("$CLI" route backend) +[ "$r" = adi ] && ok "route: owner above quota floor -> owner" || bad "route quota ok (got '$r')" + +# 6. fm_fleet_budget_ok reflects a stubbed quota-axi min headroom vs floor +STUB=$(mktemp -d) +cat > "$STUB/quota-axi" <<'Q' +#!/usr/bin/env bash +echo "$FAKE_QUOTA_JSON" +Q +chmod +x "$STUB/quota-axi" +export FM_FLEET_QUOTA_MIN=5 +FAKE_QUOTA_JSON='{"providers":[{"provider":"claude","windows":[{"percentRemaining":40}]}]}' \ + PATH="$STUB:$PATH" fm_fleet_budget_ok && ok "budget_ok: above floor passes" || bad "budget_ok above floor" +FAKE_QUOTA_JSON='{"providers":[{"provider":"claude","windows":[{"percentRemaining":2}]}]}' \ + PATH="$STUB:$PATH" fm_fleet_budget_ok && bad "budget_ok below floor should fail" || ok "budget_ok: below floor fails" + +# 7. register refuses a foreign home (cross-uid safety) +D3=$(mktemp -d); export FM_FLEET_DIR="$D3/fleet"; "$CLI" init >/dev/null +"$CLI" register evil backend /home/someoneelse/kun-agent-workspace claude-default >/dev/null 2>&1 \ + && bad "register accepted a foreign home" || ok "register refuses a foreign home" + +# 8. fm-fleet-wait.sh (token economy): a fresh claim wakes; nothing else does +D4=$(mktemp -d); export FM_FLEET_DIR="$D4/fleet"; "$CLI" init >/dev/null +"$CLI" register adi backend "$HOME/kun-agent-workspace" claude-default >/dev/null 2>&1 +"$CLI" queue W-1 backend "wake item" >/dev/null; "$CLI" claim W-1 adi >/dev/null +out=$(bin/fm-fleet-wait.sh adi --once --no-heartbeat); rc=$? +{ [ "$rc" -eq 0 ] && printf '%s' "$out" | grep -q 'W-1'; } && ok "wait --once: fresh claim wakes (exit 0 + id)" || bad "wait fresh claim (rc=$rc out='$out')" +bin/fm-fleet-wait.sh royce --once --no-heartbeat >/dev/null 2>&1 && bad "wait woke with no claim" || ok "wait --once: no claim -> exit 1 (LLM stays idle)" +sed -i 's/\(W-1.*\)status:claimed/\1status:in-flight/' "$FM_FLEET_DIR/backlog.md" +bin/fm-fleet-wait.sh adi --once --no-heartbeat >/dev/null 2>&1 && bad "wait woke on in-flight (already started)" || ok "wait --once: in-flight item is not a fresh wake" + +# 9. fm-fleet-join.sh: self-onboard writes config/fleet-dir + registers; idempotent. +# HOME is overridden to a temp home so the own-home guard passes for the fixture. +JH=$(mktemp -d)/home; mkdir -p "$JH"; JF=$(mktemp -d)/fleet +FM_FLEET_DIR="$JF" "$CLI" init >/dev/null +out=$(HOME="$JH" FM_HOME="$JH" FM_FLEET_DIR="$JF" bin/fm-fleet-join.sh adi backend claude-default 2>&1); rc=$? +{ [ "$rc" -eq 0 ] && [ "$(cat "$JH/config/fleet-dir" 2>/dev/null)" = "$JF" ] && grep -qE "^\| *adi *\|" "$JF/operators.md"; } \ + && ok "join: writes config/fleet-dir + registers self" || bad "join (rc=$rc)" +HOME="$JH" FM_HOME="$JH" FM_FLEET_DIR="$JF" bin/fm-fleet-join.sh adi backend claude-default >/dev/null 2>&1 +n=$(grep -cE "^\| *adi *\|" "$JF/operators.md"); [ "$n" -eq 1 ] && ok "join: idempotent (one row on rejoin)" || bad "join dup (n=$n)" + +echo "-----" +[ "$fails" -eq 0 ] && { echo "ALL PASS"; exit 0; } || { echo "$fails FAILURE(S)"; exit 1; } diff --git a/tests/federation/test_quota_surfaces.sh b/tests/federation/test_quota_surfaces.sh new file mode 100755 index 00000000000..6dc4e7555ed --- /dev/null +++ b/tests/federation/test_quota_surfaces.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# Tests the per-surface quota view + model->surface map + failover selector + authed +# override hook. Hermetic: a temp FM_HOME supplies a controlled model map + a stub +# `cursor` quota-source (honoring config/quota-overrides.json), and `quota-axi` is +# stubbed on PATH, so the LOGIC is asserted (not live numbers): grok drained (2%), +# cursor healthy (80% via its authed reader) -> a grok task fails over to cursor. +set -uo pipefail +REAL="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +pass=0; fail=0 +ok(){ echo "PASS: $1"; pass=$((pass+1)); } +no(){ echo "FAIL: $1"; fail=$((fail+1)); } + +HOMEDIR=$(mktemp -d) +mkdir -p "$HOMEDIR/config" "$HOMEDIR/bin/quota-sources" +cp "$REAL/docs/examples/model-surfaces.json" "$HOMEDIR/config/model-surfaces.json" +# stub cursor source: headroom comes from the override command (mirrors the real reader) +cat > "$HOMEDIR/bin/quota-sources/cursor.sh" <<'EOF' +#!/usr/bin/env bash +set -uo pipefail +ov="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/config/quota-overrides.json" +hr=null +if command -v jq >/dev/null 2>&1 && [ -f "$ov" ]; then + cmd=$(jq -r '.cursor // ""' "$ov" 2>/dev/null) + [ -n "$cmd" ] && { o=$(bash -c "$cmd" 2>/dev/null | tr -dc '0-9'); [ -n "$o" ] && hr=$o; } +fi +printf '{"surface":"cursor","status":"logged_in","headroom":%s,"unit":"requests","models":["grok"],"note":"test"}\n' "$hr" +EOF +chmod +x "$HOMEDIR/bin/quota-sources/cursor.sh" +# stub copilot source: same override contract. Mirrors the real one, whose reason for +# existing is that quota-axi's native copilot provider only probes the OLD IDE credential +# path (~/.config/github-copilot/apps.json) and so stays auth_required for a CLI login. +cat > "$HOMEDIR/bin/quota-sources/copilot.sh" <<'EOF' +#!/usr/bin/env bash +set -uo pipefail +ov="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/config/quota-overrides.json" +hr=null +if command -v jq >/dev/null 2>&1 && [ -f "$ov" ]; then + cmd=$(jq -r '.copilot // ""' "$ov" 2>/dev/null) + [ -n "$cmd" ] && { o=$(bash -c "$cmd" 2>/dev/null | tr -dc '0-9'); [ -n "$o" ] && hr=$o; } +fi +printf '{"surface":"copilot","status":"logged_in","headroom":%s,"unit":"premium interactions","models":["claude","gpt"],"note":"test"}\n' "$hr" +EOF +chmod +x "$HOMEDIR/bin/quota-sources/copilot.sh" + +STUB=$(mktemp -d) +cat > "$STUB/quota-axi" <<'EOF' +#!/usr/bin/env bash +cat <<'JSON' +{"providers":[ + {"provider":"grok","source":"oauth","state":{"status":"fresh"},"windows":[{"percentRemaining":2}]}, + {"provider":"cursor","source":"oauth","state":{"status":"fresh"},"windows":[{"percentRemaining":15}]}, + {"provider":"claude","source":"oauth","state":{"status":"fresh"},"windows":[{"percentRemaining":90}]}, + {"provider":"copilot","source":"unavailable","state":{"status":"auth_required"},"windows":[]}, + {"provider":"codex","source":"cli-rpc","state":{"status":"fresh"},"windows":[{"percentRemaining":90}]} +]} +JSON +EOF +chmod +x "$STUB/quota-axi" +export PATH="$STUB:$PATH" +export FM_HOME="$HOMEDIR" +FLEET=$(mktemp -d); export FM_FLEET_DIR="$FLEET" +cd "$REAL" +Q(){ bin/fm-fleet.sh "$@" 2>&1; } + +# cursor's authed reader reports 80 -> supersedes quota-axi's 15 +printf '{"cursor":"echo 80"}\n' > "$HOMEDIR/config/quota-overrides.json" +[ "$(Q pick grok)" = cursor ] && ok "pick grok fails over to cursor when grok drained" || no "pick grok -> $(Q pick grok)" +[ "$(Q pick claude)" = claude ] && ok "pick claude -> claude (has headroom)" || no "pick claude -> $(Q pick claude)" +[ "$(Q pick kimi)" = kimi ] && ok "pick kimi -> kimi (only surface; cline unmonitored)" || no "pick kimi -> $(Q pick kimi)" +b=$(Q pick bogus); printf '%s' "$b" | grep -qi unknown && ok "pick unknown family is flagged" || no "pick bogus not flagged (got: $b)" +Q quota | grep -E '^cursor' | grep -q custom && ok "quota view includes cursor (custom source)" || no "cursor missing from quota view" +Q models | grep -E '^grok' | grep -q cursor && ok "models view: grok reachable via cursor" || no "grok->cursor missing in models view" +Q quota | grep -qE '^cline' && no "cline should be removed from monitoring" || ok "cline is not a monitored surface" +# authed override precedence: cursor reader -> 55 shows as 55% +printf '{"cursor":"echo 55"}\n' > "$HOMEDIR/config/quota-overrides.json" +Q quota | grep -E '^cursor' | grep -q '55%' && ok "authed override: cursor headroom reads 55% from its reader" || no "override not applied ($(Q quota | grep -E '^cursor'))" + +# --- copilot surface (GitHub Copilot CLI) ------------------------------------------- +# Its custom source must SUPERSEDE quota-axi's native auth_required row (the native probe +# reads the old IDE credential path and can never see a standalone CLI login). +printf '{"cursor":"echo 55","copilot":"echo 88"}\n' > "$HOMEDIR/config/quota-overrides.json" +Q quota | grep -E '^copilot' | grep -q '88%' \ + && ok "copilot custom source supersedes quota-axi auth_required row (88%)" \ + || no "copilot row wrong ($(Q quota | grep -E '^copilot'))" +Q models | grep -E '^claude' | grep -q copilot \ + && ok "models view: claude reachable via copilot" \ + || no "claude->copilot missing in models view" +Q models | grep -E '^gpt' | grep -q copilot \ + && ok "models view: gpt reachable via copilot" \ + || no "gpt->copilot missing in models view" + +# Failover INTO copilot: drain claude's native pool, copilot stays healthy. +cat > "$STUB/quota-axi" <<'EOF' +#!/usr/bin/env bash +cat <<'JSON' +{"providers":[ + {"provider":"claude","source":"oauth","state":{"status":"fresh"},"windows":[{"percentRemaining":1}]}, + {"provider":"cursor","source":"oauth","state":{"status":"fresh"},"windows":[{"percentRemaining":15}]}, + {"provider":"copilot","source":"unavailable","state":{"status":"auth_required"},"windows":[]}, + {"provider":"codex","source":"cli-rpc","state":{"status":"fresh"},"windows":[{"percentRemaining":90}]} +]} +JSON +EOF +chmod +x "$STUB/quota-axi" +[ "$(Q pick claude)" = copilot ] \ + && ok "pick claude fails over to copilot when the claude pool is drained" \ + || no "pick claude -> $(Q pick claude) (expected copilot)" + +rm -rf "$STUB" "$FLEET" "$HOMEDIR" +echo "-----" +[ "$fail" -eq 0 ] && echo "ALL PASS ($pass)" || { echo "$fail FAILED"; exit 1; } diff --git a/tests/federation/test_spawn_account.sh b/tests/federation/test_spawn_account.sh new file mode 100644 index 00000000000..46d16322fc3 --- /dev/null +++ b/tests/federation/test_spawn_account.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# --account axis test (Phase 4, Task 11). Run from ~/kun-agent-workspace: +# bash tests/federation/test_spawn_account.sh +# Exercises: launch-command composition for each config-dir isolation method, +# model/effort folding, api-key refusal (no secret on argv), unknown-account +# refusal, and the wrapper handing the composed command to fm-spawn (stubbed). +set -uo pipefail +cd "$(dirname "$0")/../.." || exit 2 +FM_HOME="$(pwd)"; export FM_HOME +# shellcheck source=bin/fm-account-env.sh disable=SC1091 +. bin/fm-account-env.sh +fails=0 +ok(){ echo "PASS: $1"; } +bad(){ echo "FAIL: $1"; fails=$((fails+1)); } + +TMP=$(mktemp -d); export FM_ACCOUNTS_FILE="$TMP/accounts.json" +CD="$TMP/cd"; mkdir -p "$CD" +echo "sk-fake-not-a-real-key" > "$TMP/grok.key" + +cat > "$FM_ACCOUNTS_FILE" </dev/null); rc=$? +{ [ "$rc" -eq 2 ] && [ -z "$out" ]; } && ok "api-key compose refused (no key on argv)" || bad "api-key refusal (rc=$rc out='$out')" + +# 6. unknown account refused +fm_account_compose_launch nope >/dev/null 2>&1 && bad "unknown account composed" || ok "unknown account refused" + +# 7. wrapper hands composed command to fm-spawn (stub captures argv) +STUB="$TMP/spawn-stub.sh" +cat > "$STUB" <<'S' +#!/usr/bin/env bash +: > "$FM_STUB_OUT" +for a in "$@"; do printf '%s\n' "$a" >> "$FM_STUB_OUT"; done +S +chmod +x "$STUB" +FM_STUB_OUT="$TMP/out.txt" FM_SPAWN_BIN="$STUB" \ + bash bin/fm-spawn-acct.sh T-1 /proj --account claude-alt --model opus >/dev/null 2>&1 +n=$(wc -l < "$TMP/out.txt") +a1=$(sed -n '1p' "$TMP/out.txt"); a2=$(sed -n '2p' "$TMP/out.txt"); a3=$(sed -n '3p' "$TMP/out.txt") +{ [ "$n" -eq 3 ] && [ "$a1" = "T-1" ] && [ "$a2" = "/proj" ] && [ "$a3" = "CLAUDE_CONFIG_DIR=$CD claude --model opus" ]; } \ + && ok "wrapper passes (id, dir, composed-launch) to fm-spawn" || bad "wrapper passthrough (n=$n a1='$a1' a2='$a2' a3='$a3')" + +# 8. wrapper refuses api-key account (fail-closed; stub NOT invoked) +: > "$TMP/out2.txt" +FM_STUB_OUT="$TMP/out2.txt" FM_SPAWN_BIN="$STUB" \ + bash bin/fm-spawn-acct.sh T-2 /proj --account grok-x >/dev/null 2>&1; rc=$? +{ [ "$rc" -ne 0 ] && [ ! -s "$TMP/out2.txt" ]; } && ok "wrapper fail-closed on api-key account" || bad "wrapper api-key (rc=$rc, stub-called=$( [ -s "$TMP/out2.txt" ] && echo yes || echo no ))" + +# 9. apply_env exports in the CALLER's shell (regression: must NOT be a subshell) +( unset CLAUDE_CONFIG_DIR; fm_account_apply_env claude-alt && [ "$CLAUDE_CONFIG_DIR" = "$CD" ] ) \ + && ok "apply_env exports config-dir-env in caller shell" || bad "apply_env export (subshell regression)" + +# 10. config-dir-flag sets FM_ACCT_ARGV_SUFFIX (not stdout) +( fm_account_apply_env cline-x && [ "$FM_ACCT_ARGV_SUFFIX" = "--config $CD" ] ) \ + && ok "apply_env sets argv suffix for flag method" || bad "apply_env suffix" + +rm -rf "$TMP" +echo "-----"; [ "$fails" -eq 0 ] && { echo "ALL PASS"; exit 0; } || { echo "$fails FAILURE(S)"; exit 1; }