diff --git a/.agents/skills/quota-array-dispatch/SKILL.md b/.agents/skills/quota-array-dispatch/SKILL.md index 56b8a7ba5a4..0b336fb530a 100644 --- a/.agents/skills/quota-array-dispatch/SKILL.md +++ b/.agents/skills/quota-array-dispatch/SKILL.md @@ -59,8 +59,11 @@ Grok prepaid `credits` are unrelated to paid-window headroom; never read them as The selector is the mechanical owner of dispatch capacity and of ranking among remaining eligible candidates. It does not replace reasoning-class fit: keep only candidates that meet the required reasoning class before passing the set, and never use `spendPriority` or remaining quota to silently replace that class. When every remaining candidate is tight, dispatch inside the strongest-reasoning class if one of those candidates can proceed, or stop and report that the strongest-class choice cannot proceed rather than downgrading it to spend or conserve quota. +That rule governs this selection, which is the initial dispatch decision. +The separate in-run `modelFallback` response to a model that depletes after dispatch (`AGENTS.md` section 4) walks the configured chain for the class already dispatched, so it never re-opens class choice. +An exhausted chain for the required class stops and reports there too, rather than relaunching beneath that class. -Providers exposed by quota-axi, including Claude, Codex, Grok, and Cursor, require fresh telemetry within the configured maximum age and a tightest live percentage strictly above `reservePercent`. +Providers exposed by quota-axi, including Claude, Codex, Grok, Cursor, and agy, require fresh telemetry within the configured maximum age and a tightest live percentage strictly above `reservePercent`. Stale, unavailable, malformed, or windowless telemetry makes that provider ineligible for a new dispatch. A provider whose pools are billed separately would be priced by its worst pool under that rule, so a profile may declare the one window it draws on with `quotaWindow`; `docs/configuration.md` owns that field's semantics. Confirm the declared window against the provider's live telemetry before relying on it, because a declared window the telemetry does not carry blocks that candidate rather than repricing it. @@ -84,7 +87,8 @@ Apply only among candidates satisfying required fit and strongest reasoning clas 2. Pass that exact object or array to `FM_HOME= bin/fm-dispatch-select.mjs select`. 3. Read its sanitized per-provider diagnostics and selected JSON profile. 4. Pass the selected `harness`, `provider`, `model`, and `effort` axes to `fm-spawn.sh`; it records `provider` as routing evidence without forwarding it to the harness CLI. - Its `--provider` accepts `claude`, `codex`, and `grok` only, so a selected profile whose provider is native to its harness and outside that set, such as `cursor`, is spawned without the redundant flag; the recorded harness still establishes that provider for a later `record-failure`. + Its `--provider` accepts every routable provider, including `cursor` and `agy`, and a native harness refuses any provider but its own; `docs/configuration.md` owns which adapters are native. + Omitting the field on a native harness is equally safe, because the recorded harness alone establishes that provider for a later `record-failure`. 5. If it exits 3, stop and report that no candidate has current dispatch-capacity evidence rather than choosing manually around the reserve, cooldown, or telemetry refusal. 6. If a running task with recorded routing-provider metadata records provider rate-limit or quota-exhaustion evidence in its status log, run `fm-dispatch-select.mjs record-failure --provider --task ` before retrying the candidate set. 7. Use `clear --provider ` only after the credential or provider condition is known to be corrected; it clears the cooldown, not dispatch history. diff --git a/AGENTS.md b/AGENTS.md index abc48e884d8..80480bcef9e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -187,8 +187,7 @@ A silent bootstrap section needs no action; for any printed actionable diagnosti ## 4. Harness and runtime dispatch Load `harness-adapters` before every spawn or recovery and before trust handling, skill invocation, interrupt, exit, resume, or adapter verification. -The verified harnesses are `claude`, `codex`, `opencode`, `pi`, `pi-signed`, `grok`, `kimi`, `cline`, `cursor-agent`, `cursor`, and `copilot`, plus `muse` for crewmates and scouts only and `agy` for crewmate launches only; never dispatch on an unverified adapter. -`agy` is verified for crewmate launches only, and a `--secondmate` spawn refuses it. +The verified harnesses are `claude`, `codex`, `opencode`, `pi`, `pi-signed`, `grok`, `kimi`, `cursor-agent`, and `cursor`, plus `muse`, `agy`, `cline`, and `copilot` for crewmate and scout launches only; never dispatch on an unverified adapter, and never select one of those four for a secondmate (`docs/configuration.md` "Harness support" owns the per-kind verified set). If static `config/crew-harness` or `config/secondmate-harness` names an unverified adapter, report it and fall back only to a verified adapter rather than launching it. `docs/configuration.md` owns dispatch-profile and runtime-backend schemas, `bin/fm-harness.sh` owns static resolution, and `bin/fm-spawn.sh` owns launch flags and fail-closed validation. @@ -209,7 +208,15 @@ Do not add model-specific versions of that policy. `secondmate-provisioning` owns secondmate harness pins and inherited local material, while `harness-adapters` owns the harness consequences. Dispatch only on a backend that `fm-spawn` validates as spawn-capable; pass an explicit per-spawn `--backend` only under that exact task's own authority, never as later-task precedent (selection contract: [`docs/configuration.md`](docs/configuration.md) "Runtime backend"). A missing dependency, authentication failure, unsupported backend, or version refusal is a blocker; never silently retry on another backend. -When an active ship or scout session is blocked due to token/quota exhaustion or harness limits, Firstmate may adopt and relaunch the blocked session in place using `bin/fm-runtime-handoff.sh --harness [--model ] [--effort ] [--progress-note ]`. This cleanly exits the blocked agent, preserves the existing worktree, lease, PR metadata, and work-in-progress without loss, and relaunches the replacement agent in the same worktree to continue execution seamlessly. +When an active ship or scout session is blocked due to token/quota exhaustion or harness limits, Firstmate may adopt and relaunch the blocked session in place using `bin/fm-runtime-handoff.sh --harness [--model ] [--effort ] [--progress-note ]`. +This cleanly exits the blocked agent, preserves the existing worktree, lease, PR metadata, and work-in-progress without loss, and relaunches the replacement agent in the same worktree to continue execution seamlessly. +When a worker's model depletes mid-run, switch models within the same harness automatically (relaunch in place via `bin/fm-runtime-handoff.sh` with `--model`) instead of blocking, parking, or escalating a routine depletion. +Read model fallback chains from `config/crew-dispatch.json` `modelFallback` (legacy alias `_model_fallback`) without hardcoding a duplicate copy, and move work to the next harness lane only when a harness's whole model chain is exhausted. +Depletion detection for a provider quota-axi exposes is the recorded `record-failure` telemetry contract above; live 429, limit, or quota errors in the pane or status log are the trigger of record only for runtimes without quota telemetry (such as ClinePass). +Every automatic model switch must be logged and visible in status reporting rather than silently downgrading reasoning class. +The fail-closed capacity contract (reserve, cooldown, and telemetry freshness) remains enforced. +The strongest-reasoning-class rule governs which candidate is dispatched in the first place, so it is never traded away to conserve quota at selection time; model fallback is the separate in-run response to a model that depleted after dispatch, and it walks the configured chain rather than choosing a class. +If the chain for the required class is exhausted, stop and report that the strongest-class choice cannot proceed rather than relaunching beneath it. ## 5. Recovery diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index efb8adb4582..e3bfc42afad 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -1029,6 +1029,18 @@ crew_dispatch_validate() { or ($items | any(has("quotaWindow") and (((.quotaWindow | type) != "string") or (.quotaWindow | length) == 0))); def malformed_provider($items): ($items | any(has("provider") and (((.provider | type) != "string") or (.provider | length) == 0))); + def model_fallback: (.modelFallback // ._model_fallback); + def bad_fallback_harnesses: + (model_fallback // {}) | keys | map(select(. as $h | verified($h) | not)) | unique; + def bad_fallback_chains: + (model_fallback // {}) + | to_entries + | map(select( + ((.value | type) != "array") + or ((.value | length) == 0) + or (.value | any((type != "string") or (length == 0))))) + | map(.key) + | unique; def routing_setting_ok($key; $value): if ($value | type) != "number" or ($value | floor) != $value then false elif $key == "reservePercent" then $value >= 0 and $value <= 99 @@ -1050,6 +1062,10 @@ crew_dispatch_validate() { "subscriptionRouting has unknown field: " + ([.subscriptionRouting | keys[] | . as $key | select((["reservePercent","telemetryMaxAgeSeconds","cooldownSeconds"] | index($key)) == null)] | sort | join(", ")) elif has("subscriptionRouting") and ([.subscriptionRouting | to_entries[] | select(. as $entry | routing_setting_ok($entry.key; $entry.value) | not)] | length) > 0 then "subscriptionRouting setting is out of range: " + ([.subscriptionRouting | to_entries[] | select(. as $entry | routing_setting_ok($entry.key; $entry.value) | not) | .key] | sort | join(", ")) + elif has("modelFallback") and has("_model_fallback") then "modelFallback and its legacy alias _model_fallback cannot both be declared" + elif (has("modelFallback") or has("_model_fallback")) and (model_fallback | type) != "object" then "modelFallback must be an object mapping a harness to its ordered model chain" + elif (bad_fallback_harnesses | length) > 0 then "modelFallback has an unverified harness: " + (bad_fallback_harnesses | join(", ")) + elif (bad_fallback_chains | length) > 0 then "modelFallback chain must be a non-empty array of non-empty model ids: " + (bad_fallback_chains | join(", ")) elif has("rules") and (.rules | type) != "array" then "rules must be an array" elif [(.rules // [])[]? | select(type != "object")] | length > 0 then "each rule must be an object" elif [(.rules // [])[]? | select((.when? | type) != "string" or (.when | length) == 0)] | length > 0 then "each rule needs non-empty when" @@ -1074,9 +1090,9 @@ crew_dispatch_validate() { | map(select(. != null)) | map(select(. as $h | verified($h) | not)) | unique) as $bad_harnesses - | (configured_profiles | map(.provider? // empty) | map(. as $provider | select((["claude","codex","grok"] | index($provider)) == null)) | unique) as $bad_providers + | (configured_profiles | map(.provider? // empty) | map(. as $provider | select((["claude","codex","grok","cursor","agy"] | index($provider)) == null)) | unique) as $bad_providers | (configured_profiles | map(select(.harness == "kimi" or .provider == "kimi")) | length) as $bad_kimi_routes - | (configured_profiles | map(select((.harness == "claude" or .harness == "codex" or .harness == "grok") and .provider? != null and .provider != .harness) | "\(.harness):\(.provider)") | unique) as $mismatched_native_providers + | (configured_profiles | map(select((.harness == "claude" or .harness == "codex" or .harness == "grok" or .harness == "cursor" or .harness == "agy") and .provider? != null and .provider != .harness) | "\(.harness):\(.provider)") | unique) as $mismatched_native_providers | if ($bad_harnesses | length) > 0 then "unverified harness: " + ($bad_harnesses | join(", ")) elif $bad_kimi_routes > 0 then "Kimi is unsupported for subscription dispatch" elif ($mismatched_native_providers | length) > 0 then "native harness/provider mismatch: " + ($mismatched_native_providers | join(", ")) diff --git a/bin/fm-control-lib.sh b/bin/fm-control-lib.sh index 008c340cd64..f2b9bb8f8df 100644 --- a/bin/fm-control-lib.sh +++ b/bin/fm-control-lib.sh @@ -63,7 +63,7 @@ fm_control_verb_allowed() { # # than guessed at, exactly as a spawn on it would be. fm_control_harness_supported() { # case "${1-}" in - claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|muse|cline) return 0 ;; + claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|muse|cline|copilot|agy) return 0 ;; esac return 1 } @@ -88,13 +88,15 @@ fm_control_harness_family() { # cursor*) printf 'cursor' ;; muse*) printf 'muse' ;; cline*) printf 'cline' ;; + copilot*) printf 'copilot' ;; + agy*) printf 'agy' ;; *) return 1 ;; esac } -# Which task kinds an adapter is verified to run. muse and cline are -# crewmate/scout adapters only: neither has a primary supervision protocol, and -# bin/fm-spawn.sh refuses a --secondmate launch on either. The control plane +# Which task kinds an adapter is verified to run. muse, cline, copilot, and agy +# are crewmate/scout adapters only: none has a primary supervision protocol, and +# bin/fm-spawn.sh refuses a --secondmate launch on any of them. The control plane # asks this BEFORE it stops anything, so an incompatible relaunch target is # refused while the current agent is still running rather than after it has # been stopped. @@ -102,7 +104,7 @@ fm_control_harness_supports_kind() { # local harness=${1-} kind=${2-} fm_control_harness_supported "$harness" || return 1 case "$harness" in - muse|cline) [ "$kind" != secondmate ] || return 1 ;; + muse|cline|copilot|agy) [ "$kind" != secondmate ] || return 1 ;; esac return 0 } @@ -114,8 +116,8 @@ fm_control_harness_supports_kind() { # # borrowing grok's interrupt key here would stop the agent instead of its turn. fm_control_interrupt_key() { # case "${1-}" in - claude|codex|opencode|pi|pi-signed|kimi|cursor|muse|cline) printf 'Escape' ;; - grok) printf 'C-c' ;; + claude|codex|opencode|pi|pi-signed|kimi|cursor|muse|cline|agy) printf 'Escape' ;; + grok|copilot) printf 'C-c' ;; *) return 1 ;; esac } @@ -125,7 +127,7 @@ fm_control_interrupt_key() { # fm_control_interrupt_repeat() { # case "${1-}" in opencode) printf '2' ;; - claude|codex|pi|pi-signed|grok|kimi|cursor|muse|cline) printf '1' ;; + claude|codex|pi|pi-signed|grok|kimi|cursor|muse|cline|copilot|agy) printf '1' ;; *) return 1 ;; esac } @@ -143,7 +145,7 @@ fm_control_interrupt_repeat() { # fm_control_interrupt_clear_key() { # case "${1-}" in muse) printf 'C-u' ;; - claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|cline) ;; + claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|cline|copilot|agy) ;; *) return 1 ;; esac } @@ -155,7 +157,7 @@ fm_control_interrupt_ack_source() { # # after an interrupt was measured as variable - sometimes seconds, sometimes # not within 20 - so a cancellation claim built on it would be unreliable. # Normal turn completion is prompt, which is what the busy fold depends on. - claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|cline) printf 'none' ;; + claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|cline|copilot|agy) printf 'none' ;; *) return 1 ;; esac } @@ -171,7 +173,7 @@ fm_control_interrupt_ack_source() { # # nothing for an adapter that exits on a key instead. fm_control_exit_command() { # case "${1-}" in - claude|opencode|grok|kimi|cursor|muse) printf '/exit' ;; + claude|opencode|grok|kimi|cursor|muse|copilot|agy) printf '/exit' ;; codex|pi|pi-signed) printf '/quit' ;; cline) ;; *) return 1 ;; @@ -186,7 +188,7 @@ fm_control_exit_command() { # fm_control_exit_key() { # case "${1-}" in cline) printf 'C-c' ;; - claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|muse) ;; + claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|muse|copilot|agy) ;; *) return 1 ;; esac } diff --git a/bin/fm-dispatch-select.mjs b/bin/fm-dispatch-select.mjs index db44b795140..74b5b673947 100755 --- a/bin/fm-dispatch-select.mjs +++ b/bin/fm-dispatch-select.mjs @@ -3,8 +3,8 @@ // // Usage: // fm-dispatch-select.mjs select [--quota-json ] [--now ] [] -// fm-dispatch-select.mjs record-failure --provider --task [--now ] -// fm-dispatch-select.mjs clear --provider +// fm-dispatch-select.mjs record-failure --provider --task [--now ] +// fm-dispatch-select.mjs clear --provider // // `select` accepts a full rule object with `use`, one profile object, or a // non-empty profile array on the command line or stdin. It prints exactly one @@ -15,14 +15,13 @@ // model support discovery, and provider identity for non-native adapters. This // script owns only subscription readiness and deterministic distribution: // -// - Native claude, codex, and grok profiles resolve to their same-named provider. -// Other harnesses need an explicit `provider` field. -// - Providers exposed by quota-axi, including Claude, Codex, and Grok, require -// fresh telemetry no older than the configured maximum. Their tightest -// reported live percentage must remain strictly above the configured reserve. -// Stale, absent, malformed, or -// windowless telemetry makes that provider ineligible; it never falls back to -// an unmetered guess. +// - Native claude, codex, grok, cursor, and agy profiles resolve to their +// same-named provider. Other harnesses need an explicit `provider` field. +// - Providers exposed by quota-axi, including Claude, Codex, Grok, Cursor, and +// agy, require fresh telemetry no older than the configured maximum. Their +// tightest reported live percentage must remain strictly above the +// configured reserve. Stale, absent, malformed, or windowless telemetry makes +// that provider ineligible; it never falls back to an unmetered guess. // - A profile may declare the one quota window it is actually drawn from with // an optional `quotaWindow` field naming a `windows[].id` in that provider's // telemetry. The candidate is then priced on that window alone instead of the @@ -39,6 +38,11 @@ // - Rate-limit or quota-exhaustion evidence creates a provider cooldown. // `record-failure` verifies the evidence in the named task's status file and // verifies that task's recorded routing provider before changing state. +// Evidence must read in subscription vocabulary - a framed 429, an explicit +// rate limit, or a named quota/credit/allowance being exhausted, depleted, +// or reached. A context-window or tool-output ceiling is an ordinary working +// state, so a bare `limit`, `token`, or unframed `429` is refused rather +// than parking the provider for a whole cooldown. // - Among eligible candidates, a known spendPriority from quota-axi is the // quota-perspective ranker: the highest known scalar wins. When every // remaining eligible candidate lacks a known scalar, or when known scalars @@ -51,6 +55,7 @@ // // A profile priced on its own pool looks like this: // { "harness": "cursor", "model": "cursor-grok-4.6-high", "quotaWindow": "auto_usage" } +// { "harness": "agy", "model": "gemini-3.7-flash-high", "quotaWindow": "gemini_5h" } // // config/crew-dispatch.json may contain this optional settings object: // "subscriptionRouting": { @@ -74,14 +79,13 @@ import path from 'node:path'; import { spawnSync } from 'node:child_process'; // Routable providers must have a credit-identity in quota-axi so the selector -// can test capacity and redirect. quota-axi reports claude, codex, grok AND -// cursor (its `cursor` provider is the Cursor subscription). cline is -// deliberately absent: it is BYO-API-key with no subscription window quota-axi -// can read, so it stays spawn-only (a single non-array profile), never a -// credit-routed candidate. Same for pi/opencode. copilot has telemetry too and -// could be added the same way if wanted. -const PROVIDERS = new Set(['claude', 'codex', 'grok', 'cursor']); -const VERIFIED_HARNESSES = new Set(['claude', 'codex', 'opencode', 'pi', 'pi-signed', 'grok', 'kimi', 'cline', 'cursor', 'copilot']); +// can test capacity and redirect. quota-axi reports claude, codex, grok, +// cursor, and agy. cline is deliberately absent: it is BYO-API-key with no +// subscription window quota-axi can read, so it stays spawn-only (a single +// non-array profile), never a credit-routed candidate. Same for pi/opencode. +// copilot has telemetry too and could be added the same way if wanted. +const PROVIDERS = new Set(['claude', 'codex', 'grok', 'cursor', 'agy']); +const VERIFIED_HARNESSES = new Set(['claude', 'codex', 'opencode', 'pi', 'pi-signed', 'grok', 'kimi', 'cursor', 'muse', 'agy', 'cline', 'copilot']); const NATIVE_PROVIDER = new Map([ ['claude', 'claude'], ['codex', 'codex'], @@ -89,6 +93,8 @@ const NATIVE_PROVIDER = new Map([ // The cursor harness draws on the Cursor subscription, which quota-axi // reports under the provider name `cursor`. ['cursor', 'cursor'], + // agy draws on the Google AI subscription, reported as provider `agy`. + ['agy', 'agy'], ]); const DEFAULTS = Object.freeze({ reservePercent: 20, @@ -100,7 +106,21 @@ const LIMITS = Object.freeze({ telemetryMaxAgeSeconds: [1, 3600], cooldownSeconds: [60, 86400], }); -const RATE_LIMIT_RE = /rate[ _-]?limit|too many requests|(?:quota|usage)[^\n]{0,80}(?:exhaust|limit|deplet)|(?:exhaust|deplet)[^\n]{0,80}(?:quota|usage)/i; +// Keep this narrow: both consumers of this gate park a provider for a whole +// cooldown on a single match, so an ordinary working state ("context token +// limit reached", "exceeded the tool output limit") must not reach it. The +// subscription-vocabulary contract these alternatives encode is stated in the +// evidence bullet of the header help above. +const RATE_LIMIT_RE = new RegExp([ + '(?:http|status|code|error|response)[^\\n]{0,16}\\b429\\b', + 'rate[ _-]?limit', + 'too many requests', + 'resource[ _-]?exhausted', + 'insufficient[ _-]?(?:quota|credits?|balance|funds)', + 'out of (?:quota|credits?|tokens?|balance)', + '(?:quota|usage|spending|allowance|subscription|credits?|balance|monthly|weekly|daily|session)[^\\n]{0,80}(?:exhaust|deplet|used up|limit|reach|exceed|zero)', + '(?:exhaust|deplet|reach|exceed)[^\\n]{0,80}(?:quota|usage|spending|allowance|credits?|balance)', +].join('|'), 'i'); class CliError extends Error { constructor(message, code = 2) { @@ -580,7 +600,7 @@ function taskMetaProvider(paths, task) { })); const harness = entries.get('harness'); const provider = entries.get('provider') || NATIVE_PROVIDER.get(harness); - if (!provider || !PROVIDERS.has(provider)) die('record-failure requires a recorded claude, codex, or grok routing provider'); + if (!provider || !PROVIDERS.has(provider)) die('record-failure requires a recorded claude, codex, grok, cursor, or agy routing provider'); const nativeProvider = NATIVE_PROVIDER.get(harness); if (nativeProvider && provider !== nativeProvider) die(`task ${task} has mismatched native harness and provider metadata`); if (harness === 'kimi') die('record-failure does not support Kimi tasks'); @@ -589,7 +609,7 @@ function taskMetaProvider(paths, task) { } function recordFailure(options, paths, settings, now, state) { - if (!options.provider || !PROVIDERS.has(options.provider)) die('record-failure needs --provider claude, codex, or grok'); + if (!options.provider || !PROVIDERS.has(options.provider)) die('record-failure needs --provider claude, codex, grok, cursor, or agy'); if (!options.task) die('record-failure needs --task'); const actual = taskMetaProvider(paths, options.task); if (actual !== options.provider) die(`task ${options.task} is recorded on provider ${actual}, not ${options.provider}`); @@ -599,7 +619,7 @@ function recordFailure(options, paths, settings, now, state) { } function clearProvider(options, paths, state) { - if (!options.provider || !PROVIDERS.has(options.provider)) die('clear needs --provider claude, codex, or grok'); + if (!options.provider || !PROVIDERS.has(options.provider)) die('clear needs --provider claude, codex, grok, cursor, or agy'); delete state.cooldowns[options.provider]; saveState(paths.stateFile, state); log(`provider=${options.provider} cooldown cleared`); diff --git a/bin/fm-runtime-handoff.sh b/bin/fm-runtime-handoff.sh index a3ac7e9b345..d7abd38c46d 100755 --- a/bin/fm-runtime-handoff.sh +++ b/bin/fm-runtime-handoff.sh @@ -191,7 +191,7 @@ fi # Verified harness only: reuse spawn's launch_template gate by requiring a known name. case "$HARNESS" in - claude|codex|opencode|pi|pi-signed|grok|kimi) ;; + claude|codex|opencode|pi|pi-signed|grok|kimi|cursor|muse|cline|copilot|agy) ;; *) echo "error: target harness '$HARNESS' is not a verified adapter; refuse rather than launching it" >&2 exit 1 @@ -209,8 +209,9 @@ fi # Exit command facts from harness-adapters (do not invent adapters here). handoff_exit_spec() { # -> prints "text:" or "key:" case "$1" in - claude|opencode|grok|kimi) printf 'text:/exit\n' ;; + claude|opencode|grok|kimi|cursor|muse|copilot|agy) printf 'text:/exit\n' ;; codex|pi|pi-signed) printf 'text:/quit\n' ;; + cline) printf 'key:C-c\n' ;; *) return 1 ;; esac } diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 7edffe4fc8d..769f64ebaa6 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -380,8 +380,8 @@ case "$EFFORT" in *) echo "error: --effort must be one of low, medium, high, xhigh, max" >&2; exit 1 ;; esac case "$PROVIDER" in - ''|claude|codex|grok) ;; - *) echo "error: --provider must be one of claude, codex, grok" >&2; exit 1 ;; + ''|claude|codex|grok|cursor|agy) ;; + *) echo "error: --provider must be one of claude, codex, grok, cursor, agy" >&2; exit 1 ;; esac if [ "$REUSE_WORKTREE" = 1 ]; then if [ "$KIND" = secondmate ]; then @@ -1418,7 +1418,7 @@ case "$HARNESS" in kimi) [ -z "$PROVIDER" ] || { echo "error: Kimi cannot carry a subscription routing provider" >&2; exit 1; } ;; - claude|codex|grok) + claude|codex|grok|cursor|agy) [ -z "$PROVIDER" ] || [ "$PROVIDER" = "$HARNESS" ] || { echo "error: native harness $HARNESS requires provider $HARNESS" >&2; exit 1; } ;; esac diff --git a/docs/architecture.md b/docs/architecture.md index e0bb724ed56..9ab9ea564cf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -195,6 +195,7 @@ When the file exists, `fm-spawn.sh` refuses crewmate and scout launches without Secondmate launches are exempt because they resolve the secondmate harness and any optional secondmate model or effort tokens instead. Unsupported effort values are still recorded in task meta when passed to `fm-spawn.sh`, but the launch template omits any effort flag that the selected harness does not accept. After Firstmate filters a matched array for task fit and reasoning class, `fm-dispatch-select.mjs` uses fresh metered evidence, excludes provider cooldowns, and rotates eligible subscriptions through private home-local state. +That selector runs only at dispatch; a model that depletes after dispatch is answered by the separate `modelFallback` chain walked through `bin/fm-runtime-handoff.sh` in place, whose schema and boundary are owned by [`docs/configuration.md`](configuration.md). Static and explicit dispatch keep spawn launch compatible across claude, codex, opencode, pi, pi-signed, grok, kimi, cursor, muse, cline, copilot, and agy, while automatic subscription arrays exclude Kimi and preserve the selected profile for later audit. agy is the one exception to plain omission: its ceiling is `high`, so `xhigh` and `max` are resolved against the requested model id rather than simply dropped. A base model id refuses to launch without `--effort`, so those tiers clamp down to `--effort high`; an id that already bakes a `-low`, `-medium`, or `-high` suffix launches on its own and rejects a non-matching `--effort`, so the flag is withheld and the baked tier stands. diff --git a/docs/configuration.md b/docs/configuration.md index 64a7717407c..be1df41a863 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -259,13 +259,12 @@ The full cmux home label also includes a short hash of the resolved `FM_ROOT` pa ## Harness support -claude, codex, opencode, pi, pi-signed, grok, kimi, and cursor are empirically verified for crewmate and secondmate launches; cline and copilot are verified for crewmate launches only and are not yet wired for secondmate use; [README requirements](../README.md#requirements) own the set supported for the primary session. +claude, codex, opencode, pi, pi-signed, grok, kimi, and cursor are empirically verified for crewmate and secondmate launches; muse, agy, cline, and copilot are verified for crewmate and scout launches only, and `fm-spawn.sh` refuses each of them for a `--secondmate` spawn because none of them can arm a primary supervision protocol; [README requirements](../README.md#requirements) own the set supported for the primary session. A cursor secondmate or primary runs the tracked project-scope `.cursor/hooks.json` in its own home and must be launched with `--trust`, or no project hook loads; [`docs/supervision-protocols/cursor.md`](supervision-protocols/cursor.md) owns its supervision protocol. Cursor delivery confirmation is verified on tmux and Herdr only. On Zellij, cmux, and Orca a Cursor steer lands, but `fm-send` reports delivery unconfirmed and exits non-zero because their shared submit core does not consult the busy footer; [runtime backend verification](verification/runtime-backends.md#cursor-agent-cli) owns the evidence and transcript-state boundary. -muse is verified for crewmate and scout launches ONLY, and `fm-spawn.sh` refuses it for a secondmate, because muse ships no usable hook surface for a primary session's turn-end supervision; [`docs/verification/muse.md`](verification/muse.md) owns that evidence. +muse's secondmate refusal is specifically because it ships no usable hook surface for a primary session's turn-end supervision; [`docs/verification/muse.md`](verification/muse.md) owns that evidence. muse also needs a worker-reachable credential before spawning, and the portable fleet path is the `/muse/auth.json` credential stored by `muse login`, because a caller-only `META_API_KEY` does not cross a long-lived backend daemon. -claude, codex, opencode, pi, pi-signed, grok, and kimi are empirically verified for crewmate and secondmate launches; agy is verified for crewmate launches only, and a `--secondmate` spawn refuses it; [README requirements](../README.md#requirements) own the set supported for the primary session. New harnesses get verified through a supervised trial task before joining the set. The verified adapter knowledge - each harness's busy-state source, interrupt and exit commands, skill-invocation syntax, and per-harness quirks - lives in [`.agents/skills/harness-adapters/SKILL.md`](../.agents/skills/harness-adapters/SKILL.md). Launch mechanics, including the verified command templates, live in [`bin/fm-spawn.sh`](../bin/fm-spawn.sh). @@ -317,14 +316,17 @@ This section is the single owner of the canonical schema and its per-field seman { "when": "", "use": [ - { "harness": "", "provider": "", "model": "", "effort": "", "quotaWindow": "" } + { "harness": "", "provider": "", "model": "", "effort": "", "quotaWindow": "" } ], "why": "" } ], "default": [ { "harness": "", "provider": "", "model": "", "effort": "", "quotaWindow": "" } - ] + ], + "modelFallback": { + "": ["", "", ""] + } } ``` @@ -337,7 +339,7 @@ An omitted model or effort means the selected harness uses its own default for t Declaring it is the only way to price a candidate on a single window; no mapping from model name to pool is inferred, because a declaration in config is checkable and correctable while an inferred one silently rots. An omitted `quotaWindow` keeps the conservative provider-wide minimum, and a declared window that the live telemetry does not carry makes that candidate ineligible rather than falling back to a rosier figure. Read the current window ids from `quota-axi --json`, and the current model ids from `bin/fm-model-refresh.sh`, before writing either field. -Native `claude`, `codex`, `grok`, and `cursor` profiles establish the same-named provider without a redundant field; for `claude`, `codex`, and `grok` a provider that is present must match the harness. +Native `claude`, `codex`, `grok`, `cursor`, and `agy` profiles establish the same-named provider without a redundant field; for `claude`, `codex`, `grok`, `cursor`, and `agy` a provider that is present must match the harness. A non-native adapter needs an explicit provider when it participates in subscription-aware selection, because model spelling does not establish account identity. Kimi 0.29.1 is rejected from subscription-aware profiles because its guarded Herdr lifecycle exit was not deterministic after interrupt; no other Moonshot route is substituted. Every profile array is an implicit subscription-aware choice resolved through `quota-array-dispatch` and `bin/fm-dispatch-select.mjs` after firstmate removes candidates that do not meet task fit or the strongest required reasoning class. @@ -346,7 +348,7 @@ If a selected profile carries an effort value the chosen harness does not accept See [`docs/examples/crew-dispatch.json`](examples/crew-dispatch.json) for a starting point to copy into local `config/crew-dispatch.json`. When the file exists, bootstrap validates it with `jq`. Valid files stay silent by default; with `FM_BOOTSTRAP_VERBOSE_FACTS=1`, bootstrap emits `BOOTSTRAP_INFO: crew dispatch active config/crew-dispatch.json`, one `BOOTSTRAP_INFO:` fact per rule, and one fact for the optional default profile set. -Malformed JSON, an empty or malformed rule/default array, an unverified harness, an unsupported provider relationship, an invalid subscription setting, or an effort value unsupported by that harness is reported as `CREW_DISPATCH: invalid config/crew-dispatch.json - ...`; missing `jq` is reported through the normal `MISSING: jq` install-consent flow. +Malformed JSON, an empty or malformed rule/default array, an unverified harness, an unsupported provider relationship, an invalid subscription setting, a malformed model fallback chain, or an effort value unsupported by that harness is reported as `CREW_DISPATCH: invalid config/crew-dispatch.json - ...`; missing `jq` is reported through the normal `MISSING: jq` install-consent flow. While the file remains present, no crewmate or scout spawn may proceed without an explicit resolved harness; malformed configuration must be reported and corrected rather than selected around. Secondmate homes inherit this file from the primary, so a secondmate's own crewmates apply the same dispatch profile behavior. @@ -356,9 +358,17 @@ Secondmate homes inherit this file from the primary, so a secondmate's own crewm `cooldownSeconds` is an integer from 60 through 86400 and defaults to 1800. Unknown fields fail bootstrap validation instead of being ignored. -Providers exposed by quota-axi, including Claude, Codex, Grok, and Cursor, require fresh telemetry with a usable percentage above the reserve. +Providers exposed by quota-axi, including Claude, Codex, Grok, Cursor, and agy, require fresh telemetry with a usable percentage above the reserve. The selector ranks remaining eligible candidates by a known `spendPriority` scalar when quota-axi publishes one, then persists rotation and cooldown in private `state/.dispatch-routing.json` through a serialized atomic update. Verified rate-limit or quota-exhaustion evidence from a task carrying recorded routing-provider metadata can be recorded with `bin/fm-dispatch-select.mjs record-failure`; exact flags, evidence checks, exit codes, and clear behavior are owned by the script's help. +When an already-dispatched worker's model depletes, firstmate switches models within the same harness automatically (relaunch in place via `bin/fm-runtime-handoff.sh --harness --model `) following the configured `modelFallback` chain before moving to the next harness. + +`modelFallback` is an optional top-level object mapping each harness name to its own ordered model chain, strongest entry first. +`_model_fallback` is accepted as a legacy alias for the same object; a file declaring both is invalid, because two chains for one harness cannot both be authoritative. +Every key must be a harness verified for dispatch under the same rule as a profile `harness`, and every value must be a non-empty array of non-empty model-id strings; a malformed chain is reported as a `CREW_DISPATCH` diagnostic rather than silently ignored. +Read the current model ids from `bin/fm-model-refresh.sh` before writing a chain, because an id the harness does not accept fails at launch instead of being repriced. +Firstmate consults this object only after dispatch, when a running task's model depletes: it relaunches in place on the entry after the model recorded for that task, and moves to the next harness lane only once the chain's last entry is reached. +It is never consulted at selection time, so it neither overrides a profile `model` nor re-opens the strongest-reasoning-class rule that governs the initial dispatch. ## Fleet add-on (config/fleet-dir / config/admiral / config/accounts.json / FM_FLEET_*) diff --git a/docs/examples/crew-dispatch.json b/docs/examples/crew-dispatch.json index f85dbf2d931..26158015b35 100644 --- a/docs/examples/crew-dispatch.json +++ b/docs/examples/crew-dispatch.json @@ -22,6 +22,13 @@ ], "why": "Cursor bills its included, auto, and API pools separately, so this route is priced on the pool it actually draws on instead of on whichever pool is emptiest. Confirm the window id against quota-axi --json before copying this." }, + { + "when": "The task is standard coding work suited for Gemini or Claude models on Antigravity.", + "use": [ + { "harness": "agy", "model": "gemini-3.7-flash-high", "effort": "high", "quotaWindow": "gemini_5h" } + ], + "why": "Antigravity bills Gemini and Claude/GPT pools separately; quotaWindow prices on the Gemini pool. Confirm both the model id against agy models (or bin/fm-model-refresh.sh) and the window id against quota-axi --json before copying this, because a stale model id surfaces only as an agy trust-gate timeout at launch." + }, { "when": "The task is a big or ambiguous multi-file feature, a risky refactor, or work that requires holding many moving parts in mind.", "use": [ @@ -34,5 +41,9 @@ "default": [ { "harness": "claude", "provider": "claude", "model": "sonnet", "effort": "medium" }, { "harness": "codex", "provider": "codex", "model": "gpt-5.5", "effort": "medium" } - ] + ], + "modelFallback": { + "claude": ["claude-sonnet-5", "sonnet", "haiku"], + "agy": ["gemini-3.7-flash-high", "gemini-3.6-flash-high", "gemini-3.5-flash-high"] + } } diff --git a/tests/fm-bootstrap.test.sh b/tests/fm-bootstrap.test.sh index f3e67116669..25ff37979a2 100755 --- a/tests/fm-bootstrap.test.sh +++ b/tests/fm-bootstrap.test.sh @@ -1154,6 +1154,15 @@ empty default array is flagged^{"default":[]}^exact^CREW_DISPATCH: invalid confi non-object default array entry is flagged^{"default":["codex"]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - each default profile must be an object default array profile without harness is flagged^{"default":[{"model":"gpt-5.5"}]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - each default profile needs harness default array malformed effort is flagged^{"default":[{"harness":"codex","effort":3}]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - default profile model, effort, and quotaWindow must be non-empty strings when present +agy model profile with quotaWindow is accepted^{"rules":[{"when":"agy work","use":[{"harness":"agy","model":"gemini-3.7-flash-high","effort":"high","quotaWindow":"gemini_5h"}]}]}^empty^ +agy native subscription provider mismatch is flagged^{"default":[{"harness":"agy","provider":"claude"}]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - native harness/provider mismatch: agy:claude +model fallback chains are accepted^{"default":{"harness":"agy"},"modelFallback":{"agy":["gemini-3.6-flash-high","gemini-3.5-flash-high"]}}^empty^ +legacy _model_fallback alias is accepted^{"default":{"harness":"claude"},"_model_fallback":{"claude":["claude-sonnet-5","haiku"]}}^empty^ +declaring both model fallback spellings is flagged^{"modelFallback":{"claude":["a"]},"_model_fallback":{"claude":["b"]}}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - modelFallback and its legacy alias _model_fallback cannot both be declared +non-object model fallback is flagged^{"modelFallback":["claude"]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - modelFallback must be an object mapping a harness to its ordered model chain +model fallback unverified harness is flagged^{"modelFallback":{"spaceship":["a"]}}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - modelFallback has an unverified harness: spaceship +empty model fallback chain is flagged^{"modelFallback":{"claude":[]}}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - modelFallback chain must be a non-empty array of non-empty model ids: claude +model fallback chain with a malformed id is flagged^{"modelFallback":{"claude":["claude-sonnet-5",""]}}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - modelFallback chain must be a non-empty array of non-empty model ids: claude ROWS pass "bootstrap validates crew-dispatch.json and reports malformed or unverified configs" } diff --git a/tests/fm-control.test.sh b/tests/fm-control.test.sh index 0daef79c97c..8bd58ef16d6 100755 --- a/tests/fm-control.test.sh +++ b/tests/fm-control.test.sh @@ -52,6 +52,8 @@ verified_adapter_contract() { # -> exit command, interrupt key, repea kimi) printf '/exit\tEscape\t1\t\n' ;; cursor) printf '/exit\tEscape\t1\t\n' ;; muse) printf '/exit\tEscape\t1\tC-u\n' ;; + copilot) printf '/exit\tC-c\t1\t\n' ;; + agy) printf '/exit\tEscape\t1\t\n' ;; *) return 1 ;; esac } @@ -267,7 +269,7 @@ test_harness_family_resolution() { for pair in claude:claude claude-latest:claude codex:codex codex-cli:codex \ opencode:opencode grok:grok grok-2:grok kimi:kimi cursor:cursor \ cursor-agent:cursor muse:muse muse-bin-0.1.0:muse pi:pi \ - pi-signed:pi-signed; do + pi-signed:pi-signed cline:cline copilot:copilot agy:agy; do recorded=${pair%%:*} want=${pair#*:} got=$(fm_control_harness_family "$recorded") \ diff --git a/tests/fm-dispatch-select.test.sh b/tests/fm-dispatch-select.test.sh index 022adb83fff..3c93708e591 100755 --- a/tests/fm-dispatch-select.test.sh +++ b/tests/fm-dispatch-select.test.sh @@ -425,6 +425,145 @@ JSON pass "tied known spendPriority still rotates by least-recent use" } +test_agy_with_declared_quota_windows_prices_separate_pools() { + local home fakebin quota out rc + home=$(make_home agy-windows) + fakebin=$(make_fakebin agy-windows) + quota="$home/quota.json" + cat > "$quota" <&1) || rc=$? + expect_code 3 "$rc" "undeclared agy profile must price on the worst window and fail closed" + assert_contains "$out" "quota headroom 0% is at or below 20% reserve" \ + "the provider-wide refusal must name the figure it priced on" + + # Declared healthy window (gemini_5h at 100%): succeeds. + out=$(run_select "$home" "$fakebin" "$quota" agy-gemini.json \ + '[{"harness":"agy","model":"gemini-3.7-flash-high","quotaWindow":"gemini_5h"}]' 2>"$home/gemini.err") + [ "$(printf '%s\n' "$out" | jq -r .model)" = "gemini-3.7-flash-high" ] \ + || fail "agy candidate with declared healthy gemini_5h window was refused: $out" + assert_contains "$(cat "$home/gemini.err")" "window gemini_5h headroom=100%" \ + "the honoured gemini window must be inspectable in the diagnostic" + [ "$(printf '%s\n' "$out" | jq -r '.provider')" = "agy" ] \ + || fail "native agy profile must establish agy provider identity: $out" + [ "$(printf '%s\n' "$out" | jq -r '.quotaWindow // "none"')" = none ] \ + || fail "quotaWindow must not reach the launch profile: $out" + + # Declared exhausted window (claude_gpt_5h at 0%): refused. + rc=0 + out=$(run_select "$home" "$fakebin" "$quota" agy-claude.json \ + '[{"harness":"agy","model":"claude-sonnet-4-6","quotaWindow":"claude_gpt_5h"}]' 2>&1) || rc=$? + expect_code 3 "$rc" "agy candidate with exhausted claude_gpt_5h window must be refused" + assert_contains "$out" "window claude_gpt_5h headroom 0% is at or below 20% reserve" \ + "exhausted declared window refusal must name the window" + pass "agy declared quota windows price Gemini and Claude pools separately" +} + +test_agy_record_failure_and_cooldown() { + local home fakebin quota profiles out rc + home=$(make_home agy-cooldown) + fakebin=$(make_fakebin agy-cooldown) + quota="$home/quota.json" + cat > "$quota" < "$home/state/agy-task.meta" + printf 'failed: 429 Too Many Requests - hit your 5-hour limit\n' > "$home/state/agy-task.status" + + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_CONFIG_OVERRIDE="$home/config" \ + PATH="$fakebin:$BASE_PATH" "$SELECTOR" record-failure --provider agy --task agy-task --now 1000 \ + >/dev/null 2>&1 || fail "verified 429 status evidence on agy did not create cooldown" + + out=$(run_select "$home" "$fakebin" "$quota" .dispatch-routing.json "$profiles" 1001 2>"$home/select.err") + [ "$(printf '%s\n' "$out" | jq -r .harness)" = "codex" ] || fail "cooldown did not fail over from agy to codex: $out" + assert_contains "$(cat "$home/select.err")" "candidate provider=agy unavailable: cooldown until epoch 2800" \ + "cooldown diagnostic on agy omitted its bound" + + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_CONFIG_OVERRIDE="$home/config" \ + PATH="$fakebin:$BASE_PATH" "$SELECTOR" clear --provider agy \ + >/dev/null 2>&1 || fail "clear --provider agy failed" + + # Rotation is least-recently-used and agy has never been dispatched here, so + # only a genuinely cleared cooldown can put it back at the front. + out=$(run_select "$home" "$fakebin" "$quota" .dispatch-routing.json "$profiles" 1002 2>/dev/null) + [ "$(printf '%s\n' "$out" | jq -r .harness)" = "agy" ] \ + || fail "cleared agy cooldown did not restore candidate eligibility: $out" + pass "agy verified failure records cooldown and clear restores eligibility" +} + +test_depletion_evidence_gate_separates_quota_from_working_limits() { + local home fakebin quota out rc name status + home=$(make_home evidence-gate) + fakebin=$(make_fakebin evidence-gate) + quota="$home/quota.json" + write_quota "$quota" fresh 80 fresh 80 + + # Authentic subscription depletion must be accepted as cooldown evidence. + while IFS='^' read -r name status; do + [ -n "$name" ] || continue + printf 'harness=codex\n' > "$home/state/$name.meta" + printf '%s\n' "$status" > "$home/state/$name.status" + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_CONFIG_OVERRIDE="$home/config" \ + PATH="$fakebin:$BASE_PATH" "$SELECTOR" record-failure --provider codex --task "$name" --now 1000 \ + >/dev/null 2>&1 || fail "quota depletion evidence was rejected: $status" + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_CONFIG_OVERRIDE="$home/config" \ + PATH="$fakebin:$BASE_PATH" "$SELECTOR" clear --provider codex >/dev/null 2>&1 \ + || fail "clear after $name failed" + done <<'ROWS' +http-429^failed: request failed with status code 429 +insufficient-quota^failed: insufficient_quota - add credits to continue +out-of-credits^failed: you are out of credits for this billing period +weekly-allowance^failed: your weekly usage allowance is exhausted +ROWS + + # A working ceiling is not spent quota: parking the provider on it would push + # dispatch onto a weaker lane for a whole cooldown with full headroom left. + while IFS='^' read -r name status; do + [ -n "$name" ] || continue + printf 'harness=codex\n' > "$home/state/$name.meta" + printf '%s\n' "$status" > "$home/state/$name.status" + rc=0 + out=$(FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_CONFIG_OVERRIDE="$home/config" \ + PATH="$fakebin:$BASE_PATH" "$SELECTOR" record-failure --provider codex --task "$name" --now 1000 2>&1) || rc=$? + expect_code 2 "$rc" "benign working text must not qualify as quota evidence: $status" + assert_contains "$out" "contains no rate-limit or quota-exhaustion evidence" \ + "benign refusal was unclear for: $status" + done <<'ROWS' +context-window^working: context token limit reached; compacting +tool-output^failed: exceeded the tool output limit +line-number^working: applying the hunk at line 429 of the diff +max-tokens^working: max output tokens limit hit; continuing +ROWS + + # The refused cases must have left dispatch untouched. + out=$(run_select "$home" "$fakebin" "$quota" .dispatch-routing.json \ + '[{"harness":"codex"}]' 1001 2>/dev/null) + [ "$(printf '%s\n' "$out" | jq -r .harness)" = "codex" ] \ + || fail "a benign working line parked a healthy provider: $out" + pass "depletion evidence accepts quota vocabulary and rejects working ceilings" +} + test_distribution_is_deterministic_balanced_and_array_order_independent test_stale_unavailable_and_reserve_thresholds_fail_closed test_a_declared_quota_window_is_priced_instead_of_the_provider_minimum @@ -436,5 +575,8 @@ test_invalid_profiles_and_settings_are_actionable test_existing_wrapper_and_grok_routes_remain_selectable test_grok_without_a_resolvable_quota_window_is_not_priced test_new_verified_adapters_with_providers_are_selectable +test_agy_with_declared_quota_windows_prices_separate_pools +test_agy_record_failure_and_cooldown +test_depletion_evidence_gate_separates_quota_from_working_limits echo "# all fm-dispatch-select tests passed" diff --git a/tests/fm-pi-watch-extension.test.sh b/tests/fm-pi-watch-extension.test.sh index 5af0ef9106d..d6f6bb97d7e 100755 --- a/tests/fm-pi-watch-extension.test.sh +++ b/tests/fm-pi-watch-extension.test.sh @@ -504,7 +504,7 @@ trap 'exit 0' TERM INT while :; do sleep 0.02; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_PI_ARM_READY_TIMEOUT_MS=250 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_PI_ARM_READY_TIMEOUT_MS=1500 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -528,7 +528,12 @@ writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); const mod = await import(pathToFileURL(process.env.PLUGIN).href); mod.default(pi); await tool.execute("tool-call-hung-successor", {}, undefined, undefined, {}); -for (let i = 0; i < 500 && !prompt; i += 1) { +// Wall-clock deadline, not an iteration count. This path deliberately burns one +// whole arm-ready window per attempt (successor + two retries), so a fixed +// 500x10ms budget can expire before the wake is even due once the window is +// wide enough to survive a loaded runner's process-start latency. +const promptDeadline = Date.now() + 60000; +while (!prompt && Date.now() < promptDeadline) { await new Promise((resolve) => setTimeout(resolve, 10)); } const rows = existsSync(process.env.FM_ARM_LOG) @@ -544,7 +549,7 @@ if (stableRows.length !== 4) throw new Error(`single-flight recovery launched ${ EOF ) status=$? - expect_code 0 "$status" "Pi must deliver the actionable wake after bounded hung-successor recovery" + expect_code 0 "$status" "Pi must deliver the actionable wake after bounded hung-successor recovery: $out" [ -z "$out" ] || fail "Pi hung-successor test printed output: $out" pass "Pi hung successor falls back to one typed actionable wake" } @@ -576,7 +581,7 @@ printf 'arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" while [ ! -e "$FM_RELEASE_FILE" ]; do sleep 0.1; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_RELEASE_FILE="$release" FM_PI_ARM_READY_TIMEOUT_MS=250 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_RELEASE_FILE="$release" FM_PI_ARM_READY_TIMEOUT_MS=1500 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -654,7 +659,7 @@ trap 'exit 0' TERM INT while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_UNRETIRED_READY_FILE="$ready" FM_UNRETIRED_RETIRE_FILE="$retired" FM_RELEASE_FILE="$release" FM_STOP_FILE="$stop" FM_LATE_KIND="$kind" FM_PI_ARM_READY_TIMEOUT_MS=250 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_UNRETIRED_READY_FILE="$ready" FM_UNRETIRED_RETIRE_FILE="$retired" FM_RELEASE_FILE="$release" FM_STOP_FILE="$stop" FM_LATE_KIND="$kind" FM_PI_ARM_READY_TIMEOUT_MS=1500 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node --input-type=module 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -1412,7 +1417,17 @@ const hooks = await mod.FmPrimaryWatchArm({ const event = { event: { type: "session.idle", properties: { sessionID: "session-test" } } }; writeFileSync(`${process.env.FM_HOME}/state/.lock`, "999999\n"); await hooks.event(event); -await new Promise((resolve) => setTimeout(resolve, 120)); +// The event hook starts the arm attempt without awaiting it, and a foreign-lock +// refusal costs ~10 git/ps probes before it settles. Drain that attempt through +// the coordinator instead of sleeping a fixed budget: on a loaded runner the +// probes outlast any wall-clock guess, and flipping the lock underneath an +// in-flight attempt makes the next caller coalesce onto its stale refusal and +// never arm. Awaiting also asserts the exact reason the gate refused. +const refused = await globalThis.__firstmateOpenCodeWatchArm.ensureArmed("session-test", client); +if (refused !== "read-only") { + console.error(`expected read-only while another session holds the lock, got ${refused}`); + process.exit(1); +} if (existsSync(process.env.FM_ARM_LOG)) { console.error("watch arm ran without owning the session lock"); process.exit(1); @@ -1680,7 +1695,7 @@ trap 'exit 0' TERM INT while :; do sleep 0.02; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_OPENCODE_ARM_READY_TIMEOUT_MS=250 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_OPENCODE_ARM_READY_TIMEOUT_MS=1500 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -1704,7 +1719,12 @@ const hooks = await mod.FmPrimaryWatchArm({ }); writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); await hooks.event({ event: { type: "session.idle", properties: { sessionID: "session-test" } } }); -for (let i = 0; i < 500 && !prompt; i += 1) { +// Wall-clock deadline, not an iteration count. This path deliberately burns one +// whole arm-ready window per attempt (successor + two retries), so a fixed +// 500x10ms budget can expire before the wake is even due once the window is +// wide enough to survive a loaded runner's process-start latency. +const promptDeadline = Date.now() + 60000; +while (!prompt && Date.now() < promptDeadline) { await new Promise((resolve) => setTimeout(resolve, 10)); } const rows = existsSync(process.env.FM_ARM_LOG) @@ -1720,7 +1740,7 @@ if (stableRows.length !== 4) throw new Error(`single-flight recovery launched ${ EOF ) status=$? - expect_code 0 "$status" "OpenCode must deliver the actionable wake after bounded hung-successor recovery" + expect_code 0 "$status" "OpenCode must deliver the actionable wake after bounded hung-successor recovery: $out" [ -z "$out" ] || fail "OpenCode hung-successor test printed output: $out" pass "OpenCode hung successor falls back to one typed actionable wake" } @@ -1754,7 +1774,7 @@ printf 'arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" while [ ! -e "$FM_RELEASE_FILE" ]; do sleep 0.1; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_RELEASE_FILE="$release" FM_OPENCODE_ARM_READY_TIMEOUT_MS=250 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_RELEASE_FILE="$release" FM_OPENCODE_ARM_READY_TIMEOUT_MS=1500 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; @@ -1834,7 +1854,7 @@ trap 'exit 0' TERM INT while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done SH chmod +x "$repo/bin/fm-watch-arm.sh" - out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_UNRETIRED_READY_FILE="$ready" FM_UNRETIRED_RETIRE_FILE="$retired" FM_RELEASE_FILE="$release" FM_STOP_FILE="$stop" FM_LATE_KIND="$kind" FM_OPENCODE_ARM_READY_TIMEOUT_MS=250 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' + out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_UNRETIRED_READY_FILE="$ready" FM_UNRETIRED_RETIRE_FILE="$retired" FM_RELEASE_FILE="$release" FM_STOP_FILE="$stop" FM_LATE_KIND="$kind" FM_OPENCODE_ARM_READY_TIMEOUT_MS=1500 FM_WATCH_ARM_RETIRE_TIMEOUT_MS=20 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 FM_WATCH_REARM_RETRY_LIMIT=2 node 2>&1 <<'EOF' import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; diff --git a/tests/fm-runtime-handoff.test.sh b/tests/fm-runtime-handoff.test.sh index eb2c70e1757..c56615fb780 100755 --- a/tests/fm-runtime-handoff.test.sh +++ b/tests/fm-runtime-handoff.test.sh @@ -112,6 +112,15 @@ exit 0 SH chmod +x "$fakebin/treehouse" + # Stub every admitted harness so a fixture never depends on what happens to be + # installed on the runner. cursor is resolved by EXECUTABLE name, not by + # harness name: bin/fm-cursor-lib.sh accepts only `cursor-agent` (or an + # `agent` that proves itself Cursor), so a stub named `cursor` would leave the + # handoff to find - or on a bare CI runner fail to find - the real binary. + for tool in pi-signed opencode cline copilot agy cursor cursor-agent muse grok kimi pi codex claude; do + fm_fake_exit0 "$fakebin" "$tool" + done + printf '%s\n' "$fakebin" } @@ -337,7 +346,7 @@ setup_case() { export FM_FAKE_PANE_CMD=codex : > "$FM_FAKE_SEND_LOG" farm=$(make_bin_farm "$CASE_DIR") - for h in cline cursor-agent copilot; do + for h in spaceship cursor-agent unknown-harness; do set +e out=$(FM_ROOT_OVERRIDE="$ROOT" "$farm/fm-runtime-handoff.sh" task-u2 --harness "$h" 2>&1) rc=$? @@ -580,6 +589,51 @@ setup_case() { pass "sends the recorded harness's exit command and completes once the pane is dead" } +# --- success: a key-exit recorded harness hands off to a newly admitted target - + +{ + setup_case exit-path-key task-x3 + # cline is the one admitted adapter with no exit COMMAND: it leaves on C-c. + # cursor is a newly admitted handoff target, so this covers both new edges. + sed -i.bak 's/^harness=codex$/harness=cline/' "$CASE_HOME/state/task-x3.meta" + rm -f "$CASE_HOME/state/task-x3.meta.bak" + export FM_FAKE_WINDOW_FILE="$CASE_DIR/window.present" + : > "$FM_FAKE_WINDOW_FILE" + export FM_FAKE_EXIT_MARKER="$CASE_DIR/exited" + export FM_FAKE_SEND_MARKS_EXIT="$FM_FAKE_EXIT_MARKER" + export FM_FAKE_PANE_CMD=cline + : > "$FM_FAKE_SEND_LOG" + farm=$(make_bin_farm "$CASE_DIR") + head_before=$(git -C "$CASE_WT" rev-parse HEAD) + + set +e + out=$( + FM_ROOT_OVERRIDE="$ROOT" FM_SPAWN_SETTLE_POLLS=2 \ + FM_HANDOFF_EXIT_POLLS=5 FM_HANDOFF_EXIT_SLEEP=0 \ + "$farm/fm-runtime-handoff.sh" task-x3 --harness cursor \ + --progress-note "ClinePass depleted; work is unlanded." 2>&1 + ) + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "handoff from a key-exit harness to cursor should succeed: $out" + assert_contains "$out" "handed-off task-x3" "handoff success line" + + send_log=$(cat "$FM_FAKE_SEND_LOG") + assert_contains "$send_log" "task-x3 --key C-c" "cline must be exited by key, never by an invented /exit" + case "$send_log" in + *"task-x3 /exit"*) fail "cline has no exit command; a text exit was sent: $send_log" ;; + esac + [ -f "$FM_FAKE_EXIT_MARKER" ] || fail "exit key should have been sent" + [ ! -f "$FM_FAKE_WINDOW_FILE" ] || fail "dead endpoint husk should have been killed" + + [ "$(git -C "$CASE_WT" rev-parse HEAD)" = "$head_before" ] || fail "key-exit path must preserve HEAD" + [ -f "$CASE_WT/dirty.txt" ] || fail "key-exit path must preserve uncommitted changes" + meta=$(cat "$CASE_HOME/state/task-x3.meta") + assert_contains "$meta" "harness=cursor" "meta harness updated to the newly admitted target" + assert_contains "$meta" "pr=https://example.test/pr/1" "pr= preserved over the key-exit path" + pass "exits a key-only harness by key and relaunches on a newly admitted target" +} + # --- refusal: harness ignores the exit command and stays alive -------------- {