diff --git a/.agents/skills/harness-adapters/SKILL.md b/.agents/skills/harness-adapters/SKILL.md index c5ba453f95b..9ea4112153c 100644 --- a/.agents/skills/harness-adapters/SKILL.md +++ b/.agents/skills/harness-adapters/SKILL.md @@ -12,6 +12,7 @@ Use this reference before any harness-specific firstmate operation: spawn, recov Crewmates default to the same harness firstmate is running on unless `config/crew-harness` records an adapter name. Optional dispatch profiles in `config/crew-dispatch.json` can override that static default for one crewmate or scout dispatch by selecting concrete harness, model, and effort axes at intake. +When a matched rule or default is a profile array, load `quota-array-dispatch` for the pace-aware candidate choice after this skill establishes harness and model/provider facts. The captain may override that file at session start or later; a per-task instruction such as "run this one on codex" overrides it for that dispatch only. `default` means mirror firstmate's own harness. diff --git a/.agents/skills/quota-array-dispatch/SKILL.md b/.agents/skills/quota-array-dispatch/SKILL.md new file mode 100644 index 00000000000..a5fe06d6ec6 --- /dev/null +++ b/.agents/skills/quota-array-dispatch/SKILL.md @@ -0,0 +1,170 @@ +--- +name: quota-array-dispatch +description: >- + Agent-only decision procedure for resolving a matched crew-dispatch profile + array from current quota-axi output, including quota-window pace signals. + Load when a dispatch rule or default resolves to more than one profile candidate. +user-invocable: false +metadata: + internal: true +--- + +# quota-array-dispatch + +This skill is the single owner of the pace-aware profile-array selection procedure. +The concise always-loaded intake boundary remains in `AGENTS.md` section 4. +`docs/configuration.md` owns the `config/crew-dispatch.json` schema only. +`quota-axi` remains data-only and never recommends a route. +Firstmate owns the judgment. +Do not add a daemon, opaque composite score, routing wrapper, hard-coded model-specific policy, or producer-side route recommendation. + +## When to load + +Load this skill whenever a matched dispatch rule or the configured default resolves to a profile array (more than one candidate), before choosing the concrete `--harness`, `--model`, and `--effort` passed to `fm-spawn`. +Keep using `harness-adapters` for harness verification, model/provider discovery, and effort fallback. + +## Intake boundary this skill does not relax + +1. Explicit per-task captain overrides still win over configured profiles. +2. Configured profile matching precedence is unchanged: best-fit rule, then configured default, then static crewmate harness. +3. Malformed `config/crew-dispatch.json` remains an actionable error; never select around it. +4. Every configured candidate in the matched array must be accounted for. +5. If any harness/model/provider relationship, applicable quota data, or interpretation cannot be established, stop and report that candidate instead of omitting it, guessing, falling back, or calling the result quota-informed. +6. When every candidate is tight, preserve the captain's strongest-reasoning class rather than silently downgrading it solely to conserve quota; stop and report the tight choice if that class cannot proceed. +7. Genuine ties must remain free of array-order or harness bias. + +## Collect inspectable facts for every candidate + +For each candidate profile: + +1. Establish the harness/model/provider relationship from current authoritative discovery owned by `harness-adapters`. + Fail loudly on an unresolved relationship. +2. Run `quota-axi --json` once per intake and reuse that snapshot for every candidate. +3. Require a current provider report with known quota semantics and a known applicable effective-availability record for that candidate's provider and model scope. + Stale raw windows remain diagnostic evidence only and are never current headroom. +4. Read every bounding window relevant to that candidate, including windows named by `boundedBy`, `limitingWindowIds`, `aheadWindowIds`, `behindWindowIds`, `onPaceWindowIds`, and `unknownWindowIds` on the effective record. +5. Record these inspectable facts, never a hidden score: + - task/profile fit + - reasoning class required by the captain request or task ambiguity + - raw applicable headroom (`effectivePercentRemaining` or the tightest applicable remaining percentage) + - effective pace status when present + - signed reserve for each applicable window and the effective worst reserve when present + - whether any applicable window or effective summary is ahead of reset + - whether any applicable pace is `unknown` + - schema compatibility note when pace fields are absent + +## Pace signals + +quota-axi `schemaVersion` 3 window pace uses: + +- `reservePercentPoints = percentRemaining - timeRemainingPercent` +- Negative reserve means usage is ahead of reset pace and creates conservation pressure. +- Positive reserve means usage is behind reset pace. +- `on_pace` is neutral. + +Effective-availability pace summaries may report `ahead`, `behind`, `on_pace`, `mixed`, or `unknown`. + +Treat conservation pressure as present when: + +- effective pace status is `ahead`, or +- effective pace status is `mixed` and any `aheadWindowIds` remain, or +- any applicable bounding window itself has pace status `ahead`. + +An effective `mixed` result is never healthy merely because one window is behind. +Any remaining `aheadWindowIds` keep conservation pressure. + +Signed reserve comparison uses the worst applicable reserve, preferring the producer field `worstReservePercentPoints` when present and otherwise the minimum signed reserve across applicable bounding windows. + +## Selection procedure + +Apply these steps only among candidates that already satisfy required task/profile fit and the strongest reasoning class the request genuinely needs. +Never use pace or raw headroom to silently replace that reasoning class with a weaker one. + +1. **Unresolved relationship or quota data** + Stop and report the blocked candidate. +2. **Strongest-reasoning / all-tight** + If every remaining candidate is tight, keep the strongest-reasoning class and either dispatch inside that class or stop and report that the tight choice cannot proceed. + Do not conserve quota through an unapproved downgrade. +3. **Conservation pressure vs sustainable pace** + When fit and reasoning class are comparable, prefer a candidate without ahead-of-reset conservation pressure over one with conservation pressure, even when the pressured candidate has somewhat higher raw remaining percentage. +4. **Among pressured candidates** + Prefer the least-negative worst applicable reserve. + Example: worst reserve `-4` is safer than `-18` when other inspectable facts are comparable. +5. **Among sustainable candidates** + Use known behind/on-pace evidence plus raw headroom transparently. + Do not collapse those facts into an opaque composite score. + Prefer known sustainable evidence over `unknown` pace when otherwise comparable. + Between known sustainable candidates, prefer the clearly better inspectable pair of pace reserve and raw headroom; state both facts in the choice rationale. +6. **Unknown pace** + `unknown` is valid explicit uncertainty from quota-axi, not a parser failure and not permission to assume the window is healthy or exhausted. + Inspect `unknownWindowIds` and each window's pace `reason` so the rationale preserves the producer's stated uncertainty. + Prefer known sustainable evidence when otherwise comparable. + If the dispatch choice materially hinges on unresolved pace, report the uncertainty rather than inventing a conclusion. +7. **Absent pace / older schema** + `schemaVersion` 2 payloads or missing pace fields must degrade explicitly and safely. + Do not crash, fabricate pace, or silently reinterpret absence as healthy/`on_pace`. + Compare raw applicable headroom only, using known effective availability rather than stale or isolated window percentages, state that pace is unavailable, and keep every other safety rule above. +8. **Genuine ties** + If every inspectable selection fact is equal, stop and report every tied candidate for captain choice. + Do not select by array order, harness name, or another arbitrary identity ordering. + Report duplicate concrete profiles as a configuration error. + +The intake rationale must name the inspectable facts used for every candidate. +Never conclude with an unexplained "best quota" label. + +## Acceptance scenarios + +These scenarios are normative examples of the procedure above. + +### Higher raw quota but materially ahead vs lower raw quota on/behind pace + +Candidate A has higher `effectivePercentRemaining` but conservation pressure from an ahead bounding window. +Candidate B has lower raw headroom, no conservation pressure, and known behind or on-pace evidence. +Choose B when fit and reasoning class are comparable. + +### Mixed effective pace with an ahead bound + +Effective pace status is `mixed` and `aheadWindowIds` is non-empty. +Treat the candidate as conservation-pressured even if another window is behind or on pace. + +### Both candidates ahead with different worst reserves + +Both candidates have conservation pressure. +Choose the least-negative worst applicable reserve when fit and reasoning class are comparable. + +### Known sustainable versus unknown + +Candidate A has known behind or on-pace evidence. +Candidate B has comparable fit, reasoning class, and raw headroom but `unknown` pace. +Prefer A. +If the only way to prefer one side depends on unresolved pace and no known sustainable candidate remains, report the uncertainty. + +### Every candidate tight while strongest-reasoning applies + +All candidates are tight on real headroom. +Keep the strongest reasoning class required by the request. +Do not pick a weaker class only to save quota. +Dispatch inside that class or stop and report that the tight strongest-class choice cannot proceed. + +### Genuine tie without array-order or harness bias + +Two candidates match on fit, reasoning class, conservation pressure, worst reserve, pace class, raw headroom, and unknown flags. +Choosing either array order or a standing harness preference is forbidden. +Stop and report both tied candidates for captain choice. + +### schemaVersion 2 or absent-pace compatibility + +Older quota-axi output or missing pace fields still allow array resolution. +Compare raw headroom only, state that pace is unavailable, and do not invent ahead/behind/on_pace. + +## Sanitized producer shape + +Validate consumers against a sanitized `schemaVersion` 3 shape derived from quota-axi 0.1.15: + +- top level: `schemaVersion`, `generatedAt`, `providers[]` +- each provider: `provider`, `state`, `windows[]`, and optional `quotaSemantics` with `status` and `effectiveAvailability[]` +- each window: `id`, `label`, `kind`, and optional `percentRemaining` and `pace`; pace has `status` plus optional `reason`, `timeRemainingPercent`, and `reservePercentPoints` +- each effective-availability entry: `scope`, `status`, `boundedBy`, optional `effectivePercentRemaining`, optional `limitingWindowIds`, and optional pace summary +- each effective pace summary: `status` plus optional `aheadWindowIds`, `behindWindowIds`, `onPaceWindowIds`, `unknownWindowIds`, `worstReservePercentPoints`, and `worstReserveWindowId` + +Never persist live provider balances, reset timestamps, account identifiers, or other private account details in tracked fixtures. diff --git a/AGENTS.md b/AGENTS.md index d85e90b8eaa..f838dfb27ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -164,12 +164,13 @@ If static `config/crew-harness` or `config/secondmate-harness` names an unverifi `docs/configuration.md` owns dispatch-profile and runtime-backend schemas, `bin/fm-harness.sh` owns static resolution, and `bin/fm-spawn.sh` owns launch flags and fail-closed validation. When dispatch profiles exist, consult them at every crewmate or scout intake and pass the resolved concrete profile required by `fm-spawn`. Routing precedence is an explicit per-task captain override, then the best-fit configured rule, then the configured default, then the static crewmate harness. -Firstmate alone resolves a matched profile array: run `quota-axi --json` at that intake, evaluate every configured candidate against that current output, and choose the candidate with the most real headroom. +Firstmate alone resolves a matched profile array: run `quota-axi --json` at that intake, evaluate every configured candidate against that current output, and choose with inspectable real headroom including quota-window pace. Account for every candidate; if any harness/model/provider relationship, applicable quota data, or interpretation cannot be established, stop and report that candidate instead of omitting it, guessing, falling back, or calling the result quota-informed. Preserve malformed profile configuration as an actionable error rather than selecting around it. When every candidate is tight, preserve the captain's strongest-reasoning class rather than silently downgrading it solely to conserve quota; stop and report the tight choice if that class cannot proceed. Break genuine headroom ties without array-order or harness bias. -`quota-axi` owns how model or product windows relate to bounding account windows. +`quota-axi` owns how model or product windows relate to bounding account windows and remains data-only. +Load `quota-array-dispatch` before choosing among a matched profile array; that skill is the single owner of the pace-aware selection procedure. The generic effort fallback and its precedence are owned by `harness-adapters`: explicit captain and standing configured effort win; otherwise use low for well-understood explicit work, xhigh for ambiguous investigation or design, intermediate levels proportionally, and never max without explicit captain preference. Do not add model-specific versions of that policy. @@ -472,6 +473,7 @@ These skills are not captain-invocable; load them only at their precise triggers - `bootstrap-diagnostics` - load whenever the session-start digest's bootstrap section prints an actionable diagnostic line (`MISSING:`, `MISSING_MANUAL:`, `BACKEND_INVALID:`, `NEEDS_GH_AUTH`, `TANGLE:`, `CREW_DISPATCH: invalid`, `FLEET_SYNC:`, `PR_CHECK_MIGRATION:`, `SECONDMATE_SYNC:`, `SECONDMATE_LIVENESS:`, `NUDGE_SECONDMATES:`, or `FMX:`); silence and `BOOTSTRAP_INFO:` need no load. - `diagnostic-reasoning` - load before scoping a reported bug and before acting on a diagnostic report. - `ask-user-authority` - load before deciding any ask-user finding, regardless of the project's `yolo` posture. +- `quota-array-dispatch` - load before choosing among a matched crew-dispatch profile array from current quota-axi output. - `harness-adapters` - load before spawning or recovering a crewmate or secondmate, handling a trust dialog, sending a harness-specific skill invocation, interrupting or exiting an agent, resuming an exited agent, or verifying a new harness adapter. - `firstmate-orca` - load before switching to Orca, spawning or supervising Orca-backed work, smoke-testing Orca backend behavior, debugging Orca task state, or reconciling Orca-backed task metadata. - `project-management` - load before adding, creating, removing, or initializing a project. diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index 8685b2e2bbd..c86b7e839ab 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -53,7 +53,8 @@ # with update --archive-body and mv [...]); an installed but # incompatible build reports MISSING like no-mistakes. A compatible # tasks-axi default backend is silent. quota-axi is required for the -# agent-owned dispatch-profile array procedure in AGENTS.md section 4. +# agent-owned dispatch-profile array procedure in AGENTS.md section 4 +# and .agents/skills/quota-array-dispatch/SKILL.md. # X mode is OPTIONAL and inert unless FM_HOME/.env has a non-empty # FMX_PAIRING_TOKEN. When opted in, bootstrap requires curl+jq, writes # the relay poll shim and 30s cadence config, and prints an FMX line. diff --git a/docs/architecture.md b/docs/architecture.md index 4840ef77a02..da1519a0d44 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -142,7 +142,7 @@ The intake and authority contract in `AGENTS.md` owns when separate scout resear ## Dispatch profiles Crewmate and scout dispatch can stay on the static crewmate harness resolved by `config/crew-harness`, or it can use local dispatch profiles in `config/crew-dispatch.json`. -The dispatch file is intentionally judgment-based: firstmate reads the natural-language rules at intake, chooses the best matching rule, resolves profile arrays itself from current quota output under `AGENTS.md` section 4, and passes only concrete `--harness`, `--model`, and `--effort` axes to `fm-spawn.sh`. +The dispatch file is intentionally judgment-based: firstmate reads the natural-language rules at intake, chooses the best matching rule, resolves profile arrays itself from current quota output under the `AGENTS.md` section 4 intake boundary and the `quota-array-dispatch` selection procedure, and passes only concrete `--harness`, `--model`, and `--effort` axes to `fm-spawn.sh`. The shell scripts validate the JSON shape and verified harness/effort combinations, but they do not parse task intent, match natural-language rules, or own array selection. The session-start bootstrap step keeps valid dispatch configuration silent unless verbose facts are enabled and surfaces a concise invalid-config line when validation fails. When the file exists, `fm-spawn.sh` refuses crewmate and scout launches without an explicit harness, so `config/crew-harness` is only automatic when no dispatch profile file is active. diff --git a/docs/configuration.md b/docs/configuration.md index fed683343ea..7f23866c334 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -207,11 +207,12 @@ For Pi and pi-signed secondmate launches, `fm-spawn.sh` starts the selected exec ## Crew dispatch profiles (config/crew-dispatch.json) `config/crew-dispatch.json` is an optional local, gitignored file containing natural-language rules that firstmate reads before dispatching a crewmate or scout. -The shell scripts do not match those rules; firstmate chooses the best matching rule with judgment, resolves its profile object or array under the operating contract in `AGENTS.md` section 4, and passes only concrete `--harness`, `--model`, and `--effort` flags to `fm-spawn.sh`. +The shell scripts do not match those rules; firstmate chooses the best matching rule with judgment, resolves its profile object or array under the operating contract in `AGENTS.md` section 4 and `quota-array-dispatch`, and passes only concrete `--harness`, `--model`, and `--effort` flags to `fm-spawn.sh`. When the file exists, `fm-spawn.sh` enforces that contract by refusing crewmate and scout spawns that lack an explicit harness (`--harness`, a positional adapter, or a raw launch command). Batch spawns satisfy the same requirement with a shared `--harness`. Secondmate spawns are exempt and still resolve through `config/secondmate-harness` and its optional model and effort tokens. -This section is the single owner of the canonical schema and its per-field semantics; `AGENTS.md` section 4 owns the dispatch and array-selection procedure. +This section is the single owner of the canonical schema and its per-field semantics. +`AGENTS.md` section 4 owns the always-loaded dispatch intake boundary, and `quota-array-dispatch` owns the pace-aware profile-array selection procedure. ```json { @@ -235,7 +236,7 @@ Both `use` and the optional top-level `default` accept either one profile object The single-object form stays fully backward-compatible, and every profile needs `harness`. Profile `model` and `effort` fields and rule `why` are optional. An omitted model or effort means the selected harness uses its own default for that axis. -Every profile array is an implicit quota-aware choice. +Every profile array is an implicit quota-aware choice resolved through `quota-array-dispatch`. If no dispatch rule fits, firstmate resolves `default` through the same object-or-array path before falling back to `config/crew-harness`. If a selected profile carries an effort value the chosen harness does not accept, `fm-spawn.sh` records the requested `effort=` in task meta for traceability but omits the launch flag, and bootstrap reports the invalid harness/effort pair as a `CREW_DISPATCH` diagnostic when it is visible in the file. See [`docs/examples/crew-dispatch.json`](examples/crew-dispatch.json) for a starting point to copy into local `config/crew-dispatch.json`. diff --git a/docs/documentation-audiences.json b/docs/documentation-audiences.json index 60773b0d451..54b2190f6c8 100644 --- a/docs/documentation-audiences.json +++ b/docs/documentation-audiences.json @@ -155,6 +155,10 @@ "path": ".agents/skills/project-management/SKILL.md", "audience": "agent-runtime" }, + { + "path": ".agents/skills/quota-array-dispatch/SKILL.md", + "audience": "agent-runtime" + }, { "path": ".agents/skills/secondmate-provisioning/SKILL.md", "audience": "agent-runtime" diff --git a/docs/examples/crew-dispatch.json b/docs/examples/crew-dispatch.json index 4c8fc36993c..23a5391d20a 100644 --- a/docs/examples/crew-dispatch.json +++ b/docs/examples/crew-dispatch.json @@ -16,7 +16,7 @@ { "harness": "claude", "model": "claude-sonnet-5", "effort": "high" }, { "harness": "codex", "model": "gpt-5.5", "effort": "high" } ], - "why": "Firstmate compares every candidate with current relevant quota before dispatch, so use a strong coding profile." + "why": "Firstmate compares every candidate with current relevant quota and pace before dispatch, so use a strong coding profile." } ], "default": [ diff --git a/tests/fixtures/quota-array-dispatch/cases.json b/tests/fixtures/quota-array-dispatch/cases.json new file mode 100644 index 00000000000..23d097be463 --- /dev/null +++ b/tests/fixtures/quota-array-dispatch/cases.json @@ -0,0 +1,352 @@ +{ + "cases": [ + { + "id": "higher-raw-ahead-vs-lower-raw-sustainable", + "expect": "B", + "reason": "prefer sustainable pace over higher raw headroom with conservation pressure", + "candidates": [ + { + "id": "A", + "harness": "claude", + "model": "strong-a", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 80, + "paceStatus": "ahead", + "aheadWindowIds": ["weekly"], + "worstReserve": -12.0, + "unknownPace": false, + "paceAvailable": true + }, + { + "id": "B", + "harness": "codex", + "model": "strong-b", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 55, + "paceStatus": "behind", + "aheadWindowIds": [], + "worstReserve": 18.0, + "unknownPace": false, + "paceAvailable": true + } + ] + }, + { + "id": "mixed-effective-with-ahead-bound", + "expect": "B", + "reason": "mixed with aheadWindowIds is conservation pressure", + "candidates": [ + { + "id": "A", + "harness": "claude", + "model": "mixed-a", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 75, + "paceStatus": "mixed", + "aheadWindowIds": ["seven_day"], + "worstReserve": -8.0, + "unknownPace": false, + "paceAvailable": true + }, + { + "id": "B", + "harness": "codex", + "model": "steady-b", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 60, + "paceStatus": "on_pace", + "aheadWindowIds": [], + "worstReserve": 0.0, + "unknownPace": false, + "paceAvailable": true + } + ] + }, + { + "id": "both-ahead-least-negative-reserve", + "expect": "B", + "reason": "among pressured candidates prefer least-negative worst reserve", + "candidates": [ + { + "id": "A", + "harness": "claude", + "model": "pressured-a", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 50, + "paceStatus": "ahead", + "aheadWindowIds": ["weekly"], + "worstReserve": -22.0, + "unknownPace": false, + "paceAvailable": true + }, + { + "id": "B", + "harness": "codex", + "model": "pressured-b", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 48, + "paceStatus": "ahead", + "aheadWindowIds": ["weekly"], + "worstReserve": -5.0, + "unknownPace": false, + "paceAvailable": true + } + ] + }, + { + "id": "ahead-bounding-window-overrides-neutral-effective-summary", + "expect": "B", + "reason": "an ahead applicable bounding window creates conservation pressure even when the effective summary is neutral", + "candidates": [ + { + "id": "A", + "harness": "claude", + "model": "bounded-a", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 72, + "paceStatus": "on_pace", + "aheadWindowIds": [], + "boundingWindows": [ + { + "id": "weekly", + "paceStatus": "ahead", + "reservePercentPoints": -9.0 + } + ], + "worstReserve": -9.0, + "unknownPace": false, + "paceAvailable": true + }, + { + "id": "B", + "harness": "codex", + "model": "steady-b", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 58, + "paceStatus": "behind", + "aheadWindowIds": [], + "boundingWindows": [ + { + "id": "weekly", + "paceStatus": "behind", + "reservePercentPoints": 7.0 + } + ], + "worstReserve": 7.0, + "unknownPace": false, + "paceAvailable": true + } + ] + }, + { + "id": "known-sustainable-vs-unknown", + "expect": "A", + "reason": "prefer known sustainable evidence over unknown pace", + "candidates": [ + { + "id": "A", + "harness": "codex", + "model": "known-a", + "effort": "medium", + "fit": "comparable", + "reasoningClass": "standard", + "tight": false, + "rawHeadroom": 40, + "paceStatus": "behind", + "aheadWindowIds": [], + "worstReserve": 10.0, + "unknownPace": false, + "paceAvailable": true + }, + { + "id": "B", + "harness": "claude", + "model": "unknown-b", + "effort": "medium", + "fit": "comparable", + "reasoningClass": "standard", + "tight": false, + "rawHeadroom": 42, + "paceStatus": "unknown", + "aheadWindowIds": [], + "worstReserve": null, + "unknownPace": true, + "paceAvailable": true + } + ] + }, + { + "id": "all-tight-strongest-reasoning", + "expect": "A", + "reason": "preserve strongest-reasoning class when every candidate is tight", + "requiredReasoningClass": "strong", + "candidates": [ + { + "id": "A", + "harness": "claude", + "model": "strong-tight", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": true, + "rawHeadroom": 8, + "paceStatus": "behind", + "aheadWindowIds": [], + "worstReserve": 2.0, + "unknownPace": false, + "paceAvailable": true + }, + { + "id": "B", + "harness": "codex", + "model": "weaker-roomier", + "effort": "medium", + "fit": "comparable", + "reasoningClass": "standard", + "tight": true, + "rawHeadroom": 25, + "paceStatus": "behind", + "aheadWindowIds": [], + "worstReserve": 12.0, + "unknownPace": false, + "paceAvailable": true + } + ] + }, + { + "id": "genuine-tie-captain-choice", + "expectError": "genuine tie requires captain choice", + "reason": "report genuine ties instead of selecting by array order or harness identity", + "candidates": [ + { + "id": "A", + "harness": "claude", + "model": "same-model", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 50, + "paceStatus": "on_pace", + "aheadWindowIds": [], + "worstReserve": 0.0, + "unknownPace": false, + "paceAvailable": true + }, + { + "id": "B", + "harness": "codex", + "model": "same-model", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 50, + "paceStatus": "on_pace", + "aheadWindowIds": [], + "worstReserve": 0.0, + "unknownPace": false, + "paceAvailable": true + } + ] + }, + { + "id": "genuine-tie-reversed-array-order", + "expectError": "genuine tie requires captain choice", + "reason": "reversing a genuine tie must still require captain choice", + "candidates": [ + { + "id": "B", + "harness": "codex", + "model": "same-model", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 50, + "paceStatus": "on_pace", + "aheadWindowIds": [], + "worstReserve": 0.0, + "unknownPace": false, + "paceAvailable": true + }, + { + "id": "A", + "harness": "claude", + "model": "same-model", + "effort": "high", + "fit": "comparable", + "reasoningClass": "strong", + "tight": false, + "rawHeadroom": 50, + "paceStatus": "on_pace", + "aheadWindowIds": [], + "worstReserve": 0.0, + "unknownPace": false, + "paceAvailable": true + } + ] + }, + { + "id": "schema-v2-absent-pace", + "expect": "A", + "reason": "absent pace degrades to raw headroom without fabricating pace health", + "candidates": [ + { + "id": "A", + "harness": "codex", + "model": "legacy-a", + "effort": "medium", + "fit": "comparable", + "reasoningClass": "standard", + "tight": false, + "rawHeadroom": 70, + "paceStatus": null, + "aheadWindowIds": [], + "worstReserve": null, + "unknownPace": false, + "paceAvailable": false + }, + { + "id": "B", + "harness": "claude", + "model": "legacy-b", + "effort": "medium", + "fit": "comparable", + "reasoningClass": "standard", + "tight": false, + "rawHeadroom": 40, + "paceStatus": null, + "aheadWindowIds": [], + "worstReserve": null, + "unknownPace": false, + "paceAvailable": false + } + ] + } + ] +} diff --git a/tests/fixtures/quota-array-dispatch/schema-v3-shape.json b/tests/fixtures/quota-array-dispatch/schema-v3-shape.json new file mode 100644 index 00000000000..a79f86aca32 --- /dev/null +++ b/tests/fixtures/quota-array-dispatch/schema-v3-shape.json @@ -0,0 +1,103 @@ +{ + "schemaVersion": 3, + "generatedAt": "1970-01-01T00:00:00.000Z", + "providers": [ + { + "provider": "claude", + "label": "Claude", + "source": "test", + "plan": "test", + "windows": [ + { + "id": "five_hour", + "label": "session", + "kind": "session", + "percentUsed": 20, + "percentRemaining": 80, + "windowSeconds": 18000, + "pace": { + "status": "behind", + "timeRemainingPercent": 40.0, + "elapsedPercent": 60.0, + "reservePercentPoints": 40.0 + } + }, + { + "id": "seven_day", + "label": "week", + "kind": "weekly", + "percentUsed": 55, + "percentRemaining": 45, + "windowSeconds": 604800, + "pace": { + "status": "ahead", + "timeRemainingPercent": 60.0, + "elapsedPercent": 40.0, + "reservePercentPoints": -15.0 + } + } + ], + "quotaSemantics": { + "status": "known", + "description": "sanitized representative schemaVersion 3 shape", + "effectiveAvailability": [ + { + "scope": "all_models", + "status": "known", + "effectivePercentRemaining": 45, + "boundedBy": ["five_hour", "seven_day"], + "limitingWindowIds": ["seven_day"], + "pace": { + "status": "mixed", + "aheadWindowIds": ["seven_day"], + "behindWindowIds": ["five_hour"], + "worstReservePercentPoints": -15.0, + "worstReserveWindowId": "seven_day" + } + } + ] + } + }, + { + "provider": "codex", + "label": "Codex", + "source": "test", + "plan": "test", + "windows": [ + { + "id": "weekly", + "label": "week", + "kind": "weekly", + "percentUsed": 30, + "percentRemaining": 70, + "windowSeconds": 604800, + "pace": { + "status": "behind", + "timeRemainingPercent": 50.0, + "elapsedPercent": 50.0, + "reservePercentPoints": 20.0 + } + } + ], + "quotaSemantics": { + "status": "known", + "description": "sanitized representative schemaVersion 3 shape", + "effectiveAvailability": [ + { + "scope": "all_models", + "status": "known", + "effectivePercentRemaining": 70, + "boundedBy": ["weekly"], + "limitingWindowIds": ["weekly"], + "pace": { + "status": "behind", + "behindWindowIds": ["weekly"], + "worstReservePercentPoints": 20.0, + "worstReserveWindowId": "weekly" + } + } + ] + } + } + ] +} diff --git a/tests/fm-instruction-owners.test.sh b/tests/fm-instruction-owners.test.sh index f55f3e905f9..754e00ddc83 100755 --- a/tests/fm-instruction-owners.test.sh +++ b/tests/fm-instruction-owners.test.sh @@ -109,13 +109,15 @@ test_agent_owned_quota_array_dispatch_contract() { 'Firstmate alone resolves a matched profile array' \ 'run `quota-axi --json` at that intake' \ 'evaluate every configured candidate against that current output' \ - 'choose the candidate with the most real headroom' \ + 'inspectable real headroom including quota-window pace' \ 'if any harness/model/provider relationship, applicable quota data, or interpretation cannot be established, stop and report that candidate' \ 'instead of omitting it, guessing, falling back, or calling the result quota-informed' \ 'Preserve malformed profile configuration as an actionable error' \ "preserve the captain's strongest-reasoning class rather than silently downgrading it" \ 'Break genuine headroom ties without array-order or harness bias' \ - '`quota-axi` owns how model or product windows relate to bounding account windows'; do + '`quota-axi` owns how model or product windows relate to bounding account windows' \ + 'remains data-only' \ + 'Load `quota-array-dispatch` before choosing among a matched profile array'; do assert_grep "$phrase" "$AGENTS" "array-dispatch contract lost '$phrase'" done @@ -131,12 +133,16 @@ test_agent_owned_quota_array_dispatch_contract() { done assert_grep 'not as a permanent namespace or provider mapping' "$HARNESS" \ "model discovery guidance permits a fixed provider table" - assert_grep '`AGENTS.md` section 4 owns the dispatch and array-selection procedure.' "$CONFIG" \ - "configuration docs do not point to the agent-owned array procedure" + assert_grep 'load `quota-array-dispatch` for the pace-aware candidate choice' "$HARNESS" \ + "harness-adapters lost the quota-array-dispatch handoff" + assert_grep '`quota-array-dispatch` owns the pace-aware profile-array selection procedure' "$CONFIG" \ + "configuration docs do not point to quota-array-dispatch" assert_grep 'quota-axi is required for the' "$BOOTSTRAP" \ "bootstrap docs lost the quota-axi dependency pointer" - assert_grep 'agent-owned dispatch-profile array procedure in AGENTS.md section 4.' "$BOOTSTRAP" \ + assert_grep 'agent-owned dispatch-profile array procedure in AGENTS.md section 4' "$BOOTSTRAP" \ "bootstrap docs do not point to the agent-owned array procedure" + assert_grep 'quota-array-dispatch/SKILL.md' "$BOOTSTRAP" \ + "bootstrap docs do not point to quota-array-dispatch" pass "firstmate directly compares every quota candidate with authoritative model discovery" } diff --git a/tests/fm-quota-array-dispatch.test.sh b/tests/fm-quota-array-dispatch.test.sh new file mode 100755 index 00000000000..a958e56c306 --- /dev/null +++ b/tests/fm-quota-array-dispatch.test.sh @@ -0,0 +1,278 @@ +#!/usr/bin/env bash +# Contract and deterministic fixture tests for quota-array-dispatch. +# +# The skill owns the agent-facing decision procedure. +# This test encodes the same inspectable comparison rules against sanitized +# fixtures so acceptance cases stay deterministic without introducing a +# production routing wrapper. +# shellcheck disable=SC2016 +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +AGENTS="$ROOT/AGENTS.md" +OWNER="$ROOT/.agents/skills/quota-array-dispatch/SKILL.md" +HARNESS="$ROOT/.agents/skills/harness-adapters/SKILL.md" +CONFIG="$ROOT/docs/configuration.md" +ARCHITECTURE="$ROOT/docs/architecture.md" +BOOTSTRAP="$ROOT/bin/fm-bootstrap.sh" +AUDIENCES="$ROOT/docs/documentation-audiences.json" +CASES="$ROOT/tests/fixtures/quota-array-dispatch/cases.json" +SHAPE="$ROOT/tests/fixtures/quota-array-dispatch/schema-v3-shape.json" + +intake_boundary() { + awk ' + /^## 4\. Harness and runtime dispatch$/ { found = 1; next } + found && /^## 5\. Recovery$/ { exit } + found { print } + ' "$AGENTS" +} + +select_candidate_py() { + python3 - "$@" <<'PY' +import json, sys + +def conservation_pressure(c): + if not c.get("paceAvailable", True): + return False + status = c.get("paceStatus") + ahead_ids = c.get("aheadWindowIds") or [] + bounding_windows = c.get("boundingWindows") or [] + if status == "ahead": + return True + if status == "mixed" and ahead_ids: + return True + if any(window.get("paceStatus") == "ahead" for window in bounding_windows): + return True + return False + +def select(case): + required = case.get("requiredReasoningClass") + cands = list(case["candidates"]) + if required: + matching = [c for c in cands if c.get("reasoningClass") == required] + if not matching: + return {"error": "required reasoning class unavailable"} + # Strongest-reasoning rule: never drop to a weaker class for quota. + cands = matching + + # Fit filter: fixtures mark comparable; keep only comparable for these cases. + cands = [c for c in cands if c.get("fit") == "comparable"] + if not cands: + return {"error": "no comparable candidates"} + + def sort_key(c): + pressured = conservation_pressure(c) + unknown = bool(c.get("unknownPace")) or c.get("paceStatus") == "unknown" + pace_available = bool(c.get("paceAvailable", True)) + reserve = c.get("worstReserve") + if reserve is None: + reserve_key = float("-inf") + else: + reserve_key = float(reserve) + raw = float(c.get("rawHeadroom") or 0) + # Sort ascending by preference rank components that python min understands + # via a tuple where lower is better only for pressure/unknown flags. + return ( + 1 if pressured else 0, + 1 if (unknown and pace_available) else 0, + 0 if pace_available else 1, # when pace absent, still comparable via raw only + # Among pressured: least-negative reserve => higher reserve first => negate + (-reserve_key if pressured else 0), + # Among sustainable with pace: prefer higher reserve then higher raw + (-reserve_key if (not pressured and pace_available and not unknown) else 0), + -raw, + ) + + # Special-case all-tight already constrained to required class above. + best_key = min(sort_key(c) for c in cands) + winners = [c for c in cands if sort_key(c) == best_key] + if len(winners) > 1: + return { + "error": "genuine tie requires captain choice", + "candidates": sorted(c["id"] for c in winners), + } + winner = winners[0] + return { + "id": winner["id"], + "pressured": conservation_pressure(winner), + } + +case = json.loads(sys.argv[1]) +print(json.dumps(select(case))) +PY +} + +test_owner_and_always_loaded_boundary() { + local boundary trigger_count + boundary=$(intake_boundary) + + assert_present "$OWNER" "quota-array-dispatch owner is missing" + assert_grep 'name: quota-array-dispatch' "$OWNER" "quota-array-dispatch skill has the wrong name" + assert_grep 'user-invocable: false' "$OWNER" "quota-array-dispatch skill must be agent-only" + assert_grep 'single owner of the pace-aware profile-array selection procedure' "$OWNER" \ + "quota-array-dispatch skill does not declare ownership" + + assert_contains "$boundary" 'Firstmate alone resolves a matched profile array' \ + "intake boundary lost agent-owned array resolution" + assert_contains "$boundary" 'run `quota-axi --json` at that intake' \ + "intake boundary lost quota-axi intake read" + assert_contains "$boundary" 'evaluate every configured candidate against that current output' \ + "intake boundary lost full-candidate accounting" + assert_contains "$boundary" 'inspectable real headroom including quota-window pace' \ + "intake boundary lost pace-aware headroom wording" + assert_contains "$boundary" 'if any harness/model/provider relationship, applicable quota data, or interpretation cannot be established, stop and report that candidate' \ + "intake boundary lost unresolved-candidate refusal" + assert_contains "$boundary" 'instead of omitting it, guessing, falling back, or calling the result quota-informed' \ + "intake boundary lost no-guess wording" + assert_contains "$boundary" 'Preserve malformed profile configuration as an actionable error' \ + "intake boundary lost malformed-config refusal" + assert_contains "$boundary" "preserve the captain's strongest-reasoning class rather than silently downgrading it" \ + "intake boundary lost strongest-reasoning rule" + assert_contains "$boundary" 'Break genuine headroom ties without array-order or harness bias' \ + "intake boundary lost genuine-tie rule" + assert_contains "$boundary" '`quota-axi` owns how model or product windows relate to bounding account windows' \ + "intake boundary lost quota-axi window ownership" + assert_contains "$boundary" 'remains data-only' \ + "intake boundary lost data-only producer boundary" + assert_contains "$boundary" 'Load `quota-array-dispatch` before choosing among a matched profile array' \ + "intake boundary lost quota-array-dispatch load trigger" + + trigger_count=$(grep -Fc -- '- `quota-array-dispatch` -' "$AGENTS") + [ "$trigger_count" -eq 1 ] || fail "quota-array-dispatch must have exactly one section 13 trigger, found $trigger_count" + + # Full pace procedure stays out of AGENTS.md. + if printf '%s\n' "$boundary" | grep -q 'reservePercentPoints'; then + fail "AGENTS.md intake boundary duplicated pace formula detail" + fi + if printf '%s\n' "$boundary" | grep -q 'aheadWindowIds'; then + fail "AGENTS.md intake boundary duplicated aheadWindowIds detail" + fi + + pass "quota-array-dispatch has one conditional owner and a concise always-loaded boundary" +} + +test_owner_contains_acceptance_procedure() { + local phrase + for phrase in \ + 'reservePercentPoints = percentRemaining - timeRemainingPercent' \ + 'Negative reserve means usage is ahead of reset pace and creates conservation pressure' \ + 'Positive reserve means usage is behind reset pace' \ + '`on_pace` is neutral' \ + 'effective pace status is `mixed` and any `aheadWindowIds` remain' \ + 'prefer a candidate without ahead-of-reset conservation pressure over one with conservation pressure' \ + 'even when the pressured candidate has somewhat higher raw remaining percentage' \ + 'Prefer the least-negative worst applicable reserve' \ + 'Use known behind/on-pace evidence plus raw headroom transparently' \ + 'Do not collapse those facts into an opaque composite score' \ + '`unknown` is valid explicit uncertainty from quota-axi' \ + 'Prefer known sustainable evidence over `unknown` pace when otherwise comparable' \ + 'If the dispatch choice materially hinges on unresolved pace, report the uncertainty' \ + 'Do not crash, fabricate pace, or silently reinterpret absence as healthy' \ + 'stop and report every tied candidate for captain choice' \ + 'Do not select by array order, harness name, or another arbitrary identity ordering' \ + 'Do not add a daemon, opaque composite score, routing wrapper, hard-coded model-specific policy'; do + assert_grep "$phrase" "$OWNER" "quota-array-dispatch procedure lost '$phrase'" + done + + for phrase in \ + 'Higher raw quota but materially ahead vs lower raw quota on/behind pace' \ + 'Mixed effective pace with an ahead bound' \ + 'Both candidates ahead with different worst reserves' \ + 'Known sustainable versus unknown' \ + 'Every candidate tight while strongest-reasoning applies' \ + 'Genuine tie without array-order or harness bias' \ + 'schemaVersion 2 or absent-pace compatibility'; do + assert_grep "$phrase" "$OWNER" "acceptance scenario missing: $phrase" + done + pass "quota-array-dispatch owns the full pace procedure and acceptance scenarios" +} + +test_cross_references_stay_pointers() { + assert_grep '`quota-array-dispatch` owns the pace-aware profile-array selection procedure' "$CONFIG" \ + "configuration docs do not point to quota-array-dispatch" + assert_no_grep '`AGENTS.md` section 4 owns the dispatch and array-selection procedure.' "$CONFIG" \ + "configuration docs still claim AGENTS.md owns the full array-selection procedure" + assert_grep 'quota-array-dispatch' "$ARCHITECTURE" \ + "architecture docs lost the quota-array-dispatch pointer" + assert_grep 'quota-array-dispatch' "$BOOTSTRAP" \ + "bootstrap header lost the quota-array-dispatch pointer" + assert_grep 'load `quota-array-dispatch` for the pace-aware candidate choice' "$HARNESS" \ + "harness-adapters lost the array-selection handoff" + assert_grep '.agents/skills/quota-array-dispatch/SKILL.md' "$AUDIENCES" \ + "documentation audience inventory missing quota-array-dispatch" + pass "cross-references point at the single procedure owner" +} + +test_schema_v3_shape_fixture() { + python3 - "$SHAPE" <<'PY' || fail "schema v3 shape fixture is invalid" +import json, sys +path = sys.argv[1] +data = json.load(open(path)) +assert data.get("schemaVersion") == 3, data.get("schemaVersion") +assert isinstance(data.get("providers"), list) and data["providers"], "providers" +provider = data["providers"][0] +assert "windows" in provider and provider["windows"], "windows" +window = provider["windows"][0] +assert "pace" in window and "status" in window["pace"], window +eff = provider["quotaSemantics"]["effectiveAvailability"][0] +assert "pace" in eff and "status" in eff["pace"], eff +assert "effectivePercentRemaining" in eff +# Privacy: no live account residue markers. +blob = json.dumps(data) +for bad in ("sk-", "@", "Bearer ", "accountId", "organizationId"): + assert bad not in blob, bad +PY + pass "sanitized schemaVersion 3 fixture preserves producer pace shape without private details" +} + +test_deterministic_acceptance_cases() { + local raw case_json case_id expect expect_error got reason + raw=$(cat "$CASES") + while IFS= read -r case_json; do + case_id=$(python3 -c 'import json,sys; print(json.loads(sys.argv[1])["id"])' "$case_json") + expect=$(python3 -c 'import json,sys; print(json.loads(sys.argv[1]).get("expect", ""))' "$case_json") + expect_error=$(python3 -c 'import json,sys; print(json.loads(sys.argv[1]).get("expectError", ""))' "$case_json") + reason=$(python3 -c 'import json,sys; print(json.loads(sys.argv[1])["reason"])' "$case_json") + got=$(select_candidate_py "$case_json") + python3 -c ' +import json,sys +got=json.loads(sys.argv[1]) +expect=sys.argv[2] +expect_error=sys.argv[3] +case_id=sys.argv[4] +err=got.get("error") +if expect_error: + if err != expect_error: + raise SystemExit("%s: expected error %s, got %s" % (case_id, expect_error, got)) +elif err: + raise SystemExit("%s: selector error: %s" % (case_id, err)) +elif got.get("id") != expect: + raise SystemExit("%s: expected %s, got %s" % (case_id, expect, got)) +' "$got" "$expect" "$expect_error" "$case_id" \ + || fail "case $case_id failed ($reason); selector returned $got" + if [ -n "$expect_error" ]; then + pass "case $case_id -> $expect_error ($reason)" + else + pass "case $case_id -> $expect ($reason)" + fi + done < <(python3 -c 'import json,sys; data=json.load(sys.stdin); [print(json.dumps(c, separators=(",", ":"))) for c in data["cases"]]' <<<"$raw") +} + +test_no_duplicate_procedure_in_agents() { + # Guard against re-expanding the full procedure into AGENTS.md. + local count + count=$(grep -c 'conservation pressure' "$AGENTS" || true) + [ "$count" -eq 0 ] || fail "AGENTS.md should not restate conservation-pressure procedure detail" + count=$(grep -c 'worst applicable reserve' "$AGENTS" || true) + [ "$count" -eq 0 ] || fail "AGENTS.md should not restate worst-reserve procedure detail" + pass "AGENTS.md does not duplicate the pace procedure body" +} + +test_owner_and_always_loaded_boundary +test_owner_contains_acceptance_procedure +test_cross_references_stay_pointers +test_schema_v3_shape_fixture +test_deterministic_acceptance_cases +test_no_duplicate_procedure_in_agents