From abe237a6e1395e744dda59b73a3a9fb52020bc7d Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Wed, 7 Oct 2026 17:16:42 -0500 Subject: [PATCH 01/12] feat: pick a project skill for every ship and scout launch Every ship and scout launch and relaunch now runs bin/fm-skill-pick.sh once its task copy exists. It judges the copy's .agents/skills and .claude/skills (an earlier folder wins a duplicate name) and adds the one picked skill to the launch instructions as a skill to read in full and follow, after the existing mandatory triggers. When nothing fits or the picker cannot run, the launch continues and the instructions and task record say why. The recipe is the TypeSafe skill-suggestion cookbook (rank the whole roster with three need gates, then rerank the top three with 700-character excerpts and one fits noul each; 0.30 and 0.30 thresholds), with the cookbook's question texts verbatim. It runs on the vendored hyper-jev starter client, imported unchanged from .agents/skills/hyper-jev/templates/starter/src/core/. This replaces bin/fm-skill-suggest.sh, bin/fm-skill-catalog.jq and their test, which ran only when a brief carried a "Skill selection input" section. The brief text sent is the text dispatch resolution may send: fm_typesafe_brief_task moves from bin/fm-dispatch-resolve.sh into bin/fm-typesafe-lib.sh and is shared, and every string either request can carry passes the dispatch-never-send check first. Keys come from the home .env through fm_typesafe_key and the new fm_openrouter_key, and reach Node on stdin. System assessment Dependency map: - bin/fm-spawn.sh -> bin/fm-skill-pick.sh (bounded by fm_run_timed at 30 s, process-group kill) -> bin/fm-typesafe-lib.sh (brief text, never-send, keys) and bin/fm-skill-pick.mjs -> vendored hyper-jev client -> TypeSafe direct, then OpenRouter on a failed direct call. - bin/fm-dispatch-resolve.sh -> fm_typesafe_brief_task (moved, unchanged behavior; tests/fm-dispatch-resolve.test.sh passes). - The launch section is written before the no-mistakes intent overlay, so it never becomes --intent text. - Runtime needs: node with .ts type stripping (Node 24+, as the vendored starter declares) and jq; without either the launch proceeds and records the reason. Blast radius: - Every ship and scout launch and relaunch gains one picker run and one "# Skill selection" section; secondmate charters are not touched. - No key: the picker stops after the key check (0.05 s), makes no network call, and records status unavailable; launch otherwise unchanged. - Hung or failing provider: each call is bounded at 10 s, the whole pick at 25 s inside the 30 s spawn bound. - The task record gains skill_selection=, skill_selection_reason= and skill_selection_picked=, owned and preserved across relaunch. Spawn latency: - Live, firstmate's 29 skills, synthetic task, TypeSafe direct: 0.90 s, 1.14 s, 1.50 s for the whole picker (jev-1.13.0, picked afk). - Offline, Vernant's real catalog: 245 skills in one Choice; request 1 is 48,689 bytes, request 2 is 3,903 bytes; 1.40 s local wall time at load average about 265 on 18 cores. Deviations from the examples: - bin/fm-skill-pick.mjs:169: a failed direct TypeSafe call is retried once through OpenRouter, which serves the rest of the pick; the starter picks one provider by key presence and never switches, and OpenRouter is the required fallback for a failed direct call. - bin/fm-typesafe-lib.sh:34: the OpenRouter key is read from the home .env only, never the ambient environment, where another project's key may live. - bin/fm-skill-pick.mjs:50-51: per-call timeoutMs 10 s and an overall 25 s signal, because the starter's 30 s default per call exceeds the spawn bound. - bin/fm-skill-pick.mjs:184-187: above 255 skills the Choice is split into which::N questions in the same request, as the cookbook advises; no skill is dropped. - bin/fm-skill-pick.mjs:72: SKILL.md frontmatter replaces the cookbook's prepared roster file; one description serves as description and description_full. - bin/fm-skill-pick.mjs:129: only Git-tracked SKILL.md files are sent, as before; other skills are listed by name as not judged. - bin/fm-skill-pick.sh:53: "read this skill in full and follow it" replaces the cookbook's "Ignore this if it does not fit". - The starter's default jev-latest replaces the cookbook's pinned jev-1.12. --- .../references/common/dispatch.md | 6 +- .../skills/operational-home-layout/SKILL.md | 2 +- bin/fm-brief-heading-lib.sh | 4 +- bin/fm-dispatch-resolve.sh | 82 +- bin/fm-skill-catalog.jq | 24 - bin/fm-skill-pick.mjs | 243 ++++ bin/fm-skill-pick.sh | 94 ++ bin/fm-skill-suggest.sh | 259 ---- bin/fm-spawn.sh | 108 +- bin/fm-test-run.sh | 21 +- bin/fm-typesafe-lib.sh | 116 +- docs/configuration.md | 72 +- docs/scripts.md | 2 +- tests/fm-skill-pick.test.sh | 228 ++++ tests/fm-skill-suggest.test.sh | 1116 ----------------- tests/fm-test-run.test.sh | 6 +- 16 files changed, 815 insertions(+), 1568 deletions(-) delete mode 100644 bin/fm-skill-catalog.jq create mode 100644 bin/fm-skill-pick.mjs create mode 100644 bin/fm-skill-pick.sh delete mode 100755 bin/fm-skill-suggest.sh create mode 100644 tests/fm-skill-pick.test.sh delete mode 100755 tests/fm-skill-suggest.test.sh diff --git a/.agents/skills/harness-adapters/references/common/dispatch.md b/.agents/skills/harness-adapters/references/common/dispatch.md index 328304cecb8..d171425da5a 100644 --- a/.agents/skills/harness-adapters/references/common/dispatch.md +++ b/.agents/skills/harness-adapters/references/common/dispatch.md @@ -11,10 +11,10 @@ When the opt-in `bin/fm-dispatch-resolve.sh` is on, its `clear` answer already n [`secondmate-provisioning`](../secondmate-provisioning/SKILL.md) owns inherited harness defaults and routing configuration. -## Skill advice at intake +## Skill selection at launch -Apply every existing mandatory explicit/named and safety trigger before considering optional advice. -Follow [advisory skill selection](../../../docs/configuration.md#advisory-skill-selection) at intake and on material intent changes; it owns permitted input, worker overlays, privacy, role coverage, and supported limits. +Every ship and scout spawn adds a picked project skill to the worker's launch instructions, after every existing mandatory explicit/named and safety trigger. +[Worker skill selection](../../../docs/configuration.md#worker-skill-selection) owns its input, privacy, provider order, and recorded result. ## Owners diff --git a/.agents/skills/operational-home-layout/SKILL.md b/.agents/skills/operational-home-layout/SKILL.md index 963c5741c6f..cd9a281b4f5 100644 --- a/.agents/skills/operational-home-layout/SKILL.md +++ b/.agents/skills/operational-home-layout/SKILL.md @@ -19,7 +19,7 @@ README.md public overview and development notes .claude/mods/ Claude Code mods (function-hooks plugins), committed; Calm's module may load through CLAUDE_CODE_ENABLE_FUNCTION_HOOKS or tengu_plugin_hooks_modules, but activates only under the exact opt-in docs/calm.md owns and is otherwise a complete no-op skills/ standalone public installer-facing skills, committed; not loaded by firstmate bin/ helper scripts, committed; read each script's header before first use -.env optional Relay pairing token (presence-gates section 14), mail-plane credentials (schema: docs/configuration.md "Mail plane"), and TypeSafe key TYPESAFE_API_KEY, kept once in the primary home and read by its secondmate homes through their parent record (docs/configuration.md "Typed dispatch resolution" and "Advisory skill selection"); LOCAL, gitignored +.env optional Relay pairing token (presence-gates section 14), mail-plane credentials (schema: docs/configuration.md "Mail plane"), TypeSafe key TYPESAFE_API_KEY, kept once in the primary home and read by its secondmate homes through their parent record, and OpenRouter fallback key OPENROUTER_API_KEY (docs/configuration.md "Typed dispatch resolution" and "Worker skill selection"); LOCAL, gitignored config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "default" = same as firstmate. Inherited as the literal file: a concrete primary adapter value also controls a secondmate home's own crewmates (section 4) config/claude-permission-mode optional one-token permission posture for every Claude worker launch: absent or "bypass" keeps --dangerously-skip-permissions, "auto" launches with --permission-mode auto; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Claude permission mode" config/claude-launcher optional Claude worker launcher; see docs/configuration.md "Claude launcher" for selection, proxy requirements, and inheritance diff --git a/bin/fm-brief-heading-lib.sh b/bin/fm-brief-heading-lib.sh index 60891f51107..e635a017749 100644 --- a/bin/fm-brief-heading-lib.sh +++ b/bin/fm-brief-heading-lib.sh @@ -3,8 +3,8 @@ # Usage: . bin/fm-brief-heading-lib.sh # # This file is the single owner of brief-section parsing for intent, -# validation, dispatch resolution, and advisory skill selection. -# Every consumer shares parsing semantics; resolver-only privacy preprocessing +# validation, dispatch resolution, and worker skill selection. +# Every consumer shares parsing semantics; never-send privacy preprocessing # is owned by docs/configuration.md "Never-send list". # Parse an exact ATX heading outside fenced blocks. Body mode prints through diff --git a/bin/fm-dispatch-resolve.sh b/bin/fm-dispatch-resolve.sh index f663be7f5f7..c349a245d4d 100755 --- a/bin/fm-dispatch-resolve.sh +++ b/bin/fm-dispatch-resolve.sh @@ -310,63 +310,13 @@ never_send_off() { } # Snapshot and validate privacy policy before reading any outgoing brief text. -NEVER_SEND_LIST='' MARKED_SECTIONS=0 +NEVER_SEND_LIST='' never_send_load() { - local value n=0 - fm_typesafe_policy_inspect "$NEVER_SEND_PATH" \ + fm_typesafe_policy_marked_sections "$NEVER_SEND_PATH" \ || never_send_off "$FM_TYPESAFE_WITHHELD_REASON" [ -e "$NEVER_SEND_PATH" ] || [ -L "$NEVER_SEND_PATH" ] || return 0 NEVER_SEND_LIST=$(jq -Rr 'gsub("\\s+"; " ")' "$NEVER_SEND_PATH" 2>/dev/null) \ || never_send_off "could not read $NEVER_SEND_PATH" - while IFS= read -r value; do - n=$((n + 1)) - value=${value# } - value=${value% } - case "$value" in - '# dispatch-never-send marked-sections') MARKED_SECTIONS=1 ;; - '#'*) - case "$(printf '%s' "${value#'#'}" | tr '[:upper:]' '[:lower:]')" in - dispatch-never-send*|' dispatch-never-send'*) - never_send_off "invalid privacy directive in $NEVER_SEND_PATH line $n" ;; - esac - ;; - esac - done <<<"$NEVER_SEND_LIST" -} - -# Markers are interpreted on the original brief, even inside Markdown fences, -# so protected headings cannot change extraction or cause a whole-brief fallback. -never_send_brief() { - local rc=0 - grep -qiE -e '") { - if (hidden) exit 1 - hidden = 1 - next - } - if (marker == "") { - if (!hidden) exit 1 - hidden = 0 - next - } - if (tolower($0) ~ /") { + if (hidden) exit 1 + hidden = 1 + next + } + if (marker == "") { + if (!hidden) exit 1 + hidden = 0 + next + } + if (tolower($0) ~ /`), unspaced (``), or differently cased (``) markers stop the entire request rather than being corrected. A brief without such text is sent as before. This option protects only the marked occurrences in the brief, not copies elsewhere or dispatch-rule text; use literals when those must also be withheld. -Do not send real Vernant/customer text until authorized: TypeSafe's public terms have not established the required `standard_confidential/v1` processor protections of deletion within 30 days and no training. +On 2026-10-07, the captain authorized worker skill selection to send the same sanitized, never-send-filtered task text used by `bin/fm-dispatch-resolve.sh` to TypeSafe and its OpenRouter fallback, ruling “I asked for A already.” This authorization does not widen any other sending boundary. +Outside that worker skill-selection authorization, do not send real Vernant/customer text until separately authorized: TypeSafe's public terms have not established the required `standard_confidential/v1` processor protections of deletion within 30 days and no training. -Before each request is sent, every remaining string in it is checked for literal matches: for dispatch resolution, the project name, the sanitized task text, each rule's `when`, and the fixed question text; for worker skill selection, the sanitized task text and every skill name, description, and opening excerpt either request can carry; belay checks every request string through `bin/fm-typesafe-lib.sh`. +Before each request is sent, every remaining string in it is checked for literal matches through `bin/fm-typesafe-lib.sh`: for dispatch resolution, the project name, the sanitized task text, each rule's `when`, and the fixed question text; for worker skill selection, each complete outbound request, including task text, question instructions, assembled skill criteria, and the requested model ID, before either direct or fallback transport; belay checks every request string. A literal match stops the request: the resolver behaves exactly as when it is off, printing one `dispatch-resolve: off (...; nothing sent)` line on stderr and nothing on stdout, making no network or quota call, and exiting 0, so firstmate dispatches through its existing intake. A list that cannot be inspected through its ancestors, is present but not a readable regular file, contains an invalid directive, or has a marker problem also stops the request the same way rather than sending unchecked text. That one diagnostic names the list line number at most and never prints the listed value or the matching text. @@ -1701,21 +1702,28 @@ To move to a newer upstream version, review it, update the pinned commit in this ## Worker skill selection -Every ship and scout launch and relaunch runs `bin/fm-skill-pick.sh` once its task copy exists, so the worker starts with the one project skill that fits its task. +Every ship and scout launch and relaunch with brief delivery runs `bin/fm-skill-pick.sh` once its task copy exists, so the worker starts with the one project skill that fits its task. Secondmate charters are not judged. The picker reads the task copy's `.agents/skills` and `.claude/skills`; a firstmate task's copy holds firstmate's own skills. It runs the [TypeSafe skill-suggestion cookbook](https://docs.typesafe.ai/cookbooks/skill_suggestion) recipe on the vendored [hyper-jev](../.agents/skills/hyper-jev/SOURCE.md) client, and `bin/fm-skill-pick.mjs`'s header owns the recipe, roster rules, and provider order. +For rosters over 255 skills, chunk-local top-three candidates enter a common Choice ranking before the detailed shortlist step; probabilities from independent chunks are never compared. This follows the cookbook's shortlist-over-chunk-winners guidance, retaining three candidates per chunk to preserve near-winners, and adds ranking requests to the ordinary two-request flow (`bin/fm-skill-pick.mjs:237-256`). + +**Named cookbook deviation — highest-fit delivery:** the rerank Choice is retained, but the delivered pick is the highest-fit shortlisted skill rather than the Choice winner, and its fit must be at least 0.30 (`bin/fm-skill-pick.mjs:270-280`). The captain chose this deviation because workers must read and follow the delivered skill, rather than receive the cookbook's ignorable advisory suggestion. + The pick is added to the launch instructions as a skill to read in full and follow. Existing mandatory skill triggers and the worker's own skill index still apply first and unchanged. A skill the picker could not send, such as one that is not a Git-tracked file in the task copy, is listed by name for the worker to check. When nothing fits, the instructions say so; when the picker cannot run, the launch continues and the instructions say why. -The task record carries `skill_selection=` (picked, none, or unavailable), `skill_selection_reason=`, and `skill_selection_picked=`. +The task record carries `skill_selection=` (picked, none, unavailable, or undelivered), `skill_selection_reason=`, and `skill_selection_picked=` when a pick is delivered. +Raw commands receive the selection only when they contain `__BRIEF__`; without that placeholder, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. Failure to publish the selection overlay also records undelivered and clears the pick. `bin/fm-spawn.sh` stops the picker after 30 seconds. -Only the brief text dispatch resolution may send is sent, after the same never-send checks; see the [never-send list](#typed-dispatch-resolution-env-typesafe_api_key). +Only the brief text dispatch resolution may send is sent, after the same never-send checks; see the [never-send list](#typed-dispatch-resolution-env-typesafe_api_key). For a promoted scout's ship relaunch, skill selection uses the recorded ship kind and current ship Firstmate spec rather than the superseded scout spec; ordinary dispatch extraction is unchanged. Privacy-hidden promotion instructions never revive the superseded spec. The `TYPESAFE_API_KEY` opt-in enables it, and TypeSafe is asked directly first. -When the direct call fails and `OPENROUTER_API_KEY` is set in the same home `.env`, the same request and the rest of that pick go through OpenRouter. +When the direct call is unavailable because its key is absent or fails and `OPENROUTER_API_KEY` is set in the same home `.env`, the same request and the rest of that pick go through OpenRouter. This is the captain's accepted policy: “openrouter is a fallback from directly using the typesafe api.” +Fallback provenance is retained for picked, no-selection, and unavailable outcomes. +Missing optional catalog directories are skipped; failures inspecting existing catalogs are unavailable with their actionable reason, not successful no-selection results. Keys reach the picker on stdin, never in its environment or arguments. [`tests/fm-skill-pick.test.sh`](../tests/fm-skill-pick.test.sh) exercises selection, both thresholds, the fallback, the never-send boundary, and chunking through the public tool. @@ -1733,7 +1741,7 @@ Required tools come in two parts: a universal toolchain every home needs regardl Every home requires: -- node and git. +- Node 24+ with native TypeScript loading enabled, and git. Worker skill selection capability-checks imports of its vendored TypeScript client and reports an unavailable prerequisite when they fail. - gh, with GitHub authentication through `gh auth login`. - no-mistakes v1.46.0 or newer. - Compatible gh-axi. diff --git a/tests/fm-skill-pick.test.sh b/tests/fm-skill-pick.test.sh index f235d2a9121..4dbdff4235e 100644 --- a/tests/fm-skill-pick.test.sh +++ b/tests/fm-skill-pick.test.sh @@ -7,6 +7,10 @@ set -eu # shellcheck source=tests/fixtures.sh . "$(dirname "${BASH_SOURCE[0]}")/fixtures.sh" command -v node >/dev/null 2>&1 || { printf 'ok - skipped: node is not installed\n'; exit 0; } +if ! prerequisite=$(node "$ROOT/bin/fm-skill-pick.mjs" check 2>&1); then + printf 'ok - skipped: %s\n' "$prerequisite" + exit 0 +fi TMP_ROOT=$(fm_test_tmproot fm-skill-pick) TOOL="$ROOT/bin/fm-skill-pick.sh" HOME_DIR="$TMP_ROOT/home" @@ -21,11 +25,14 @@ unset TYPESAFE_API_KEY TYPESAFE_API_KEY_PRIVATE OPENROUTER_API_KEY_PRIVATE cat > "$FAKE_FETCH" <<'JS' import { appendFileSync } from 'node:fs'; const env = process.env; +let directCalls = 0; globalThis.fetch = async (url, init) => { const provider = String(url).includes('openrouter') ? 'openrouter' : 'typesafe'; const body = JSON.parse(init.body); appendFileSync(env.FAKE_LOG, JSON.stringify({ provider, auth: init.headers.Authorization, body }) + '\n'); - const mode = provider === 'openrouter' ? env.FAKE_OPENROUTER || 'ok' : env.FAKE_TYPESAFE || 'ok'; + if (provider === 'typesafe') directCalls++; + const mode = provider === 'openrouter' ? env.FAKE_OPENROUTER || 'ok' : + env.FAKE_RERANK_FAIL && directCalls > 1 ? env.FAKE_RERANK_FAIL : env.FAKE_TYPESAFE || 'ok'; if (mode === 'hang') { return new Promise((_, reject) => init.signal.addEventListener('abort', () => reject(init.signal.reason))); } @@ -34,13 +41,26 @@ globalThis.fetch = async (url, init) => { for (const [id, q] of Object.entries(body.questions)) { if (q.type === 'noul') { const gate = Number(env.FAKE_GATE || 0.8); - const value = id === 'gate::prose_suffices' ? 1 - gate : id.startsWith('gate::') ? gate : Number(env.FAKE_FIT || 0.7); + const fits = JSON.parse(env.FAKE_FITS || '{}'); + const value = id === 'gate::prose_suffices' ? 1 - gate : id.startsWith('gate::') ? gate : + fits[id.slice('fits::'.length)] ?? Number(env.FAKE_FIT || 0.7); answers[id] = { type: 'noul', noul: value }; } else { const keys = Object.keys(q.criteria); const pick = keys.includes(env.FAKE_PICK) ? env.FAKE_PICK : keys[0]; - answers[id] = { type: 'choice', choice: pick, confidence: 1, - probabilities: Object.fromEntries(keys.map((k) => [k, k === pick ? 1 : 0])) }; + let probabilities = Object.fromEntries(keys.map((k) => [k, k === pick ? 1 : 0])); + if (env.FAKE_WEIGHTED) { + if (id === 'which::1') { + const leading = [0.30, 0.29, 0.28]; + probabilities = Object.fromEntries(keys.map((k, i) => [k, leading[i] ?? 0.13 / (keys.length - 3)])); + } else if (id === 'which::2') { + probabilities = Object.fromEntries(keys.map((k, i) => [k, [0.34, 0.33, 0.33][i]])); + } else { + probabilities = Object.fromEntries(keys.map((k) => [k, k === 'skill-1' ? 0.60 : 0.40 / (keys.length - 1)])); + } + } + const winner = Object.entries(probabilities).sort((a, b) => b[1] - a[1])[0][0]; + answers[id] = { type: 'choice', choice: winner, confidence: probabilities[winner], probabilities }; } } return new Response(JSON.stringify({ model: `${provider}-jev-test`, answers, usage: { input_tokens: 1, output_tokens: 1 } })); @@ -123,6 +143,14 @@ assert_contains "$out" "(best fit 0.20 below 0.3)" "a low best fit picks nothing assert_equals "status=none" "$(sed -n 1p "$TMP_ROOT/record")" "no fit is recorded as none" pass "the cookbook's two thresholds can each pick nothing" +reset; keys +out=$(pick FAKE_PICK=alpha FAKE_FITS='{"alpha":0.1,"beta":0.8,"gamma":0.4}') +assert_contains "$out" "- Picked for this task: beta" "the highest fitting shortlist skill wins even when Choice prefers a low-fit skill" +reset; keys +out=$(pick FAKE_FIT=0.30) +assert_equals "status=picked" "$(sed -n 1p "$TMP_ROOT/record")" "a fit exactly at .30 is accepted" +pass "selection uses the highest fit and the inclusive threshold" + reset; keys out=$(pick FAKE_TYPESAFE=500 FAKE_PICK=gamma) assert_contains "$out" "- Picked for this task: gamma" "the OpenRouter fallback still picks" @@ -148,6 +176,22 @@ out=$(OPENROUTER_API_KEY=ambient-key pick FAKE_TYPESAFE=500) assert_not_contains "$(cat "$LOG")" ambient-key "an ambient OpenRouter key is never used" pass "TypeSafe direct first, OpenRouter on a failed direct call, keys from the home .env only" +for exit_case in gate fit picked unavailable; do + reset; keys + case "$exit_case" in + gate) out=$(pick FAKE_TYPESAFE=500 FAKE_GATE=0.1) ;; + fit) out=$(pick FAKE_TYPESAFE=500 FAKE_FIT=0.1) ;; + picked) out=$(pick FAKE_TYPESAFE=500) ;; + unavailable) out=$(pick FAKE_TYPESAFE=500 FAKE_OPENROUTER=500) ;; + esac + assert_contains "$(sed -n 2p "$TMP_ROOT/record")" "TypeSafe direct failed" "fallback provenance survives $exit_case finalization" +done +reset; keys +out=$(pick FAKE_RERANK_FAIL=500) +assert_contains "$(sed -n 2p "$TMP_ROOT/record")" "TypeSafe direct failed" "rerank fallback provenance is attached" +assert_contains "$out" "through openrouter" "rerank fallback serves the final pick" +pass "fallback provenance is finalized for every outcome" + reset; keys printf 'Use for delta work\n' > "$HOME_DIR/config/dispatch-never-send" out=$(pick) @@ -160,18 +204,49 @@ assert_contains "$out" "withheld by dispatch-never-send policy" "a never-send li assert_equals 0 "$(requests)" "a withheld task sends nothing" pass "the dispatch-never-send policy covers every string a request can carry" +for forbidden in "It is described as: Use for alpha work." "Use for alpha work. — # alpha" "Exactly one of these skills" "Is the assistant being asked" "jev-latest"; do + reset; keys + printf '%s\n' "$forbidden" > "$HOME_DIR/config/dispatch-never-send" + out=$(pick) + assert_contains "$out" "withheld by dispatch-never-send policy" "the assembled body checks $forbidden" + case "$forbidden" in + "It is described"*|"Use for alpha"*|"Exactly one"*) expected=1 ;; + *) expected=0 ;; + esac + assert_equals "$expected" "$(requests)" "withheld assembled bodies never reach either provider" +done +reset; keys +printf '~typesafe/jev-latest\n' > "$HOME_DIR/config/dispatch-never-send" +out=$(pick FAKE_TYPESAFE=500) +assert_equals 1 "$(requests)" "the fallback model is separately checked before transport" +assert_contains "$out" "withheld by dispatch-never-send policy" "fallback privacy rejection is reported" +assert_contains "$(sed -n 2p "$TMP_ROOT/record")" "TypeSafe direct failed" "fallback privacy rejection retains direct provenance" +pass "actual assembled criteria, instructions and both provider models are checked" +reset; keys +skill "$COPY/.agents/skills" gamma 'description: Use for report work.' +printf 'gamma\n' > "$HOME_DIR/config/dispatch-never-send" +out=$(pick) +assert_contains "$out" "withheld by dispatch-never-send policy" "skill identities in Choice keys are checked" +assert_equals 0 "$(requests)" "a forbidden Choice key stops the first request" +skill "$COPY/.agents/skills" gamma 'description: "Use for gamma work."' + reset; keys BIG="$TMP_ROOT/big" git init -q "$BIG" -for i in $(seq 1 300); do skill "$BIG/skills" "skill-$i" "description: Use for numbered task $i."; done +for i in $(seq 1 258); do skill "$BIG/skills" "skill-$i" "description: Use for numbered task $i."; done git -C "$BIG" add -- skills -out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" FAKE_PICK=skill-7 \ - bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$BIG/skills") -assert_contains "$out" "- Picked for this task: skill-7" "a skill in a later chunk can win" +out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" FAKE_WEIGHTED=1 \ + FAKE_FITS='{"skill-1":0.9}' FAKE_FIT=0.1 bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$BIG/skills") +assert_contains "$out" "- Picked for this task: skill-1" "a lower local probability survives into a common ranking" jq -se '.[0].body.questions | (keys | map(select(startswith("which"))) | sort) == ["which::1","which::2"] - and ([.["which::1"].criteria, .["which::2"].criteria | keys | length] | add) == 300' "$LOG" >/dev/null \ - || fail "300 skills are split into Choice questions of at most 255, none dropped" -pass "a roster above 255 skills is ranked in chunks" + and ([.["which::1"].criteria, .["which::2"].criteria | keys | length] | add) == 258' "$LOG" >/dev/null \ + || fail "258 skills are split into Choice questions of at most 255, none dropped" +jq -se '.[1].body.questions.which.criteria | has("skill-1") and has("skill-99") and length == 6' "$LOG" >/dev/null \ + || fail "chunk shortlists enter a common ranking before detailed rerank" +jq -se 'all(.[]; all(.body.questions[]; .type != "choice" or (.criteria | length) <= 255))' "$LOG" >/dev/null \ + || fail "every Choice stays within the transport limit" +assert_equals 3 "$(requests)" "a chunked roster adds the common ranking request" +pass "non-point-mass chunk probabilities are not globally flattened" reset; keys out=$(pick FAKE_TYPESAFE=hang FAKE_PICK=alpha) @@ -225,4 +300,185 @@ grep -qx 'skill_selection_reason=no TypeSafe or OpenRouter key' "$SPAWN_HOME/sta assert_equals 0 "$(requests)" "nothing is sent without a key" pass "a launch with no key proceeds and records why nothing was picked" + +reset; keys +out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$TMP_ROOT/missing" --catalog "$COPY/.agents/skills") +assert_contains "$out" "- Picked for this task:" "a missing optional catalog is skipped" +reset; keys +out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$TMP_ROOT/brief.md") +assert_contains "$out" "could not enumerate skill catalog" "an existing non-directory catalog is unavailable" +assert_contains "$out" "$TMP_ROOT/brief.md" "the catalog failure identifies its path" +assert_equals 0 "$(requests)" "a broken catalog never sends a partial roster" +cat > "$TMP_ROOT/io-failure.mjs" <<'JS' +import fs from 'node:fs'; +import { syncBuiltinESMExports } from 'node:module'; +const original = fs[process.env.FAIL_OPERATION]; +fs[process.env.FAIL_OPERATION] = (path, ...args) => { + if (String(path) === process.env.FAIL_PATH) { + throw Object.assign(new Error(`permission denied: ${path}`), { code: 'EACCES' }); + } + return original(path, ...args); +}; +syncBuiltinESMExports(); +JS +for operation in readdirSync openSync; do + reset; keys + case "$operation" in + readdirSync) failure_path="$(cd "$COPY" && pwd -P)/.agents/skills" ;; + openSync) failure_path="$(cd "$COPY" && pwd -P)/.agents/skills/alpha/SKILL.md" ;; + esac + out=$(env FAIL_OPERATION="$operation" FAIL_PATH="$failure_path" FAKE_LOG="$LOG" \ + NODE_OPTIONS="--import=$FAKE_FETCH --import=$TMP_ROOT/io-failure.mjs" FM_HOME="$HOME_DIR" \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$COPY/.agents/skills") + assert_contains "$out" "permission denied" "existing catalog I/O failures remain actionable" + assert_contains "$out" "$failure_path" "the unreadable entry is identified" + assert_equals 0 "$(requests)" "I/O errors prevent transport" +done +pass "only absent catalogs are optional" + +cat > "$TMP_ROOT/unsupported-loader.mjs" <<'JS' +export async function resolve(specifier, context, nextResolve) { + if (specifier.endsWith('.ts')) { + throw Object.assign(new Error('Unknown file extension ".ts"'), { code: 'ERR_UNKNOWN_FILE_EXTENSION' }); + } + return nextResolve(specifier, context); +} +JS +reset; keys +out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH --loader=$TMP_ROOT/unsupported-loader.mjs" FM_HOME="$HOME_DIR" \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$COPY/.agents/skills") +assert_contains "$out" "unsupported Node runtime" "failed imports explain the TypeScript runtime prerequisite" +assert_equals 0 "$(requests)" "unsupported runtimes never start selection" +pass "the prerequisite imports the actual vendored modules" + +for exit_case in gate fit picked unavailable; do + reset + printf 'OPENROUTER_API_KEY=or-test-key\n' > "$HOME_DIR/.env" + case "$exit_case" in + gate) out=$(pick FAKE_GATE=0.1) ;; + fit) out=$(pick FAKE_FIT=0.1) ;; + picked) out=$(pick) ;; + unavailable) out=$(pick FAKE_OPENROUTER=500) ;; + esac + assert_contains "$(sed -n 2p "$TMP_ROOT/record")" "no TypeSafe key; used OpenRouter" "key-only fallback provenance survives $exit_case" +done +pass "OpenRouter key-only fallback retains provenance" + +cp "$TMP_ROOT/brief.md" "$TMP_ROOT/ordinary-brief.md" +cat > "$TMP_ROOT/brief.md" <<'EOF' +# Task +This is a SCOUT task: the deliverable is a written report, not a PR. +## Captain's intent +PERMITTED-ORIGINAL-INTENT +## Firstmate spec +STALE-SCOUT-SPEC +# Current ship Firstmate spec +PERMITTED-CURRENT-SHIP-SPEC +EOF +reset; keys +out=$(pick) +sent=$(jq -r '.body.state.request' "$LOG") +assert_contains "$sent" PERMITTED-ORIGINAL-INTENT "promoted ships keep the original intent" +assert_contains "$sent" PERMITTED-CURRENT-SHIP-SPEC "promoted ships send the current spec" +assert_not_contains "$sent" STALE-SCOUT-SPEC "promoted ships omit the stale scout spec" +assert_not_contains "$sent" "Brief kind: scout" "promoted ships omit the scout tag" +. "$ROOT/bin/fm-typesafe-lib.sh" +fm_typesafe_brief_task "$TMP_ROOT/brief.md" "$HOME_DIR/config/dispatch-never-send" "$TMP_ROOT/default-task" +assert_contains "$(cat "$TMP_ROOT/default-task")" STALE-SCOUT-SPEC "dispatch's three-argument extraction remains unchanged" +assert_contains "$(cat "$TMP_ROOT/default-task")" "Brief kind: scout" "dispatch retains the scout tag" +assert_not_contains "$(cat "$TMP_ROOT/default-task")" PERMITTED-CURRENT-SHIP-SPEC "dispatch does not switch specs" +fm_typesafe_brief_task "$TMP_ROOT/brief.md" "$HOME_DIR/config/dispatch-never-send" "$TMP_ROOT/scout-task" scout +assert_contains "$(cat "$TMP_ROOT/scout-task")" STALE-SCOUT-SPEC "explicit scouts retain their spec" +for hidden in whole partial; do + reset; keys + printf '# dispatch-never-send marked-sections\n' > "$HOME_DIR/config/dispatch-never-send" + case "$hidden" in + whole) marker=''$'\n''# Current ship Firstmate spec' ;; + partial) marker='# Current ship Firstmate spec'$'\n''' ;; + esac + printf '# Task\n## Captain'"'"'s intent\nPERMITTED-ORIGINAL-INTENT\n## Firstmate spec\nSTALE-SCOUT-SPEC\n%s\nHIDDEN-CURRENT-SPEC\n\n' "$marker" > "$TMP_ROOT/brief.md" + [ "$hidden" != partial ] || printf 'PERMITTED-SPEC-REMAINDER\n' >> "$TMP_ROOT/brief.md" + out=$(pick) + sent=$(jq -r '.body.state.request' "$LOG") + assert_contains "$sent" PERMITTED-ORIGINAL-INTENT "hidden overrides retain permitted intent" + assert_not_contains "$sent" HIDDEN-CURRENT-SPEC "marked override text never leaves" + assert_not_contains "$sent" STALE-SCOUT-SPEC "hidden overrides never revive stale instructions" + [ "$hidden" != partial ] || assert_contains "$sent" PERMITTED-SPEC-REMAINDER "permitted override remainder is retained" +done +mv "$TMP_ROOT/ordinary-brief.md" "$TMP_ROOT/brief.md" +pass "promotion and marked regions preserve effective task privacy" + +for raw in plain brief; do + reset + spawn_case "raw-$raw" + printf 'TYPESAFE_API_KEY=ts-spawn-key\n' > "$SPAWN_HOME/.env" + case "$raw" in + plain) harness='custom-agent --flag' ;; + brief) harness='custom-agent --brief __BRIEF__' ;; + esac + out=$(FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FAKE_PICK=alpha FM_FAKE_LAUNCH_LOG="$TMP_ROOT/raw-$raw.launch" \ + fm_test_run_spawn "$SPAWN_HOME" "$SPAWN_POOL" "$SPAWN_FAKEBIN" "raw-$raw" "$SPAWN_PROJECT" --harness "$harness" --mode no-mistakes --yolo off) + meta="$SPAWN_HOME/state/raw-$raw.meta" + case "$raw" in + plain) + assert_equals 0 "$(requests)" "raw launches without brief transport do not invoke the picker" + grep -qx 'skill_selection=undelivered' "$meta" || fail "raw launch records undelivered" + assert_not_contains "$(cat "$meta")" "skill_selection_picked=" "undelivered launches never record a pick" + assert_contains "$(cat "$TMP_ROOT/raw-$raw.launch")" 'custom-agent --flag' "the raw command is retained" + ;; + brief) + assert_equals 2 "$(requests)" "raw launches with a brief perform normal selection" + grep -qx 'skill_selection=picked' "$meta" || fail "raw brief launch records the pick" + assert_contains "$(cat "$SPAWN_HOME/data/raw-$raw/launch-brief.md")" "- Picked for this task: alpha" "raw brief transport receives the section" + ;; + esac +done +pass "raw launch records match delivery capability" + +reset +spawn_case failed-overlay +printf 'TYPESAFE_API_KEY=ts-spawn-key\n' > "$SPAWN_HOME/.env" +real_mv=$(command -v mv) +cat > "$SPAWN_FAKEBIN/mv" <<'SH' +#!/usr/bin/env bash +if [[ "${2:-}" == */launch-brief.md ]]; then + if [ -e "$PUBLICATION_COUNT" ]; then exit 1; fi + : > "$PUBLICATION_COUNT" +fi +exec "$REAL_MV" "$@" +SH +chmod +x "$SPAWN_FAKEBIN/mv" +out=$(FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FAKE_PICK=alpha \ + REAL_MV="$real_mv" PUBLICATION_COUNT="$TMP_ROOT/publications" \ + fm_test_run_spawn "$SPAWN_HOME" "$SPAWN_POOL" "$SPAWN_FAKEBIN" failed-overlay "$SPAWN_PROJECT" --mode no-mistakes --yolo off) +assert_equals 2 "$(requests)" "selection completed before the overlay publication failed" +grep -qx 'skill_selection=undelivered' "$SPAWN_HOME/state/failed-overlay.meta" || fail "overlay failure is recorded as undelivered" +assert_not_contains "$(cat "$SPAWN_HOME/state/failed-overlay.meta")" "skill_selection_picked=" "an unpublished pick is not recorded" +assert_not_contains "$(cat "$SPAWN_HOME/data/failed-overlay/launch-brief.md")" "# Skill selection" "publication failure retains the original launch brief" +pass "successful picks are not claimed when overlay publication fails" + +reset; keys +fm_typesafe_brief_task "$TMP_ROOT/brief.md" "$HOME_DIR/config/dispatch-never-send" "$TMP_ROOT/explicit-scout-task" scout +assert_contains "$(cat "$TMP_ROOT/explicit-scout-task")" "Brief kind: scout" "recorded scout kind does not depend on a brief sentinel" +cat > "$TMP_ROOT/fully-hidden.md" <<'EOF' +# Task +This is a SCOUT task: the deliverable is a written report, not a PR. + +## Captain's intent +HIDDEN-INTENT + +## Firstmate spec +STALE-SCOUT-SPEC + +# Current ship Firstmate spec +HIDDEN-CURRENT-SPEC + +EOF +printf '# dispatch-never-send marked-sections\n' > "$HOME_DIR/config/dispatch-never-send" +fm_typesafe_brief_task "$TMP_ROOT/fully-hidden.md" "$HOME_DIR/config/dispatch-never-send" "$TMP_ROOT/hidden-task" ship +assert_equals "" "$(cat "$TMP_ROOT/hidden-task")" "fully hidden effective promoted task does not fall back to stale boilerplate" +pass "explicit kind and entirely hidden promotion preserve task semantics" + printf '# all fm-skill-pick tests passed\n' From 70c102d19131e8198e84213ed19f0757b315c1ad Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Wed, 7 Oct 2026 20:24:02 -0500 Subject: [PATCH 03/12] no-mistakes(review): Fix picker path fidelity, catalog outcomes, and prerequisite skips --- bin/fm-skill-pick.mjs | 22 ++++-- bin/fm-skill-pick.sh | 4 +- docs/configuration.md | 8 +-- tests/fm-skill-pick.test.sh | 138 ++++++++++++++++++++++++++++++++++-- 4 files changed, 153 insertions(+), 19 deletions(-) diff --git a/bin/fm-skill-pick.mjs b/bin/fm-skill-pick.mjs index 868dec70ca1..27b269e3c74 100644 --- a/bin/fm-skill-pick.mjs +++ b/bin/fm-skill-pick.mjs @@ -14,7 +14,7 @@ // sent only when /SKILL.md is a regular Git-tracked file with a // description, and every other one is listed in not_judged with its reason. // pick prints status=, reason=, picked=, path=, fit=, provider= and model= -// lines; status is picked, none or unavailable. +// lines; path is a JSON string, and status is picked, none or unavailable. import { execFileSync } from 'node:child_process'; import { closeSync, lstatSync, openSync, readFileSync, readSync, readdirSync, realpathSync } from 'node:fs'; import { join } from 'node:path'; @@ -25,7 +25,11 @@ try { ({ choice, noul } = await import('../.agents/skills/hyper-jev/templates/starter/src/core/helpers.ts')); ({ LIMITS } = await import('../.agents/skills/hyper-jev/templates/starter/src/core/types.ts')); } catch (error) { - process.stderr.write(`unsupported Node runtime: importing the vendored TypeScript client requires Node with TypeScript support (${error.message})\n`); + if (error.code === 'ERR_UNKNOWN_FILE_EXTENSION' && error.message.includes('".ts"')) { + process.stderr.write(`unsupported Node runtime: importing the vendored TypeScript client requires Node with TypeScript support (${error.message})\n`); + process.exit(77); + } + process.stderr.write(`could not load vendored TypeScript client: ${error.message}\n`); process.exit(1); } @@ -154,7 +158,7 @@ function line(value) { async function pick(taskFile, rosterFile, policyPath, scratchPath) { const request = readFileSync(taskFile, 'utf8'); - const { skills } = JSON.parse(readFileSync(rosterFile, 'utf8')); + const { skills, not_judged = [] } = JSON.parse(readFileSync(rosterFile, 'utf8')); const [typesafe = '', openrouter = ''] = readFileSync(0, 'utf8').split('\n'); const keys = { typesafe: typesafe.trim(), openrouter: openrouter.trim() }; const byName = new Map(skills.map((skill) => [skill.name, skill])); @@ -164,10 +168,14 @@ async function pick(taskFile, rosterFile, policyPath, scratchPath) { const done = () => { if (directFailure) out.reason = [out.reason, `TypeSafe direct failed (${directFailure})`].filter(Boolean).join('; '); if (fallbackReason) out.reason = [out.reason, fallbackReason].filter(Boolean).join('; '); - for (const [key, value] of Object.entries(out)) process.stdout.write(`${key}=${line(value)}\n`); + for (const [key, value] of Object.entries(out)) process.stdout.write(`${key}=${key === 'path' ? JSON.stringify(value) : line(value)}\n`); }; if (!skills.length) { - Object.assign(out, { status: 'none', reason: 'this project has no skills to judge' }); + if (not_judged.length) { + out.reason = `no project skills could be judged: ${not_judged.map(({ name, reason }) => `${name} (${reason})`).join(', ')}`; + } else { + Object.assign(out, { status: 'none', reason: 'this project has no skills to judge' }); + } return done(); } const client = (provider) => { @@ -231,7 +239,7 @@ async function pick(taskFile, rosterFile, policyPath, scratchPath) { const gate = oriented.reduce((sum, value) => sum + value, 0) / oriented.length; Object.assign(out, { provider: current.provider, model: wide.meta.resolvedModel }); if (gate < GATE_THRESHOLD) { - Object.assign(out, { status: 'none', reason: `need ${gate.toFixed(2)} below ${GATE_THRESHOLD}` }); + Object.assign(out, { status: 'none', reason: `no judged project skill fits this task: need ${gate.toFixed(2)} below ${GATE_THRESHOLD}` }); return done(); } const rank = (answers) => Object.entries(answers.probabilities).sort((a, b) => b[1] - a[1]); @@ -272,7 +280,7 @@ async function pick(taskFile, rosterFile, policyPath, scratchPath) { const best = result.answers[`fits::${winner.name}`].noul; Object.assign(out, { provider: current.provider, model: result.meta.resolvedModel }); if (best < FITS_THRESHOLD) { - Object.assign(out, { status: 'none', reason: `best fit ${best.toFixed(2)} below ${FITS_THRESHOLD}` }); + Object.assign(out, { status: 'none', reason: `no judged project skill fits this task: best fit ${best.toFixed(2)} below ${FITS_THRESHOLD}` }); return done(); } Object.assign(out, { diff --git a/bin/fm-skill-pick.sh b/bin/fm-skill-pick.sh index ee14392cb01..2abfab93c50 100644 --- a/bin/fm-skill-pick.sh +++ b/bin/fm-skill-pick.sh @@ -56,7 +56,7 @@ finish() { printf -- '- Picked for this task: %s - read %s\n' "$PICKED" "$SKILL_PATH" printf "Picked by %s through %s from this project's skills (fit %s); your skill index still applies for anything else this task needs.\n" "$MODEL" "$PROVIDER" "$FIT" ;; - none) printf 'Skill selection found no project skill that fits this task (%s); use your skill index as usual.\n' "$REASON" ;; + none) printf 'Skill selection found no fit among the judged project skills (%s); use your skill index as usual.\n' "$REASON" ;; *) printf 'Skill selection was unavailable for this task (%s). This does not mean no skill applies: check your skill index for skills that fit this task before starting work.\n' "$REASON" ;; esac [ ! -s "$WORK/not-judged" ] || printf 'Not judged, so check them yourself if relevant: %s.\n' "$(cat "$WORK/not-judged")" @@ -84,7 +84,7 @@ printf '%s\n%s\n' "$TYPESAFE_API_KEY_PRIVATE" "$OPENROUTER_API_KEY_PRIVATE" | node "$SCRIPT_DIR/fm-skill-pick.mjs" pick "$WORK/task" "$WORK/roster.json" "$CONFIG/dispatch-never-send" "$WORK/scan" > "$WORK/result" 2> "$WORK/error" || unavailable "the skill picker stopped with an error: $(cat "$WORK/error")" field() { sed -n "s/^$1=//p" "$WORK/result" | head -n 1; } -STATUS=$(field status) REASON=$(field reason) PICKED=$(field picked) SKILL_PATH=$(field path) +STATUS=$(field status) REASON=$(field reason) PICKED=$(field picked) SKILL_PATH=$(field path | jq -r '.') FIT=$(field fit) PROVIDER=$(field provider) MODEL=$(field model) case "$STATUS" in picked|none) ;; diff --git a/docs/configuration.md b/docs/configuration.md index ba84e7cad26..de4e0390092 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1707,14 +1707,14 @@ Secondmate charters are not judged. The picker reads the task copy's `.agents/skills` and `.claude/skills`; a firstmate task's copy holds firstmate's own skills. It runs the [TypeSafe skill-suggestion cookbook](https://docs.typesafe.ai/cookbooks/skill_suggestion) recipe on the vendored [hyper-jev](../.agents/skills/hyper-jev/SOURCE.md) client, and `bin/fm-skill-pick.mjs`'s header owns the recipe, roster rules, and provider order. -For rosters over 255 skills, chunk-local top-three candidates enter a common Choice ranking before the detailed shortlist step; probabilities from independent chunks are never compared. This follows the cookbook's shortlist-over-chunk-winners guidance, retaining three candidates per chunk to preserve near-winners, and adds ranking requests to the ordinary two-request flow (`bin/fm-skill-pick.mjs:237-256`). +For rosters over 255 skills, chunk-local top-three candidates enter a common Choice ranking before the detailed shortlist step; probabilities from independent chunks are never compared. This follows the cookbook's shortlist-over-chunk-winners guidance, retaining three candidates per chunk to preserve near-winners, and adds ranking requests to the ordinary two-request flow (`bin/fm-skill-pick.mjs:245-264`). -**Named cookbook deviation — highest-fit delivery:** the rerank Choice is retained, but the delivered pick is the highest-fit shortlisted skill rather than the Choice winner, and its fit must be at least 0.30 (`bin/fm-skill-pick.mjs:270-280`). The captain chose this deviation because workers must read and follow the delivered skill, rather than receive the cookbook's ignorable advisory suggestion. +**Named cookbook deviation — highest-fit delivery:** the rerank Choice is retained, but the delivered pick is the highest-fit shortlisted skill rather than the Choice winner, and its fit must be at least 0.30 (`bin/fm-skill-pick.mjs:278-288`). The captain chose this deviation because workers must read and follow the delivered skill, rather than receive the cookbook's ignorable advisory suggestion. The pick is added to the launch instructions as a skill to read in full and follow. Existing mandatory skill triggers and the worker's own skill index still apply first and unchanged. A skill the picker could not send, such as one that is not a Git-tracked file in the task copy, is listed by name for the worker to check. -When nothing fits, the instructions say so; when the picker cannot run, the launch continues and the instructions say why. +No-fit outcomes apply only to judged skills; when every discovered skill is excluded, selection is unavailable with the exclusion reasons, not a finding that no skill fits. A genuinely empty catalog reports that there are no skills to judge. When the picker cannot run, the launch continues and the instructions say why. The task record carries `skill_selection=` (picked, none, unavailable, or undelivered), `skill_selection_reason=`, and `skill_selection_picked=` when a pick is delivered. Raw commands receive the selection only when they contain `__BRIEF__`; without that placeholder, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. Failure to publish the selection overlay also records undelivered and clears the pick. `bin/fm-spawn.sh` stops the picker after 30 seconds. @@ -1741,7 +1741,7 @@ Required tools come in two parts: a universal toolchain every home needs regardl Every home requires: -- Node 24+ with native TypeScript loading enabled, and git. Worker skill selection capability-checks imports of its vendored TypeScript client and reports an unavailable prerequisite when they fail. +- Node 24+ with native TypeScript loading enabled, and git. Worker skill selection capability-checks imports of its vendored TypeScript client; unsupported TypeScript loading is a runtime prerequisite, while other import failures report a client-loading error. The picker test suite emits canonical `skip:` output only for missing Node or unsupported TypeScript loading; other client-loading errors fail the suite. - gh, with GitHub authentication through `gh auth login`. - no-mistakes v1.46.0 or newer. - Compatible gh-axi. diff --git a/tests/fm-skill-pick.test.sh b/tests/fm-skill-pick.test.sh index 4dbdff4235e..e3f996e7bba 100644 --- a/tests/fm-skill-pick.test.sh +++ b/tests/fm-skill-pick.test.sh @@ -6,10 +6,17 @@ set -eu # shellcheck source=tests/fixtures.sh . "$(dirname "${BASH_SOURCE[0]}")/fixtures.sh" -command -v node >/dev/null 2>&1 || { printf 'ok - skipped: node is not installed\n'; exit 0; } -if ! prerequisite=$(node "$ROOT/bin/fm-skill-pick.mjs" check 2>&1); then - printf 'ok - skipped: %s\n' "$prerequisite" - exit 0 +command -v node >/dev/null 2>&1 || { printf 'skip: node not found\n'; exit 0; } +if prerequisite=$(node "$ROOT/bin/fm-skill-pick.mjs" check 2>&1); then + : +else + prerequisite_status=$? + if [ "$prerequisite_status" -eq 77 ]; then + printf 'skip: unsupported Node runtime\n%s\n' "$prerequisite" + exit 0 + fi + printf '%s\n' "$prerequisite" >&2 + exit "$prerequisite_status" fi TMP_ROOT=$(fm_test_tmproot fm-skill-pick) TOOL="$ROOT/bin/fm-skill-pick.sh" @@ -133,13 +140,87 @@ assert_not_contains "$(cat "$LOG")" CLAUDE-COPY-DESCRIPTION "a later folder's co assert_not_contains "$out" ts-test-key "the key is never printed" pass "a clear task picks one project skill through TypeSafe direct" +for whitespace in spaces tab newline; do + reset; keys + case "$whitespace" in + spaces) path_project="$TMP_ROOT/project repeated spaces" ;; + tab) path_project="$TMP_ROOT/"$'project\twith\ttabs' ;; + newline) path_project="$TMP_ROOT/"$'Client Reports\tQuarter\nNotes' ;; + esac + git init -q "$path_project" + skill "$path_project/skills" alpha 'description: Use for alpha work.' + git -C "$path_project" add -- skills + selected_path="$(cd "$path_project" && pwd -P)/skills/alpha/SKILL.md" + out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$path_project/skills" --record "$TMP_ROOT/record") + assert_contains "$out" "- Picked for this task: alpha - read $selected_path" "the launch instruction preserves $whitespace in the project path" + [ -f "$selected_path" ] || fail "the emitted selected path identifies the existing skill" + grep -qx 'status=picked' "$TMP_ROOT/record" || fail "the whitespace project selection is recorded" + assert_equals 2 "$(requests)" "a whitespace project path completes selection" +done +pass "selected paths preserve repeated spaces, tabs and embedded newlines" + +for roster_case in absent empty untracked undescribed partial; do + roster="$TMP_ROOT/roster-$roster_case" + git init -q "$roster" + case "$roster_case" in + absent) ;; + empty) mkdir -p "$roster/skills" ;; + untracked) skill "$roster/skills" scratch 'description: Untracked local skill.' ;; + undescribed) skill "$roster/skills" blank ''; git -C "$roster" add -- skills ;; + partial) + skill "$roster/skills" alpha 'description: Use for alpha work.' + skill "$roster/skills" blank '' + git -C "$roster" add -- skills + skill "$roster/skills" scratch 'description: Untracked local skill.' + ;; + esac + thresholds=(gate) + [ "$roster_case" != partial ] || thresholds+=(fit) + for threshold in "${thresholds[@]}"; do + reset; keys + if [ "$threshold" = gate ]; then threshold_env=FAKE_GATE=0.1 + else threshold_env=FAKE_FIT=0.2; fi + out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" "$threshold_env" \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$roster/skills" --record "$TMP_ROOT/record") + case "$roster_case" in + absent|empty) + status=none; reason='this project has no skills to judge'; count=0 + assert_contains "$out" "Skill selection found no fit among the judged project skills ($reason)" "an $roster_case roster reports no skills" + ;; + untracked|undescribed) + status=unavailable; count=0 + if [ "$roster_case" = untracked ]; then excluded='scratch (not a Git-tracked file in this project)' + else excluded='blank (no readable description)'; fi + reason="no project skills could be judged: $excluded" + assert_contains "$out" "Skill selection was unavailable for this task ($reason)" "an entirely excluded roster is unavailable" + assert_contains "$out" "Not judged, so check them yourself if relevant: $excluded." "excluded skills remain visible" + ;; + partial) + status=none + if [ "$threshold" = gate ]; then reason='no judged project skill fits this task: need 0.10 below 0.3'; count=1 + else reason='no judged project skill fits this task: best fit 0.20 below 0.3'; count=2; fi + assert_contains "$out" "Skill selection found no fit among the judged project skills ($reason)" "a low $threshold applies only to judged skills" + assert_contains "$out" "blank (no readable description)" "an undescribed skill remains visible beside a low $threshold" + assert_contains "$out" "scratch (not a Git-tracked file in this project)" "an untracked skill remains visible beside a low $threshold" + jq -se 'all(.[]; (.body.questions.which.criteria | keys) == ["alpha"])' "$LOG" >/dev/null \ + || fail "only the judged skill enters requests" + ;; + esac + grep -Fqx "status=$status" "$TMP_ROOT/record" || fail "the $roster_case $threshold status is recorded" + grep -Fqx "reason=$reason" "$TMP_ROOT/record" || fail "the $roster_case $threshold reason is recorded" + assert_equals "$count" "$(requests)" "the $roster_case $threshold request count" + done +done +pass "empty, excluded and partially judged rosters have distinct outcomes" + reset; keys out=$(pick FAKE_GATE=0.1) -assert_contains "$out" "Skill selection found no project skill that fits this task (need 0.10 below 0.3)" "a low need picks nothing" +assert_contains "$out" "Skill selection found no fit among the judged project skills (no judged project skill fits this task: need 0.10 below 0.3)" "a low need picks nothing" assert_equals 1 "$(requests)" "a low need stops after request 1" reset; keys out=$(pick FAKE_FIT=0.2) -assert_contains "$out" "(best fit 0.20 below 0.3)" "a low best fit picks nothing" +assert_contains "$out" "Skill selection found no fit among the judged project skills (no judged project skill fits this task: best fit 0.20 below 0.3)" "a low best fit is limited to judged skills" assert_equals "status=none" "$(sed -n 1p "$TMP_ROOT/record")" "no fit is recorded as none" pass "the cookbook's two thresholds can each pick nothing" @@ -352,6 +433,51 @@ out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH --loader=$TMP_ROOT/ assert_contains "$out" "unsupported Node runtime" "failed imports explain the TypeScript runtime prerequisite" assert_equals 0 "$(requests)" "unsupported runtimes never start selection" pass "the prerequisite imports the actual vendored modules" +cat > "$TMP_ROOT/missing-module-loader.mjs" <<'JS' +export async function resolve(specifier, context, nextResolve) { + if (specifier.endsWith('.ts')) { + throw Object.assign(new Error('Cannot find vendored TypeScript module'), { code: 'ERR_MODULE_NOT_FOUND' }); + } + return nextResolve(specifier, context); +} +JS +for loader_case in unsupported missing-module; do + loader_options="--no-warnings --loader=$TMP_ROOT/$loader_case-loader.mjs" + if check_out=$(env NODE_OPTIONS="$loader_options" node "$ROOT/bin/fm-skill-pick.mjs" check 2>&1); then + check_status=0 + else check_status=$?; fi + if [ "$loader_case" = unsupported ]; then + assert_equals 77 "$check_status" "unsupported TypeScript imports have a distinct check exit" + assert_contains "$check_out" "unsupported Node runtime:" "the executable check diagnoses an unsupported runtime" + expected_suite_status=0 + else + assert_equals 1 "$check_status" "missing modules fail the executable check" + assert_contains "$check_out" "could not load vendored TypeScript client: Cannot find vendored TypeScript module" "missing modules are loading failures" + assert_not_contains "$check_out" "unsupported Node runtime" "a missing module is not an unsupported runtime" + expected_suite_status=1 + fi + reset; keys + out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH $loader_options" FM_HOME="$HOME_DIR" \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$COPY/.agents/skills" --record "$TMP_ROOT/record") + assert_contains "$out" "$check_out" "the picker reports the actual loader failure" + grep -qx 'status=unavailable' "$TMP_ROOT/record" || fail "loader failure is recorded as unavailable" + assert_equals 0 "$(requests)" "loader failures send no requests" + if suite_out=$(env NODE_OPTIONS="$loader_options" bash "$ROOT/tests/fm-skill-pick.test.sh" 2>&1); then + suite_status=0 + else suite_status=$?; fi + assert_equals "$expected_suite_status" "$suite_status" "the suite distinguishes $loader_case prerequisites" + if [ "$loader_case" = unsupported ]; then + assert_equals "skip: unsupported Node runtime"$'\n'"$check_out" "$suite_out" "unsupported runtime suite skips are canonical" + else + assert_equals "$check_out" "$suite_out" "a broken vendored import fails the suite without skipping" + fi +done +mkdir -p "$TMP_ROOT/no-node-bin" +ln -s "$(command -v dirname)" "$TMP_ROOT/no-node-bin/dirname" +suite_out=$(env PATH="$TMP_ROOT/no-node-bin" ROOT="$ROOT" FM_TEST_LIB_SOURCED=1 FM_TEST_FIXTURES_SOURCED=1 \ + /bin/bash "$ROOT/tests/fm-skill-pick.test.sh" 2>&1) +assert_equals 'skip: node not found' "$suite_out" "a missing Node executable produces the canonical suite skip" +pass "check and suite entry distinguish unsupported runtimes from broken imports" for exit_case in gate fit picked unavailable; do reset From 2637ba943e8475dbb3572bc283e21916d9b5070f Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Wed, 7 Oct 2026 20:32:02 -0500 Subject: [PATCH 04/12] no-mistakes(review): Correct skill picker credential transport documentation --- docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index de4e0390092..ec635f22601 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1606,7 +1606,7 @@ Firstmate passes its profile line unless it states a reason to override, such as **Key handling and fixed settings** - The resolver, skill picker, and bootstrap copy an environment-provided key into a non-exported private variable and unset `TYPESAFE_API_KEY` before launching child processes, so the secret is absent from child environments. -- The resolver and skill picker send the key to `curl` only as a header read from a file descriptor, never on argv, and neither prints, logs, or writes it. +- The resolver sends its key to `curl` only as a header read from a file descriptor, never on argv. The skill picker pipes its keys on stdin to Node, which supplies them to the vendored JevClient's fetch transport; they never enter Node's environment or arguments. Neither tool prints, logs, or writes keys to files. - The resolver fixes the endpoint at `https://api.typesafe.ai`, model at `jev-latest`, default confidence floor at 0.6, and request timeout at 5 seconds; `TYPESAFE_API_KEY` is its only resolver-specific environment setting. The live rule-match evidence is recorded in [`verification/dispatch-resolve.md`](verification/dispatch-resolve.md). From 9e14fcffab531d2d7d2ed3a57f45476f33dc0726 Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Wed, 7 Oct 2026 21:25:09 -0500 Subject: [PATCH 05/12] no-mistakes(review): Fix worker brief delivery and linked skill exclusions --- .../references/common/dispatch.md | 2 +- bin/fm-skill-pick.mjs | 15 +- bin/fm-spawn.sh | 4 +- docs/configuration.md | 4 +- tests/fm-skill-pick.test.sh | 174 ++++++++++++++++++ 5 files changed, 189 insertions(+), 10 deletions(-) diff --git a/.agents/skills/harness-adapters/references/common/dispatch.md b/.agents/skills/harness-adapters/references/common/dispatch.md index c2abe0f9b7a..7b5ff8126e5 100644 --- a/.agents/skills/harness-adapters/references/common/dispatch.md +++ b/.agents/skills/harness-adapters/references/common/dispatch.md @@ -13,7 +13,7 @@ When the opt-in `bin/fm-dispatch-resolve.sh` is on, its `clear` answer already n ## Skill selection at launch -Ship and scout spawns with brief delivery add the project skill selection outcome to the worker's launch instructions, after every existing mandatory explicit/named and safety trigger. Raw commands without `__BRIEF__` instead record selection as undelivered and remain unchanged. +Ship and scout spawns with brief delivery add the project skill selection outcome to the worker's launch instructions, after every existing mandatory explicit/named and safety trigger. Raw commands support delivery through `__BRIEF__`, `__BRIEFDOORBELL__`, or a detected Kimi/Rovo harness's brief-file pointer; only raw commands without any supported brief transport record selection as undelivered and remain unchanged. [Worker skill selection](../../../docs/configuration.md#worker-skill-selection) owns its input, privacy, provider order, and recorded result. diff --git a/bin/fm-skill-pick.mjs b/bin/fm-skill-pick.mjs index 27b269e3c74..b77ecbf238d 100644 --- a/bin/fm-skill-pick.mjs +++ b/bin/fm-skill-pick.mjs @@ -128,13 +128,18 @@ function roster(catalogs) { let skill = null; let reason = null; try { - if (!lstatSync(join(dir, child)).isDirectory()) continue; - if (!lstatSync(path, { throwIfNoEntry: false })) continue; - if (!lstatSync(path).isFile()) { + const entry = lstatSync(join(dir, child)); + if (entry.isSymbolicLink()) { reason = 'not a Git-tracked file in this project'; } else { - const text = readHead(path); - try { skill = frontmatter(text); } catch { reason = 'no readable description'; } + if (!entry.isDirectory()) continue; + if (!lstatSync(path, { throwIfNoEntry: false })) continue; + if (!lstatSync(path).isFile()) { + reason = 'not a Git-tracked file in this project'; + } else { + const text = readHead(path); + try { skill = frontmatter(text); } catch { reason = 'no readable description'; } + } } } catch (error) { throw new Error(`could not read skill catalog entry ${path}: ${error.message}`); diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 22333687794..597512d7317 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -4614,9 +4614,9 @@ spawn_assert_agent_worktree # record say plainly why. SKILL_SELECTION_BOUND=30 SKILL_SELECTION_STATUS='' SKILL_SELECTION_REASON='' SKILL_SELECTION_PICKED='' -if { [ "$KIND" = ship ] || [ "$KIND" = scout ]; } && [ "$RAW_LAUNCH" = 1 ] && [[ "$LAUNCH" != *'__BRIEF__'* ]]; then +if { [ "$KIND" = ship ] || [ "$KIND" = scout ]; } && [ "$RAW_LAUNCH" = 1 ] && [[ "$LAUNCH" != *'__BRIEF__'* ]] && [[ "$LAUNCH" != *'__BRIEFDOORBELL__'* ]] && [ "$HARNESS" != kimi ] && [ "$HARNESS" != rovo ]; then SKILL_SELECTION_STATUS=undelivered - SKILL_SELECTION_REASON="raw launch command has no __BRIEF__ placeholder" + SKILL_SELECTION_REASON="raw launch command has no supported brief transport" echo "skill selection for $ID: $SKILL_SELECTION_STATUS - $SKILL_SELECTION_REASON" >&2 elif [ "$KIND" = ship ] || [ "$KIND" = scout ]; then skill_section="$DATA/$ID/.skill-selection.md.${BASHPID:-$$}" diff --git a/docs/configuration.md b/docs/configuration.md index ec635f22601..d261fa62a70 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1713,10 +1713,10 @@ For rosters over 255 skills, chunk-local top-three candidates enter a common Cho The pick is added to the launch instructions as a skill to read in full and follow. Existing mandatory skill triggers and the worker's own skill index still apply first and unchanged. -A skill the picker could not send, such as one that is not a Git-tracked file in the task copy, is listed by name for the worker to check. +A skill the picker could not send, such as one that is not a Git-tracked file in the task copy or a linked skill directory, is listed by its local name for the worker to check; linked directories are not followed or sent. No-fit outcomes apply only to judged skills; when every discovered skill is excluded, selection is unavailable with the exclusion reasons, not a finding that no skill fits. A genuinely empty catalog reports that there are no skills to judge. When the picker cannot run, the launch continues and the instructions say why. The task record carries `skill_selection=` (picked, none, unavailable, or undelivered), `skill_selection_reason=`, and `skill_selection_picked=` when a pick is delivered. -Raw commands receive the selection only when they contain `__BRIEF__`; without that placeholder, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. Failure to publish the selection overlay also records undelivered and clears the pick. +Raw commands receive the selection when they contain `__BRIEF__` or `__BRIEFDOORBELL__`, or when their detected harness is Kimi or Rovo, which receives a brief-file pointer. Without any supported brief transport, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. Failure to publish the selection overlay also records undelivered and clears the pick. `bin/fm-spawn.sh` stops the picker after 30 seconds. Only the brief text dispatch resolution may send is sent, after the same never-send checks; see the [never-send list](#typed-dispatch-resolution-env-typesafe_api_key). For a promoted scout's ship relaunch, skill selection uses the recorded ship kind and current ship Firstmate spec rather than the superseded scout spec; ordinary dispatch extraction is unchanged. Privacy-hidden promotion instructions never revive the superseded spec. diff --git a/tests/fm-skill-pick.test.sh b/tests/fm-skill-pick.test.sh index e3f996e7bba..0b6443d15ce 100644 --- a/tests/fm-skill-pick.test.sh +++ b/tests/fm-skill-pick.test.sh @@ -212,6 +212,52 @@ for roster_case in absent empty untracked undescribed partial; do assert_equals "$count" "$(requests)" "the $roster_case $threshold request count" done done + +linked_target="$TMP_ROOT/linked-target" +skill "$linked_target" TARGET-FRONTMATTER-SENTINEL 'description: LINKED-DESCRIPTION-SENTINEL' +printf 'LINKED-BODY-SENTINEL\n' >> "$linked_target/TARGET-FRONTMATTER-SENTINEL/SKILL.md" +for linked_case in entirely mixed precedence; do + reset; keys + roster="$TMP_ROOT/linked-$linked_case" + git init -q "$roster" + mkdir -p "$roster/.agents/skills" "$roster/.claude/skills" + ln -s "$linked_target/TARGET-FRONTMATTER-SENTINEL" "$roster/.agents/skills/agents-link" + ln -s "$linked_target/TARGET-FRONTMATTER-SENTINEL" "$roster/.claude/skills/claude-link" + if [ "$linked_case" != entirely ]; then + skill "$roster/.agents/skills" alpha 'description: Use for alpha work.' + fi + if [ "$linked_case" = precedence ]; then + skill "$roster/.claude/skills" agents-link 'description: LATER-DUPLICATE-SENTINEL' + fi + git -C "$roster" add -- .agents .claude + roster_json=$(node "$ROOT/bin/fm-skill-pick.mjs" roster "$roster/.agents/skills" "$roster/.claude/skills") + printf '%s\n' "$roster_json" | jq -e ' + (.not_judged | map(.name) | sort) == ["agents-link","claude-link"] + and all(.not_judged[]; .reason == "not a Git-tracked file in this project") + and all(.skills[]; .name == "alpha")' >/dev/null \ + || fail "linked directories keep their local names and exclusion reasons in the public roster" + assert_not_contains "$roster_json" SENTINEL "the roster never reads linked target frontmatter or body" + out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" FAKE_PICK=alpha \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$roster/.agents/skills" \ + --catalog "$roster/.claude/skills" --record "$TMP_ROOT/record") + for name in agents-link claude-link; do + assert_contains "$out" "$name (not a Git-tracked file in this project)" "linked skills remain visible for local checking" + done + if [ "$linked_case" = entirely ]; then + grep -qx 'status=unavailable' "$TMP_ROOT/record" || fail "entirely linked catalogs are unavailable" + assert_contains "$out" "no project skills could be judged:" "entirely linked catalogs report their exclusions" + assert_equals 0 "$(requests)" "entirely linked catalogs make no requests" + else + grep -qx 'status=picked' "$TMP_ROOT/record" || fail "normal tracked skills can still be selected" + grep -qx 'picked=alpha' "$TMP_ROOT/record" || fail "only the normal skill is picked" + assert_contains "$out" "- Picked for this task: alpha" "mixed catalogs deliver the normal skill" + assert_equals 2 "$(requests)" "mixed catalogs judge the normal tracked skill" + jq -se 'all(.[]; (.body.questions.which.criteria | keys) == ["alpha"])' "$LOG" >/dev/null \ + || fail "links and later duplicates never enter ranking or reranking" + fi + assert_not_contains "$(cat "$LOG")" SENTINEL "no linked target or shadowed duplicate content reaches transport" +done +pass "linked skill directories are visible but never read or judged, with earlier catalog precedence" pass "empty, excluded and partially judged rosters have distinct outcomes" reset; keys @@ -563,6 +609,134 @@ for raw in plain brief; do done pass "raw launch records match delivery capability" +raw_pointer_fake() { + mv "$SPAWN_FAKEBIN/tmux" "$SPAWN_FAKEBIN/tmux-base" + fm_fake_exit0 "$SPAWN_FAKEBIN" rovo + cat > "$SPAWN_FAKEBIN/tmux" <<'SH' +#!/usr/bin/env bash +set -eu +state=$(cat "$POINTER_STATE" 2>/dev/null || true) +screen() { + if [ "$state" = delivered ]; then + printf 'Read the brief at %s and follow it exactly.\nContext: | 3.3%% 30.1K/922K\n' "$POINTER_BRIEF" + else + printf 'Welcome to Rovo!\nContext: | 0.0%% 0/922K\n? for shortcuts.\n' + fi + printf '╭────────────────────────────────╮\n' + if [ "$state" = typed ]; then printf '│ > Read the brief and follow it │\n' + else printf '│ > │\n'; fi + printf '╰────────────────────────────────╯\n' +} +case "$*" in + *"#{cursor_y}"*) printf '4\n'; exit 0 ;; +esac +case "${1:-}" in + send-keys) + prev= + for arg in "$@"; do + if [ "$prev" = -l ] && [[ "$arg" = 'Read the brief at '* ]]; then + printf '%s\n' "$arg" >> "$POINTER_LOG" + printf 'typed\n' > "$POINTER_STATE" + exit 0 + fi + prev=$arg + done + if [ "$state" = typed ]; then + case " $* " in *' Enter '*) printf 'delivered\n' > "$POINTER_STATE" ;; esac + fi + ;; + capture-pane) + start= end= prev= + for arg in "$@"; do + case "$prev" in -S) start=$arg ;; -E) end=$arg ;; esac + case "$arg" in -S|-E) prev=$arg ;; *) prev= ;; esac + done + case "$start:$end" in + *[!0-9:]*|'':*|*:'') screen ;; + *) screen | awk -v start="$start" -v end="$end" 'NR - 1 >= start && NR - 1 <= end' ;; + esac + exit 0 + ;; +esac +exec "$(dirname "$0")/tmux-base" "$@" +SH + chmod +x "$SPAWN_FAKEBIN/tmux" +} + +for transport in doorbell pointer; do + for outcome in picked none unavailable; do + reset + id="raw-$transport-$outcome" + spawn_case "$id" + printf 'TYPESAFE_API_KEY=ts-spawn-key\n' > "$SPAWN_HOME/.env" + gate=0.8 + case "$outcome" in + none) gate=0.1 ;; + unavailable) rm "$SPAWN_HOME/.env" ;; + esac + brief="$SPAWN_HOME/data/$id/launch-brief.md" + pointer_log="$TMP_ROOT/$id.pointer" + case "$transport" in + doorbell) harness='custom-agent --brief __BRIEFDOORBELL__' ;; + pointer) harness='rovo run --yolo'; raw_pointer_fake ;; + esac + launch_log="$TMP_ROOT/$id.launch" + out=$(FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FAKE_PICK=alpha FAKE_GATE="$gate" \ + FM_FAKE_LAUNCH_LOG="$launch_log" POINTER_LOG="$pointer_log" POINTER_STATE="$TMP_ROOT/$id.pointer-state" \ + POINTER_BRIEF="$brief" FM_ROVO_READY_POLLS=3 FM_ROVO_DELIVERY_POLLS=3 FM_ROVO_POLL_INTERVAL=0 \ + FM_ROVO_SUBMIT_SLEEP=0 \ + fm_test_run_spawn "$SPAWN_HOME" "$SPAWN_POOL" "$SPAWN_FAKEBIN" "$id" "$SPAWN_PROJECT" \ + --harness "$harness" --mode no-mistakes --yolo off) + assert_contains "$out" "spawned $id" "the raw $transport $outcome launch succeeds" + meta="$SPAWN_HOME/state/$id.meta" + grep -qx "skill_selection=$outcome" "$meta" || fail "raw $transport metadata records $outcome" + payload=$(cat "$brief") + case "$transport" in + doorbell) + probe="$TMP_ROOT/$id.argv" + mkdir -p "$probe" + cat > "$probe/custom-agent" <<'SH' +#!/bin/sh +[ "$1" = --brief ] || exit 1 +printf '%s' "$2" +SH + chmod +x "$probe/custom-agent" + doorbell=$(fm_eval_launch "$(cat "$launch_log")" "$SPAWN_POOL" "$probe") + record=$(printf '%s' "$doorbell" | sed -n "s/.*Firstmate operational input waiting: read '\\([^']*\\)'.*/\\1/p") + [ -f "$record" ] || fail "raw doorbell command names a published inbox record" + payload=$(FM_STATE_OVERRIDE="$SPAWN_HOME/state" "$ROOT/bin/fm-operational-input.sh" open "$record") + assert_equals "$(cat "$brief")" "$payload" "raw doorbell inbox holds the generated brief" + ;; + pointer) + assert_contains "$(cat "$launch_log")" 'rovo run --yolo' "the raw pointer launch command is retained" + assert_equals "Read the brief at $brief and follow it exactly." "$(cat "$pointer_log")" "the pointer names the generated brief" + ;; + esac + case "$outcome" in + picked) + assert_contains "$payload" "- Picked for this task: alpha - read $SPAWN_POOL/.agents/skills/alpha/SKILL.md" "raw $transport delivers the project pick" + grep -qx 'skill_selection_picked=alpha' "$meta" || fail "raw $transport metadata carries the delivered pick" + assert_equals 2 "$(requests)" "raw $transport picked outcome uses normal selection" + ;; + none) + assert_contains "$payload" "Skill selection found no fit among the judged project skills" "raw $transport delivers the no-fit outcome" + assert_equals 1 "$(requests)" "raw $transport no-fit outcome stops after gating" + ;; + unavailable) + assert_contains "$payload" "Skill selection was unavailable for this task (no TypeSafe or OpenRouter key)" "raw $transport delivers the unavailable reason" + assert_equals 0 "$(requests)" "raw $transport without keys sends nothing" + ;; + esac + if [ "$outcome" != picked ]; then + reason=$(sed -n 's/^skill_selection_reason=//p' "$meta") + [ -n "$reason" ] || fail "raw $transport metadata has an outcome reason" + assert_contains "$payload" "$reason" "raw $transport brief and metadata agree on the reason" + assert_not_contains "$(cat "$meta")" 'skill_selection_picked=' "raw $transport without a pick claims none" + fi + done +done +pass "raw doorbell and pointer transports deliver picked, none and unavailable outcomes" + reset spawn_case failed-overlay printf 'TYPESAFE_API_KEY=ts-spawn-key\n' > "$SPAWN_HOME/.env" From 6574b5bdac1afd5eeacf7fb4501326af4e520613 Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Wed, 7 Oct 2026 23:07:35 -0500 Subject: [PATCH 06/12] no-mistakes(review): Fix linked-skill accounting and failed-delivery selection metadata --- bin/fm-skill-pick.mjs | 4 +- bin/fm-spawn.sh | 26 ++++- docs/configuration.md | 4 +- tests/fm-skill-pick.test.sh | 198 +++++++++++++++++++++++++++++++++++- 4 files changed, 226 insertions(+), 6 deletions(-) diff --git a/bin/fm-skill-pick.mjs b/bin/fm-skill-pick.mjs index b77ecbf238d..8db6ac7744d 100644 --- a/bin/fm-skill-pick.mjs +++ b/bin/fm-skill-pick.mjs @@ -16,7 +16,7 @@ // pick prints status=, reason=, picked=, path=, fit=, provider= and model= // lines; path is a JSON string, and status is picked, none or unavailable. import { execFileSync } from 'node:child_process'; -import { closeSync, lstatSync, openSync, readFileSync, readSync, readdirSync, realpathSync } from 'node:fs'; +import { closeSync, lstatSync, openSync, readFileSync, readSync, readdirSync, realpathSync, statSync } from 'node:fs'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; let JevClient, choice, noul, LIMITS; @@ -130,6 +130,8 @@ function roster(catalogs) { try { const entry = lstatSync(join(dir, child)); if (entry.isSymbolicLink()) { + if (!statSync(join(dir, child), { throwIfNoEntry: false })?.isDirectory()) continue; + if (!lstatSync(path, { throwIfNoEntry: false })) continue; reason = 'not a Git-tracked file in this project'; } else { if (!entry.isDirectory()) continue; diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 597512d7317..34a5e9816d8 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -1365,6 +1365,8 @@ CONFIG_INHERIT_LOCK= CONFIG_INHERIT_LOCK_HELD=0 GIT_HOOKS_DIR= SPAWN_LAUNCH_SENT=0 +SPAWN_BRIEF_DELIVERED=0 +SPAWN_BRIEF_FAILURE_REASON= SPAWN_ENDPOINT_CLOSED=0 SPAWN_TREEHOUSE_ABORT_TARGET= SPAWN_ROUTING_PAIR= @@ -1397,7 +1399,7 @@ parse_orca_worktree_result() { } spawn_abort_cleanup() { - local status=$? + local status=$? skill_meta_tmp if [ "$RELAUNCH_REPLACEMENT_PENDING" = 1 ] && [ "$SPAWN_META_PUBLISH_STARTED" = 1 ] && [ -n "$SPAWN_META_TMP" ] && @@ -1516,6 +1518,24 @@ spawn_abort_cleanup() { status=1 fi fi + if [ "$status" -ne 0 ] && [ "$SPAWN_BRIEF_DELIVERED" = 0 ] && + [ "$SPAWN_META_LOCK_HELD" = 1 ] && [ -n "${SKILL_SELECTION_STATUS:-}" ] && + [ -n "${SPAWN_GEN:-}" ] && + grep -Fqx "spawn_gen=$SPAWN_GEN" "$STATE/$ID.meta" 2>/dev/null; then + skill_meta_tmp=$(mktemp "$STATE/.$ID.meta.skill-selection.XXXXXX") || skill_meta_tmp= + if [ -n "$skill_meta_tmp" ] && + awk -F= '$1 != "skill_selection" && $1 != "skill_selection_reason" && $1 != "skill_selection_picked"' \ + "$STATE/$ID.meta" >"$skill_meta_tmp" && + printf 'skill_selection=undelivered\nskill_selection_reason=%s\n' \ + "${SPAWN_BRIEF_FAILURE_REASON:-launch aborted before brief delivery (exit status $status)}" >>"$skill_meta_tmp" && + fm_backlog_atomic_transition publish "$skill_meta_tmp" "$STATE/$ID.meta" "task record" "$STATE"; then + : + else + echo "error: could not record undelivered skill selection for $ID" >&2 + status=1 + fi + [ -z "$skill_meta_tmp" ] || rm -f "$skill_meta_tmp" + fi if [ "$SPAWN_META_LOCK_HELD" = 1 ]; then SPAWN_META_LOCK_HELD=0 fm_lock_release "$SPAWN_META_LOCK" || true @@ -4333,6 +4353,7 @@ kimi_wait_for_delivery() { } kimi_spawn_fail() { # + SPAWN_BRIEF_FAILURE_REASON=$1 printf '%s\n' "$(status_stamp_line "failed: $1")" >>"$STATE/$ID.status" echo "error: $1; inspect window $T" >&2 } @@ -4401,6 +4422,7 @@ rovo_wait_for_delivery() { } rovo_spawn_fail() { # + SPAWN_BRIEF_FAILURE_REASON=$1 printf '%s\n' "$(status_stamp_line "failed: $1")" >>"$STATE/$ID.status" echo "error: $1; inspect window $T" >&2 rovo_endpoint_cleanup @@ -5466,6 +5488,7 @@ case "$LAUNCH" in *) brief_opstate=$STATE ;; esac brief_doorbell=$(FM_STATE_OVERRIDE="$brief_opstate" "$FM_ROOT/bin/fm-operational-input.sh" record launch-brief <"$BRIEF") || { + SPAWN_BRIEF_FAILURE_REASON="could not publish the launch brief as an operational-inbox record" echo "error: could not publish the launch brief for $ID as an operational-inbox record under $brief_opstate; $HARNESS strips the typed operational marker, so the worker was not launched" >&2 exit 1 } @@ -5812,6 +5835,7 @@ if [ "$HARNESS" = agy ]; then exit 1 fi fi +SPAWN_BRIEF_DELIVERED=1 if [ "$KIND" = secondmate ] && [ "${FM_SKIP_SECONDMATE_INHERIT:-0}" != 1 ]; then if ! fm_config_reread_discard_pending "$PROJ_ABS" "$ID" "$FM_HOME"; then diff --git a/docs/configuration.md b/docs/configuration.md index d261fa62a70..7bd2c35660a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1713,10 +1713,10 @@ For rosters over 255 skills, chunk-local top-three candidates enter a common Cho The pick is added to the launch instructions as a skill to read in full and follow. Existing mandatory skill triggers and the worker's own skill index still apply first and unchanged. -A skill the picker could not send, such as one that is not a Git-tracked file in the task copy or a linked skill directory, is listed by its local name for the worker to check; linked directories are not followed or sent. +A skill the picker could not send, such as one that is not a Git-tracked file in the task copy or a linked skill directory containing `SKILL.md`, is listed by its local name for the worker to check; linked skill contents are never read or sent. Links without `SKILL.md` are not skills: they reserve no name and produce no manual-check instruction. No-fit outcomes apply only to judged skills; when every discovered skill is excluded, selection is unavailable with the exclusion reasons, not a finding that no skill fits. A genuinely empty catalog reports that there are no skills to judge. When the picker cannot run, the launch continues and the instructions say why. The task record carries `skill_selection=` (picked, none, unavailable, or undelivered), `skill_selection_reason=`, and `skill_selection_picked=` when a pick is delivered. -Raw commands receive the selection when they contain `__BRIEF__` or `__BRIEFDOORBELL__`, or when their detected harness is Kimi or Rovo, which receives a brief-file pointer. Without any supported brief transport, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. Failure to publish the selection overlay also records undelivered and clears the pick. +Raw commands receive the selection when they contain `__BRIEF__` or `__BRIEFDOORBELL__`, or when their detected harness is Kimi or Rovo, which receives a brief-file pointer. Without any supported brief transport, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. Failure to publish the selection overlay also records undelivered and clears the pick. If launch delivery fails, any retained record for that launch is marked undelivered with the failure reason and no claimed pick; an abort before replacement-record publication preserves the prior record. `bin/fm-spawn.sh` stops the picker after 30 seconds. Only the brief text dispatch resolution may send is sent, after the same never-send checks; see the [never-send list](#typed-dispatch-resolution-env-typesafe_api_key). For a promoted scout's ship relaunch, skill selection uses the recorded ship kind and current ship Firstmate spec rather than the superseded scout spec; ordinary dispatch extraction is unchanged. Privacy-hidden promotion instructions never revive the superseded spec. diff --git a/tests/fm-skill-pick.test.sh b/tests/fm-skill-pick.test.sh index 0b6443d15ce..ce6b076eccd 100644 --- a/tests/fm-skill-pick.test.sh +++ b/tests/fm-skill-pick.test.sh @@ -213,6 +213,52 @@ for roster_case in absent empty untracked undescribed partial; do done done +reset; keys +roster="$TMP_ROOT/non-skill-links" +non_skill_target="$TMP_ROOT/non-skill-target" +git init -q "$roster" +mkdir -p "$roster/.agents/skills" "$roster/.claude/skills" "$non_skill_target" +printf 'NON-SKILL-TARGET-SENTINEL\n' > "$non_skill_target/README" +ln -s "$non_skill_target" "$roster/.agents/skills/alpha" +ln -s "$non_skill_target" "$roster/.claude/skills/plugin-helper" +for catalog in .agents .claude; do + ln -s "$non_skill_target/README" "$roster/$catalog/skills/file-link" + ln -s "$TMP_ROOT/missing-non-skill-target" "$roster/$catalog/skills/dangling-link" +done +git -C "$roster" add -- .agents .claude +roster_json=$(node "$ROOT/bin/fm-skill-pick.mjs" roster "$roster/.agents/skills" "$roster/.claude/skills") +printf '%s\n' "$roster_json" | jq -e '.skills == [] and .not_judged == []' >/dev/null \ + || fail "non-skill links in both catalogs emit no phantom skill names or exclusions" +out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$roster/.agents/skills" \ + --catalog "$roster/.claude/skills" --record "$TMP_ROOT/record") +grep -qx 'status=none' "$TMP_ROOT/record" || fail "a plugin-only catalog records no skill selection" +grep -qx 'reason=this project has no skills to judge' "$TMP_ROOT/record" \ + || fail "non-skill links leave the catalog empty rather than unavailable" +assert_not_contains "$out" "Not judged, so check them yourself if relevant:" "non-skill links do not require manual checking" +assert_not_contains "$out" "Skill selection was unavailable" "non-skill links do not make selection unavailable" +assert_equals 0 "$(requests)" "a plugin-only catalog makes no requests" +reset; keys +skill "$roster/.claude/skills" alpha 'description: Use for alpha work.' +git -C "$roster" add -- .claude/skills/alpha/SKILL.md +roster_json=$(node "$ROOT/bin/fm-skill-pick.mjs" roster "$roster/.agents/skills" "$roster/.claude/skills") +printf '%s\n' "$roster_json" | jq -e ' + (.skills | map(.name)) == ["alpha"] and .not_judged == []' >/dev/null \ + || fail "a non-skill link does not reserve the name of a later tracked skill" +out=$(env FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FM_HOME="$HOME_DIR" FAKE_PICK=alpha \ + bash "$TOOL" --brief "$TMP_ROOT/brief.md" --catalog "$roster/.agents/skills" \ + --catalog "$roster/.claude/skills" --record "$TMP_ROOT/record") +grep -qx 'status=picked' "$TMP_ROOT/record" || fail "the later tracked skill can be picked" +grep -qx 'picked=alpha' "$TMP_ROOT/record" || fail "the non-skill link does not shadow the picked name" +assert_contains "$out" "- Picked for this task: alpha - read $(cd "$roster" && pwd -P)/.claude/skills/alpha/SKILL.md" "selection resolves to the later tracked skill" +assert_not_contains "$out" "Not judged, so check them yourself if relevant:" "non-skill links stay absent beside a tracked skill" +assert_equals 2 "$(requests)" "the later tracked skill completes normal selection" +jq -se 'all(.[]; (.body.questions.which.criteria | keys) == ["alpha"])' "$LOG" >/dev/null \ + || fail "only the actual later skill enters ranking and reranking" +assert_not_contains "$roster_json" NON-SKILL-TARGET-SENTINEL "non-skill linked target contents stay out of the roster" +assert_not_contains "$(cat "$LOG")" NON-SKILL-TARGET-SENTINEL "non-skill linked target contents stay out of transport" +pass "non-skill links leave catalogs empty and do not reserve later tracked skill names" + linked_target="$TMP_ROOT/linked-target" skill "$linked_target" TARGET-FRONTMATTER-SENTINEL 'description: LINKED-DESCRIPTION-SENTINEL' printf 'LINKED-BODY-SENTINEL\n' >> "$linked_target/TARGET-FRONTMATTER-SENTINEL/SKILL.md" @@ -611,14 +657,24 @@ pass "raw launch records match delivery capability" raw_pointer_fake() { mv "$SPAWN_FAKEBIN/tmux" "$SPAWN_FAKEBIN/tmux-base" - fm_fake_exit0 "$SPAWN_FAKEBIN" rovo + fm_fake_exit0 "$SPAWN_FAKEBIN" rovo kimi cat > "$SPAWN_FAKEBIN/tmux" <<'SH' #!/usr/bin/env bash set -eu state=$(cat "$POINTER_STATE" 2>/dev/null || true) screen() { + if [ "${POINTER_RELAUNCH:-0}" = 1 ] && [ ! -f "$FM_FAKE_LAUNCH_LOG" ]; then + printf 'test-shell$ \n' + return + fi + if [ "${POINTER_FAILURE:-}" = ready ]; then + printf 'Starting agent...\n' + return + fi if [ "$state" = delivered ]; then printf 'Read the brief at %s and follow it exactly.\nContext: | 3.3%% 30.1K/922K\n' "$POINTER_BRIEF" + elif [ "${POINTER_HARNESS:-rovo}" = kimi ]; then + printf 'Welcome to Kimi Code!\ncontext: 0.0%%\n' else printf 'Welcome to Rovo!\nContext: | 0.0%% 0/922K\n? for shortcuts.\n' fi @@ -629,12 +685,21 @@ screen() { } case "$*" in *"#{cursor_y}"*) printf '4\n'; exit 0 ;; + *"#{pane_current_command}"*) printf 'zsh\n'; exit 0 ;; + *"#{pane_tty}"*) exit 0 ;; esac case "${1:-}" in + list-windows) + if [ "${POINTER_RELAUNCH:-0}" = 1 ]; then + printf '%s\n' "$POINTER_WINDOW" + exit 0 + fi + ;; send-keys) prev= for arg in "$@"; do if [ "$prev" = -l ] && [[ "$arg" = 'Read the brief at '* ]]; then + [ "${POINTER_FAILURE:-}" != submit ] || exit 1 printf '%s\n' "$arg" >> "$POINTER_LOG" printf 'typed\n' > "$POINTER_STATE" exit 0 @@ -642,7 +707,12 @@ case "${1:-}" in prev=$arg done if [ "$state" = typed ]; then - case " $* " in *' Enter '*) printf 'delivered\n' > "$POINTER_STATE" ;; esac + case " $* " in + *' Enter '*) + if [ "${POINTER_FAILURE:-}" = confirmation ]; then printf 'submitted\n' > "$POINTER_STATE" + else printf 'delivered\n' > "$POINTER_STATE"; fi + ;; + esac fi ;; capture-pane) @@ -737,6 +807,130 @@ SH done pass "raw doorbell and pointer transports deliver picked, none and unavailable outcomes" +relaunch_selection_case() { + local id=$1 kind=$2 + spawn_case "$id" + mkdir -p "$SPAWN_HOME/user-home/.kimi-code" + printf 'default_model = "kimi-for-coding"\n' > "$SPAWN_HOME/user-home/.kimi-code/config.toml" + meta="$SPAWN_HOME/state/$id.meta" + { + printf 'window=firstmate:fm-%s\nendpoint_task_id=%s\n' "$id" "$id" + printf 'worktree=%s\nproject=%s\n' "$SPAWN_POOL" "$SPAWN_PROJECT" + printf 'harness=codex\nkind=%s\n' "$kind" + printf 'mode=no-mistakes\nyolo=off\nbranch=fm/%s\nmodel=default\neffort=default\n' "$id" + printf 'spawn_gen=prior-incarnation\nskill_selection=picked\nskill_selection_picked=prior-skill\nskill_selection_reason=prior-reason\n' + printf 'owner_note=preserve this owner metadata\n' + } > "$meta" + raw_pointer_fake +} + +relaunch_publish_failure_fake() { + cat > "$SPAWN_FAKEBIN/mv" <<'SH' +#!/usr/bin/env bash +set -eu +for arg in "$@"; do + case "${PUBLICATION_FAILURE:-}:$arg" in + metadata:*/.*.meta.relaunch.*) exit 1 ;; + esac +done +exec "$REAL_MV" "$@" +SH + chmod +x "$SPAWN_FAKEBIN/mv" +} + +real_mv=$(command -v mv) +for transport in kimi rovo doorbell; do + for failure in ready submit confirmation; do + [ "$transport" != doorbell ] || [ "$failure" = ready ] || continue + for outcome in picked none unavailable; do + reset + id="relaunch-$transport-$failure-$outcome" + case "$outcome" in none) kind=scout ;; *) kind=ship ;; esac + relaunch_selection_case "$id" "$kind" + printf 'TYPESAFE_API_KEY=ts-spawn-key\n' > "$SPAWN_HOME/.env" + gate=0.8 + case "$outcome" in + none) gate=0.1 ;; + unavailable) rm "$SPAWN_HOME/.env" ;; + esac + brief="$SPAWN_HOME/data/$id/launch-brief.md" + case "$transport" in + kimi|rovo) + harness="$transport" + recorded_harness="$transport" + case "$failure" in + ready) failure_reason="$transport did not show a verified ready signal before brief delivery" ;; + submit) failure_reason="$transport brief pointer could not be submitted" ;; + confirmation) failure_reason="$transport brief pointer delivery was not confirmed" ;; + esac + ;; + doorbell) + harness='custom-agent --brief __BRIEFDOORBELL__' + recorded_harness=custom-agent + failure_reason='could not publish the launch brief as an operational-inbox record' + : > "$SPAWN_HOME/state/operational-inbox" + ;; + esac + if out=$(FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FAKE_PICK=alpha FAKE_GATE="$gate" \ + FM_FAKE_LAUNCH_LOG="$TMP_ROOT/$id.launch" POINTER_LOG="$TMP_ROOT/$id.pointer" \ + POINTER_STATE="$TMP_ROOT/$id.pointer-state" POINTER_BRIEF="$brief" POINTER_RELAUNCH=1 \ + POINTER_HARNESS="$transport" POINTER_FAILURE="$failure" POINTER_WINDOW="fm-$id" \ + FM_KIMI_READY_POLLS=2 FM_KIMI_DELIVERY_POLLS=2 FM_KIMI_POLL_INTERVAL=0 FM_KIMI_SUBMIT_SLEEP=0 \ + FM_ROVO_READY_POLLS=2 FM_ROVO_DELIVERY_POLLS=2 FM_ROVO_POLL_INTERVAL=0 FM_ROVO_SUBMIT_SLEEP=0 \ + REAL_MV="$real_mv" \ + fm_test_run_spawn "$SPAWN_HOME" "$SPAWN_POOL" "$SPAWN_FAKEBIN" "$id" --relaunch --harness "$harness"); then + fail "$transport $failure relaunch must fail after selection" + fi + if [ "$transport" = doorbell ]; then + assert_contains "$out" "could not publish the launch brief for $id" "the relaunch reports its actual publication failure" + else + assert_contains "$out" "$failure_reason" "the relaunch reports its actual delivery failure" + fi + grep -qx "harness=$recorded_harness" "$meta" || fail "failed relaunch retains the replacement harness record" + grep -qx "kind=$kind" "$meta" || fail "failed relaunch preserves the instruction-owner kind" + grep -qx 'skill_selection=undelivered' "$meta" || fail "failed $transport $failure relaunch marks $outcome undelivered" + assert_contains "$(sed -n 's/^skill_selection_reason=//p' "$meta")" "$failure_reason" "undelivered metadata records the actual failure" + assert_not_contains "$(cat "$meta")" 'skill_selection_picked=' "failed delivery clears old and new picked fields" + assert_not_contains "$(cat "$meta")" 'prior-reason' "failed delivery clears the prior selection reason" + assert_not_contains "$(cat "$meta")" 'spawn_gen=prior-incarnation' "failed delivery retains the replacement incarnation" + grep -qx 'owner_note=preserve this owner metadata' "$meta" || fail "failed delivery preserves unrelated owner metadata" + case "$outcome" in + picked) + assert_equals 2 "$(requests)" "selection completed before the picked relaunch failed" + assert_contains "$(cat "$brief")" "- Picked for this task: alpha" "the generated brief already contains the completed alpha pick" + ;; + none) + assert_equals 1 "$(requests)" "no-fit selection completed before relaunch delivery failed" + assert_contains "$(cat "$brief")" "Skill selection found no fit among the judged project skills" "the generated brief contains the no-fit outcome" + ;; + unavailable) + assert_equals 0 "$(requests)" "unavailable selection makes no requests before relaunch failure" + assert_contains "$(cat "$brief")" "Skill selection was unavailable for this task" "the generated brief contains the unavailable outcome" + ;; + esac + done + done +done +pass "failed Kimi, Rovo and doorbell relaunch delivery records undelivered selection for ships and scouts" + +reset +id=relaunch-before-publication +relaunch_selection_case "$id" ship +printf 'TYPESAFE_API_KEY=ts-spawn-key\n' > "$SPAWN_HOME/.env" +prior_meta=$(cat "$meta") +relaunch_publish_failure_fake +if out=$(FAKE_LOG="$LOG" NODE_OPTIONS="--import=$FAKE_FETCH" FAKE_PICK=alpha \ + FM_FAKE_LAUNCH_LOG="$TMP_ROOT/$id.launch" POINTER_LOG="$TMP_ROOT/$id.pointer" \ + POINTER_STATE="$TMP_ROOT/$id.pointer-state" POINTER_BRIEF="$SPAWN_HOME/data/$id/launch-brief.md" \ + POINTER_RELAUNCH=1 POINTER_WINDOW="fm-$id" REAL_MV="$real_mv" PUBLICATION_FAILURE=metadata \ + fm_test_run_spawn "$SPAWN_HOME" "$SPAWN_POOL" "$SPAWN_FAKEBIN" "$id" --relaunch --harness rovo); then + fail "relaunch must fail when replacement metadata cannot be published" +fi +assert_equals "$prior_meta" "$(cat "$meta")" "pre-publication abort preserves the entire previous selection and owner record" +assert_equals 2 "$(requests)" "the pre-publication abort occurs after normal selection" +[ ! -f "$TMP_ROOT/$id.launch" ] || fail "replacement launch is not sent before metadata publication" +pass "pre-publication relaunch abort leaves prior selection metadata unchanged" + reset spawn_case failed-overlay printf 'TYPESAFE_API_KEY=ts-spawn-key\n' > "$SPAWN_HOME/.env" From 2a2985bc32f9917bda5fac1dd793962cd7794fb7 Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Wed, 7 Oct 2026 23:30:33 -0500 Subject: [PATCH 07/12] no-mistakes(review): Preserve private extraction and legacy promoted captain intent --- bin/fm-brief-heading-lib.sh | 37 ++++++++++++++++++++++++++++ bin/fm-dod-lib.sh | 38 ----------------------------- bin/fm-spawn.sh | 2 +- bin/fm-typesafe-lib.sh | 16 ++++++------- docs/configuration.md | 3 ++- docs/scripts.md | 2 +- tests/fm-skill-pick.test.sh | 48 +++++++++++++++++++++++++++++++++++++ 7 files changed, 96 insertions(+), 50 deletions(-) diff --git a/bin/fm-brief-heading-lib.sh b/bin/fm-brief-heading-lib.sh index e635a017749..9414563b235 100644 --- a/bin/fm-brief-heading-lib.sh +++ b/bin/fm-brief-heading-lib.sh @@ -92,3 +92,40 @@ fm_brief_task_heading_present() { # printf '%s\n' "$task" | fm_brief_heading_parse - "$2" present >/dev/null } +# Print the words of every provenance-marked line in a legacy `# Task` body. +# The marker is read the way bin/fm-brief-heading-lib.sh reads a heading: a +# line inside a ``` or ~~~ fenced block, or indented four spaces or a tab as an +# indented example, is never a marked line, so a fenced `Captain:` sample cannot +# pass the provenance gate as the ship contract's intent (issue 3608). +fm_brief_marked_captain_words() { # + printf '%s\n' "$1" | awk ' + { + scan = $0 + spaces = 0 + while (spaces < 3 && substr(scan, 1, 1) == " ") { + scan = substr(scan, 2) + spaces++ + } + marker = substr(scan, 1, 1) + marker_len = 0 + if (marker == "`" || marker == "~") { + while (substr(scan, marker_len + 1, 1) == marker) marker_len++ + } + if (marker_len >= 3) { + if (!fenced) { + fenced = 1 + fence_marker = marker + fence_len = marker_len + } else if (marker == fence_marker && marker_len >= fence_len && substr(scan, marker_len + 1) ~ /^[[:space:]]*$/) { + fenced = 0 + } + next + } + if (fenced || substr(scan, 1, 1) ~ /^[ \t]$/) next + if (match(scan, /^(\[captain\]|Captain('\''s (words|ask|intent))?:)[[:space:]]*/)) { + words = substr(scan, RLENGTH + 1) + if (words ~ /[^[:space:]]/) print words + } + } + ' +} diff --git a/bin/fm-dod-lib.sh b/bin/fm-dod-lib.sh index a454402bce7..cd52375b3b9 100755 --- a/bin/fm-dod-lib.sh +++ b/bin/fm-dod-lib.sh @@ -196,44 +196,6 @@ fm_brief_task_placeholders_present() { # return 1 } -# Print the words of every provenance-marked line in a legacy `# Task` body. -# The marker is read the way bin/fm-brief-heading-lib.sh reads a heading: a -# line inside a ``` or ~~~ fenced block, or indented four spaces or a tab as an -# indented example, is never a marked line, so a fenced `Captain:` sample cannot -# pass the provenance gate as the ship contract's intent (issue 3608). -fm_brief_marked_captain_words() { # - printf '%s\n' "$1" | awk ' - { - scan = $0 - spaces = 0 - while (spaces < 3 && substr(scan, 1, 1) == " ") { - scan = substr(scan, 2) - spaces++ - } - marker = substr(scan, 1, 1) - marker_len = 0 - if (marker == "`" || marker == "~") { - while (substr(scan, marker_len + 1, 1) == marker) marker_len++ - } - if (marker_len >= 3) { - if (!fenced) { - fenced = 1 - fence_marker = marker - fence_len = marker_len - } else if (marker == fence_marker && marker_len >= fence_len && substr(scan, marker_len + 1) ~ /^[[:space:]]*$/) { - fenced = 0 - } - next - } - if (fenced || substr(scan, 1, 1) ~ /^[ \t]$/) next - if (match(scan, /^(\[captain\]|Captain('\''s (words|ask|intent))?:)[[:space:]]*/)) { - words = substr(scan, RLENGTH + 1) - if (words ~ /[^[:space:]]/) print words - } - } - ' -} - fm_brief_intent_overlay() { # cat <<'EOF' diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 34a5e9816d8..c3d1826c0bb 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -26,7 +26,7 @@ # it also carries the current `--intent` contract and the extracted captain # intent. A legacy mixed Task is accepted there only under bin/fm-dod-lib.sh's # provenance-marking rules; unmarked legacy Tasks stop for migration rather -# than becoming intent. That library owns the parsing and intent rules. When +# than becoming intent. That library owns the intent rules. When # the explicit mode carries less rigor than the project's standing posture, a # loud one-line deviation notice is printed and the spawn continues. # no-mistakes-prod-only is a registry policy rather than a task mode and is diff --git a/bin/fm-typesafe-lib.sh b/bin/fm-typesafe-lib.sh index 0a695435bfb..f3015bed548 100644 --- a/bin/fm-typesafe-lib.sh +++ b/bin/fm-typesafe-lib.sh @@ -218,20 +218,18 @@ fm_typesafe_brief_task() { if [ "$promoted" -eq 1 ] && [ "$heading" = "## Firstmate spec" ]; then fm_brief_heading_present "$output" "# Current ship Firstmate spec" || continue printf '%s\n%s\n\n' '# Current ship Firstmate spec' "$(fm_brief_heading_body "$output" "# Current ship Firstmate spec")" - else - fm_brief_task_heading_present "$output" "$heading" || continue + elif fm_brief_task_heading_present "$output" "$heading"; then printf '%s\n%s\n\n' "$heading" "$(fm_brief_task_heading_body "$output" "$heading")" + elif [ "$promoted" -eq 1 ] && [ "$heading" = "## Captain's intent" ]; then + fm_brief_marked_captain_words "$(fm_brief_heading_body "$output" "# Task")" fi done ) [ -n "$sections" ] || [ "$promoted" -eq 1 ] || return 0 - if ! { - if [ "$kind" = scout ] || { [ -z "$kind" ] && grep -qxF 'This is a SCOUT task: the deliverable is a written report, not a PR.' "$output"; }; then - printf 'Brief kind: scout (report only)\n\n' - fi - printf '%s\n' "$sections" - } > "$output.sections" || ! mv "$output.sections" "$output"; then - rm -f "$output.sections" + if [ "$kind" = scout ] || { [ -z "$kind" ] && grep -qxF 'This is a SCOUT task: the deliverable is a written report, not a PR.' "$output"; }; then + sections=$'Brief kind: scout (report only)\n\n'"$sections" + fi + if ! printf '%s\n' "$sections" > "$output"; then FM_TYPESAFE_WITHHELD_REASON="could not read the brief" return 1 fi diff --git a/docs/configuration.md b/docs/configuration.md index 7bd2c35660a..8f529ec9d9a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1467,6 +1467,7 @@ Firstmate invokes the resolve path directly after writing the brief, without a p When on and at least one rule exists, the tool sends the project name and the brief's task-specific text as state and asks one Choice question whose options are every rule's `when` plus the fixed neutral option for no matching rule; the model never sees quota, catalogs, `why`, `use`, approvals, or confidence floors. The task-specific text is the brief's `## Captain's intent` and `## Firstmate spec` sections under `# Task` that `bin/fm-brief.sh` scaffolds, read by the same parser that feeds `fm-spawn.sh` validation and the no-mistakes `--intent` contract; a brief with neither section is sent whole. Section extraction and the whole-brief fallback both read the brief after any enabled never-send marked regions are removed (see the never-send list below). +Extraction rewrites the caller's task-text file in place without relaxing its existing permissions. When the sections are sent from a scout brief, the line `Brief kind: scout (report only)` comes first, taken from the scaffold's scout contract line; ship briefs and briefs sent whole get no kind line. A ship brief's delivery mode is deliberately not sent, because in live runs naming it pushed a routine ship brief toward the hardest tier (see [the verification record](verification/dispatch-resolve.md)). @@ -1719,7 +1720,7 @@ The task record carries `skill_selection=` (picked, none, unavailable, or undeli Raw commands receive the selection when they contain `__BRIEF__` or `__BRIEFDOORBELL__`, or when their detected harness is Kimi or Rovo, which receives a brief-file pointer. Without any supported brief transport, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. Failure to publish the selection overlay also records undelivered and clears the pick. If launch delivery fails, any retained record for that launch is marked undelivered with the failure reason and no claimed pick; an abort before replacement-record publication preserves the prior record. `bin/fm-spawn.sh` stops the picker after 30 seconds. -Only the brief text dispatch resolution may send is sent, after the same never-send checks; see the [never-send list](#typed-dispatch-resolution-env-typesafe_api_key). For a promoted scout's ship relaunch, skill selection uses the recorded ship kind and current ship Firstmate spec rather than the superseded scout spec; ordinary dispatch extraction is unchanged. Privacy-hidden promotion instructions never revive the superseded spec. +Only the brief text dispatch resolution may send is sent, after the same never-send checks; see the [never-send list](#typed-dispatch-resolution-env-typesafe_api_key). For a promoted scout's ship relaunch, skill selection retains the original Captain's intent or provenance-marked legacy Task words and uses the recorded ship kind and current ship Firstmate spec rather than the superseded scout spec; ordinary dispatch extraction is unchanged. Legacy provenance uses the same fence and indentation exclusions as promotion. Privacy-hidden promotion instructions never revive the superseded spec. The `TYPESAFE_API_KEY` opt-in enables it, and TypeSafe is asked directly first. When the direct call is unavailable because its key is absent or fails and `OPENROUTER_API_KEY` is set in the same home `.env`, the same request and the rest of that pick go through OpenRouter. This is the captain's accepted policy: “openrouter is a fallback from directly using the typesafe api.” Fallback provenance is retained for picked, no-selection, and unavailable outcomes. diff --git a/docs/scripts.md b/docs/scripts.md index d27efa5b82c..fa0976c2ec0 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -36,7 +36,7 @@ The shared no-mistakes gate lifecycle boundary is summarized in [architecture.md | `fm-decision-hold.sh` | One-release compatibility shim mapping the retired decision commands onto fm-captain-hold.sh | | `fm-brief.sh` | Scaffold ship (explicit `--mode`, plus the project's registered `--forge`), scout, secondmate-charter, and Herdr-lab briefs, with Captain's intent and Firstmate spec subsections on ship/scout | | [`fm-dod-lib.sh`](../bin/fm-dod-lib.sh) | Own ship/scout worker role scope, ship definitions of done, the named-head reachability gate on ship `done:` acceptance, and the no-mistakes `--intent` contract | -| [`fm-brief-heading-lib.sh`](../bin/fm-brief-heading-lib.sh) | Shared brief-section reader | +| [`fm-brief-heading-lib.sh`](../bin/fm-brief-heading-lib.sh) | Shared brief-section and legacy captain-provenance reader | | [`fm-skill-pick.sh`](../bin/fm-skill-pick.sh) | Pick the project skill a ship or scout worker loads at launch ([operator contract](configuration.md#worker-skill-selection)) | | [`fm-typesafe-lib.sh`](../bin/fm-typesafe-lib.sh) | Shared TypeSafe credential, transport, and never-send boundary ([operator contract](configuration.md#typed-dispatch-resolution-env-typesafe_api_key)) | | [`fm-jev-belay-hook.sh`](../bin/fm-jev-belay-hook.sh) | Claude worker wrapper for the pinned upstream belay Stop hook ([contract](configuration.md#jev-belay-stop-hook)) | diff --git a/tests/fm-skill-pick.test.sh b/tests/fm-skill-pick.test.sh index ce6b076eccd..eebf4015268 100644 --- a/tests/fm-skill-pick.test.sh +++ b/tests/fm-skill-pick.test.sh @@ -603,6 +603,14 @@ assert_contains "$sent" PERMITTED-CURRENT-SHIP-SPEC "promoted ships send the cur assert_not_contains "$sent" STALE-SCOUT-SPEC "promoted ships omit the stale scout spec" assert_not_contains "$sent" "Brief kind: scout" "promoted ships omit the scout tag" . "$ROOT/bin/fm-typesafe-lib.sh" +( + umask 022 + private_task=$(mktemp "$TMP_ROOT/private-task.XXXXXX") + fm_typesafe_brief_task "$TMP_ROOT/brief.md" "$HOME_DIR/config/dispatch-never-send" "$private_task" + assert_contains "$(cat "$private_task")" PERMITTED-ORIGINAL-INTENT "private output contains the extracted intent" + assert_equals 600 "$(node -e 'console.log((require("node:fs").statSync(process.argv[1]).mode & 0o777).toString(8))' "$private_task")" "extraction preserves owner-only output under umask 022" +) +pass "shared task extraction preserves private caller permissions" fm_typesafe_brief_task "$TMP_ROOT/brief.md" "$HOME_DIR/config/dispatch-never-send" "$TMP_ROOT/default-task" assert_contains "$(cat "$TMP_ROOT/default-task")" STALE-SCOUT-SPEC "dispatch's three-argument extraction remains unchanged" assert_contains "$(cat "$TMP_ROOT/default-task")" "Brief kind: scout" "dispatch retains the scout tag" @@ -625,6 +633,46 @@ for hidden in whole partial; do assert_not_contains "$sent" STALE-SCOUT-SPEC "hidden overrides never revive stale instructions" [ "$hidden" != partial ] || assert_contains "$sent" PERMITTED-SPEC-REMAINDER "permitted override remainder is retained" done +cat > "$TMP_ROOT/brief.md" <<'EOF' +# Task +This is a SCOUT task: the deliverable is a written report, not a PR. +[captain] Investigate the session-floor refusal. + [captain] Preserve the existing launch path. +Captain's ask: Retain the legacy provenance contract. +UNMARKED-STALE-SCOUT-INSTRUCTIONS +```markdown +[captain] FENCED-CAPTAIN-EXAMPLE +``` +~~~markdown +[captain] TILDE-FENCED-CAPTAIN-EXAMPLE +~~~ + [captain] INDENTED-CAPTAIN-EXAMPLE + +[captain] HIDDEN-LEGACY-INTENT + +# Outside Task +[captain] OUTSIDE-TASK-CAPTAIN-TEXT +# Current ship Firstmate spec +PERMITTED-CURRENT-SHIP-SPEC +EOF +reset; keys +printf '# dispatch-never-send marked-sections\n' > "$HOME_DIR/config/dispatch-never-send" +out=$(pick) +assert_equals 2 "$(requests)" "legacy promoted intent completes both cookbook requests" +jq -se 'all(.[]; .body.state.request | + contains("Investigate the session-floor refusal.") and + contains("Preserve the existing launch path.") and + contains("Retain the legacy provenance contract.") and + contains("PERMITTED-CURRENT-SHIP-SPEC") and + (contains("[captain]") | not) and + (contains("Captain'"'"'s ask:") | not) and + (contains("UNMARKED-STALE-SCOUT-INSTRUCTIONS") | not) and + (contains("FENCED-CAPTAIN-EXAMPLE") | not) and + (contains("INDENTED-CAPTAIN-EXAMPLE") | not) and + (contains("HIDDEN-LEGACY-INTENT") | not) and + (contains("OUTSIDE-TASK-CAPTAIN-TEXT") | not) and + (contains("Brief kind: scout") | not))' "$LOG" >/dev/null \ + || fail "each promoted request carries only permitted legacy captain words and the current ship spec" mv "$TMP_ROOT/ordinary-brief.md" "$TMP_ROOT/brief.md" pass "promotion and marked regions preserve effective task privacy" From 7ab87c91b7abab610e984cf6968b2d9ca4eb7ea0 Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Thu, 8 Oct 2026 01:40:47 -0500 Subject: [PATCH 08/12] no-mistakes(review): Make skill picker CLI executable --- bin/fm-skill-pick.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 bin/fm-skill-pick.sh diff --git a/bin/fm-skill-pick.sh b/bin/fm-skill-pick.sh old mode 100644 new mode 100755 From 4b422486de15b5719e3f240454c8712f9aae957b Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Thu, 8 Oct 2026 01:53:24 -0500 Subject: [PATCH 09/12] no-mistakes(review): Select picker regressions for spawn changes only --- bin/fm-test-run.sh | 8 ++++++-- tests/fm-test-run.test.sh | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 836df338fba..ce879c7d321 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -1645,8 +1645,12 @@ families_for_changed_path() { printf '%s\n' pure-contract-unit printf '%s\n' live-harness-optin ;; - bin/fm-spawn.sh|bin/fm-send.sh|bin/fm-harness.sh|\ - bin/fm-peek.sh|bin/fm-composer*) + bin/fm-spawn.sh) + printf '%s\n' backend-dispatch + printf '%s\n' pure-contract-unit + printf '%s\n' "__script__:fm-skill-pick.test.sh" + ;; + bin/fm-send.sh|bin/fm-harness.sh|bin/fm-peek.sh|bin/fm-composer*) printf '%s\n' backend-dispatch printf '%s\n' pure-contract-unit ;; diff --git a/tests/fm-test-run.test.sh b/tests/fm-test-run.test.sh index 619c58724a7..802440b4ee5 100755 --- a/tests/fm-test-run.test.sh +++ b/tests/fm-test-run.test.sh @@ -305,6 +305,37 @@ test_shell_line_ending_policy_selects_runner_contract() { pass "shell line-ending policy selects runner coverage" } +test_changed_spawn_selects_picker_without_broadening_siblings() { + local tmp repo listed source + tmp=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-run-spawn-selection.XXXXXX") + repo="$tmp/repo" + init_changed_fixture_repo "$repo" + printf '#!/usr/bin/env bash\n' >"$repo/tests/fm-skill-pick.test.sh" + chmod +x "$repo/tests/fm-skill-pick.test.sh" + for source in fm-spawn.sh fm-send.sh fm-harness.sh fm-peek.sh fm-composer.sh fm-composer-lib.sh; do + : >"$repo/bin/$source" + done + git -C "$repo" add bin tests/fm-skill-pick.test.sh + git -C "$repo" -c user.name=test -c user.email=test@example.invalid commit -qm spawn-selection-fixture + + for source in fm-spawn.sh fm-send.sh fm-harness.sh fm-peek.sh fm-composer.sh fm-composer-lib.sh; do + printf '\n' >"$repo/bin/$source" + listed=$(cd "$repo" && bin/fm-test-run.sh --list --changed --base HEAD) \ + || fail "$source changed selection failed" + assert_contains "$listed" "tests/fm-backend.test.sh" "$source retains backend-dispatch coverage" + assert_contains "$listed" "tests/fm-brief.test.sh" "$source retains pure-contract-unit coverage" + if [ "$source" = fm-spawn.sh ]; then + assert_contains "$listed" "tests/fm-skill-pick.test.sh" "spawn selects worker picker coverage" + else + assert_not_contains "$listed" "tests/fm-skill-pick.test.sh" "$source does not select worker picker coverage" + fi + : >"$repo/bin/$source" + done + + rm -rf "$tmp" + pass "spawn changes select picker coverage without broadening sibling commands" +} + test_changed_dependency_selection_and_unmapped_failure() { local tmp repo listed rc tmp=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-run-changed.XXXXXX") @@ -2058,6 +2089,7 @@ test_changed_runner_surfaces_select_their_family test_shell_line_ending_policy_selects_runner_contract test_changed_dependency_selection_and_unmapped_failure test_changed_fleet_snapshot_selects_only_its_exact_ledger_consumer +test_changed_spawn_selects_picker_without_broadening_siblings test_changed_status_owners_select_all_consuming_tests test_changed_bin_reference_selects_per_script_not_per_family test_changed_uses_bounded_automatic_concurrency From 3558ae9b2851ccef35de798aa3048188acd3799c Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Thu, 8 Oct 2026 03:22:11 -0500 Subject: [PATCH 10/12] no-mistakes(document): Clarify worker skill-picker contracts and documentation ownership --- .../references/common/dispatch.md | 4 +- bin/fm-skill-pick.mjs | 24 +++++----- bin/fm-skill-pick.sh | 4 +- bin/fm-spawn.sh | 18 +++----- bin/fm-typesafe-lib.sh | 11 ++--- docs/configuration.md | 45 ++++++++++++------- 6 files changed, 60 insertions(+), 46 deletions(-) diff --git a/.agents/skills/harness-adapters/references/common/dispatch.md b/.agents/skills/harness-adapters/references/common/dispatch.md index 7b5ff8126e5..927308affec 100644 --- a/.agents/skills/harness-adapters/references/common/dispatch.md +++ b/.agents/skills/harness-adapters/references/common/dispatch.md @@ -13,8 +13,8 @@ When the opt-in `bin/fm-dispatch-resolve.sh` is on, its `clear` answer already n ## Skill selection at launch -Ship and scout spawns with brief delivery add the project skill selection outcome to the worker's launch instructions, after every existing mandatory explicit/named and safety trigger. Raw commands support delivery through `__BRIEF__`, `__BRIEFDOORBELL__`, or a detected Kimi/Rovo harness's brief-file pointer; only raw commands without any supported brief transport record selection as undelivered and remain unchanged. -[Worker skill selection](../../../docs/configuration.md#worker-skill-selection) owns its input, privacy, provider order, and recorded result. +Apply every existing mandatory explicit/named and safety trigger before the launch's skill selection. +Follow [Worker skill selection](../../../docs/configuration.md#worker-skill-selection) for ship and scout launches; it owns input, brief transport, privacy, provider policy, and recorded outcomes. ## Owners diff --git a/bin/fm-skill-pick.mjs b/bin/fm-skill-pick.mjs index 8db6ac7744d..5b73f14d70b 100644 --- a/bin/fm-skill-pick.mjs +++ b/bin/fm-skill-pick.mjs @@ -3,16 +3,20 @@ // Usage: node bin/fm-skill-pick.mjs roster ... > roster.json // node bin/fm-skill-pick.mjs pick < keys // Recipe: https://docs.typesafe.ai/cookbooks/skill_suggestion ("rank the whole -// roster", "rerank the top three", suggest()); question texts, thresholds, -// shortlist and excerpt are retained; the highest-fit candidate is selected. Transport, validation, -// retries and provider endpoints are .agents/skills/hyper-jev's starter client, -// unchanged. keys is two lines, the TypeSafe key then the OpenRouter key; -// either may be empty. TypeSafe is asked directly first; when that call fails and an -// OpenRouter key exists, the same request and the rest of this run go through -// OpenRouter. Keys arrive on stdin only and are never printed. -// roster: one entry per skill name, the earlier catalog winning; a skill is -// sent only when /SKILL.md is a regular Git-tracked file with a -// description, and every other one is listed in not_judged with its reason. +// roster", "rerank the top three", suggest()); question texts, the 0.30 need +// and fit thresholds, three-skill shortlist and 700-character excerpt are retained. +// Above the client's 255-option Choice limit, each chunk's top three enter a +// common ranking before detailed reranking; independent chunk probabilities +// are never compared. Additional reduction rounds keep every Choice in bounds. +// Named deviation: deliver the highest-fit shortlisted candidate, not the +// rerank Choice winner, only when its fit meets the inclusive fit threshold. +// Transport, validation, retries and endpoints are the unchanged vendored client. +// docs/configuration.md "Worker skill selection" owns provider policy and setup. +// keys is two lines, the TypeSafe key then the OpenRouter key; either may be empty. +// roster: one entry per skill name, the earlier catalog winning; entries without +// SKILL.md are not skills and reserve no name. Only regular Git-tracked SKILL.md +// files with a readable description are judged; other discovered skills appear +// in not_judged. Linked skill contents are never read or sent. // pick prints status=, reason=, picked=, path=, fit=, provider= and model= // lines; path is a JSON string, and status is picked, none or unavailable. import { execFileSync } from 'node:child_process'; diff --git a/bin/fm-skill-pick.sh b/bin/fm-skill-pick.sh index 2abfab93c50..9c12f3fa3f0 100755 --- a/bin/fm-skill-pick.sh +++ b/bin/fm-skill-pick.sh @@ -2,8 +2,8 @@ # Pick the one project skill a worker should load, for its launch instructions. # Usage: fm-skill-pick.sh --brief --catalog ... [--record ] [--kind ship|scout] # bin/fm-skill-pick.mjs runs the TypeSafe skill-suggestion cookbook recipe on -# the vendored hyper-jev client over the catalogs' skills (an earlier catalog -# wins a duplicate name); its header owns the recipe, roster and provider order. +# the vendored hyper-jev client; its header owns recipe and roster mechanics. +# docs/configuration.md "Worker skill selection" owns provider policy and setup. # This wrapper supplies what the example cannot: the brief text dispatch # resolution may send (fm_typesafe_brief_task), the dispatch-never-send check # over every string a request can carry, and the keys from fm-typesafe-lib.sh, diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index c3d1826c0bb..c695badb605 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -42,13 +42,10 @@ # first in the private launch-brief overlay, including the exact task-owned # steering inbox. This never rewrites a project's instruction files or a # secondmate's charter. -# Every ship/scout launch and relaunch also runs bin/fm-skill-pick.sh -# once its task copy exists, over that copy's .agents/skills and -# .claude/skills, and adds the picked skill to the same overlay as one to -# load and follow; source intent and mandatory skill triggers remain -# unchanged. The picker is bounded at 30 seconds and never stops a launch: -# when it is unavailable the overlay says why, and the task record carries -# skill_selection=, skill_selection_reason=, and skill_selection_picked=. +# Ship/scout launches and relaunches integrate bin/fm-skill-pick.sh through +# that overlay. docs/configuration.md "Worker skill selection" owns selection, +# supported brief transports, and recorded outcomes. The picker is bounded +# at 30 seconds and never stops a launch. # fm-spawn.sh --relaunch [--harness ] [--model ] [--effort ] [--claude-debug] [--reconcile-only] # --claude-debug is off by default and applies to --relaunch only; a fresh ship, scout, secondmate, or batch spawn refuses it. It adds Claude Code's own --debug to a claude launch and sets CLAUDE_CODE_DIAGNOSTICS_FILE to state/.claude-diagnostics.jsonl, where Claude writes the shutdown_signal event that names the signal. It is refused unless the resolved harness is claude. # --reconcile-only applies only to --relaunch; bin/fm-control.sh's header owns @@ -4629,11 +4626,8 @@ fi spawn_enter_recorded_worktree spawn_assert_agent_worktree -# Every ship and scout launch picks from the skills of the project copy it -# works in (bin/fm-skill-pick.sh), and the pick becomes part of the launch -# instructions. The picker is bounded and only ever adds that section: when it -# cannot run, the launch continues and both the instructions and the task -# record say plainly why. +# Judge only after entering the task copy so skill paths target the worker's +# actual files. Raw commands without supported brief transport stay untouched. SKILL_SELECTION_BOUND=30 SKILL_SELECTION_STATUS='' SKILL_SELECTION_REASON='' SKILL_SELECTION_PICKED='' if { [ "$KIND" = ship ] || [ "$KIND" = scout ]; } && [ "$RAW_LAUNCH" = 1 ] && [[ "$LAUNCH" != *'__BRIEF__'* ]] && [[ "$LAUNCH" != *'__BRIEFDOORBELL__'* ]] && [ "$HARNESS" != kimi ] && [ "$HARNESS" != rovo ]; then diff --git a/bin/fm-typesafe-lib.sh b/bin/fm-typesafe-lib.sh index f3015bed548..c61242b48bf 100644 --- a/bin/fm-typesafe-lib.sh +++ b/bin/fm-typesafe-lib.sh @@ -13,11 +13,12 @@ # The private key is never exported or placed on argv. # fm_typesafe_permitted checks # every request string against the existing dispatch-never-send policy. -# fm_typesafe_brief_task writes the -# one permitted task text Jev may receive for a brief: marked never-send -# regions removed, then the scout tag plus `## Captain's intent` and -# `## Firstmate spec`, or the whole remaining brief when it has neither. -# docs/configuration.md "Never-send list" owns the marker rules. +# fm_typesafe_brief_task [ship|scout] +# writes the permitted brief task text without relaxing existing output-file +# permissions. The optional kind selects current ship instructions for promoted +# briefs or the scout tag; omitting it preserves dispatch extraction. +# docs/configuration.md "What the model receives", "Never-send list", and +# "Worker skill selection" own the extraction and privacy contracts. # A refusal sets FM_TYPESAFE_WITHHELD_REASON and returns 1, without echoing text. TYPESAFE_API_KEY_PRIVATE=${TYPESAFE_API_KEY_PRIVATE:-${TYPESAFE_API_KEY:-}} diff --git a/docs/configuration.md b/docs/configuration.md index 8f529ec9d9a..72dfc8b027a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1515,7 +1515,8 @@ A brief containing ``), unspaced (``), or differently cased (``) markers stop the entire request rather than being corrected. A brief without such text is sent as before. This option protects only the marked occurrences in the brief, not copies elsewhere or dispatch-rule text; use literals when those must also be withheld. -On 2026-10-07, the captain authorized worker skill selection to send the same sanitized, never-send-filtered task text used by `bin/fm-dispatch-resolve.sh` to TypeSafe and its OpenRouter fallback, ruling “I asked for A already.” This authorization does not widen any other sending boundary. +On 2026-10-07, the captain authorized worker skill selection to send the same sanitized, never-send-filtered task text used by `bin/fm-dispatch-resolve.sh` to TypeSafe and its OpenRouter fallback, ruling “I asked for A already.” +This authorization does not widen any other sending boundary. Outside that worker skill-selection authorization, do not send real Vernant/customer text until separately authorized: TypeSafe's public terms have not established the required `standard_confidential/v1` processor protections of deletion within 30 days and no training. Before each request is sent, every remaining string in it is checked for literal matches through `bin/fm-typesafe-lib.sh`: for dispatch resolution, the project name, the sanitized task text, each rule's `when`, and the fixed question text; for worker skill selection, each complete outbound request, including task text, question instructions, assembled skill criteria, and the requested model ID, before either direct or fallback transport; belay checks every request string. @@ -1607,7 +1608,9 @@ Firstmate passes its profile line unless it states a reason to override, such as **Key handling and fixed settings** - The resolver, skill picker, and bootstrap copy an environment-provided key into a non-exported private variable and unset `TYPESAFE_API_KEY` before launching child processes, so the secret is absent from child environments. -- The resolver sends its key to `curl` only as a header read from a file descriptor, never on argv. The skill picker pipes its keys on stdin to Node, which supplies them to the vendored JevClient's fetch transport; they never enter Node's environment or arguments. Neither tool prints, logs, or writes keys to files. +- The resolver sends its key to `curl` only as a header read from a file descriptor, never on argv. + The skill picker pipes its keys on stdin to Node, which supplies them to the vendored JevClient's fetch transport; they never enter Node's environment or arguments. + Neither tool prints, logs, or writes keys to files. - The resolver fixes the endpoint at `https://api.typesafe.ai`, model at `jev-latest`, default confidence floor at 0.6, and request timeout at 5 seconds; `TYPESAFE_API_KEY` is its only resolver-specific environment setting. The live rule-match evidence is recorded in [`verification/dispatch-resolve.md`](verification/dispatch-resolve.md). @@ -1703,29 +1706,40 @@ To move to a newer upstream version, review it, update the pinned commit in this ## Worker skill selection -Every ship and scout launch and relaunch with brief delivery runs `bin/fm-skill-pick.sh` once its task copy exists, so the worker starts with the one project skill that fits its task. +Every ship and scout launch and relaunch with brief delivery runs `bin/fm-skill-pick.sh` once its task copy exists, to select a project skill for the task. Secondmate charters are not judged. The picker reads the task copy's `.agents/skills` and `.claude/skills`; a firstmate task's copy holds firstmate's own skills. -It runs the [TypeSafe skill-suggestion cookbook](https://docs.typesafe.ai/cookbooks/skill_suggestion) recipe on the vendored [hyper-jev](../.agents/skills/hyper-jev/SOURCE.md) client, and `bin/fm-skill-pick.mjs`'s header owns the recipe, roster rules, and provider order. +It runs the [TypeSafe skill-suggestion cookbook](https://docs.typesafe.ai/cookbooks/skill_suggestion) recipe on the vendored [hyper-jev](../.agents/skills/hyper-jev/SOURCE.md) client; the [picker header](../bin/fm-skill-pick.mjs) owns recipe mechanics, large-roster chunking, and roster rules. -For rosters over 255 skills, chunk-local top-three candidates enter a common Choice ranking before the detailed shortlist step; probabilities from independent chunks are never compared. This follows the cookbook's shortlist-over-chunk-winners guidance, retaining three candidates per chunk to preserve near-winners, and adds ranking requests to the ordinary two-request flow (`bin/fm-skill-pick.mjs:245-264`). - -**Named cookbook deviation — highest-fit delivery:** the rerank Choice is retained, but the delivered pick is the highest-fit shortlisted skill rather than the Choice winner, and its fit must be at least 0.30 (`bin/fm-skill-pick.mjs:278-288`). The captain chose this deviation because workers must read and follow the delivered skill, rather than receive the cookbook's ignorable advisory suggestion. +**Named cookbook deviation - highest-fit delivery:** the captain chose a highest-fit pick that workers must read and follow, rather than the cookbook's ignorable advisory suggestion. +The picker header owns the precise selection rule and thresholds. The pick is added to the launch instructions as a skill to read in full and follow. +The launch instructions take precedence over a conflicting picked skill. Existing mandatory skill triggers and the worker's own skill index still apply first and unchanged. -A skill the picker could not send, such as one that is not a Git-tracked file in the task copy or a linked skill directory containing `SKILL.md`, is listed by its local name for the worker to check; linked skill contents are never read or sent. Links without `SKILL.md` are not skills: they reserve no name and produce no manual-check instruction. -No-fit outcomes apply only to judged skills; when every discovered skill is excluded, selection is unavailable with the exclusion reasons, not a finding that no skill fits. A genuinely empty catalog reports that there are no skills to judge. When the picker cannot run, the launch continues and the instructions say why. +A skill the picker could not send, such as one that is not a Git-tracked file in the task copy or a linked skill directory containing `SKILL.md`, is listed by its local name for the worker to check; linked skill contents are never read or sent. +Links without `SKILL.md` are not skills: they reserve no name and produce no manual-check instruction. +No-fit outcomes apply only to judged skills; when every discovered skill is excluded, selection is unavailable with the exclusion reasons, not a finding that no skill fits. +A genuinely empty catalog reports that there are no skills to judge. +When the picker cannot run, the launch continues and the instructions say why. The task record carries `skill_selection=` (picked, none, unavailable, or undelivered), `skill_selection_reason=`, and `skill_selection_picked=` when a pick is delivered. -Raw commands receive the selection when they contain `__BRIEF__` or `__BRIEFDOORBELL__`, or when their detected harness is Kimi or Rovo, which receives a brief-file pointer. Without any supported brief transport, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. Failure to publish the selection overlay also records undelivered and clears the pick. If launch delivery fails, any retained record for that launch is marked undelivered with the failure reason and no claimed pick; an abort before replacement-record publication preserves the prior record. +Raw commands receive the selection when they contain `__BRIEF__` or `__BRIEFDOORBELL__`, or when their detected harness is Kimi or Rovo, which receives a brief-file pointer. +Without any supported brief transport, selection is recorded as undelivered with its reason, no pick is claimed, and the command is unchanged. +Failure to publish the selection overlay also records undelivered and clears the pick. +If launch delivery fails, any retained record for that launch is marked undelivered with the failure reason and no claimed pick; an abort before replacement-record publication preserves the prior record. `bin/fm-spawn.sh` stops the picker after 30 seconds. -Only the brief text dispatch resolution may send is sent, after the same never-send checks; see the [never-send list](#typed-dispatch-resolution-env-typesafe_api_key). For a promoted scout's ship relaunch, skill selection retains the original Captain's intent or provenance-marked legacy Task words and uses the recorded ship kind and current ship Firstmate spec rather than the superseded scout spec; ordinary dispatch extraction is unchanged. Legacy provenance uses the same fence and indentation exclusions as promotion. Privacy-hidden promotion instructions never revive the superseded spec. -The `TYPESAFE_API_KEY` opt-in enables it, and TypeSafe is asked directly first. -When the direct call is unavailable because its key is absent or fails and `OPENROUTER_API_KEY` is set in the same home `.env`, the same request and the rest of that pick go through OpenRouter. This is the captain's accepted policy: “openrouter is a fallback from directly using the typesafe api.” +Only the brief text dispatch resolution may send is sent, after the same never-send checks; see the [never-send list](#typed-dispatch-resolution-env-typesafe_api_key). +For a promoted scout's ship relaunch, skill selection retains the original Captain's intent or provenance-marked legacy Task words and uses the recorded ship kind and current ship Firstmate spec rather than the superseded scout spec; ordinary dispatch extraction is unchanged. +Legacy provenance uses the same fence and indentation exclusions as promotion. +Privacy-hidden promotion instructions never revive the superseded spec. +The picker uses the shared [TypeSafe key lookup](#typed-dispatch-resolution-env-typesafe_api_key), including the local primary-home fallback, and asks TypeSafe directly first when that key is available. +When the direct call is unavailable because its key is absent or fails and `OPENROUTER_API_KEY` is set in the calling home's `.env`, the same request and the rest of that pick go through OpenRouter. +Unlike the TypeSafe key, the OpenRouter key is read only from that home's `.env`, with no primary-home or ambient-environment fallback. +This is the captain's accepted policy: “openrouter is a fallback from directly using the typesafe api.” Fallback provenance is retained for picked, no-selection, and unavailable outcomes. Missing optional catalog directories are skipped; failures inspecting existing catalogs are unavailable with their actionable reason, not successful no-selection results. -Keys reach the picker on stdin, never in its environment or arguments. +The shared [key-handling contract](#typed-dispatch-resolution-env-typesafe_api_key) owns credential transport and secrecy. [`tests/fm-skill-pick.test.sh`](../tests/fm-skill-pick.test.sh) exercises selection, both thresholds, the fallback, the never-send boundary, and chunking through the public tool. The cookbook's published rates are external design context, not validated performance evidence for this caller. @@ -1742,7 +1756,8 @@ Required tools come in two parts: a universal toolchain every home needs regardl Every home requires: -- Node 24+ with native TypeScript loading enabled, and git. Worker skill selection capability-checks imports of its vendored TypeScript client; unsupported TypeScript loading is a runtime prerequisite, while other import failures report a client-loading error. The picker test suite emits canonical `skip:` output only for missing Node or unsupported TypeScript loading; other client-loading errors fail the suite. +- Node 24+ with native TypeScript loading enabled, and git. + Worker skill selection capability-checks imports of its vendored TypeScript client; unsupported TypeScript loading is a runtime prerequisite, while other import failures report a client-loading error. - gh, with GitHub authentication through `gh auth login`. - no-mistakes v1.46.0 or newer. - Compatible gh-axi. From 1625f6e696546b4c2b8f25ba998bf7edf883eb0e Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Thu, 8 Oct 2026 04:58:02 -0500 Subject: [PATCH 11/12] =?UTF-8?q?no-mistakes(ci):=20Diagnosed=20Lint=201:?= =?UTF-8?q?=20bin/fm-spawn.sh=20exhausted=20ShellCheck=E2=80=99s=20memory?= =?UTF-8?q?=20limit=20(exit=20251,=20peak=20RSS=208,388,000=20KiB).=20Remo?= =?UTF-8?q?ved=20redundant=20direct=20imports=20of=20fm-pr-lib.sh=20and=20?= =?UTF-8?q?fm-classify-lib.sh;=20fm-dod-lib.sh=20still=20loads=20both=20be?= =?UTF-8?q?fore=20use,=20preserving=20full=20source-aware=20analysis=20and?= =?UTF-8?q?=20launch=20behavior.=20Updated=20lint=20ownership=20documentat?= =?UTF-8?q?ion=20without=20changing=20CI=20limits=20or=20diagnostic=20rule?= =?UTF-8?q?s.=20Verification:=20uncached=20full=20spawn=20lint=20passed=20?= =?UTF-8?q?with=20peak=20RSS=207,767,568=20KiB=20on=20Darwin;=20the=20comp?= =?UTF-8?q?lete=20fm-skill-pick=20behavioral=20suite=20passed=20when=20run?= =?UTF-8?q?=20without=20concurrent=20heavy=20lint;=20pinned=20actionlint?= =?UTF-8?q?=20validated=20all=20three=20workflows.=20The=20broader=20lint?= =?UTF-8?q?=20regression=20suite=20and=20canonical=20partition=20exceeded?= =?UTF-8?q?=20local=20command=20deadlines.=20Linux=20address-space=20enfor?= =?UTF-8?q?cement=20remains=20unverified=20locally=20because=20macOS=20can?= =?UTF-8?q?not=20enforce=20that=20CI=20bound.=20Temporary=20verification?= =?UTF-8?q?=20files=20were=20removed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- bin/fm-spawn.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index c695badb605..824e30138ba 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -696,6 +696,7 @@ fm_backlog_directory_present "$STATE" "state directory" || { . "$SCRIPT_DIR/fm-busy-lib.sh" # shellcheck source=bin/fm-cursor-lib.sh . "$SCRIPT_DIR/fm-cursor-lib.sh" +# fm-dod-lib.sh also owns the PR and classification imports needed below. # shellcheck source=bin/fm-dod-lib.sh . "$SCRIPT_DIR/fm-dod-lib.sh" # shellcheck source=bin/fm-trace-context-lib.sh From 555f6d44341ba11d7f854817fbf749940e8126d2 Mon Sep 17 00:00:00 2001 From: Charles Brooker Date: Thu, 8 Oct 2026 06:25:29 -0500 Subject: [PATCH 12/12] no-mistakes(document): Clarify skill picker credential transport and fallback boundaries --- docs/configuration.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 72dfc8b027a..be55a0081b1 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1609,7 +1609,7 @@ Firstmate passes its profile line unless it states a reason to override, such as - The resolver, skill picker, and bootstrap copy an environment-provided key into a non-exported private variable and unset `TYPESAFE_API_KEY` before launching child processes, so the secret is absent from child environments. - The resolver sends its key to `curl` only as a header read from a file descriptor, never on argv. - The skill picker pipes its keys on stdin to Node, which supplies them to the vendored JevClient's fetch transport; they never enter Node's environment or arguments. + The skill picker pipes its resolved keys on stdin to Node, which supplies them to the vendored JevClient's fetch transport; the picker does not export the resolved key variables or put their values on argv. Neither tool prints, logs, or writes keys to files. - The resolver fixes the endpoint at `https://api.typesafe.ai`, model at `jev-latest`, default confidence floor at 0.6, and request timeout at 5 seconds; `TYPESAFE_API_KEY` is its only resolver-specific environment setting. @@ -1734,7 +1734,8 @@ For a promoted scout's ship relaunch, skill selection retains the original Capta Legacy provenance uses the same fence and indentation exclusions as promotion. Privacy-hidden promotion instructions never revive the superseded spec. The picker uses the shared [TypeSafe key lookup](#typed-dispatch-resolution-env-typesafe_api_key), including the local primary-home fallback, and asks TypeSafe directly first when that key is available. -When the direct call is unavailable because its key is absent or fails and `OPENROUTER_API_KEY` is set in the calling home's `.env`, the same request and the rest of that pick go through OpenRouter. +When the TypeSafe key is absent or a direct request fails and `OPENROUTER_API_KEY` is set in the calling home's `.env`, the same request and the rest of that pick go through OpenRouter. +Privacy refusals, question-validation failures, and expiry of the picker's overall deadline stop selection without triggering provider fallback. Unlike the TypeSafe key, the OpenRouter key is read only from that home's `.env`, with no primary-home or ambient-environment fallback. This is the captain's accepted policy: “openrouter is a fallback from directly using the typesafe api.” Fallback provenance is retained for picked, no-selection, and unavailable outcomes.