diff --git a/AGENTS.md b/AGENTS.md index 27b91b6b750..665e1b6efb4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ Hard rules, in priority order: 1. **Never write to a project.** Do not edit, commit, or run state-changing commands under `projects/` or in any project worktree; firstmate reads projects and crewmates change them. - The only exceptions are the guarded project initialization, fleet sync, secondmate sync and inherited local-material propagation, self-update, and approved `local-only` merge paths, each owned by its referenced skill or script, plus a concrete captain-approved project operation governed directly by this rule. + The only exceptions are the guarded project initialization, fleet sync, secondmate sync and inherited local-material propagation, self-update, approved `local-only` merge, and the purely additive teardown harvest of crew-generated untracked files into the project (`bin/fm-teardown.sh`, never overwriting an existing project file) paths, each owned by its referenced skill or script, plus a concrete captain-approved project operation governed directly by this rule. Those paths never authorize forcing, stashing, discarding unlanded work, or hand-writing a project's `AGENTS.md`. Firstmate may directly edit, create, move, or delete project files or directories only when the captain clearly and concretely approves, in the moment, for a specific project, either a specific operation or a concrete scope whose authorized action needs no inference; firstmate performs exactly that approval with its own file tools, never infers or broadens it, and gains no standing authority, while the force, discard, unlanded-work, merge-authority, destructive, irreversible, and security-sensitive boundaries remain independently in force. 2. **Never merge a PR without the captain's explicit word.** @@ -327,7 +327,7 @@ Fill the task subsections according to section 11. ### Dispatch and supervision handoff Spawn only through `bin/fm-spawn.sh` after the profile and backend checks in section 4. -The spawn must resolve a genuine isolated task worktree distinct from the primary checkout; a failed isolation assertion stops the task. +The spawn must resolve a genuine isolated task worktree distinct from the primary checkout and belonging to that same project, so an unrelated git repo the pane transiently sits in is never mistaken for the task worktree (`bin/fm-spawn.sh`'s header states the full rule); a failed isolation assertion stops the task. When the configured tasks-axi backlog gate applies, the spawn itself moves the work item to In flight and refuses rather than dispatching work this home has no item for, so recording the dispatch is never a separate step to remember; a manual-backend home retains the hand-editing contract in `docs/configuration.md`. After spawning, confirm the worker is processing the brief and handle any trust dialog through `harness-adapters`. A persistent secondmate is recorded in the secondmate registry and runtime state, never as a backlog work item. @@ -400,6 +400,7 @@ Retire a custom check only through `bin/fm-check-unregister.sh ` (or `bin/fm Tear down a ship task only after landing is confirmed. A teardown refusal for uncommitted or unlanded work is a stop-and-investigate result, never an obstacle to bypass. +Before that check, a ship teardown harvests every untracked, non-ignored file the crew generated into the project's primary checkout at the same relative path (never overwriting an existing file) and removes it from the worktree, so generated notes, docs, and scratch survive the worktree hard-reset; leftover untracked files therefore no longer refuse a ship teardown, while committed-but-unlanded work still does. Never force teardown without explicit discard authority. After successful teardown, record completion, retain only the configured recent Done history, and re-evaluate queued work whose blockers and time gates have cleared. @@ -454,7 +455,7 @@ A forced repair must use the home-scoped owner path emitted by supervision instr Guard warnings do not replace the contract. Queued wakes must be presented before other action and acknowledged only after handling, stale liveness must be repaired through the emitted protocol, and the worktree-tangle warning must be resolved without touching unlanded work. -The spawn assertion and generated ship brief must both enforce that project work starts in an isolated disposable worktree, never the primary checkout. +The spawn assertion and generated ship brief must both enforce that project work starts in an isolated disposable worktree of the project being spawned into, never the primary checkout or an unrelated repo. Harness-aware turn-end guards are structural backstops, not permission to omit the live cycle. ### Away-mode and quiet-mode stub @@ -528,6 +529,7 @@ Batch non-urgent updates into the next natural reply. Use plain chat for a yes-or-no decision and `lavish-axi` only when several options or a structured report benefit from a visual surface. Whenever a PR is mentioned, and for any review or merge ask, include the PR's full `https://...` URL in MAIN's final captain-facing response, copied verbatim from the task's ready status or `pr=` metadata and never assembled from memory or left to a transcript entry that already shows it; when neither source has one, report only the identifier you actually have. Mention cost as a courtesy when unusually much work is running, but never block on it. +When a claude crewmate or scout finishes, report its exact token usage with `bin/fm-token-usage.sh ` (claude-harness only; see `docs/token-usage.md`). ## 10. Backlog contract diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 518e905f60f..951e1344ae6 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -209,6 +209,11 @@ # itself a linked worktree of the project repository still launches. A pane # that never reaches an isolated worktree refuses at the end of that wait, # naming the last path seen and why it was rejected. +# Isolation alone is not enough: the path must also be a worktree of that +# SAME project (matching --git-common-dir), so an unrelated git repo the pane +# transiently sits in can never be mistaken for the task worktree. +# FM_SPAWN_WORKTREE_TIMEOUT (default 60) bounds, in seconds, how long the +# post-`treehouse get` poll waits for that worktree before giving up. # That placement is proven only at launch. Every ship or scout pane therefore # also receives `export FM_TASK_ID=` before the launch command, on # the same channel as GOTMPDIR, and bin/fm-test-run.sh refuses to execute the @@ -2827,6 +2832,48 @@ real_path_or_raw() { # fi } +# The project repo's shared git dir. Every linked worktree of this project - +# which is exactly what `treehouse get` hands out - reports this same path as +# its --git-common-dir, while an unrelated repo reports its own. That is the +# property that identifies a real task worktree. +PROJ_GIT_COMMON=$(git -C "$PROJ_ABS" rev-parse --path-format=absolute --git-common-dir 2>/dev/null || true) +# Fail-open is deliberate (see is_project_worktree), but it must never be +# silent: without this line nothing reveals that the worktree-identity check is +# inert until another agent launches somewhere it should not have. Any failure +# of the rev-parse above lands here, so the message names every cause it can +# have rather than asserting one. Emitted once per spawn, here rather than +# inside the predicate, because the discovery poll calls it up to +# FM_SPAWN_WORKTREE_TIMEOUT times. +if [ -z "$PROJ_GIT_COMMON" ] && [ "$KIND" != secondmate ]; then + echo "warning: could not determine the git common dir of $PROJ_ABS (git may be missing from PATH, the directory may not be a git repo, git may have refused it over safe.directory ownership, or git may predate 2.31's rev-parse --path-format); the worktree-identity check is DISABLED for this spawn, which proceeds on the isolation check alone" >&2 +fi + +# is_project_worktree: true when is a worktree of the SAME repository as +# the project. +# +# Why this exists: the worktree-discovery poll below waits for the pane's +# foreground cwd to differ from the project, then treats whatever it sees as +# the task worktree. That is too weak. oh-my-zsh's oh-my-zsh.sh does +# `builtin cd -q "$ZSH"` on EVERY shell startup to read its revision for the +# zcompdump stamp, so a freshly spawned pane transiently reports +# ~/.oh-my-zsh as its foreground cwd. The poll latched onto it and launched +# the agent there. The old isolation guard passed it too, because ~/.oh-my-zsh +# IS a git repo and IS not the primary checkout - the guard tested "some other +# git repo" when it meant "a worktree treehouse gave us". +is_project_worktree() { # + # NB: the local is deliberately NOT named `path`. In zsh `path` is tied to + # PATH, so `local path=...` empties PATH for the function's scope and every + # command lookup inside it fails silently. This script is bash, but the + # surrounding fleet is zsh and the same helper shape gets copied around. + local candidate=$1 common + [ -n "$candidate" ] || return 1 + # cannot tell; do not block the spawn - the condition is reported once at + # spawn time where PROJ_GIT_COMMON is resolved. + [ -n "$PROJ_GIT_COMMON" ] || return 0 + common=$(git -C "$candidate" rev-parse --path-format=absolute --git-common-dir 2>/dev/null || true) + [ -n "$common" ] && [ "$common" = "$PROJ_GIT_COMMON" ] +} + # Session-provider container-ensure + task creation. tmux stays exactly as P1 # left it (same session-name / new-window sequence, see bin/backends/tmux.sh); # a herdr spawn goes through the version-gated, workspace-per-HOME, @@ -2838,7 +2885,9 @@ real_path_or_raw() { # # True when is an isolated worktree of the spawning project: a real # directory that is its own worktree root, is not the spawning project itself, -# and does not share the project repository's common git dir. SPAWN_WT_TOP is +# does not use the project repository's common git dir as its own git dir, and +# reports that same common git dir as its --git-common-dir (so it belongs to +# this project and is not some unrelated repo; see is_project_worktree). SPAWN_WT_TOP is # left holding the worktree root the check read, and SPAWN_WT_REASON a short # phrase naming why a rejected path failed, both for the refusal messages. # @@ -2900,13 +2949,20 @@ spawn_worktree_isolated() { # SPAWN_WT_REASON="it is the repository's primary checkout (its git dir is the spawning project's common git dir)" return 1 fi + # Being *a* git repo that is not the primary is not enough: it must be a + # worktree of THIS project. Without this, any unrelated repo the pane + # transiently sits in passes (see is_project_worktree). + if ! is_project_worktree "$wt_real"; then + SPAWN_WT_REASON="it is a git repo but NOT a worktree of the spawning project (its git common dir is '$(git -C "$wt_real" rev-parse --path-format=absolute --git-common-dir 2>/dev/null || echo unknown)', expected '$PROJ_GIT_COMMON')" + return 1 + fi return 0 } validate_spawn_worktree() { # local source=$1 inspect_target=$2 if ! spawn_worktree_isolated "$WT"; then - echo "error: $source did not yield an isolated worktree (resolved '$WT'; worktree root '${SPAWN_WT_TOP:-none}'; spawning project '$PROJ_ABS'); refusing to launch to avoid tangling the primary checkout. Inspect target $inspect_target" >&2 + echo "error: $source did not yield an isolated worktree (resolved '$WT'; worktree root '${SPAWN_WT_TOP:-none}'; spawning project '$PROJ_ABS'; ${SPAWN_WT_REASON:-rejected}); refusing to launch to avoid tangling the primary checkout. Inspect target $inspect_target" >&2 exit 1 fi } @@ -3874,10 +3930,14 @@ elif [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then # misconfiguration would need machinery this path does not want - so the # refusal has to be self-explaining instead: carry the last path seen and the # reason it was rejected, and report both at the deadline. + # A wrong path is never terminal - it may simply be transient - so the loop + # waits it out and FM_SPAWN_WORKTREE_TIMEOUT (default 60) is the only give-up + # point. candidate="" last_seen="" last_reason="the pane reported no path" - for _ in $(seq 1 60); do + SPAWN_WT_TIMEOUT=${FM_SPAWN_WORKTREE_TIMEOUT:-60} + for _ in $(seq 1 "$SPAWN_WT_TIMEOUT"); do p=$(spawn_current_path "$WT_TARGET" || true) [ -z "$p" ] || last_seen="$p" if [ -n "$p" ] && spawn_worktree_isolated "$p"; then @@ -3895,7 +3955,7 @@ elif [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then sleep 1 done if [ -z "$WT" ]; then - echo "error: treehouse get did not enter an isolated worktree within 60s (last seen '${last_seen:-none}': $last_reason; spawning project '$PROJ_ABS'); inspect window $T" >&2 + echo "error: treehouse get did not enter an isolated worktree of $PROJ_ABS within ${SPAWN_WT_TIMEOUT}s (last seen '${last_seen:-none}': $last_reason; spawning project '$PROJ_ABS'); inspect window $T" >&2 exit 1 fi diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index a5a41e8a451..497c9edf7ab 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -66,6 +66,14 @@ # A gh lookup error falls back to the content check; if that is also inconclusive, # teardown refuses rather than risk discarding unlanded work. # Uncommitted changes are never landed. +# Before the safety check and removal, a ship task's worktree is harvested: every +# untracked, non-ignored file the crew generated is copied into the project's +# primary checkout (never overwriting an existing path) and then removed from the +# worktree, so generated notes/docs/scratch survive the hard-reset and the tree is +# clean for the safety check. This is a captain-requested write into the project +# (AGENTS.md section 1 write exception); it is purely additive, makes no git-state +# change, and never touches committed-but-unlanded work (which still refuses). Scout +# (scratch worktree, report is the deliverable) and secondmate teardowns are exempt. # local-only projects additionally accept work merged into the local default # branch (firstmate performs that merge after configured approval) as a fallback # for the common case where there is no remote at all. @@ -1777,6 +1785,53 @@ teardown_treehouse_return() { return 1 } +# Harvest crew-generated files before the worktree is destroyed. treehouse return +# hard-resets the worktree, so any file the crew left untracked - generated notes, +# docs, scratch outputs - would be lost with the pool slot. Copy every untracked, +# non-ignored file into the project's primary checkout at the same relative path, +# then remove it from the worktree so the existing dirty check sees a clean tree and +# teardown proceeds instead of refusing on the untracked files. Purely additive at +# the destination: a path that already exists is NEVER overwritten (the worktree +# copy is dropped with the worktree, exactly as it would be without this feature), +# and no git-state change is made to either repo. --exclude-standard drops +# firstmate's own gitignored hook files, so they are never harvested. A file that +# cannot be copied is left in place, so a copy failure surfaces as the normal +# dirty-worktree refusal rather than a silent loss. Committed-but-unlanded work is +# untouched here and still blocks teardown via the landed-work check. +# This is the captain-requested teardown harvest (AGENTS.md section 1 write +# exception); it runs for ship tasks only, never for scout (scratch worktree, the +# report is the deliverable) or secondmate (a home, not a project worktree). +harvest_untracked_into_project() { + local wt=$1 proj=$2 rel src dst copied=0 skipped=0 wt_abs proj_abs + [ -n "$wt" ] && [ -d "$wt" ] || return 0 + [ -n "$proj" ] && [ -d "$proj" ] || return 0 + wt_abs=$(cd "$wt" 2>/dev/null && pwd -P) || return 0 + proj_abs=$(cd "$proj" 2>/dev/null && pwd -P) || return 0 + [ "$wt_abs" != "$proj_abs" ] || return 0 + git -C "$wt" rev-parse --is-inside-work-tree >/dev/null 2>&1 || return 0 + while IFS= read -r rel; do + [ -n "$rel" ] || continue + src="$wt/$rel" + [ -f "$src" ] || continue + dst="$proj/$rel" + if [ -e "$dst" ]; then + skipped=$((skipped + 1)) + echo "harvest: skip (already at destination) $rel" >&2 + elif mkdir -p "$(dirname "$dst")" 2>/dev/null && cp "$src" "$dst" 2>/dev/null; then + copied=$((copied + 1)) + echo "harvest: kept $rel" >&2 + else + echo "harvest: could not copy $rel (leaving it in the worktree)" >&2 + continue + fi + rm -f "$src" 2>/dev/null || true + done < <(git -C "$wt" ls-files --others --exclude-standard 2>/dev/null) + if [ "$copied" -gt 0 ] || [ "$skipped" -gt 0 ]; then + echo "harvest: $copied file(s) copied into $proj, $skipped already present" >&2 + fi + return 0 +} + validate_worktree_teardown_safety() { local dirty_raw dirty unpushed_raw unpushed DEFAULT unmerged_raw unmerged branch [ -d "$WT" ] || return 0 @@ -3340,6 +3395,17 @@ if [ "$BACKEND" = orca ] && [ "$KIND" != scout ] && [ "$KIND" != secondmate ] && ORCA_PATH_MATCH_VERIFIED=1 fi +# Harvest crew-generated untracked files into the project before any safety check +# or removal, so they survive the worktree being hard-reset. Runs on every ship +# teardown (including --force) whose recorded slot this teardown still owns; a +# slot reassigned to another task is never read or touched. Removing the +# harvested files leaves the worktree clean of non-hook untracked files, so the +# safety check below no longer refuses on them; committed-but-unlanded work is +# untouched and still refuses. +if [ "$KIND" != secondmate ] && [ "$KIND" != scout ] && teardown_owns_worktree; then + harvest_untracked_into_project "$WT" "$PROJ" +fi + if teardown_owns_worktree && [ -d "$WT" ] && [ "$FORCE" != "--force" ]; then if validate_worktree_teardown_safety; then : diff --git a/bin/fm-token-usage.sh b/bin/fm-token-usage.sh new file mode 100755 index 00000000000..76d55d3428b --- /dev/null +++ b/bin/fm-token-usage.sh @@ -0,0 +1,188 @@ +#!/usr/bin/env bash +# fm-token-usage.sh - report claude token usage for a crewmate/scout task. +# +# Why this exists: firstmate supervises crewmates but had no precise accounting of +# what a crewmate cost in tokens. The claude harness pane shows only per-subagent +# counters, not a session total, so eyeballing the pane undercounts. This reads +# claude's own session transcripts, which survive teardown, and sums the exact +# per-message usage. +# +# Data source (claude Code, verified 2026-07-07 on Claude Code 2.1.x): +# ~/.claude/projects//.jsonl main agent +# ~/.claude/projects///subagents/*.jsonl each subagent +# The is the absolute worktree path with every '/' and '.' +# replaced by '-'. Subagent (Task) usage lives ONLY in the subagents/ dir, never +# in the main jsonl (its lines carry no isSidechain usage), so both must be +# summed or the total undercounts by the whole subagent fan-out. +# See docs/token-usage.md for the empirical evidence. +# +# Usage: +# fm-token-usage.sh [--json] +# fm-token-usage.sh --cwd [--session ] [--all] [--since ] [--json] +# +# task-id mode reads worktree= and harness= from state/.meta. Only +# harness=claude is supported; any other harness exits non-zero with a clear +# message, because non-claude adapters record usage differently. +# +# Session scoping (a pooled worktree slot is reused across tasks, so a cwd can +# hold several sessions over time): +# default the NEWEST session in the cwd (by main-jsonl mtime) + its subagents +# --session ID exactly that session + its subagents +# --since EPOCH every session whose main jsonl mtime >= EPOCH + their subagents +# --all every session ever recorded for the cwd + all subagents +# At a crewmate's done (the normal call site) the newest session IS this task's, +# so the default needs no flags. +# +# Env: FM_CLAUDE_PROJECTS_DIR overrides ~/.claude/projects (used by the tests). +# +# Output: a human summary by default, or a JSON object with --json: +# {"input":N,"output":N,"cache_create":N,"cache_read":N,"total":N,"sessions":N,"files":N} +# Read-only. Exit 0 on a successful read (even zero usage), 2 on a usage error, +# 3 on an unsupported harness, 4 when no session data exists for the cwd. +set -u + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +PROJECTS_DIR="${FM_CLAUDE_PROJECTS_DIR:-$HOME/.claude/projects}" + +usage() { + echo "usage: fm-token-usage.sh [--json]" >&2 + echo " fm-token-usage.sh --cwd [--session ] [--all] [--since ] [--json]" >&2 +} + +CWD="" +ID="" +SESSION="" +SINCE="" +ALL=0 +JSON=0 +while [ $# -gt 0 ]; do + case "$1" in + --cwd) CWD=${2:-}; shift 2 ;; + --session) SESSION=${2:-}; shift 2 ;; + --since) SINCE=${2:-}; shift 2 ;; + --all) ALL=1; shift ;; + --json) JSON=1; shift ;; + -h|--help) usage; exit 0 ;; + -*) echo "fm-token-usage.sh: unknown option $1" >&2; usage; exit 2 ;; + *) + if [ -z "$ID" ]; then ID=$1; else + echo "fm-token-usage.sh: unexpected argument $1" >&2; usage; exit 2 + fi + shift ;; + esac +done + +HARNESS="claude" +if [ -n "$ID" ]; then + META="$STATE/$ID.meta" + [ -f "$META" ] || { echo "fm-token-usage.sh: no meta for task '$ID' at $META" >&2; exit 2; } + CWD=$(sed -n 's/^worktree=//p' "$META" | head -1) + HARNESS=$(sed -n 's/^harness=//p' "$META" | head -1) + [ -n "$CWD" ] || { echo "fm-token-usage.sh: meta for '$ID' has no worktree=" >&2; exit 2; } +elif [ -z "$CWD" ]; then + usage; exit 2 +fi + +case "$HARNESS" in + claude|claude*) : ;; + *) echo "fm-token-usage.sh: token usage is only supported for claude crewmates (harness=$HARNESS)" >&2; exit 3 ;; +esac + +command -v jq >/dev/null 2>&1 || { echo "fm-token-usage.sh: jq is required" >&2; exit 2; } + +# Encode the absolute cwd the way claude names its project dir: every '/' and +# '.' becomes '-'. +enc=$(printf '%s' "$CWD" | tr './' '-') +PROJDIR="$PROJECTS_DIR/$enc" +[ -d "$PROJDIR" ] || { echo "fm-token-usage.sh: no claude session data for $CWD (looked in $PROJDIR)" >&2; exit 4; } + +# Epoch mtime of a file. GNU stat (Linux) is tried first because BSD's +# "-f " spelling means "--file-system" to GNU stat, which then dumps a +# whole file-system record instead of failing; BSD stat rejects "-c" cleanly. +# Anything that is not a bare integer is treated as no reading. +_mtime() { + local m + m=$(stat -c %Y "$1" 2>/dev/null) || m=$(stat -f %m "$1" 2>/dev/null) || return 1 + case $m in ''|*[!0-9]*) return 1 ;; esac + printf '%s\n' "$m" +} + +# Collect the main-session jsonl files that are in scope. +mains=() +if [ -n "$SESSION" ]; then + [ -f "$PROJDIR/$SESSION.jsonl" ] && mains+=("$PROJDIR/$SESSION.jsonl") +else + all_mains=() + while IFS= read -r f; do all_mains+=("$f"); done < <(find "$PROJDIR" -maxdepth 1 -type f -name '*.jsonl' 2>/dev/null | sort) + if [ "${#all_mains[@]}" -eq 0 ]; then + echo "fm-token-usage.sh: no session transcripts under $PROJDIR" >&2; exit 4 + fi + if [ "$ALL" -eq 1 ]; then + mains=("${all_mains[@]}") + elif [ -n "$SINCE" ]; then + for f in "${all_mains[@]}"; do + m=$(_mtime "$f"); [ -n "$m" ] && [ "$m" -ge "$SINCE" ] && mains+=("$f") + done + else + # newest single session by mtime + newest=""; newest_m=-1 + for f in "${all_mains[@]}"; do + m=$(_mtime "$f"); [ -n "$m" ] || continue + if [ "$m" -gt "$newest_m" ]; then newest_m=$m; newest=$f; fi + done + [ -n "$newest" ] && mains+=("$newest") + fi +fi + +[ "${#mains[@]}" -gt 0 ] || { echo "fm-token-usage.sh: no session transcript in scope for $CWD" >&2; exit 4; } + +# For every in-scope session, add its subagents/*.jsonl (subagent usage lives +# there, not in the main jsonl). +files=() +for m in "${mains[@]}"; do + files+=("$m") + sid=$(basename "$m" .jsonl) + subdir="$PROJDIR/$sid/subagents" + if [ -d "$subdir" ]; then + while IFS= read -r sf; do files+=("$sf"); done < <(find "$subdir" -type f -name '*.jsonl' 2>/dev/null) + fi +done + +# Sum per-message usage across every in-scope file. A malformed or non-message +# line contributes nothing (guarded by `objects`). +read -r I O CC CR < <( + jq -n -r ' + reduce inputs as $l ( + {i:0,o:0,cc:0,cr:0}; + # `// {}` is load-bearing: a line without a usage object must yield {}, + # not empty. An empty result here would make this reduce step produce no + # output, which jq treats as resetting the accumulator to null - silently + # discarding every token counted so far (real transcripts interleave many + # non-usage lines, so this would zero out the total). + (($l.message | objects | .usage | objects) // {}) as $u + | { i:(.i + ($u.input_tokens // 0)), + o:(.o + ($u.output_tokens // 0)), + cc:(.cc + ($u.cache_creation_input_tokens // 0)), + cr:(.cr + ($u.cache_read_input_tokens // 0)) } + ) | "\(.i) \(.o) \(.cc) \(.cr)" + ' "${files[@]}" +) +TOTAL=$((I + O + CC + CR)) +NSESS=${#mains[@]} +NFILES=${#files[@]} + +if [ "$JSON" -eq 1 ]; then + printf '{"input":%d,"output":%d,"cache_create":%d,"cache_read":%d,"total":%d,"sessions":%d,"files":%d}\n' \ + "$I" "$O" "$CC" "$CR" "$TOTAL" "$NSESS" "$NFILES" +else + label=${ID:-$CWD} + printf 'token usage for %s (claude):\n' "$label" + printf ' input %12d\n' "$I" + printf ' output %12d\n' "$O" + printf ' cache-create %12d\n' "$CC" + printf ' cache-read %12d\n' "$CR" + printf ' TOTAL %12d (%d session(s), %d transcript file(s))\n' "$TOTAL" "$NSESS" "$NFILES" +fi diff --git a/docs/architecture.md b/docs/architecture.md index 4e6c3e6e667..a0545042564 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -263,7 +263,7 @@ Codex App support is recorded in `docs/codex-app-backend.md`; it is not selectab ## Worktrees, not branches in your checkout Crewmates never intentionally touch your project clone; [treehouse](https://github.com/kunchenguid/treehouse) pools clean worktrees for tmux, herdr, zellij, and cmux tasks, while Orca creates its own worktrees for `backend=orca`. -The [`fm-spawn.sh` header](../bin/fm-spawn.sh) owns ship/scout worktree isolation and fresh-base refusal rules, including spawns from linked homes. +The [`fm-spawn.sh` header](../bin/fm-spawn.sh) owns ship/scout worktree isolation and fresh-base refusal rules, including spawns from linked homes and the requirement that the resolved worktree belong to the spawning project rather than an unrelated repo the pane transiently sits in. Portable regressions live in [`tests/fm-spawn-pool-base-freshen.test.sh`](../tests/fm-spawn-pool-base-freshen.test.sh) for spawn isolation and base freshness, and [`tests/fm-control-relaunch.test.sh`](../tests/fm-control-relaunch.test.sh) for preserving the recorded copy on relaunch. The firstmate repo has one extra exposure because it can dispatch crewmates to work on itself. diff --git a/docs/configuration.md b/docs/configuration.md index 4803123e1bc..40135f4280d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1207,6 +1207,7 @@ FM_COMPOSER_CAPTURE_LINES=20 # fleet-wide bound for tail-capture composer read FM_COMPOSER_PI_MAX_LINES=8 # fleet-wide: maximum rows admitted between Pi's identity-corroborated separator pair; taller or ambiguous candidates stay unknown FM_COMPOSER_GHOST_LUMA_MAX=128 # fleet-wide: max perceived luminance (0.299R+0.587G+0.114B, 0-255) for a TRUECOLOR foreground to count as de-emphasised ghost/placeholder text and be stripped; dim/faint (SGR 2) is stripped regardless. Assumes a dark terminal theme (bin/fm-composer-lib.sh's fm_composer_strip_ghost, used by styled tmux, herdr, and Zellij reads) GROK_HOME= # optional Grok config home for firstmate's global grok turn-end hook; defaults to ~/.grok +FM_SPAWN_WORKTREE_TIMEOUT=60 # seconds fm-spawn's post-'treehouse get' poll waits for the pane to enter a worktree of the project before giving up FM_SEND_RETRIES=3 # fm-send typed-plane Enter-retry attempts after typing the line once; agy typed targets use a longer per-harness default owned by bin/fm-send.sh FM_SEND_SLEEP=0.4 # seconds between fm-send typed-plane submit checks FM_SEND_SETTLE=1 # seconds fm-send waits after a successful typed-plane submit; 0 disables diff --git a/docs/documentation-audiences.json b/docs/documentation-audiences.json index e459e95006a..0de223bcecc 100644 --- a/docs/documentation-audiences.json +++ b/docs/documentation-audiences.json @@ -432,6 +432,10 @@ "path": "docs/tmux-backend.md", "audience": "operator-current" }, + { + "path": "docs/token-usage.md", + "audience": "operator-current" + }, { "path": "docs/trace-context.md", "audience": "maintainer-architecture" diff --git a/docs/orca-backend.md b/docs/orca-backend.md index 000782a3536..810c49936bd 100644 --- a/docs/orca-backend.md +++ b/docs/orca-backend.md @@ -76,6 +76,7 @@ Reinstall the CLI and rerun; [`verification/runtime-backends.md`](verification/r - Orca exposes no stable CLI version or protocol marker, so readiness is the compatibility gate rather than a version floor. - Only the verified terminal-handle and worktree result fields are accepted; speculative response shapes are rejected. - Orca's worktree shape is unverified against the spawn-time Claude workspace-trust check in `bin/fm-claude-trust.sh`, which refuses any path that is not a linked git worktree sharing the project's git common dir, so a claude spawn on Orca fails loudly at that check rather than launching if Orca clones instead of linking. +- The same shape is a known unknown for every harness at `validate_spawn_worktree`, which requires an Orca-provided worktree to share the project's `--git-common-dir` (the full contract is stated once in `bin/fm-spawn.sh`'s header); that assumes `orca worktree create --repo id:` yields a linked git worktree of the registered repo, which is NOT smoke-proven. The probe that settles it is `git -C rev-parse --path-format=absolute --git-common-dir` compared against the same command run in the project directory; if Orca provisions from its own clone or mirror, the correct fix is to give the Orca path its own identity rule rather than to drop the check. The fake-Orca tests feed a hand-built linked worktree, so they pin the predicate's accept and refuse behaviour but cannot confirm the shape real Orca provisions. ## Regression entry points diff --git a/docs/token-usage.md b/docs/token-usage.md new file mode 100644 index 00000000000..db38417699c --- /dev/null +++ b/docs/token-usage.md @@ -0,0 +1,74 @@ +# Crewmate token accounting (`bin/fm-token-usage.sh`) + +Precise per-crewmate token usage for `claude`-harness crewmates and scouts, read +from claude's own session transcripts. +The claude pane shows only per-subagent counters, never a session total, so +reading it undercounts; the transcripts are authoritative and survive teardown. + +## Data source (verified 2026-07-07, Claude Code 2.1.x) + +Claude Code writes one directory per working directory under +`~/.claude/projects/`, named by replacing every `/` and `.` in the absolute cwd +with `-`. +For example the worktree `/Users/me/.treehouse/repo-abc/1/repo` becomes +`-Users-me--treehouse-repo-abc-1-repo`. + +Inside that directory: + +``` +.jsonl the main agent's transcript +/subagents/agent-*.jsonl one transcript per Task subagent +/tool-results/ large tool outputs (no usage data) +``` + +Each assistant message line carries `.message.usage` with `input_tokens`, +`output_tokens`, `cache_creation_input_tokens`, and `cache_read_input_tokens`. +Non-assistant lines (user turns, summaries, string-valued `message`) carry no +usage and contribute nothing. + +**Subagent usage lives only in `subagents/`.** +The main jsonl's lines do NOT carry the subagents' usage (no `isSidechain` +usage rows appear there), so a sum of only the main transcript undercounts by the +entire subagent fan-out. +`fm-token-usage.sh` sums the main transcript plus every `subagents/*.jsonl` for +each in-scope session. + +## Evidence + +Three crewmates run on 2026-07-07 (discogs-rn-app MAPP tickets), each dispatching +6-7 subagents, summed by the tool (`--all`) and cross-checked against an +independent Python parse of the same files: + +| Task | files | input | output | cache-create | cache-read | total | +| --- | --- | --- | --- | --- | --- | --- | +| MAPP-3307 | 1 main + 7 sub | 64,927 | 204,768 | 1,271,817 | 17,172,801 | 18,714,313 | +| MAPP-3315 | 1 main + 6 sub | 74,322 | 253,939 | 1,266,306 | 28,667,638 | 30,262,205 | +| MAPP-3329 | 1 main + 6 sub | 134,292 | 230,444 | 2,823,366 | 39,496,130 | 42,684,232 | + +Summing only the main transcript (the pre-`subagents/` bug) reported 34,801 for +MAPP-3307 instead of 18,714,313; the regression test +`tests/fm-token-usage.test.sh` interleaves non-usage lines between usage lines to +pin the jq accumulator against silently resetting on them. + +## Session scoping + +A treehouse pool slot is reused across tasks, so one cwd can accumulate several +sessions over time. +The tool scopes by: + +- default: the newest session (by main-jsonl mtime) plus its subagents - correct + at a crewmate's done, when the newest session is that task's; +- `--session `: exactly that session; +- `--since `: every session whose main jsonl is at/after the epoch; +- `--all`: every session recorded for the cwd. + +## Usage + +``` +fm-token-usage.sh [--json] # reads worktree=/harness= from state/.meta +fm-token-usage.sh --cwd [--all|--since E|--session ID] [--json] +``` + +Only `harness=claude` is supported; other adapters record usage differently and +exit 3. +`FM_CLAUDE_PROJECTS_DIR` overrides `~/.claude/projects` (used by the tests). diff --git a/tests/fm-backend-orca.test.sh b/tests/fm-backend-orca.test.sh index a62043a76f0..55c0594f80f 100755 --- a/tests/fm-backend-orca.test.sh +++ b/tests/fm-backend-orca.test.sh @@ -659,6 +659,41 @@ test_spawn_refuses_orca_nonisolated_worktree() { pass "fm-spawn.sh --backend orca: refuses non-isolated worktrees and closes implicit terminals" } +test_spawn_refuses_orca_foreign_repo_worktree() { + local proj foreign data state config id out status + id="orcaforeignz5" + proj="$TMP_ROOT/foreign-spawn-project" + foreign="$TMP_ROOT/foreign-spawn-other-repo" + data="$TMP_ROOT/foreign-spawn-data" + state="$TMP_ROOT/foreign-spawn-state" + config="$TMP_ROOT/foreign-spawn-config" + fm_git_init_commit "$proj" + # A separate repository: its own root, not the primary checkout. Under the old + # rule ("a git repo whose root is itself and is not the primary") this passed. + fm_git_init_commit "$foreign" + mkdir -p "$data/$id" "$state" "$config" + write_spawn_brief "$data" "$id" + touch "$state/.last-watcher-beat" + orca_case foreign-spawn + printf '1\n' > "$RESP/1.exit" + printf '{"ok":true,"result":{"repo":{"id":"repo-foreign"}}}\n' > "$RESP/2.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-foreign","path":"%s"},"terminal":{"handle":"term-foreign"}}}\n' "$foreign" > "$RESP/3.out" + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --mode no-mistakes --yolo off --backend orca 2>&1 ) + status=$? + expect_code 1 "$status" "fm-spawn.sh --backend orca should refuse a worktree from another repository" + assert_contains "$out" "NOT a worktree of" \ + "Orca spawn should name the foreign-repository refusal" + assert_absent "$state/$id.meta" "aborted Orca spawn must not record meta" + assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-foreign'$'\x1f''--json' \ + "Orca spawn should close the implicit terminal after validation aborts" + assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-foreign'$'\x1f''--force'$'\x1f''--json' \ + "Orca spawn should remove the foreign worktree instead of proceeding to launch in it" + pass "fm-spawn.sh --backend orca: refuses a worktree belonging to a different repository" +} + test_spawn_removes_orca_worktree_when_terminal_create_fails() { local proj wt data state config id out status id="orcatermfailz8" @@ -1380,6 +1415,7 @@ test_spawn_writes_orca_metadata_and_launches_harness test_spawn_refuses_orca_secondmate_before_home_mutation test_spawn_refuses_orca_when_runtime_not_ready test_spawn_refuses_orca_nonisolated_worktree +test_spawn_refuses_orca_foreign_repo_worktree test_spawn_removes_orca_worktree_when_terminal_create_fails test_spawn_preserves_orca_metadata_when_abort_cleanup_fails test_spawn_releases_orca_resources_when_metadata_write_fails diff --git a/tests/fm-no-mistakes-required.test.sh b/tests/fm-no-mistakes-required.test.sh index 4807371289a..07b067615cc 100755 --- a/tests/fm-no-mistakes-required.test.sh +++ b/tests/fm-no-mistakes-required.test.sh @@ -1,11 +1,17 @@ #!/usr/bin/env bash # Regression tests for the pinned shared no-mistakes gate action. +# +# The ref is read out of .github/workflows/no-mistakes-required.yml rather than +# repeated here, so these tests always exercise the verifier the required check +# actually runs. A second copy of the pin drifts silently: rolling the workflow +# to v1.80.1 left this script fetching the previous action, so the tests kept +# passing against a verifier no PR was ever graded by. set -u # shellcheck source=tests/lib.sh disable=SC1091 . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" -ACTION_REF=32d396ac0f29135daf7fcb9964aba9d5f4e796d6 +GATE_WORKFLOW="$ROOT/.github/workflows/no-mistakes-required.yml" TMP_ROOT=$(fm_test_tmproot fm-no-mistakes-required) VERIFY="$TMP_ROOT/verify.py" OLD_SHA=1111111111111111111111111111111111111111 @@ -13,9 +19,28 @@ NEW_SHA=2222222222222222222222222222222222222222 SIGNATURE='Updates from [git push no-mistakes](https://github.com/kunchenguid/no-mistakes)' COMPLETED_STEPS='[{"step":"review","status":"completed"},{"step":"test","status":"completed"},{"step":"document","status":"completed"}]' +# Echo the immutable ref the required check is pinned to, resolved from the +# workflow's parsed step list rather than from how the file happens to be typed. +resolve_pinned_action_ref() { + ruby -ryaml -e ' +doc = YAML.load_file(ARGV[0]) +uses = doc.fetch("jobs").values.flat_map { |job| job.fetch("steps", []) } + .map { |step| step["uses"] }.compact + .select { |u| u.start_with?("kunchenguid/no-mistakes/.github/actions/require-no-mistakes@") } +abort "expected exactly one require-no-mistakes step, found #{uses.length}" unless uses.length == 1 +ref = uses.first.split("@", 2).last +abort "require-no-mistakes must be pinned to a full commit SHA, got #{ref}" unless ref =~ /\A[0-9a-f]{40}\z/ +puts ref +' "$GATE_WORKFLOW" +} + fetch_shared_verifier() { command -v curl >/dev/null 2>&1 || fail "curl is required to exercise the pinned shared action" command -v python3 >/dev/null 2>&1 || fail "python3 is required to exercise the pinned shared action" + command -v ruby >/dev/null 2>&1 || fail "ruby is required to parse the required-check workflow as YAML" + assert_present "$GATE_WORKFLOW" ".github/workflows/no-mistakes-required.yml is missing" + ACTION_REF=$(resolve_pinned_action_ref) \ + || fail "could not resolve the pinned require-no-mistakes action ref" curl --fail --silent --show-error --location \ "https://raw.githubusercontent.com/kunchenguid/no-mistakes/${ACTION_REF}/.github/actions/require-no-mistakes/verify.py" \ > "$VERIFY" || fail "could not fetch the pinned shared action verifier" diff --git a/tests/fm-tangle-guard.test.sh b/tests/fm-tangle-guard.test.sh index 6f80e3078e0..04075d6171b 100755 --- a/tests/fm-tangle-guard.test.sh +++ b/tests/fm-tangle-guard.test.sh @@ -7,12 +7,14 @@ # is a crewmate branching/committing in the primary instead of its own worktree, # stranding the primary on a feature branch. Two guards cover it: # GUARD 1 (prevention) - the brief asserts isolation before its branch step, and -# fm-spawn refuses to launch unless the resolved worktree is isolated. +# fm-spawn refuses to launch unless the resolved worktree is isolated +# AND a worktree of the project being spawned into. # GUARD 2 (detection) - fm-guard and fm-bootstrap alarm when the primary is on # a feature branch, and stay silent on the default branch or detached. # These cases pin: the shared lib's branch classification, the fm-guard banner, -# the fm-bootstrap problem line, the brief assertion ordering, and the fm-spawn -# abort - all hermetic over temp git repos and fakebins. +# the fm-bootstrap problem line, the brief assertion ordering, the fm-spawn +# abort, and the worktree-discovery poll's refusal to latch onto an unrelated +# git repo - all hermetic over temp git repos and fakebins. set -u # shellcheck source=tests/fixtures.sh @@ -159,6 +161,58 @@ run_spawn() { "$id" "$proj" codex --mode no-mistakes --yolo off } +# A sequence-driven variant of the shared spawn fakebin: FM_FAKE_PANE_PATH_SEQ +# names a file of one pane cwd per line, consumed one per poll, with the last +# line repeating forever after. That is what reproduces the real shape of the +# oh-my-zsh bug, where the pane sits somewhere else for the first polls and only +# then lands in the task worktree. Everything else matches the shared stub. +make_spawn_seq_fakebin() { + local dir=$1 fakebin + fakebin=$(make_spawn_fakebin "$dir") + cat > "$fakebin/tmux" <<'SH' +#!/usr/bin/env bash +set -u +case "$*" in + *"#{pane_current_path}"*) + if [ -n "${FM_FAKE_PANE_PATH_SEQ:-}" ]; then + # One line per poll. A sibling .n file carries the cursor across the + # separate fake-tmux processes the poll loop spawns. + n_file="$FM_FAKE_PANE_PATH_SEQ.n" + n=$(cat "$n_file" 2>/dev/null || echo 1) + total=$(wc -l < "$FM_FAKE_PANE_PATH_SEQ") + [ "$n" -le "$total" ] || n=$total + sed -n "${n}p" "$FM_FAKE_PANE_PATH_SEQ" + echo $((n + 1)) > "$n_file" + exit 0 + fi + printf '%s\n' "${FM_FAKE_PANE_PATH:-}" + exit 0 + ;; +esac +case "${1:-}" in + display-message) printf 'firstmate\n'; exit 0 ;; + list-windows) exit 0 ;; + has-session|new-session|new-window|kill-window|set-window-option|send-keys) exit 0 ;; +esac +exit 0 +SH + chmod +x "$fakebin/tmux" + printf '%s\n' "$fakebin" +} + +# run_spawn_seq +# Same as run_spawn, but the pane reports one path per poll from . +# FM_SPAWN_WORKTREE_TIMEOUT bounds the poll so a sequence that never settles +# fails in a few polls instead of the 60-poll production bound. +run_spawn_seq() { + local home=$1 id=$2 proj=$3 seq=$4 fakebin=$5 + fm_test_spawn_brief "$home" "$id" brief + rm -f "$seq.n" + FM_FAKE_PANE_PATH_SEQ="$seq" FM_SPAWN_WORKTREE_TIMEOUT="${FM_TEST_SPAWN_TIMEOUT:-8}" \ + fm_test_run_spawn "$home" '' "$fakebin" \ + "$id" "$proj" codex --mode no-mistakes --yolo off +} + test_spawn_isolation_abort() { local home proj fakebin out status home="$TMP_ROOT/spawn-home" @@ -191,6 +245,8 @@ test_spawn_isolation_abort() { assert_absent "$home/state/abort-notgit-dd4.meta" "aborted spawn must not record meta" # Abort: the pane resolves INTO the primary checkout (a subdir of PROJ_ABS). + # This one DOES share the project's git dir, so the poll accepts it and the + # isolation rule is what refuses it - the two checks are independent. out=$(run_spawn "$home" abort-primary-ee5 "$proj" "$proj/sub" "$fakebin"); status=$? expect_code 1 "$status" "spawn landing inside the primary checkout should abort" assert_contains "$out" "did not enter an isolated worktree" "primary-checkout spawn lacked the isolation error" @@ -287,9 +343,107 @@ test_spawn_tmux_window_construction() { pass "fm-spawn: appends windows by session-colon, pins the name, and targets the window id" } +# --- GUARD 1d: the discovery poll must not latch onto a foreign repo -------- + +# Regression for the oh-my-zsh spawn incident. oh-my-zsh.sh runs +# `builtin cd -q "$ZSH"` on EVERY shell startup to stamp the zcompdump +# revision, so a freshly spawned pane transiently reports ~/.oh-my-zsh as its +# foreground cwd. The poll used to accept the first path that merely DIFFERED +# from the project, and the isolation rule used to ask only "a git repo whose +# root is itself, and not the primary" - which ~/.oh-my-zsh satisfies. Five +# agents launched in the user's shell framework directory before it was +# diagnosed. The property that actually identifies a task worktree is a shared +# --git-common-dir with the project, so the standing behaviour is: a foreign +# repo is never accepted, and the poll keeps waiting for the real worktree. +test_spawn_rejects_foreign_repo_cwd() { + local home proj fakebin foreign wt seq out status + home="$TMP_ROOT/foreign-home" + mkdir -p "$home/data" + proj=$(make_repo "$TMP_ROOT/foreign-proj") + fakebin=$(make_spawn_seq_fakebin "$TMP_ROOT/foreign-fake") + # The assertions concern identity, not how long an unchanged cwd is polled. + fm_test_fake_sleep_noop "$fakebin" + # Stands in for ~/.oh-my-zsh: a real git repo, its own root, not the primary. + foreign=$(make_repo "$TMP_ROOT/foreign-omz") + wt="$TMP_ROOT/foreign-wt" + git -C "$proj" worktree add -q --detach "$wt" >/dev/null 2>&1 + + # Never launch into a git repo that is not this project's, however long it + # sits there. Without the common-dir check this spawn SUCCEEDS into $foreign. + out=$(run_spawn "$home" abort-foreign-gg7 "$proj" "$foreign" "$fakebin"); status=$? + expect_code 1 "$status" "spawn into an unrelated git repo should abort" + assert_contains "$out" "did not enter an isolated worktree" "foreign-repo spawn lacked the resolution error" + assert_contains "$out" "NOT a worktree of the spawning project" "foreign-repo spawn did not say why the path was rejected" + assert_contains "$out" "$foreign" "error should name the impostor path that was seen" + assert_absent "$home/state/abort-foreign-gg7.meta" "aborted spawn must not record meta" + + # The live shape: the pane sits in the foreign repo for the first polls, then + # treehouse lands it in the real worktree. The poll must wait that out and + # resolve the worktree, not latch onto the first different path it saw. The + # last line repeats, which also satisfies the poll's two-consecutive-reads rule. + seq="$TMP_ROOT/foreign-seq" + printf '%s\n%s\n%s\n' "$foreign" "$foreign" "$wt" > "$seq" + out=$(run_spawn_seq "$home" ok-waited-hh8 "$proj" "$seq" "$fakebin"); status=$? + expect_code 0 "$status" "spawn should succeed once the pane reaches the real worktree" + assert_present "$home/state/ok-waited-hh8.meta" "successful spawn must record meta" + assert_grep "worktree=$wt" "$home/state/ok-waited-hh8.meta" \ + "spawn resolved the wrong worktree; it must wait past the impostor cwd" + assert_no_grep "worktree=$foreign" "$home/state/ok-waited-hh8.meta" \ + "spawn latched onto the impostor repo instead of the task worktree" + pass "fm-spawn: the worktree poll waits past an unrelated git repo instead of launching in it" +} + +# --- GUARD 1e: the fail-open branch ----------------------------------------- + +# The identity predicate deliberately fails OPEN: when the project's own +# --git-common-dir cannot be resolved it never blocks a spawn by itself, and +# the operator is told on stderr that the check is off. On this base the +# isolation guard's own unresolved-git-dir rule (spawn_worktree_isolated) then +# refuses every candidate, genuine worktree or not, so the spawn does not +# launch; that refusal is the isolation guard's, not the identity check's. +# Both halves of the identity contract are pinned here: the warning fires, and +# the reason reported is never the identity predicate's. +test_spawn_fail_open_when_common_dir_unknown() { + local home proj fakebin foreign real_git out status + home="$TMP_ROOT/failopen-home" + mkdir -p "$home/data" + proj=$(make_repo "$TMP_ROOT/failopen-proj") + fakebin=$(make_spawn_fakebin "$TMP_ROOT/failopen-fake") + fm_test_fake_sleep_noop "$fakebin" + foreign=$(make_repo "$TMP_ROOT/failopen-omz") + real_git=$(command -v git) + + # Shadow git so only `rev-parse --path-format=...` fails, exactly as a git + # older than 2.31 would; every other invocation passes through to real git. + cat > "$fakebin/git" <&2; exit 129 ;; + esac +done +exec "$real_git" "\$@" +SH + chmod +x "$fakebin/git" + + out=$(run_spawn "$home" failopen-jj9 "$proj" "$foreign" "$fakebin"); status=$? + assert_contains "$out" "worktree-identity check is DISABLED" \ + "fail-open spawn did not warn that the identity check is off" + assert_not_contains "$out" "NOT a worktree of the spawning project" \ + "the identity predicate must not block when the project's common dir is unknown" + expect_code 1 "$status" "the isolation guard's own unresolved-git-dir rule still refuses the spawn" + assert_contains "$out" "its git directory could not be resolved" \ + "the refusal must come from the isolation guard's unresolved-git-dir rule" + assert_absent "$home/state/failopen-jj9.meta" "refused spawn must not record meta" + pass "fm-spawn: an unresolvable project git common dir disables the identity check loudly and never blocks by itself" +} + test_lib_classification test_guard_banner test_bootstrap_line test_brief_assertion_precedes_branch test_spawn_isolation_abort test_spawn_tmux_window_construction +test_spawn_rejects_foreign_repo_cwd +test_spawn_fail_open_when_common_dir_unknown diff --git a/tests/fm-teardown.test.sh b/tests/fm-teardown.test.sh index 7b2a86df631..9840764e39e 100755 --- a/tests/fm-teardown.test.sh +++ b/tests/fm-teardown.test.sh @@ -3922,3 +3922,77 @@ test_process_spawned_during_grace_is_reaped_on_later_pass test_persistent_scan_refuses_after_bounded_retries test_process_exit_during_identity_lookup_does_not_refuse test_run_abort_precedes_process_reap_precedes_worktree_removal + +# --- teardown harvest of crew-generated untracked files --------------------- +# On a ship teardown, every untracked non-ignored file the crew left is copied +# into the project's primary checkout (never overwriting) and removed from the +# worktree, so leftover untracked files no longer refuse teardown and generated +# work survives the hard-reset. Scout/secondmate worktrees are exempt. + +test_harvest_copies_untracked_into_project() { + local case_dir rc + case_dir=$(make_case harvest-copy) + write_meta "$case_dir" no-mistakes ship + wt_commit "$case_dir" "shippable work" + git -C "$case_dir/wt" push -q origin fm/task-x1 + git -C "$case_dir/project" fetch -q origin + printf 'notes\n' > "$case_dir/wt/NOTES.md" + mkdir -p "$case_dir/wt/e2e/flows" + printf 'flow\n' > "$case_dir/wt/e2e/flows/a.yaml" + + set +e + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "harvest-copy: teardown should succeed (untracked harvested, work landed)" + ! grep -q REFUSED "$case_dir/stderr" || fail "harvest-copy: teardown printed REFUSED" + assert_present "$case_dir/project/NOTES.md" "harvest-copy: NOTES.md should be copied into project" + assert_present "$case_dir/project/e2e/flows/a.yaml" "harvest-copy: nested file should be copied into project" + assert_absent "$case_dir/wt/NOTES.md" "harvest-copy: source should be removed from worktree" + grep -q "harvest: kept NOTES.md" "$case_dir/stderr" || fail "harvest-copy: should log kept NOTES.md" + pass "harvest copies untracked crew files into project and allows an otherwise-dirty teardown" +} + +test_harvest_no_clobber_preserves_existing() { + local case_dir rc + case_dir=$(make_case harvest-clobber) + write_meta "$case_dir" no-mistakes ship + wt_commit "$case_dir" "shippable work" + git -C "$case_dir/wt" push -q origin fm/task-x1 + git -C "$case_dir/project" fetch -q origin + printf 'ORIGINAL\n' > "$case_dir/project/KEEP.md" # pre-existing at destination + printf 'NEW\n' > "$case_dir/wt/KEEP.md" # crew's untracked version + + set +e + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "harvest-clobber: teardown should succeed" + [ "$(cat "$case_dir/project/KEEP.md")" = "ORIGINAL" ] \ + || fail "harvest-clobber: existing project file must NOT be overwritten" + grep -q "harvest: skip (already at destination) KEEP.md" "$case_dir/stderr" \ + || fail "harvest-clobber: should log the no-clobber skip" + pass "harvest never overwrites an existing project file (no-clobber)" +} + +test_harvest_skipped_for_scout() { + local case_dir rc + case_dir=$(make_case harvest-scout) + write_meta "$case_dir" no-mistakes scout + printf 'scratch\n' > "$case_dir/wt/SCRATCH.md" + + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "harvest-scout: scout teardown should succeed with --force" + assert_absent "$case_dir/project/SCRATCH.md" "harvest-scout: scout scratch must NOT be harvested" + pass "scout teardown does not harvest (scratch-worktree exemption)" +} + +test_harvest_copies_untracked_into_project +test_harvest_no_clobber_preserves_existing +test_harvest_skipped_for_scout diff --git a/tests/fm-token-usage.test.sh b/tests/fm-token-usage.test.sh new file mode 100755 index 00000000000..e6a83164c19 --- /dev/null +++ b/tests/fm-token-usage.test.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# Behavior tests for bin/fm-token-usage.sh - the claude token-accounting helper. +# +# The helper sums per-message token usage from claude's own session transcripts, +# which live at: +# $FM_CLAUDE_PROJECTS_DIR//.jsonl main agent +# $FM_CLAUDE_PROJECTS_DIR///subagents/*.jsonl subagents +# where is the absolute cwd with every '/' and '.' turned into '-'. +# These cases pin the contract hermetically over synthetic transcripts: +# (a) task-id mode sums main + subagents from state/.meta's worktree +# (b) non-claude harness is refused with exit 3 +# (c) --json emits the machine object +# (d) default scoping counts only the NEWEST session; --all counts every one +# (e) --cwd mode works without a task/meta +# (f) a cwd with no transcripts exits 4 +# (g) malformed / non-usage lines contribute nothing (guarded) +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +TOOL="$ROOT/bin/fm-token-usage.sh" +command -v jq >/dev/null 2>&1 || { echo "1..0 # SKIP jq not installed"; exit 0; } + +TMP=$(fm_test_tmproot fm-token-usage) +HOME_DIR="$TMP/home" +PROJECTS="$TMP/projects" +mkdir -p "$HOME_DIR/state" "$PROJECTS" + +# A synthetic crewmate worktree path (need not exist; only its encoding matters). +CWD="$TMP/wt/discogs-rn-app" +enc=$(printf '%s' "$CWD" | tr './' '-') +PROJDIR="$PROJECTS/$enc" + +usage_line() { # + printf '{"type":"assistant","message":{"role":"assistant","usage":{"input_tokens":%d,"output_tokens":%d,"cache_creation_input_tokens":%d,"cache_read_input_tokens":%d}}}\n' "$1" "$2" "$3" "$4" +} + +# --- build a session: main jsonl (2 usage lines + guard lines) + 1 subagent --- +mkdir -p "$PROJDIR/sess-main/subagents" +{ + echo '{"type":"summary","summary":"no message here"}' # guard: no .message + usage_line 10 20 5 100 + echo '{"message":"plain string message"}' # guard: .message is a string, INTERLEAVED + echo '{"type":"user","message":{"role":"user","content":"hi"}}' # guard: message but no usage, INTERLEAVED + usage_line 1 2 0 50 +} > "$PROJDIR/sess-main.jsonl" +usage_line 3 4 7 200 > "$PROJDIR/sess-main/subagents/agent-x.jsonl" +# main: in=11 out=22 cc=5 cr=150 ; sub: in=3 out=4 cc=7 cr=200 +# totals: in=14 out=26 cc=12 cr=350 -> grand=402 + +fm_write_meta "$HOME_DIR/state/tok-a.meta" "worktree=$CWD" "harness=claude" "kind=ship" + +export FM_HOME="$HOME_DIR" +export FM_CLAUDE_PROJECTS_DIR="$PROJECTS" + +# (a) task-id mode sums main + subagents +out=$("$TOOL" tok-a); code=$? +expect_code 0 "$code" "task-id mode should succeed" +assert_contains "$out" "input 14" "(a) input total wrong" +assert_contains "$out" "output 26" "(a) output total wrong" +assert_contains "$out" "cache-create 12" "(a) cache-create total wrong" +assert_contains "$out" "cache-read 350" "(a) cache-read total wrong" +assert_contains "$out" "TOTAL 402" "(a) grand total wrong" +pass "(a) task-id mode sums main + subagents" + +# (b) non-claude harness refused +fm_write_meta "$HOME_DIR/state/tok-codex.meta" "worktree=$CWD" "harness=codex" "kind=ship" +out=$("$TOOL" tok-codex 2>&1); code=$? +expect_code 3 "$code" "(b) non-claude harness should exit 3" +assert_contains "$out" "only supported for claude" "(b) should explain claude-only" +pass "(b) non-claude harness refused with exit 3" + +# (c) --json emits the machine object +out=$("$TOOL" tok-a --json); code=$? +expect_code 0 "$code" "(c) --json should succeed" +assert_contains "$out" '"total":402' "(c) json total wrong" +assert_contains "$out" '"input":14' "(c) json input wrong" +assert_contains "$out" '"sessions":1' "(c) json sessions wrong" +pass "(c) --json emits machine object" + +# (d) default counts newest session only; --all counts every session +mkdir -p "$PROJDIR/sess-old/subagents" +usage_line 1000 2000 3000 4000 > "$PROJDIR/sess-old.jsonl" +touch -t 202001010000 "$PROJDIR/sess-old.jsonl" +touch -t 202601010000 "$PROJDIR/sess-main.jsonl" +out=$("$TOOL" --cwd "$CWD" --json); code=$? +expect_code 0 "$code" "(d) default cwd mode should succeed" +assert_contains "$out" '"total":402' "(d) default should count only newest session (402), not the old one" +out=$("$TOOL" --cwd "$CWD" --all --json); code=$? +expect_code 0 "$code" "(d) --all should succeed" +# --all adds old session: +1000/2000/3000/4000 = +10000 -> 10402 +assert_contains "$out" '"total":10402' "(d) --all should sum every session" +assert_contains "$out" '"sessions":2' "(d) --all should report 2 sessions" +pass "(d) newest-session default vs --all" + +# (e) --cwd mode without a task/meta already exercised above; confirm --session +out=$("$TOOL" --cwd "$CWD" --session sess-old --json); code=$? +expect_code 0 "$code" "(e) --session should succeed" +assert_contains "$out" '"total":10000' "(e) --session should scope to just that session" +pass "(e) --cwd / --session scoping" + +# (f) no transcripts for a cwd -> exit 4 +out=$("$TOOL" --cwd "$TMP/wt/nonexistent" 2>&1); code=$? +expect_code 4 "$code" "(f) missing session data should exit 4" +pass "(f) missing session data exits 4" + +# (g) guard lines already interleaved in sess-main proved they contribute 0 in (a). +pass "(g) malformed / non-usage lines contribute nothing" + +echo "1..7"