From aa434629008c93f1d1301b627d9ef0e29769b6fc Mon Sep 17 00:00:00 2001 From: Lloyd Date: Wed, 2 Sep 2026 22:55:23 +0200 Subject: [PATCH 01/61] fix(session-lock): identify a harness session on Windows/Git Bash Every session start on Cygwin (Git for Windows) refused the fleet lock and dropped to read-only with "cannot locate harness process in ancestry", so spawning, steering, merging, the wake-queue drain, and supervision repair were skipped on every start. Two independent causes, both on the identity path. Cygwin's ps has no -o option at all and fails the whole invocation with "unknown option -- o", so the ancestry walk aborted on its first hop. The walk now reads comm, args, and ppid through accessors that fall back to Cygwin's fixed ps columns, leaving the procps/BSD path unchanged. That alone does not resolve the session: the parent link from a shell the harness spawns does not cross the Cygwin boundary, and Cygwin reports that shell's PPID as 1, so no walk can reach a harness that is a native Windows process. Identity is instead taken from the session pid the harness publishes and confirmed against the Windows process table before it is used - the pid must still be live and its executable must independently identify a verified harness - so an absent, stale, or non-harness value is discarded rather than bound. Walking the real Windows parent chain was implemented and then removed as unsafe. MSYS emulates exec by spawning a fresh Windows process and exiting the old one, so intermediate shells vanish and a child's recorded parent is routinely a pid that no longer exists; Windows never reparents an orphan, so that dangling id stays and can be reissued to an unrelated process. Following it can bind a home's lock to the wrong process, which is the failure this file exists to prevent. A harness that publishes nothing stays unresolved, which leaves the session read-only exactly as before. Windows pids are tagged rather than stored bare. They are a different namespace: kill -0 reports a live Windows process as dead, and the number can collide with an unrelated live Cygwin pid. The tag makes the value non-numeric, so a consumer that treats it as a local pid - including a future kill - refuses it instead of acting on the wrong process. fm-sessionstart-nudge.sh carried a private second copy of the ownership walk and so stayed wrong after the owner was fixed, nudging a session that already held the lock. It now asks the owning function. Verified on Windows 11 (Git Bash, Cygwin ps 3.4.10): the lock is acquired, reports its holder, is idempotent, and refuses a bogus, dead, or non-harness published pid. Regressions cover both platform departures behind a fake process table, so they run on Linux and macOS CI too. The pre-existing e2e failure in this suite on Windows is unchanged from main; it cannot exec its symlinked fixture. Refs #3396 Claude-Session: https://claude.ai/code/session_01F9T29YDqYSkQeDTC2Nfi7h (cherry picked from commit 8b281b7b5507c342ac7870e8ccbefa56c55d05eb) --- bin/fm-lock.sh | 12 +- bin/fm-session-lock-lib.sh | 163 ++++++++++++++++++++++++- bin/fm-sessionstart-nudge.sh | 34 ++---- tests/fm-session-lock-ancestry.test.sh | 137 +++++++++++++++++++++ 4 files changed, 313 insertions(+), 33 deletions(-) diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index 52d7c8aee4b..6058afe2334 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -33,7 +33,17 @@ if [ "${1:-}" = "status" ]; then exit 0 fi -me=$(fm_harness_ancestry_pid) || { echo "error: cannot locate harness process in ancestry" >&2; exit 1; } +me=$(fm_harness_ancestry_pid) || { + if fm_win_boundary_applies; then + # Here the parent link does not reach the harness at all, so "not in the + # ancestry" would describe the wrong problem and send the reader hunting a + # process tree that can never contain the answer. + echo "error: cannot identify this harness session on Windows: it publishes no session pid this build recognizes (see FM_WIN_HARNESS_PID_VARS in bin/fm-session-lock-lib.sh); operate read-only until resolved" >&2 + else + echo "error: cannot locate harness process in ancestry" >&2 + fi + exit 1 +} probe=$(mktemp "$STATE/.lock-write.XXXXXX" 2>/dev/null) || { echo "error: cannot write session lock; operate read-only until resolved" >&2 exit 1 diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index 91c901f820b..86653f62617 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -47,6 +47,41 @@ fm_harness_path_name() { # return 1 } +# Portable single-pid process introspection. +# +# procps/BSD `ps -o comm=/args=/ppid=` custom-format columns are the primary +# path and cover Linux and macOS. Cygwin's ps, which Git for Windows ships, has +# no -o option at all and exits with "unknown option -- o", so the primary path +# fails outright there and the ancestry walk aborts on its first hop. The +# fallbacks parse Cygwin's fixed columns instead: +# ps -p PID PID PPID PGID WINPID TTY UID STIME COMMAND (executable path) +# ps -f -p PID UID PID PPID TTY STIME COMMAND (full argv) +# A genuinely dead or inaccessible pid still fails both paths, so this widens +# nothing: it only stops a supported platform from failing on ps syntax alone. +fm_ps_comm() { # -> executable name or path + local pid=$1 out + out=$(ps -o comm= -p "$pid" 2>/dev/null) && [ -n "$out" ] && { printf '%s' "$out"; return 0; } + out=$(ps -p "$pid" 2>/dev/null | awk 'NR == 2 { for (i = 8; i <= NF; i++) printf "%s%s", (i > 8 ? " " : ""), $i; exit }') + [ -n "$out" ] || return 1 + printf '%s' "$out" +} + +fm_ps_args() { # -> full command line + local pid=$1 out + out=$(ps -o args= -p "$pid" 2>/dev/null) && [ -n "$out" ] && { printf '%s' "$out"; return 0; } + out=$(ps -f -p "$pid" 2>/dev/null | awk 'NR == 2 { for (i = 6; i <= NF; i++) printf "%s%s", (i > 6 ? " " : ""), $i; exit }') + [ -n "$out" ] || return 1 + printf '%s' "$out" +} + +fm_ps_ppid() { # -> parent pid + local pid=$1 out + out=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d ' ') && [ -n "$out" ] && { printf '%s' "$out"; return 0; } + out=$(ps -f -p "$pid" 2>/dev/null | awk 'NR == 2 { print $3; exit }') + [ -n "$out" ] || return 1 + printf '%s' "$out" +} + # True when the process described by command name $1 and full argument string $2 # is a verified harness. Sets FM_HARNESS_IS_CLAUDE for the ancestry walk. # @@ -90,6 +125,105 @@ fm_harness_process_matches() { # return 1 } +# --- Windows process-boundary bridge ----------------------------------------- +# +# On Cygwin (Git for Windows) the harness is a native Windows process, and the +# parent link from a shell it spawns does NOT cross the Cygwin boundary: Cygwin +# reports that shell's PPID as 1. So no amount of walking ppid can ever reach the +# harness, and the contiguous-run model below cannot be satisfied by the Cygwin +# process table alone. The Windows process table does hold the real parent chain, +# and `ps -W` lists Windows processes keyed by WINPID, so identity is recovered +# from there instead. +# +# Windows pids live in a DIFFERENT namespace from Cygwin pids: `kill -0` on a +# Windows pid reports "No such process" even while that process is running, and a +# Windows pid can collide with an unrelated live Cygwin pid. A bare number is +# therefore ambiguous and unsafe to store. Every pid resolved through this bridge +# is tagged, which makes the namespace explicit for readers and makes the value +# non-numeric so that any consumer treating it as a Cygwin pid - including a +# future `kill` - refuses it instead of acting on the wrong process. +FM_WIN_PID_PREFIX='win:' + +# True on a Cygwin-family userspace, where the boundary above applies. +fm_win_boundary_applies() { + case "$(uname -s 2>/dev/null)" in + CYGWIN*|MINGW*|MSYS*) return 0 ;; + esac + return 1 +} + +# Strip the namespace tag from $1, or return 1 when $1 is not a tagged pid. +fm_win_untag_pid() { # + case "$1" in + "$FM_WIN_PID_PREFIX"[0-9]*) printf '%s' "${1#"$FM_WIN_PID_PREFIX"}"; return 0 ;; + esac + return 1 +} + +# Windows command paths are backslash-separated and .exe-suffixed, neither of +# which the path-component matcher above understands. Normalizing here is what +# keeps that matcher's whole-component safety intact: without it the harness +# regex would fall back to matching the entire unsplit path, so an unrelated +# C:\claude-notes\tool.exe would read as a harness process. +fm_win_normalize_command() { # + local path=${1//\\//} + printf '%s' "${path%.exe}" +} + +# Print the normalized executable path of live Windows process $1, or return 1. +# Presence in `ps -W` is also this bridge's liveness test, because kill -0 cannot +# answer that question across the namespace boundary. +fm_win_command() { # + local winpid=$1 out + case "$winpid" in + ''|*[!0-9]*) return 1 ;; + esac + out=$(ps -W 2>/dev/null | awk -v w="$winpid" '$4 == w { for (i = 8; i <= NF; i++) printf "%s%s", (i > 8 ? " " : ""), $i; exit }') + [ -n "$out" ] || return 1 + fm_win_normalize_command "$out" +} + +# Environment variables through which a verified harness publishes the pid of +# its own session process. Extend only with a variable confirmed to name the +# session-long harness process on Windows, because the identity check below is +# only as narrow as this table. +# +# Walking the real Windows parent chain is deliberately NOT the fallback here. +# Two properties of this platform make it unusable for identity: +# - MSYS emulates exec by spawning a fresh Windows process and exiting the old +# one, so intermediate shells vanish constantly and a child's recorded parent +# is routinely a pid that no longer exists. The chain simply breaks. +# - Windows never reparents an orphan, so that dangling parent id stays on the +# child and Windows is free to reissue it. Following it can therefore land on +# an unrelated live process and bind a home's session lock to it, which is +# the wrong-process-binding failure this file exists to prevent. +# A harness that publishes nothing is reported as unresolved instead, which +# leaves the session read-only exactly as before - the safe direction. +FM_WIN_HARNESS_PID_VARS=(CLAUDE_PID) + +# Print this session's harness as one tagged Windows pid, or return 1. +# +# The published pid is a claim, not evidence, so it is never trusted on its own: +# it is confirmed against the Windows process table, and accepted only when that +# pid is still live AND its executable independently identifies a verified +# harness by the same rules every other platform uses. A value that is absent, +# malformed, stale, or naming a non-harness process is discarded rather than +# used, so a wrong or recycled pid fails closed instead of binding the lock. +fm_win_harness_ancestry_pids() { + local var winpid comm + for var in "${FM_WIN_HARNESS_PID_VARS[@]}"; do + winpid=${!var:-} + case "$winpid" in + ''|*[!0-9]*) continue ;; + esac + comm=$(fm_win_command "$winpid") || continue + fm_harness_process_matches "$comm" "$comm" || continue + printf '%s%s\n' "$FM_WIN_PID_PREFIX" "$winpid" + return 0 + done + return 1 +} + # Walk the current process ancestry (up to 16 hops) and print this session's # contiguous verified-harness ancestry, innermost pid first. # @@ -111,8 +245,8 @@ fm_harness_process_matches() { # fm_harness_ancestry_pids() { local pid=$$ comm args extending=0 printed=0 for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do - comm=$(ps -o comm= -p "$pid" 2>/dev/null) || break - args=$(ps -o args= -p "$pid" 2>/dev/null) + comm=$(fm_ps_comm "$pid") || break + args=$(fm_ps_args "$pid") if fm_harness_process_matches "$comm" "$args"; then printf '%s\n' "$pid" printed=1 @@ -121,7 +255,7 @@ fm_harness_ancestry_pids() { elif [ "$extending" -eq 1 ]; then break fi - pid=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d ' ') + pid=$(fm_ps_ppid "$pid") # Examine the top of the chain before stopping. Inside a PID namespace the # harness itself is pid 1, so stopping as soon as the next pid is 1 hides the # very process this walk exists to find. A host's real pid 1 (init, systemd, @@ -129,6 +263,14 @@ fm_harness_ancestry_pids() { case "$pid" in '' | *[!0-9]*) break ;; esac [ "$pid" -ge 1 ] || break done + # A harness living in the same process table is always preferred, so an + # ordinary POSIX ancestry keeps resolving to plain pids and nothing about the + # existing platforms changes. The Windows bridge is consulted only after that + # walk finds nothing, which on Cygwin is what the severed parent link + # guarantees it will do. + if [ "$printed" -eq 0 ] && fm_win_boundary_applies; then + fm_win_harness_ancestry_pids && return 0 + fi [ "$printed" -eq 1 ] } @@ -151,11 +293,19 @@ EOF } # True if $1 is a live process that looks like a verified harness. +# A tagged Windows pid is answered from the Windows process table, because +# kill -0 cannot see across that boundary and would report a live harness as +# dead - which would hand a running session's home to a second one. fm_harness_pid_alive() { - local pid=$1 comm args + local pid=$1 comm args winpid + if winpid=$(fm_win_untag_pid "$pid"); then + comm=$(fm_win_command "$winpid") || return 1 + fm_harness_process_matches "$comm" "$comm" + return + fi kill -0 "$pid" 2>/dev/null || return 1 - comm=$(ps -o comm= -p "$pid" 2>/dev/null) || return 1 - args=$(ps -o args= -p "$pid" 2>/dev/null) + comm=$(fm_ps_comm "$pid") || return 1 + args=$(fm_ps_args "$pid") fm_harness_process_matches "$comm" "$args" } @@ -171,6 +321,7 @@ fm_session_lock_owned_by_self() { local state=$1 lock_pid pids pid lock_pid=$(cat "$state/.lock" 2>/dev/null || true) case "$lock_pid" in + "$FM_WIN_PID_PREFIX"[0-9]*) : ;; ''|*[!0-9]*) return 1 ;; esac pids=$(fm_harness_ancestry_pids) || return 1 diff --git a/bin/fm-sessionstart-nudge.sh b/bin/fm-sessionstart-nudge.sh index a12aa3e4628..56acdd9fb91 100755 --- a/bin/fm-sessionstart-nudge.sh +++ b/bin/fm-sessionstart-nudge.sh @@ -16,36 +16,18 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" . "$SCRIPT_DIR/fm-primary-scope-lib.sh" # shellcheck source=bin/fm-operational-input.sh . "$SCRIPT_DIR/fm-operational-input.sh" +# shellcheck source=bin/fm-session-lock-lib.sh +. "$SCRIPT_DIR/fm-session-lock-lib.sh" fm_is_gate_agent "$FM_ROOT" && exit 0 fm_primary_scope_matches "$FM_ROOT" "$STATE" || exit 0 -lock_is_in_ancestry() { - local lock_pid pid=$$ _ - [ -f "$STATE/.lock" ] || return 1 - IFS= read -r lock_pid < "$STATE/.lock" 2>/dev/null || return 1 - case "$lock_pid" in - # A lock pid of 1 is legitimate inside a PID namespace, where the harness - # holding the home lock IS pid 1, so it is no longer rejected outright; the - # liveness check below still gates it. On a host, a lock file that wrongly - # names pid 1 can now make this hook conclude the lock is already held and - # stay silent, which is the safe direction for a SessionStart hook whose only - # outputs are one nudge line or nothing. - ''|*[!0-9]*) return 1 ;; - esac - kill -0 "$lock_pid" 2>/dev/null || return 1 - for _ in 1 2 3 4 5 6 7 8; do - [ "$pid" = "$lock_pid" ] && return 0 - pid=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d ' ') - # Stop only after the top of the chain has been compared, for the same - # namespace reason as bin/fm-session-lock-lib.sh's walk. - case "$pid" in '' | *[!0-9]*) return 1 ;; esac - [ "$pid" -ge 1 ] || return 1 - done - return 1 -} - -lock_is_in_ancestry && exit 0 +# "Has this session already acquired the home lock?" is one question with one +# owner. Asking it here with a private ancestry walk kept a second copy of every +# platform assumption that owner makes, so this script stayed wrong on Cygwin +# after the owner itself was fixed - and nudged a session that already held the +# lock to start over. +fm_session_lock_owned_by_self "$STATE" && exit 0 nudge= fm_operational_input_encode session-start \ "Run \`bin/fm-session-start.sh\` now, exactly once, before executing any other instructions." \ diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index dbf1e683f77..8a067431f64 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -266,6 +266,139 @@ SH pass "session-lock: a live version-named session holding the lock is not mistaken for a stale owner" } +# --- Cygwin / Git-for-Windows layer ------------------------------------------ +# +# Both of this platform's departures are reproduced here rather than assumed, so +# these run identically on Linux and macOS CI: its ps rejects -o outright, and +# every shell it reports has its parent link severed at 1 because the real parent +# is a Windows process Cygwin cannot see. + +# A fakebin whose ps behaves like Cygwin's and whose uname reports MINGW. +# The Windows-side table is supplied per case through FM_TEST_WIN_TABLE, in the +# same column order the real `ps -W` prints. +cygwin_fakebin() { # + local dir=$1 fakebin + fakebin=$(fm_fakebin "$dir") + cat > "$fakebin/uname" <<'SH' +#!/usr/bin/env bash +printf '%s\n' 'MINGW64_NT-10.0-26200' +SH + cat > "$fakebin/ps" <<'SH' +#!/usr/bin/env bash +set -u +for a in "$@"; do + # Cygwin's ps has no -o option at all; it fails the whole invocation. + [ "$a" = "-o" ] && { echo "ps: unknown option -- o" >&2; exit 1; } +done +mode=p full=0 pid= +while [ "$#" -gt 0 ]; do + case "$1" in + -W) mode=W; shift ;; + -f) full=1; shift ;; + -p) pid=$2; shift 2 ;; + *) shift ;; + esac +done +if [ "$mode" = W ]; then + printf '%s\n' ' PID PPID PGID WINPID TTY UID STIME COMMAND' + [ -n "${FM_TEST_WIN_TABLE:-}" ] && printf '%s\n' "$FM_TEST_WIN_TABLE" + exit 0 +fi +# The Cygwin-side table. 700 is an MSYS-native harness; everything else is an +# ordinary shell whose parent link is severed exactly as the real ps reports it. +case "$pid" in + 700) comm='/opt/claude/versions/2.1.220'; ppid=1 ;; + *) comm='/usr/bin/bash'; ppid=${FM_TEST_CYG_PPID:-1} ;; +esac +if [ "$full" = 1 ]; then + printf '%s\n' ' UID PID PPID TTY STIME COMMAND' + printf 'u %s %s ? 00:00:00 %s\n' "$pid" "$ppid" "$comm" +else + printf '%s\n' ' PID PPID PGID WINPID TTY UID STIME COMMAND' + printf '%s %s %s 9999 ? 0 00:00:00 %s\n' "$pid" "$ppid" "$pid" "$comm" +fi +SH + chmod +x "$fakebin/uname" "$fakebin/ps" + printf '%s\n' "$fakebin" +} + +# WINPID 7204 is the session harness. 4321 is an ordinary desktop process, and +# 5150 is the path-shaped lookalike that must never read as a harness. +WIN_TABLE=' 4201508 0 0 7204 ? 0 22:24:48 C:\Users\u\.local\bin\claude.exe + 4198625 0 0 4321 ? 0 22:24:48 C:\Windows\explorer.exe + 4199454 0 0 5150 ? 0 22:24:48 C:\tools\claude-notes\helper.exe' + +test_windows_session_is_identified_from_its_published_pid() { + local dir fakebin got + dir="$TMP_ROOT/win-published" + fakebin=$(cygwin_fakebin "$dir") + mkdir -p "$dir/state" + printf 'win:7204\n' > "$dir/state/.lock" + + got=$(FM_TEST_WIN_TABLE="$WIN_TABLE" CLAUDE_PID=7204 lib_eval "$fakebin" 'fm_harness_ancestry_pid') \ + || fail "the session was not identified at all on a severed Cygwin parent link" + [ "$got" = 'win:7204' ] || fail "expected the tagged Windows session pid win:7204, got '$got'" + FM_TEST_WIN_TABLE="$WIN_TABLE" CLAUDE_PID=7204 lib_eval "$fakebin" 'fm_harness_pid_alive win:7204' \ + || fail "a live Windows-side session was classified as a dead lock owner" + FM_TEST_WIN_TABLE="$WIN_TABLE" CLAUDE_PID=7204 lib_eval "$fakebin" "fm_session_lock_owned_by_self '$dir/state'" \ + || fail "the session holding the lock did not recognize itself as the owner" + pass "session-lock: a Windows session is identified from its published pid across the severed parent link" +} + +test_windows_published_pid_is_confirmed_before_it_is_trusted() { + local dir fakebin + dir="$TMP_ROOT/win-unconfirmed" + fakebin=$(cygwin_fakebin "$dir") + mkdir -p "$dir/state" + + # A published pid is a claim. Each of these shapes must be discarded rather + # than bound: a process that is gone, one that is not a harness at all, and a + # path that merely contains the harness name inside a longer component. + for claimed in 9999 4321 5150; do + if FM_TEST_WIN_TABLE="$WIN_TABLE" CLAUDE_PID="$claimed" lib_eval "$fakebin" 'fm_harness_ancestry_pid'; then + fail "published pid $claimed was bound as this session's harness without confirmation" + fi + if FM_TEST_WIN_TABLE="$WIN_TABLE" lib_eval "$fakebin" "fm_harness_pid_alive win:$claimed"; then + fail "published pid $claimed passed the harness-liveness predicate" + fi + done + pass "session-lock: a published Windows pid is confirmed against the process table before it is trusted" +} + +test_windows_pid_is_never_resolved_as_a_cygwin_pid() { + local dir fakebin + dir="$TMP_ROOT/win-namespace" + fakebin=$(cygwin_fakebin "$dir") + mkdir -p "$dir/state" + + # 700 is a harness in the CYGWIN table and absent from the Windows one. The + # two namespaces overlap numerically, so resolving a tagged pid through the + # local table would bind a home to whatever unrelated process holds that + # number - which is exactly what the tag exists to prevent. + FM_TEST_WIN_TABLE="$WIN_TABLE" lib_eval "$fakebin" 'fm_harness_pid_alive 700' \ + || fail "fixture is vacuous: pid 700 must be a live harness in the Cygwin table" + if FM_TEST_WIN_TABLE="$WIN_TABLE" lib_eval "$fakebin" 'fm_harness_pid_alive win:700'; then + fail "a tagged Windows pid was resolved against the Cygwin process table" + fi + pass "session-lock: a tagged Windows pid is never resolved against the Cygwin process table" +} + +test_cygwin_ps_without_o_still_resolves_a_local_harness() { + local dir fakebin got + dir="$TMP_ROOT/cygwin-local" + fakebin=$(cygwin_fakebin "$dir") + mkdir -p "$dir/state" + + # An MSYS-native harness lives in the same process table as this shell, so it + # must resolve through the ordinary walk and stay an untagged pid. This is + # what proves the walk survives a ps with no -o option at all. + got=$(FM_TEST_CYG_PPID=700 FM_TEST_WIN_TABLE="$WIN_TABLE" CLAUDE_PID=7204 \ + lib_eval "$fakebin" 'fm_harness_ancestry_pid') \ + || fail "a harness in the local process table was not resolved when ps rejected -o" + [ "$got" = 700 ] || fail "expected the untagged local harness pid 700, got '$got'" + pass "session-lock: a harness in the local process table resolves untagged when ps has no -o option" +} + # --- end-to-end layer: the real Stop auto-arm in real process trees ---------- install_autoarm_scripts() { @@ -409,6 +542,10 @@ test_harness_at_namespace_pid1_is_examined test_ordinary_paths_are_never_harness_processes test_harness_beyond_a_gap_never_owns_the_lock test_competing_version_named_session_is_seen_as_live +test_windows_session_is_identified_from_its_published_pid +test_windows_published_pid_is_confirmed_before_it_is_trusted +test_windows_pid_is_never_resolved_as_a_cygwin_pid +test_cygwin_ps_without_o_still_resolves_a_local_harness test_e2e_version_named_session_claims_the_home test_e2e_daemon_parented_session_claims_the_home test_e2e_daemon_parented_version_named_session_keeps_its_lock From a975ea4c24321cf97af50e7943e69ec6d227219b Mon Sep 17 00:00:00 2001 From: Lloyd Date: Wed, 2 Sep 2026 21:02:49 +0000 Subject: [PATCH 02/61] fix(session-lock): keep the nudge's own ancestry question intact Delegating the nudge to fm_session_lock_owned_by_self changed the question it asks. The nudge asks whether a process in this ancestry took the lock; the ownership predicate additionally requires a verified harness in that ancestry, so a session whose lock was written by a plain shell started being nudged to run session start again. tests/fm-sessionstart-nudge.test.sh pins the looser contract deliberately. The walk stays local, now reading ppid through the portable accessor so it also survives a ps with no -o option, and defers to the ownership predicate only for a Windows-tagged holder, which is not in this process table at all. (cherry picked from commit 0baf64f71ffea275292a0fd88d033caccc041fc9) --- bin/fm-sessionstart-nudge.sh | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/bin/fm-sessionstart-nudge.sh b/bin/fm-sessionstart-nudge.sh index 56acdd9fb91..c996e023d94 100755 --- a/bin/fm-sessionstart-nudge.sh +++ b/bin/fm-sessionstart-nudge.sh @@ -22,12 +22,30 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" fm_is_gate_agent "$FM_ROOT" && exit 0 fm_primary_scope_matches "$FM_ROOT" "$STATE" || exit 0 -# "Has this session already acquired the home lock?" is one question with one -# owner. Asking it here with a private ancestry walk kept a second copy of every -# platform assumption that owner makes, so this script stayed wrong on Cygwin -# after the owner itself was fixed - and nudged a session that already held the -# lock to start over. -fm_session_lock_owned_by_self "$STATE" && exit 0 +lock_is_in_ancestry() { + local lock_pid pid=$$ _ + [ -f "$STATE/.lock" ] || return 1 + IFS= read -r lock_pid < "$STATE/.lock" 2>/dev/null || return 1 + case "$lock_pid" in + # A Windows-tagged holder is not in this process table at all, so a local + # ancestry comparison cannot answer for it. Defer to the owner of harness + # identity, which is the only thing that can read across that boundary. + "$FM_WIN_PID_PREFIX"[0-9]*) fm_session_lock_owned_by_self "$STATE"; return ;; + # PID 1 can own the session inside a PID namespace; compare it below. + ''|*[!0-9]*) return 1 ;; + esac + kill -0 "$lock_pid" 2>/dev/null || return 1 + for _ in 1 2 3 4 5 6 7 8; do + [ "$pid" = "$lock_pid" ] && return 0 + pid=$(fm_ps_ppid "$pid") + # Compare the top of the chain before stopping, as in the owning library. + case "$pid" in '' | *[!0-9]*) return 1 ;; esac + [ "$pid" -ge 1 ] || return 1 + done + return 1 +} + +lock_is_in_ancestry && exit 0 nudge= fm_operational_input_encode session-start \ "Run \`bin/fm-session-start.sh\` now, exactly once, before executing any other instructions." \ From 36dbddc321884944a0a466b164718e9f23b6dc67 Mon Sep 17 00:00:00 2001 From: Lloyd Date: Wed, 2 Sep 2026 23:23:59 +0200 Subject: [PATCH 03/61] fix(session-lock): accept a tagged identity in every gate that reads the lock The Windows identity added in this branch introduced a second shape into state/.lock. Six gates read that field, and each one answered "is this value usable" with its own inline numeric test, so every one of them read a valid Windows holder as malformed. The visible cost was concentrated in startup completion: the record was never written, and the clear/compact check that consumes it could never match, so every clear or compact on Windows repeated the full startup sequence. The deferred network sweeps reported ownership as changed when it had not, and the Stop auto-arm treated a dead Windows session as an unreadable lock rather than a recoverable one. fm-lease.sh was the outlier: rather than refusing a value it could not use, `tr -cd '0-9'` reduced the tag to its digits and produced a number naming an unrelated process in the local table. It happened to fail closed downstream, but deriving a wrong-namespace pid is precisely what the tag exists to prevent, so it now takes the value whole and requires a local pid. fm_session_pid_valid is now the single owner of that question and every gate delegates to it, so a third identity shape cannot split them again. Verified against the shipped bytes of each gate with a tagged lock: startup completion now records and is recognized (main's gate reruns the full startup on the same input), both network-ownership gates authorize their sweeps, and a tagged lock yields no lease holder pid instead of 7204. (cherry picked from commit 92561828fb9540ab6c77437def67a954e44b0418) --- bin/fm-bootstrap.sh | 6 ++++- bin/fm-claude-stop-autoarm.sh | 10 +++---- bin/fm-lease.sh | 8 +++++- bin/fm-session-lock-lib.sh | 19 +++++++++++--- bin/fm-session-start.sh | 4 +-- bin/fm-sessionstart-run.sh | 2 +- bin/fm-startup-network.sh | 2 +- tests/fm-session-lock-ancestry.test.sh | 36 ++++++++++++++++++++++++++ 8 files changed, 71 insertions(+), 16 deletions(-) diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index 98b791e52f7..2ec0d432431 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -191,6 +191,10 @@ DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" # deferred network stage sets, so an ordinary bootstrap run records nothing. # shellcheck source=bin/fm-timing-lib.sh disable=SC1091 . "$SCRIPT_DIR/fm-timing-lib.sh" +# Sourced only for fm_session_pid_valid: the lock identity this script compares +# is not always a local pid. +# shellcheck source=bin/fm-session-lock-lib.sh disable=SC1091 +. "$SCRIPT_DIR/fm-session-lock-lib.sh" # Network-phase selection (see the header). An unrecognized value resolves to # `all` so a malformed override runs every step rather than silently dropping a @@ -205,7 +209,7 @@ network_phase() { [ "$FM_BOOTSTRAP_NETWORK_PHASE" != skip ]; } network_mutation_authorized() { local expected=${FM_BOOTSTRAP_NETWORK_LOCK_PID:-} current [ -n "$expected" ] || return 0 - case "$expected" in *[!0-9]*) return 1 ;; esac + fm_session_pid_valid "$expected" || return 1 [ -f "$STATE/.lock" ] && [ ! -L "$STATE/.lock" ] || return 1 current=$(cat "$STATE/.lock" 2>/dev/null) || return 1 [ "$current" = "$expected" ] diff --git a/bin/fm-claude-stop-autoarm.sh b/bin/fm-claude-stop-autoarm.sh index df1100ba988..a96eb4c96ec 100755 --- a/bin/fm-claude-stop-autoarm.sh +++ b/bin/fm-claude-stop-autoarm.sh @@ -120,17 +120,17 @@ fm_hook_payload_is_foreign_host "$PAYLOAD" && exit 0 fm_primary_scope_matches "$FM_ROOT" "$STATE" || exit 0 # --- identity: only the lock-owning session's hooks may arm ------------------ -# A prior session may have died after leaving its numeric harness pid in .lock. -# Use the shared liveness predicate to recognize only that stale-owner case. +# A prior session may have died after leaving its harness identity in .lock. +# Use the shared liveness predicate to recognize only that stale-owner case; it +# resolves a tagged identity too, so a dead Windows session is still recoverable +# rather than being read as a malformed lock nobody may ever clear. # Defer the mutating claim until after the unchanged AFK and need gates, so an # idle or away home remains byte-for-byte inert. Missing or malformed locks are # uncertainty rather than stale-owner evidence and remain inert. RECOVER_SESSION_LOCK=0 if ! fm_session_lock_owned_by_self "$STATE"; then LOCK_PID=$(cat "$STATE/.lock" 2>/dev/null || true) - case "$LOCK_PID" in - ''|*[!0-9]*) exit 0 ;; - esac + fm_session_pid_valid "$LOCK_PID" || exit 0 fm_harness_pid_alive "$LOCK_PID" && exit 0 RECOVER_SESSION_LOCK=1 fi diff --git a/bin/fm-lease.sh b/bin/fm-lease.sh index b90c205d425..1b83f6a55bf 100755 --- a/bin/fm-lease.sh +++ b/bin/fm-lease.sh @@ -121,10 +121,16 @@ case "$CMD" in # provides one (the Pi branch extension passes the session-lock holder), # else the session-lock holder (state/.lock is the harness pid), else this # shell; without a matching session lock the resulting lease is stale. + # This pid is checked with kill -0, so only a pid in THIS process table can + # serve. Take the lock's value whole and require it to be one: a lock holding + # an identity from another namespace must yield nothing and fall through to + # the shell below, never be reduced to its digits, because those digits name + # an unrelated local process rather than the holder. HOLDER_PID=${FM_LEASE_HOLDER_PID:-} case "$HOLDER_PID" in *[!0-9]*) HOLDER_PID= ;; esac if [ -z "$HOLDER_PID" ]; then - HOLDER_PID=$(head -n 1 "$STATE/.lock" 2>/dev/null | tr -cd '0-9' || true) + HOLDER_PID=$(head -n 1 "$STATE/.lock" 2>/dev/null || true) + case "$HOLDER_PID" in *[!0-9]*) HOLDER_PID= ;; esac fi [ -n "$HOLDER_PID" ] || HOLDER_PID=$$ TMP=$(mktemp "$STATE/.fm-lease-tmp.XXXXXX") diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index 86653f62617..73caeecaf05 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -160,6 +160,20 @@ fm_win_untag_pid() { # return 1 } +# True when $1 is a well-formed session-lock identity: a local pid, or a tagged +# Windows pid. Every reader of state/.lock decides "is this value usable at all" +# through this one predicate, because the readers are spread across several +# scripts and a private numeric test in any one of them silently rejects a valid +# holder - which reads as "startup never completed" and repeats the whole +# sequence on every clear or compact. +fm_session_pid_valid() { # + case "$1" in + "$FM_WIN_PID_PREFIX"[0-9]*) return 0 ;; + ''|*[!0-9]*) return 1 ;; + esac + return 0 +} + # Windows command paths are backslash-separated and .exe-suffixed, neither of # which the path-component matcher above understands. Normalizing here is what # keeps that matcher's whole-component safety intact: without it the harness @@ -320,10 +334,7 @@ fm_harness_pid_alive() { fm_session_lock_owned_by_self() { local state=$1 lock_pid pids pid lock_pid=$(cat "$state/.lock" 2>/dev/null || true) - case "$lock_pid" in - "$FM_WIN_PID_PREFIX"[0-9]*) : ;; - ''|*[!0-9]*) return 1 ;; - esac + fm_session_pid_valid "$lock_pid" || return 1 pids=$(fm_harness_ancestry_pids) || return 1 while IFS= read -r pid; do [ "$pid" = "$lock_pid" ] && return 0 diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index 3a994b86030..c7ae855edb9 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -996,9 +996,7 @@ EOF if [ "$READ_ONLY" -eq 0 ] && [ "$REEMIT" -eq 0 ]; then COMPLETION_RECORDED=0 COMPLETION_PID=$(cat "$STATE/.lock" 2>/dev/null || true) - case "$COMPLETION_PID" in - ''|*[!0-9]*) COMPLETION_PID= ;; - esac + fm_session_pid_valid "$COMPLETION_PID" || COMPLETION_PID= COMPLETION_TMP=$(mktemp "$STATE/.session-start-complete.XXXXXX" 2>/dev/null || true) if [ -n "$COMPLETION_PID" ] && [ -n "$COMPLETION_TMP" ] \ && printf '%s\n' "$COMPLETION_PID" > "$COMPLETION_TMP" 2>/dev/null \ diff --git a/bin/fm-sessionstart-run.sh b/bin/fm-sessionstart-run.sh index a970eced675..4aab178e96a 100755 --- a/bin/fm-sessionstart-run.sh +++ b/bin/fm-sessionstart-run.sh @@ -96,7 +96,7 @@ session_start_completed() { fm_session_lock_owned_by_self "$STATE" || return 1 lock_pid=$(cat "$STATE/.lock" 2>/dev/null) || return 1 completion_pid=$(cat "$COMPLETION_FILE" 2>/dev/null) || return 1 - case "$lock_pid" in ''|*[!0-9]*) return 1 ;; esac + fm_session_pid_valid "$lock_pid" || return 1 [ "$completion_pid" = "$lock_pid" ] } diff --git a/bin/fm-startup-network.sh b/bin/fm-startup-network.sh index 380138ae25f..4f08661673c 100755 --- a/bin/fm-startup-network.sh +++ b/bin/fm-startup-network.sh @@ -310,7 +310,7 @@ EOF # fail closed to the read-only probe. lock_unchanged() { # local expected=$1 current - case "$expected" in ''|*[!0-9]*) return 1 ;; esac + fm_session_pid_valid "$expected" || return 1 [ -f "$STATE/.lock" ] && [ ! -L "$STATE/.lock" ] || return 1 current=$(cat "$STATE/.lock" 2>/dev/null) || return 1 [ "$current" = "$expected" ] diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index 8a067431f64..5dcee2fea75 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -383,6 +383,41 @@ test_windows_pid_is_never_resolved_as_a_cygwin_pid() { pass "session-lock: a tagged Windows pid is never resolved against the Cygwin process table" } +test_a_published_identity_is_accepted_by_the_gates_that_read_the_lock() { + local dir fakebin identity + dir="$TMP_ROOT/win-identity-gates" + fakebin=$(cygwin_fakebin "$dir") + mkdir -p "$dir/state" + + # The identity a session writes into the lock is read back by gates spread + # across several scripts - startup-completion recording and its clear/compact + # validation, the deferred network sweeps, and the Stop auto-arm. Each one + # asks only "is this value a usable identity", and each answered that with its + # own numeric test, so introducing a second identity shape made every one of + # them silently read a valid holder as malformed. The visible cost was a + # completion record that was never written, which reads as "startup never + # finished" and repeats the whole sequence on every clear or compact. One + # predicate owns the question so a third shape can never split them again. + identity=$(FM_TEST_WIN_TABLE="$WIN_TABLE" CLAUDE_PID=7204 lib_eval "$fakebin" 'fm_harness_ancestry_pid') \ + || fail "fixture is vacuous: no identity was published to gate on" + lib_eval "$fakebin" "fm_session_pid_valid '$identity'" \ + || fail "the identity '$identity' this session publishes is rejected by the gates that read it back" + + # A local pid stays equally valid: the Windows shape is additional, not a + # replacement, and the same predicate serves both platforms. + lib_eval "$fakebin" 'fm_session_pid_valid 700' \ + || fail "an ordinary local pid was rejected as an invalid lock identity" + + # Still fail closed on the shapes a torn or hand-edited lock produces, and on + # a bare tag carrying no pid at all. + for bad in '' 'win:' 'win:abc' 'abc' '70 0' '-1'; do + if lib_eval "$fakebin" "fm_session_pid_valid '$bad'"; then + fail "the malformed lock value '$bad' was accepted as a usable identity" + fi + done + pass "session-lock: a published identity is accepted by every gate that reads the lock" +} + test_cygwin_ps_without_o_still_resolves_a_local_harness() { local dir fakebin got dir="$TMP_ROOT/cygwin-local" @@ -545,6 +580,7 @@ test_competing_version_named_session_is_seen_as_live test_windows_session_is_identified_from_its_published_pid test_windows_published_pid_is_confirmed_before_it_is_trusted test_windows_pid_is_never_resolved_as_a_cygwin_pid +test_a_published_identity_is_accepted_by_the_gates_that_read_the_lock test_cygwin_ps_without_o_still_resolves_a_local_harness test_e2e_version_named_session_claims_the_home test_e2e_daemon_parented_session_claims_the_home From a24f8d109c80a292fe5a36380f110a025d0376c2 Mon Sep 17 00:00:00 2001 From: Cristian Date: Tue, 15 Sep 2026 23:51:26 +1200 Subject: [PATCH 04/61] feat(native-owner): consolidate experimental ownership core and Codex tool policy --- .gitattributes | 4 + bin/fm-test-run.sh | 4 +- bin/native-owner/NativeHomeLease.cs | 97 +++++ bin/native-owner/NativeOwner.cs | 161 ++++++++ bin/native-owner/codex-tool-gate.mjs | 91 +++++ docs/verification/runtime-backends.md | 35 ++ tests/fixtures/native-owner/AppHost.mjs | 119 ++++++ tests/fixtures/native-owner/Build.ps1 | 26 ++ tests/fixtures/native-owner/NativeDriver.cs | 362 ++++++++++++++++++ tests/fixtures/native-owner/Run-Cycle.mjs | 27 ++ tests/fixtures/native-owner/Verify-Cycle.mjs | 48 +++ .../native-owner/consumer-adapter.patch | 96 +++++ tests/fixtures/native-owner/exercise.sh | 39 ++ tests/fixtures/native-owner/jq | 7 + .../fixtures/native-owner/notification-ack.sh | 24 ++ .../native-owner/notification-check.sh | 33 ++ .../fixtures/native-owner/tool-gate.test.mjs | 64 ++++ tests/fm-native-owner-codex-live-e2e.test.sh | 17 + tests/fm-native-owner-tool-gate.test.sh | 5 + 19 files changed, 1257 insertions(+), 2 deletions(-) create mode 100644 bin/native-owner/NativeHomeLease.cs create mode 100644 bin/native-owner/NativeOwner.cs create mode 100644 bin/native-owner/codex-tool-gate.mjs create mode 100644 tests/fixtures/native-owner/AppHost.mjs create mode 100644 tests/fixtures/native-owner/Build.ps1 create mode 100644 tests/fixtures/native-owner/NativeDriver.cs create mode 100644 tests/fixtures/native-owner/Run-Cycle.mjs create mode 100644 tests/fixtures/native-owner/Verify-Cycle.mjs create mode 100644 tests/fixtures/native-owner/consumer-adapter.patch create mode 100644 tests/fixtures/native-owner/exercise.sh create mode 100644 tests/fixtures/native-owner/jq create mode 100644 tests/fixtures/native-owner/notification-ack.sh create mode 100644 tests/fixtures/native-owner/notification-check.sh create mode 100644 tests/fixtures/native-owner/tool-gate.test.mjs create mode 100644 tests/fm-native-owner-codex-live-e2e.test.sh create mode 100644 tests/fm-native-owner-tool-gate.test.sh diff --git a/.gitattributes b/.gitattributes index 7bdc6ed6b5a..a075a637e90 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1,6 @@ # Bash parses shell scripts with LF line endings on every supported platform. *.sh text eol=lf +# Test overlays and the extensionless Bash fixture must survive Windows clones. +# Unified diff context includes intentional space-only lines. +tests/fixtures/native-owner/consumer-adapter.patch text eol=lf whitespace=-blank-at-eol +tests/fixtures/native-owner/jq text eol=lf diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index bcfac4f3ed5..b1c57307aa0 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -282,7 +282,7 @@ family_for_basename() { fm-composer-ghost.test.sh|fm-composer-lib.test.sh|\ fm-crew-state.test.sh|fm-captain-hold-lifecycle.test.sh|\ fm-documentation-audiences.test.sh|fm-ensure-agents-md.test.sh|fm-grok-harness.test.sh|\ - fm-harness-precedence.test.sh|\ + fm-harness-precedence.test.sh|fm-native-owner-tool-gate.test.sh|\ fm-kimi-harness.test.sh|fm-muse-harness.test.sh|fm-rovo-harness.test.sh|fm-agy-harness.test.sh|fm-omp-harness.test.sh|fm-herdr-lab.test.sh|fm-lint.test.sh|\ fm-lint-workflows.test.sh|\ fm-operational-input.test.sh|fm-pi-primary-types.test.sh|\ @@ -342,7 +342,7 @@ family_for_basename() { fm-claude-stop-autoarm-live-e2e.test.sh|\ fm-cmux-claude-composer-live-e2e.test.sh|\ fm-composer-matrix-live-e2e.test.sh|\ - fm-codex-continuity-live-e2e.test.sh|fm-grok-continuity-live-e2e.test.sh|\ + fm-codex-continuity-live-e2e.test.sh|fm-native-owner-codex-live-e2e.test.sh|fm-grok-continuity-live-e2e.test.sh|\ fm-cursor-primary-live-e2e.test.sh|\ fm-grok-stop-live-e2e.test.sh|fm-harness-adapter-instructions-live-e2e.test.sh|\ fm-harness-liveness-drift-live-e2e.test.sh|\ diff --git a/bin/native-owner/NativeHomeLease.cs b/bin/native-owner/NativeHomeLease.cs new file mode 100644 index 00000000000..09a056660c8 --- /dev/null +++ b/bin/native-owner/NativeHomeLease.cs @@ -0,0 +1,97 @@ +// Experimental home reservation. The temporary-home restriction deliberately +// remains until the production lifecycle and path checks are validated. +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Diagnostics; +using System.IO; +using System.Runtime.InteropServices; +using System.Security.AccessControl; +using System.Security.Principal; +using System.Text; +using System.Web.Script.Serialization; +public sealed class NativeHomeLease : IDisposable { + static readonly JavaScriptSerializer Json=new JavaScriptSerializer(); + FileStream file; + public readonly string Home; + public string PreviousGeneration { get; private set; } + [StructLayout(LayoutKind.Sequential)] struct FT { public uint low,high; } + [DllImport("kernel32.dll",SetLastError=true)] static extern IntPtr OpenProcess(uint access,bool inherit,uint pid); + [DllImport("kernel32.dll",SetLastError=true)] static extern bool GetProcessTimes(IntPtr process,out FT created,out FT exited,out FT kernel,out FT user); + [DllImport("kernel32.dll",SetLastError=true)] static extern uint WaitForSingleObject(IntPtr handle,uint milliseconds); + [DllImport("kernel32.dll")] static extern bool CloseHandle(IntPtr handle); + static string Filename(string home) { + string full=Path.GetFullPath(home); + if(!full.StartsWith(Path.GetFullPath(Path.GetTempPath()),StringComparison.OrdinalIgnoreCase)) throw new InvalidOperationException("Probe leases require a temporary home"); + return Path.Combine(full,"owner-probe.json"); + } + static Dictionary Read(Stream stream) { + stream.Position=0; + using(var reader=new StreamReader(stream,Encoding.UTF8,false,1024,true)) return Json.Deserialize>(reader.ReadToEnd()); + } + static bool RootAlive(Dictionary record) { + if(record==null || !record.ContainsKey("state") || (string)record["state"]!="live") throw new InvalidOperationException("Ambiguous or incomplete owner record; preserved"); + uint pid=Convert.ToUInt32(record["rootPid"]); + if(pid==0) throw new InvalidOperationException("Invalid recorded owner"); + IntPtr handle=OpenProcess(0x1000|0x100000,false,pid); + if(handle==IntPtr.Zero) { + int error=Marshal.GetLastWin32Error(); + if(error==87) return false; + throw new Win32Exception(error,"Recorded owner unreadable; preserved"); + } + try { + FT born,exit,kernel,user; + if(!GetProcessTimes(handle,out born,out exit,out kernel,out user)) throw new Win32Exception(Marshal.GetLastWin32Error()); + ulong creation=((ulong)born.high<<32)|born.low; + if(creation!=Convert.ToUInt64(record["rootCreationFileTime"])) return false; + uint wait=WaitForSingleObject(handle,0); + if(wait==0) return false; + if(wait!=258) throw new InvalidOperationException("Recorded owner liveness unreadable"); + return true; + } finally { CloseHandle(handle); } + } + public NativeHomeLease(string home) { + Home=Path.GetFullPath(home); + string name=Filename(Home); + Directory.CreateDirectory(Home); + var security=new FileSecurity(); security.SetAccessRuleProtection(true,false); + security.AddAccessRule(new FileSystemAccessRule(WindowsIdentity.GetCurrent().User,FileSystemRights.FullControl,AccessControlType.Allow)); + // The OS arbitrates concurrent controllers before either reads or writes. + bool created=false; + try { file=new FileStream(name,FileMode.CreateNew,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security); created=true; } + catch(IOException) { file=new FileStream(name,FileMode.Open,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security); } + try { + if(!created && file.Length==0) throw new InvalidOperationException("Empty existing owner record is ambiguous; preserved"); + if(file.Length>0) { + var previous=Read(file); + if(RootAlive(previous)) throw new InvalidOperationException("Recorded primary is still alive; refusing replacement"); + PreviousGeneration=previous.ContainsKey("generation") ? (string)previous["generation"] : null; + } + Write(new Dictionary{{"state","pending"},{"controllerPid",Process.GetCurrentProcess().Id}}); + } catch { Dispose(); throw; } + } + void Write(Dictionary value) { + byte[] bytes=Encoding.UTF8.GetBytes(Json.Serialize(value)); + file.Position=0; file.SetLength(0); file.Write(bytes,0,bytes.Length); file.Flush(true); + } + public void Publish(uint pid,ulong created,string generation,string pipe) { + using(var self=Process.GetCurrentProcess()) Write(new Dictionary{{"state","live"},{"rootPid",pid},{"rootCreationFileTime",created},{"generation",generation},{"pipe",pipe},{"controllerPid",self.Id},{"controllerCreated",self.StartTime.ToUniversalTime().ToFileTimeUtc()}}); + } + public static Dictionary Binding(string state) { + string canonical=Path.GetFullPath(state).TrimEnd(Path.DirectorySeparatorChar,Path.AltDirectorySeparatorChar); + if(!string.Equals(Path.GetFileName(canonical),"state",StringComparison.OrdinalIgnoreCase)) throw new InvalidOperationException("Unexpected state directory"); + using(var reader=new FileStream(Filename(Path.GetDirectoryName(canonical)),FileMode.Open,FileAccess.Read,FileShare.ReadWrite)) { + var record=Read(reader); + if(!RootAlive(record)) throw new InvalidOperationException("Primary no longer live"); + return record; + } + } + public static string Check(string home,string generation) { + using(var reader=new FileStream(Filename(home),FileMode.Open,FileAccess.Read,FileShare.ReadWrite)) { + var record=Read(reader); + bool same=record.ContainsKey("generation") && (string)record["generation"]==generation; + return Json.Serialize(new Dictionary{{"probeOwnerCurrent",same && RootAlive(record)},{"generationMatches",same},{"authorityGranted",false}}); + } + } + public void Dispose() { if(file!=null) { file.Dispose(); file=null; } } +} diff --git a/bin/native-owner/NativeOwner.cs b/bin/native-owner/NativeOwner.cs new file mode 100644 index 00000000000..cefee3c370f --- /dev/null +++ b/bin/native-owner/NativeOwner.cs @@ -0,0 +1,161 @@ +// Experimental native ownership core; no installed launcher is provided. +// The fixture driver registers scopes; association alone never grants ownership. +using System; +using System.Collections; +using System.Collections.Generic; +using System.ComponentModel; +using System.Diagnostics; +using System.IO; +using System.IO.Pipes; +using System.Runtime.InteropServices; +using System.Security.AccessControl; +using System.Security.Principal; +using System.Text; +using System.Threading; +using System.Web.Script.Serialization; + +public static partial class NativeOwner { + static JavaScriptSerializer Json = new JavaScriptSerializer(); + static string RegisteredHarness="unknown"; + const uint WAIT_TIMEOUT = 258; + [StructLayout(LayoutKind.Sequential)] struct SA { public int length; public IntPtr descriptor; public int inherit; } + [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)] struct SI { + public int cb; public string reserved, desktop, title; + public uint x,y,xSize,ySize,xCount,yCount,fill,flags; + public short show, reserved2; public IntPtr reservedBytes, input, output, error; + } + [StructLayout(LayoutKind.Sequential)] struct PI { public IntPtr process, thread; public uint pid, tid; } + [StructLayout(LayoutKind.Sequential)] struct FT { public uint low, high; } + [DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)] static extern IntPtr CreateJobObject(IntPtr security, string name); + [DllImport("kernel32.dll", SetLastError=true)] static extern bool AssignProcessToJobObject(IntPtr job, IntPtr process); + [DllImport("kernel32.dll", SetLastError=true)] static extern bool IsProcessInJob(IntPtr process, IntPtr job, out bool result); + [DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)] static extern bool CreateProcess(string exe, StringBuilder args, IntPtr psa, IntPtr tsa, bool inherit, uint flags, IntPtr env, string cwd, ref SI startup, out PI process); + [DllImport("kernel32.dll", SetLastError=true)] static extern uint ResumeThread(IntPtr thread); + [DllImport("kernel32.dll", SetLastError=true)] static extern uint WaitForSingleObject(IntPtr handle, uint milliseconds); + [DllImport("kernel32.dll", SetLastError=true)] static extern bool GetExitCodeProcess(IntPtr process, out uint code); + [DllImport("kernel32.dll", SetLastError=true)] static extern bool GetProcessTimes(IntPtr process, out FT created, out FT exited, out FT kernel, out FT user); + [DllImport("kernel32.dll", SetLastError=true)] static extern IntPtr OpenProcess(uint access, bool inherit, uint pid); + [DllImport("kernel32.dll", SetLastError=true)] static extern bool GetNamedPipeClientProcessId(IntPtr pipe, out uint pid); + [DllImport("kernel32.dll", SetLastError=true)] static extern bool CloseHandle(IntPtr handle); + [DllImport("kernel32.dll", SetLastError=true)] static extern bool TerminateProcess(IntPtr process, uint code); + [DllImport("kernel32.dll", SetLastError=true)] static extern bool TerminateJobObject(IntPtr job, uint code); + [DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)] static extern IntPtr CreateFile(string name, uint access, uint share, ref SA sa, uint creation, uint flags, IntPtr template); + + [StructLayout(LayoutKind.Sequential)] struct SidAttributes { public IntPtr sid; public uint attributes; } + [StructLayout(LayoutKind.Sequential)] struct TokenGroupsFirst { public uint count; public SidAttributes first; } + [DllImport("advapi32.dll", SetLastError=true)] static extern bool GetTokenInformation(IntPtr token, int kind, IntPtr data, int size, out int needed); + static List> TokenGroups(IntPtr token, int kind) { + int needed; GetTokenInformation(token,kind,IntPtr.Zero,0,out needed); + if (needed==0) throw Error("GetTokenInformation size"); + IntPtr data=Marshal.AllocHGlobal(needed); + try { + if (!GetTokenInformation(token,kind,data,needed,out needed)) throw Error("GetTokenInformation"); + int count=Marshal.ReadInt32(data), offset=(int)Marshal.OffsetOf(typeof(TokenGroupsFirst),"first"); + int stride=Marshal.SizeOf(typeof(SidAttributes)); + var rows=new List>(); + for(int i=0;i{{"sid",new SecurityIdentifier(entry.sid).Value},{"attributes",entry.attributes}}); + } + return rows; + } finally { Marshal.FreeHGlobal(data); } + } + static Exception Error(string call) { return new Win32Exception(Marshal.GetLastWin32Error(), call); } + static string OwnExe { get { return Process.GetCurrentProcess().MainModule.FileName; } } + static string Quote(string s) { return "\"" + s.Replace("\"", "\\\"") + "\""; } + sealed class ChildScope { public IntPtr job; public PI process; public string role; public string purpose; } + static ChildScope pendingOperation; + static Dictionary delivered; + static string receipt; + static bool consumed; + static int checkStarts,ackStarts; + static Dictionary Verdict(Dictionary request, uint pid, IntPtr root, IntPtr job, uint rootPid, List scopes, string session, string home, string nonce) { + string reason = "unverified", classification="none"; + if (WaitForSingleObject(root, 0) != WAIT_TIMEOUT) reason = "session-exited"; + else if (!request.ContainsKey("session") || (string)request["session"] != session) reason = "wrong-session"; + else if (!request.ContainsKey("home") || (string)request["home"] != home) reason = "wrong-home"; + else if (!request.ContainsKey("nonce") || (string)request["nonce"] != nonce) reason = "wrong-capability"; + else { + IntPtr client = OpenProcess(0x1000 | 0x100000, false, pid); + if (client == IntPtr.Zero) reason = "client-unreadable"; + else try { + bool member; + if (!IsProcessInJob(client, job, out member)) reason = "membership-unreadable"; + else if (WaitForSingleObject(client, 0) != WAIT_TIMEOUT) reason = "client-exited"; + else { + reason = member ? "associated" : "outside-session-job"; + if(member) { + classification=pid==rootPid ? "registered-primary" : "unclassified-descendant"; + foreach(var scope in scopes) { + bool scoped; + if(!IsProcessInJob(client,scope.job,out scoped)) { classification="scope-unreadable"; break; } + if(!scoped) continue; + // A restrictive child scope always defeats an enclosing grant. + if(scope.role!="owner-operation") { classification="registered-"+scope.role; break; } + classification=WaitForSingleObject(scope.process.process,0)==WAIT_TIMEOUT ? "registered-owner-operation" : "expired-owner-operation"; + } + } + } + } finally { CloseHandle(client); } + } + return new Dictionary { + {"clientPid",pid}, {"case", request.ContainsKey("case") ? request["case"] : "unnamed"}, + {"association",reason}, {"hostClassification",classification}, {"authorityGranted",false} + }; + } + [DllImport("kernel32.dll",SetLastError=true)] static extern bool GetNamedPipeServerProcessId(IntPtr pipe,out uint pid); + static string Canonical(string value) { return Path.GetFullPath(value).TrimEnd('\\','/'); } + static void AuthorizeOwner(Dictionary request,Dictionary verdict,NamedPipeServerStream server,NativeHomeLease lease,string session) { + bool unrestricted=false; + server.RunAsClient(delegate { + using(var identity=WindowsIdentity.GetCurrent(true)) unrestricted=identity!=null && TokenGroups(identity.Token,11).Count==0; + }); + bool allowed=lease!=null && (string)verdict["association"]=="associated" && unrestricted && + ((string)verdict["hostClassification"]=="registered-primary" || (string)verdict["hostClassification"]=="registered-owner-operation") && + request.ContainsKey("state") && string.Equals(Canonical((string)request["state"]),Canonical(Path.Combine(lease.Home,"state")),StringComparison.OrdinalIgnoreCase); + string verb=request.ContainsKey("verb") ? (string)request["verb"] : ""; + allowed=allowed && (verb=="identity" || verb=="alive" || verb=="owns" || verb=="harness"); + string id="native:"+session, value=""; + if(allowed) { + if(verb=="identity") value=id; + else if(verb=="harness") value=RegisteredHarness; + else if(verb=="alive") { + string requested=request.ContainsKey("id") ? (string)request["id"] : ""; + value=requested==id ? "true" : lease.PreviousGeneration!=null && requested=="native:"+lease.PreviousGeneration ? "false" : "unknown"; + } + else { + string filename=Path.Combine(lease.Home,"state",".lock"); + value=File.Exists(filename) && (File.GetAttributes(filename)&FileAttributes.ReparsePoint)==0 && File.ReadAllText(filename).Trim()==id ? "true" : "false"; + } + } + verdict["ownerAuthorized"]=allowed; + verdict["ownerValue"]=value; + // Limited to the isolated state path and the three operations above. + verdict["authorityGranted"]=allowed; + } + static int OwnerClient(string verb,string state,string id) { + var binding=NativeHomeLease.Binding(state); + using(var pipe=new NamedPipeClientStream(".",(string)binding["pipe"],PipeAccessRights.ReadData|PipeAccessRights.WriteData|PipeAccessRights.Synchronize,PipeOptions.None,TokenImpersonationLevel.Identification,HandleInheritability.None)) { + pipe.Connect(8000); + uint serverPid; + if(!GetNamedPipeServerProcessId(pipe.SafePipeHandle.DangerousGetHandle(),out serverPid) || serverPid!=Convert.ToUInt32(binding["controllerPid"])) throw new InvalidOperationException("Owner server identity mismatch"); + IntPtr server=OpenProcess(0x1000|0x100000,false,serverPid); + if(server==IntPtr.Zero) throw Error("Owner server unreadable"); + try { + FT born,exit,kernel,user; + if(!GetProcessTimes(server,out born,out exit,out kernel,out user) || (((ulong)born.high<<32)|born.low)!=Convert.ToUInt64(binding["controllerCreated"]) || WaitForSingleObject(server,0)!=WAIT_TIMEOUT) throw new InvalidOperationException("Owner server instance mismatch"); + var request=new Dictionary{{"session",(string)binding["generation"]},{"home",Environment.GetEnvironmentVariable("FM_PROBE_HOME")},{"nonce",Environment.GetEnvironmentVariable("FM_PROBE_NONCE")},{"case","owner-"+verb},{"kind","owner"},{"verb",verb},{"state",Canonical(state)},{"id",id}}; + using(var writer=new StreamWriter(pipe,new UTF8Encoding(false),1024,true)) + using(var reader=new StreamReader(pipe,Encoding.UTF8,false,1024,true)) { + writer.AutoFlush=true; writer.WriteLine(Json.Serialize(request)); + var answer=Json.Deserialize>(reader.ReadLine()); + if(!answer.ContainsKey("ownerAuthorized") || !(bool)answer["ownerAuthorized"]) { Console.Error.WriteLine("Owner operation refused"); return 2; } + string value=(string)answer["ownerValue"]; + if(verb=="identity" || verb=="harness") { Console.WriteLine(value); return 0; } + if(value=="unknown") { Console.Error.WriteLine("Owner liveness unknown"); return 2; } + return value=="true" ? 0 : 1; + } + } finally { CloseHandle(server); } + } + } +} diff --git a/bin/native-owner/codex-tool-gate.mjs b/bin/native-owner/codex-tool-gate.mjs new file mode 100644 index 00000000000..4abb4dc412b --- /dev/null +++ b/bin/native-owner/codex-tool-gate.mjs @@ -0,0 +1,91 @@ +// Request policy for a controller-owned Codex app-server connection. +// The host supplies protocol-envelope identity and a fixed operation adapter; +// model arguments never select a process, command, home, or thread. +export function createNotificationGate({ primaryThread, operate, isAlive }) { + if (typeof primaryThread !== 'string' || !primaryThread || + typeof operate !== 'function' || typeof isAlive !== 'function') { + throw new TypeError('A primary thread, operation adapter, and liveness source are required'); + } + let activeTurn = null; + let closed = false; + let busy = false; + let receipt = null; + let challenge = null; + let acknowledged = false; + const seen = new Set(); + const deny = reason => ({ success: false, value: { denied: reason } }); + const valid = params => !closed && isAlive() && params.threadId === primaryThread && + typeof params.turnId === 'string' && params.turnId === activeTurn; + + return Object.freeze({ + beginTurn(thread, turn) { + if (closed || thread !== primaryThread || typeof turn !== 'string' || !turn || activeTurn) { + throw new Error('Cannot register this turn'); + } + activeTurn = turn; + }, + endTurn(thread, turn) { + if (thread === primaryThread && turn === activeTurn) activeTurn = null; + }, + close() { + closed = true; + activeTurn = null; + }, + async handle(params) { + if (!params || !valid(params) || typeof params.callId !== 'string' || !params.callId) { + return deny('wrong-thread-turn-or-replay'); + } + const key = JSON.stringify([params.threadId, params.turnId, params.callId]); + if (seen.has(key)) return deny('wrong-thread-turn-or-replay'); + seen.add(key); + if (params.namespace != null) return deny('unexpected-namespace'); + const args = params.arguments; + if (!args || Array.isArray(args) || typeof args !== 'object') return deny('invalid-arguments'); + if (busy) return deny('operation-in-progress'); + + if (params.tool === 'fm_notification_check') { + if (Object.keys(args).length || receipt) return deny('check-already-used-or-invalid-arguments'); + } else if (params.tool === 'fm_notification_ack') { + if (Object.keys(args).sort().join(',') !== 'observed,receipt' || !receipt || + args.receipt !== receipt || args.observed !== challenge) { + return deny('wrong-receipt-or-unhandled-notification'); + } + if (acknowledged) return deny('receipt-already-consumed'); + } else { + return deny('unknown-tool'); + } + + busy = true; + try { + if (params.tool === 'fm_notification_check') { + const result = await operate('check'); + const note = result?.notification; + if (result?.operationState !== 'delivered' || !note || + typeof note.receipt !== 'string' || !note.receipt || + typeof note.challenge !== 'string' || !note.challenge || + typeof note.message !== 'string') { + throw new Error('Missing notification delivery'); + } + receipt = note.receipt; + challenge = note.challenge; + if (!valid(params)) return deny('wrong-thread-turn-or-replay'); + return { success: true, value: { + message: note.message, receipt, challenge, checkpointExit: note.checkpointExit, + } }; + } + const result = await operate('ack', { receipt, observed: args.observed }); + if (result?.operationState !== 'acknowledged') throw new Error('Acknowledgement did not complete'); + acknowledged = true; + if (!valid(params)) return deny('wrong-thread-turn-or-replay'); + return { success: true, value: { acknowledged: true } }; + } catch (error) { + // A failed mutation may have partially completed. Never retry it based + // on a missing response or manufacture a success; retain durable work. + closed = true; + return { success: false, value: { error: error.message } }; + } finally { + busy = false; + } + }, + }); +} diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index d0f84d25c7c..d7a87fc97ab 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -6,6 +6,41 @@ This record contains reusable version-scoped evidence for active runtime guarant The backend guides own current setup, safety boundaries, and limitations. Exact task chronology, branch names, temporary homes, local paths, process ids, thread ids, and delivery transcripts remain in private reports or PR evidence. +## Experimental native Windows ownership candidate + +Verified on 2026-09-15 with Codex app-server 0.154.0, Windows 10.0.26200 x86_64, and the saved unelevated Windows sandbox. +This is an isolated candidate verification, not an installed runtime backend or a claim that ordinary Codex shell tools can run Git Bash in that sandbox. +The native ownership core and Codex request policy live under `bin/native-owner/`; launchers, controlled messages, model prompts, Docker jq, and the temporary consumer integration patch remain test-only under `tests/fixtures/native-owner/`. +The home reservation deliberately still rejects non-temporary homes, and normal Firstmate startup does not load the candidate. + +Refresh the portable request-policy regression with: + +```sh +bash tests/fm-native-owner-tool-gate.test.sh +``` + +The 17 behavioral cases cover primary-thread/turn binding, duplicate calls, consumed receipts, invalid arguments, arbitrary-command rejection, dead connections, overlapping calls, late delivery, and operation failure without blind retry. +Refresh the actual Windows integration with the explicit two-model-turn guard: + +```sh +FM_LIVE_NATIVE_CODEX=1 bash tests/fm-native-owner-codex-live-e2e.test.sh +``` + +Observed terminal results: + +```text +PASS: model-free registered operation bridge. +PASS: real app-server notification cycle, handling, acknowledgement, replay refusal, and foreign-thread denial. +``` + +The actual primary thread read and acknowledged one controlled inbox notification after the startup operation expired. +A new fixed operation performed each check and acknowledgement through unchanged Firstmate command owners; the queue became empty and the note moved to handled. +A repeated receipt and a request from another real thread were refused without another native operation. +The checkpoint exercised its three-second timeout followed by a drain, not immediate interrupt-driven delivery. +Both app-server threads reported read-only filesystem policy, disabled network access, and approval policy `never`; the two explicitly authorized host operations execute outside ordinary model shell tools. +Dynamic tool registration requires the experimental API capability in this version. +This does not establish persistent receipt recovery, cancellation of an already accepted mutation, adversarial Windows path/process races, populated-fleet behavior, or other harness support. + ## Harness detection precedence Firstmate's own harness comes from two kinds of evidence, and `bin/fm-harness.sh` owns how they combine: an environment marker names its harness, and the nearest harness process in the parent chain proves who owns the process tree. diff --git a/tests/fixtures/native-owner/AppHost.mjs b/tests/fixtures/native-owner/AppHost.mjs new file mode 100644 index 00000000000..41150bf633a --- /dev/null +++ b/tests/fixtures/native-owner/AppHost.mjs @@ -0,0 +1,119 @@ +// Disposable host adapter. Only this process owns the app-server connection. +// Dynamic tool arguments never select a thread, executable, home, or command. +import {createNotificationGate} from './codex-tool-gate.mjs'; +import fs from 'node:fs'; +import path from 'node:path'; +import net from 'node:net'; +import {spawn} from 'node:child_process'; +import {createInterface} from 'node:readline'; +const home=process.env.FM_PROBE_HOME; +const executable=path.join(process.env.APPDATA,'npm/node_modules/@openai/codex/node_modules/@openai/codex-win32-x64/vendor/x86_64-pc-windows-msvc/bin/codex.exe'); +const evidence={frames:[],tools:[],native:[],primary:null,foreign:null,passed:false}; +const save=()=>fs.writeFileSync(path.join(home,'app-host-evidence.json'),JSON.stringify(evidence,null,2)); +const pause=ms=>new Promise(resolve=>setTimeout(resolve,ms)); +const base={kind:'notification',session:process.env.FM_PROBE_SESSION,home,nonce:process.env.FM_PROBE_NONCE}; +async function native(action,extra={}) { + const until=Date.now()+12000; + for(;;) { + try { + const result=await new Promise((resolve,reject)=>{ + const socket=net.createConnection('\\\\.\\pipe\\'+process.env.FM_PROBE_PIPE);let buffer='',done=false; + socket.setTimeout(9000,()=>socket.destroy(Error('Native operation query timed out'))); + socket.on('connect',()=>socket.write(JSON.stringify({...base,action,...extra})+'\n')); + socket.on('data',chunk=>{buffer+=chunk;const end=buffer.indexOf('\n');if(end>=0&&!done){done=true;try{resolve(JSON.parse(buffer.slice(0,end)));}catch(error){reject(error);}socket.end();}}); + socket.on('error',reject);socket.on('close',()=>{if(!done)reject(Error('Native channel closed without a result'));}); + }); + if(!result.notificationAuthorized)throw Error('Native controller denied host adapter'); + evidence.native.push({action,state:result.operationState,startupExpired:result.startupExpired}); + return result; + } catch(error) {if(!['EBUSY','ENOENT'].includes(error.code)||Date.now()>until)throw error;await pause(40);} + } +} +async function operation(action,extra={}) { + const start=await native(action,extra); + if(start.operationState!=='pending')throw Error('Operation did not start: '+start.operationState); + for(let i=0;i<700;i++) { + await pause(100);const result=await native('result'); + if(result.operationState==='failed')throw Error('Native notification command failed'); + if(result.operationState!=='pending')return result; + } + throw Error('Bounded operation exceeded 70 seconds'); +} +const child=spawn(executable,['app-server','--stdio','--disable','hooks','-c','windows.sandbox=unelevated','-c','model_reasoning_effort=high'],{cwd:home,stdio:['pipe','pipe','pipe']}); +let alive=true,next=0,stderr='',primary=null,receipt=null,challenge=null,acknowledged=false; +let gate=null; +const pending=new Map(),completed=new Map(); +const failPending=error=>{for(const value of pending.values())value.reject(error);pending.clear();}; +const timer=setTimeout(()=>{child.kill();process.exitCode=1;},220000); +child.stderr.on('data',chunk=>stderr+=chunk); +child.on('error',error=>{alive=false;failPending(error);}); +child.on('exit',code=>{alive=false;failPending(Error('App-server exited '+code));}); +const send=value=>{if(!alive)throw Error('App-server is no longer live');child.stdin.write(JSON.stringify(value)+'\n');}; +const request=(method,params)=>new Promise((resolve,reject)=>{const id=++next;pending.set(id,{resolve,reject,method,params});send({id,method,params});}); +async function tool(frame) { + const p=frame.params; + const record={requestId:frame.id,threadId:p.threadId,turnId:p.turnId,callId:p.callId,tool:p.tool,arguments:p.arguments}; + evidence.tools.push(record); + const respond=(success,value)=>{record.success=success;record.result=value;send({id:frame.id,result:{success,contentItems:[{type:'inputText',text:JSON.stringify(value)}]}});save();}; + if(!gate)return respond(false,{denied:'primary-not-registered'}); + const result=await gate.handle(p); + if(result.success&&p.tool==='fm_notification_check'){receipt=result.value.receipt;challenge=result.value.challenge;} + if(result.success&&p.tool==='fm_notification_ack')acknowledged=true; + respond(result.success,result.value); +} +createInterface({input:child.stdout}).on('line',line=>{ + let event;try{event=JSON.parse(line);}catch{alive=false;failPending(Error('Invalid app-server frame'));return;} + evidence.frames.push(event); + if(event.method==='item/tool/call'&&event.id!==undefined){tool(event).catch(error=>{evidence.fatal=error.message;save();child.kill();});return;} + if(event.id!==undefined&&pending.has(event.id)) { + const value=pending.get(event.id);pending.delete(event.id); + if(event.error)value.reject(Error(JSON.stringify(event.error))); + else {if(value.method==='turn/start'&&value.params.threadId===primary)gate.beginTurn(primary,event.result.turn.id);value.resolve(event.result);} + } else if(event.method==='turn/completed'){completed.set(event.params.turn.id,event.params.turn);gate?.endTurn(event.params.threadId,event.params.turn.id);} + else if(event.id!==undefined)send({id:event.id,error:{code:-32601,message:'No other host operations are authorized'}}); +}); +const tools=[ + {name:'fm_notification_check',description:'Read exactly one controlled notification through the registered Firstmate operation.',inputSchema:{type:'object',properties:{},additionalProperties:false}}, + {name:'fm_notification_ack',description:'After observing the notification, acknowledge its receipt and exact observed challenge. Receipts are single use.',inputSchema:{type:'object',properties:{receipt:{type:'string'},observed:{type:'string'}},required:['receipt','observed'],additionalProperties:false}}, +]; +async function turn(threadId,text) { + const started=await request('turn/start',{threadId,input:[{type:'text',text,text_elements:[]}]}); + for(let i=0;i<1500;i++){if(!alive)throw Error('App-server lost during turn');if(completed.has(started.turn.id)){const end=completed.get(started.turn.id);if(end.status!=='completed')throw Error('Turn failed: '+JSON.stringify(end));return started.turn.id;}await pause(100);} + throw Error('Turn exceeded bounded wait'); +} +try { + await request('initialize',{clientInfo:{name:'firstmate-scoped-notification-test',version:'0.0.0'},capabilities:{experimentalApi:true}}); + send({method:'initialized',params:{}}); + let ready=false; + for(let i=0;i<1200;i++){const state=await native('result');if(state.startupExpired){ready=true;break;}await pause(100);} + if(!ready||!fs.existsSync(path.join(home,'owner-operation.complete')))throw Error('Startup did not finish and expire'); + const params={cwd:home,model:'gpt-5.6-terra',sandbox:'read-only',approvalPolicy:'never',ephemeral:true,dynamicTools:tools}; + primary=(await request('thread/start',params)).thread.id;evidence.primary=primary; + gate=createNotificationGate({primaryThread:primary,operate:operation,isAlive:()=>alive}); + evidence.foreign=(await request('thread/start',params)).thread.id; + if(process.env.FM_PROBE_API_DRY==='1') { + const delivery=await operation('check'); + const ack=await operation('ack',{receipt:delivery.notification.receipt,observed:delivery.notification.challenge}); + const replay=await native('ack',{receipt:delivery.notification.receipt,observed:delivery.notification.challenge}); + if(ack.operationState!=='acknowledged'||replay.operationState!=='denied')throw Error('Native bridge preflight failed'); + evidence.passed=true;evidence.modelFree=true; + console.log('PASS: registered post-startup operations deliver and acknowledge using the native bridge; model-free only.'); + } else { + evidence.primaryTurn=await turn(primary,'This is a bounded integration test in an empty disposable Firstmate home. Use only the supplied fm_notification tools; do not use shell, file, browser, or other tools. Call fm_notification_check once. Read the message, then call fm_notification_ack with its receipt and the observed challenge. After successful acknowledgement, repeat that same acknowledgement exactly once to test replay rejection. Report the three results and stop. Do not retry anything else.'); + if(!acknowledged)throw Error('Primary did not acknowledge notification'); + const primaryCalls=evidence.tools.filter(row=>row.threadId===primary); + if(primaryCalls.length!==3||!primaryCalls[0].success||!primaryCalls[1].success||primaryCalls[2].result?.denied!=='receipt-already-consumed')throw Error('Primary/check/ack/replay sequence differed'); + // The native owner independently refuses a replay even from the trusted host. + const nativeReplay=await native('ack',{receipt,observed:challenge}); + if(nativeReplay.operationState!=='denied')throw Error('Native owner accepted replay'); + evidence.nativeReplayDenied=true; + evidence.foreignTurn=await turn(evidence.foreign,'This is a deliberate authorization negative control. Call fm_notification_check exactly once with no arguments. It should be denied because this is not the registered primary thread. Do not use any other tool, retry, or change settings. Report the result and stop.'); + const foreignCalls=evidence.tools.filter(row=>row.threadId===evidence.foreign); + if(foreignCalls.length!==1||foreignCalls[0].success||foreignCalls[0].result?.denied!=='wrong-thread-turn-or-replay')throw Error('Foreign thread rejection not demonstrated'); + evidence.passed=true;evidence.receipt=receipt;evidence.challenge=challenge; + console.log('PASS: primary received and acknowledged the notification; receipt replay and a second real thread were denied.'); + } +} catch(error) {evidence.fatal=error.stack;process.exitCode=1;console.error(error.stack);} finally { + save();fs.writeFileSync(path.join(home,'app-server.stderr'),stderr);child.stdin.end(); + const closeTimer=setTimeout(()=>child.kill(),3000);child.once('exit',()=>{clearTimeout(closeTimer);clearTimeout(timer);});if(!alive){clearTimeout(closeTimer);clearTimeout(timer);} +} diff --git a/tests/fixtures/native-owner/Build.ps1 b/tests/fixtures/native-owner/Build.ps1 new file mode 100644 index 00000000000..8ab6bc1f38d --- /dev/null +++ b/tests/fixtures/native-owner/Build.ps1 @@ -0,0 +1,26 @@ +# Build an isolated test assembly and home; never install the native provider. +# Usage: powershell -NoProfile -File tests/fixtures/native-owner/Build.ps1 +# Requires native Git symlinks and the pre-existing Docker validation image. +$ErrorActionPreference = 'Stop' +$env:MSYS = 'winsymlinks:nativestrict' +$repo = [IO.Path]::GetFullPath((Join-Path $PSScriptRoot '../../..')) +$root = Join-Path ([IO.Path]::GetTempPath()) ('fm-native-candidate-' + [guid]::NewGuid().ToString('N')) +New-Item -ItemType Directory $root | Out-Null +$binary = Join-Path $root 'SessionProbe.exe' +$sources = @((Join-Path $repo 'bin/native-owner/NativeOwner.cs'), (Join-Path $repo 'bin/native-owner/NativeHomeLease.cs'), (Join-Path $PSScriptRoot 'NativeDriver.cs')) +Add-Type -Path $sources -OutputAssembly $binary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll +$copy = Join-Path $root 'firstmate' +& git -c core.symlinks=true clone --quiet --no-local --single-branch $repo $copy +if ($LASTEXITCODE -ne 0) { throw 'Disposable clone failed' } +& git -C $copy apply --whitespace=error (Join-Path $PSScriptRoot 'consumer-adapter.patch') +if ($LASTEXITCODE -ne 0) { throw 'Consumer test integration no longer applies; reconcile it explicitly' } +foreach ($name in @('exercise.sh','notification-check.sh','notification-ack.sh')) { Copy-Item (Join-Path $PSScriptRoot $name) (Join-Path $copy $name) } +Copy-Item (Join-Path $PSScriptRoot 'AppHost.mjs') (Join-Path $root 'AppHost.mjs') +Copy-Item (Join-Path $repo 'bin/native-owner/codex-tool-gate.mjs') (Join-Path $root 'codex-tool-gate.mjs') +Copy-Item $binary (Join-Path $copy 'bin/fm-native-owner.exe') +New-Item -ItemType Directory (Join-Path $root 'tools') | Out-Null +Copy-Item (Join-Path $PSScriptRoot 'jq') (Join-Path $root 'tools/jq') +$state = Join-Path $repo 'data/native-candidate-validation' +New-Item -ItemType Directory -Force $state | Out-Null +@{ root=$root; binary=$binary; repo=$repo; hashes=@($sources | ForEach-Object { @{ file=$_; hash=(Get-FileHash $_).Hash } }) } | ConvertTo-Json -Depth 5 | Set-Content -Encoding UTF8 (Join-Path $state 'build.json') +Write-Output $binary diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs new file mode 100644 index 00000000000..3eae4fd9aeb --- /dev/null +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -0,0 +1,362 @@ +// Test-only launchers, command dispatch, ACL experiments, and lifecycle controls. +using System; +using System.Collections; +using System.Collections.Generic; +using System.ComponentModel; +using System.Diagnostics; +using System.IO; +using System.IO.Pipes; +using System.Runtime.InteropServices; +using System.Security.AccessControl; +using System.Security.Principal; +using System.Text; +using System.Threading; +using System.Web.Script.Serialization; + +public static partial class NativeOwner { + static string LogonSid() { + using(var identity=WindowsIdentity.GetCurrent()) { + foreach(var row in TokenGroups(identity.Token,2)) + if (((uint)row["attributes"] & 0xc0000000U)==0xc0000000U) return (string)row["sid"]; + } + throw new InvalidOperationException("No kernel-marked logon SID available"); + } + static Dictionary TokenFacts() { + using(var identity=WindowsIdentity.GetCurrent()) return new Dictionary { + {"kind","token-facts"},{"pid",Process.GetCurrentProcess().Id},{"user",identity.User.Value}, + {"groups",TokenGroups(identity.Token,2)},{"restricting",TokenGroups(identity.Token,11)} + }; + } + static IntPtr FileHandle(string path, uint access, uint creation) { + SA sa = new SA { length=Marshal.SizeOf(typeof(SA)), inherit=1 }; + IntPtr h = CreateFile(path, access, 3, ref sa, creation, 0x80, IntPtr.Zero); + if (h == new IntPtr(-1)) throw Error("CreateFile"); + return h; + } + static SortedDictionary EnvironmentFor(string pipe, string session, string home, string nonce) { + var result = new SortedDictionary(StringComparer.OrdinalIgnoreCase); + foreach (DictionaryEntry e in Environment.GetEnvironmentVariables()) { + string k = (string)e.Key; + if (k.StartsWith("FM_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("PI_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("NO_MISTAKES", StringComparison.OrdinalIgnoreCase) || k == "CLAUDE_PID" || k == "CLAUDECODE") continue; + result[k] = (string)e.Value; + } + result["FM_PROBE_PIPE"] = pipe; result["FM_PROBE_SESSION"] = session; + result["FM_PROBE_HOME"] = home; result["FM_PROBE_NONCE"] = nonce; + result["FM_PROBE_EXE"] = OwnExe; + return result; + } + static ChildScope StartScope(string role, IntPtr parentJob, IntPtr environment, string home, ref SI startup, string purpose="startup") { + var scope=new ChildScope { job=CreateJobObject(IntPtr.Zero,null), role=role, purpose=purpose }; + IntPtr operationEnvironment=IntPtr.Zero; + if(scope.job==IntPtr.Zero) throw Error("CreateJobObject child scope"); + try { + string operation=role=="owner-operation" ? (purpose=="startup" ? "owner-operation" : "notification-operation "+purpose) : "scoped-client "+role; + if(role=="owner-operation") { + // Only this fixed, non-extensible test operation receives the grant. + var values=new SortedDictionary(StringComparer.OrdinalIgnoreCase); + foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { + string value=Environment.GetEnvironmentVariable(key); if(value!=null) values[key]=value; + } + values["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); + // Copy only the controller's registration fields from the prepared block. + int offset=0; + while(Marshal.ReadInt16(environment,offset)!=0) { + string entry=Marshal.PtrToStringUni(IntPtr.Add(environment,offset)); offset+=(entry.Length+1)*2; + int split=entry.IndexOf('='); if(split<=0) continue; + string key=entry.Substring(0,split); + if(key.StartsWith("FM_PROBE_",StringComparison.Ordinal) || key=="FM_HOME" || key=="MSYS") values[key]=entry.Substring(split+1); + } + var block=new StringBuilder(); foreach(var value in values) block.Append(value.Key).Append('=').Append(value.Value).Append('\0'); block.Append('\0'); + operationEnvironment=Marshal.StringToHGlobalUni(block.ToString()); + } + if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" "+operation),IntPtr.Zero,IntPtr.Zero,true,0x4|0x400,operationEnvironment==IntPtr.Zero ? environment : operationEnvironment,home,ref startup,out scope.process)) throw Error("CreateProcess child scope"); + if(!AssignProcessToJobObject(parentJob,scope.process.process) || !AssignProcessToJobObject(scope.job,scope.process.process)) throw Error("Assign child scope"); + // The caller records this scope before resuming the process. + return scope; + } catch { + if(scope.process.process!=IntPtr.Zero) { TerminateProcess(scope.process.process,125); CloseHandle(scope.process.thread); CloseHandle(scope.process.process); } + CloseHandle(scope.job); throw; + } finally { if(operationEnvironment!=IntPtr.Zero) Marshal.FreeHGlobal(operationEnvironment); } + } + static void ReleaseFixtureWhenScopesEnd(List scopes,string home) { + if(scopes.Count==0) return; + foreach(var scope in scopes) if(WaitForSingleObject(scope.process.process,0)==WAIT_TIMEOUT) return; + string done=Path.Combine(home,"boundaries-done"); + if(!File.Exists(done)) File.WriteAllText(done,"complete"); + } + static int Client(string which) { + Console.WriteLine(Json.Serialize(TokenFacts())); + var request = new Dictionary { + {"session",Environment.GetEnvironmentVariable("FM_PROBE_SESSION")}, + {"home",Environment.GetEnvironmentVariable("FM_PROBE_HOME")}, + {"nonce",Environment.GetEnvironmentVariable("FM_PROBE_NONCE")}, {"case",which}, {"claimedRole","primary"} + }; + if (which == "wrong-session") request["session"] = Guid.NewGuid().ToString("N"); + if (which == "wrong-home") request["home"] = "C:\\not-the-test-home"; + if (which == "wrong-capability") request["nonce"] = "copied-invalid-value"; + using (var pipe = new NamedPipeClientStream(".", Environment.GetEnvironmentVariable("FM_PROBE_PIPE"), PipeAccessRights.ReadData | PipeAccessRights.WriteData | PipeAccessRights.Synchronize, PipeOptions.None, TokenImpersonationLevel.Identification, HandleInheritability.None)) { + pipe.Connect(8000); + using (var writer = new StreamWriter(pipe, new UTF8Encoding(false), 1024, true)) + using (var reader = new StreamReader(pipe, Encoding.UTF8, false, 1024, true)) { + writer.AutoFlush = true; writer.WriteLine(Json.Serialize(request)); + string line = reader.ReadLine(); + if (line == null) throw new IOException("No association response"); + Console.WriteLine(line); + } + } + return 0; + } + static void RunFirstmate(string role,bool shouldSucceed) { + string root=Path.Combine(Path.GetDirectoryName(OwnExe),"firstmate"); + string script=Path.Combine(root,shouldSucceed ? "exercise.sh" : "bin/fm-lock.sh").Replace('\\','/'); + using(var p=Process.Start(new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script)) { UseShellExecute=false,RedirectStandardOutput=!shouldSucceed,RedirectStandardError=!shouldSucceed })) { + var stdout=shouldSucceed ? System.Threading.Tasks.Task.FromResult("") : p.StandardOutput.ReadToEndAsync(); var stderr=shouldSucceed ? System.Threading.Tasks.Task.FromResult("") : p.StandardError.ReadToEndAsync(); + if(!p.WaitForExit(240000)) { p.Kill(); throw new IOException("Bounded Firstmate test timed out"); } + File.WriteAllText(Path.Combine(Environment.GetEnvironmentVariable("FM_PROBE_HOME"),"firstmate-"+role+".json"),Json.Serialize(new Dictionary{{"exit",p.ExitCode},{"stdout",stdout.Result},{"stderr",stderr.Result}})); + if((p.ExitCode==0)!=shouldSucceed) throw new IOException("Unexpected Firstmate operation result for "+role); + } + } + static int Fixture() { + foreach (string c in new [] {"root", "wrong-session", "wrong-home", "wrong-capability"}) Client(c); + var child = Process.Start(new ProcessStartInfo(OwnExe, "client inherited-child") { UseShellExecute=false }); + child.WaitForExit(); if (child.ExitCode != 0) return child.ExitCode; + // Exercise MSYS exec without trying to infer its disappearing ancestors. + var bash = Process.Start(new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe", "-c " + Quote("exec \"$FM_PROBE_EXE\" client msys-exec")) { UseShellExecute=false }); + bash.WaitForExit(); + if(Environment.GetEnvironmentVariable("FM_PROBE_BOUNDARIES")=="1") { + DateTime deadline=DateTime.UtcNow.AddSeconds(15); + string done=Path.Combine(Environment.GetEnvironmentVariable("FM_PROBE_HOME"),"boundaries-done"); + while(!File.Exists(done) && DateTime.UtcNow>(File.ReadAllText(configPath)); + string home = Path.GetFullPath((string)config["home"]); + string executable = (string)config["executable"], arguments = (string)config["arguments"]; + int seconds = Convert.ToInt32(config["timeoutSeconds"]); + if(config.ContainsKey("registeredHarness")) { + string expected=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),@"npm\node_modules\@openai\codex\node_modules\@openai\codex-win32-x64\vendor\x86_64-pc-windows-msvc\bin\codex.exe"); + string host=Path.Combine(Path.GetDirectoryName(OwnExe),"AppHost.mjs"); + string node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); + bool direct=(string)config["registeredHarness"]=="codex" && string.Equals(Path.GetFullPath(executable),Path.GetFullPath(expected),StringComparison.OrdinalIgnoreCase); + bool adapter=(string)config["registeredHarness"]=="codex-app-server" && string.Equals(Path.GetFullPath(executable),Path.GetFullPath(node),StringComparison.OrdinalIgnoreCase) && arguments==Quote(host); + if(!direct && !adapter) throw new InvalidOperationException("Requested runtime does not match the fixed native Codex launch target"); + RegisteredHarness="codex"; + } + Directory.CreateDirectory(home); + string session = Guid.NewGuid().ToString("N"), pipeName = "fm-private-probe-" + session, nonce = Guid.NewGuid().ToString("N"); + IntPtr job = CreateJobObject(IntPtr.Zero, null); + if (job == IntPtr.Zero) throw Error("CreateJobObject"); + IntPtr env = IntPtr.Zero, stdout=IntPtr.Zero, stderr=IntPtr.Zero, stdin=IntPtr.Zero; + PI child = new PI(); bool assigned=false, timedOut=false; + var observations = new List>(); + Process outsider = null; + var scopes=new List(); + NativeHomeLease lease=null; + try { + if(config.ContainsKey("leaseHome")) lease=new NativeHomeLease((string)config["leaseHome"]); + var values = EnvironmentFor(pipeName, session, home, nonce); + values["MSYS"]="winsymlinks:nativestrict"; + if(config.ContainsKey("apiDry") && (bool)config["apiDry"]) values["FM_PROBE_API_DRY"]="1"; + if(lease!=null) { values["FM_PROBE_LEASE_HOME"]=lease.Home; values["FM_PROBE_LEASE_GENERATION"]=session; values["FM_HOME"]=lease.Home.Replace('\\','/'); } + if(config.ContainsKey("ownerExercise") && (bool)config["ownerExercise"]) values["FM_PROBE_EXERCISE"]="1"; + if(config.ContainsKey("boundaries") && (bool)config["boundaries"]) values["FM_PROBE_BOUNDARIES"]="1"; + var block = new StringBuilder(); foreach (var e in values) block.Append(e.Key).Append('=').Append(e.Value).Append('\0'); block.Append('\0'); + env = Marshal.StringToHGlobalUni(block.ToString()); + stdout = FileHandle(Path.Combine(home,"stdout.log"),0x40000000,2); + stderr = FileHandle(Path.Combine(home,"stderr.log"),0x40000000,2); + stdin = FileHandle("NUL",0x80000000,3); + SI startup = new SI { cb=Marshal.SizeOf(typeof(SI)), flags=0x100, input=stdin, output=stdout, error=stderr }; + if (!CreateProcess(executable, new StringBuilder(Quote(executable)+" "+arguments), IntPtr.Zero, IntPtr.Zero, true, 0x4 | 0x400, env, home, ref startup, out child)) throw Error("CreateProcess suspended"); + if (!config.ContainsKey("assignJob") || (bool)config["assignJob"]) { + if (!AssignProcessToJobObject(job,child.process)) throw Error("AssignProcessToJobObject"); + assigned=true; + } + FT born, exited, kernel, user; + if (!GetProcessTimes(child.process,out born,out exited,out kernel,out user)) throw Error("GetProcessTimes"); + if(lease!=null) lease.Publish(child.pid,((ulong)born.high<<32)|born.low,session,pipeName); + if(config.ContainsKey("ownerOperation") && (bool)config["ownerOperation"]) { + var operation=StartScope("owner-operation",job,env,home,ref startup); + scopes.Add(operation); + if(ResumeThread(operation.process.thread)==0xffffffff) throw Error("Resume owner operation"); + } + if(values.ContainsKey("FM_PROBE_BOUNDARIES")) { + foreach(string role in new [] {"worker","nested-primary"}) scopes.Add(StartScope(role,job,env,home,ref startup)); + foreach(var scope in scopes) if(ResumeThread(scope.process.thread)==0xffffffff) throw Error("Resume child scope"); + } + if (ResumeThread(child.thread) == 0xffffffff) throw Error("ResumeThread"); + if(config.ContainsKey("abandonAfterLaunch") && (bool)config["abandonAfterLaunch"]) { + if(executable!=OwnExe || !arguments.StartsWith("sleep ",StringComparison.Ordinal)) throw new InvalidOperationException("Abandon test only permits bounded disposable sleepers"); + Console.WriteLine("DISPOSABLE_CONTROLLER_EXIT"); + Environment.Exit(86); + } + DateTime deadline=DateTime.UtcNow.AddSeconds(seconds); + var security = new PipeSecurity(); security.SetAccessRuleProtection(true,false); + security.AddAccessRule(new PipeAccessRule(WindowsIdentity.GetCurrent().User, PipeAccessRights.FullControl, AccessControlType.Allow)); + string pipeAcl=config.ContainsKey("pipeAcl") ? (string)config["pipeAcl"] : "UserOnly"; + if(pipeAcl=="LogonData") { + // Only this ephemeral endpoint gains data exchange rights for + // this kernel-authenticated Windows logon, never Everyone or + // server-instance creation, ACL changes, or filesystem rights. + security.AddAccessRule(new PipeAccessRule(new SecurityIdentifier(LogonSid()), PipeAccessRights.ReadData | PipeAccessRights.WriteData | PipeAccessRights.Synchronize, AccessControlType.Allow)); + } else if(pipeAcl!="UserOnly") throw new InvalidOperationException("Unknown pipe ACL test mode"); + bool outsiderStarted=false; + while (WaitForSingleObject(child.process,0) == WAIT_TIMEOUT && DateTime.UtcNow < deadline) { + using (var server = new NamedPipeServerStream(pipeName, PipeDirection.InOut, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous, 4096, 4096, security)) { + var pending = server.BeginWaitForConnection(null,null); + if (!outsiderStarted && config.ContainsKey("outsider") && (bool)config["outsider"]) { + var info = new ProcessStartInfo(OwnExe,"client copied-outsider") { UseShellExecute=false, RedirectStandardOutput=true, RedirectStandardError=true }; + foreach (var e in values) info.EnvironmentVariables[e.Key]=e.Value; + outsider=Process.Start(info); outsiderStarted=true; + } + while (!pending.AsyncWaitHandle.WaitOne(50) && WaitForSingleObject(child.process,0) == WAIT_TIMEOUT && DateTime.UtcNow < deadline) { ReleaseFixtureWhenScopesEnd(scopes,home); } + if (!pending.IsCompleted) break; + server.EndWaitForConnection(pending); + uint clientPid; + if (!GetNamedPipeClientProcessId(server.SafePipeHandle.DangerousGetHandle(), out clientPid)) throw Error("GetNamedPipeClientProcessId"); + using (var reader = new StreamReader(server,Encoding.UTF8,false,1024,true)) + using (var writer = new StreamWriter(server,new UTF8Encoding(false),1024,true)) { + var read = reader.ReadLineAsync(); + if (!read.Wait(8000)) throw new IOException("Client request timeout"); + if (read.Result == null || read.Result.Length > 4096) throw new IOException("Invalid request"); + var request=Json.Deserialize>(read.Result); + var verdict=Verdict(request,clientPid,child.process,job,child.pid,scopes,session,home,nonce); + if(request.ContainsKey("kind") && (string)request["kind"]=="owner") AuthorizeOwner(request,verdict,server,lease,session); + if(request.ContainsKey("kind") && (string)request["kind"]=="notification") NotificationRequest(request,verdict,lease,job,env,home,ref startup,scopes); + observations.Add(verdict); writer.AutoFlush=true; writer.WriteLine(Json.Serialize(verdict)); + } + } + } + timedOut=WaitForSingleObject(child.process,0)==WAIT_TIMEOUT; + if (timedOut) { + if (assigned) TerminateJobObject(job,124); else TerminateProcess(child.process,124); + WaitForSingleObject(child.process,5000); + } + uint code; if (!GetExitCodeProcess(child.process,out code)) throw Error("GetExitCodeProcess"); + var stale=new Dictionary{{"session",session},{"home",home},{"nonce",nonce},{"case","after-root-exit"}}; + observations.Add(Verdict(stale,(uint)Process.GetCurrentProcess().Id,child.process,job,child.pid,scopes,session,home,nonce)); + var result = new Dictionary { + {"rootPid",child.pid},{"registeredHarness",RegisteredHarness},{"rootCreationFileTime",((ulong)born.high<<32)|born.low}, + {"rootExit",code},{"timedOut",timedOut},{"jobAssigned",assigned},{"pipeAcl",pipeAcl}, + {"probeGeneration",session},{"probeLeaseHeld",lease!=null}, + {"pipeDacl",security.GetSecurityDescriptorSddlForm(AccessControlSections.Access)}, + {"authorityImplemented",false},{"notificationCheckStarts",checkStarts},{"notificationAckStarts",ackStarts},{"notificationConsumed",consumed},{"observations",observations} + }; + if (outsider != null) { + if (!outsider.WaitForExit(10000)) throw new IOException("Outsider probe did not stop"); + result["outsiderExit"]=outsider.ExitCode; result["outsiderOutput"]=outsider.StandardOutput.ReadToEnd(); result["outsiderError"]=outsider.StandardError.ReadToEnd(); + } + File.WriteAllText(Path.Combine(home,"result.json"),Json.Serialize(result)); + Console.WriteLine(Json.Serialize(result)); + return timedOut ? 124 : (int)code; + } finally { + // Only this disposable probe's retained native handles are eligible. + // Never use a PID lookup to stop an unrelated or reused process. + if (child.process != IntPtr.Zero && WaitForSingleObject(child.process,0)==WAIT_TIMEOUT) { + if (assigned) TerminateJobObject(job,125); else TerminateProcess(child.process,125); + } + if(config.ContainsKey("ownerExercise") && (bool)config["ownerExercise"]) TerminateJobObject(job,125); + foreach(var scope in scopes) { + if(WaitForSingleObject(scope.process.process,0)==WAIT_TIMEOUT) TerminateJobObject(scope.job,125); + CloseHandle(scope.process.thread); CloseHandle(scope.process.process); CloseHandle(scope.job); + } + foreach (IntPtr h in new [] {child.thread,child.process,stdout,stderr,stdin,job}) if (h!=IntPtr.Zero) CloseHandle(h); + if(env!=IntPtr.Zero) Marshal.FreeHGlobal(env); + if(lease!=null) lease.Dispose(); + } + } + static void NotificationRequest(Dictionary request,Dictionary verdict,NativeHomeLease lease,IntPtr job,IntPtr environment,string home,ref SI startup,List scopes) { + bool allowed=lease!=null && (string)verdict["association"]=="associated" && (string)verdict["hostClassification"]=="registered-primary"; + verdict["notificationAuthorized"]=allowed; + if(!allowed) return; + bool ready=true; + foreach(var scope in scopes) if(scope.purpose=="startup" && WaitForSingleObject(scope.process.process,0)==WAIT_TIMEOUT) ready=false; + verdict["startupExpired"]=ready; + if(pendingOperation!=null && WaitForSingleObject(pendingOperation.process.process,0)!=WAIT_TIMEOUT) { + uint code; if(!GetExitCodeProcess(pendingOperation.process.process,out code)) throw Error("Operation exit"); + verdict["operationExit"]=code; + if(code!=0) { verdict["operationState"]="failed"; return; } + string file=Path.Combine(home,"notification-"+pendingOperation.purpose+".json"); + var output=Json.Deserialize>(File.ReadAllText(file)); + if(pendingOperation.purpose=="check") { + delivered=output; receipt=Guid.NewGuid().ToString("N"); + delivered["receipt"]=receipt; + } else consumed=true; + pendingOperation=null; + } + string action=request.ContainsKey("action") ? (string)request["action"] : ""; + if(action=="result") { + verdict["operationState"]=pendingOperation!=null ? "pending" : consumed ? "acknowledged" : delivered!=null ? "delivered" : "idle"; + if(delivered!=null) verdict["notification"]=delivered; + return; + } + if(!ready || pendingOperation!=null) { verdict["operationState"]="busy"; return; } + if(action=="check" && checkStarts==0) checkStarts++; + else if(action=="ack" && delivered!=null && !consumed && ackStarts==0 && request.ContainsKey("receipt") && (string)request["receipt"]==receipt && request.ContainsKey("observed") && (string)request["observed"]==(string)delivered["challenge"]) { + ackStarts++; + File.WriteAllText(Path.Combine(home,"notification-ack-request.json"),Json.Serialize(delivered)); + } else { verdict["operationState"]="denied"; return; } + pendingOperation=StartScope("owner-operation",job,environment,home,ref startup,action); + scopes.Add(pendingOperation); + if(ResumeThread(pendingOperation.process.thread)==0xffffffff) throw Error("Resume notification operation"); + verdict["operationState"]="pending"; + } + public static int Main(string[] args) { + try { + if(args.Length>=3 && args[0]=="owner") return OwnerClient(args[1],args[2],args.Length>3 ? args[3] : ""); + if(args.Length>0 && args[0]=="client") return Client(args.Length>1 ? args[1] : "agent-tool"); + if(args.Length==2 && args[0]=="sleep") { int ms=int.Parse(args[1]); if(ms<0 || ms>15000) throw new ArgumentException("Sleep must be bounded"); Thread.Sleep(ms); return 0; } + if(args.Length==3 && args[0]=="lease-check") { Console.WriteLine(NativeHomeLease.Check(args[1],args[2])); return 0; } + if(args.Length==2 && args[0]=="notification-operation") { + if(args[1]!="check" && args[1]!="ack") throw new ArgumentException("Unsupported notification operation"); + string script=Path.Combine(Path.GetDirectoryName(OwnExe),"firstmate","notification-"+args[1]+".sh"); + using(var operation=Process.Start(new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script)) {UseShellExecute=false})) { + if(!operation.WaitForExit(60000)) throw new IOException("Notification operation exceeded its bound"); + return operation.ExitCode; + } + } + if(args.Length>0 && args[0]=="owner-operation") { + RunFirstmate("owner-operation",true); + string state=Path.Combine(Environment.GetEnvironmentVariable("FM_HOME"),"state"); + int live=OwnerClient("alive",state,"native:"+Environment.GetEnvironmentVariable("FM_PROBE_SESSION")); + int unknown=OwnerClient("alive",state,"native:00000000000000000000000000000000"); + if(live!=0 || unknown!=2) throw new IOException("Liveness classification failed"); + int verb=OwnerClient("unregistered-verb",state,""); + if(verb!=2) throw new IOException("Unknown verb granted"); + File.WriteAllText(Path.Combine(Environment.GetEnvironmentVariable("FM_PROBE_HOME"),"owner-operation.complete"),Json.Serialize(new {live=live,unknown=unknown,forbiddenVerb=verb})); + return 0; + } + if(args.Length>0 && args[0]=="bounded-fixture") { + string home=Environment.GetEnvironmentVariable("FM_PROBE_HOME"); + DateTime limit=DateTime.UtcNow.AddSeconds(260); + while(!File.Exists(Path.Combine(home,"owner-operation.complete")) && DateTime.UtcNow>(File.ReadAllText(outcome)); + if(Convert.ToInt32(recorded["exit"])!=0) throw new IOException("Real startup operation failed; see startup.log"); + } + Thread.Sleep(25); + } + if(!File.Exists(Path.Combine(home,"owner-operation.complete"))) throw new IOException("Owner operation did not complete"); + RunFirstmate("unregistered",false); + while(!File.Exists(Path.Combine(home,"boundaries-done")) && DateTime.UtcNow0 && args[0]=="fixture") return Fixture(); + if(args.Length==2 && args[0]=="owner-fixture") { Client("scope-"+args[1]); RunFirstmate(args[1],false); return 0; } + if(args.Length==2 && args[0]=="scoped-client") { + Client("scope-"+args[1]); + bool exercise=Environment.GetEnvironmentVariable("FM_PROBE_EXERCISE")=="1"; + if(exercise) RunFirstmate(args[1],false); + var descendant=Process.Start(new ProcessStartInfo(OwnExe,(exercise ? "owner-fixture " : "client scope-")+args[1]+"-descendant") { UseShellExecute=false }); + descendant.WaitForExit(); return descendant.ExitCode; + } + if(args.Length==2 && args[0]=="run") return Run(args[1]); + Console.Error.WriteLine("usage: NativeOwner run | client [case] | fixture"); return 2; + } catch(Exception e) { Console.Error.WriteLine(e.ToString()); return 2; } + } +} diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs new file mode 100644 index 00000000000..b8fe608d847 --- /dev/null +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -0,0 +1,27 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import {fileURLToPath} from 'node:url'; +import {spawnSync} from 'node:child_process'; +import {randomUUID,createHash} from 'node:crypto'; +const repo=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..'); +const dir=path.join(repo,'data/native-candidate-validation'); +const read=file=>JSON.parse(fs.readFileSync(file,'utf8').replace(/^\uFEFF/,'')); +const build=read(path.join(dir,'build.json')); +const dry=process.argv.includes('--dry'); +if(!dry&&process.env.FM_LIVE_NATIVE_CODEX!=='1')throw Error('Live model test requires FM_LIVE_NATIVE_CODEX=1'); +if(!dry){const preflight=read(path.join(dir,'bridge-preflight.json'));if(!preflight.passed||preflight.binaryHash!==createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex'))throw Error('Run model-free bridge preflight for this binary first');} +const home=path.join(build.root,'appserver-'+randomUUID());fs.mkdirSync(home); +const script=path.join(build.root,'AppHost.mjs'); +const spec={home,leaseHome:path.join(home,'home'),executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'UserOnly',apiDry:dry}; +const file=path.join(home,'spec.json');fs.writeFileSync(file,JSON.stringify(spec,null,2)); +const run=spawnSync(build.binary,['run',file],{encoding:'utf8',timeout:310000}); +fs.writeFileSync(path.join(home,'controller.stdout'),run.stdout||'');fs.writeFileSync(path.join(home,'controller.stderr'),run.stderr||''); +fs.writeFileSync(path.join(dir,dry?'bridge-latest.json':'cycle-latest.json'),JSON.stringify({home,exit:run.status},null,2)); +console.log(JSON.stringify({home,exit:run.status,dry})); +if(run.status!==0)throw Error('Bounded app-server run failed; inspect evidence, do not start another model attempt'); +const host=read(path.join(home,'app-host-evidence.json')),native=read(path.join(home,'result.json')); +if(!host.passed||native.notificationCheckStarts!==1||native.notificationAckStarts!==1||!native.notificationConsumed)throw Error('Notification cycle incomplete'); +if(!host.native.filter(row=>row.action==='check'||row.action==='ack').every(row=>row.startupExpired))throw Error('Startup scope still active'); +if(fs.existsSync(path.join(spec.leaseHome,'state/.wake-queue'))&&fs.readFileSync(path.join(spec.leaseHome,'state/.wake-queue'),'utf8').trim())throw Error('Queue was not acknowledged'); +if(dry)fs.writeFileSync(path.join(dir,'bridge-preflight.json'),JSON.stringify({passed:true,home,binaryHash:createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex')},null,2)); +console.log(dry?'PASS: model-free registered operation bridge.':'PASS: real app-server notification cycle, handling, acknowledgement, replay refusal, and foreign-thread denial.'); diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs new file mode 100644 index 00000000000..ab8f45bee7b --- /dev/null +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -0,0 +1,48 @@ +// Verify captured protocol and durable command outcomes without another model turn. +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import {fileURLToPath} from 'node:url'; +import {createHash} from 'node:crypto'; +const repo=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..'); +const state=path.join(repo,'data/native-candidate-validation'); +const read=file=>JSON.parse(fs.readFileSync(file,'utf8').replace(/^\uFEFF/,'')); +const hash=file=>createHash('sha256').update(fs.readFileSync(file)).digest('hex'); +const build=read(path.join(state,'build.json')); +const latest=read(path.join(state,'cycle-latest.json')); +const home=latest.home; +const host=read(path.join(home,'app-host-evidence.json')); +const native=read(path.join(home,'result.json')); +assert.equal(latest.exit,0);assert.equal(native.rootExit,0);assert.equal(host.passed,true); +assert.notEqual(host.primary,host.foreign);assert.equal(host.tools.length,4); +const requests=host.frames.filter(frame=>frame.method==='item/tool/call'); +assert.equal(requests.length,4); +for(const tool of host.tools)assert.ok(requests.some(frame=>frame.id===tool.requestId&&frame.params.threadId===tool.threadId&&frame.params.turnId===tool.turnId&&frame.params.callId===tool.callId&&JSON.stringify(frame.params.arguments)===JSON.stringify(tool.arguments))); +assert.equal(host.frames.filter(frame=>frame.method==='turn/completed'&&frame.params.turn.status==='completed').length,2); +const primary=host.tools.filter(tool=>tool.threadId===host.primary); +assert.deepEqual(primary.map(tool=>tool.success),[true,true,false]); +assert.equal(primary[1].arguments.receipt,primary[0].result.receipt); +assert.equal(primary[1].arguments.observed,primary[0].result.challenge); +assert.equal(primary[2].result.denied,'receipt-already-consumed'); +assert.equal(host.tools.find(tool=>tool.threadId===host.foreign).result.denied,'wrong-thread-turn-or-replay'); +assert.equal(host.nativeReplayDenied,true); +assert.equal(native.notificationCheckStarts,1);assert.equal(native.notificationAckStarts,1);assert.equal(native.notificationConsumed,true); +assert.ok(host.native.filter(row=>row.action==='check'||row.action==='ack').every(row=>row.startupExpired)); +const delivered=read(path.join(home,'notification-check.json')); +const acknowledgement=read(path.join(home,'notification-ack-request.json')); +for(const key of ['seq','generation','note','challenge'])assert.equal(acknowledgement[key],delivered[key]); +assert.equal(delivered.challenge,primary[1].arguments.observed); +assert.ok(fs.readFileSync(path.join(home,'cycle-delivery.log'),'utf8').includes(`--ack-through ${delivered.seq} --recovery-generation ${delivered.generation}`)); +assert.equal(read(path.join(home,'notification-ack.json')).acknowledged,true); +const operational=path.join(home,'home','state'); +assert.equal(fs.readFileSync(path.join(operational,'.wake-queue'),'utf8').trim(),''); +assert.ok(fs.existsSync(path.join(operational,'inbox/handled',delivered.note+'.note'))); +const threads=host.frames.filter(frame=>frame.result?.thread); +assert.equal(threads.length,2); +for(const thread of threads){assert.equal(thread.result.sandbox.type,'readOnly');assert.equal(thread.result.sandbox.networkAccess,false);assert.equal(thread.result.approvalPolicy,'never');} +const archive=path.join(state,'evidence');fs.mkdirSync(archive,{recursive:true}); +fs.cpSync(home,path.join(archive,'live'),{recursive:true}); +fs.cpSync(path.join(repo,'bin/native-owner'),path.join(archive,'tested-core'),{recursive:true}); +fs.cpSync(path.join(repo,'tests/fixtures/native-owner'),path.join(archive,'tested-fixture'),{recursive:true}); +fs.writeFileSync(path.join(state,'verification.json'),JSON.stringify({passed:true,realModelTurns:2,realToolCalls:4,threadAndReplayRejection:true,postStartup:true,buildSources:build.hashes,binaryHash:hash(build.binary),gateHash:hash(path.join(build.root,'codex-tool-gate.mjs'))},null,2)); +console.log('PASS: consolidated candidate protocol, ownership, and durable acknowledgement evidence.'); diff --git a/tests/fixtures/native-owner/consumer-adapter.patch b/tests/fixtures/native-owner/consumer-adapter.patch new file mode 100644 index 00000000000..0a87b9be8ed --- /dev/null +++ b/tests/fixtures/native-owner/consumer-adapter.patch @@ -0,0 +1,96 @@ +diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh +index 73caeeca..549cb15a 100644 +--- a/bin/fm-session-lock-lib.sh ++++ b/bin/fm-session-lock-lib.sh +@@ -143,6 +143,14 @@ fm_harness_process_matches() { # + # non-numeric so that any consumer treating it as a Cygwin pid - including a + # future `kill` - refuses it instead of acting on the wrong process. + FM_WIN_PID_PREFIX='win:' ++# Isolated native-owner adapter; executable location is code-owned, not supplied ++# by the caller's environment. Unknown/unavailable ownership never falls back. ++FM_NATIVE_OWNER_BIN="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-native-owner.exe" ++fm_native_owner_call() { # [id] ++ local native_state ++ native_state=$(cygpath -w "${FM_STATE_OVERRIDE:-${FM_HOME:?}/state}") || return 2 ++ MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner "$1" "$native_state" "${2:-}" ++} + + # True on a Cygwin-family userspace, where the boundary above applies. + fm_win_boundary_applies() { +@@ -167,7 +175,13 @@ fm_win_untag_pid() { # + # holder - which reads as "startup never completed" and repeats the whole + # sequence on every clear or compact. + fm_session_pid_valid() { # ++ local native_id + case "$1" in ++ native:*) ++ native_id=${1#native:} ++ [ "${#native_id}" -eq 32 ] || return 1 ++ case "$native_id" in *[!0-9a-f]*) return 1 ;; esac ++ return 0 ;; + "$FM_WIN_PID_PREFIX"[0-9]*) return 0 ;; + ''|*[!0-9]*) return 1 ;; + esac +@@ -257,6 +271,7 @@ fm_win_harness_ancestry_pids() { + # session cannot be read off the ancestry at all, so the whole contiguous run is + # reported and the callers below decide what they need from it. + fm_harness_ancestry_pids() { ++ if fm_win_boundary_applies; then fm_native_owner_call identity; return; fi + local pid=$$ comm args extending=0 printed=0 + for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do + comm=$(fm_ps_comm "$pid") || break +@@ -311,7 +326,14 @@ EOF + # kill -0 cannot see across that boundary and would report a live harness as + # dead - which would hand a running session's home to a second one. + fm_harness_pid_alive() { +- local pid=$1 comm args winpid ++ local pid=$1 comm args winpid owner_rc ++ if fm_win_boundary_applies; then ++ fm_native_owner_call alive "$pid" ++ owner_rc=$? ++ # Only an explicit proven-dead answer permits replacing an old record. ++ [ "$owner_rc" -ne 1 ] ++ return ++ fi + if winpid=$(fm_win_untag_pid "$pid"); then + comm=$(fm_win_command "$winpid") || return 1 + fm_harness_process_matches "$comm" "$comm" +@@ -332,7 +354,12 @@ fm_harness_pid_alive() { + # lock, a malformed lock, a lock held by a harness outside this ancestry, or an + # ancestry that cannot be resolved all fail closed. + fm_session_lock_owned_by_self() { +- local state=$1 lock_pid pids pid ++ local state=$1 lock_pid pids pid native_state ++ if fm_win_boundary_applies; then ++ native_state=$(cygpath -w "$state") || return 1 ++ MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner owns "$native_state" ++ return ++ fi + lock_pid=$(cat "$state/.lock" 2>/dev/null || true) + fm_session_pid_valid "$lock_pid" || return 1 + pids=$(fm_harness_ancestry_pids) || return 1 +diff --git a/bin/fm-harness.sh b/bin/fm-harness.sh +index 7989643f..201b61cb 100644 +--- a/bin/fm-harness.sh ++++ b/bin/fm-harness.sh +@@ -395,7 +395,19 @@ harness_family() { + # a harness-shaped path in some node process's arguments is weaker evidence + # than a harness publishing its own identity. + detect_own() { +- local marker ancestry strength harness ++ local marker ancestry strength harness native_state ++ # Disposable native integration: ask the registered launch, not this sibling ++ # operation's ancestry. Denial never adopts an inherited agent marker. ++ case "$(uname -s)" in ++ MINGW*|MSYS*|CYGWIN*) ++ if [ -f "$SCRIPT_DIR/fm-native-owner.exe" ]; then ++ native_state=$(cygpath -w "${FM_STATE_OVERRIDE:-$FM_HOME/state}") || { echo unknown; return; } ++ harness=$(MSYS2_ARG_CONV_EXCL='*' "$SCRIPT_DIR/fm-native-owner.exe" owner harness "$native_state" 2>/dev/null) || { echo unknown; return; } ++ case "$harness" in codex|unknown) echo "$harness" ;; *) echo unknown ;; esac ++ return ++ fi ++ ;; ++ esac + marker=$(harness_marker) + ancestry=$(harness_ancestry) + if [ -z "$ancestry" ]; then diff --git a/tests/fixtures/native-owner/exercise.sh b/tests/fixtures/native-owner/exercise.sh new file mode 100644 index 00000000000..88bd27d8a88 --- /dev/null +++ b/tests/fixtures/native-owner/exercise.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Real startup in an empty disposable home; no mocked startup/deferred owners. +set -eu +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +BUILD=$(cd "$ROOT/.." && pwd) +export FM_HOME +FM_HOME=$(cygpath -u "${FM_HOME:?}") +export PATH="$BUILD/tools:$PATH" +LOG=$(cygpath -u "${FM_PROBE_HOME:?}") +mkdir -p "$FM_HOME/state" "$FM_HOME/config" "$FM_HOME/data" +# Explicit empty manual backlog avoids inventing work or invoking task migration. +printf 'manual\n' > "$FM_HOME/config/backlog-backend" +printf '# Backlog\n' > "$FM_HOME/data/backlog.md" +cd "$ROOT" +printf '%s\n' "$(bin/fm-harness.sh)" > "$LOG/detected-harness.txt" +jq --version > "$LOG/jq-version.txt" +bin/fm-sessionstart-run.sh --source startup > "$LOG/startup.log" 2>&1 +. "$ROOT/bin/fm-session-lock-lib.sh" +fm_session_lock_owned_by_self "$FM_HOME/state" +identity=$(<"$FM_HOME/state/.lock") +[ "$(<"$FM_HOME/state/.session-start-complete")" = "$identity" ] +! grep -q 'READ-ONLY SESSION\|SESSION START INCOMPLETE' "$LOG/startup.log" +printf 'STARTUP_COMPLETION_PASS\n' +# Observe the real detached worker; do not reimplement or invoke its work twice. +for ((i=0; i<150; i++)); do + phase=$(awk -F= '$1=="state" {print $2}' "$FM_HOME/state/.startup-network.status" 2>/dev/null || true) + case "$phase" in done|failed|timeout) break ;; esac + sleep 1 +done +bin/fm-startup-network.sh report > "$LOG/deferred-report.log" 2>&1 +cp "$FM_HOME/state/.startup-network.status" "$LOG/deferred-status.txt" +[ "$phase" = 'done' ] +grep -qx 'locked=1' "$LOG/deferred-status.txt" +grep -qx 'rc=0' "$LOG/deferred-status.txt" +# A later check, after startup and deferred work, must retain the same owner. +fm_session_lock_owned_by_self "$FM_HOME/state" +[ "$(<"$FM_HOME/state/.lock")" = "$identity" ] +printf '%s\n' "$identity" > "$LOG/final-owner.txt" +printf 'DEFERRED_WORK_PASS\nSUBSEQUENT_OWNER_CHECK_PASS\nACTUAL_FIRSTMATE_OPERATION_PASS\n' diff --git a/tests/fixtures/native-owner/jq b/tests/fixtures/native-owner/jq new file mode 100644 index 00000000000..a9eb382cda0 --- /dev/null +++ b/tests/fixtures/native-owner/jq @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# jq stays in the existing Docker image. Only this disposable build is mounted. +set -eu +root=$(cd "$(dirname "$0")/.." && pwd) +exec env MSYS2_ARG_CONV_EXCL='*' 'C:/Program Files/Docker/Docker/resources/bin/docker.exe' run --rm -i \ + --mount "type=bind,source=$(cygpath -m "$root"),target=$root,readonly" \ + --workdir "$root" firstmate-pr3553-validation:tools-v1 jq "$@" diff --git a/tests/fixtures/native-owner/notification-ack.sh b/tests/fixtures/native-owner/notification-ack.sh new file mode 100644 index 00000000000..a7ab57f3b33 --- /dev/null +++ b/tests/fixtures/native-owner/notification-ack.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Only controller-validated receipt data is accepted; no command text is parsed. +set -eu +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +LOG=$(cygpath -u "${FM_PROBE_HOME:?}") +export FM_HOME +FM_HOME=$(cygpath -u "${FM_HOME:?}") +BUILD=$(cd "$ROOT/.." && pwd) +export PATH="$BUILD/tools:$PATH" +cd "$ROOT" +. bin/fm-session-lock-lib.sh +fm_session_lock_owned_by_self "$FM_HOME/state" +request="$LOG/notification-ack-request.json" +note=$(jq -r .note "$request") +seq=$(jq -r .seq "$request") +generation=$(jq -r .generation "$request") +case "$note" in ''|*[!A-Za-z0-9._-]*) exit 2 ;; esac +case "$seq" in ''|*[!0-9]*) exit 2 ;; esac +case "$generation" in ''|*[!A-Za-z0-9._-]*) exit 2 ;; esac +bin/fm-inbox.sh drain --ack "$note" > "$LOG/cycle-inbox-ack.log" +bin/fm-wake-drain.sh --ack-through "$seq" --recovery-generation "$generation" > "$LOG/cycle-ack.log" 2>&1 +[ ! -s "$FM_HOME/state/.wake-queue" ] +[ -f "$FM_HOME/state/inbox/handled/$note.note" ] +printf '{"acknowledged":true,"queueEmpty":true}\n' > "$LOG/notification-ack.json" diff --git a/tests/fixtures/native-owner/notification-check.sh b/tests/fixtures/native-owner/notification-check.sh new file mode 100644 index 00000000000..84efdf98e60 --- /dev/null +++ b/tests/fixtures/native-owner/notification-check.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Fixed notification operation, created and scoped by the existing controller. +set -eu +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +LOG=$(cygpath -u "${FM_PROBE_HOME:?}") +export FM_HOME +FM_HOME=$(cygpath -u "${FM_HOME:?}") +BUILD=$(cd "$ROOT/.." && pwd) +export PATH="$BUILD/tools:$PATH" +cd "$ROOT" +. bin/fm-session-lock-lib.sh +fm_session_lock_owned_by_self "$FM_HOME/state" +[ -f "$LOG/owner-operation.complete" ] +bin/fm-wake-drain.sh > "$LOG/cycle-initial-drain.log" 2>&1 +challenge=$(od -An -N12 -tx1 /dev/urandom | tr -d ' \n') +bin/fm-inbox.sh note "Controlled notification. Confirm observation of $challenge; no project action is requested." > "$LOG/cycle-enqueue.log" +set +e +bin/fm-watch-checkpoint.sh --seconds 3 > "$LOG/cycle-checkpoint.log" 2>&1 +rc=$? +set -e +case "$rc" in 0|124) ;; *) exit "$rc" ;; esac +bin/fm-wake-drain.sh > "$LOG/cycle-delivery.log" 2>&1 +bin/fm-inbox.sh drain > "$LOG/cycle-message.log" +message=$(<"$LOG/cycle-message.log") +grep -q "$challenge" "$LOG/cycle-message.log" +note=$(awk '/^queued / {print $2}' "$LOG/cycle-enqueue.log") +ack=$(grep 'WAKE_ACK_REQUIRED:' "$LOG/cycle-delivery.log" | tail -1) +seq=$(awk '{for(i=1;i "$LOG/notification-check.json" diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs new file mode 100644 index 00000000000..80cf639095f --- /dev/null +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -0,0 +1,64 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {createNotificationGate} from '../../../bin/native-owner/codex-tool-gate.mjs'; +const message={receipt:'receipt',challenge:'observed',message:'Controlled message',checkpointExit:124}; +function fixture(operate) { + const calls=[];let alive=true; + const gate=createNotificationGate({primaryThread:'primary',isAlive:()=>alive,operate:async(...args)=>{ + calls.push(args); + return operate?operate(...args):args[0]==='check'?{operationState:'delivered',notification:message}:{operationState:'acknowledged'}; + }}); + gate.beginTurn('primary','turn'); + return {gate,calls,die:()=>{alive=false;}}; +} +const check=(overrides={})=>({threadId:'primary',turnId:'turn',callId:'check',namespace:null,tool:'fm_notification_check',arguments:{},...overrides}); +const ack=(overrides={})=>check({callId:'ack',tool:'fm_notification_ack',arguments:{receipt:'receipt',observed:'observed'},...overrides}); + +test('observation precedes one acknowledgement; receipt replay cannot execute twice',async()=>{ + const {gate,calls}=fixture(); + assert.equal((await gate.handle(ack())).success,false); + assert.equal(calls.length,0); + assert.equal((await gate.handle(check())).success,true); + assert.equal((await gate.handle(ack({callId:'valid-ack'}))).success,true); + assert.equal((await gate.handle(ack({callId:'replayed-receipt'}))).value.denied,'receipt-already-consumed'); + assert.deepEqual(calls,[['check'],['ack',{receipt:'receipt',observed:'observed'}]]); +}); +for(const [name,request] of Object.entries({ + foreignThread:check({threadId:'foreign'}), foreignTurn:check({turnId:'foreign'}), + emptyCall:check({callId:''}), namespace:check({namespace:'other'}), + arbitraryCommand:check({arguments:{command:'write files'}}), forgedIdentity:check({arguments:{threadId:'primary'}}), + unknownTool:check({tool:'shell'}), nullArguments:check({arguments:null}), arrayArguments:check({arguments:[]}), +}))test(`${name} cannot reach the native operation`,async()=>{ + const {gate,calls}=fixture();assert.equal((await gate.handle(request)).success,false);assert.equal(calls.length,0); +}); +test('duplicate call ID rejected independently of receipt state',async()=>{ + const {gate,calls}=fixture();await gate.handle(check()); + assert.equal((await gate.handle(check())).value.denied,'wrong-thread-turn-or-replay');assert.equal(calls.length,1); +}); +test('wrong observation, receipt, and extra arguments cannot acknowledge',async()=>{ + const {gate,calls}=fixture();await gate.handle(check()); + for(const [i,args] of [{receipt:'wrong',observed:'observed'},{receipt:'receipt',observed:'wrong'},{...message,command:'other'}].entries())assert.equal((await gate.handle(ack({callId:'bad'+i,arguments:args}))).success,false); + assert.equal(calls.length,1); +}); +test('dead connection and completed turn deny even with correct IDs',async()=>{ + const f=fixture();f.die();assert.equal((await f.gate.handle(check())).success,false);assert.equal(f.calls.length,0); + const g=fixture();g.gate.endTurn('primary','turn');assert.equal((await g.gate.handle(check())).success,false);assert.equal(g.calls.length,0); +}); +test('closed gate cannot be rebound to another thread',()=>{ + const {gate}=fixture();assert.throws(()=>gate.beginTurn('foreign','turn2'));gate.close();assert.throws(()=>gate.beginTurn('primary','turn2')); +}); +test('concurrent calls cannot start overlapping operations',async()=>{ + let finish;const {gate,calls}=fixture(()=>new Promise(resolve=>{finish=resolve;})); + const first=gate.handle(check()); + assert.equal((await gate.handle(check({callId:'concurrent'}))).value.denied,'operation-in-progress'); + finish({operationState:'delivered',notification:message});assert.equal((await first).success,true);assert.equal(calls.length,1); +}); +test('late delivery after turn completion grants no further action',async()=>{ + let finish;const {gate,calls}=fixture(()=>new Promise(resolve=>{finish=resolve;})); + const first=gate.handle(check());gate.endTurn('primary','turn');finish({operationState:'delivered',notification:message}); + assert.equal((await first).success,false);assert.equal((await gate.handle(ack())).success,false);assert.equal(calls.length,1); +}); +test('operation failure is not reported as success and cannot be blindly retried',async()=>{ + const {gate,calls}=fixture(()=>{throw Error('partial operation requires reconciliation');}); + assert.equal((await gate.handle(check())).success,false);assert.equal((await gate.handle(check({callId:'retry'}))).success,false);assert.equal(calls.length,1); +}); diff --git a/tests/fm-native-owner-codex-live-e2e.test.sh b/tests/fm-native-owner-codex-live-e2e.test.sh new file mode 100644 index 00000000000..ce0115f332c --- /dev/null +++ b/tests/fm-native-owner-codex-live-e2e.test.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# Explicitly opt-in, two-turn Codex test of the consolidated native core. +# Uses a disposable home only; no sandbox changes or provider installation. +set -eu +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +fm_live_gate opt-in FM_LIVE_NATIVE_CODEX node powershell.exe codex docker +case "$(uname -s)" in + MINGW*|MSYS*|CYGWIN*) ;; + *) printf '%s\n' 'Native Codex ownership test requires Windows' >&2; exit 1 ;; +esac +export FM_LIVE_NATIVE_CODEX=1 +fixture="$ROOT/tests/fixtures/native-owner" +powershell.exe -NoProfile -NonInteractive -File "$(cygpath -w "$fixture/Build.ps1")" +node "$fixture/Run-Cycle.mjs" --dry +node "$fixture/Run-Cycle.mjs" +node "$fixture/Verify-Cycle.mjs" diff --git a/tests/fm-native-owner-tool-gate.test.sh b/tests/fm-native-owner-tool-gate.test.sh new file mode 100644 index 00000000000..f52139a0d18 --- /dev/null +++ b/tests/fm-native-owner-tool-gate.test.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +# Portable behavioral tests of the host-side notification request policy. +set -eu +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +node --test "$ROOT/tests/fixtures/native-owner/tool-gate.test.mjs" From 351345e49b24b473bec786a4cc6d4688ff0b1857 Mon Sep 17 00:00:00 2001 From: Cristian Date: Tue, 15 Sep 2026 23:54:46 +1200 Subject: [PATCH 05/61] test(native-owner): bind evidence to its exact build across rebuilds --- tests/fixtures/native-owner/Run-Cycle.mjs | 1 + tests/fixtures/native-owner/Verify-Cycle.mjs | 12 +++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs index b8fe608d847..4c4f2e87078 100644 --- a/tests/fixtures/native-owner/Run-Cycle.mjs +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -11,6 +11,7 @@ const dry=process.argv.includes('--dry'); if(!dry&&process.env.FM_LIVE_NATIVE_CODEX!=='1')throw Error('Live model test requires FM_LIVE_NATIVE_CODEX=1'); if(!dry){const preflight=read(path.join(dir,'bridge-preflight.json'));if(!preflight.passed||preflight.binaryHash!==createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex'))throw Error('Run model-free bridge preflight for this binary first');} const home=path.join(build.root,'appserver-'+randomUUID());fs.mkdirSync(home); +fs.writeFileSync(path.join(home,'build.json'),JSON.stringify(build,null,2)); const script=path.join(build.root,'AppHost.mjs'); const spec={home,leaseHome:path.join(home,'home'),executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'UserOnly',apiDry:dry}; const file=path.join(home,'spec.json');fs.writeFileSync(file,JSON.stringify(spec,null,2)); diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index ab8f45bee7b..320ed3f5aa2 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -8,9 +8,9 @@ const repo=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..') const state=path.join(repo,'data/native-candidate-validation'); const read=file=>JSON.parse(fs.readFileSync(file,'utf8').replace(/^\uFEFF/,'')); const hash=file=>createHash('sha256').update(fs.readFileSync(file)).digest('hex'); -const build=read(path.join(state,'build.json')); const latest=read(path.join(state,'cycle-latest.json')); const home=latest.home; +const build=read(path.join(home,'build.json')); const host=read(path.join(home,'app-host-evidence.json')); const native=read(path.join(home,'result.json')); assert.equal(latest.exit,0);assert.equal(native.rootExit,0);assert.equal(host.passed,true); @@ -40,9 +40,11 @@ assert.ok(fs.existsSync(path.join(operational,'inbox/handled',delivered.note+'.n const threads=host.frames.filter(frame=>frame.result?.thread); assert.equal(threads.length,2); for(const thread of threads){assert.equal(thread.result.sandbox.type,'readOnly');assert.equal(thread.result.sandbox.networkAccess,false);assert.equal(thread.result.approvalPolicy,'never');} -const archive=path.join(state,'evidence');fs.mkdirSync(archive,{recursive:true}); -fs.cpSync(home,path.join(archive,'live'),{recursive:true}); -fs.cpSync(path.join(repo,'bin/native-owner'),path.join(archive,'tested-core'),{recursive:true}); -fs.cpSync(path.join(repo,'tests/fixtures/native-owner'),path.join(archive,'tested-fixture'),{recursive:true}); +const archive=path.join(state,'evidence',path.basename(home)); +if(!fs.existsSync(archive)) { + fs.mkdirSync(archive,{recursive:true}); + fs.cpSync(home,path.join(archive,'live'),{recursive:true}); +} + fs.writeFileSync(path.join(state,'verification.json'),JSON.stringify({passed:true,realModelTurns:2,realToolCalls:4,threadAndReplayRejection:true,postStartup:true,buildSources:build.hashes,binaryHash:hash(build.binary),gateHash:hash(path.join(build.root,'codex-tool-gate.mjs'))},null,2)); console.log('PASS: consolidated candidate protocol, ownership, and durable acknowledgement evidence.'); From 54aa6736568a2d24cd5148f8ac154e75f9759cf8 Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 08:43:39 +1200 Subject: [PATCH 06/61] feat(native-owner): persist receipt lifecycle and preserve interrupted work --- bin/fm-test-run.sh | 7 +- bin/native-owner/NativeHomeLease.cs | 1 + bin/native-owner/NativeReceiptJournal.cs | 115 ++++++++++++++++++ bin/native-owner/codex-tool-gate.mjs | 20 ++- docs/verification/runtime-backends.md | 13 +- tests/fixtures/native-owner/Build.ps1 | 4 +- tests/fixtures/native-owner/Lifecycle.mjs | 86 +++++++++++++ tests/fixtures/native-owner/NativeDriver.cs | 22 ++-- tests/fixtures/native-owner/ReceiptTests.cs | 107 ++++++++++++++++ tests/fixtures/native-owner/Verify-Cycle.mjs | 9 +- .../fixtures/native-owner/tool-gate.test.mjs | 26 ++++ .../fm-native-owner-receipt-live-e2e.test.sh | 12 ++ 12 files changed, 404 insertions(+), 18 deletions(-) create mode 100644 bin/native-owner/NativeReceiptJournal.cs create mode 100644 tests/fixtures/native-owner/Lifecycle.mjs create mode 100644 tests/fixtures/native-owner/ReceiptTests.cs create mode 100644 tests/fm-native-owner-receipt-live-e2e.test.sh diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index b1c57307aa0..321e7de71da 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -342,7 +342,7 @@ family_for_basename() { fm-claude-stop-autoarm-live-e2e.test.sh|\ fm-cmux-claude-composer-live-e2e.test.sh|\ fm-composer-matrix-live-e2e.test.sh|\ - fm-codex-continuity-live-e2e.test.sh|fm-native-owner-codex-live-e2e.test.sh|fm-grok-continuity-live-e2e.test.sh|\ + fm-codex-continuity-live-e2e.test.sh|fm-native-owner-codex-live-e2e.test.sh|fm-native-owner-receipt-live-e2e.test.sh|fm-grok-continuity-live-e2e.test.sh|\ fm-cursor-primary-live-e2e.test.sh|\ fm-grok-stop-live-e2e.test.sh|fm-harness-adapter-instructions-live-e2e.test.sh|\ fm-harness-liveness-drift-live-e2e.test.sh|\ @@ -1359,6 +1359,11 @@ families_for_changed_path() { printf '%s\n' backend-dispatch printf '%s\n' real-herdr-gated ;; + bin/native-owner/*|tests/fixtures/native-owner/*) + printf '%s\n' __script__:fm-native-owner-tool-gate.test.sh + printf '%s\n' __script__:fm-native-owner-receipt-live-e2e.test.sh + printf '%s\n' __script__:fm-native-owner-codex-live-e2e.test.sh + ;; bin/fm-agent-process-lib.sh) # The shared harness-process classifier feeds both the tmux and Herdr # liveness verdicts, so a change to it is proven by both backends' suites. diff --git a/bin/native-owner/NativeHomeLease.cs b/bin/native-owner/NativeHomeLease.cs index 09a056660c8..0c53936982b 100644 --- a/bin/native-owner/NativeHomeLease.cs +++ b/bin/native-owner/NativeHomeLease.cs @@ -14,6 +14,7 @@ public sealed class NativeHomeLease : IDisposable { static readonly JavaScriptSerializer Json=new JavaScriptSerializer(); FileStream file; public readonly string Home; + internal bool IsHeld { get { return file!=null; } } public string PreviousGeneration { get; private set; } [StructLayout(LayoutKind.Sequential)] struct FT { public uint low,high; } [DllImport("kernel32.dll",SetLastError=true)] static extern IntPtr OpenProcess(uint access,bool inherit,uint pid); diff --git a/bin/native-owner/NativeReceiptJournal.cs b/bin/native-owner/NativeReceiptJournal.cs new file mode 100644 index 00000000000..a42322bb82c --- /dev/null +++ b/bin/native-owner/NativeReceiptJournal.cs @@ -0,0 +1,115 @@ +// Durable receipt transitions for a controller that already holds the home lease. +// This is not an authority source: only the authenticated controller may call it. +// Append and flush intent before mutation; ambiguous attempts require reconciliation. +using System; +using System.Collections.Generic; +using System.IO; +using System.Runtime.InteropServices; +using System.Security.AccessControl; +using System.Security.Principal; +using System.Text; +using System.Web.Script.Serialization; + +public sealed class NativeReceiptJournal : IDisposable { + readonly JavaScriptSerializer json = new JavaScriptSerializer(); + readonly Dictionary> receipts = new Dictionary>(); + readonly string home, generation; + readonly NativeHomeLease lease; + FileStream file; + const long Limit = 16 * 1024 * 1024; + [StructLayout(LayoutKind.Sequential)] struct Info { + public uint attributes, createdLow, createdHigh, accessLow, accessHigh, writeLow, writeHigh; + public uint volume, sizeHigh, sizeLow, links, indexHigh, indexLow; + } + [DllImport("kernel32.dll", SetLastError=true)] static extern bool GetFileInformationByHandle(IntPtr handle, out Info info); + public NativeReceiptJournal(NativeHomeLease ownedLease, string ownerGeneration) { + if(ownedLease==null || !ownedLease.IsHeld) throw new InvalidOperationException("An active native home lease is required"); + lease=ownedLease; + if(ownerGeneration==null || ownerGeneration.Length!=32 || !IsHex(ownerGeneration)) throw new ArgumentException("Invalid owner generation"); + home=Path.GetFullPath(ownedLease.Home).TrimEnd('\\','/'); generation=ownerGeneration; + // The native home lease owns directory validation and exclusion. Never + // create a directory or select another home based on tool arguments. + if(!Directory.Exists(home)) throw new IOException("Owned home is absent"); + string name=Path.Combine(home,"owner-receipts.jsonl"); + var security=new FileSecurity();security.SetAccessRuleProtection(true,false); + var user=WindowsIdentity.GetCurrent().User; + security.SetOwner(user); + security.AddAccessRule(new FileSystemAccessRule(user,FileSystemRights.FullControl,AccessControlType.Allow)); + bool created=false; + try { + try { file=new FileStream(name,FileMode.CreateNew,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security);created=true; } + catch(IOException) { + if((File.GetAttributes(name)&FileAttributes.ReparsePoint)!=0) throw new IOException("Receipt journal is a reparse point"); + file=new FileStream(name,FileMode.Open,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security); + } + var access=file.GetAccessControl(); + if(!access.AreAccessRulesProtected || !access.GetOwner(typeof(SecurityIdentifier)).Equals(user)) throw new IOException("Receipt journal security differs; preserved"); + foreach(FileSystemAccessRule rule in access.GetAccessRules(true,true,typeof(SecurityIdentifier))) if(rule.AccessControlType==AccessControlType.Allow && !rule.IdentityReference.Equals(user)) throw new IOException("Receipt journal grants unexpected access; preserved"); + Info info; + if(!GetFileInformationByHandle(file.SafeFileHandle.DangerousGetHandle(),out info) || info.links!=1 || (info.attributes&0x400)!=0) throw new IOException("Receipt journal file identity is unsafe"); + if(file.Length>Limit || (!created && file.Length==0)) throw new IOException("Receipt journal is empty or oversized; preserved"); + if(!created) { + string content; + using(var reader=new StreamReader(file,new UTF8Encoding(false,true),true,4096,true)) content=reader.ReadToEnd(); + if(!content.EndsWith("\n",StringComparison.Ordinal)) throw new IOException("Interrupted receipt record; preserved"); + foreach(string line in content.Split(new [] {'\n'},StringSplitOptions.RemoveEmptyEntries)) Apply(json.Deserialize>(line)); + } + Append("session",null,null); + } catch { Dispose();throw; } + } + static bool IsHex(string value) { foreach(char c in value) if(!(c>='0'&&c<='9')&&!(c>='a'&&c<='f')) return false;return true; } + public bool NeedsReconciliation { get { foreach(var row in receipts.Values) if((string)row["event"]=="ack-started") return true;return false; } } + Dictionary Copy(Dictionary value) { return json.Deserialize>(json.Serialize(value)); } + void EnsureOpen() { if(file==null || !lease.IsHeld) throw new InvalidOperationException("Receipt journal or owning lease is closed"); } + public Dictionary Present(Dictionary payload) { + EnsureOpen(); + if(NeedsReconciliation) throw new IOException("An acknowledgement was interrupted; reconcile durable work before proceeding"); + foreach(var row in receipts.Values) if((string)row["generation"]==generation && (string)row["event"]=="presented") return Delivery(row); + if(payload==null || !payload.ContainsKey("challenge") || !(payload["challenge"] is string)) throw new ArgumentException("Notification payload is incomplete"); + string receipt=Guid.NewGuid().ToString("N"); + Append("presented",receipt,Copy(payload)); + return Delivery(receipts[receipt]); + } + Dictionary Delivery(Dictionary row) { + var result=Copy((Dictionary)row["payload"]);result["receipt"]=row["receipt"];return result; + } + public Dictionary BeginAcknowledgement(string receipt,string observed) { + EnsureOpen(); + Dictionary row; + if(NeedsReconciliation || receipt==null || !receipts.TryGetValue(receipt,out row) || (string)row["generation"]!=generation || (string)row["event"]!="presented") throw new InvalidOperationException("Receipt is not eligible"); + var payload=(Dictionary)row["payload"]; + if((string)payload["challenge"]!=observed) throw new InvalidOperationException("Notification was not observed"); + Append("ack-started",receipt,Copy(payload)); + return Delivery(receipts[receipt]); + } + public void CompleteAcknowledgement(string receipt) { + EnsureOpen(); + Dictionary row; + if(receipt==null || !receipts.TryGetValue(receipt,out row) || (string)row["generation"]!=generation || (string)row["event"]!="ack-started") throw new InvalidOperationException("No matching acknowledgement attempt"); + Append("acknowledged",receipt,Copy((Dictionary)row["payload"])); + } + void Append(string kind,string receipt,Dictionary payload) { + var row=new Dictionary{{"version",1},{"home",home},{"generation",generation},{"event",kind},{"receipt",receipt},{"payload",payload}}; + byte[] bytes=new UTF8Encoding(false,true).GetBytes(json.Serialize(row)+"\n"); + if(bytes.Length>65536 || file.Length+bytes.Length>Limit) throw new IOException("Receipt journal capacity exceeded; durable work preserved"); + try { file.Position=file.Length;file.Write(bytes,0,bytes.Length);file.Flush(true);Apply(row); } + catch { Dispose();throw; } + } + void Apply(Dictionary row) { + if(row==null || Convert.ToInt32(row["version"])!=1 || !string.Equals((string)row["home"],home,StringComparison.OrdinalIgnoreCase)) throw new IOException("Receipt journal binding is invalid"); + string gen=(string)row["generation"],kind=(string)row["event"]; + if(gen==null || gen.Length!=32 || !IsHex(gen)) throw new IOException("Invalid receipt generation"); + if(kind=="session") return; + string id=(string)row["receipt"]; + if(id==null || id.Length!=32 || !IsHex(id) || !(row["payload"] is Dictionary)) throw new IOException("Malformed receipt; preserved"); + Dictionary previous; + bool exists=receipts.TryGetValue(id,out previous); + if(kind=="presented") { if(exists) throw new IOException("Duplicate receipt; preserved"); } + else if(kind=="ack-started" || kind=="acknowledged") { + string expected=kind=="ack-started" ? "presented" : "ack-started"; + if(!exists || (string)previous["event"]!=expected || (string)previous["generation"]!=gen || json.Serialize(previous["payload"])!=json.Serialize(row["payload"])) throw new IOException("Invalid receipt transition; preserved"); + } else throw new IOException("Unknown receipt transition; preserved"); + receipts[id]=row; + } + public void Dispose() { if(file!=null) { file.Dispose();file=null; } } +} diff --git a/bin/native-owner/codex-tool-gate.mjs b/bin/native-owner/codex-tool-gate.mjs index 4abb4dc412b..cab0dbfe667 100644 --- a/bin/native-owner/codex-tool-gate.mjs +++ b/bin/native-owner/codex-tool-gate.mjs @@ -12,20 +12,25 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { let receipt = null; let challenge = null; let acknowledged = false; + let offered = null; const seen = new Set(); + const retiredTurns = new Set(); const deny = reason => ({ success: false, value: { denied: reason } }); const valid = params => !closed && isAlive() && params.threadId === primaryThread && typeof params.turnId === 'string' && params.turnId === activeTurn; return Object.freeze({ beginTurn(thread, turn) { - if (closed || thread !== primaryThread || typeof turn !== 'string' || !turn || activeTurn) { + if (closed || thread !== primaryThread || typeof turn !== 'string' || !turn || activeTurn || retiredTurns.has(turn)) { throw new Error('Cannot register this turn'); } activeTurn = turn; }, endTurn(thread, turn) { - if (thread === primaryThread && turn === activeTurn) activeTurn = null; + if (thread === primaryThread && turn === activeTurn) { + retiredTurns.add(turn); + activeTurn = null; + } }, close() { closed = true; @@ -44,7 +49,10 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { if (busy) return deny('operation-in-progress'); if (params.tool === 'fm_notification_check') { - if (Object.keys(args).length || receipt) return deny('check-already-used-or-invalid-arguments'); + if (Object.keys(args).length) return deny('invalid-arguments'); + // Redelivery is read-only: cancellation or a lost response must not + // strand pending work or start another native check before handling it. + if (receipt && !acknowledged) return { success: true, value: { ...offered } }; } else if (params.tool === 'fm_notification_ack') { if (Object.keys(args).sort().join(',') !== 'observed,receipt' || !receipt || args.receipt !== receipt || args.observed !== challenge) { @@ -68,10 +76,10 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { } receipt = note.receipt; challenge = note.challenge; + acknowledged = false; + offered = Object.freeze({ message: note.message, receipt, challenge, checkpointExit: note.checkpointExit }); if (!valid(params)) return deny('wrong-thread-turn-or-replay'); - return { success: true, value: { - message: note.message, receipt, challenge, checkpointExit: note.checkpointExit, - } }; + return { success: true, value: { ...offered } }; } const result = await operate('ack', { receipt, observed: args.observed }); if (result?.operationState !== 'acknowledged') throw new Error('Acknowledgement did not complete'); diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index d7a87fc97ab..ab1cc111b8b 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -19,7 +19,14 @@ Refresh the portable request-policy regression with: bash tests/fm-native-owner-tool-gate.test.sh ``` -The 17 behavioral cases cover primary-thread/turn binding, duplicate calls, consumed receipts, invalid arguments, arbitrary-command rejection, dead connections, overlapping calls, late delivery, and operation failure without blind retry. +The 21 behavioral cases cover primary-thread/turn binding, duplicate calls, consumed receipts, invalid arguments, arbitrary-command rejection, dead connections, overlapping calls, late delivery, repeated cycles, retired turns, read-only redelivery after cancellation, and operation failure without blind retry. +The native controller now persists receipt presentation, acknowledgement intent, and completion under its existing exclusive home lease. +It flushes acknowledgement intent before invoking the mutation and preserves interrupted attempts for reconciliation rather than retrying them. +The token-free Windows guard exercises 12 receipt cases, including restart/generation behavior, interrupted and completed attempts, multiple cycles, torn records, file links, and lease revocation, followed by exclusive-owner, normal-exit, controller-loss, orphan-recovery, ambiguous-record, and child-scope tests: + +```sh +bash tests/fm-native-owner-receipt-live-e2e.test.sh +``` Refresh the actual Windows integration with the explicit two-model-turn guard: ```sh @@ -39,7 +46,9 @@ A repeated receipt and a request from another real thread were refused without a The checkpoint exercised its three-second timeout followed by a drain, not immediate interrupt-driven delivery. Both app-server threads reported read-only filesystem policy, disabled network access, and approval policy `never`; the two explicitly authorized host operations execute outside ordinary model shell tools. Dynamic tool registration requires the experimental API capability in this version. -This does not establish persistent receipt recovery, cancellation of an already accepted mutation, adversarial Windows path/process races, populated-fleet behavior, or other harness support. +The actual app-server acknowledgement is also checked against the durable journal's owner generation and target notification. +This does not establish automatic reconciliation of an interrupted mutation, cancellation of an already accepted mutation, adversarial Windows path/process races, populated-fleet behavior, or other harness support. +The production launcher and ordinary-startup integration remain disabled while those boundaries are incomplete. ## Harness detection precedence diff --git a/tests/fixtures/native-owner/Build.ps1 b/tests/fixtures/native-owner/Build.ps1 index 8ab6bc1f38d..8739aa4d472 100644 --- a/tests/fixtures/native-owner/Build.ps1 +++ b/tests/fixtures/native-owner/Build.ps1 @@ -7,8 +7,10 @@ $repo = [IO.Path]::GetFullPath((Join-Path $PSScriptRoot '../../..')) $root = Join-Path ([IO.Path]::GetTempPath()) ('fm-native-candidate-' + [guid]::NewGuid().ToString('N')) New-Item -ItemType Directory $root | Out-Null $binary = Join-Path $root 'SessionProbe.exe' -$sources = @((Join-Path $repo 'bin/native-owner/NativeOwner.cs'), (Join-Path $repo 'bin/native-owner/NativeHomeLease.cs'), (Join-Path $PSScriptRoot 'NativeDriver.cs')) +$sources = @((Join-Path $repo 'bin/native-owner/NativeOwner.cs'), (Join-Path $repo 'bin/native-owner/NativeHomeLease.cs'), (Join-Path $PSScriptRoot 'NativeDriver.cs'), (Join-Path $repo 'bin/native-owner/NativeReceiptJournal.cs'), (Join-Path $PSScriptRoot 'ReceiptTests.cs')) Add-Type -Path $sources -OutputAssembly $binary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll +& $binary receipt-tests +if ($LASTEXITCODE -ne 0) { throw 'Durable receipt lifecycle tests failed' } $copy = Join-Path $root 'firstmate' & git -c core.symlinks=true clone --quiet --no-local --single-branch $repo $copy if ($LASTEXITCODE -ne 0) { throw 'Disposable clone failed' } diff --git a/tests/fixtures/native-owner/Lifecycle.mjs b/tests/fixtures/native-owner/Lifecycle.mjs new file mode 100644 index 00000000000..06a125bcddc --- /dev/null +++ b/tests/fixtures/native-owner/Lifecycle.mjs @@ -0,0 +1,86 @@ +// Bounded, model-free native lease/lifetime controls; no production home involved. +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { spawn, spawnSync } from 'node:child_process'; +import { randomUUID, createHash } from 'node:crypto'; +const directory = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../data/native-candidate-validation'); +const read = name => JSON.parse(fs.readFileSync(name, 'utf8').replace(/^\uFEFF/, '')); +const build = read(path.join(directory, 'build.json')); +const root = path.join(build.root, 'lifecycle-' + randomUUID()); fs.mkdirSync(root); +const records = []; +const sleep = ms => new Promise(resolve => setTimeout(resolve, ms)); +const hash = name => createHash('sha256').update(fs.readFileSync(name)).digest('hex'); +function start(name, leaseHome, milliseconds = 100, abandonAfterLaunch = false, boundaries = false) { + const home = path.join(root, name); fs.mkdirSync(home); + const spec = { home, leaseHome, executable: build.binary, arguments: boundaries ? 'fixture' : `sleep ${milliseconds}`, timeoutSeconds: 20, outsider: boundaries, boundaries, assignJob: true, pipeAcl: 'UserOnly', abandonAfterLaunch }; + const config = path.join(home, 'spec.json'); fs.writeFileSync(config, JSON.stringify(spec, null, 2)); + const child = spawn(build.binary, ['run', config], { stdio: ['ignore', 'pipe', 'pipe'] }); + let stdout = '', stderr = ''; + child.stdout.on('data', data => stdout += data); child.stderr.on('data', data => stderr += data); + const done = new Promise((resolve, reject) => { child.on('error', reject); child.on('exit', code => { + const result = { name, home, code, stdout, stderr }; records.push(result); + fs.writeFileSync(path.join(home, 'controller.json'), JSON.stringify(result, null, 2)); resolve(result); + }); }); + return { home, done }; +} +async function waitOwner(home, oldGeneration) { + for (let attempt = 0; attempt < 200; attempt++) { + try { const value = read(path.join(home, 'owner-probe.json')); if (value.state === 'live' && value.generation !== oldGeneration) return value; } catch (error) { if (error.code !== 'ENOENT' && !(error instanceof SyntaxError)) throw error; } + await sleep(20); + } + throw Error('Owner did not publish in time'); +} +function check(home, generation) { + const result = spawnSync(build.binary, ['lease-check', home, generation], { encoding: 'utf8' }); + if (result.status !== 0) throw Error(result.stderr); + return JSON.parse(result.stdout); +} +function assert(condition, message) { if (!condition) throw Error(message); } +const shared = path.join(root, 'shared'); +const first = start('first', shared, 2500); +const firstOwner = await waitOwner(shared); +assert(check(shared, firstOwner.generation).probeOwnerCurrent, 'Initial current-owner check failed'); +const original = hash(path.join(shared, 'owner-probe.json')); +const competitor = await start('competitor', shared).done; +assert(competitor.code !== 0 && !fs.existsSync(path.join(competitor.home, 'result.json')), 'Competing controller started a root'); +assert(hash(path.join(shared, 'owner-probe.json')) === original, 'Competing controller changed owner record'); +assert((await first.done).code === 0, 'First owner failed'); +assert(!check(shared, firstOwner.generation).probeOwnerCurrent, 'Exited owner remained current'); +const next = start('restart', shared, 1800); +const nextOwner = await waitOwner(shared, firstOwner.generation); +assert(nextOwner.generation !== firstOwner.generation, 'Generation reused'); +assert(!check(shared, firstOwner.generation).probeOwnerCurrent, 'Old generation accepted after restart'); +assert(check(shared, nextOwner.generation).probeOwnerCurrent, 'New generation rejected'); +assert((await next.done).code === 0, 'Restart failed'); +const crashHome = path.join(root, 'controller-loss'); +const abandoned = await start('abandon-controller', crashHome, 3500, true).done; +assert(abandoned.code === 86, 'Controlled broker-loss fixture failed'); +const survivor = read(path.join(crashHome, 'owner-probe.json')); +assert(check(crashHome, survivor.generation).probeOwnerCurrent, 'Primary did not survive its controller'); +const before = hash(path.join(crashHome, 'owner-probe.json')); +const refused = await start('live-primary-refusal', crashHome).done; +assert(refused.code !== 0 && refused.stderr.includes('Recorded primary is still alive'), 'Live orphan was not protected'); +assert(hash(path.join(crashHome, 'owner-probe.json')) === before, 'Live orphan record changed'); +let alive = true; +for (let attempt = 0; attempt < 100; attempt++) { alive = check(crashHome, survivor.generation).probeOwnerCurrent; if (!alive) break; await sleep(50); } +assert(!alive, 'Bounded orphan did not exit'); +assert((await start('recover-after-primary-exit', crashHome).done).code === 0, 'Proven-dead owner prevented new session'); +for (const [name, content] of [['pending', '{"state":"pending"}'], ['empty', ''], ['malformed', '{broken']]) { + const home = path.join(root, 'ambiguous-' + name); fs.mkdirSync(home); + const filename = path.join(home, 'owner-probe.json'); fs.writeFileSync(filename, content); + const beforeHash = hash(filename); + assert((await start('reject-' + name, home).done).code !== 0, 'Ambiguous record was accepted'); + assert(hash(filename) === beforeHash, 'Ambiguous record was overwritten'); +} +const boundary = await start('registered-boundaries', path.join(root, 'boundary-owner'), 100, false, true).done; +assert(boundary.code === 0, 'Boundary fixture failed'); +const result = read(path.join(boundary.home, 'result.json')); +assert(result.observations.length === 12 && result.probeLeaseHeld && !result.authorityImplemented, 'Boundary evidence incomplete'); +for (const role of ['worker', 'nested-primary']) for (const suffix of ['', '-descendant']) { + const row = result.observations.find(row => row.case === `scope-${role}${suffix}`); + assert(row?.hostClassification === `registered-${role}` && !row.authorityGranted, 'Inherited role escaped its registration'); +} +assert(result.observations.every(row => row.authorityGranted === false), 'Production authority unexpectedly granted'); +fs.writeFileSync(path.join(directory, 'lifecycle-latest.json'), JSON.stringify({ root, passed: true, records }, null, 2)); +console.log('PASS: exclusive acquisition, unchanged competing record, normal exit, fresh generation, stale-generation rejection, controller-loss protection, recovery after root exit, three ambiguous-record refusals, and twelve child-boundary observations.'); diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 3eae4fd9aeb..8311ed98d81 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -14,6 +14,7 @@ using System.Web.Script.Serialization; public static partial class NativeOwner { + static NativeReceiptJournal operationJournal; static string LogonSid() { using(var identity=WindowsIdentity.GetCurrent()) { foreach(var row in TokenGroups(identity.Token,2)) @@ -157,7 +158,10 @@ static int Run(string configPath) { var scopes=new List(); NativeHomeLease lease=null; try { - if(config.ContainsKey("leaseHome")) lease=new NativeHomeLease((string)config["leaseHome"]); + if(config.ContainsKey("leaseHome")) { + lease=new NativeHomeLease((string)config["leaseHome"]); + operationJournal=new NativeReceiptJournal(lease,session); + } var values = EnvironmentFor(pipeName, session, home, nonce); values["MSYS"]="winsymlinks:nativestrict"; if(config.ContainsKey("apiDry") && (bool)config["apiDry"]) values["FM_PROBE_API_DRY"]="1"; @@ -265,6 +269,7 @@ static int Run(string configPath) { } foreach (IntPtr h in new [] {child.thread,child.process,stdout,stderr,stdin,job}) if (h!=IntPtr.Zero) CloseHandle(h); if(env!=IntPtr.Zero) Marshal.FreeHGlobal(env); + if(operationJournal!=null) { operationJournal.Dispose();operationJournal=null; } if(lease!=null) lease.Dispose(); } } @@ -282,9 +287,9 @@ static void NotificationRequest(Dictionary request,Dictionary>(File.ReadAllText(file)); if(pendingOperation.purpose=="check") { - delivered=output; receipt=Guid.NewGuid().ToString("N"); - delivered["receipt"]=receipt; - } else consumed=true; + delivered=operationJournal.Present(output); receipt=(string)delivered["receipt"]; + consumed=false; + } else { operationJournal.CompleteAcknowledgement(receipt);consumed=true; } pendingOperation=null; } string action=request.ContainsKey("action") ? (string)request["action"] : ""; @@ -294,10 +299,12 @@ static void NotificationRequest(Dictionary request,Dictionary request,Dictionary=3 && args[0]=="owner") return OwnerClient(args[1],args[2],args.Length>3 ? args[3] : ""); if(args.Length>0 && args[0]=="client") return Client(args.Length>1 ? args[1] : "agent-tool"); if(args.Length==2 && args[0]=="sleep") { int ms=int.Parse(args[1]); if(ms<0 || ms>15000) throw new ArgumentException("Sleep must be bounded"); Thread.Sleep(ms); return 0; } diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs new file mode 100644 index 00000000000..31e8ad8d0d2 --- /dev/null +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -0,0 +1,107 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Runtime.InteropServices; +using System.Security.AccessControl; +public static class ReceiptTests { + [DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] static extern bool CreateHardLink(string link,string target,IntPtr security); + [DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] [return: MarshalAs(UnmanagedType.I1)] static extern bool CreateSymbolicLink(string link,string target,uint flags); + static readonly string A=new string('a',32),B=new string('b',32); + static int passed; + static Dictionary Payload(string value) { return new Dictionary{{"challenge",value},{"message","pending notification"},{"seq","1"},{"generation","recovery"},{"note","note-id"}}; } + static void Expect(bool value,string message) { if(!value) throw new Exception(message); } + static void Refuses(Action action,string message) { bool refused=false;try{action();}catch{refused=true;}Expect(refused,message); } + static void Case(string name,Action test) { + string home=Path.Combine(Path.GetTempPath(),"fm-receipts-"+Guid.NewGuid().ToString("N")); + using(var lease=new NativeHomeLease(home)) test(lease); + passed++;Console.WriteLine("PASS: "+name); + } + public static int Run() { + Case("one writer and unobserved acknowledgement refusal",lease=>{ + using(var journal=new NativeReceiptJournal(lease,A)) { + Refuses(()=>{using(var other=new NativeReceiptJournal(lease,A)){}},"Concurrent writer accepted"); + var note=journal.Present(Payload("first"));string receipt=(string)note["receipt"]; + Refuses(()=>journal.BeginAcknowledgement(receipt,"wrong"),"Unobserved message accepted"); + journal.BeginAcknowledgement(receipt,"first");journal.CompleteAcknowledgement(receipt); + Refuses(()=>journal.BeginAcknowledgement(receipt,"first"),"Consumed receipt accepted"); + } + }); + Case("pending delivery survives reopen and is not replaced",lease=>{ + string receipt; + using(var journal=new NativeReceiptJournal(lease,A)) receipt=(string)journal.Present(Payload("first"))["receipt"]; + using(var journal=new NativeReceiptJournal(lease,A)) { + var replay=journal.Present(Payload("second"));Expect((string)replay["receipt"]==receipt && (string)replay["challenge"]=="first","Pending work replaced"); + } + }); + Case("new generation cannot consume predecessor receipt",lease=>{ + string old; + using(var journal=new NativeReceiptJournal(lease,A)) old=(string)journal.Present(Payload("first"))["receipt"]; + using(var journal=new NativeReceiptJournal(lease,B)) { + Refuses(()=>journal.BeginAcknowledgement(old,"first"),"Previous generation authorized"); + Expect((string)journal.Present(Payload("first"))["receipt"]!=old,"Receipt reused across generation"); + } + }); + Case("interrupted acknowledgement blocks fresh mutation",lease=>{ + string receipt; + using(var journal=new NativeReceiptJournal(lease,A)) {receipt=(string)journal.Present(Payload("first"))["receipt"];journal.BeginAcknowledgement(receipt,"first");} + using(var journal=new NativeReceiptJournal(lease,B)) { + Expect(journal.NeedsReconciliation,"Interrupted attempt lost"); + Refuses(()=>journal.Present(Payload("second")),"New work replaced ambiguous attempt"); + Refuses(()=>journal.CompleteAcknowledgement(receipt),"New generation invented completion"); + } + }); + Case("completed receipt remains consumed after restart",lease=>{ + string receipt; + using(var journal=new NativeReceiptJournal(lease,A)) {receipt=(string)journal.Present(Payload("first"))["receipt"];journal.BeginAcknowledgement(receipt,"first");journal.CompleteAcknowledgement(receipt);} + using(var journal=new NativeReceiptJournal(lease,B)) { + Expect(!journal.NeedsReconciliation,"Completed attempt became ambiguous"); + Refuses(()=>journal.BeginAcknowledgement(receipt,"first"),"Completed predecessor replay accepted"); + journal.Present(Payload("second")); + } + }); + Case("multiple cycles retain independent consumed receipts",lease=>{ + using(var journal=new NativeReceiptJournal(lease,A)) { + string first=(string)journal.Present(Payload("first"))["receipt"]; + journal.BeginAcknowledgement(first,"first");journal.CompleteAcknowledgement(first); + string second=(string)journal.Present(Payload("second"))["receipt"]; + Refuses(()=>journal.BeginAcknowledgement(first,"first"),"Earlier cycle replay accepted"); + journal.BeginAcknowledgement(second,"second");journal.CompleteAcknowledgement(second); + } + }); + Case("returned objects cannot change persisted target",lease=>{ + using(var journal=new NativeReceiptJournal(lease,A)) { + var source=Payload("first");var result=journal.Present(source);source["challenge"]="changed";result["challenge"]="changed"; + var actual=journal.BeginAcknowledgement((string)result["receipt"],"first");Expect((string)actual["challenge"]=="first","Caller changed target"); + } + }); + Case("torn tail is preserved rather than skipped",lease=>{ + using(var journal=new NativeReceiptJournal(lease,A)) journal.Present(Payload("first")); + string file=Path.Combine(lease.Home,"owner-receipts.jsonl");File.AppendAllText(file,"{\"version\":1");string before=File.ReadAllText(file); + Refuses(()=>{using(var journal=new NativeReceiptJournal(lease,B)){}},"Torn record accepted");Expect(before==File.ReadAllText(file),"Torn record changed"); + }); + Case("empty existing file is not adopted",lease=>{ + using(var journal=new NativeReceiptJournal(lease,A)) {} + string file=Path.Combine(lease.Home,"owner-receipts.jsonl");File.WriteAllText(file,""); + Refuses(()=>{using(var journal=new NativeReceiptJournal(lease,A)){}},"Empty journal adopted");Expect(new FileInfo(file).Length==0,"Ambiguous file overwritten"); + }); + Case("hard-linked receipt file is refused",lease=>{ + using(var journal=new NativeReceiptJournal(lease,A)) journal.Present(Payload("first")); + string file=Path.Combine(lease.Home,"owner-receipts.jsonl"); + Expect(CreateHardLink(Path.Combine(lease.Home,"alias"),file,IntPtr.Zero),"Hard-link fixture failed"); + Refuses(()=>{using(var journal=new NativeReceiptJournal(lease,A)){}},"Hard-linked receipt accepted"); + }); + Case("symbolic receipt file is refused",lease=>{ + using(var journal=new NativeReceiptJournal(lease,A)) journal.Present(Payload("first")); + string file=Path.Combine(lease.Home,"owner-receipts.jsonl"),target=Path.Combine(lease.Home,"target");File.Move(file,target); + Expect(CreateSymbolicLink(file,target,2),"Symlink fixture requires Windows Developer Mode"); + Refuses(()=>{using(var journal=new NativeReceiptJournal(lease,A)){}},"Symbolic receipt accepted"); + }); + Case("released home lease revokes journal operations",lease=>{ + using(var journal=new NativeReceiptJournal(lease,A)) { + var note=journal.Present(Payload("first"));lease.Dispose(); + Refuses(()=>journal.BeginAcknowledgement((string)note["receipt"],"first"),"Released lease authorized mutation"); + } + }); + Console.WriteLine("RECEIPT_TESTS_PASS "+passed);return 0; + } +} diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index 320ed3f5aa2..265d713acdf 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -32,6 +32,13 @@ const delivered=read(path.join(home,'notification-check.json')); const acknowledgement=read(path.join(home,'notification-ack-request.json')); for(const key of ['seq','generation','note','challenge'])assert.equal(acknowledgement[key],delivered[key]); assert.equal(delivered.challenge,primary[1].arguments.observed); +const journal=fs.readFileSync(path.join(home,'home/owner-receipts.jsonl'),'utf8').trim().split('\n').map(line=>JSON.parse(line)); +assert.deepEqual(journal.map(row=>row.event),['session','presented','ack-started','acknowledged']); +assert.ok(journal.every(row=>row.generation===native.probeGeneration)); +for(const row of journal.slice(1)) { + assert.equal(row.receipt,primary[0].result.receipt); + for(const key of ['seq','generation','note','challenge'])assert.equal(row.payload[key],delivered[key]); +} assert.ok(fs.readFileSync(path.join(home,'cycle-delivery.log'),'utf8').includes(`--ack-through ${delivered.seq} --recovery-generation ${delivered.generation}`)); assert.equal(read(path.join(home,'notification-ack.json')).acknowledged,true); const operational=path.join(home,'home','state'); @@ -46,5 +53,5 @@ if(!fs.existsSync(archive)) { fs.cpSync(home,path.join(archive,'live'),{recursive:true}); } -fs.writeFileSync(path.join(state,'verification.json'),JSON.stringify({passed:true,realModelTurns:2,realToolCalls:4,threadAndReplayRejection:true,postStartup:true,buildSources:build.hashes,binaryHash:hash(build.binary),gateHash:hash(path.join(build.root,'codex-tool-gate.mjs'))},null,2)); +fs.writeFileSync(path.join(state,'verification.json'),JSON.stringify({passed:true,realModelTurns:2,realToolCalls:4,threadAndReplayRejection:true,postStartup:true,durableAcknowledgement:true,buildSources:build.hashes,binaryHash:hash(build.binary),gateHash:hash(path.join(build.root,'codex-tool-gate.mjs'))},null,2)); console.log('PASS: consolidated candidate protocol, ownership, and durable acknowledgement evidence.'); diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index 80cf639095f..ba4848a1f76 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -44,6 +44,20 @@ test('dead connection and completed turn deny even with correct IDs',async()=>{ const f=fixture();f.die();assert.equal((await f.gate.handle(check())).success,false);assert.equal(f.calls.length,0); const g=fixture();g.gate.endTurn('primary','turn');assert.equal((await g.gate.handle(check())).success,false);assert.equal(g.calls.length,0); }); +test('completed turn cannot be resurrected',()=>{ + const {gate}=fixture();gate.endTurn('primary','turn');assert.throws(()=>gate.beginTurn('primary','turn')); +}); +test('next cycle works without reauthorizing an earlier receipt',async()=>{ + let cycle=0; + const {gate,calls}=fixture(action=>action==='check'?{operationState:'delivered',notification:{...message,receipt:'receipt'+(++cycle)}}:{operationState:'acknowledged'}); + await gate.handle(check()); + assert.equal((await gate.handle(ack({arguments:{receipt:'receipt1',observed:'observed'}}))).success,true); + gate.endTurn('primary','turn');gate.beginTurn('primary','turn2'); + assert.equal((await gate.handle(check({turnId:'turn2'}))).success,true); + assert.equal((await gate.handle(ack({turnId:'turn2',arguments:{receipt:'receipt1',observed:'observed'}}))).success,false); + assert.equal((await gate.handle(ack({turnId:'turn2',callId:'second-ack',arguments:{receipt:'receipt2',observed:'observed'}}))).success,true); + assert.equal(calls.length,4); +}); test('closed gate cannot be rebound to another thread',()=>{ const {gate}=fixture();assert.throws(()=>gate.beginTurn('foreign','turn2'));gate.close();assert.throws(()=>gate.beginTurn('primary','turn2')); }); @@ -58,6 +72,18 @@ test('late delivery after turn completion grants no further action',async()=>{ const first=gate.handle(check());gate.endTurn('primary','turn');finish({operationState:'delivered',notification:message}); assert.equal((await first).success,false);assert.equal((await gate.handle(ack())).success,false);assert.equal(calls.length,1); }); +test('pending delivery can be reread without a new native operation',async()=>{ + const {gate,calls}=fixture();const first=await gate.handle(check());first.value.receipt='caller-change'; + const second=await gate.handle(check({callId:'redelivery'})); + assert.equal(second.value.receipt,'receipt');assert.equal(calls.length,1); +}); +test('notification arriving after cancellation remains available next turn',async()=>{ + let finish;const {gate,calls}=fixture(()=>new Promise(resolve=>{finish=resolve;})); + const first=gate.handle(check());gate.endTurn('primary','turn');finish({operationState:'delivered',notification:message}); + assert.equal((await first).success,false);gate.beginTurn('primary','next'); + const recovered=await gate.handle(check({turnId:'next',callId:'redelivery'})); + assert.equal(recovered.success,true);assert.equal(recovered.value.receipt,'receipt');assert.equal(calls.length,1); +}); test('operation failure is not reported as success and cannot be blindly retried',async()=>{ const {gate,calls}=fixture(()=>{throw Error('partial operation requires reconciliation');}); assert.equal((await gate.handle(check())).success,false);assert.equal((await gate.handle(check({callId:'retry'}))).success,false);assert.equal(calls.length,1); diff --git a/tests/fm-native-owner-receipt-live-e2e.test.sh b/tests/fm-native-owner-receipt-live-e2e.test.sh new file mode 100644 index 00000000000..a4f083f2374 --- /dev/null +++ b/tests/fm-native-owner-receipt-live-e2e.test.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +# Token-free native Windows receipt persistence and file-boundary tests. +set -eu +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +fm_live_gate default-on FM_LIVE_NATIVE_RECEIPTS powershell.exe git node +case "$(uname -s)" in + MINGW*|MSYS*|CYGWIN*) ;; + *) printf '%s\n' 'Native receipt tests require Windows' >&2; exit 1 ;; +esac +powershell.exe -NoProfile -NonInteractive -File "$(cygpath -w "$ROOT/tests/fixtures/native-owner/Build.ps1")" +node "$ROOT/tests/fixtures/native-owner/Lifecycle.mjs" From fd1f7f7237c082a5a8ef57fc8cd73552164365fd Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 10:17:03 +1200 Subject: [PATCH 07/61] feat(native-owner): recover completed receipts and bound shutdown --- .../NativeAcknowledgementEvidence.cs | 83 +++++++++++++++++++ bin/native-owner/NativeOperationLifetime.cs | 44 ++++++++++ bin/native-owner/NativeReceiptJournal.cs | 30 +++++-- bin/native-owner/host-lifecycle.mjs | 43 ++++++++++ docs/verification/runtime-backends.md | 17 ++-- tests/fixtures/native-owner/AppHost.mjs | 51 +++++++++--- tests/fixtures/native-owner/Build.ps1 | 3 +- tests/fixtures/native-owner/NativeDriver.cs | 26 +++++- .../native-owner/OperationLifetimeTests.cs | 39 +++++++++ tests/fixtures/native-owner/ReceiptTests.cs | 50 +++++++++++ tests/fixtures/native-owner/Run-Cycle.mjs | 27 +++++- tests/fixtures/native-owner/Verify-Cycle.mjs | 11 ++- tests/fixtures/native-owner/exercise.sh | 5 +- .../native-owner/host-lifecycle.test.mjs | 70 ++++++++++++++++ .../fixtures/native-owner/notification-ack.sh | 10 +++ tests/fm-native-owner-codex-live-e2e.test.sh | 2 + tests/fm-native-owner-tool-gate.test.sh | 2 +- 17 files changed, 479 insertions(+), 34 deletions(-) create mode 100644 bin/native-owner/NativeAcknowledgementEvidence.cs create mode 100644 bin/native-owner/NativeOperationLifetime.cs create mode 100644 bin/native-owner/host-lifecycle.mjs create mode 100644 tests/fixtures/native-owner/OperationLifetimeTests.cs create mode 100644 tests/fixtures/native-owner/host-lifecycle.test.mjs diff --git a/bin/native-owner/NativeAcknowledgementEvidence.cs b/bin/native-owner/NativeAcknowledgementEvidence.cs new file mode 100644 index 00000000000..5e6969cef19 --- /dev/null +++ b/bin/native-owner/NativeAcknowledgementEvidence.cs @@ -0,0 +1,83 @@ +// Read-only evidence for an interrupted inbox + wake acknowledgement. +// Only complete, matching durable postconditions permit recovery. Partial, +// missing, malformed, or changed evidence never authorizes a retry or deletion. +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Security.Cryptography; +using System.Text; +using System.Text.RegularExpressions; +using System.Web.Script.Serialization; + +public sealed class NativeAcknowledgementEvidence { + internal readonly NativeHomeLease Lease; + readonly Dictionary record; + NativeAcknowledgementEvidence(NativeHomeLease lease,Dictionary value) { Lease=lease;record=value; } + internal Dictionary Record() { var json=new JavaScriptSerializer();return json.Deserialize>(json.Serialize(record)); } + static string Note(Dictionary payload) { + object value; + if(!payload.TryGetValue("note",out value) || !(value is string) || !Regex.IsMatch((string)value,@"\A[A-Za-z0-9_-]+\z")) throw new IOException("Invalid inbox target"); + return (string)value; + } + static byte[] Read(string home,string relative) { + home=Path.GetFullPath(home).TrimEnd('\\','/'); + if((File.GetAttributes(home)&FileAttributes.ReparsePoint)!=0) throw new IOException("Reparse point in acknowledgement home"); + string full=Path.Combine(home,relative), parent=full; + while(!string.Equals(parent,home,StringComparison.OrdinalIgnoreCase)) { + if((File.GetAttributes(parent)&FileAttributes.ReparsePoint)!=0) throw new IOException("Reparse point in acknowledgement evidence"); + parent=Path.GetDirectoryName(parent); + if(parent==null) throw new IOException("Evidence escaped its home"); + } + using(var file=new FileStream(full,FileMode.Open,FileAccess.Read,FileShare.Read)) { + if(file.Length>16*1024*1024) throw new IOException("Acknowledgement evidence exceeds its bound"); + using(var data=new MemoryStream()) { file.CopyTo(data);return data.ToArray(); } + } + } + static string Hash(byte[] value) { using(var hash=SHA256.Create()) return BitConverter.ToString(hash.ComputeHash(value)).Replace("-","").ToLowerInvariant(); } + static List Rows(byte[] bytes,ulong cutoff) { + string text=new UTF8Encoding(false,true).GetString(bytes); + if(text.Length>0 && !text.EndsWith("\n",StringComparison.Ordinal)) throw new IOException("Incomplete wake queue"); + var result=new List(); + foreach(string line in text.Split(new [] {'\n'},StringSplitOptions.RemoveEmptyEntries)) { + string[] fields=line.Split('\t');ulong sequence,epoch; + if(fields.Length<5 || !ulong.TryParse(fields[0],NumberStyles.None,CultureInfo.InvariantCulture,out epoch) || !ulong.TryParse(fields[1],NumberStyles.None,CultureInfo.InvariantCulture,out sequence) || sequence==0 || fields[3].Length==0 || (fields[2]!="check" && fields[2]!="signal" && fields[2]!="stale" && fields[2]!="heartbeat")) throw new IOException("Malformed wake queue"); + if(sequence<=cutoff) result.Add(line); + } + return result; + } + public static NativeAcknowledgementEvidence Capture(NativeHomeLease lease,Dictionary payload) { + if(lease==null || !lease.IsHeld) throw new InvalidOperationException("An active home lease is required"); + string note=Note(payload);ulong cutoff; + if(!payload.ContainsKey("seq") || !(payload["seq"] is string) || !ulong.TryParse((string)payload["seq"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff) || cutoff==0) throw new IOException("Invalid wake cutoff"); + var rows=Rows(Read(lease.Home,Path.Combine("state",".wake-queue")),cutoff); + bool found=false;foreach(string row in rows) if(row.Split('\t')[3]=="inbox:"+note) found=true; + if(!found) throw new IOException("Inbox wake is not present at the acknowledged cutoff"); + if(File.Exists(Path.Combine(lease.Home,"state","inbox","handled",note+".note"))) throw new IOException("Inbox target is already handled or ambiguous"); + string digest=Hash(Read(lease.Home,Path.Combine("state","inbox",note+".note"))); + return new NativeAcknowledgementEvidence(lease,new Dictionary{{"version",1},{"note",note},{"cutoff",cutoff.ToString(CultureInfo.InvariantCulture)},{"noteSha256",digest},{"rows",rows.ToArray()}}); + } + internal static bool Completed(NativeHomeLease lease,Dictionary evidence) { + if(lease==null || !lease.IsHeld || evidence==null) return false; + try { + if(Convert.ToInt32(evidence["version"])!=1) return false; + string note=Note(evidence), digest=(string)evidence["noteSha256"];ulong cutoff; + if(!Regex.IsMatch(digest,@"\A[0-9a-f]{64}\z") || !ulong.TryParse((string)evidence["cutoff"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff) || cutoff==0) return false; + // A captured, nonempty target set is mandatory; an empty queue on + // its own is not evidence that an acknowledgement happened. + var targets=evidence["rows"] as System.Collections.IList; + if(targets==null || targets.Count==0) return false; + var saved=new List();foreach(object target in targets) { if(!(target is string)) return false;saved.Add((string)target); } + var validated=Rows(new UTF8Encoding(false,true).GetBytes(string.Join("\n",saved.ToArray())+"\n"),cutoff); + if(validated.Count!=targets.Count || !validated.Exists(row=>row.Split('\t')[3]=="inbox:"+note)) return false; + if(File.Exists(Path.Combine(lease.Home,"state","inbox",note+".note"))) return false; + if(Hash(Read(lease.Home,Path.Combine("state","inbox","handled",note+".note")))!=digest) return false; + return Rows(Read(lease.Home,Path.Combine("state",".wake-queue")),cutoff).Count==0; + } catch(IOException) { return false; } + catch(UnauthorizedAccessException) { return false; } + catch(ArgumentException) { return false; } + catch(KeyNotFoundException) { return false; } + catch(InvalidCastException) { return false; } + catch(FormatException) { return false; } + } +} diff --git a/bin/native-owner/NativeOperationLifetime.cs b/bin/native-owner/NativeOperationLifetime.cs new file mode 100644 index 00000000000..240a3960035 --- /dev/null +++ b/bin/native-owner/NativeOperationLifetime.cs @@ -0,0 +1,44 @@ +// Only fixed owner-operation jobs use kill-on-close. Never apply it to the +// encompassing session job or to independently owned worker jobs. +using System; +using System.ComponentModel; +using System.Runtime.InteropServices; +using System.Threading; +public static class NativeOperationLifetime { + [StructLayout(LayoutKind.Sequential)] struct Limits { + public long processTime,jobTime; + public uint flags; + public UIntPtr minimum,maximum; + public uint active; + public UIntPtr affinity; + public uint priority,scheduling; + } + [StructLayout(LayoutKind.Sequential)] struct Extended { + public Limits basic; + public ulong readOps,writeOps,otherOps,readBytes,writeBytes,otherBytes; + public UIntPtr processMemory,jobMemory,peakProcess,peakJob; + } + [StructLayout(LayoutKind.Sequential)] struct Accounting { + public long user,kernel,periodUser,periodKernel; + public uint faults,total,active,terminated; + } + [DllImport("kernel32.dll",SetLastError=true)] static extern bool SetInformationJobObject(IntPtr job,int kind,ref Extended value,uint size); + [DllImport("kernel32.dll",SetLastError=true)] static extern bool QueryInformationJobObject(IntPtr job,int kind,out Accounting value,uint size,IntPtr returned); + [DllImport("kernel32.dll",SetLastError=true)] static extern bool TerminateJobObject(IntPtr job,uint code); + public static void Configure(IntPtr ownedOperationJob) { + var value=new Extended();value.basic.flags=0x2000; + if(!SetInformationJobObject(ownedOperationJob,9,ref value,(uint)Marshal.SizeOf(typeof(Extended)))) throw new Win32Exception(Marshal.GetLastWin32Error(),"Operation kill-on-close configuration failed"); + } + public static void Stop(IntPtr ownedOperationJob,int milliseconds) { + if(milliseconds<0 || milliseconds>10000) throw new ArgumentOutOfRangeException("milliseconds"); + if(!TerminateJobObject(ownedOperationJob,125)) throw new Win32Exception(Marshal.GetLastWin32Error(),"Fixed operation stop failed"); + DateTime limit=DateTime.UtcNow.AddMilliseconds(milliseconds); + do { + Accounting value; + if(!QueryInformationJobObject(ownedOperationJob,1,out value,(uint)Marshal.SizeOf(typeof(Accounting)),IntPtr.Zero)) throw new Win32Exception(Marshal.GetLastWin32Error(),"Operation liveness is unknown"); + if(value.active==0) return; + Thread.Sleep(10); + } while(DateTime.UtcNow Present(Dictionary payload) { Dictionary Delivery(Dictionary row) { var result=Copy((Dictionary)row["payload"]);result["receipt"]=row["receipt"];return result; } - public Dictionary BeginAcknowledgement(string receipt,string observed) { + public Dictionary BeginAcknowledgement(string receipt,string observed,NativeAcknowledgementEvidence evidence=null) { EnsureOpen(); Dictionary row; if(NeedsReconciliation || receipt==null || !receipts.TryGetValue(receipt,out row) || (string)row["generation"]!=generation || (string)row["event"]!="presented") throw new InvalidOperationException("Receipt is not eligible"); var payload=(Dictionary)row["payload"]; if((string)payload["challenge"]!=observed) throw new InvalidOperationException("Notification was not observed"); - Append("ack-started",receipt,Copy(payload)); + if(evidence!=null && !object.ReferenceEquals(evidence.Lease,lease)) throw new InvalidOperationException("Evidence belongs to another home lease"); + Append("ack-started",receipt,Copy(payload),evidence==null ? null : evidence.Record()); return Delivery(receipts[receipt]); } public void CompleteAcknowledgement(string receipt) { EnsureOpen(); Dictionary row; if(receipt==null || !receipts.TryGetValue(receipt,out row) || (string)row["generation"]!=generation || (string)row["event"]!="ack-started") throw new InvalidOperationException("No matching acknowledgement attempt"); - Append("acknowledged",receipt,Copy((Dictionary)row["payload"])); + Append("acknowledged",receipt,Copy((Dictionary)row["payload"]),Evidence(row)); } - void Append(string kind,string receipt,Dictionary payload) { - var row=new Dictionary{{"version",1},{"home",home},{"generation",generation},{"event",kind},{"receipt",receipt},{"payload",payload}}; + static Dictionary Evidence(Dictionary row) { + object value;return row.TryGetValue("targetEvidence",out value) ? value as Dictionary : null; + } + public int ReconcileCompletedAcknowledgements() { + EnsureOpen(); + var pending=new List(); + foreach(var entry in receipts) if((string)entry.Value["event"]=="ack-started") pending.Add(entry.Key); + int completed=0; + foreach(string id in pending) { + var previous=receipts[id];var evidence=Evidence(previous); + if(!NativeAcknowledgementEvidence.Completed(lease,evidence)) continue; + Append("recovered-acknowledged",id,Copy((Dictionary)previous["payload"]),evidence,(string)previous["generation"]); + completed++; + } + return completed; + } + void Append(string kind,string receipt,Dictionary payload,Dictionary evidence=null,string ackGeneration=null) { + var row=new Dictionary{{"version",1},{"home",home},{"generation",generation},{"event",kind},{"receipt",receipt},{"payload",payload},{"targetEvidence",evidence},{"ackGeneration",ackGeneration}}; byte[] bytes=new UTF8Encoding(false,true).GetBytes(json.Serialize(row)+"\n"); if(bytes.Length>65536 || file.Length+bytes.Length>Limit) throw new IOException("Receipt journal capacity exceeded; durable work preserved"); try { file.Position=file.Length;file.Write(bytes,0,bytes.Length);file.Flush(true);Apply(row); } @@ -108,6 +125,9 @@ void Apply(Dictionary row) { else if(kind=="ack-started" || kind=="acknowledged") { string expected=kind=="ack-started" ? "presented" : "ack-started"; if(!exists || (string)previous["event"]!=expected || (string)previous["generation"]!=gen || json.Serialize(previous["payload"])!=json.Serialize(row["payload"])) throw new IOException("Invalid receipt transition; preserved"); + if(kind=="acknowledged" && json.Serialize(Evidence(previous))!=json.Serialize(Evidence(row))) throw new IOException("Acknowledgement evidence changed; preserved"); + } else if(kind=="recovered-acknowledged") { + if(!exists || (string)previous["event"]!="ack-started" || !row.ContainsKey("ackGeneration") || (string)row["ackGeneration"]!=(string)previous["generation"] || json.Serialize(previous["payload"])!=json.Serialize(row["payload"]) || Evidence(row)==null || json.Serialize(Evidence(previous))!=json.Serialize(Evidence(row))) throw new IOException("Invalid recovery transition; preserved"); } else throw new IOException("Unknown receipt transition; preserved"); receipts[id]=row; } diff --git a/bin/native-owner/host-lifecycle.mjs b/bin/native-owner/host-lifecycle.mjs new file mode 100644 index 00000000000..c6eded0231c --- /dev/null +++ b/bin/native-owner/host-lifecycle.mjs @@ -0,0 +1,43 @@ +// Host-owned lifecycle, never a model tool. Callbacks close the registered +// operation jobs and the retained app-server process, not a shared process tree. +export function createHostLifecycle({ gate, interrupt, stopOperations, closeInput, waitForExit, terminate, graceMs = 2000 }) { + if (!Number.isInteger(graceMs) || graceMs < 1 || graceMs > 10000) throw new TypeError('Invalid shutdown bound'); + let stopping = null; + async function bounded(action, label) { + let timer; + const controller = new AbortController(); + try { + return await Promise.race([ + Promise.resolve().then(() => action(controller.signal)), + new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(`${label} timed out`)), graceMs); }), + ]); + } finally { clearTimeout(timer); controller.abort(); } + } + return Object.freeze({ + shutdown() { + if (stopping) return stopping; + // Revoke new tool calls synchronously, before any await or interrupt. + gate.close(); + stopping = (async () => { + const errors = []; + let operationsStopped = false; + try { await bounded(interrupt, 'Turn interruption'); } catch (error) { errors.push(error.message); } + try { + operationsStopped = (await bounded(stopOperations, 'Operation shutdown')) === true; + if (!operationsStopped) errors.push('Operation shutdown was not confirmed'); + } catch (error) { errors.push(error.message); } + try { closeInput(); } catch (error) { errors.push(error.message); } + let exited = false, forced = false; + try { exited = (await bounded(waitForExit, 'App-server exit')) === true; } catch { /* Escalate to its retained handle only. */ } + if (!exited) { + forced = true; + try { terminate(); } catch (error) { errors.push(error.message); } + try { exited = (await bounded(waitForExit, 'App-server termination')) === true; } catch (error) { errors.push(error.message); } + } + if (!exited) errors.push('App-server exit was not confirmed'); + return { stopped: operationsStopped && exited, operationsStopped, exited, forced, errors }; + })(); + return stopping; + }, + }); +} diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index ab1cc111b8b..17caacbf239 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -19,10 +19,12 @@ Refresh the portable request-policy regression with: bash tests/fm-native-owner-tool-gate.test.sh ``` -The 21 behavioral cases cover primary-thread/turn binding, duplicate calls, consumed receipts, invalid arguments, arbitrary-command rejection, dead connections, overlapping calls, late delivery, repeated cycles, retired turns, read-only redelivery after cancellation, and operation failure without blind retry. +The 29 behavioral cases cover the request policy and bounded host shutdown, including synchronous revocation, single-flight shutdown, failed interruption, cancelled shutdown requests, retained-process termination, and unconfirmed-exit refusal. The native controller now persists receipt presentation, acknowledgement intent, and completion under its existing exclusive home lease. It flushes acknowledgement intent before invoking the mutation and preserves interrupted attempts for reconciliation rather than retrying them. -The token-free Windows guard exercises 12 receipt cases, including restart/generation behavior, interrupted and completed attempts, multiple cycles, torn records, file links, and lease revocation, followed by exclusive-owner, normal-exit, controller-loss, orphan-recovery, ambiguous-record, and child-scope tests: +The token-free Windows guard exercises 25 receipt cases, including complete-versus-partial recovery, preservation of newer work, missing or changed evidence, generation separation, file links, and lease revocation. +Two native operation-lifetime cases verify bounded stop and last-handle-close termination while an independent process remains alive. +The guard also retains the exclusive-owner, normal-exit, controller-loss, orphan-recovery, ambiguous-record, and child-scope tests: ```sh bash tests/fm-native-owner-receipt-live-e2e.test.sh @@ -46,9 +48,14 @@ A repeated receipt and a request from another real thread were refused without a The checkpoint exercised its three-second timeout followed by a drain, not immediate interrupt-driven delivery. Both app-server threads reported read-only filesystem policy, disabled network access, and approval policy `never`; the two explicitly authorized host operations execute outside ordinary model shell tools. Dynamic tool registration requires the experimental API capability in this version. -The actual app-server acknowledgement is also checked against the durable journal's owner generation and target notification. -This does not establish automatic reconciliation of an interrupted mutation, cancellation of an already accepted mutation, adversarial Windows path/process races, populated-fleet behavior, or other harness support. -The production launcher and ordinary-startup integration remain disabled while those boundaries are incomplete. +The actual app-server acknowledgement is also checked against the durable journal's owner generation, captured queue targets, and handled-note content hash. +The host revokes new calls before shutdown, stops only its fixed operation jobs, and confirms app-server exit through its retained process object. +Kill-on-close is applied only to fixed operation jobs, not the encompassing session or independently owned worker jobs. +Two additional model-free cases interrupt the real acknowledgement scripts after the inbox move and after the queue acknowledgement, respectively. +A new controller preserves the partial case as unresolved and reconciles the completed case from matching durable effects without replaying either mutation. +Missing, changed, malformed, reparse-point, or incomplete evidence remains unresolved; recovery does not mean retrying or undoing a partial acknowledgement. +These results do not establish active-model-turn interruption, production-wide shutdown, adversarial Windows path/process races, populated-fleet behavior, or other harness support. +The production launcher and ordinary-startup integration remain unfinished and disabled while those boundaries are incomplete. ## Harness detection precedence diff --git a/tests/fixtures/native-owner/AppHost.mjs b/tests/fixtures/native-owner/AppHost.mjs index 41150bf633a..aeb2956be42 100644 --- a/tests/fixtures/native-owner/AppHost.mjs +++ b/tests/fixtures/native-owner/AppHost.mjs @@ -1,6 +1,7 @@ // Disposable host adapter. Only this process owns the app-server connection. // Dynamic tool arguments never select a thread, executable, home, or command. import {createNotificationGate} from './codex-tool-gate.mjs'; +import {createHostLifecycle} from './host-lifecycle.mjs'; import fs from 'node:fs'; import path from 'node:path'; import net from 'node:net'; @@ -12,16 +13,18 @@ const evidence={frames:[],tools:[],native:[],primary:null,foreign:null,passed:fa const save=()=>fs.writeFileSync(path.join(home,'app-host-evidence.json'),JSON.stringify(evidence,null,2)); const pause=ms=>new Promise(resolve=>setTimeout(resolve,ms)); const base={kind:'notification',session:process.env.FM_PROBE_SESSION,home,nonce:process.env.FM_PROBE_NONCE}; -async function native(action,extra={}) { +async function native(action,extra={},signal) { const until=Date.now()+12000; for(;;) { + if(signal?.aborted)throw Error('Native request cancelled'); try { const result=await new Promise((resolve,reject)=>{ const socket=net.createConnection('\\\\.\\pipe\\'+process.env.FM_PROBE_PIPE);let buffer='',done=false; socket.setTimeout(9000,()=>socket.destroy(Error('Native operation query timed out'))); socket.on('connect',()=>socket.write(JSON.stringify({...base,action,...extra})+'\n')); socket.on('data',chunk=>{buffer+=chunk;const end=buffer.indexOf('\n');if(end>=0&&!done){done=true;try{resolve(JSON.parse(buffer.slice(0,end)));}catch(error){reject(error);}socket.end();}}); - socket.on('error',reject);socket.on('close',()=>{if(!done)reject(Error('Native channel closed without a result'));}); + const abort=()=>socket.destroy(Error('Native request cancelled'));signal?.addEventListener('abort',abort,{once:true}); + socket.on('error',reject);socket.on('close',()=>{signal?.removeEventListener('abort',abort);if(!done)reject(Error('Native channel closed without a result'));}); }); if(!result.notificationAuthorized)throw Error('Native controller denied host adapter'); evidence.native.push({action,state:result.operationState,startupExpired:result.startupExpired}); @@ -33,7 +36,13 @@ async function operation(action,extra={}) { const start=await native(action,extra); if(start.operationState!=='pending')throw Error('Operation did not start: '+start.operationState); for(let i=0;i<700;i++) { - await pause(100);const result=await native('result'); + await pause(100); + if(action==='ack'&&process.env.FM_PROBE_ACK_FAULT&&fs.existsSync(path.join(home,'ack-fault-ready'))) { + const stopped=await native('shutdown'); + if(stopped.operationState!=='stopped')throw Error('Interrupted operation did not stop'); + return {operationState:'interrupted'}; + } + const result=await native('result'); if(result.operationState==='failed')throw Error('Native notification command failed'); if(result.operationState!=='pending')return result; } @@ -41,7 +50,7 @@ async function operation(action,extra={}) { } const child=spawn(executable,['app-server','--stdio','--disable','hooks','-c','windows.sandbox=unelevated','-c','model_reasoning_effort=high'],{cwd:home,stdio:['pipe','pipe','pipe']}); let alive=true,next=0,stderr='',primary=null,receipt=null,challenge=null,acknowledged=false; -let gate=null; +let gate=null,activeTurn=null,closing=false; const pending=new Map(),completed=new Map(); const failPending=error=>{for(const value of pending.values())value.reject(error);pending.clear();}; const timer=setTimeout(()=>{child.kill();process.exitCode=1;},220000); @@ -54,7 +63,7 @@ async function tool(frame) { const p=frame.params; const record={requestId:frame.id,threadId:p.threadId,turnId:p.turnId,callId:p.callId,tool:p.tool,arguments:p.arguments}; evidence.tools.push(record); - const respond=(success,value)=>{record.success=success;record.result=value;send({id:frame.id,result:{success,contentItems:[{type:'inputText',text:JSON.stringify(value)}]}});save();}; + const respond=(success,value)=>{record.success=success;record.result=value;if(!closing&&alive)send({id:frame.id,result:{success,contentItems:[{type:'inputText',text:JSON.stringify(value)}]}});save();}; if(!gate)return respond(false,{denied:'primary-not-registered'}); const result=await gate.handle(p); if(result.success&&p.tool==='fm_notification_check'){receipt=result.value.receipt;challenge=result.value.challenge;} @@ -68,9 +77,9 @@ createInterface({input:child.stdout}).on('line',line=>{ if(event.id!==undefined&&pending.has(event.id)) { const value=pending.get(event.id);pending.delete(event.id); if(event.error)value.reject(Error(JSON.stringify(event.error))); - else {if(value.method==='turn/start'&&value.params.threadId===primary)gate.beginTurn(primary,event.result.turn.id);value.resolve(event.result);} - } else if(event.method==='turn/completed'){completed.set(event.params.turn.id,event.params.turn);gate?.endTurn(event.params.threadId,event.params.turn.id);} - else if(event.id!==undefined)send({id:event.id,error:{code:-32601,message:'No other host operations are authorized'}}); + else {if(value.method==='turn/start'&&value.params.threadId===primary){gate.beginTurn(primary,event.result.turn.id);activeTurn=event.result.turn.id;}value.resolve(event.result);} + } else if(event.method==='turn/completed'){completed.set(event.params.turn.id,event.params.turn);gate?.endTurn(event.params.threadId,event.params.turn.id);if(event.params.turn.id===activeTurn)activeTurn=null;} + else if(event.id!==undefined&&event.method)send({id:event.id,error:{code:-32601,message:'No other host operations are authorized'}}); }); const tools=[ {name:'fm_notification_check',description:'Read exactly one controlled notification through the registered Firstmate operation.',inputSchema:{type:'object',properties:{},additionalProperties:false}}, @@ -94,10 +103,15 @@ try { if(process.env.FM_PROBE_API_DRY==='1') { const delivery=await operation('check'); const ack=await operation('ack',{receipt:delivery.notification.receipt,observed:delivery.notification.challenge}); - const replay=await native('ack',{receipt:delivery.notification.receipt,observed:delivery.notification.challenge}); - if(ack.operationState!=='acknowledged'||replay.operationState!=='denied')throw Error('Native bridge preflight failed'); + if(process.env.FM_PROBE_ACK_FAULT) { + if(ack.operationState!=='interrupted')throw Error('Fault injection did not interrupt the acknowledgement'); + evidence.ackFault=process.env.FM_PROBE_ACK_FAULT; + } else { + const replay=await native('ack',{receipt:delivery.notification.receipt,observed:delivery.notification.challenge}); + if(ack.operationState!=='acknowledged'||replay.operationState!=='denied')throw Error('Native bridge preflight failed'); + } evidence.passed=true;evidence.modelFree=true; - console.log('PASS: registered post-startup operations deliver and acknowledge using the native bridge; model-free only.'); + console.log(evidence.ackFault?'PASS: fixed acknowledgement interrupted at the requested durable boundary; model-free only.':'PASS: registered post-startup operations deliver and acknowledge using the native bridge; model-free only.'); } else { evidence.primaryTurn=await turn(primary,'This is a bounded integration test in an empty disposable Firstmate home. Use only the supplied fm_notification tools; do not use shell, file, browser, or other tools. Call fm_notification_check once. Read the message, then call fm_notification_ack with its receipt and the observed challenge. After successful acknowledgement, repeat that same acknowledgement exactly once to test replay rejection. Report the three results and stop. Do not retry anything else.'); if(!acknowledged)throw Error('Primary did not acknowledge notification'); @@ -114,6 +128,17 @@ try { console.log('PASS: primary received and acknowledged the notification; receipt replay and a second real thread were denied.'); } } catch(error) {evidence.fatal=error.stack;process.exitCode=1;console.error(error.stack);} finally { - save();fs.writeFileSync(path.join(home,'app-server.stderr'),stderr);child.stdin.end(); - const closeTimer=setTimeout(()=>child.kill(),3000);child.once('exit',()=>{clearTimeout(closeTimer);clearTimeout(timer);});if(!alive){clearTimeout(closeTimer);clearTimeout(timer);} + closing=true; + const lifecycle=createHostLifecycle({ + gate:{close:()=>gate?.close()}, + interrupt:()=>{if(activeTurn&&alive)void request('turn/interrupt',{threadId:primary,turnId:activeTurn}).catch(()=>{});}, + stopOperations:async signal=>(await native('shutdown',{},signal)).operationState==='stopped', + closeInput:()=>child.stdin.end(), + waitForExit:signal=>!alive?Promise.resolve(true):new Promise(resolve=>{const stop=()=>{child.removeListener('exit',exit);resolve(false);};const exit=()=>{signal.removeEventListener('abort',stop);resolve(true);};child.once('exit',exit);signal.addEventListener('abort',stop,{once:true});}), + terminate:()=>child.kill(),graceMs:5000, + }); + evidence.shutdown=await lifecycle.shutdown(); + if(!evidence.shutdown.stopped){evidence.passed=false;process.exitCode=1;} + clearTimeout(timer); + save();fs.writeFileSync(path.join(home,'app-server.stderr'),stderr); } diff --git a/tests/fixtures/native-owner/Build.ps1 b/tests/fixtures/native-owner/Build.ps1 index 8739aa4d472..c281f1c931f 100644 --- a/tests/fixtures/native-owner/Build.ps1 +++ b/tests/fixtures/native-owner/Build.ps1 @@ -8,6 +8,7 @@ $root = Join-Path ([IO.Path]::GetTempPath()) ('fm-native-candidate-' + [guid]::N New-Item -ItemType Directory $root | Out-Null $binary = Join-Path $root 'SessionProbe.exe' $sources = @((Join-Path $repo 'bin/native-owner/NativeOwner.cs'), (Join-Path $repo 'bin/native-owner/NativeHomeLease.cs'), (Join-Path $PSScriptRoot 'NativeDriver.cs'), (Join-Path $repo 'bin/native-owner/NativeReceiptJournal.cs'), (Join-Path $PSScriptRoot 'ReceiptTests.cs')) +$sources += @((Join-Path $repo 'bin/native-owner/NativeAcknowledgementEvidence.cs'), (Join-Path $repo 'bin/native-owner/NativeOperationLifetime.cs'), (Join-Path $PSScriptRoot 'OperationLifetimeTests.cs')) Add-Type -Path $sources -OutputAssembly $binary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll & $binary receipt-tests if ($LASTEXITCODE -ne 0) { throw 'Durable receipt lifecycle tests failed' } @@ -18,7 +19,7 @@ if ($LASTEXITCODE -ne 0) { throw 'Disposable clone failed' } if ($LASTEXITCODE -ne 0) { throw 'Consumer test integration no longer applies; reconcile it explicitly' } foreach ($name in @('exercise.sh','notification-check.sh','notification-ack.sh')) { Copy-Item (Join-Path $PSScriptRoot $name) (Join-Path $copy $name) } Copy-Item (Join-Path $PSScriptRoot 'AppHost.mjs') (Join-Path $root 'AppHost.mjs') -Copy-Item (Join-Path $repo 'bin/native-owner/codex-tool-gate.mjs') (Join-Path $root 'codex-tool-gate.mjs') +foreach ($module in @('codex-tool-gate.mjs','host-lifecycle.mjs')) { Copy-Item (Join-Path $repo ('bin/native-owner/' + $module)) (Join-Path $root $module) } Copy-Item $binary (Join-Path $copy 'bin/fm-native-owner.exe') New-Item -ItemType Directory (Join-Path $root 'tools') | Out-Null Copy-Item (Join-Path $PSScriptRoot 'jq') (Join-Path $root 'tools/jq') diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 8311ed98d81..add309411b0 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -15,6 +15,7 @@ public static partial class NativeOwner { static NativeReceiptJournal operationJournal; + static bool shutdownRequested; static string LogonSid() { using(var identity=WindowsIdentity.GetCurrent()) { foreach(var row in TokenGroups(identity.Token,2)) @@ -51,6 +52,7 @@ static ChildScope StartScope(string role, IntPtr parentJob, IntPtr environment, IntPtr operationEnvironment=IntPtr.Zero; if(scope.job==IntPtr.Zero) throw Error("CreateJobObject child scope"); try { + if(role=="owner-operation") NativeOperationLifetime.Configure(scope.job); string operation=role=="owner-operation" ? (purpose=="startup" ? "owner-operation" : "notification-operation "+purpose) : "scoped-client "+role; if(role=="owner-operation") { // Only this fixed, non-extensible test operation receives the grant. @@ -157,14 +159,21 @@ static int Run(string configPath) { Process outsider = null; var scopes=new List(); NativeHomeLease lease=null; + int recoveredAcknowledgements=0; try { if(config.ContainsKey("leaseHome")) { lease=new NativeHomeLease((string)config["leaseHome"]); operationJournal=new NativeReceiptJournal(lease,session); + recoveredAcknowledgements=operationJournal.ReconcileCompletedAcknowledgements(); } var values = EnvironmentFor(pipeName, session, home, nonce); values["MSYS"]="winsymlinks:nativestrict"; if(config.ContainsKey("apiDry") && (bool)config["apiDry"]) values["FM_PROBE_API_DRY"]="1"; + if(config.ContainsKey("ackFault")) { + string fault=(string)config["ackFault"]; + if(!values.ContainsKey("FM_PROBE_API_DRY") || (fault!="partial" && fault!="complete")) throw new ArgumentException("Fault injection is limited to the model-free fixture"); + values["FM_PROBE_ACK_FAULT"]=fault; + } if(lease!=null) { values["FM_PROBE_LEASE_HOME"]=lease.Home; values["FM_PROBE_LEASE_GENERATION"]=session; values["FM_HOME"]=lease.Home.Replace('\\','/'); } if(config.ContainsKey("ownerExercise") && (bool)config["ownerExercise"]) values["FM_PROBE_EXERCISE"]="1"; if(config.ContainsKey("boundaries") && (bool)config["boundaries"]) values["FM_PROBE_BOUNDARIES"]="1"; @@ -247,6 +256,7 @@ static int Run(string configPath) { {"rootExit",code},{"timedOut",timedOut},{"jobAssigned",assigned},{"pipeAcl",pipeAcl}, {"probeGeneration",session},{"probeLeaseHeld",lease!=null}, {"pipeDacl",security.GetSecurityDescriptorSddlForm(AccessControlSections.Access)}, + {"recoveredAcknowledgements",recoveredAcknowledgements},{"receiptNeedsReconciliation",operationJournal!=null && operationJournal.NeedsReconciliation}, {"authorityImplemented",false},{"notificationCheckStarts",checkStarts},{"notificationAckStarts",ackStarts},{"notificationConsumed",consumed},{"observations",observations} }; if (outsider != null) { @@ -277,6 +287,14 @@ static void NotificationRequest(Dictionary request,Dictionary request,Dictionary request,Dictionary request,Dictionary=3 && args[0]=="owner") return OwnerClient(args[1],args[2],args.Length>3 ? args[3] : ""); if(args.Length>0 && args[0]=="client") return Client(args.Length>1 ? args[1] : "agent-tool"); if(args.Length==2 && args[0]=="sleep") { int ms=int.Parse(args[1]); if(ms<0 || ms>15000) throw new ArgumentException("Sleep must be bounded"); Thread.Sleep(ms); return 0; } diff --git a/tests/fixtures/native-owner/OperationLifetimeTests.cs b/tests/fixtures/native-owner/OperationLifetimeTests.cs new file mode 100644 index 00000000000..cff46072573 --- /dev/null +++ b/tests/fixtures/native-owner/OperationLifetimeTests.cs @@ -0,0 +1,39 @@ +using System; +using System.Diagnostics; +using System.IO; +using System.Runtime.InteropServices; +using System.Text; +public static partial class NativeOwner { + static PI LifetimeChild(IntPtr job) { + PI child=new PI();var startup=new SI { cb=Marshal.SizeOf(typeof(SI)) }; + if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" sleep 10000"),IntPtr.Zero,IntPtr.Zero,false,0x4|0x400,IntPtr.Zero,Path.GetTempPath(),ref startup,out child)) throw Error("Create lifetime child"); + try { + if(!AssignProcessToJobObject(job,child.process)) throw Error("Assign lifetime child"); + if(ResumeThread(child.thread)==0xffffffff) throw Error("Resume lifetime child"); + return child; + } catch { TerminateProcess(child.process,125);CloseHandle(child.thread);CloseHandle(child.process);throw; } + } + static int TestOperationLifetime() { + using(var independent=Process.Start(new ProcessStartInfo(OwnExe,"sleep 15000") { UseShellExecute=false })) { + try { + foreach(bool close in new [] {false,true}) { + IntPtr job=CreateJobObject(IntPtr.Zero,null);PI child=new PI(); + if(job==IntPtr.Zero) throw Error("Create lifetime job"); + try { + NativeOperationLifetime.Configure(job);child=LifetimeChild(job); + if(WaitForSingleObject(child.process,0)!=WAIT_TIMEOUT) throw new Exception("Operation was not initially live"); + if(close) { CloseHandle(job);job=IntPtr.Zero; } + else NativeOperationLifetime.Stop(job,3000); + if(WaitForSingleObject(child.process,3000)!=0) throw new Exception("Fixed operation survived shutdown"); + if(independent.HasExited) throw new Exception("Independent process was stopped"); + Console.WriteLine("PASS: operation "+(close ? "last-handle close" : "bounded stop")+" preserves independent process"); + } finally { + if(child.process!=IntPtr.Zero) { if(WaitForSingleObject(child.process,0)==WAIT_TIMEOUT) TerminateProcess(child.process,125);CloseHandle(child.thread);CloseHandle(child.process); } + if(job!=IntPtr.Zero) CloseHandle(job); + } + } + } finally { if(!independent.HasExited) independent.Kill();independent.WaitForExit(3000); } + } + return 0; + } +} diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index 31e8ad8d0d2..660060a0ee4 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -16,6 +16,14 @@ static void Case(string name,Action test) { using(var lease=new NativeHomeLease(home)) test(lease); passed++;Console.WriteLine("PASS: "+name); } + static string Queue(NativeHomeLease lease) { return Path.Combine(lease.Home,"state",".wake-queue"); } + static string Pending(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","note-id.note"); } + static string Handled(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","handled","note-id.note"); } + static void Targets(NativeHomeLease lease) { + Directory.CreateDirectory(Path.GetDirectoryName(Handled(lease))); + File.WriteAllText(Pending(lease),"original captured inbox record\n"); + File.WriteAllText(Queue(lease),"1\t1\tcheck\tinbox:note-id\tcaptain inbox note\n"); + } public static int Run() { Case("one writer and unobserved acknowledgement refusal",lease=>{ using(var journal=new NativeReceiptJournal(lease,A)) { @@ -102,6 +110,48 @@ public static int Run() { Refuses(()=>journal.BeginAcknowledgement((string)note["receipt"],"first"),"Released lease authorized mutation"); } }); + foreach(string scenario in new [] {"complete","newer","pending","note-only","wake-only","changed-note","missing-queue","malformed-queue","old-row-remains","no-evidence","duplicate-pending","reparse-handled"}) { + Case("interrupted recovery: "+scenario,lease=>{ + Targets(lease);string receipt; + using(var journal=new NativeReceiptJournal(lease,A)) { + var delivery=journal.Present(Payload("first"));receipt=(string)delivery["receipt"]; + var evidence=NativeAcknowledgementEvidence.Capture(lease,delivery); + journal.BeginAcknowledgement(receipt,"first",scenario=="no-evidence" ? null : evidence); + } + if(scenario!="pending" && scenario!="wake-only") File.Move(Pending(lease),Handled(lease)); + if(scenario!="pending" && scenario!="note-only") File.WriteAllText(Queue(lease),""); + if(scenario=="newer") File.WriteAllText(Queue(lease),"2\t2\tcheck\tnew-work\tuntouched\n"); + if(scenario=="changed-note") File.AppendAllText(Handled(lease),"changed"); + if(scenario=="missing-queue") File.Delete(Queue(lease)); + if(scenario=="malformed-queue") File.WriteAllText(Queue(lease),"torn"); + if(scenario=="old-row-remains") File.WriteAllText(Queue(lease),"1\t1\tcheck\tdifferent-key\tunknown\n"); + if(scenario=="duplicate-pending") File.Copy(Handled(lease),Pending(lease)); + if(scenario=="reparse-handled") { + string original=Path.GetDirectoryName(Handled(lease)),other=Path.Combine(lease.Home,"other-handled");Directory.Move(original,other); + Expect(CreateSymbolicLink(original,other,3),"Directory symlink fixture failed"); + } + string before=File.Exists(Queue(lease)) ? File.ReadAllText(Queue(lease)) : null; + bool complete=scenario=="complete" || scenario=="newer"; + using(var journal=new NativeReceiptJournal(lease,B)) { + Expect(journal.ReconcileCompletedAcknowledgements()==(complete ? 1 : 0),"Incorrect recovery classification: "+scenario); + Expect(journal.NeedsReconciliation!=complete,"Incorrect reconciliation obligation"); + Refuses(()=>journal.BeginAcknowledgement(receipt,"first"),"Old receipt became reusable"); + Expect(journal.ReconcileCompletedAcknowledgements()==0,"Recovery replay changed history"); + } + using(var journal=new NativeReceiptJournal(lease,B)) Expect(journal.NeedsReconciliation!=complete,"Recovery result did not survive reopening"); + Expect(before==(File.Exists(Queue(lease)) ? File.ReadAllText(Queue(lease)) : null),"Recovery mutated wake data"); + }); + } + Case("acknowledgement evidence cannot cross homes",lease=>{ + Targets(lease); + string otherHome=Path.Combine(Path.GetTempPath(),"fm-receipts-other-"+Guid.NewGuid().ToString("N")); + using(var other=new NativeHomeLease(otherHome)) using(var journal=new NativeReceiptJournal(lease,A)) { + Targets(other);var delivery=journal.Present(Payload("first")); + var evidence=NativeAcknowledgementEvidence.Capture(other,delivery); + Refuses(()=>journal.BeginAcknowledgement((string)delivery["receipt"],"first",evidence),"Foreign home evidence accepted"); + Expect(!journal.NeedsReconciliation,"Rejected evidence created an attempt"); + } + }); Console.WriteLine("RECEIPT_TESTS_PASS "+passed);return 0; } } diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs index 4c4f2e87078..a46143c32ec 100644 --- a/tests/fixtures/native-owner/Run-Cycle.mjs +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -8,21 +8,44 @@ const dir=path.join(repo,'data/native-candidate-validation'); const read=file=>JSON.parse(fs.readFileSync(file,'utf8').replace(/^\uFEFF/,'')); const build=read(path.join(dir,'build.json')); const dry=process.argv.includes('--dry'); +const fault=process.argv.find(value=>value.startsWith('--fault='))?.slice(8); +if(fault&&(!dry||!['partial','complete'].includes(fault)))throw Error('Fault cases require --dry and partial or complete'); if(!dry&&process.env.FM_LIVE_NATIVE_CODEX!=='1')throw Error('Live model test requires FM_LIVE_NATIVE_CODEX=1'); if(!dry){const preflight=read(path.join(dir,'bridge-preflight.json'));if(!preflight.passed||preflight.binaryHash!==createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex'))throw Error('Run model-free bridge preflight for this binary first');} const home=path.join(build.root,'appserver-'+randomUUID());fs.mkdirSync(home); fs.writeFileSync(path.join(home,'build.json'),JSON.stringify(build,null,2)); const script=path.join(build.root,'AppHost.mjs'); const spec={home,leaseHome:path.join(home,'home'),executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'UserOnly',apiDry:dry}; +if(fault)spec.ackFault=fault; const file=path.join(home,'spec.json');fs.writeFileSync(file,JSON.stringify(spec,null,2)); const run=spawnSync(build.binary,['run',file],{encoding:'utf8',timeout:310000}); fs.writeFileSync(path.join(home,'controller.stdout'),run.stdout||'');fs.writeFileSync(path.join(home,'controller.stderr'),run.stderr||''); -fs.writeFileSync(path.join(dir,dry?'bridge-latest.json':'cycle-latest.json'),JSON.stringify({home,exit:run.status},null,2)); +fs.writeFileSync(path.join(dir,fault?`fault-${fault}-latest.json`:dry?'bridge-latest.json':'cycle-latest.json'),JSON.stringify({home,exit:run.status},null,2)); console.log(JSON.stringify({home,exit:run.status,dry})); if(run.status!==0)throw Error('Bounded app-server run failed; inspect evidence, do not start another model attempt'); const host=read(path.join(home,'app-host-evidence.json')),native=read(path.join(home,'result.json')); -if(!host.passed||native.notificationCheckStarts!==1||native.notificationAckStarts!==1||!native.notificationConsumed)throw Error('Notification cycle incomplete'); +if(!host.shutdown?.stopped||!host.shutdown.operationsStopped||!host.shutdown.exited)throw Error('Host shutdown was not confirmed'); +if(!host.passed||native.notificationCheckStarts!==1||native.notificationAckStarts!==1||native.notificationConsumed!==!fault)throw Error('Notification cycle incomplete'); if(!host.native.filter(row=>row.action==='check'||row.action==='ack').every(row=>row.startupExpired))throw Error('Startup scope still active'); +if(fault) { + const queue=path.join(spec.leaseHome,'state/.wake-queue'),before=fs.readFileSync(queue); + const journal=()=>fs.readFileSync(path.join(spec.leaseHome,'owner-receipts.jsonl'),'utf8').trim().split('\n').map(JSON.parse); + if(journal().at(-1).event!=='ack-started'||!native.receiptNeedsReconciliation)throw Error('Interrupted intent was not preserved'); + if((before.length===0)!==(fault==='complete'))throw Error('Fault did not land at the requested mutation boundary'); + const recovery=path.join(home,'recovery');fs.mkdirSync(recovery); + const recoverySpec={home:recovery,leaseHome:spec.leaseHome,executable:build.binary,arguments:'sleep 100',timeoutSeconds:10,pipeAcl:'UserOnly'}; + const recoveryFile=path.join(recovery,'spec.json');fs.writeFileSync(recoveryFile,JSON.stringify(recoverySpec)); + const restarted=spawnSync(build.binary,['run',recoveryFile],{encoding:'utf8',timeout:15000}); + fs.writeFileSync(path.join(recovery,'controller.stdout'),restarted.stdout||'');fs.writeFileSync(path.join(recovery,'controller.stderr'),restarted.stderr||''); + if(restarted.status!==0)throw Error('Recovery controller failed'); + const recovered=read(path.join(recovery,'result.json')); + if(recovered.recoveredAcknowledgements!==(fault==='complete'?1:0)||recovered.receiptNeedsReconciliation!==(fault==='partial')||recovered.notificationAckStarts!==0)throw Error('Incorrect interrupted acknowledgement recovery'); + if(!fs.readFileSync(queue).equals(before))throw Error('Recovery replayed a wake mutation'); + const history=journal(); + if(fault==='complete'&&(history.at(-1).event!=='recovered-acknowledged'||history.at(-1).ackGeneration!==native.probeGeneration||history.at(-1).generation!==recovered.probeGeneration))throw Error('Recovery generations are not bound'); + console.log(`PASS: actual ${fault} acknowledgement interruption; recovery ${fault==='complete'?'confirmed completed effects without replay':'preserved the unresolved partial mutation'}.`); + process.exit(0); +} if(fs.existsSync(path.join(spec.leaseHome,'state/.wake-queue'))&&fs.readFileSync(path.join(spec.leaseHome,'state/.wake-queue'),'utf8').trim())throw Error('Queue was not acknowledged'); if(dry)fs.writeFileSync(path.join(dir,'bridge-preflight.json'),JSON.stringify({passed:true,home,binaryHash:createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex')},null,2)); console.log(dry?'PASS: model-free registered operation bridge.':'PASS: real app-server notification cycle, handling, acknowledgement, replay refusal, and foreign-thread denial.'); diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index 265d713acdf..2d749c97055 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -14,6 +14,7 @@ const build=read(path.join(home,'build.json')); const host=read(path.join(home,'app-host-evidence.json')); const native=read(path.join(home,'result.json')); assert.equal(latest.exit,0);assert.equal(native.rootExit,0);assert.equal(host.passed,true); +assert.deepEqual(host.shutdown,{stopped:true,operationsStopped:true,exited:true,forced:false,errors:[]}); assert.notEqual(host.primary,host.foreign);assert.equal(host.tools.length,4); const requests=host.frames.filter(frame=>frame.method==='item/tool/call'); assert.equal(requests.length,4); @@ -43,7 +44,13 @@ assert.ok(fs.readFileSync(path.join(home,'cycle-delivery.log'),'utf8').includes( assert.equal(read(path.join(home,'notification-ack.json')).acknowledged,true); const operational=path.join(home,'home','state'); assert.equal(fs.readFileSync(path.join(operational,'.wake-queue'),'utf8').trim(),''); -assert.ok(fs.existsSync(path.join(operational,'inbox/handled',delivered.note+'.note'))); +const handled=path.join(operational,'inbox/handled',delivered.note+'.note'); +assert.ok(fs.existsSync(handled)); +const targets=journal[2].targetEvidence; +assert.equal(targets.note,delivered.note);assert.equal(targets.cutoff,delivered.seq); +assert.equal(targets.noteSha256,hash(handled)); +assert.ok(targets.rows.some(row=>row.split('\t')[3]===`inbox:${delivered.note}`)); +assert.deepEqual(journal[3].targetEvidence,targets); const threads=host.frames.filter(frame=>frame.result?.thread); assert.equal(threads.length,2); for(const thread of threads){assert.equal(thread.result.sandbox.type,'readOnly');assert.equal(thread.result.sandbox.networkAccess,false);assert.equal(thread.result.approvalPolicy,'never');} @@ -53,5 +60,5 @@ if(!fs.existsSync(archive)) { fs.cpSync(home,path.join(archive,'live'),{recursive:true}); } -fs.writeFileSync(path.join(state,'verification.json'),JSON.stringify({passed:true,realModelTurns:2,realToolCalls:4,threadAndReplayRejection:true,postStartup:true,durableAcknowledgement:true,buildSources:build.hashes,binaryHash:hash(build.binary),gateHash:hash(path.join(build.root,'codex-tool-gate.mjs'))},null,2)); +fs.writeFileSync(path.join(state,'verification.json'),JSON.stringify({passed:true,realModelTurns:2,realToolCalls:4,threadAndReplayRejection:true,postStartup:true,durableAcknowledgement:true,buildSources:build.hashes,binaryHash:hash(build.binary),gateHash:hash(path.join(build.root,'codex-tool-gate.mjs')),lifecycleHash:hash(path.join(build.root,'host-lifecycle.mjs')),confirmedShutdown:host.shutdown},null,2)); console.log('PASS: consolidated candidate protocol, ownership, and durable acknowledgement evidence.'); diff --git a/tests/fixtures/native-owner/exercise.sh b/tests/fixtures/native-owner/exercise.sh index 88bd27d8a88..a43ecc54b83 100644 --- a/tests/fixtures/native-owner/exercise.sh +++ b/tests/fixtures/native-owner/exercise.sh @@ -19,7 +19,10 @@ bin/fm-sessionstart-run.sh --source startup > "$LOG/startup.log" 2>&1 fm_session_lock_owned_by_self "$FM_HOME/state" identity=$(<"$FM_HOME/state/.lock") [ "$(<"$FM_HOME/state/.session-start-complete")" = "$identity" ] -! grep -q 'READ-ONLY SESSION\|SESSION START INCOMPLETE' "$LOG/startup.log" +if grep -q 'READ-ONLY SESSION\|SESSION START INCOMPLETE' "$LOG/startup.log"; then + printf 'Startup did not complete with ownership\n' >&2 + exit 1 +fi printf 'STARTUP_COMPLETION_PASS\n' # Observe the real detached worker; do not reimplement or invoke its work twice. for ((i=0; i<150; i++)); do diff --git a/tests/fixtures/native-owner/host-lifecycle.test.mjs b/tests/fixtures/native-owner/host-lifecycle.test.mjs new file mode 100644 index 00000000000..fce0fcf05af --- /dev/null +++ b/tests/fixtures/native-owner/host-lifecycle.test.mjs @@ -0,0 +1,70 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {createHostLifecycle} from '../../../bin/native-owner/host-lifecycle.mjs'; +function setup(overrides = {}) { + const events = []; + const lifecycle = createHostLifecycle({ + gate: {close: () => events.push('revoke')}, + interrupt: async () => events.push('interrupt'), + stopOperations: async () => { events.push('operations'); return true; }, + closeInput: () => events.push('eof'), + waitForExit: async () => { events.push('exited'); return true; }, + terminate: () => events.push('terminate'), graceMs: 25, ...overrides, + }); + return {lifecycle, events}; +} +test('shutdown revokes synchronously and is single-flight', async () => { + const {lifecycle, events} = setup(); + const first = lifecycle.shutdown(), second = lifecycle.shutdown(); + assert.equal(first, second); + assert.deepEqual(events, ['revoke']); + const result = await first; + assert.deepEqual(events, ['revoke', 'interrupt', 'operations', 'eof', 'exited']); + assert.deepEqual(result, {stopped:true, operationsStopped:true, exited:true, forced:false, errors:[]}); +}); +test('unconfirmed operation shutdown is not reported as success', async () => { + const {lifecycle, events} = setup({stopOperations: async () => false}); + const result = await lifecycle.shutdown(); + assert.equal(result.stopped, false); + assert.equal(result.exited, true); + assert(events.includes('eof')); +}); +test('failed interruption does not prevent cleanup', async () => { + const {lifecycle, events} = setup({interrupt: async () => { throw Error('interrupt unavailable'); }}); + const result = await lifecycle.shutdown(); + assert.equal(result.stopped, true); + assert.deepEqual(result.errors, ['interrupt unavailable']); + assert(events.includes('operations')); +}); +test('operation timeout aborts its request and still closes app-server', async () => { + let aborted = false; + const {lifecycle, events} = setup({stopOperations: signal => new Promise(() => { signal.addEventListener('abort', () => { aborted = true; }); })}); + const result = await lifecycle.shutdown(); + assert.equal(result.stopped, false); + assert.equal(aborted, true); + assert(events.includes('eof')); +}); +test('grace timeout terminates only through supplied retained-process action', async () => { + let waits = 0; + const {lifecycle, events} = setup({waitForExit: () => ++waits === 1 ? new Promise(() => {}) : Promise.resolve(true)}); + const result = await lifecycle.shutdown(); + assert.equal(result.stopped, true); + assert.equal(result.forced, true); + assert.equal(events.filter(value => value === 'terminate').length, 1); +}); +test('unconfirmed process exit remains a failure after termination', async () => { + const {lifecycle, events} = setup({waitForExit: async () => false}); + const result = await lifecycle.shutdown(); + assert.equal(result.stopped, false); + assert.equal(result.exited, false); + assert(events.includes('terminate')); +}); +test('termination error is retained and not mistaken for exit', async () => { + const {lifecycle} = setup({waitForExit: async () => false, terminate: () => { throw Error('termination denied'); }}); + const result = await lifecycle.shutdown(); + assert.equal(result.stopped, false); + assert(result.errors.includes('termination denied')); +}); +test('invalid bounds are refused before lifecycle effects', () => { + for (const graceMs of [0, -1, 10001, Infinity, 1.5]) assert.throws(() => setup({graceMs})); +}); diff --git a/tests/fixtures/native-owner/notification-ack.sh b/tests/fixtures/native-owner/notification-ack.sh index a7ab57f3b33..2baca49551c 100644 --- a/tests/fixtures/native-owner/notification-ack.sh +++ b/tests/fixtures/native-owner/notification-ack.sh @@ -18,7 +18,17 @@ case "$note" in ''|*[!A-Za-z0-9._-]*) exit 2 ;; esac case "$seq" in ''|*[!0-9]*) exit 2 ;; esac case "$generation" in ''|*[!A-Za-z0-9._-]*) exit 2 ;; esac bin/fm-inbox.sh drain --ack "$note" > "$LOG/cycle-inbox-ack.log" +if [ "${FM_PROBE_ACK_FAULT:-}" = partial ]; then + printf 'partial\n' > "$LOG/ack-fault-ready" + sleep 30 + exit 125 +fi bin/fm-wake-drain.sh --ack-through "$seq" --recovery-generation "$generation" > "$LOG/cycle-ack.log" 2>&1 [ ! -s "$FM_HOME/state/.wake-queue" ] [ -f "$FM_HOME/state/inbox/handled/$note.note" ] +if [ "${FM_PROBE_ACK_FAULT:-}" = complete ]; then + printf 'complete\n' > "$LOG/ack-fault-ready" + sleep 30 + exit 125 +fi printf '{"acknowledged":true,"queueEmpty":true}\n' > "$LOG/notification-ack.json" diff --git a/tests/fm-native-owner-codex-live-e2e.test.sh b/tests/fm-native-owner-codex-live-e2e.test.sh index ce0115f332c..1e2af4ae9cd 100644 --- a/tests/fm-native-owner-codex-live-e2e.test.sh +++ b/tests/fm-native-owner-codex-live-e2e.test.sh @@ -13,5 +13,7 @@ export FM_LIVE_NATIVE_CODEX=1 fixture="$ROOT/tests/fixtures/native-owner" powershell.exe -NoProfile -NonInteractive -File "$(cygpath -w "$fixture/Build.ps1")" node "$fixture/Run-Cycle.mjs" --dry +node "$fixture/Run-Cycle.mjs" --dry --fault=partial +node "$fixture/Run-Cycle.mjs" --dry --fault=complete node "$fixture/Run-Cycle.mjs" node "$fixture/Verify-Cycle.mjs" diff --git a/tests/fm-native-owner-tool-gate.test.sh b/tests/fm-native-owner-tool-gate.test.sh index f52139a0d18..cd30e9413e9 100644 --- a/tests/fm-native-owner-tool-gate.test.sh +++ b/tests/fm-native-owner-tool-gate.test.sh @@ -2,4 +2,4 @@ # Portable behavioral tests of the host-side notification request policy. set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) -node --test "$ROOT/tests/fixtures/native-owner/tool-gate.test.mjs" +node --test "$ROOT/tests/fixtures/native-owner/tool-gate.test.mjs" "$ROOT/tests/fixtures/native-owner/host-lifecycle.test.mjs" From d4c848cfa10655d591fb5d790fde3869bbcb8e56 Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 12:39:35 +1200 Subject: [PATCH 08/61] feat(native-owner): integrate experimental Codex launcher --- .gitattributes | 5 +- .gitignore | 3 + bin/fm-harness.sh | 11 ++ bin/fm-native-codex.ps1 | 70 +++++++ bin/fm-session-lock-lib.sh | 44 ++++- bin/fm-sessionstart-nudge.sh | 2 +- bin/fm-test-run.sh | 5 +- .../NativeAcknowledgementEvidence.cs | 66 ++++--- bin/native-owner/NativeHomeLease.cs | 24 ++- bin/native-owner/NativeLauncher.cs | 149 +++++++++++++++ bin/native-owner/NativeOperations.cs | 124 ++++++++++++ bin/native-owner/NativeOwner.cs | 2 +- bin/native-owner/ack.sh | 25 +++ bin/native-owner/check.sh | 38 ++++ bin/native-owner/codex-host.mjs | 178 ++++++++++++++++++ bin/native-owner/codex-tool-gate.mjs | 3 + bin/native-owner/startup.sh | 25 +++ bin/native-owner/tools/jq | 11 ++ docs/verification/runtime-backends.md | 64 ++++++- tests/fixtures/native-owner/Build.ps1 | 6 +- tests/fixtures/native-owner/Launcher.mjs | 127 +++++++++++++ tests/fixtures/native-owner/NativeDriver.cs | 100 ---------- tests/fixtures/native-owner/ReceiptTests.cs | 23 +++ .../native-owner/consumer-adapter.patch | 96 ---------- .../fixtures/native-owner/tool-gate.test.mjs | 12 ++ .../fm-native-owner-launcher-live-e2e.test.sh | 10 + tests/fm-session-lock-ancestry.test.sh | 27 +++ 27 files changed, 1010 insertions(+), 240 deletions(-) create mode 100644 bin/fm-native-codex.ps1 create mode 100644 bin/native-owner/NativeLauncher.cs create mode 100644 bin/native-owner/NativeOperations.cs create mode 100755 bin/native-owner/ack.sh create mode 100755 bin/native-owner/check.sh create mode 100644 bin/native-owner/codex-host.mjs create mode 100755 bin/native-owner/startup.sh create mode 100755 bin/native-owner/tools/jq create mode 100644 tests/fixtures/native-owner/Launcher.mjs delete mode 100644 tests/fixtures/native-owner/consumer-adapter.patch create mode 100755 tests/fm-native-owner-launcher-live-e2e.test.sh diff --git a/.gitattributes b/.gitattributes index a075a637e90..3299242ecc8 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,6 +1,5 @@ # Bash parses shell scripts with LF line endings on every supported platform. *.sh text eol=lf -# Test overlays and the extensionless Bash fixture must survive Windows clones. -# Unified diff context includes intentional space-only lines. -tests/fixtures/native-owner/consumer-adapter.patch text eol=lf whitespace=-blank-at-eol +# Extensionless Bash helpers must survive Windows clones. tests/fixtures/native-owner/jq text eol=lf +bin/native-owner/tools/jq text eol=lf diff --git a/.gitignore b/.gitignore index 3eece43c35f..99a0005ffb2 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,6 @@ __pycache__/ config/ .tools/ +# Local native candidate build; never distributed as a tracked executable. +/bin/fm-native-owner.exe +/bin/fm-native-owner.build diff --git a/bin/fm-harness.sh b/bin/fm-harness.sh index 7989643f1b6..eda6c5f374a 100755 --- a/bin/fm-harness.sh +++ b/bin/fm-harness.sh @@ -395,6 +395,17 @@ harness_family() { # a harness-shaped path in some node process's arguments is weaker evidence # than a harness publishing its own identity. detect_own() { + local native_state native_home native_harness + native_home="${FM_STATE_OVERRIDE:-$FM_HOME/state}" + native_home=${native_home%/state} + case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) + if [ -f "$native_home/owner-probe.json" ]; then + native_state=$(cygpath -w "${FM_STATE_OVERRIDE:-$FM_HOME/state}") || { echo unknown; return; } + native_harness=$(MSYS2_ARG_CONV_EXCL='*' "$SCRIPT_DIR/fm-native-owner.exe" owner harness "$native_state" 2>/dev/null) || { echo unknown; return; } + case "$native_harness" in codex) echo codex ;; *) echo unknown ;; esac + return + fi ;; + esac local marker ancestry strength harness marker=$(harness_marker) ancestry=$(harness_ancestry) diff --git a/bin/fm-native-codex.ps1 b/bin/fm-native-codex.ps1 new file mode 100644 index 00000000000..27b49618bd6 --- /dev/null +++ b/bin/fm-native-codex.ps1 @@ -0,0 +1,70 @@ +# Explicit experimental launcher; does not install hooks or change user settings. +# -BuildOnly compiles the code-owned provider after all its sessions have stopped. +# Launch requires an empty-fleet temporary home and a local Docker image with jq. +<# +.SYNOPSIS +Build or explicitly launch the experimental native Windows Codex host. +.DESCRIPTION +Requires Windows PowerShell 5.1, native Node and Codex, Git Bash, and Docker at + their standard installation paths; Codex app-server 0.154.0 was verified. +Only empty-fleet homes beneath the user's Windows temporary directory are +accepted. Existing projects, fleet registrations, Relay and process sources +are not supported. Ordinary startup never selects this launcher. +No hooks, global settings, sandbox settings, packages, or Docker images are +installed or changed. The two fixed notification operations execute outside +the model's read-only, network-disabled sandbox under native authorization. +.PARAMETER Experimental +Required consent to launch this temporary-home-only candidate. +.PARAMETER BuildOnly +Compile the local provider and source stamp without launching a session. +.PARAMETER VerifyOnly +Connect startup and app-server without starting any model turns. +.PARAMETER OperationalHome +The Windows path to the temporary operational home; Home is an alias. +.PARAMETER JqImage +An existing local Docker image containing jq and GNU timeout. No image is pulled. +Read-only helper containers self-expire even if their native client is stopped. +.NOTES +Use /interrupt to interrupt the current model turn and /quit to end the session. +Interrupted acknowledgements remain pending for evidence-based reconciliation. +See docs/verification/runtime-backends.md for the tested boundary and guards. +#> +param([switch]$Experimental,[switch]$BuildOnly,[switch]$VerifyOnly,[Alias("Home")][string]$OperationalHome,[string]$JqImage) +$ErrorActionPreference='Stop' +$root=[IO.Path]::GetFullPath((Join-Path $PSScriptRoot '..')) +$names=@('NativeOwner','NativeHomeLease','NativeReceiptJournal','NativeAcknowledgementEvidence','NativeOperationLifetime','NativeOperations','NativeLauncher') +$sources=@($names | ForEach-Object { Join-Path $PSScriptRoot ('native-owner/'+$_+'.cs') }) +$fingerprint=($sources | ForEach-Object {(Get-FileHash $_ -Algorithm SHA256).Hash}) -join ':' +$binary=Join-Path $PSScriptRoot 'fm-native-owner.exe' +$stamp=Join-Path $PSScriptRoot 'fm-native-owner.build' +if ($BuildOnly) { + $temporary=Join-Path ([IO.Path]::GetTempPath()) ('fm-native-build-'+[guid]::NewGuid().ToString('N')+'.exe') + Add-Type -Path $sources -OutputAssembly $temporary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll + # Windows refuses replacement of a running image; never bypass that refusal. + if (Test-Path $binary) { [IO.File]::Delete($binary) } + [IO.File]::Move($temporary,$binary) + [IO.File]::WriteAllText($stamp,$fingerprint) + Write-Output 'Native candidate compiled; no session started.' + exit 0 +} +if (!$Experimental) { throw 'Explicit -Experimental opt-in is required; production use remains disabled.' } +if (!$OperationalHome) { throw '-Home must name a temporary empty-fleet home.' } +if (!$JqImage -or $JqImage -notmatch '^[A-Za-z0-9][A-Za-z0-9./:_@-]+$') { throw '-JqImage must name an existing local Docker image containing jq.' } +if (!(Test-Path $binary) -or !(Test-Path $stamp) -or [IO.File]::ReadAllText($stamp) -ne $fingerprint) { throw 'Provider missing or out of date; run -BuildOnly after stopping native sessions.' } +& docker image inspect $JqImage *> $null +if ($LASTEXITCODE -ne 0) { throw 'The selected Docker jq image is not available locally; no image was pulled.' } +$previous=$env:FM_NATIVE_JQ_IMAGE +$previousVerify=$env:FM_NATIVE_VERIFY_ONLY +try { + $env:FM_NATIVE_JQ_IMAGE=$JqImage + $env:FM_NATIVE_VERIFY_ONLY=if ($VerifyOnly) { '1' } else { '' } + # Direct inherited handles preserve interactive input; PowerShell's native + # pipeline adapter can buffer piped input until EOF instead of forwarding it. + $info=New-Object Diagnostics.ProcessStartInfo + $info.FileName=$binary + $info.Arguments='launch --experimental "'+[IO.Path]::GetFullPath($OperationalHome).TrimEnd('\')+'"' + $info.UseShellExecute=$false + $process=[Diagnostics.Process]::Start($info) + try { $process.WaitForExit(); $result=$process.ExitCode } finally { $process.Dispose() } + exit $result +} finally { $env:FM_NATIVE_JQ_IMAGE=$previous; $env:FM_NATIVE_VERIFY_ONLY=$previousVerify } diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index 73caeecaf05..cdd9097da1a 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -143,6 +143,29 @@ fm_harness_process_matches() { # # non-numeric so that any consumer treating it as a Cygwin pid - including a # future `kill` - refuses it instead of acting on the wrong process. FM_WIN_PID_PREFIX='win:' +# Native routing is selected by durable home records, never an inherited role. +FM_NATIVE_OWNER_BIN="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-native-owner.exe" +fm_native_owner_selected() { + local state="${FM_STATE_OVERRIDE:-${FM_HOME:-}/state}" value + [ -f "${state%/state}/owner-probe.json" ] && return 0 + value=$(cat "$state/.lock" 2>/dev/null || true) + case "$value" in native:*) return 0 ;; esac + return 1 +} +fm_native_owner_call() { + local native_state + native_state=$(cygpath -w "${FM_STATE_OVERRIDE:-${FM_HOME:?}/state}") || return 2 + MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner "$1" "$native_state" "${2:-}" +} +# Three states: 0 is live, 1 proven dead, 2 unknown. Do not turn exclusion +# (unknown must preserve occupancy) into a positive health assertion. +fm_native_owner_state() { + local rc + if fm_native_owner_call alive "$1"; then return 0; else rc=$?; fi + [ "$rc" -eq 1 ] && return 1 + return 2 +} + # True on a Cygwin-family userspace, where the boundary above applies. fm_win_boundary_applies() { @@ -167,7 +190,13 @@ fm_win_untag_pid() { # # holder - which reads as "startup never completed" and repeats the whole # sequence on every clear or compact. fm_session_pid_valid() { # + local native_id case "$1" in + native:*) + native_id=${1#native:} + [ "${#native_id}" -eq 32 ] || return 1 + case "$native_id" in *[!0-9a-f]*) return 1 ;; esac + return 0 ;; "$FM_WIN_PID_PREFIX"[0-9]*) return 0 ;; ''|*[!0-9]*) return 1 ;; esac @@ -257,6 +286,7 @@ fm_win_harness_ancestry_pids() { # session cannot be read off the ancestry at all, so the whole contiguous run is # reported and the callers below decide what they need from it. fm_harness_ancestry_pids() { + if fm_win_boundary_applies && fm_native_owner_selected; then fm_native_owner_call identity; return; fi local pid=$$ comm args extending=0 printed=0 for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do comm=$(fm_ps_comm "$pid") || break @@ -311,7 +341,12 @@ EOF # kill -0 cannot see across that boundary and would report a live harness as # dead - which would hand a running session's home to a second one. fm_harness_pid_alive() { - local pid=$1 comm args winpid + local pid=$1 comm args winpid owner_rc + case "$pid" in native:*) + if fm_native_owner_state "$pid"; then return 0; else owner_rc=$?; fi + # Compatibility for exclusion readers only; native health uses the state API. + [ "$owner_rc" -ne 1 ]; return ;; + esac if winpid=$(fm_win_untag_pid "$pid"); then comm=$(fm_win_command "$winpid") || return 1 fm_harness_process_matches "$comm" "$comm" @@ -332,7 +367,12 @@ fm_harness_pid_alive() { # lock, a malformed lock, a lock held by a harness outside this ancestry, or an # ancestry that cannot be resolved all fail closed. fm_session_lock_owned_by_self() { - local state=$1 lock_pid pids pid + local state=$1 lock_pid pids pid native_state + if fm_win_boundary_applies && FM_STATE_OVERRIDE="$state" fm_native_owner_selected; then + native_state=$(cygpath -w "$state") || return 1 + MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner owns "$native_state" + return + fi lock_pid=$(cat "$state/.lock" 2>/dev/null || true) fm_session_pid_valid "$lock_pid" || return 1 pids=$(fm_harness_ancestry_pids) || return 1 diff --git a/bin/fm-sessionstart-nudge.sh b/bin/fm-sessionstart-nudge.sh index c996e023d94..976010424a3 100755 --- a/bin/fm-sessionstart-nudge.sh +++ b/bin/fm-sessionstart-nudge.sh @@ -30,7 +30,7 @@ lock_is_in_ancestry() { # A Windows-tagged holder is not in this process table at all, so a local # ancestry comparison cannot answer for it. Defer to the owner of harness # identity, which is the only thing that can read across that boundary. - "$FM_WIN_PID_PREFIX"[0-9]*) fm_session_lock_owned_by_self "$STATE"; return ;; + native:*|"$FM_WIN_PID_PREFIX"[0-9]*) fm_session_lock_owned_by_self "$STATE"; return ;; # PID 1 can own the session inside a PID namespace; compare it below. ''|*[!0-9]*) return 1 ;; esac diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 321e7de71da..1d3fb108ba3 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -342,7 +342,7 @@ family_for_basename() { fm-claude-stop-autoarm-live-e2e.test.sh|\ fm-cmux-claude-composer-live-e2e.test.sh|\ fm-composer-matrix-live-e2e.test.sh|\ - fm-codex-continuity-live-e2e.test.sh|fm-native-owner-codex-live-e2e.test.sh|fm-native-owner-receipt-live-e2e.test.sh|fm-grok-continuity-live-e2e.test.sh|\ + fm-codex-continuity-live-e2e.test.sh|fm-native-owner-codex-live-e2e.test.sh|fm-native-owner-receipt-live-e2e.test.sh|fm-native-owner-launcher-live-e2e.test.sh|fm-grok-continuity-live-e2e.test.sh|\ fm-cursor-primary-live-e2e.test.sh|\ fm-grok-stop-live-e2e.test.sh|fm-harness-adapter-instructions-live-e2e.test.sh|\ fm-harness-liveness-drift-live-e2e.test.sh|\ @@ -1359,10 +1359,11 @@ families_for_changed_path() { printf '%s\n' backend-dispatch printf '%s\n' real-herdr-gated ;; - bin/native-owner/*|tests/fixtures/native-owner/*) + bin/native-owner/*|bin/fm-native-codex.ps1|tests/fixtures/native-owner/*) printf '%s\n' __script__:fm-native-owner-tool-gate.test.sh printf '%s\n' __script__:fm-native-owner-receipt-live-e2e.test.sh printf '%s\n' __script__:fm-native-owner-codex-live-e2e.test.sh + printf '%s\n' __script__:fm-native-owner-launcher-live-e2e.test.sh ;; bin/fm-agent-process-lib.sh) # The shared harness-process classifier feeds both the tmux and Herdr diff --git a/bin/native-owner/NativeAcknowledgementEvidence.cs b/bin/native-owner/NativeAcknowledgementEvidence.cs index 5e6969cef19..59c73734c1b 100644 --- a/bin/native-owner/NativeAcknowledgementEvidence.cs +++ b/bin/native-owner/NativeAcknowledgementEvidence.cs @@ -48,36 +48,56 @@ static List Rows(byte[] bytes,ulong cutoff) { } public static NativeAcknowledgementEvidence Capture(NativeHomeLease lease,Dictionary payload) { if(lease==null || !lease.IsHeld) throw new InvalidOperationException("An active home lease is required"); - string note=Note(payload);ulong cutoff; + ulong cutoff; if(!payload.ContainsKey("seq") || !(payload["seq"] is string) || !ulong.TryParse((string)payload["seq"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff) || cutoff==0) throw new IOException("Invalid wake cutoff"); var rows=Rows(Read(lease.Home,Path.Combine("state",".wake-queue")),cutoff); - bool found=false;foreach(string row in rows) if(row.Split('\t')[3]=="inbox:"+note) found=true; - if(!found) throw new IOException("Inbox wake is not present at the acknowledged cutoff"); - if(File.Exists(Path.Combine(lease.Home,"state","inbox","handled",note+".note"))) throw new IOException("Inbox target is already handled or ambiguous"); - string digest=Hash(Read(lease.Home,Path.Combine("state","inbox",note+".note"))); - return new NativeAcknowledgementEvidence(lease,new Dictionary{{"version",1},{"note",note},{"cutoff",cutoff.ToString(CultureInfo.InvariantCulture)},{"noteSha256",digest},{"rows",rows.ToArray()}}); + if(rows.Count==0) throw new IOException("No queued targets remain"); + var ids=new HashSet(); + if(payload.ContainsKey("notes")) { + var input=payload["notes"] as System.Collections.IList; + if(input==null)throw new IOException("Invalid inbox target list"); + foreach(object id in input)ids.Add(Note(new Dictionary{{"note",id}})); + } else ids.Add(Note(payload)); + var queued=new HashSet(); + foreach(string row in rows) {string key=row.Split(' ')[3];if(key.StartsWith("inbox:",StringComparison.Ordinal))queued.Add(key.Substring(6));} + if(!ids.SetEquals(queued))throw new IOException("Inbox targets differ from the captured queue"); + var notes=new List>(); + foreach(string id in ids) { + if(File.Exists(Path.Combine(lease.Home,"state","inbox","handled",id+".note")))throw new IOException("Inbox target is already handled or ambiguous"); + notes.Add(new Dictionary{{"note",id},{"noteSha256",Hash(Read(lease.Home,Path.Combine("state","inbox",id+".note")))}}); + } + var record=new Dictionary{{"version",2},{"notes",notes},{"cutoff",cutoff.ToString(CultureInfo.InvariantCulture)},{"rows",rows.ToArray()}}; + if(notes.Count==1) {record["note"]=notes[0]["note"];record["noteSha256"]=notes[0]["noteSha256"];} + return new NativeAcknowledgementEvidence(lease,record); } internal static bool Completed(NativeHomeLease lease,Dictionary evidence) { if(lease==null || !lease.IsHeld || evidence==null) return false; try { - if(Convert.ToInt32(evidence["version"])!=1) return false; - string note=Note(evidence), digest=(string)evidence["noteSha256"];ulong cutoff; - if(!Regex.IsMatch(digest,@"\A[0-9a-f]{64}\z") || !ulong.TryParse((string)evidence["cutoff"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff) || cutoff==0) return false; - // A captured, nonempty target set is mandatory; an empty queue on - // its own is not evidence that an acknowledgement happened. + int version=Convert.ToInt32(evidence["version"]);ulong cutoff; + if(version!=1 && version!=2)return false; + if(!ulong.TryParse((string)evidence["cutoff"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff)||cutoff==0)return false; var targets=evidence["rows"] as System.Collections.IList; - if(targets==null || targets.Count==0) return false; - var saved=new List();foreach(object target in targets) { if(!(target is string)) return false;saved.Add((string)target); } + if(targets==null||targets.Count==0)return false; + var saved=new List();foreach(object target in targets){if(!(target is string))return false;saved.Add((string)target);} var validated=Rows(new UTF8Encoding(false,true).GetBytes(string.Join("\n",saved.ToArray())+"\n"),cutoff); - if(validated.Count!=targets.Count || !validated.Exists(row=>row.Split('\t')[3]=="inbox:"+note)) return false; - if(File.Exists(Path.Combine(lease.Home,"state","inbox",note+".note"))) return false; - if(Hash(Read(lease.Home,Path.Combine("state","inbox","handled",note+".note")))!=digest) return false; - return Rows(Read(lease.Home,Path.Combine("state",".wake-queue")),cutoff).Count==0; - } catch(IOException) { return false; } - catch(UnauthorizedAccessException) { return false; } - catch(ArgumentException) { return false; } - catch(KeyNotFoundException) { return false; } - catch(InvalidCastException) { return false; } - catch(FormatException) { return false; } + if(validated.Count!=targets.Count)return false; + var notes=new List>(); + if(version==1)notes.Add(evidence); + else { + var list=evidence["notes"] as System.Collections.IList;if(list==null)return false; + foreach(object entry in list){var note=entry as Dictionary;if(note==null)return false;notes.Add(note);} + } + var ids=new HashSet(); + foreach(var entry in notes) { + string note=Note(entry),digest=(string)entry["noteSha256"]; + if(!ids.Add(note)||!Regex.IsMatch(digest,@"\A[0-9a-f]{64}\z"))return false; + if(File.Exists(Path.Combine(lease.Home,"state","inbox",note+".note")))return false; + if(Hash(Read(lease.Home,Path.Combine("state","inbox","handled",note+".note")))!=digest)return false; + } + var queued=new HashSet();foreach(string row in validated){string key=row.Split(' ')[3];if(key.StartsWith("inbox:",StringComparison.Ordinal))queued.Add(key.Substring(6));} + return ids.SetEquals(queued)&&Rows(Read(lease.Home,Path.Combine("state",".wake-queue")),cutoff).Count==0; + } catch(IOException){return false;} catch(UnauthorizedAccessException){return false;} + catch(ArgumentException){return false;} catch(KeyNotFoundException){return false;} + catch(InvalidCastException){return false;} catch(FormatException){return false;} catch(OverflowException){return false;} } } diff --git a/bin/native-owner/NativeHomeLease.cs b/bin/native-owner/NativeHomeLease.cs index 0c53936982b..de97050a020 100644 --- a/bin/native-owner/NativeHomeLease.cs +++ b/bin/native-owner/NativeHomeLease.cs @@ -16,6 +16,13 @@ public sealed class NativeHomeLease : IDisposable { public readonly string Home; internal bool IsHeld { get { return file!=null; } } public string PreviousGeneration { get; private set; } + readonly HashSet deadGenerations=new HashSet(); + static bool Generation(string value) { + if(value==null||value.Length!=32)return false; + foreach(char c in value)if(!(c>='0'&&c<='9')&&!(c>='a'&&c<='f'))return false; + return true; + } + public bool ProvenDeadGeneration(string value) { return IsHeld&&deadGenerations.Contains(value); } [StructLayout(LayoutKind.Sequential)] struct FT { public uint low,high; } [DllImport("kernel32.dll",SetLastError=true)] static extern IntPtr OpenProcess(uint access,bool inherit,uint pid); [DllImport("kernel32.dll",SetLastError=true)] static extern bool GetProcessTimes(IntPtr process,out FT created,out FT exited,out FT kernel,out FT user); @@ -23,7 +30,11 @@ public sealed class NativeHomeLease : IDisposable { [DllImport("kernel32.dll")] static extern bool CloseHandle(IntPtr handle); static string Filename(string home) { string full=Path.GetFullPath(home); - if(!full.StartsWith(Path.GetFullPath(Path.GetTempPath()),StringComparison.OrdinalIgnoreCase)) throw new InvalidOperationException("Probe leases require a temporary home"); + string temporary=Path.GetFullPath(Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData),"Temp")).TrimEnd('\\','/')+Path.DirectorySeparatorChar; + if(!full.StartsWith(temporary,StringComparison.OrdinalIgnoreCase)) throw new InvalidOperationException("Experimental leases require a home beneath the user's Windows temporary directory"); + for(string parent=full;parent!=null;parent=Path.GetDirectoryName(parent)) { + if(Directory.Exists(parent)&&(File.GetAttributes(parent)&FileAttributes.ReparsePoint)!=0) throw new InvalidOperationException("Reparse-point homes are not supported"); + } return Path.Combine(full,"owner-probe.json"); } static Dictionary Read(Stream stream) { @@ -67,6 +78,14 @@ public NativeHomeLease(string home) { var previous=Read(file); if(RootAlive(previous)) throw new InvalidOperationException("Recorded primary is still alive; refusing replacement"); PreviousGeneration=previous.ContainsKey("generation") ? (string)previous["generation"] : null; + if(!Generation(PreviousGeneration))throw new InvalidOperationException("Invalid predecessor generation; preserved"); + if(previous.ContainsKey("deadGenerations")) { + var prior=previous["deadGenerations"] as System.Collections.IList; + if(prior==null)throw new InvalidOperationException("Invalid predecessor history; preserved"); + foreach(object item in prior){string value=item as string;if(!Generation(value))throw new InvalidOperationException("Invalid predecessor history; preserved");deadGenerations.Add(value);} + } + deadGenerations.Add(PreviousGeneration); + if(deadGenerations.Count>256)throw new InvalidOperationException("Predecessor history requires maintenance; preserved"); } Write(new Dictionary{{"state","pending"},{"controllerPid",Process.GetCurrentProcess().Id}}); } catch { Dispose(); throw; } @@ -76,7 +95,8 @@ void Write(Dictionary value) { file.Position=0; file.SetLength(0); file.Write(bytes,0,bytes.Length); file.Flush(true); } public void Publish(uint pid,ulong created,string generation,string pipe) { - using(var self=Process.GetCurrentProcess()) Write(new Dictionary{{"state","live"},{"rootPid",pid},{"rootCreationFileTime",created},{"generation",generation},{"pipe",pipe},{"controllerPid",self.Id},{"controllerCreated",self.StartTime.ToUniversalTime().ToFileTimeUtc()}}); + var previous=new string[deadGenerations.Count];deadGenerations.CopyTo(previous); + using(var self=Process.GetCurrentProcess()) Write(new Dictionary{{"deadGenerations",previous},{"state","live"},{"rootPid",pid},{"rootCreationFileTime",created},{"generation",generation},{"pipe",pipe},{"controllerPid",self.Id},{"controllerCreated",self.StartTime.ToUniversalTime().ToFileTimeUtc()}}); } public static Dictionary Binding(string state) { string canonical=Path.GetFullPath(state).TrimEnd(Path.DirectorySeparatorChar,Path.AltDirectorySeparatorChar); diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs new file mode 100644 index 00000000000..a2ea26c2613 --- /dev/null +++ b/bin/native-owner/NativeLauncher.cs @@ -0,0 +1,149 @@ +// Opt-in native launcher. No fixture commands, configurable executable, or +// arbitrary operation entry point is exposed by this assembly. +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.IO.Pipes; +using System.Runtime.InteropServices; +using System.Security.AccessControl; +using System.Security.Principal; +using System.Text; +using System.Threading; +public static partial class NativeOwner { + [DllImport("kernel32.dll")] static extern IntPtr GetStdHandle(int kind); + static string CodeRoot { get { return Path.GetDirectoryName(Path.GetDirectoryName(OwnExe)); } } + static void EmptyFleet(string home) { + string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); + if((Directory.Exists(state)&&Directory.GetFiles(state,"*.meta").Length!=0) || (Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); + } + static int Launch(string selectedHome) { + string home=Path.GetFullPath(selectedHome), node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); + string host=Path.Combine(CodeRoot,"bin","native-owner","codex-host.mjs"); + if(!File.Exists(node)||!File.Exists(host)) throw new IOException("Native Node or the code-owned host is missing"); + EmptyFleet(home); + string session=Guid.NewGuid().ToString("N"),nonce=Guid.NewGuid().ToString("N"),pipeName="fm-native-"+session; + IntPtr job=CreateJobObject(IntPtr.Zero,null),env=IntPtr.Zero,output=IntPtr.Zero,input=IntPtr.Zero; + if(job==IntPtr.Zero) throw Error("Create session job"); + PI primary=new PI();var scopes=new List(); + NativeHomeLease lease=null; + NamedPipeServerStream ownerServer=null,controlServer=null;Thread controlThread=null; + object sync=new object();Exception controlFailure=null;DateTime failureAt=DateTime.MaxValue; + ConsoleCancelEventHandler cancel=(sender,args)=>{args.Cancel=true;};Console.CancelKeyPress+=cancel; + try { + lease=new NativeHomeLease(home);operationJournal=new NativeReceiptJournal(lease,session); + int recovered=operationJournal.ReconcileCompletedAcknowledgements(); + string runtime=Path.Combine(home,"state","native-runtime",session);Directory.CreateDirectory(runtime); + var security=new PipeSecurity();security.SetAccessRuleProtection(true,false); + security.AddAccessRule(new PipeAccessRule(WindowsIdentity.GetCurrent().User,PipeAccessRights.FullControl,AccessControlType.Allow)); + // Retain both instances before publishing or resuming a client. + // The host connects once and never reconnects to a replacement. + ownerServer=new NamedPipeServerStream(pipeName,PipeDirection.InOut,1,PipeTransmissionMode.Byte,PipeOptions.Asynchronous,4096,4096,security); + string controlName="fm-native-host-"+session; + controlServer=new NamedPipeServerStream(controlName,PipeDirection.InOut,1,PipeTransmissionMode.Byte,PipeOptions.Asynchronous,4096,4096,security); + var values=EnvironmentFor(controlName,session,runtime,nonce); + values["FM_HOME"]=home;values["FM_PROBE_CODE_ROOT"]=CodeRoot; + values["FM_PROBE_LEASE_HOME"]=home;values["FM_PROBE_LEASE_GENERATION"]=session; + values["FM_PROBE_RECOVERED"]=recovered.ToString(); + values["FM_PROBE_JQ_IMAGE"]=Environment.GetEnvironmentVariable("FM_NATIVE_JQ_IMAGE")??""; + values["FM_PROBE_VERIFY_ONLY"]=Environment.GetEnvironmentVariable("FM_NATIVE_VERIFY_ONLY")??""; + values["MSYS"]="winsymlinks:nativestrict"; + var block=new StringBuilder();foreach(var entry in values)block.Append(entry.Key).Append('=').Append(entry.Value).Append('\0');block.Append('\0'); + env=Marshal.StringToHGlobalUni(block.ToString()); + var startup=new SI {cb=Marshal.SizeOf(typeof(SI)),flags=0x100,input=GetStdHandle(-10),output=GetStdHandle(-11),error=GetStdHandle(-12)}; + if(!CreateProcess(node,new StringBuilder(Quote(node)+" "+Quote(host)),IntPtr.Zero,IntPtr.Zero,true,0x4|0x400,env,CodeRoot,ref startup,out primary))throw Error("Create suspended host"); + if(!AssignProcessToJobObject(job,primary.process))throw Error("Assign host"); + FT born,exit,kernel,user;if(!GetProcessTimes(primary.process,out born,out exit,out kernel,out user))throw Error("Read host instance"); + lease.Publish(primary.pid,((ulong)born.high<<32)|born.low,session,pipeName);RegisteredHarness="codex"; + output=FileHandle(Path.Combine(runtime,"operations.log"),0x40000000,2);input=FileHandle("NUL",0x80000000,3); + var operationStartup=new SI {cb=Marshal.SizeOf(typeof(SI)),flags=0x100,input=input,output=output,error=output}; + if(!operationJournal.NeedsReconciliation) { + var operation=StartScope("owner-operation",job,env,runtime,ref operationStartup);scopes.Add(operation); + if(ResumeThread(operation.process.thread)==0xffffffff)throw Error("Resume startup"); + } + controlThread=new Thread(()=>{ + try { + var connect=controlServer.BeginWaitForConnection(null,null); + while(!connect.AsyncWaitHandle.WaitOne(50)&&WaitForSingleObject(primary.process,0)==WAIT_TIMEOUT){} + if(!connect.IsCompleted)return;controlServer.EndWaitForConnection(connect); + uint peer;if(!GetNamedPipeClientProcessId(controlServer.SafePipeHandle.DangerousGetHandle(),out peer)||peer!=primary.pid)throw new IOException("Host channel peer mismatch"); + using(var reader=new StreamReader(controlServer,Encoding.UTF8,false,1024,true))using(var writer=new StreamWriter(controlServer,new UTF8Encoding(false),1024,true)) { + writer.AutoFlush=true; + while(WaitForSingleObject(primary.process,0)==WAIT_TIMEOUT) { + var line=reader.ReadLineAsync(); + while(!line.Wait(50)&&WaitForSingleObject(primary.process,0)==WAIT_TIMEOUT){} + if(!line.IsCompleted||line.Result==null)break; + if(line.Result.Length>4096)throw new IOException("Oversized host request"); + lock(sync) { + var request=Json.Deserialize>(line.Result); + var verdict=Verdict(request,peer,primary.process,job,primary.pid,scopes,session,runtime,nonce); + if(!request.ContainsKey("kind")||(string)request["kind"]!="notification")throw new IOException("Invalid host operation plane"); + NotificationRequest(request,verdict,lease,job,env,runtime,ref operationStartup,scopes); + writer.WriteLine(Json.Serialize(verdict)); + } + } + } + } catch(Exception error){Interlocked.CompareExchange(ref controlFailure,error,null);} + finally {controlServer.Dispose();} + });controlThread.IsBackground=true;controlThread.Start(); + if(ResumeThread(primary.thread)==0xffffffff)throw Error("Resume host"); + while(WaitForSingleObject(primary.process,0)==WAIT_TIMEOUT) { + var pending=ownerServer.BeginWaitForConnection(null,null); + while(!pending.AsyncWaitHandle.WaitOne(50)&&WaitForSingleObject(primary.process,0)==WAIT_TIMEOUT) { + if(Volatile.Read(ref controlFailure)!=null) { + if(failureAt==DateTime.MaxValue)failureAt=DateTime.UtcNow; + if((DateTime.UtcNow-failureAt).TotalSeconds>12)break; + } + } + if(!pending.IsCompleted)break;ownerServer.EndWaitForConnection(pending); + uint pid;if(!GetNamedPipeClientProcessId(ownerServer.SafePipeHandle.DangerousGetHandle(),out pid))throw Error("Read native peer"); + using(var reader=new StreamReader(ownerServer,Encoding.UTF8,false,1024,true))using(var writer=new StreamWriter(ownerServer,new UTF8Encoding(false),1024,true)) { + var read=reader.ReadLineAsync();if(!read.Wait(8000)||read.Result==null||read.Result.Length>4096)throw new IOException("Invalid native request"); + lock(sync) { + var request=Json.Deserialize>(read.Result); + var verdict=Verdict(request,pid,primary.process,job,primary.pid,scopes,session,runtime,nonce); + if(request.ContainsKey("kind")&&(string)request["kind"]=="owner")AuthorizeOwner(request,verdict,ownerServer,lease,session); + writer.AutoFlush=true;writer.WriteLine(Json.Serialize(verdict)); + } + } + ownerServer.Disconnect(); + } + if(controlFailure!=null)throw new IOException("Native host connection failed",controlFailure); + uint code;if(!GetExitCodeProcess(primary.process,out code))throw Error("Read host exit");return (int)code; + } finally { + if(ownerServer!=null)ownerServer.Dispose(); + if(controlServer!=null)controlServer.Dispose(); + if(controlThread!=null&&!controlThread.Join(3000)) { + Console.Error.WriteLine("Host dispatch did not stop; preserving interrupted work and ending its controller"); + Environment.Exit(125); + } + // Never terminate the enclosing job: unrelated fleet processes may + // outlive a host. Only registered fixed operations are stopped here. + foreach(var scope in scopes) { + try {NativeOperationLifetime.Stop(scope.job,2000);} finally {CloseHandle(scope.process.thread);CloseHandle(scope.process.process);CloseHandle(scope.job);} + } + if(primary.process!=IntPtr.Zero&&WaitForSingleObject(primary.process,0)==WAIT_TIMEOUT){TerminateProcess(primary.process,125);WaitForSingleObject(primary.process,3000);} + foreach(IntPtr handle in new [] {primary.thread,primary.process,job,output,input})if(handle!=IntPtr.Zero)CloseHandle(handle); + if(env!=IntPtr.Zero)Marshal.FreeHGlobal(env); + if(operationJournal!=null)operationJournal.Dispose();if(lease!=null)lease.Dispose();Console.CancelKeyPress-=cancel; + } + } + static int FixedOperation(string purpose) { + if(purpose!="startup"&&purpose!="check"&&purpose!="ack")throw new ArgumentException("Unknown fixed operation"); + string home=Environment.GetEnvironmentVariable("FM_HOME");EmptyFleet(home); + if(OwnerClient(purpose=="startup" ? "identity" : "owns",Path.Combine(home,"state"),"")!=0)throw new InvalidOperationException("Operation is not registered"); + string script=Path.Combine(CodeRoot,"bin","native-owner",purpose+".sh"); + using(var process=Process.Start(new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script.Replace('\\','/'))) {UseShellExecute=false})) { + if(!process.WaitForExit(purpose=="startup" ? 240000 : 60000))throw new TimeoutException("Fixed operation exceeded its bound");return process.ExitCode; + } + } + public static int Main(string[] args) { + try { + if(args.Length==3&&args[0]=="launch"&&args[1]=="--experimental")return Launch(args[2]); + if(args.Length>=3&&args[0]=="owner")return OwnerClient(args[1],args[2],args.Length>3?args[3]:""); + if(args.Length==1&&args[0]=="owner-operation")return FixedOperation("startup"); + if(args.Length==2&&args[0]=="notification-operation")return FixedOperation(args[1]); + throw new ArgumentException("Use fm-native-codex.ps1 -Experimental -Home "); + } catch(Exception error){Console.Error.WriteLine(error.Message);return 2;} + } +} diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs new file mode 100644 index 00000000000..133d2db822b --- /dev/null +++ b/bin/native-owner/NativeOperations.cs @@ -0,0 +1,124 @@ +// Shared registered operation dispatch; external callers never choose commands. +using System; +using System.Collections; +using System.Collections.Generic; +using System.IO; +using System.Runtime.InteropServices; +using System.Text; +public static partial class NativeOwner { + static NativeReceiptJournal operationJournal; + static bool shutdownRequested; + static IntPtr FileHandle(string path, uint access, uint creation) { + SA sa = new SA { length=Marshal.SizeOf(typeof(SA)), inherit=1 }; + IntPtr h = CreateFile(path, access, 3, ref sa, creation, 0x80, IntPtr.Zero); + if (h == new IntPtr(-1)) throw Error("CreateFile"); + return h; + } + static SortedDictionary EnvironmentFor(string pipe, string session, string home, string nonce) { + var result = new SortedDictionary(StringComparer.OrdinalIgnoreCase); + foreach (DictionaryEntry e in Environment.GetEnvironmentVariables()) { + string k = (string)e.Key; + if (k=="NODE_OPTIONS" || k=="NODE_PATH" || k=="BASH_ENV" || k=="ENV") continue; + if (k.StartsWith("FM_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("PI_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("NO_MISTAKES", StringComparison.OrdinalIgnoreCase) || k == "CLAUDE_PID" || k == "CLAUDECODE") continue; + result[k] = (string)e.Value; + } + result["FM_PROBE_PIPE"] = pipe; result["FM_PROBE_SESSION"] = session; + result["FM_PROBE_HOME"] = home; result["FM_PROBE_NONCE"] = nonce; + result["FM_PROBE_EXE"] = OwnExe; + return result; + } + static ChildScope StartScope(string role, IntPtr parentJob, IntPtr environment, string home, ref SI startup, string purpose="startup") { + var scope=new ChildScope { job=CreateJobObject(IntPtr.Zero,null), role=role, purpose=purpose }; + IntPtr operationEnvironment=IntPtr.Zero; + if(scope.job==IntPtr.Zero) throw Error("CreateJobObject child scope"); + try { + if(role=="owner-operation") NativeOperationLifetime.Configure(scope.job); + string operation=role=="owner-operation" ? (purpose=="startup" ? "owner-operation" : "notification-operation "+purpose) : "scoped-client "+role; + if(role=="owner-operation") { + // Only fixed operations receive the grant; caller claims never select it. + var values=new SortedDictionary(StringComparer.OrdinalIgnoreCase); + foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { + string value=Environment.GetEnvironmentVariable(key); if(value!=null) values[key]=value; + } + values["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); + // Copy only the controller's registration fields from the prepared block. + int offset=0; + while(Marshal.ReadInt16(environment,offset)!=0) { + string entry=Marshal.PtrToStringUni(IntPtr.Add(environment,offset)); offset+=(entry.Length+1)*2; + int split=entry.IndexOf('='); if(split<=0) continue; + string key=entry.Substring(0,split); + if(key.StartsWith("FM_PROBE_",StringComparison.Ordinal) || key=="FM_HOME" || key=="MSYS") values[key]=entry.Substring(split+1); + } + var block=new StringBuilder(); foreach(var value in values) block.Append(value.Key).Append('=').Append(value.Value).Append('\0'); block.Append('\0'); + operationEnvironment=Marshal.StringToHGlobalUni(block.ToString()); + } + if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" "+operation),IntPtr.Zero,IntPtr.Zero,true,0x4|0x400|0x200,operationEnvironment==IntPtr.Zero ? environment : operationEnvironment,home,ref startup,out scope.process)) throw Error("CreateProcess child scope"); + if(!AssignProcessToJobObject(parentJob,scope.process.process) || !AssignProcessToJobObject(scope.job,scope.process.process)) throw Error("Assign child scope"); + // The caller records this scope before resuming the process. + return scope; + } catch { + if(scope.process.process!=IntPtr.Zero) { TerminateProcess(scope.process.process,125); CloseHandle(scope.process.thread); CloseHandle(scope.process.process); } + CloseHandle(scope.job); throw; + } finally { if(operationEnvironment!=IntPtr.Zero) Marshal.FreeHGlobal(operationEnvironment); } + } + static void NotificationRequest(Dictionary request,Dictionary verdict,NativeHomeLease lease,IntPtr job,IntPtr environment,string home,ref SI startup,List scopes) { + bool allowed=lease!=null && (string)verdict["association"]=="associated" && (string)verdict["hostClassification"]=="registered-primary"; + verdict["notificationAuthorized"]=allowed; + if(!allowed) return; + string action=request.ContainsKey("action") ? (string)request["action"] : ""; + if(action=="shutdown") { + shutdownRequested=true; + foreach(var scope in scopes) if(scope.role=="owner-operation") NativeOperationLifetime.Stop(scope.job,1500); + verdict["operationState"]="stopped"; + return; + } + if(shutdownRequested) { verdict["operationState"]="stopped";return; } + bool ready=true,startupFailed=false; + foreach(var scope in scopes) if(scope.purpose=="startup") { + if(WaitForSingleObject(scope.process.process,0)==WAIT_TIMEOUT) ready=false; + else {uint code;if(!GetExitCodeProcess(scope.process.process,out code)||code!=0)startupFailed=true;} + } + verdict["startupFailed"]=startupFailed; + if(action=="status") { + verdict["startupExpired"]=ready; + verdict["operationState"]=operationJournal.NeedsReconciliation ? "reconciliation-required" : startupFailed ? "startup-failed" : ready ? "ready" : "starting"; + return; + } + verdict["startupExpired"]=ready; + if(pendingOperation!=null && WaitForSingleObject(pendingOperation.process.process,0)!=WAIT_TIMEOUT) { + uint code; if(!GetExitCodeProcess(pendingOperation.process.process,out code)) throw Error("Operation exit"); + verdict["operationExit"]=code; + if(code!=0) { verdict["operationState"]="failed"; return; } + string file=Path.Combine(home,"notification-"+pendingOperation.purpose+".json"); + var output=Json.Deserialize>(File.ReadAllText(file)); + if(pendingOperation.purpose=="check") { + if(output.ContainsKey("quiet") && (bool)output["quiet"]) { delivered=null;receipt=null; } + else {delivered=operationJournal.Present(output);receipt=(string)delivered["receipt"];} + consumed=false; + } else { operationJournal.CompleteAcknowledgement(receipt);consumed=true; } + NativeOperationLifetime.Stop(pendingOperation.job,1500); + scopes.Remove(pendingOperation); + CloseHandle(pendingOperation.process.thread);CloseHandle(pendingOperation.process.process);CloseHandle(pendingOperation.job); + pendingOperation=null; + } + if(action=="result") { + verdict["operationState"]=pendingOperation!=null ? "pending" : operationJournal.NeedsReconciliation ? "reconciliation-required" : consumed ? "acknowledged" : delivered!=null ? "delivered" : "quiet"; + if(delivered!=null) verdict["notification"]=delivered; + return; + } + if(startupFailed) {verdict["operationState"]="startup-failed";return;} + if(!ready || pendingOperation!=null) { verdict["operationState"]="busy"; return; } + if(operationJournal.NeedsReconciliation) { verdict["operationState"]="reconciliation-required";return; } + if(action=="check" && (delivered==null || consumed)) checkStarts++; + else if(action=="ack" && delivered!=null && !consumed && request.ContainsKey("receipt") && (string)request["receipt"]==receipt && request.ContainsKey("observed") && (string)request["observed"]==(string)delivered["challenge"]) { + var targetEvidence=NativeAcknowledgementEvidence.Capture(lease,delivered); + var acknowledged=operationJournal.BeginAcknowledgement(receipt,(string)request["observed"],targetEvidence); + ackStarts++; + File.WriteAllText(Path.Combine(home,"notification-ack-request.json"),Json.Serialize(acknowledged)); + } else { verdict["operationState"]="denied"; return; } + pendingOperation=StartScope("owner-operation",job,environment,home,ref startup,action); + scopes.Add(pendingOperation); + if(ResumeThread(pendingOperation.process.thread)==0xffffffff) throw Error("Resume notification operation"); + verdict["operationState"]="pending"; + } +} diff --git a/bin/native-owner/NativeOwner.cs b/bin/native-owner/NativeOwner.cs index cefee3c370f..6e0468432a4 100644 --- a/bin/native-owner/NativeOwner.cs +++ b/bin/native-owner/NativeOwner.cs @@ -121,7 +121,7 @@ static void AuthorizeOwner(Dictionary request,Dictionary "$LOG/notification-ack.json" diff --git a/bin/native-owner/check.sh b/bin/native-owner/check.sh new file mode 100755 index 00000000000..32ab48f1da0 --- /dev/null +++ b/bin/native-owner/check.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Read actual work only. No synthetic note or scripted model prompt is generated. +set -euo pipefail +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) +LOG=$(cygpath -u "${FM_PROBE_HOME:?}") +export FM_HOME +FM_HOME=$(cygpath -u "${FM_HOME:?}") +export PATH="$ROOT/bin/native-owner/tools:$PATH" +cd "$ROOT" +. bin/fm-session-lock-lib.sh +fm_session_lock_owned_by_self "$FM_HOME/state" +set +e +bin/fm-watch-checkpoint.sh --seconds 1 > "$LOG/checkpoint.log" 2>&1 +rc=$? +set -e +case "$rc" in 0|124) ;; *) exit "$rc" ;; esac +bin/fm-wake-drain.sh > "$LOG/delivery.log" 2>&1 +ack=$(grep 'WAKE_ACK_REQUIRED:' "$LOG/delivery.log" | tail -1 || true) +if [ -z "$ack" ]; then + if grep -Eq 'OPEN DECISIONS|UNREAD STATUS|RECORD DIVERGENCE|STATUS OUTCOME BACKSTOP' "$LOG/checkpoint.log" "$LOG/delivery.log"; then + printf 'Unqueued work requires reconciliation; captured in %s\n' "$LOG" >&2 + exit 2 + fi + printf '{"quiet":true}\n' > "$LOG/notification-check.json" + exit 0 +fi +bin/fm-inbox.sh drain > "$LOG/inbox.log" +seq=$(awk '{for(i=1;i0))') +challenge=$(od -An -N12 -tx1 /dev/urandom | tr -d ' \n') +message="$(<"$LOG/checkpoint.log") +$(<"$LOG/delivery.log") +$(<"$LOG/inbox.log")" +jq -n --arg message "$message" --arg challenge "$challenge" --arg seq "$seq" --arg generation "$generation" --argjson notes "$notes" \ + '{message:$message,challenge:$challenge,seq:$seq,generation:$generation,notes:$notes}' > "$LOG/notification-check.json" diff --git a/bin/native-owner/codex-host.mjs b/bin/native-owner/codex-host.mjs new file mode 100644 index 00000000000..8566173dcc2 --- /dev/null +++ b/bin/native-owner/codex-host.mjs @@ -0,0 +1,178 @@ +// Interactive native host. The controller has created the private pipe before +// this process resumes. Connect once: EOF or timeout ends this session, never +// reconnect to an endpoint that could now belong to another controller. +import fs from 'node:fs'; +import path from 'node:path'; +import net from 'node:net'; +import {spawn} from 'node:child_process'; +import {createInterface} from 'node:readline'; +import {createNotificationGate} from './codex-tool-gate.mjs'; +import {createHostLifecycle} from './host-lifecycle.mjs'; +const runtime=process.env.FM_PROBE_HOME,root=process.env.FM_PROBE_CODE_ROOT; +const pause=ms=>new Promise(resolve=>setTimeout(resolve,ms)); +let closing=false,gate=null,server=null,alive=false,thread=null,activeTurn=null,ended=false; +let lifecycle=null,terminal=null,announced=null,next=0,nativeWaiter=null,nativeTail=Promise.resolve(); +const input=[],pending=new Map(),turns=new Map(); +const evidence={ready:false,turns:[],tools:[],digestDeliveredBeforeDeferred:false}; +const save=()=>{const file=path.join(runtime,'host.json');fs.writeFileSync(file+'.tmp',JSON.stringify(evidence,null,2));fs.renameSync(file+'.tmp',file);}; +const consoleInput=createInterface({input:process.stdin}); +consoleInput.on('line',line=>{ + if(line==='/quit'){void stop();return;} + if(line==='/interrupt'){void interrupt().catch(fail);return;} + input.push(line); +}); +consoleInput.on('close',()=>{ended=true;}); +process.on('SIGINT',()=>{if(activeTurn)void interrupt().catch(fail);else void stop();}); +process.on('SIGTERM',()=>{void stop();}); +const socket=net.createConnection('\\\\.\\pipe\\'+process.env.FM_PROBE_PIPE); +const connection=new Promise((resolve,reject)=>{ + const timer=setTimeout(()=>{socket.destroy();reject(Error('Native connection timed out'));},10000); + socket.once('connect',()=>{clearTimeout(timer);resolve();});socket.once('error',error=>{clearTimeout(timer);reject(error);}); +}); +const channel=createInterface({input:socket}); +channel.on('line',line=>{ + if(!nativeWaiter){fail(Error('Unexpected native response'));return;} + const waiter=nativeWaiter;nativeWaiter=null; + try {const value=JSON.parse(line);if(!value.notificationAuthorized)throw Error('Native host authorization refused');waiter.resolve(value);}catch(error){waiter.reject(error);} +}); +socket.on('error',error=>{nativeWaiter?.reject(error);nativeWaiter=null;if(!closing)fail(error);}); +socket.on('close',()=>{const error=Error('Native controller connection closed');nativeWaiter?.reject(error);nativeWaiter=null;if(!closing)fail(error);}); +function native(action,extra={},signal){ + const run=async()=>{ + await connection;if(socket.destroyed||signal?.aborted||(closing&&action!=='shutdown'))throw Error('Native session unavailable'); + return new Promise((resolve,reject)=>{ + let timer; + const abort=()=>{socket.destroy();finish(reject,Error('Native request interrupted; durable work preserved'));}; + const finish=(callback,value)=>{clearTimeout(timer);signal?.removeEventListener('abort',abort);callback(value);}; + nativeWaiter={resolve:value=>finish(resolve,value),reject:error=>finish(reject,error)}; + timer=setTimeout(abort,10000);signal?.addEventListener('abort',abort,{once:true}); + socket.write(JSON.stringify({kind:'notification',session:process.env.FM_PROBE_SESSION,home:runtime,nonce:process.env.FM_PROBE_NONCE,action,...extra})+'\n'); + }); + }; + const result=nativeTail.then(run);nativeTail=result.catch(()=>{});return result; +} +function send(frame){if(!alive)throw Error('App-server is not live');server.stdin.write(JSON.stringify(frame)+'\n');} +function request(method,params,signal){ + return new Promise((resolve,reject)=>{ + if(signal?.aborted){reject(Error('Request cancelled'));return;} + const id=++next; + let timer; + const clean=()=>{clearTimeout(timer);signal?.removeEventListener('abort',abort);}; + const abort=()=>{pending.delete(id);clean();reject(Error('App-server request cancelled'));}; + timer=setTimeout(()=>{pending.delete(id);clean();reject(Error('App-server '+method+' timed out'));},30000); + pending.set(id,{method,resolve:value=>{clean();resolve(value);},reject:error=>{clean();reject(error);}}); + signal?.addEventListener('abort',abort,{once:true}); + try {send({id,method,params});}catch(error){pending.delete(id);clean();reject(error);} + }); +} +async function interrupt(signal){ + if(!activeTurn||!alive)return; + const target=activeTurn; + await request('turn/interrupt',{threadId:thread,turnId:target},signal); + const limit=Date.now()+10000; + while(alive&&!turns.has(target)&&!signal?.aborted&&Date.now()gate?.close()},interrupt, + stopOperations:async signal=>(await native('shutdown',{},signal)).operationState==='stopped', + closeInput:()=>{if(server)server.stdin.end();}, + waitForExit:signal=>!alive?Promise.resolve(true):new Promise(resolve=>{ + const exit=()=>{signal.removeEventListener('abort',abort);resolve(true);}; + const abort=()=>{server.removeListener('exit',exit);resolve(false);}; + server.once('exit',exit);signal.addEventListener('abort',abort,{once:true}); + }),terminate:()=>server?.kill(),graceMs:3000}); + const result=lifecycle.shutdown(); + void result.then(value=>{ + fs.writeFileSync(path.join(runtime,'shutdown.json'),JSON.stringify(value)); + if(!value.stopped){console.error('Shutdown was not fully confirmed; durable work was preserved.');process.exitCode=1;} + channel.close();socket.destroy(); + }); + return result; +} +try { + await connection; + let status=await native('status'); + if(status.operationState==='reconciliation-required')throw Error('An earlier acknowledgement is incomplete. Its records are preserved; review '+path.join(process.env.FM_HOME,'owner-receipts.jsonl')+' before starting more work.'); + const limit=Date.now()+200000; + while(!closing&&!fs.existsSync(path.join(runtime,'digest.ready'))){ + status=await native('status');if(status.operationState==='startup-failed'||Date.now()>limit)throw Error('Startup failed; inspect '+path.join(runtime,'startup.log')); + await pause(100); + } + if(!closing){ + const executable=path.join(process.env.APPDATA,'npm/node_modules/@openai/codex/node_modules/@openai/codex-win32-x64/vendor/x86_64-pc-windows-msvc/bin/codex.exe'); + server=spawn(executable,['app-server','--stdio','--disable','hooks','-c','windows.sandbox=unelevated'],{cwd:root,stdio:['pipe','pipe','pipe'],detached:true,windowsHide:true});alive=true; + server.stderr.on('data',data=>fs.appendFileSync(path.join(runtime,'app-server.stderr'),data)); + const lost=error=>{alive=false;for(const waiter of pending.values())waiter.reject(error);pending.clear();if(!closing)fail(error);}; + server.on('error',lost);server.on('exit',()=>lost(Error('App-server exited')));server.stdin.on('error',error=>{if(!closing)fail(error);}); + createInterface({input:server.stdout}).on('line',line=>{ + try { + const frame=JSON.parse(line); + if(frame.method==='item/tool/call'&&frame.id!==undefined){void tool(frame).catch(fail);return;} + if(frame.id!==undefined&&pending.has(frame.id)){ + const waiter=pending.get(frame.id);pending.delete(frame.id); + if(frame.error)waiter.reject(Error(JSON.stringify(frame.error))); + else {if(waiter.method==='turn/start'){activeTurn=frame.result.turn.id;evidence.activeTurn=activeTurn;save();gate.beginTurn(thread,activeTurn);}waiter.resolve(frame.result);}return; + } + if(frame.method==='turn/completed'&&frame.params.threadId===thread){turns.set(frame.params.turn.id,frame.params.turn);evidence.turns.push({id:frame.params.turn.id,status:frame.params.turn.status});save();gate?.endTurn(thread,frame.params.turn.id);if(activeTurn===frame.params.turn.id){activeTurn=null;evidence.activeTurn=null;save();}} + if(frame.method==='item/agentMessage/delta'&&frame.params.threadId===thread)process.stdout.write(frame.params.delta); + if(frame.id!==undefined&&frame.method)send({id:frame.id,error:{code:-32601,message:'No other host operations are authorized'}}); + }catch(error){fail(error);} + }); + await request('initialize',{clientInfo:{name:'firstmate-native',version:'0.1.0'},capabilities:{experimentalApi:true}});send({method:'initialized',params:{}}); + const dynamicTools=[ + {name:'fm_notification_check',description:'Read pending Firstmate notifications. Quiet means there is no new delivery.',inputSchema:{type:'object',properties:{},additionalProperties:false}}, + {name:'fm_notification_ack',description:'Acknowledge an observed and handled delivery. Never acknowledge unresolved decisions or unperformed work.',inputSchema:{type:'object',properties:{receipt:{type:'string'},observed:{type:'string'}},required:['receipt','observed'],additionalProperties:false}}, + ]; + const instructions='The native host already ran startup exactly once. Do not rerun startup or arm another supervisor. This experimental empty-fleet session supports only the two notification tools; do not claim to dispatch project work. The host continues notification checks after startup finishes. Use the supplied receipt and observed challenge only after handling the entire delivery. Unresolved work must remain pending. Startup digest follows:\n'+fs.readFileSync(path.join(runtime,'startup.log'),'utf8'); + const started=await request('thread/start',{cwd:root,sandbox:'read-only',approvalPolicy:'never',ephemeral:true,developerInstructions:instructions,dynamicTools}); + if(started.sandbox?.type!=='readOnly'||started.sandbox.networkAccess!==false||started.approvalPolicy!=='never')throw Error('App-server returned an unexpected security policy'); + thread=started.thread.id;evidence.thread=thread;evidence.ready=true;evidence.policy={sandbox:started.sandbox,approval:started.approvalPolicy};evidence.digestDeliveredBeforeDeferred=!fs.existsSync(path.join(runtime,'startup.finished'));save(); + gate=createNotificationGate({primaryThread:thread,operate:operation,isAlive:()=>alive&&!closing}); + console.error('Experimental native session ready. /interrupt stops the current turn; /quit ends the session.'); + while(!closing&&alive){ + if(process.env.FM_PROBE_VERIFY_ONLY==='1'){ + if(input.length)throw Error('Verify-only mode does not start model turns'); + if(ended)break;await pause(100);continue; + } + if(input.length){await turn(input.shift());continue;} + if(ended)break; + const result=await operation('check'); + if(result.operationState==='delivered'&&result.notification.receipt!==announced){announced=result.notification.receipt;await turn('A new durable notification is available. Read it with fm_notification_check, handle it within the available authority, and acknowledge only if fully handled.');} + else await pause(1000); + } + } +}catch(error){if(!closing)fail(error);}finally{await stop();} diff --git a/bin/native-owner/codex-tool-gate.mjs b/bin/native-owner/codex-tool-gate.mjs index cab0dbfe667..0d62b0ec0aa 100644 --- a/bin/native-owner/codex-tool-gate.mjs +++ b/bin/native-owner/codex-tool-gate.mjs @@ -67,6 +67,9 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { try { if (params.tool === 'fm_notification_check') { const result = await operate('check'); + if (result?.operationState === 'quiet') { + return valid(params) ? { success: true, value: { quiet: true } } : deny('wrong-thread-turn-or-replay'); + } const note = result?.notification; if (result?.operationState !== 'delivered' || !note || typeof note.receipt !== 'string' || !note.receipt || diff --git a/bin/native-owner/startup.sh b/bin/native-owner/startup.sh new file mode 100755 index 00000000000..43f5e934fd3 --- /dev/null +++ b/bin/native-owner/startup.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Run startup once, publish its digest promptly, retain only bounded deferred authority. +set -eu +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) +LOG=$(cygpath -u "${FM_PROBE_HOME:?}") +export FM_HOME +FM_HOME=$(cygpath -u "${FM_HOME:?}") +export PATH="$ROOT/bin/native-owner/tools:$PATH" +cd "$ROOT" +bin/fm-sessionstart-run.sh --source startup > "$LOG/startup.log" 2>&1 +. bin/fm-session-lock-lib.sh +fm_session_lock_owned_by_self "$FM_HOME/state" +[ "$(<"$FM_HOME/state/.session-start-complete")" = "$(<"$FM_HOME/state/.lock")" ] +if grep -q 'READ-ONLY SESSION\|SESSION START INCOMPLETE' "$LOG/startup.log"; then exit 1; fi +printf 'ready\n' > "$LOG/digest.ready" +# The existing worker owns network work. This scope neither repeats that work +# nor delays delivery of the digest to the host. +for ((i=0; i<180; i++)); do + phase=$(awk -F= '$1=="state" {print $2}' "$FM_HOME/state/.startup-network.status" 2>/dev/null || true) + case "$phase" in done|failed|timeout) break ;; esac + sleep 1 +done +case "$phase" in done|failed|timeout) ;; *) printf 'Deferred startup exceeded its bound\n' >&2; exit 1 ;; esac +bin/fm-startup-network.sh report > "$LOG/deferred.log" 2>&1 +printf 'finished\n' > "$LOG/startup.finished" diff --git a/bin/native-owner/tools/jq b/bin/native-owner/tools/jq new file mode 100755 index 00000000000..23d962ff8b5 --- /dev/null +++ b/bin/native-owner/tools/jq @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# Container-only jq; mount only the code root and operational home, read-only. +set -euo pipefail +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) +HOME_PATH=$(cygpath -u "${FM_HOME:?}") +image=${FM_PROBE_JQ_IMAGE:?An explicit local jq image is required} +export MSYS_NO_PATHCONV=1 +exec 'C:/Program Files/Docker/Docker/resources/bin/docker.exe' run --rm --pull never -i --network none \ + --mount "type=bind,source=$(cygpath -w "$ROOT"),target=$ROOT,readonly" \ + --mount "type=bind,source=$(cygpath -w "$HOME_PATH"),target=$HOME_PATH,readonly" \ + --workdir "$ROOT" "$image" timeout --kill-after=3s 45s jq "$@" diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 17caacbf239..ba88beaba72 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -8,10 +8,16 @@ Exact task chronology, branch names, temporary homes, local paths, process ids, ## Experimental native Windows ownership candidate -Verified on 2026-09-15 with Codex app-server 0.154.0, Windows 10.0.26200 x86_64, and the saved unelevated Windows sandbox. +Verified on 2026-09-15 and 2026-09-16 with Codex app-server 0.154.0, Windows 10.0.26200 x86_64, and the saved unelevated Windows sandbox. This is an isolated candidate verification, not an installed runtime backend or a claim that ordinary Codex shell tools can run Git Bash in that sandbox. -The native ownership core and Codex request policy live under `bin/native-owner/`; launchers, controlled messages, model prompts, Docker jq, and the temporary consumer integration patch remain test-only under `tests/fixtures/native-owner/`. -The home reservation deliberately still rejects non-temporary homes, and normal Firstmate startup does not load the candidate. +The native controller, interactive app-server host, fixed operation scripts, and request policy live under `bin/native-owner/`. +`bin/fm-native-codex.ps1` owns explicit launch and local compilation; its PowerShell help documents prerequisites and flags. +It does not install hooks, alter saved settings, pull images, or select a new default backend. +The home reservation rejects locations outside the user's Windows temporary directory and existing reparse-point ancestors. +Existing fleet metadata, projects, registrations, Relay configuration, and process-event sources are refused. +Those checks retain the experimental boundary rather than proving race-free paths or populated-fleet support. +Ordinary homes do not opt in; the real session-lock, harness, and startup-nudge readers select the native provider only from native home records. +The test-only consumer patch has been removed. Refresh the portable request-policy regression with: @@ -19,10 +25,10 @@ Refresh the portable request-policy regression with: bash tests/fm-native-owner-tool-gate.test.sh ``` -The 29 behavioral cases cover the request policy and bounded host shutdown, including synchronous revocation, single-flight shutdown, failed interruption, cancelled shutdown requests, retained-process termination, and unconfirmed-exit refusal. +The 31 behavioral cases cover the request policy and bounded host shutdown, including synchronous revocation, single-flight shutdown, failed interruption, cancelled shutdown requests, retained-process termination, and unconfirmed-exit refusal. The native controller now persists receipt presentation, acknowledgement intent, and completion under its existing exclusive home lease. It flushes acknowledgement intent before invoking the mutation and preserves interrupted attempts for reconciliation rather than retrying them. -The token-free Windows guard exercises 25 receipt cases, including complete-versus-partial recovery, preservation of newer work, missing or changed evidence, generation separation, file links, and lease revocation. +The token-free Windows guard exercises 28 receipt cases, including complete-versus-partial recovery, preservation of newer work, missing or changed evidence, generation separation, file links, and lease revocation. Two native operation-lifetime cases verify bounded stop and last-handle-close termination while an independent process remains alive. The guard also retains the exclusive-owner, normal-exit, controller-loss, orphan-recovery, ambiguous-record, and child-scope tests: @@ -54,8 +60,52 @@ Kill-on-close is applied only to fixed operation jobs, not the encompassing sess Two additional model-free cases interrupt the real acknowledgement scripts after the inbox move and after the queue acknowledgement, respectively. A new controller preserves the partial case as unresolved and reconciles the completed case from matching durable effects without replaying either mutation. Missing, changed, malformed, reparse-point, or incomplete evidence remains unresolved; recovery does not mean retrying or undoing a partial acknowledgement. -These results do not establish active-model-turn interruption, production-wide shutdown, adversarial Windows path/process races, populated-fleet behavior, or other harness support. -The production launcher and ordinary-startup integration remain unfinished and disabled while those boundaries are incomplete. +The receipt evidence also supports multiple inbox targets and general wake records with no inbox target; an unrelated note is never consumed. + +### Explicit launcher integration + +The opt-in launcher now runs real startup, delivers its digest before deferred work finishes, and retains the deferred operation's authorization until completion or bounded cancellation. +It starts an ephemeral read-only, network-disabled primary thread and verifies the returned app-server policy. +Its interactive host accepts ordinary input, `/interrupt`, and `/quit` and initiates model handling when a new durable notification arrives. +Only controller-selected startup, notification check, and acknowledgement scripts run with registered native operation authority. +Docker jq mounts are read-only, networking and image pulls are disabled, and its read-only helper process has a separate container-side expiry. +The launch-bound owner record preserves verified-dead predecessor generations across failed startup attempts so an intervening failed launch does not strand the previous session lock. +That history is bounded and refuses further acquisition rather than silently forgetting evidence. +An unresolved acknowledgement prevents more work and identifies the preserved journal for review; no partial attempt is retried or rolled back automatically. + +Build without launching, then explicitly test an empty temporary home from Windows PowerShell: + +```powershell +.\bin\fm-native-codex.ps1 -BuildOnly +.\bin\fm-native-codex.ps1 -Experimental -VerifyOnly ` + -Home "$env:LOCALAPPDATA\Temp\firstmate-native-example" -JqImage +``` + +Omit `-VerifyOnly` for an interactive model session. +The local image must contain jq and GNU timeout; this command does not install them. +The native provider must be rebuilt when its source stamp changes. + +Refresh the actual launcher without model turns, then optionally exercise two notification turns and active-turn cancellation: + +```sh +FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 \ + bash tests/fm-native-owner-launcher-live-e2e.test.sh +FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 FM_LIVE_NATIVE_CODEX=1 \ + bash tests/fm-native-owner-launcher-live-e2e.test.sh +``` + +The actual launcher refused a competing launch without changing its record, restarted after an intervening startup failure, refused a populated home unchanged, and delivered startup before a deliberately delayed deferred worker completed. +Non-temporary homes and pre-existing reparse-point ancestors were refused before creating a home. +Its cancellation stopped the registered deferred operation while an independent process stayed alive. +Two later inbox notifications each initiated a real model turn without another startup or a test-generated model prompt; each was observed and acknowledged through the fixed operations. +A separate shutdown interrupted an active model turn, confirmed app-server exit, and preserved the pending notification. +The production launcher also refused a real partially acknowledged journal from the fault-injection fixture and surfaced the reconciliation requirement without starting the model. +Fixture delays and controlled notifications remain under `tests/fixtures/native-owner/`; the launcher contains no test message generator. + +Production use remains disabled. +These results do not establish populated-fleet shutdown, forced app-server descendant cleanup, adversarial Windows path/process races, PID reuse and peer-handle races, integration of every numeric identity reader, other harness support, or installation and packaging. +The shared-reader rollout remains deliberately limited to the explicit empty-fleet launcher; Pi, OMP, OpenCode, and fleet lease consumers are not claimed supported. +Full no-mistakes validation has not run, and no publication or merge is authorized by these targeted tests. ## Harness detection precedence diff --git a/tests/fixtures/native-owner/Build.ps1 b/tests/fixtures/native-owner/Build.ps1 index c281f1c931f..d778056c6f0 100644 --- a/tests/fixtures/native-owner/Build.ps1 +++ b/tests/fixtures/native-owner/Build.ps1 @@ -8,15 +8,15 @@ $root = Join-Path ([IO.Path]::GetTempPath()) ('fm-native-candidate-' + [guid]::N New-Item -ItemType Directory $root | Out-Null $binary = Join-Path $root 'SessionProbe.exe' $sources = @((Join-Path $repo 'bin/native-owner/NativeOwner.cs'), (Join-Path $repo 'bin/native-owner/NativeHomeLease.cs'), (Join-Path $PSScriptRoot 'NativeDriver.cs'), (Join-Path $repo 'bin/native-owner/NativeReceiptJournal.cs'), (Join-Path $PSScriptRoot 'ReceiptTests.cs')) -$sources += @((Join-Path $repo 'bin/native-owner/NativeAcknowledgementEvidence.cs'), (Join-Path $repo 'bin/native-owner/NativeOperationLifetime.cs'), (Join-Path $PSScriptRoot 'OperationLifetimeTests.cs')) +$sources += @((Join-Path $repo 'bin/native-owner/NativeOperations.cs'), (Join-Path $repo 'bin/native-owner/NativeAcknowledgementEvidence.cs'), (Join-Path $repo 'bin/native-owner/NativeOperationLifetime.cs'), (Join-Path $PSScriptRoot 'OperationLifetimeTests.cs')) Add-Type -Path $sources -OutputAssembly $binary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll & $binary receipt-tests if ($LASTEXITCODE -ne 0) { throw 'Durable receipt lifecycle tests failed' } $copy = Join-Path $root 'firstmate' & git -c core.symlinks=true clone --quiet --no-local --single-branch $repo $copy if ($LASTEXITCODE -ne 0) { throw 'Disposable clone failed' } -& git -C $copy apply --whitespace=error (Join-Path $PSScriptRoot 'consumer-adapter.patch') -if ($LASTEXITCODE -ne 0) { throw 'Consumer test integration no longer applies; reconcile it explicitly' } +# Exercise the real opt-in consumers, including local changes before commit. +foreach ($name in @('fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh')) { Copy-Item (Join-Path $repo ('bin/' + $name)) (Join-Path $copy ('bin/' + $name)) } foreach ($name in @('exercise.sh','notification-check.sh','notification-ack.sh')) { Copy-Item (Join-Path $PSScriptRoot $name) (Join-Path $copy $name) } Copy-Item (Join-Path $PSScriptRoot 'AppHost.mjs') (Join-Path $root 'AppHost.mjs') foreach ($module in @('codex-tool-gate.mjs','host-lifecycle.mjs')) { Copy-Item (Join-Path $repo ('bin/native-owner/' + $module)) (Join-Path $root $module) } diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs new file mode 100644 index 00000000000..15eda1fbfc3 --- /dev/null +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -0,0 +1,127 @@ +// Actual launcher integration, not the fixture controller or a scripted model. +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import {fileURLToPath} from 'node:url'; +import {spawn,spawnSync} from 'node:child_process'; +const repo=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..'); +const area=fs.mkdtempSync(path.join(os.tmpdir(),'fm-launcher-e2e-')),code=path.join(area,'code'); +const image=process.env.FM_NATIVE_TEST_JQ_IMAGE; +if(!image)throw Error('FM_NATIVE_TEST_JQ_IMAGE must name an existing local image with jq and GNU timeout'); +const read=file=>JSON.parse(fs.readFileSync(file,'utf8').replace(/^\uFEFF/,'')); +const sleep=ms=>new Promise(resolve=>setTimeout(resolve,ms)); +function command(exe,args){const result=spawnSync(exe,args,{encoding:'utf8',timeout:120000});if(result.status!==0)throw Error(result.stderr||result.stdout);return result;} +command('git',['-c','core.symlinks=true','clone','--quiet','--no-local','--single-branch',repo,code]); +fs.cpSync(path.join(repo,'bin/native-owner'),path.join(code,'bin/native-owner'),{recursive:true}); +for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); +const launcher=path.join(code,'bin/fm-native-codex.ps1'); +command('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly']); +function start(home,verify=true){ + const args=['-NoProfile','-File',launcher,'-Experimental','-Home',home,'-JqImage',image];if(verify)args.push('-VerifyOnly'); + const child=spawn('powershell.exe',args,{stdio:['pipe','pipe','pipe']});let stdout='',stderr=''; + child.stdout.on('data',data=>stdout+=data);child.stderr.on('data',data=>stderr+=data);child.stdin.on('error',()=>{}); + const done=new Promise((resolve,reject)=>{child.on('error',reject);child.on('exit',exit=>resolve({exit,stdout,stderr}));}); + return {child,done,home}; +} +async function bound(promise,session,ms=230000){let timer;try{return await Promise.race([promise,new Promise((_,reject)=>{timer=setTimeout(()=>{session.child.stdin.write('/quit\n');reject(Error('Launcher exceeded its bound; inspect '+session.home));},ms);})]);}finally{clearTimeout(timer);}} +async function ready(session,previous){ + for(let i=0;i<2200;i++){ + if(session.child.exitCode!==null)throw Error(JSON.stringify(await session.done)); + try {const owner=read(path.join(session.home,'owner-probe.json'));if(owner.state==='live'&&typeof owner.generation==='string'&&owner.generation!==previous){const runtime=path.join(session.home,'state/native-runtime',owner.generation);if(read(path.join(runtime,'host.json')).ready)return {owner,runtime};}}catch(error){if(error.code!=='ENOENT'&&!(error instanceof SyntaxError))throw error;} + await sleep(100); + } + throw Error('Launcher readiness timed out'); +} +const records=[]; +const live=process.argv.includes('--live'); +if(live&&process.env.FM_LIVE_NATIVE_CODEX!=='1')throw Error('Live launcher tests require FM_LIVE_NATIVE_CODEX=1'); +const posix=value=>value.replaceAll('\\','/').replace(/^([A-Za-z]):/,(_,drive)=>'/'+drive.toLowerCase()); +function enqueue(home,message){ + fs.mkdirSync(home,{recursive:true}); + const env=Object.fromEntries(Object.entries(process.env).filter(([key])=>!key.startsWith('FM_')&&!key.startsWith('PI_'))); + Object.assign(env,{FM_HOME:posix(home),FM_PROBE_JQ_IMAGE:image,MSYS:'winsymlinks:nativestrict'}); + const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export PATH="$1/bin/native-owner/tools:/usr/bin:/bin:$PATH"; export FM_HOME; FM_HOME=$(cygpath -u "$3"); exec /usr/bin/bash "$1/bin/fm-inbox.sh" note "$2"','launcher-test',posix(code),message,home],{env,encoding:'utf8',timeout:30000}); + assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr}));return result.stdout.trim().split(/\s+/)[1]; +} +const home=path.join(area,'home'); +const first=start(home);const initial=await ready(first);console.error('first ready',area); +const competitor=start(home);competitor.child.stdin.end();const refused=await bound(competitor.done,competitor,20000); +console.error('competitor returned',refused);assert.notEqual(refused.exit,0);assert.equal(read(path.join(home,'owner-probe.json')).generation,initial.owner.generation); +console.error('quitting first');first.child.stdin.write('/quit\n');assert.equal((await bound(first.done,first,20000)).exit,0); +assert.equal(read(path.join(initial.runtime,'shutdown.json')).stopped,true); +records.push('exclusive launch and confirmed shutdown'); +const startup=path.join(code,'bin/native-owner/startup.sh'),startupSource=fs.readFileSync(startup,'utf8'); +fs.writeFileSync(startup,'#!/usr/bin/env bash\nexit 7\n'); +const failed=start(home);failed.child.stdin.end();assert.notEqual((await bound(failed.done,failed,20000)).exit,0); +assert.equal(fs.readFileSync(path.join(home,'state/.lock'),'utf8').trim(),'native:'+initial.owner.generation); +fs.writeFileSync(startup,startupSource); +const again=start(home);const restarted=await ready(again,initial.owner.generation);assert.notEqual(restarted.owner.generation,initial.owner.generation); +assert.equal(fs.readFileSync(path.join(home,'state/.lock'),'utf8').trim(),'native:'+restarted.owner.generation); +again.child.stdin.end();assert.equal((await bound(again.done,again,20000)).exit,0); +records.push('restart after an intervening failed startup retains proven-dead ownership history'); +const populated=path.join(area,'populated');fs.mkdirSync(path.join(populated,'state'),{recursive:true});fs.writeFileSync(path.join(populated,'state/work.meta'),'preserve'); +const blocked=start(populated);blocked.child.stdin.end();assert.notEqual((await bound(blocked.done,blocked,20000)).exit,0); +assert.equal(fs.readFileSync(path.join(populated,'state/work.meta'),'utf8'),'preserve');assert.equal(fs.existsSync(path.join(populated,'owner-probe.json')),false); +records.push('populated home refused without changing its records'); +const outside=path.join(repo,'data/native-launcher',path.basename(area)+'-outside'); +assert.equal(fs.existsSync(outside),false); +const external=start(outside);external.child.stdin.end();assert.notEqual((await bound(external.done,external,20000)).exit,0);assert.equal(fs.existsSync(outside),false); +const target=path.join(area,'junction-target'),alias=path.join(area,'junction');fs.mkdirSync(target);fs.symlinkSync(target,alias,'junction'); +const linked=start(path.join(alias,'home'));linked.child.stdin.end();assert.notEqual((await bound(linked.done,linked,20000)).exit,0);assert.equal(fs.existsSync(path.join(target,'home')),false); +records.push('non-temporary and pre-existing reparse-point homes refused before creation'); +// Delay the existing network owner only in this disposable code copy. The +// production launcher has no delay/mock switch and still invokes that owner. +const network=path.join(code,'bin/fm-startup-network.sh');fs.renameSync(network,network+'.actual'); +fs.writeFileSync(network,'#!/usr/bin/env bash\nif [ "${1:-}" = run ]; then sleep 15; fi\nexec "$(dirname "$0")/fm-startup-network.sh.actual" "$@"\n'); +// Its start command invokes the original script's self path. Delay the worker +// entry inside that preserved copy, before any original script logic executes. +const original=fs.readFileSync(network+'.actual','utf8');fs.writeFileSync(network+'.actual',original.replace(/^#![^\n]*\n/,'#!/usr/bin/env bash\nif [ "${1:-}" = run ]; then sleep 15; fi\n')); +const deferred=start(path.join(area,'deferred'));const early=await ready(deferred); +const earlyEvidence=read(path.join(early.runtime,'host.json')); +assert.equal(earlyEvidence.digestDeliveredBeforeDeferred,true); +const independent=spawn(process.execPath,['-e','setTimeout(()=>{},60000)'],{stdio:'ignore'}); +try { + deferred.child.stdin.write('/quit\n');assert.equal((await bound(deferred.done,deferred,20000)).exit,0); + assert.equal(read(path.join(early.runtime,'shutdown.json')).stopped,true);assert.equal(independent.exitCode,null); + records.push('digest delivered before deferred completion; cancellation preserves an independent process'); +}finally{independent.kill();} +fs.writeFileSync(network,original);fs.unlinkSync(network+'.actual'); +if(live){ + const messageHome=path.join(area,'live-message'); + const model=start(messageHome,false);const liveReady=await ready(model);const runtime=liveReady.runtime; + const notes=[]; + for(let cycle=1;cycle<=2;cycle++){ + notes.push(enqueue(messageHome,'Launcher integration notification '+cycle+': no project action is requested. Read and acknowledge this notification.')); + let completed=false; + for(let i=0;i<1800;i++){ + if(model.child.exitCode!==null)throw Error(JSON.stringify(await model.done)); + const evidence=read(path.join(runtime,'host.json')); + if(evidence.turns.length>=cycle&&evidence.turns[cycle-1].status==='completed'){completed=true;break;} + await sleep(100); + } + assert(completed,'The notification did not produce a completed model turn'); + } + model.child.stdin.write('/quit\n');const result=await bound(model.done,model,20000);assert.equal(result.exit,0,result.stderr); + const host=read(path.join(runtime,'host.json'));assert.equal(host.turns.length,2); + assert(host.tools.some(tool=>tool.tool==='fm_notification_check'&&tool.success));assert(host.tools.some(tool=>tool.tool==='fm_notification_ack'&&tool.success)); + for(const note of notes)assert(fs.existsSync(path.join(messageHome,'state/inbox/handled',note+'.note'))); + assert.equal(fs.readFileSync(path.join(messageHome,'state/.wake-queue'),'utf8').trim(),''); + fs.writeFileSync(path.join(runtime,'console.json'),JSON.stringify(result,null,2)); + records.push('two real post-startup notification cycles were automatically delivered, observed, and acknowledged'); + const cancelHome=path.join(area,'live-cancel');const pendingNote=enqueue(cancelHome,'Cancellation test: leave this notification pending; do not acknowledge it.'); + const active=start(cancelHome,false);active.child.stdin.write('Call fm_notification_check once, but do not acknowledge anything. Explain what remains pending. Do not use other tools.\n'); + const current=await ready(active);let began=false; + for(let i=0;i<1200;i++){if(read(path.join(current.runtime,'host.json')).activeTurn){began=true;break;}await sleep(50);} + assert(began,'No active turn to cancel');active.child.stdin.write('/quit\n'); + const stopped=await bound(active.done,active,20000);assert.equal(stopped.exit,0,stopped.stderr); + assert(read(path.join(current.runtime,'host.json')).turns.some(turn=>turn.status==='interrupted')); + assert.equal(read(path.join(current.runtime,'shutdown.json')).stopped,true); + assert(fs.existsSync(path.join(cancelHome,'state/inbox',pendingNote+'.note'))); + records.push('shutdown interrupted a real active turn and preserved its pending notification'); +} +const state=path.join(repo,'data/native-launcher');fs.mkdirSync(state,{recursive:true}); +const summary=JSON.stringify({area,code,home,records,live,passed:true},null,2); +fs.writeFileSync(path.join(state,'launcher-latest.json'),summary); +fs.writeFileSync(path.join(state,live?'launcher-live-latest.json':'launcher-smoke-latest.json'),summary); +console.log('PASS: '+records.join('; ')); diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index add309411b0..b3fdd37bc58 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -14,8 +14,6 @@ using System.Web.Script.Serialization; public static partial class NativeOwner { - static NativeReceiptJournal operationJournal; - static bool shutdownRequested; static string LogonSid() { using(var identity=WindowsIdentity.GetCurrent()) { foreach(var row in TokenGroups(identity.Token,2)) @@ -29,58 +27,6 @@ static Dictionary TokenFacts() { {"groups",TokenGroups(identity.Token,2)},{"restricting",TokenGroups(identity.Token,11)} }; } - static IntPtr FileHandle(string path, uint access, uint creation) { - SA sa = new SA { length=Marshal.SizeOf(typeof(SA)), inherit=1 }; - IntPtr h = CreateFile(path, access, 3, ref sa, creation, 0x80, IntPtr.Zero); - if (h == new IntPtr(-1)) throw Error("CreateFile"); - return h; - } - static SortedDictionary EnvironmentFor(string pipe, string session, string home, string nonce) { - var result = new SortedDictionary(StringComparer.OrdinalIgnoreCase); - foreach (DictionaryEntry e in Environment.GetEnvironmentVariables()) { - string k = (string)e.Key; - if (k.StartsWith("FM_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("PI_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("NO_MISTAKES", StringComparison.OrdinalIgnoreCase) || k == "CLAUDE_PID" || k == "CLAUDECODE") continue; - result[k] = (string)e.Value; - } - result["FM_PROBE_PIPE"] = pipe; result["FM_PROBE_SESSION"] = session; - result["FM_PROBE_HOME"] = home; result["FM_PROBE_NONCE"] = nonce; - result["FM_PROBE_EXE"] = OwnExe; - return result; - } - static ChildScope StartScope(string role, IntPtr parentJob, IntPtr environment, string home, ref SI startup, string purpose="startup") { - var scope=new ChildScope { job=CreateJobObject(IntPtr.Zero,null), role=role, purpose=purpose }; - IntPtr operationEnvironment=IntPtr.Zero; - if(scope.job==IntPtr.Zero) throw Error("CreateJobObject child scope"); - try { - if(role=="owner-operation") NativeOperationLifetime.Configure(scope.job); - string operation=role=="owner-operation" ? (purpose=="startup" ? "owner-operation" : "notification-operation "+purpose) : "scoped-client "+role; - if(role=="owner-operation") { - // Only this fixed, non-extensible test operation receives the grant. - var values=new SortedDictionary(StringComparer.OrdinalIgnoreCase); - foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { - string value=Environment.GetEnvironmentVariable(key); if(value!=null) values[key]=value; - } - values["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); - // Copy only the controller's registration fields from the prepared block. - int offset=0; - while(Marshal.ReadInt16(environment,offset)!=0) { - string entry=Marshal.PtrToStringUni(IntPtr.Add(environment,offset)); offset+=(entry.Length+1)*2; - int split=entry.IndexOf('='); if(split<=0) continue; - string key=entry.Substring(0,split); - if(key.StartsWith("FM_PROBE_",StringComparison.Ordinal) || key=="FM_HOME" || key=="MSYS") values[key]=entry.Substring(split+1); - } - var block=new StringBuilder(); foreach(var value in values) block.Append(value.Key).Append('=').Append(value.Value).Append('\0'); block.Append('\0'); - operationEnvironment=Marshal.StringToHGlobalUni(block.ToString()); - } - if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" "+operation),IntPtr.Zero,IntPtr.Zero,true,0x4|0x400,operationEnvironment==IntPtr.Zero ? environment : operationEnvironment,home,ref startup,out scope.process)) throw Error("CreateProcess child scope"); - if(!AssignProcessToJobObject(parentJob,scope.process.process) || !AssignProcessToJobObject(scope.job,scope.process.process)) throw Error("Assign child scope"); - // The caller records this scope before resuming the process. - return scope; - } catch { - if(scope.process.process!=IntPtr.Zero) { TerminateProcess(scope.process.process,125); CloseHandle(scope.process.thread); CloseHandle(scope.process.process); } - CloseHandle(scope.job); throw; - } finally { if(operationEnvironment!=IntPtr.Zero) Marshal.FreeHGlobal(operationEnvironment); } - } static void ReleaseFixtureWhenScopesEnd(List scopes,string home) { if(scopes.Count==0) return; foreach(var scope in scopes) if(WaitForSingleObject(scope.process.process,0)==WAIT_TIMEOUT) return; @@ -283,52 +229,6 @@ static int Run(string configPath) { if(lease!=null) lease.Dispose(); } } - static void NotificationRequest(Dictionary request,Dictionary verdict,NativeHomeLease lease,IntPtr job,IntPtr environment,string home,ref SI startup,List scopes) { - bool allowed=lease!=null && (string)verdict["association"]=="associated" && (string)verdict["hostClassification"]=="registered-primary"; - verdict["notificationAuthorized"]=allowed; - if(!allowed) return; - string action=request.ContainsKey("action") ? (string)request["action"] : ""; - if(action=="shutdown") { - shutdownRequested=true; - foreach(var scope in scopes) if(scope.role=="owner-operation") NativeOperationLifetime.Stop(scope.job,1500); - verdict["operationState"]="stopped"; - return; - } - if(shutdownRequested) { verdict["operationState"]="stopped";return; } - bool ready=true; - foreach(var scope in scopes) if(scope.purpose=="startup" && WaitForSingleObject(scope.process.process,0)==WAIT_TIMEOUT) ready=false; - verdict["startupExpired"]=ready; - if(pendingOperation!=null && WaitForSingleObject(pendingOperation.process.process,0)!=WAIT_TIMEOUT) { - uint code; if(!GetExitCodeProcess(pendingOperation.process.process,out code)) throw Error("Operation exit"); - verdict["operationExit"]=code; - if(code!=0) { verdict["operationState"]="failed"; return; } - string file=Path.Combine(home,"notification-"+pendingOperation.purpose+".json"); - var output=Json.Deserialize>(File.ReadAllText(file)); - if(pendingOperation.purpose=="check") { - delivered=operationJournal.Present(output); receipt=(string)delivered["receipt"]; - consumed=false; - } else { operationJournal.CompleteAcknowledgement(receipt);consumed=true; } - pendingOperation=null; - } - if(action=="result") { - verdict["operationState"]=pendingOperation!=null ? "pending" : operationJournal.NeedsReconciliation ? "reconciliation-required" : consumed ? "acknowledged" : delivered!=null ? "delivered" : "idle"; - if(delivered!=null) verdict["notification"]=delivered; - return; - } - if(!ready || pendingOperation!=null) { verdict["operationState"]="busy"; return; } - if(operationJournal.NeedsReconciliation) { verdict["operationState"]="reconciliation-required";return; } - if(action=="check" && (delivered==null || consumed)) checkStarts++; - else if(action=="ack" && delivered!=null && !consumed && request.ContainsKey("receipt") && (string)request["receipt"]==receipt && request.ContainsKey("observed") && (string)request["observed"]==(string)delivered["challenge"]) { - var targetEvidence=NativeAcknowledgementEvidence.Capture(lease,delivered); - var acknowledged=operationJournal.BeginAcknowledgement(receipt,(string)request["observed"],targetEvidence); - ackStarts++; - File.WriteAllText(Path.Combine(home,"notification-ack-request.json"),Json.Serialize(acknowledged)); - } else { verdict["operationState"]="denied"; return; } - pendingOperation=StartScope("owner-operation",job,environment,home,ref startup,action); - scopes.Add(pendingOperation); - if(ResumeThread(pendingOperation.process.thread)==0xffffffff) throw Error("Resume notification operation"); - verdict["operationState"]="pending"; - } public static int Main(string[] args) { try { if(args.Length==1 && args[0]=="receipt-tests") { ReceiptTests.Run();return TestOperationLifetime(); } diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index 660060a0ee4..4db9b897df2 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -152,6 +152,29 @@ public static int Run() { Expect(!journal.NeedsReconciliation,"Rejected evidence created an attempt"); } }); + foreach(string scenario in new [] {"no-inbox-targets","multiple-complete","multiple-partial"}) { + Case("general wake recovery: "+scenario,lease=>{ + Targets(lease);var payload=Payload("general");payload.Remove("note"); + if(scenario=="no-inbox-targets") { + payload["notes"]=new string[0];File.WriteAllText(Queue(lease),"1\t1\tcheck\tdiagnostic\treport\n"); + } else { + payload["notes"]=new [] {"note-id","second"};payload["seq"]="2"; + File.WriteAllText(Path.Combine(lease.Home,"state","inbox","second.note"),"second notification"); + File.AppendAllText(Queue(lease),"2\t2\tcheck\tinbox:second\tsecond\n"); + } + using(var journal=new NativeReceiptJournal(lease,A)) { + var delivery=journal.Present(payload); + journal.BeginAcknowledgement((string)delivery["receipt"],"general",NativeAcknowledgementEvidence.Capture(lease,delivery)); + } + if(scenario!="no-inbox-targets") { + File.Move(Pending(lease),Handled(lease)); + if(scenario=="multiple-complete")File.Move(Path.Combine(lease.Home,"state","inbox","second.note"),Path.Combine(lease.Home,"state","inbox","handled","second.note")); + } + File.WriteAllText(Queue(lease),""); + using(var journal=new NativeReceiptJournal(lease,B)) Expect(journal.ReconcileCompletedAcknowledgements()==(scenario=="multiple-partial"?0:1),"Incorrect general-wake recovery"); + if(scenario=="no-inbox-targets")Expect(File.Exists(Pending(lease)),"Unrelated inbox note was consumed"); + }); + } Console.WriteLine("RECEIPT_TESTS_PASS "+passed);return 0; } } diff --git a/tests/fixtures/native-owner/consumer-adapter.patch b/tests/fixtures/native-owner/consumer-adapter.patch deleted file mode 100644 index 0a87b9be8ed..00000000000 --- a/tests/fixtures/native-owner/consumer-adapter.patch +++ /dev/null @@ -1,96 +0,0 @@ -diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh -index 73caeeca..549cb15a 100644 ---- a/bin/fm-session-lock-lib.sh -+++ b/bin/fm-session-lock-lib.sh -@@ -143,6 +143,14 @@ fm_harness_process_matches() { # - # non-numeric so that any consumer treating it as a Cygwin pid - including a - # future `kill` - refuses it instead of acting on the wrong process. - FM_WIN_PID_PREFIX='win:' -+# Isolated native-owner adapter; executable location is code-owned, not supplied -+# by the caller's environment. Unknown/unavailable ownership never falls back. -+FM_NATIVE_OWNER_BIN="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-native-owner.exe" -+fm_native_owner_call() { # [id] -+ local native_state -+ native_state=$(cygpath -w "${FM_STATE_OVERRIDE:-${FM_HOME:?}/state}") || return 2 -+ MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner "$1" "$native_state" "${2:-}" -+} - - # True on a Cygwin-family userspace, where the boundary above applies. - fm_win_boundary_applies() { -@@ -167,7 +175,13 @@ fm_win_untag_pid() { # - # holder - which reads as "startup never completed" and repeats the whole - # sequence on every clear or compact. - fm_session_pid_valid() { # -+ local native_id - case "$1" in -+ native:*) -+ native_id=${1#native:} -+ [ "${#native_id}" -eq 32 ] || return 1 -+ case "$native_id" in *[!0-9a-f]*) return 1 ;; esac -+ return 0 ;; - "$FM_WIN_PID_PREFIX"[0-9]*) return 0 ;; - ''|*[!0-9]*) return 1 ;; - esac -@@ -257,6 +271,7 @@ fm_win_harness_ancestry_pids() { - # session cannot be read off the ancestry at all, so the whole contiguous run is - # reported and the callers below decide what they need from it. - fm_harness_ancestry_pids() { -+ if fm_win_boundary_applies; then fm_native_owner_call identity; return; fi - local pid=$$ comm args extending=0 printed=0 - for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do - comm=$(fm_ps_comm "$pid") || break -@@ -311,7 +326,14 @@ EOF - # kill -0 cannot see across that boundary and would report a live harness as - # dead - which would hand a running session's home to a second one. - fm_harness_pid_alive() { -- local pid=$1 comm args winpid -+ local pid=$1 comm args winpid owner_rc -+ if fm_win_boundary_applies; then -+ fm_native_owner_call alive "$pid" -+ owner_rc=$? -+ # Only an explicit proven-dead answer permits replacing an old record. -+ [ "$owner_rc" -ne 1 ] -+ return -+ fi - if winpid=$(fm_win_untag_pid "$pid"); then - comm=$(fm_win_command "$winpid") || return 1 - fm_harness_process_matches "$comm" "$comm" -@@ -332,7 +354,12 @@ fm_harness_pid_alive() { - # lock, a malformed lock, a lock held by a harness outside this ancestry, or an - # ancestry that cannot be resolved all fail closed. - fm_session_lock_owned_by_self() { -- local state=$1 lock_pid pids pid -+ local state=$1 lock_pid pids pid native_state -+ if fm_win_boundary_applies; then -+ native_state=$(cygpath -w "$state") || return 1 -+ MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner owns "$native_state" -+ return -+ fi - lock_pid=$(cat "$state/.lock" 2>/dev/null || true) - fm_session_pid_valid "$lock_pid" || return 1 - pids=$(fm_harness_ancestry_pids) || return 1 -diff --git a/bin/fm-harness.sh b/bin/fm-harness.sh -index 7989643f..201b61cb 100644 ---- a/bin/fm-harness.sh -+++ b/bin/fm-harness.sh -@@ -395,7 +395,19 @@ harness_family() { - # a harness-shaped path in some node process's arguments is weaker evidence - # than a harness publishing its own identity. - detect_own() { -- local marker ancestry strength harness -+ local marker ancestry strength harness native_state -+ # Disposable native integration: ask the registered launch, not this sibling -+ # operation's ancestry. Denial never adopts an inherited agent marker. -+ case "$(uname -s)" in -+ MINGW*|MSYS*|CYGWIN*) -+ if [ -f "$SCRIPT_DIR/fm-native-owner.exe" ]; then -+ native_state=$(cygpath -w "${FM_STATE_OVERRIDE:-$FM_HOME/state}") || { echo unknown; return; } -+ harness=$(MSYS2_ARG_CONV_EXCL='*' "$SCRIPT_DIR/fm-native-owner.exe" owner harness "$native_state" 2>/dev/null) || { echo unknown; return; } -+ case "$harness" in codex|unknown) echo "$harness" ;; *) echo unknown ;; esac -+ return -+ fi -+ ;; -+ esac - marker=$(harness_marker) - ancestry=$(harness_ancestry) - if [ -z "$ancestry" ]; then diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index ba4848a1f76..291919ad668 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -14,6 +14,18 @@ function fixture(operate) { const check=(overrides={})=>({threadId:'primary',turnId:'turn',callId:'check',namespace:null,tool:'fm_notification_check',arguments:{},...overrides}); const ack=(overrides={})=>check({callId:'ack',tool:'fm_notification_ack',arguments:{receipt:'receipt',observed:'observed'},...overrides}); +test('quiet checks create no receipt and permit a later delivery',async()=>{ + let count=0;const {gate,calls}=fixture(()=>++count===1?{operationState:'quiet'}:{operationState:'delivered',notification:message}); + assert.deepEqual((await gate.handle(check())).value,{quiet:true}); + assert.equal((await gate.handle(ack())).success,false); + assert.equal((await gate.handle(check({callId:'later'}))).value.receipt,'receipt'); + assert.equal(calls.length,2); +}); +test('a cancelled quiet check cannot authorize another turn',async()=>{ + let finish;const {gate}=fixture(()=>new Promise(resolve=>{finish=resolve;})); + const pending=gate.handle(check());gate.endTurn('primary','turn');finish({operationState:'quiet'}); + assert.equal((await pending).success,false); +}); test('observation precedes one acknowledgement; receipt replay cannot execute twice',async()=>{ const {gate,calls}=fixture(); assert.equal((await gate.handle(ack())).success,false); diff --git a/tests/fm-native-owner-launcher-live-e2e.test.sh b/tests/fm-native-owner-launcher-live-e2e.test.sh new file mode 100755 index 00000000000..9219454837d --- /dev/null +++ b/tests/fm-native-owner-launcher-live-e2e.test.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# Actual native launcher. Default opt-out; model turns require a second opt-in. +set -eu +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +fm_live_gate opt-in FM_LIVE_NATIVE_LAUNCHER node powershell.exe codex docker +case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) ;; *) printf 'Native launcher tests require Windows\n' >&2; exit 1 ;; esac +args=() +if [ "${FM_LIVE_NATIVE_CODEX:-}" = 1 ]; then args+=(--live); fi +node "$ROOT/tests/fixtures/native-owner/Launcher.mjs" "${args[@]}" diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index 5dcee2fea75..d609e6d6811 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -572,6 +572,33 @@ test_e2e_daemon_parented_version_named_session_keeps_its_lock() { pass "session-lock e2e: a version-named session under a harness-named daemon keeps its own lock" } +test_native_state_contract() ( + # shellcheck source=bin/fm-session-lock-lib.sh + . "$LIB" + local answer rc identity=native:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + fm_native_owner_call() { return "$answer"; } + fm_session_pid_valid "$identity" || fail "valid native identity rejected" + if fm_session_pid_valid native:123; then fail "malformed native identity accepted"; fi + for answer in 0 1 2; do + rc=0; fm_native_owner_state "$identity" || rc=$? + [ "$rc" -eq "$answer" ] || fail "native three-state result collapsed" + rc=0; fm_harness_pid_alive "$identity" || rc=$? + if [ "$answer" -eq 1 ]; then + [ "$rc" -ne 0 ] || fail "proven-dead owner retained occupancy" + else + [ "$rc" -eq 0 ] || fail "live or unknown owner lost exclusion" + fi + done + FM_HOME="$TMP_ROOT/native-selection" + FM_STATE_OVERRIDE="$FM_HOME/state" + mkdir -p "$FM_STATE_OVERRIDE" + if fm_native_owner_selected; then fail "ordinary home opted in without records"; fi + printf '%s\n' "$identity" > "$FM_STATE_OVERRIDE/.lock" + fm_native_owner_selected || fail "native lock lost routing without its binding" + pass "native identity routing preserves unknown exclusion without certifying health" +) + +test_native_state_contract test_version_named_session_is_identified_on_both_platforms test_harness_at_namespace_pid1_is_examined test_ordinary_paths_are_never_harness_processes From 9b7efa231ce8bdbccb816df6c8bf608f4efdc1c2 Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 15:02:33 +1200 Subject: [PATCH 09/61] no-mistakes(review): Fix native ownership and app isolation regressions --- CONTRIBUTING.md | 7 +- bin/fm-lock.sh | 16 +++-- bin/fm-native-codex.ps1 | 8 +-- bin/fm-session-lock-lib.sh | 23 +++++-- bin/native-owner/NativeHomeLease.cs | 7 +- bin/native-owner/NativeLauncher.cs | 9 ++- bin/native-owner/NativeOperations.cs | 4 +- bin/native-owner/NativeOwner.cs | 10 +++ bin/native-owner/app-server-policy.mjs | 67 +++++++++++++++++++ bin/native-owner/codex-host.mjs | 7 +- docs/documentation-audiences.json | 8 +++ docs/native-windows-codex.md | 37 ++++++++++ docs/verification/runtime-backends.md | 40 ++++------- tests/fixtures/native-owner/AppHost.mjs | 6 +- .../fixtures/native-owner/AppServerPolicy.mjs | 46 +++++++++++++ tests/fixtures/native-owner/Build.ps1 | 15 ++++- tests/fixtures/native-owner/Launcher.mjs | 4 +- tests/fixtures/native-owner/NativeDriver.cs | 32 ++++++++- tests/fixtures/native-owner/Verify-Cycle.mjs | 1 + ...e-owner-app-server-policy-live-e2e.test.sh | 8 +++ tests/fm-session-lock-ancestry.test.sh | 53 +++++++++++++++ 21 files changed, 348 insertions(+), 60 deletions(-) create mode 100644 bin/native-owner/app-server-policy.mjs create mode 100644 docs/native-windows-codex.md create mode 100644 tests/fixtures/native-owner/AppServerPolicy.mjs create mode 100644 tests/fm-native-owner-app-server-policy-live-e2e.test.sh diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9567425893b..0ee66758e70 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -44,8 +44,11 @@ See the [no-mistakes quick start](https://kunchenguid.github.io/no-mistakes/star A local `config/backend` file explicitly overrides runtime auto-detection for new task endpoints and stays gitignored; spawn-supported values are `tmux`, `herdr` (which has its own required CI lane), and `zellij`, `orca`, and `cmux`, which remain experimental with no dedicated real-backend CI lane, while `codex-app` is documented only in `docs/codex-app-backend.md`. It does not make `data/` tracked. - Helper scripts in `bin/` are plain bash. - Each starts with a usage header comment; keep it accurate when you change behavior. - Test scripts and helpers in `tests/` are plain bash too. + The restricted experimental native Windows Codex launcher is the sole scoped exception: `bin/fm-native-codex.ps1` and the C# and JavaScript implementation under `bin/native-owner/` may use PowerShell, C#, and JavaScript. + Matching fixtures under `tests/fixtures/native-owner/` may use those languages too. + This exception does not extend to other helpers, other platforms, or future implementations. + Each Bash helper starts with a usage header comment; keep it accurate when you change behavior. + Test scripts and helpers in `tests/` are plain bash too, except for the matching native Windows fixtures above. `bin/fm-lint.sh` must pass: it is the single owner of the lint definition (the shellcheck file set, config, pinned shellcheck version, pinned actionlint workflow lint, and the backend-purity check rejecting direct Beads CLI calls in core `bin/` scripts), and both CI and the no-mistakes pre-push gate invoke it with no arguments. Its header and `--help` output own the exact local lint modes, file-set selection, and analysis flags. A malformed `.github/workflows/*.yml`, including a self-broken `ci.yml`, fails that local lint path before merge because a broken workflow cannot report its own breakage. diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index 6058afe2334..da98d3f78a1 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -29,7 +29,13 @@ if [ "${1:-}" = "status" ]; then echo "lock: unreadable" exit 0 } - if fm_harness_pid_alive "$old"; then echo "lock: held by live harness pid $old"; else echo "lock: stale (pid $old dead or not a harness)"; fi + if fm_harness_pid_alive "$old"; then + echo "lock: held by live harness pid $old" + elif fm_harness_pid_excludes "$old"; then + echo "lock: held by native owner with unconfirmed health $old" + else + echo "lock: stale (pid $old dead or not a harness)" + fi exit 0 fi @@ -71,8 +77,8 @@ if [ -f "$LOCK" ] && [ ! -L "$LOCK" ]; then echo "lock acquired: harness pid $me" exit 0 fi - if fm_harness_pid_alive "$old"; then - echo "error: another live firstmate session holds the lock (pid $old); operate read-only until resolved" >&2 + if fm_harness_pid_excludes "$old"; then + echo "error: another firstmate session may hold the lock (pid $old); operate read-only until resolved" >&2 exit 1 fi fi @@ -96,8 +102,8 @@ if [ -e "$LOCK" ] || [ -L "$LOCK" ]; then echo "error: session lock is unreadable; operate read-only until resolved" >&2 exit 1 } - if [ "$old" != "$me" ] && fm_harness_pid_alive "$old"; then - echo "error: another live firstmate session holds the lock (pid $old); operate read-only until resolved" >&2 + if [ "$old" != "$me" ] && fm_harness_pid_excludes "$old"; then + echo "error: another firstmate session may hold the lock (pid $old); operate read-only until resolved" >&2 exit 1 fi fi diff --git a/bin/fm-native-codex.ps1 b/bin/fm-native-codex.ps1 index 27b49618bd6..5ec6255dc62 100644 --- a/bin/fm-native-codex.ps1 +++ b/bin/fm-native-codex.ps1 @@ -20,16 +20,16 @@ Compile the local provider and source stamp without launching a session. .PARAMETER VerifyOnly Connect startup and app-server without starting any model turns. .PARAMETER OperationalHome -The Windows path to the temporary operational home; Home is an alias. +The Windows path to the temporary operational home. .PARAMETER JqImage An existing local Docker image containing jq and GNU timeout. No image is pulled. Read-only helper containers self-expire even if their native client is stopped. .NOTES Use /interrupt to interrupt the current model turn and /quit to end the session. Interrupted acknowledgements remain pending for evidence-based reconciliation. -See docs/verification/runtime-backends.md for the tested boundary and guards. +See docs/native-windows-codex.md for setup and supported limits. #> -param([switch]$Experimental,[switch]$BuildOnly,[switch]$VerifyOnly,[Alias("Home")][string]$OperationalHome,[string]$JqImage) +param([switch]$Experimental,[switch]$BuildOnly,[switch]$VerifyOnly,[string]$OperationalHome,[string]$JqImage) $ErrorActionPreference='Stop' $root=[IO.Path]::GetFullPath((Join-Path $PSScriptRoot '..')) $names=@('NativeOwner','NativeHomeLease','NativeReceiptJournal','NativeAcknowledgementEvidence','NativeOperationLifetime','NativeOperations','NativeLauncher') @@ -48,7 +48,7 @@ if ($BuildOnly) { exit 0 } if (!$Experimental) { throw 'Explicit -Experimental opt-in is required; production use remains disabled.' } -if (!$OperationalHome) { throw '-Home must name a temporary empty-fleet home.' } +if (!$OperationalHome) { throw '-OperationalHome must name a temporary empty-fleet home.' } if (!$JqImage -or $JqImage -notmatch '^[A-Za-z0-9][A-Za-z0-9./:_@-]+$') { throw '-JqImage must name an existing local Docker image containing jq.' } if (!(Test-Path $binary) -or !(Test-Path $stamp) -or [IO.File]::ReadAllText($stamp) -ne $fingerprint) { throw 'Provider missing or out of date; run -BuildOnly after stopping native sessions.' } & docker image inspect $JqImage *> $null diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index cdd9097da1a..be459467846 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -155,7 +155,11 @@ fm_native_owner_selected() { fm_native_owner_call() { local native_state native_state=$(cygpath -w "${FM_STATE_OVERRIDE:-${FM_HOME:?}/state}") || return 2 - MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner "$1" "$native_state" "${2:-}" + case "$1" in + alive) MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner "$1" "$native_state" "${2:?}" ;; + identity|owns|harness) MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner "$1" "$native_state" ;; + *) return 2 ;; + esac } # Three states: 0 is live, 1 proven dead, 2 unknown. Do not turn exclusion # (unknown must preserve occupancy) into a positive health assertion. @@ -341,11 +345,10 @@ EOF # kill -0 cannot see across that boundary and would report a live harness as # dead - which would hand a running session's home to a second one. fm_harness_pid_alive() { - local pid=$1 comm args winpid owner_rc + local pid=$1 comm args winpid case "$pid" in native:*) - if fm_native_owner_state "$pid"; then return 0; else owner_rc=$?; fi - # Compatibility for exclusion readers only; native health uses the state API. - [ "$owner_rc" -ne 1 ]; return ;; + fm_native_owner_state "$pid" + return ;; esac if winpid=$(fm_win_untag_pid "$pid"); then comm=$(fm_win_command "$winpid") || return 1 @@ -358,6 +361,16 @@ fm_harness_pid_alive() { fm_harness_process_matches "$comm" "$args" } +fm_harness_pid_excludes() { + local pid=$1 owner_rc + case "$pid" in native:*) + if fm_native_owner_state "$pid"; then return 0; else owner_rc=$?; fi + [ "$owner_rc" -ne 1 ] + return ;; + esac + fm_harness_pid_alive "$pid" +} + # True when state dir $1 holds a session lock whose pid is ANY harness ancestor # of the current process: this script runs inside the session that owns the # home's fleet lock. Membership is the honest test of that question, because the diff --git a/bin/native-owner/NativeHomeLease.cs b/bin/native-owner/NativeHomeLease.cs index de97050a020..cbf5bdf6cae 100644 --- a/bin/native-owner/NativeHomeLease.cs +++ b/bin/native-owner/NativeHomeLease.cs @@ -15,7 +15,6 @@ public sealed class NativeHomeLease : IDisposable { FileStream file; public readonly string Home; internal bool IsHeld { get { return file!=null; } } - public string PreviousGeneration { get; private set; } readonly HashSet deadGenerations=new HashSet(); static bool Generation(string value) { if(value==null||value.Length!=32)return false; @@ -77,14 +76,14 @@ public NativeHomeLease(string home) { if(file.Length>0) { var previous=Read(file); if(RootAlive(previous)) throw new InvalidOperationException("Recorded primary is still alive; refusing replacement"); - PreviousGeneration=previous.ContainsKey("generation") ? (string)previous["generation"] : null; - if(!Generation(PreviousGeneration))throw new InvalidOperationException("Invalid predecessor generation; preserved"); + string previousGeneration=previous.ContainsKey("generation") ? (string)previous["generation"] : null; + if(!Generation(previousGeneration))throw new InvalidOperationException("Invalid predecessor generation; preserved"); if(previous.ContainsKey("deadGenerations")) { var prior=previous["deadGenerations"] as System.Collections.IList; if(prior==null)throw new InvalidOperationException("Invalid predecessor history; preserved"); foreach(object item in prior){string value=item as string;if(!Generation(value))throw new InvalidOperationException("Invalid predecessor history; preserved");deadGenerations.Add(value);} } - deadGenerations.Add(PreviousGeneration); + deadGenerations.Add(previousGeneration); if(deadGenerations.Count>256)throw new InvalidOperationException("Predecessor history requires maintenance; preserved"); } Write(new Dictionary{{"state","pending"},{"controllerPid",Process.GetCurrentProcess().Id}}); diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs index a2ea26c2613..c6175249359 100644 --- a/bin/native-owner/NativeLauncher.cs +++ b/bin/native-owner/NativeLauncher.cs @@ -32,7 +32,7 @@ static int Launch(string selectedHome) { ConsoleCancelEventHandler cancel=(sender,args)=>{args.Cancel=true;};Console.CancelKeyPress+=cancel; try { lease=new NativeHomeLease(home);operationJournal=new NativeReceiptJournal(lease,session); - int recovered=operationJournal.ReconcileCompletedAcknowledgements(); + operationJournal.ReconcileCompletedAcknowledgements(); string runtime=Path.Combine(home,"state","native-runtime",session);Directory.CreateDirectory(runtime); var security=new PipeSecurity();security.SetAccessRuleProtection(true,false); security.AddAccessRule(new PipeAccessRule(WindowsIdentity.GetCurrent().User,PipeAccessRights.FullControl,AccessControlType.Allow)); @@ -43,8 +43,6 @@ static int Launch(string selectedHome) { controlServer=new NamedPipeServerStream(controlName,PipeDirection.InOut,1,PipeTransmissionMode.Byte,PipeOptions.Asynchronous,4096,4096,security); var values=EnvironmentFor(controlName,session,runtime,nonce); values["FM_HOME"]=home;values["FM_PROBE_CODE_ROOT"]=CodeRoot; - values["FM_PROBE_LEASE_HOME"]=home;values["FM_PROBE_LEASE_GENERATION"]=session; - values["FM_PROBE_RECOVERED"]=recovered.ToString(); values["FM_PROBE_JQ_IMAGE"]=Environment.GetEnvironmentVariable("FM_NATIVE_JQ_IMAGE")??""; values["FM_PROBE_VERIFY_ONLY"]=Environment.GetEnvironmentVariable("FM_NATIVE_VERIFY_ONLY")??""; values["MSYS"]="winsymlinks:nativestrict"; @@ -139,11 +137,12 @@ static int FixedOperation(string purpose) { } public static int Main(string[] args) { try { + int ownerResult; if(args.Length==3&&args[0]=="launch"&&args[1]=="--experimental")return Launch(args[2]); - if(args.Length>=3&&args[0]=="owner")return OwnerClient(args[1],args[2],args.Length>3?args[3]:""); + if(TryOwnerCommand(args,out ownerResult))return ownerResult; if(args.Length==1&&args[0]=="owner-operation")return FixedOperation("startup"); if(args.Length==2&&args[0]=="notification-operation")return FixedOperation(args[1]); - throw new ArgumentException("Use fm-native-codex.ps1 -Experimental -Home "); + throw new ArgumentException("Use fm-native-codex.ps1 -Experimental -OperationalHome "); } catch(Exception error){Console.Error.WriteLine(error.Message);return 2;} } } diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index 133d2db822b..dd605cf2407 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -18,8 +18,8 @@ static SortedDictionary EnvironmentFor(string pipe, string sessio var result = new SortedDictionary(StringComparer.OrdinalIgnoreCase); foreach (DictionaryEntry e in Environment.GetEnvironmentVariables()) { string k = (string)e.Key; - if (k=="NODE_OPTIONS" || k=="NODE_PATH" || k=="BASH_ENV" || k=="ENV") continue; - if (k.StartsWith("FM_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("PI_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("NO_MISTAKES", StringComparison.OrdinalIgnoreCase) || k == "CLAUDE_PID" || k == "CLAUDECODE") continue; + if (string.Equals(k,"NODE_OPTIONS",StringComparison.OrdinalIgnoreCase) || string.Equals(k,"NODE_PATH",StringComparison.OrdinalIgnoreCase) || string.Equals(k,"BASH_ENV",StringComparison.OrdinalIgnoreCase) || string.Equals(k,"ENV",StringComparison.OrdinalIgnoreCase)) continue; + if (k.StartsWith("FM_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("PI_", StringComparison.OrdinalIgnoreCase) || k.StartsWith("NO_MISTAKES", StringComparison.OrdinalIgnoreCase) || string.Equals(k,"CLAUDE_PID",StringComparison.OrdinalIgnoreCase) || string.Equals(k,"CLAUDECODE",StringComparison.OrdinalIgnoreCase)) continue; result[k] = (string)e.Value; } result["FM_PROBE_PIPE"] = pipe; result["FM_PROBE_SESSION"] = session; diff --git a/bin/native-owner/NativeOwner.cs b/bin/native-owner/NativeOwner.cs index 6e0468432a4..4c51f37166a 100644 --- a/bin/native-owner/NativeOwner.cs +++ b/bin/native-owner/NativeOwner.cs @@ -158,4 +158,14 @@ static int OwnerClient(string verb,string state,string id) { } finally { CloseHandle(server); } } } + static bool TryOwnerCommand(string[] args,out int result) { + result=0; + if(args.Length==3 && args[0]=="owner" && (args[1]=="identity" || args[1]=="owns" || args[1]=="harness")) { + result=OwnerClient(args[1],args[2],"");return true; + } + if(args.Length==4 && args[0]=="owner" && args[1]=="alive") { + result=OwnerClient(args[1],args[2],args[3]);return true; + } + return false; + } } diff --git a/bin/native-owner/app-server-policy.mjs b/bin/native-owner/app-server-policy.mjs new file mode 100644 index 00000000000..b6b4256c51d --- /dev/null +++ b/bin/native-owner/app-server-policy.mjs @@ -0,0 +1,67 @@ +import {spawn} from 'node:child_process'; + +function uniqueNames(value) { + if(!Array.isArray(value)||value.length>256||value.some(name=>typeof name!=='string'||!name.length||name.length>256))throw Error('Invalid inherited MCP server catalog'); + const names=[...new Set(value)]; + if(names.length!==value.length)throw Error('Duplicate inherited MCP server name'); + return names.sort(); +} + +export function discoverMcpServerNames(executable,cwd,env=process.env) { + return new Promise((resolve,reject)=>{ + const child=spawn(executable,['mcp','list','--json','--disable','apps','--disable','plugins'],{cwd,env,stdio:['ignore','pipe','pipe'],windowsHide:true}); + let stdout='',stderr='',settled=false; + const finish=(callback,value)=>{if(settled)return;settled=true;clearTimeout(timer);callback(value);}; + const fail=error=>{child.kill();finish(reject,error);}; + const timer=setTimeout(()=>fail(Error('MCP configuration discovery timed out')),30000); + child.stdout.on('data',data=>{stdout+=data;if(stdout.length>1048576)fail(Error('MCP configuration discovery exceeded its bound'));}); + child.stderr.on('data',data=>{stderr+=data;if(stderr.length>1048576)fail(Error('MCP configuration diagnostics exceeded their bound'));}); + child.on('error',fail); + child.on('exit',code=>{ + if(settled)return; + if(code!==0){finish(reject,Error('MCP configuration discovery failed: '+stderr.trim()));return;} + try { + const rows=JSON.parse(stdout); + if(!Array.isArray(rows)||rows.some(row=>!row||typeof row!=='object'))throw Error('Invalid MCP configuration response'); + finish(resolve,uniqueNames(rows.map(row=>row.name))); + }catch(error){finish(reject,error);} + }); + }); +} + +export function isolatedAppServerArgs(mcpServerNames,extra=[]) { + const names=uniqueNames(mcpServerNames); + const disabled=names.length?['-c','mcp_servers={'+names.map(name=>JSON.stringify(name)+'={enabled=false}').join(',')+'}']:[]; + return ['app-server','--stdio','--disable','hooks','--disable','apps','--disable','plugins',...disabled,...extra]; +} + +export async function verifyExternalToolConfiguration(request,mcpServerNames) { + const expected=uniqueNames(mcpServerNames); + const effective=(await request('config/read',{includeLayers:false})).config??{}; + const configured=effective.mcp_servers&&typeof effective.mcp_servers==='object'?effective.mcp_servers:{}; + const configuredNames=Object.keys(configured).sort(); + const enabledMcpServers=configuredNames.filter(name=>configured[name]?.enabled!==false); + const result={ + appsFeatureEnabled:effective.features?.apps, + pluginsFeatureEnabled:effective.features?.plugins, + configuredMcpServers:configuredNames, + enabledMcpServers, + }; + if(result.appsFeatureEnabled!==false||result.pluginsFeatureEnabled!==false||JSON.stringify(configuredNames)!==JSON.stringify(expected)||enabledMcpServers.length)throw Error('External app-server configuration is not isolated: '+JSON.stringify(result)); + return result; +} + +export async function verifyExternalToolIsolation(request,threadId,configuration) { + const installed=await request('app/installed',{threadId,forceRefresh:false}); + const servers=await request('mcpServerStatus/list',{cursor:null,limit:100,detail:'toolsAndAuthOnly'}); + const apps=Array.isArray(installed.apps)?installed.apps:[]; + const mcpServers=Array.isArray(servers.data)?servers.data:[]; + const activeMcpServers=mcpServers.filter(server=>server.runtimeStatus!==null||server.serverInfo!==null||server.toolsError!==null||Object.keys(server.tools??{}).length||(server.resources??[]).length||(server.resourceTemplates??[]).length); + const result={ + ...configuration, + exposedApps:apps.map(app=>app.id??app.name??'unknown'), + activeMcpServers:activeMcpServers.map(server=>server.name??server.id??'unknown'), + }; + if(result.exposedApps.length||result.activeMcpServers.length)throw Error('External app-server tools are not isolated: '+JSON.stringify(result)); + return result; +} diff --git a/bin/native-owner/codex-host.mjs b/bin/native-owner/codex-host.mjs index 8566173dcc2..c53966ebed2 100644 --- a/bin/native-owner/codex-host.mjs +++ b/bin/native-owner/codex-host.mjs @@ -8,6 +8,7 @@ import {spawn} from 'node:child_process'; import {createInterface} from 'node:readline'; import {createNotificationGate} from './codex-tool-gate.mjs'; import {createHostLifecycle} from './host-lifecycle.mjs'; +import {discoverMcpServerNames,isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from './app-server-policy.mjs'; const runtime=process.env.FM_PROBE_HOME,root=process.env.FM_PROBE_CODE_ROOT; const pause=ms=>new Promise(resolve=>setTimeout(resolve,ms)); let closing=false,gate=null,server=null,alive=false,thread=null,activeTurn=null,ended=false; @@ -134,7 +135,8 @@ try { } if(!closing){ const executable=path.join(process.env.APPDATA,'npm/node_modules/@openai/codex/node_modules/@openai/codex-win32-x64/vendor/x86_64-pc-windows-msvc/bin/codex.exe'); - server=spawn(executable,['app-server','--stdio','--disable','hooks','-c','windows.sandbox=unelevated'],{cwd:root,stdio:['pipe','pipe','pipe'],detached:true,windowsHide:true});alive=true; + const mcpServerNames=await discoverMcpServerNames(executable,root); + server=spawn(executable,isolatedAppServerArgs(mcpServerNames,['-c','windows.sandbox=unelevated']),{cwd:root,stdio:['pipe','pipe','pipe'],detached:true,windowsHide:true});alive=true; server.stderr.on('data',data=>fs.appendFileSync(path.join(runtime,'app-server.stderr'),data)); const lost=error=>{alive=false;for(const waiter of pending.values())waiter.reject(error);pending.clear();if(!closing)fail(error);}; server.on('error',lost);server.on('exit',()=>lost(Error('App-server exited')));server.stdin.on('error',error=>{if(!closing)fail(error);}); @@ -158,9 +160,10 @@ try { {name:'fm_notification_ack',description:'Acknowledge an observed and handled delivery. Never acknowledge unresolved decisions or unperformed work.',inputSchema:{type:'object',properties:{receipt:{type:'string'},observed:{type:'string'}},required:['receipt','observed'],additionalProperties:false}}, ]; const instructions='The native host already ran startup exactly once. Do not rerun startup or arm another supervisor. This experimental empty-fleet session supports only the two notification tools; do not claim to dispatch project work. The host continues notification checks after startup finishes. Use the supplied receipt and observed challenge only after handling the entire delivery. Unresolved work must remain pending. Startup digest follows:\n'+fs.readFileSync(path.join(runtime,'startup.log'),'utf8'); + const externalConfiguration=await verifyExternalToolConfiguration(request,mcpServerNames); const started=await request('thread/start',{cwd:root,sandbox:'read-only',approvalPolicy:'never',ephemeral:true,developerInstructions:instructions,dynamicTools}); if(started.sandbox?.type!=='readOnly'||started.sandbox.networkAccess!==false||started.approvalPolicy!=='never')throw Error('App-server returned an unexpected security policy'); - thread=started.thread.id;evidence.thread=thread;evidence.ready=true;evidence.policy={sandbox:started.sandbox,approval:started.approvalPolicy};evidence.digestDeliveredBeforeDeferred=!fs.existsSync(path.join(runtime,'startup.finished'));save(); + thread=started.thread.id;evidence.thread=thread;evidence.externalTools=await verifyExternalToolIsolation(request,thread,externalConfiguration);evidence.ready=true;evidence.policy={sandbox:started.sandbox,approval:started.approvalPolicy};evidence.digestDeliveredBeforeDeferred=!fs.existsSync(path.join(runtime,'startup.finished'));save(); gate=createNotificationGate({primaryThread:thread,operate:operation,isAlive:()=>alive&&!closing}); console.error('Experimental native session ready. /interrupt stops the current turn; /quit ends the session.'); while(!closing&&alive){ diff --git a/docs/documentation-audiences.json b/docs/documentation-audiences.json index 618caa941ec..b4927ddc11d 100644 --- a/docs/documentation-audiences.json +++ b/docs/documentation-audiences.json @@ -102,6 +102,10 @@ "source": "docs/codex-app-backend.md", "target": "docs/verification/runtime-backends.md" }, + { + "source": "docs/native-windows-codex.md", + "target": "docs/verification/runtime-backends.md" + }, { "source": "docs/trace-context.md", "target": "docs/verification/trace-context.md" @@ -368,6 +372,10 @@ "path": "docs/herdr-backend.md", "audience": "operator-current" }, + { + "path": "docs/native-windows-codex.md", + "audience": "operator-current" + }, { "path": "docs/orca-backend.md", "audience": "operator-current" diff --git a/docs/native-windows-codex.md b/docs/native-windows-codex.md new file mode 100644 index 00000000000..5fe5e890ffc --- /dev/null +++ b/docs/native-windows-codex.md @@ -0,0 +1,37 @@ +# Experimental native Windows Codex launcher + +Audience: operators. + +This explicit opt-in launcher is a restricted experimental candidate, not an installed runtime backend or a production-ready integration. +Ordinary startup never selects it, and it does not install hooks, alter saved or global Codex settings, pull images, or select a default backend. +[`verification/runtime-backends.md`](verification/runtime-backends.md#experimental-native-windows-ownership-candidate) records the dated empirical evidence and refresh commands. + +## Setup + +The launcher requires Windows PowerShell 5.1, native Node and Codex, Git Bash, and Docker at their standard installation paths. +It also requires an existing local Docker image containing jq and GNU timeout; the launcher does not install or pull it. +The PowerShell help for `bin/fm-native-codex.ps1` owns the exact flags and prerequisites. + +Build without launching, then verify an empty temporary operational home: + +```powershell +.\bin\fm-native-codex.ps1 -BuildOnly +.\bin\fm-native-codex.ps1 -Experimental -VerifyOnly ` + -OperationalHome "$env:LOCALAPPDATA\Temp\firstmate-native-example" -JqImage +``` + +Omit `-VerifyOnly` for an interactive model session. +Rebuild the native provider after its source stamp changes and after all native sessions have stopped. +Use `/interrupt` to interrupt the current model turn and `/quit` to end the session. + +## Safety boundary and limits + +Only empty-fleet homes beneath the current user's Windows temporary directory are accepted. +Existing fleet metadata, projects, registrations, Relay configuration, process-event sources, non-temporary homes, and existing reparse-point ancestors are refused. +The app-server thread is ephemeral, read-only, network-disabled, and approval-never; Apps, plugins, and configured MCP servers are disabled for this host and their effective catalogs are checked before readiness. +Only controller-selected startup, notification check, and acknowledgement scripts receive registered native operation authority. +Interrupted or ambiguous acknowledgements remain preserved for evidence-based reconciliation and are never replayed or rolled back automatically. + +Production use remains disabled. +Populated-fleet shutdown, forced app-server descendant cleanup, adversarial Windows path and process races, remaining numeric identity readers, other harnesses, installation, and packaging are not supported or claimed. +Ordinary unelevated Codex shell execution of Git Bash remains a known limitation; the narrow authenticated host operations are not a general sandbox fix. diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index ba88beaba72..ebb9533f606 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -9,15 +9,7 @@ Exact task chronology, branch names, temporary homes, local paths, process ids, ## Experimental native Windows ownership candidate Verified on 2026-09-15 and 2026-09-16 with Codex app-server 0.154.0, Windows 10.0.26200 x86_64, and the saved unelevated Windows sandbox. -This is an isolated candidate verification, not an installed runtime backend or a claim that ordinary Codex shell tools can run Git Bash in that sandbox. -The native controller, interactive app-server host, fixed operation scripts, and request policy live under `bin/native-owner/`. -`bin/fm-native-codex.ps1` owns explicit launch and local compilation; its PowerShell help documents prerequisites and flags. -It does not install hooks, alter saved settings, pull images, or select a new default backend. -The home reservation rejects locations outside the user's Windows temporary directory and existing reparse-point ancestors. -Existing fleet metadata, projects, registrations, Relay configuration, and process-event sources are refused. -Those checks retain the experimental boundary rather than proving race-free paths or populated-fleet support. -Ordinary homes do not opt in; the real session-lock, harness, and startup-nudge readers select the native provider only from native home records. -The test-only consumer patch has been removed. +[`../native-windows-codex.md`](../native-windows-codex.md) owns current setup, safety boundaries, and supported limits for this isolated candidate. Refresh the portable request-policy regression with: @@ -35,6 +27,19 @@ The guard also retains the exclusive-owner, normal-exit, controller-loss, orphan ```sh bash tests/fm-native-owner-receipt-live-e2e.test.sh ``` + +Refresh effective app and MCP isolation without a model turn: + +```sh +FM_LIVE_NATIVE_APP_POLICY=1 bash tests/fm-native-owner-app-server-policy-live-e2e.test.sh +``` + +Observed terminal result: + +```text +PASS: effective app and MCP catalogs are isolated {"appsFeatureEnabled":false,"pluginsFeatureEnabled":false,"configuredMcpServers":["inherited_probe"],"enabledMcpServers":[],"exposedApps":[],"activeMcpServers":[]} +``` + Refresh the actual Windows integration with the explicit two-model-turn guard: ```sh @@ -73,18 +78,6 @@ The launch-bound owner record preserves verified-dead predecessor generations ac That history is bounded and refuses further acquisition rather than silently forgetting evidence. An unresolved acknowledgement prevents more work and identifies the preserved journal for review; no partial attempt is retried or rolled back automatically. -Build without launching, then explicitly test an empty temporary home from Windows PowerShell: - -```powershell -.\bin\fm-native-codex.ps1 -BuildOnly -.\bin\fm-native-codex.ps1 -Experimental -VerifyOnly ` - -Home "$env:LOCALAPPDATA\Temp\firstmate-native-example" -JqImage -``` - -Omit `-VerifyOnly` for an interactive model session. -The local image must contain jq and GNU timeout; this command does not install them. -The native provider must be rebuilt when its source stamp changes. - Refresh the actual launcher without model turns, then optionally exercise two notification turns and active-turn cancellation: ```sh @@ -102,11 +95,6 @@ A separate shutdown interrupted an active model turn, confirmed app-server exit, The production launcher also refused a real partially acknowledged journal from the fault-injection fixture and surfaced the reconciliation requirement without starting the model. Fixture delays and controlled notifications remain under `tests/fixtures/native-owner/`; the launcher contains no test message generator. -Production use remains disabled. -These results do not establish populated-fleet shutdown, forced app-server descendant cleanup, adversarial Windows path/process races, PID reuse and peer-handle races, integration of every numeric identity reader, other harness support, or installation and packaging. -The shared-reader rollout remains deliberately limited to the explicit empty-fleet launcher; Pi, OMP, OpenCode, and fleet lease consumers are not claimed supported. -Full no-mistakes validation has not run, and no publication or merge is authorized by these targeted tests. - ## Harness detection precedence Firstmate's own harness comes from two kinds of evidence, and `bin/fm-harness.sh` owns how they combine: an environment marker names its harness, and the nearest harness process in the parent chain proves who owns the process tree. diff --git a/tests/fixtures/native-owner/AppHost.mjs b/tests/fixtures/native-owner/AppHost.mjs index aeb2956be42..02b5e57f536 100644 --- a/tests/fixtures/native-owner/AppHost.mjs +++ b/tests/fixtures/native-owner/AppHost.mjs @@ -2,6 +2,7 @@ // Dynamic tool arguments never select a thread, executable, home, or command. import {createNotificationGate} from './codex-tool-gate.mjs'; import {createHostLifecycle} from './host-lifecycle.mjs'; +import {discoverMcpServerNames,isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from './app-server-policy.mjs'; import fs from 'node:fs'; import path from 'node:path'; import net from 'node:net'; @@ -48,7 +49,8 @@ async function operation(action,extra={}) { } throw Error('Bounded operation exceeded 70 seconds'); } -const child=spawn(executable,['app-server','--stdio','--disable','hooks','-c','windows.sandbox=unelevated','-c','model_reasoning_effort=high'],{cwd:home,stdio:['pipe','pipe','pipe']}); +const mcpServerNames=await discoverMcpServerNames(executable,home); +const child=spawn(executable,isolatedAppServerArgs(mcpServerNames,['-c','windows.sandbox=unelevated','-c','model_reasoning_effort=high']),{cwd:home,stdio:['pipe','pipe','pipe']}); let alive=true,next=0,stderr='',primary=null,receipt=null,challenge=null,acknowledged=false; let gate=null,activeTurn=null,closing=false; const pending=new Map(),completed=new Map(); @@ -97,7 +99,9 @@ try { for(let i=0;i<1200;i++){const state=await native('result');if(state.startupExpired){ready=true;break;}await pause(100);} if(!ready||!fs.existsSync(path.join(home,'owner-operation.complete')))throw Error('Startup did not finish and expire'); const params={cwd:home,model:'gpt-5.6-terra',sandbox:'read-only',approvalPolicy:'never',ephemeral:true,dynamicTools:tools}; + const externalConfiguration=await verifyExternalToolConfiguration(request,mcpServerNames); primary=(await request('thread/start',params)).thread.id;evidence.primary=primary; + evidence.externalTools=await verifyExternalToolIsolation(request,primary,externalConfiguration); gate=createNotificationGate({primaryThread:primary,operate:operation,isAlive:()=>alive}); evidence.foreign=(await request('thread/start',params)).thread.id; if(process.env.FM_PROBE_API_DRY==='1') { diff --git a/tests/fixtures/native-owner/AppServerPolicy.mjs b/tests/fixtures/native-owner/AppServerPolicy.mjs new file mode 100644 index 00000000000..30c80a7a23a --- /dev/null +++ b/tests/fixtures/native-owner/AppServerPolicy.mjs @@ -0,0 +1,46 @@ +import path from 'node:path'; +import fs from 'node:fs'; +import {spawn} from 'node:child_process'; +import {createInterface} from 'node:readline'; +import {fileURLToPath} from 'node:url'; +import {discoverMcpServerNames,isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from '../../../bin/native-owner/app-server-policy.mjs'; + +const root=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..'); +const area=fs.mkdtempSync(path.join(process.env.TEMP,'fm-native-app-policy-')); +const codexHome=path.join(area,'codex-home'); +fs.mkdirSync(codexHome); +const probe=process.execPath.replaceAll('\\','/'); +fs.writeFileSync(path.join(codexHome,'config.toml'),`[features]\napps = true\nplugins = true\n[mcp_servers.inherited_probe]\ncommand = ${JSON.stringify(probe)}\nargs = ["-e", "process.exit(0)"]\nenabled = true\n`); +const executable=path.join(process.env.APPDATA,'npm/node_modules/@openai/codex/node_modules/@openai/codex-win32-x64/vendor/x86_64-pc-windows-msvc/bin/codex.exe'); +const childEnvironment={...process.env,CODEX_HOME:codexHome}; +const mcpServerNames=await discoverMcpServerNames(executable,root,childEnvironment); +const child=spawn(executable,isolatedAppServerArgs(mcpServerNames,['-c','windows.sandbox=unelevated']),{cwd:root,env:childEnvironment,stdio:['pipe','pipe','pipe'],windowsHide:true}); +let next=0,stderr=''; +const pending=new Map(); +child.stderr.on('data',data=>stderr+=data); +const fail=error=>{for(const waiter of pending.values())waiter.reject(error);pending.clear();}; +child.on('error',fail);child.on('exit',code=>fail(Error('App-server exited '+code+': '+stderr))); +createInterface({input:child.stdout}).on('line',line=>{ + const frame=JSON.parse(line); + if(frame.id!==undefined&&pending.has(frame.id)){ + const waiter=pending.get(frame.id);pending.delete(frame.id); + frame.error?waiter.reject(Error(JSON.stringify(frame.error))):waiter.resolve(frame.result); + }else if(frame.id!==undefined&&frame.method)child.stdin.write(JSON.stringify({id:frame.id,error:{code:-32601,message:'Unsupported test request'}})+'\n'); +}); +const request=(method,params)=>new Promise((resolve,reject)=>{ + const id=++next,timer=setTimeout(()=>{pending.delete(id);reject(Error(method+' timed out'));},30000); + pending.set(id,{resolve:value=>{clearTimeout(timer);resolve(value);},reject:error=>{clearTimeout(timer);reject(error);}}); + child.stdin.write(JSON.stringify({id,method,params})+'\n'); +}); + +try { + await request('initialize',{clientInfo:{name:'firstmate-native-policy-test',version:'0.1.0'},capabilities:{experimentalApi:true}}); + child.stdin.write(JSON.stringify({method:'initialized',params:{}})+'\n'); + const externalConfiguration=await verifyExternalToolConfiguration(request,mcpServerNames); + const started=await request('thread/start',{cwd:root,sandbox:'read-only',approvalPolicy:'never',ephemeral:true,dynamicTools:[]}); + const result=await verifyExternalToolIsolation(request,started.thread.id,externalConfiguration); + console.log('PASS: effective app and MCP catalogs are isolated '+JSON.stringify(result)); +}finally{ + child.stdin.end(); + await new Promise(resolve=>{const timer=setTimeout(()=>{child.kill();resolve();},5000);child.once('exit',()=>{clearTimeout(timer);resolve();});}); +} diff --git a/tests/fixtures/native-owner/Build.ps1 b/tests/fixtures/native-owner/Build.ps1 index d778056c6f0..0794457b2b6 100644 --- a/tests/fixtures/native-owner/Build.ps1 +++ b/tests/fixtures/native-owner/Build.ps1 @@ -12,6 +12,19 @@ $sources += @((Join-Path $repo 'bin/native-owner/NativeOperations.cs'), (Join-Pa Add-Type -Path $sources -OutputAssembly $binary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll & $binary receipt-tests if ($LASTEXITCODE -ne 0) { throw 'Durable receipt lifecycle tests failed' } +& $binary environment-tests +if ($LASTEXITCODE -ne 0) { throw 'Native environment filtering tests failed' } +$invalidState = Join-Path $root 'missing-state' +$savedErrorPreference = $ErrorActionPreference +$ErrorActionPreference = 'Continue' +try { + $invalidOwner = @(& $binary owner alive $invalidState 'native:00000000000000000000000000000000' trailing 2>&1) + if ($LASTEXITCODE -ne 2 -or ($invalidOwner -join "`n") -notmatch 'usage: NativeOwner') { throw 'Owner CLI accepted a trailing argument' } + $invalidOwner = @(& $binary owner harness $invalidState trailing 2>&1) + if ($LASTEXITCODE -ne 2 -or ($invalidOwner -join "`n") -notmatch 'usage: NativeOwner') { throw 'Owner CLI accepted an ID for a fixed-shape verb' } +} finally { + $ErrorActionPreference = $savedErrorPreference +} $copy = Join-Path $root 'firstmate' & git -c core.symlinks=true clone --quiet --no-local --single-branch $repo $copy if ($LASTEXITCODE -ne 0) { throw 'Disposable clone failed' } @@ -19,7 +32,7 @@ if ($LASTEXITCODE -ne 0) { throw 'Disposable clone failed' } foreach ($name in @('fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh')) { Copy-Item (Join-Path $repo ('bin/' + $name)) (Join-Path $copy ('bin/' + $name)) } foreach ($name in @('exercise.sh','notification-check.sh','notification-ack.sh')) { Copy-Item (Join-Path $PSScriptRoot $name) (Join-Path $copy $name) } Copy-Item (Join-Path $PSScriptRoot 'AppHost.mjs') (Join-Path $root 'AppHost.mjs') -foreach ($module in @('codex-tool-gate.mjs','host-lifecycle.mjs')) { Copy-Item (Join-Path $repo ('bin/native-owner/' + $module)) (Join-Path $root $module) } +foreach ($module in @('codex-tool-gate.mjs','host-lifecycle.mjs','app-server-policy.mjs')) { Copy-Item (Join-Path $repo ('bin/native-owner/' + $module)) (Join-Path $root $module) } Copy-Item $binary (Join-Path $copy 'bin/fm-native-owner.exe') New-Item -ItemType Directory (Join-Path $root 'tools') | Out-Null Copy-Item (Join-Path $PSScriptRoot 'jq') (Join-Path $root 'tools/jq') diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 15eda1fbfc3..13dd89f90bd 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -17,8 +17,10 @@ fs.cpSync(path.join(repo,'bin/native-owner'),path.join(code,'bin/native-owner'), for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); const launcher=path.join(code,'bin/fm-native-codex.ps1'); command('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly']); +const alias=spawnSync('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly','-Home',path.join(area,'alias')],{encoding:'utf8',timeout:120000}); +assert.notEqual(alias.status,0,'The removed -Home alias was still accepted'); function start(home,verify=true){ - const args=['-NoProfile','-File',launcher,'-Experimental','-Home',home,'-JqImage',image];if(verify)args.push('-VerifyOnly'); + const args=['-NoProfile','-File',launcher,'-Experimental','-OperationalHome',home,'-JqImage',image];if(verify)args.push('-VerifyOnly'); const child=spawn('powershell.exe',args,{stdio:['pipe','pipe','pipe']});let stdout='',stderr=''; child.stdout.on('data',data=>stdout+=data);child.stderr.on('data',data=>stderr+=data);child.stdin.on('error',()=>{}); const done=new Promise((resolve,reject)=>{child.on('error',reject);child.on('exit',exit=>resolve({exit,stdout,stderr}));}); diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index b3fdd37bc58..156e683a37d 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -81,6 +81,32 @@ static int Fixture() { if(Environment.GetEnvironmentVariable("FM_PROBE_EXERCISE")=="1") RunFirstmate("primary",true); return bash.ExitCode; } + static int EnvironmentTests() { + string directory=Path.Combine(Path.GetTempPath(),"fm-native-environment-"+Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(directory); + string payload=Path.Combine(directory,"payload.js"),main=Path.Combine(directory,"main.js"),injected=Path.Combine(directory,"injected"),result=Path.Combine(directory,"result.json"); + File.WriteAllText(payload,"require('fs').writeFileSync("+Json.Serialize(injected.Replace('\\','/'))+",'injected')"); + File.WriteAllText(main,"const fs=require('fs');const denied=['NODE_OPTIONS','NODE_PATH','BASH_ENV','ENV','CLAUDE_PID','CLAUDECODE'];const leaked=Object.keys(process.env).filter(k=>denied.includes(k.toUpperCase())||k.toUpperCase().startsWith('FM_')||k.toUpperCase().startsWith('PI_')||k.toUpperCase().startsWith('NO_MISTAKES'));fs.writeFileSync("+Json.Serialize(result.Replace('\\','/'))+",JSON.stringify(leaked));"); + var poison=new Dictionary(StringComparer.OrdinalIgnoreCase){{"node_options","--require=\""+payload.Replace('\\','/')+"\""},{"node_path",directory},{"bash_env",payload},{"env",payload},{"claude_pid","123"},{"claudecode","1"},{"fm_poison","1"},{"pi_poison","1"},{"no_mistakes_poison","1"}}; + var original=new Dictionary(StringComparer.OrdinalIgnoreCase); + try { + foreach(var entry in poison){original[entry.Key]=Environment.GetEnvironmentVariable(entry.Key);Environment.SetEnvironmentVariable(entry.Key,entry.Value);} + var values=EnvironmentFor("pipe","session",directory,"nonce"); + var info=new ProcessStartInfo(Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"),Quote(main)){UseShellExecute=false}; + info.EnvironmentVariables.Clear();foreach(var entry in values)info.EnvironmentVariables[entry.Key]=entry.Value; + using(var process=Process.Start(info)){ + if(!process.WaitForExit(10000))throw new TimeoutException("Environment test child exceeded its bound"); + if(process.ExitCode!=0)throw new InvalidOperationException("Environment test child failed"); + } + var leaked=Json.Deserialize(File.ReadAllText(result)); + if(File.Exists(injected)||leaked.Length!=5)throw new InvalidOperationException("Denied inherited environment reached the native host"); + foreach(string key in leaked)if(!key.StartsWith("FM_PROBE_",StringComparison.Ordinal))throw new InvalidOperationException("Unexpected inherited environment reached the native host"); + Console.WriteLine("PASS: inherited Windows environment denylist is case-insensitive"); + return 0; + } finally { + foreach(var entry in original)Environment.SetEnvironmentVariable(entry.Key,entry.Value); + } + } static int Run(string configPath) { var config = Json.Deserialize>(File.ReadAllText(configPath)); string home = Path.GetFullPath((string)config["home"]); @@ -120,7 +146,7 @@ static int Run(string configPath) { if(!values.ContainsKey("FM_PROBE_API_DRY") || (fault!="partial" && fault!="complete")) throw new ArgumentException("Fault injection is limited to the model-free fixture"); values["FM_PROBE_ACK_FAULT"]=fault; } - if(lease!=null) { values["FM_PROBE_LEASE_HOME"]=lease.Home; values["FM_PROBE_LEASE_GENERATION"]=session; values["FM_HOME"]=lease.Home.Replace('\\','/'); } + if(lease!=null) values["FM_HOME"]=lease.Home.Replace('\\','/'); if(config.ContainsKey("ownerExercise") && (bool)config["ownerExercise"]) values["FM_PROBE_EXERCISE"]="1"; if(config.ContainsKey("boundaries") && (bool)config["boundaries"]) values["FM_PROBE_BOUNDARIES"]="1"; var block = new StringBuilder(); foreach (var e in values) block.Append(e.Key).Append('=').Append(e.Value).Append('\0'); block.Append('\0'); @@ -231,8 +257,10 @@ static int Run(string configPath) { } public static int Main(string[] args) { try { + int ownerResult; if(args.Length==1 && args[0]=="receipt-tests") { ReceiptTests.Run();return TestOperationLifetime(); } - if(args.Length>=3 && args[0]=="owner") return OwnerClient(args[1],args[2],args.Length>3 ? args[3] : ""); + if(args.Length==1 && args[0]=="environment-tests") return EnvironmentTests(); + if(TryOwnerCommand(args,out ownerResult)) return ownerResult; if(args.Length>0 && args[0]=="client") return Client(args.Length>1 ? args[1] : "agent-tool"); if(args.Length==2 && args[0]=="sleep") { int ms=int.Parse(args[1]); if(ms<0 || ms>15000) throw new ArgumentException("Sleep must be bounded"); Thread.Sleep(ms); return 0; } if(args.Length==3 && args[0]=="lease-check") { Console.WriteLine(NativeHomeLease.Check(args[1],args[2])); return 0; } diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index 2d749c97055..4f72ac97459 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -14,6 +14,7 @@ const build=read(path.join(home,'build.json')); const host=read(path.join(home,'app-host-evidence.json')); const native=read(path.join(home,'result.json')); assert.equal(latest.exit,0);assert.equal(native.rootExit,0);assert.equal(host.passed,true); +assert.deepEqual(host.externalTools,{appsFeatureEnabled:false,pluginsFeatureEnabled:false,enabledApps:[],configuredMcpServers:[],activeMcpServers:[]}); assert.deepEqual(host.shutdown,{stopped:true,operationsStopped:true,exited:true,forced:false,errors:[]}); assert.notEqual(host.primary,host.foreign);assert.equal(host.tools.length,4); const requests=host.frames.filter(frame=>frame.method==='item/tool/call'); diff --git a/tests/fm-native-owner-app-server-policy-live-e2e.test.sh b/tests/fm-native-owner-app-server-policy-live-e2e.test.sh new file mode 100644 index 00000000000..11cf1260e02 --- /dev/null +++ b/tests/fm-native-owner-app-server-policy-live-e2e.test.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# Token-free real app-server effective-catalog guard for the native host policy. +set -eu +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +fm_live_gate default-on FM_LIVE_NATIVE_APP_POLICY node codex +case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) ;; *) printf 'Native app-server policy tests require Windows\n' >&2; exit 1 ;; esac +node "$ROOT/tests/fixtures/native-owner/AppServerPolicy.mjs" diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index d609e6d6811..83e11dfc66c 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -583,6 +583,12 @@ test_native_state_contract() ( rc=0; fm_native_owner_state "$identity" || rc=$? [ "$rc" -eq "$answer" ] || fail "native three-state result collapsed" rc=0; fm_harness_pid_alive "$identity" || rc=$? + if [ "$answer" -eq 0 ]; then + [ "$rc" -eq 0 ] || fail "proven-live native owner lost positive health" + else + [ "$rc" -ne 0 ] || fail "dead or unknown native owner reported positive health" + fi + rc=0; fm_harness_pid_excludes "$identity" || rc=$? if [ "$answer" -eq 1 ]; then [ "$rc" -ne 0 ] || fail "proven-dead owner retained occupancy" else @@ -598,7 +604,54 @@ test_native_state_contract() ( pass "native identity routing preserves unknown exclusion without certifying health" ) +test_native_status_and_acquisition_behavior() { + local dir bindir fakebin identity out rc + dir="$TMP_ROOT/native-status" + bindir="$dir/bin" + fakebin="$dir/fakebin" + identity=native:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + mkdir -p "$bindir" "$fakebin" "$dir/state" + cp "$ROOT/bin/fm-lock.sh" "$ROOT/bin/fm-session-lock-lib.sh" "$ROOT/bin/fm-wake-lib.sh" "$bindir/" + cat > "$bindir/fm-native-owner.exe" <<'SH' +#!/usr/bin/env bash +case "${2:-}" in + alive) exit "${FM_TEST_NATIVE_STATE:?}" ;; + identity) printf '%s\n' 'native:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'; exit 0 ;; + *) exit 2 ;; +esac +SH + cat > "$fakebin/cygpath" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "${@: -1}" +SH + cat > "$fakebin/ps" <<'SH' +#!/usr/bin/env bash +case "$*" in + *comm=*) printf '%s\n' codex ;; + *args=*) printf '%s\n' codex ;; + *ppid=*) printf '%s\n' 1 ;; + *) exit 1 ;; +esac +SH + chmod +x "$bindir/fm-native-owner.exe" "$bindir/fm-lock.sh" "$fakebin/cygpath" "$fakebin/ps" + printf '%s\n' "$identity" > "$dir/state/.lock" + out=$(PATH="$fakebin:$PATH" FM_HOME="$dir" FM_STATE_OVERRIDE="$dir/state" FM_TEST_NATIVE_STATE=2 "$bindir/fm-lock.sh" status) + case "$out" in + *'held by native owner with unconfirmed health'*) ;; + *) fail "unknown native owner status was not distinguished: $out" ;; + esac + case "$out" in *'held by live harness'*) fail "unknown native owner status reported positive health" ;; esac + set +e + out=$(PATH="$fakebin:$PATH" FM_HOME="$dir" FM_STATE_OVERRIDE="$dir/state" FM_TEST_NATIVE_STATE=2 "$bindir/fm-lock.sh" 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "unknown native owner did not exclude acquisition" + [ "$(cat "$dir/state/.lock")" = "$identity" ] || fail "unknown native owner was overwritten during acquisition" + pass "native lock status distinguishes unknown health while acquisition remains excluded" +} + test_native_state_contract +test_native_status_and_acquisition_behavior test_version_named_session_is_identified_on_both_platforms test_harness_at_namespace_pid1_is_examined test_ordinary_paths_are_never_harness_processes From 1be41519ef72438cd0eedb31de12b91d1d8c992f Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 15:30:00 +1200 Subject: [PATCH 10/61] no-mistakes(review): Harden native launcher isolation and regression coverage --- bin/fm-native-codex.ps1 | 10 +-- bin/fm-session-lock-lib.sh | 14 +++-- bin/fm-test-run.sh | 3 +- bin/native-owner/NativeLauncher.cs | 2 +- docs/native-windows-codex.md | 4 +- tests/fixtures/native-owner/Launcher.mjs | 15 +++-- tests/fixtures/native-owner/Verify-Cycle.mjs | 5 +- .../native-owner/app-server-policy.test.mjs | 61 +++++++++++++++++++ tests/fm-live-gate.test.sh | 29 +++++++++ ...e-owner-app-server-policy-live-e2e.test.sh | 12 +++- tests/fm-native-owner-tool-gate.test.sh | 4 +- tests/fm-session-lock-ancestry.test.sh | 19 ++++++ tests/fm-test-run.test.sh | 13 ++++ 13 files changed, 164 insertions(+), 27 deletions(-) create mode 100644 tests/fixtures/native-owner/app-server-policy.test.mjs diff --git a/bin/fm-native-codex.ps1 b/bin/fm-native-codex.ps1 index 5ec6255dc62..31157400cb0 100644 --- a/bin/fm-native-codex.ps1 +++ b/bin/fm-native-codex.ps1 @@ -5,14 +5,8 @@ .SYNOPSIS Build or explicitly launch the experimental native Windows Codex host. .DESCRIPTION -Requires Windows PowerShell 5.1, native Node and Codex, Git Bash, and Docker at - their standard installation paths; Codex app-server 0.154.0 was verified. -Only empty-fleet homes beneath the user's Windows temporary directory are -accepted. Existing projects, fleet registrations, Relay and process sources -are not supported. Ordinary startup never selects this launcher. -No hooks, global settings, sandbox settings, packages, or Docker images are -installed or changed. The two fixed notification operations execute outside -the model's read-only, network-disabled sandbox under native authorization. +Builds the native provider or launches it with explicit experimental opt-in. +See docs/native-windows-codex.md for setup, safety boundaries, and supported limits. .PARAMETER Experimental Required consent to launch this temporary-home-only candidate. .PARAMETER BuildOnly diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index be459467846..fa7a31ef271 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -187,12 +187,14 @@ fm_win_untag_pid() { # return 1 } -# True when $1 is a well-formed session-lock identity: a local pid, or a tagged -# Windows pid. Every reader of state/.lock decides "is this value usable at all" -# through this one predicate, because the readers are spread across several -# scripts and a private numeric test in any one of them silently rejects a valid -# holder - which reads as "startup never completed" and repeats the whole -# sequence on every clear or compact. +# True when $1 is a well-formed session-lock identity: a local pid, a tagged +# Windows pid, or an explicitly registered native generation. This validates a +# lock identity, not a process-id argument; PID-only interfaces keep their own +# numeric validation. Every reader of state/.lock decides "is this value usable +# at all" through this one predicate, because the readers are spread across +# several scripts and a private numeric test in any one of them silently rejects +# a valid holder - which reads as "startup never completed" and repeats the +# whole sequence on every clear or compact. fm_session_pid_valid() { # local native_id case "$1" in diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 1d3fb108ba3..757fee9551d 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -342,7 +342,7 @@ family_for_basename() { fm-claude-stop-autoarm-live-e2e.test.sh|\ fm-cmux-claude-composer-live-e2e.test.sh|\ fm-composer-matrix-live-e2e.test.sh|\ - fm-codex-continuity-live-e2e.test.sh|fm-native-owner-codex-live-e2e.test.sh|fm-native-owner-receipt-live-e2e.test.sh|fm-native-owner-launcher-live-e2e.test.sh|fm-grok-continuity-live-e2e.test.sh|\ + fm-codex-continuity-live-e2e.test.sh|fm-native-owner-app-server-policy-live-e2e.test.sh|fm-native-owner-codex-live-e2e.test.sh|fm-native-owner-receipt-live-e2e.test.sh|fm-native-owner-launcher-live-e2e.test.sh|fm-grok-continuity-live-e2e.test.sh|\ fm-cursor-primary-live-e2e.test.sh|\ fm-grok-stop-live-e2e.test.sh|fm-harness-adapter-instructions-live-e2e.test.sh|\ fm-harness-liveness-drift-live-e2e.test.sh|\ @@ -1361,6 +1361,7 @@ families_for_changed_path() { ;; bin/native-owner/*|bin/fm-native-codex.ps1|tests/fixtures/native-owner/*) printf '%s\n' __script__:fm-native-owner-tool-gate.test.sh + printf '%s\n' __script__:fm-native-owner-app-server-policy-live-e2e.test.sh printf '%s\n' __script__:fm-native-owner-receipt-live-e2e.test.sh printf '%s\n' __script__:fm-native-owner-codex-live-e2e.test.sh printf '%s\n' __script__:fm-native-owner-launcher-live-e2e.test.sh diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs index c6175249359..c2c61a07902 100644 --- a/bin/native-owner/NativeLauncher.cs +++ b/bin/native-owner/NativeLauncher.cs @@ -15,7 +15,7 @@ public static partial class NativeOwner { static string CodeRoot { get { return Path.GetDirectoryName(Path.GetDirectoryName(OwnExe)); } } static void EmptyFleet(string home) { string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); - if((Directory.Exists(state)&&Directory.GetFiles(state,"*.meta").Length!=0) || (Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); + if((Directory.Exists(state)&&Directory.GetFiles(state,"*.meta").Length!=0) || (Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || File.Exists(Path.Combine(state,"x-watch.check.sh")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); } static int Launch(string selectedHome) { string home=Path.GetFullPath(selectedHome), node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); diff --git a/docs/native-windows-codex.md b/docs/native-windows-codex.md index 5fe5e890ffc..596997b1607 100644 --- a/docs/native-windows-codex.md +++ b/docs/native-windows-codex.md @@ -1,7 +1,5 @@ # Experimental native Windows Codex launcher -Audience: operators. - This explicit opt-in launcher is a restricted experimental candidate, not an installed runtime backend or a production-ready integration. Ordinary startup never selects it, and it does not install hooks, alter saved or global Codex settings, pull images, or select a default backend. [`verification/runtime-backends.md`](verification/runtime-backends.md#experimental-native-windows-ownership-candidate) records the dated empirical evidence and refresh commands. @@ -10,7 +8,7 @@ Ordinary startup never selects it, and it does not install hooks, alter saved or The launcher requires Windows PowerShell 5.1, native Node and Codex, Git Bash, and Docker at their standard installation paths. It also requires an existing local Docker image containing jq and GNU timeout; the launcher does not install or pull it. -The PowerShell help for `bin/fm-native-codex.ps1` owns the exact flags and prerequisites. +The PowerShell help for `bin/fm-native-codex.ps1` owns the exact flag mechanics. Build without launching, then verify an empty temporary operational home: diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 13dd89f90bd..1e1c2cd6590 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -17,8 +17,8 @@ fs.cpSync(path.join(repo,'bin/native-owner'),path.join(code,'bin/native-owner'), for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); const launcher=path.join(code,'bin/fm-native-codex.ps1'); command('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly']); -const alias=spawnSync('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly','-Home',path.join(area,'alias')],{encoding:'utf8',timeout:120000}); -assert.notEqual(alias.status,0,'The removed -Home alias was still accepted'); +const removedAlias=spawnSync('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly','-Home',path.join(area,'alias')],{encoding:'utf8',timeout:120000}); +assert.notEqual(removedAlias.status,0,'The removed -Home alias was still accepted'); function start(home,verify=true){ const args=['-NoProfile','-File',launcher,'-Experimental','-OperationalHome',home,'-JqImage',image];if(verify)args.push('-VerifyOnly'); const child=spawn('powershell.exe',args,{stdio:['pipe','pipe','pipe']});let stdout='',stderr=''; @@ -66,11 +66,18 @@ const populated=path.join(area,'populated');fs.mkdirSync(path.join(populated,'st const blocked=start(populated);blocked.child.stdin.end();assert.notEqual((await bound(blocked.done,blocked,20000)).exit,0); assert.equal(fs.readFileSync(path.join(populated,'state/work.meta'),'utf8'),'preserve');assert.equal(fs.existsSync(path.join(populated,'owner-probe.json')),false); records.push('populated home refused without changing its records'); +for(const [name,relative] of [['relay-config','config/x-mode.env'],['relay-watch','state/x-watch.check.sh']]){ + const relayHome=path.join(area,name),record=path.join(relayHome,relative),contents='preserve relay state'; + fs.mkdirSync(path.dirname(record),{recursive:true});fs.writeFileSync(record,contents); + const refusedRelay=start(relayHome);refusedRelay.child.stdin.end();assert.notEqual((await bound(refusedRelay.done,refusedRelay,20000)).exit,0); + assert.equal(fs.readFileSync(record,'utf8'),contents);assert.equal(fs.existsSync(path.join(relayHome,'owner-probe.json')),false); +} +records.push('generated Relay state refused before lease acquisition and preserved'); const outside=path.join(repo,'data/native-launcher',path.basename(area)+'-outside'); assert.equal(fs.existsSync(outside),false); const external=start(outside);external.child.stdin.end();assert.notEqual((await bound(external.done,external,20000)).exit,0);assert.equal(fs.existsSync(outside),false); -const target=path.join(area,'junction-target'),alias=path.join(area,'junction');fs.mkdirSync(target);fs.symlinkSync(target,alias,'junction'); -const linked=start(path.join(alias,'home'));linked.child.stdin.end();assert.notEqual((await bound(linked.done,linked,20000)).exit,0);assert.equal(fs.existsSync(path.join(target,'home')),false); +const target=path.join(area,'junction-target'),junction=path.join(area,'junction');fs.mkdirSync(target);fs.symlinkSync(target,junction,'junction'); +const linked=start(path.join(junction,'home'));linked.child.stdin.end();assert.notEqual((await bound(linked.done,linked,20000)).exit,0);assert.equal(fs.existsSync(path.join(target,'home')),false); records.push('non-temporary and pre-existing reparse-point homes refused before creation'); // Delay the existing network owner only in this disposable code copy. The // production launcher has no delay/mock switch and still invokes that owner. diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index 4f72ac97459..7332660cbe5 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -14,7 +14,10 @@ const build=read(path.join(home,'build.json')); const host=read(path.join(home,'app-host-evidence.json')); const native=read(path.join(home,'result.json')); assert.equal(latest.exit,0);assert.equal(native.rootExit,0);assert.equal(host.passed,true); -assert.deepEqual(host.externalTools,{appsFeatureEnabled:false,pluginsFeatureEnabled:false,enabledApps:[],configuredMcpServers:[],activeMcpServers:[]}); +assert.deepEqual(Object.keys(host.externalTools).sort(),['activeMcpServers','appsFeatureEnabled','configuredMcpServers','enabledMcpServers','exposedApps','pluginsFeatureEnabled']); +assert.equal(host.externalTools.appsFeatureEnabled,false);assert.equal(host.externalTools.pluginsFeatureEnabled,false); +assert.ok(Array.isArray(host.externalTools.configuredMcpServers));assert.deepEqual(host.externalTools.enabledMcpServers,[]); +assert.deepEqual(host.externalTools.exposedApps,[]);assert.deepEqual(host.externalTools.activeMcpServers,[]); assert.deepEqual(host.shutdown,{stopped:true,operationsStopped:true,exited:true,forced:false,errors:[]}); assert.notEqual(host.primary,host.foreign);assert.equal(host.tools.length,4); const requests=host.frames.filter(frame=>frame.method==='item/tool/call'); diff --git a/tests/fixtures/native-owner/app-server-policy.test.mjs b/tests/fixtures/native-owner/app-server-policy.test.mjs new file mode 100644 index 00000000000..4fd8b4e4e39 --- /dev/null +++ b/tests/fixtures/native-owner/app-server-policy.test.mjs @@ -0,0 +1,61 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from '../../../bin/native-owner/app-server-policy.mjs'; + +function requestBoundary(responses) { + const calls=[]; + const request=async(method,params)=>{ + calls.push({method,params}); + if(!(method in responses))throw Error('Unexpected request '+method); + return responses[method]; + }; + return {request,calls}; +} + +test('isolated app-server arguments disable every inherited external capability',()=>{ + assert.deepEqual(isolatedAppServerArgs(['zeta','alpha'],['-c','windows.sandbox=unelevated']),[ + 'app-server','--stdio','--disable','hooks','--disable','apps','--disable','plugins', + '-c','mcp_servers={"alpha"={enabled=false},"zeta"={enabled=false}}', + '-c','windows.sandbox=unelevated', + ]); +}); + +test('the fake request boundary observes a fully isolated effective catalog',async()=>{ + const boundary=requestBoundary({ + 'config/read':{config:{features:{apps:false,plugins:false},mcp_servers:{alpha:{enabled:false}}}}, + 'app/installed':{apps:[]}, + 'mcpServerStatus/list':{data:[]}, + }); + const configuration=await verifyExternalToolConfiguration(boundary.request,['alpha']); + const result=await verifyExternalToolIsolation(boundary.request,'thread-1',configuration); + assert.deepEqual(result,{ + appsFeatureEnabled:false,pluginsFeatureEnabled:false,configuredMcpServers:['alpha'],enabledMcpServers:[],exposedApps:[],activeMcpServers:[], + }); + assert.deepEqual(boundary.calls,[ + {method:'config/read',params:{includeLayers:false}}, + {method:'app/installed',params:{threadId:'thread-1',forceRefresh:false}}, + {method:'mcpServerStatus/list',params:{cursor:null,limit:100,detail:'toolsAndAuthOnly'}}, + ]); +}); + +test('enabled app features and MCP servers are rejected at the effective configuration boundary',async()=>{ + for(const config of [ + {features:{apps:true,plugins:false},mcp_servers:{alpha:{enabled:false}}}, + {features:{apps:false,plugins:true},mcp_servers:{alpha:{enabled:false}}}, + {features:{apps:false,plugins:false},mcp_servers:{alpha:{enabled:true}}}, + ]) { + const {request}=requestBoundary({'config/read':{config}}); + await assert.rejects(verifyExternalToolConfiguration(request,['alpha']),/External app-server configuration is not isolated/); + } +}); + +test('exposed apps and active MCP tools are rejected at the thread boundary',async()=>{ + const configuration={appsFeatureEnabled:false,pluginsFeatureEnabled:false,configuredMcpServers:['alpha'],enabledMcpServers:[]}; + for(const responses of [ + {'app/installed':{apps:[{id:'connected-app'}]},'mcpServerStatus/list':{data:[]}}, + {'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{name:'alpha',runtimeStatus:null,serverInfo:null,toolsError:null,tools:{write:{}},resources:[],resourceTemplates:[]}]}}, + ]) { + const {request}=requestBoundary(responses); + await assert.rejects(verifyExternalToolIsolation(request,'thread-1',configuration),/External app-server tools are not isolated/); + } +}); diff --git a/tests/fm-live-gate.test.sh b/tests/fm-live-gate.test.sh index c2e3b4e1ca1..e6a35ece3aa 100755 --- a/tests/fm-live-gate.test.sh +++ b/tests/fm-live-gate.test.sh @@ -199,6 +199,34 @@ EOF pass "all $checked live guards refuse together on FM_LIVE=0" } +test_native_app_policy_capability_skips_non_windows_by_default() { + local platform_bin result rc tool + platform_bin="$TMP_ROOT/non-windows-bin" + mkdir -p "$platform_bin" + for tool in node codex; do + printf '#!/usr/bin/env bash\nexit 0\n' > "$platform_bin/$tool" + chmod +x "$platform_bin/$tool" + done + printf '#!/usr/bin/env bash\nprintf "Linux\\n"\n' > "$platform_bin/uname" + chmod +x "$platform_bin/uname" + + set +e + result=$(clean_env PATH="$platform_bin:/usr/bin:/bin" bash "$ROOT/tests/fm-native-owner-app-server-policy-live-e2e.test.sh" 2>&1) + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "default native app-policy selection failed on an unsupported platform: $result" + assert_contains "$result" "skip: live: Windows required" "unsupported default selection must capability-skip" + + set +e + result=$(clean_env PATH="$platform_bin:/usr/bin:/bin" FM_LIVE_NATIVE_APP_POLICY=1 bash "$ROOT/tests/fm-native-owner-app-server-policy-live-e2e.test.sh" 2>&1) + rc=$? + set -e + [ "$rc" -eq 1 ] || fail "forced native app-policy selection did not refuse an unsupported platform: $result" + assert_contains "$result" "was requested but native app-server policy tests require Windows" "forced platform refusal must name the unsupported capability" + assert_not_contains "$result" "skip:" "a forced unsupported-platform run must not report a skip" + pass "native app policy skips unsupported platforms unless explicitly forced" +} + test_default_on_runs_when_the_tool_is_installed pass "a default-on guard runs wherever its tools are installed" test_default_on_skips_and_names_the_absent_tool @@ -218,3 +246,4 @@ pass "any entry point of a multi-mode guard turns it on" test_gate_lets_a_guard_drive_the_real_fleet_scripts_under_a_gate_marker pass "the shared gate carries the gate-refusal bypass into every live guard" test_every_live_guard_is_wired_to_the_shared_gate +test_native_app_policy_capability_skips_non_windows_by_default diff --git a/tests/fm-native-owner-app-server-policy-live-e2e.test.sh b/tests/fm-native-owner-app-server-policy-live-e2e.test.sh index 11cf1260e02..3c8ba5e7574 100644 --- a/tests/fm-native-owner-app-server-policy-live-e2e.test.sh +++ b/tests/fm-native-owner-app-server-policy-live-e2e.test.sh @@ -4,5 +4,15 @@ set -eu # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" fm_live_gate default-on FM_LIVE_NATIVE_APP_POLICY node codex -case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) ;; *) printf 'Native app-server policy tests require Windows\n' >&2; exit 1 ;; esac +case "$(uname -s)" in + MINGW*|MSYS*|CYGWIN*) ;; + *) + if [ "${FM_LIVE_NATIVE_APP_POLICY:-}" = 1 ] || [ "${FM_LIVE:-}" = 1 ]; then + printf 'not ok - FM_LIVE_NATIVE_APP_POLICY was requested but native app-server policy tests require Windows\n' >&2 + exit 1 + fi + printf 'skip: live: Windows required for native app-server policy tests\n' + exit 0 + ;; +esac node "$ROOT/tests/fixtures/native-owner/AppServerPolicy.mjs" diff --git a/tests/fm-native-owner-tool-gate.test.sh b/tests/fm-native-owner-tool-gate.test.sh index cd30e9413e9..433955587fd 100644 --- a/tests/fm-native-owner-tool-gate.test.sh +++ b/tests/fm-native-owner-tool-gate.test.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Portable behavioral tests of the host-side notification request policy. +# Portable behavioral tests of native host request and app-server policy. set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) -node --test "$ROOT/tests/fixtures/native-owner/tool-gate.test.mjs" "$ROOT/tests/fixtures/native-owner/host-lifecycle.test.mjs" +node --test "$ROOT/tests/fixtures/native-owner/tool-gate.test.mjs" "$ROOT/tests/fixtures/native-owner/host-lifecycle.test.mjs" "$ROOT/tests/fixtures/native-owner/app-server-policy.test.mjs" diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index 83e11dfc66c..bbafcaf5302 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -650,8 +650,27 @@ SH pass "native lock status distinguishes unknown health while acquisition remains excluded" } +test_numeric_ancestry_interfaces_reject_native_identity() { + local identity=native:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa out rc + set +e + out=$("$ROOT/bin/fm-harness.sh" ancestry "$identity" 2>&1) + rc=$? + set -e + [ "$rc" -eq 2 ] || fail "ancestry accepted a native lock identity as a process id: $out" + case "$out" in *'ancestry takes a numeric pid'*) ;; *) fail "ancestry refusal did not preserve its numeric contract: $out" ;; esac + + set +e + out=$("$ROOT/bin/fm-harness.sh" ancestry-descent 700 "$identity" 2>&1) + rc=$? + set -e + [ "$rc" -eq 2 ] || fail "ancestry-descent accepted a native lock identity as a process id: $out" + case "$out" in *'ancestry-descent takes numeric pids'*) ;; *) fail "ancestry-descent refusal did not preserve its numeric contract: $out" ;; esac + pass "numeric ancestry interfaces reject native lock identities" +} + test_native_state_contract test_native_status_and_acquisition_behavior +test_numeric_ancestry_interfaces_reject_native_identity test_version_named_session_is_identified_on_both_platforms test_harness_at_namespace_pid1_is_examined test_ordinary_paths_are_never_harness_processes diff --git a/tests/fm-test-run.test.sh b/tests/fm-test-run.test.sh index bb7c6b3c5fa..46c2e56cf30 100755 --- a/tests/fm-test-run.test.sh +++ b/tests/fm-test-run.test.sh @@ -105,6 +105,8 @@ init_changed_fixture_repo() { fm-daemon.test.sh \ fm-harness-adapter-instructions-live-e2e.test.sh \ fm-harness-adapter-references.test.sh \ + fm-native-owner-app-server-policy-live-e2e.test.sh \ + fm-native-owner-tool-gate.test.sh \ fm-backend-herdr-smoke.test.sh \ fm-secondmate-safety.test.sh \ fm-session-start.test.sh \ @@ -132,6 +134,8 @@ init_changed_fixture_repo() { : >"$repo/bin/fm-procevent-quota.sh" : >"$repo/bin/fm-quota-axi-lib.sh" : >"$repo/bin/fm-quota-choose.sh" + mkdir -p "$repo/bin/native-owner" + : >"$repo/bin/native-owner/app-server-policy.mjs" : >"$repo/bin/unmapped-source.sh" # A shared top-level test fixture read by two suites in different families, # beside a tests/ file nothing reads at all. @@ -412,6 +416,15 @@ test_changed_dependency_selection_and_unmapped_failure() { git -C "$repo" add bin/fm-timeout-lib.sh git -C "$repo" -c user.name=test -c user.email=test@example.invalid commit -qm timeout-lib-change + printf '\n' >>"$repo/bin/native-owner/app-server-policy.mjs" + listed=$(cd "$repo" && bin/fm-test-run.sh --list --changed --base HEAD) + assert_contains "$listed" "tests/fm-native-owner-tool-gate.test.sh" \ + "native owner policy selects portable behavioral coverage" + assert_contains "$listed" "tests/fm-native-owner-app-server-policy-live-e2e.test.sh" \ + "native owner policy selects the real app-server isolation guard" + git -C "$repo" add bin/native-owner/app-server-policy.mjs + git -C "$repo" -c user.name=test -c user.email=test@example.invalid commit -qm native-owner-policy-change + printf '\n' >>"$repo/src/unmapped.ts" set +e (cd "$repo" && bin/fm-test-run.sh --list --changed --base HEAD) >"$tmp/out" 2>"$tmp/err" From 99bb4532e361d0aab0b465f8cb944e6f65ab4ae8 Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 15:46:39 +1200 Subject: [PATCH 11/61] no-mistakes(review): Harden app policy pagination and response validation --- bin/native-owner/app-server-policy.mjs | 47 ++++++++++++-- .../native-owner/app-server-policy.test.mjs | 62 +++++++++++++++++-- 2 files changed, 97 insertions(+), 12 deletions(-) diff --git a/bin/native-owner/app-server-policy.mjs b/bin/native-owner/app-server-policy.mjs index b6b4256c51d..3a09756a029 100644 --- a/bin/native-owner/app-server-policy.mjs +++ b/bin/native-owner/app-server-policy.mjs @@ -1,5 +1,40 @@ import {spawn} from 'node:child_process'; +const MAX_MCP_STATUS_PAGES=256; + +function isRecord(value) { + return value!==null&&typeof value==='object'&&!Array.isArray(value); +} + +function validInstalledApp(value) { + return isRecord(value)&&typeof value.id==='string'&&(value.runtimeName===null||typeof value.runtimeName==='string')&&typeof value.enabled==='boolean'&&typeof value.callable==='boolean'; +} + +function validMcpServerStatus(value) { + return isRecord(value)&&typeof value.name==='string'&& + (value.runtimeStatus===null||['notStarted','starting','connected','authenticationRequired','failed','cancelled','disabled'].includes(value.runtimeStatus))&& + (value.pluginId===null||typeof value.pluginId==='string')&& + (value.serverInfo===null||isRecord(value.serverInfo))&& + isRecord(value.tools)&&(value.toolsError===null||typeof value.toolsError==='string')&& + Array.isArray(value.resources)&&Array.isArray(value.resourceTemplates)&& + ['unknown','unsupported','notLoggedIn','bearerToken','oAuth'].includes(value.authStatus); +} + +async function readMcpServerStatuses(request) { + const statuses=[],seenCursors=new Set(); + let cursor=null; + for(let page=0;page!validMcpServerStatus(status))||(response.nextCursor!==null&&typeof response.nextCursor!=='string'))throw Error('Invalid MCP server status response'); + statuses.push(...response.data); + if(response.nextCursor===null)return statuses; + if(seenCursors.has(response.nextCursor))throw Error('Repeated MCP server status cursor'); + seenCursors.add(response.nextCursor); + cursor=response.nextCursor; + } + throw Error('MCP server status pagination exceeded its bound'); +} + function uniqueNames(value) { if(!Array.isArray(value)||value.length>256||value.some(name=>typeof name!=='string'||!name.length||name.length>256))throw Error('Invalid inherited MCP server catalog'); const names=[...new Set(value)]; @@ -53,14 +88,14 @@ export async function verifyExternalToolConfiguration(request,mcpServerNames) { export async function verifyExternalToolIsolation(request,threadId,configuration) { const installed=await request('app/installed',{threadId,forceRefresh:false}); - const servers=await request('mcpServerStatus/list',{cursor:null,limit:100,detail:'toolsAndAuthOnly'}); - const apps=Array.isArray(installed.apps)?installed.apps:[]; - const mcpServers=Array.isArray(servers.data)?servers.data:[]; - const activeMcpServers=mcpServers.filter(server=>server.runtimeStatus!==null||server.serverInfo!==null||server.toolsError!==null||Object.keys(server.tools??{}).length||(server.resources??[]).length||(server.resourceTemplates??[]).length); + if(!isRecord(installed)||!Array.isArray(installed.apps)||installed.apps.some(app=>!validInstalledApp(app)))throw Error('Invalid installed app catalog response'); + const apps=installed.apps; + const mcpServers=await readMcpServerStatuses(request); + const activeMcpServers=mcpServers.filter(server=>server.runtimeStatus!==null||server.serverInfo!==null||server.toolsError!==null||Object.keys(server.tools).length||server.resources.length||server.resourceTemplates.length); const result={ ...configuration, - exposedApps:apps.map(app=>app.id??app.name??'unknown'), - activeMcpServers:activeMcpServers.map(server=>server.name??server.id??'unknown'), + exposedApps:apps.map(app=>app.id), + activeMcpServers:activeMcpServers.map(server=>server.name), }; if(result.exposedApps.length||result.activeMcpServers.length)throw Error('External app-server tools are not isolated: '+JSON.stringify(result)); return result; diff --git a/tests/fixtures/native-owner/app-server-policy.test.mjs b/tests/fixtures/native-owner/app-server-policy.test.mjs index 4fd8b4e4e39..de7618e7c63 100644 --- a/tests/fixtures/native-owner/app-server-policy.test.mjs +++ b/tests/fixtures/native-owner/app-server-policy.test.mjs @@ -7,11 +7,18 @@ function requestBoundary(responses) { const request=async(method,params)=>{ calls.push({method,params}); if(!(method in responses))throw Error('Unexpected request '+method); - return responses[method]; + const response=responses[method]; + return typeof response==='function'?response(params):response; }; return {request,calls}; } +function inactiveMcpServer(name) { + return {name,runtimeStatus:null,pluginId:null,serverInfo:null,tools:{},toolsError:null,resources:[],resourceTemplates:[],authStatus:'unknown'}; +} + +const isolatedConfiguration={appsFeatureEnabled:false,pluginsFeatureEnabled:false,configuredMcpServers:['alpha'],enabledMcpServers:[]}; + test('isolated app-server arguments disable every inherited external capability',()=>{ assert.deepEqual(isolatedAppServerArgs(['zeta','alpha'],['-c','windows.sandbox=unelevated']),[ 'app-server','--stdio','--disable','hooks','--disable','apps','--disable','plugins', @@ -24,7 +31,7 @@ test('the fake request boundary observes a fully isolated effective catalog',asy const boundary=requestBoundary({ 'config/read':{config:{features:{apps:false,plugins:false},mcp_servers:{alpha:{enabled:false}}}}, 'app/installed':{apps:[]}, - 'mcpServerStatus/list':{data:[]}, + 'mcpServerStatus/list':{data:[],nextCursor:null}, }); const configuration=await verifyExternalToolConfiguration(boundary.request,['alpha']); const result=await verifyExternalToolIsolation(boundary.request,'thread-1',configuration); @@ -50,12 +57,55 @@ test('enabled app features and MCP servers are rejected at the effective configu }); test('exposed apps and active MCP tools are rejected at the thread boundary',async()=>{ - const configuration={appsFeatureEnabled:false,pluginsFeatureEnabled:false,configuredMcpServers:['alpha'],enabledMcpServers:[]}; for(const responses of [ - {'app/installed':{apps:[{id:'connected-app'}]},'mcpServerStatus/list':{data:[]}}, - {'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{name:'alpha',runtimeStatus:null,serverInfo:null,toolsError:null,tools:{write:{}},resources:[],resourceTemplates:[]}]}}, + {'app/installed':{apps:[{id:'connected-app',runtimeName:'Connected App',enabled:true,callable:true}]},'mcpServerStatus/list':{data:[],nextCursor:null}}, + {'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{...inactiveMcpServer('alpha'),tools:{write:{}}}],nextCursor:null}}, ]) { const {request}=requestBoundary(responses); - await assert.rejects(verifyExternalToolIsolation(request,'thread-1',configuration),/External app-server tools are not isolated/); + await assert.rejects(verifyExternalToolIsolation(request,'thread-1',isolatedConfiguration),/External app-server tools are not isolated/); } }); + +test('a prohibited MCP capability on a later page is rejected',async()=>{ + const pages=new Map([ + [null,{data:Array.from({length:100},(_,index)=>inactiveMcpServer('inactive-'+index)),nextCursor:'page-2'}], + ['page-2',{data:[{...inactiveMcpServer('active'),runtimeStatus:'connected'}],nextCursor:null}], + ]); + const {request,calls}=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':({cursor})=>pages.get(cursor)}); + await assert.rejects(verifyExternalToolIsolation(request,'thread-1',isolatedConfiguration),/External app-server tools are not isolated/); + assert.equal(calls.filter(call=>call.method==='mcpServerStatus/list').length,2); +}); + +test('valid empty and multi-page catalogs are accepted',async()=>{ + const empty=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[],nextCursor:null}}); + assert.deepEqual((await verifyExternalToolIsolation(empty.request,'thread-1',isolatedConfiguration)).activeMcpServers,[]); + const pages=new Map([ + [null,{data:[inactiveMcpServer('alpha')],nextCursor:'page-2'}], + ['page-2',{data:[inactiveMcpServer('beta')],nextCursor:null}], + ]); + const multiple=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':({cursor})=>pages.get(cursor)}); + assert.deepEqual((await verifyExternalToolIsolation(multiple.request,'thread-1',isolatedConfiguration)).activeMcpServers,[]); + assert.deepEqual(multiple.calls.filter(call=>call.method==='mcpServerStatus/list').map(call=>call.params.cursor),[null,'page-2']); +}); + +test('malformed catalog responses are rejected',async()=>{ + for(const responses of [ + {'app/installed':{},'mcpServerStatus/list':{data:[],nextCursor:null}}, + {'app/installed':{apps:[{}]},'mcpServerStatus/list':{data:[],nextCursor:null}}, + {'app/installed':{apps:[]},'mcpServerStatus/list':{}}, + {'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{}],nextCursor:null}}, + {'app/installed':{apps:[]},'mcpServerStatus/list':{data:[],nextCursor:7}}, + ]) { + const {request}=requestBoundary(responses); + await assert.rejects(verifyExternalToolIsolation(request,'thread-1',isolatedConfiguration),/Invalid (installed app catalog|MCP server status) response/); + } +}); + +test('repeated and unbounded MCP pagination are rejected',async()=>{ + const repeated=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[],nextCursor:'same'}}); + await assert.rejects(verifyExternalToolIsolation(repeated.request,'thread-1',isolatedConfiguration),/Repeated MCP server status cursor/); + let page=0; + const unbounded=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':()=>({data:[],nextCursor:String(++page)})}); + await assert.rejects(verifyExternalToolIsolation(unbounded.request,'thread-1',isolatedConfiguration),/MCP server status pagination exceeded its bound/); + assert.equal(unbounded.calls.filter(call=>call.method==='mcpServerStatus/list').length,256); +}); From 5807ed2e5f2de268ad9bfb1409944216f2391c29 Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 16:23:39 +1200 Subject: [PATCH 12/61] no-mistakes(review): Harden native isolation and remove fixture-only production seams --- bin/fm-lock.sh | 31 +++++++---- bin/fm-native-codex.ps1 | 4 -- bin/native-owner/NativeHomeLease.cs | 7 --- bin/native-owner/NativeLauncher.cs | 2 +- bin/native-owner/NativeOperations.cs | 52 +++++++++---------- bin/native-owner/NativeOwner.cs | 1 - bin/native-owner/app-server-policy.mjs | 3 +- docs/verification/runtime-backends.md | 28 ++-------- tests/fixtures/native-owner/NativeDriver.cs | 36 +++++++++++-- tests/fixtures/native-owner/Run-Cycle.mjs | 11 ++-- tests/fixtures/native-owner/Verify-Cycle.mjs | 4 +- .../native-owner/app-server-policy.test.mjs | 12 ++++- tests/fixtures/native-owner/jq | 5 +- tests/fm-native-owner-codex-live-e2e.test.sh | 1 + tests/fm-session-lock-ancestry.test.sh | 7 +++ 15 files changed, 120 insertions(+), 84 deletions(-) diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index da98d3f78a1..fbc464e7b3d 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -1,8 +1,7 @@ #!/usr/bin/env bash # Acquire or inspect the per-home firstmate session lock. -# Writes the harness (agent) process PID found by walking the shell's ancestry, -# which lives as long as the firstmate session - unlike the transient subshell -# PID of any one tool call, which is dead moments after it is written. +# Writes the harness session identity, normally the process PID found by walking +# the shell's ancestry and an opaque launch-bound identity for the native owner. # Usage: fm-lock.sh acquire; exit 1 unless ownership is verified # fm-lock.sh status print holder and liveness; always exits 0 set -u @@ -23,6 +22,20 @@ mkdir -p "$STATE" 2>/dev/null || { # shellcheck source=bin/fm-session-lock-lib.sh . "$SCRIPT_DIR/fm-session-lock-lib.sh" +fm_lock_owner_label() { + case "$1" in + native:*) printf 'native owner identity %s' "$1" ;; + *) printf 'harness pid %s' "$1" ;; + esac +} + +fm_lock_holder_label() { + case "$1" in + native:*) printf 'native owner identity %s' "$1" ;; + *) printf 'pid %s' "$1" ;; + esac +} + if [ "${1:-}" = "status" ]; then if [ ! -f "$LOCK" ]; then echo "lock: free"; exit 0; fi old=$(cat "$LOCK" 2>/dev/null) || { @@ -30,11 +43,11 @@ if [ "${1:-}" = "status" ]; then exit 0 } if fm_harness_pid_alive "$old"; then - echo "lock: held by live harness pid $old" + echo "lock: held by live $(fm_lock_owner_label "$old")" elif fm_harness_pid_excludes "$old"; then echo "lock: held by native owner with unconfirmed health $old" else - echo "lock: stale (pid $old dead or not a harness)" + echo "lock: stale ($(fm_lock_holder_label "$old") dead or not a harness)" fi exit 0 fi @@ -74,11 +87,11 @@ trap 'exit 1' HUP INT TERM if [ -f "$LOCK" ] && [ ! -L "$LOCK" ]; then old=$(cat "$LOCK" 2>/dev/null || true) if [ "$old" = "$me" ]; then - echo "lock acquired: harness pid $me" + echo "lock acquired: $(fm_lock_owner_label "$me")" exit 0 fi if fm_harness_pid_excludes "$old"; then - echo "error: another firstmate session may hold the lock (pid $old); operate read-only until resolved" >&2 + echo "error: another firstmate session may hold the lock ($(fm_lock_holder_label "$old")); operate read-only until resolved" >&2 exit 1 fi fi @@ -103,7 +116,7 @@ if [ -e "$LOCK" ] || [ -L "$LOCK" ]; then exit 1 } if [ "$old" != "$me" ] && fm_harness_pid_excludes "$old"; then - echo "error: another firstmate session may hold the lock (pid $old); operate read-only until resolved" >&2 + echo "error: another firstmate session may hold the lock ($(fm_lock_holder_label "$old")); operate read-only until resolved" >&2 exit 1 fi fi @@ -120,4 +133,4 @@ if [ ! -f "$LOCK" ] || [ -L "$LOCK" ] || [ "$written" != "$me" ]; then exit 1 fi release_claim_lock -echo "lock acquired: harness pid $me" +echo "lock acquired: $(fm_lock_owner_label "$me")" diff --git a/bin/fm-native-codex.ps1 b/bin/fm-native-codex.ps1 index 31157400cb0..6ee7eb785fa 100644 --- a/bin/fm-native-codex.ps1 +++ b/bin/fm-native-codex.ps1 @@ -18,10 +18,6 @@ The Windows path to the temporary operational home. .PARAMETER JqImage An existing local Docker image containing jq and GNU timeout. No image is pulled. Read-only helper containers self-expire even if their native client is stopped. -.NOTES -Use /interrupt to interrupt the current model turn and /quit to end the session. -Interrupted acknowledgements remain pending for evidence-based reconciliation. -See docs/native-windows-codex.md for setup and supported limits. #> param([switch]$Experimental,[switch]$BuildOnly,[switch]$VerifyOnly,[string]$OperationalHome,[string]$JqImage) $ErrorActionPreference='Stop' diff --git a/bin/native-owner/NativeHomeLease.cs b/bin/native-owner/NativeHomeLease.cs index cbf5bdf6cae..66c38a5feb5 100644 --- a/bin/native-owner/NativeHomeLease.cs +++ b/bin/native-owner/NativeHomeLease.cs @@ -106,12 +106,5 @@ public static Dictionary Binding(string state) { return record; } } - public static string Check(string home,string generation) { - using(var reader=new FileStream(Filename(home),FileMode.Open,FileAccess.Read,FileShare.ReadWrite)) { - var record=Read(reader); - bool same=record.ContainsKey("generation") && (string)record["generation"]==generation; - return Json.Serialize(new Dictionary{{"probeOwnerCurrent",same && RootAlive(record)},{"generationMatches",same},{"authorityGranted",false}}); - } - } public void Dispose() { if(file!=null) { file.Dispose(); file=null; } } } diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs index c2c61a07902..31b763d7eec 100644 --- a/bin/native-owner/NativeLauncher.cs +++ b/bin/native-owner/NativeLauncher.cs @@ -56,7 +56,7 @@ static int Launch(string selectedHome) { output=FileHandle(Path.Combine(runtime,"operations.log"),0x40000000,2);input=FileHandle("NUL",0x80000000,3); var operationStartup=new SI {cb=Marshal.SizeOf(typeof(SI)),flags=0x100,input=input,output=output,error=output}; if(!operationJournal.NeedsReconciliation) { - var operation=StartScope("owner-operation",job,env,runtime,ref operationStartup);scopes.Add(operation); + var operation=StartOwnerOperation(job,env,runtime,ref operationStartup);scopes.Add(operation); if(ResumeThread(operation.process.thread)==0xffffffff)throw Error("Resume startup"); } controlThread=new Thread(()=>{ diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index dd605cf2407..ee7be5e880e 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -27,32 +27,29 @@ static SortedDictionary EnvironmentFor(string pipe, string sessio result["FM_PROBE_EXE"] = OwnExe; return result; } - static ChildScope StartScope(string role, IntPtr parentJob, IntPtr environment, string home, ref SI startup, string purpose="startup") { - var scope=new ChildScope { job=CreateJobObject(IntPtr.Zero,null), role=role, purpose=purpose }; + static ChildScope StartOwnerOperation(IntPtr parentJob, IntPtr environment, string home, ref SI startup, string purpose="startup") { + if(purpose!="startup" && purpose!="check" && purpose!="ack") throw new ArgumentException("Unknown fixed operation"); + var scope=new ChildScope { job=CreateJobObject(IntPtr.Zero,null), role="owner-operation", purpose=purpose }; IntPtr operationEnvironment=IntPtr.Zero; if(scope.job==IntPtr.Zero) throw Error("CreateJobObject child scope"); try { - if(role=="owner-operation") NativeOperationLifetime.Configure(scope.job); - string operation=role=="owner-operation" ? (purpose=="startup" ? "owner-operation" : "notification-operation "+purpose) : "scoped-client "+role; - if(role=="owner-operation") { - // Only fixed operations receive the grant; caller claims never select it. - var values=new SortedDictionary(StringComparer.OrdinalIgnoreCase); - foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { - string value=Environment.GetEnvironmentVariable(key); if(value!=null) values[key]=value; - } - values["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); - // Copy only the controller's registration fields from the prepared block. - int offset=0; - while(Marshal.ReadInt16(environment,offset)!=0) { - string entry=Marshal.PtrToStringUni(IntPtr.Add(environment,offset)); offset+=(entry.Length+1)*2; - int split=entry.IndexOf('='); if(split<=0) continue; - string key=entry.Substring(0,split); - if(key.StartsWith("FM_PROBE_",StringComparison.Ordinal) || key=="FM_HOME" || key=="MSYS") values[key]=entry.Substring(split+1); - } - var block=new StringBuilder(); foreach(var value in values) block.Append(value.Key).Append('=').Append(value.Value).Append('\0'); block.Append('\0'); - operationEnvironment=Marshal.StringToHGlobalUni(block.ToString()); + NativeOperationLifetime.Configure(scope.job); + string operation=purpose=="startup" ? "owner-operation" : "notification-operation "+purpose; + var values=new SortedDictionary(StringComparer.OrdinalIgnoreCase); + foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { + string value=Environment.GetEnvironmentVariable(key); if(value!=null) values[key]=value; } - if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" "+operation),IntPtr.Zero,IntPtr.Zero,true,0x4|0x400|0x200,operationEnvironment==IntPtr.Zero ? environment : operationEnvironment,home,ref startup,out scope.process)) throw Error("CreateProcess child scope"); + values["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); + int offset=0; + while(Marshal.ReadInt16(environment,offset)!=0) { + string entry=Marshal.PtrToStringUni(IntPtr.Add(environment,offset)); offset+=(entry.Length+1)*2; + int split=entry.IndexOf('='); if(split<=0) continue; + string key=entry.Substring(0,split); + if(key.StartsWith("FM_PROBE_",StringComparison.Ordinal) || key=="FM_HOME" || key=="MSYS") values[key]=entry.Substring(split+1); + } + var block=new StringBuilder(); foreach(var value in values) block.Append(value.Key).Append('=').Append(value.Value).Append('\0'); block.Append('\0'); + operationEnvironment=Marshal.StringToHGlobalUni(block.ToString()); + if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" "+operation),IntPtr.Zero,IntPtr.Zero,true,0x4|0x400|0x200,operationEnvironment,home,ref startup,out scope.process)) throw Error("CreateProcess child scope"); if(!AssignProcessToJobObject(parentJob,scope.process.process) || !AssignProcessToJobObject(scope.job,scope.process.process)) throw Error("Assign child scope"); // The caller records this scope before resuming the process. return scope; @@ -109,14 +106,15 @@ static void NotificationRequest(Dictionary request,Dictionary delivered; static string receipt; static bool consumed; - static int checkStarts,ackStarts; static Dictionary Verdict(Dictionary request, uint pid, IntPtr root, IntPtr job, uint rootPid, List scopes, string session, string home, string nonce) { string reason = "unverified", classification="none"; if (WaitForSingleObject(root, 0) != WAIT_TIMEOUT) reason = "session-exited"; diff --git a/bin/native-owner/app-server-policy.mjs b/bin/native-owner/app-server-policy.mjs index 3a09756a029..b51edfb57bb 100644 --- a/bin/native-owner/app-server-policy.mjs +++ b/bin/native-owner/app-server-policy.mjs @@ -15,6 +15,7 @@ function validMcpServerStatus(value) { (value.runtimeStatus===null||['notStarted','starting','connected','authenticationRequired','failed','cancelled','disabled'].includes(value.runtimeStatus))&& (value.pluginId===null||typeof value.pluginId==='string')&& (value.serverInfo===null||isRecord(value.serverInfo))&& + (value.serverCapabilities===null||isRecord(value.serverCapabilities))&& isRecord(value.tools)&&(value.toolsError===null||typeof value.toolsError==='string')&& Array.isArray(value.resources)&&Array.isArray(value.resourceTemplates)&& ['unknown','unsupported','notLoggedIn','bearerToken','oAuth'].includes(value.authStatus); @@ -91,7 +92,7 @@ export async function verifyExternalToolIsolation(request,threadId,configuration if(!isRecord(installed)||!Array.isArray(installed.apps)||installed.apps.some(app=>!validInstalledApp(app)))throw Error('Invalid installed app catalog response'); const apps=installed.apps; const mcpServers=await readMcpServerStatuses(request); - const activeMcpServers=mcpServers.filter(server=>server.runtimeStatus!==null||server.serverInfo!==null||server.toolsError!==null||Object.keys(server.tools).length||server.resources.length||server.resourceTemplates.length); + const activeMcpServers=mcpServers.filter(server=>server.runtimeStatus!==null||server.serverInfo!==null||server.serverCapabilities!==null||server.toolsError!==null||Object.keys(server.tools).length||server.resources.length||server.resourceTemplates.length); const result={ ...configuration, exposedApps:apps.map(app=>app.id), diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index ebb9533f606..9ed3519ef8a 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -17,12 +17,8 @@ Refresh the portable request-policy regression with: bash tests/fm-native-owner-tool-gate.test.sh ``` -The 31 behavioral cases cover the request policy and bounded host shutdown, including synchronous revocation, single-flight shutdown, failed interruption, cancelled shutdown requests, retained-process termination, and unconfirmed-exit refusal. -The native controller now persists receipt presentation, acknowledgement intent, and completion under its existing exclusive home lease. -It flushes acknowledgement intent before invoking the mutation and preserves interrupted attempts for reconciliation rather than retrying them. -The token-free Windows guard exercises 28 receipt cases, including complete-versus-partial recovery, preservation of newer work, missing or changed evidence, generation separation, file links, and lease revocation. -Two native operation-lifetime cases verify bounded stop and last-handle-close termination while an independent process remains alive. -The guard also retains the exclusive-owner, normal-exit, controller-loss, orphan-recovery, ambiguous-record, and child-scope tests: +The portable policy command above and native receipt command below remain refresh entry points. +No attributable terminal output was retained for their latest runs, so this record does not claim those results as current evidence: ```sh bash tests/fm-native-owner-receipt-live-e2e.test.sh @@ -43,7 +39,8 @@ PASS: effective app and MCP catalogs are isolated {"appsFeatureEnabled":false,"p Refresh the actual Windows integration with the explicit two-model-turn guard: ```sh -FM_LIVE_NATIVE_CODEX=1 bash tests/fm-native-owner-codex-live-e2e.test.sh +FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_CODEX=1 \ + bash tests/fm-native-owner-codex-live-e2e.test.sh ``` Observed terminal results: @@ -69,15 +66,6 @@ The receipt evidence also supports multiple inbox targets and general wake recor ### Explicit launcher integration -The opt-in launcher now runs real startup, delivers its digest before deferred work finishes, and retains the deferred operation's authorization until completion or bounded cancellation. -It starts an ephemeral read-only, network-disabled primary thread and verifies the returned app-server policy. -Its interactive host accepts ordinary input, `/interrupt`, and `/quit` and initiates model handling when a new durable notification arrives. -Only controller-selected startup, notification check, and acknowledgement scripts run with registered native operation authority. -Docker jq mounts are read-only, networking and image pulls are disabled, and its read-only helper process has a separate container-side expiry. -The launch-bound owner record preserves verified-dead predecessor generations across failed startup attempts so an intervening failed launch does not strand the previous session lock. -That history is bounded and refuses further acquisition rather than silently forgetting evidence. -An unresolved acknowledgement prevents more work and identifies the preserved journal for review; no partial attempt is retried or rolled back automatically. - Refresh the actual launcher without model turns, then optionally exercise two notification turns and active-turn cancellation: ```sh @@ -87,13 +75,7 @@ FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 FM_LIVE bash tests/fm-native-owner-launcher-live-e2e.test.sh ``` -The actual launcher refused a competing launch without changing its record, restarted after an intervening startup failure, refused a populated home unchanged, and delivered startup before a deliberately delayed deferred worker completed. -Non-temporary homes and pre-existing reparse-point ancestors were refused before creating a home. -Its cancellation stopped the registered deferred operation while an independent process stayed alive. -Two later inbox notifications each initiated a real model turn without another startup or a test-generated model prompt; each was observed and acknowledged through the fixed operations. -A separate shutdown interrupted an active model turn, confirmed app-server exit, and preserved the pending notification. -The production launcher also refused a real partially acknowledged journal from the fault-injection fixture and surfaced the reconciliation requirement without starting the model. -Fixture delays and controlled notifications remain under `tests/fixtures/native-owner/`; the launcher contains no test message generator. +No attributable terminal output was retained for the latest launcher runs, so this record does not claim their results as current evidence. ## Harness detection precedence diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 156e683a37d..e1243a6dace 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -107,6 +107,30 @@ static int EnvironmentTests() { foreach(var entry in original)Environment.SetEnvironmentVariable(entry.Key,entry.Value); } } + static ChildScope StartFixtureScope(string role,IntPtr parentJob,IntPtr environment,string home,ref SI startup) { + if(role!="worker" && role!="nested-primary") throw new ArgumentException("Unsupported fixture scope"); + var scope=new ChildScope {job=CreateJobObject(IntPtr.Zero,null),role=role,purpose="fixture"}; + if(scope.job==IntPtr.Zero) throw Error("Create fixture scope job"); + try { + if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" scoped-client "+role),IntPtr.Zero,IntPtr.Zero,true,0x4|0x400|0x200,environment,home,ref startup,out scope.process)) throw Error("Create fixture scope"); + if(!AssignProcessToJobObject(parentJob,scope.process.process) || !AssignProcessToJobObject(scope.job,scope.process.process)) throw Error("Assign fixture scope"); + return scope; + } catch { + if(scope.process.process!=IntPtr.Zero) {TerminateProcess(scope.process.process,125);CloseHandle(scope.process.thread);CloseHandle(scope.process.process);} + CloseHandle(scope.job);throw; + } + } + static int LeaseCheck(string home,string generation) { + bool current=false; + try { + var binding=NativeHomeLease.Binding(Path.Combine(home,"state")); + current=binding.ContainsKey("generation") && (string)binding["generation"]==generation; + } catch(InvalidOperationException error) { + if(error.Message!="Primary no longer live") throw; + } + Console.WriteLine(Json.Serialize(new Dictionary{{"probeOwnerCurrent",current}})); + return 0; + } static int Run(string configPath) { var config = Json.Deserialize>(File.ReadAllText(configPath)); string home = Path.GetFullPath((string)config["home"]); @@ -139,6 +163,10 @@ static int Run(string configPath) { recoveredAcknowledgements=operationJournal.ReconcileCompletedAcknowledgements(); } var values = EnvironmentFor(pipeName, session, home, nonce); + if(config.ContainsKey("ownerExercise") && (bool)config["ownerExercise"]) { + if(!config.ContainsKey("jqImage") || string.IsNullOrWhiteSpace((string)config["jqImage"])) throw new ArgumentException("Owner exercise requires an explicit local jq image"); + values["FM_PROBE_JQ_IMAGE"]=(string)config["jqImage"]; + } values["MSYS"]="winsymlinks:nativestrict"; if(config.ContainsKey("apiDry") && (bool)config["apiDry"]) values["FM_PROBE_API_DRY"]="1"; if(config.ContainsKey("ackFault")) { @@ -164,12 +192,12 @@ static int Run(string configPath) { if (!GetProcessTimes(child.process,out born,out exited,out kernel,out user)) throw Error("GetProcessTimes"); if(lease!=null) lease.Publish(child.pid,((ulong)born.high<<32)|born.low,session,pipeName); if(config.ContainsKey("ownerOperation") && (bool)config["ownerOperation"]) { - var operation=StartScope("owner-operation",job,env,home,ref startup); + var operation=StartOwnerOperation(job,env,home,ref startup); scopes.Add(operation); if(ResumeThread(operation.process.thread)==0xffffffff) throw Error("Resume owner operation"); } if(values.ContainsKey("FM_PROBE_BOUNDARIES")) { - foreach(string role in new [] {"worker","nested-primary"}) scopes.Add(StartScope(role,job,env,home,ref startup)); + foreach(string role in new [] {"worker","nested-primary"}) scopes.Add(StartFixtureScope(role,job,env,home,ref startup)); foreach(var scope in scopes) if(ResumeThread(scope.process.thread)==0xffffffff) throw Error("Resume child scope"); } if (ResumeThread(child.thread) == 0xffffffff) throw Error("ResumeThread"); @@ -229,7 +257,7 @@ static int Run(string configPath) { {"probeGeneration",session},{"probeLeaseHeld",lease!=null}, {"pipeDacl",security.GetSecurityDescriptorSddlForm(AccessControlSections.Access)}, {"recoveredAcknowledgements",recoveredAcknowledgements},{"receiptNeedsReconciliation",operationJournal!=null && operationJournal.NeedsReconciliation}, - {"authorityImplemented",false},{"notificationCheckStarts",checkStarts},{"notificationAckStarts",ackStarts},{"notificationConsumed",consumed},{"observations",observations} + {"authorityImplemented",false},{"notificationConsumed",consumed},{"observations",observations} }; if (outsider != null) { if (!outsider.WaitForExit(10000)) throw new IOException("Outsider probe did not stop"); @@ -263,7 +291,7 @@ public static int Main(string[] args) { if(TryOwnerCommand(args,out ownerResult)) return ownerResult; if(args.Length>0 && args[0]=="client") return Client(args.Length>1 ? args[1] : "agent-tool"); if(args.Length==2 && args[0]=="sleep") { int ms=int.Parse(args[1]); if(ms<0 || ms>15000) throw new ArgumentException("Sleep must be bounded"); Thread.Sleep(ms); return 0; } - if(args.Length==3 && args[0]=="lease-check") { Console.WriteLine(NativeHomeLease.Check(args[1],args[2])); return 0; } + if(args.Length==3 && args[0]=="lease-check") return LeaseCheck(args[1],args[2]); if(args.Length==2 && args[0]=="notification-operation") { if(args[1]!="check" && args[1]!="ack") throw new ArgumentException("Unsupported notification operation"); string script=Path.Combine(Path.GetDirectoryName(OwnExe),"firstmate","notification-"+args[1]+".sh"); diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs index a46143c32ec..d877fe3bd89 100644 --- a/tests/fixtures/native-owner/Run-Cycle.mjs +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -15,7 +15,7 @@ if(!dry){const preflight=read(path.join(dir,'bridge-preflight.json'));if(!prefli const home=path.join(build.root,'appserver-'+randomUUID());fs.mkdirSync(home); fs.writeFileSync(path.join(home,'build.json'),JSON.stringify(build,null,2)); const script=path.join(build.root,'AppHost.mjs'); -const spec={home,leaseHome:path.join(home,'home'),executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'UserOnly',apiDry:dry}; +const spec={home,leaseHome:path.join(home,'home'),executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'UserOnly',apiDry:dry,jqImage:process.env.FM_NATIVE_TEST_JQ_IMAGE}; if(fault)spec.ackFault=fault; const file=path.join(home,'spec.json');fs.writeFileSync(file,JSON.stringify(spec,null,2)); const run=spawnSync(build.binary,['run',file],{encoding:'utf8',timeout:310000}); @@ -25,12 +25,14 @@ console.log(JSON.stringify({home,exit:run.status,dry})); if(run.status!==0)throw Error('Bounded app-server run failed; inspect evidence, do not start another model attempt'); const host=read(path.join(home,'app-host-evidence.json')),native=read(path.join(home,'result.json')); if(!host.shutdown?.stopped||!host.shutdown.operationsStopped||!host.shutdown.exited)throw Error('Host shutdown was not confirmed'); -if(!host.passed||native.notificationCheckStarts!==1||native.notificationAckStarts!==1||native.notificationConsumed!==!fault)throw Error('Notification cycle incomplete'); +const starts=action=>host.native.filter(row=>row.action===action&&row.state==='pending').length; +if(!host.passed||starts('check')!==1||starts('ack')!==1||native.notificationConsumed!==!fault)throw Error('Notification cycle incomplete'); if(!host.native.filter(row=>row.action==='check'||row.action==='ack').every(row=>row.startupExpired))throw Error('Startup scope still active'); if(fault) { const queue=path.join(spec.leaseHome,'state/.wake-queue'),before=fs.readFileSync(queue); const journal=()=>fs.readFileSync(path.join(spec.leaseHome,'owner-receipts.jsonl'),'utf8').trim().split('\n').map(JSON.parse); - if(journal().at(-1).event!=='ack-started'||!native.receiptNeedsReconciliation)throw Error('Interrupted intent was not preserved'); + const historyBefore=journal(); + if(historyBefore.at(-1).event!=='ack-started'||!native.receiptNeedsReconciliation)throw Error('Interrupted intent was not preserved'); if((before.length===0)!==(fault==='complete'))throw Error('Fault did not land at the requested mutation boundary'); const recovery=path.join(home,'recovery');fs.mkdirSync(recovery); const recoverySpec={home:recovery,leaseHome:spec.leaseHome,executable:build.binary,arguments:'sleep 100',timeoutSeconds:10,pipeAcl:'UserOnly'}; @@ -39,9 +41,10 @@ if(fault) { fs.writeFileSync(path.join(recovery,'controller.stdout'),restarted.stdout||'');fs.writeFileSync(path.join(recovery,'controller.stderr'),restarted.stderr||''); if(restarted.status!==0)throw Error('Recovery controller failed'); const recovered=read(path.join(recovery,'result.json')); - if(recovered.recoveredAcknowledgements!==(fault==='complete'?1:0)||recovered.receiptNeedsReconciliation!==(fault==='partial')||recovered.notificationAckStarts!==0)throw Error('Incorrect interrupted acknowledgement recovery'); + if(recovered.recoveredAcknowledgements!==(fault==='complete'?1:0)||recovered.receiptNeedsReconciliation!==(fault==='partial'))throw Error('Incorrect interrupted acknowledgement recovery'); if(!fs.readFileSync(queue).equals(before))throw Error('Recovery replayed a wake mutation'); const history=journal(); + if(history.length!==historyBefore.length+(fault==='complete'?1:0))throw Error('Recovery started or recorded an unexpected acknowledgement'); if(fault==='complete'&&(history.at(-1).event!=='recovered-acknowledged'||history.at(-1).ackGeneration!==native.probeGeneration||history.at(-1).generation!==recovered.probeGeneration))throw Error('Recovery generations are not bound'); console.log(`PASS: actual ${fault} acknowledgement interruption; recovery ${fault==='complete'?'confirmed completed effects without replay':'preserved the unresolved partial mutation'}.`); process.exit(0); diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index 7332660cbe5..15f079cd88e 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -31,7 +31,9 @@ assert.equal(primary[1].arguments.observed,primary[0].result.challenge); assert.equal(primary[2].result.denied,'receipt-already-consumed'); assert.equal(host.tools.find(tool=>tool.threadId===host.foreign).result.denied,'wrong-thread-turn-or-replay'); assert.equal(host.nativeReplayDenied,true); -assert.equal(native.notificationCheckStarts,1);assert.equal(native.notificationAckStarts,1);assert.equal(native.notificationConsumed,true); +assert.equal(host.native.filter(row=>row.action==='check'&&row.state==='pending').length,1); +assert.equal(host.native.filter(row=>row.action==='ack'&&row.state==='pending').length,1); +assert.equal(native.notificationConsumed,true); assert.ok(host.native.filter(row=>row.action==='check'||row.action==='ack').every(row=>row.startupExpired)); const delivered=read(path.join(home,'notification-check.json')); const acknowledgement=read(path.join(home,'notification-ack-request.json')); diff --git a/tests/fixtures/native-owner/app-server-policy.test.mjs b/tests/fixtures/native-owner/app-server-policy.test.mjs index de7618e7c63..0f0adfb7b29 100644 --- a/tests/fixtures/native-owner/app-server-policy.test.mjs +++ b/tests/fixtures/native-owner/app-server-policy.test.mjs @@ -14,7 +14,7 @@ function requestBoundary(responses) { } function inactiveMcpServer(name) { - return {name,runtimeStatus:null,pluginId:null,serverInfo:null,tools:{},toolsError:null,resources:[],resourceTemplates:[],authStatus:'unknown'}; + return {name,runtimeStatus:null,pluginId:null,serverInfo:null,serverCapabilities:null,tools:{},toolsError:null,resources:[],resourceTemplates:[],authStatus:'unknown'}; } const isolatedConfiguration={appsFeatureEnabled:false,pluginsFeatureEnabled:false,configuredMcpServers:['alpha'],enabledMcpServers:[]}; @@ -76,6 +76,16 @@ test('a prohibited MCP capability on a later page is rejected',async()=>{ assert.equal(calls.filter(call=>call.method==='mcpServerStatus/list').length,2); }); +test('server capabilities are required and only null is inactive',async()=>{ + const valid=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[inactiveMcpServer('alpha')],nextCursor:null}}); + assert.deepEqual((await verifyExternalToolIsolation(valid.request,'thread-1',isolatedConfiguration)).activeMcpServers,[]); + const missing=inactiveMcpServer('missing');delete missing.serverCapabilities; + const absent=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[missing],nextCursor:null}}); + await assert.rejects(verifyExternalToolIsolation(absent.request,'thread-1',isolatedConfiguration),/Invalid MCP server status response/); + const exposed=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{...inactiveMcpServer('active'),serverCapabilities:{tools:{}}}],nextCursor:null}}); + await assert.rejects(verifyExternalToolIsolation(exposed.request,'thread-1',isolatedConfiguration),/External app-server tools are not isolated/); +}); + test('valid empty and multi-page catalogs are accepted',async()=>{ const empty=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[],nextCursor:null}}); assert.deepEqual((await verifyExternalToolIsolation(empty.request,'thread-1',isolatedConfiguration)).activeMcpServers,[]); diff --git a/tests/fixtures/native-owner/jq b/tests/fixtures/native-owner/jq index a9eb382cda0..3b891a815f1 100644 --- a/tests/fixtures/native-owner/jq +++ b/tests/fixtures/native-owner/jq @@ -2,6 +2,9 @@ # jq stays in the existing Docker image. Only this disposable build is mounted. set -eu root=$(cd "$(dirname "$0")/.." && pwd) +image=${FM_PROBE_JQ_IMAGE:?FM_PROBE_JQ_IMAGE must name an existing local image with jq and GNU timeout} +env MSYS2_ARG_CONV_EXCL='*' 'C:/Program Files/Docker/Docker/resources/bin/docker.exe' image inspect "$image" >/dev/null exec env MSYS2_ARG_CONV_EXCL='*' 'C:/Program Files/Docker/Docker/resources/bin/docker.exe' run --rm -i \ + --pull never --network none \ --mount "type=bind,source=$(cygpath -m "$root"),target=$root,readonly" \ - --workdir "$root" firstmate-pr3553-validation:tools-v1 jq "$@" + --workdir "$root" "$image" timeout --kill-after=3s 45s jq "$@" diff --git a/tests/fm-native-owner-codex-live-e2e.test.sh b/tests/fm-native-owner-codex-live-e2e.test.sh index 1e2af4ae9cd..14361cde0bf 100644 --- a/tests/fm-native-owner-codex-live-e2e.test.sh +++ b/tests/fm-native-owner-codex-live-e2e.test.sh @@ -5,6 +5,7 @@ set -eu # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" fm_live_gate opt-in FM_LIVE_NATIVE_CODEX node powershell.exe codex docker +: "${FM_NATIVE_TEST_JQ_IMAGE:?Set FM_NATIVE_TEST_JQ_IMAGE to an existing local image with jq and GNU timeout}" case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) ;; *) printf '%s\n' 'Native Codex ownership test requires Windows' >&2; exit 1 ;; diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index bbafcaf5302..6152c890dc9 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -641,12 +641,19 @@ SH *) fail "unknown native owner status was not distinguished: $out" ;; esac case "$out" in *'held by live harness'*) fail "unknown native owner status reported positive health" ;; esac + out=$(PATH="$fakebin:$PATH" FM_HOME="$dir" FM_STATE_OVERRIDE="$dir/state" FM_TEST_NATIVE_STATE=1 "$bindir/fm-lock.sh" status) + case "$out" in *"stale (native owner identity $identity dead or not a harness)"*) ;; *) fail "dead native owner status mislabeled its identity: $out" ;; esac set +e out=$(PATH="$fakebin:$PATH" FM_HOME="$dir" FM_STATE_OVERRIDE="$dir/state" FM_TEST_NATIVE_STATE=2 "$bindir/fm-lock.sh" 2>&1) rc=$? set -e [ "$rc" -ne 0 ] || fail "unknown native owner did not exclude acquisition" [ "$(cat "$dir/state/.lock")" = "$identity" ] || fail "unknown native owner was overwritten during acquisition" + case "$out" in *"native owner identity $identity"*) ;; *) fail "native exclusion mislabeled its owner identity: $out" ;; esac + case "$out" in *"pid $identity"*) fail "native exclusion labeled an opaque identity as a pid: $out" ;; esac + printf '%s\n' 'native:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' > "$dir/state/.lock" + out=$(PATH="$fakebin:$PATH" FM_HOME="$dir" FM_STATE_OVERRIDE="$dir/state" FM_TEST_NATIVE_STATE=0 "$bindir/fm-lock.sh") + case "$out" in *'lock acquired: native owner identity native:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'*) ;; *) fail "native acquisition mislabeled its owner identity: $out" ;; esac pass "native lock status distinguishes unknown health while acquisition remains excluded" } From 7d9ef2bd2e7fd0aa6567613ae02ecb08d1a3a85f Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 17:31:14 +1200 Subject: [PATCH 13/61] no-mistakes(review): Fix native startup, shutdown, and lock compatibility --- bin/fm-harness.sh | 10 ++++-- bin/fm-lock.sh | 27 +++++++++++--- bin/native-owner/NativeOperations.cs | 3 +- bin/native-owner/codex-host.mjs | 7 ++-- bin/native-owner/codex-tool-gate.mjs | 3 ++ bin/native-owner/host-lifecycle.mjs | 14 ++++++-- docs/configuration.md | 1 + docs/native-windows-codex.md | 1 + docs/verification/runtime-backends.md | 36 ++----------------- tests/fixtures/native-owner/AppHost.mjs | 6 ++-- tests/fixtures/native-owner/NativeDriver.cs | 5 +++ .../native-owner/OperationLifetimeTests.cs | 19 +++++++--- tests/fixtures/native-owner/Run-Cycle.mjs | 1 + tests/fixtures/native-owner/Verify-Cycle.mjs | 2 +- .../native-owner/host-lifecycle.test.mjs | 11 ++++-- .../fixtures/native-owner/tool-gate.test.mjs | 6 ++++ 16 files changed, 94 insertions(+), 58 deletions(-) diff --git a/bin/fm-harness.sh b/bin/fm-harness.sh index eda6c5f374a..19ef1e017be 100755 --- a/bin/fm-harness.sh +++ b/bin/fm-harness.sh @@ -52,9 +52,13 @@ # Ancestry - the nearest harness process in this process's parent chain. This # is the structural fact about who actually owns the process tree, # so it is what settles a disagreement. -# detect_own is the single owner of how the two combine; harness_marker and -# harness_ancestry only report evidence. Record each newly verified env marker -# in harness_marker, and each newly verified command name in harness_ancestry. +# Native owner - a launch-bound identity published by the experimental Windows +# launcher and authenticated through its native owner endpoint. +# When selected by its home record, it precedes marker and ancestry. +# detect_own is the single owner of how the three combine; harness_marker and +# harness_ancestry only report their evidence. Record each newly verified env +# marker in harness_marker, and each newly verified command name in +# harness_ancestry. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index fbc464e7b3d..c205a20dbdb 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -36,6 +36,21 @@ fm_lock_holder_label() { esac } +fm_lock_conflict_message() { + if fm_harness_pid_alive "$1"; then + case "$1" in + native:*) printf 'error: another live firstmate session holds the lock (native owner identity %s); operate read-only until resolved' "$1" ;; + *) printf 'error: another live firstmate session holds the lock (pid %s); operate read-only until resolved' "$1" ;; + esac + return 0 + fi + if fm_harness_pid_excludes "$1"; then + printf 'error: another firstmate session may hold the lock (%s); operate read-only until resolved' "$(fm_lock_holder_label "$1")" + return 0 + fi + return 1 +} + if [ "${1:-}" = "status" ]; then if [ ! -f "$LOCK" ]; then echo "lock: free"; exit 0; fi old=$(cat "$LOCK" 2>/dev/null) || { @@ -90,8 +105,8 @@ if [ -f "$LOCK" ] && [ ! -L "$LOCK" ]; then echo "lock acquired: $(fm_lock_owner_label "$me")" exit 0 fi - if fm_harness_pid_excludes "$old"; then - echo "error: another firstmate session may hold the lock ($(fm_lock_holder_label "$old")); operate read-only until resolved" >&2 + if conflict=$(fm_lock_conflict_message "$old"); then + echo "$conflict" >&2 exit 1 fi fi @@ -115,9 +130,11 @@ if [ -e "$LOCK" ] || [ -L "$LOCK" ]; then echo "error: session lock is unreadable; operate read-only until resolved" >&2 exit 1 } - if [ "$old" != "$me" ] && fm_harness_pid_excludes "$old"; then - echo "error: another firstmate session may hold the lock ($(fm_lock_holder_label "$old")); operate read-only until resolved" >&2 - exit 1 + if [ "$old" != "$me" ]; then + if conflict=$(fm_lock_conflict_message "$old"); then + echo "$conflict" >&2 + exit 1 + fi fi fi if ! { printf '%s\n' "$me" > "$LOCK"; } 2>/dev/null; then diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index ee7be5e880e..4bae7518189 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -67,9 +67,10 @@ static void NotificationRequest(Dictionary request,Dictionarynew Promise(resolve=>setTimeout(resolve,ms)); @@ -77,7 +77,7 @@ async function interrupt(signal){ async function operation(action,extra={}){ if(action==='check'){ const status=await native('status'); - if(status.operationState==='starting')return {operationState:'quiet'}; + if(status.operationState==='starting')return status; if(status.operationState!=='ready')throw Error('Native work unavailable: '+status.operationState); const previous=await native('result');if(previous.operationState==='delivered')return previous; } @@ -109,7 +109,7 @@ function stop(){ if(lifecycle)return lifecycle.shutdown(); closing=true;consoleInput.close();process.stdin.destroy(); lifecycle=createHostLifecycle({gate:{close:()=>gate?.close()},interrupt, - stopOperations:async signal=>(await native('shutdown',{},signal)).operationState==='stopped', + stopOperations:async signal=>{const value=await native('shutdown',{},signal);return {stopped:value.operationState==='stopped',reconciliationRequired:value.reconciliationRequired===true};}, closeInput:()=>{if(server)server.stdin.end();}, waitForExit:signal=>!alive?Promise.resolve(true):new Promise(resolve=>{ const exit=()=>{signal.removeEventListener('abort',abort);resolve(true);}; @@ -119,6 +119,7 @@ function stop(){ const result=lifecycle.shutdown(); void result.then(value=>{ fs.writeFileSync(path.join(runtime,'shutdown.json'),JSON.stringify(value)); + if(value.reconciliationRequired)console.error(reconciliationWarning(path.join(process.env.FM_HOME,'owner-receipts.jsonl'))); if(!value.stopped){console.error('Shutdown was not fully confirmed; durable work was preserved.');process.exitCode=1;} channel.close();socket.destroy(); }); diff --git a/bin/native-owner/codex-tool-gate.mjs b/bin/native-owner/codex-tool-gate.mjs index 0d62b0ec0aa..ad14550fa29 100644 --- a/bin/native-owner/codex-tool-gate.mjs +++ b/bin/native-owner/codex-tool-gate.mjs @@ -67,6 +67,9 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { try { if (params.tool === 'fm_notification_check') { const result = await operate('check'); + if (result?.operationState === 'starting') { + return valid(params) ? { success: false, value: { unavailable: 'startup-in-progress' } } : deny('wrong-thread-turn-or-replay'); + } if (result?.operationState === 'quiet') { return valid(params) ? { success: true, value: { quiet: true } } : deny('wrong-thread-turn-or-replay'); } diff --git a/bin/native-owner/host-lifecycle.mjs b/bin/native-owner/host-lifecycle.mjs index c6eded0231c..6d9078627ef 100644 --- a/bin/native-owner/host-lifecycle.mjs +++ b/bin/native-owner/host-lifecycle.mjs @@ -1,5 +1,9 @@ // Host-owned lifecycle, never a model tool. Callbacks close the registered // operation jobs and the retained app-server process, not a shared process tree. +export function reconciliationWarning(journal) { + if (typeof journal !== 'string' || !journal) throw new TypeError('Receipt journal path is required'); + return `Acknowledgement completion is unconfirmed. Its records are preserved and require reconciliation: ${journal}`; +} export function createHostLifecycle({ gate, interrupt, stopOperations, closeInput, waitForExit, terminate, graceMs = 2000 }) { if (!Number.isInteger(graceMs) || graceMs < 1 || graceMs > 10000) throw new TypeError('Invalid shutdown bound'); let stopping = null; @@ -20,10 +24,14 @@ export function createHostLifecycle({ gate, interrupt, stopOperations, closeInpu gate.close(); stopping = (async () => { const errors = []; - let operationsStopped = false; + let operationsStopped = false, reconciliationRequired = false; try { await bounded(interrupt, 'Turn interruption'); } catch (error) { errors.push(error.message); } try { - operationsStopped = (await bounded(stopOperations, 'Operation shutdown')) === true; + const operationResult = await bounded(stopOperations, 'Operation shutdown'); + if (operationResult && typeof operationResult === 'object') { + operationsStopped = operationResult.stopped === true; + reconciliationRequired = operationResult.reconciliationRequired === true; + } else operationsStopped = operationResult === true; if (!operationsStopped) errors.push('Operation shutdown was not confirmed'); } catch (error) { errors.push(error.message); } try { closeInput(); } catch (error) { errors.push(error.message); } @@ -35,7 +43,7 @@ export function createHostLifecycle({ gate, interrupt, stopOperations, closeInpu try { exited = (await bounded(waitForExit, 'App-server termination')) === true; } catch (error) { errors.push(error.message); } } if (!exited) errors.push('App-server exit was not confirmed'); - return { stopped: operationsStopped && exited, operationsStopped, exited, forced, errors }; + return { stopped: operationsStopped && exited, operationsStopped, reconciliationRequired, exited, forced, errors }; })(); return stopping; }, diff --git a/docs/configuration.md b/docs/configuration.md index e1797073646..f6709ab1830 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -13,6 +13,7 @@ The tracked code root contains the shared instruction, skill, documentation, wor `data/` holds durable private fleet records such as the project and secondmate registries, captain preferences, optional shared captain preferences, learnings, backlog, briefs, scout reports, and explicitly installed content-addressed extension packages under `data/extensions/packages/`. `state/` holds runtime records such as task metadata, append-only status events, endpoint signals, watcher and wake-queue coordination, inactive terminal-outcome receipts under `state/terminal-outcomes/`, enabled extension working namespaces under `state/extensions/`, away-mode state, generated Relay artifacts, parent-side remote ledger copies under `state/secondmate-summary-cache/`, one-shot Bearings reconcile requests under `state/reconcile-notify/`, private secondmate config-reread generations with their retry and quarantine state, per-task steering-inbox records under `state/.inbox/` (`bin/fm-task-inbox-lib.sh`), and parent-owned secondmate pending-reply records under `state/pending-replies/` (`bin/fm-pending-reply-lib.sh`). `config/` holds local gitignored operating choices, including explicit extension bindings under `config/extensions.d/`, and `projects/` holds the local project clones that Firstmate reads but changes only through the narrow guarded and concrete captain-approved exceptions in `AGENTS.md`. +The explicit experimental native Windows launcher adds top-level `owner-probe.json` and `owner-receipts.jsonl` records, while [`native-windows-codex.md`](native-windows-codex.md) owns their supported scope and their producer code owns record mechanics. Untracked files and directories whose names begin with `scratchpad` are also gitignored, so temporary scratch does not make porcelain-based secondmate sync guards treat a home as dirty. `bin/fm-spawn.sh` owns the base task-metadata fields it emits, while the runtime-backend section below owns backend-specific fields and selector interpretation. diff --git a/docs/native-windows-codex.md b/docs/native-windows-codex.md index 596997b1607..162245cf3c1 100644 --- a/docs/native-windows-codex.md +++ b/docs/native-windows-codex.md @@ -28,6 +28,7 @@ Only empty-fleet homes beneath the current user's Windows temporary directory ar Existing fleet metadata, projects, registrations, Relay configuration, process-event sources, non-temporary homes, and existing reparse-point ancestors are refused. The app-server thread is ephemeral, read-only, network-disabled, and approval-never; Apps, plugins, and configured MCP servers are disabled for this host and their effective catalogs are checked before readiness. Only controller-selected startup, notification check, and acknowledgement scripts receive registered native operation authority. +Those operations and their descendants are owned by the native session: deferred startup may outlive the digest shell but `/quit` cancels it without terminating independently owned workers, and unfinished startup may run again after restart. Interrupted or ambiguous acknowledgements remain preserved for evidence-based reconciliation and are never replayed or rolled back automatically. Production use remains disabled. diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 9ed3519ef8a..831f8caaef5 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -8,7 +8,7 @@ Exact task chronology, branch names, temporary homes, local paths, process ids, ## Experimental native Windows ownership candidate -Verified on 2026-09-15 and 2026-09-16 with Codex app-server 0.154.0, Windows 10.0.26200 x86_64, and the saved unelevated Windows sandbox. +This section retains refresh entry points only; no current-build output has been recorded by the verification owner. [`../native-windows-codex.md`](../native-windows-codex.md) owns current setup, safety boundaries, and supported limits for this isolated candidate. Refresh the portable request-policy regression with: @@ -17,8 +17,7 @@ Refresh the portable request-policy regression with: bash tests/fm-native-owner-tool-gate.test.sh ``` -The portable policy command above and native receipt command below remain refresh entry points. -No attributable terminal output was retained for their latest runs, so this record does not claim those results as current evidence: +Refresh native receipt persistence and operation lifetime with: ```sh bash tests/fm-native-owner-receipt-live-e2e.test.sh @@ -30,12 +29,6 @@ Refresh effective app and MCP isolation without a model turn: FM_LIVE_NATIVE_APP_POLICY=1 bash tests/fm-native-owner-app-server-policy-live-e2e.test.sh ``` -Observed terminal result: - -```text -PASS: effective app and MCP catalogs are isolated {"appsFeatureEnabled":false,"pluginsFeatureEnabled":false,"configuredMcpServers":["inherited_probe"],"enabledMcpServers":[],"exposedApps":[],"activeMcpServers":[]} -``` - Refresh the actual Windows integration with the explicit two-model-turn guard: ```sh @@ -43,27 +36,6 @@ FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_CODEX=1 \ bash tests/fm-native-owner-codex-live-e2e.test.sh ``` -Observed terminal results: - -```text -PASS: model-free registered operation bridge. -PASS: real app-server notification cycle, handling, acknowledgement, replay refusal, and foreign-thread denial. -``` - -The actual primary thread read and acknowledged one controlled inbox notification after the startup operation expired. -A new fixed operation performed each check and acknowledgement through unchanged Firstmate command owners; the queue became empty and the note moved to handled. -A repeated receipt and a request from another real thread were refused without another native operation. -The checkpoint exercised its three-second timeout followed by a drain, not immediate interrupt-driven delivery. -Both app-server threads reported read-only filesystem policy, disabled network access, and approval policy `never`; the two explicitly authorized host operations execute outside ordinary model shell tools. -Dynamic tool registration requires the experimental API capability in this version. -The actual app-server acknowledgement is also checked against the durable journal's owner generation, captured queue targets, and handled-note content hash. -The host revokes new calls before shutdown, stops only its fixed operation jobs, and confirms app-server exit through its retained process object. -Kill-on-close is applied only to fixed operation jobs, not the encompassing session or independently owned worker jobs. -Two additional model-free cases interrupt the real acknowledgement scripts after the inbox move and after the queue acknowledgement, respectively. -A new controller preserves the partial case as unresolved and reconciles the completed case from matching durable effects without replaying either mutation. -Missing, changed, malformed, reparse-point, or incomplete evidence remains unresolved; recovery does not mean retrying or undoing a partial acknowledgement. -The receipt evidence also supports multiple inbox targets and general wake records with no inbox target; an unrelated note is never consumed. - ### Explicit launcher integration Refresh the actual launcher without model turns, then optionally exercise two notification turns and active-turn cancellation: @@ -75,11 +47,9 @@ FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 FM_LIVE bash tests/fm-native-owner-launcher-live-e2e.test.sh ``` -No attributable terminal output was retained for the latest launcher runs, so this record does not claim their results as current evidence. - ## Harness detection precedence -Firstmate's own harness comes from two kinds of evidence, and `bin/fm-harness.sh` owns how they combine: an environment marker names its harness, and the nearest harness process in the parent chain proves who owns the process tree. +`bin/fm-harness.sh` owns native-owner, marker, and ancestry precedence; the evidence below covers marker and ancestry only, not the experimental native candidate. A marker alone is not proof of ownership, because it is ordinary environment state that a child inherits and a terminal multiplexer can replay into an unrelated session. Verified on 2026-09-02 on Linux 7.1.12 with the portable regression, which builds every case from real renamed processes and no installed harness: diff --git a/tests/fixtures/native-owner/AppHost.mjs b/tests/fixtures/native-owner/AppHost.mjs index 02b5e57f536..0e9a61dca4f 100644 --- a/tests/fixtures/native-owner/AppHost.mjs +++ b/tests/fixtures/native-owner/AppHost.mjs @@ -1,7 +1,7 @@ // Disposable host adapter. Only this process owns the app-server connection. // Dynamic tool arguments never select a thread, executable, home, or command. import {createNotificationGate} from './codex-tool-gate.mjs'; -import {createHostLifecycle} from './host-lifecycle.mjs'; +import {createHostLifecycle,reconciliationWarning} from './host-lifecycle.mjs'; import {discoverMcpServerNames,isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from './app-server-policy.mjs'; import fs from 'node:fs'; import path from 'node:path'; @@ -41,6 +41,7 @@ async function operation(action,extra={}) { if(action==='ack'&&process.env.FM_PROBE_ACK_FAULT&&fs.existsSync(path.join(home,'ack-fault-ready'))) { const stopped=await native('shutdown'); if(stopped.operationState!=='stopped')throw Error('Interrupted operation did not stop'); + if(!stopped.reconciliationRequired)throw Error('Interrupted acknowledgement did not require reconciliation'); return {operationState:'interrupted'}; } const result=await native('result'); @@ -136,12 +137,13 @@ try { const lifecycle=createHostLifecycle({ gate:{close:()=>gate?.close()}, interrupt:()=>{if(activeTurn&&alive)void request('turn/interrupt',{threadId:primary,turnId:activeTurn}).catch(()=>{});}, - stopOperations:async signal=>(await native('shutdown',{},signal)).operationState==='stopped', + stopOperations:async signal=>{const value=await native('shutdown',{},signal);return {stopped:value.operationState==='stopped',reconciliationRequired:value.reconciliationRequired===true};}, closeInput:()=>child.stdin.end(), waitForExit:signal=>!alive?Promise.resolve(true):new Promise(resolve=>{const stop=()=>{child.removeListener('exit',exit);resolve(false);};const exit=()=>{signal.removeEventListener('abort',stop);resolve(true);};child.once('exit',exit);signal.addEventListener('abort',stop,{once:true});}), terminate:()=>child.kill(),graceMs:5000, }); evidence.shutdown=await lifecycle.shutdown(); + if(evidence.shutdown.reconciliationRequired)console.error(reconciliationWarning(path.join(process.env.FM_HOME,'owner-receipts.jsonl'))); if(!evidence.shutdown.stopped){evidence.passed=false;process.exitCode=1;} clearTimeout(timer); save();fs.writeFileSync(path.join(home,'app-server.stderr'),stderr); diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index e1243a6dace..195489dac27 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -291,6 +291,11 @@ public static int Main(string[] args) { if(TryOwnerCommand(args,out ownerResult)) return ownerResult; if(args.Length>0 && args[0]=="client") return Client(args.Length>1 ? args[1] : "agent-tool"); if(args.Length==2 && args[0]=="sleep") { int ms=int.Parse(args[1]); if(ms<0 || ms>15000) throw new ArgumentException("Sleep must be bounded"); Thread.Sleep(ms); return 0; } + if(args.Length==2 && args[0]=="operation-parent") { + using(var descendant=Process.Start(new ProcessStartInfo(OwnExe,"sleep 10000") {UseShellExecute=false})) { + File.WriteAllText(args[1],descendant.Id.ToString());descendant.WaitForExit();return descendant.ExitCode; + } + } if(args.Length==3 && args[0]=="lease-check") return LeaseCheck(args[1],args[2]); if(args.Length==2 && args[0]=="notification-operation") { if(args[1]!="check" && args[1]!="ack") throw new ArgumentException("Unsupported notification operation"); diff --git a/tests/fixtures/native-owner/OperationLifetimeTests.cs b/tests/fixtures/native-owner/OperationLifetimeTests.cs index cff46072573..55f5079ac36 100644 --- a/tests/fixtures/native-owner/OperationLifetimeTests.cs +++ b/tests/fixtures/native-owner/OperationLifetimeTests.cs @@ -4,9 +4,9 @@ using System.Runtime.InteropServices; using System.Text; public static partial class NativeOwner { - static PI LifetimeChild(IntPtr job) { + static PI LifetimeChild(IntPtr job,string marker) { PI child=new PI();var startup=new SI { cb=Marshal.SizeOf(typeof(SI)) }; - if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" sleep 10000"),IntPtr.Zero,IntPtr.Zero,false,0x4|0x400,IntPtr.Zero,Path.GetTempPath(),ref startup,out child)) throw Error("Create lifetime child"); + if(!CreateProcess(OwnExe,new StringBuilder(Quote(OwnExe)+" operation-parent "+Quote(marker)),IntPtr.Zero,IntPtr.Zero,false,0x4|0x400,IntPtr.Zero,Path.GetTempPath(),ref startup,out child)) throw Error("Create lifetime child"); try { if(!AssignProcessToJobObject(job,child.process)) throw Error("Assign lifetime child"); if(ResumeThread(child.thread)==0xffffffff) throw Error("Resume lifetime child"); @@ -17,19 +17,28 @@ static int TestOperationLifetime() { using(var independent=Process.Start(new ProcessStartInfo(OwnExe,"sleep 15000") { UseShellExecute=false })) { try { foreach(bool close in new [] {false,true}) { - IntPtr job=CreateJobObject(IntPtr.Zero,null);PI child=new PI(); + IntPtr job=CreateJobObject(IntPtr.Zero,null);PI child=new PI();Process descendant=null; + string marker=Path.Combine(Path.GetTempPath(),"fm-native-operation-worker-"+Guid.NewGuid().ToString("N")); if(job==IntPtr.Zero) throw Error("Create lifetime job"); try { - NativeOperationLifetime.Configure(job);child=LifetimeChild(job); + NativeOperationLifetime.Configure(job);child=LifetimeChild(job,marker); + DateTime limit=DateTime.UtcNow.AddSeconds(3); + while(!File.Exists(marker) && DateTime.UtcNowfs.readFileSync(path.join(spec.leaseHome,'owner-receipts.jsonl'),'utf8').trim().split('\n').map(JSON.parse); const historyBefore=journal(); if(historyBefore.at(-1).event!=='ack-started'||!native.receiptNeedsReconciliation)throw Error('Interrupted intent was not preserved'); + if(!host.shutdown.reconciliationRequired||!run.stderr.includes('Acknowledgement completion is unconfirmed. Its records are preserved and require reconciliation:'))throw Error('Shutdown did not surface the preserved reconciliation requirement'); if((before.length===0)!==(fault==='complete'))throw Error('Fault did not land at the requested mutation boundary'); const recovery=path.join(home,'recovery');fs.mkdirSync(recovery); const recoverySpec={home:recovery,leaseHome:spec.leaseHome,executable:build.binary,arguments:'sleep 100',timeoutSeconds:10,pipeAcl:'UserOnly'}; diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index 15f079cd88e..1fdf9d45578 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -18,7 +18,7 @@ assert.deepEqual(Object.keys(host.externalTools).sort(),['activeMcpServers','app assert.equal(host.externalTools.appsFeatureEnabled,false);assert.equal(host.externalTools.pluginsFeatureEnabled,false); assert.ok(Array.isArray(host.externalTools.configuredMcpServers));assert.deepEqual(host.externalTools.enabledMcpServers,[]); assert.deepEqual(host.externalTools.exposedApps,[]);assert.deepEqual(host.externalTools.activeMcpServers,[]); -assert.deepEqual(host.shutdown,{stopped:true,operationsStopped:true,exited:true,forced:false,errors:[]}); +assert.deepEqual(host.shutdown,{stopped:true,operationsStopped:true,reconciliationRequired:false,exited:true,forced:false,errors:[]}); assert.notEqual(host.primary,host.foreign);assert.equal(host.tools.length,4); const requests=host.frames.filter(frame=>frame.method==='item/tool/call'); assert.equal(requests.length,4); diff --git a/tests/fixtures/native-owner/host-lifecycle.test.mjs b/tests/fixtures/native-owner/host-lifecycle.test.mjs index fce0fcf05af..9e8f4323cb1 100644 --- a/tests/fixtures/native-owner/host-lifecycle.test.mjs +++ b/tests/fixtures/native-owner/host-lifecycle.test.mjs @@ -1,6 +1,6 @@ import test from 'node:test'; import assert from 'node:assert/strict'; -import {createHostLifecycle} from '../../../bin/native-owner/host-lifecycle.mjs'; +import {createHostLifecycle,reconciliationWarning} from '../../../bin/native-owner/host-lifecycle.mjs'; function setup(overrides = {}) { const events = []; const lifecycle = createHostLifecycle({ @@ -20,7 +20,14 @@ test('shutdown revokes synchronously and is single-flight', async () => { assert.deepEqual(events, ['revoke']); const result = await first; assert.deepEqual(events, ['revoke', 'interrupt', 'operations', 'eof', 'exited']); - assert.deepEqual(result, {stopped:true, operationsStopped:true, exited:true, forced:false, errors:[]}); + assert.deepEqual(result, {stopped:true, operationsStopped:true, reconciliationRequired:false, exited:true, forced:false, errors:[]}); +}); +test('shutdown preserves a reconciliation requirement after operations stop', async () => { + const {lifecycle} = setup({stopOperations: async () => ({stopped:true,reconciliationRequired:true})}); + const result = await lifecycle.shutdown(); + assert.equal(result.stopped, true); + assert.equal(result.reconciliationRequired, true); + assert.match(reconciliationWarning('C:\\home\\owner-receipts.jsonl'), /completion is unconfirmed.*preserved.*require reconciliation/); }); test('unconfirmed operation shutdown is not reported as success', async () => { const {lifecycle, events} = setup({stopOperations: async () => false}); diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index 291919ad668..a05ba16e72e 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -21,6 +21,12 @@ test('quiet checks create no receipt and permit a later delivery',async()=>{ assert.equal((await gate.handle(check({callId:'later'}))).value.receipt,'receipt'); assert.equal(calls.length,2); }); +test('startup in progress is explicit and a queued note remains deliverable',async()=>{ + let count=0;const {gate,calls}=fixture(()=>++count===1?{operationState:'starting'}:{operationState:'delivered',notification:message}); + assert.deepEqual(await gate.handle(check()),{success:false,value:{unavailable:'startup-in-progress'}}); + const delivered=await gate.handle(check({callId:'after-startup'})); + assert.equal(delivered.success,true);assert.equal(delivered.value.receipt,'receipt');assert.equal(calls.length,2); +}); test('a cancelled quiet check cannot authorize another turn',async()=>{ let finish;const {gate}=fixture(()=>new Promise(resolve=>{finish=resolve;})); const pending=gate.handle(check());gate.endTurn('primary','turn');finish({operationState:'quiet'}); From 89b0aba6a4532d45c9513d303ac412bd62b70203 Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 18:01:12 +1200 Subject: [PATCH 14/61] no-mistakes(review): Fix native notification recovery and isolation semantics --- bin/fm-session-lock-lib.sh | 32 +++++---- .../NativeAcknowledgementEvidence.cs | 38 +++++++++-- bin/native-owner/app-server-policy.mjs | 14 +++- bin/native-owner/codex-host.mjs | 8 ++- tests/fixtures/native-owner/AppHost.mjs | 10 ++- tests/fixtures/native-owner/Build.ps1 | 5 +- tests/fixtures/native-owner/Launcher.mjs | 34 ++++++++-- tests/fixtures/native-owner/NativeDriver.cs | 6 ++ tests/fixtures/native-owner/ReceiptTests.cs | 65 +++++++++++++++++++ tests/fixtures/native-owner/Run-Cycle.mjs | 25 ++++++- .../native-owner/app-server-policy.test.mjs | 24 ++++++- tests/fixtures/native-owner/exercise.sh | 7 ++ .../native-owner/notification-check.sh | 12 ++-- tests/fixtures/native-owner/zero-recovery.sh | 30 +++++++++ tests/fm-native-owner-codex-live-e2e.test.sh | 1 + 15 files changed, 274 insertions(+), 37 deletions(-) create mode 100644 tests/fixtures/native-owner/zero-recovery.sh diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index fa7a31ef271..08b96df23ce 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -1,8 +1,10 @@ #!/usr/bin/env bash # Shared session-lock harness identity. # -# ONE owner of the "which verified-harness process holds this home's session -# lock, and does the current process descend from that same harness?" decision. +# ONE owner of the "which verified-harness identity holds this home's session +# lock, and does the current session prove that same ownership?" decision. +# A session owner identity is a numeric local pid, a tagged Windows pid, or an +# opaque native: value; process-introspection helpers remain numeric. # bin/fm-lock.sh uses it to acquire and inspect state/.lock; # bin/fm-claude-stop-autoarm.sh uses it to prove a Stop hook fires inside the # lock-owning primary session before it may arm or rewake. @@ -273,8 +275,10 @@ fm_win_harness_ancestry_pids() { return 1 } -# Walk the current process ancestry (up to 16 hops) and print this session's -# contiguous verified-harness ancestry, innermost pid first. +# Print this session's verified owner identities, innermost first. POSIX and +# tagged-Windows routes walk at most 16 ancestry hops and print pids; the native +# route prints the single opaque native: identity registered for the +# home instead of treating it as a pid. # # The walk climbs freely until the first harness match, because the caller is # normally an ordinary shell several levels below its session. After that first @@ -324,12 +328,12 @@ fm_harness_ancestry_pids() { [ "$printed" -eq 1 ] } -# Print the one pid that identifies this session when the session lock is being -# WRITTEN: the outermost pid of the contiguous run. That is the pid that lives as +# Print the one owner identity written to the session lock: the opaque native +# generation, or the outermost pid of the contiguous run. The latter lives as # long as the session - a Claude worker several levels in is reaped when its hook # returns, and a lock naming it would look stale moments later while the session -# is still running. Every non-Claude harness reports a single pid, so this is its -# innermost match unchanged. +# is still running. Every non-Claude, non-native route reports a single pid, so +# this is its innermost match unchanged. fm_harness_ancestry_pid() { local pids pid outermost='' pids=$(fm_harness_ancestry_pids) || return 1 @@ -342,7 +346,9 @@ EOF printf '%s\n' "$outermost" } -# True if $1 is a live process that looks like a verified harness. +# Classify a session-lock owner identity: 0 positively live, 1 proven dead, or 2 +# unknown. POSIX and tagged-Windows pid routes can prove only live or dead; the +# native generation route preserves unreadable or ambiguous ownership as unknown. # A tagged Windows pid is answered from the Windows process table, because # kill -0 cannot see across that boundary and would report a live harness as # dead - which would hand a running session's home to a second one. @@ -363,6 +369,8 @@ fm_harness_pid_alive() { fm_harness_process_matches "$comm" "$args" } +# Test exclusion rather than positive health: return 0 for a live or unknown +# owner identity, and 1 only when the owner is proven dead. fm_harness_pid_excludes() { local pid=$1 owner_rc case "$pid" in native:*) @@ -373,14 +381,14 @@ fm_harness_pid_excludes() { fm_harness_pid_alive "$pid" } -# True when state dir $1 holds a session lock whose pid is ANY harness ancestor -# of the current process: this script runs inside the session that owns the +# True when state dir $1 holds the owner identity of this native session or ANY +# harness ancestor of the current process: this script runs inside the session that owns the # home's fleet lock. Membership is the honest test of that question, because the # lock owner sits at an unknown depth in a contiguous Claude run - it is the # outermost pid when the hook fires inside the session's own nested worker chain, # and an inner pid when a harness-named daemon parents the session. A missing # lock, a malformed lock, a lock held by a harness outside this ancestry, or an -# ancestry that cannot be resolved all fail closed. +# ancestry that cannot be resolved all refuse ownership. fm_session_lock_owned_by_self() { local state=$1 lock_pid pids pid native_state if fm_win_boundary_applies && FM_STATE_OVERRIDE="$state" fm_native_owner_selected; then diff --git a/bin/native-owner/NativeAcknowledgementEvidence.cs b/bin/native-owner/NativeAcknowledgementEvidence.cs index 59c73734c1b..470ce1f201b 100644 --- a/bin/native-owner/NativeAcknowledgementEvidence.cs +++ b/bin/native-owner/NativeAcknowledgementEvidence.cs @@ -35,6 +35,16 @@ static byte[] Read(string home,string relative) { } } static string Hash(byte[] value) { using(var hash=SHA256.Create()) return BitConverter.ToString(hash.ComputeHash(value)).Replace("-","").ToLowerInvariant(); } + static string RecoveryGeneration(Dictionary payload) { + object value; + if(!payload.TryGetValue("generation",out value) || !(value is string) || !Regex.IsMatch((string)value,@"\A[A-Za-z0-9._-]+\z")) throw new IOException("Invalid recovery generation"); + return (string)value; + } + static string RecoveryMarker(NativeHomeLease lease) { + string value=new UTF8Encoding(false,true).GetString(Read(lease.Home,Path.Combine("state",".watcher-down"))); + if(!value.EndsWith("\n",StringComparison.Ordinal) || value.IndexOf('\n')!=value.Length-1) throw new IOException("Invalid recovery marker"); + return value.Substring(0,value.Length-1); + } static List Rows(byte[] bytes,ulong cutoff) { string text=new UTF8Encoding(false,true).GetString(bytes); if(text.Length>0 && !text.EndsWith("\n",StringComparison.Ordinal)) throw new IOException("Incomplete wake queue"); @@ -49,9 +59,9 @@ static List Rows(byte[] bytes,ulong cutoff) { public static NativeAcknowledgementEvidence Capture(NativeHomeLease lease,Dictionary payload) { if(lease==null || !lease.IsHeld) throw new InvalidOperationException("An active home lease is required"); ulong cutoff; - if(!payload.ContainsKey("seq") || !(payload["seq"] is string) || !ulong.TryParse((string)payload["seq"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff) || cutoff==0) throw new IOException("Invalid wake cutoff"); - var rows=Rows(Read(lease.Home,Path.Combine("state",".wake-queue")),cutoff); - if(rows.Count==0) throw new IOException("No queued targets remain"); + if(!payload.ContainsKey("seq") || !(payload["seq"] is string) || !ulong.TryParse((string)payload["seq"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff)) throw new IOException("Invalid wake cutoff"); + var queue=Read(lease.Home,Path.Combine("state",".wake-queue")); + var rows=Rows(queue,cutoff); var ids=new HashSet(); if(payload.ContainsKey("notes")) { var input=payload["notes"] as System.Collections.IList; @@ -61,6 +71,13 @@ public static NativeAcknowledgementEvidence Capture(NativeHomeLease lease,Dictio var queued=new HashSet(); foreach(string row in rows) {string key=row.Split(' ')[3];if(key.StartsWith("inbox:",StringComparison.Ordinal))queued.Add(key.Substring(6));} if(!ids.SetEquals(queued))throw new IOException("Inbox targets differ from the captured queue"); + if(cutoff==0) { + if(queue.Length!=0 || rows.Count!=0 || ids.Count!=0)throw new IOException("Invalid zero-row recovery target"); + string generation=RecoveryGeneration(payload),marker=RecoveryMarker(lease); + if(marker!="pending:handling:"+generation && marker!="announced:handling:"+generation)throw new IOException("Recovery target differs from the captured generation"); + return new NativeAcknowledgementEvidence(lease,new Dictionary{{"version",3},{"notes",new object[0]},{"cutoff","0"},{"rows",new object[0]},{"recoveryGeneration",generation},{"recoveryMarker",marker}}); + } + if(rows.Count==0) throw new IOException("No queued targets remain"); var notes=new List>(); foreach(string id in ids) { if(File.Exists(Path.Combine(lease.Home,"state","inbox","handled",id+".note")))throw new IOException("Inbox target is already handled or ambiguous"); @@ -74,10 +91,19 @@ internal static bool Completed(NativeHomeLease lease,Dictionary e if(lease==null || !lease.IsHeld || evidence==null) return false; try { int version=Convert.ToInt32(evidence["version"]);ulong cutoff; - if(version!=1 && version!=2)return false; - if(!ulong.TryParse((string)evidence["cutoff"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff)||cutoff==0)return false; + if(version!=1 && version!=2 && version!=3)return false; + if(!ulong.TryParse((string)evidence["cutoff"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff))return false; var targets=evidence["rows"] as System.Collections.IList; - if(targets==null||targets.Count==0)return false; + if(targets==null)return false; + if(cutoff==0) { + if(version!=3 || targets.Count!=0)return false; + var zeroNotes=evidence["notes"] as System.Collections.IList; + string generation=evidence["recoveryGeneration"] as string,marker=evidence["recoveryMarker"] as string; + if(zeroNotes==null || zeroNotes.Count!=0 || generation==null || !Regex.IsMatch(generation,@"\A[A-Za-z0-9._-]+\z"))return false; + if(marker!="pending:handling:"+generation && marker!="announced:handling:"+generation)return false; + return RecoveryMarker(lease)=="acked:handling:"+generation && Read(lease.Home,Path.Combine("state",".wake-queue")).Length==0; + } + if(version==3 || targets.Count==0)return false; var saved=new List();foreach(object target in targets){if(!(target is string))return false;saved.Add((string)target);} var validated=Rows(new UTF8Encoding(false,true).GetBytes(string.Join("\n",saved.ToArray())+"\n"),cutoff); if(validated.Count!=targets.Count)return false; diff --git a/bin/native-owner/app-server-policy.mjs b/bin/native-owner/app-server-policy.mjs index b51edfb57bb..67a4aaf83a7 100644 --- a/bin/native-owner/app-server-policy.mjs +++ b/bin/native-owner/app-server-policy.mjs @@ -1,6 +1,16 @@ import {spawn} from 'node:child_process'; const MAX_MCP_STATUS_PAGES=256; +const MCP_RUNTIME_STATUS_ACTIVE=new Map([ + [null,false], + ['notStarted',true], + ['starting',true], + ['connected',true], + ['authenticationRequired',true], + ['failed',true], + ['cancelled',true], + ['disabled',false], +]); function isRecord(value) { return value!==null&&typeof value==='object'&&!Array.isArray(value); @@ -12,7 +22,7 @@ function validInstalledApp(value) { function validMcpServerStatus(value) { return isRecord(value)&&typeof value.name==='string'&& - (value.runtimeStatus===null||['notStarted','starting','connected','authenticationRequired','failed','cancelled','disabled'].includes(value.runtimeStatus))&& + MCP_RUNTIME_STATUS_ACTIVE.has(value.runtimeStatus)&& (value.pluginId===null||typeof value.pluginId==='string')&& (value.serverInfo===null||isRecord(value.serverInfo))&& (value.serverCapabilities===null||isRecord(value.serverCapabilities))&& @@ -92,7 +102,7 @@ export async function verifyExternalToolIsolation(request,threadId,configuration if(!isRecord(installed)||!Array.isArray(installed.apps)||installed.apps.some(app=>!validInstalledApp(app)))throw Error('Invalid installed app catalog response'); const apps=installed.apps; const mcpServers=await readMcpServerStatuses(request); - const activeMcpServers=mcpServers.filter(server=>server.runtimeStatus!==null||server.serverInfo!==null||server.serverCapabilities!==null||server.toolsError!==null||Object.keys(server.tools).length||server.resources.length||server.resourceTemplates.length); + const activeMcpServers=mcpServers.filter(server=>MCP_RUNTIME_STATUS_ACTIVE.get(server.runtimeStatus)||server.serverInfo!==null||server.serverCapabilities!==null||server.toolsError!==null||Object.keys(server.tools).length||server.resources.length||server.resourceTemplates.length); const result={ ...configuration, exposedApps:apps.map(app=>app.id), diff --git a/bin/native-owner/codex-host.mjs b/bin/native-owner/codex-host.mjs index 16e79fa884c..21ab8101f20 100644 --- a/bin/native-owner/codex-host.mjs +++ b/bin/native-owner/codex-host.mjs @@ -103,6 +103,7 @@ async function turn(text){ const result=turns.get(id); if(!closing&&(!result||!['completed','interrupted'].includes(result.status)))throw Error('The model turn did not complete'); process.stdout.write('\n'); + return result; } function fail(error){if(!terminal){terminal=error;console.error(error.message);process.exitCode=1;}void stop();} function stop(){ @@ -175,7 +176,12 @@ try { if(input.length){await turn(input.shift());continue;} if(ended)break; const result=await operation('check'); - if(result.operationState==='delivered'&&result.notification.receipt!==announced){announced=result.notification.receipt;await turn('A new durable notification is available. Read it with fm_notification_check, handle it within the available authority, and acknowledge only if fully handled.');} + if(result.operationState==='delivered'&&result.notification.receipt!==announced){ + const receipt=result.notification.receipt; + const handled=await turn('A new durable notification is available. Read it with fm_notification_check, handle it within the available authority, and acknowledge only if fully handled.'); + if(!closing&&alive&&handled?.status==='completed')announced=receipt; + else if(!closing&&alive&&handled?.status==='interrupted')await pause(1000); + } else await pause(1000); } } diff --git a/tests/fixtures/native-owner/AppHost.mjs b/tests/fixtures/native-owner/AppHost.mjs index 0e9a61dca4f..6dee31e3314 100644 --- a/tests/fixtures/native-owner/AppHost.mjs +++ b/tests/fixtures/native-owner/AppHost.mjs @@ -94,7 +94,15 @@ async function turn(threadId,text) { throw Error('Turn exceeded bounded wait'); } try { - await request('initialize',{clientInfo:{name:'firstmate-scoped-notification-test',version:'0.0.0'},capabilities:{experimentalApi:true}}); + if(process.env.FM_PROBE_STARTUP_QUEUED==='1') { + const status=await native('status'),unavailable=await native('check'); + const note=process.env.FM_PROBE_STARTUP_NOTE; + const preserved=fs.existsSync(path.join(process.env.FM_HOME,'state','inbox',note+'.note')); + if(status.operationState!=='starting'||unavailable.operationState!=='busy'||!preserved)throw Error('Queued startup notification was not preserved while work was unavailable'); + evidence.startupQueued={status:status.operationState,unavailable:unavailable.operationState,preserved,note};save(); + fs.writeFileSync(path.join(home,'startup-unavailable-observed'),'observed'); + } + await request('initialize',{clientInfo:{name:'firstmate-scoped-notification-test',version:'0.0.0'},capabilities:{experimentalApi:true}}); send({method:'initialized',params:{}}); let ready=false; for(let i=0;i<1200;i++){const state=await native('result');if(state.startupExpired){ready=true;break;}await pause(100);} diff --git a/tests/fixtures/native-owner/Build.ps1 b/tests/fixtures/native-owner/Build.ps1 index 0794457b2b6..8a1f7dcbd21 100644 --- a/tests/fixtures/native-owner/Build.ps1 +++ b/tests/fixtures/native-owner/Build.ps1 @@ -10,8 +10,11 @@ $binary = Join-Path $root 'SessionProbe.exe' $sources = @((Join-Path $repo 'bin/native-owner/NativeOwner.cs'), (Join-Path $repo 'bin/native-owner/NativeHomeLease.cs'), (Join-Path $PSScriptRoot 'NativeDriver.cs'), (Join-Path $repo 'bin/native-owner/NativeReceiptJournal.cs'), (Join-Path $PSScriptRoot 'ReceiptTests.cs')) $sources += @((Join-Path $repo 'bin/native-owner/NativeOperations.cs'), (Join-Path $repo 'bin/native-owner/NativeAcknowledgementEvidence.cs'), (Join-Path $repo 'bin/native-owner/NativeOperationLifetime.cs'), (Join-Path $PSScriptRoot 'OperationLifetimeTests.cs')) Add-Type -Path $sources -OutputAssembly $binary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll +$env:FM_PROBE_CODE_ROOT = $repo & $binary receipt-tests -if ($LASTEXITCODE -ne 0) { throw 'Durable receipt lifecycle tests failed' } +$receiptExit = $LASTEXITCODE +Remove-Item Env:FM_PROBE_CODE_ROOT +if ($receiptExit -ne 0) { throw 'Durable receipt lifecycle tests failed' } & $binary environment-tests if ($LASTEXITCODE -ne 0) { throw 'Native environment filtering tests failed' } $invalidState = Join-Path $root 'missing-state' diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 1e1c2cd6590..546b040b6cd 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -115,10 +115,36 @@ if(live){ const host=read(path.join(runtime,'host.json'));assert.equal(host.turns.length,2); assert(host.tools.some(tool=>tool.tool==='fm_notification_check'&&tool.success));assert(host.tools.some(tool=>tool.tool==='fm_notification_ack'&&tool.success)); for(const note of notes)assert(fs.existsSync(path.join(messageHome,'state/inbox/handled',note+'.note'))); - assert.equal(fs.readFileSync(path.join(messageHome,'state/.wake-queue'),'utf8').trim(),''); - fs.writeFileSync(path.join(runtime,'console.json'),JSON.stringify(result,null,2)); - records.push('two real post-startup notification cycles were automatically delivered, observed, and acknowledged'); - const cancelHome=path.join(area,'live-cancel');const pendingNote=enqueue(cancelHome,'Cancellation test: leave this notification pending; do not acknowledge it.'); + assert.equal(fs.readFileSync(path.join(messageHome,'state/.wake-queue'),'utf8').trim(),''); + fs.writeFileSync(path.join(runtime,'console.json'),JSON.stringify(result,null,2)); + records.push('two real post-startup notification cycles were automatically delivered, observed, and acknowledged'); + const retryHome=path.join(area,'live-interrupted-redelivery'); + const retryNote=enqueue(retryHome,'Interrupted automatic handling test: read and acknowledge this notification when handling resumes.'); + const retry=start(retryHome,false);const retryReady=await ready(retry);let interruptSent=false,interrupted=false,completed=false; + for(let i=0;i<1800;i++){ + if(retry.child.exitCode!==null)throw Error(JSON.stringify(await retry.done)); + const host=read(path.join(retryReady.runtime,'host.json')); + if(host.activeTurn){retry.child.stdin.write('/interrupt\n');interruptSent=true;break;} + await sleep(50); + } + assert(interruptSent,'No automatic notification turn became active for interruption'); + for(let i=0;i<1800;i++){ + if(retry.child.exitCode!==null)throw Error(JSON.stringify(await retry.done)); + const turns=read(path.join(retryReady.runtime,'host.json')).turns; + interrupted=turns.some(turn=>turn.status==='interrupted');completed=interrupted&&turns.some(turn=>turn.status==='completed'); + if(completed&&fs.existsSync(path.join(retryHome,'state/inbox/handled',retryNote+'.note')))break; + await sleep(100); + } + assert(interrupted,'The automatic notification turn was not interrupted'); + assert(completed,'The interrupted notification was not offered to a later automatic turn'); + await sleep(1500); + assert.deepEqual(read(path.join(retryReady.runtime,'host.json')).turns.map(turn=>turn.status),['interrupted','completed']); + retry.child.stdin.write('/quit\n');const retryResult=await bound(retry.done,retry,20000);assert.equal(retryResult.exit,0,retryResult.stderr); + assert.equal(read(path.join(retryReady.runtime,'host.json')).turns.length,2); + assert.equal(read(path.join(retryReady.runtime,'shutdown.json')).stopped,true); + assert.equal(fs.readFileSync(path.join(retryHome,'state/.wake-queue'),'utf8').trim(),''); + records.push('interrupted automatic handling re-offered the pending receipt once, then stopped after completion and quit'); + const cancelHome=path.join(area,'live-cancel');const pendingNote=enqueue(cancelHome,'Cancellation test: leave this notification pending; do not acknowledge it.'); const active=start(cancelHome,false);active.child.stdin.write('Call fm_notification_check once, but do not acknowledge anything. Explain what remains pending. Do not use other tools.\n'); const current=await ready(active);let began=false; for(let i=0;i<1200;i++){if(read(path.join(current.runtime,'host.json')).activeTurn){began=true;break;}await sleep(50);} diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 195489dac27..0817f0cddb7 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -169,6 +169,12 @@ static int Run(string configPath) { } values["MSYS"]="winsymlinks:nativestrict"; if(config.ContainsKey("apiDry") && (bool)config["apiDry"]) values["FM_PROBE_API_DRY"]="1"; + if(config.ContainsKey("startupQueued") && (bool)config["startupQueued"]) { + string note=config.ContainsKey("startupNote") ? (string)config["startupNote"] : null; + if(!values.ContainsKey("FM_PROBE_API_DRY") || string.IsNullOrEmpty(note)) throw new ArgumentException("Queued-startup fixture requires model-free mode and a note"); + foreach(char c in note) if(!char.IsLetterOrDigit(c) && c!='_' && c!='-') throw new ArgumentException("Invalid queued-startup note"); + values["FM_PROBE_STARTUP_QUEUED"]="1";values["FM_PROBE_STARTUP_NOTE"]=note; + } if(config.ContainsKey("ackFault")) { string fault=(string)config["ackFault"]; if(!values.ContainsKey("FM_PROBE_API_DRY") || (fault!="partial" && fault!="complete")) throw new ArgumentException("Fault injection is limited to the model-free fixture"); diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index 4db9b897df2..43180355e30 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -1,5 +1,6 @@ using System; using System.Collections.Generic; +using System.Diagnostics; using System.IO; using System.Runtime.InteropServices; using System.Security.AccessControl; @@ -24,6 +25,25 @@ static void Targets(NativeHomeLease lease) { File.WriteAllText(Pending(lease),"original captured inbox record\n"); File.WriteAllText(Queue(lease),"1\t1\tcheck\tinbox:note-id\tcaptain inbox note\n"); } + static string Quote(string value) { return "\""+value.Replace("\"","\\\"")+"\""; } + static string ZeroRecovery(NativeHomeLease lease,string action,string generation=null) { + string root=Environment.GetEnvironmentVariable("FM_PROBE_CODE_ROOT"); + if(string.IsNullOrEmpty(root))throw new InvalidOperationException("Zero-recovery fixture root is required"); + string script=Path.Combine(root,"tests","fixtures","native-owner","zero-recovery.sh"); + var start=new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script)+" "+action+" "+Quote(lease.Home)+(generation==null ? "" : " "+Quote(generation))) {UseShellExecute=false,RedirectStandardOutput=true,RedirectStandardError=true}; + start.EnvironmentVariables["MSYS"]="winsymlinks:nativestrict"; + using(var process=Process.Start(start)) { + var output=process.StandardOutput.ReadToEndAsync();var error=process.StandardError.ReadToEndAsync(); + if(!process.WaitForExit(30000)){process.Kill();throw new IOException("Zero-recovery fixture exceeded its bound");} + if(process.ExitCode!=0)throw new IOException("Zero-recovery fixture failed: "+error.Result); + return output.Result.Trim(); + } + } + static Dictionary ZeroPayload(NativeHomeLease lease) { + string[] target=ZeroRecovery(lease,"present").Split('\t'); + Expect(target.Length==2 && target[0]=="0","Real recovery owner did not produce a zero-row target"); + return new Dictionary{{"challenge","zero"},{"message","recovery"},{"seq",target[0]},{"generation",target[1]},{"notes",new string[0]}}; + } public static int Run() { Case("one writer and unobserved acknowledgement refusal",lease=>{ using(var journal=new NativeReceiptJournal(lease,A)) { @@ -175,6 +195,51 @@ public static int Run() { if(scenario=="no-inbox-targets")Expect(File.Exists(Pending(lease)),"Unrelated inbox note was consumed"); }); } + Case("zero-row recovery rejects an unproven empty target",lease=>{ + Directory.CreateDirectory(Path.GetDirectoryName(Queue(lease)));File.WriteAllText(Queue(lease),""); + var payload=new Dictionary{{"challenge","zero"},{"message","recovery"},{"seq","0"},{"generation","missing"},{"notes",new string[0]}}; + using(var journal=new NativeReceiptJournal(lease,A)) { + var delivery=journal.Present(payload); + Refuses(()=>NativeAcknowledgementEvidence.Capture(lease,delivery),"Arbitrary empty target accepted"); + Expect(!journal.NeedsReconciliation,"Rejected empty target created an attempt"); + } + }); + Case("zero-row recovery rejects a mismatched generation",lease=>{ + var payload=ZeroPayload(lease);payload["generation"]="different"; + using(var journal=new NativeReceiptJournal(lease,A)) { + var delivery=journal.Present(payload); + Refuses(()=>NativeAcknowledgementEvidence.Capture(lease,delivery),"Mismatched recovery generation accepted"); + } + }); + Case("zero-row recovery interruption preserves the obligation",lease=>{ + var payload=ZeroPayload(lease);string receipt; + using(var journal=new NativeReceiptJournal(lease,A)) { + var delivery=journal.Present(payload);receipt=(string)delivery["receipt"]; + journal.BeginAcknowledgement(receipt,"zero",NativeAcknowledgementEvidence.Capture(lease,delivery)); + } + string marker=File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")); + using(var journal=new NativeReceiptJournal(lease,B)) { + Expect(journal.ReconcileCompletedAcknowledgements()==0,"Unperformed recovery target was invented"); + Expect(journal.NeedsReconciliation,"Interrupted recovery target was lost"); + } + Expect(File.ReadAllText(Queue(lease))=="","Interrupted recovery changed the queue"); + Expect(marker==File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")),"Interrupted recovery rolled back the target"); + }); + Case("completed zero-row recovery reconciles without replay",lease=>{ + var payload=ZeroPayload(lease);string receipt,generation=(string)payload["generation"]; + using(var journal=new NativeReceiptJournal(lease,A)) { + var delivery=journal.Present(payload);receipt=(string)delivery["receipt"]; + journal.BeginAcknowledgement(receipt,"zero",NativeAcknowledgementEvidence.Capture(lease,delivery)); + } + ZeroRecovery(lease,"acknowledge",generation); + string queue=File.ReadAllText(Queue(lease)),marker=File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")); + using(var journal=new NativeReceiptJournal(lease,B)) { + Expect(journal.ReconcileCompletedAcknowledgements()==1,"Completed zero-row target was not reconciled"); + Expect(!journal.NeedsReconciliation,"Completed zero-row target remained ambiguous"); + Expect(journal.ReconcileCompletedAcknowledgements()==0,"Completed zero-row target replayed"); + } + Expect(queue==File.ReadAllText(Queue(lease)) && marker==File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")),"Reconciliation changed completed effects"); + }); Console.WriteLine("RECEIPT_TESTS_PASS "+passed);return 0; } } diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs index 0b2cc542eca..1b2d6c051e5 100644 --- a/tests/fixtures/native-owner/Run-Cycle.mjs +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -6,16 +6,30 @@ import {randomUUID,createHash} from 'node:crypto'; const repo=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..'); const dir=path.join(repo,'data/native-candidate-validation'); const read=file=>JSON.parse(fs.readFileSync(file,'utf8').replace(/^\uFEFF/,'')); +const posix=value=>value.replaceAll('\\','/').replace(/^([A-Za-z]):/,(_,drive)=>'/'+drive.toLowerCase()); const build=read(path.join(dir,'build.json')); const dry=process.argv.includes('--dry'); +const startupQueued=process.argv.includes('--startup-queued'); const fault=process.argv.find(value=>value.startsWith('--fault='))?.slice(8); if(fault&&(!dry||!['partial','complete'].includes(fault)))throw Error('Fault cases require --dry and partial or complete'); +if(startupQueued&&!dry)throw Error('Queued-startup case requires model-free mode'); if(!dry&&process.env.FM_LIVE_NATIVE_CODEX!=='1')throw Error('Live model test requires FM_LIVE_NATIVE_CODEX=1'); if(!dry){const preflight=read(path.join(dir,'bridge-preflight.json'));if(!preflight.passed||preflight.binaryHash!==createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex'))throw Error('Run model-free bridge preflight for this binary first');} const home=path.join(build.root,'appserver-'+randomUUID());fs.mkdirSync(home); fs.writeFileSync(path.join(home,'build.json'),JSON.stringify(build,null,2)); const script=path.join(build.root,'AppHost.mjs'); -const spec={home,leaseHome:path.join(home,'home'),executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'UserOnly',apiDry:dry,jqImage:process.env.FM_NATIVE_TEST_JQ_IMAGE}; +const leaseHome=path.join(home,'home'); +let startupNote=null; +if(startupQueued){ + fs.mkdirSync(leaseHome); + const env={...process.env,FM_HOME:posix(leaseHome),MSYS:'winsymlinks:nativestrict'}; + const queued=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',posix(path.join(build.root,'firstmate/bin/fm-inbox.sh')),'note','Queued before native startup readiness; preserve and deliver this exact note.'],{env,encoding:'utf8',timeout:30000}); + if(queued.status!==0)throw Error('Could not queue the pre-startup note: '+queued.stderr); + startupNote=queued.stdout.trim().split(/\s+/)[1]; + if(!startupNote)throw Error('Pre-startup note ID was not returned'); +} +const spec={home,leaseHome,executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'UserOnly',apiDry:dry,jqImage:process.env.FM_NATIVE_TEST_JQ_IMAGE}; +if(startupQueued){spec.startupQueued=true;spec.startupNote=startupNote;} if(fault)spec.ackFault=fault; const file=path.join(home,'spec.json');fs.writeFileSync(file,JSON.stringify(spec,null,2)); const run=spawnSync(build.binary,['run',file],{encoding:'utf8',timeout:310000}); @@ -27,7 +41,12 @@ const host=read(path.join(home,'app-host-evidence.json')),native=read(path.join( if(!host.shutdown?.stopped||!host.shutdown.operationsStopped||!host.shutdown.exited)throw Error('Host shutdown was not confirmed'); const starts=action=>host.native.filter(row=>row.action===action&&row.state==='pending').length; if(!host.passed||starts('check')!==1||starts('ack')!==1||native.notificationConsumed!==!fault)throw Error('Notification cycle incomplete'); -if(!host.native.filter(row=>row.action==='check'||row.action==='ack').every(row=>row.startupExpired))throw Error('Startup scope still active'); +if(!host.native.filter(row=>(row.action==='check'||row.action==='ack')&&row.state==='pending').every(row=>row.startupExpired))throw Error('Startup scope still active'); +if(startupQueued){ + const delivered=read(path.join(home,'notification-check.json')); + if(host.startupQueued?.status!=='starting'||host.startupQueued?.unavailable!=='busy'||!host.startupQueued?.preserved||host.startupQueued.note!==startupNote)throw Error('Queued-startup unavailable response was not demonstrated'); + if(delivered.note!==startupNote||!fs.existsSync(path.join(leaseHome,'state/inbox/handled',startupNote+'.note')))throw Error('Pre-startup note was not normally delivered and acknowledged'); +} if(fault) { const queue=path.join(spec.leaseHome,'state/.wake-queue'),before=fs.readFileSync(queue); const journal=()=>fs.readFileSync(path.join(spec.leaseHome,'owner-receipts.jsonl'),'utf8').trim().split('\n').map(JSON.parse); @@ -52,4 +71,4 @@ if(fault) { } if(fs.existsSync(path.join(spec.leaseHome,'state/.wake-queue'))&&fs.readFileSync(path.join(spec.leaseHome,'state/.wake-queue'),'utf8').trim())throw Error('Queue was not acknowledged'); if(dry)fs.writeFileSync(path.join(dir,'bridge-preflight.json'),JSON.stringify({passed:true,home,binaryHash:createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex')},null,2)); -console.log(dry?'PASS: model-free registered operation bridge.':'PASS: real app-server notification cycle, handling, acknowledgement, replay refusal, and foreign-thread denial.'); +console.log(startupQueued?'PASS: queued notification stayed unavailable and preserved until startup, then delivered normally.':dry?'PASS: model-free registered operation bridge.':'PASS: real app-server notification cycle, handling, acknowledgement, replay refusal, and foreign-thread denial.'); diff --git a/tests/fixtures/native-owner/app-server-policy.test.mjs b/tests/fixtures/native-owner/app-server-policy.test.mjs index 0f0adfb7b29..98d82dddf4f 100644 --- a/tests/fixtures/native-owner/app-server-policy.test.mjs +++ b/tests/fixtures/native-owner/app-server-policy.test.mjs @@ -76,9 +76,26 @@ test('a prohibited MCP capability on a later page is rejected',async()=>{ assert.equal(calls.filter(call=>call.method==='mcpServerStatus/list').length,2); }); -test('server capabilities are required and only null is inactive',async()=>{ - const valid=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[inactiveMcpServer('alpha')],nextCursor:null}}); - assert.deepEqual((await verifyExternalToolIsolation(valid.request,'thread-1',isolatedConfiguration)).activeMcpServers,[]); +test('supported runtime statuses have explicit activity semantics',async()=>{ + const semantics=new Map([[null,false],['disabled',false],['notStarted',true],['starting',true],['connected',true],['authenticationRequired',true],['failed',true],['cancelled',true]]); + for(const [runtimeStatus,active] of semantics){ + const boundary=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{...inactiveMcpServer(String(runtimeStatus)),runtimeStatus}],nextCursor:null}}); + if(active)await assert.rejects(verifyExternalToolIsolation(boundary.request,'thread-1',isolatedConfiguration),/External app-server tools are not isolated/); + else assert.deepEqual((await verifyExternalToolIsolation(boundary.request,'thread-1',isolatedConfiguration)).activeMcpServers,[]); + } +}); + +test('disabled status is inactive only when the complete response exposes nothing',async()=>{ + for(const exposed of [ + {serverInfo:{}},{serverCapabilities:{}},{toolsError:'failed to enumerate'}, {tools:{read:{}}}, {resources:[{}]}, {resourceTemplates:[{}]}, + ]){ + const server={...inactiveMcpServer('disabled'),runtimeStatus:'disabled',...exposed}; + const boundary=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[server],nextCursor:null}}); + await assert.rejects(verifyExternalToolIsolation(boundary.request,'thread-1',isolatedConfiguration),/External app-server tools are not isolated/); + } +}); + +test('server capabilities are required',async()=>{ const missing=inactiveMcpServer('missing');delete missing.serverCapabilities; const absent=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[missing],nextCursor:null}}); await assert.rejects(verifyExternalToolIsolation(absent.request,'thread-1',isolatedConfiguration),/Invalid MCP server status response/); @@ -104,6 +121,7 @@ test('malformed catalog responses are rejected',async()=>{ {'app/installed':{apps:[{}]},'mcpServerStatus/list':{data:[],nextCursor:null}}, {'app/installed':{apps:[]},'mcpServerStatus/list':{}}, {'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{}],nextCursor:null}}, + {'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{...inactiveMcpServer('future'),runtimeStatus:'future'}],nextCursor:null}}, {'app/installed':{apps:[]},'mcpServerStatus/list':{data:[],nextCursor:7}}, ]) { const {request}=requestBoundary(responses); diff --git a/tests/fixtures/native-owner/exercise.sh b/tests/fixtures/native-owner/exercise.sh index a43ecc54b83..da7d81a78cc 100644 --- a/tests/fixtures/native-owner/exercise.sh +++ b/tests/fixtures/native-owner/exercise.sh @@ -7,6 +7,13 @@ export FM_HOME FM_HOME=$(cygpath -u "${FM_HOME:?}") export PATH="$BUILD/tools:$PATH" LOG=$(cygpath -u "${FM_PROBE_HOME:?}") +if [ "${FM_PROBE_STARTUP_QUEUED:-}" = 1 ]; then + for ((i=0; i<100; i++)); do + [ -f "$LOG/startup-unavailable-observed" ] && break + sleep 0.1 + done + [ -f "$LOG/startup-unavailable-observed" ] +fi mkdir -p "$FM_HOME/state" "$FM_HOME/config" "$FM_HOME/data" # Explicit empty manual backlog avoids inventing work or invoking task migration. printf 'manual\n' > "$FM_HOME/config/backlog-backend" diff --git a/tests/fixtures/native-owner/notification-check.sh b/tests/fixtures/native-owner/notification-check.sh index 84efdf98e60..e2e7839bfdb 100644 --- a/tests/fixtures/native-owner/notification-check.sh +++ b/tests/fixtures/native-owner/notification-check.sh @@ -11,9 +11,14 @@ cd "$ROOT" . bin/fm-session-lock-lib.sh fm_session_lock_owned_by_self "$FM_HOME/state" [ -f "$LOG/owner-operation.complete" ] -bin/fm-wake-drain.sh > "$LOG/cycle-initial-drain.log" 2>&1 challenge=$(od -An -N12 -tx1 /dev/urandom | tr -d ' \n') -bin/fm-inbox.sh note "Controlled notification. Confirm observation of $challenge; no project action is requested." > "$LOG/cycle-enqueue.log" +if [ "${FM_PROBE_STARTUP_QUEUED:-}" = 1 ]; then + note=${FM_PROBE_STARTUP_NOTE:?} +else + bin/fm-wake-drain.sh > "$LOG/cycle-initial-drain.log" 2>&1 + bin/fm-inbox.sh note "Controlled notification. Confirm observation of $challenge; no project action is requested." > "$LOG/cycle-enqueue.log" + note=$(awk '/^queued / {print $2}' "$LOG/cycle-enqueue.log") +fi set +e bin/fm-watch-checkpoint.sh --seconds 3 > "$LOG/cycle-checkpoint.log" 2>&1 rc=$? @@ -22,8 +27,7 @@ case "$rc" in 0|124) ;; *) exit "$rc" ;; esac bin/fm-wake-drain.sh > "$LOG/cycle-delivery.log" 2>&1 bin/fm-inbox.sh drain > "$LOG/cycle-message.log" message=$(<"$LOG/cycle-message.log") -grep -q "$challenge" "$LOG/cycle-message.log" -note=$(awk '/^queued / {print $2}' "$LOG/cycle-enqueue.log") +if [ "${FM_PROBE_STARTUP_QUEUED:-}" != 1 ]; then grep -q "$challenge" "$LOG/cycle-message.log"; fi ack=$(grep 'WAKE_ACK_REQUIRED:' "$LOG/cycle-delivery.log" | tail -1) seq=$(awk '{for(i=1;i "$FM_HOME/state/.wake-queue" + . bin/fm-wake-lib.sh + fm_recovery_marker_publish "$FM_HOME/state/.watcher-down" downtime + error=$(mktemp "$FM_HOME/state/.zero-recovery.XXXXXX") + trap 'rm -f -- "$error"' EXIT + bin/fm-wake-drain.sh > /dev/null 2> "$error" + ack=$(grep '^WAKE_ACK_REQUIRED:' "$error" | tail -1) + seq=$(awk '{for(i=1;i Date: Wed, 16 Sep 2026 18:28:09 +1200 Subject: [PATCH 15/61] no-mistakes(review): Preserve unoffered notifications and reject populated backlogs --- bin/native-owner/NativeLauncher.cs | 10 +++++++++ bin/native-owner/codex-host.mjs | 4 ++-- bin/native-owner/codex-tool-gate.mjs | 18 ++++++++++++++- docs/native-windows-codex.md | 2 +- tests/fixtures/native-owner/Launcher.mjs | 12 +++++++++- .../fixtures/native-owner/tool-gate.test.mjs | 22 ++++++++++++++++--- 6 files changed, 60 insertions(+), 8 deletions(-) diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs index 31b763d7eec..6a05fed9082 100644 --- a/bin/native-owner/NativeLauncher.cs +++ b/bin/native-owner/NativeLauncher.cs @@ -13,9 +13,19 @@ public static partial class NativeOwner { [DllImport("kernel32.dll")] static extern IntPtr GetStdHandle(int kind); static string CodeRoot { get { return Path.GetDirectoryName(Path.GetDirectoryName(OwnExe)); } } + static void EmptyBacklog(string home) { + string data=Path.Combine(home,"data"),backlog=Path.Combine(data,"backlog.md"); + if(!Directory.Exists(data)||Directory.GetFileSystemEntries(data,"backlog.md").Length==0)return; + if(!File.Exists(backlog))throw new InvalidOperationException("This experimental launcher requires a canonical empty backlog; the existing backlog was preserved"); + string content; + try {content=File.ReadAllText(backlog,new UTF8Encoding(false,true));} + catch(Exception error){throw new InvalidOperationException("This experimental launcher could not validate the existing backlog; it was preserved",error);} + if(content!="## In flight\n\n## Queued\n\n## Done\n")throw new InvalidOperationException("This experimental launcher requires a canonical empty backlog; the existing backlog was preserved"); + } static void EmptyFleet(string home) { string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); if((Directory.Exists(state)&&Directory.GetFiles(state,"*.meta").Length!=0) || (Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || File.Exists(Path.Combine(state,"x-watch.check.sh")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); + EmptyBacklog(home); } static int Launch(string selectedHome) { string home=Path.GetFullPath(selectedHome), node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); diff --git a/bin/native-owner/codex-host.mjs b/bin/native-owner/codex-host.mjs index 21ab8101f20..1546daf96ab 100644 --- a/bin/native-owner/codex-host.mjs +++ b/bin/native-owner/codex-host.mjs @@ -6,7 +6,7 @@ import path from 'node:path'; import net from 'node:net'; import {spawn} from 'node:child_process'; import {createInterface} from 'node:readline'; -import {createNotificationGate} from './codex-tool-gate.mjs'; +import {confirmAutomaticNotificationOffer,createNotificationGate} from './codex-tool-gate.mjs'; import {createHostLifecycle,reconciliationWarning} from './host-lifecycle.mjs'; import {discoverMcpServerNames,isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from './app-server-policy.mjs'; const runtime=process.env.FM_PROBE_HOME,root=process.env.FM_PROBE_CODE_ROOT; @@ -179,7 +179,7 @@ try { if(result.operationState==='delivered'&&result.notification.receipt!==announced){ const receipt=result.notification.receipt; const handled=await turn('A new durable notification is available. Read it with fm_notification_check, handle it within the available authority, and acknowledge only if fully handled.'); - if(!closing&&alive&&handled?.status==='completed')announced=receipt; + if(!closing&&alive&&confirmAutomaticNotificationOffer(gate,thread,handled,receipt))announced=receipt; else if(!closing&&alive&&handled?.status==='interrupted')await pause(1000); } else await pause(1000); diff --git a/bin/native-owner/codex-tool-gate.mjs b/bin/native-owner/codex-tool-gate.mjs index ad14550fa29..cb362abd2e2 100644 --- a/bin/native-owner/codex-tool-gate.mjs +++ b/bin/native-owner/codex-tool-gate.mjs @@ -13,6 +13,7 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { let challenge = null; let acknowledged = false; let offered = null; + const offers = new Map(); const seen = new Set(); const retiredTurns = new Set(); const deny = reason => ({ success: false, value: { denied: reason } }); @@ -52,7 +53,10 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { if (Object.keys(args).length) return deny('invalid-arguments'); // Redelivery is read-only: cancellation or a lost response must not // strand pending work or start another native check before handling it. - if (receipt && !acknowledged) return { success: true, value: { ...offered } }; + if (receipt && !acknowledged) { + offers.set(params.turnId, receipt); + return { success: true, value: { ...offered } }; + } } else if (params.tool === 'fm_notification_ack') { if (Object.keys(args).sort().join(',') !== 'observed,receipt' || !receipt || args.receipt !== receipt || args.observed !== challenge) { @@ -85,6 +89,7 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { acknowledged = false; offered = Object.freeze({ message: note.message, receipt, challenge, checkpointExit: note.checkpointExit }); if (!valid(params)) return deny('wrong-thread-turn-or-replay'); + offers.set(params.turnId, receipt); return { success: true, value: { ...offered } }; } const result = await operate('ack', { receipt, observed: args.observed }); @@ -101,5 +106,16 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { busy = false; } }, + wasOffered(thread, turn, expectedReceipt) { + return thread === primaryThread && offers.get(turn) === expectedReceipt; + }, }); } + +export function confirmAutomaticNotificationOffer(gate, thread, turn, receipt) { + if (turn?.status !== 'completed') return false; + if (!gate?.wasOffered(thread, turn.id, receipt)) { + throw new Error('Automatic notification turn completed without receiving its pending receipt; durable work was preserved'); + } + return true; +} diff --git a/docs/native-windows-codex.md b/docs/native-windows-codex.md index 162245cf3c1..21d4d0efe52 100644 --- a/docs/native-windows-codex.md +++ b/docs/native-windows-codex.md @@ -2,7 +2,7 @@ This explicit opt-in launcher is a restricted experimental candidate, not an installed runtime backend or a production-ready integration. Ordinary startup never selects it, and it does not install hooks, alter saved or global Codex settings, pull images, or select a default backend. -[`verification/runtime-backends.md`](verification/runtime-backends.md#experimental-native-windows-ownership-candidate) records the dated empirical evidence and refresh commands. +[`verification/runtime-backends.md`](verification/runtime-backends.md#experimental-native-windows-ownership-candidate) provides the verification refresh entry points; no current-build output is recorded there yet. ## Setup diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 546b040b6cd..6abcda4fbb0 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -46,7 +46,8 @@ function enqueue(home,message){ const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export PATH="$1/bin/native-owner/tools:/usr/bin:/bin:$PATH"; export FM_HOME; FM_HOME=$(cygpath -u "$3"); exec /usr/bin/bash "$1/bin/fm-inbox.sh" note "$2"','launcher-test',posix(code),message,home],{env,encoding:'utf8',timeout:30000}); assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr}));return result.stdout.trim().split(/\s+/)[1]; } -const home=path.join(area,'home'); +const home=path.join(area,'home');fs.mkdirSync(path.join(home,'data'),{recursive:true}); +fs.writeFileSync(path.join(home,'data/backlog.md'),'## In flight\n\n## Queued\n\n## Done\n'); const first=start(home);const initial=await ready(first);console.error('first ready',area); const competitor=start(home);competitor.child.stdin.end();const refused=await bound(competitor.done,competitor,20000); console.error('competitor returned',refused);assert.notEqual(refused.exit,0);assert.equal(read(path.join(home,'owner-probe.json')).generation,initial.owner.generation); @@ -66,6 +67,15 @@ const populated=path.join(area,'populated');fs.mkdirSync(path.join(populated,'st const blocked=start(populated);blocked.child.stdin.end();assert.notEqual((await bound(blocked.done,blocked,20000)).exit,0); assert.equal(fs.readFileSync(path.join(populated,'state/work.meta'),'utf8'),'preserve');assert.equal(fs.existsSync(path.join(populated,'owner-probe.json')),false); records.push('populated home refused without changing its records'); +for(const [name,contents] of [ + ['queued-backlog','## In flight\n\n## Queued\n- [ ] queued-work - preserved project work (repo: firstmate) (kind: ship)\n\n## Done\n'], + ['unrecognized-backlog','# Backlog\n\nproject work in an unrecognized form\n'], +]){ + const backlogHome=path.join(area,name),backlog=path.join(backlogHome,'data/backlog.md');fs.mkdirSync(path.dirname(backlog),{recursive:true});fs.writeFileSync(backlog,contents); + const refusedBacklog=start(backlogHome);refusedBacklog.child.stdin.end();assert.notEqual((await bound(refusedBacklog.done,refusedBacklog,20000)).exit,0); + assert.equal(fs.readFileSync(backlog,'utf8'),contents);assert.equal(fs.existsSync(path.join(backlogHome,'owner-probe.json')),false); +} +records.push('queued and unrecognized backlogs refused before lease acquisition and preserved'); for(const [name,relative] of [['relay-config','config/x-mode.env'],['relay-watch','state/x-watch.check.sh']]){ const relayHome=path.join(area,name),record=path.join(relayHome,relative),contents='preserve relay state'; fs.mkdirSync(path.dirname(record),{recursive:true});fs.writeFileSync(record,contents); diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index a05ba16e72e..2961f1b6013 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -1,6 +1,6 @@ import test from 'node:test'; import assert from 'node:assert/strict'; -import {createNotificationGate} from '../../../bin/native-owner/codex-tool-gate.mjs'; +import {confirmAutomaticNotificationOffer,createNotificationGate} from '../../../bin/native-owner/codex-tool-gate.mjs'; const message={receipt:'receipt',challenge:'observed',message:'Controlled message',checkpointExit:124}; function fixture(operate) { const calls=[];let alive=true; @@ -103,6 +103,22 @@ test('notification arriving after cancellation remains available next turn',asyn assert.equal(recovered.success,true);assert.equal(recovered.value.receipt,'receipt');assert.equal(calls.length,1); }); test('operation failure is not reported as success and cannot be blindly retried',async()=>{ - const {gate,calls}=fixture(()=>{throw Error('partial operation requires reconciliation');}); - assert.equal((await gate.handle(check())).success,false);assert.equal((await gate.handle(check({callId:'retry'}))).success,false);assert.equal(calls.length,1); + const {gate,calls}=fixture(()=>{throw Error('partial operation requires reconciliation');}); + assert.equal((await gate.handle(check())).success,false);assert.equal((await gate.handle(check({callId:'retry'}))).success,false);assert.equal(calls.length,1); +}); +test('completed prose-only automatic turn preserves the unoffered receipt',()=>{ + const {gate}=fixture();gate.endTurn('primary','turn'); + assert.throws(()=>confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt'),/durable work was preserved/); +}); +for(const [name,request] of [ + ['denied',check({namespace:'other'})], + ['malformed',check({arguments:null})], +])test(`completed automatic turn with a ${name} check preserves the receipt`,async()=>{ + const {gate}=fixture();assert.equal((await gate.handle(request)).success,false);gate.endTurn('primary','turn'); + assert.throws(()=>confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt'),/durable work was preserved/); +}); +test('completed automatic turn suppresses only the exact successfully offered receipt',async()=>{ + const {gate}=fixture();assert.equal((await gate.handle(check())).success,true);gate.endTurn('primary','turn'); + assert.equal(confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt'),true); + assert.throws(()=>confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'other')); }); From b47d18f0b6ad5647620a3ca37b3f97eaad0a6016 Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 22:29:36 +1200 Subject: [PATCH 16/61] no-mistakes(review): Consolidate native admission, recovery, and host-loop coverage --- bin/fm-backlog-transition-lib.sh | 82 +++++ bin/fm-wake-lib.sh | 322 ++++++++++++++++++ .../NativeAcknowledgementEvidence.cs | 159 +++------ bin/native-owner/NativeLauncher.cs | 21 +- bin/native-owner/NativeReceiptJournal.cs | 6 +- bin/native-owner/ack-evidence.sh | 120 +++++++ bin/native-owner/ack.sh | 14 +- bin/native-owner/admit.sh | 12 + bin/native-owner/check.sh | 13 +- bin/native-owner/codex-host-runtime.mjs | 209 ++++++++++++ bin/native-owner/codex-host.mjs | 194 +---------- docs/native-windows-codex.md | 37 +- docs/verification/runtime-backends.md | 41 --- tests/fixtures/native-owner/Launcher.mjs | 12 + tests/fixtures/native-owner/ReceiptTests.cs | 3 + tests/fixtures/native-owner/Run-Cycle.mjs | 5 +- tests/fixtures/native-owner/Verify-Cycle.mjs | 6 +- .../fixtures/native-owner/fake-app-server.mjs | 58 ++++ .../fixtures/native-owner/tool-gate.test.mjs | 85 ++++- 19 files changed, 1001 insertions(+), 398 deletions(-) create mode 100644 bin/native-owner/ack-evidence.sh create mode 100644 bin/native-owner/admit.sh create mode 100644 bin/native-owner/codex-host-runtime.mjs create mode 100644 tests/fixtures/native-owner/fake-app-server.mjs diff --git a/bin/fm-backlog-transition-lib.sh b/bin/fm-backlog-transition-lib.sh index c116d016b0e..cc4e3897579 100644 --- a/bin/fm-backlog-transition-lib.sh +++ b/bin/fm-backlog-transition-lib.sh @@ -72,6 +72,7 @@ FM_BACKLOG_ROW_HOLD_KIND= # retained_incomplete | answered | stale | noop. # shellcheck disable=SC2034 # Output global, read by the sourcing caller. FM_BACKLOG_CLOSE_REPLAY_RESULT= +FM_BACKLOG_EMPTY_ERROR= # Bounded execution is fm-timeout-lib.sh's alone; source it rather than # re-deriving a deadline here. It is stateless, so the memoisation reason this @@ -149,6 +150,87 @@ fm_backlog_file() { # fi } +fm_backlog_markdown_empty() { # + local file=$1 + FM_BACKLOG_EMPTY_ERROR= + if [ ! -r "$file" ] || [ ! -f "$file" ] || [ -L "$file" ]; then + FM_BACKLOG_EMPTY_ERROR="backlog is not a readable regular file at $file" + return 1 + fi + if ! LC_ALL=C awk ' + { + sub(/\r$/, "") + if (length($0) != 0) lines[++count]=$0 + } + END { + start=1 + if (lines[1] == "# Backlog") start=2 + if (count == 1 && start == 2) exit 0 + if (count - start + 1 != 3) exit 1 + if (lines[start] != "## In flight") exit 1 + if (lines[start + 1] != "## Queued") exit 1 + if (lines[start + 2] != "## Done") exit 1 + } + ' "$file"; then + FM_BACKLOG_EMPTY_ERROR="backlog contains work or an unrecognized empty skeleton at $file" + return 1 + fi +} + +fm_backlog_empty_fleet_preflight() { # + local state=$1 data=$2 record root backend file + FM_BACKLOG_EMPTY_ERROR= + if [ -e "$state" ] || [ -L "$state" ]; then + if ! fm_backlog_directory_present "$state" "state directory"; then + FM_BACKLOG_EMPTY_ERROR=$FM_BACKLOG_TRANSITION_ERROR + return 1 + fi + for record in "$state"/*.meta "$state"/*.status "$state"/*.backlog-close; do + if [ -e "$record" ] || [ -L "$record" ]; then + FM_BACKLOG_EMPTY_ERROR="work-bearing task record is present at $record" + return 1 + fi + done + fi + if [ ! -e "$data" ] && [ ! -L "$data" ]; then + root=${data%/*} + [ -n "$root" ] || root=/ + if [ -e "$root/.tasks.toml" ] || [ -L "$root/.tasks.toml" ]; then + FM_BACKLOG_EMPTY_ERROR="the experimental empty-fleet preflight cannot validate a configured backlog without its data directory" + return 1 + fi + return 0 + fi + if ! fm_backlog_directory_present "$data" "data directory"; then + FM_BACKLOG_EMPTY_ERROR=$FM_BACKLOG_TRANSITION_ERROR + return 1 + fi + root=$(fm_backlog_root "$data") || { + FM_BACKLOG_EMPTY_ERROR=${FM_BACKLOG_TRANSITION_ERROR:-"data directory cannot be resolved: $data"} + return 1 + } + if ! fm_backlog_config_present "$root" "$(fm_backlog_authorized_root "$data")"; then + FM_BACKLOG_EMPTY_ERROR=$FM_BACKLOG_TRANSITION_ERROR + return 1 + fi + backend=$(fm_tasks_axi_backend "$root" 2>&1) || { + FM_BACKLOG_EMPTY_ERROR=$backend + return 1 + } + if [ "$backend" != markdown ]; then + FM_BACKLOG_EMPTY_ERROR="the experimental empty-fleet preflight does not accept a non-markdown backlog backend" + return 1 + fi + file=$(fm_backlog_file "$data") || { + FM_BACKLOG_EMPTY_ERROR=${FM_BACKLOG_TRANSITION_ERROR:-"backlog path cannot be resolved"} + return 1 + } + if [ ! -e "$file" ] && [ ! -L "$file" ]; then + return 0 + fi + fm_backlog_markdown_empty "$file" +} + # The directory a backlog's own `.tasks.toml` is resolved from. fm_backlog_root() { # local data parent diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index e3582cbc4da..789c018ccee 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -2063,6 +2063,328 @@ fm_wake_actor_pending_count() { # [ ] printf '%s\n' "$count" } +FM_WAKE_ACK_EVIDENCE_TOKEN= +FM_WAKE_ACK_EVIDENCE_CUTOFF= +FM_WAKE_ACK_EVIDENCE_GENERATION= +FM_WAKE_ACK_EVIDENCE_MARKER= +FM_WAKE_ACK_EVIDENCE_ROWS= +FM_WAKE_ACK_EVIDENCE_NOTES= +FM_WAKE_ACK_EVIDENCE_LEGACY=0 + +fm_wake_ack_hash() { # + if command -v sha256sum >/dev/null 2>&1; then + sha256sum "$1" | awk '{print $1}' + elif command -v shasum >/dev/null 2>&1; then + shasum -a 256 "$1" | awk '{print $1}' + else + return 1 + fi +} + +fm_wake_ack_note_safe() { # + local path=$1 + [ -d "$STATE/inbox" ] && [ ! -L "$STATE/inbox" ] || return 1 + if [ -e "$STATE/inbox/handled" ] || [ -L "$STATE/inbox/handled" ]; then + [ -d "$STATE/inbox/handled" ] && [ ! -L "$STATE/inbox/handled" ] || return 1 + fi + case "$path" in + "$STATE/inbox/handled"/*) [ -d "$STATE/inbox/handled" ] || return 1 ;; + esac + [ -f "$path" ] && [ ! -L "$path" ] +} + +fm_wake_ack_evidence_clear() { + [ -z "$FM_WAKE_ACK_EVIDENCE_ROWS" ] || rm -f -- "$FM_WAKE_ACK_EVIDENCE_ROWS" + [ -z "$FM_WAKE_ACK_EVIDENCE_NOTES" ] || rm -f -- "$FM_WAKE_ACK_EVIDENCE_NOTES" \ + "$FM_WAKE_ACK_EVIDENCE_NOTES.hashes" "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" "$FM_WAKE_ACK_EVIDENCE_NOTES.actual" + FM_WAKE_ACK_EVIDENCE_TOKEN= + FM_WAKE_ACK_EVIDENCE_CUTOFF= + FM_WAKE_ACK_EVIDENCE_GENERATION= + FM_WAKE_ACK_EVIDENCE_MARKER= + FM_WAKE_ACK_EVIDENCE_ROWS= + FM_WAKE_ACK_EVIDENCE_NOTES= + FM_WAKE_ACK_EVIDENCE_LEGACY=0 +} + +fm_wake_ack_evidence_capture() { + local rows_file="$STATE/.main-eligible-rows" marker="$STATE/.watcher-down" + local payload seq_count note_count note id digest encoded + fm_wake_ack_evidence_clear + FM_WAKE_ACK_EVIDENCE_ROWS=$(mktemp "${TMPDIR:-/tmp}/fm-wake-ack-rows.XXXXXX") || return 1 + FM_WAKE_ACK_EVIDENCE_NOTES=$(mktemp "${TMPDIR:-/tmp}/fm-wake-ack-notes.XXXXXX") || { + fm_wake_ack_evidence_clear + return 1 + } + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" || { fm_wake_ack_evidence_clear; return 1; } + if ! fm_recovery_marker_snapshot "$marker"; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + fi + FM_WAKE_ACK_EVIDENCE_MARKER=$FM_RECOVERY_MARKER_TOKEN + case "$FM_WAKE_ACK_EVIDENCE_MARKER" in + pending:handling:*|announced:handling:*) ;; + *) + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + ;; + esac + FM_WAKE_ACK_EVIDENCE_GENERATION=${FM_WAKE_ACK_EVIDENCE_MARKER##*:} + if fm_wake_grant_rows_valid "$rows_file" 2>/dev/null; then + if ! awk -F '\t' -v seqs="$rows_file" ' + BEGIN { while ((getline seq < seqs) > 0) wanted[seq]=1 } + NF >= 5 && $2 ~ /^[0-9]+$/ && ($2 in wanted) { + if (seen[$2]++) bad=1 + print + } + END { + for (seq in wanted) if (!seen[seq]) bad=1 + exit bad + } + ' "$FM_WAKE_QUEUE" > "$FM_WAKE_ACK_EVIDENCE_ROWS"; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + fi + FM_WAKE_ACK_EVIDENCE_CUTOFF=$(awk -F '\t' '$2 > max { max=$2 } END { print max + 0 }' "$FM_WAKE_ACK_EVIDENCE_ROWS") + else + if { [ -e "$rows_file" ] || [ -L "$rows_file" ]; } || [ -s "$FM_WAKE_QUEUE" ]; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + fi + FM_WAKE_ACK_EVIDENCE_CUTOFF=0 + fi + if ! awk -F '\t' ' + $4 ~ /^inbox:/ { + id=substr($4,7) + if (id !~ /^[A-Za-z0-9_-]+$/) exit 1 + if (!seen[id]++) print id + } + ' "$FM_WAKE_ACK_EVIDENCE_ROWS" > "$FM_WAKE_ACK_EVIDENCE_NOTES"; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + fi + : > "$FM_WAKE_ACK_EVIDENCE_NOTES.hashes" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + } + while IFS= read -r id; do + [ -n "$id" ] || continue + note="$STATE/inbox/$id.note" + [ ! -e "$STATE/inbox/handled/$id.note" ] && [ ! -L "$STATE/inbox/handled/$id.note" ] \ + && fm_wake_ack_note_safe "$note" && digest=$(fm_wake_ack_hash "$note") || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + } + printf '%s\t%s\n' "$id" "$digest" >> "$FM_WAKE_ACK_EVIDENCE_NOTES.hashes" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + } + done < "$FM_WAKE_ACK_EVIDENCE_NOTES" + mv -f -- "$FM_WAKE_ACK_EVIDENCE_NOTES.hashes" "$FM_WAKE_ACK_EVIDENCE_NOTES" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + } + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + payload=$(mktemp "${TMPDIR:-/tmp}/fm-wake-ack-payload.XXXXXX") || { fm_wake_ack_evidence_clear; return 1; } + seq_count=$(awk 'END { print NR + 0 }' "$FM_WAKE_ACK_EVIDENCE_ROWS") + note_count=$(awk 'END { print NR + 0 }' "$FM_WAKE_ACK_EVIDENCE_NOTES") + { + printf 'fm-wake-ack-evidence-v1\n' + printf 'cutoff\t%s\n' "$FM_WAKE_ACK_EVIDENCE_CUTOFF" + printf 'generation\t%s\n' "$FM_WAKE_ACK_EVIDENCE_GENERATION" + printf 'marker\t%s\n' "$(printf '%s' "$FM_WAKE_ACK_EVIDENCE_MARKER" | base64 | tr -d '\r\n')" + printf 'rows\t%s\n' "$seq_count" + while IFS= read -r seq; do + printf 'row\t%s\n' "$(printf '%s' "$seq" | base64 | tr -d '\r\n')" + done < "$FM_WAKE_ACK_EVIDENCE_ROWS" + printf 'notes\t%s\n' "$note_count" + while IFS=$'\t' read -r id digest; do + printf 'note\t%s\t%s\n' "$id" "$digest" + done < "$FM_WAKE_ACK_EVIDENCE_NOTES" + } > "$payload" || { rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; } + encoded=$(base64 < "$payload" | tr -d '\r\n') || { rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; } + rm -f -- "$payload" + FM_WAKE_ACK_EVIDENCE_TOKEN="v1.$encoded" +} + +fm_wake_ack_evidence_load() { # + local token=$1 encoded payload header tag value extra count index row notes_derived expected_notes + fm_wake_ack_evidence_clear + [ "${#token}" -le 8388608 ] || return 1 + case "$token" in + v1.*) encoded=${token#v1.}; header=fm-wake-ack-evidence-v1 ;; + legacy.*) encoded=${token#legacy.}; header=fm-wake-ack-evidence-legacy-v1; FM_WAKE_ACK_EVIDENCE_LEGACY=1 ;; + *) return 1 ;; + esac + payload=$(mktemp "${TMPDIR:-/tmp}/fm-wake-ack-payload.XXXXXX") || return 1 + FM_WAKE_ACK_EVIDENCE_ROWS=$(mktemp "${TMPDIR:-/tmp}/fm-wake-ack-rows.XXXXXX") || { rm -f -- "$payload"; return 1; } + FM_WAKE_ACK_EVIDENCE_NOTES=$(mktemp "${TMPDIR:-/tmp}/fm-wake-ack-notes.XXXXXX") || { + rm -f -- "$payload" + fm_wake_ack_evidence_clear + return 1 + } + if ! printf '%s' "$encoded" | base64 -d > "$payload" 2>/dev/null; then + rm -f -- "$payload" + fm_wake_ack_evidence_clear + return 1 + fi + exec 7< "$payload" + IFS= read -r value <&7 || value= + if [ "$value" != "$header" ]; then exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; fi + IFS=$'\t' read -r tag FM_WAKE_ACK_EVIDENCE_CUTOFF extra <&7 || true + [ "$tag" = cutoff ] && [ -z "$extra" ] || { exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; } + case "$FM_WAKE_ACK_EVIDENCE_CUTOFF" in ''|*[!0-9]*) exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1 ;; esac + IFS=$'\t' read -r tag FM_WAKE_ACK_EVIDENCE_GENERATION extra <&7 || true + [ "$tag" = generation ] && [ -z "$extra" ] || { exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; } + case "$FM_WAKE_ACK_EVIDENCE_GENERATION" in ''|*[!A-Za-z0-9._-]*) exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1 ;; esac + IFS=$'\t' read -r tag value extra <&7 || true + if [ "$tag" != marker ] || [ -n "$extra" ] || ! FM_WAKE_ACK_EVIDENCE_MARKER=$(printf '%s' "$value" | base64 -d 2>/dev/null); then + exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1 + fi + IFS=$'\t' read -r tag count extra <&7 || true + case "$count" in ''|*[!0-9]*) count=-1 ;; esac + if [ "$tag" != rows ] || [ -n "$extra" ] || [ "$count" -lt 0 ] || [ "$count" -gt 100000 ]; then + exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1 + fi + for ((index=0; index/dev/null); then + exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1 + fi + printf '%s\n' "$row" >> "$FM_WAKE_ACK_EVIDENCE_ROWS" + done + IFS=$'\t' read -r tag count extra <&7 || true + case "$count" in ''|*[!0-9]*) count=-1 ;; esac + if [ "$tag" != notes ] || [ -n "$extra" ] || [ "$count" -lt 0 ] || [ "$count" -gt 100000 ]; then + exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1 + fi + for ((index=0; index> "$FM_WAKE_ACK_EVIDENCE_NOTES" + done + if IFS= read -r value <&7; then exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; fi + exec 7<&- + rm -f -- "$payload" + if ! awk -F '\t' -v cutoff="$FM_WAKE_ACK_EVIDENCE_CUTOFF" ' + NF < 5 || $2 !~ /^[0-9]+$/ || $2 == 0 || $2 > cutoff || seen[$2]++ { bad=1 } + $2 > max { max=$2 } + END { if ((NR == 0 && cutoff != 0) || (NR > 0 && max != cutoff) || bad) exit 1 } + ' "$FM_WAKE_ACK_EVIDENCE_ROWS"; then fm_wake_ack_evidence_clear; return 1; fi + case "$FM_WAKE_ACK_EVIDENCE_MARKER" in + pending:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|announced:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|acked:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION") ;; + '') [ "$FM_WAKE_ACK_EVIDENCE_LEGACY" = 1 ] && [ "$FM_WAKE_ACK_EVIDENCE_CUTOFF" -gt 0 ] || { fm_wake_ack_evidence_clear; return 1; } ;; + *) fm_wake_ack_evidence_clear; return 1 ;; + esac + notes_derived=$(mktemp "${TMPDIR:-/tmp}/fm-wake-ack-derived.XXXXXX") || { fm_wake_ack_evidence_clear; return 1; } + expected_notes=$(mktemp "${TMPDIR:-/tmp}/fm-wake-ack-expected.XXXXXX") || { rm -f -- "$notes_derived"; fm_wake_ack_evidence_clear; return 1; } + if ! awk -F '\t' '$4 ~ /^inbox:/ { id=substr($4,7); if (id !~ /^[A-Za-z0-9_-]+$/) exit 1; print id }' "$FM_WAKE_ACK_EVIDENCE_ROWS" | LC_ALL=C sort -u > "$notes_derived" \ + || ! cut -f1 "$FM_WAKE_ACK_EVIDENCE_NOTES" | LC_ALL=C sort -u > "$expected_notes" \ + || ! cmp -s "$notes_derived" "$expected_notes" \ + || [ "$(wc -l < "$expected_notes" | tr -d '[:space:]')" != "$(wc -l < "$FM_WAKE_ACK_EVIDENCE_NOTES" | tr -d '[:space:]')" ]; then + rm -f -- "$notes_derived" "$expected_notes" + fm_wake_ack_evidence_clear + return 1 + fi + rm -f -- "$notes_derived" "$expected_notes" + FM_WAKE_ACK_EVIDENCE_TOKEN=$token +} + +fm_wake_ack_evidence_precondition() { # + local token=$1 marker="$STATE/.watcher-down" id digest note handled + fm_wake_ack_evidence_load "$token" || return 1 + [ "$FM_WAKE_ACK_EVIDENCE_LEGACY" = 0 ] || { fm_wake_ack_evidence_clear; return 1; } + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" || { fm_wake_ack_evidence_clear; return 1; } + fm_recovery_marker_snapshot "$marker" || true + case "$FM_RECOVERY_MARKER_TOKEN" in + pending:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|announced:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION") ;; + *) fm_lock_release "$FM_WAKE_QUEUE_LOCK"; fm_wake_ack_evidence_clear; return 1 ;; + esac + if [ "$FM_WAKE_ACK_EVIDENCE_CUTOFF" = 0 ] && [ -s "$FM_WAKE_QUEUE" ]; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + fi + if ! awk -F '\t' -v expected="$FM_WAKE_ACK_EVIDENCE_ROWS" ' + BEGIN { while ((getline row < expected) > 0) { split(row,f,"\t"); saved[f[2]]=row; count++ } } + $2 in saved { if ($0 != saved[$2] || seen[$2]++) bad=1 } + END { for (seq in saved) if (!seen[seq]) bad=1; exit bad } + ' "$FM_WAKE_QUEUE"; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + fi + while IFS=$'\t' read -r id digest; do + [ -n "$id" ] || continue + note="$STATE/inbox/$id.note"; handled="$STATE/inbox/handled/$id.note" + if [ -e "$handled" ] || [ -L "$handled" ] || ! fm_wake_ack_note_safe "$note" || [ "$(fm_wake_ack_hash "$note" 2>/dev/null)" != "$digest" ]; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + fi + done < "$FM_WAKE_ACK_EVIDENCE_NOTES" + fm_lock_release "$FM_WAKE_QUEUE_LOCK" +} + +fm_wake_ack_evidence_completed() { # + local token=$1 id digest note handled marker="$STATE/.watcher-down" + fm_wake_ack_evidence_load "$token" || return 1 + if [ ! -f "$FM_WAKE_QUEUE" ] || [ -L "$FM_WAKE_QUEUE" ]; then + fm_wake_ack_evidence_clear + return 1 + fi + if [ "$FM_WAKE_ACK_EVIDENCE_CUTOFF" = 0 ]; then + fm_recovery_marker_snapshot "$marker" || true + if [ "$FM_RECOVERY_MARKER_TOKEN" != "acked:handling:$FM_WAKE_ACK_EVIDENCE_GENERATION" ] || [ -s "$FM_WAKE_QUEUE" ]; then + fm_wake_ack_evidence_clear + return 1 + fi + elif ! awk -F '\t' -v cutoff="$FM_WAKE_ACK_EVIDENCE_CUTOFF" ' + NF < 5 || $2 !~ /^[0-9]+$/ { bad=1 } + $2 <= cutoff { old=1 } + END { exit bad || old } + ' "$FM_WAKE_QUEUE"; then + fm_wake_ack_evidence_clear + return 1 + fi + while IFS=$'\t' read -r id digest; do + [ -n "$id" ] || continue + note="$STATE/inbox/$id.note"; handled="$STATE/inbox/handled/$id.note" + if [ -e "$note" ] || [ -L "$note" ] || ! fm_wake_ack_note_safe "$handled" || [ "$(fm_wake_ack_hash "$handled" 2>/dev/null)" != "$digest" ]; then + fm_wake_ack_evidence_clear + return 1 + fi + done < "$FM_WAKE_ACK_EVIDENCE_NOTES" + fm_wake_ack_evidence_clear + return 0 +} + +fm_wake_ack_evidence_acknowledge() { # + local token=$1 id digest notes=() + fm_wake_ack_evidence_precondition "$token" || return 1 + while IFS=$'\t' read -r id digest; do + [ -n "$id" ] && notes+=("$id") + done < "$FM_WAKE_ACK_EVIDENCE_NOTES" + if [ "${#notes[@]}" -gt 0 ]; then + "$FM_WAKE_LIB_DIR/fm-inbox.sh" drain --ack "${notes[@]}" || { fm_wake_ack_evidence_clear; return 1; } + fi + "$FM_WAKE_LIB_DIR/fm-wake-drain.sh" --ack-through "$FM_WAKE_ACK_EVIDENCE_CUTOFF" \ + --recovery-generation "$FM_WAKE_ACK_EVIDENCE_GENERATION" || { fm_wake_ack_evidence_clear; return 1; } + fm_wake_ack_evidence_clear + fm_wake_ack_evidence_completed "$token" +} + # --- signal announcement signatures ----------------------------------------- # # The watcher's per-file signal scan (bin/fm-watch.sh scan_signals) detects a diff --git a/bin/native-owner/NativeAcknowledgementEvidence.cs b/bin/native-owner/NativeAcknowledgementEvidence.cs index 470ce1f201b..35fed435dca 100644 --- a/bin/native-owner/NativeAcknowledgementEvidence.cs +++ b/bin/native-owner/NativeAcknowledgementEvidence.cs @@ -1,129 +1,62 @@ -// Read-only evidence for an interrupted inbox + wake acknowledgement. -// Only complete, matching durable postconditions permit recovery. Partial, -// missing, malformed, or changed evidence never authorizes a retry or deletion. using System; using System.Collections.Generic; -using System.Globalization; +using System.ComponentModel; +using System.Diagnostics; using System.IO; -using System.Security.Cryptography; -using System.Text; -using System.Text.RegularExpressions; using System.Web.Script.Serialization; public sealed class NativeAcknowledgementEvidence { internal readonly NativeHomeLease Lease; - readonly Dictionary record; - NativeAcknowledgementEvidence(NativeHomeLease lease,Dictionary value) { Lease=lease;record=value; } - internal Dictionary Record() { var json=new JavaScriptSerializer();return json.Deserialize>(json.Serialize(record)); } - static string Note(Dictionary payload) { - object value; - if(!payload.TryGetValue("note",out value) || !(value is string) || !Regex.IsMatch((string)value,@"\A[A-Za-z0-9_-]+\z")) throw new IOException("Invalid inbox target"); - return (string)value; + readonly string record; + NativeAcknowledgementEvidence(NativeHomeLease lease,string value) { Lease=lease;record=value; } + internal object Record() { return record; } + static string Quote(string value) { return "\""+value.Replace("\"","\\\"")+"\""; } + static string CodeRoot() { + string configured=Environment.GetEnvironmentVariable("FM_PROBE_CODE_ROOT"); + if(!string.IsNullOrEmpty(configured))return Path.GetFullPath(configured); + return Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory,"..")); } - static byte[] Read(string home,string relative) { - home=Path.GetFullPath(home).TrimEnd('\\','/'); - if((File.GetAttributes(home)&FileAttributes.ReparsePoint)!=0) throw new IOException("Reparse point in acknowledgement home"); - string full=Path.Combine(home,relative), parent=full; - while(!string.Equals(parent,home,StringComparison.OrdinalIgnoreCase)) { - if((File.GetAttributes(parent)&FileAttributes.ReparsePoint)!=0) throw new IOException("Reparse point in acknowledgement evidence"); - parent=Path.GetDirectoryName(parent); - if(parent==null) throw new IOException("Evidence escaped its home"); + static int Invoke(NativeHomeLease lease,string mode,string input,out string output) { + if(lease==null || !lease.IsHeld)throw new InvalidOperationException("An active home lease is required"); + string script=Path.Combine(CodeRoot(),"bin","native-owner","ack-evidence.sh"); + var start=new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script.Replace('\\','/'))+" "+mode) { + UseShellExecute=false,RedirectStandardInput=true,RedirectStandardOutput=true,RedirectStandardError=true,CreateNoWindow=true + }; + start.EnvironmentVariables["FM_HOME"]=lease.Home; + start.EnvironmentVariables["MSYS"]="winsymlinks:nativestrict"; + using(var process=Process.Start(start)) { + var stdout=process.StandardOutput.ReadToEndAsync();var stderr=process.StandardError.ReadToEndAsync(); + if(input!=null)process.StandardInput.Write(input); + process.StandardInput.Close(); + if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("Acknowledgement evidence owner exceeded its bound");} + output=stdout.Result.Trim(); + if(process.ExitCode!=0 && mode!="verify-token" && mode!="verify-legacy")throw new IOException("Acknowledgement evidence owner refused the target: "+stderr.Result.Trim()); + return process.ExitCode; } - using(var file=new FileStream(full,FileMode.Open,FileAccess.Read,FileShare.Read)) { - if(file.Length>16*1024*1024) throw new IOException("Acknowledgement evidence exceeds its bound"); - using(var data=new MemoryStream()) { file.CopyTo(data);return data.ToArray(); } - } - } - static string Hash(byte[] value) { using(var hash=SHA256.Create()) return BitConverter.ToString(hash.ComputeHash(value)).Replace("-","").ToLowerInvariant(); } - static string RecoveryGeneration(Dictionary payload) { - object value; - if(!payload.TryGetValue("generation",out value) || !(value is string) || !Regex.IsMatch((string)value,@"\A[A-Za-z0-9._-]+\z")) throw new IOException("Invalid recovery generation"); - return (string)value; - } - static string RecoveryMarker(NativeHomeLease lease) { - string value=new UTF8Encoding(false,true).GetString(Read(lease.Home,Path.Combine("state",".watcher-down"))); - if(!value.EndsWith("\n",StringComparison.Ordinal) || value.IndexOf('\n')!=value.Length-1) throw new IOException("Invalid recovery marker"); - return value.Substring(0,value.Length-1); - } - static List Rows(byte[] bytes,ulong cutoff) { - string text=new UTF8Encoding(false,true).GetString(bytes); - if(text.Length>0 && !text.EndsWith("\n",StringComparison.Ordinal)) throw new IOException("Incomplete wake queue"); - var result=new List(); - foreach(string line in text.Split(new [] {'\n'},StringSplitOptions.RemoveEmptyEntries)) { - string[] fields=line.Split('\t');ulong sequence,epoch; - if(fields.Length<5 || !ulong.TryParse(fields[0],NumberStyles.None,CultureInfo.InvariantCulture,out epoch) || !ulong.TryParse(fields[1],NumberStyles.None,CultureInfo.InvariantCulture,out sequence) || sequence==0 || fields[3].Length==0 || (fields[2]!="check" && fields[2]!="signal" && fields[2]!="stale" && fields[2]!="heartbeat")) throw new IOException("Malformed wake queue"); - if(sequence<=cutoff) result.Add(line); - } - return result; } public static NativeAcknowledgementEvidence Capture(NativeHomeLease lease,Dictionary payload) { - if(lease==null || !lease.IsHeld) throw new InvalidOperationException("An active home lease is required"); - ulong cutoff; - if(!payload.ContainsKey("seq") || !(payload["seq"] is string) || !ulong.TryParse((string)payload["seq"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff)) throw new IOException("Invalid wake cutoff"); - var queue=Read(lease.Home,Path.Combine("state",".wake-queue")); - var rows=Rows(queue,cutoff); - var ids=new HashSet(); - if(payload.ContainsKey("notes")) { - var input=payload["notes"] as System.Collections.IList; - if(input==null)throw new IOException("Invalid inbox target list"); - foreach(object id in input)ids.Add(Note(new Dictionary{{"note",id}})); - } else ids.Add(Note(payload)); - var queued=new HashSet(); - foreach(string row in rows) {string key=row.Split(' ')[3];if(key.StartsWith("inbox:",StringComparison.Ordinal))queued.Add(key.Substring(6));} - if(!ids.SetEquals(queued))throw new IOException("Inbox targets differ from the captured queue"); - if(cutoff==0) { - if(queue.Length!=0 || rows.Count!=0 || ids.Count!=0)throw new IOException("Invalid zero-row recovery target"); - string generation=RecoveryGeneration(payload),marker=RecoveryMarker(lease); - if(marker!="pending:handling:"+generation && marker!="announced:handling:"+generation)throw new IOException("Recovery target differs from the captured generation"); - return new NativeAcknowledgementEvidence(lease,new Dictionary{{"version",3},{"notes",new object[0]},{"cutoff","0"},{"rows",new object[0]},{"recoveryGeneration",generation},{"recoveryMarker",marker}}); - } - if(rows.Count==0) throw new IOException("No queued targets remain"); - var notes=new List>(); - foreach(string id in ids) { - if(File.Exists(Path.Combine(lease.Home,"state","inbox","handled",id+".note")))throw new IOException("Inbox target is already handled or ambiguous"); - notes.Add(new Dictionary{{"note",id},{"noteSha256",Hash(Read(lease.Home,Path.Combine("state","inbox",id+".note")))}}); - } - var record=new Dictionary{{"version",2},{"notes",notes},{"cutoff",cutoff.ToString(CultureInfo.InvariantCulture)},{"rows",rows.ToArray()}}; - if(notes.Count==1) {record["note"]=notes[0]["note"];record["noteSha256"]=notes[0]["noteSha256"];} - return new NativeAcknowledgementEvidence(lease,record); + if(payload==null)throw new IOException("Notification payload is incomplete"); + object value;string opaque=null,output; + if(payload.TryGetValue("ownerEvidence",out value))opaque=value as string; + if(string.IsNullOrEmpty(opaque)) { + var json=new JavaScriptSerializer(); + if(Invoke(lease,"capture-payload",json.Serialize(payload),out output)!=0 || string.IsNullOrEmpty(output))throw new IOException("Acknowledgement evidence was not captured"); + opaque=output; + } else if(Invoke(lease,"preflight-token",opaque,out output)!=0)throw new IOException("Acknowledgement evidence no longer matches its owner target"); + return new NativeAcknowledgementEvidence(lease,opaque); } - internal static bool Completed(NativeHomeLease lease,Dictionary evidence) { - if(lease==null || !lease.IsHeld || evidence==null) return false; + internal static bool Completed(NativeHomeLease lease,object evidence) { + if(lease==null || !lease.IsHeld || evidence==null)return false; try { - int version=Convert.ToInt32(evidence["version"]);ulong cutoff; - if(version!=1 && version!=2 && version!=3)return false; - if(!ulong.TryParse((string)evidence["cutoff"],NumberStyles.None,CultureInfo.InvariantCulture,out cutoff))return false; - var targets=evidence["rows"] as System.Collections.IList; - if(targets==null)return false; - if(cutoff==0) { - if(version!=3 || targets.Count!=0)return false; - var zeroNotes=evidence["notes"] as System.Collections.IList; - string generation=evidence["recoveryGeneration"] as string,marker=evidence["recoveryMarker"] as string; - if(zeroNotes==null || zeroNotes.Count!=0 || generation==null || !Regex.IsMatch(generation,@"\A[A-Za-z0-9._-]+\z"))return false; - if(marker!="pending:handling:"+generation && marker!="announced:handling:"+generation)return false; - return RecoveryMarker(lease)=="acked:handling:"+generation && Read(lease.Home,Path.Combine("state",".wake-queue")).Length==0; - } - if(version==3 || targets.Count==0)return false; - var saved=new List();foreach(object target in targets){if(!(target is string))return false;saved.Add((string)target);} - var validated=Rows(new UTF8Encoding(false,true).GetBytes(string.Join("\n",saved.ToArray())+"\n"),cutoff); - if(validated.Count!=targets.Count)return false; - var notes=new List>(); - if(version==1)notes.Add(evidence); - else { - var list=evidence["notes"] as System.Collections.IList;if(list==null)return false; - foreach(object entry in list){var note=entry as Dictionary;if(note==null)return false;notes.Add(note);} - } - var ids=new HashSet(); - foreach(var entry in notes) { - string note=Note(entry),digest=(string)entry["noteSha256"]; - if(!ids.Add(note)||!Regex.IsMatch(digest,@"\A[0-9a-f]{64}\z"))return false; - if(File.Exists(Path.Combine(lease.Home,"state","inbox",note+".note")))return false; - if(Hash(Read(lease.Home,Path.Combine("state","inbox","handled",note+".note")))!=digest)return false; - } - var queued=new HashSet();foreach(string row in validated){string key=row.Split(' ')[3];if(key.StartsWith("inbox:",StringComparison.Ordinal))queued.Add(key.Substring(6));} - return ids.SetEquals(queued)&&Rows(Read(lease.Home,Path.Combine("state",".wake-queue")),cutoff).Count==0; + string output; + string opaque=evidence as string; + if(opaque!=null)return Invoke(lease,"verify-token",opaque,out output)==0; + var json=new JavaScriptSerializer(); + return Invoke(lease,"verify-legacy",json.Serialize(evidence),out output)==0; } catch(IOException){return false;} catch(UnauthorizedAccessException){return false;} - catch(ArgumentException){return false;} catch(KeyNotFoundException){return false;} - catch(InvalidCastException){return false;} catch(FormatException){return false;} catch(OverflowException){return false;} + catch(Win32Exception){return false;} + catch(ArgumentException){return false;} catch(InvalidOperationException){return false;} + catch(FormatException){return false;} catch(OverflowException){return false;} + catch(TimeoutException){return false;} } } diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs index 6a05fed9082..bad1bf53e5f 100644 --- a/bin/native-owner/NativeLauncher.cs +++ b/bin/native-owner/NativeLauncher.cs @@ -13,19 +13,20 @@ public static partial class NativeOwner { [DllImport("kernel32.dll")] static extern IntPtr GetStdHandle(int kind); static string CodeRoot { get { return Path.GetDirectoryName(Path.GetDirectoryName(OwnExe)); } } - static void EmptyBacklog(string home) { - string data=Path.Combine(home,"data"),backlog=Path.Combine(data,"backlog.md"); - if(!Directory.Exists(data)||Directory.GetFileSystemEntries(data,"backlog.md").Length==0)return; - if(!File.Exists(backlog))throw new InvalidOperationException("This experimental launcher requires a canonical empty backlog; the existing backlog was preserved"); - string content; - try {content=File.ReadAllText(backlog,new UTF8Encoding(false,true));} - catch(Exception error){throw new InvalidOperationException("This experimental launcher could not validate the existing backlog; it was preserved",error);} - if(content!="## In flight\n\n## Queued\n\n## Done\n")throw new InvalidOperationException("This experimental launcher requires a canonical empty backlog; the existing backlog was preserved"); + static void OwnerAdmission(string home) { + string script=Path.Combine(CodeRoot,"bin","native-owner","admit.sh"); + var start=new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script.Replace('\\','/'))) {UseShellExecute=false,RedirectStandardError=true,CreateNoWindow=true}; + start.EnvironmentVariables["FM_HOME"]=home; + using(var process=Process.Start(start)) { + var error=process.StandardError.ReadToEndAsync(); + if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("The empty-fleet owner preflight exceeded its bound");} + if(process.ExitCode!=0)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); + } } static void EmptyFleet(string home) { string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); - if((Directory.Exists(state)&&Directory.GetFiles(state,"*.meta").Length!=0) || (Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || File.Exists(Path.Combine(state,"x-watch.check.sh")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); - EmptyBacklog(home); + if((Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || File.Exists(Path.Combine(state,"x-watch.check.sh")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); + OwnerAdmission(home); } static int Launch(string selectedHome) { string home=Path.GetFullPath(selectedHome), node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); diff --git a/bin/native-owner/NativeReceiptJournal.cs b/bin/native-owner/NativeReceiptJournal.cs index 20b66720016..9562e06057f 100644 --- a/bin/native-owner/NativeReceiptJournal.cs +++ b/bin/native-owner/NativeReceiptJournal.cs @@ -89,8 +89,8 @@ public void CompleteAcknowledgement(string receipt) { if(receipt==null || !receipts.TryGetValue(receipt,out row) || (string)row["generation"]!=generation || (string)row["event"]!="ack-started") throw new InvalidOperationException("No matching acknowledgement attempt"); Append("acknowledged",receipt,Copy((Dictionary)row["payload"]),Evidence(row)); } - static Dictionary Evidence(Dictionary row) { - object value;return row.TryGetValue("targetEvidence",out value) ? value as Dictionary : null; + static object Evidence(Dictionary row) { + object value;return row.TryGetValue("targetEvidence",out value) ? value : null; } public int ReconcileCompletedAcknowledgements() { EnsureOpen(); @@ -105,7 +105,7 @@ public int ReconcileCompletedAcknowledgements() { } return completed; } - void Append(string kind,string receipt,Dictionary payload,Dictionary evidence=null,string ackGeneration=null) { + void Append(string kind,string receipt,Dictionary payload,object evidence=null,string ackGeneration=null) { var row=new Dictionary{{"version",1},{"home",home},{"generation",generation},{"event",kind},{"receipt",receipt},{"payload",payload},{"targetEvidence",evidence},{"ackGeneration",ackGeneration}}; byte[] bytes=new UTF8Encoding(false,true).GetBytes(json.Serialize(row)+"\n"); if(bytes.Length>65536 || file.Length+bytes.Length>Limit) throw new IOException("Receipt journal capacity exceeded; durable work preserved"); diff --git a/bin/native-owner/ack-evidence.sh b/bin/native-owner/ack-evidence.sh new file mode 100644 index 00000000000..6fe3a8c67a5 --- /dev/null +++ b/bin/native-owner/ack-evidence.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) +export FM_HOME +FM_HOME=$(cygpath -u "${FM_HOME:?}") +export FM_STATE_OVERRIDE="$FM_HOME/state" +cd "$ROOT" +. bin/fm-wake-lib.sh + +read_token() { + local token + token=$(cat) + [ -n "$token" ] || exit 2 + printf '%s' "$token" +} + +legacy_token() { + node -e ' + let input=""; + process.stdin.setEncoding("utf8"); + process.stdin.on("data",chunk=>input+=chunk); + process.stdin.on("end",()=>{ + const value=JSON.parse(input); + if(!value||typeof value!=="object"||Array.isArray(value))throw Error("legacy evidence must be an object"); + const version=Number(value.version); + if(![1,2,3].includes(version))throw Error("unsupported legacy evidence"); + if(typeof value.cutoff!=="string")throw Error("invalid legacy cutoff"); + const cutoff=String(value.cutoff); + const rows=value.rows; + if(!Array.isArray(rows)||rows.some(row=>typeof row!=="string"))throw Error("invalid legacy rows"); + let notes=[]; + if(version===1)notes=[value]; + else notes=value.notes; + if(!Array.isArray(notes)||notes.some(note=>!note||typeof note!=="object"||Array.isArray(note)))throw Error("invalid legacy notes"); + if(version===3&&(cutoff!=="0"||rows.length!==0||notes.length!==0))throw Error("invalid legacy recovery target"); + const generation=version===3?value.recoveryGeneration:"legacy"; + const marker=version===3?value.recoveryMarker:""; + if(typeof generation!=="string"||typeof marker!=="string")throw Error("invalid legacy recovery evidence"); + const line=["fm-wake-ack-evidence-legacy-v1",`cutoff\t${cutoff}`,`generation\t${generation}`,`marker\t${Buffer.from(marker).toString("base64")}`,`rows\t${rows.length}`]; + for(const row of rows)line.push(`row\t${Buffer.from(row).toString("base64")}`); + line.push(`notes\t${notes.length}`); + for(const note of notes){ + if(typeof note.note!=="string"||typeof note.noteSha256!=="string")throw Error("invalid legacy note"); + line.push(`note\t${note.note}\t${note.noteSha256}`); + } + process.stdout.write("legacy."+Buffer.from(line.join("\n")+"\n").toString("base64")); + }); + ' +} + +case "${1:-}" in + capture-json) + fm_wake_ack_evidence_capture + printf '{"seq":"%s","generation":"%s","notes":[' \ + "$FM_WAKE_ACK_EVIDENCE_CUTOFF" "$FM_WAKE_ACK_EVIDENCE_GENERATION" + separator= + while IFS=$'\t' read -r id digest; do + [ -n "$id" ] || continue + printf '%s"%s"' "$separator" "$id" + separator=, + done < "$FM_WAKE_ACK_EVIDENCE_NOTES" + printf '],"ownerEvidence":"%s"}\n' "$FM_WAKE_ACK_EVIDENCE_TOKEN" + fm_wake_ack_evidence_clear + ;; + capture-token) + fm_wake_ack_evidence_capture + printf '%s\n' "$FM_WAKE_ACK_EVIDENCE_TOKEN" + fm_wake_ack_evidence_clear + ;; + capture-payload) + mapfile -t expected < <(node -e ' + let input=""; + process.stdin.setEncoding("utf8"); + process.stdin.on("data",chunk=>input+=chunk); + process.stdin.on("end",()=>{ + const value=JSON.parse(input); + if(!value||typeof value!=="object"||Array.isArray(value))throw Error("invalid payload"); + const seq=String(value.seq),generation=value.generation; + const notes=value.notes===undefined?[value.note]:value.notes; + if(!/^\d+$/.test(seq)||typeof generation!=="string"||!Array.isArray(notes)||notes.some(note=>typeof note!=="string"))throw Error("invalid payload target"); + process.stdout.write([seq,generation,...new Set(notes)].join("\n")+"\n"); + }); + ') + [ "${#expected[@]}" -ge 2 ] || exit 2 + fm_wake_ack_evidence_capture + [ "${expected[0]}" = "$FM_WAKE_ACK_EVIDENCE_CUTOFF" ] \ + && [ "${expected[1]}" = "$FM_WAKE_ACK_EVIDENCE_GENERATION" ] || exit 2 + if [ "${#expected[@]}" -gt 2 ]; then + printf '%s\n' "${expected[@]:2}" | LC_ALL=C sort -u > "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" + else + : > "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" + fi + cut -f1 "$FM_WAKE_ACK_EVIDENCE_NOTES" | LC_ALL=C sort -u > "$FM_WAKE_ACK_EVIDENCE_NOTES.actual" + cmp -s "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" "$FM_WAKE_ACK_EVIDENCE_NOTES.actual" || exit 2 + rm -f -- "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" "$FM_WAKE_ACK_EVIDENCE_NOTES.actual" + printf '%s\n' "$FM_WAKE_ACK_EVIDENCE_TOKEN" + fm_wake_ack_evidence_clear + ;; + preflight-token) + token=$(read_token) + fm_wake_ack_evidence_precondition "$token" + fm_wake_ack_evidence_clear + ;; + verify-token) + token=$(read_token) + fm_wake_ack_evidence_completed "$token" + ;; + verify-legacy) + token=$(legacy_token) + fm_wake_ack_evidence_completed "$token" + ;; + acknowledge-token) + token=$(read_token) + fm_wake_ack_evidence_acknowledge "$token" + ;; + *) + printf 'usage: ack-evidence.sh capture-json|capture-token|capture-payload|preflight-token|verify-token|verify-legacy|acknowledge-token\n' >&2 + exit 2 + ;; +esac diff --git a/bin/native-owner/ack.sh b/bin/native-owner/ack.sh index d1d59c35106..275ed40ed94 100755 --- a/bin/native-owner/ack.sh +++ b/bin/native-owner/ack.sh @@ -10,16 +10,6 @@ cd "$ROOT" . bin/fm-session-lock-lib.sh fm_session_lock_owned_by_self "$FM_HOME/state" request="$LOG/notification-ack-request.json" -seq=$(jq -r .seq "$request") -generation=$(jq -r .generation "$request") -case "$seq" in ''|*[!0-9]*) exit 2 ;; esac -case "$generation" in ''|*[!A-Za-z0-9._-]*) exit 2 ;; esac -notes=$(jq -r '.notes[]' "$request") -if [ -n "$notes" ]; then - while IFS= read -r note; do - case "$note" in ''|*[!A-Za-z0-9_-]*) exit 2 ;; esac - bin/fm-inbox.sh drain --ack "$note" - done <<< "$notes" -fi -bin/fm-wake-drain.sh --ack-through "$seq" --recovery-generation "$generation" +evidence=$(jq -er '.ownerEvidence | select(type == "string" and length > 0)' "$request") +printf '%s' "$evidence" | bash bin/native-owner/ack-evidence.sh acknowledge-token printf '{"acknowledged":true}\n' > "$LOG/notification-ack.json" diff --git a/bin/native-owner/admit.sh b/bin/native-owner/admit.sh new file mode 100644 index 00000000000..7d005325af7 --- /dev/null +++ b/bin/native-owner/admit.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) +export FM_HOME +FM_HOME=$(cygpath -u "${FM_HOME:?}") +cd "$ROOT" +. bin/fm-tasks-axi-lib.sh +. bin/fm-backlog-transition-lib.sh +if ! fm_backlog_empty_fleet_preflight "$FM_HOME/state" "$FM_HOME/data"; then + printf '%s\n' "${FM_BACKLOG_EMPTY_ERROR:-the home contains work-bearing records}" >&2 + exit 2 +fi diff --git a/bin/native-owner/check.sh b/bin/native-owner/check.sh index 32ab48f1da0..9501be63841 100755 --- a/bin/native-owner/check.sh +++ b/bin/native-owner/check.sh @@ -15,8 +15,7 @@ rc=$? set -e case "$rc" in 0|124) ;; *) exit "$rc" ;; esac bin/fm-wake-drain.sh > "$LOG/delivery.log" 2>&1 -ack=$(grep 'WAKE_ACK_REQUIRED:' "$LOG/delivery.log" | tail -1 || true) -if [ -z "$ack" ]; then +if ! grep -q 'WAKE_ACK_REQUIRED:' "$LOG/delivery.log"; then if grep -Eq 'OPEN DECISIONS|UNREAD STATUS|RECORD DIVERGENCE|STATUS OUTCOME BACKSTOP' "$LOG/checkpoint.log" "$LOG/delivery.log"; then printf 'Unqueued work requires reconciliation; captured in %s\n' "$LOG" >&2 exit 2 @@ -25,14 +24,10 @@ if [ -z "$ack" ]; then exit 0 fi bin/fm-inbox.sh drain > "$LOG/inbox.log" -seq=$(awk '{for(i=1;i0))') +target=$(bash bin/native-owner/ack-evidence.sh capture-json) challenge=$(od -An -N12 -tx1 /dev/urandom | tr -d ' \n') message="$(<"$LOG/checkpoint.log") $(<"$LOG/delivery.log") $(<"$LOG/inbox.log")" -jq -n --arg message "$message" --arg challenge "$challenge" --arg seq "$seq" --arg generation "$generation" --argjson notes "$notes" \ - '{message:$message,challenge:$challenge,seq:$seq,generation:$generation,notes:$notes}' > "$LOG/notification-check.json" +jq -n --arg message "$message" --arg challenge "$challenge" --argjson target "$target" \ + '$target + {message:$message,challenge:$challenge}' > "$LOG/notification-check.json" diff --git a/bin/native-owner/codex-host-runtime.mjs b/bin/native-owner/codex-host-runtime.mjs new file mode 100644 index 00000000000..858dde291e0 --- /dev/null +++ b/bin/native-owner/codex-host-runtime.mjs @@ -0,0 +1,209 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import net from 'node:net'; +import {spawn} from 'node:child_process'; +import {createInterface} from 'node:readline'; +import {confirmAutomaticNotificationOffer,createNotificationGate} from './codex-tool-gate.mjs'; +import {createHostLifecycle,reconciliationWarning} from './host-lifecycle.mjs'; +import {discoverMcpServerNames,isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from './app-server-policy.mjs'; + +export async function runCodexHost(options={}) { + const env=options.env??process.env; + const runtime=env.FM_PROBE_HOME,root=env.FM_PROBE_CODE_ROOT; + const inputStream=options.input??process.stdin,outputStream=options.output??process.stdout,errorStream=options.error??process.stderr; + const pause=options.pause??(ms=>new Promise(resolve=>setTimeout(resolve,ms))); + let closing=false,gate=null,server=null,alive=false,thread=null,activeTurn=null,ended=false; + let lifecycle=null,terminal=null,announced=null,next=0,nativeWaiter=null,nativeTail=Promise.resolve(); + let socket=null,channel=null; + const input=[],pending=new Map(),turns=new Map(); + const evidence={ready:false,turns:[],tools:[],automatic:[],digestDeliveredBeforeDeferred:false}; + const save=()=>{const file=path.join(runtime,'host.json');fs.writeFileSync(file+'.tmp',JSON.stringify(evidence,null,2));fs.renameSync(file+'.tmp',file);}; + const consoleInput=createInterface({input:inputStream}); + consoleInput.on('line',line=>{ + if(line==='/quit'){void stop();return;} + if(line==='/interrupt'){void interrupt().catch(fail);return;} + input.push(line); + }); + consoleInput.on('close',()=>{ended=true;}); + const onSigint=()=>{if(activeTurn)void interrupt().catch(fail);else void stop();}; + const onSigterm=()=>{void stop();}; + if(options.installSignalHandlers!==false){process.on('SIGINT',onSigint);process.on('SIGTERM',onSigterm);} + + let connection; + if(options.native){ + connection=Promise.resolve(); + }else{ + socket=net.createConnection('\\\\.\\pipe\\'+env.FM_PROBE_PIPE); + connection=new Promise((resolve,reject)=>{ + const timer=setTimeout(()=>{socket.destroy();reject(Error('Native connection timed out'));},10000); + socket.once('connect',()=>{clearTimeout(timer);resolve();});socket.once('error',error=>{clearTimeout(timer);reject(error);}); + }); + channel=createInterface({input:socket}); + channel.on('line',line=>{ + if(!nativeWaiter){fail(Error('Unexpected native response'));return;} + const waiter=nativeWaiter;nativeWaiter=null; + try {const value=JSON.parse(line);if(!value.notificationAuthorized)throw Error('Native host authorization refused');waiter.resolve(value);}catch(error){waiter.reject(error);} + }); + socket.on('error',error=>{nativeWaiter?.reject(error);nativeWaiter=null;if(!closing)fail(error);}); + socket.on('close',()=>{const error=Error('Native controller connection closed');nativeWaiter?.reject(error);nativeWaiter=null;if(!closing)fail(error);}); + } + function native(action,extra={},signal){ + if(options.native)return Promise.resolve().then(()=>options.native(action,extra,signal)); + const run=async()=>{ + await connection;if(socket.destroyed||signal?.aborted||(closing&&action!=='shutdown'))throw Error('Native session unavailable'); + return new Promise((resolve,reject)=>{ + let timer; + const abort=()=>{socket.destroy();finish(reject,Error('Native request interrupted; durable work preserved'));}; + const finish=(callback,value)=>{clearTimeout(timer);signal?.removeEventListener('abort',abort);callback(value);}; + nativeWaiter={resolve:value=>finish(resolve,value),reject:error=>finish(reject,error)}; + timer=setTimeout(abort,10000);signal?.addEventListener('abort',abort,{once:true}); + socket.write(JSON.stringify({kind:'notification',session:env.FM_PROBE_SESSION,home:runtime,nonce:env.FM_PROBE_NONCE,action,...extra})+'\n'); + }); + }; + const result=nativeTail.then(run);nativeTail=result.catch(()=>{});return result; + } + function send(frame){if(!alive)throw Error('App-server is not live');server.stdin.write(JSON.stringify(frame)+'\n');} + function request(method,params,signal){ + return new Promise((resolve,reject)=>{ + if(signal?.aborted){reject(Error('Request cancelled'));return;} + const id=++next;let timer; + const clean=()=>{clearTimeout(timer);signal?.removeEventListener('abort',abort);}; + const abort=()=>{pending.delete(id);clean();reject(Error('App-server request cancelled'));}; + timer=setTimeout(()=>{pending.delete(id);clean();reject(Error('App-server '+method+' timed out'));},30000); + pending.set(id,{method,resolve:value=>{clean();resolve(value);},reject:error=>{clean();reject(error);}}); + signal?.addEventListener('abort',abort,{once:true}); + try {send({id,method,params});}catch(error){pending.delete(id);clean();reject(error);} + }); + } + async function interrupt(signal){ + if(!activeTurn||!alive)return; + const target=activeTurn; + await request('turn/interrupt',{threadId:thread,turnId:target},signal); + const limit=Date.now()+10000; + while(alive&&!turns.has(target)&&!signal?.aborted&&Date.now()gate?.close()},interrupt, + stopOperations:async signal=>{const value=await native('shutdown',{},signal);return {stopped:value.operationState==='stopped',reconciliationRequired:value.reconciliationRequired===true};}, + closeInput:()=>{if(server)server.stdin.end();}, + waitForExit:signal=>!alive?Promise.resolve(true):new Promise(resolve=>{ + const exit=()=>{signal.removeEventListener('abort',abort);resolve(true);}; + const abort=()=>{server.removeListener('exit',exit);resolve(false);}; + server.once('exit',exit);signal.addEventListener('abort',abort,{once:true}); + }),terminate:()=>server?.kill(),graceMs:3000}); + const result=lifecycle.shutdown(); + void result.then(value=>{ + fs.writeFileSync(path.join(runtime,'shutdown.json'),JSON.stringify(value)); + if(value.reconciliationRequired)errorStream.write(reconciliationWarning(path.join(env.FM_HOME,'owner-receipts.jsonl'))+'\n'); + if(!value.stopped){ + if(terminal)errorStream.write('Shutdown was not fully confirmed; durable work was preserved.\n'); + else terminal=Error('Shutdown was not fully confirmed; durable work was preserved.'); + } + channel?.close();socket?.destroy(); + }); + return result; + } + try { + await connection; + let status=await native('status'); + if(status.operationState==='reconciliation-required')throw Error('An earlier acknowledgement is incomplete. Its records are preserved; review '+path.join(env.FM_HOME,'owner-receipts.jsonl')+' before starting more work.'); + const limit=Date.now()+200000; + while(!closing&&!fs.existsSync(path.join(runtime,'digest.ready'))){ + status=await native('status');if(status.operationState==='startup-failed'||Date.now()>limit)throw Error('Startup failed; inspect '+path.join(runtime,'startup.log')); + await pause(100); + } + if(!closing){ + const executable=path.join(env.APPDATA,'npm/node_modules/@openai/codex/node_modules/@openai/codex-win32-x64/vendor/x86_64-pc-windows-msvc/bin/codex.exe'); + const mcpServerNames=options.mcpServerNames??await discoverMcpServerNames(executable,root,env); + server=options.spawnAppServer?options.spawnAppServer({executable,mcpServerNames,root,env}):spawn(executable,isolatedAppServerArgs(mcpServerNames,['-c','windows.sandbox=unelevated']),{cwd:root,stdio:['pipe','pipe','pipe'],detached:true,windowsHide:true});alive=true; + server.stderr.on('data',data=>fs.appendFileSync(path.join(runtime,'app-server.stderr'),data)); + const lost=error=>{alive=false;for(const waiter of pending.values())waiter.reject(error);pending.clear();if(!closing)fail(error);}; + server.on('error',lost);server.on('exit',()=>lost(Error('App-server exited')));server.stdin.on('error',error=>{if(!closing)fail(error);}); + createInterface({input:server.stdout}).on('line',line=>{ + try { + const frame=JSON.parse(line); + if(frame.method==='item/tool/call'&&frame.id!==undefined){void tool(frame).catch(fail);return;} + if(frame.id!==undefined&&pending.has(frame.id)){ + const waiter=pending.get(frame.id);pending.delete(frame.id); + if(frame.error)waiter.reject(Error(JSON.stringify(frame.error))); + else {if(waiter.method==='turn/start'){activeTurn=frame.result.turn.id;evidence.activeTurn=activeTurn;save();gate.beginTurn(thread,activeTurn);}waiter.resolve(frame.result);}return; + } + if(frame.method==='turn/completed'&&frame.params.threadId===thread){turns.set(frame.params.turn.id,frame.params.turn);evidence.turns.push({id:frame.params.turn.id,status:frame.params.turn.status});save();gate?.endTurn(thread,frame.params.turn.id);if(activeTurn===frame.params.turn.id){activeTurn=null;evidence.activeTurn=null;save();}} + if(frame.method==='item/agentMessage/delta'&&frame.params.threadId===thread)outputStream.write(frame.params.delta); + if(frame.id!==undefined&&frame.method)send({id:frame.id,error:{code:-32601,message:'No other host operations are authorized'}}); + }catch(error){fail(error);} + }); + await request('initialize',{clientInfo:{name:'firstmate-native',version:'0.1.0'},capabilities:{experimentalApi:true}});send({method:'initialized',params:{}}); + const dynamicTools=[ + {name:'fm_notification_check',description:'Read pending Firstmate notifications. Quiet means there is no new delivery.',inputSchema:{type:'object',properties:{},additionalProperties:false}}, + {name:'fm_notification_ack',description:'Acknowledge an observed and handled delivery. Never acknowledge unresolved decisions or unperformed work.',inputSchema:{type:'object',properties:{receipt:{type:'string'},observed:{type:'string'}},required:['receipt','observed'],additionalProperties:false}}, + ]; + const instructions='The native host already ran startup exactly once. Do not rerun startup or arm another supervisor. This experimental empty-fleet session supports only the two notification tools; do not claim to dispatch project work. The host continues notification checks after startup finishes. Use the supplied receipt and observed challenge only after handling the entire delivery. Unresolved work must remain pending. Startup digest follows:\n'+fs.readFileSync(path.join(runtime,'startup.log'),'utf8'); + const externalConfiguration=await verifyExternalToolConfiguration(request,mcpServerNames); + const started=await request('thread/start',{cwd:root,sandbox:'read-only',approvalPolicy:'never',ephemeral:true,developerInstructions:instructions,dynamicTools}); + if(started.sandbox?.type!=='readOnly'||started.sandbox.networkAccess!==false||started.approvalPolicy!=='never')throw Error('App-server returned an unexpected security policy'); + thread=started.thread.id;evidence.thread=thread;evidence.externalTools=await verifyExternalToolIsolation(request,thread,externalConfiguration);evidence.ready=true;evidence.policy={sandbox:started.sandbox,approval:started.approvalPolicy};evidence.digestDeliveredBeforeDeferred=!fs.existsSync(path.join(runtime,'startup.finished'));save(); + gate=createNotificationGate({primaryThread:thread,operate:operation,isAlive:()=>alive&&!closing}); + errorStream.write('Experimental native session ready. /interrupt stops the current turn; /quit ends the session.\n'); + while(!closing&&alive){ + if(env.FM_PROBE_VERIFY_ONLY==='1'){ + if(input.length)throw Error('Verify-only mode does not start model turns'); + if(ended)break;await pause(100);continue; + } + if(input.length){await turn(input.shift());continue;} + if(ended)break; + const result=await operation('check'); + if(result.operationState==='delivered'&&result.notification.receipt!==announced){ + const receipt=result.notification.receipt; + const handled=await turn('A new durable notification is available. Read it with fm_notification_check, handle it within the available authority, and acknowledge only if fully handled.'); + let outcome='interrupted'; + if(!closing&&alive&&handled?.status==='completed'){ + if(confirmAutomaticNotificationOffer(gate,thread,handled,receipt)){announced=receipt;outcome='offered';} + }else if(!closing&&alive&&handled?.status==='interrupted')await pause(1000); + evidence.automatic.push({turn:handled?.id,status:handled?.status,receipt,outcome});save(); + if(options.afterAutomaticTurn&&await options.afterAutomaticTurn({handled,receipt,outcome,evidence})===false)break; + }else await pause(1000); + } + } + }catch(error){if(!closing)fail(error);}finally{ + await stop(); + if(options.installSignalHandlers!==false){process.removeListener('SIGINT',onSigint);process.removeListener('SIGTERM',onSigterm);} + } + if(terminal)throw terminal; + return evidence; +} diff --git a/bin/native-owner/codex-host.mjs b/bin/native-owner/codex-host.mjs index 1546daf96ab..04fd85559de 100644 --- a/bin/native-owner/codex-host.mjs +++ b/bin/native-owner/codex-host.mjs @@ -1,188 +1,8 @@ -// Interactive native host. The controller has created the private pipe before -// this process resumes. Connect once: EOF or timeout ends this session, never -// reconnect to an endpoint that could now belong to another controller. -import fs from 'node:fs'; -import path from 'node:path'; -import net from 'node:net'; -import {spawn} from 'node:child_process'; -import {createInterface} from 'node:readline'; -import {confirmAutomaticNotificationOffer,createNotificationGate} from './codex-tool-gate.mjs'; -import {createHostLifecycle,reconciliationWarning} from './host-lifecycle.mjs'; -import {discoverMcpServerNames,isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from './app-server-policy.mjs'; -const runtime=process.env.FM_PROBE_HOME,root=process.env.FM_PROBE_CODE_ROOT; -const pause=ms=>new Promise(resolve=>setTimeout(resolve,ms)); -let closing=false,gate=null,server=null,alive=false,thread=null,activeTurn=null,ended=false; -let lifecycle=null,terminal=null,announced=null,next=0,nativeWaiter=null,nativeTail=Promise.resolve(); -const input=[],pending=new Map(),turns=new Map(); -const evidence={ready:false,turns:[],tools:[],digestDeliveredBeforeDeferred:false}; -const save=()=>{const file=path.join(runtime,'host.json');fs.writeFileSync(file+'.tmp',JSON.stringify(evidence,null,2));fs.renameSync(file+'.tmp',file);}; -const consoleInput=createInterface({input:process.stdin}); -consoleInput.on('line',line=>{ - if(line==='/quit'){void stop();return;} - if(line==='/interrupt'){void interrupt().catch(fail);return;} - input.push(line); -}); -consoleInput.on('close',()=>{ended=true;}); -process.on('SIGINT',()=>{if(activeTurn)void interrupt().catch(fail);else void stop();}); -process.on('SIGTERM',()=>{void stop();}); -const socket=net.createConnection('\\\\.\\pipe\\'+process.env.FM_PROBE_PIPE); -const connection=new Promise((resolve,reject)=>{ - const timer=setTimeout(()=>{socket.destroy();reject(Error('Native connection timed out'));},10000); - socket.once('connect',()=>{clearTimeout(timer);resolve();});socket.once('error',error=>{clearTimeout(timer);reject(error);}); -}); -const channel=createInterface({input:socket}); -channel.on('line',line=>{ - if(!nativeWaiter){fail(Error('Unexpected native response'));return;} - const waiter=nativeWaiter;nativeWaiter=null; - try {const value=JSON.parse(line);if(!value.notificationAuthorized)throw Error('Native host authorization refused');waiter.resolve(value);}catch(error){waiter.reject(error);} -}); -socket.on('error',error=>{nativeWaiter?.reject(error);nativeWaiter=null;if(!closing)fail(error);}); -socket.on('close',()=>{const error=Error('Native controller connection closed');nativeWaiter?.reject(error);nativeWaiter=null;if(!closing)fail(error);}); -function native(action,extra={},signal){ - const run=async()=>{ - await connection;if(socket.destroyed||signal?.aborted||(closing&&action!=='shutdown'))throw Error('Native session unavailable'); - return new Promise((resolve,reject)=>{ - let timer; - const abort=()=>{socket.destroy();finish(reject,Error('Native request interrupted; durable work preserved'));}; - const finish=(callback,value)=>{clearTimeout(timer);signal?.removeEventListener('abort',abort);callback(value);}; - nativeWaiter={resolve:value=>finish(resolve,value),reject:error=>finish(reject,error)}; - timer=setTimeout(abort,10000);signal?.addEventListener('abort',abort,{once:true}); - socket.write(JSON.stringify({kind:'notification',session:process.env.FM_PROBE_SESSION,home:runtime,nonce:process.env.FM_PROBE_NONCE,action,...extra})+'\n'); - }); - }; - const result=nativeTail.then(run);nativeTail=result.catch(()=>{});return result; -} -function send(frame){if(!alive)throw Error('App-server is not live');server.stdin.write(JSON.stringify(frame)+'\n');} -function request(method,params,signal){ - return new Promise((resolve,reject)=>{ - if(signal?.aborted){reject(Error('Request cancelled'));return;} - const id=++next; - let timer; - const clean=()=>{clearTimeout(timer);signal?.removeEventListener('abort',abort);}; - const abort=()=>{pending.delete(id);clean();reject(Error('App-server request cancelled'));}; - timer=setTimeout(()=>{pending.delete(id);clean();reject(Error('App-server '+method+' timed out'));},30000); - pending.set(id,{method,resolve:value=>{clean();resolve(value);},reject:error=>{clean();reject(error);}}); - signal?.addEventListener('abort',abort,{once:true}); - try {send({id,method,params});}catch(error){pending.delete(id);clean();reject(error);} - }); -} -async function interrupt(signal){ - if(!activeTurn||!alive)return; - const target=activeTurn; - await request('turn/interrupt',{threadId:thread,turnId:target},signal); - const limit=Date.now()+10000; - while(alive&&!turns.has(target)&&!signal?.aborted&&Date.now()gate?.close()},interrupt, - stopOperations:async signal=>{const value=await native('shutdown',{},signal);return {stopped:value.operationState==='stopped',reconciliationRequired:value.reconciliationRequired===true};}, - closeInput:()=>{if(server)server.stdin.end();}, - waitForExit:signal=>!alive?Promise.resolve(true):new Promise(resolve=>{ - const exit=()=>{signal.removeEventListener('abort',abort);resolve(true);}; - const abort=()=>{server.removeListener('exit',exit);resolve(false);}; - server.once('exit',exit);signal.addEventListener('abort',abort,{once:true}); - }),terminate:()=>server?.kill(),graceMs:3000}); - const result=lifecycle.shutdown(); - void result.then(value=>{ - fs.writeFileSync(path.join(runtime,'shutdown.json'),JSON.stringify(value)); - if(value.reconciliationRequired)console.error(reconciliationWarning(path.join(process.env.FM_HOME,'owner-receipts.jsonl'))); - if(!value.stopped){console.error('Shutdown was not fully confirmed; durable work was preserved.');process.exitCode=1;} - channel.close();socket.destroy(); - }); - return result; -} +import {runCodexHost} from './codex-host-runtime.mjs'; + try { - await connection; - let status=await native('status'); - if(status.operationState==='reconciliation-required')throw Error('An earlier acknowledgement is incomplete. Its records are preserved; review '+path.join(process.env.FM_HOME,'owner-receipts.jsonl')+' before starting more work.'); - const limit=Date.now()+200000; - while(!closing&&!fs.existsSync(path.join(runtime,'digest.ready'))){ - status=await native('status');if(status.operationState==='startup-failed'||Date.now()>limit)throw Error('Startup failed; inspect '+path.join(runtime,'startup.log')); - await pause(100); - } - if(!closing){ - const executable=path.join(process.env.APPDATA,'npm/node_modules/@openai/codex/node_modules/@openai/codex-win32-x64/vendor/x86_64-pc-windows-msvc/bin/codex.exe'); - const mcpServerNames=await discoverMcpServerNames(executable,root); - server=spawn(executable,isolatedAppServerArgs(mcpServerNames,['-c','windows.sandbox=unelevated']),{cwd:root,stdio:['pipe','pipe','pipe'],detached:true,windowsHide:true});alive=true; - server.stderr.on('data',data=>fs.appendFileSync(path.join(runtime,'app-server.stderr'),data)); - const lost=error=>{alive=false;for(const waiter of pending.values())waiter.reject(error);pending.clear();if(!closing)fail(error);}; - server.on('error',lost);server.on('exit',()=>lost(Error('App-server exited')));server.stdin.on('error',error=>{if(!closing)fail(error);}); - createInterface({input:server.stdout}).on('line',line=>{ - try { - const frame=JSON.parse(line); - if(frame.method==='item/tool/call'&&frame.id!==undefined){void tool(frame).catch(fail);return;} - if(frame.id!==undefined&&pending.has(frame.id)){ - const waiter=pending.get(frame.id);pending.delete(frame.id); - if(frame.error)waiter.reject(Error(JSON.stringify(frame.error))); - else {if(waiter.method==='turn/start'){activeTurn=frame.result.turn.id;evidence.activeTurn=activeTurn;save();gate.beginTurn(thread,activeTurn);}waiter.resolve(frame.result);}return; - } - if(frame.method==='turn/completed'&&frame.params.threadId===thread){turns.set(frame.params.turn.id,frame.params.turn);evidence.turns.push({id:frame.params.turn.id,status:frame.params.turn.status});save();gate?.endTurn(thread,frame.params.turn.id);if(activeTurn===frame.params.turn.id){activeTurn=null;evidence.activeTurn=null;save();}} - if(frame.method==='item/agentMessage/delta'&&frame.params.threadId===thread)process.stdout.write(frame.params.delta); - if(frame.id!==undefined&&frame.method)send({id:frame.id,error:{code:-32601,message:'No other host operations are authorized'}}); - }catch(error){fail(error);} - }); - await request('initialize',{clientInfo:{name:'firstmate-native',version:'0.1.0'},capabilities:{experimentalApi:true}});send({method:'initialized',params:{}}); - const dynamicTools=[ - {name:'fm_notification_check',description:'Read pending Firstmate notifications. Quiet means there is no new delivery.',inputSchema:{type:'object',properties:{},additionalProperties:false}}, - {name:'fm_notification_ack',description:'Acknowledge an observed and handled delivery. Never acknowledge unresolved decisions or unperformed work.',inputSchema:{type:'object',properties:{receipt:{type:'string'},observed:{type:'string'}},required:['receipt','observed'],additionalProperties:false}}, - ]; - const instructions='The native host already ran startup exactly once. Do not rerun startup or arm another supervisor. This experimental empty-fleet session supports only the two notification tools; do not claim to dispatch project work. The host continues notification checks after startup finishes. Use the supplied receipt and observed challenge only after handling the entire delivery. Unresolved work must remain pending. Startup digest follows:\n'+fs.readFileSync(path.join(runtime,'startup.log'),'utf8'); - const externalConfiguration=await verifyExternalToolConfiguration(request,mcpServerNames); - const started=await request('thread/start',{cwd:root,sandbox:'read-only',approvalPolicy:'never',ephemeral:true,developerInstructions:instructions,dynamicTools}); - if(started.sandbox?.type!=='readOnly'||started.sandbox.networkAccess!==false||started.approvalPolicy!=='never')throw Error('App-server returned an unexpected security policy'); - thread=started.thread.id;evidence.thread=thread;evidence.externalTools=await verifyExternalToolIsolation(request,thread,externalConfiguration);evidence.ready=true;evidence.policy={sandbox:started.sandbox,approval:started.approvalPolicy};evidence.digestDeliveredBeforeDeferred=!fs.existsSync(path.join(runtime,'startup.finished'));save(); - gate=createNotificationGate({primaryThread:thread,operate:operation,isAlive:()=>alive&&!closing}); - console.error('Experimental native session ready. /interrupt stops the current turn; /quit ends the session.'); - while(!closing&&alive){ - if(process.env.FM_PROBE_VERIFY_ONLY==='1'){ - if(input.length)throw Error('Verify-only mode does not start model turns'); - if(ended)break;await pause(100);continue; - } - if(input.length){await turn(input.shift());continue;} - if(ended)break; - const result=await operation('check'); - if(result.operationState==='delivered'&&result.notification.receipt!==announced){ - const receipt=result.notification.receipt; - const handled=await turn('A new durable notification is available. Read it with fm_notification_check, handle it within the available authority, and acknowledge only if fully handled.'); - if(!closing&&alive&&confirmAutomaticNotificationOffer(gate,thread,handled,receipt))announced=receipt; - else if(!closing&&alive&&handled?.status==='interrupted')await pause(1000); - } - else await pause(1000); - } - } -}catch(error){if(!closing)fail(error);}finally{await stop();} + await runCodexHost(); +} catch(error) { + console.error(error.message); + process.exitCode=1; +} diff --git a/docs/native-windows-codex.md b/docs/native-windows-codex.md index 21d4d0efe52..23d4c01154d 100644 --- a/docs/native-windows-codex.md +++ b/docs/native-windows-codex.md @@ -2,7 +2,6 @@ This explicit opt-in launcher is a restricted experimental candidate, not an installed runtime backend or a production-ready integration. Ordinary startup never selects it, and it does not install hooks, alter saved or global Codex settings, pull images, or select a default backend. -[`verification/runtime-backends.md`](verification/runtime-backends.md#experimental-native-windows-ownership-candidate) provides the verification refresh entry points; no current-build output is recorded there yet. ## Setup @@ -22,6 +21,42 @@ Omit `-VerifyOnly` for an interactive model session. Rebuild the native provider after its source stamp changes and after all native sessions have stopped. Use `/interrupt` to interrupt the current model turn and `/quit` to end the session. +## Verification entry points + +Run the portable request-policy regression with: + +```sh +bash tests/fm-native-owner-tool-gate.test.sh +``` + +Run native receipt persistence and operation-lifetime checks with: + +```sh +bash tests/fm-native-owner-receipt-live-e2e.test.sh +``` + +Run effective app and MCP isolation without a model turn with: + +```sh +FM_LIVE_NATIVE_APP_POLICY=1 bash tests/fm-native-owner-app-server-policy-live-e2e.test.sh +``` + +Run the actual Windows integration with the explicit two-model-turn guard with: + +```sh +FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_CODEX=1 \ + bash tests/fm-native-owner-codex-live-e2e.test.sh +``` + +Run the explicit launcher without model turns, then optionally exercise two notification turns and active-turn cancellation with: + +```sh +FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 \ + bash tests/fm-native-owner-launcher-live-e2e.test.sh +FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 FM_LIVE_NATIVE_CODEX=1 \ + bash tests/fm-native-owner-launcher-live-e2e.test.sh +``` + ## Safety boundary and limits Only empty-fleet homes beneath the current user's Windows temporary directory are accepted. diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 831f8caaef5..8beaa43a981 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -6,47 +6,6 @@ This record contains reusable version-scoped evidence for active runtime guarant The backend guides own current setup, safety boundaries, and limitations. Exact task chronology, branch names, temporary homes, local paths, process ids, thread ids, and delivery transcripts remain in private reports or PR evidence. -## Experimental native Windows ownership candidate - -This section retains refresh entry points only; no current-build output has been recorded by the verification owner. -[`../native-windows-codex.md`](../native-windows-codex.md) owns current setup, safety boundaries, and supported limits for this isolated candidate. - -Refresh the portable request-policy regression with: - -```sh -bash tests/fm-native-owner-tool-gate.test.sh -``` - -Refresh native receipt persistence and operation lifetime with: - -```sh -bash tests/fm-native-owner-receipt-live-e2e.test.sh -``` - -Refresh effective app and MCP isolation without a model turn: - -```sh -FM_LIVE_NATIVE_APP_POLICY=1 bash tests/fm-native-owner-app-server-policy-live-e2e.test.sh -``` - -Refresh the actual Windows integration with the explicit two-model-turn guard: - -```sh -FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_CODEX=1 \ - bash tests/fm-native-owner-codex-live-e2e.test.sh -``` - -### Explicit launcher integration - -Refresh the actual launcher without model turns, then optionally exercise two notification turns and active-turn cancellation: - -```sh -FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 \ - bash tests/fm-native-owner-launcher-live-e2e.test.sh -FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 FM_LIVE_NATIVE_CODEX=1 \ - bash tests/fm-native-owner-launcher-live-e2e.test.sh -``` - ## Harness detection precedence `bin/fm-harness.sh` owns native-owner, marker, and ancestry precedence; the evidence below covers marker and ancestry only, not the experimental native candidate. diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 6abcda4fbb0..2c253086de0 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -63,10 +63,22 @@ const again=start(home);const restarted=await ready(again,initial.owner.generati assert.equal(fs.readFileSync(path.join(home,'state/.lock'),'utf8').trim(),'native:'+restarted.owner.generation); again.child.stdin.end();assert.equal((await bound(again.done,again,20000)).exit,0); records.push('restart after an intervening failed startup retains proven-dead ownership history'); +const titled=path.join(area,'titled-empty');fs.mkdirSync(path.join(titled,'data'),{recursive:true}); +fs.writeFileSync(path.join(titled,'data/backlog.md'),'# Backlog\r\n'); +const titledSession=start(titled);await ready(titledSession);titledSession.child.stdin.end(); +assert.equal((await bound(titledSession.done,titledSession,20000)).exit,0); +records.push('owner-provided backlog admission accepts the canonical CRLF title-only skeleton'); const populated=path.join(area,'populated');fs.mkdirSync(path.join(populated,'state'),{recursive:true});fs.writeFileSync(path.join(populated,'state/work.meta'),'preserve'); const blocked=start(populated);blocked.child.stdin.end();assert.notEqual((await bound(blocked.done,blocked,20000)).exit,0); assert.equal(fs.readFileSync(path.join(populated,'state/work.meta'),'utf8'),'preserve');assert.equal(fs.existsSync(path.join(populated,'owner-probe.json')),false); records.push('populated home refused without changing its records'); +for(const [name,relative] of [['orphan-status','state/orphan.status'],['interrupted-close','state/orphan.backlog-close']]){ + const residualHome=path.join(area,name),record=path.join(residualHome,relative),contents='preserve residual task state'; + fs.mkdirSync(path.dirname(record),{recursive:true});fs.writeFileSync(record,contents); + const refusedResidual=start(residualHome);refusedResidual.child.stdin.end();assert.notEqual((await bound(refusedResidual.done,refusedResidual,20000)).exit,0); + assert.equal(fs.readFileSync(record,'utf8'),contents);assert.equal(fs.existsSync(path.join(residualHome,'owner-probe.json')),false); +} +records.push('orphan status and interrupted-close records refused before lease acquisition and preserved'); for(const [name,contents] of [ ['queued-backlog','## In flight\n\n## Queued\n- [ ] queued-work - preserved project work (repo: firstmate) (kind: ship)\n\n## Done\n'], ['unrecognized-backlog','# Backlog\n\nproject work in an unrecognized form\n'], diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index 43180355e30..e02d2e8a113 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -24,6 +24,8 @@ static void Targets(NativeHomeLease lease) { Directory.CreateDirectory(Path.GetDirectoryName(Handled(lease))); File.WriteAllText(Pending(lease),"original captured inbox record\n"); File.WriteAllText(Queue(lease),"1\t1\tcheck\tinbox:note-id\tcaptain inbox note\n"); + File.WriteAllText(Path.Combine(lease.Home,"state",".main-eligible-rows"),"1\n"); + File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"pending:handling:recovery\n"); } static string Quote(string value) { return "\""+value.Replace("\"","\\\"")+"\""; } static string ZeroRecovery(NativeHomeLease lease,string action,string generation=null) { @@ -181,6 +183,7 @@ public static int Run() { payload["notes"]=new [] {"note-id","second"};payload["seq"]="2"; File.WriteAllText(Path.Combine(lease.Home,"state","inbox","second.note"),"second notification"); File.AppendAllText(Queue(lease),"2\t2\tcheck\tinbox:second\tsecond\n"); + File.WriteAllText(Path.Combine(lease.Home,"state",".main-eligible-rows"),"1\n2\n"); } using(var journal=new NativeReceiptJournal(lease,A)) { var delivery=journal.Present(payload); diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs index 1b2d6c051e5..c29769c5ddd 100644 --- a/tests/fixtures/native-owner/Run-Cycle.mjs +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -32,7 +32,8 @@ const spec={home,leaseHome,executable:process.execPath,arguments:'"'+script+'"', if(startupQueued){spec.startupQueued=true;spec.startupNote=startupNote;} if(fault)spec.ackFault=fault; const file=path.join(home,'spec.json');fs.writeFileSync(file,JSON.stringify(spec,null,2)); -const run=spawnSync(build.binary,['run',file],{encoding:'utf8',timeout:310000}); +const controllerEnv={...process.env,FM_PROBE_CODE_ROOT:repo}; +const run=spawnSync(build.binary,['run',file],{env:controllerEnv,encoding:'utf8',timeout:310000}); fs.writeFileSync(path.join(home,'controller.stdout'),run.stdout||'');fs.writeFileSync(path.join(home,'controller.stderr'),run.stderr||''); fs.writeFileSync(path.join(dir,fault?`fault-${fault}-latest.json`:dry?'bridge-latest.json':'cycle-latest.json'),JSON.stringify({home,exit:run.status},null,2)); console.log(JSON.stringify({home,exit:run.status,dry})); @@ -57,7 +58,7 @@ if(fault) { const recovery=path.join(home,'recovery');fs.mkdirSync(recovery); const recoverySpec={home:recovery,leaseHome:spec.leaseHome,executable:build.binary,arguments:'sleep 100',timeoutSeconds:10,pipeAcl:'UserOnly'}; const recoveryFile=path.join(recovery,'spec.json');fs.writeFileSync(recoveryFile,JSON.stringify(recoverySpec)); - const restarted=spawnSync(build.binary,['run',recoveryFile],{encoding:'utf8',timeout:15000}); + const restarted=spawnSync(build.binary,['run',recoveryFile],{env:controllerEnv,encoding:'utf8',timeout:15000}); fs.writeFileSync(path.join(recovery,'controller.stdout'),restarted.stdout||'');fs.writeFileSync(path.join(recovery,'controller.stderr'),restarted.stderr||''); if(restarted.status!==0)throw Error('Recovery controller failed'); const recovered=read(path.join(recovery,'result.json')); diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index 1fdf9d45578..fe58f20ff7c 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -53,10 +53,8 @@ assert.equal(fs.readFileSync(path.join(operational,'.wake-queue'),'utf8').trim() const handled=path.join(operational,'inbox/handled',delivered.note+'.note'); assert.ok(fs.existsSync(handled)); const targets=journal[2].targetEvidence; -assert.equal(targets.note,delivered.note);assert.equal(targets.cutoff,delivered.seq); -assert.equal(targets.noteSha256,hash(handled)); -assert.ok(targets.rows.some(row=>row.split('\t')[3]===`inbox:${delivered.note}`)); -assert.deepEqual(journal[3].targetEvidence,targets); +assert.equal(typeof targets,'string');assert.ok(targets.length>0); +assert.equal(journal[3].targetEvidence,targets); const threads=host.frames.filter(frame=>frame.result?.thread); assert.equal(threads.length,2); for(const thread of threads){assert.equal(thread.result.sandbox.type,'readOnly');assert.equal(thread.result.sandbox.networkAccess,false);assert.equal(thread.result.approvalPolicy,'never');} diff --git a/tests/fixtures/native-owner/fake-app-server.mjs b/tests/fixtures/native-owner/fake-app-server.mjs new file mode 100644 index 00000000000..8e6252ae08e --- /dev/null +++ b/tests/fixtures/native-owner/fake-app-server.mjs @@ -0,0 +1,58 @@ +import {EventEmitter} from 'node:events'; +import {PassThrough} from 'node:stream'; +import {createInterface} from 'node:readline'; + +export function createFakeAppServer(scenario){ + const server=new EventEmitter(),input=new PassThrough(); + server.stdin=input;server.stdout=new PassThrough();server.stderr=new PassThrough(); + let toolStage='',exited=false; + const finish=()=>{ + if(exited)return; + exited=true;server.stdout.end();server.stderr.end(); + setImmediate(()=>server.emit('exit',0,null)); + }; + const fail=error=>{server.stderr.write(error.message+'\n');server.emit('error',error);finish();}; + const send=value=>server.stdout.write(JSON.stringify(value)+'\n'); + const thread='primary',turn='automatic-turn'; + const complete=()=>send({method:'turn/completed',params:{threadId:thread,turn:{id:turn,status:'completed'}}}); + const call=(id,tool,args,overrides={})=>send({id,method:'item/tool/call',params:{threadId:thread,turnId:turn,callId:'call-'+id,namespace:null,tool,arguments:args,...overrides}}); + const lines=createInterface({input}); + lines.on('line',line=>{ + try { + const frame=JSON.parse(line); + if(frame.method==='initialized')return; + if(frame.id===900){ + if(scenario==='success'){ + const result=JSON.parse(frame.result.contentItems[0].text); + toolStage='ack';call(901,'fm_notification_ack',{receipt:result.receipt,observed:result.challenge}); + }else complete(); + return; + } + if(frame.id===901){complete();return;} + if(frame.method==='initialize'){send({id:frame.id,result:{}});return;} + if(frame.method==='config/read'){send({id:frame.id,result:{config:{features:{apps:false,plugins:false},mcp_servers:{}}}});return;} + if(frame.method==='thread/start'){ + send({id:frame.id,result:{thread:{id:thread},sandbox:{type:'readOnly',networkAccess:false},approvalPolicy:'never'}});return; + } + if(frame.method==='app/installed'){send({id:frame.id,result:{apps:[]}});return;} + if(frame.method==='mcpServerStatus/list'){send({id:frame.id,result:{data:[],nextCursor:null}});return;} + if(frame.method==='turn/start'){ + send({id:frame.id,result:{turn:{id:turn}}}); + queueMicrotask(()=>{ + if(scenario==='prose')complete(); + else if(scenario==='denied'){toolStage='check';call(900,'fm_notification_check',{}, {namespace:'other'});} + else if(scenario==='malformed'){toolStage='check';call(900,'fm_notification_check',null);} + else if(scenario==='success'){toolStage='check';call(900,'fm_notification_check',{});} + else fail(Error('unknown scenario '+scenario)); + }); + return; + } + if(frame.method==='turn/interrupt'){send({id:frame.id,result:{}});return;} + if(frame.id!==undefined&&frame.error)return; + throw Error('unexpected frame '+JSON.stringify({frame,toolStage})); + }catch(error){fail(error);} + }); + lines.on('close',finish); + server.kill=()=>{lines.close();input.destroy();finish();return true;}; + return server; +} diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index 2961f1b6013..688444c6aeb 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -1,6 +1,13 @@ import test from 'node:test'; import assert from 'node:assert/strict'; -import {confirmAutomaticNotificationOffer,createNotificationGate} from '../../../bin/native-owner/codex-tool-gate.mjs'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import {PassThrough} from 'node:stream'; +import {fileURLToPath} from 'node:url'; +import {createNotificationGate} from '../../../bin/native-owner/codex-tool-gate.mjs'; +import {runCodexHost} from '../../../bin/native-owner/codex-host-runtime.mjs'; +import {createFakeAppServer} from './fake-app-server.mjs'; const message={receipt:'receipt',challenge:'observed',message:'Controlled message',checkpointExit:124}; function fixture(operate) { const calls=[];let alive=true; @@ -106,19 +113,65 @@ test('operation failure is not reported as success and cannot be blindly retried const {gate,calls}=fixture(()=>{throw Error('partial operation requires reconciliation');}); assert.equal((await gate.handle(check())).success,false);assert.equal((await gate.handle(check({callId:'retry'}))).success,false);assert.equal(calls.length,1); }); -test('completed prose-only automatic turn preserves the unoffered receipt',()=>{ - const {gate}=fixture();gate.endTurn('primary','turn'); - assert.throws(()=>confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt'),/durable work was preserved/); -}); -for(const [name,request] of [ - ['denied',check({namespace:'other'})], - ['malformed',check({arguments:null})], -])test(`completed automatic turn with a ${name} check preserves the receipt`,async()=>{ - const {gate}=fixture();assert.equal((await gate.handle(request)).success,false);gate.endTurn('primary','turn'); - assert.throws(()=>confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt'),/durable work was preserved/); -}); -test('completed automatic turn suppresses only the exact successfully offered receipt',async()=>{ - const {gate}=fixture();assert.equal((await gate.handle(check())).success,true);gate.endTurn('primary','turn'); - assert.equal(confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt'),true); - assert.throws(()=>confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'other')); +const repo=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..'); +async function hostScenario(scenario){ + const area=fs.mkdtempSync(path.join(os.tmpdir(),'fm-native-host-loop-')); + const runtime=path.join(area,'runtime'),home=path.join(area,'home'); + fs.mkdirSync(runtime,{recursive:true});fs.mkdirSync(home,{recursive:true}); + fs.writeFileSync(path.join(runtime,'digest.ready'),'ready\n'); + fs.writeFileSync(path.join(runtime,'startup.log'),'deterministic startup digest\n'); + fs.writeFileSync(path.join(runtime,'startup.finished'),'finished\n'); + const input=new PassThrough(),output=new PassThrough(),error=new PassThrough(); + const notification={receipt:'receipt',challenge:'observed',message:'Controlled message',checkpointExit:124}; + let ackPending=false,acknowledgements=0,shutdowns=0,afterShutdown=0; + const native=async(action,extra)=>{ + if(shutdowns&&action!=='shutdown')afterShutdown++; + if(action==='status')return {operationState:'ready'}; + if(action==='result'){ + if(ackPending){ackPending=false;return {operationState:'acknowledged'};} + return {operationState:'delivered',notification}; + } + if(action==='ack'){ + assert.deepEqual(extra,{receipt:'receipt',observed:'observed'}); + acknowledgements++;ackPending=true;return {operationState:'pending'}; + } + if(action==='shutdown'){shutdowns++;return {operationState:'stopped',reconciliationRequired:false};} + throw Error('unexpected native action '+action); + }; + let result,failure; + try { + result=await runCodexHost({ + env:{...process.env,FM_PROBE_HOME:runtime,FM_PROBE_CODE_ROOT:repo,FM_HOME:home,FM_PROBE_SESSION:'session',FM_PROBE_NONCE:'nonce'}, + input,output,error,native,mcpServerNames:[],spawnAppServer:()=>createFakeAppServer(scenario),installSignalHandlers:false, + afterAutomaticTurn:()=>{ + setTimeout(()=>input.write('/quit\n'),20); + return true; + }, + }); + }catch(errorValue){failure=errorValue;} + return {result,failure,acknowledgements,shutdowns,afterShutdown,host:JSON.parse(fs.readFileSync(path.join(runtime,'host.json'),'utf8'))}; +} + +for(const scenario of ['prose','denied','malformed'])test(`actual host loop preserves a ${scenario} completed turn`,async()=>{ + const result=await hostScenario(scenario); + assert.match(result.failure?.message??'',/durable work was preserved/); + assert.equal(result.acknowledgements,0); + assert.equal(result.shutdowns,1); + assert.equal(result.afterShutdown,0); + assert.deepEqual(result.host.turns.map(turn=>turn.status),['completed']); + if(scenario==='prose')assert.equal(result.host.tools.length,0); + else assert.equal(result.host.tools[0].success,false); +}); + +test('actual host loop suppresses only the exact successfully offered receipt',async()=>{ + const result=await hostScenario('success'); + assert.equal(result.failure,undefined); + assert.equal(result.acknowledgements,1); + assert.equal(result.shutdowns,1); + assert.equal(result.afterShutdown,0); + assert.deepEqual(result.result.automatic.map(item=>item.outcome),['offered']); + assert.deepEqual(result.host.turns.map(turn=>turn.status),['completed']); + assert.deepEqual(result.host.tools.map(tool=>[tool.tool,tool.success]),[ + ['fm_notification_check',true],['fm_notification_ack',true], + ]); }); From c96913103ba83189dd265cb0c2e9f107d163d60c Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 23:18:56 +1200 Subject: [PATCH 17/61] no-mistakes(review): Harden native admission and acknowledgement recovery --- bin/fm-wake-lib.sh | 103 +++++++++++++++++- .../NativeAcknowledgementEvidence.cs | 22 +--- bin/native-owner/NativeLauncher.cs | 18 +-- bin/native-owner/NativeOperations.cs | 32 ++++++ bin/native-owner/ack-evidence.sh | 38 +------ bin/native-owner/ack.sh | 3 +- bin/native-owner/admit.sh | 8 ++ bin/native-owner/check.sh | 3 +- bin/native-owner/codex-host-runtime.mjs | 2 +- bin/native-owner/startup.sh | 3 +- bin/native-owner/tools/jq | 3 +- docs/documentation-audiences.json | 4 - docs/watcher-continuity.md | 2 + tests/fixtures/native-owner/Build.ps1 | 19 ++-- tests/fixtures/native-owner/Launcher.mjs | 17 ++- tests/fixtures/native-owner/NativeDriver.cs | 23 +++- tests/fixtures/native-owner/ReceiptTests.cs | 69 +++++++++--- tests/fixtures/native-owner/Run-Cycle.mjs | 7 +- tests/fixtures/native-owner/Verify-Cycle.mjs | 2 +- tests/fixtures/native-owner/zero-recovery.sh | 5 + 20 files changed, 263 insertions(+), 120 deletions(-) diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 789c018ccee..863f9bf7b83 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -623,9 +623,63 @@ fm_recovery_marker_read() { case "${line##*:}" in ''|*[!A-Za-z0-9._-]*) return 1 ;; esac + [ "${#line}" -le 160 ] || return 1 FM_RECOVERY_MARKER_TOKEN=$line } +_fm_recovery_completion_has_locked() { + local marker=$1 generation=$2 file="${1}.ack-completions" size + [ "${#generation}" -le 128 ] || return 2 + if [ ! -e "$file" ] && [ ! -L "$file" ]; then + return 1 + fi + [ -f "$file" ] && [ ! -L "$file" ] || return 2 + size=$(wc -c < "$file" 2>/dev/null | tr -d '[:space:]') || return 2 + case "$size" in ''|*[!0-9]*) return 2 ;; esac + [ "$size" -le 262144 ] || return 2 + LC_ALL=C awk -v target="$generation" ' + NR == 1 { if ($0 != "fm-wake-ack-completions-v1") bad=1; next } + length($0) > 128 || $0 !~ /^[A-Za-z0-9._-]+$/ || seen[$0]++ { bad=1 } + $0 == target { found=1 } + NR > 1025 { bad=1 } + END { + if (NR < 1 || bad) exit 2 + if (found) exit 0 + exit 1 + } + ' "$file" +} + +_fm_recovery_completion_preserve_locked() { + local marker=$1 generation=$2 file="${1}.ack-completions" status count tmp + [ "${#generation}" -le 128 ] || return 1 + if _fm_recovery_completion_has_locked "$marker" "$generation"; then + return 0 + else + status=$? + fi + case "$status" in + 1) ;; + *) return 1 ;; + esac + if [ -e "$file" ] || [ -L "$file" ]; then + count=$(awk 'END { print NR - 1 }' "$file") || return 1 + [ "$count" -lt 1024 ] || return 1 + fi + tmp=$(mktemp "${file}.tmp.XXXXXX") || return 1 + if [ -e "$file" ]; then + cp "$file" "$tmp" || { rm -f -- "$tmp"; return 1; } + else + printf 'fm-wake-ack-completions-v1\n' > "$tmp" || { rm -f -- "$tmp"; return 1; } + fi + if ! printf '%s\n' "$generation" >> "$tmp" \ + || ! chmod 0600 "$tmp" \ + || ! _fm_atomic_replace "$tmp" "$file"; then + rm -f -- "$tmp" + return 1 + fi +} + _fm_atomic_replace() { mv -f -- "$1" "$2" } @@ -673,6 +727,16 @@ _fm_recovery_marker_publish() { generation=${FM_RECOVERY_MARKER_TOKEN##*:} status=announced ;; + acked:handling:*|acked:downtime:*) + generation=${FM_RECOVERY_MARKER_TOKEN##*:} + if ! _fm_recovery_completion_preserve_locked "$marker" "$generation"; then + FM_RECOVERY_MARKER_TOKEN=$saved_token + fm_lock_release "$lock" + return 1 + fi + generation='' + status=pending + ;; esac fi FM_RECOVERY_MARKER_TOKEN=$saved_token @@ -728,6 +792,34 @@ fm_recovery_marker_snapshot() { fm_lock_release "$lock" } +FM_RECOVERY_COMPLETION_SOURCE= +fm_recovery_marker_completed() { + local marker=$1 generation=$2 lock status + FM_RECOVERY_COMPLETION_SOURCE= + lock="${marker}.lock" + fm_lock_acquire_wait "$lock" || return 1 + if ! fm_recovery_marker_read "$marker"; then + fm_lock_release "$lock" + return 1 + fi + if [ "$FM_RECOVERY_MARKER_TOKEN" = "acked:handling:$generation" ] \ + || [ "$FM_RECOVERY_MARKER_TOKEN" = "acked:downtime:$generation" ]; then + FM_RECOVERY_COMPLETION_SOURCE=current + fm_lock_release "$lock" + return 0 + fi + if _fm_recovery_completion_has_locked "$marker" "$generation"; then + status=0 + else + status=$? + fi + if [ "$status" -eq 0 ]; then + FM_RECOVERY_COMPLETION_SOURCE=history + fi + fm_lock_release "$lock" + return "$status" +} + _fm_recovery_marker_ack() { local marker=$1 expected_generation=$2 lock tmp line [ -n "$expected_generation" ] || return 2 @@ -752,6 +844,10 @@ _fm_recovery_marker_ack() { fm_lock_release "$lock" return 1 fi + if ! _fm_recovery_completion_preserve_locked "$marker" "$expected_generation"; then + fm_lock_release "$lock" + return 1 + fi fm_lock_release "$lock" } @@ -2096,7 +2192,7 @@ fm_wake_ack_note_safe() { # fm_wake_ack_evidence_clear() { [ -z "$FM_WAKE_ACK_EVIDENCE_ROWS" ] || rm -f -- "$FM_WAKE_ACK_EVIDENCE_ROWS" [ -z "$FM_WAKE_ACK_EVIDENCE_NOTES" ] || rm -f -- "$FM_WAKE_ACK_EVIDENCE_NOTES" \ - "$FM_WAKE_ACK_EVIDENCE_NOTES.hashes" "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" "$FM_WAKE_ACK_EVIDENCE_NOTES.actual" + "$FM_WAKE_ACK_EVIDENCE_NOTES.hashes" FM_WAKE_ACK_EVIDENCE_TOKEN= FM_WAKE_ACK_EVIDENCE_CUTOFF= FM_WAKE_ACK_EVIDENCE_GENERATION= @@ -2245,6 +2341,7 @@ fm_wake_ack_evidence_load() { # IFS=$'\t' read -r tag FM_WAKE_ACK_EVIDENCE_GENERATION extra <&7 || true [ "$tag" = generation ] && [ -z "$extra" ] || { exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; } case "$FM_WAKE_ACK_EVIDENCE_GENERATION" in ''|*[!A-Za-z0-9._-]*) exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1 ;; esac + [ "${#FM_WAKE_ACK_EVIDENCE_GENERATION}" -le 128 ] || { exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; } IFS=$'\t' read -r tag value extra <&7 || true if [ "$tag" != marker ] || [ -n "$extra" ] || ! FM_WAKE_ACK_EVIDENCE_MARKER=$(printf '%s' "$value" | base64 -d 2>/dev/null); then exec 7<&-; rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1 @@ -2345,8 +2442,8 @@ fm_wake_ack_evidence_completed() { # return 1 fi if [ "$FM_WAKE_ACK_EVIDENCE_CUTOFF" = 0 ]; then - fm_recovery_marker_snapshot "$marker" || true - if [ "$FM_RECOVERY_MARKER_TOKEN" != "acked:handling:$FM_WAKE_ACK_EVIDENCE_GENERATION" ] || [ -s "$FM_WAKE_QUEUE" ]; then + if ! fm_recovery_marker_completed "$marker" "$FM_WAKE_ACK_EVIDENCE_GENERATION" \ + || { [ "$FM_RECOVERY_COMPLETION_SOURCE" = current ] && [ -s "$FM_WAKE_QUEUE" ]; }; then fm_wake_ack_evidence_clear return 1 fi diff --git a/bin/native-owner/NativeAcknowledgementEvidence.cs b/bin/native-owner/NativeAcknowledgementEvidence.cs index 35fed435dca..39bcbfce212 100644 --- a/bin/native-owner/NativeAcknowledgementEvidence.cs +++ b/bin/native-owner/NativeAcknowledgementEvidence.cs @@ -10,20 +10,11 @@ public sealed class NativeAcknowledgementEvidence { readonly string record; NativeAcknowledgementEvidence(NativeHomeLease lease,string value) { Lease=lease;record=value; } internal object Record() { return record; } - static string Quote(string value) { return "\""+value.Replace("\"","\\\"")+"\""; } - static string CodeRoot() { - string configured=Environment.GetEnvironmentVariable("FM_PROBE_CODE_ROOT"); - if(!string.IsNullOrEmpty(configured))return Path.GetFullPath(configured); - return Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory,"..")); - } static int Invoke(NativeHomeLease lease,string mode,string input,out string output) { if(lease==null || !lease.IsHeld)throw new InvalidOperationException("An active home lease is required"); - string script=Path.Combine(CodeRoot(),"bin","native-owner","ack-evidence.sh"); - var start=new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script.Replace('\\','/'))+" "+mode) { - UseShellExecute=false,RedirectStandardInput=true,RedirectStandardOutput=true,RedirectStandardError=true,CreateNoWindow=true - }; - start.EnvironmentVariables["FM_HOME"]=lease.Home; - start.EnvironmentVariables["MSYS"]="winsymlinks:nativestrict"; + string script=Path.Combine(NativeOwner.CodeRoot,"bin","native-owner","ack-evidence.sh"); + var start=NativeOwner.BashHelper(script,mode,lease.Home); + start.RedirectStandardInput=true;start.RedirectStandardOutput=true;start.RedirectStandardError=true; using(var process=Process.Start(start)) { var stdout=process.StandardOutput.ReadToEndAsync();var stderr=process.StandardError.ReadToEndAsync(); if(input!=null)process.StandardInput.Write(input); @@ -38,11 +29,8 @@ public static NativeAcknowledgementEvidence Capture(NativeHomeLease lease,Dictio if(payload==null)throw new IOException("Notification payload is incomplete"); object value;string opaque=null,output; if(payload.TryGetValue("ownerEvidence",out value))opaque=value as string; - if(string.IsNullOrEmpty(opaque)) { - var json=new JavaScriptSerializer(); - if(Invoke(lease,"capture-payload",json.Serialize(payload),out output)!=0 || string.IsNullOrEmpty(output))throw new IOException("Acknowledgement evidence was not captured"); - opaque=output; - } else if(Invoke(lease,"preflight-token",opaque,out output)!=0)throw new IOException("Acknowledgement evidence no longer matches its owner target"); + if(string.IsNullOrEmpty(opaque))throw new IOException("Acknowledgement evidence is missing"); + if(Invoke(lease,"preflight-token",opaque,out output)!=0)throw new IOException("Acknowledgement evidence no longer matches its owner target"); return new NativeAcknowledgementEvidence(lease,opaque); } internal static bool Completed(NativeHomeLease lease,object evidence) { diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs index bad1bf53e5f..798687c5601 100644 --- a/bin/native-owner/NativeLauncher.cs +++ b/bin/native-owner/NativeLauncher.cs @@ -12,22 +12,6 @@ using System.Threading; public static partial class NativeOwner { [DllImport("kernel32.dll")] static extern IntPtr GetStdHandle(int kind); - static string CodeRoot { get { return Path.GetDirectoryName(Path.GetDirectoryName(OwnExe)); } } - static void OwnerAdmission(string home) { - string script=Path.Combine(CodeRoot,"bin","native-owner","admit.sh"); - var start=new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script.Replace('\\','/'))) {UseShellExecute=false,RedirectStandardError=true,CreateNoWindow=true}; - start.EnvironmentVariables["FM_HOME"]=home; - using(var process=Process.Start(start)) { - var error=process.StandardError.ReadToEndAsync(); - if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("The empty-fleet owner preflight exceeded its bound");} - if(process.ExitCode!=0)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); - } - } - static void EmptyFleet(string home) { - string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); - if((Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || File.Exists(Path.Combine(state,"x-watch.check.sh")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); - OwnerAdmission(home); - } static int Launch(string selectedHome) { string home=Path.GetFullPath(selectedHome), node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); string host=Path.Combine(CodeRoot,"bin","native-owner","codex-host.mjs"); @@ -142,7 +126,7 @@ static int FixedOperation(string purpose) { string home=Environment.GetEnvironmentVariable("FM_HOME");EmptyFleet(home); if(OwnerClient(purpose=="startup" ? "identity" : "owns",Path.Combine(home,"state"),"")!=0)throw new InvalidOperationException("Operation is not registered"); string script=Path.Combine(CodeRoot,"bin","native-owner",purpose+".sh"); - using(var process=Process.Start(new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script.Replace('\\','/'))) {UseShellExecute=false})) { + using(var process=Process.Start(BashHelper(script,"",home,true))) { if(!process.WaitForExit(purpose=="startup" ? 240000 : 60000))throw new TimeoutException("Fixed operation exceeded its bound");return process.ExitCode; } } diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index 4bae7518189..08638177901 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -2,12 +2,44 @@ using System; using System.Collections; using System.Collections.Generic; +using System.Diagnostics; using System.IO; using System.Runtime.InteropServices; using System.Text; public static partial class NativeOwner { static NativeReceiptJournal operationJournal; static bool shutdownRequested; + internal static string CodeRoot { get { return Path.GetDirectoryName(Path.GetDirectoryName(OwnExe)); } } + internal static ProcessStartInfo BashHelper(string script,string arguments,string home,bool includeProbe=false) { + var start=new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script.Replace('\\','/'))+(string.IsNullOrEmpty(arguments) ? "" : " "+arguments)) {UseShellExecute=false,CreateNoWindow=true}; + start.EnvironmentVariables.Clear(); + foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { + string value=Environment.GetEnvironmentVariable(key);if(value!=null)start.EnvironmentVariables[key]=value; + } + start.EnvironmentVariables["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); + if(includeProbe) { + foreach(string key in new [] {"FM_PROBE_PIPE","FM_PROBE_SESSION","FM_PROBE_HOME","FM_PROBE_NONCE","FM_PROBE_EXE","FM_PROBE_JQ_IMAGE","FM_PROBE_VERIFY_ONLY"}) { + string value=Environment.GetEnvironmentVariable(key);if(value!=null)start.EnvironmentVariables[key]=value; + } + } + start.EnvironmentVariables["FM_HOME"]=home; + start.EnvironmentVariables["MSYS"]="winsymlinks:nativestrict"; + return start; + } + static void OwnerAdmission(string home) { + string script=Path.Combine(CodeRoot,"bin","native-owner","admit.sh"); + var start=BashHelper(script,"",home);start.RedirectStandardError=true; + using(var process=Process.Start(start)) { + var error=process.StandardError.ReadToEndAsync(); + if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("The empty-fleet owner preflight exceeded its bound");} + if(process.ExitCode!=0)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); + } + } + internal static void EmptyFleet(string home) { + string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); + if((Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); + OwnerAdmission(home); + } static IntPtr FileHandle(string path, uint access, uint creation) { SA sa = new SA { length=Marshal.SizeOf(typeof(SA)), inherit=1 }; IntPtr h = CreateFile(path, access, 3, ref sa, creation, 0x80, IntPtr.Zero); diff --git a/bin/native-owner/ack-evidence.sh b/bin/native-owner/ack-evidence.sh index 6fe3a8c67a5..0be15313200 100644 --- a/bin/native-owner/ack-evidence.sh +++ b/bin/native-owner/ack-evidence.sh @@ -1,4 +1,6 @@ #!/usr/bin/env bash +# Usage: FM_HOME= ack-evidence.sh capture-json|preflight-token|verify-token|verify-legacy|acknowledge-token +# Required environment: FM_HOME; token modes read their evidence from standard input. set -euo pipefail ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) export FM_HOME @@ -62,40 +64,6 @@ case "${1:-}" in printf '],"ownerEvidence":"%s"}\n' "$FM_WAKE_ACK_EVIDENCE_TOKEN" fm_wake_ack_evidence_clear ;; - capture-token) - fm_wake_ack_evidence_capture - printf '%s\n' "$FM_WAKE_ACK_EVIDENCE_TOKEN" - fm_wake_ack_evidence_clear - ;; - capture-payload) - mapfile -t expected < <(node -e ' - let input=""; - process.stdin.setEncoding("utf8"); - process.stdin.on("data",chunk=>input+=chunk); - process.stdin.on("end",()=>{ - const value=JSON.parse(input); - if(!value||typeof value!=="object"||Array.isArray(value))throw Error("invalid payload"); - const seq=String(value.seq),generation=value.generation; - const notes=value.notes===undefined?[value.note]:value.notes; - if(!/^\d+$/.test(seq)||typeof generation!=="string"||!Array.isArray(notes)||notes.some(note=>typeof note!=="string"))throw Error("invalid payload target"); - process.stdout.write([seq,generation,...new Set(notes)].join("\n")+"\n"); - }); - ') - [ "${#expected[@]}" -ge 2 ] || exit 2 - fm_wake_ack_evidence_capture - [ "${expected[0]}" = "$FM_WAKE_ACK_EVIDENCE_CUTOFF" ] \ - && [ "${expected[1]}" = "$FM_WAKE_ACK_EVIDENCE_GENERATION" ] || exit 2 - if [ "${#expected[@]}" -gt 2 ]; then - printf '%s\n' "${expected[@]:2}" | LC_ALL=C sort -u > "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" - else - : > "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" - fi - cut -f1 "$FM_WAKE_ACK_EVIDENCE_NOTES" | LC_ALL=C sort -u > "$FM_WAKE_ACK_EVIDENCE_NOTES.actual" - cmp -s "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" "$FM_WAKE_ACK_EVIDENCE_NOTES.actual" || exit 2 - rm -f -- "$FM_WAKE_ACK_EVIDENCE_NOTES.expected" "$FM_WAKE_ACK_EVIDENCE_NOTES.actual" - printf '%s\n' "$FM_WAKE_ACK_EVIDENCE_TOKEN" - fm_wake_ack_evidence_clear - ;; preflight-token) token=$(read_token) fm_wake_ack_evidence_precondition "$token" @@ -114,7 +82,7 @@ case "${1:-}" in fm_wake_ack_evidence_acknowledge "$token" ;; *) - printf 'usage: ack-evidence.sh capture-json|capture-token|capture-payload|preflight-token|verify-token|verify-legacy|acknowledge-token\n' >&2 + printf 'usage: ack-evidence.sh capture-json|preflight-token|verify-token|verify-legacy|acknowledge-token\n' >&2 exit 2 ;; esac diff --git a/bin/native-owner/ack.sh b/bin/native-owner/ack.sh index 275ed40ed94..ebcdd928269 100755 --- a/bin/native-owner/ack.sh +++ b/bin/native-owner/ack.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Mutate only the controller's captured receipt targets, using existing owners. +# Usage: FM_HOME= FM_PROBE_HOME= ack.sh +# Required environment: FM_HOME and FM_PROBE_HOME. set -euo pipefail ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) LOG=$(cygpath -u "${FM_PROBE_HOME:?}") diff --git a/bin/native-owner/admit.sh b/bin/native-owner/admit.sh index 7d005325af7..35614aa954e 100644 --- a/bin/native-owner/admit.sh +++ b/bin/native-owner/admit.sh @@ -1,4 +1,6 @@ #!/usr/bin/env bash +# Usage: FM_HOME= admit.sh +# Required environment: FM_HOME. set -euo pipefail ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) export FM_HOME @@ -6,7 +8,13 @@ FM_HOME=$(cygpath -u "${FM_HOME:?}") cd "$ROOT" . bin/fm-tasks-axi-lib.sh . bin/fm-backlog-transition-lib.sh +. bin/fm-supervision-lib.sh if ! fm_backlog_empty_fleet_preflight "$FM_HOME/state" "$FM_HOME/data"; then printf '%s\n' "${FM_BACKLOG_EMPTY_ERROR:-the home contains work-bearing records}" >&2 exit 2 fi +fm_supervision_status "$FM_HOME/state" +if [ "$FM_SUP_NEEDED" = true ]; then + printf 'the home contains registered work requiring supervision\n' >&2 + exit 2 +fi diff --git a/bin/native-owner/check.sh b/bin/native-owner/check.sh index 9501be63841..8d34edb9ebc 100755 --- a/bin/native-owner/check.sh +++ b/bin/native-owner/check.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Read actual work only. No synthetic note or scripted model prompt is generated. +# Usage: FM_HOME= FM_PROBE_HOME= FM_PROBE_JQ_IMAGE= check.sh +# Required environment: FM_HOME, FM_PROBE_HOME, and FM_PROBE_JQ_IMAGE. set -euo pipefail ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) LOG=$(cygpath -u "${FM_PROBE_HOME:?}") diff --git a/bin/native-owner/codex-host-runtime.mjs b/bin/native-owner/codex-host-runtime.mjs index 858dde291e0..db936c52789 100644 --- a/bin/native-owner/codex-host-runtime.mjs +++ b/bin/native-owner/codex-host-runtime.mjs @@ -11,7 +11,7 @@ export async function runCodexHost(options={}) { const env=options.env??process.env; const runtime=env.FM_PROBE_HOME,root=env.FM_PROBE_CODE_ROOT; const inputStream=options.input??process.stdin,outputStream=options.output??process.stdout,errorStream=options.error??process.stderr; - const pause=options.pause??(ms=>new Promise(resolve=>setTimeout(resolve,ms))); + const pause=ms=>new Promise(resolve=>setTimeout(resolve,ms)); let closing=false,gate=null,server=null,alive=false,thread=null,activeTurn=null,ended=false; let lifecycle=null,terminal=null,announced=null,next=0,nativeWaiter=null,nativeTail=Promise.resolve(); let socket=null,channel=null; diff --git a/bin/native-owner/startup.sh b/bin/native-owner/startup.sh index 43f5e934fd3..7b72d33d6fe 100755 --- a/bin/native-owner/startup.sh +++ b/bin/native-owner/startup.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Run startup once, publish its digest promptly, retain only bounded deferred authority. +# Usage: FM_HOME= FM_PROBE_HOME= FM_PROBE_JQ_IMAGE= startup.sh +# Required environment: FM_HOME, FM_PROBE_HOME, and FM_PROBE_JQ_IMAGE. set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) LOG=$(cygpath -u "${FM_PROBE_HOME:?}") diff --git a/bin/native-owner/tools/jq b/bin/native-owner/tools/jq index 23d962ff8b5..4bd9e900d38 100755 --- a/bin/native-owner/tools/jq +++ b/bin/native-owner/tools/jq @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Container-only jq; mount only the code root and operational home, read-only. +# Usage: FM_HOME= FM_PROBE_JQ_IMAGE= jq [arguments...] +# Required environment: FM_HOME and FM_PROBE_JQ_IMAGE. set -euo pipefail ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) HOME_PATH=$(cygpath -u "${FM_HOME:?}") diff --git a/docs/documentation-audiences.json b/docs/documentation-audiences.json index b4927ddc11d..d333e30cadb 100644 --- a/docs/documentation-audiences.json +++ b/docs/documentation-audiences.json @@ -102,10 +102,6 @@ "source": "docs/codex-app-backend.md", "target": "docs/verification/runtime-backends.md" }, - { - "source": "docs/native-windows-codex.md", - "target": "docs/verification/runtime-backends.md" - }, { "source": "docs/trace-context.md", "target": "docs/verification/trace-context.md" diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index a5a4554f5d3..bc99b15acf5 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -60,6 +60,8 @@ The acknowledgement retires the marker only when no rows remain after sequence-b A concurrently appended wake has a higher sequence, remains queued, and keeps the episode pending for presentation. Consequently, an empty-queue downtime publication during handling can be retired by the outstanding acknowledgement without a dedicated recovery turn. An acknowledged episode does not freeze the generation, because the next downtime after it opens an episode of its own. +Before that later episode replaces an acknowledged marker, the wake owner retains the completed generation in the versioned newline-delimited `state/.watcher-down.ack-completions` owner record so restart reconciliation can still prove the earlier effect. +The record retains at most 1,024 unique generations without automatic pruning; an unreadable, malformed, or full record prevents marker replacement and preserves the current proof instead of permitting new work to erase it. ## Per-actor acknowledgement diff --git a/tests/fixtures/native-owner/Build.ps1 b/tests/fixtures/native-owner/Build.ps1 index 8a1f7dcbd21..266b8323153 100644 --- a/tests/fixtures/native-owner/Build.ps1 +++ b/tests/fixtures/native-owner/Build.ps1 @@ -6,14 +6,17 @@ $env:MSYS = 'winsymlinks:nativestrict' $repo = [IO.Path]::GetFullPath((Join-Path $PSScriptRoot '../../..')) $root = Join-Path ([IO.Path]::GetTempPath()) ('fm-native-candidate-' + [guid]::NewGuid().ToString('N')) New-Item -ItemType Directory $root | Out-Null -$binary = Join-Path $root 'SessionProbe.exe' +$copy = Join-Path $root 'firstmate' +& git -c core.symlinks=true clone --quiet --no-local --single-branch $repo $copy +if ($LASTEXITCODE -ne 0) { throw 'Disposable clone failed' } +Copy-Item -Path (Join-Path $repo 'bin/*') -Destination (Join-Path $copy 'bin') -Recurse -Force +Copy-Item -Path (Join-Path $repo 'tests/fixtures/native-owner/*') -Destination (Join-Path $copy 'tests/fixtures/native-owner') -Recurse -Force +$binary = Join-Path $copy 'bin/SessionProbe.exe' $sources = @((Join-Path $repo 'bin/native-owner/NativeOwner.cs'), (Join-Path $repo 'bin/native-owner/NativeHomeLease.cs'), (Join-Path $PSScriptRoot 'NativeDriver.cs'), (Join-Path $repo 'bin/native-owner/NativeReceiptJournal.cs'), (Join-Path $PSScriptRoot 'ReceiptTests.cs')) $sources += @((Join-Path $repo 'bin/native-owner/NativeOperations.cs'), (Join-Path $repo 'bin/native-owner/NativeAcknowledgementEvidence.cs'), (Join-Path $repo 'bin/native-owner/NativeOperationLifetime.cs'), (Join-Path $PSScriptRoot 'OperationLifetimeTests.cs')) Add-Type -Path $sources -OutputAssembly $binary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll -$env:FM_PROBE_CODE_ROOT = $repo & $binary receipt-tests $receiptExit = $LASTEXITCODE -Remove-Item Env:FM_PROBE_CODE_ROOT if ($receiptExit -ne 0) { throw 'Durable receipt lifecycle tests failed' } & $binary environment-tests if ($LASTEXITCODE -ne 0) { throw 'Native environment filtering tests failed' } @@ -28,18 +31,14 @@ try { } finally { $ErrorActionPreference = $savedErrorPreference } -$copy = Join-Path $root 'firstmate' -& git -c core.symlinks=true clone --quiet --no-local --single-branch $repo $copy -if ($LASTEXITCODE -ne 0) { throw 'Disposable clone failed' } # Exercise the real opt-in consumers, including local changes before commit. -foreach ($name in @('fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh')) { Copy-Item (Join-Path $repo ('bin/' + $name)) (Join-Path $copy ('bin/' + $name)) } foreach ($name in @('exercise.sh','notification-check.sh','notification-ack.sh')) { Copy-Item (Join-Path $PSScriptRoot $name) (Join-Path $copy $name) } -Copy-Item (Join-Path $PSScriptRoot 'AppHost.mjs') (Join-Path $root 'AppHost.mjs') -foreach ($module in @('codex-tool-gate.mjs','host-lifecycle.mjs','app-server-policy.mjs')) { Copy-Item (Join-Path $repo ('bin/native-owner/' + $module)) (Join-Path $root $module) } +Copy-Item (Join-Path $PSScriptRoot 'AppHost.mjs') (Join-Path $copy 'AppHost.mjs') +foreach ($module in @('codex-tool-gate.mjs','host-lifecycle.mjs','app-server-policy.mjs')) { Copy-Item (Join-Path $repo ('bin/native-owner/' + $module)) (Join-Path $copy $module) } Copy-Item $binary (Join-Path $copy 'bin/fm-native-owner.exe') New-Item -ItemType Directory (Join-Path $root 'tools') | Out-Null Copy-Item (Join-Path $PSScriptRoot 'jq') (Join-Path $root 'tools/jq') $state = Join-Path $repo 'data/native-candidate-validation' New-Item -ItemType Directory -Force $state | Out-Null -@{ root=$root; binary=$binary; repo=$repo; hashes=@($sources | ForEach-Object { @{ file=$_; hash=(Get-FileHash $_).Hash } }) } | ConvertTo-Json -Depth 5 | Set-Content -Encoding UTF8 (Join-Path $state 'build.json') +@{ root=$root; code=$copy; binary=$binary; repo=$repo; hashes=@($sources | ForEach-Object { @{ file=$_; hash=(Get-FileHash $_).Hash } }) } | ConvertTo-Json -Depth 5 | Set-Content -Encoding UTF8 (Join-Path $state 'build.json') Write-Output $binary diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 2c253086de0..5dcebd5a2be 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -19,9 +19,9 @@ const launcher=path.join(code,'bin/fm-native-codex.ps1'); command('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly']); const removedAlias=spawnSync('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly','-Home',path.join(area,'alias')],{encoding:'utf8',timeout:120000}); assert.notEqual(removedAlias.status,0,'The removed -Home alias was still accepted'); -function start(home,verify=true){ +function start(home,verify=true,env=process.env){ const args=['-NoProfile','-File',launcher,'-Experimental','-OperationalHome',home,'-JqImage',image];if(verify)args.push('-VerifyOnly'); - const child=spawn('powershell.exe',args,{stdio:['pipe','pipe','pipe']});let stdout='',stderr=''; + const child=spawn('powershell.exe',args,{stdio:['pipe','pipe','pipe'],env});let stdout='',stderr=''; child.stdout.on('data',data=>stdout+=data);child.stderr.on('data',data=>stderr+=data);child.stdin.on('error',()=>{}); const done=new Promise((resolve,reject)=>{child.on('error',reject);child.on('exit',exit=>resolve({exit,stdout,stderr}));}); return {child,done,home}; @@ -79,6 +79,19 @@ for(const [name,relative] of [['orphan-status','state/orphan.status'],['interrup assert.equal(fs.readFileSync(record,'utf8'),contents);assert.equal(fs.existsSync(path.join(residualHome,'owner-probe.json')),false); } records.push('orphan status and interrupted-close records refused before lease acquisition and preserved'); +const maskedHome=path.join(area,'bash-env-mask'),maskedStatus=path.join(maskedHome,'state/orphan.status'),mask=path.join(area,'bash-env-exit.sh'); +fs.mkdirSync(path.dirname(maskedStatus),{recursive:true});fs.writeFileSync(maskedStatus,'preserve masked residual state');fs.writeFileSync(mask,'exit 0\n'); +const masked=start(maskedHome,true,{...process.env,BASH_ENV:mask});masked.child.stdin.end();assert.notEqual((await bound(masked.done,masked,20000)).exit,0); +assert.equal(fs.readFileSync(maskedStatus,'utf8'),'preserve masked residual state');assert.equal(fs.existsSync(path.join(maskedHome,'owner-probe.json')),false); +records.push('ambient Bash startup hooks cannot bypass admission before lease acquisition'); +const customHome=path.join(area,'registered-custom'),customState=path.join(customHome,'state'),canary=path.join(customHome,'executed'); +fs.mkdirSync(customState,{recursive:true});fs.writeFileSync(path.join(customState,'custom.check.sh'),`#!/usr/bin/env bash\nprintf executed > "${posix(canary)}"\n`); +fs.chmodSync(path.join(customState,'custom.check.sh'),0o700); +const register=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',posix(path.join(code,'bin/fm-check-register.sh')),'custom'],{env:{...process.env,FM_HOME:posix(customHome),MSYS:'winsymlinks:nativestrict'},encoding:'utf8',timeout:30000}); +assert.equal(register.status,0,register.stderr); +const custom=start(customHome);custom.child.stdin.end();assert.notEqual((await bound(custom.done,custom,20000)).exit,0); +assert.equal(fs.existsSync(canary),false);assert.equal(fs.existsSync(path.join(customHome,'owner-probe.json')),false); +records.push('registered custom work is refused before lease acquisition without execution'); for(const [name,contents] of [ ['queued-backlog','## In flight\n\n## Queued\n- [ ] queued-work - preserved project work (repo: firstmate) (kind: ship)\n\n## Done\n'], ['unrecognized-backlog','# Backlog\n\nproject work in an unrecognized form\n'], diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 0817f0cddb7..08051235a08 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -56,7 +56,7 @@ static int Client(string which) { return 0; } static void RunFirstmate(string role,bool shouldSucceed) { - string root=Path.Combine(Path.GetDirectoryName(OwnExe),"firstmate"); + string root=CodeRoot; string script=Path.Combine(root,shouldSucceed ? "exercise.sh" : "bin/fm-lock.sh").Replace('\\','/'); using(var p=Process.Start(new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script)) { UseShellExecute=false,RedirectStandardOutput=!shouldSucceed,RedirectStandardError=!shouldSucceed })) { var stdout=shouldSucceed ? System.Threading.Tasks.Task.FromResult("") : p.StandardOutput.ReadToEndAsync(); var stderr=shouldSucceed ? System.Threading.Tasks.Task.FromResult("") : p.StandardError.ReadToEndAsync(); @@ -84,10 +84,11 @@ static int Fixture() { static int EnvironmentTests() { string directory=Path.Combine(Path.GetTempPath(),"fm-native-environment-"+Guid.NewGuid().ToString("N")); Directory.CreateDirectory(directory); - string payload=Path.Combine(directory,"payload.js"),main=Path.Combine(directory,"main.js"),injected=Path.Combine(directory,"injected"),result=Path.Combine(directory,"result.json"); + string payload=Path.Combine(directory,"payload.js"),bashMask=Path.Combine(directory,"bash-mask.sh"),main=Path.Combine(directory,"main.js"),injected=Path.Combine(directory,"injected"),result=Path.Combine(directory,"result.json"); File.WriteAllText(payload,"require('fs').writeFileSync("+Json.Serialize(injected.Replace('\\','/'))+",'injected')"); + File.WriteAllText(bashMask,"exit 0\n"); File.WriteAllText(main,"const fs=require('fs');const denied=['NODE_OPTIONS','NODE_PATH','BASH_ENV','ENV','CLAUDE_PID','CLAUDECODE'];const leaked=Object.keys(process.env).filter(k=>denied.includes(k.toUpperCase())||k.toUpperCase().startsWith('FM_')||k.toUpperCase().startsWith('PI_')||k.toUpperCase().startsWith('NO_MISTAKES'));fs.writeFileSync("+Json.Serialize(result.Replace('\\','/'))+",JSON.stringify(leaked));"); - var poison=new Dictionary(StringComparer.OrdinalIgnoreCase){{"node_options","--require=\""+payload.Replace('\\','/')+"\""},{"node_path",directory},{"bash_env",payload},{"env",payload},{"claude_pid","123"},{"claudecode","1"},{"fm_poison","1"},{"pi_poison","1"},{"no_mistakes_poison","1"}}; + var poison=new Dictionary(StringComparer.OrdinalIgnoreCase){{"node_options","--require=\""+payload.Replace('\\','/')+"\""},{"node_path",directory},{"bAsH_eNv",bashMask},{"env",payload},{"claude_pid","123"},{"claudecode","1"},{"fm_poison","1"},{"pi_poison","1"},{"no_mistakes_poison","1"}}; var original=new Dictionary(StringComparer.OrdinalIgnoreCase); try { foreach(var entry in poison){original[entry.Key]=Environment.GetEnvironmentVariable(entry.Key);Environment.SetEnvironmentVariable(entry.Key,entry.Value);} @@ -102,6 +103,18 @@ static int EnvironmentTests() { if(File.Exists(injected)||leaked.Length!=5)throw new InvalidOperationException("Denied inherited environment reached the native host"); foreach(string key in leaked)if(!key.StartsWith("FM_PROBE_",StringComparison.Ordinal))throw new InvalidOperationException("Unexpected inherited environment reached the native host"); Console.WriteLine("PASS: inherited Windows environment denylist is case-insensitive"); + string residual=Path.Combine(directory,"residual"),residualState=Path.Combine(residual,"state"),status=Path.Combine(residualState,"orphan.status"); + Directory.CreateDirectory(residualState);File.WriteAllText(status,"preserve\n"); + bool refused=false;try{EmptyFleet(residual);}catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(status)!="preserve\n")throw new InvalidOperationException("Mixed-case BASH_ENV bypassed empty-home admission"); + Console.WriteLine("PASS: fixed Bash admission ignores mixed-case ambient authority"); + string registered=Path.Combine(directory,"registered"),registeredState=Path.Combine(registered,"state"),canary=Path.Combine(registered,"executed"); + Directory.CreateDirectory(registeredState); + File.WriteAllText(Path.Combine(registeredState,"custom.check.sh"),"#!/usr/bin/env bash\nprintf executed > \""+canary.Replace('\\','/')+"\"\n"); + File.WriteAllText(Path.Combine(registeredState,"custom.check-trust"),"fm-custom-check-v1\n"+new string('0',64)+"\n"); + refused=false;try{EmptyFleet(registered);}catch(InvalidOperationException){refused=true;} + if(!refused||File.Exists(canary))throw new InvalidOperationException("Registered custom work passed admission or executed during inspection"); + Console.WriteLine("PASS: registered custom checks are refused without execution"); return 0; } finally { foreach(var entry in original)Environment.SetEnvironmentVariable(entry.Key,entry.Value); @@ -138,7 +151,7 @@ static int Run(string configPath) { int seconds = Convert.ToInt32(config["timeoutSeconds"]); if(config.ContainsKey("registeredHarness")) { string expected=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),@"npm\node_modules\@openai\codex\node_modules\@openai\codex-win32-x64\vendor\x86_64-pc-windows-msvc\bin\codex.exe"); - string host=Path.Combine(Path.GetDirectoryName(OwnExe),"AppHost.mjs"); + string host=Path.Combine(CodeRoot,"AppHost.mjs"); string node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); bool direct=(string)config["registeredHarness"]=="codex" && string.Equals(Path.GetFullPath(executable),Path.GetFullPath(expected),StringComparison.OrdinalIgnoreCase); bool adapter=(string)config["registeredHarness"]=="codex-app-server" && string.Equals(Path.GetFullPath(executable),Path.GetFullPath(node),StringComparison.OrdinalIgnoreCase) && arguments==Quote(host); @@ -305,7 +318,7 @@ public static int Main(string[] args) { if(args.Length==3 && args[0]=="lease-check") return LeaseCheck(args[1],args[2]); if(args.Length==2 && args[0]=="notification-operation") { if(args[1]!="check" && args[1]!="ack") throw new ArgumentException("Unsupported notification operation"); - string script=Path.Combine(Path.GetDirectoryName(OwnExe),"firstmate","notification-"+args[1]+".sh"); + string script=Path.Combine(CodeRoot,"notification-"+args[1]+".sh"); using(var operation=Process.Start(new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script)) {UseShellExecute=false})) { if(!operation.WaitForExit(60000)) throw new IOException("Notification operation exceeded its bound"); return operation.ExitCode; diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index e02d2e8a113..ccba3b95b21 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -4,6 +4,8 @@ using System.IO; using System.Runtime.InteropServices; using System.Security.AccessControl; +using System.Text; +using System.Web.Script.Serialization; public static class ReceiptTests { [DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] static extern bool CreateHardLink(string link,string target,IntPtr security); [DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] [return: MarshalAs(UnmanagedType.I1)] static extern bool CreateSymbolicLink(string link,string target,uint flags); @@ -18,6 +20,7 @@ static void Case(string name,Action test) { passed++;Console.WriteLine("PASS: "+name); } static string Queue(NativeHomeLease lease) { return Path.Combine(lease.Home,"state",".wake-queue"); } + static string CompletionHistory(NativeHomeLease lease) { return Path.Combine(lease.Home,"state",".watcher-down.ack-completions"); } static string Pending(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","note-id.note"); } static string Handled(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","handled","note-id.note"); } static void Targets(NativeHomeLease lease) { @@ -27,13 +30,10 @@ static void Targets(NativeHomeLease lease) { File.WriteAllText(Path.Combine(lease.Home,"state",".main-eligible-rows"),"1\n"); File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"pending:handling:recovery\n"); } - static string Quote(string value) { return "\""+value.Replace("\"","\\\"")+"\""; } static string ZeroRecovery(NativeHomeLease lease,string action,string generation=null) { - string root=Environment.GetEnvironmentVariable("FM_PROBE_CODE_ROOT"); - if(string.IsNullOrEmpty(root))throw new InvalidOperationException("Zero-recovery fixture root is required"); - string script=Path.Combine(root,"tests","fixtures","native-owner","zero-recovery.sh"); - var start=new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script)+" "+action+" "+Quote(lease.Home)+(generation==null ? "" : " "+Quote(generation))) {UseShellExecute=false,RedirectStandardOutput=true,RedirectStandardError=true}; - start.EnvironmentVariables["MSYS"]="winsymlinks:nativestrict"; + string script=Path.Combine(NativeOwner.CodeRoot,"tests","fixtures","native-owner","zero-recovery.sh"); + string arguments=action+" \""+lease.Home.Replace("\"","\\\"")+"\""+(generation==null ? "" : " \""+generation.Replace("\"","\\\"")+"\""); + var start=NativeOwner.BashHelper(script,arguments,lease.Home);start.RedirectStandardOutput=true;start.RedirectStandardError=true; using(var process=Process.Start(start)) { var output=process.StandardOutput.ReadToEndAsync();var error=process.StandardError.ReadToEndAsync(); if(!process.WaitForExit(30000)){process.Kill();throw new IOException("Zero-recovery fixture exceeded its bound");} @@ -41,10 +41,24 @@ static string ZeroRecovery(NativeHomeLease lease,string action,string generation return output.Result.Trim(); } } + static Dictionary OwnerPayload(NativeHomeLease lease,string challenge) { + string script=Path.Combine(NativeOwner.CodeRoot,"bin","native-owner","ack-evidence.sh"); + var start=NativeOwner.BashHelper(script,"capture-json",lease.Home);start.RedirectStandardOutput=true;start.RedirectStandardError=true; + using(var process=Process.Start(start)) { + var output=process.StandardOutput.ReadToEndAsync();var error=process.StandardError.ReadToEndAsync(); + if(!process.WaitForExit(30000)){process.Kill();throw new IOException("Acknowledgement target fixture exceeded its bound");} + if(process.ExitCode!=0)throw new IOException("Acknowledgement target fixture failed: "+error.Result); + var payload=new JavaScriptSerializer().Deserialize>(output.Result); + payload["challenge"]=challenge;payload["message"]="pending notification"; + return payload; + } + } static Dictionary ZeroPayload(NativeHomeLease lease) { string[] target=ZeroRecovery(lease,"present").Split('\t'); Expect(target.Length==2 && target[0]=="0","Real recovery owner did not produce a zero-row target"); - return new Dictionary{{"challenge","zero"},{"message","recovery"},{"seq",target[0]},{"generation",target[1]},{"notes",new string[0]}}; + var payload=OwnerPayload(lease,"zero"); + Expect((string)payload["seq"]==target[0] && (string)payload["generation"]==target[1],"Owner evidence did not bind the recovery target"); + payload["message"]="recovery";return payload; } public static int Run() { Case("one writer and unobserved acknowledgement refusal",lease=>{ @@ -136,9 +150,9 @@ public static int Run() { Case("interrupted recovery: "+scenario,lease=>{ Targets(lease);string receipt; using(var journal=new NativeReceiptJournal(lease,A)) { - var delivery=journal.Present(Payload("first"));receipt=(string)delivery["receipt"]; - var evidence=NativeAcknowledgementEvidence.Capture(lease,delivery); - journal.BeginAcknowledgement(receipt,"first",scenario=="no-evidence" ? null : evidence); + var delivery=journal.Present(OwnerPayload(lease,"first"));receipt=(string)delivery["receipt"]; + var evidence=scenario=="no-evidence" ? null : NativeAcknowledgementEvidence.Capture(lease,delivery); + journal.BeginAcknowledgement(receipt,"first",evidence); } if(scenario!="pending" && scenario!="wake-only") File.Move(Pending(lease),Handled(lease)); if(scenario!="pending" && scenario!="note-only") File.WriteAllText(Queue(lease),""); @@ -168,7 +182,7 @@ public static int Run() { Targets(lease); string otherHome=Path.Combine(Path.GetTempPath(),"fm-receipts-other-"+Guid.NewGuid().ToString("N")); using(var other=new NativeHomeLease(otherHome)) using(var journal=new NativeReceiptJournal(lease,A)) { - Targets(other);var delivery=journal.Present(Payload("first")); + Targets(other);var delivery=journal.Present(OwnerPayload(other,"first")); var evidence=NativeAcknowledgementEvidence.Capture(other,delivery); Refuses(()=>journal.BeginAcknowledgement((string)delivery["receipt"],"first",evidence),"Foreign home evidence accepted"); Expect(!journal.NeedsReconciliation,"Rejected evidence created an attempt"); @@ -176,17 +190,16 @@ public static int Run() { }); foreach(string scenario in new [] {"no-inbox-targets","multiple-complete","multiple-partial"}) { Case("general wake recovery: "+scenario,lease=>{ - Targets(lease);var payload=Payload("general");payload.Remove("note"); + Targets(lease); if(scenario=="no-inbox-targets") { - payload["notes"]=new string[0];File.WriteAllText(Queue(lease),"1\t1\tcheck\tdiagnostic\treport\n"); + File.WriteAllText(Queue(lease),"1\t1\tcheck\tdiagnostic\treport\n"); } else { - payload["notes"]=new [] {"note-id","second"};payload["seq"]="2"; File.WriteAllText(Path.Combine(lease.Home,"state","inbox","second.note"),"second notification"); File.AppendAllText(Queue(lease),"2\t2\tcheck\tinbox:second\tsecond\n"); File.WriteAllText(Path.Combine(lease.Home,"state",".main-eligible-rows"),"1\n2\n"); } using(var journal=new NativeReceiptJournal(lease,A)) { - var delivery=journal.Present(payload); + var delivery=journal.Present(OwnerPayload(lease,"general")); journal.BeginAcknowledgement((string)delivery["receipt"],"general",NativeAcknowledgementEvidence.Capture(lease,delivery)); } if(scenario!="no-inbox-targets") { @@ -208,10 +221,12 @@ public static int Run() { } }); Case("zero-row recovery rejects a mismatched generation",lease=>{ - var payload=ZeroPayload(lease);payload["generation"]="different"; + var payload=ZeroPayload(lease);string generation=(string)payload["generation"]; + ZeroRecovery(lease,"acknowledge",generation); + ZeroRecovery(lease,"append"); using(var journal=new NativeReceiptJournal(lease,A)) { var delivery=journal.Present(payload); - Refuses(()=>NativeAcknowledgementEvidence.Capture(lease,delivery),"Mismatched recovery generation accepted"); + Refuses(()=>NativeAcknowledgementEvidence.Capture(lease,delivery),"Foreign recovery generation accepted"); } }); Case("zero-row recovery interruption preserves the obligation",lease=>{ @@ -235,6 +250,7 @@ public static int Run() { journal.BeginAcknowledgement(receipt,"zero",NativeAcknowledgementEvidence.Capture(lease,delivery)); } ZeroRecovery(lease,"acknowledge",generation); + ZeroRecovery(lease,"append"); string queue=File.ReadAllText(Queue(lease)),marker=File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")); using(var journal=new NativeReceiptJournal(lease,B)) { Expect(journal.ReconcileCompletedAcknowledgements()==1,"Completed zero-row target was not reconciled"); @@ -242,6 +258,25 @@ public static int Run() { Expect(journal.ReconcileCompletedAcknowledgements()==0,"Completed zero-row target replayed"); } Expect(queue==File.ReadAllText(Queue(lease)) && marker==File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")),"Reconciliation changed completed effects"); + Expect(queue.Contains("later-notification"),"A later notification was not kept pending"); + }); + Case("zero-row completion capacity preserves current proof",lease=>{ + Directory.CreateDirectory(Path.GetDirectoryName(Queue(lease)));File.WriteAllText(Queue(lease),""); + File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"acked:handling:capacity-current\n"); + var history=new StringBuilder("fm-wake-ack-completions-v1\n"); + for(int i=0;i<1024;i++)history.Append("stored-").Append(i).Append('\n'); + File.WriteAllText(CompletionHistory(lease),history.ToString()); + string marker=File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")); + Refuses(()=>ZeroRecovery(lease,"append"),"A full completion history allowed proof replacement"); + Expect(marker==File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")) && File.ReadAllText(Queue(lease))=="","Capacity refusal changed current proof or queued new work"); + }); + Case("malformed zero-row completion history preserves current proof",lease=>{ + Directory.CreateDirectory(Path.GetDirectoryName(Queue(lease)));File.WriteAllText(Queue(lease),""); + File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"acked:handling:malformed-current\n"); + File.WriteAllText(CompletionHistory(lease),"unrecognized\n"); + string marker=File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")); + Refuses(()=>ZeroRecovery(lease,"append"),"Malformed completion history allowed proof replacement"); + Expect(marker==File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")) && File.ReadAllText(Queue(lease))=="","Malformed-history refusal changed current proof or queued new work"); }); Console.WriteLine("RECEIPT_TESTS_PASS "+passed);return 0; } diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs index c29769c5ddd..03bf8ae4101 100644 --- a/tests/fixtures/native-owner/Run-Cycle.mjs +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -17,7 +17,7 @@ if(!dry&&process.env.FM_LIVE_NATIVE_CODEX!=='1')throw Error('Live model test req if(!dry){const preflight=read(path.join(dir,'bridge-preflight.json'));if(!preflight.passed||preflight.binaryHash!==createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex'))throw Error('Run model-free bridge preflight for this binary first');} const home=path.join(build.root,'appserver-'+randomUUID());fs.mkdirSync(home); fs.writeFileSync(path.join(home,'build.json'),JSON.stringify(build,null,2)); -const script=path.join(build.root,'AppHost.mjs'); +const script=path.join(build.code,'AppHost.mjs'); const leaseHome=path.join(home,'home'); let startupNote=null; if(startupQueued){ @@ -32,8 +32,7 @@ const spec={home,leaseHome,executable:process.execPath,arguments:'"'+script+'"', if(startupQueued){spec.startupQueued=true;spec.startupNote=startupNote;} if(fault)spec.ackFault=fault; const file=path.join(home,'spec.json');fs.writeFileSync(file,JSON.stringify(spec,null,2)); -const controllerEnv={...process.env,FM_PROBE_CODE_ROOT:repo}; -const run=spawnSync(build.binary,['run',file],{env:controllerEnv,encoding:'utf8',timeout:310000}); +const run=spawnSync(build.binary,['run',file],{env:process.env,encoding:'utf8',timeout:310000}); fs.writeFileSync(path.join(home,'controller.stdout'),run.stdout||'');fs.writeFileSync(path.join(home,'controller.stderr'),run.stderr||''); fs.writeFileSync(path.join(dir,fault?`fault-${fault}-latest.json`:dry?'bridge-latest.json':'cycle-latest.json'),JSON.stringify({home,exit:run.status},null,2)); console.log(JSON.stringify({home,exit:run.status,dry})); @@ -58,7 +57,7 @@ if(fault) { const recovery=path.join(home,'recovery');fs.mkdirSync(recovery); const recoverySpec={home:recovery,leaseHome:spec.leaseHome,executable:build.binary,arguments:'sleep 100',timeoutSeconds:10,pipeAcl:'UserOnly'}; const recoveryFile=path.join(recovery,'spec.json');fs.writeFileSync(recoveryFile,JSON.stringify(recoverySpec)); - const restarted=spawnSync(build.binary,['run',recoveryFile],{env:controllerEnv,encoding:'utf8',timeout:15000}); + const restarted=spawnSync(build.binary,['run',recoveryFile],{env:process.env,encoding:'utf8',timeout:15000}); fs.writeFileSync(path.join(recovery,'controller.stdout'),restarted.stdout||'');fs.writeFileSync(path.join(recovery,'controller.stderr'),restarted.stderr||''); if(restarted.status!==0)throw Error('Recovery controller failed'); const recovered=read(path.join(recovery,'result.json')); diff --git a/tests/fixtures/native-owner/Verify-Cycle.mjs b/tests/fixtures/native-owner/Verify-Cycle.mjs index fe58f20ff7c..45795586b1e 100644 --- a/tests/fixtures/native-owner/Verify-Cycle.mjs +++ b/tests/fixtures/native-owner/Verify-Cycle.mjs @@ -64,5 +64,5 @@ if(!fs.existsSync(archive)) { fs.cpSync(home,path.join(archive,'live'),{recursive:true}); } -fs.writeFileSync(path.join(state,'verification.json'),JSON.stringify({passed:true,realModelTurns:2,realToolCalls:4,threadAndReplayRejection:true,postStartup:true,durableAcknowledgement:true,buildSources:build.hashes,binaryHash:hash(build.binary),gateHash:hash(path.join(build.root,'codex-tool-gate.mjs')),lifecycleHash:hash(path.join(build.root,'host-lifecycle.mjs')),confirmedShutdown:host.shutdown},null,2)); +fs.writeFileSync(path.join(state,'verification.json'),JSON.stringify({passed:true,realModelTurns:2,realToolCalls:4,threadAndReplayRejection:true,postStartup:true,durableAcknowledgement:true,buildSources:build.hashes,binaryHash:hash(build.binary),gateHash:hash(path.join(build.code,'codex-tool-gate.mjs')),lifecycleHash:hash(path.join(build.code,'host-lifecycle.mjs')),confirmedShutdown:host.shutdown},null,2)); console.log('PASS: consolidated candidate protocol, ownership, and durable acknowledgement evidence.'); diff --git a/tests/fixtures/native-owner/zero-recovery.sh b/tests/fixtures/native-owner/zero-recovery.sh index ad73912c90e..0317fe92a72 100644 --- a/tests/fixtures/native-owner/zero-recovery.sh +++ b/tests/fixtures/native-owner/zero-recovery.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# Usage: zero-recovery.sh present|acknowledge|append [generation] set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) export FM_HOME @@ -26,5 +27,9 @@ case "${1:-}" in case "$generation" in ''|*[!A-Za-z0-9._-]*) exit 2 ;; esac bin/fm-wake-drain.sh --ack-through 0 --recovery-generation "$generation" ;; + append) + . bin/fm-wake-lib.sh + fm_wake_append check later-notification 'later notification remains pending' + ;; *) exit 2 ;; esac From 2df27f179b5b95a031611efb04221c44dadacbeb Mon Sep 17 00:00:00 2001 From: Cristian Date: Wed, 16 Sep 2026 23:45:02 +1200 Subject: [PATCH 18/61] no-mistakes(review): Reject residual work and document fixture usage --- bin/fm-backlog-transition-lib.sh | 10 +++++++++- tests/fixtures/native-owner/NativeDriver.cs | 10 ++++++++++ tests/fixtures/native-owner/exercise.sh | 2 ++ tests/fixtures/native-owner/jq | 2 ++ tests/fixtures/native-owner/notification-ack.sh | 2 ++ tests/fixtures/native-owner/notification-check.sh | 2 ++ 6 files changed, 27 insertions(+), 1 deletion(-) diff --git a/bin/fm-backlog-transition-lib.sh b/bin/fm-backlog-transition-lib.sh index cc4e3897579..5c83bd50c5c 100644 --- a/bin/fm-backlog-transition-lib.sh +++ b/bin/fm-backlog-transition-lib.sh @@ -185,7 +185,9 @@ fm_backlog_empty_fleet_preflight() { # FM_BACKLOG_EMPTY_ERROR=$FM_BACKLOG_TRANSITION_ERROR return 1 fi - for record in "$state"/*.meta "$state"/*.status "$state"/*.backlog-close; do + for record in "$state"/*.meta "$state"/*.status "$state"/*.backlog-close \ + "$state"/*.inbox "$state"/.backlog-handoff-*.wake-pending \ + "$state"/handoff/*.outbox.md; do if [ -e "$record" ] || [ -L "$record" ]; then FM_BACKLOG_EMPTY_ERROR="work-bearing task record is present at $record" return 1 @@ -205,6 +207,12 @@ fm_backlog_empty_fleet_preflight() { # FM_BACKLOG_EMPTY_ERROR=$FM_BACKLOG_TRANSITION_ERROR return 1 fi + for record in "$data"/handoff/*.outbox.md; do + if [ -e "$record" ] || [ -L "$record" ]; then + FM_BACKLOG_EMPTY_ERROR="work-bearing handoff record is present at $record" + return 1 + fi + done root=$(fm_backlog_root "$data") || { FM_BACKLOG_EMPTY_ERROR=${FM_BACKLOG_TRANSITION_ERROR:-"data directory cannot be resolved: $data"} return 1 diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 08051235a08..5c8140ecf6e 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -115,6 +115,16 @@ static int EnvironmentTests() { refused=false;try{EmptyFleet(registered);}catch(InvalidOperationException){refused=true;} if(!refused||File.Exists(canary))throw new InvalidOperationException("Registered custom work passed admission or executed during inspection"); Console.WriteLine("PASS: registered custom checks are refused without execution"); + string handoff=Path.Combine(directory,"handoff"),outbox=Path.Combine(handoff,"data","handoff","agent.outbox.md"),outboxBody="# Backlog\n\n## Queued\n\n- [ ] routed-work\n"; + Directory.CreateDirectory(Path.GetDirectoryName(outbox));File.WriteAllText(outbox,outboxBody); + refused=false;try{EmptyFleet(handoff);using(var lease=new NativeHomeLease(handoff)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(outbox)!=outboxBody||File.Exists(Path.Combine(handoff,"owner-probe.json"))||Directory.Exists(Path.Combine(handoff,"state")))throw new InvalidOperationException("Pending handoff work passed admission or caused lease or route activity"); + Console.WriteLine("PASS: pending handoff work is preserved without lease or route activity"); + string steering=Path.Combine(directory,"steering"),steeringState=Path.Combine(steering,"state"),inbox=Path.Combine(steeringState,"agent.inbox"),message=Path.Combine(inbox,"001.msg"),messageBody="schema=fm-task-inbox.v1\n--\npreserve\n"; + Directory.CreateDirectory(inbox);File.WriteAllText(message,messageBody); + refused=false;try{EmptyFleet(steering);using(var lease=new NativeHomeLease(steering)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(message)!=messageBody||File.Exists(Path.Combine(steering,"owner-probe.json"))||File.Exists(Path.Combine(steeringState,".lock")))throw new InvalidOperationException("Orphan steering work passed admission or caused lease or route activity"); + Console.WriteLine("PASS: orphan steering work is preserved without lease or route activity"); return 0; } finally { foreach(var entry in original)Environment.SetEnvironmentVariable(entry.Key,entry.Value); diff --git a/tests/fixtures/native-owner/exercise.sh b/tests/fixtures/native-owner/exercise.sh index da7d81a78cc..a5d8da9e662 100644 --- a/tests/fixtures/native-owner/exercise.sh +++ b/tests/fixtures/native-owner/exercise.sh @@ -1,4 +1,6 @@ #!/usr/bin/env bash +# Usage: FM_HOME= FM_PROBE_HOME= FM_PROBE_JQ_IMAGE= exercise.sh +# Required environment: FM_HOME, FM_PROBE_HOME, FM_PROBE_JQ_IMAGE. # Real startup in an empty disposable home; no mocked startup/deferred owners. set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) diff --git a/tests/fixtures/native-owner/jq b/tests/fixtures/native-owner/jq index 3b891a815f1..3a3f8af0a24 100644 --- a/tests/fixtures/native-owner/jq +++ b/tests/fixtures/native-owner/jq @@ -1,4 +1,6 @@ #!/usr/bin/env bash +# Usage: FM_PROBE_JQ_IMAGE= jq +# Required environment: FM_PROBE_JQ_IMAGE. # jq stays in the existing Docker image. Only this disposable build is mounted. set -eu root=$(cd "$(dirname "$0")/.." && pwd) diff --git a/tests/fixtures/native-owner/notification-ack.sh b/tests/fixtures/native-owner/notification-ack.sh index 2baca49551c..714116618b3 100644 --- a/tests/fixtures/native-owner/notification-ack.sh +++ b/tests/fixtures/native-owner/notification-ack.sh @@ -1,4 +1,6 @@ #!/usr/bin/env bash +# Usage: FM_HOME= FM_PROBE_HOME= FM_PROBE_JQ_IMAGE= notification-ack.sh +# Required environment: FM_HOME, FM_PROBE_HOME, FM_PROBE_JQ_IMAGE. # Only controller-validated receipt data is accepted; no command text is parsed. set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) diff --git a/tests/fixtures/native-owner/notification-check.sh b/tests/fixtures/native-owner/notification-check.sh index e2e7839bfdb..8c43c36a88d 100644 --- a/tests/fixtures/native-owner/notification-check.sh +++ b/tests/fixtures/native-owner/notification-check.sh @@ -1,4 +1,6 @@ #!/usr/bin/env bash +# Usage: FM_HOME= FM_PROBE_HOME= FM_PROBE_JQ_IMAGE= notification-check.sh +# Required environment: FM_HOME, FM_PROBE_HOME, FM_PROBE_JQ_IMAGE. # Fixed notification operation, created and scoped by the existing controller. set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) From 788f4f57870e2db47a9974ea3fb18ae06fc5c1f6 Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 00:15:41 +1200 Subject: [PATCH 19/61] no-mistakes(review): Reject owner-managed residual work and preserve concurrent acknowledgements --- bin/fm-guard.sh | 9 +++++ bin/fm-supervision-lib.sh | 39 ++++++++++++++++++--- bin/fm-turnend-guard.sh | 12 +++++++ bin/fm-wake-lib.sh | 6 ++-- tests/fixtures/native-owner/NativeDriver.cs | 24 +++++++++++-- tests/fixtures/native-owner/ReceiptTests.cs | 27 ++++++++++++++ tests/fm-turnend-guard.test.sh | 31 ++++++++++++---- 7 files changed, 132 insertions(+), 16 deletions(-) diff --git a/bin/fm-guard.sh b/bin/fm-guard.sh index ba9ee330465..a9b81ab51f7 100755 --- a/bin/fm-guard.sh +++ b/bin/fm-guard.sh @@ -173,6 +173,9 @@ fm_supervision_status "$STATE" "$GRACE" in_flight=$FM_SUP_IN_FLIGHT sources=$FM_SUP_SOURCES checks=$FM_SUP_CHECKS +check_inputs=$FM_SUP_CHECK_INPUTS +pending_replies=$FM_SUP_PENDING_REPLIES +reconcile_requests=$FM_SUP_RECONCILE_REQUESTS needed=$FM_SUP_NEEDED beacon_desc=$FM_SUP_BEACON_DESC fm_watcher_supervision_verdict "$STATE" "$WATCH" "$GRACE" "$FM_HOME" "$FM_ROOT" @@ -240,6 +243,12 @@ if [ "$watcher_healthy" = false ]; then printf '● %s process-event source(s) registered, but %s.\n' "$sources" "$watcher_cause" elif [ "$checks" -gt 0 ]; then printf '● %s registered custom check(s), but %s.\n' "$checks" "$watcher_cause" + elif [ "$check_inputs" -gt 0 ]; then + printf '● %s state check input(s), but %s.\n' "$check_inputs" "$watcher_cause" + elif [ "$pending_replies" -gt 0 ]; then + printf '● %s pending secondmate reply record(s), but %s.\n' "$pending_replies" "$watcher_cause" + elif [ "$reconcile_requests" -gt 0 ]; then + printf '● %s secondmate reconcile request(s), but %s.\n' "$reconcile_requests" "$watcher_cause" else printf '● X-mode relay polling needs supervision, but %s.\n' "$watcher_cause" fi diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index 1bbc5708834..fc082950c26 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -35,15 +35,37 @@ fm_sup_stat_mtime() { # sweep's call at execution time, and a home whose check # no longer validates needs the watcher precisely so the # sweep can report the rejection instead of going quiet. -# FM_SUP_NEEDED true/false - in-flight work, an X-mode relay poll, a +# FM_SUP_CHECK_INPUTS count of state/*.check.sh inputs, including relay, +# registered, and unregistered checks +# FM_SUP_PENDING_REPLIES count of parent-owned pending-reply inputs +# FM_SUP_RECONCILE_REQUESTS count of secondmate reconcile-notify inputs +# FM_SUP_NEEDED true/false - in-flight work, a state check input, a # registered event source (a source is a wait on an # external process, not a task, so it has no metadata), -# or a registered custom check +# a pending reply, or a reconcile-notify request # FM_SUP_WATCHER_FRESH true/false - a watcher beacon within the grace window # FM_SUP_BEACON_DESC human-readable beacon age, for banners ("never" if absent) # FM_SUP_QUEUE_PENDING true/false - state/.wake-queue has unread records # grace-seconds defaults to $FM_GUARD_GRACE, then 300, matching fm-guard.sh. # Always returns 0; callers read the vars, or use fm_supervision_unhealthy below. +fm_sup_directory_entry_count() { + local directory=$1 entry count=0 + if [ ! -e "$directory" ] && [ ! -L "$directory" ]; then + printf '0\n' + return 0 + fi + if [ ! -d "$directory" ] || [ -L "$directory" ]; then + printf '1\n' + return 0 + fi + for entry in "$directory"/* "$directory"/.[!.]* "$directory"/..?*; do + if [ -e "$entry" ] || [ -L "$entry" ]; then + count=$((count + 1)) + fi + done + printf '%s\n' "$count" +} + fm_supervision_status() { local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} meta source check id beat m age FM_SUP_IN_FLIGHT=0 @@ -62,8 +84,12 @@ fm_supervision_status() { FM_SUP_SOURCES=$((FM_SUP_SOURCES + 1)) done FM_SUP_CHECKS=0 + FM_SUP_CHECK_INPUTS=0 for check in "$state"/*.check.sh; do - [ -e "$check" ] || continue + if [ ! -e "$check" ] && [ ! -L "$check" ]; then + continue + fi + FM_SUP_CHECK_INPUTS=$((FM_SUP_CHECK_INPUTS + 1)) id=${check##*/} id=${id%.check.sh} if [ "$id" = x-watch ]; then @@ -72,10 +98,13 @@ fm_supervision_status() { [ -e "$state/$id.check-trust" ] || continue FM_SUP_CHECKS=$((FM_SUP_CHECKS + 1)) done + FM_SUP_PENDING_REPLIES=$(fm_sup_directory_entry_count "$state/pending-replies") + FM_SUP_RECONCILE_REQUESTS=$(fm_sup_directory_entry_count "$state/reconcile-notify") if [ "$FM_SUP_IN_FLIGHT" -gt 0 ] \ - || [ -f "$state/x-watch.check.sh" ] \ || [ "$FM_SUP_SOURCES" -gt 0 ] \ - || [ "$FM_SUP_CHECKS" -gt 0 ]; then + || [ "$FM_SUP_CHECK_INPUTS" -gt 0 ] \ + || [ "$FM_SUP_PENDING_REPLIES" -gt 0 ] \ + || [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then FM_SUP_NEEDED=true fi diff --git a/bin/fm-turnend-guard.sh b/bin/fm-turnend-guard.sh index ffceafaee51..7d06d29f2be 100755 --- a/bin/fm-turnend-guard.sh +++ b/bin/fm-turnend-guard.sh @@ -234,6 +234,12 @@ block_stop() { printf '● %s process-event source(s) registered, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_SOURCES" "$FM_SUP_BEACON_DESC" elif [ "$FM_SUP_CHECKS" -gt 0 ]; then printf '● %s registered custom check(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_CHECKS" "$FM_SUP_BEACON_DESC" + elif [ "$FM_SUP_CHECK_INPUTS" -gt 0 ]; then + printf '● %s state check input(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_CHECK_INPUTS" "$FM_SUP_BEACON_DESC" + elif [ "$FM_SUP_PENDING_REPLIES" -gt 0 ]; then + printf '● %s pending secondmate reply record(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_PENDING_REPLIES" "$FM_SUP_BEACON_DESC" + elif [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then + printf '● %s secondmate reconcile request(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_RECONCILE_REQUESTS" "$FM_SUP_BEACON_DESC" else printf '● X-mode relay polling needs supervision, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_BEACON_DESC" fi @@ -496,6 +502,12 @@ if [ "$terminal_status" -eq 0 ]; then NEED_DESC="$FM_SUP_SOURCES process-event source(s) registered" elif [ "$FM_SUP_CHECKS" -gt 0 ]; then NEED_DESC="$FM_SUP_CHECKS registered custom check(s)" + elif [ "$FM_SUP_CHECK_INPUTS" -gt 0 ]; then + NEED_DESC="$FM_SUP_CHECK_INPUTS state check input(s)" + elif [ "$FM_SUP_PENDING_REPLIES" -gt 0 ]; then + NEED_DESC="$FM_SUP_PENDING_REPLIES pending secondmate reply record(s)" + elif [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then + NEED_DESC="$FM_SUP_RECONCILE_REQUESTS secondmate reconcile request(s)" else NEED_DESC="X-mode relay polling active" fi diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 863f9bf7b83..c88c74b87e7 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -2219,7 +2219,7 @@ fm_wake_ack_evidence_capture() { fi FM_WAKE_ACK_EVIDENCE_MARKER=$FM_RECOVERY_MARKER_TOKEN case "$FM_WAKE_ACK_EVIDENCE_MARKER" in - pending:handling:*|announced:handling:*) ;; + pending:handling:*|announced:handling:*|pending:downtime:*|announced:downtime:*) ;; *) fm_lock_release "$FM_WAKE_QUEUE_LOCK" fm_wake_ack_evidence_clear @@ -2380,7 +2380,7 @@ fm_wake_ack_evidence_load() { # END { if ((NR == 0 && cutoff != 0) || (NR > 0 && max != cutoff) || bad) exit 1 } ' "$FM_WAKE_ACK_EVIDENCE_ROWS"; then fm_wake_ack_evidence_clear; return 1; fi case "$FM_WAKE_ACK_EVIDENCE_MARKER" in - pending:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|announced:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|acked:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION") ;; + pending:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|announced:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|pending:downtime:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|announced:downtime:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|acked:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION") ;; '') [ "$FM_WAKE_ACK_EVIDENCE_LEGACY" = 1 ] && [ "$FM_WAKE_ACK_EVIDENCE_CUTOFF" -gt 0 ] || { fm_wake_ack_evidence_clear; return 1; } ;; *) fm_wake_ack_evidence_clear; return 1 ;; esac @@ -2405,7 +2405,7 @@ fm_wake_ack_evidence_precondition() { # fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" || { fm_wake_ack_evidence_clear; return 1; } fm_recovery_marker_snapshot "$marker" || true case "$FM_RECOVERY_MARKER_TOKEN" in - pending:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|announced:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION") ;; + pending:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|announced:handling:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|pending:downtime:"$FM_WAKE_ACK_EVIDENCE_GENERATION"|announced:downtime:"$FM_WAKE_ACK_EVIDENCE_GENERATION") ;; *) fm_lock_release "$FM_WAKE_QUEUE_LOCK"; fm_wake_ack_evidence_clear; return 1 ;; esac if [ "$FM_WAKE_ACK_EVIDENCE_CUTOFF" = 0 ] && [ -s "$FM_WAKE_QUEUE" ]; then diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 5c8140ecf6e..56326fddde5 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -112,9 +112,24 @@ static int EnvironmentTests() { Directory.CreateDirectory(registeredState); File.WriteAllText(Path.Combine(registeredState,"custom.check.sh"),"#!/usr/bin/env bash\nprintf executed > \""+canary.Replace('\\','/')+"\"\n"); File.WriteAllText(Path.Combine(registeredState,"custom.check-trust"),"fm-custom-check-v1\n"+new string('0',64)+"\n"); - refused=false;try{EmptyFleet(registered);}catch(InvalidOperationException){refused=true;} - if(!refused||File.Exists(canary))throw new InvalidOperationException("Registered custom work passed admission or executed during inspection"); + refused=false;try{EmptyFleet(registered);using(var lease=new NativeHomeLease(registered)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.Exists(canary)||File.Exists(Path.Combine(registered,"owner-probe.json"))||File.Exists(Path.Combine(registeredState,".watch.lock")))throw new InvalidOperationException("Registered custom work passed admission, acquired a lease, or executed during inspection"); Console.WriteLine("PASS: registered custom checks are refused without execution"); + string unregistered=Path.Combine(directory,"unregistered"),unregisteredState=Path.Combine(unregistered,"state"),unregisteredCanary=Path.Combine(unregistered,"executed"),unregisteredCheck=Path.Combine(unregisteredState,"orphan.check.sh"),unregisteredBody="#!/usr/bin/env bash\nprintf executed > \""+unregisteredCanary.Replace('\\','/')+"\"\n"; + Directory.CreateDirectory(unregisteredState);File.WriteAllText(unregisteredCheck,unregisteredBody); + refused=false;try{EmptyFleet(unregistered);using(var lease=new NativeHomeLease(unregistered)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(unregisteredCheck)!=unregisteredBody||File.Exists(unregisteredCanary)||File.Exists(Path.Combine(unregistered,"owner-probe.json"))||File.Exists(Path.Combine(unregisteredState,".watch.lock")))throw new InvalidOperationException("Unregistered custom work passed admission, changed, acquired a lease, or executed during inspection"); + Console.WriteLine("PASS: unregistered custom checks are preserved and refused without execution"); + string pendingReply=Path.Combine(directory,"pending-reply"),pendingState=Path.Combine(pendingReply,"state"),pendingDirectory=Path.Combine(pendingState,"pending-replies"),pendingRecord=Path.Combine(pendingDirectory,"0123456789abcdef"),pendingBody="schema=fm-pending-reply.v1\nphase=awaiting_report\n"; + Directory.CreateDirectory(pendingDirectory);File.WriteAllText(pendingRecord,pendingBody); + refused=false;try{EmptyFleet(pendingReply);using(var lease=new NativeHomeLease(pendingReply)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(pendingRecord)!=pendingBody||File.Exists(Path.Combine(pendingReply,"owner-probe.json"))||File.Exists(Path.Combine(pendingState,".watch.lock"))||Directory.GetFiles(pendingState,"*",SearchOption.AllDirectories).Length!=1)throw new InvalidOperationException("Pending reply work passed admission, changed, or caused lease or route activity"); + Console.WriteLine("PASS: pending replies are preserved without lease or route activity"); + string reconcile=Path.Combine(directory,"reconcile-request"),reconcileState=Path.Combine(reconcile,"state"),reconcileDirectory=Path.Combine(reconcileState,"reconcile-notify"),reconcileRecord=Path.Combine(reconcileDirectory,"request-fixture.json"),reconcileBody="{\"version\":1}\n"; + Directory.CreateDirectory(reconcileDirectory);File.WriteAllText(reconcileRecord,reconcileBody); + refused=false;try{EmptyFleet(reconcile);using(var lease=new NativeHomeLease(reconcile)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(reconcileRecord)!=reconcileBody||File.Exists(Path.Combine(reconcile,"owner-probe.json"))||File.Exists(Path.Combine(reconcileState,".watch.lock"))||File.Exists(Path.Combine(reconcileState,".reconcile-notify-process.lock"))||Directory.GetFiles(reconcileState,"*",SearchOption.AllDirectories).Length!=1)throw new InvalidOperationException("Reconcile request passed admission, changed, or caused lease or route activity"); + Console.WriteLine("PASS: reconcile requests are preserved without lease or route activity"); string handoff=Path.Combine(directory,"handoff"),outbox=Path.Combine(handoff,"data","handoff","agent.outbox.md"),outboxBody="# Backlog\n\n## Queued\n\n- [ ] routed-work\n"; Directory.CreateDirectory(Path.GetDirectoryName(outbox));File.WriteAllText(outbox,outboxBody); refused=false;try{EmptyFleet(handoff);using(var lease=new NativeHomeLease(handoff)){} }catch(InvalidOperationException){refused=true;} @@ -125,6 +140,11 @@ static int EnvironmentTests() { refused=false;try{EmptyFleet(steering);using(var lease=new NativeHomeLease(steering)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(message)!=messageBody||File.Exists(Path.Combine(steering,"owner-probe.json"))||File.Exists(Path.Combine(steeringState,".lock")))throw new InvalidOperationException("Orphan steering work passed admission or caused lease or route activity"); Console.WriteLine("PASS: orphan steering work is preserved without lease or route activity"); + string supported=Path.Combine(directory,"supported-notification"),supportedState=Path.Combine(supported,"state"),supportedInbox=Path.Combine(supportedState,"inbox"),supportedNote=Path.Combine(supportedInbox,"note-id.note"),supportedQueue=Path.Combine(supportedState,".wake-queue"),supportedMarker=Path.Combine(supportedState,".watcher-down"),supportedBody="preserve notification\n"; + Directory.CreateDirectory(supportedInbox);File.WriteAllText(supportedNote,supportedBody);File.WriteAllText(supportedQueue,"1\t1\tcheck\tinbox:note-id\tcaptain inbox note\n");File.WriteAllText(supportedMarker,"pending:handling:supported\n"); + EmptyFleet(supported); + if(File.ReadAllText(supportedNote)!=supportedBody||!File.ReadAllText(supportedQueue).Contains("inbox:note-id")||File.ReadAllText(supportedMarker)!="pending:handling:supported\n"||File.Exists(Path.Combine(supported,"owner-probe.json")))throw new InvalidOperationException("Supported top-level notification state was changed or leased during admission"); + Console.WriteLine("PASS: supported top-level notification state remains admissible and unchanged"); return 0; } finally { foreach(var entry in original)Environment.SetEnvironmentVariable(entry.Key,entry.Value); diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index ccba3b95b21..f2279760932 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -53,6 +53,16 @@ static Dictionary OwnerPayload(NativeHomeLease lease,string chall return payload; } } + static void OwnerAcknowledge(NativeHomeLease lease,Dictionary payload) { + string script=Path.Combine(NativeOwner.CodeRoot,"bin","native-owner","ack-evidence.sh"),opaque=(string)payload["ownerEvidence"]; + var start=NativeOwner.BashHelper(script,"acknowledge-token",lease.Home);start.RedirectStandardInput=true;start.RedirectStandardOutput=true;start.RedirectStandardError=true; + using(var process=Process.Start(start)) { + var output=process.StandardOutput.ReadToEndAsync();var error=process.StandardError.ReadToEndAsync(); + process.StandardInput.Write(opaque);process.StandardInput.Close(); + if(!process.WaitForExit(30000)){process.Kill();throw new IOException("Acknowledgement owner exceeded its bound");} + if(process.ExitCode!=0)throw new IOException("Acknowledgement owner refused its captured target: "+error.Result+output.Result); + } + } static Dictionary ZeroPayload(NativeHomeLease lease) { string[] target=ZeroRecovery(lease,"present").Split('\t'); Expect(target.Length==2 && target[0]=="0","Real recovery owner did not produce a zero-row target"); @@ -211,6 +221,23 @@ public static int Run() { if(scenario=="no-inbox-targets")Expect(File.Exists(Pending(lease)),"Unrelated inbox note was consumed"); }); } + foreach(string timing in new [] {"before-capture","after-ack-started"}) { + Case("concurrent wake remains pending: "+timing,lease=>{ + Targets(lease); + if(timing=="before-capture")ZeroRecovery(lease,"append"); + using(var journal=new NativeReceiptJournal(lease,A)) { + var delivery=journal.Present(OwnerPayload(lease,"concurrent"));string receipt=(string)delivery["receipt"],generation=(string)delivery["generation"]; + var evidence=NativeAcknowledgementEvidence.Capture(lease,delivery); + journal.BeginAcknowledgement(receipt,"concurrent",evidence); + if(timing=="after-ack-started")ZeroRecovery(lease,"append"); + Expect(File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")).Contains(":"+generation+"\n"),"Concurrent wake changed the recovery generation"); + OwnerAcknowledge(lease,delivery);journal.CompleteAcknowledgement(receipt); + } + string queue=File.ReadAllText(Queue(lease)); + Expect(queue.Contains("later-notification")&&!queue.Contains("inbox:note-id"),"Acknowledgement did not retain only the later wake"); + Expect(!File.Exists(Pending(lease))&&File.ReadAllText(Handled(lease))=="original captured inbox record\n","Acknowledgement did not consume exactly the captured note"); + }); + } Case("zero-row recovery rejects an unproven empty target",lease=>{ Directory.CreateDirectory(Path.GetDirectoryName(Queue(lease)));File.WriteAllText(Queue(lease),""); var payload=new Dictionary{{"challenge","zero"},{"message","recovery"},{"seq","0"},{"generation","missing"},{"notes",new string[0]}}; diff --git a/tests/fm-turnend-guard.test.sh b/tests/fm-turnend-guard.test.sh index f0245f6a827..b4bbd1ad519 100755 --- a/tests/fm-turnend-guard.test.sh +++ b/tests/fm-turnend-guard.test.sh @@ -140,16 +140,33 @@ test_predicate_registered_check_survives_rebinding_drift() { pass "fm_supervision_needed: a registered check whose bytes drifted still needs supervision" } -test_predicate_unregistered_check_needs_nothing() { +test_predicate_unregistered_check_needs_supervision() { local state="$TMP_ROOT/pred-check-unregistered/state" mkdir -p "$state" printf '#!/usr/bin/env bash\nexit 0\n' > "$state/rogue.check.sh" chmod 700 "$state/rogue.check.sh" - if fm_supervision_needed "$state" 300; then - fail "a check with no trust binding must not arm supervision" - fi + fm_supervision_needed "$state" 300 || fail "an unregistered check did not keep supervision active for rejection" [ "$FM_SUP_CHECKS" -eq 0 ] || fail "an unregistered check must not be counted, got $FM_SUP_CHECKS" - pass "fm_supervision_needed: false for a check.sh with no registration binding" + [ "$FM_SUP_CHECK_INPUTS" -eq 1 ] || fail "expected one state check input, got $FM_SUP_CHECK_INPUTS" + pass "fm_supervision_needed: an unregistered check remains visible for rejection" +} + +test_predicate_pending_reply_needs_supervision() { + local state="$TMP_ROOT/pred-pending-reply/state" + mkdir -p "$state/pending-replies" + printf 'schema=fm-pending-reply.v1\n' > "$state/pending-replies/0123456789abcdef" + fm_supervision_needed "$state" 300 || fail "a pending secondmate reply did not keep supervision active" + [ "$FM_SUP_PENDING_REPLIES" -eq 1 ] || fail "expected one pending reply input, got $FM_SUP_PENDING_REPLIES" + pass "fm_supervision_needed: a pending secondmate reply needs supervision" +} + +test_predicate_reconcile_request_needs_supervision() { + local state="$TMP_ROOT/pred-reconcile-request/state" + mkdir -p "$state/reconcile-notify" + printf '{}\n' > "$state/reconcile-notify/request-fixture.json" + fm_supervision_needed "$state" 300 || fail "a reconcile request did not keep supervision active" + [ "$FM_SUP_RECONCILE_REQUESTS" -eq 1 ] || fail "expected one reconcile request input, got $FM_SUP_RECONCILE_REQUESTS" + pass "fm_supervision_needed: a secondmate reconcile request needs supervision" } test_predicate_task_pr_poll_is_not_a_custom_check() { @@ -2208,7 +2225,9 @@ test_predicate_x_mode_needs_supervision test_predicate_source_needs_supervision test_predicate_registered_check_needs_supervision test_predicate_registered_check_survives_rebinding_drift -test_predicate_unregistered_check_needs_nothing +test_predicate_unregistered_check_needs_supervision +test_predicate_pending_reply_needs_supervision +test_predicate_reconcile_request_needs_supervision test_predicate_task_pr_poll_is_not_a_custom_check test_predicate_relay_shim_is_not_a_custom_check test_hook_silent_when_no_work_in_flight From 30d765d671fe65b08941d15f5987be58109eb8b9 Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 00:46:08 +1200 Subject: [PATCH 20/61] no-mistakes(review): Harden native admission and acknowledgement evidence --- bin/fm-guard.sh | 22 ++------- bin/fm-supervision-lib.sh | 41 +++++++++++++++- bin/fm-turnend-guard.sh | 28 ++--------- bin/native-owner/NativeOperations.cs | 33 +++++++------ bin/native-owner/NativeReceiptJournal.cs | 10 +++- bin/native-owner/ack.sh | 2 +- tests/fixtures/native-owner/AppHost.mjs | 15 +++--- tests/fixtures/native-owner/Launcher.mjs | 4 +- tests/fixtures/native-owner/NativeDriver.cs | 10 +++- tests/fixtures/native-owner/ReceiptTests.cs | 48 +++++++++++++++---- tests/fixtures/native-owner/Run-Cycle.mjs | 7 +-- .../fixtures/native-owner/notification-ack.sh | 21 +++++++- tests/fm-native-owner-codex-live-e2e.test.sh | 4 ++ tests/fm-turnend-guard.test.sh | 15 ++++++ 14 files changed, 175 insertions(+), 85 deletions(-) diff --git a/bin/fm-guard.sh b/bin/fm-guard.sh index a9b81ab51f7..5ae3b529ae5 100755 --- a/bin/fm-guard.sh +++ b/bin/fm-guard.sh @@ -170,12 +170,8 @@ fi # grace-based predicate (bin/fm-supervision-lib.sh), which owns what needs # supervision. fm_supervision_status "$STATE" "$GRACE" -in_flight=$FM_SUP_IN_FLIGHT -sources=$FM_SUP_SOURCES -checks=$FM_SUP_CHECKS -check_inputs=$FM_SUP_CHECK_INPUTS -pending_replies=$FM_SUP_PENDING_REPLIES -reconcile_requests=$FM_SUP_RECONCILE_REQUESTS +reason_count=$FM_SUP_REASON_COUNT +reason_label=$FM_SUP_REASON_LABEL needed=$FM_SUP_NEEDED beacon_desc=$FM_SUP_BEACON_DESC fm_watcher_supervision_verdict "$STATE" "$WATCH" "$GRACE" "$FM_HOME" "$FM_ROOT" @@ -237,18 +233,8 @@ if [ "$watcher_healthy" = false ]; then else watcher_cause=$(printf 'no watcher has a fresh beacon (last beat: %s, grace %ss)' "$beacon_desc" "$GRACE") fi - if [ "$in_flight" -gt 0 ]; then - printf '● %s task(s) in flight, but %s.\n' "$in_flight" "$watcher_cause" - elif [ "$sources" -gt 0 ]; then - printf '● %s process-event source(s) registered, but %s.\n' "$sources" "$watcher_cause" - elif [ "$checks" -gt 0 ]; then - printf '● %s registered custom check(s), but %s.\n' "$checks" "$watcher_cause" - elif [ "$check_inputs" -gt 0 ]; then - printf '● %s state check input(s), but %s.\n' "$check_inputs" "$watcher_cause" - elif [ "$pending_replies" -gt 0 ]; then - printf '● %s pending secondmate reply record(s), but %s.\n' "$pending_replies" "$watcher_cause" - elif [ "$reconcile_requests" -gt 0 ]; then - printf '● %s secondmate reconcile request(s), but %s.\n' "$reconcile_requests" "$watcher_cause" + if [ "$reason_count" -gt 0 ]; then + printf '● %s %s, but %s.\n' "$reason_count" "$reason_label" "$watcher_cause" else printf '● X-mode relay polling needs supervision, but %s.\n' "$watcher_cause" fi diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index fc082950c26..70536970bc9 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -39,10 +39,15 @@ fm_sup_stat_mtime() { # registered, and unregistered checks # FM_SUP_PENDING_REPLIES count of parent-owned pending-reply inputs # FM_SUP_RECONCILE_REQUESTS count of secondmate reconcile-notify inputs +# FM_SUP_TURN_ENDS count of state/*.turn-ended watcher inputs # FM_SUP_NEEDED true/false - in-flight work, a state check input, a # registered event source (a source is a wait on an # external process, not a task, so it has no metadata), -# a pending reply, or a reconcile-notify request +# a pending reply, a reconcile-notify request, or a +# turn-ended notification +# FM_SUP_REASON_COUNT count for the highest-priority active reason, or zero +# for Relay polling +# FM_SUP_REASON_LABEL caller-neutral label for that selected reason # FM_SUP_WATCHER_FRESH true/false - a watcher beacon within the grace window # FM_SUP_BEACON_DESC human-readable beacon age, for banners ("never" if absent) # FM_SUP_QUEUE_PENDING true/false - state/.wake-queue has unread records @@ -67,7 +72,7 @@ fm_sup_directory_entry_count() { } fm_supervision_status() { - local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} meta source check id beat m age + local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} meta source check turn_end id beat m age FM_SUP_IN_FLIGHT=0 FM_SUP_NEEDED=false FM_SUP_WATCHER_FRESH=false @@ -100,13 +105,45 @@ fm_supervision_status() { done FM_SUP_PENDING_REPLIES=$(fm_sup_directory_entry_count "$state/pending-replies") FM_SUP_RECONCILE_REQUESTS=$(fm_sup_directory_entry_count "$state/reconcile-notify") + FM_SUP_TURN_ENDS=0 + for turn_end in "$state"/*.turn-ended; do + if [ -e "$turn_end" ] || [ -L "$turn_end" ]; then + FM_SUP_TURN_ENDS=$((FM_SUP_TURN_ENDS + 1)) + fi + done if [ "$FM_SUP_IN_FLIGHT" -gt 0 ] \ + || [ "$FM_SUP_TURN_ENDS" -gt 0 ] \ || [ "$FM_SUP_SOURCES" -gt 0 ] \ || [ "$FM_SUP_CHECK_INPUTS" -gt 0 ] \ || [ "$FM_SUP_PENDING_REPLIES" -gt 0 ] \ || [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then FM_SUP_NEEDED=true fi + if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then + FM_SUP_REASON_COUNT=$FM_SUP_IN_FLIGHT + FM_SUP_REASON_LABEL='task(s) in flight' + elif [ "$FM_SUP_TURN_ENDS" -gt 0 ]; then + FM_SUP_REASON_COUNT=$FM_SUP_TURN_ENDS + FM_SUP_REASON_LABEL='turn-end notification(s) pending' + elif [ "$FM_SUP_SOURCES" -gt 0 ]; then + FM_SUP_REASON_COUNT=$FM_SUP_SOURCES + FM_SUP_REASON_LABEL='process-event source(s) registered' + elif [ "$FM_SUP_CHECKS" -gt 0 ]; then + FM_SUP_REASON_COUNT=$FM_SUP_CHECKS + FM_SUP_REASON_LABEL='registered custom check(s)' + elif [ "$FM_SUP_CHECK_INPUTS" -gt 0 ]; then + FM_SUP_REASON_COUNT=$FM_SUP_CHECK_INPUTS + FM_SUP_REASON_LABEL='state check input(s)' + elif [ "$FM_SUP_PENDING_REPLIES" -gt 0 ]; then + FM_SUP_REASON_COUNT=$FM_SUP_PENDING_REPLIES + FM_SUP_REASON_LABEL='pending secondmate reply record(s)' + elif [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then + FM_SUP_REASON_COUNT=$FM_SUP_RECONCILE_REQUESTS + FM_SUP_REASON_LABEL='secondmate reconcile request(s)' + else + FM_SUP_REASON_COUNT=0 + FM_SUP_REASON_LABEL='X-mode relay polling' + fi beat="$state/.last-watcher-beat" if [ -e "$beat" ]; then diff --git a/bin/fm-turnend-guard.sh b/bin/fm-turnend-guard.sh index 7d06d29f2be..ca52b16af8c 100755 --- a/bin/fm-turnend-guard.sh +++ b/bin/fm-turnend-guard.sh @@ -228,18 +228,8 @@ block_stop() { { printf '●%s\n' "$rule" printf '● TURN WOULD END BLIND - SUPERVISION IS OFF\n' - if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then - printf '● %s task(s) in flight, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_IN_FLIGHT" "$FM_SUP_BEACON_DESC" - elif [ "$FM_SUP_SOURCES" -gt 0 ]; then - printf '● %s process-event source(s) registered, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_SOURCES" "$FM_SUP_BEACON_DESC" - elif [ "$FM_SUP_CHECKS" -gt 0 ]; then - printf '● %s registered custom check(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_CHECKS" "$FM_SUP_BEACON_DESC" - elif [ "$FM_SUP_CHECK_INPUTS" -gt 0 ]; then - printf '● %s state check input(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_CHECK_INPUTS" "$FM_SUP_BEACON_DESC" - elif [ "$FM_SUP_PENDING_REPLIES" -gt 0 ]; then - printf '● %s pending secondmate reply record(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_PENDING_REPLIES" "$FM_SUP_BEACON_DESC" - elif [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then - printf '● %s secondmate reconcile request(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_RECONCILE_REQUESTS" "$FM_SUP_BEACON_DESC" + if [ "$FM_SUP_REASON_COUNT" -gt 0 ]; then + printf '● %s %s, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_REASON_COUNT" "$FM_SUP_REASON_LABEL" "$FM_SUP_BEACON_DESC" else printf '● X-mode relay polling needs supervision, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_BEACON_DESC" fi @@ -496,18 +486,8 @@ budget_account_current_epoch block || block_stop terminal_fail_open terminal_status=$? if [ "$terminal_status" -eq 0 ]; then - if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then - NEED_DESC="$FM_SUP_IN_FLIGHT task(s) in flight" - elif [ "$FM_SUP_SOURCES" -gt 0 ]; then - NEED_DESC="$FM_SUP_SOURCES process-event source(s) registered" - elif [ "$FM_SUP_CHECKS" -gt 0 ]; then - NEED_DESC="$FM_SUP_CHECKS registered custom check(s)" - elif [ "$FM_SUP_CHECK_INPUTS" -gt 0 ]; then - NEED_DESC="$FM_SUP_CHECK_INPUTS state check input(s)" - elif [ "$FM_SUP_PENDING_REPLIES" -gt 0 ]; then - NEED_DESC="$FM_SUP_PENDING_REPLIES pending secondmate reply record(s)" - elif [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then - NEED_DESC="$FM_SUP_RECONCILE_REQUESTS secondmate reconcile request(s)" + if [ "$FM_SUP_REASON_COUNT" -gt 0 ]; then + NEED_DESC="$FM_SUP_REASON_COUNT $FM_SUP_REASON_LABEL" else NEED_DESC="X-mode relay polling active" fi diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index 08638177901..50060fc8dba 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -10,15 +10,20 @@ public static partial class NativeOwner { static NativeReceiptJournal operationJournal; static bool shutdownRequested; internal static string CodeRoot { get { return Path.GetDirectoryName(Path.GetDirectoryName(OwnExe)); } } + static SortedDictionary TrustedBaseEnvironment() { + var values=new SortedDictionary(StringComparer.OrdinalIgnoreCase); + foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { + string value=Environment.GetEnvironmentVariable(key);if(value!=null)values[key]=value; + } + values["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); + return values; + } internal static ProcessStartInfo BashHelper(string script,string arguments,string home,bool includeProbe=false) { var start=new ProcessStartInfo(@"C:\Program Files\Git\bin\bash.exe","--noprofile --norc "+Quote(script.Replace('\\','/'))+(string.IsNullOrEmpty(arguments) ? "" : " "+arguments)) {UseShellExecute=false,CreateNoWindow=true}; start.EnvironmentVariables.Clear(); - foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { - string value=Environment.GetEnvironmentVariable(key);if(value!=null)start.EnvironmentVariables[key]=value; - } - start.EnvironmentVariables["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); + foreach(var entry in TrustedBaseEnvironment())start.EnvironmentVariables[entry.Key]=entry.Value; if(includeProbe) { - foreach(string key in new [] {"FM_PROBE_PIPE","FM_PROBE_SESSION","FM_PROBE_HOME","FM_PROBE_NONCE","FM_PROBE_EXE","FM_PROBE_JQ_IMAGE","FM_PROBE_VERIFY_ONLY"}) { + foreach(string key in new [] {"FM_PROBE_PIPE","FM_PROBE_SESSION","FM_PROBE_HOME","FM_PROBE_NONCE","FM_PROBE_JQ_IMAGE","FM_PROBE_VERIFY_ONLY"}) { string value=Environment.GetEnvironmentVariable(key);if(value!=null)start.EnvironmentVariables[key]=value; } } @@ -56,7 +61,6 @@ static SortedDictionary EnvironmentFor(string pipe, string sessio } result["FM_PROBE_PIPE"] = pipe; result["FM_PROBE_SESSION"] = session; result["FM_PROBE_HOME"] = home; result["FM_PROBE_NONCE"] = nonce; - result["FM_PROBE_EXE"] = OwnExe; return result; } static ChildScope StartOwnerOperation(IntPtr parentJob, IntPtr environment, string home, ref SI startup, string purpose="startup") { @@ -67,17 +71,13 @@ static ChildScope StartOwnerOperation(IntPtr parentJob, IntPtr environment, stri try { NativeOperationLifetime.Configure(scope.job); string operation=purpose=="startup" ? "owner-operation" : "notification-operation "+purpose; - var values=new SortedDictionary(StringComparer.OrdinalIgnoreCase); - foreach(string key in new [] {"SystemRoot","WINDIR","TEMP","TMP","USERPROFILE","APPDATA","LOCALAPPDATA"}) { - string value=Environment.GetEnvironmentVariable(key); if(value!=null) values[key]=value; - } - values["PATH"]=@"C:\Program Files\Git\usr\bin;C:\Windows\System32;C:\Windows;C:\Program Files\nodejs;C:\Program Files\GitHub CLI;"+Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),"npm"); + var values=TrustedBaseEnvironment(); int offset=0; while(Marshal.ReadInt16(environment,offset)!=0) { string entry=Marshal.PtrToStringUni(IntPtr.Add(environment,offset)); offset+=(entry.Length+1)*2; int split=entry.IndexOf('='); if(split<=0) continue; string key=entry.Substring(0,split); - if(key.StartsWith("FM_PROBE_",StringComparison.Ordinal) || key=="FM_HOME" || key=="MSYS") values[key]=entry.Substring(split+1); + if(key.StartsWith("FM_PROBE_",StringComparison.OrdinalIgnoreCase) || string.Equals(key,"FM_HOME",StringComparison.OrdinalIgnoreCase) || string.Equals(key,"MSYS",StringComparison.OrdinalIgnoreCase)) values[key]=entry.Substring(split+1); } var block=new StringBuilder(); foreach(var value in values) block.Append(value.Key).Append('=').Append(value.Value).Append('\0'); block.Append('\0'); operationEnvironment=Marshal.StringToHGlobalUni(block.ToString()); @@ -120,16 +120,21 @@ static void NotificationRequest(Dictionary request,Dictionary>(File.ReadAllText(file)); if(pendingOperation.purpose=="check") { + var output=Json.Deserialize>(File.ReadAllText(file)); if(output.ContainsKey("quiet") && (bool)output["quiet"]) { delivered=null;receipt=null; } else {delivered=operationJournal.Present(output);receipt=(string)delivered["receipt"];} consumed=false; - } else { operationJournal.CompleteAcknowledgement(receipt);consumed=true; } + } else { + try { operationJournal.CompleteAcknowledgement(receipt,File.ReadAllText(file));consumed=true; } + catch(IOException) { verdict["operationState"]="reconciliation-required";verdict["reconciliationRequired"]=true; } + catch(UnauthorizedAccessException) { verdict["operationState"]="reconciliation-required";verdict["reconciliationRequired"]=true; } + } NativeOperationLifetime.Stop(pendingOperation.job,1500); scopes.Remove(pendingOperation); CloseHandle(pendingOperation.process.thread);CloseHandle(pendingOperation.process.process);CloseHandle(pendingOperation.job); pendingOperation=null; + if(verdict.ContainsKey("reconciliationRequired")) return; } if(action=="result") { verdict["operationState"]=pendingOperation!=null ? "pending" : operationJournal.NeedsReconciliation ? "reconciliation-required" : consumed ? "acknowledged" : delivered!=null ? "delivered" : "quiet"; diff --git a/bin/native-owner/NativeReceiptJournal.cs b/bin/native-owner/NativeReceiptJournal.cs index 9562e06057f..dddae6c7a42 100644 --- a/bin/native-owner/NativeReceiptJournal.cs +++ b/bin/native-owner/NativeReceiptJournal.cs @@ -83,11 +83,17 @@ public Dictionary BeginAcknowledgement(string receipt,string obse Append("ack-started",receipt,Copy(payload),evidence==null ? null : evidence.Record()); return Delivery(receipts[receipt]); } - public void CompleteAcknowledgement(string receipt) { + public void CompleteAcknowledgement(string receipt,string completion) { EnsureOpen(); Dictionary row; if(receipt==null || !receipts.TryGetValue(receipt,out row) || (string)row["generation"]!=generation || (string)row["event"]!="ack-started") throw new InvalidOperationException("No matching acknowledgement attempt"); - Append("acknowledged",receipt,Copy((Dictionary)row["payload"]),Evidence(row)); + Dictionary response; + try { response=json.Deserialize>(completion); } + catch(Exception error) { throw new IOException("Acknowledgement completion response is malformed; preserved",error); } + object acknowledged,responseEvidence;string evidence=Evidence(row) as string; + if(response==null || !response.TryGetValue("acknowledged",out acknowledged) || !(acknowledged is bool) || !(bool)acknowledged || !response.TryGetValue("ownerEvidence",out responseEvidence) || !(responseEvidence is string) || string.IsNullOrEmpty(evidence) || (string)responseEvidence!=evidence) throw new IOException("Acknowledgement completion evidence does not match the persisted target; preserved"); + if(!NativeAcknowledgementEvidence.Completed(lease,evidence)) throw new IOException("Acknowledgement effect lacks affirmative owner evidence; preserved"); + Append("acknowledged",receipt,Copy((Dictionary)row["payload"]),evidence); } static object Evidence(Dictionary row) { object value;return row.TryGetValue("targetEvidence",out value) ? value : null; diff --git a/bin/native-owner/ack.sh b/bin/native-owner/ack.sh index ebcdd928269..1b59ab6b954 100755 --- a/bin/native-owner/ack.sh +++ b/bin/native-owner/ack.sh @@ -13,4 +13,4 @@ fm_session_lock_owned_by_self "$FM_HOME/state" request="$LOG/notification-ack-request.json" evidence=$(jq -er '.ownerEvidence | select(type == "string" and length > 0)' "$request") printf '%s' "$evidence" | bash bin/native-owner/ack-evidence.sh acknowledge-token -printf '{"acknowledged":true}\n' > "$LOG/notification-ack.json" +jq -n --arg ownerEvidence "$evidence" '{acknowledged:true,ownerEvidence:$ownerEvidence}' > "$LOG/notification-ack.json" diff --git a/tests/fixtures/native-owner/AppHost.mjs b/tests/fixtures/native-owner/AppHost.mjs index 6dee31e3314..ed554034e67 100644 --- a/tests/fixtures/native-owner/AppHost.mjs +++ b/tests/fixtures/native-owner/AppHost.mjs @@ -28,7 +28,7 @@ async function native(action,extra={},signal) { socket.on('error',reject);socket.on('close',()=>{signal?.removeEventListener('abort',abort);if(!done)reject(Error('Native channel closed without a result'));}); }); if(!result.notificationAuthorized)throw Error('Native controller denied host adapter'); - evidence.native.push({action,state:result.operationState,startupExpired:result.startupExpired}); + evidence.native.push({action,state:result.operationState,startupExpired:result.startupExpired,operationExit:result.operationExit}); return result; } catch(error) {if(!['EBUSY','ENOENT'].includes(error.code)||Date.now()>until)throw error;await pause(40);} } @@ -116,15 +116,18 @@ try { if(process.env.FM_PROBE_API_DRY==='1') { const delivery=await operation('check'); const ack=await operation('ack',{receipt:delivery.notification.receipt,observed:delivery.notification.challenge}); - if(process.env.FM_PROBE_ACK_FAULT) { - if(ack.operationState!=='interrupted')throw Error('Fault injection did not interrupt the acknowledgement'); - evidence.ackFault=process.env.FM_PROBE_ACK_FAULT; - } else { + if(['partial','complete'].includes(process.env.FM_PROBE_ACK_FAULT)) { + if(ack.operationState!=='interrupted')throw Error('Fault injection did not interrupt the acknowledgement'); + evidence.ackFault=process.env.FM_PROBE_ACK_FAULT; + } else if(process.env.FM_PROBE_ACK_FAULT) { + if(ack.operationState!=='reconciliation-required')throw Error('Zero-exit acknowledgement without exact completion evidence was accepted'); + evidence.ackFault=process.env.FM_PROBE_ACK_FAULT; + } else { const replay=await native('ack',{receipt:delivery.notification.receipt,observed:delivery.notification.challenge}); if(ack.operationState!=='acknowledged'||replay.operationState!=='denied')throw Error('Native bridge preflight failed'); } evidence.passed=true;evidence.modelFree=true; - console.log(evidence.ackFault?'PASS: fixed acknowledgement interrupted at the requested durable boundary; model-free only.':'PASS: registered post-startup operations deliver and acknowledge using the native bridge; model-free only.'); + console.log(evidence.ackFault?'PASS: fixed acknowledgement fault preserved the reconciliation obligation; model-free only.':'PASS: registered post-startup operations deliver and acknowledge using the native bridge; model-free only.'); } else { evidence.primaryTurn=await turn(primary,'This is a bounded integration test in an empty disposable Firstmate home. Use only the supplied fm_notification tools; do not use shell, file, browser, or other tools. Call fm_notification_check once. Read the message, then call fm_notification_ack with its receipt and the observed challenge. After successful acknowledgement, repeat that same acknowledgement exactly once to test replay rejection. Report the three results and stop. Do not retry anything else.'); if(!acknowledged)throw Error('Primary did not acknowledge notification'); diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 5dcebd5a2be..a6f3aee81b8 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -72,13 +72,13 @@ const populated=path.join(area,'populated');fs.mkdirSync(path.join(populated,'st const blocked=start(populated);blocked.child.stdin.end();assert.notEqual((await bound(blocked.done,blocked,20000)).exit,0); assert.equal(fs.readFileSync(path.join(populated,'state/work.meta'),'utf8'),'preserve');assert.equal(fs.existsSync(path.join(populated,'owner-probe.json')),false); records.push('populated home refused without changing its records'); -for(const [name,relative] of [['orphan-status','state/orphan.status'],['interrupted-close','state/orphan.backlog-close']]){ +for(const [name,relative] of [['orphan-status','state/orphan.status'],['interrupted-close','state/orphan.backlog-close'],['residual-turn-end','state/orphan.turn-ended']]){ const residualHome=path.join(area,name),record=path.join(residualHome,relative),contents='preserve residual task state'; fs.mkdirSync(path.dirname(record),{recursive:true});fs.writeFileSync(record,contents); const refusedResidual=start(residualHome);refusedResidual.child.stdin.end();assert.notEqual((await bound(refusedResidual.done,refusedResidual,20000)).exit,0); assert.equal(fs.readFileSync(record,'utf8'),contents);assert.equal(fs.existsSync(path.join(residualHome,'owner-probe.json')),false); } -records.push('orphan status and interrupted-close records refused before lease acquisition and preserved'); +records.push('orphan status, interrupted-close, and turn-end records refused before lease acquisition and preserved'); const maskedHome=path.join(area,'bash-env-mask'),maskedStatus=path.join(maskedHome,'state/orphan.status'),mask=path.join(area,'bash-env-exit.sh'); fs.mkdirSync(path.dirname(maskedStatus),{recursive:true});fs.writeFileSync(maskedStatus,'preserve masked residual state');fs.writeFileSync(mask,'exit 0\n'); const masked=start(maskedHome,true,{...process.env,BASH_ENV:mask});masked.child.stdin.end();assert.notEqual((await bound(masked.done,masked,20000)).exit,0); diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 56326fddde5..08f392efbb8 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -100,7 +100,7 @@ static int EnvironmentTests() { if(process.ExitCode!=0)throw new InvalidOperationException("Environment test child failed"); } var leaked=Json.Deserialize(File.ReadAllText(result)); - if(File.Exists(injected)||leaked.Length!=5)throw new InvalidOperationException("Denied inherited environment reached the native host"); + if(File.Exists(injected)||leaked.Length!=4)throw new InvalidOperationException("Denied inherited environment reached the native host"); foreach(string key in leaked)if(!key.StartsWith("FM_PROBE_",StringComparison.Ordinal))throw new InvalidOperationException("Unexpected inherited environment reached the native host"); Console.WriteLine("PASS: inherited Windows environment denylist is case-insensitive"); string residual=Path.Combine(directory,"residual"),residualState=Path.Combine(residual,"state"),status=Path.Combine(residualState,"orphan.status"); @@ -140,6 +140,11 @@ static int EnvironmentTests() { refused=false;try{EmptyFleet(steering);using(var lease=new NativeHomeLease(steering)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(message)!=messageBody||File.Exists(Path.Combine(steering,"owner-probe.json"))||File.Exists(Path.Combine(steeringState,".lock")))throw new InvalidOperationException("Orphan steering work passed admission or caused lease or route activity"); Console.WriteLine("PASS: orphan steering work is preserved without lease or route activity"); + string turnEnded=Path.Combine(directory,"turn-ended"),turnEndedState=Path.Combine(turnEnded,"state"),turnEndedRecord=Path.Combine(turnEndedState,"orphan.turn-ended"); + Directory.CreateDirectory(turnEndedState);File.WriteAllText(turnEndedRecord,"preserve\n"); + refused=false;try{EmptyFleet(turnEnded);using(var lease=new NativeHomeLease(turnEnded)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(turnEndedRecord)!="preserve\n"||File.Exists(Path.Combine(turnEnded,"owner-probe.json"))||File.Exists(Path.Combine(turnEndedState,".lock")))throw new InvalidOperationException("Residual turn-end work passed admission or caused lease activity"); + Console.WriteLine("PASS: residual turn-end work is preserved without lease activity"); string supported=Path.Combine(directory,"supported-notification"),supportedState=Path.Combine(supported,"state"),supportedInbox=Path.Combine(supportedState,"inbox"),supportedNote=Path.Combine(supportedInbox,"note-id.note"),supportedQueue=Path.Combine(supportedState,".wake-queue"),supportedMarker=Path.Combine(supportedState,".watcher-down"),supportedBody="preserve notification\n"; Directory.CreateDirectory(supportedInbox);File.WriteAllText(supportedNote,supportedBody);File.WriteAllText(supportedQueue,"1\t1\tcheck\tinbox:note-id\tcaptain inbox note\n");File.WriteAllText(supportedMarker,"pending:handling:supported\n"); EmptyFleet(supported); @@ -206,6 +211,7 @@ static int Run(string configPath) { recoveredAcknowledgements=operationJournal.ReconcileCompletedAcknowledgements(); } var values = EnvironmentFor(pipeName, session, home, nonce); + values["FM_PROBE_EXE"]=OwnExe; if(config.ContainsKey("ownerExercise") && (bool)config["ownerExercise"]) { if(!config.ContainsKey("jqImage") || string.IsNullOrWhiteSpace((string)config["jqImage"])) throw new ArgumentException("Owner exercise requires an explicit local jq image"); values["FM_PROBE_JQ_IMAGE"]=(string)config["jqImage"]; @@ -220,7 +226,7 @@ static int Run(string configPath) { } if(config.ContainsKey("ackFault")) { string fault=(string)config["ackFault"]; - if(!values.ContainsKey("FM_PROBE_API_DRY") || (fault!="partial" && fault!="complete")) throw new ArgumentException("Fault injection is limited to the model-free fixture"); + if(!values.ContainsKey("FM_PROBE_API_DRY") || (fault!="partial" && fault!="complete" && fault!="zero-missing" && fault!="zero-malformed" && fault!="zero-mismatched" && fault!="zero-unproven")) throw new ArgumentException("Fault injection is limited to the model-free fixture"); values["FM_PROBE_ACK_FAULT"]=fault; } if(lease!=null) values["FM_HOME"]=lease.Home.Replace('\\','/'); diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index f2279760932..9d2d82c39d5 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -10,6 +10,7 @@ public static class ReceiptTests { [DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] static extern bool CreateHardLink(string link,string target,IntPtr security); [DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] [return: MarshalAs(UnmanagedType.I1)] static extern bool CreateSymbolicLink(string link,string target,uint flags); static readonly string A=new string('a',32),B=new string('b',32); + static readonly JavaScriptSerializer Json=new JavaScriptSerializer(); static int passed; static Dictionary Payload(string value) { return new Dictionary{{"challenge",value},{"message","pending notification"},{"seq","1"},{"generation","recovery"},{"note","note-id"}}; } static void Expect(bool value,string message) { if(!value) throw new Exception(message); } @@ -63,6 +64,15 @@ static void OwnerAcknowledge(NativeHomeLease lease,Dictionary pay if(process.ExitCode!=0)throw new IOException("Acknowledgement owner refused its captured target: "+error.Result+output.Result); } } + static string Completion(Dictionary payload) { + return Json.Serialize(new Dictionary{{"acknowledged",true},{"ownerEvidence",payload["ownerEvidence"]}}); + } + static void Acknowledge(NativeHomeLease lease,NativeReceiptJournal journal,Dictionary delivery,string observed) { + string receipt=(string)delivery["receipt"]; + journal.BeginAcknowledgement(receipt,observed,NativeAcknowledgementEvidence.Capture(lease,delivery)); + OwnerAcknowledge(lease,delivery); + journal.CompleteAcknowledgement(receipt,Completion(delivery)); + } static Dictionary ZeroPayload(NativeHomeLease lease) { string[] target=ZeroRecovery(lease,"present").Split('\t'); Expect(target.Length==2 && target[0]=="0","Real recovery owner did not produce a zero-row target"); @@ -72,11 +82,12 @@ static Dictionary ZeroPayload(NativeHomeLease lease) { } public static int Run() { Case("one writer and unobserved acknowledgement refusal",lease=>{ + Targets(lease); using(var journal=new NativeReceiptJournal(lease,A)) { Refuses(()=>{using(var other=new NativeReceiptJournal(lease,A)){}},"Concurrent writer accepted"); - var note=journal.Present(Payload("first"));string receipt=(string)note["receipt"]; + var note=journal.Present(OwnerPayload(lease,"first"));string receipt=(string)note["receipt"]; Refuses(()=>journal.BeginAcknowledgement(receipt,"wrong"),"Unobserved message accepted"); - journal.BeginAcknowledgement(receipt,"first");journal.CompleteAcknowledgement(receipt); + Acknowledge(lease,journal,note,"first"); Refuses(()=>journal.BeginAcknowledgement(receipt,"first"),"Consumed receipt accepted"); } }); @@ -101,12 +112,12 @@ public static int Run() { using(var journal=new NativeReceiptJournal(lease,B)) { Expect(journal.NeedsReconciliation,"Interrupted attempt lost"); Refuses(()=>journal.Present(Payload("second")),"New work replaced ambiguous attempt"); - Refuses(()=>journal.CompleteAcknowledgement(receipt),"New generation invented completion"); + Refuses(()=>journal.CompleteAcknowledgement(receipt,null),"New generation invented completion"); } }); Case("completed receipt remains consumed after restart",lease=>{ - string receipt; - using(var journal=new NativeReceiptJournal(lease,A)) {receipt=(string)journal.Present(Payload("first"))["receipt"];journal.BeginAcknowledgement(receipt,"first");journal.CompleteAcknowledgement(receipt);} + Targets(lease);string receipt; + using(var journal=new NativeReceiptJournal(lease,A)) {var delivery=journal.Present(OwnerPayload(lease,"first"));receipt=(string)delivery["receipt"];Acknowledge(lease,journal,delivery,"first");} using(var journal=new NativeReceiptJournal(lease,B)) { Expect(!journal.NeedsReconciliation,"Completed attempt became ambiguous"); Refuses(()=>journal.BeginAcknowledgement(receipt,"first"),"Completed predecessor replay accepted"); @@ -114,12 +125,29 @@ public static int Run() { } }); Case("multiple cycles retain independent consumed receipts",lease=>{ + Targets(lease); using(var journal=new NativeReceiptJournal(lease,A)) { - string first=(string)journal.Present(Payload("first"))["receipt"]; - journal.BeginAcknowledgement(first,"first");journal.CompleteAcknowledgement(first); - string second=(string)journal.Present(Payload("second"))["receipt"]; + var firstDelivery=journal.Present(OwnerPayload(lease,"first"));string first=(string)firstDelivery["receipt"]; + Acknowledge(lease,journal,firstDelivery,"first");Targets(lease);File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"pending:handling:recovery-second\n"); + var secondDelivery=journal.Present(OwnerPayload(lease,"second"));string second=(string)secondDelivery["receipt"]; Refuses(()=>journal.BeginAcknowledgement(first,"first"),"Earlier cycle replay accepted"); - journal.BeginAcknowledgement(second,"second");journal.CompleteAcknowledgement(second); + Acknowledge(lease,journal,secondDelivery,"second"); + } + }); + Case("zero-exit completion requires exact affirmative owner evidence",lease=>{ + Targets(lease); + using(var journal=new NativeReceiptJournal(lease,A)) { + var delivery=journal.Present(OwnerPayload(lease,"first"));string receipt=(string)delivery["receipt"],evidence=(string)delivery["ownerEvidence"]; + journal.BeginAcknowledgement(receipt,"first",NativeAcknowledgementEvidence.Capture(lease,delivery)); + string before=File.ReadAllText(Path.Combine(lease.Home,"owner-receipts.jsonl")); + Refuses(()=>journal.CompleteAcknowledgement(receipt,null),"Missing zero-exit response completed an acknowledgement"); + Refuses(()=>journal.CompleteAcknowledgement(receipt,"{broken"),"Malformed zero-exit response completed an acknowledgement"); + Refuses(()=>journal.CompleteAcknowledgement(receipt,Json.Serialize(new Dictionary{{"acknowledged",true},{"ownerEvidence","mismatch"}})),"Mismatched zero-exit response completed an acknowledgement"); + Refuses(()=>journal.CompleteAcknowledgement(receipt,Json.Serialize(new Dictionary{{"acknowledged",false},{"ownerEvidence",evidence}})),"Negative zero-exit response completed an acknowledgement"); + Refuses(()=>journal.CompleteAcknowledgement(receipt,Completion(delivery)),"Unperformed effect completed from response data alone"); + Expect(before==File.ReadAllText(Path.Combine(lease.Home,"owner-receipts.jsonl")),"Rejected completion response changed the journal"); + OwnerAcknowledge(lease,delivery);journal.CompleteAcknowledgement(receipt,Completion(delivery)); + Expect(!journal.NeedsReconciliation,"Affirmatively proven completion remained unresolved"); } }); Case("returned objects cannot change persisted target",lease=>{ @@ -231,7 +259,7 @@ public static int Run() { journal.BeginAcknowledgement(receipt,"concurrent",evidence); if(timing=="after-ack-started")ZeroRecovery(lease,"append"); Expect(File.ReadAllText(Path.Combine(lease.Home,"state",".watcher-down")).Contains(":"+generation+"\n"),"Concurrent wake changed the recovery generation"); - OwnerAcknowledge(lease,delivery);journal.CompleteAcknowledgement(receipt); + OwnerAcknowledge(lease,delivery);journal.CompleteAcknowledgement(receipt,Completion(delivery)); } string queue=File.ReadAllText(Queue(lease)); Expect(queue.Contains("later-notification")&&!queue.Contains("inbox:note-id"),"Acknowledgement did not retain only the later wake"); diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs index 03bf8ae4101..a82114c66db 100644 --- a/tests/fixtures/native-owner/Run-Cycle.mjs +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -11,7 +11,7 @@ const build=read(path.join(dir,'build.json')); const dry=process.argv.includes('--dry'); const startupQueued=process.argv.includes('--startup-queued'); const fault=process.argv.find(value=>value.startsWith('--fault='))?.slice(8); -if(fault&&(!dry||!['partial','complete'].includes(fault)))throw Error('Fault cases require --dry and partial or complete'); +if(fault&&(!dry||!['partial','complete','zero-missing','zero-malformed','zero-mismatched','zero-unproven'].includes(fault)))throw Error('Fault cases require --dry and a registered fault mode'); if(startupQueued&&!dry)throw Error('Queued-startup case requires model-free mode'); if(!dry&&process.env.FM_LIVE_NATIVE_CODEX!=='1')throw Error('Live model test requires FM_LIVE_NATIVE_CODEX=1'); if(!dry){const preflight=read(path.join(dir,'bridge-preflight.json'));if(!preflight.passed||preflight.binaryHash!==createHash('sha256').update(fs.readFileSync(build.binary)).digest('hex'))throw Error('Run model-free bridge preflight for this binary first');} @@ -51,8 +51,9 @@ if(fault) { const queue=path.join(spec.leaseHome,'state/.wake-queue'),before=fs.readFileSync(queue); const journal=()=>fs.readFileSync(path.join(spec.leaseHome,'owner-receipts.jsonl'),'utf8').trim().split('\n').map(JSON.parse); const historyBefore=journal(); - if(historyBefore.at(-1).event!=='ack-started'||!native.receiptNeedsReconciliation)throw Error('Interrupted intent was not preserved'); + if(historyBefore.at(-1).event!=='ack-started'||!native.receiptNeedsReconciliation)throw Error('Unresolved acknowledgement intent was not preserved'); if(!host.shutdown.reconciliationRequired||!run.stderr.includes('Acknowledgement completion is unconfirmed. Its records are preserved and require reconciliation:'))throw Error('Shutdown did not surface the preserved reconciliation requirement'); + if(fault.startsWith('zero-')&&!host.native.some(row=>row.action==='result'&&row.state==='reconciliation-required'&&row.operationExit===0))throw Error('The zero-exit completion refusal was not observed through the native controller'); if((before.length===0)!==(fault==='complete'))throw Error('Fault did not land at the requested mutation boundary'); const recovery=path.join(home,'recovery');fs.mkdirSync(recovery); const recoverySpec={home:recovery,leaseHome:spec.leaseHome,executable:build.binary,arguments:'sleep 100',timeoutSeconds:10,pipeAcl:'UserOnly'}; @@ -66,7 +67,7 @@ if(fault) { const history=journal(); if(history.length!==historyBefore.length+(fault==='complete'?1:0))throw Error('Recovery started or recorded an unexpected acknowledgement'); if(fault==='complete'&&(history.at(-1).event!=='recovered-acknowledged'||history.at(-1).ackGeneration!==native.probeGeneration||history.at(-1).generation!==recovered.probeGeneration))throw Error('Recovery generations are not bound'); - console.log(`PASS: actual ${fault} acknowledgement interruption; recovery ${fault==='complete'?'confirmed completed effects without replay':'preserved the unresolved partial mutation'}.`); + console.log(`PASS: actual ${fault} acknowledgement fault; recovery ${fault==='complete'?'confirmed completed effects without replay':'preserved the unresolved mutation'}.`); process.exit(0); } if(fs.existsSync(path.join(spec.leaseHome,'state/.wake-queue'))&&fs.readFileSync(path.join(spec.leaseHome,'state/.wake-queue'),'utf8').trim())throw Error('Queue was not acknowledged'); diff --git a/tests/fixtures/native-owner/notification-ack.sh b/tests/fixtures/native-owner/notification-ack.sh index 714116618b3..d549d410edf 100644 --- a/tests/fixtures/native-owner/notification-ack.sh +++ b/tests/fixtures/native-owner/notification-ack.sh @@ -16,9 +16,28 @@ request="$LOG/notification-ack-request.json" note=$(jq -r .note "$request") seq=$(jq -r .seq "$request") generation=$(jq -r .generation "$request") +evidence=$(jq -er '.ownerEvidence | select(type == "string" and length > 0)' "$request") case "$note" in ''|*[!A-Za-z0-9._-]*) exit 2 ;; esac case "$seq" in ''|*[!0-9]*) exit 2 ;; esac case "$generation" in ''|*[!A-Za-z0-9._-]*) exit 2 ;; esac +case "${FM_PROBE_ACK_FAULT:-}" in + zero-missing) + rm -f "$LOG/notification-ack.json" + exit 0 + ;; + zero-malformed) + printf '{broken\n' > "$LOG/notification-ack.json" + exit 0 + ;; + zero-mismatched) + jq -n --arg ownerEvidence mismatch '{acknowledged:true,ownerEvidence:$ownerEvidence}' > "$LOG/notification-ack.json" + exit 0 + ;; + zero-unproven) + jq -n --arg ownerEvidence "$evidence" '{acknowledged:true,ownerEvidence:$ownerEvidence}' > "$LOG/notification-ack.json" + exit 0 + ;; +esac bin/fm-inbox.sh drain --ack "$note" > "$LOG/cycle-inbox-ack.log" if [ "${FM_PROBE_ACK_FAULT:-}" = partial ]; then printf 'partial\n' > "$LOG/ack-fault-ready" @@ -33,4 +52,4 @@ if [ "${FM_PROBE_ACK_FAULT:-}" = complete ]; then sleep 30 exit 125 fi -printf '{"acknowledged":true,"queueEmpty":true}\n' > "$LOG/notification-ack.json" +jq -n --arg ownerEvidence "$evidence" '{acknowledged:true,queueEmpty:true,ownerEvidence:$ownerEvidence}' > "$LOG/notification-ack.json" diff --git a/tests/fm-native-owner-codex-live-e2e.test.sh b/tests/fm-native-owner-codex-live-e2e.test.sh index 4fdabcef13b..6ffba1bcdad 100644 --- a/tests/fm-native-owner-codex-live-e2e.test.sh +++ b/tests/fm-native-owner-codex-live-e2e.test.sh @@ -17,5 +17,9 @@ node "$fixture/Run-Cycle.mjs" --dry node "$fixture/Run-Cycle.mjs" --dry --startup-queued node "$fixture/Run-Cycle.mjs" --dry --fault=partial node "$fixture/Run-Cycle.mjs" --dry --fault=complete +node "$fixture/Run-Cycle.mjs" --dry --fault=zero-missing +node "$fixture/Run-Cycle.mjs" --dry --fault=zero-malformed +node "$fixture/Run-Cycle.mjs" --dry --fault=zero-mismatched +node "$fixture/Run-Cycle.mjs" --dry --fault=zero-unproven node "$fixture/Run-Cycle.mjs" node "$fixture/Verify-Cycle.mjs" diff --git a/tests/fm-turnend-guard.test.sh b/tests/fm-turnend-guard.test.sh index b4bbd1ad519..2718bc9f0a8 100755 --- a/tests/fm-turnend-guard.test.sh +++ b/tests/fm-turnend-guard.test.sh @@ -104,9 +104,23 @@ test_predicate_source_needs_supervision() { fm_supervision_unhealthy "$state" 300 || fail "registered source with no beacon must be unhealthy" [ "$FM_SUP_IN_FLIGHT" -eq 0 ] || fail "a process-event source must not count as a task" [ "$FM_SUP_SOURCES" -eq 1 ] || fail "expected one registered process-event source" + [ "$FM_SUP_REASON_COUNT" -eq 1 ] && [ "$FM_SUP_REASON_LABEL" = 'process-event source(s) registered' ] \ + || fail "the selected supervision reason did not identify the source" pass "fm_supervision_unhealthy: source-only home needs supervision" } +test_predicate_turn_end_needs_supervision() { + local state="$TMP_ROOT/pred-turn-end/state" + mkdir -p "$state" + printf 'preserve\n' > "$state/orphan.turn-ended" + fm_supervision_unhealthy "$state" 300 || fail "a residual turn-end notification did not keep supervision active" + [ "$FM_SUP_TURN_ENDS" -eq 1 ] || fail "expected one turn-end notification, got $FM_SUP_TURN_ENDS" + [ "$FM_SUP_REASON_COUNT" -eq 1 ] && [ "$FM_SUP_REASON_LABEL" = 'turn-end notification(s) pending' ] \ + || fail "the selected supervision reason did not identify the turn-end notification" + [ "$(cat "$state/orphan.turn-ended")" = preserve ] || fail "the supervision read changed the turn-end notification" + pass "fm_supervision_needed: a residual turn-end notification needs supervision" +} + # Register a custom check the way an operator does, through the real # bin/fm-check-register.sh, so these cases bind to the shipped registration # artifacts rather than to a hand-written imitation of them. @@ -2223,6 +2237,7 @@ test_predicate_healthy_fresh_beacon test_predicate_queue_pending_flag test_predicate_x_mode_needs_supervision test_predicate_source_needs_supervision +test_predicate_turn_end_needs_supervision test_predicate_registered_check_needs_supervision test_predicate_registered_check_survives_rebinding_drift test_predicate_unregistered_check_needs_supervision From 087163db76173eb3ffaa91d7d9b53e8c1844540b Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 09:25:32 +1200 Subject: [PATCH 21/61] no-mistakes(review): Clarify host authority and consolidate owner contracts --- bin/fm-backlog-transition-lib.sh | 14 ++++++++++++++ bin/fm-wake-lib.sh | 12 ++++++++++++ bin/native-owner/codex-host-runtime.mjs | 2 +- docs/native-windows-codex.md | 1 + docs/watcher-continuity.md | 3 +-- tests/fixtures/native-owner/AppHost.mjs | 15 +++++++++++++++ tests/fixtures/native-owner/NativeDriver.cs | 4 +++- tests/fixtures/native-owner/Run-Cycle.mjs | 4 ++-- 8 files changed, 49 insertions(+), 6 deletions(-) diff --git a/bin/fm-backlog-transition-lib.sh b/bin/fm-backlog-transition-lib.sh index 5c83bd50c5c..38d137dbcdf 100644 --- a/bin/fm-backlog-transition-lib.sh +++ b/bin/fm-backlog-transition-lib.sh @@ -55,6 +55,18 @@ # closes a row that reads as an open captain call. An answer that closes the row # first applies any supported retained artifact from the validated record, then # replay simply retires the record. +# +# EMPTY-FLEET ADMISSION. fm_backlog_markdown_empty accepts a +# readable, non-symlink markdown file only when its nonblank LF or CRLF lines are +# either `# Backlog` alone or the optional `# Backlog` title followed by the +# `## In flight`, `## Queued`, and `## Done` empty section skeleton. +# fm_backlog_empty_fleet_preflight additionally rejects +# task, close-recovery, inbox, and handoff work records, requires the configured +# backlog owner to resolve to markdown, and applies that semantic empty-file +# check when the backlog exists. Both functions write nothing, return 0 only for +# recognized empty state, and otherwise return non-zero with the refusal in +# FM_BACKLOG_EMPTY_ERROR. Callers must source fm-tasks-axi-lib.sh first so backend +# selection and configuration errors retain their owning contract. # Set by fm_backlog_transition_applies for a return-1 exemption. # shellcheck disable=SC2034 # Output global, read by the sourcing caller. @@ -72,6 +84,8 @@ FM_BACKLOG_ROW_HOLD_KIND= # retained_incomplete | answered | stale | noop. # shellcheck disable=SC2034 # Output global, read by the sourcing caller. FM_BACKLOG_CLOSE_REPLAY_RESULT= +# Set by the empty-markdown and empty-fleet admission helpers when they refuse. +# shellcheck disable=SC2034 # Output global, read by the sourcing caller. FM_BACKLOG_EMPTY_ERROR= # Bounded execution is fm-timeout-lib.sh's alone; source it rather than diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index c88c74b87e7..1761ac112e7 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1,5 +1,17 @@ #!/usr/bin/env bash # Shared durable wake queue and portable lock helpers. +# +# RECOVERY COMPLETION HISTORY. Before a current `acked:handling:` or +# `acked:downtime:` marker is replaced, this library retains that +# generation in `${marker}.ack-completions`. The file is newline-delimited: its +# first line is exactly `fm-wake-ack-completions-v1`, followed by at most 1,024 +# unique generation lines, each 1-128 ASCII characters from `[A-Za-z0-9._-]`. +# The complete file is bounded at 262,144 bytes and is never pruned +# automatically. fm_recovery_marker_completed accepts either the matching +# current acknowledged marker or a matching history row. A symlink, non-regular, +# unreadable, oversized, malformed, duplicate, or full history makes completion +# unresolved and prevents acknowledgement or marker replacement from erasing +# the current proof. FM_WAKE_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_WAKE_DEFAULT_ROOT="$(cd "$FM_WAKE_LIB_DIR/.." && pwd)" diff --git a/bin/native-owner/codex-host-runtime.mjs b/bin/native-owner/codex-host-runtime.mjs index db936c52789..48eb1e089a4 100644 --- a/bin/native-owner/codex-host-runtime.mjs +++ b/bin/native-owner/codex-host-runtime.mjs @@ -173,7 +173,7 @@ export async function runCodexHost(options={}) { {name:'fm_notification_check',description:'Read pending Firstmate notifications. Quiet means there is no new delivery.',inputSchema:{type:'object',properties:{},additionalProperties:false}}, {name:'fm_notification_ack',description:'Acknowledge an observed and handled delivery. Never acknowledge unresolved decisions or unperformed work.',inputSchema:{type:'object',properties:{receipt:{type:'string'},observed:{type:'string'}},required:['receipt','observed'],additionalProperties:false}}, ]; - const instructions='The native host already ran startup exactly once. Do not rerun startup or arm another supervisor. This experimental empty-fleet session supports only the two notification tools; do not claim to dispatch project work. The host continues notification checks after startup finishes. Use the supplied receipt and observed challenge only after handling the entire delivery. Unresolved work must remain pending. Startup digest follows:\n'+fs.readFileSync(path.join(runtime,'startup.log'),'utf8'); + const instructions='The native host already ran startup exactly once. Do not rerun startup or arm another supervisor. This experimental empty-fleet session exposes two privileged notification tools. Ordinary Codex tools remain confined to the read-only, network-disabled sandbox and have no native host authority; do not claim to dispatch project work. The host continues notification checks after startup finishes. Use the supplied receipt and observed challenge only after handling the entire delivery. Unresolved work must remain pending. Startup digest follows:\n'+fs.readFileSync(path.join(runtime,'startup.log'),'utf8'); const externalConfiguration=await verifyExternalToolConfiguration(request,mcpServerNames); const started=await request('thread/start',{cwd:root,sandbox:'read-only',approvalPolicy:'never',ephemeral:true,developerInstructions:instructions,dynamicTools}); if(started.sandbox?.type!=='readOnly'||started.sandbox.networkAccess!==false||started.approvalPolicy!=='never')throw Error('App-server returned an unexpected security policy'); diff --git a/docs/native-windows-codex.md b/docs/native-windows-codex.md index 23d4c01154d..0f36929f357 100644 --- a/docs/native-windows-codex.md +++ b/docs/native-windows-codex.md @@ -63,6 +63,7 @@ Only empty-fleet homes beneath the current user's Windows temporary directory ar Existing fleet metadata, projects, registrations, Relay configuration, process-event sources, non-temporary homes, and existing reparse-point ancestors are refused. The app-server thread is ephemeral, read-only, network-disabled, and approval-never; Apps, plugins, and configured MCP servers are disabled for this host and their effective catalogs are checked before readiness. Only controller-selected startup, notification check, and acknowledgement scripts receive registered native operation authority. +The two notification tools are the model-facing path to those registered host operations, not the complete Codex tool catalog; ordinary tools cannot acquire native authority from the inherited endpoint, claims, callback identity, or session job. Those operations and their descendants are owned by the native session: deferred startup may outlive the digest shell but `/quit` cancels it without terminating independently owned workers, and unfinished startup may run again after restart. Interrupted or ambiguous acknowledgements remain preserved for evidence-based reconciliation and are never replayed or rolled back automatically. diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index bc99b15acf5..0710c8628ce 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -60,8 +60,7 @@ The acknowledgement retires the marker only when no rows remain after sequence-b A concurrently appended wake has a higher sequence, remains queued, and keeps the episode pending for presentation. Consequently, an empty-queue downtime publication during handling can be retired by the outstanding acknowledgement without a dedicated recovery turn. An acknowledged episode does not freeze the generation, because the next downtime after it opens an episode of its own. -Before that later episode replaces an acknowledged marker, the wake owner retains the completed generation in the versioned newline-delimited `state/.watcher-down.ack-completions` owner record so restart reconciliation can still prove the earlier effect. -The record retains at most 1,024 unique generations without automatic pruning; an unreadable, malformed, or full record prevents marker replacement and preserves the current proof instead of permitting new work to erase it. +`bin/fm-wake-lib.sh`'s header owns the completion-history record used to prove an earlier acknowledged generation across later marker replacement, including its exact format, bounds, and refusal behavior. ## Per-actor acknowledgement diff --git a/tests/fixtures/native-owner/AppHost.mjs b/tests/fixtures/native-owner/AppHost.mjs index ed554034e67..736bc212297 100644 --- a/tests/fixtures/native-owner/AppHost.mjs +++ b/tests/fixtures/native-owner/AppHost.mjs @@ -107,6 +107,21 @@ try { let ready=false; for(let i=0;i<1200;i++){const state=await native('result');if(state.startupExpired){ready=true;break;}await pause(100);} if(!ready||!fs.existsSync(path.join(home,'owner-operation.complete')))throw Error('Startup did not finish and expire'); + const protectedFiles=[path.join(process.env.FM_HOME,'owner-receipts.jsonl'),path.join(process.env.FM_HOME,'state','.wake-queue')]; + const protectedSnapshot=()=>protectedFiles.map(file=>fs.existsSync(file)?fs.readFileSync(file).toString('base64'):null); + const beforeOrdinaryTool=protectedSnapshot(); + const ordinaryTool=await request('command/exec',{command:[process.env.FM_PROBE_EXE,'privilege-probe'],cwd:home,sandboxPolicy:{type:'readOnly',access:{type:'fullAccess'}},timeoutMs:10000}); + if(ordinaryTool.exitCode!==0)throw Error('Ordinary sandbox command did not execute: '+ordinaryTool.stderr); + const ordinaryLines=ordinaryTool.stdout.trim().split(/\r?\n/).filter(Boolean); + if(ordinaryLines.length!==2)throw Error('Ordinary sandbox command returned unexpected evidence'); + const ordinaryToken=JSON.parse(ordinaryLines[0]),ordinaryVerdict=JSON.parse(ordinaryLines[1]); + if(!Array.isArray(ordinaryToken.restricting)||ordinaryToken.restricting.length===0)throw Error('Command did not execute under the ordinary restricted sandbox token'); + if(ordinaryVerdict.association!=='associated'||ordinaryVerdict.hostClassification!=='unclassified-descendant')throw Error('Ordinary sandbox command did not reach the controller with copied claims as a session descendant'); + if(ordinaryVerdict.notificationAuthorized!==false||ordinaryVerdict.authorityGranted!==false)throw Error('Ordinary sandbox command acquired privileged host authority'); + await pause(250); + const primaryControl=await native('result'); + if(primaryControl.operationState!=='quiet'||JSON.stringify(protectedSnapshot())!==JSON.stringify(beforeOrdinaryTool))throw Error('Ordinary sandbox command caused a protected notification effect'); + evidence.ordinaryToolRefusal={exitCode:ordinaryTool.exitCode,association:ordinaryVerdict.association,hostClassification:ordinaryVerdict.hostClassification,restricted:true,protectedEffects:false,registeredPrimaryState:primaryControl.operationState};save(); const params={cwd:home,model:'gpt-5.6-terra',sandbox:'read-only',approvalPolicy:'never',ephemeral:true,dynamicTools:tools}; const externalConfiguration=await verifyExternalToolConfiguration(request,mcpServerNames); primary=(await request('thread/start',params)).thread.id;evidence.primary=primary; diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 08f392efbb8..fdfaf278dc9 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -33,13 +33,14 @@ static void ReleaseFixtureWhenScopesEnd(List scopes,string home) { string done=Path.Combine(home,"boundaries-done"); if(!File.Exists(done)) File.WriteAllText(done,"complete"); } - static int Client(string which) { + static int Client(string which, bool notificationProbe=false) { Console.WriteLine(Json.Serialize(TokenFacts())); var request = new Dictionary { {"session",Environment.GetEnvironmentVariable("FM_PROBE_SESSION")}, {"home",Environment.GetEnvironmentVariable("FM_PROBE_HOME")}, {"nonce",Environment.GetEnvironmentVariable("FM_PROBE_NONCE")}, {"case",which}, {"claimedRole","primary"} }; + if(notificationProbe) { request["kind"]="notification";request["action"]="check"; } if (which == "wrong-session") request["session"] = Guid.NewGuid().ToString("N"); if (which == "wrong-home") request["home"] = "C:\\not-the-test-home"; if (which == "wrong-capability") request["nonce"] = "copied-invalid-value"; @@ -344,6 +345,7 @@ public static int Main(string[] args) { if(args.Length==1 && args[0]=="receipt-tests") { ReceiptTests.Run();return TestOperationLifetime(); } if(args.Length==1 && args[0]=="environment-tests") return EnvironmentTests(); if(TryOwnerCommand(args,out ownerResult)) return ownerResult; + if(args.Length==1 && args[0]=="privilege-probe") return Client("ordinary-sandbox-command",true); if(args.Length>0 && args[0]=="client") return Client(args.Length>1 ? args[1] : "agent-tool"); if(args.Length==2 && args[0]=="sleep") { int ms=int.Parse(args[1]); if(ms<0 || ms>15000) throw new ArgumentException("Sleep must be bounded"); Thread.Sleep(ms); return 0; } if(args.Length==2 && args[0]=="operation-parent") { diff --git a/tests/fixtures/native-owner/Run-Cycle.mjs b/tests/fixtures/native-owner/Run-Cycle.mjs index a82114c66db..dbee3ef20a3 100644 --- a/tests/fixtures/native-owner/Run-Cycle.mjs +++ b/tests/fixtures/native-owner/Run-Cycle.mjs @@ -28,7 +28,7 @@ if(startupQueued){ startupNote=queued.stdout.trim().split(/\s+/)[1]; if(!startupNote)throw Error('Pre-startup note ID was not returned'); } -const spec={home,leaseHome,executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'UserOnly',apiDry:dry,jqImage:process.env.FM_NATIVE_TEST_JQ_IMAGE}; +const spec={home,leaseHome,executable:process.execPath,arguments:'"'+script+'"',registeredHarness:'codex-app-server',timeoutSeconds:280,ownerExercise:true,ownerOperation:true,pipeAcl:'LogonData',apiDry:dry,jqImage:process.env.FM_NATIVE_TEST_JQ_IMAGE}; if(startupQueued){spec.startupQueued=true;spec.startupNote=startupNote;} if(fault)spec.ackFault=fault; const file=path.join(home,'spec.json');fs.writeFileSync(file,JSON.stringify(spec,null,2)); @@ -40,7 +40,7 @@ if(run.status!==0)throw Error('Bounded app-server run failed; inspect evidence, const host=read(path.join(home,'app-host-evidence.json')),native=read(path.join(home,'result.json')); if(!host.shutdown?.stopped||!host.shutdown.operationsStopped||!host.shutdown.exited)throw Error('Host shutdown was not confirmed'); const starts=action=>host.native.filter(row=>row.action===action&&row.state==='pending').length; -if(!host.passed||starts('check')!==1||starts('ack')!==1||native.notificationConsumed!==!fault)throw Error('Notification cycle incomplete'); +if(!host.passed||host.ordinaryToolRefusal?.restricted!==true||host.ordinaryToolRefusal?.association!=='associated'||host.ordinaryToolRefusal?.hostClassification!=='unclassified-descendant'||host.ordinaryToolRefusal?.protectedEffects!==false||host.ordinaryToolRefusal?.registeredPrimaryState!=='quiet'||starts('check')!==1||starts('ack')!==1||native.notificationConsumed!==!fault)throw Error('Notification cycle or ordinary-tool authority refusal incomplete'); if(!host.native.filter(row=>(row.action==='check'||row.action==='ack')&&row.state==='pending').every(row=>row.startupExpired))throw Error('Startup scope still active'); if(startupQueued){ const delivered=read(path.join(home,'notification-check.json')); From 9b9cedf71afe3120fad5171dcee24ca4493e72d3 Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 09:57:27 +1200 Subject: [PATCH 22/61] no-mistakes(review): Harden native admission and restore supervision semantics --- bin/fm-guard.sh | 13 +- bin/fm-startup-network.sh | 8 +- bin/fm-supervision-lib.sh | 79 ++---- bin/fm-turnend-guard.sh | 16 +- bin/fm-wake-lib.sh | 235 ++++++++++++++++++ bin/native-owner/admit.sh | 13 +- tests/fixtures/native-owner/Launcher.mjs | 22 +- tests/fixtures/native-owner/NativeDriver.cs | 13 +- tests/fm-live-gate.test.sh | 36 +++ .../fm-native-owner-receipt-live-e2e.test.sh | 9 +- tests/fm-startup-network.test.sh | 18 +- tests/fm-turnend-guard.test.sh | 48 ++-- 12 files changed, 410 insertions(+), 100 deletions(-) diff --git a/bin/fm-guard.sh b/bin/fm-guard.sh index 5ae3b529ae5..ba9ee330465 100755 --- a/bin/fm-guard.sh +++ b/bin/fm-guard.sh @@ -170,8 +170,9 @@ fi # grace-based predicate (bin/fm-supervision-lib.sh), which owns what needs # supervision. fm_supervision_status "$STATE" "$GRACE" -reason_count=$FM_SUP_REASON_COUNT -reason_label=$FM_SUP_REASON_LABEL +in_flight=$FM_SUP_IN_FLIGHT +sources=$FM_SUP_SOURCES +checks=$FM_SUP_CHECKS needed=$FM_SUP_NEEDED beacon_desc=$FM_SUP_BEACON_DESC fm_watcher_supervision_verdict "$STATE" "$WATCH" "$GRACE" "$FM_HOME" "$FM_ROOT" @@ -233,8 +234,12 @@ if [ "$watcher_healthy" = false ]; then else watcher_cause=$(printf 'no watcher has a fresh beacon (last beat: %s, grace %ss)' "$beacon_desc" "$GRACE") fi - if [ "$reason_count" -gt 0 ]; then - printf '● %s %s, but %s.\n' "$reason_count" "$reason_label" "$watcher_cause" + if [ "$in_flight" -gt 0 ]; then + printf '● %s task(s) in flight, but %s.\n' "$in_flight" "$watcher_cause" + elif [ "$sources" -gt 0 ]; then + printf '● %s process-event source(s) registered, but %s.\n' "$sources" "$watcher_cause" + elif [ "$checks" -gt 0 ]; then + printf '● %s registered custom check(s), but %s.\n' "$checks" "$watcher_cause" else printf '● X-mode relay polling needs supervision, but %s.\n' "$watcher_cause" fi diff --git a/bin/fm-startup-network.sh b/bin/fm-startup-network.sh index 4f08661673c..791a4a7c5e3 100755 --- a/bin/fm-startup-network.sh +++ b/bin/fm-startup-network.sh @@ -358,9 +358,7 @@ EOF fi if [ "$claim_live" -eq 0 ]; then if report_requires_wake "$state"; then - fm_wake_append check startup-network \ - "check: startup-network: deferred startup network checks finished ($state); read them with $FM_ROOT/bin/fm-startup-network.sh report" \ - || true + fm_wake_append_startup_network "$state" || true fi fm_lock_release "$PUBLISH_LOCK" return 0 @@ -375,9 +373,7 @@ EOF return 0 fi if report_requires_wake "$state"; then - fm_wake_append check startup-network \ - "check: startup-network: deferred startup network checks finished ($state); read them with $FM_ROOT/bin/fm-startup-network.sh report" \ - || true + fm_wake_append_startup_network "$state" || true fi fm_lock_release "$PUBLISH_LOCK" } diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index 70536970bc9..8d8d34184f4 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -35,19 +35,10 @@ fm_sup_stat_mtime() { # sweep's call at execution time, and a home whose check # no longer validates needs the watcher precisely so the # sweep can report the rejection instead of going quiet. -# FM_SUP_CHECK_INPUTS count of state/*.check.sh inputs, including relay, -# registered, and unregistered checks -# FM_SUP_PENDING_REPLIES count of parent-owned pending-reply inputs -# FM_SUP_RECONCILE_REQUESTS count of secondmate reconcile-notify inputs -# FM_SUP_TURN_ENDS count of state/*.turn-ended watcher inputs -# FM_SUP_NEEDED true/false - in-flight work, a state check input, a +# FM_SUP_NEEDED true/false - in-flight work, an X-mode relay poll, a # registered event source (a source is a wait on an # external process, not a task, so it has no metadata), -# a pending reply, a reconcile-notify request, or a -# turn-ended notification -# FM_SUP_REASON_COUNT count for the highest-priority active reason, or zero -# for Relay polling -# FM_SUP_REASON_LABEL caller-neutral label for that selected reason +# or a registered custom check # FM_SUP_WATCHER_FRESH true/false - a watcher beacon within the grace window # FM_SUP_BEACON_DESC human-readable beacon age, for banners ("never" if absent) # FM_SUP_QUEUE_PENDING true/false - state/.wake-queue has unread records @@ -72,7 +63,7 @@ fm_sup_directory_entry_count() { } fm_supervision_status() { - local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} meta source check turn_end id beat m age + local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} meta source check id beat m age FM_SUP_IN_FLIGHT=0 FM_SUP_NEEDED=false FM_SUP_WATCHER_FRESH=false @@ -89,12 +80,8 @@ fm_supervision_status() { FM_SUP_SOURCES=$((FM_SUP_SOURCES + 1)) done FM_SUP_CHECKS=0 - FM_SUP_CHECK_INPUTS=0 for check in "$state"/*.check.sh; do - if [ ! -e "$check" ] && [ ! -L "$check" ]; then - continue - fi - FM_SUP_CHECK_INPUTS=$((FM_SUP_CHECK_INPUTS + 1)) + [ -e "$check" ] || continue id=${check##*/} id=${id%.check.sh} if [ "$id" = x-watch ]; then @@ -103,47 +90,12 @@ fm_supervision_status() { [ -e "$state/$id.check-trust" ] || continue FM_SUP_CHECKS=$((FM_SUP_CHECKS + 1)) done - FM_SUP_PENDING_REPLIES=$(fm_sup_directory_entry_count "$state/pending-replies") - FM_SUP_RECONCILE_REQUESTS=$(fm_sup_directory_entry_count "$state/reconcile-notify") - FM_SUP_TURN_ENDS=0 - for turn_end in "$state"/*.turn-ended; do - if [ -e "$turn_end" ] || [ -L "$turn_end" ]; then - FM_SUP_TURN_ENDS=$((FM_SUP_TURN_ENDS + 1)) - fi - done if [ "$FM_SUP_IN_FLIGHT" -gt 0 ] \ - || [ "$FM_SUP_TURN_ENDS" -gt 0 ] \ + || [ -f "$state/x-watch.check.sh" ] \ || [ "$FM_SUP_SOURCES" -gt 0 ] \ - || [ "$FM_SUP_CHECK_INPUTS" -gt 0 ] \ - || [ "$FM_SUP_PENDING_REPLIES" -gt 0 ] \ - || [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then + || [ "$FM_SUP_CHECKS" -gt 0 ]; then FM_SUP_NEEDED=true fi - if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then - FM_SUP_REASON_COUNT=$FM_SUP_IN_FLIGHT - FM_SUP_REASON_LABEL='task(s) in flight' - elif [ "$FM_SUP_TURN_ENDS" -gt 0 ]; then - FM_SUP_REASON_COUNT=$FM_SUP_TURN_ENDS - FM_SUP_REASON_LABEL='turn-end notification(s) pending' - elif [ "$FM_SUP_SOURCES" -gt 0 ]; then - FM_SUP_REASON_COUNT=$FM_SUP_SOURCES - FM_SUP_REASON_LABEL='process-event source(s) registered' - elif [ "$FM_SUP_CHECKS" -gt 0 ]; then - FM_SUP_REASON_COUNT=$FM_SUP_CHECKS - FM_SUP_REASON_LABEL='registered custom check(s)' - elif [ "$FM_SUP_CHECK_INPUTS" -gt 0 ]; then - FM_SUP_REASON_COUNT=$FM_SUP_CHECK_INPUTS - FM_SUP_REASON_LABEL='state check input(s)' - elif [ "$FM_SUP_PENDING_REPLIES" -gt 0 ]; then - FM_SUP_REASON_COUNT=$FM_SUP_PENDING_REPLIES - FM_SUP_REASON_LABEL='pending secondmate reply record(s)' - elif [ "$FM_SUP_RECONCILE_REQUESTS" -gt 0 ]; then - FM_SUP_REASON_COUNT=$FM_SUP_RECONCILE_REQUESTS - FM_SUP_REASON_LABEL='secondmate reconcile request(s)' - else - FM_SUP_REASON_COUNT=0 - FM_SUP_REASON_LABEL='X-mode relay polling' - fi beat="$state/.last-watcher-beat" if [ -e "$beat" ]; then @@ -163,6 +115,25 @@ fm_supervision_status() { return 0 } +FM_SUP_RESIDUAL_ERROR= +fm_supervision_residual_inputs_absent() { # + local state=$1 record count + FM_SUP_RESIDUAL_ERROR= + for record in "$state"/*.check.sh "$state"/*.check-trust "$state"/*.turn-ended; do + if [ -e "$record" ] || [ -L "$record" ]; then + FM_SUP_RESIDUAL_ERROR="supervision input is present at $record" + return 1 + fi + done + for record in "$state/pending-replies" "$state/reconcile-notify"; do + count=$(fm_sup_directory_entry_count "$record") + if [ "$count" -gt 0 ]; then + FM_SUP_RESIDUAL_ERROR="supervision input is present under $record" + return 1 + fi + done +} + # fm_supervision_needed [grace-seconds] # Exit 0 (true) exactly when the home needs a watcher. fm_supervision_needed() { diff --git a/bin/fm-turnend-guard.sh b/bin/fm-turnend-guard.sh index ca52b16af8c..ffceafaee51 100755 --- a/bin/fm-turnend-guard.sh +++ b/bin/fm-turnend-guard.sh @@ -228,8 +228,12 @@ block_stop() { { printf '●%s\n' "$rule" printf '● TURN WOULD END BLIND - SUPERVISION IS OFF\n' - if [ "$FM_SUP_REASON_COUNT" -gt 0 ]; then - printf '● %s %s, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_REASON_COUNT" "$FM_SUP_REASON_LABEL" "$FM_SUP_BEACON_DESC" + if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then + printf '● %s task(s) in flight, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_IN_FLIGHT" "$FM_SUP_BEACON_DESC" + elif [ "$FM_SUP_SOURCES" -gt 0 ]; then + printf '● %s process-event source(s) registered, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_SOURCES" "$FM_SUP_BEACON_DESC" + elif [ "$FM_SUP_CHECKS" -gt 0 ]; then + printf '● %s registered custom check(s), but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_CHECKS" "$FM_SUP_BEACON_DESC" else printf '● X-mode relay polling needs supervision, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_BEACON_DESC" fi @@ -486,8 +490,12 @@ budget_account_current_epoch block || block_stop terminal_fail_open terminal_status=$? if [ "$terminal_status" -eq 0 ]; then - if [ "$FM_SUP_REASON_COUNT" -gt 0 ]; then - NEED_DESC="$FM_SUP_REASON_COUNT $FM_SUP_REASON_LABEL" + if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then + NEED_DESC="$FM_SUP_IN_FLIGHT task(s) in flight" + elif [ "$FM_SUP_SOURCES" -gt 0 ]; then + NEED_DESC="$FM_SUP_SOURCES process-event source(s) registered" + elif [ "$FM_SUP_CHECKS" -gt 0 ]; then + NEED_DESC="$FM_SUP_CHECKS registered custom check(s)" else NEED_DESC="X-mode relay polling active" fi diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 1761ac112e7..c4aa46d7d0c 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -12,6 +12,12 @@ # unreadable, oversized, malformed, duplicate, or full history makes completion # unresolved and prevents acknowledgement or marker replacement from erasing # the current proof. +# +# NATIVE EMPTY-FLEET ADMISSION. fm_wake_native_empty_fleet_preflight accepts +# only bounded producer rows for top-level inbox notes and deferred startup +# completion, plus valid main-presentation and recovery evidence. It is +# read-only, rejects branch grants and unknown or malformed rows, and reports +# refusals in FM_WAKE_NATIVE_ADMISSION_ERROR. FM_WAKE_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_WAKE_DEFAULT_ROOT="$(cd "$FM_WAKE_LIB_DIR/.." && pwd)" @@ -1918,6 +1924,19 @@ fm_wake_append() { return "$status" } +fm_wake_startup_network_payload() { # + local state=$1 + case "$state" in done|failed|timeout) ;; *) return 1 ;; esac + printf 'check: startup-network: deferred startup network checks finished (%s); read them with %s/bin/fm-startup-network.sh report\n' \ + "$state" "$FM_ROOT" +} + +fm_wake_append_startup_network() { # + local payload + payload=$(fm_wake_startup_network_payload "$1") || return 1 + fm_wake_append check startup-network "$payload" +} + # fm_wake_append_locked # Locked core of fm_wake_append: appends the wake row under an already-held # FM_WAKE_QUEUE_LOCK. Callers that must commit another durable record atomically @@ -2103,6 +2122,222 @@ fm_wake_grant_rows_valid() { # [ -s "$1" ] && awk 'BEGIN { ok=1 } !/^[0-9]+$/ || seen[$0]++ { ok=0 } END { exit !ok }' "$1" } +FM_WAKE_NATIVE_ADMISSION_ERROR= +fm_wake_native_empty_fleet_preflight() { # + local state=$1 queue marker seq_file main_rows history record size counter + local epoch seq kind key payload extra id note handled expected row_state status + FM_WAKE_NATIVE_ADMISSION_ERROR= + queue="$state/.wake-queue" + marker="$state/.watcher-down" + seq_file="$state/.wake-queue.seq" + main_rows="$state/.main-eligible-rows" + history="$marker.ack-completions" + if [ ! -d "$state" ] || [ -L "$state" ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="wake state directory is unavailable at $state" + return 1 + fi + for record in "$state/.branch-eligible-rows" "$state/.branch-eligible-owner"; do + if [ -e "$record" ] || [ -L "$record" ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="unsupported branch wake state is present at $record" + return 1 + fi + done + if [ -e "$seq_file" ] || [ -L "$seq_file" ]; then + if [ ! -f "$seq_file" ] || [ ! -r "$seq_file" ] || [ -L "$seq_file" ] \ + || ! awk 'NR != 1 || !/^[0-9]+$/ { bad=1 } END { exit bad || NR != 1 }' "$seq_file"; then + FM_WAKE_NATIVE_ADMISSION_ERROR="wake sequence state is unrecognized at $seq_file" + return 1 + fi + counter=$(cat "$seq_file") + if [ "${#counter}" -gt 16 ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="wake sequence state is out of bounds at $seq_file" + return 1 + fi + else + counter= + fi + if [ -e "$main_rows" ] || [ -L "$main_rows" ]; then + if [ -L "$main_rows" ] || [ -z "$counter" ] || ! fm_wake_grant_rows_valid "$main_rows" \ + || ! awk -v ceiling="$counter" 'length($0) > 16 || $0 > ceiling { bad=1 } END { exit bad }' "$main_rows"; then + FM_WAKE_NATIVE_ADMISSION_ERROR="native wake presentation state is unrecognized at $main_rows" + return 1 + fi + fi + if [ -e "$queue" ] || [ -L "$queue" ]; then + if [ ! -f "$queue" ] || [ ! -r "$queue" ] || [ -L "$queue" ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue is not a readable regular file at $queue" + return 1 + fi + size=$(wc -c < "$queue" 2>/dev/null | tr -d '[:space:]') || size= + case "$size" in ''|*[!0-9]*) size=8388609 ;; esac + if [ "$size" -gt 8388608 ] || ! LC_ALL=C awk -F '\t' ' + NF != 5 || $1 !~ /^[0-9]+$/ || $2 !~ /^[0-9]+$/ || $2 == 0 || length($2) > 16 || + $3 !~ /^(signal|stale|check|heartbeat)$/ || length($4) == 0 || length($5) == 0 || + seen[$2]++ || (previous && $2 <= previous) { bad=1 } + { previous=$2; count++ } + END { exit bad || count > 100000 } + ' "$queue"; then + FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue contains malformed or oversized records at $queue" + return 1 + fi + fi + if [ -s "$queue" ] && [ -z "$counter" ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue has no producer sequence state at $seq_file" + return 1 + fi + if [ -s "$queue" ]; then + while IFS=$'\t' read -r epoch seq kind key payload extra || [ -n "$epoch$seq$kind$key$payload$extra" ]; do + [ -n "$epoch$seq$kind$key$payload$extra" ] || continue + [ "$seq" -le "$counter" ] || { + FM_WAKE_NATIVE_ADMISSION_ERROR="wake row $seq exceeds its producer sequence at $seq_file" + return 1 + } + case "$key" in + inbox:*) + id=${key#inbox:} + case "$id" in ''|*[!A-Za-z0-9_-]*) + FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue contains an invalid inbox identity" + return 1 + ;; + esac + case "$payload" in "check: captain inbox note $id - "?*) ;; *) + FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue contains an unrecognized inbox record" + return 1 + ;; + esac + [ "$kind" = check ] || { + FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue contains an unsupported inbox wake kind" + return 1 + } + note="$state/inbox/$id.note" + handled="$state/inbox/handled/$id.note" + if { [ ! -f "$note" ] || [ -L "$note" ]; } && { [ ! -f "$handled" ] || [ -L "$handled" ]; }; then + FM_WAKE_NATIVE_ADMISSION_ERROR="inbox wake has no safe note record for $id" + return 1 + fi + if { [ -e "$note" ] || [ -L "$note" ]; } && { [ -e "$handled" ] || [ -L "$handled" ]; }; then + FM_WAKE_NATIVE_ADMISSION_ERROR="inbox wake has ambiguous note records for $id" + return 1 + fi + ;; + startup-network) + [ "$kind" = check ] || { + FM_WAKE_NATIVE_ADMISSION_ERROR="startup completion used an unsupported wake kind" + return 1 + } + row_state= + for status in done failed timeout; do + expected=$(fm_wake_startup_network_payload "$status") || return 1 + [ "$payload" != "$expected" ] || row_state=$status + done + if [ -z "$row_state" ] || [ ! -f "$state/.startup-network.status" ] \ + || [ ! -r "$state/.startup-network.status" ] || [ -L "$state/.startup-network.status" ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="startup completion wake lacks recognized owner state" + return 1 + fi + if ! awk -F= -v expected="$row_state" \ + '$1 == "state" && $2 == expected { count++ } END { exit count != 1 }' \ + "$state/.startup-network.status"; then + FM_WAKE_NATIVE_ADMISSION_ERROR="startup completion owner state is unrecognized" + return 1 + fi + ;; + *) + FM_WAKE_NATIVE_ADMISSION_ERROR="unsupported work-bearing wake is queued at $queue" + return 1 + ;; + esac + done < "$queue" + fi + if [ -d "$state/inbox" ] && [ ! -L "$state/inbox" ]; then + for record in "$state/inbox"/* "$state/inbox"/.[!.]* "$state/inbox"/..?*; do + [ -e "$record" ] || [ -L "$record" ] || continue + if [ "$record" = "$state/inbox/handled" ]; then + [ -d "$record" ] && [ ! -L "$record" ] || { + FM_WAKE_NATIVE_ADMISSION_ERROR="inbox handled state is unsafe at $record" + return 1 + } + continue + fi + case "$record" in "$state/inbox"/*.note) ;; *) + FM_WAKE_NATIVE_ADMISSION_ERROR="unrecognized inbox state is present at $record" + return 1 + ;; + esac + [ -f "$record" ] && [ ! -L "$record" ] || { + FM_WAKE_NATIVE_ADMISSION_ERROR="inbox note is unsafe at $record" + return 1 + } + id=${record##*/} + id=${id%.note} + case "$id" in ''|*[!A-Za-z0-9_-]*) + FM_WAKE_NATIVE_ADMISSION_ERROR="inbox note has an invalid identity at $record" + return 1 + ;; + esac + if ! awk -F '\t' -v key="inbox:$id" '$3 == "check" && $4 == key { found=1 } END { exit !found }' "$queue"; then + FM_WAKE_NATIVE_ADMISSION_ERROR="pending inbox note has no owner wake for $id" + return 1 + fi + done + if [ -d "$state/inbox/handled" ] && [ ! -L "$state/inbox/handled" ]; then + for record in "$state/inbox/handled"/* "$state/inbox/handled"/.[!.]* "$state/inbox/handled"/..?*; do + [ -e "$record" ] || [ -L "$record" ] || continue + case "$record" in "$state/inbox/handled"/*.note) ;; *) + FM_WAKE_NATIVE_ADMISSION_ERROR="unrecognized handled inbox state is present at $record" + return 1 + ;; + esac + [ -f "$record" ] && [ ! -L "$record" ] || { + FM_WAKE_NATIVE_ADMISSION_ERROR="handled inbox note is unsafe at $record" + return 1 + } + id=${record##*/} + id=${id%.note} + case "$id" in ''|*[!A-Za-z0-9_-]*) + FM_WAKE_NATIVE_ADMISSION_ERROR="handled inbox note has an invalid identity at $record" + return 1 + ;; + esac + done + fi + elif [ -e "$state/inbox" ] || [ -L "$state/inbox" ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="inbox state is unsafe at $state/inbox" + return 1 + fi + if [ -e "$marker" ] || [ -L "$marker" ]; then + if ! fm_recovery_marker_read "$marker"; then + FM_WAKE_NATIVE_ADMISSION_ERROR="wake recovery state is unrecognized at $marker" + return 1 + fi + if [ -s "$queue" ]; then + case "$FM_RECOVERY_MARKER_TOKEN" in pending:*|announced:*) ;; *) + FM_WAKE_NATIVE_ADMISSION_ERROR="queued wakes lack a pending recovery generation" + return 1 + ;; + esac + fi + elif [ -s "$queue" ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="queued wakes have no recovery generation at $marker" + return 1 + fi + if [ -e "$history" ] || [ -L "$history" ]; then + [ -e "$marker" ] && [ ! -L "$marker" ] || { + FM_WAKE_NATIVE_ADMISSION_ERROR="wake completion history has no current recovery marker" + return 1 + } + if _fm_recovery_completion_has_locked "$marker" fm-native-admission-probe; then + : + else + status=$? + if [ "$status" -ne 1 ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="wake completion history is unrecognized at $history" + return 1 + fi + fi + fi +} + # 0 when holds the supported record, names a live process whose # identity still matches what was recorded, and matches any expected pid and # generation the caller pins. An unreadable, malformed, or superseded record is diff --git a/bin/native-owner/admit.sh b/bin/native-owner/admit.sh index 35614aa954e..c05036b2fc2 100644 --- a/bin/native-owner/admit.sh +++ b/bin/native-owner/admit.sh @@ -13,8 +13,15 @@ if ! fm_backlog_empty_fleet_preflight "$FM_HOME/state" "$FM_HOME/data"; then printf '%s\n' "${FM_BACKLOG_EMPTY_ERROR:-the home contains work-bearing records}" >&2 exit 2 fi -fm_supervision_status "$FM_HOME/state" -if [ "$FM_SUP_NEEDED" = true ]; then - printf 'the home contains registered work requiring supervision\n' >&2 +if ! fm_supervision_residual_inputs_absent "$FM_HOME/state"; then + printf '%s\n' "${FM_SUP_RESIDUAL_ERROR:-the home contains residual supervision work}" >&2 exit 2 fi +if [ -e "$FM_HOME/state" ] || [ -L "$FM_HOME/state" ]; then + export FM_STATE_OVERRIDE="$FM_HOME/state" + . bin/fm-wake-lib.sh + if ! fm_wake_native_empty_fleet_preflight "$FM_HOME/state"; then + printf '%s\n' "${FM_WAKE_NATIVE_ADMISSION_ERROR:-the home contains unsupported wake state}" >&2 + exit 2 + fi +fi diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index a6f3aee81b8..174019400fe 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -14,7 +14,7 @@ const sleep=ms=>new Promise(resolve=>setTimeout(resolve,ms)); function command(exe,args){const result=spawnSync(exe,args,{encoding:'utf8',timeout:120000});if(result.status!==0)throw Error(result.stderr||result.stdout);return result;} command('git',['-c','core.symlinks=true','clone','--quiet','--no-local','--single-branch',repo,code]); fs.cpSync(path.join(repo,'bin/native-owner'),path.join(code,'bin/native-owner'),{recursive:true}); -for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); +for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh','fm-backlog-transition-lib.sh','fm-supervision-lib.sh','fm-wake-lib.sh','fm-startup-network.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); const launcher=path.join(code,'bin/fm-native-codex.ps1'); command('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly']); const removedAlias=spawnSync('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly','-Home',path.join(area,'alias')],{encoding:'utf8',timeout:120000}); @@ -79,6 +79,16 @@ for(const [name,relative] of [['orphan-status','state/orphan.status'],['interrup assert.equal(fs.readFileSync(record,'utf8'),contents);assert.equal(fs.existsSync(path.join(residualHome,'owner-probe.json')),false); } records.push('orphan status, interrupted-close, and turn-end records refused before lease acquisition and preserved'); +const queuedHome=path.join(area,'supported-queued-restart'),queuedNote=enqueue(queuedHome,'Preserve this supported notification across a native restart.'); +const queuedBody=fs.readFileSync(path.join(queuedHome,'state/inbox',queuedNote+'.note'),'utf8'); +const queuedSession=start(queuedHome);const queuedReady=await ready(queuedSession);queuedSession.child.stdin.end(); +assert.equal((await bound(queuedSession.done,queuedSession,20000)).exit,0); +assert.equal(fs.readFileSync(path.join(queuedHome,'state/inbox',queuedNote+'.note'),'utf8'),queuedBody); +assert(fs.readFileSync(path.join(queuedHome,'state/.wake-queue'),'utf8').includes('inbox:'+queuedNote)); +const queuedRestart=start(queuedHome);await ready(queuedRestart,queuedReady.owner.generation);queuedRestart.child.stdin.end(); +assert.equal((await bound(queuedRestart.done,queuedRestart,20000)).exit,0); +assert.equal(fs.readFileSync(path.join(queuedHome,'state/inbox',queuedNote+'.note'),'utf8'),queuedBody); +records.push('producer-created inbox and native handling recovery remain admissible across restart'); const maskedHome=path.join(area,'bash-env-mask'),maskedStatus=path.join(maskedHome,'state/orphan.status'),mask=path.join(area,'bash-env-exit.sh'); fs.mkdirSync(path.dirname(maskedStatus),{recursive:true});fs.writeFileSync(maskedStatus,'preserve masked residual state');fs.writeFileSync(mask,'exit 0\n'); const masked=start(maskedHome,true,{...process.env,BASH_ENV:mask});masked.child.stdin.end();assert.notEqual((await bound(masked.done,masked,20000)).exit,0); @@ -108,6 +118,16 @@ for(const [name,relative] of [['relay-config','config/x-mode.env'],['relay-watch assert.equal(fs.readFileSync(record,'utf8'),contents);assert.equal(fs.existsSync(path.join(relayHome,'owner-probe.json')),false); } records.push('generated Relay state refused before lease acquisition and preserved'); +for(const [name,row] of [ + ['unsupported-wake','1\t1\tcheck\torphan-work\tcheck: unsupported residual work\n'], + ['malformed-wake','malformed wake row\n'], +]){ + const wakeHome=path.join(area,name),wakeState=path.join(wakeHome,'state'),queue=path.join(wakeState,'.wake-queue'),marker=path.join(wakeState,'.watcher-down'); + fs.mkdirSync(wakeState,{recursive:true});fs.writeFileSync(path.join(wakeState,'.wake-queue.seq'),'1\n');fs.writeFileSync(queue,row);fs.writeFileSync(marker,'pending:downtime:preserve\n'); + const wakeSession=start(wakeHome);wakeSession.child.stdin.end();assert.notEqual((await bound(wakeSession.done,wakeSession,20000)).exit,0); + assert.equal(fs.readFileSync(queue,'utf8'),row);assert.equal(fs.readFileSync(marker,'utf8'),'pending:downtime:preserve\n');assert.equal(fs.existsSync(path.join(wakeHome,'owner-probe.json')),false); +} +records.push('unsupported and malformed wake records refused unchanged before lease acquisition'); const outside=path.join(repo,'data/native-launcher',path.basename(area)+'-outside'); assert.equal(fs.existsSync(outside),false); const external=start(outside);external.child.stdin.end();assert.notEqual((await bound(external.done,external,20000)).exit,0);assert.equal(fs.existsSync(outside),false); diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index fdfaf278dc9..32201f794bc 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -147,10 +147,21 @@ static int EnvironmentTests() { if(!refused||File.ReadAllText(turnEndedRecord)!="preserve\n"||File.Exists(Path.Combine(turnEnded,"owner-probe.json"))||File.Exists(Path.Combine(turnEndedState,".lock")))throw new InvalidOperationException("Residual turn-end work passed admission or caused lease activity"); Console.WriteLine("PASS: residual turn-end work is preserved without lease activity"); string supported=Path.Combine(directory,"supported-notification"),supportedState=Path.Combine(supported,"state"),supportedInbox=Path.Combine(supportedState,"inbox"),supportedNote=Path.Combine(supportedInbox,"note-id.note"),supportedQueue=Path.Combine(supportedState,".wake-queue"),supportedMarker=Path.Combine(supportedState,".watcher-down"),supportedBody="preserve notification\n"; - Directory.CreateDirectory(supportedInbox);File.WriteAllText(supportedNote,supportedBody);File.WriteAllText(supportedQueue,"1\t1\tcheck\tinbox:note-id\tcaptain inbox note\n");File.WriteAllText(supportedMarker,"pending:handling:supported\n"); + Directory.CreateDirectory(supportedInbox);File.WriteAllText(supportedNote,supportedBody);File.WriteAllText(Path.Combine(supportedState,".wake-queue.seq"),"1\n");File.WriteAllText(supportedQueue,"1\t1\tcheck\tinbox:note-id\tcheck: captain inbox note note-id - native admission test\n");File.WriteAllText(supportedMarker,"pending:downtime:supported\n"); + EmptyFleet(supported); + File.WriteAllText(Path.Combine(supportedState,".main-eligible-rows"),"1\n");File.WriteAllText(supportedMarker,"pending:handling:supported\n"); EmptyFleet(supported); if(File.ReadAllText(supportedNote)!=supportedBody||!File.ReadAllText(supportedQueue).Contains("inbox:note-id")||File.ReadAllText(supportedMarker)!="pending:handling:supported\n"||File.Exists(Path.Combine(supported,"owner-probe.json")))throw new InvalidOperationException("Supported top-level notification state was changed or leased during admission"); Console.WriteLine("PASS: supported top-level notification state remains admissible and unchanged"); + foreach(string shape in new [] {"unsupported","malformed"}) { + string wakeHome=Path.Combine(directory,shape+"-wake"),wakeState=Path.Combine(wakeHome,"state"),wakeQueue=Path.Combine(wakeState,".wake-queue"),wakeMarker=Path.Combine(wakeState,".watcher-down"); + Directory.CreateDirectory(wakeState);File.WriteAllText(Path.Combine(wakeState,".wake-queue.seq"),"1\n");File.WriteAllText(wakeMarker,"pending:downtime:preserve\n"); + File.WriteAllText(wakeQueue,shape=="unsupported" ? "1\t1\tcheck\torphan-work\tcheck: unsupported residual work\n" : "malformed wake row\n"); + string queueBefore=File.ReadAllText(wakeQueue),markerBefore=File.ReadAllText(wakeMarker); + refused=false;try{EmptyFleet(wakeHome);using(var lease=new NativeHomeLease(wakeHome)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(wakeQueue)!=queueBefore||File.ReadAllText(wakeMarker)!=markerBefore||File.Exists(Path.Combine(wakeHome,"owner-probe.json")))throw new InvalidOperationException("Unsupported wake state passed admission, changed, or acquired a lease"); + } + Console.WriteLine("PASS: unsupported and malformed wake state is preserved and refused before lease acquisition"); return 0; } finally { foreach(var entry in original)Environment.SetEnvironmentVariable(entry.Key,entry.Value); diff --git a/tests/fm-live-gate.test.sh b/tests/fm-live-gate.test.sh index e6a35ece3aa..347923fa0e1 100755 --- a/tests/fm-live-gate.test.sh +++ b/tests/fm-live-gate.test.sh @@ -227,6 +227,41 @@ test_native_app_policy_capability_skips_non_windows_by_default() { pass "native app policy skips unsupported platforms unless explicitly forced" } +test_native_receipts_capability_skips_non_windows_by_default() { + local platform_bin result rc tool + platform_bin="$TMP_ROOT/non-windows-receipts-bin" + mkdir -p "$platform_bin" + for tool in powershell.exe git node; do + printf '#!/usr/bin/env bash\nexit 0\n' > "$platform_bin/$tool" + chmod +x "$platform_bin/$tool" + done + printf '#!/usr/bin/env bash\nprintf "Linux\\n"\n' > "$platform_bin/uname" + chmod +x "$platform_bin/uname" + + set +e + result=$(clean_env PATH="$platform_bin:/usr/bin:/bin" bash "$ROOT/tests/fm-native-owner-receipt-live-e2e.test.sh" 2>&1) + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "default native receipt selection failed on an unsupported platform: $result" + assert_contains "$result" "skip: live: Windows required" "unsupported default receipt selection must capability-skip" + + set +e + result=$(clean_env PATH="$platform_bin:/usr/bin:/bin" FM_LIVE_NATIVE_RECEIPTS=1 bash "$ROOT/tests/fm-native-owner-receipt-live-e2e.test.sh" 2>&1) + rc=$? + set -e + [ "$rc" -eq 1 ] || fail "forced native receipt selection did not refuse an unsupported platform: $result" + assert_contains "$result" "was requested but native receipt tests require Windows" "forced receipt platform refusal must name the unsupported capability" + assert_not_contains "$result" "skip:" "a forced unsupported receipt run must not report a skip" + + set +e + result=$(clean_env PATH="$platform_bin:/usr/bin:/bin" FM_LIVE=1 bash "$ROOT/tests/fm-native-owner-receipt-live-e2e.test.sh" 2>&1) + rc=$? + set -e + [ "$rc" -eq 1 ] || fail "family-forced native receipt selection did not refuse an unsupported platform: $result" + assert_not_contains "$result" "skip:" "FM_LIVE=1 must not turn an unsupported receipt run into a skip" + pass "native receipts skip unsupported platforms unless explicitly forced" +} + test_default_on_runs_when_the_tool_is_installed pass "a default-on guard runs wherever its tools are installed" test_default_on_skips_and_names_the_absent_tool @@ -247,3 +282,4 @@ test_gate_lets_a_guard_drive_the_real_fleet_scripts_under_a_gate_marker pass "the shared gate carries the gate-refusal bypass into every live guard" test_every_live_guard_is_wired_to_the_shared_gate test_native_app_policy_capability_skips_non_windows_by_default +test_native_receipts_capability_skips_non_windows_by_default diff --git a/tests/fm-native-owner-receipt-live-e2e.test.sh b/tests/fm-native-owner-receipt-live-e2e.test.sh index a4f083f2374..972340afaf1 100644 --- a/tests/fm-native-owner-receipt-live-e2e.test.sh +++ b/tests/fm-native-owner-receipt-live-e2e.test.sh @@ -6,7 +6,14 @@ set -eu fm_live_gate default-on FM_LIVE_NATIVE_RECEIPTS powershell.exe git node case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) ;; - *) printf '%s\n' 'Native receipt tests require Windows' >&2; exit 1 ;; + *) + if [ "${FM_LIVE_NATIVE_RECEIPTS:-}" = 1 ] || [ "${FM_LIVE:-}" = 1 ]; then + printf 'not ok - FM_LIVE_NATIVE_RECEIPTS was requested but native receipt tests require Windows\n' >&2 + exit 1 + fi + printf 'skip: live: Windows required for native receipt tests\n' + exit 0 + ;; esac powershell.exe -NoProfile -NonInteractive -File "$(cygpath -w "$ROOT/tests/fixtures/native-owner/Build.ps1")" node "$ROOT/tests/fixtures/native-owner/Lifecycle.mjs" diff --git a/tests/fm-startup-network.test.sh b/tests/fm-startup-network.test.sh index 346b71e4277..5eed63f0db4 100755 --- a/tests/fm-startup-network.test.sh +++ b/tests/fm-startup-network.test.sh @@ -323,7 +323,7 @@ EOF # test above: an actionable report (here, a MISSING: line bootstrap-diagnostics # would load a skill for) still reaches the wake queue even when unclaimed. test_an_actionable_successful_result_still_queues_a_wake() { - local rec home root log claimant + local rec home root log claimant status_tmp rec=$(new_world actionable-result-wakes) IFS='|' read -r home root log < "$status_tmp" + mv "$status_tmp" "$home/state/.startup-network.status" + if ( + FM_ROOT_OVERRIDE="$root" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" . "$root/bin/fm-wake-lib.sh" + fm_wake_native_empty_fleet_preflight "$home/state" + ); then + fail "native admission accepted a startup wake whose owner state did not match" + fi - pass "fm-startup-network: an actionable state=done report still queues a wake" + pass "fm-startup-network: actionable completion wakes stay producer-bound for native admission" } test_deferred_invalid_secondmate_markers_queue_durable_findings() { diff --git a/tests/fm-turnend-guard.test.sh b/tests/fm-turnend-guard.test.sh index 2718bc9f0a8..238b7b2c16d 100755 --- a/tests/fm-turnend-guard.test.sh +++ b/tests/fm-turnend-guard.test.sh @@ -104,21 +104,18 @@ test_predicate_source_needs_supervision() { fm_supervision_unhealthy "$state" 300 || fail "registered source with no beacon must be unhealthy" [ "$FM_SUP_IN_FLIGHT" -eq 0 ] || fail "a process-event source must not count as a task" [ "$FM_SUP_SOURCES" -eq 1 ] || fail "expected one registered process-event source" - [ "$FM_SUP_REASON_COUNT" -eq 1 ] && [ "$FM_SUP_REASON_LABEL" = 'process-event source(s) registered' ] \ - || fail "the selected supervision reason did not identify the source" pass "fm_supervision_unhealthy: source-only home needs supervision" } -test_predicate_turn_end_needs_supervision() { +test_predicate_turn_end_does_not_expand_supervision() { local state="$TMP_ROOT/pred-turn-end/state" mkdir -p "$state" printf 'preserve\n' > "$state/orphan.turn-ended" - fm_supervision_unhealthy "$state" 300 || fail "a residual turn-end notification did not keep supervision active" - [ "$FM_SUP_TURN_ENDS" -eq 1 ] || fail "expected one turn-end notification, got $FM_SUP_TURN_ENDS" - [ "$FM_SUP_REASON_COUNT" -eq 1 ] && [ "$FM_SUP_REASON_LABEL" = 'turn-end notification(s) pending' ] \ - || fail "the selected supervision reason did not identify the turn-end notification" + if fm_supervision_needed "$state" 300; then + fail "a residual turn-end notification changed ordinary supervision arming" + fi [ "$(cat "$state/orphan.turn-ended")" = preserve ] || fail "the supervision read changed the turn-end notification" - pass "fm_supervision_needed: a residual turn-end notification needs supervision" + pass "fm_supervision_needed: residual turn-end state does not change ordinary arming" } # Register a custom check the way an operator does, through the real @@ -154,33 +151,36 @@ test_predicate_registered_check_survives_rebinding_drift() { pass "fm_supervision_needed: a registered check whose bytes drifted still needs supervision" } -test_predicate_unregistered_check_needs_supervision() { +test_predicate_unregistered_check_needs_nothing() { local state="$TMP_ROOT/pred-check-unregistered/state" mkdir -p "$state" printf '#!/usr/bin/env bash\nexit 0\n' > "$state/rogue.check.sh" chmod 700 "$state/rogue.check.sh" - fm_supervision_needed "$state" 300 || fail "an unregistered check did not keep supervision active for rejection" + if fm_supervision_needed "$state" 300; then + fail "a check with no trust binding must not arm supervision" + fi [ "$FM_SUP_CHECKS" -eq 0 ] || fail "an unregistered check must not be counted, got $FM_SUP_CHECKS" - [ "$FM_SUP_CHECK_INPUTS" -eq 1 ] || fail "expected one state check input, got $FM_SUP_CHECK_INPUTS" - pass "fm_supervision_needed: an unregistered check remains visible for rejection" + pass "fm_supervision_needed: false for a check.sh with no registration binding" } -test_predicate_pending_reply_needs_supervision() { +test_predicate_pending_reply_does_not_expand_supervision() { local state="$TMP_ROOT/pred-pending-reply/state" mkdir -p "$state/pending-replies" printf 'schema=fm-pending-reply.v1\n' > "$state/pending-replies/0123456789abcdef" - fm_supervision_needed "$state" 300 || fail "a pending secondmate reply did not keep supervision active" - [ "$FM_SUP_PENDING_REPLIES" -eq 1 ] || fail "expected one pending reply input, got $FM_SUP_PENDING_REPLIES" - pass "fm_supervision_needed: a pending secondmate reply needs supervision" + if fm_supervision_needed "$state" 300; then + fail "a pending secondmate reply changed ordinary supervision arming" + fi + pass "fm_supervision_needed: pending replies do not change ordinary arming" } -test_predicate_reconcile_request_needs_supervision() { +test_predicate_reconcile_request_does_not_expand_supervision() { local state="$TMP_ROOT/pred-reconcile-request/state" mkdir -p "$state/reconcile-notify" printf '{}\n' > "$state/reconcile-notify/request-fixture.json" - fm_supervision_needed "$state" 300 || fail "a reconcile request did not keep supervision active" - [ "$FM_SUP_RECONCILE_REQUESTS" -eq 1 ] || fail "expected one reconcile request input, got $FM_SUP_RECONCILE_REQUESTS" - pass "fm_supervision_needed: a secondmate reconcile request needs supervision" + if fm_supervision_needed "$state" 300; then + fail "a reconcile request changed ordinary supervision arming" + fi + pass "fm_supervision_needed: reconcile requests do not change ordinary arming" } test_predicate_task_pr_poll_is_not_a_custom_check() { @@ -2237,12 +2237,12 @@ test_predicate_healthy_fresh_beacon test_predicate_queue_pending_flag test_predicate_x_mode_needs_supervision test_predicate_source_needs_supervision -test_predicate_turn_end_needs_supervision +test_predicate_turn_end_does_not_expand_supervision test_predicate_registered_check_needs_supervision test_predicate_registered_check_survives_rebinding_drift -test_predicate_unregistered_check_needs_supervision -test_predicate_pending_reply_needs_supervision -test_predicate_reconcile_request_needs_supervision +test_predicate_unregistered_check_needs_nothing +test_predicate_pending_reply_does_not_expand_supervision +test_predicate_reconcile_request_does_not_expand_supervision test_predicate_task_pr_poll_is_not_a_custom_check test_predicate_relay_shim_is_not_a_custom_check test_hook_silent_when_no_work_in_flight From ce60983e8194e7e43edc9723b8855dc0d263760d Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 10:23:51 +1200 Subject: [PATCH 23/61] no-mistakes(review): Restore native restart reconciliation paths --- bin/fm-inbox.sh | 53 ++++++++ bin/fm-startup-network.sh | 23 ++++ bin/fm-wake-lib.sh | 165 +++++------------------ tests/fixtures/native-owner/Launcher.mjs | 37 ++++- tests/fm-startup-network.test.sh | 8 +- 5 files changed, 147 insertions(+), 139 deletions(-) diff --git a/bin/fm-inbox.sh b/bin/fm-inbox.sh index f314a12f7a1..50e35833b7b 100755 --- a/bin/fm-inbox.sh +++ b/bin/fm-inbox.sh @@ -377,6 +377,58 @@ cmd_drain() { printf '\nAck with: fm-inbox.sh drain --ack ...\n' } +native_admission_predicate() { + local queue=${1:-} source count=0 epoch seq kind key payload extra id note handled record + [ -n "$queue" ] || return 2 + if [ ! -e "$queue" ] && [ ! -L "$queue" ]; then + source=/dev/null + else + source=$queue + fi + while IFS=$'\t' read -r epoch seq kind key payload extra || [ -n "$epoch$seq$kind$key$payload$extra" ]; do + case "$key" in inbox:*) ;; *) continue ;; esac + id=${key#inbox:} + case "$id" in ''|*[!A-Za-z0-9_-]*) return 1 ;; esac + [ "$kind" = check ] || return 1 + case "$payload" in "check: captain inbox note $id - "?*) ;; *) return 1 ;; esac + note="$INBOX/$id.note" + handled="$INBOX/handled/$id.note" + if { [ ! -f "$note" ] || [ -L "$note" ]; } && { [ ! -f "$handled" ] || [ -L "$handled" ]; }; then + return 1 + fi + if { [ -e "$note" ] || [ -L "$note" ]; } && { [ -e "$handled" ] || [ -L "$handled" ]; }; then + return 1 + fi + count=$((count + 1)) + done < "$source" + if [ -d "$INBOX" ] && [ ! -L "$INBOX" ]; then + for record in "$INBOX"/* "$INBOX"/.[!.]* "$INBOX"/..?*; do + [ -e "$record" ] || [ -L "$record" ] || continue + if [ "$record" = "$INBOX/handled" ]; then + [ -d "$record" ] && [ ! -L "$record" ] || return 1 + continue + fi + case "$record" in "$INBOX"/*.note) ;; *) return 1 ;; esac + [ -f "$record" ] && [ ! -L "$record" ] || return 1 + id=${record##*/}; id=${id%.note} + case "$id" in ''|*[!A-Za-z0-9_-]*) return 1 ;; esac + awk -F '\t' -v key="inbox:$id" '$3 == "check" && $4 == key { found=1 } END { exit !found }' "$source" || return 1 + done + if [ -d "$INBOX/handled" ] && [ ! -L "$INBOX/handled" ]; then + for record in "$INBOX/handled"/* "$INBOX/handled"/.[!.]* "$INBOX/handled"/..?*; do + [ -e "$record" ] || [ -L "$record" ] || continue + case "$record" in "$INBOX/handled"/*.note) ;; *) return 1 ;; esac + [ -f "$record" ] && [ ! -L "$record" ] || return 1 + id=${record##*/}; id=${id%.note} + case "$id" in ''|*[!A-Za-z0-9_-]*) return 1 ;; esac + done + fi + elif [ -e "$INBOX" ] || [ -L "$INBOX" ]; then + return 1 + fi + printf '%s\n' "$count" +} + # ---------------------------------------------------------------- dispatch case "${1:-}" in @@ -386,6 +438,7 @@ case "${1:-}" in ask) shift; cmd_ask "$@" ;; list) shift; cmd_list ;; drain) shift; cmd_drain "$@" ;; + native-admission-predicate) shift; native_admission_predicate "$@" ;; ''|-h|--help|help) # The whole header block, found rather than counted: everything after the # shebang up to the first line that is not a comment. A fixed line range diff --git a/bin/fm-startup-network.sh b/bin/fm-startup-network.sh index 791a4a7c5e3..88700acd586 100755 --- a/bin/fm-startup-network.sh +++ b/bin/fm-startup-network.sh @@ -329,6 +329,28 @@ report_requires_wake() { # "$REPORT_FILE" 2>/dev/null } +native_admission_predicate() { + local queue=${1:-} source count=0 epoch seq kind key payload extra state expected recognized + [ -n "$queue" ] || return 2 + if [ ! -e "$queue" ] && [ ! -L "$queue" ]; then + source=/dev/null + else + source=$queue + fi + while IFS=$'\t' read -r epoch seq kind key payload extra || [ -n "$epoch$seq$kind$key$payload$extra" ]; do + [ "$key" = startup-network ] || continue + [ "$kind" = check ] || return 1 + recognized=false + for state in done failed timeout; do + expected=$(fm_wake_startup_network_payload "$state") || return 1 + [ "$payload" != "$expected" ] || recognized=true + done + [ "$recognized" = true ] || return 1 + count=$((count + 1)) + done < "$source" + printf '%s\n' "$count" +} + await_delivery() { # local generation=$1 state=$2 limit waited=0 claim_record claim_generation claim_pid claim_live limit=$(( $(delivery_budget) * 10 )) @@ -653,6 +675,7 @@ case "$MODE" in harvest) cmd_harvest "${HARVEST_PID:-}" ;; report) print_state; print_timings ;; wait) cmd_wait "${1:-120}" || exit $? ;; + native-admission-predicate) native_admission_predicate "${1:-}" ;; -h|--help) usage ;; *) printf 'fm-startup-network: unknown mode: %s\n' "${MODE:-}" >&2 diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index c4aa46d7d0c..6863d537395 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -13,11 +13,11 @@ # unresolved and prevents acknowledgement or marker replacement from erasing # the current proof. # -# NATIVE EMPTY-FLEET ADMISSION. fm_wake_native_empty_fleet_preflight accepts -# only bounded producer rows for top-level inbox notes and deferred startup -# completion, plus valid main-presentation and recovery evidence. It is -# read-only, rejects branch grants and unknown or malformed rows, and reports -# refusals in FM_WAKE_NATIVE_ADMISSION_ERROR. +# NATIVE EMPTY-FLEET ADMISSION. fm_wake_native_empty_fleet_preflight validates +# bounded queue, presentation, and recovery state, while the inbox and deferred +# startup owners decide which of their rows are supported. It is read-only, +# rejects branch grants and unknown or malformed rows, and reports refusals in +# FM_WAKE_NATIVE_ADMISSION_ERROR. FM_WAKE_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_WAKE_DEFAULT_ROOT="$(cd "$FM_WAKE_LIB_DIR/.." && pwd)" @@ -2125,7 +2125,7 @@ fm_wake_grant_rows_valid() { # FM_WAKE_NATIVE_ADMISSION_ERROR= fm_wake_native_empty_fleet_preflight() { # local state=$1 queue marker seq_file main_rows history record size counter - local epoch seq kind key payload extra id note handled expected row_state status + local row_count inbox_count startup_count status FM_WAKE_NATIVE_ADMISSION_ERROR= queue="$state/.wake-queue" marker="$state/.watcher-down" @@ -2170,10 +2170,10 @@ fm_wake_native_empty_fleet_preflight() { # fi size=$(wc -c < "$queue" 2>/dev/null | tr -d '[:space:]') || size= case "$size" in ''|*[!0-9]*) size=8388609 ;; esac - if [ "$size" -gt 8388608 ] || ! LC_ALL=C awk -F '\t' ' + if [ "$size" -gt 8388608 ] || ! LC_ALL=C awk -F '\t' -v ceiling="$counter" ' NF != 5 || $1 !~ /^[0-9]+$/ || $2 !~ /^[0-9]+$/ || $2 == 0 || length($2) > 16 || $3 !~ /^(signal|stale|check|heartbeat)$/ || length($4) == 0 || length($5) == 0 || - seen[$2]++ || (previous && $2 <= previous) { bad=1 } + seen[$2]++ || (previous && $2 <= previous) || (ceiling != "" && $2 > ceiling) { bad=1 } { previous=$2; count++ } END { exit bad || count > 100000 } ' "$queue"; then @@ -2185,124 +2185,30 @@ fm_wake_native_empty_fleet_preflight() { # FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue has no producer sequence state at $seq_file" return 1 fi - if [ -s "$queue" ]; then - while IFS=$'\t' read -r epoch seq kind key payload extra || [ -n "$epoch$seq$kind$key$payload$extra" ]; do - [ -n "$epoch$seq$kind$key$payload$extra" ] || continue - [ "$seq" -le "$counter" ] || { - FM_WAKE_NATIVE_ADMISSION_ERROR="wake row $seq exceeds its producer sequence at $seq_file" - return 1 - } - case "$key" in - inbox:*) - id=${key#inbox:} - case "$id" in ''|*[!A-Za-z0-9_-]*) - FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue contains an invalid inbox identity" - return 1 - ;; - esac - case "$payload" in "check: captain inbox note $id - "?*) ;; *) - FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue contains an unrecognized inbox record" - return 1 - ;; - esac - [ "$kind" = check ] || { - FM_WAKE_NATIVE_ADMISSION_ERROR="wake queue contains an unsupported inbox wake kind" - return 1 - } - note="$state/inbox/$id.note" - handled="$state/inbox/handled/$id.note" - if { [ ! -f "$note" ] || [ -L "$note" ]; } && { [ ! -f "$handled" ] || [ -L "$handled" ]; }; then - FM_WAKE_NATIVE_ADMISSION_ERROR="inbox wake has no safe note record for $id" - return 1 - fi - if { [ -e "$note" ] || [ -L "$note" ]; } && { [ -e "$handled" ] || [ -L "$handled" ]; }; then - FM_WAKE_NATIVE_ADMISSION_ERROR="inbox wake has ambiguous note records for $id" - return 1 - fi - ;; - startup-network) - [ "$kind" = check ] || { - FM_WAKE_NATIVE_ADMISSION_ERROR="startup completion used an unsupported wake kind" - return 1 - } - row_state= - for status in done failed timeout; do - expected=$(fm_wake_startup_network_payload "$status") || return 1 - [ "$payload" != "$expected" ] || row_state=$status - done - if [ -z "$row_state" ] || [ ! -f "$state/.startup-network.status" ] \ - || [ ! -r "$state/.startup-network.status" ] || [ -L "$state/.startup-network.status" ]; then - FM_WAKE_NATIVE_ADMISSION_ERROR="startup completion wake lacks recognized owner state" - return 1 - fi - if ! awk -F= -v expected="$row_state" \ - '$1 == "state" && $2 == expected { count++ } END { exit count != 1 }' \ - "$state/.startup-network.status"; then - FM_WAKE_NATIVE_ADMISSION_ERROR="startup completion owner state is unrecognized" - return 1 - fi - ;; - *) - FM_WAKE_NATIVE_ADMISSION_ERROR="unsupported work-bearing wake is queued at $queue" - return 1 - ;; - esac - done < "$queue" - fi - if [ -d "$state/inbox" ] && [ ! -L "$state/inbox" ]; then - for record in "$state/inbox"/* "$state/inbox"/.[!.]* "$state/inbox"/..?*; do - [ -e "$record" ] || [ -L "$record" ] || continue - if [ "$record" = "$state/inbox/handled" ]; then - [ -d "$record" ] && [ ! -L "$record" ] || { - FM_WAKE_NATIVE_ADMISSION_ERROR="inbox handled state is unsafe at $record" - return 1 - } - continue - fi - case "$record" in "$state/inbox"/*.note) ;; *) - FM_WAKE_NATIVE_ADMISSION_ERROR="unrecognized inbox state is present at $record" - return 1 - ;; - esac - [ -f "$record" ] && [ ! -L "$record" ] || { - FM_WAKE_NATIVE_ADMISSION_ERROR="inbox note is unsafe at $record" - return 1 - } - id=${record##*/} - id=${id%.note} - case "$id" in ''|*[!A-Za-z0-9_-]*) - FM_WAKE_NATIVE_ADMISSION_ERROR="inbox note has an invalid identity at $record" - return 1 - ;; - esac - if ! awk -F '\t' -v key="inbox:$id" '$3 == "check" && $4 == key { found=1 } END { exit !found }' "$queue"; then - FM_WAKE_NATIVE_ADMISSION_ERROR="pending inbox note has no owner wake for $id" - return 1 - fi - done - if [ -d "$state/inbox/handled" ] && [ ! -L "$state/inbox/handled" ]; then - for record in "$state/inbox/handled"/* "$state/inbox/handled"/.[!.]* "$state/inbox/handled"/..?*; do - [ -e "$record" ] || [ -L "$record" ] || continue - case "$record" in "$state/inbox/handled"/*.note) ;; *) - FM_WAKE_NATIVE_ADMISSION_ERROR="unrecognized handled inbox state is present at $record" - return 1 - ;; - esac - [ -f "$record" ] && [ ! -L "$record" ] || { - FM_WAKE_NATIVE_ADMISSION_ERROR="handled inbox note is unsafe at $record" - return 1 - } - id=${record##*/} - id=${id%.note} - case "$id" in ''|*[!A-Za-z0-9_-]*) - FM_WAKE_NATIVE_ADMISSION_ERROR="handled inbox note has an invalid identity at $record" - return 1 - ;; - esac - done - fi - elif [ -e "$state/inbox" ] || [ -L "$state/inbox" ]; then - FM_WAKE_NATIVE_ADMISSION_ERROR="inbox state is unsafe at $state/inbox" + if [ -e "$queue" ]; then + row_count=$(awk 'END { print NR + 0 }' "$queue" 2>/dev/null) || row_count= + else + row_count=0 + fi + inbox_count=$(FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$state" \ + "$FM_WAKE_LIB_DIR/fm-inbox.sh" native-admission-predicate "$queue" 2>/dev/null) || { + FM_WAKE_NATIVE_ADMISSION_ERROR="inbox owner refused native admission state" + return 1 + } + startup_count=$(FM_ROOT_OVERRIDE="$FM_ROOT" FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$state" \ + "$FM_WAKE_LIB_DIR/fm-startup-network.sh" native-admission-predicate "$queue" 2>/dev/null) || { + FM_WAKE_NATIVE_ADMISSION_ERROR="startup owner refused native admission state" + return 1 + } + case "$row_count:$inbox_count:$startup_count" in + *[!0-9:]*) + FM_WAKE_NATIVE_ADMISSION_ERROR="wake producer admission evidence is unrecognized" + return 1 + ;; + esac + if [ -z "$row_count" ] || [ -z "$inbox_count" ] || [ -z "$startup_count" ] \ + || [ "$row_count" -ne $((inbox_count + startup_count)) ]; then + FM_WAKE_NATIVE_ADMISSION_ERROR="unsupported work-bearing wake is queued at $queue" return 1 fi if [ -e "$marker" ] || [ -L "$marker" ]; then @@ -2310,13 +2216,6 @@ fm_wake_native_empty_fleet_preflight() { # FM_WAKE_NATIVE_ADMISSION_ERROR="wake recovery state is unrecognized at $marker" return 1 fi - if [ -s "$queue" ]; then - case "$FM_RECOVERY_MARKER_TOKEN" in pending:*|announced:*) ;; *) - FM_WAKE_NATIVE_ADMISSION_ERROR="queued wakes lack a pending recovery generation" - return 1 - ;; - esac - fi elif [ -s "$queue" ]; then FM_WAKE_NATIVE_ADMISSION_ERROR="queued wakes have no recovery generation at $marker" return 1 diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 174019400fe..8f924967259 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -14,7 +14,7 @@ const sleep=ms=>new Promise(resolve=>setTimeout(resolve,ms)); function command(exe,args){const result=spawnSync(exe,args,{encoding:'utf8',timeout:120000});if(result.status!==0)throw Error(result.stderr||result.stdout);return result;} command('git',['-c','core.symlinks=true','clone','--quiet','--no-local','--single-branch',repo,code]); fs.cpSync(path.join(repo,'bin/native-owner'),path.join(code,'bin/native-owner'),{recursive:true}); -for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh','fm-backlog-transition-lib.sh','fm-supervision-lib.sh','fm-wake-lib.sh','fm-startup-network.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); +for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh','fm-backlog-transition-lib.sh','fm-supervision-lib.sh','fm-wake-lib.sh','fm-startup-network.sh','fm-inbox.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); const launcher=path.join(code,'bin/fm-native-codex.ps1'); command('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly']); const removedAlias=spawnSync('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly','-Home',path.join(area,'alias')],{encoding:'utf8',timeout:120000}); @@ -46,6 +46,17 @@ function enqueue(home,message){ const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export PATH="$1/bin/native-owner/tools:/usr/bin:/bin:$PATH"; export FM_HOME; FM_HOME=$(cygpath -u "$3"); exec /usr/bin/bash "$1/bin/fm-inbox.sh" note "$2"','launcher-test',posix(code),message,home],{env,encoding:'utf8',timeout:30000}); assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr}));return result.stdout.trim().split(/\s+/)[1]; } +function appendStartupWake(home,state){ + const env={...process.env,FM_HOME:home,FM_ROOT_OVERRIDE:posix(code),FM_STATE_OVERRIDE:posix(path.join(home,'state')),MSYS:'winsymlinks:nativestrict'}; + const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','. "$1/bin/fm-wake-lib.sh"; fm_wake_append_startup_network "$2"','startup-wake',posix(code),state],{env,encoding:'utf8',timeout:30000}); + assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); +} +function captureAcknowledgement(home){ + const env={...process.env,FM_HOME:home,MSYS:'winsymlinks:nativestrict'}; + const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',posix(path.join(code,'bin/native-owner/ack-evidence.sh')),'capture-json'],{env,encoding:'utf8',timeout:30000}); + assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); + return JSON.parse(result.stdout); +} const home=path.join(area,'home');fs.mkdirSync(path.join(home,'data'),{recursive:true}); fs.writeFileSync(path.join(home,'data/backlog.md'),'## In flight\n\n## Queued\n\n## Done\n'); const first=start(home);const initial=await ready(first);console.error('first ready',area); @@ -89,6 +100,30 @@ const queuedRestart=start(queuedHome);await ready(queuedRestart,queuedReady.owne assert.equal((await bound(queuedRestart.done,queuedRestart,20000)).exit,0); assert.equal(fs.readFileSync(path.join(queuedHome,'state/inbox',queuedNote+'.note'),'utf8'),queuedBody); records.push('producer-created inbox and native handling recovery remain admissible across restart'); +const historicalHome=path.join(area,'historical-startup-completion'),historicalState=path.join(historicalHome,'state'),historicalStatus=path.join(historicalState,'.startup-network.status'); +fs.mkdirSync(historicalState,{recursive:true});fs.writeFileSync(historicalStatus,'generation=historical\nstate=failed\n');appendStartupWake(historicalHome,'failed'); +const historicalRow=fs.readFileSync(path.join(historicalState,'.wake-queue'),'utf8');fs.writeFileSync(historicalStatus,'generation=newer\nstate=done\n'); +const historicalSession=start(historicalHome);await ready(historicalSession);historicalSession.child.stdin.end();assert.equal((await bound(historicalSession.done,historicalSession,20000)).exit,0); +assert(fs.readFileSync(path.join(historicalState,'.wake-queue'),'utf8').includes(historicalRow.trim())); +records.push('queued startup failure survives a newer startup status and remains admissible'); +const interruptedHome=path.join(area,'interrupted-ack-restart'),interruptedNote=enqueue(interruptedHome,'Preserve this interrupted acknowledgement for reconciliation.'); +const preparedSession=start(interruptedHome);const prepared=await ready(preparedSession);preparedSession.child.stdin.end();assert.equal((await bound(preparedSession.done,preparedSession,20000)).exit,0); +const target=captureAcknowledgement(interruptedHome),receipt='c'.repeat(32),payload={...target,challenge:'interrupted-restart',message:'preserved partial acknowledgement'}; +const journal=path.join(interruptedHome,'owner-receipts.jsonl'),journalRows=[ + {version:1,home:path.resolve(interruptedHome),generation:prepared.owner.generation,event:'presented',receipt,payload,targetEvidence:null,ackGeneration:null}, + {version:1,home:path.resolve(interruptedHome),generation:prepared.owner.generation,event:'ack-started',receipt,payload,targetEvidence:target.ownerEvidence,ackGeneration:null}, +]; +fs.appendFileSync(journal,journalRows.map(row=>JSON.stringify(row)).join('\n')+'\n'); +const interruptedQueue=path.join(interruptedHome,'state/.wake-queue'),queueBeforeRestart=fs.readFileSync(interruptedQueue,'utf8'),pendingNote=path.join(interruptedHome,'state/inbox',interruptedNote+'.note'),handledNote=path.join(interruptedHome,'state/inbox/handled',interruptedNote+'.note'); +fs.mkdirSync(path.dirname(handledNote),{recursive:true});fs.renameSync(pendingNote,handledNote);const handledBody=fs.readFileSync(handledNote,'utf8'); +const interruptedMarker=path.join(interruptedHome,'state/.watcher-down'),markerBody=`acked:handling:${target.generation}\n`;fs.writeFileSync(interruptedMarker,markerBody); +const interruptedRestart=start(interruptedHome);interruptedRestart.child.stdin.end();const reconciliation=await bound(interruptedRestart.done,interruptedRestart,20000); +assert.notEqual(reconciliation.exit,0);assert(reconciliation.stderr.includes('earlier acknowledgement is incomplete'),reconciliation.stderr); +const reconciledOwner=read(path.join(interruptedHome,'owner-probe.json'));assert.notEqual(reconciledOwner.generation,prepared.owner.generation); +const reconciliationRuntime=path.join(interruptedHome,'state/native-runtime',reconciledOwner.generation); +assert.equal(fs.existsSync(path.join(reconciliationRuntime,'startup.log')),false);assert.equal(fs.existsSync(path.join(reconciliationRuntime,'startup.finished')),false); +assert.equal(fs.readFileSync(interruptedQueue,'utf8'),queueBeforeRestart);assert.equal(fs.readFileSync(interruptedMarker,'utf8'),markerBody);assert.equal(fs.readFileSync(handledNote,'utf8'),handledBody); +records.push('partial acknowledgement reaches reconciliation unchanged without restarting startup'); const maskedHome=path.join(area,'bash-env-mask'),maskedStatus=path.join(maskedHome,'state/orphan.status'),mask=path.join(area,'bash-env-exit.sh'); fs.mkdirSync(path.dirname(maskedStatus),{recursive:true});fs.writeFileSync(maskedStatus,'preserve masked residual state');fs.writeFileSync(mask,'exit 0\n'); const masked=start(maskedHome,true,{...process.env,BASH_ENV:mask});masked.child.stdin.end();assert.notEqual((await bound(masked.done,masked,20000)).exit,0); diff --git a/tests/fm-startup-network.test.sh b/tests/fm-startup-network.test.sh index 5eed63f0db4..8633f76fbf3 100755 --- a/tests/fm-startup-network.test.sh +++ b/tests/fm-startup-network.test.sh @@ -349,14 +349,12 @@ EOF status_tmp="$home/state/.startup-network.status.mismatch" sed 's/^state=done$/state=failed/' "$home/state/.startup-network.status" > "$status_tmp" mv "$status_tmp" "$home/state/.startup-network.status" - if ( + ( FM_ROOT_OVERRIDE="$root" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" . "$root/bin/fm-wake-lib.sh" fm_wake_native_empty_fleet_preflight "$home/state" - ); then - fail "native admission accepted a startup wake whose owner state did not match" - fi + ) || fail "native admission tied a queued startup completion to mutable latest-run status" - pass "fm-startup-network: actionable completion wakes stay producer-bound for native admission" + pass "fm-startup-network: producer-owned completion wakes survive later status publication" } test_deferred_invalid_secondmate_markers_queue_durable_findings() { From a339214a8f2953e43b1e9c6381d7fb4055f80c28 Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 10:54:43 +1200 Subject: [PATCH 24/61] no-mistakes(review): Harden native launch admission and recovery ownership --- bin/fm-inbox.sh | 9 +- bin/fm-lock.sh | 33 ++++++++ bin/fm-startup-network.sh | 10 ++- bin/fm-supervision-lib.sh | 3 +- bin/fm-wake-lib.sh | 29 ++++++- bin/native-owner/NativeLauncher.cs | 4 +- bin/native-owner/NativeOperations.cs | 13 ++- bin/native-owner/NativeReceiptJournal.cs | 48 ++++++++--- bin/native-owner/ack-evidence.sh | 7 +- bin/native-owner/admit.sh | 25 +++++- tests/fixtures/native-owner/Launcher.mjs | 94 ++++++++++++++++----- tests/fixtures/native-owner/NativeDriver.cs | 22 ++--- 12 files changed, 235 insertions(+), 62 deletions(-) diff --git a/bin/fm-inbox.sh b/bin/fm-inbox.sh index 50e35833b7b..914ca53a46a 100755 --- a/bin/fm-inbox.sh +++ b/bin/fm-inbox.sh @@ -25,6 +25,9 @@ # fm-inbox.sh ask ... # fm-inbox.sh list # fm-inbox.sh drain [--ack ...] +# fm-inbox.sh native-admission-predicate +# Internal read-only mode: print the owned row count; exit 1 for +# unrecognized state and 2 for invalid usage. # # Configuration. A region, a model id and an AWS profile name somebody's account # and somebody's choices, so this file carries no default for any of them. Each is @@ -438,7 +441,11 @@ case "${1:-}" in ask) shift; cmd_ask "$@" ;; list) shift; cmd_list ;; drain) shift; cmd_drain "$@" ;; - native-admission-predicate) shift; native_admission_predicate "$@" ;; + native-admission-predicate) + shift + [ "$#" -eq 1 ] || { printf 'usage: fm-inbox.sh native-admission-predicate \n' >&2; exit 2; } + native_admission_predicate "$@" + ;; ''|-h|--help|help) # The whole header block, found rather than counted: everything after the # shebang up to the first line that is not a comment. A fixed line range diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index c205a20dbdb..33cf7a34bd1 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -4,6 +4,8 @@ # the shell's ancestry and an opaque launch-bound identity for the native owner. # Usage: fm-lock.sh acquire; exit 1 unless ownership is verified # fm-lock.sh status print holder and liveness; always exits 0 +# fm-lock.sh native-admission-predicate +# exit 0 only when no owner excludes a native launch set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -51,6 +53,37 @@ fm_lock_conflict_message() { return 1 } +if [ "${1:-}" = "native-admission-predicate" ]; then + [ "$#" -eq 1 ] || { + echo "usage: fm-lock.sh native-admission-predicate" >&2 + exit 2 + } + if [ ! -e "$LOCK" ] && [ ! -L "$LOCK" ]; then + exit 0 + fi + if [ ! -f "$LOCK" ] || [ -L "$LOCK" ]; then + echo "error: session lock is not a readable regular file; native launch refused" >&2 + exit 1 + fi + old=$(cat "$LOCK" 2>/dev/null) || { + echo "error: session lock is unreadable; native launch refused" >&2 + exit 1 + } + if ! fm_session_pid_valid "$old"; then + echo "error: session lock owner is unrecognized; native launch refused" >&2 + exit 1 + fi + if fm_harness_pid_excludes "$old"; then + if conflict=$(fm_lock_conflict_message "$old"); then + echo "$conflict" >&2 + else + echo "error: another firstmate session may hold the lock; native launch refused" >&2 + fi + exit 1 + fi + exit 0 +fi + if [ "${1:-}" = "status" ]; then if [ ! -f "$LOCK" ]; then echo "lock: free"; exit 0; fi old=$(cat "$LOCK" 2>/dev/null) || { diff --git a/bin/fm-startup-network.sh b/bin/fm-startup-network.sh index 88700acd586..a3acd1bf141 100755 --- a/bin/fm-startup-network.sh +++ b/bin/fm-startup-network.sh @@ -73,6 +73,9 @@ # fm-startup-network.sh wait [] # Block until the report is published, up to (default 120). # For operators and tests only; a session start never waits. +# fm-startup-network.sh native-admission-predicate +# Internal read-only mode: print the owned row count; exit 1 for +# unrecognized state and 2 for invalid usage. # # STATE, all under this home's state/ and gitignored with it: # .startup-network.status key=value record - generation, lock_pid, state, @@ -675,11 +678,14 @@ case "$MODE" in harvest) cmd_harvest "${HARVEST_PID:-}" ;; report) print_state; print_timings ;; wait) cmd_wait "${1:-120}" || exit $? ;; - native-admission-predicate) native_admission_predicate "${1:-}" ;; + native-admission-predicate) + [ "$#" -eq 1 ] || { printf 'usage: fm-startup-network.sh native-admission-predicate \n' >&2; exit 2; } + native_admission_predicate "$1" + ;; -h|--help) usage ;; *) printf 'fm-startup-network: unknown mode: %s\n' "${MODE:-}" >&2 - printf 'usage: fm-startup-network.sh start|run|harvest|report|wait\n' >&2 + printf 'usage: fm-startup-network.sh start|run|harvest|report|wait|native-admission-predicate\n' >&2 exit 2 ;; esac diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index 8d8d34184f4..595d2107d1e 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -119,7 +119,8 @@ FM_SUP_RESIDUAL_ERROR= fm_supervision_residual_inputs_absent() { # local state=$1 record count FM_SUP_RESIDUAL_ERROR= - for record in "$state"/*.check.sh "$state"/*.check-trust "$state"/*.turn-ended; do + for record in "$state"/*.check.sh "$state"/*.check-trust "$state"/*.turn-ended \ + "$state/.afk" "$state/.afk-contract"; do if [ -e "$record" ] || [ -L "$record" ]; then FM_SUP_RESIDUAL_ERROR="supervision input is present at $record" return 1 diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 6863d537395..9741b474df8 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -17,7 +17,8 @@ # bounded queue, presentation, and recovery state, while the inbox and deferred # startup owners decide which of their rows are supported. It is read-only, # rejects branch grants and unknown or malformed rows, and reports refusals in -# FM_WAKE_NATIVE_ADMISSION_ERROR. +# FM_WAKE_NATIVE_ADMISSION_ERROR. An acknowledged marker is accepted only with +# matching evidence selected by the native receipt-journal owner. FM_WAKE_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_WAKE_DEFAULT_ROOT="$(cd "$FM_WAKE_LIB_DIR/.." && pwd)" @@ -2123,8 +2124,8 @@ fm_wake_grant_rows_valid() { # } FM_WAKE_NATIVE_ADMISSION_ERROR= -fm_wake_native_empty_fleet_preflight() { # - local state=$1 queue marker seq_file main_rows history record size counter +fm_wake_native_empty_fleet_preflight() { # [] + local state=$1 native_evidence=${2:-} queue marker seq_file main_rows history record size counter local row_count inbox_count startup_count status FM_WAKE_NATIVE_ADMISSION_ERROR= queue="$state/.wake-queue" @@ -2216,6 +2217,14 @@ fm_wake_native_empty_fleet_preflight() { # FM_WAKE_NATIVE_ADMISSION_ERROR="wake recovery state is unrecognized at $marker" return 1 fi + case "$FM_RECOVERY_MARKER_TOKEN" in + acked:*) + if [ -z "$native_evidence" ] || ! fm_wake_ack_evidence_native_recovery "$native_evidence" "$marker"; then + FM_WAKE_NATIVE_ADMISSION_ERROR="acknowledged wake recovery is not owned by the native receipt journal" + return 1 + fi + ;; + esac elif [ -s "$queue" ]; then FM_WAKE_NATIVE_ADMISSION_ERROR="queued wakes have no recovery generation at $marker" return 1 @@ -2544,6 +2553,20 @@ fm_wake_ack_evidence_load() { # FM_WAKE_ACK_EVIDENCE_TOKEN=$token } +fm_wake_ack_evidence_native_recovery() { # + local token=$1 marker=$2 current expected + fm_wake_ack_evidence_load "$token" || return 1 + fm_recovery_marker_read "$marker" || { fm_wake_ack_evidence_clear; return 1; } + current=$FM_RECOVERY_MARKER_TOKEN + if [ "$FM_WAKE_ACK_EVIDENCE_LEGACY" = 1 ] && [ -z "$FM_WAKE_ACK_EVIDENCE_MARKER" ]; then + case "$current" in acked:handling:*|acked:downtime:*) ;; *) fm_wake_ack_evidence_clear; return 1 ;; esac + else + expected="acked:${FM_WAKE_ACK_EVIDENCE_MARKER#*:}" + [ "$current" = "$expected" ] || { fm_wake_ack_evidence_clear; return 1; } + fi + fm_wake_ack_evidence_clear +} + fm_wake_ack_evidence_precondition() { # local token=$1 marker="$STATE/.watcher-down" id digest note handled fm_wake_ack_evidence_load "$token" || return 1 diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs index 798687c5601..3f5c6a095a8 100644 --- a/bin/native-owner/NativeLauncher.cs +++ b/bin/native-owner/NativeLauncher.cs @@ -16,7 +16,7 @@ static int Launch(string selectedHome) { string home=Path.GetFullPath(selectedHome), node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); string host=Path.Combine(CodeRoot,"bin","native-owner","codex-host.mjs"); if(!File.Exists(node)||!File.Exists(host)) throw new IOException("Native Node or the code-owned host is missing"); - EmptyFleet(home); + EmptyFleet(home,true); string session=Guid.NewGuid().ToString("N"),nonce=Guid.NewGuid().ToString("N"),pipeName="fm-native-"+session; IntPtr job=CreateJobObject(IntPtr.Zero,null),env=IntPtr.Zero,output=IntPtr.Zero,input=IntPtr.Zero; if(job==IntPtr.Zero) throw Error("Create session job"); @@ -123,7 +123,7 @@ static int Launch(string selectedHome) { } static int FixedOperation(string purpose) { if(purpose!="startup"&&purpose!="check"&&purpose!="ack")throw new ArgumentException("Unknown fixed operation"); - string home=Environment.GetEnvironmentVariable("FM_HOME");EmptyFleet(home); + string home=Environment.GetEnvironmentVariable("FM_HOME");EmptyFleet(home,false); if(OwnerClient(purpose=="startup" ? "identity" : "owns",Path.Combine(home,"state"),"")!=0)throw new InvalidOperationException("Operation is not registered"); string script=Path.Combine(CodeRoot,"bin","native-owner",purpose+".sh"); using(var process=Process.Start(BashHelper(script,"",home,true))) { diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index 50060fc8dba..d1dc4a60126 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -31,19 +31,24 @@ internal static ProcessStartInfo BashHelper(string script,string arguments,strin start.EnvironmentVariables["MSYS"]="winsymlinks:nativestrict"; return start; } - static void OwnerAdmission(string home) { + static void OwnerAdmission(string home,bool launch) { string script=Path.Combine(CodeRoot,"bin","native-owner","admit.sh"); - var start=BashHelper(script,"",home);start.RedirectStandardError=true; + object evidence=NativeReceiptJournal.AdmissionEvidence(home); + var start=BashHelper(script,launch ? "launch" : "owned-operation",home);start.RedirectStandardInput=true;start.RedirectStandardError=true; + string input=""; + if(evidence is string) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="token";input=(string)evidence;} + else if(evidence!=null) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="legacy";input=Json.Serialize(evidence);} using(var process=Process.Start(start)) { var error=process.StandardError.ReadToEndAsync(); + process.StandardInput.Write(input);process.StandardInput.Close(); if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("The empty-fleet owner preflight exceeded its bound");} if(process.ExitCode!=0)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); } } - internal static void EmptyFleet(string home) { + internal static void EmptyFleet(string home,bool launch) { string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); if((Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); - OwnerAdmission(home); + OwnerAdmission(home,launch); } static IntPtr FileHandle(string path, uint access, uint creation) { SA sa = new SA { length=Marshal.SizeOf(typeof(SA)), inherit=1 }; diff --git a/bin/native-owner/NativeReceiptJournal.cs b/bin/native-owner/NativeReceiptJournal.cs index dddae6c7a42..a4596c0f03f 100644 --- a/bin/native-owner/NativeReceiptJournal.cs +++ b/bin/native-owner/NativeReceiptJournal.cs @@ -15,6 +15,7 @@ public sealed class NativeReceiptJournal : IDisposable { readonly Dictionary> receipts = new Dictionary>(); readonly string home, generation; readonly NativeHomeLease lease; + object latestAcknowledgementEvidence; FileStream file; const long Limit = 16 * 1024 * 1024; [StructLayout(LayoutKind.Sequential)] struct Info { @@ -22,6 +23,38 @@ public sealed class NativeReceiptJournal : IDisposable { public uint volume, sizeHigh, sizeLow, links, indexHigh, indexLow; } [DllImport("kernel32.dll", SetLastError=true)] static extern bool GetFileInformationByHandle(IntPtr handle, out Info info); + NativeReceiptJournal(string selectedHome) { + home=Path.GetFullPath(selectedHome).TrimEnd('\\','/'); + } + static void ValidateFile(FileStream stream,SecurityIdentifier user) { + var access=stream.GetAccessControl(); + if(!access.AreAccessRulesProtected || !access.GetOwner(typeof(SecurityIdentifier)).Equals(user)) throw new IOException("Receipt journal security differs; preserved"); + foreach(FileSystemAccessRule rule in access.GetAccessRules(true,true,typeof(SecurityIdentifier))) if(rule.AccessControlType==AccessControlType.Allow && !rule.IdentityReference.Equals(user)) throw new IOException("Receipt journal grants unexpected access; preserved"); + Info info; + if(!GetFileInformationByHandle(stream.SafeFileHandle.DangerousGetHandle(),out info) || info.links!=1 || (info.attributes&0x400)!=0) throw new IOException("Receipt journal file identity is unsafe"); + } + void Load(FileStream stream) { + if(stream.Length>Limit || stream.Length==0) throw new IOException("Receipt journal is empty or oversized; preserved"); + string content; + stream.Position=0; + using(var reader=new StreamReader(stream,new UTF8Encoding(false,true),true,4096,true)) content=reader.ReadToEnd(); + if(!content.EndsWith("\n",StringComparison.Ordinal)) throw new IOException("Interrupted receipt record; preserved"); + foreach(string line in content.Split(new [] {'\n'},StringSplitOptions.RemoveEmptyEntries)) Apply(json.Deserialize>(line)); + } + public static object AdmissionEvidence(string selectedHome) { + string home=Path.GetFullPath(selectedHome).TrimEnd('\\','/'),name=Path.Combine(home,"owner-receipts.jsonl"); + FileAttributes attributes; + try { attributes=File.GetAttributes(name); } + catch(FileNotFoundException) { return null; } + catch(DirectoryNotFoundException) { return null; } + if((attributes&FileAttributes.ReparsePoint)!=0 || (attributes&FileAttributes.Directory)!=0) throw new IOException("Receipt journal path is unsafe; preserved"); + var parser=new NativeReceiptJournal(home); + using(var stream=new FileStream(name,FileMode.Open,FileAccess.Read,FileShare.ReadWrite)) { + ValidateFile(stream,WindowsIdentity.GetCurrent().User); + parser.Load(stream); + } + return parser.latestAcknowledgementEvidence; + } public NativeReceiptJournal(NativeHomeLease ownedLease, string ownerGeneration) { if(ownedLease==null || !ownedLease.IsHeld) throw new InvalidOperationException("An active native home lease is required"); lease=ownedLease; @@ -42,18 +75,8 @@ public NativeReceiptJournal(NativeHomeLease ownedLease, string ownerGeneration) if((File.GetAttributes(name)&FileAttributes.ReparsePoint)!=0) throw new IOException("Receipt journal is a reparse point"); file=new FileStream(name,FileMode.Open,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security); } - var access=file.GetAccessControl(); - if(!access.AreAccessRulesProtected || !access.GetOwner(typeof(SecurityIdentifier)).Equals(user)) throw new IOException("Receipt journal security differs; preserved"); - foreach(FileSystemAccessRule rule in access.GetAccessRules(true,true,typeof(SecurityIdentifier))) if(rule.AccessControlType==AccessControlType.Allow && !rule.IdentityReference.Equals(user)) throw new IOException("Receipt journal grants unexpected access; preserved"); - Info info; - if(!GetFileInformationByHandle(file.SafeFileHandle.DangerousGetHandle(),out info) || info.links!=1 || (info.attributes&0x400)!=0) throw new IOException("Receipt journal file identity is unsafe"); - if(file.Length>Limit || (!created && file.Length==0)) throw new IOException("Receipt journal is empty or oversized; preserved"); - if(!created) { - string content; - using(var reader=new StreamReader(file,new UTF8Encoding(false,true),true,4096,true)) content=reader.ReadToEnd(); - if(!content.EndsWith("\n",StringComparison.Ordinal)) throw new IOException("Interrupted receipt record; preserved"); - foreach(string line in content.Split(new [] {'\n'},StringSplitOptions.RemoveEmptyEntries)) Apply(json.Deserialize>(line)); - } + ValidateFile(file,user); + if(!created) Load(file); Append("session",null,null); } catch { Dispose();throw; } } @@ -136,6 +159,7 @@ void Apply(Dictionary row) { if(!exists || (string)previous["event"]!="ack-started" || !row.ContainsKey("ackGeneration") || (string)row["ackGeneration"]!=(string)previous["generation"] || json.Serialize(previous["payload"])!=json.Serialize(row["payload"]) || Evidence(row)==null || json.Serialize(Evidence(previous))!=json.Serialize(Evidence(row))) throw new IOException("Invalid recovery transition; preserved"); } else throw new IOException("Unknown receipt transition; preserved"); receipts[id]=row; + if(kind=="ack-started" || kind=="acknowledged" || kind=="recovered-acknowledged") latestAcknowledgementEvidence=Evidence(row); } public void Dispose() { if(file!=null) { file.Dispose();file=null; } } } diff --git a/bin/native-owner/ack-evidence.sh b/bin/native-owner/ack-evidence.sh index 0be15313200..5ef1ec1f211 100644 --- a/bin/native-owner/ack-evidence.sh +++ b/bin/native-owner/ack-evidence.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Usage: FM_HOME= ack-evidence.sh capture-json|preflight-token|verify-token|verify-legacy|acknowledge-token +# Usage: FM_HOME= ack-evidence.sh capture-json|legacy-token|preflight-token|verify-token|verify-legacy|acknowledge-token # Required environment: FM_HOME; token modes read their evidence from standard input. set -euo pipefail ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) @@ -64,6 +64,9 @@ case "${1:-}" in printf '],"ownerEvidence":"%s"}\n' "$FM_WAKE_ACK_EVIDENCE_TOKEN" fm_wake_ack_evidence_clear ;; + legacy-token) + legacy_token + ;; preflight-token) token=$(read_token) fm_wake_ack_evidence_precondition "$token" @@ -82,7 +85,7 @@ case "${1:-}" in fm_wake_ack_evidence_acknowledge "$token" ;; *) - printf 'usage: ack-evidence.sh capture-json|preflight-token|verify-token|verify-legacy|acknowledge-token\n' >&2 + printf 'usage: ack-evidence.sh capture-json|legacy-token|preflight-token|verify-token|verify-legacy|acknowledge-token\n' >&2 exit 2 ;; esac diff --git a/bin/native-owner/admit.sh b/bin/native-owner/admit.sh index c05036b2fc2..1a8bea25f47 100644 --- a/bin/native-owner/admit.sh +++ b/bin/native-owner/admit.sh @@ -1,11 +1,30 @@ #!/usr/bin/env bash -# Usage: FM_HOME= admit.sh -# Required environment: FM_HOME. +# Usage: FM_HOME= admit.sh launch|owned-operation +# Required environment: FM_HOME; acknowledgement evidence is read from standard input. set -euo pipefail ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) +mode=${1:-} +[ "$#" -eq 1 ] || { printf 'usage: admit.sh launch|owned-operation\n' >&2; exit 2; } +case "$mode" in launch|owned-operation) ;; *) printf 'usage: admit.sh launch|owned-operation\n' >&2; exit 2 ;; esac +native_evidence=$(cat) +case "${FM_NATIVE_ACK_EVIDENCE_KIND:-}" in + '') [ -z "$native_evidence" ] || { printf 'unexpected native acknowledgement evidence\n' >&2; exit 2; } ;; + token) [ -n "$native_evidence" ] || { printf 'native acknowledgement evidence is empty\n' >&2; exit 2; } ;; + legacy) + native_evidence=$(printf '%s' "$native_evidence" | bash "$ROOT/bin/native-owner/ack-evidence.sh" legacy-token) || { + printf 'legacy native acknowledgement evidence is unrecognized\n' >&2 + exit 2 + } + ;; + *) printf 'native acknowledgement evidence kind is unrecognized\n' >&2; exit 2 ;; +esac export FM_HOME FM_HOME=$(cygpath -u "${FM_HOME:?}") cd "$ROOT" +if [ "$mode" = launch ] && ! FM_STATE_OVERRIDE="$FM_HOME/state" bin/fm-lock.sh native-admission-predicate; then + printf 'the home has a live or unresolved session owner\n' >&2 + exit 2 +fi . bin/fm-tasks-axi-lib.sh . bin/fm-backlog-transition-lib.sh . bin/fm-supervision-lib.sh @@ -20,7 +39,7 @@ fi if [ -e "$FM_HOME/state" ] || [ -L "$FM_HOME/state" ]; then export FM_STATE_OVERRIDE="$FM_HOME/state" . bin/fm-wake-lib.sh - if ! fm_wake_native_empty_fleet_preflight "$FM_HOME/state"; then + if ! fm_wake_native_empty_fleet_preflight "$FM_HOME/state" "$native_evidence"; then printf '%s\n' "${FM_WAKE_NATIVE_ADMISSION_ERROR:-the home contains unsupported wake state}" >&2 exit 2 fi diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 8f924967259..690faac7459 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -14,7 +14,7 @@ const sleep=ms=>new Promise(resolve=>setTimeout(resolve,ms)); function command(exe,args){const result=spawnSync(exe,args,{encoding:'utf8',timeout:120000});if(result.status!==0)throw Error(result.stderr||result.stdout);return result;} command('git',['-c','core.symlinks=true','clone','--quiet','--no-local','--single-branch',repo,code]); fs.cpSync(path.join(repo,'bin/native-owner'),path.join(code,'bin/native-owner'),{recursive:true}); -for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh','fm-backlog-transition-lib.sh','fm-supervision-lib.sh','fm-wake-lib.sh','fm-startup-network.sh','fm-inbox.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); +for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh','fm-backlog-transition-lib.sh','fm-supervision-lib.sh','fm-wake-lib.sh','fm-startup-network.sh','fm-inbox.sh','fm-lock.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); const launcher=path.join(code,'bin/fm-native-codex.ps1'); command('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly']); const removedAlias=spawnSync('powershell.exe',['-NoProfile','-File',launcher,'-BuildOnly','-Home',path.join(area,'alias')],{encoding:'utf8',timeout:120000}); @@ -35,10 +35,27 @@ async function ready(session,previous){ } throw Error('Launcher readiness timed out'); } +async function waitUntil(session,label,predicate,ms=40000){ + const limit=Date.now()+ms; + while(Date.now()value.replaceAll('\\','/').replace(/^([A-Za-z]):/,(_,drive)=>'/'+drive.toLowerCase()); +const contractHome=path.join(area,'predicate-contracts'),contractEnv={...process.env,FM_HOME:posix(contractHome),MSYS:'winsymlinks:nativestrict'}; +for(const script of ['fm-inbox.sh','fm-startup-network.sh']){ + const executable=posix(path.join(code,'bin',script)),help=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',executable,'--help'],{env:contractEnv,encoding:'utf8',timeout:30000}); + assert.equal(help.status,0,help.stderr);assert(help.stdout.includes('native-admission-predicate '),help.stdout); + const invalid=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',executable,'native-admission-predicate'],{env:contractEnv,encoding:'utf8',timeout:30000}); + assert.equal(invalid.status,2,invalid.stderr);assert(invalid.stderr.includes('native-admission-predicate '),invalid.stderr); +} +records.push('internal owner predicates publish and enforce their usage, output, and exit contracts'); function enqueue(home,message){ fs.mkdirSync(home,{recursive:true}); const env=Object.fromEntries(Object.entries(process.env).filter(([key])=>!key.startsWith('FM_')&&!key.startsWith('PI_'))); @@ -51,12 +68,7 @@ function appendStartupWake(home,state){ const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','. "$1/bin/fm-wake-lib.sh"; fm_wake_append_startup_network "$2"','startup-wake',posix(code),state],{env,encoding:'utf8',timeout:30000}); assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); } -function captureAcknowledgement(home){ - const env={...process.env,FM_HOME:home,MSYS:'winsymlinks:nativestrict'}; - const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',posix(path.join(code,'bin/native-owner/ack-evidence.sh')),'capture-json'],{env,encoding:'utf8',timeout:30000}); - assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); - return JSON.parse(result.stdout); -} +const journalRows=home=>fs.readFileSync(path.join(home,'owner-receipts.jsonl'),'utf8').trim().split(/\r?\n/).filter(Boolean).map(JSON.parse); const home=path.join(area,'home');fs.mkdirSync(path.join(home,'data'),{recursive:true}); fs.writeFileSync(path.join(home,'data/backlog.md'),'## In flight\n\n## Queued\n\n## Done\n'); const first=start(home);const initial=await ready(first);console.error('first ready',area); @@ -83,13 +95,31 @@ const populated=path.join(area,'populated');fs.mkdirSync(path.join(populated,'st const blocked=start(populated);blocked.child.stdin.end();assert.notEqual((await bound(blocked.done,blocked,20000)).exit,0); assert.equal(fs.readFileSync(path.join(populated,'state/work.meta'),'utf8'),'preserve');assert.equal(fs.existsSync(path.join(populated,'owner-probe.json')),false); records.push('populated home refused without changing its records'); -for(const [name,relative] of [['orphan-status','state/orphan.status'],['interrupted-close','state/orphan.backlog-close'],['residual-turn-end','state/orphan.turn-ended']]){ +for(const [name,relative] of [['orphan-status','state/orphan.status'],['interrupted-close','state/orphan.backlog-close'],['residual-turn-end','state/orphan.turn-ended'],['away-marker','state/.afk'],['away-contract','state/.afk-contract']]){ const residualHome=path.join(area,name),record=path.join(residualHome,relative),contents='preserve residual task state'; fs.mkdirSync(path.dirname(record),{recursive:true});fs.writeFileSync(record,contents); const refusedResidual=start(residualHome);refusedResidual.child.stdin.end();assert.notEqual((await bound(refusedResidual.done,refusedResidual,20000)).exit,0); assert.equal(fs.readFileSync(record,'utf8'),contents);assert.equal(fs.existsSync(path.join(residualHome,'owner-probe.json')),false); } -records.push('orphan status, interrupted-close, and turn-end records refused before lease acquisition and preserved'); +records.push('orphan status, interrupted-close, turn-end, and away records refused before lease acquisition and preserved'); +const namedHarness=path.join(area,'codex.exe'),holderPidFile=path.join(area,'ordinary-holder.pid'),liveLockHome=path.join(area,'live-lock'),liveLock=path.join(liveLockHome,'state/.lock'); +fs.copyFileSync('C:/Program Files/Git/usr/bin/sleep.exe',namedHarness);fs.mkdirSync(path.dirname(liveLock),{recursive:true}); +const holder=spawn('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','"$1" 60 & child=$!; printf "%s\\n" "$child" > "$2"; wait "$child"','lock-holder',posix(namedHarness),posix(holderPidFile)],{stdio:'ignore'}); +const holderDone=new Promise(resolve=>holder.on('exit',resolve)); +let holderPid; +try { + await waitUntil({child:holder,done:holderDone,home:liveLockHome},'ordinary lock holder',()=>{holderPid=fs.readFileSync(holderPidFile,'utf8').trim();return /^[0-9]+$/.test(holderPid);},10000); + const holderAlive=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','kill -0 "$1"','holder-check',holderPid],{encoding:'utf8',timeout:10000});assert.equal(holderAlive.status,0,holderAlive.stderr); + fs.writeFileSync(liveLock,holderPid+'\n'); + const locked=start(liveLockHome);locked.child.stdin.end();const lockedResult=await bound(locked.done,locked,20000); + assert.notEqual(lockedResult.exit,0);assert.equal(fs.readFileSync(liveLock,'utf8'),holderPid+'\n');assert.equal(fs.existsSync(path.join(liveLockHome,'owner-probe.json')),false); +}finally{ + if(holderPid)spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','kill "$1" 2>/dev/null || true','holder-stop',holderPid],{encoding:'utf8',timeout:10000}); + await Promise.race([holderDone,sleep(3000)]); +} +const unknownLockHome=path.join(area,'unknown-lock'),unknownLock=path.join(unknownLockHome,'state/.lock');fs.mkdirSync(path.dirname(unknownLock),{recursive:true});fs.writeFileSync(unknownLock,'native:'+'0'.repeat(32)+'\n'); +const unknownLocked=start(unknownLockHome);unknownLocked.child.stdin.end();assert.notEqual((await bound(unknownLocked.done,unknownLocked,20000)).exit,0);assert.equal(fs.readFileSync(unknownLock,'utf8'),'native:'+'0'.repeat(32)+'\n');assert.equal(fs.existsSync(path.join(unknownLockHome,'owner-probe.json')),false); +records.push('live ordinary and unresolved native session locks refuse launch before native ownership publication'); const queuedHome=path.join(area,'supported-queued-restart'),queuedNote=enqueue(queuedHome,'Preserve this supported notification across a native restart.'); const queuedBody=fs.readFileSync(path.join(queuedHome,'state/inbox',queuedNote+'.note'),'utf8'); const queuedSession=start(queuedHome);const queuedReady=await ready(queuedSession);queuedSession.child.stdin.end(); @@ -100,30 +130,52 @@ const queuedRestart=start(queuedHome);await ready(queuedRestart,queuedReady.owne assert.equal((await bound(queuedRestart.done,queuedRestart,20000)).exit,0); assert.equal(fs.readFileSync(path.join(queuedHome,'state/inbox',queuedNote+'.note'),'utf8'),queuedBody); records.push('producer-created inbox and native handling recovery remain admissible across restart'); +const uncorrelatedHome=path.join(area,'uncorrelated-acked'),uncorrelatedNote=enqueue(uncorrelatedHome,'Preserve this ordinary interrupted acknowledgement.'); +const uncorrelatedMarker=path.join(uncorrelatedHome,'state/.watcher-down'),uncorrelatedMarkerBefore=fs.readFileSync(uncorrelatedMarker,'utf8'); +const uncorrelatedAcked=uncorrelatedMarkerBefore.replace(/^(pending|announced):/,'acked:');assert.notEqual(uncorrelatedAcked,uncorrelatedMarkerBefore);fs.writeFileSync(uncorrelatedMarker,uncorrelatedAcked); +const uncorrelatedQueue=fs.readFileSync(path.join(uncorrelatedHome,'state/.wake-queue'),'utf8'),uncorrelatedBody=fs.readFileSync(path.join(uncorrelatedHome,'state/inbox',uncorrelatedNote+'.note'),'utf8'); +const uncorrelated=start(uncorrelatedHome);uncorrelated.child.stdin.end();assert.notEqual((await bound(uncorrelated.done,uncorrelated,20000)).exit,0); +assert.equal(fs.existsSync(path.join(uncorrelatedHome,'owner-probe.json')),false);assert.equal(fs.readFileSync(uncorrelatedMarker,'utf8'),uncorrelatedAcked);assert.equal(fs.readFileSync(path.join(uncorrelatedHome,'state/.wake-queue'),'utf8'),uncorrelatedQueue);assert.equal(fs.readFileSync(path.join(uncorrelatedHome,'state/inbox',uncorrelatedNote+'.note'),'utf8'),uncorrelatedBody); +records.push('uncorrelated acknowledged recovery is refused and preserved before lease acquisition'); const historicalHome=path.join(area,'historical-startup-completion'),historicalState=path.join(historicalHome,'state'),historicalStatus=path.join(historicalState,'.startup-network.status'); fs.mkdirSync(historicalState,{recursive:true});fs.writeFileSync(historicalStatus,'generation=historical\nstate=failed\n');appendStartupWake(historicalHome,'failed'); const historicalRow=fs.readFileSync(path.join(historicalState,'.wake-queue'),'utf8');fs.writeFileSync(historicalStatus,'generation=newer\nstate=done\n'); const historicalSession=start(historicalHome);await ready(historicalSession);historicalSession.child.stdin.end();assert.equal((await bound(historicalSession.done,historicalSession,20000)).exit,0); assert(fs.readFileSync(path.join(historicalState,'.wake-queue'),'utf8').includes(historicalRow.trim())); records.push('queued startup failure survives a newer startup status and remains admissible'); +const hostScript=path.join(code,'bin/native-owner/codex-host.mjs'),wakeDrain=path.join(code,'bin/fm-wake-drain.sh'),hostSource=fs.readFileSync(hostScript,'utf8'),wakeDrainSource=fs.readFileSync(wakeDrain,'utf8'); +fs.copyFileSync(path.join(repo,'tests/fixtures/native-owner/fake-app-server.mjs'),path.join(code,'bin/native-owner/fake-app-server.mjs')); +fs.writeFileSync(hostScript,"import {runCodexHost} from './codex-host-runtime.mjs';\nimport {createFakeAppServer} from './fake-app-server.mjs';\ntry { await runCodexHost({spawnAppServer:()=>createFakeAppServer('success'),mcpServerNames:[]}); } catch(error) { console.error(error.message); process.exitCode=1; }\n"); +const completedHome=path.join(area,'completed-ack-restart'),completedNote=enqueue(completedHome,'Complete this controlled acknowledgement before restart.'); +const completedSession=start(completedHome,false);const completedReady=await ready(completedSession); +await waitUntil(completedSession,'completed acknowledgement',()=>journalRows(completedHome).at(-1)?.event==='acknowledged'&&fs.existsSync(path.join(completedHome,'state/inbox/handled',completedNote+'.note'))&&fs.readFileSync(path.join(completedHome,'state/.wake-queue'),'utf8').trim()===''); +completedSession.child.stdin.write('/quit\n');assert.equal((await bound(completedSession.done,completedSession,20000)).exit,0); +const completedRestart=start(completedHome);await ready(completedRestart,completedReady.owner.generation);completedRestart.child.stdin.end();assert.equal((await bound(completedRestart.done,completedRestart,20000)).exit,0); +records.push('journal-correlated completed acknowledgement remains admissible across restart'); +const acknowledgementBoundary=' if ! _fm_atomic_replace "$DRAIN_TMP" "$FM_WAKE_QUEUE"; then'; +if(!wakeDrainSource.includes(acknowledgementBoundary))throw Error('Acknowledgement boundary fixture could not be installed'); +fs.writeFileSync(wakeDrain,wakeDrainSource.replace(acknowledgementBoundary,' printf "ready\\n" > "${FM_PROBE_HOME:?}/ack-boundary-ready"\n sleep 30\n'+acknowledgementBoundary)); const interruptedHome=path.join(area,'interrupted-ack-restart'),interruptedNote=enqueue(interruptedHome,'Preserve this interrupted acknowledgement for reconciliation.'); -const preparedSession=start(interruptedHome);const prepared=await ready(preparedSession);preparedSession.child.stdin.end();assert.equal((await bound(preparedSession.done,preparedSession,20000)).exit,0); -const target=captureAcknowledgement(interruptedHome),receipt='c'.repeat(32),payload={...target,challenge:'interrupted-restart',message:'preserved partial acknowledgement'}; -const journal=path.join(interruptedHome,'owner-receipts.jsonl'),journalRows=[ - {version:1,home:path.resolve(interruptedHome),generation:prepared.owner.generation,event:'presented',receipt,payload,targetEvidence:null,ackGeneration:null}, - {version:1,home:path.resolve(interruptedHome),generation:prepared.owner.generation,event:'ack-started',receipt,payload,targetEvidence:target.ownerEvidence,ackGeneration:null}, -]; -fs.appendFileSync(journal,journalRows.map(row=>JSON.stringify(row)).join('\n')+'\n'); -const interruptedQueue=path.join(interruptedHome,'state/.wake-queue'),queueBeforeRestart=fs.readFileSync(interruptedQueue,'utf8'),pendingNote=path.join(interruptedHome,'state/inbox',interruptedNote+'.note'),handledNote=path.join(interruptedHome,'state/inbox/handled',interruptedNote+'.note'); -fs.mkdirSync(path.dirname(handledNote),{recursive:true});fs.renameSync(pendingNote,handledNote);const handledBody=fs.readFileSync(handledNote,'utf8'); -const interruptedMarker=path.join(interruptedHome,'state/.watcher-down'),markerBody=`acked:handling:${target.generation}\n`;fs.writeFileSync(interruptedMarker,markerBody); +const interruptedQueue=path.join(interruptedHome,'state/.wake-queue'),queueBeforeRestart=fs.readFileSync(interruptedQueue,'utf8'),handledNote=path.join(interruptedHome,'state/inbox/handled',interruptedNote+'.note'),interruptedMarker=path.join(interruptedHome,'state/.watcher-down'); +let interruptedSession,interruptedReady,interruptedResult,markerBody,handledBody; +try { + interruptedSession=start(interruptedHome,false);interruptedReady=await ready(interruptedSession); + await waitUntil(interruptedSession,'acknowledgement marker boundary',()=>fs.existsSync(path.join(interruptedReady.runtime,'ack-boundary-ready'))); + assert.deepEqual(journalRows(interruptedHome).map(row=>row.event),['session','presented','ack-started']); + markerBody=fs.readFileSync(interruptedMarker,'utf8');assert.match(markerBody,/^acked:handling:[A-Za-z0-9._-]+\n$/); + handledBody=fs.readFileSync(handledNote,'utf8');assert.equal(fs.readFileSync(interruptedQueue,'utf8'),queueBeforeRestart); + interruptedSession.child.stdin.write('/quit\n');interruptedResult=await bound(interruptedSession.done,interruptedSession,20000);assert.equal(interruptedResult.exit,0,interruptedResult.stderr); +}finally{ + if(interruptedSession?.child.exitCode===null){interruptedSession.child.stdin.write('/quit\n');try{await bound(interruptedSession.done,interruptedSession,20000);}catch{}} + fs.writeFileSync(hostScript,hostSource);fs.writeFileSync(wakeDrain,wakeDrainSource); +} const interruptedRestart=start(interruptedHome);interruptedRestart.child.stdin.end();const reconciliation=await bound(interruptedRestart.done,interruptedRestart,20000); assert.notEqual(reconciliation.exit,0);assert(reconciliation.stderr.includes('earlier acknowledgement is incomplete'),reconciliation.stderr); -const reconciledOwner=read(path.join(interruptedHome,'owner-probe.json'));assert.notEqual(reconciledOwner.generation,prepared.owner.generation); +const reconciledOwner=read(path.join(interruptedHome,'owner-probe.json'));assert.notEqual(reconciledOwner.generation,interruptedReady.owner.generation); const reconciliationRuntime=path.join(interruptedHome,'state/native-runtime',reconciledOwner.generation); assert.equal(fs.existsSync(path.join(reconciliationRuntime,'startup.log')),false);assert.equal(fs.existsSync(path.join(reconciliationRuntime,'startup.finished')),false); assert.equal(fs.readFileSync(interruptedQueue,'utf8'),queueBeforeRestart);assert.equal(fs.readFileSync(interruptedMarker,'utf8'),markerBody);assert.equal(fs.readFileSync(handledNote,'utf8'),handledBody); -records.push('partial acknowledgement reaches reconciliation unchanged without restarting startup'); +records.push('real acknowledgement interrupted after marker commit reaches reconciliation unchanged without restarting startup'); const maskedHome=path.join(area,'bash-env-mask'),maskedStatus=path.join(maskedHome,'state/orphan.status'),mask=path.join(area,'bash-env-exit.sh'); fs.mkdirSync(path.dirname(maskedStatus),{recursive:true});fs.writeFileSync(maskedStatus,'preserve masked residual state');fs.writeFileSync(mask,'exit 0\n'); const masked=start(maskedHome,true,{...process.env,BASH_ENV:mask});masked.child.stdin.end();assert.notEqual((await bound(masked.done,masked,20000)).exit,0); diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 32201f794bc..1247a5c0179 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -106,51 +106,51 @@ static int EnvironmentTests() { Console.WriteLine("PASS: inherited Windows environment denylist is case-insensitive"); string residual=Path.Combine(directory,"residual"),residualState=Path.Combine(residual,"state"),status=Path.Combine(residualState,"orphan.status"); Directory.CreateDirectory(residualState);File.WriteAllText(status,"preserve\n"); - bool refused=false;try{EmptyFleet(residual);}catch(InvalidOperationException){refused=true;} + bool refused=false;try{EmptyFleet(residual,true);}catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(status)!="preserve\n")throw new InvalidOperationException("Mixed-case BASH_ENV bypassed empty-home admission"); Console.WriteLine("PASS: fixed Bash admission ignores mixed-case ambient authority"); string registered=Path.Combine(directory,"registered"),registeredState=Path.Combine(registered,"state"),canary=Path.Combine(registered,"executed"); Directory.CreateDirectory(registeredState); File.WriteAllText(Path.Combine(registeredState,"custom.check.sh"),"#!/usr/bin/env bash\nprintf executed > \""+canary.Replace('\\','/')+"\"\n"); File.WriteAllText(Path.Combine(registeredState,"custom.check-trust"),"fm-custom-check-v1\n"+new string('0',64)+"\n"); - refused=false;try{EmptyFleet(registered);using(var lease=new NativeHomeLease(registered)){} }catch(InvalidOperationException){refused=true;} + refused=false;try{EmptyFleet(registered,true);using(var lease=new NativeHomeLease(registered)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.Exists(canary)||File.Exists(Path.Combine(registered,"owner-probe.json"))||File.Exists(Path.Combine(registeredState,".watch.lock")))throw new InvalidOperationException("Registered custom work passed admission, acquired a lease, or executed during inspection"); Console.WriteLine("PASS: registered custom checks are refused without execution"); string unregistered=Path.Combine(directory,"unregistered"),unregisteredState=Path.Combine(unregistered,"state"),unregisteredCanary=Path.Combine(unregistered,"executed"),unregisteredCheck=Path.Combine(unregisteredState,"orphan.check.sh"),unregisteredBody="#!/usr/bin/env bash\nprintf executed > \""+unregisteredCanary.Replace('\\','/')+"\"\n"; Directory.CreateDirectory(unregisteredState);File.WriteAllText(unregisteredCheck,unregisteredBody); - refused=false;try{EmptyFleet(unregistered);using(var lease=new NativeHomeLease(unregistered)){} }catch(InvalidOperationException){refused=true;} + refused=false;try{EmptyFleet(unregistered,true);using(var lease=new NativeHomeLease(unregistered)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(unregisteredCheck)!=unregisteredBody||File.Exists(unregisteredCanary)||File.Exists(Path.Combine(unregistered,"owner-probe.json"))||File.Exists(Path.Combine(unregisteredState,".watch.lock")))throw new InvalidOperationException("Unregistered custom work passed admission, changed, acquired a lease, or executed during inspection"); Console.WriteLine("PASS: unregistered custom checks are preserved and refused without execution"); string pendingReply=Path.Combine(directory,"pending-reply"),pendingState=Path.Combine(pendingReply,"state"),pendingDirectory=Path.Combine(pendingState,"pending-replies"),pendingRecord=Path.Combine(pendingDirectory,"0123456789abcdef"),pendingBody="schema=fm-pending-reply.v1\nphase=awaiting_report\n"; Directory.CreateDirectory(pendingDirectory);File.WriteAllText(pendingRecord,pendingBody); - refused=false;try{EmptyFleet(pendingReply);using(var lease=new NativeHomeLease(pendingReply)){} }catch(InvalidOperationException){refused=true;} + refused=false;try{EmptyFleet(pendingReply,true);using(var lease=new NativeHomeLease(pendingReply)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(pendingRecord)!=pendingBody||File.Exists(Path.Combine(pendingReply,"owner-probe.json"))||File.Exists(Path.Combine(pendingState,".watch.lock"))||Directory.GetFiles(pendingState,"*",SearchOption.AllDirectories).Length!=1)throw new InvalidOperationException("Pending reply work passed admission, changed, or caused lease or route activity"); Console.WriteLine("PASS: pending replies are preserved without lease or route activity"); string reconcile=Path.Combine(directory,"reconcile-request"),reconcileState=Path.Combine(reconcile,"state"),reconcileDirectory=Path.Combine(reconcileState,"reconcile-notify"),reconcileRecord=Path.Combine(reconcileDirectory,"request-fixture.json"),reconcileBody="{\"version\":1}\n"; Directory.CreateDirectory(reconcileDirectory);File.WriteAllText(reconcileRecord,reconcileBody); - refused=false;try{EmptyFleet(reconcile);using(var lease=new NativeHomeLease(reconcile)){} }catch(InvalidOperationException){refused=true;} + refused=false;try{EmptyFleet(reconcile,true);using(var lease=new NativeHomeLease(reconcile)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(reconcileRecord)!=reconcileBody||File.Exists(Path.Combine(reconcile,"owner-probe.json"))||File.Exists(Path.Combine(reconcileState,".watch.lock"))||File.Exists(Path.Combine(reconcileState,".reconcile-notify-process.lock"))||Directory.GetFiles(reconcileState,"*",SearchOption.AllDirectories).Length!=1)throw new InvalidOperationException("Reconcile request passed admission, changed, or caused lease or route activity"); Console.WriteLine("PASS: reconcile requests are preserved without lease or route activity"); string handoff=Path.Combine(directory,"handoff"),outbox=Path.Combine(handoff,"data","handoff","agent.outbox.md"),outboxBody="# Backlog\n\n## Queued\n\n- [ ] routed-work\n"; Directory.CreateDirectory(Path.GetDirectoryName(outbox));File.WriteAllText(outbox,outboxBody); - refused=false;try{EmptyFleet(handoff);using(var lease=new NativeHomeLease(handoff)){} }catch(InvalidOperationException){refused=true;} + refused=false;try{EmptyFleet(handoff,true);using(var lease=new NativeHomeLease(handoff)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(outbox)!=outboxBody||File.Exists(Path.Combine(handoff,"owner-probe.json"))||Directory.Exists(Path.Combine(handoff,"state")))throw new InvalidOperationException("Pending handoff work passed admission or caused lease or route activity"); Console.WriteLine("PASS: pending handoff work is preserved without lease or route activity"); string steering=Path.Combine(directory,"steering"),steeringState=Path.Combine(steering,"state"),inbox=Path.Combine(steeringState,"agent.inbox"),message=Path.Combine(inbox,"001.msg"),messageBody="schema=fm-task-inbox.v1\n--\npreserve\n"; Directory.CreateDirectory(inbox);File.WriteAllText(message,messageBody); - refused=false;try{EmptyFleet(steering);using(var lease=new NativeHomeLease(steering)){} }catch(InvalidOperationException){refused=true;} + refused=false;try{EmptyFleet(steering,true);using(var lease=new NativeHomeLease(steering)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(message)!=messageBody||File.Exists(Path.Combine(steering,"owner-probe.json"))||File.Exists(Path.Combine(steeringState,".lock")))throw new InvalidOperationException("Orphan steering work passed admission or caused lease or route activity"); Console.WriteLine("PASS: orphan steering work is preserved without lease or route activity"); string turnEnded=Path.Combine(directory,"turn-ended"),turnEndedState=Path.Combine(turnEnded,"state"),turnEndedRecord=Path.Combine(turnEndedState,"orphan.turn-ended"); Directory.CreateDirectory(turnEndedState);File.WriteAllText(turnEndedRecord,"preserve\n"); - refused=false;try{EmptyFleet(turnEnded);using(var lease=new NativeHomeLease(turnEnded)){} }catch(InvalidOperationException){refused=true;} + refused=false;try{EmptyFleet(turnEnded,true);using(var lease=new NativeHomeLease(turnEnded)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(turnEndedRecord)!="preserve\n"||File.Exists(Path.Combine(turnEnded,"owner-probe.json"))||File.Exists(Path.Combine(turnEndedState,".lock")))throw new InvalidOperationException("Residual turn-end work passed admission or caused lease activity"); Console.WriteLine("PASS: residual turn-end work is preserved without lease activity"); string supported=Path.Combine(directory,"supported-notification"),supportedState=Path.Combine(supported,"state"),supportedInbox=Path.Combine(supportedState,"inbox"),supportedNote=Path.Combine(supportedInbox,"note-id.note"),supportedQueue=Path.Combine(supportedState,".wake-queue"),supportedMarker=Path.Combine(supportedState,".watcher-down"),supportedBody="preserve notification\n"; Directory.CreateDirectory(supportedInbox);File.WriteAllText(supportedNote,supportedBody);File.WriteAllText(Path.Combine(supportedState,".wake-queue.seq"),"1\n");File.WriteAllText(supportedQueue,"1\t1\tcheck\tinbox:note-id\tcheck: captain inbox note note-id - native admission test\n");File.WriteAllText(supportedMarker,"pending:downtime:supported\n"); - EmptyFleet(supported); + EmptyFleet(supported,true); File.WriteAllText(Path.Combine(supportedState,".main-eligible-rows"),"1\n");File.WriteAllText(supportedMarker,"pending:handling:supported\n"); - EmptyFleet(supported); + EmptyFleet(supported,true); if(File.ReadAllText(supportedNote)!=supportedBody||!File.ReadAllText(supportedQueue).Contains("inbox:note-id")||File.ReadAllText(supportedMarker)!="pending:handling:supported\n"||File.Exists(Path.Combine(supported,"owner-probe.json")))throw new InvalidOperationException("Supported top-level notification state was changed or leased during admission"); Console.WriteLine("PASS: supported top-level notification state remains admissible and unchanged"); foreach(string shape in new [] {"unsupported","malformed"}) { @@ -158,7 +158,7 @@ static int EnvironmentTests() { Directory.CreateDirectory(wakeState);File.WriteAllText(Path.Combine(wakeState,".wake-queue.seq"),"1\n");File.WriteAllText(wakeMarker,"pending:downtime:preserve\n"); File.WriteAllText(wakeQueue,shape=="unsupported" ? "1\t1\tcheck\torphan-work\tcheck: unsupported residual work\n" : "malformed wake row\n"); string queueBefore=File.ReadAllText(wakeQueue),markerBefore=File.ReadAllText(wakeMarker); - refused=false;try{EmptyFleet(wakeHome);using(var lease=new NativeHomeLease(wakeHome)){} }catch(InvalidOperationException){refused=true;} + refused=false;try{EmptyFleet(wakeHome,true);using(var lease=new NativeHomeLease(wakeHome)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(wakeQueue)!=queueBefore||File.ReadAllText(wakeMarker)!=markerBefore||File.Exists(Path.Combine(wakeHome,"owner-probe.json")))throw new InvalidOperationException("Unsupported wake state passed admission, changed, or acquired a lease"); } Console.WriteLine("PASS: unsupported and malformed wake state is preserved and refused before lease acquisition"); From de8ee11ae22d9186eac673ddcc1eab074494c994 Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 11:21:23 +1200 Subject: [PATCH 25/61] no-mistakes(review): Fix native ownership; Git Bash verification permission-blocked --- bin/fm-lock.sh | 35 ++++++++-- bin/fm-wake-lib.sh | 34 +++++++++ bin/native-owner/NativeHomeLease.cs | 70 ++++++++++++++----- bin/native-owner/NativeLauncher.cs | 5 +- bin/native-owner/NativeOperations.cs | 7 +- bin/native-owner/ack-evidence.sh | 38 +--------- bin/native-owner/admit.sh | 2 +- docs/native-windows-codex.md | 12 ++-- tests/fixtures/native-owner/Launcher.mjs | 15 ++-- tests/fixtures/native-owner/NativeDriver.cs | 14 ++++ tests/fixtures/native-owner/ReceiptTests.cs | 10 +++ .../native-owner/fm-wake-lib-interrupt.sh | 17 +++++ 12 files changed, 185 insertions(+), 74 deletions(-) create mode 100644 tests/fixtures/native-owner/fm-wake-lib-interrupt.sh diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index 33cf7a34bd1..fc80b9a4507 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -13,10 +13,6 @@ FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" LOCK="$STATE/.lock" -mkdir -p "$STATE" 2>/dev/null || { - echo "error: cannot create session-lock state directory $STATE; operate read-only until resolved" >&2 - exit 1 -} # Harness identity (FM_HARNESS_RE, ancestry walk, holder liveness) is owned by # the shared session-lock lib so the Claude Stop auto-arm applies the exact @@ -53,6 +49,19 @@ fm_lock_conflict_message() { return 1 } +fm_lock_native_admission_proves_dead() { + local wanted=${1#native:} list=${FM_NATIVE_PROVEN_DEAD_GENERATIONS:-} generation found=1 + local IFS=, + [ -n "$list" ] || return 1 + case "$list" in ,*|*,|*,,*) return 2 ;; esac + for generation in $list; do + [ "${#generation}" -eq 32 ] || return 2 + case "$generation" in *[!0-9a-f]*) return 2 ;; esac + [ "$generation" != "$wanted" ] || found=0 + done + return "$found" +} + if [ "${1:-}" = "native-admission-predicate" ]; then [ "$#" -eq 1 ] || { echo "usage: fm-lock.sh native-admission-predicate" >&2 @@ -73,6 +82,19 @@ if [ "${1:-}" = "native-admission-predicate" ]; then echo "error: session lock owner is unrecognized; native launch refused" >&2 exit 1 fi + case "$old" in + native:*) + if fm_lock_native_admission_proves_dead "$old"; then + exit 0 + else + dead_rc=$? + fi + if [ "$dead_rc" -eq 2 ]; then + echo "error: native dead-generation evidence is unrecognized; native launch refused" >&2 + exit 1 + fi + ;; + esac if fm_harness_pid_excludes "$old"; then if conflict=$(fm_lock_conflict_message "$old"); then echo "$conflict" >&2 @@ -84,6 +106,11 @@ if [ "${1:-}" = "native-admission-predicate" ]; then exit 0 fi +mkdir -p "$STATE" 2>/dev/null || { + echo "error: cannot create session-lock state directory $STATE; operate read-only until resolved" >&2 + exit 1 +} + if [ "${1:-}" = "status" ]; then if [ ! -f "$LOCK" ]; then echo "lock: free"; exit 0; fi old=$(cat "$LOCK" 2>/dev/null) || { diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 9741b474df8..d73f4bd8fc6 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -2357,6 +2357,40 @@ fm_wake_ack_evidence_clear() { FM_WAKE_ACK_EVIDENCE_LEGACY=0 } +fm_wake_ack_evidence_legacy_token() { + node -e ' + let input=""; + process.stdin.setEncoding("utf8"); + process.stdin.on("data",chunk=>input+=chunk); + process.stdin.on("end",()=>{ + const value=JSON.parse(input); + if(!value||typeof value!=="object"||Array.isArray(value))throw Error("legacy evidence must be an object"); + const version=Number(value.version); + if(![1,2,3].includes(version))throw Error("unsupported legacy evidence"); + if(typeof value.cutoff!=="string")throw Error("invalid legacy cutoff"); + const cutoff=String(value.cutoff); + const rows=value.rows; + if(!Array.isArray(rows)||rows.some(row=>typeof row!=="string"))throw Error("invalid legacy rows"); + let notes=[]; + if(version===1)notes=[value]; + else notes=value.notes; + if(!Array.isArray(notes)||notes.some(note=>!note||typeof note!=="object"||Array.isArray(note)))throw Error("invalid legacy notes"); + if(version===3&&(cutoff!=="0"||rows.length!==0||notes.length!==0))throw Error("invalid legacy recovery target"); + const generation=version===3?value.recoveryGeneration:"legacy"; + const marker=version===3?value.recoveryMarker:""; + if(typeof generation!=="string"||typeof marker!=="string")throw Error("invalid legacy recovery evidence"); + const line=["fm-wake-ack-evidence-legacy-v1",`cutoff\t${cutoff}`,`generation\t${generation}`,`marker\t${Buffer.from(marker).toString("base64")}`,`rows\t${rows.length}`]; + for(const row of rows)line.push(`row\t${Buffer.from(row).toString("base64")}`); + line.push(`notes\t${notes.length}`); + for(const note of notes){ + if(typeof note.note!=="string"||typeof note.noteSha256!=="string")throw Error("invalid legacy note"); + line.push(`note\t${note.note}\t${note.noteSha256}`); + } + process.stdout.write("legacy."+Buffer.from(line.join("\n")+"\n").toString("base64")); + }); + ' +} + fm_wake_ack_evidence_capture() { local rows_file="$STATE/.main-eligible-rows" marker="$STATE/.watcher-down" local payload seq_count note_count note id digest encoded diff --git a/bin/native-owner/NativeHomeLease.cs b/bin/native-owner/NativeHomeLease.cs index 66c38a5feb5..631182900e0 100644 --- a/bin/native-owner/NativeHomeLease.cs +++ b/bin/native-owner/NativeHomeLease.cs @@ -23,18 +23,38 @@ static bool Generation(string value) { } public bool ProvenDeadGeneration(string value) { return IsHeld&&deadGenerations.Contains(value); } [StructLayout(LayoutKind.Sequential)] struct FT { public uint low,high; } + [StructLayout(LayoutKind.Sequential)] struct Info { + public uint attributes,createdLow,createdHigh,accessLow,accessHigh,writeLow,writeHigh; + public uint volume,sizeHigh,sizeLow,links,indexHigh,indexLow; + } [DllImport("kernel32.dll",SetLastError=true)] static extern IntPtr OpenProcess(uint access,bool inherit,uint pid); [DllImport("kernel32.dll",SetLastError=true)] static extern bool GetProcessTimes(IntPtr process,out FT created,out FT exited,out FT kernel,out FT user); [DllImport("kernel32.dll",SetLastError=true)] static extern uint WaitForSingleObject(IntPtr handle,uint milliseconds); [DllImport("kernel32.dll")] static extern bool CloseHandle(IntPtr handle); - static string Filename(string home) { + [DllImport("kernel32.dll",SetLastError=true)] static extern bool GetFileInformationByHandle(IntPtr handle,out Info info); + public static string ValidateHomePath(string home) { string full=Path.GetFullPath(home); string temporary=Path.GetFullPath(Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData),"Temp")).TrimEnd('\\','/')+Path.DirectorySeparatorChar; if(!full.StartsWith(temporary,StringComparison.OrdinalIgnoreCase)) throw new InvalidOperationException("Experimental leases require a home beneath the user's Windows temporary directory"); for(string parent=full;parent!=null;parent=Path.GetDirectoryName(parent)) { if(Directory.Exists(parent)&&(File.GetAttributes(parent)&FileAttributes.ReparsePoint)!=0) throw new InvalidOperationException("Reparse-point homes are not supported"); } - return Path.Combine(full,"owner-probe.json"); + return full; + } + static string Filename(string home) { return Path.Combine(ValidateHomePath(home),"owner-probe.json"); } + static void ValidateFile(FileStream stream,SecurityIdentifier user) { + var access=stream.GetAccessControl(); + if(!access.AreAccessRulesProtected || !access.GetOwner(typeof(SecurityIdentifier)).Equals(user)) throw new IOException("Owner record security differs; preserved"); + foreach(FileSystemAccessRule rule in access.GetAccessRules(true,true,typeof(SecurityIdentifier))) if(rule.AccessControlType==AccessControlType.Allow && !rule.IdentityReference.Equals(user)) throw new IOException("Owner record grants unexpected access; preserved"); + Info info; + if(!GetFileInformationByHandle(stream.SafeFileHandle.DangerousGetHandle(),out info) || info.links!=1 || (info.attributes&0x400)!=0) throw new IOException("Owner record file identity is unsafe; preserved"); + } + static FileStream OpenExisting(string name,FileSystemRights rights,FileAccess access,FileShare share) { + FileAttributes attributes=File.GetAttributes(name); + if((attributes&FileAttributes.ReparsePoint)!=0 || (attributes&FileAttributes.Directory)!=0) throw new IOException("Owner record path is unsafe; preserved"); + FileStream stream=rights==0 ? new FileStream(name,FileMode.Open,access,share) : new FileStream(name,FileMode.Open,rights,share,4096,FileOptions.None); + try { ValidateFile(stream,WindowsIdentity.GetCurrent().User);return stream; } + catch { stream.Dispose();throw; } } static Dictionary Read(Stream stream) { stream.Position=0; @@ -61,30 +81,48 @@ static bool RootAlive(Dictionary record) { return true; } finally { CloseHandle(handle); } } + static void CollectDeadGenerations(Dictionary record,HashSet generations) { + string previousGeneration=record.ContainsKey("generation") ? (string)record["generation"] : null; + if(!Generation(previousGeneration))throw new InvalidOperationException("Invalid predecessor generation; preserved"); + if(record.ContainsKey("deadGenerations")) { + var prior=record["deadGenerations"] as System.Collections.IList; + if(prior==null)throw new InvalidOperationException("Invalid predecessor history; preserved"); + foreach(object item in prior){string value=item as string;if(!Generation(value))throw new InvalidOperationException("Invalid predecessor history; preserved");generations.Add(value);} + } + generations.Add(previousGeneration); + if(generations.Count>256)throw new InvalidOperationException("Predecessor history requires maintenance; preserved"); + } + public static string[] ProvenDeadGenerationsForAdmission(string home) { + string name=Filename(home); + try { + using(var reader=OpenExisting(name,0,FileAccess.Read,FileShare.ReadWrite)) { + if(reader.Length==0)throw new InvalidOperationException("Empty existing owner record is ambiguous; preserved"); + var previous=Read(reader); + if(RootAlive(previous))return new string[0]; + var generations=new HashSet();CollectDeadGenerations(previous,generations); + var result=new string[generations.Count];generations.CopyTo(result);return result; + } + } catch(FileNotFoundException) { return new string[0]; } + catch(DirectoryNotFoundException) { return new string[0]; } + } public NativeHomeLease(string home) { - Home=Path.GetFullPath(home); + Home=ValidateHomePath(home); string name=Filename(Home); Directory.CreateDirectory(Home); var security=new FileSecurity(); security.SetAccessRuleProtection(true,false); - security.AddAccessRule(new FileSystemAccessRule(WindowsIdentity.GetCurrent().User,FileSystemRights.FullControl,AccessControlType.Allow)); + var user=WindowsIdentity.GetCurrent().User;security.SetOwner(user); + security.AddAccessRule(new FileSystemAccessRule(user,FileSystemRights.FullControl,AccessControlType.Allow)); // The OS arbitrates concurrent controllers before either reads or writes. bool created=false; - try { file=new FileStream(name,FileMode.CreateNew,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security); created=true; } - catch(IOException) { file=new FileStream(name,FileMode.Open,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security); } try { + try { file=new FileStream(name,FileMode.CreateNew,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security);created=true; } + catch(IOException) { file=OpenExisting(name,FileSystemRights.Read|FileSystemRights.Write,0,FileShare.Read); } + ValidateFile(file,user); if(!created && file.Length==0) throw new InvalidOperationException("Empty existing owner record is ambiguous; preserved"); if(file.Length>0) { var previous=Read(file); if(RootAlive(previous)) throw new InvalidOperationException("Recorded primary is still alive; refusing replacement"); - string previousGeneration=previous.ContainsKey("generation") ? (string)previous["generation"] : null; - if(!Generation(previousGeneration))throw new InvalidOperationException("Invalid predecessor generation; preserved"); - if(previous.ContainsKey("deadGenerations")) { - var prior=previous["deadGenerations"] as System.Collections.IList; - if(prior==null)throw new InvalidOperationException("Invalid predecessor history; preserved"); - foreach(object item in prior){string value=item as string;if(!Generation(value))throw new InvalidOperationException("Invalid predecessor history; preserved");deadGenerations.Add(value);} - } - deadGenerations.Add(previousGeneration); - if(deadGenerations.Count>256)throw new InvalidOperationException("Predecessor history requires maintenance; preserved"); + CollectDeadGenerations(previous,deadGenerations); } Write(new Dictionary{{"state","pending"},{"controllerPid",Process.GetCurrentProcess().Id}}); } catch { Dispose(); throw; } @@ -100,7 +138,7 @@ public void Publish(uint pid,ulong created,string generation,string pipe) { public static Dictionary Binding(string state) { string canonical=Path.GetFullPath(state).TrimEnd(Path.DirectorySeparatorChar,Path.AltDirectorySeparatorChar); if(!string.Equals(Path.GetFileName(canonical),"state",StringComparison.OrdinalIgnoreCase)) throw new InvalidOperationException("Unexpected state directory"); - using(var reader=new FileStream(Filename(Path.GetDirectoryName(canonical)),FileMode.Open,FileAccess.Read,FileShare.ReadWrite)) { + using(var reader=OpenExisting(Filename(Path.GetDirectoryName(canonical)),0,FileAccess.Read,FileShare.ReadWrite)) { var record=Read(reader); if(!RootAlive(record)) throw new InvalidOperationException("Primary no longer live"); return record; diff --git a/bin/native-owner/NativeLauncher.cs b/bin/native-owner/NativeLauncher.cs index 3f5c6a095a8..d7f61897a3a 100644 --- a/bin/native-owner/NativeLauncher.cs +++ b/bin/native-owner/NativeLauncher.cs @@ -13,10 +13,11 @@ public static partial class NativeOwner { [DllImport("kernel32.dll")] static extern IntPtr GetStdHandle(int kind); static int Launch(string selectedHome) { - string home=Path.GetFullPath(selectedHome), node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); + string home=NativeHomeLease.ValidateHomePath(selectedHome), node=Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),@"nodejs\node.exe"); string host=Path.Combine(CodeRoot,"bin","native-owner","codex-host.mjs"); if(!File.Exists(node)||!File.Exists(host)) throw new IOException("Native Node or the code-owned host is missing"); - EmptyFleet(home,true); + string[] provenDeadGenerations=NativeHomeLease.ProvenDeadGenerationsForAdmission(home); + EmptyFleet(home,true,provenDeadGenerations); string session=Guid.NewGuid().ToString("N"),nonce=Guid.NewGuid().ToString("N"),pipeName="fm-native-"+session; IntPtr job=CreateJobObject(IntPtr.Zero,null),env=IntPtr.Zero,output=IntPtr.Zero,input=IntPtr.Zero; if(job==IntPtr.Zero) throw Error("Create session job"); diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index d1dc4a60126..917e19892ab 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -31,10 +31,11 @@ internal static ProcessStartInfo BashHelper(string script,string arguments,strin start.EnvironmentVariables["MSYS"]="winsymlinks:nativestrict"; return start; } - static void OwnerAdmission(string home,bool launch) { + static void OwnerAdmission(string home,bool launch,string[] provenDeadGenerations) { string script=Path.Combine(CodeRoot,"bin","native-owner","admit.sh"); object evidence=NativeReceiptJournal.AdmissionEvidence(home); var start=BashHelper(script,launch ? "launch" : "owned-operation",home);start.RedirectStandardInput=true;start.RedirectStandardError=true; + if(launch && provenDeadGenerations!=null && provenDeadGenerations.Length>0)start.EnvironmentVariables["FM_NATIVE_PROVEN_DEAD_GENERATIONS"]=string.Join(",",provenDeadGenerations); string input=""; if(evidence is string) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="token";input=(string)evidence;} else if(evidence!=null) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="legacy";input=Json.Serialize(evidence);} @@ -45,10 +46,10 @@ static void OwnerAdmission(string home,bool launch) { if(process.ExitCode!=0)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); } } - internal static void EmptyFleet(string home,bool launch) { + internal static void EmptyFleet(string home,bool launch,string[] provenDeadGenerations=null) { string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); if((Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); - OwnerAdmission(home,launch); + OwnerAdmission(home,launch,provenDeadGenerations); } static IntPtr FileHandle(string path, uint access, uint creation) { SA sa = new SA { length=Marshal.SizeOf(typeof(SA)), inherit=1 }; diff --git a/bin/native-owner/ack-evidence.sh b/bin/native-owner/ack-evidence.sh index 5ef1ec1f211..e731ac673a0 100644 --- a/bin/native-owner/ack-evidence.sh +++ b/bin/native-owner/ack-evidence.sh @@ -16,40 +16,6 @@ read_token() { printf '%s' "$token" } -legacy_token() { - node -e ' - let input=""; - process.stdin.setEncoding("utf8"); - process.stdin.on("data",chunk=>input+=chunk); - process.stdin.on("end",()=>{ - const value=JSON.parse(input); - if(!value||typeof value!=="object"||Array.isArray(value))throw Error("legacy evidence must be an object"); - const version=Number(value.version); - if(![1,2,3].includes(version))throw Error("unsupported legacy evidence"); - if(typeof value.cutoff!=="string")throw Error("invalid legacy cutoff"); - const cutoff=String(value.cutoff); - const rows=value.rows; - if(!Array.isArray(rows)||rows.some(row=>typeof row!=="string"))throw Error("invalid legacy rows"); - let notes=[]; - if(version===1)notes=[value]; - else notes=value.notes; - if(!Array.isArray(notes)||notes.some(note=>!note||typeof note!=="object"||Array.isArray(note)))throw Error("invalid legacy notes"); - if(version===3&&(cutoff!=="0"||rows.length!==0||notes.length!==0))throw Error("invalid legacy recovery target"); - const generation=version===3?value.recoveryGeneration:"legacy"; - const marker=version===3?value.recoveryMarker:""; - if(typeof generation!=="string"||typeof marker!=="string")throw Error("invalid legacy recovery evidence"); - const line=["fm-wake-ack-evidence-legacy-v1",`cutoff\t${cutoff}`,`generation\t${generation}`,`marker\t${Buffer.from(marker).toString("base64")}`,`rows\t${rows.length}`]; - for(const row of rows)line.push(`row\t${Buffer.from(row).toString("base64")}`); - line.push(`notes\t${notes.length}`); - for(const note of notes){ - if(typeof note.note!=="string"||typeof note.noteSha256!=="string")throw Error("invalid legacy note"); - line.push(`note\t${note.note}\t${note.noteSha256}`); - } - process.stdout.write("legacy."+Buffer.from(line.join("\n")+"\n").toString("base64")); - }); - ' -} - case "${1:-}" in capture-json) fm_wake_ack_evidence_capture @@ -65,7 +31,7 @@ case "${1:-}" in fm_wake_ack_evidence_clear ;; legacy-token) - legacy_token + fm_wake_ack_evidence_legacy_token ;; preflight-token) token=$(read_token) @@ -77,7 +43,7 @@ case "${1:-}" in fm_wake_ack_evidence_completed "$token" ;; verify-legacy) - token=$(legacy_token) + token=$(fm_wake_ack_evidence_legacy_token) fm_wake_ack_evidence_completed "$token" ;; acknowledge-token) diff --git a/bin/native-owner/admit.sh b/bin/native-owner/admit.sh index 1a8bea25f47..9b3d15e0b78 100644 --- a/bin/native-owner/admit.sh +++ b/bin/native-owner/admit.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Usage: FM_HOME= admit.sh launch|owned-operation -# Required environment: FM_HOME; acknowledgement evidence is read from standard input. +# Required environment: FM_HOME; standard input must be empty with FM_NATIVE_ACK_EVIDENCE_KIND unset, or nonempty with the discriminator set to token|legacy. set -euo pipefail ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) mode=${1:-} diff --git a/docs/native-windows-codex.md b/docs/native-windows-codex.md index 0f36929f357..8bc2529befd 100644 --- a/docs/native-windows-codex.md +++ b/docs/native-windows-codex.md @@ -26,35 +26,35 @@ Use `/interrupt` to interrupt the current model turn and `/quit` to end the sess Run the portable request-policy regression with: ```sh -bash tests/fm-native-owner-tool-gate.test.sh +bin/fm-test-run.sh tests/fm-native-owner-tool-gate.test.sh ``` Run native receipt persistence and operation-lifetime checks with: ```sh -bash tests/fm-native-owner-receipt-live-e2e.test.sh +bin/fm-test-run.sh tests/fm-native-owner-receipt-live-e2e.test.sh ``` Run effective app and MCP isolation without a model turn with: ```sh -FM_LIVE_NATIVE_APP_POLICY=1 bash tests/fm-native-owner-app-server-policy-live-e2e.test.sh +FM_LIVE_NATIVE_APP_POLICY=1 bin/fm-test-run.sh tests/fm-native-owner-app-server-policy-live-e2e.test.sh ``` Run the actual Windows integration with the explicit two-model-turn guard with: ```sh FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_CODEX=1 \ - bash tests/fm-native-owner-codex-live-e2e.test.sh + bin/fm-test-run.sh tests/fm-native-owner-codex-live-e2e.test.sh ``` Run the explicit launcher without model turns, then optionally exercise two notification turns and active-turn cancellation with: ```sh FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 \ - bash tests/fm-native-owner-launcher-live-e2e.test.sh + bin/fm-test-run.sh tests/fm-native-owner-launcher-live-e2e.test.sh FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 FM_LIVE_NATIVE_CODEX=1 \ - bash tests/fm-native-owner-launcher-live-e2e.test.sh + bin/fm-test-run.sh tests/fm-native-owner-launcher-live-e2e.test.sh ``` ## Safety boundary and limits diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 690faac7459..a282bdf0d4d 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -143,7 +143,7 @@ const historicalRow=fs.readFileSync(path.join(historicalState,'.wake-queue'),'ut const historicalSession=start(historicalHome);await ready(historicalSession);historicalSession.child.stdin.end();assert.equal((await bound(historicalSession.done,historicalSession,20000)).exit,0); assert(fs.readFileSync(path.join(historicalState,'.wake-queue'),'utf8').includes(historicalRow.trim())); records.push('queued startup failure survives a newer startup status and remains admissible'); -const hostScript=path.join(code,'bin/native-owner/codex-host.mjs'),wakeDrain=path.join(code,'bin/fm-wake-drain.sh'),hostSource=fs.readFileSync(hostScript,'utf8'),wakeDrainSource=fs.readFileSync(wakeDrain,'utf8'); +const hostScript=path.join(code,'bin/native-owner/codex-host.mjs'),hostSource=fs.readFileSync(hostScript,'utf8'); fs.copyFileSync(path.join(repo,'tests/fixtures/native-owner/fake-app-server.mjs'),path.join(code,'bin/native-owner/fake-app-server.mjs')); fs.writeFileSync(hostScript,"import {runCodexHost} from './codex-host-runtime.mjs';\nimport {createFakeAppServer} from './fake-app-server.mjs';\ntry { await runCodexHost({spawnAppServer:()=>createFakeAppServer('success'),mcpServerNames:[]}); } catch(error) { console.error(error.message); process.exitCode=1; }\n"); const completedHome=path.join(area,'completed-ack-restart'),completedNote=enqueue(completedHome,'Complete this controlled acknowledgement before restart.'); @@ -152,9 +152,8 @@ await waitUntil(completedSession,'completed acknowledgement',()=>journalRows(com completedSession.child.stdin.write('/quit\n');assert.equal((await bound(completedSession.done,completedSession,20000)).exit,0); const completedRestart=start(completedHome);await ready(completedRestart,completedReady.owner.generation);completedRestart.child.stdin.end();assert.equal((await bound(completedRestart.done,completedRestart,20000)).exit,0); records.push('journal-correlated completed acknowledgement remains admissible across restart'); -const acknowledgementBoundary=' if ! _fm_atomic_replace "$DRAIN_TMP" "$FM_WAKE_QUEUE"; then'; -if(!wakeDrainSource.includes(acknowledgementBoundary))throw Error('Acknowledgement boundary fixture could not be installed'); -fs.writeFileSync(wakeDrain,wakeDrainSource.replace(acknowledgementBoundary,' printf "ready\\n" > "${FM_PROBE_HOME:?}/ack-boundary-ready"\n sleep 30\n'+acknowledgementBoundary)); +const wakeLib=path.join(code,'bin/fm-wake-lib.sh'),wakeLibActual=wakeLib+'.actual'; +fs.renameSync(wakeLib,wakeLibActual);fs.copyFileSync(path.join(repo,'tests/fixtures/native-owner/fm-wake-lib-interrupt.sh'),wakeLib); const interruptedHome=path.join(area,'interrupted-ack-restart'),interruptedNote=enqueue(interruptedHome,'Preserve this interrupted acknowledgement for reconciliation.'); const interruptedQueue=path.join(interruptedHome,'state/.wake-queue'),queueBeforeRestart=fs.readFileSync(interruptedQueue,'utf8'),handledNote=path.join(interruptedHome,'state/inbox/handled',interruptedNote+'.note'),interruptedMarker=path.join(interruptedHome,'state/.watcher-down'); let interruptedSession,interruptedReady,interruptedResult,markerBody,handledBody; @@ -167,7 +166,7 @@ try { interruptedSession.child.stdin.write('/quit\n');interruptedResult=await bound(interruptedSession.done,interruptedSession,20000);assert.equal(interruptedResult.exit,0,interruptedResult.stderr); }finally{ if(interruptedSession?.child.exitCode===null){interruptedSession.child.stdin.write('/quit\n');try{await bound(interruptedSession.done,interruptedSession,20000);}catch{}} - fs.writeFileSync(hostScript,hostSource);fs.writeFileSync(wakeDrain,wakeDrainSource); + fs.writeFileSync(hostScript,hostSource);fs.unlinkSync(wakeLib);fs.renameSync(wakeLibActual,wakeLib); } const interruptedRestart=start(interruptedHome);interruptedRestart.child.stdin.end();const reconciliation=await bound(interruptedRestart.done,interruptedRestart,20000); assert.notEqual(reconciliation.exit,0);assert(reconciliation.stderr.includes('earlier acknowledgement is incomplete'),reconciliation.stderr); @@ -220,7 +219,11 @@ assert.equal(fs.existsSync(outside),false); const external=start(outside);external.child.stdin.end();assert.notEqual((await bound(external.done,external,20000)).exit,0);assert.equal(fs.existsSync(outside),false); const target=path.join(area,'junction-target'),junction=path.join(area,'junction');fs.mkdirSync(target);fs.symlinkSync(target,junction,'junction'); const linked=start(path.join(junction,'home'));linked.child.stdin.end();assert.notEqual((await bound(linked.done,linked,20000)).exit,0);assert.equal(fs.existsSync(path.join(target,'home')),false); -records.push('non-temporary and pre-existing reparse-point homes refused before creation'); +const externalOwner=path.join(area,'external-owner.json'),linkedOwnerHome=path.join(area,'linked-owner'),linkedOwner=path.join(linkedOwnerHome,'owner-probe.json'); +const externalOwnerBody=JSON.stringify({deadGenerations:[],state:'live',rootPid:4294967295,rootCreationFileTime:0,generation:'d'.repeat(32),pipe:'unreachable',controllerPid:4294967295,controllerCreated:0}); +fs.mkdirSync(linkedOwnerHome);fs.writeFileSync(externalOwner,externalOwnerBody);fs.linkSync(externalOwner,linkedOwner); +const linkedOwnerSession=start(linkedOwnerHome);linkedOwnerSession.child.stdin.end();assert.notEqual((await bound(linkedOwnerSession.done,linkedOwnerSession,20000)).exit,0);assert.equal(fs.readFileSync(externalOwner,'utf8'),externalOwnerBody); +records.push('non-temporary, reparse-point, and hard-linked owner homes are refused without external changes'); // Delay the existing network owner only in this disposable code copy. The // production launcher has no delay/mock switch and still invokes that owner. const network=path.join(code,'bin/fm-startup-network.sh');fs.renameSync(network,network+'.actual'); diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 1247a5c0179..75e314e0291 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -109,6 +109,20 @@ static int EnvironmentTests() { bool refused=false;try{EmptyFleet(residual,true);}catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(status)!="preserve\n")throw new InvalidOperationException("Mixed-case BASH_ENV bypassed empty-home admission"); Console.WriteLine("PASS: fixed Bash admission ignores mixed-case ambient authority"); + string restart=Path.Combine(directory,"dead-owner-restart"),restartState=Path.Combine(restart,"state"),restartGeneration=new string('d',32); + uint departedPid;ulong departedCreated; + using(var departed=Process.Start(new ProcessStartInfo("cmd.exe","/c exit 0") {UseShellExecute=false,CreateNoWindow=true})) { + departedPid=(uint)departed.Id;departedCreated=(ulong)departed.StartTime.ToUniversalTime().ToFileTimeUtc();departed.WaitForExit(); + } + using(var predecessor=new NativeHomeLease(restart))predecessor.Publish(departedPid,departedCreated,restartGeneration,"unreachable"); + Directory.CreateDirectory(restartState);File.WriteAllText(Path.Combine(restartState,".lock"),"native:"+restartGeneration+"\n"); + string[] deadProof=NativeHomeLease.ProvenDeadGenerationsForAdmission(restart); + if(Array.IndexOf(deadProof,restartGeneration)<0)throw new InvalidOperationException("Exact departed owner was not proven dead"); + refused=false;try{EmptyFleet(restart,true,new [] {new string('e',32)});}catch(InvalidOperationException){refused=true;} + if(!refused)throw new InvalidOperationException("Unrelated dead-generation claim bypassed native owner exclusion"); + EmptyFleet(restart,true,deadProof); + using(var replacement=new NativeHomeLease(restart))if(!replacement.ProvenDeadGeneration(restartGeneration))throw new InvalidOperationException("Replacement lease lost proven-dead ownership history"); + Console.WriteLine("PASS: exact dead native generation admits restart while an unreachable endpoint alone remains exclusionary"); string registered=Path.Combine(directory,"registered"),registeredState=Path.Combine(registered,"state"),canary=Path.Combine(registered,"executed"); Directory.CreateDirectory(registeredState); File.WriteAllText(Path.Combine(registeredState,"custom.check.sh"),"#!/usr/bin/env bash\nprintf executed > \""+canary.Replace('\\','/')+"\"\n"); diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index 9d2d82c39d5..ad691e21382 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -20,6 +20,15 @@ static void Case(string name,Action test) { using(var lease=new NativeHomeLease(home)) test(lease); passed++;Console.WriteLine("PASS: "+name); } + static void OwnerProbeLinkCase() { + string root=Path.Combine(Path.GetTempPath(),"fm-owner-link-"+Guid.NewGuid().ToString("N")),home=Path.Combine(root,"home"),external=Path.Combine(root,"external-owner.json"),owner=Path.Combine(home,"owner-probe.json"); + Directory.CreateDirectory(home); + string body=Json.Serialize(new Dictionary{{"deadGenerations",new string[0]},{"state","live"},{"rootPid",uint.MaxValue},{"rootCreationFileTime",0L},{"generation",A},{"pipe","unreachable"},{"controllerPid",uint.MaxValue},{"controllerCreated",0L}}); + File.WriteAllText(external,body);Expect(CreateHardLink(owner,external,IntPtr.Zero),"Owner hard-link fixture failed"); + Refuses(()=>{using(var lease=new NativeHomeLease(home)){}},"Hard-linked owner record accepted"); + Expect(File.ReadAllText(external)==body,"Hard-linked external owner record changed"); + passed++;Console.WriteLine("PASS: hard-linked owner record is preserved and refused"); + } static string Queue(NativeHomeLease lease) { return Path.Combine(lease.Home,"state",".wake-queue"); } static string CompletionHistory(NativeHomeLease lease) { return Path.Combine(lease.Home,"state",".watcher-down.ack-completions"); } static string Pending(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","note-id.note"); } @@ -81,6 +90,7 @@ static Dictionary ZeroPayload(NativeHomeLease lease) { payload["message"]="recovery";return payload; } public static int Run() { + OwnerProbeLinkCase(); Case("one writer and unobserved acknowledgement refusal",lease=>{ Targets(lease); using(var journal=new NativeReceiptJournal(lease,A)) { diff --git a/tests/fixtures/native-owner/fm-wake-lib-interrupt.sh b/tests/fixtures/native-owner/fm-wake-lib-interrupt.sh new file mode 100644 index 00000000000..c62d073c22d --- /dev/null +++ b/tests/fixtures/native-owner/fm-wake-lib-interrupt.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# shellcheck disable=SC1091 +. "$(dirname "${BASH_SOURCE[0]}")/fm-wake-lib.sh.actual" + +_fm_atomic_replace() { + case "$1:$2" in + *.wake-queue.ack.*:"$FM_WAKE_QUEUE") + case "$(cat "$STATE/.watcher-down" 2>/dev/null || true)" in + acked:*) + printf 'ready\n' > "$(cygpath -u "${FM_PROBE_HOME:?}")/ack-boundary-ready" + sleep 30 + ;; + esac + ;; + esac + command mv -f -- "$1" "$2" +} From 7fb52a1be253ca4e1cab006d52c3ba53b3668d97 Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 12:27:48 +1200 Subject: [PATCH 26/61] no-mistakes(review): Fix receipt target reuse and launcher argument binding --- bin/fm-native-codex.ps1 | 1 + tests/fixtures/native-owner/ReceiptTests.cs | 12 ++++++------ 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/bin/fm-native-codex.ps1 b/bin/fm-native-codex.ps1 index 6ee7eb785fa..19c59ae5f39 100644 --- a/bin/fm-native-codex.ps1 +++ b/bin/fm-native-codex.ps1 @@ -19,6 +19,7 @@ The Windows path to the temporary operational home. An existing local Docker image containing jq and GNU timeout. No image is pulled. Read-only helper containers self-expire even if their native client is stopped. #> +[CmdletBinding(PositionalBinding=$false)] param([switch]$Experimental,[switch]$BuildOnly,[switch]$VerifyOnly,[string]$OperationalHome,[string]$JqImage) $ErrorActionPreference='Stop' $root=[IO.Path]::GetFullPath((Join-Path $PSScriptRoot '..')) diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index ad691e21382..7064554c329 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -33,12 +33,12 @@ static void OwnerProbeLinkCase() { static string CompletionHistory(NativeHomeLease lease) { return Path.Combine(lease.Home,"state",".watcher-down.ack-completions"); } static string Pending(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","note-id.note"); } static string Handled(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","handled","note-id.note"); } - static void Targets(NativeHomeLease lease) { + static void Targets(NativeHomeLease lease,string note="note-id",string generation="recovery") { Directory.CreateDirectory(Path.GetDirectoryName(Handled(lease))); - File.WriteAllText(Pending(lease),"original captured inbox record\n"); - File.WriteAllText(Queue(lease),"1\t1\tcheck\tinbox:note-id\tcaptain inbox note\n"); + File.WriteAllText(Path.Combine(lease.Home,"state","inbox",note+".note"),"original captured inbox record\n"); + File.WriteAllText(Queue(lease),"1\t1\tcheck\tinbox:"+note+"\tcaptain inbox note\n"); File.WriteAllText(Path.Combine(lease.Home,"state",".main-eligible-rows"),"1\n"); - File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"pending:handling:recovery\n"); + File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"pending:handling:"+generation+"\n"); } static string ZeroRecovery(NativeHomeLease lease,string action,string generation=null) { string script=Path.Combine(NativeOwner.CodeRoot,"tests","fixtures","native-owner","zero-recovery.sh"); @@ -57,7 +57,7 @@ static Dictionary OwnerPayload(NativeHomeLease lease,string chall using(var process=Process.Start(start)) { var output=process.StandardOutput.ReadToEndAsync();var error=process.StandardError.ReadToEndAsync(); if(!process.WaitForExit(30000)){process.Kill();throw new IOException("Acknowledgement target fixture exceeded its bound");} - if(process.ExitCode!=0)throw new IOException("Acknowledgement target fixture failed: "+error.Result); + if(process.ExitCode!=0)throw new IOException("Acknowledgement target fixture exited "+process.ExitCode+": "+error.Result+output.Result); var payload=new JavaScriptSerializer().Deserialize>(output.Result); payload["challenge"]=challenge;payload["message"]="pending notification"; return payload; @@ -138,7 +138,7 @@ public static int Run() { Targets(lease); using(var journal=new NativeReceiptJournal(lease,A)) { var firstDelivery=journal.Present(OwnerPayload(lease,"first"));string first=(string)firstDelivery["receipt"]; - Acknowledge(lease,journal,firstDelivery,"first");Targets(lease);File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"pending:handling:recovery-second\n"); + Acknowledge(lease,journal,firstDelivery,"first");Targets(lease,"second-note","recovery-second"); var secondDelivery=journal.Present(OwnerPayload(lease,"second"));string second=(string)secondDelivery["receipt"]; Refuses(()=>journal.BeginAcknowledgement(first,"first"),"Earlier cycle replay accepted"); Acknowledge(lease,journal,secondDelivery,"second"); From 06231880db59c6a85b47d6253d6d6753ae0411bb Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 12:55:56 +1200 Subject: [PATCH 27/61] no-mistakes(review): Fix receipt inspection and MCP protocol validation --- bin/native-owner/app-server-policy.mjs | 15 +++++++-------- tests/fixtures/native-owner/ReceiptTests.cs | 8 ++++++-- .../native-owner/app-server-policy.test.mjs | 16 ++++------------ 3 files changed, 17 insertions(+), 22 deletions(-) diff --git a/bin/native-owner/app-server-policy.mjs b/bin/native-owner/app-server-policy.mjs index 67a4aaf83a7..93d1e946d3a 100644 --- a/bin/native-owner/app-server-policy.mjs +++ b/bin/native-owner/app-server-policy.mjs @@ -22,11 +22,10 @@ function validInstalledApp(value) { function validMcpServerStatus(value) { return isRecord(value)&&typeof value.name==='string'&& - MCP_RUNTIME_STATUS_ACTIVE.has(value.runtimeStatus)&& - (value.pluginId===null||typeof value.pluginId==='string')&& - (value.serverInfo===null||isRecord(value.serverInfo))&& - (value.serverCapabilities===null||isRecord(value.serverCapabilities))&& - isRecord(value.tools)&&(value.toolsError===null||typeof value.toolsError==='string')&& + (value.runtimeStatus===undefined||MCP_RUNTIME_STATUS_ACTIVE.has(value.runtimeStatus))&& + (value.pluginId==null||typeof value.pluginId==='string')&& + (value.serverInfo==null||isRecord(value.serverInfo))&& + isRecord(value.tools)&&(value.toolsError==null||typeof value.toolsError==='string')&& Array.isArray(value.resources)&&Array.isArray(value.resourceTemplates)&& ['unknown','unsupported','notLoggedIn','bearerToken','oAuth'].includes(value.authStatus); } @@ -36,9 +35,9 @@ async function readMcpServerStatuses(request) { let cursor=null; for(let page=0;page!validMcpServerStatus(status))||(response.nextCursor!==null&&typeof response.nextCursor!=='string'))throw Error('Invalid MCP server status response'); + if(!isRecord(response)||!Array.isArray(response.data)||response.data.some(status=>!validMcpServerStatus(status))||(response.nextCursor!=null&&typeof response.nextCursor!=='string'))throw Error('Invalid MCP server status response'); statuses.push(...response.data); - if(response.nextCursor===null)return statuses; + if(response.nextCursor==null)return statuses; if(seenCursors.has(response.nextCursor))throw Error('Repeated MCP server status cursor'); seenCursors.add(response.nextCursor); cursor=response.nextCursor; @@ -102,7 +101,7 @@ export async function verifyExternalToolIsolation(request,threadId,configuration if(!isRecord(installed)||!Array.isArray(installed.apps)||installed.apps.some(app=>!validInstalledApp(app)))throw Error('Invalid installed app catalog response'); const apps=installed.apps; const mcpServers=await readMcpServerStatuses(request); - const activeMcpServers=mcpServers.filter(server=>MCP_RUNTIME_STATUS_ACTIVE.get(server.runtimeStatus)||server.serverInfo!==null||server.serverCapabilities!==null||server.toolsError!==null||Object.keys(server.tools).length||server.resources.length||server.resourceTemplates.length); + const activeMcpServers=mcpServers.filter(server=>MCP_RUNTIME_STATUS_ACTIVE.get(server.runtimeStatus)||server.serverInfo!=null||server.toolsError!=null||Object.keys(server.tools).length||server.resources.length||server.resourceTemplates.length); const result={ ...configuration, exposedApps:apps.map(app=>app.id), diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index 7064554c329..7647d003a16 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -33,6 +33,10 @@ static void OwnerProbeLinkCase() { static string CompletionHistory(NativeHomeLease lease) { return Path.Combine(lease.Home,"state",".watcher-down.ack-completions"); } static string Pending(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","note-id.note"); } static string Handled(NativeHomeLease lease) { return Path.Combine(lease.Home,"state","inbox","handled","note-id.note"); } + static string ReadJournal(NativeHomeLease lease) { + using(var stream=new FileStream(Path.Combine(lease.Home,"owner-receipts.jsonl"),FileMode.Open,FileAccess.Read,FileShare.ReadWrite)) + using(var reader=new StreamReader(stream,Encoding.UTF8,true)) return reader.ReadToEnd(); + } static void Targets(NativeHomeLease lease,string note="note-id",string generation="recovery") { Directory.CreateDirectory(Path.GetDirectoryName(Handled(lease))); File.WriteAllText(Path.Combine(lease.Home,"state","inbox",note+".note"),"original captured inbox record\n"); @@ -149,13 +153,13 @@ public static int Run() { using(var journal=new NativeReceiptJournal(lease,A)) { var delivery=journal.Present(OwnerPayload(lease,"first"));string receipt=(string)delivery["receipt"],evidence=(string)delivery["ownerEvidence"]; journal.BeginAcknowledgement(receipt,"first",NativeAcknowledgementEvidence.Capture(lease,delivery)); - string before=File.ReadAllText(Path.Combine(lease.Home,"owner-receipts.jsonl")); + string before=ReadJournal(lease); Refuses(()=>journal.CompleteAcknowledgement(receipt,null),"Missing zero-exit response completed an acknowledgement"); Refuses(()=>journal.CompleteAcknowledgement(receipt,"{broken"),"Malformed zero-exit response completed an acknowledgement"); Refuses(()=>journal.CompleteAcknowledgement(receipt,Json.Serialize(new Dictionary{{"acknowledged",true},{"ownerEvidence","mismatch"}})),"Mismatched zero-exit response completed an acknowledgement"); Refuses(()=>journal.CompleteAcknowledgement(receipt,Json.Serialize(new Dictionary{{"acknowledged",false},{"ownerEvidence",evidence}})),"Negative zero-exit response completed an acknowledgement"); Refuses(()=>journal.CompleteAcknowledgement(receipt,Completion(delivery)),"Unperformed effect completed from response data alone"); - Expect(before==File.ReadAllText(Path.Combine(lease.Home,"owner-receipts.jsonl")),"Rejected completion response changed the journal"); + Expect(before==ReadJournal(lease),"Rejected completion response changed the journal"); OwnerAcknowledge(lease,delivery);journal.CompleteAcknowledgement(receipt,Completion(delivery)); Expect(!journal.NeedsReconciliation,"Affirmatively proven completion remained unresolved"); } diff --git a/tests/fixtures/native-owner/app-server-policy.test.mjs b/tests/fixtures/native-owner/app-server-policy.test.mjs index 98d82dddf4f..ab5ed2f57f0 100644 --- a/tests/fixtures/native-owner/app-server-policy.test.mjs +++ b/tests/fixtures/native-owner/app-server-policy.test.mjs @@ -14,7 +14,7 @@ function requestBoundary(responses) { } function inactiveMcpServer(name) { - return {name,runtimeStatus:null,pluginId:null,serverInfo:null,serverCapabilities:null,tools:{},toolsError:null,resources:[],resourceTemplates:[],authStatus:'unknown'}; + return {name,tools:{},resources:[],resourceTemplates:[],authStatus:'unknown'}; } const isolatedConfiguration={appsFeatureEnabled:false,pluginsFeatureEnabled:false,configuredMcpServers:['alpha'],enabledMcpServers:[]}; @@ -87,7 +87,7 @@ test('supported runtime statuses have explicit activity semantics',async()=>{ test('disabled status is inactive only when the complete response exposes nothing',async()=>{ for(const exposed of [ - {serverInfo:{}},{serverCapabilities:{}},{toolsError:'failed to enumerate'}, {tools:{read:{}}}, {resources:[{}]}, {resourceTemplates:[{}]}, + {serverInfo:{}},{toolsError:'failed to enumerate'}, {tools:{read:{}}}, {resources:[{}]}, {resourceTemplates:[{}]}, ]){ const server={...inactiveMcpServer('disabled'),runtimeStatus:'disabled',...exposed}; const boundary=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[server],nextCursor:null}}); @@ -95,16 +95,8 @@ test('disabled status is inactive only when the complete response exposes nothin } }); -test('server capabilities are required',async()=>{ - const missing=inactiveMcpServer('missing');delete missing.serverCapabilities; - const absent=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[missing],nextCursor:null}}); - await assert.rejects(verifyExternalToolIsolation(absent.request,'thread-1',isolatedConfiguration),/Invalid MCP server status response/); - const exposed=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[{...inactiveMcpServer('active'),serverCapabilities:{tools:{}}}],nextCursor:null}}); - await assert.rejects(verifyExternalToolIsolation(exposed.request,'thread-1',isolatedConfiguration),/External app-server tools are not isolated/); -}); - -test('valid empty and multi-page catalogs are accepted',async()=>{ - const empty=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[],nextCursor:null}}); +test('valid empty and protocol-optional-field catalogs are accepted',async()=>{ + const empty=requestBoundary({'app/installed':{apps:[]},'mcpServerStatus/list':{data:[]}}); assert.deepEqual((await verifyExternalToolIsolation(empty.request,'thread-1',isolatedConfiguration)).activeMcpServers,[]); const pages=new Map([ [null,{data:[inactiveMcpServer('alpha')],nextCursor:'page-2'}], From 2ca48a411e62bcd7d6e2f820bb341bd57a786624 Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 21:45:43 +1200 Subject: [PATCH 28/61] no-mistakes(review): Fix receipt counters and disabled app classification --- bin/native-owner/app-server-policy.mjs | 2 +- tests/fixtures/native-owner/ReceiptTests.cs | 1 + tests/fixtures/native-owner/app-server-policy.test.mjs | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/bin/native-owner/app-server-policy.mjs b/bin/native-owner/app-server-policy.mjs index 93d1e946d3a..3f36ba7a000 100644 --- a/bin/native-owner/app-server-policy.mjs +++ b/bin/native-owner/app-server-policy.mjs @@ -104,7 +104,7 @@ export async function verifyExternalToolIsolation(request,threadId,configuration const activeMcpServers=mcpServers.filter(server=>MCP_RUNTIME_STATUS_ACTIVE.get(server.runtimeStatus)||server.serverInfo!=null||server.toolsError!=null||Object.keys(server.tools).length||server.resources.length||server.resourceTemplates.length); const result={ ...configuration, - exposedApps:apps.map(app=>app.id), + exposedApps:apps.filter(app=>app.enabled||app.callable).map(app=>app.id), activeMcpServers:activeMcpServers.map(server=>server.name), }; if(result.exposedApps.length||result.activeMcpServers.length)throw Error('External app-server tools are not isolated: '+JSON.stringify(result)); diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index 7647d003a16..07f5ae68c8e 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -41,6 +41,7 @@ static void Targets(NativeHomeLease lease,string note="note-id",string generatio Directory.CreateDirectory(Path.GetDirectoryName(Handled(lease))); File.WriteAllText(Path.Combine(lease.Home,"state","inbox",note+".note"),"original captured inbox record\n"); File.WriteAllText(Queue(lease),"1\t1\tcheck\tinbox:"+note+"\tcaptain inbox note\n"); + File.WriteAllText(Path.Combine(lease.Home,"state",".wake-queue.seq"),"1\n"); File.WriteAllText(Path.Combine(lease.Home,"state",".main-eligible-rows"),"1\n"); File.WriteAllText(Path.Combine(lease.Home,"state",".watcher-down"),"pending:handling:"+generation+"\n"); } diff --git a/tests/fixtures/native-owner/app-server-policy.test.mjs b/tests/fixtures/native-owner/app-server-policy.test.mjs index ab5ed2f57f0..9ffc449f87b 100644 --- a/tests/fixtures/native-owner/app-server-policy.test.mjs +++ b/tests/fixtures/native-owner/app-server-policy.test.mjs @@ -30,7 +30,7 @@ test('isolated app-server arguments disable every inherited external capability' test('the fake request boundary observes a fully isolated effective catalog',async()=>{ const boundary=requestBoundary({ 'config/read':{config:{features:{apps:false,plugins:false},mcp_servers:{alpha:{enabled:false}}}}, - 'app/installed':{apps:[]}, + 'app/installed':{apps:[{id:'policy-disabled-app',runtimeName:'Policy Disabled App',enabled:false,callable:false}]}, 'mcpServerStatus/list':{data:[],nextCursor:null}, }); const configuration=await verifyExternalToolConfiguration(boundary.request,['alpha']); From 7fa3d28bb0b19043e85ba928462b7294a3faceed Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 22:40:52 +1200 Subject: [PATCH 29/61] no-mistakes(test): Canonicalize launcher fixture state path --- tests/fixtures/native-owner/Launcher.mjs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index a282bdf0d4d..9a24bcc66e4 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -64,8 +64,8 @@ function enqueue(home,message){ assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr}));return result.stdout.trim().split(/\s+/)[1]; } function appendStartupWake(home,state){ - const env={...process.env,FM_HOME:home,FM_ROOT_OVERRIDE:posix(code),FM_STATE_OVERRIDE:posix(path.join(home,'state')),MSYS:'winsymlinks:nativestrict'}; - const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','. "$1/bin/fm-wake-lib.sh"; fm_wake_append_startup_network "$2"','startup-wake',posix(code),state],{env,encoding:'utf8',timeout:30000}); + const env={...process.env,FM_HOME:home,FM_ROOT_OVERRIDE:posix(code),MSYS:'winsymlinks:nativestrict'}; + const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export FM_STATE_OVERRIDE; FM_STATE_OVERRIDE=$(cygpath -u "$3"); . "$1/bin/fm-wake-lib.sh"; fm_wake_append_startup_network "$2"','startup-wake',posix(code),state,path.join(home,'state')],{env,encoding:'utf8',timeout:30000}); assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); } const journalRows=home=>fs.readFileSync(path.join(home,'owner-receipts.jsonl'),'utf8').trim().split(/\r?\n/).filter(Boolean).map(JSON.parse); From b4ed6469e90636da36b922d4cd8758eae80e7b67 Mon Sep 17 00:00:00 2001 From: Cristian Date: Thu, 17 Sep 2026 22:54:11 +1200 Subject: [PATCH 30/61] no-mistakes(test): Canonicalize launcher fixture code-root path --- tests/fixtures/native-owner/Launcher.mjs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 9a24bcc66e4..db8f5719eb2 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -64,8 +64,8 @@ function enqueue(home,message){ assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr}));return result.stdout.trim().split(/\s+/)[1]; } function appendStartupWake(home,state){ - const env={...process.env,FM_HOME:home,FM_ROOT_OVERRIDE:posix(code),MSYS:'winsymlinks:nativestrict'}; - const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export FM_STATE_OVERRIDE; FM_STATE_OVERRIDE=$(cygpath -u "$3"); . "$1/bin/fm-wake-lib.sh"; fm_wake_append_startup_network "$2"','startup-wake',posix(code),state,path.join(home,'state')],{env,encoding:'utf8',timeout:30000}); + const env={...process.env,FM_HOME:home,MSYS:'winsymlinks:nativestrict'}; + const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export FM_ROOT_OVERRIDE FM_STATE_OVERRIDE; FM_ROOT_OVERRIDE=$(cygpath -u "$1"); FM_STATE_OVERRIDE=$(cygpath -u "$3"); . "$FM_ROOT_OVERRIDE/bin/fm-wake-lib.sh"; fm_wake_append_startup_network "$2"','startup-wake',code,state,path.join(home,'state')],{env,encoding:'utf8',timeout:30000}); assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); } const journalRows=home=>fs.readFileSync(path.join(home,'owner-receipts.jsonl'),'utf8').trim().split(/\r?\n/).filter(Boolean).map(JSON.parse); From 9fbdb97fed4a64403724d15954b2ae4707f5f840 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 00:12:42 +1200 Subject: [PATCH 31/61] no-mistakes(test): Prevent acknowledgement failure on closed native output --- bin/fm-wake-lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index d73f4bd8fc6..0dfd544f847 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -2677,7 +2677,7 @@ fm_wake_ack_evidence_acknowledge() { # [ -n "$id" ] && notes+=("$id") done < "$FM_WAKE_ACK_EVIDENCE_NOTES" if [ "${#notes[@]}" -gt 0 ]; then - "$FM_WAKE_LIB_DIR/fm-inbox.sh" drain --ack "${notes[@]}" || { fm_wake_ack_evidence_clear; return 1; } + "$FM_WAKE_LIB_DIR/fm-inbox.sh" drain --ack "${notes[@]}" >/dev/null || { fm_wake_ack_evidence_clear; return 1; } fi "$FM_WAKE_LIB_DIR/fm-wake-drain.sh" --ack-through "$FM_WAKE_ACK_EVIDENCE_CUTOFF" \ --recovery-generation "$FM_WAKE_ACK_EVIDENCE_GENERATION" || { fm_wake_ack_evidence_clear; return 1; } From 71a7f97788f44a14b9841b5518ce920d145c26e5 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 08:18:49 +1200 Subject: [PATCH 32/61] no-mistakes(test): Fix Windows custom-check mode setup; host retest pending --- tests/fixtures/native-owner/Launcher.mjs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index db8f5719eb2..1e48d8ae667 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -180,9 +180,10 @@ fs.mkdirSync(path.dirname(maskedStatus),{recursive:true});fs.writeFileSync(maske const masked=start(maskedHome,true,{...process.env,BASH_ENV:mask});masked.child.stdin.end();assert.notEqual((await bound(masked.done,masked,20000)).exit,0); assert.equal(fs.readFileSync(maskedStatus,'utf8'),'preserve masked residual state');assert.equal(fs.existsSync(path.join(maskedHome,'owner-probe.json')),false); records.push('ambient Bash startup hooks cannot bypass admission before lease acquisition'); -const customHome=path.join(area,'registered-custom'),customState=path.join(customHome,'state'),canary=path.join(customHome,'executed'); -fs.mkdirSync(customState,{recursive:true});fs.writeFileSync(path.join(customState,'custom.check.sh'),`#!/usr/bin/env bash\nprintf executed > "${posix(canary)}"\n`); -fs.chmodSync(path.join(customState,'custom.check.sh'),0o700); +const customHome=path.join(area,'registered-custom'),customState=path.join(customHome,'state'),customCheck=path.join(customState,'custom.check.sh'),canary=path.join(customHome,'executed'); +fs.mkdirSync(customState,{recursive:true});fs.writeFileSync(customCheck,`#!/usr/bin/env bash\nprintf executed > "${posix(canary)}"\n`); +const chmod=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','chmod 0700 -- "$1"','custom-check-mode',posix(customCheck)],{env:{...process.env,MSYS:'winsymlinks:nativestrict'},encoding:'utf8',timeout:30000}); +assert.equal(chmod.status,0,chmod.stderr); const register=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',posix(path.join(code,'bin/fm-check-register.sh')),'custom'],{env:{...process.env,FM_HOME:posix(customHome),MSYS:'winsymlinks:nativestrict'},encoding:'utf8',timeout:30000}); assert.equal(register.status,0,register.stderr); const custom=start(customHome);custom.child.stdin.end();assert.notEqual((await bound(custom.done,custom,20000)).exit,0); From eff7d0a00723e7257b87c51f17cc9e7f91e0b9c9 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 08:37:42 +1200 Subject: [PATCH 33/61] no-mistakes(test): Remove unsupported Windows custom-check registration fixture --- tests/fixtures/native-owner/Launcher.mjs | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 1e48d8ae667..0fcab8ad445 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -3,6 +3,7 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; import assert from 'node:assert/strict'; +import {createHash} from 'node:crypto'; import {fileURLToPath} from 'node:url'; import {spawn,spawnSync} from 'node:child_process'; const repo=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..'); @@ -180,15 +181,14 @@ fs.mkdirSync(path.dirname(maskedStatus),{recursive:true});fs.writeFileSync(maske const masked=start(maskedHome,true,{...process.env,BASH_ENV:mask});masked.child.stdin.end();assert.notEqual((await bound(masked.done,masked,20000)).exit,0); assert.equal(fs.readFileSync(maskedStatus,'utf8'),'preserve masked residual state');assert.equal(fs.existsSync(path.join(maskedHome,'owner-probe.json')),false); records.push('ambient Bash startup hooks cannot bypass admission before lease acquisition'); -const customHome=path.join(area,'registered-custom'),customState=path.join(customHome,'state'),customCheck=path.join(customState,'custom.check.sh'),canary=path.join(customHome,'executed'); -fs.mkdirSync(customState,{recursive:true});fs.writeFileSync(customCheck,`#!/usr/bin/env bash\nprintf executed > "${posix(canary)}"\n`); -const chmod=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','chmod 0700 -- "$1"','custom-check-mode',posix(customCheck)],{env:{...process.env,MSYS:'winsymlinks:nativestrict'},encoding:'utf8',timeout:30000}); -assert.equal(chmod.status,0,chmod.stderr); -const register=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',posix(path.join(code,'bin/fm-check-register.sh')),'custom'],{env:{...process.env,FM_HOME:posix(customHome),MSYS:'winsymlinks:nativestrict'},encoding:'utf8',timeout:30000}); -assert.equal(register.status,0,register.stderr); +const customHome=path.join(area,'custom-work'),customState=path.join(customHome,'state'),customCheck=path.join(customState,'custom.check.sh'),customTrust=path.join(customState,'custom.check-trust'),canary=path.join(customHome,'executed'); +const customCheckBody=`#!/usr/bin/env bash\nprintf executed > "${posix(canary)}"\n`,customTrustBody=`fm-custom-check-v1\n${createHash('sha256').update(customCheckBody).digest('hex')}\n`; +// Registration itself is covered on platforms that support its private-mode contract; +// this native fixture exercises the persisted custom-work admission boundary. +fs.mkdirSync(customState,{recursive:true});fs.writeFileSync(customCheck,customCheckBody);fs.writeFileSync(customTrust,customTrustBody); const custom=start(customHome);custom.child.stdin.end();assert.notEqual((await bound(custom.done,custom,20000)).exit,0); -assert.equal(fs.existsSync(canary),false);assert.equal(fs.existsSync(path.join(customHome,'owner-probe.json')),false); -records.push('registered custom work is refused before lease acquisition without execution'); +assert.equal(fs.existsSync(canary),false);assert.equal(fs.existsSync(path.join(customHome,'owner-probe.json')),false);assert.equal(fs.readFileSync(customCheck,'utf8'),customCheckBody);assert.equal(fs.readFileSync(customTrust,'utf8'),customTrustBody); +records.push('persisted custom work is refused unchanged before lease acquisition without execution'); for(const [name,contents] of [ ['queued-backlog','## In flight\n\n## Queued\n- [ ] queued-work - preserved project work (repo: firstmate) (kind: ship)\n\n## Done\n'], ['unrecognized-backlog','# Backlog\n\nproject work in an unrecognized form\n'], From 1ea5f387fc5c5e4d076b7a1db29f9b3fc46a6cfc Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 08:58:02 +1200 Subject: [PATCH 34/61] no-mistakes(test): Fix non-temporary launcher fixture path selection --- tests/fixtures/native-owner/Launcher.mjs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 0fcab8ad445..aca4e5c35c9 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -13,6 +13,12 @@ if(!image)throw Error('FM_NATIVE_TEST_JQ_IMAGE must name an existing local image const read=file=>JSON.parse(fs.readFileSync(file,'utf8').replace(/^\uFEFF/,'')); const sleep=ms=>new Promise(resolve=>setTimeout(resolve,ms)); function command(exe,args){const result=spawnSync(exe,args,{encoding:'utf8',timeout:120000});if(result.status!==0)throw Error(result.stderr||result.stdout);return result;} +function contains(root,candidate){const relative=path.relative(root,candidate);return relative===''||(!path.isAbsolute(relative)&&relative!=='..'&&!relative.startsWith('..'+path.sep));} +const localAppData=command('powershell.exe',['-NoProfile','-NonInteractive','-Command','[Environment]::GetFolderPath([Environment+SpecialFolder]::LocalApplicationData)']).stdout.trim(); +const windowsTemp=path.resolve(localAppData,'Temp'); +const outsideRoot=path.resolve(process.env.FM_NATIVE_TEST_OUTSIDE_ROOT||path.join(repo,'data/native-launcher')); +const outside=path.join(outsideRoot,path.basename(area)+'-outside'); +assert.equal(contains(windowsTemp,outside),false,'FM_NATIVE_TEST_OUTSIDE_ROOT must resolve outside the canonical Windows temporary directory'); command('git',['-c','core.symlinks=true','clone','--quiet','--no-local','--single-branch',repo,code]); fs.cpSync(path.join(repo,'bin/native-owner'),path.join(code,'bin/native-owner'),{recursive:true}); for(const name of ['fm-native-codex.ps1','fm-session-lock-lib.sh','fm-sessionstart-nudge.sh','fm-harness.sh','fm-backlog-transition-lib.sh','fm-supervision-lib.sh','fm-wake-lib.sh','fm-startup-network.sh','fm-inbox.sh','fm-lock.sh'])fs.copyFileSync(path.join(repo,'bin',name),path.join(code,'bin',name)); @@ -215,7 +221,6 @@ for(const [name,row] of [ assert.equal(fs.readFileSync(queue,'utf8'),row);assert.equal(fs.readFileSync(marker,'utf8'),'pending:downtime:preserve\n');assert.equal(fs.existsSync(path.join(wakeHome,'owner-probe.json')),false); } records.push('unsupported and malformed wake records refused unchanged before lease acquisition'); -const outside=path.join(repo,'data/native-launcher',path.basename(area)+'-outside'); assert.equal(fs.existsSync(outside),false); const external=start(outside);external.child.stdin.end();assert.notEqual((await bound(external.done,external,20000)).exit,0);assert.equal(fs.existsSync(outside),false); const target=path.join(area,'junction-target'),junction=path.join(area,'junction');fs.mkdirSync(target);fs.symlinkSync(target,junction,'junction'); From 153074787ea456f5b6ea317fb2989fa8387b4162 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 09:24:42 +1200 Subject: [PATCH 35/61] no-mistakes(test): Retry native admission during notification publication --- bin/native-owner/NativeOperations.cs | 23 ++++++++++++++--------- tests/fixtures/native-owner/Launcher.mjs | 17 ++++++++++++++--- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index 917e19892ab..b54b44691da 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -34,16 +34,21 @@ internal static ProcessStartInfo BashHelper(string script,string arguments,strin static void OwnerAdmission(string home,bool launch,string[] provenDeadGenerations) { string script=Path.Combine(CodeRoot,"bin","native-owner","admit.sh"); object evidence=NativeReceiptJournal.AdmissionEvidence(home); - var start=BashHelper(script,launch ? "launch" : "owned-operation",home);start.RedirectStandardInput=true;start.RedirectStandardError=true; - if(launch && provenDeadGenerations!=null && provenDeadGenerations.Length>0)start.EnvironmentVariables["FM_NATIVE_PROVEN_DEAD_GENERATIONS"]=string.Join(",",provenDeadGenerations); string input=""; - if(evidence is string) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="token";input=(string)evidence;} - else if(evidence!=null) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="legacy";input=Json.Serialize(evidence);} - using(var process=Process.Start(start)) { - var error=process.StandardError.ReadToEndAsync(); - process.StandardInput.Write(input);process.StandardInput.Close(); - if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("The empty-fleet owner preflight exceeded its bound");} - if(process.ExitCode!=0)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); + DateTime retryUntil=DateTime.UtcNow.AddSeconds(3); + while(true) { + var start=BashHelper(script,launch ? "launch" : "owned-operation",home);start.RedirectStandardInput=true;start.RedirectStandardError=true; + if(launch && provenDeadGenerations!=null && provenDeadGenerations.Length>0)start.EnvironmentVariables["FM_NATIVE_PROVEN_DEAD_GENERATIONS"]=string.Join(",",provenDeadGenerations); + if(evidence is string) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="token";input=(string)evidence;} + else if(evidence!=null) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="legacy";input=Json.Serialize(evidence);} + using(var process=Process.Start(start)) { + var error=process.StandardError.ReadToEndAsync(); + process.StandardInput.Write(input);process.StandardInput.Close(); + if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("The empty-fleet owner preflight exceeded its bound");} + if(process.ExitCode==0)return; + if(launch || DateTime.UtcNow>=retryUntil)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); + } + System.Threading.Thread.Sleep(100); } } internal static void EmptyFleet(string home,bool launch,string[] provenDeadGenerations=null) { diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index aca4e5c35c9..afd5e8acac6 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -75,6 +75,11 @@ function appendStartupWake(home,state){ const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export FM_ROOT_OVERRIDE FM_STATE_OVERRIDE; FM_ROOT_OVERRIDE=$(cygpath -u "$1"); FM_STATE_OVERRIDE=$(cygpath -u "$3"); . "$FM_ROOT_OVERRIDE/bin/fm-wake-lib.sh"; fm_wake_append_startup_network "$2"','startup-wake',code,state,path.join(home,'state')],{env,encoding:'utf8',timeout:30000}); assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); } +function appendInboxWake(home,id,summary){ + const env={...process.env,FM_HOME:home,MSYS:'winsymlinks:nativestrict'}; + const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export FM_ROOT_OVERRIDE FM_STATE_OVERRIDE; FM_ROOT_OVERRIDE=$(cygpath -u "$1"); FM_STATE_OVERRIDE=$(cygpath -u "$4"); . "$FM_ROOT_OVERRIDE/bin/fm-wake-lib.sh"; fm_wake_append check "inbox:$2" "check: captain inbox note $2 - $3"','inbox-wake',code,id,summary,path.join(home,'state')],{env,encoding:'utf8',timeout:30000}); + assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); +} const journalRows=home=>fs.readFileSync(path.join(home,'owner-receipts.jsonl'),'utf8').trim().split(/\r?\n/).filter(Boolean).map(JSON.parse); const home=path.join(area,'home');fs.mkdirSync(path.join(home,'data'),{recursive:true}); fs.writeFileSync(path.join(home,'data/backlog.md'),'## In flight\n\n## Queued\n\n## Done\n'); @@ -152,13 +157,19 @@ assert(fs.readFileSync(path.join(historicalState,'.wake-queue'),'utf8').includes records.push('queued startup failure survives a newer startup status and remains admissible'); const hostScript=path.join(code,'bin/native-owner/codex-host.mjs'),hostSource=fs.readFileSync(hostScript,'utf8'); fs.copyFileSync(path.join(repo,'tests/fixtures/native-owner/fake-app-server.mjs'),path.join(code,'bin/native-owner/fake-app-server.mjs')); -fs.writeFileSync(hostScript,"import {runCodexHost} from './codex-host-runtime.mjs';\nimport {createFakeAppServer} from './fake-app-server.mjs';\ntry { await runCodexHost({spawnAppServer:()=>createFakeAppServer('success'),mcpServerNames:[]}); } catch(error) { console.error(error.message); process.exitCode=1; }\n"); +fs.writeFileSync(hostScript,"import fs from 'node:fs';\nimport path from 'node:path';\nimport {runCodexHost} from './codex-host-runtime.mjs';\nimport {createFakeAppServer} from './fake-app-server.mjs';\nconst pause=ms=>new Promise(resolve=>setTimeout(resolve,ms));\ntry { await runCodexHost({spawnAppServer:()=>createFakeAppServer('success'),mcpServerNames:[],afterAutomaticTurn:async ({evidence})=>{if(evidence.automatic.length===1){fs.writeFileSync(path.join(process.env.FM_PROBE_HOME,'first-automatic-complete'),'ready');while(!fs.existsSync(path.join(process.env.FM_PROBE_HOME,'continue-after-first')))await pause(20);}}}); } catch(error) { console.error(error.message); process.exitCode=1; }\n"); const completedHome=path.join(area,'completed-ack-restart'),completedNote=enqueue(completedHome,'Complete this controlled acknowledgement before restart.'); const completedSession=start(completedHome,false);const completedReady=await ready(completedSession); -await waitUntil(completedSession,'completed acknowledgement',()=>journalRows(completedHome).at(-1)?.event==='acknowledged'&&fs.existsSync(path.join(completedHome,'state/inbox/handled',completedNote+'.note'))&&fs.readFileSync(path.join(completedHome,'state/.wake-queue'),'utf8').trim()===''); +await waitUntil(completedSession,'completed acknowledgement',()=>journalRows(completedHome).at(-1)?.event==='acknowledged'&&fs.existsSync(path.join(completedHome,'state/inbox/handled',completedNote+'.note'))&&fs.readFileSync(path.join(completedHome,'state/.wake-queue'),'utf8').trim()===''&&fs.existsSync(path.join(completedReady.runtime,'first-automatic-complete'))); +const transitionNote='transition-note',transitionMessage='Complete this notification after its producer finishes publishing.'; +fs.writeFileSync(path.join(completedHome,'state/inbox',transitionNote+'.note'),`id=${transitionNote}\nat=2026-09-18T00:00:00Z\nsource=text\n--\n${transitionMessage}\n`); +fs.writeFileSync(path.join(completedReady.runtime,'continue-after-first'),'continue'); +await sleep(1000); +appendInboxWake(completedHome,transitionNote,transitionMessage); +await waitUntil(completedSession,'notification publication transition',()=>journalRows(completedHome).filter(row=>row.event==='acknowledged').length===2&&fs.existsSync(path.join(completedHome,'state/inbox/handled',transitionNote+'.note'))&&fs.readFileSync(path.join(completedHome,'state/.wake-queue'),'utf8').trim()==='',60000); completedSession.child.stdin.write('/quit\n');assert.equal((await bound(completedSession.done,completedSession,20000)).exit,0); const completedRestart=start(completedHome);await ready(completedRestart,completedReady.owner.generation);completedRestart.child.stdin.end();assert.equal((await bound(completedRestart.done,completedRestart,20000)).exit,0); -records.push('journal-correlated completed acknowledgement remains admissible across restart'); +records.push('notification publication transitions and completed acknowledgements remain admissible across restart'); const wakeLib=path.join(code,'bin/fm-wake-lib.sh'),wakeLibActual=wakeLib+'.actual'; fs.renameSync(wakeLib,wakeLibActual);fs.copyFileSync(path.join(repo,'tests/fixtures/native-owner/fm-wake-lib-interrupt.sh'),wakeLib); const interruptedHome=path.join(area,'interrupted-ack-restart'),interruptedNote=enqueue(interruptedHome,'Preserve this interrupted acknowledgement for reconciliation.'); From 47be0f6571b403a487186bdad434c8a89cdda9ae Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 09:44:14 +1200 Subject: [PATCH 36/61] no-mistakes(test): Fix launcher notification fixture synchronization --- tests/fixtures/native-owner/Launcher.mjs | 25 +++++++++++------ .../fixtures/native-owner/fake-app-server.mjs | 3 +- .../fixtures/native-owner/tool-gate.test.mjs | 28 +++++++++++++------ 3 files changed, 38 insertions(+), 18 deletions(-) diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index afd5e8acac6..6587e23a57d 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -81,6 +81,8 @@ function appendInboxWake(home,id,summary){ assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); } const journalRows=home=>fs.readFileSync(path.join(home,'owner-receipts.jsonl'),'utf8').trim().split(/\r?\n/).filter(Boolean).map(JSON.parse); +const acknowledgedNote=(home,note)=>journalRows(home).some(row=>row.event==='acknowledged'&&Array.isArray(row.payload?.notes)&&row.payload.notes.includes(note)); +const pendingNote=(home,note)=>fs.readFileSync(path.join(home,'state/.wake-queue'),'utf8').split(/\r?\n/).filter(Boolean).some(row=>row.split('\t')[3]===`inbox:${note}`); const home=path.join(area,'home');fs.mkdirSync(path.join(home,'data'),{recursive:true}); fs.writeFileSync(path.join(home,'data/backlog.md'),'## In flight\n\n## Queued\n\n## Done\n'); const first=start(home);const initial=await ready(first);console.error('first ready',area); @@ -159,15 +161,20 @@ const hostScript=path.join(code,'bin/native-owner/codex-host.mjs'),hostSource=fs fs.copyFileSync(path.join(repo,'tests/fixtures/native-owner/fake-app-server.mjs'),path.join(code,'bin/native-owner/fake-app-server.mjs')); fs.writeFileSync(hostScript,"import fs from 'node:fs';\nimport path from 'node:path';\nimport {runCodexHost} from './codex-host-runtime.mjs';\nimport {createFakeAppServer} from './fake-app-server.mjs';\nconst pause=ms=>new Promise(resolve=>setTimeout(resolve,ms));\ntry { await runCodexHost({spawnAppServer:()=>createFakeAppServer('success'),mcpServerNames:[],afterAutomaticTurn:async ({evidence})=>{if(evidence.automatic.length===1){fs.writeFileSync(path.join(process.env.FM_PROBE_HOME,'first-automatic-complete'),'ready');while(!fs.existsSync(path.join(process.env.FM_PROBE_HOME,'continue-after-first')))await pause(20);}}}); } catch(error) { console.error(error.message); process.exitCode=1; }\n"); const completedHome=path.join(area,'completed-ack-restart'),completedNote=enqueue(completedHome,'Complete this controlled acknowledgement before restart.'); -const completedSession=start(completedHome,false);const completedReady=await ready(completedSession); -await waitUntil(completedSession,'completed acknowledgement',()=>journalRows(completedHome).at(-1)?.event==='acknowledged'&&fs.existsSync(path.join(completedHome,'state/inbox/handled',completedNote+'.note'))&&fs.readFileSync(path.join(completedHome,'state/.wake-queue'),'utf8').trim()===''&&fs.existsSync(path.join(completedReady.runtime,'first-automatic-complete'))); -const transitionNote='transition-note',transitionMessage='Complete this notification after its producer finishes publishing.'; -fs.writeFileSync(path.join(completedHome,'state/inbox',transitionNote+'.note'),`id=${transitionNote}\nat=2026-09-18T00:00:00Z\nsource=text\n--\n${transitionMessage}\n`); -fs.writeFileSync(path.join(completedReady.runtime,'continue-after-first'),'continue'); -await sleep(1000); -appendInboxWake(completedHome,transitionNote,transitionMessage); -await waitUntil(completedSession,'notification publication transition',()=>journalRows(completedHome).filter(row=>row.event==='acknowledged').length===2&&fs.existsSync(path.join(completedHome,'state/inbox/handled',transitionNote+'.note'))&&fs.readFileSync(path.join(completedHome,'state/.wake-queue'),'utf8').trim()==='',60000); -completedSession.child.stdin.write('/quit\n');assert.equal((await bound(completedSession.done,completedSession,20000)).exit,0); +const completedSession=start(completedHome,false);let completedReady; +try { + completedReady=await ready(completedSession); + await waitUntil(completedSession,'completed acknowledgement',()=>acknowledgedNote(completedHome,completedNote)&&fs.existsSync(path.join(completedHome,'state/inbox/handled',completedNote+'.note'))&&!pendingNote(completedHome,completedNote)&&fs.existsSync(path.join(completedReady.runtime,'first-automatic-complete'))); + const transitionNote='transition-note',transitionMessage='Complete this notification after its producer finishes publishing.'; + fs.writeFileSync(path.join(completedHome,'state/inbox',transitionNote+'.note'),`id=${transitionNote}\nat=2026-09-18T00:00:00Z\nsource=text\n--\n${transitionMessage}\n`); + fs.writeFileSync(path.join(completedReady.runtime,'continue-after-first'),'continue'); + await sleep(1000); + appendInboxWake(completedHome,transitionNote,transitionMessage); + await waitUntil(completedSession,'notification publication transition',()=>acknowledgedNote(completedHome,transitionNote)&&fs.existsSync(path.join(completedHome,'state/inbox/handled',transitionNote+'.note'))&&!pendingNote(completedHome,transitionNote)&&fs.readFileSync(path.join(completedHome,'state/.wake-queue'),'utf8').trim()==='',60000); + completedSession.child.stdin.write('/quit\n');assert.equal((await bound(completedSession.done,completedSession,20000)).exit,0); +}finally{ + if(completedSession.child.exitCode===null){completedSession.child.stdin.write('/quit\n');if(completedReady)fs.writeFileSync(path.join(completedReady.runtime,'continue-after-first'),'continue');const stopped=await bound(completedSession.done,completedSession,20000);assert.equal(stopped.exit,0,stopped.stderr);} +} const completedRestart=start(completedHome);await ready(completedRestart,completedReady.owner.generation);completedRestart.child.stdin.end();assert.equal((await bound(completedRestart.done,completedRestart,20000)).exit,0); records.push('notification publication transitions and completed acknowledgements remain admissible across restart'); const wakeLib=path.join(code,'bin/fm-wake-lib.sh'),wakeLibActual=wakeLib+'.actual'; diff --git a/tests/fixtures/native-owner/fake-app-server.mjs b/tests/fixtures/native-owner/fake-app-server.mjs index 8e6252ae08e..2f84a58d383 100644 --- a/tests/fixtures/native-owner/fake-app-server.mjs +++ b/tests/fixtures/native-owner/fake-app-server.mjs @@ -13,7 +13,7 @@ export function createFakeAppServer(scenario){ }; const fail=error=>{server.stderr.write(error.message+'\n');server.emit('error',error);finish();}; const send=value=>server.stdout.write(JSON.stringify(value)+'\n'); - const thread='primary',turn='automatic-turn'; + const thread='primary';let turn='',turnNumber=0; const complete=()=>send({method:'turn/completed',params:{threadId:thread,turn:{id:turn,status:'completed'}}}); const call=(id,tool,args,overrides={})=>send({id,method:'item/tool/call',params:{threadId:thread,turnId:turn,callId:'call-'+id,namespace:null,tool,arguments:args,...overrides}}); const lines=createInterface({input}); @@ -37,6 +37,7 @@ export function createFakeAppServer(scenario){ if(frame.method==='app/installed'){send({id:frame.id,result:{apps:[]}});return;} if(frame.method==='mcpServerStatus/list'){send({id:frame.id,result:{data:[],nextCursor:null}});return;} if(frame.method==='turn/start'){ + turn='automatic-turn-'+(++turnNumber); send({id:frame.id,result:{turn:{id:turn}}}); queueMicrotask(()=>{ if(scenario==='prose')complete(); diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index 688444c6aeb..47bffffaaf4 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -114,7 +114,7 @@ test('operation failure is not reported as success and cannot be blindly retried assert.equal((await gate.handle(check())).success,false);assert.equal((await gate.handle(check({callId:'retry'}))).success,false);assert.equal(calls.length,1); }); const repo=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..'); -async function hostScenario(scenario){ +async function hostScenario(scenario,cycles=1){ const area=fs.mkdtempSync(path.join(os.tmpdir(),'fm-native-host-loop-')); const runtime=path.join(area,'runtime'),home=path.join(area,'home'); fs.mkdirSync(runtime,{recursive:true});fs.mkdirSync(home,{recursive:true}); @@ -122,17 +122,17 @@ async function hostScenario(scenario){ fs.writeFileSync(path.join(runtime,'startup.log'),'deterministic startup digest\n'); fs.writeFileSync(path.join(runtime,'startup.finished'),'finished\n'); const input=new PassThrough(),output=new PassThrough(),error=new PassThrough(); - const notification={receipt:'receipt',challenge:'observed',message:'Controlled message',checkpointExit:124}; - let ackPending=false,acknowledgements=0,shutdowns=0,afterShutdown=0; + const notification=cycle=>({receipt:cycle===0?'receipt':`receipt-${cycle+1}`,challenge:'observed',message:'Controlled message',checkpointExit:124}); + let cycle=0,ackPending=false,acknowledgements=0,shutdowns=0,afterShutdown=0; const native=async(action,extra)=>{ if(shutdowns&&action!=='shutdown')afterShutdown++; if(action==='status')return {operationState:'ready'}; if(action==='result'){ - if(ackPending){ackPending=false;return {operationState:'acknowledged'};} - return {operationState:'delivered',notification}; + if(ackPending){ackPending=false;cycle++;return {operationState:'acknowledged'};} + return {operationState:'delivered',notification:notification(Math.min(cycle,cycles-1))}; } if(action==='ack'){ - assert.deepEqual(extra,{receipt:'receipt',observed:'observed'}); + assert.deepEqual(extra,{receipt:notification(cycle).receipt,observed:'observed'}); acknowledgements++;ackPending=true;return {operationState:'pending'}; } if(action==='shutdown'){shutdowns++;return {operationState:'stopped',reconciliationRequired:false};} @@ -143,8 +143,8 @@ async function hostScenario(scenario){ result=await runCodexHost({ env:{...process.env,FM_PROBE_HOME:runtime,FM_PROBE_CODE_ROOT:repo,FM_HOME:home,FM_PROBE_SESSION:'session',FM_PROBE_NONCE:'nonce'}, input,output,error,native,mcpServerNames:[],spawnAppServer:()=>createFakeAppServer(scenario),installSignalHandlers:false, - afterAutomaticTurn:()=>{ - setTimeout(()=>input.write('/quit\n'),20); + afterAutomaticTurn:({evidence})=>{ + if(evidence.automatic.length===cycles)setTimeout(()=>input.write('/quit\n'),20); return true; }, }); @@ -175,3 +175,15 @@ test('actual host loop suppresses only the exact successfully offered receipt',a ['fm_notification_check',true],['fm_notification_ack',true], ]); }); + +test('actual host loop gives consecutive automatic turns distinct protocol identities',async()=>{ + const result=await hostScenario('success',2); + assert.equal(result.failure,undefined); + assert.equal(result.acknowledgements,2); + assert.equal(new Set(result.host.turns.map(turn=>turn.id)).size,2); + assert.deepEqual(result.result.automatic.map(item=>item.outcome),['offered','offered']); + assert.deepEqual(result.host.tools.map(tool=>[tool.tool,tool.success]),[ + ['fm_notification_check',true],['fm_notification_ack',true], + ['fm_notification_check',true],['fm_notification_ack',true], + ]); +}); From 688941c7cdf565807faa7f77d34c1002c64fd61d Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 10:52:33 +1200 Subject: [PATCH 37/61] Scope publication rendezvous to its launcher scenario --- tests/fixtures/native-owner/Launcher.mjs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 6587e23a57d..96ff9155f9c 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -177,6 +177,8 @@ try { } const completedRestart=start(completedHome);await ready(completedRestart,completedReady.owner.generation);completedRestart.child.stdin.end();assert.equal((await bound(completedRestart.done,completedRestart,20000)).exit,0); records.push('notification publication transitions and completed acknowledgements remain admissible across restart'); +// Later scenarios must not inherit the publication-only continuation wait. +fs.writeFileSync(hostScript,"import {runCodexHost} from './codex-host-runtime.mjs';\nimport {createFakeAppServer} from './fake-app-server.mjs';\ntry { await runCodexHost({spawnAppServer:()=>createFakeAppServer('success'),mcpServerNames:[]}); } catch(error) { console.error(error.message); process.exitCode=1; }\n"); const wakeLib=path.join(code,'bin/fm-wake-lib.sh'),wakeLibActual=wakeLib+'.actual'; fs.renameSync(wakeLib,wakeLibActual);fs.copyFileSync(path.join(repo,'tests/fixtures/native-owner/fm-wake-lib-interrupt.sh'),wakeLib); const interruptedHome=path.join(area,'interrupted-ack-restart'),interruptedNote=enqueue(interruptedHome,'Preserve this interrupted acknowledgement for reconciliation.'); From 07437daaddcdc2975d2c39ed7c38ef66d9e74acc Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 11:44:37 +1200 Subject: [PATCH 38/61] Synchronize inbox publication with native admission --- bin/fm-inbox.sh | 33 +++---- bin/native-owner/NativeOperations.cs | 23 ++--- bin/native-owner/admit.sh | 8 ++ tests/fixtures/native-owner/Launcher.mjs | 22 +++-- .../native-owner/pause-inbox-publication.sh | 23 +++++ tests/fm-inbox-publication.test.sh | 94 +++++++++++++++++++ 6 files changed, 163 insertions(+), 40 deletions(-) create mode 100755 tests/fixtures/native-owner/pause-inbox-publication.sh create mode 100755 tests/fm-inbox-publication.test.sh diff --git a/bin/fm-inbox.sh b/bin/fm-inbox.sh index 914ca53a46a..5b81c528d20 100755 --- a/bin/fm-inbox.sh +++ b/bin/fm-inbox.sh @@ -152,23 +152,20 @@ aws_call() { # ---------------------------------------------------------------- note -# Append exactly one wake so firstmate picks the note up at its next drain. -# Failure to wake is NOT allowed to lose the note: the record is already on -# disk, so we report the wake failure and still exit non-zero loudly. -wake_for() { - local id=$1 summary=$2 lib="$FM_ROOT/bin/fm-wake-lib.sh" - if [ ! -r "$lib" ]; then - printf 'fm-inbox: note saved but NOT announced (missing %s)\n' "$lib" >&2 - return 1 - fi - # shellcheck source=/dev/null - FM_ROOT_OVERRIDE="$FM_ROOT" FM_HOME="$FM_HOME" STATE="$STATE" . "$lib" - fm_wake_append check "inbox:$id" "check: captain inbox note $id - $summary" -} - -queue_note() { - local source=$1 body=$2 extra=${3:-} +# Publish the note and its wake under the queue lock, including inbox staging. +# Failed announcement preserves the saved note and reports failure. +queue_note() ( + local source=$1 body=$2 extra=${3:-} lib="$FM_ROOT/bin/fm-wake-lib.sh" can_wake=0 [ -n "${body//[[:space:]]/}" ] || die "refusing to queue an empty note" + if [ -r "$lib" ]; then + # shellcheck source=/dev/null + FM_ROOT_OVERRIDE="$FM_ROOT" FM_HOME="$FM_HOME" STATE="$STATE" . "$lib" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + trap 'fm_lock_release "$FM_WAKE_QUEUE_LOCK"' EXIT + trap 'exit 130' INT + trap 'exit 143' TERM + can_wake=1 + fi mkdir -p "$INBOX" local tmp id summary staging_name @@ -191,12 +188,12 @@ queue_note() { summary=$(printf '%s' "$body" | tr '\n\t' ' ' | cut -c1-100) printf 'queued %s\n' "$id" printf ' %s\n' "$summary" - if wake_for "$id" "$summary"; then + if [ "$can_wake" = 1 ] && fm_wake_append_locked check "inbox:$id" "check: captain inbox note $id - $summary"; then printf ' firstmate will pick this up at its next check.\n' else die "note $id is saved at $INBOX/$id.note but firstmate was NOT woken" fi -} +) cmd_note() { local body diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index b54b44691da..0a9086d2c18 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -35,20 +35,15 @@ static void OwnerAdmission(string home,bool launch,string[] provenDeadGeneration string script=Path.Combine(CodeRoot,"bin","native-owner","admit.sh"); object evidence=NativeReceiptJournal.AdmissionEvidence(home); string input=""; - DateTime retryUntil=DateTime.UtcNow.AddSeconds(3); - while(true) { - var start=BashHelper(script,launch ? "launch" : "owned-operation",home);start.RedirectStandardInput=true;start.RedirectStandardError=true; - if(launch && provenDeadGenerations!=null && provenDeadGenerations.Length>0)start.EnvironmentVariables["FM_NATIVE_PROVEN_DEAD_GENERATIONS"]=string.Join(",",provenDeadGenerations); - if(evidence is string) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="token";input=(string)evidence;} - else if(evidence!=null) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="legacy";input=Json.Serialize(evidence);} - using(var process=Process.Start(start)) { - var error=process.StandardError.ReadToEndAsync(); - process.StandardInput.Write(input);process.StandardInput.Close(); - if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("The empty-fleet owner preflight exceeded its bound");} - if(process.ExitCode==0)return; - if(launch || DateTime.UtcNow>=retryUntil)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); - } - System.Threading.Thread.Sleep(100); + var start=BashHelper(script,launch ? "launch" : "owned-operation",home);start.RedirectStandardInput=true;start.RedirectStandardError=true; + if(launch && provenDeadGenerations!=null && provenDeadGenerations.Length>0)start.EnvironmentVariables["FM_NATIVE_PROVEN_DEAD_GENERATIONS"]=string.Join(",",provenDeadGenerations); + if(evidence is string) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="token";input=(string)evidence;} + else if(evidence!=null) {start.EnvironmentVariables["FM_NATIVE_ACK_EVIDENCE_KIND"]="legacy";input=Json.Serialize(evidence);} + using(var process=Process.Start(start)) { + var error=process.StandardError.ReadToEndAsync(); + process.StandardInput.Write(input);process.StandardInput.Close(); + if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("The empty-fleet owner preflight exceeded its bound");} + if(process.ExitCode!=0)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); } } internal static void EmptyFleet(string home,bool launch,string[] provenDeadGenerations=null) { diff --git a/bin/native-owner/admit.sh b/bin/native-owner/admit.sh index 9b3d15e0b78..1fb23702df8 100644 --- a/bin/native-owner/admit.sh +++ b/bin/native-owner/admit.sh @@ -39,6 +39,14 @@ fi if [ -e "$FM_HOME/state" ] || [ -L "$FM_HOME/state" ]; then export FM_STATE_OVERRIDE="$FM_HOME/state" . bin/fm-wake-lib.sh + # Read the inbox, queue, and recovery marker from one publication boundary. + if ! fm_lock_acquire_wait_bounded "$FM_WAKE_QUEUE_LOCK" 10; then + printf 'native admission could not acquire the wake queue lock within its bound\n' >&2 + exit 2 + fi + trap 'fm_lock_release "$FM_WAKE_QUEUE_LOCK"' EXIT + trap 'exit 130' INT + trap 'exit 143' TERM if ! fm_wake_native_empty_fleet_preflight "$FM_HOME/state" "$native_evidence"; then printf '%s\n' "${FM_WAKE_NATIVE_ADMISSION_ERROR:-the home contains unsupported wake state}" >&2 exit 2 diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 96ff9155f9c..18ca57112fa 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -70,16 +70,18 @@ function enqueue(home,message){ const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export PATH="$1/bin/native-owner/tools:/usr/bin:/bin:$PATH"; export FM_HOME; FM_HOME=$(cygpath -u "$3"); exec /usr/bin/bash "$1/bin/fm-inbox.sh" note "$2"','launcher-test',posix(code),message,home],{env,encoding:'utf8',timeout:30000}); assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr}));return result.stdout.trim().split(/\s+/)[1]; } +function pausedEnqueue(home,message,control){ + const env=Object.fromEntries(Object.entries(process.env).filter(([key])=>!key.startsWith('FM_')&&!key.startsWith('PI_'))); + Object.assign(env,{MSYS:'winsymlinks:nativestrict'}); + const child=spawn('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',posix(path.join(repo,'tests/fixtures/native-owner/pause-inbox-publication.sh')),posix(code),posix(home),posix(control),message],{env,stdio:['ignore','pipe','pipe']});let stdout='',stderr=''; + child.stdout.on('data',data=>stdout+=data);child.stderr.on('data',data=>stderr+=data); + return new Promise((resolve,reject)=>{child.on('error',reject);child.on('exit',exit=>resolve({exit,stdout,stderr}));}); +} function appendStartupWake(home,state){ const env={...process.env,FM_HOME:home,MSYS:'winsymlinks:nativestrict'}; const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export FM_ROOT_OVERRIDE FM_STATE_OVERRIDE; FM_ROOT_OVERRIDE=$(cygpath -u "$1"); FM_STATE_OVERRIDE=$(cygpath -u "$3"); . "$FM_ROOT_OVERRIDE/bin/fm-wake-lib.sh"; fm_wake_append_startup_network "$2"','startup-wake',code,state,path.join(home,'state')],{env,encoding:'utf8',timeout:30000}); assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); } -function appendInboxWake(home,id,summary){ - const env={...process.env,FM_HOME:home,MSYS:'winsymlinks:nativestrict'}; - const result=spawnSync('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','export FM_ROOT_OVERRIDE FM_STATE_OVERRIDE; FM_ROOT_OVERRIDE=$(cygpath -u "$1"); FM_STATE_OVERRIDE=$(cygpath -u "$4"); . "$FM_ROOT_OVERRIDE/bin/fm-wake-lib.sh"; fm_wake_append check "inbox:$2" "check: captain inbox note $2 - $3"','inbox-wake',code,id,summary,path.join(home,'state')],{env,encoding:'utf8',timeout:30000}); - assert.equal(result.status,0,JSON.stringify({error:result.error?.message,stdout:result.stdout,stderr:result.stderr})); -} const journalRows=home=>fs.readFileSync(path.join(home,'owner-receipts.jsonl'),'utf8').trim().split(/\r?\n/).filter(Boolean).map(JSON.parse); const acknowledgedNote=(home,note)=>journalRows(home).some(row=>row.event==='acknowledged'&&Array.isArray(row.payload?.notes)&&row.payload.notes.includes(note)); const pendingNote=(home,note)=>fs.readFileSync(path.join(home,'state/.wake-queue'),'utf8').split(/\r?\n/).filter(Boolean).some(row=>row.split('\t')[3]===`inbox:${note}`); @@ -161,18 +163,22 @@ const hostScript=path.join(code,'bin/native-owner/codex-host.mjs'),hostSource=fs fs.copyFileSync(path.join(repo,'tests/fixtures/native-owner/fake-app-server.mjs'),path.join(code,'bin/native-owner/fake-app-server.mjs')); fs.writeFileSync(hostScript,"import fs from 'node:fs';\nimport path from 'node:path';\nimport {runCodexHost} from './codex-host-runtime.mjs';\nimport {createFakeAppServer} from './fake-app-server.mjs';\nconst pause=ms=>new Promise(resolve=>setTimeout(resolve,ms));\ntry { await runCodexHost({spawnAppServer:()=>createFakeAppServer('success'),mcpServerNames:[],afterAutomaticTurn:async ({evidence})=>{if(evidence.automatic.length===1){fs.writeFileSync(path.join(process.env.FM_PROBE_HOME,'first-automatic-complete'),'ready');while(!fs.existsSync(path.join(process.env.FM_PROBE_HOME,'continue-after-first')))await pause(20);}}}); } catch(error) { console.error(error.message); process.exitCode=1; }\n"); const completedHome=path.join(area,'completed-ack-restart'),completedNote=enqueue(completedHome,'Complete this controlled acknowledgement before restart.'); +const publicationControl=path.join(area,'publication-control');fs.mkdirSync(publicationControl); const completedSession=start(completedHome,false);let completedReady; try { completedReady=await ready(completedSession); await waitUntil(completedSession,'completed acknowledgement',()=>acknowledgedNote(completedHome,completedNote)&&fs.existsSync(path.join(completedHome,'state/inbox/handled',completedNote+'.note'))&&!pendingNote(completedHome,completedNote)&&fs.existsSync(path.join(completedReady.runtime,'first-automatic-complete'))); - const transitionNote='transition-note',transitionMessage='Complete this notification after its producer finishes publishing.'; - fs.writeFileSync(path.join(completedHome,'state/inbox',transitionNote+'.note'),`id=${transitionNote}\nat=2026-09-18T00:00:00Z\nsource=text\n--\n${transitionMessage}\n`); + const transition=pausedEnqueue(completedHome,'Complete this notification after its producer finishes publishing.',publicationControl); + await waitUntil(completedSession,'producer publication boundary',()=>fs.existsSync(path.join(publicationControl,'paused'))); fs.writeFileSync(path.join(completedReady.runtime,'continue-after-first'),'continue'); await sleep(1000); - appendInboxWake(completedHome,transitionNote,transitionMessage); + fs.writeFileSync(path.join(publicationControl,'release'),'release'); + const publication=await bound(transition,completedSession,15000);assert.equal(publication.exit,0,publication.stderr); + const transitionNote=publication.stdout.match(/^queued (\S+)/m)?.[1];assert(transitionNote,publication.stdout); await waitUntil(completedSession,'notification publication transition',()=>acknowledgedNote(completedHome,transitionNote)&&fs.existsSync(path.join(completedHome,'state/inbox/handled',transitionNote+'.note'))&&!pendingNote(completedHome,transitionNote)&&fs.readFileSync(path.join(completedHome,'state/.wake-queue'),'utf8').trim()==='',60000); completedSession.child.stdin.write('/quit\n');assert.equal((await bound(completedSession.done,completedSession,20000)).exit,0); }finally{ + fs.writeFileSync(path.join(publicationControl,'release'),'release'); if(completedSession.child.exitCode===null){completedSession.child.stdin.write('/quit\n');if(completedReady)fs.writeFileSync(path.join(completedReady.runtime,'continue-after-first'),'continue');const stopped=await bound(completedSession.done,completedSession,20000);assert.equal(stopped.exit,0,stopped.stderr);} } const completedRestart=start(completedHome);await ready(completedRestart,completedReady.owner.generation);completedRestart.child.stdin.end();assert.equal((await bound(completedRestart.done,completedRestart,20000)).exit,0); diff --git a/tests/fixtures/native-owner/pause-inbox-publication.sh b/tests/fixtures/native-owner/pause-inbox-publication.sh new file mode 100755 index 00000000000..a0d9952be53 --- /dev/null +++ b/tests/fixtures/native-owner/pause-inbox-publication.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Test-only barrier around the real inbox producer's external rename. +# Usage: pause-inbox-publication.sh [before|after] +set -euo pipefail +root=$1 +export FM_HOME=$2 CONTROL=$3 PAUSE_MODE=${5:-before} +mkdir -p "$CONTROL" +mv() { + local last=${!#} i + case "$last" in + "$FM_HOME"/state/inbox/*.note) + if [ "$PAUSE_MODE" = after ]; then command mv "$@"; fi + printf '%s\n' "$BASHPID" > "$CONTROL/producer-pid" + touch "$CONTROL/paused" + for ((i=0;i<600;i++)); do [ ! -f "$CONTROL/release" ] || break; sleep .05; done + [ -f "$CONTROL/release" ] || return 90 + [ "$PAUSE_MODE" != after ] || return 0 + ;; + esac + command mv "$@" +} +export -f mv +exec bash "$root/bin/fm-inbox.sh" note "$4" diff --git a/tests/fm-inbox-publication.test.sh b/tests/fm-inbox-publication.test.sh new file mode 100755 index 00000000000..c8314805e0c --- /dev/null +++ b/tests/fm-inbox-publication.test.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# The real note producer and native admission must share one publication boundary. +set -euo pipefail +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TMP=$(mktemp -d) +cleanup() { + local child + touch "$TMP/release" + while IFS= read -r child; do kill "$child" 2>/dev/null || true; done < <(jobs -pr) + rm -rf "$TMP" +} +trap cleanup EXIT +fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; } +pass() { printf 'PASS: %s\n' "$*"; } +export MSYS=winsymlinks:nativestrict +# admit.sh's only Windows-specific shell operation is path conversion. +# On POSIX, preserve the already-canonical absolute test path. +if ! command -v cygpath >/dev/null 2>&1; then + cygpath() { [ "$1" = -u ] && printf '%s\n' "$2"; } + export -f cygpath +fi +wait_file() { + local file=$1 i + for ((i=0;i<400;i++)); do [ ! -f "$file" ] || return 0; sleep .02; done + fail "publication did not reach $file" +} +admit() { FM_HOME="$1" bash "$ROOT/bin/native-owner/admit.sh" owned-operation "$TMP/producer.out" 2>"$TMP/producer.err" & producer=$! +wait_file "$TMP/paused" +admit "$home" >"$TMP/reader.out" 2>"$TMP/reader.err" & reader=$! +sleep 1 +kill -0 "$reader" 2>/dev/null || fail 'admission returned while publication was incomplete' +touch "$TMP/release" +wait "$producer" || fail "producer failed: $(<"$TMP/producer.err")" +wait "$reader" || fail "reader failed: $(<"$TMP/reader.err")" +admit "$home" || fail 'reader leaked its queue lock' +pass 'admission waits for actual staging/publication and releases its lock' + +for kind in malformed orphan staging; do + target="$TMP/$kind"; cp -R "$home" "$target" + case "$kind" in + malformed) printf 'malformed\n' > "$target/state/.wake-queue" ;; + orphan) printf 'saved but unannounced\n' > "$target/state/inbox/orphan.note" ;; + staging) printf 'unfinished\n' > "$target/state/inbox/.staging-orphan" ;; + esac + cp -R "$target" "$TMP/$kind-before" + if admit "$target" >"$TMP/$kind.out" 2>&1; then fail "admitted $kind records"; fi + diff -r "$target" "$TMP/$kind-before" || fail "changed refused $kind records" + pass "refuses $kind records without changing them" +done + +# A producer interrupted after rename must release its lock but retain its note. +rm "$TMP/paused" "$TMP/release" "$TMP/producer-pid" +interrupted="$TMP/interrupted" +paused_note "$interrupted" after >"$TMP/interrupted.out" 2>"$TMP/interrupted.err" & producer=$! +wait_file "$TMP/paused" +kill -TERM "$(<"$TMP/producer-pid")" +if wait "$producer"; then fail 'interrupted producer reported success'; fi +notes=("$interrupted"/state/inbox/*.note) +[ -f "${notes[0]}" ] || fail 'interrupted producer lost its saved note' +[ ! -e "$interrupted/state/.wake-queue.lock" ] || fail 'interrupted producer leaked queue lock' +if admit "$interrupted" >"$TMP/interrupted-reader.out" 2>&1; then fail 'admitted interrupted incomplete publication'; fi +[ -f "${notes[0]}" ] || fail 'reader removed interrupted note' +pass 'interrupted producer preserves its note, releases lock, and remains inadmissible' + +# An unwritable sequence-counter path forces the real append to fail after saving. +failed="$TMP/append-failed"; mkdir -p "$failed/state/.wake-queue.seq" +if FM_HOME="$failed" bash "$ROOT/bin/fm-inbox.sh" note 'Keep me after append failure' >"$TMP/failed.out" 2>&1; then fail 'append failure reported success'; fi +notes=("$failed"/state/inbox/*.note) +[ -f "${notes[0]}" ] || fail 'append failure lost its saved note' +grep -q 'NOT woken' "$TMP/failed.out" || fail 'append failure was not reported' +[ ! -e "$failed/state/.wake-queue.lock" ] || fail 'append failure leaked queue lock' +if admit "$failed" >"$TMP/failed-reader.out" 2>&1; then fail 'admitted failed publication'; fi +pass 'failed append preserves the saved note and releases the lock' + +# A reader timeout must not release another live writer's lock or erase staging. +rm -f "$TMP/paused" "$TMP/release" "$TMP/producer-pid" +timeout_home="$TMP/timeout" +paused_note "$timeout_home" >"$TMP/timeout-producer.out" 2>"$TMP/timeout-producer.err" & producer=$! +wait_file "$TMP/paused" +if admit "$timeout_home" >"$TMP/timeout-reader.out" 2>&1; then fail 'contended admission unexpectedly succeeded'; fi +grep -q 'within its bound' "$TMP/timeout-reader.out" || fail 'reader did not report lock timeout' +kill -0 "$producer" || fail 'reader killed producer' +[ -e "$timeout_home/state/.wake-queue.lock" ] || fail 'reader removed producer lock' +touch "$TMP/release" +wait "$producer" || fail "producer could not finish after reader timeout: $(<"$TMP/timeout-producer.err")" +admit "$timeout_home" || fail 'completed publication refused after timeout' +pass 'bounded reader timeout preserves live producer ownership and later publication' From d51c412f35bd4f69bc7ddd74b99eab4bdadadc97 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 12:18:04 +1200 Subject: [PATCH 39/61] Canonicalize the paused inbox fixture home --- tests/fixtures/native-owner/Launcher.mjs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/fixtures/native-owner/Launcher.mjs b/tests/fixtures/native-owner/Launcher.mjs index 18ca57112fa..d8a4b5fda98 100644 --- a/tests/fixtures/native-owner/Launcher.mjs +++ b/tests/fixtures/native-owner/Launcher.mjs @@ -73,7 +73,7 @@ function enqueue(home,message){ function pausedEnqueue(home,message,control){ const env=Object.fromEntries(Object.entries(process.env).filter(([key])=>!key.startsWith('FM_')&&!key.startsWith('PI_'))); Object.assign(env,{MSYS:'winsymlinks:nativestrict'}); - const child=spawn('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc',posix(path.join(repo,'tests/fixtures/native-owner/pause-inbox-publication.sh')),posix(code),posix(home),posix(control),message],{env,stdio:['ignore','pipe','pipe']});let stdout='',stderr=''; + const child=spawn('C:/Program Files/Git/bin/bash.exe',['--noprofile','--norc','-c','home=$(cygpath -u "$3") || exit; exec /usr/bin/bash "$1" "$2" "$home" "$4" "$5"','paused-inbox',posix(path.join(repo,'tests/fixtures/native-owner/pause-inbox-publication.sh')),posix(code),home,posix(control),message],{env,stdio:['ignore','pipe','pipe']});let stdout='',stderr=''; child.stdout.on('data',data=>stdout+=data);child.stderr.on('data',data=>stderr+=data); return new Promise((resolve,reject)=>{child.on('error',reject);child.on('exit',exit=>resolve({exit,stdout,stderr}));}); } From 53b50a8a9f0179edf6ce4657d5fa925df4ba452d Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 13:00:29 +1200 Subject: [PATCH 40/61] Tolerate brief Windows reader contention when publishing host progress --- bin/native-owner/codex-host-runtime.mjs | 3 +- bin/native-owner/host-evidence.mjs | 22 ++++++ .../native-owner/host-evidence.test.mjs | 70 +++++++++++++++++++ tests/fm-native-owner-tool-gate.test.sh | 2 +- 4 files changed, 95 insertions(+), 2 deletions(-) create mode 100644 bin/native-owner/host-evidence.mjs create mode 100644 tests/fixtures/native-owner/host-evidence.test.mjs diff --git a/bin/native-owner/codex-host-runtime.mjs b/bin/native-owner/codex-host-runtime.mjs index 48eb1e089a4..e271e2d8e1c 100644 --- a/bin/native-owner/codex-host-runtime.mjs +++ b/bin/native-owner/codex-host-runtime.mjs @@ -5,6 +5,7 @@ import {spawn} from 'node:child_process'; import {createInterface} from 'node:readline'; import {confirmAutomaticNotificationOffer,createNotificationGate} from './codex-tool-gate.mjs'; import {createHostLifecycle,reconciliationWarning} from './host-lifecycle.mjs'; +import {saveHostEvidence} from './host-evidence.mjs'; import {discoverMcpServerNames,isolatedAppServerArgs,verifyExternalToolConfiguration,verifyExternalToolIsolation} from './app-server-policy.mjs'; export async function runCodexHost(options={}) { @@ -17,7 +18,7 @@ export async function runCodexHost(options={}) { let socket=null,channel=null; const input=[],pending=new Map(),turns=new Map(); const evidence={ready:false,turns:[],tools:[],automatic:[],digestDeliveredBeforeDeferred:false}; - const save=()=>{const file=path.join(runtime,'host.json');fs.writeFileSync(file+'.tmp',JSON.stringify(evidence,null,2));fs.renameSync(file+'.tmp',file);}; + const save=()=>saveHostEvidence(runtime,evidence); const consoleInput=createInterface({input:inputStream}); consoleInput.on('line',line=>{ if(line==='/quit'){void stop();return;} diff --git a/bin/native-owner/host-evidence.mjs b/bin/native-owner/host-evidence.mjs new file mode 100644 index 00000000000..129c28633ff --- /dev/null +++ b/bin/native-owner/host-evidence.mjs @@ -0,0 +1,22 @@ +// Progress snapshots only; durable notification receipts retain their own owner. +import fs from 'node:fs'; +import path from 'node:path'; +import {performance} from 'node:perf_hooks'; + +const sleeper=new Int32Array(new SharedArrayBuffer(4)); + +export function saveHostEvidence(runtime,evidence){ + const file=path.join(runtime,'host.json'),temporary=file+'.tmp'; + fs.writeFileSync(temporary,JSON.stringify(evidence,null,2)); + const deadline=performance.now()+250; + for(;;){ + try{fs.renameSync(temporary,file);return;} + catch(error){ + const remaining=deadline-performance.now(); + // Windows can refuse replacement while an ordinary reader has the old file open. + // Keep the complete old snapshot visible; persistent errors still stop the host. + if(process.platform!=='win32'||error.code!=='EPERM'||remaining<=0)throw error; + Atomics.wait(sleeper,0,0,Math.min(5,remaining)); + } + } +} diff --git a/tests/fixtures/native-owner/host-evidence.test.mjs b/tests/fixtures/native-owner/host-evidence.test.mjs new file mode 100644 index 00000000000..f4a333e6453 --- /dev/null +++ b/tests/fixtures/native-owner/host-evidence.test.mjs @@ -0,0 +1,70 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import {spawn} from 'node:child_process'; +import {saveHostEvidence} from '../../../bin/native-owner/host-evidence.mjs'; + +const readerSource=` +const fs=require('node:fs'); +const [file,mode]=process.argv.slice(1); +let fd=mode==='hold'?fs.openSync(file,'r'):null,reads=0,errors=0; +const timer=mode==='poll'?setInterval(()=>{try{JSON.parse(fs.readFileSync(file,'utf8'));reads++;}catch{errors++;}},100):null; +process.on('message',message=>{ + if(message==='release-later'){setTimeout(()=>{fs.closeSync(fd);fd=null;},75);process.send('scheduled');} + if(message==='stop'){if(timer)clearInterval(timer);if(fd!==null)fs.closeSync(fd);process.send({reads,errors});process.disconnect();} +}); +process.send('ready'); +`; +async function reader(file,mode){ + const child=spawn(process.execPath,['-e',readerSource,file,mode],{stdio:['ignore','ignore','inherit','ipc']}); + await new Promise((resolve,reject)=>{child.once('message',resolve);child.once('error',reject);}); + return {child,async releaseLater(){const scheduled=new Promise(resolve=>child.once('message',resolve));child.send('release-later');assert.equal(await scheduled,'scheduled');},async stop(){const summary=new Promise(resolve=>child.once('message',resolve));const exited=new Promise(resolve=>child.once('exit',resolve));child.send('stop');const value=await summary;assert.equal(await exited,0);return value;}}; +} +function home(t){const dir=fs.mkdtempSync(path.join(os.tmpdir(),'fm-host-evidence-'));t.after(()=>fs.rmSync(dir,{recursive:true,force:true}));return dir;} + +test('progress publication replaces complete JSON without changing receipt records',t=>{ + const dir=home(t),receipt=path.join(dir,'owner-receipts.jsonl');fs.writeFileSync(receipt,'preserved receipt\n'); + saveHostEvidence(dir,{version:1});saveHostEvidence(dir,{version:2}); + assert.deepEqual(JSON.parse(fs.readFileSync(path.join(dir,'host.json'),'utf8')),{version:2}); + assert.equal(fs.existsSync(path.join(dir,'host.json.tmp')),false);assert.equal(fs.readFileSync(receipt,'utf8'),'preserved receipt\n'); +}); + +test('unrelated write errors are propagated',t=>{ + const dir=home(t);fs.writeFileSync(path.join(dir,'host.json.tmp'),'not a directory'); + assert.throws(()=>saveHostEvidence(path.join(dir,'host.json.tmp'),{}),error=>['ENOTDIR','ENOENT'].includes(error.code)); +}); + +test('Windows reader contention clears without removing the old snapshot',{skip:process.platform!=='win32'},async t=>{ + const dir=home(t),file=path.join(dir,'host.json');saveHostEvidence(dir,{version:1}); + const held=await reader(file,'hold'); + try{ + fs.writeFileSync(file+'.tmp','{}');assert.throws(()=>fs.renameSync(file+'.tmp',file),{code:'EPERM'}); + assert.deepEqual(JSON.parse(fs.readFileSync(file,'utf8')),{version:1}); + await held.releaseLater();saveHostEvidence(dir,{version:2}); + assert.deepEqual(JSON.parse(fs.readFileSync(file,'utf8')),{version:2}); + }finally{await held.stop();} +}); + +test('persistent Windows refusal stays bounded and preserves the complete prior snapshot',{skip:process.platform!=='win32'},async t=>{ + const dir=home(t),file=path.join(dir,'host.json');saveHostEvidence(dir,{version:1}); + const held=await reader(file,'hold'); + try{ + const start=performance.now();assert.throws(()=>saveHostEvidence(dir,{version:2}),{code:'EPERM'}); + assert(performance.now()-start<2000,'Persistent replacement refusal exceeded the bound'); + assert.deepEqual(JSON.parse(fs.readFileSync(file,'utf8')),{version:1}); + assert.deepEqual(JSON.parse(fs.readFileSync(file+'.tmp','utf8')),{version:2}); + }finally{await held.stop();} + saveHostEvidence(dir,{version:2});assert.deepEqual(JSON.parse(fs.readFileSync(file,'utf8')),{version:2}); +}); + +test('Windows publication tolerates the launcher polling cadence with coherent reads',{skip:process.platform!=='win32'},async t=>{ + const dir=home(t),file=path.join(dir,'host.json');saveHostEvidence(dir,{version:0}); + const polling=await reader(file,'poll');let result; + try{ + const deadline=performance.now()+1500;let version=0; + while(performance.now()=3,'Reader did not exercise concurrent polling');assert.equal(result.errors,0); +}); diff --git a/tests/fm-native-owner-tool-gate.test.sh b/tests/fm-native-owner-tool-gate.test.sh index 433955587fd..0889568556e 100644 --- a/tests/fm-native-owner-tool-gate.test.sh +++ b/tests/fm-native-owner-tool-gate.test.sh @@ -2,4 +2,4 @@ # Portable behavioral tests of native host request and app-server policy. set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) -node --test "$ROOT/tests/fixtures/native-owner/tool-gate.test.mjs" "$ROOT/tests/fixtures/native-owner/host-lifecycle.test.mjs" "$ROOT/tests/fixtures/native-owner/app-server-policy.test.mjs" +node --test "$ROOT/tests/fixtures/native-owner/tool-gate.test.mjs" "$ROOT/tests/fixtures/native-owner/host-lifecycle.test.mjs" "$ROOT/tests/fixtures/native-owner/app-server-policy.test.mjs" "$ROOT/tests/fixtures/native-owner/host-evidence.test.mjs" From 69bef3bb05b13be66f97d2fdb3b121d1f995dce3 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 16:48:54 +1200 Subject: [PATCH 41/61] no-mistakes(review): Captain, harden ownership/validators; live proof blocked by CreateFileMapping Win32-5 --- bin/fm-session-lock-lib.sh | 13 +++++++-- bin/native-owner/NativeHomeLease.cs | 12 ++++---- bin/native-owner/NativeReceiptJournal.cs | 17 ++--------- tests/fm-claude-stop-autoarm-live-e2e.test.sh | 21 ++++++++++++++ tests/fm-session-lock-ancestry.test.sh | 29 ++++++++++++++++++- 5 files changed, 67 insertions(+), 25 deletions(-) diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index edfbbe6f1ab..f9c06f5cabf 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -183,10 +183,15 @@ fm_win_boundary_applies() { # Strip the namespace tag from $1, or return 1 when $1 is not a tagged pid. fm_win_untag_pid() { # + local winpid case "$1" in - "$FM_WIN_PID_PREFIX"[0-9]*) printf '%s' "${1#"$FM_WIN_PID_PREFIX"}"; return 0 ;; + "$FM_WIN_PID_PREFIX"*) winpid=${1#"$FM_WIN_PID_PREFIX"} ;; + *) return 1 ;; esac - return 1 + case "$winpid" in + ''|*[!0-9]*) return 1 ;; + esac + printf '%s' "$winpid" } # True when $1 is a well-formed session-lock identity: a local pid, a tagged @@ -205,7 +210,9 @@ fm_session_pid_valid() { # [ "${#native_id}" -eq 32 ] || return 1 case "$native_id" in *[!0-9a-f]*) return 1 ;; esac return 0 ;; - "$FM_WIN_PID_PREFIX"[0-9]*) return 0 ;; + "$FM_WIN_PID_PREFIX"*) + fm_win_untag_pid "$1" >/dev/null + return ;; ''|*[!0-9]*) return 1 ;; esac return 0 diff --git a/bin/native-owner/NativeHomeLease.cs b/bin/native-owner/NativeHomeLease.cs index 631182900e0..90011da1e06 100644 --- a/bin/native-owner/NativeHomeLease.cs +++ b/bin/native-owner/NativeHomeLease.cs @@ -42,18 +42,18 @@ public static string ValidateHomePath(string home) { return full; } static string Filename(string home) { return Path.Combine(ValidateHomePath(home),"owner-probe.json"); } - static void ValidateFile(FileStream stream,SecurityIdentifier user) { + internal static void ValidateFile(FileStream stream,SecurityIdentifier user,string resource) { var access=stream.GetAccessControl(); - if(!access.AreAccessRulesProtected || !access.GetOwner(typeof(SecurityIdentifier)).Equals(user)) throw new IOException("Owner record security differs; preserved"); - foreach(FileSystemAccessRule rule in access.GetAccessRules(true,true,typeof(SecurityIdentifier))) if(rule.AccessControlType==AccessControlType.Allow && !rule.IdentityReference.Equals(user)) throw new IOException("Owner record grants unexpected access; preserved"); + if(!access.AreAccessRulesProtected || !access.GetOwner(typeof(SecurityIdentifier)).Equals(user)) throw new IOException(resource+" security differs; preserved"); + foreach(FileSystemAccessRule rule in access.GetAccessRules(true,true,typeof(SecurityIdentifier))) if(rule.AccessControlType==AccessControlType.Allow && !rule.IdentityReference.Equals(user)) throw new IOException(resource+" grants unexpected access; preserved"); Info info; - if(!GetFileInformationByHandle(stream.SafeFileHandle.DangerousGetHandle(),out info) || info.links!=1 || (info.attributes&0x400)!=0) throw new IOException("Owner record file identity is unsafe; preserved"); + if(!GetFileInformationByHandle(stream.SafeFileHandle.DangerousGetHandle(),out info) || info.links!=1 || (info.attributes&0x400)!=0) throw new IOException(resource+" file identity is unsafe; preserved"); } static FileStream OpenExisting(string name,FileSystemRights rights,FileAccess access,FileShare share) { FileAttributes attributes=File.GetAttributes(name); if((attributes&FileAttributes.ReparsePoint)!=0 || (attributes&FileAttributes.Directory)!=0) throw new IOException("Owner record path is unsafe; preserved"); FileStream stream=rights==0 ? new FileStream(name,FileMode.Open,access,share) : new FileStream(name,FileMode.Open,rights,share,4096,FileOptions.None); - try { ValidateFile(stream,WindowsIdentity.GetCurrent().User);return stream; } + try { ValidateFile(stream,WindowsIdentity.GetCurrent().User,"Owner record");return stream; } catch { stream.Dispose();throw; } } static Dictionary Read(Stream stream) { @@ -117,7 +117,7 @@ public NativeHomeLease(string home) { try { try { file=new FileStream(name,FileMode.CreateNew,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security);created=true; } catch(IOException) { file=OpenExisting(name,FileSystemRights.Read|FileSystemRights.Write,0,FileShare.Read); } - ValidateFile(file,user); + ValidateFile(file,user,"Owner record"); if(!created && file.Length==0) throw new InvalidOperationException("Empty existing owner record is ambiguous; preserved"); if(file.Length>0) { var previous=Read(file); diff --git a/bin/native-owner/NativeReceiptJournal.cs b/bin/native-owner/NativeReceiptJournal.cs index a4596c0f03f..b684de90218 100644 --- a/bin/native-owner/NativeReceiptJournal.cs +++ b/bin/native-owner/NativeReceiptJournal.cs @@ -4,7 +4,6 @@ using System; using System.Collections.Generic; using System.IO; -using System.Runtime.InteropServices; using System.Security.AccessControl; using System.Security.Principal; using System.Text; @@ -18,21 +17,9 @@ public sealed class NativeReceiptJournal : IDisposable { object latestAcknowledgementEvidence; FileStream file; const long Limit = 16 * 1024 * 1024; - [StructLayout(LayoutKind.Sequential)] struct Info { - public uint attributes, createdLow, createdHigh, accessLow, accessHigh, writeLow, writeHigh; - public uint volume, sizeHigh, sizeLow, links, indexHigh, indexLow; - } - [DllImport("kernel32.dll", SetLastError=true)] static extern bool GetFileInformationByHandle(IntPtr handle, out Info info); NativeReceiptJournal(string selectedHome) { home=Path.GetFullPath(selectedHome).TrimEnd('\\','/'); } - static void ValidateFile(FileStream stream,SecurityIdentifier user) { - var access=stream.GetAccessControl(); - if(!access.AreAccessRulesProtected || !access.GetOwner(typeof(SecurityIdentifier)).Equals(user)) throw new IOException("Receipt journal security differs; preserved"); - foreach(FileSystemAccessRule rule in access.GetAccessRules(true,true,typeof(SecurityIdentifier))) if(rule.AccessControlType==AccessControlType.Allow && !rule.IdentityReference.Equals(user)) throw new IOException("Receipt journal grants unexpected access; preserved"); - Info info; - if(!GetFileInformationByHandle(stream.SafeFileHandle.DangerousGetHandle(),out info) || info.links!=1 || (info.attributes&0x400)!=0) throw new IOException("Receipt journal file identity is unsafe"); - } void Load(FileStream stream) { if(stream.Length>Limit || stream.Length==0) throw new IOException("Receipt journal is empty or oversized; preserved"); string content; @@ -50,7 +37,7 @@ public static object AdmissionEvidence(string selectedHome) { if((attributes&FileAttributes.ReparsePoint)!=0 || (attributes&FileAttributes.Directory)!=0) throw new IOException("Receipt journal path is unsafe; preserved"); var parser=new NativeReceiptJournal(home); using(var stream=new FileStream(name,FileMode.Open,FileAccess.Read,FileShare.ReadWrite)) { - ValidateFile(stream,WindowsIdentity.GetCurrent().User); + NativeHomeLease.ValidateFile(stream,WindowsIdentity.GetCurrent().User,"Receipt journal"); parser.Load(stream); } return parser.latestAcknowledgementEvidence; @@ -75,7 +62,7 @@ public NativeReceiptJournal(NativeHomeLease ownedLease, string ownerGeneration) if((File.GetAttributes(name)&FileAttributes.ReparsePoint)!=0) throw new IOException("Receipt journal is a reparse point"); file=new FileStream(name,FileMode.Open,FileSystemRights.Read|FileSystemRights.Write,FileShare.Read,4096,FileOptions.None,security); } - ValidateFile(file,user); + NativeHomeLease.ValidateFile(file,user,"Receipt journal"); if(!created) Load(file); Append("session",null,null); } catch { Dispose();throw; } diff --git a/tests/fm-claude-stop-autoarm-live-e2e.test.sh b/tests/fm-claude-stop-autoarm-live-e2e.test.sh index ae14d9f3af5..d5eaccbdead 100755 --- a/tests/fm-claude-stop-autoarm-live-e2e.test.sh +++ b/tests/fm-claude-stop-autoarm-live-e2e.test.sh @@ -65,6 +65,13 @@ cat > "$PROJECT/bin/tool-logger.sh" <<'SH' #!/usr/bin/env bash P=$(cat 2>/dev/null || true) printf '%s\n' "$P" | jq -r '.tool_input.command // "unknown"' >> "$FM_HOME/state/tool-calls.log" 2>/dev/null +if . "$CLAUDE_PROJECT_DIR/bin/fm-session-lock-lib.sh" && fm_win_boundary_applies && [ ! -e "$FM_HOME/state/windows-harness-identity.log" ]; then + identity=$(fm_harness_ancestry_pid 2>/dev/null || printf unresolved) + command=$(fm_win_command "${CLAUDE_PID:-}" 2>/dev/null || printf unresolved) + alive=no + fm_harness_pid_alive "$identity" && alive=yes + printf 'claude_pid=%s\nidentity=%s\ncommand=%s\nalive=%s\n' "${CLAUDE_PID:-}" "$identity" "$command" "$alive" > "$FM_HOME/state/windows-harness-identity.log" +fi exit 0 SH chmod +x "$PROJECT/bin/tool-logger.sh" @@ -126,6 +133,20 @@ grep -q 'stale: fixture-rapid-2' "$TRANSCRIPT" || fail "second rapid rewake reas || fail "fresh Claude session did not run session start first: $(cat "$HOME_DIR/state/tool-calls.log" 2>/dev/null)" [ "$(cat "$HOME_DIR/state/.lock" 2>/dev/null)" != 9999999 ] \ || fail "session start did not reclaim the stale dead-owner lock" +case "$(uname -s)" in + MINGW*|MSYS*|CYGWIN*) + WINDOWS_IDENTITY_LOG="$HOME_DIR/state/windows-harness-identity.log" + [ -s "$WINDOWS_IDENTITY_LOG" ] || fail "Claude Code $CLAUDE_VERSION exposed no Windows harness identity evidence" + WINDOWS_CLAUDE_PID=$(sed -n 's/^claude_pid=//p' "$WINDOWS_IDENTITY_LOG") + case "$WINDOWS_CLAUDE_PID" in ''|*[!0-9]*) fail "Claude Code $CLAUDE_VERSION published malformed CLAUDE_PID '$WINDOWS_CLAUDE_PID'" ;; esac + [ "$(sed -n 's/^identity=//p' "$WINDOWS_IDENTITY_LOG")" = "win:$WINDOWS_CLAUDE_PID" ] \ + || fail "Claude Code $CLAUDE_VERSION did not resolve its published Windows pid: $(cat "$WINDOWS_IDENTITY_LOG")" + [ "$(sed -n 's/^alive=//p' "$WINDOWS_IDENTITY_LOG")" = yes ] \ + || fail "Claude Code $CLAUDE_VERSION published pid did not identify a live harness: $(cat "$WINDOWS_IDENTITY_LOG")" + [ "$(cat "$HOME_DIR/state/.lock")" = "win:$WINDOWS_CLAUDE_PID" ] \ + || fail "session start did not bind the lock to Claude Code $CLAUDE_VERSION's published Windows pid" + ;; +esac if [ -f "$HOME_DIR/state/tool-calls.log" ]; then ! grep -q 'fm-watch-arm.sh' "$HOME_DIR/state/tool-calls.log" \ || fail "model issued an arm command despite Stop-owned continuity: $(cat "$HOME_DIR/state/tool-calls.log")" diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index b624fc6a29a..de1424e1905 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -410,14 +410,40 @@ test_a_published_identity_is_accepted_by_the_gates_that_read_the_lock() { # Still fail closed on the shapes a torn or hand-edited lock produces, and on # a bare tag carrying no pid at all. - for bad in '' 'win:' 'win:abc' 'abc' '70 0' '-1'; do + for bad in '' 'win:' 'win:abc' 'win:123oops' 'win:1:2' 'abc' '70 0' '-1'; do if lib_eval "$fakebin" "fm_session_pid_valid '$bad'"; then fail "the malformed lock value '$bad' was accepted as a usable identity" fi + case "$bad" in + win:*) + if lib_eval "$fakebin" "fm_win_untag_pid '$bad'"; then + fail "the malformed lock value '$bad' was reduced to a Windows pid" + fi + ;; + esac done pass "session-lock: a published identity is accepted by every gate that reads the lock" } +test_native_admission_preserves_a_partially_numeric_windows_identity() { + local dir out rc + dir="$TMP_ROOT/win-partial-identity" + mkdir -p "$dir/state" + printf '%s\n' 'win:123oops' > "$dir/state/.lock" + + set +e + out=$(FM_HOME="$dir" FM_STATE_OVERRIDE="$dir/state" "$ROOT/bin/fm-lock.sh" native-admission-predicate 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "native admission accepted a partially numeric Windows identity" + [ "$(cat "$dir/state/.lock")" = 'win:123oops' ] || fail "native admission changed an ambiguous Windows identity" + case "$out" in + *'session lock owner is unrecognized; native launch refused'*) ;; + *) fail "native admission did not identify the malformed owner: $out" ;; + esac + pass "session-lock: native admission preserves a partially numeric Windows identity" +} + test_cygwin_ps_without_o_still_resolves_a_local_harness() { local dir fakebin got dir="$TMP_ROOT/cygwin-local" @@ -691,6 +717,7 @@ test_windows_session_is_identified_from_its_published_pid test_windows_published_pid_is_confirmed_before_it_is_trusted test_windows_pid_is_never_resolved_as_a_cygwin_pid test_a_published_identity_is_accepted_by_the_gates_that_read_the_lock +test_native_admission_preserves_a_partially_numeric_windows_identity test_cygwin_ps_without_o_still_resolves_a_local_harness test_e2e_version_named_session_claims_the_home test_e2e_daemon_parented_session_claims_the_home From a47060a3f7fd654c2d0c4aa0172b175b61ace695 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 17:02:19 +1200 Subject: [PATCH 42/61] no-mistakes(review): Preserve malformed Windows owners during ordinary lock acquisition --- bin/fm-lock.sh | 4 ++ tests/fm-session-lock-ancestry.test.sh | 60 ++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index fc80b9a4507..4a444d4847a 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -35,6 +35,10 @@ fm_lock_holder_label() { } fm_lock_conflict_message() { + if ! fm_session_pid_valid "$1"; then + printf 'error: session lock owner is unrecognized; operate read-only until resolved' + return 0 + fi if fm_harness_pid_alive "$1"; then case "$1" in native:*) printf 'error: another live firstmate session holds the lock (native owner identity %s); operate read-only until resolved' "$1" ;; diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index de1424e1905..240f1097cd6 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -444,6 +444,65 @@ test_native_admission_preserves_a_partially_numeric_windows_identity() { pass "session-lock: native admission preserves a partially numeric Windows identity" } +test_ordinary_acquisition_preserves_malformed_windows_identities() { + local dir fakebin bad case_dir out rc index real_ln + dir="$TMP_ROOT/win-malformed-acquisition" + fakebin=$(cygwin_fakebin "$dir") + index=0 + + for bad in 'win:' 'win:abc' 'win:123oops' 'win:1:2'; do + index=$((index + 1)) + case_dir="$dir/preclaim-$index" + mkdir -p "$case_dir/state" + printf '%s\n' "$bad" > "$case_dir/state/.lock" + + set +e + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE="$WIN_TABLE" CLAUDE_PID=7204 "$ROOT/bin/fm-lock.sh" 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "ordinary acquisition accepted malformed Windows identity '$bad'" + [ "$(cat "$case_dir/state/.lock")" = "$bad" ] \ + || fail "ordinary acquisition changed malformed Windows identity '$bad'" + case "$out" in + *'session lock owner is unrecognized; operate read-only until resolved'*) ;; + *) fail "ordinary acquisition did not identify malformed owner '$bad': $out" ;; + esac + done + + case_dir="$dir/locked-recheck" + mkdir -p "$case_dir/state" + real_ln=$(command -v ln) + cat > "$fakebin/ln" <<'SH' +#!/usr/bin/env bash +set -u +target= +for target in "$@"; do + : +done +if [ "$target" = "$FM_TEST_LOCK_STATE/.lock.acquire" ]; then + printf '%s\n' 'win:123oops' > "$FM_TEST_LOCK_STATE/.lock" +fi +exec "$FM_TEST_REAL_LN" "$@" +SH + chmod +x "$fakebin/ln" + + set +e + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE="$WIN_TABLE" CLAUDE_PID=7204 FM_TEST_REAL_LN="$real_ln" \ + FM_TEST_LOCK_STATE="$case_dir/state" "$ROOT/bin/fm-lock.sh" 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "locked acquisition recheck accepted a malformed Windows identity" + [ "$(cat "$case_dir/state/.lock")" = 'win:123oops' ] \ + || fail "locked acquisition recheck changed a malformed Windows identity" + case "$out" in + *'session lock owner is unrecognized; operate read-only until resolved'*) ;; + *) fail "locked acquisition recheck did not identify the malformed owner: $out" ;; + esac + pass "session-lock: ordinary acquisition preserves malformed Windows identities at both checks" +} + test_cygwin_ps_without_o_still_resolves_a_local_harness() { local dir fakebin got dir="$TMP_ROOT/cygwin-local" @@ -718,6 +777,7 @@ test_windows_published_pid_is_confirmed_before_it_is_trusted test_windows_pid_is_never_resolved_as_a_cygwin_pid test_a_published_identity_is_accepted_by_the_gates_that_read_the_lock test_native_admission_preserves_a_partially_numeric_windows_identity +test_ordinary_acquisition_preserves_malformed_windows_identities test_cygwin_ps_without_o_still_resolves_a_local_harness test_e2e_version_named_session_claims_the_home test_e2e_daemon_parented_session_claims_the_home From 4dfb76640bf2020c5f23972612e5ce75c3838cd1 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 18:25:24 +1200 Subject: [PATCH 43/61] no-mistakes(review): Distinguish malformed owners; Claude proof remains blocked --- bin/fm-lock.sh | 4 ++- tests/fm-session-lock-ancestry.test.sh | 39 ++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index 4a444d4847a..69c06aeb0dc 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -121,7 +121,9 @@ if [ "${1:-}" = "status" ]; then echo "lock: unreadable" exit 0 } - if fm_harness_pid_alive "$old"; then + if ! fm_session_pid_valid "$old"; then + echo "lock: held by unrecognized owner with unknown health" + elif fm_harness_pid_alive "$old"; then echo "lock: held by live $(fm_lock_owner_label "$old")" elif fm_harness_pid_excludes "$old"; then echo "lock: held by native owner with unconfirmed health $old" diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index 240f1097cd6..e29a1ad0d80 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -503,6 +503,44 @@ SH pass "session-lock: ordinary acquisition preserves malformed Windows identities at both checks" } +test_status_distinguishes_malformed_windows_identities() { + local dir fakebin bad case_dir out index + dir="$TMP_ROOT/win-malformed-status" + fakebin=$(cygwin_fakebin "$dir") + index=0 + + for bad in 'win:' 'win:abc' 'win:123oops' 'win:1:2'; do + index=$((index + 1)) + case_dir="$dir/malformed-$index" + mkdir -p "$case_dir/state" + printf '%s\n' "$bad" > "$case_dir/state/.lock" + + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE="$WIN_TABLE" "$ROOT/bin/fm-lock.sh" status) + [ "$out" = 'lock: held by unrecognized owner with unknown health' ] \ + || fail "status classified malformed Windows identity '$bad' as known: $out" + [ "$(cat "$case_dir/state/.lock")" = "$bad" ] \ + || fail "status changed malformed Windows identity '$bad'" + done + + case_dir="$dir/live" + mkdir -p "$case_dir/state" + printf '%s\n' 'win:7204' > "$case_dir/state/.lock" + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE="$WIN_TABLE" "$ROOT/bin/fm-lock.sh" status) + [ "$out" = 'lock: held by live harness pid win:7204' ] \ + || fail "status lost a valid live Windows owner: $out" + + case_dir="$dir/dead" + mkdir -p "$case_dir/state" + printf '%s\n' 'win:9999' > "$case_dir/state/.lock" + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE="$WIN_TABLE" "$ROOT/bin/fm-lock.sh" status) + [ "$out" = 'lock: stale (pid win:9999 dead or not a harness)' ] \ + || fail "status lost a valid dead Windows owner: $out" + pass "session-lock: status preserves malformed Windows identities without confusing them with valid owners" +} + test_cygwin_ps_without_o_still_resolves_a_local_harness() { local dir fakebin got dir="$TMP_ROOT/cygwin-local" @@ -778,6 +816,7 @@ test_windows_pid_is_never_resolved_as_a_cygwin_pid test_a_published_identity_is_accepted_by_the_gates_that_read_the_lock test_native_admission_preserves_a_partially_numeric_windows_identity test_ordinary_acquisition_preserves_malformed_windows_identities +test_status_distinguishes_malformed_windows_identities test_cygwin_ps_without_o_still_resolves_a_local_harness test_e2e_version_named_session_claims_the_home test_e2e_daemon_parented_session_claims_the_home From 9760ce1d9258a7b4852b82a321d9afc60c661550 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 19:33:41 +1200 Subject: [PATCH 44/61] no-mistakes(review): Reject unresolved native process-event results --- bin/fm-procevent-lib.sh | 69 +++++++++++++++++++++ bin/fm-supervision-lib.sh | 4 ++ bin/native-owner/admit.sh | 2 + tests/fixtures/native-owner/NativeDriver.cs | 15 +++++ tests/fm-inbox-publication.test.sh | 68 ++++++++++++++++++++ 5 files changed, 158 insertions(+) diff --git a/bin/fm-procevent-lib.sh b/bin/fm-procevent-lib.sh index f5fce33dee1..f368933d7ab 100644 --- a/bin/fm-procevent-lib.sh +++ b/bin/fm-procevent-lib.sh @@ -1254,3 +1254,72 @@ fm_procevent_result_extension_load() { # fm_procevent_digest_valid "$FM_PROCEVENT_RESULT_EXTENSION_PACKAGE_DIGEST" || return 2 fm_procevent_digest_valid "$FM_PROCEVENT_RESULT_EXTENSION_BINDING_DIGEST" || return 2 } + +FM_PROCEVENT_INBOX_ERROR= +fm_procevent_inbox_has_only_handled_history() { # + local state=$1 inbox record name result id seq extension + inbox=$(fm_procevent_inbox_dir "$state") + FM_PROCEVENT_INBOX_ERROR= + if [ ! -e "$inbox" ] && [ ! -L "$inbox" ]; then + return 0 + fi + if [ ! -d "$inbox" ] || [ -L "$inbox" ]; then + FM_PROCEVENT_INBOX_ERROR="process-event result inbox is malformed at $inbox" + return 1 + fi + for record in "$inbox"/* "$inbox"/.[!.]* "$inbox"/..?*; do + if [ ! -e "$record" ] && [ ! -L "$record" ]; then + continue + fi + if [ ! -f "$record" ] || [ -L "$record" ]; then + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $record" + return 1 + fi + name=${record##*/} + case "$name" in + *.result) ;; + *.adapter|*.extension|*.handled) + result=${record%.*}.result + if [ ! -f "$result" ] || [ -L "$result" ]; then + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $record" + return 1 + fi + ;; + *) + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $record" + return 1 + ;; + esac + done + for result in "$inbox"/*.result; do + [ -e "$result" ] || continue + id=$(fm_procevent_result_source_id "$result") + seq=$(fm_procevent_result_sequence "$result") + if ! fm_procevent_source_id_valid "$id"; then + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" + return 1 + fi + case "$seq" in + ''|*[!0-9]*) + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" + return 1 + ;; + esac + if ! fm_procevent_result_adapter "$result" >/dev/null; then + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" + return 1 + fi + if ! fm_procevent_is_handled "$state" "$id" "$seq"; then + FM_PROCEVENT_INBOX_ERROR="unhandled process-event result is present at $result" + return 1 + fi + extension=${result%.result}.extension + if [ -e "$extension" ] || [ -L "$extension" ]; then + if ! fm_procevent_result_extension_load "$result"; then + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $extension" + return 1 + fi + fi + done + return 0 +} diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index 595d2107d1e..c22ded97a78 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -133,6 +133,10 @@ fm_supervision_residual_inputs_absent() { # return 1 fi done + if ! fm_procevent_inbox_has_only_handled_history "$state"; then + FM_SUP_RESIDUAL_ERROR=${FM_PROCEVENT_INBOX_ERROR:-"process-event result state is unresolved under $state/procevent-inbox"} + return 1 + fi } # fm_supervision_needed [grace-seconds] diff --git a/bin/native-owner/admit.sh b/bin/native-owner/admit.sh index 1fb23702df8..ed698f7d7c2 100644 --- a/bin/native-owner/admit.sh +++ b/bin/native-owner/admit.sh @@ -27,6 +27,8 @@ if [ "$mode" = launch ] && ! FM_STATE_OVERRIDE="$FM_HOME/state" bin/fm-lock.sh n fi . bin/fm-tasks-axi-lib.sh . bin/fm-backlog-transition-lib.sh +. bin/fm-pr-lib.sh +. bin/fm-procevent-lib.sh . bin/fm-supervision-lib.sh if ! fm_backlog_empty_fleet_preflight "$FM_HOME/state" "$FM_HOME/data"; then printf '%s\n' "${FM_BACKLOG_EMPTY_ERROR:-the home contains work-bearing records}" >&2 diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 75e314e0291..642ebefc372 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -160,6 +160,21 @@ static int EnvironmentTests() { refused=false;try{EmptyFleet(turnEnded,true);using(var lease=new NativeHomeLease(turnEnded)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(turnEndedRecord)!="preserve\n"||File.Exists(Path.Combine(turnEnded,"owner-probe.json"))||File.Exists(Path.Combine(turnEndedState,".lock")))throw new InvalidOperationException("Residual turn-end work passed admission or caused lease activity"); Console.WriteLine("PASS: residual turn-end work is preserved without lease activity"); + string pendingResultHome=Path.Combine(directory,"pending-process-event-result"),pendingResultState=Path.Combine(pendingResultHome,"state"),pendingResultInbox=Path.Combine(pendingResultState,"procevent-inbox"),pendingResult=Path.Combine(pendingResultInbox,"native-result.1.result"),pendingAdapter=Path.Combine(pendingResultInbox,"native-result.1.adapter"),pendingResultBody="preserve captured result\n"; + Directory.CreateDirectory(pendingResultInbox);File.WriteAllText(pendingResult,pendingResultBody);File.WriteAllText(pendingAdapter,"lavish\n"); + refused=false;try{EmptyFleet(pendingResultHome,true);using(var lease=new NativeHomeLease(pendingResultHome)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(pendingResult)!=pendingResultBody||File.Exists(Path.Combine(pendingResultHome,"owner-probe.json"))||Directory.Exists(Path.Combine(pendingResultState,"procevent")))throw new InvalidOperationException("Unhandled process-event result passed admission, changed, acquired a lease, or created source state"); + Console.WriteLine("PASS: unhandled process-event results are preserved and refused before lease acquisition"); + string handledResultHome=Path.Combine(directory,"handled-process-event-result"),handledResultState=Path.Combine(handledResultHome,"state"),handledResultInbox=Path.Combine(handledResultState,"procevent-inbox"),handledResult=Path.Combine(handledResultInbox,"native-history.1.result"),handledAdapter=Path.Combine(handledResultInbox,"native-history.1.adapter"),handledMarker=Path.Combine(handledResultInbox,"native-history.1.handled"); + Directory.CreateDirectory(handledResultInbox);File.WriteAllText(handledResult,"handled history\n");File.WriteAllText(handledAdapter,"lavish\n");File.WriteAllText(handledMarker,""); + EmptyFleet(handledResultHome,true); + if(File.ReadAllText(handledResult)!="handled history\n"||!File.Exists(handledMarker)||File.Exists(Path.Combine(handledResultHome,"owner-probe.json")))throw new InvalidOperationException("Handled process-event history was refused, changed, or acquired a lease"); + Console.WriteLine("PASS: handled process-event history remains admissible and unchanged"); + string ambiguousAckHome=Path.Combine(directory,"ambiguous-process-event-ack"),ambiguousAckState=Path.Combine(ambiguousAckHome,"state"),ambiguousAckInbox=Path.Combine(ambiguousAckState,"procevent-inbox"),ambiguousResult=Path.Combine(ambiguousAckInbox,"native-ambiguous.1.result"),ambiguousAdapter=Path.Combine(ambiguousAckInbox,"native-ambiguous.1.adapter"),ambiguousMarker=Path.Combine(ambiguousAckInbox,"native-ambiguous.1.handled"); + Directory.CreateDirectory(ambiguousAckInbox);File.WriteAllText(ambiguousResult,"ambiguous history\n");File.WriteAllText(ambiguousAdapter,"lavish\n");Directory.CreateDirectory(ambiguousMarker); + refused=false;try{EmptyFleet(ambiguousAckHome,true);using(var lease=new NativeHomeLease(ambiguousAckHome)){} }catch(InvalidOperationException){refused=true;} + if(!refused||!Directory.Exists(ambiguousMarker)||File.Exists(Path.Combine(ambiguousAckHome,"owner-probe.json")))throw new InvalidOperationException("Ambiguous process-event acknowledgement passed admission, changed, or acquired a lease"); + Console.WriteLine("PASS: ambiguous process-event acknowledgement state is preserved and refused"); string supported=Path.Combine(directory,"supported-notification"),supportedState=Path.Combine(supported,"state"),supportedInbox=Path.Combine(supportedState,"inbox"),supportedNote=Path.Combine(supportedInbox,"note-id.note"),supportedQueue=Path.Combine(supportedState,".wake-queue"),supportedMarker=Path.Combine(supportedState,".watcher-down"),supportedBody="preserve notification\n"; Directory.CreateDirectory(supportedInbox);File.WriteAllText(supportedNote,supportedBody);File.WriteAllText(Path.Combine(supportedState,".wake-queue.seq"),"1\n");File.WriteAllText(supportedQueue,"1\t1\tcheck\tinbox:note-id\tcheck: captain inbox note note-id - native admission test\n");File.WriteAllText(supportedMarker,"pending:downtime:supported\n"); EmptyFleet(supported,true); diff --git a/tests/fm-inbox-publication.test.sh b/tests/fm-inbox-publication.test.sh index c8314805e0c..7c58ec5e8d3 100755 --- a/tests/fm-inbox-publication.test.sh +++ b/tests/fm-inbox-publication.test.sh @@ -25,6 +25,74 @@ wait_file() { fail "publication did not reach $file" } admit() { FM_HOME="$1" bash "$ROOT/bin/native-owner/admit.sh" owned-operation "$payload" + bash -c ' + . "$1/bin/fm-pr-lib.sh" + . "$1/bin/fm-procevent-lib.sh" + fm_procevent_capture "$2/state" "$3" lavish "$4" + ' _ "$ROOT" "$home" "$id" "$payload" +} +handle_result() { + bash -c ' + . "$1/bin/fm-pr-lib.sh" + . "$1/bin/fm-procevent-lib.sh" + fm_procevent_mark_handled "$2/state" "$3" 1 + ' _ "$ROOT" "$1" "$2" +} +assert_refused_unchanged() { + local home=$1 label=$2 before="$1-before" + cp -R "$home" "$before" + if admit "$home" >"$TMP/$label.out" 2>&1; then fail "admitted $label"; fi + diff -r "$home" "$before" || fail "changed refused $label records" +} +assert_accepted_unchanged() { + local home=$1 label=$2 before="$1-before" + cp -R "$home" "$before" + admit "$home" >"$TMP/$label.out" 2>&1 || fail "refused $label: $(<"$TMP/$label.out")" + diff -r "$home" "$before" || fail "changed accepted $label records" +} + +empty_result_home="$TMP/empty-result-home" +mkdir -p "$empty_result_home/state" +assert_accepted_unchanged "$empty_result_home" empty-result-home +pass 'admits a home with no process-event result state' + +for source_shape in absent empty; do + result_home="$TMP/unhandled-$source_shape-source" + capture_result "$result_home" "pending-$source_shape" >/dev/null + [ "$source_shape" != empty ] || mkdir -p "$result_home/state/procevent" + assert_refused_unchanged "$result_home" "unhandled-$source_shape-source" + pass "refuses an unhandled result with an $source_shape source directory without changing it" +done + +queued_result_home="$TMP/queued-result" +FM_HOME="$queued_result_home" bash "$ROOT/bin/fm-inbox.sh" note 'Supported notification beside a captured result' >/dev/null +admit "$queued_result_home" >/dev/null || fail 'supported notification control was not admissible' +capture_result "$queued_result_home" queued-result >/dev/null +assert_refused_unchanged "$queued_result_home" queued-result +pass 'refuses an unhandled result even beside an otherwise supported queued notification' + +handled_result_home="$TMP/handled-result" +handled_result=$(capture_result "$handled_result_home" handled-result) +handle_result "$handled_result_home" handled-result +assert_accepted_unchanged "$handled_result_home" handled-result +[ -f "${handled_result%.result}.handled" ] || fail 'handled-history control lost its acknowledgement' +pass 'admits canonical handled process-event history without changing it' + +malformed_result_home="$TMP/malformed-result" +mkdir -p "$malformed_result_home/state/procevent-inbox" +printf 'ambiguous\n' > "$malformed_result_home/state/procevent-inbox/missing-sequence.result" +printf 'lavish\n' > "$malformed_result_home/state/procevent-inbox/missing-sequence.adapter" +assert_refused_unchanged "$malformed_result_home" malformed-result + +ambiguous_ack_home="$TMP/ambiguous-ack" +ambiguous_result=$(capture_result "$ambiguous_ack_home" ambiguous-ack) +mkdir "${ambiguous_result%.result}.handled" +assert_refused_unchanged "$ambiguous_ack_home" ambiguous-ack +pass 'refuses malformed results and ambiguous acknowledgement state without changing them' + # Pause an actual producer at its external rename, without production test hooks. paused_note() { bash "$ROOT/tests/fixtures/native-owner/pause-inbox-publication.sh" \ From 578ba5fb3636aaf52e7c9e8a65ccd822486538e8 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 19:55:03 +1200 Subject: [PATCH 45/61] no-mistakes(review): Reject hidden process-event results and repair fixture --- bin/fm-procevent-lib.sh | 61 +++++++++++++++--------------- tests/fm-inbox-publication.test.sh | 8 +++- 2 files changed, 37 insertions(+), 32 deletions(-) diff --git a/bin/fm-procevent-lib.sh b/bin/fm-procevent-lib.sh index f368933d7ab..8d136f72c5f 100644 --- a/bin/fm-procevent-lib.sh +++ b/bin/fm-procevent-lib.sh @@ -1277,7 +1277,36 @@ fm_procevent_inbox_has_only_handled_history() { # fi name=${record##*/} case "$name" in - *.result) ;; + *.result) + result=$record + id=$(fm_procevent_result_source_id "$result") + seq=$(fm_procevent_result_sequence "$result") + if ! fm_procevent_source_id_valid "$id"; then + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" + return 1 + fi + case "$seq" in + ''|*[!0-9]*) + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" + return 1 + ;; + esac + if ! fm_procevent_result_adapter "$result" >/dev/null; then + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" + return 1 + fi + if ! fm_procevent_is_handled "$state" "$id" "$seq"; then + FM_PROCEVENT_INBOX_ERROR="unhandled process-event result is present at $result" + return 1 + fi + extension=${result%.result}.extension + if [ -e "$extension" ] || [ -L "$extension" ]; then + if ! fm_procevent_result_extension_load "$result"; then + FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $extension" + return 1 + fi + fi + ;; *.adapter|*.extension|*.handled) result=${record%.*}.result if [ ! -f "$result" ] || [ -L "$result" ]; then @@ -1291,35 +1320,5 @@ fm_procevent_inbox_has_only_handled_history() { # ;; esac done - for result in "$inbox"/*.result; do - [ -e "$result" ] || continue - id=$(fm_procevent_result_source_id "$result") - seq=$(fm_procevent_result_sequence "$result") - if ! fm_procevent_source_id_valid "$id"; then - FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" - return 1 - fi - case "$seq" in - ''|*[!0-9]*) - FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" - return 1 - ;; - esac - if ! fm_procevent_result_adapter "$result" >/dev/null; then - FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $result" - return 1 - fi - if ! fm_procevent_is_handled "$state" "$id" "$seq"; then - FM_PROCEVENT_INBOX_ERROR="unhandled process-event result is present at $result" - return 1 - fi - extension=${result%.result}.extension - if [ -e "$extension" ] || [ -L "$extension" ]; then - if ! fm_procevent_result_extension_load "$result"; then - FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $extension" - return 1 - fi - fi - done return 0 } diff --git a/tests/fm-inbox-publication.test.sh b/tests/fm-inbox-publication.test.sh index 7c58ec5e8d3..d69ea0fb9c8 100755 --- a/tests/fm-inbox-publication.test.sh +++ b/tests/fm-inbox-publication.test.sh @@ -26,7 +26,8 @@ wait_file() { } admit() { FM_HOME="$1" bash "$ROOT/bin/native-owner/admit.sh" owned-operation "$payload" bash -c ' . "$1/bin/fm-pr-lib.sh" @@ -87,6 +88,11 @@ printf 'ambiguous\n' > "$malformed_result_home/state/procevent-inbox/missing-seq printf 'lavish\n' > "$malformed_result_home/state/procevent-inbox/missing-sequence.adapter" assert_refused_unchanged "$malformed_result_home" malformed-result +hidden_result_home="$TMP/hidden-result" +mkdir -p "$hidden_result_home/state/procevent-inbox" +printf 'preserve hidden result\n' > "$hidden_result_home/state/procevent-inbox/.lost.1.result" +assert_refused_unchanged "$hidden_result_home" hidden-result + ambiguous_ack_home="$TMP/ambiguous-ack" ambiguous_result=$(capture_result "$ambiguous_ack_home" ambiguous-ack) mkdir "${ambiguous_result%.result}.handled" From 2c8ba91db4215307ed5e1a3938fd94a74c1d91a4 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 20:38:43 +1200 Subject: [PATCH 46/61] no-mistakes(review): Fix native admission, selection, and owner-state handling --- bin/fm-harness.sh | 13 +-- bin/fm-lock.sh | 5 +- bin/fm-procevent-lib.sh | 1 + bin/fm-session-lock-lib.sh | 26 ++--- bin/fm-supervision-lib.sh | 5 + bin/fm-terminal-outcome-lib.sh | 49 +++++++++ bin/native-owner/admit.sh | 1 + tests/fixtures/native-owner/NativeDriver.cs | 22 ++++ tests/fm-session-lock-ancestry.test.sh | 111 ++++++++++++++++++++ 9 files changed, 212 insertions(+), 21 deletions(-) create mode 100644 bin/fm-terminal-outcome-lib.sh diff --git a/bin/fm-harness.sh b/bin/fm-harness.sh index 19ef1e017be..d6f3d9e760a 100755 --- a/bin/fm-harness.sh +++ b/bin/fm-harness.sh @@ -66,8 +66,8 @@ FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" -# shellcheck source=bin/fm-cursor-lib.sh -. "$SCRIPT_DIR/fm-cursor-lib.sh" +# shellcheck source=bin/fm-session-lock-lib.sh +. "$SCRIPT_DIR/fm-session-lock-lib.sh" # shellcheck source=bin/fm-gemini-lib.sh . "$SCRIPT_DIR/fm-gemini-lib.sh" @@ -399,13 +399,10 @@ harness_family() { # a harness-shaped path in some node process's arguments is weaker evidence # than a harness publishing its own identity. detect_own() { - local native_state native_home native_harness - native_home="${FM_STATE_OVERRIDE:-$FM_HOME/state}" - native_home=${native_home%/state} + local native_harness case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) - if [ -f "$native_home/owner-probe.json" ]; then - native_state=$(cygpath -w "${FM_STATE_OVERRIDE:-$FM_HOME/state}") || { echo unknown; return; } - native_harness=$(MSYS2_ARG_CONV_EXCL='*' "$SCRIPT_DIR/fm-native-owner.exe" owner harness "$native_state" 2>/dev/null) || { echo unknown; return; } + if fm_native_owner_selected; then + native_harness=$(fm_native_owner_call harness 2>/dev/null) || { echo unknown; return; } case "$native_harness" in codex) echo codex ;; *) echo unknown ;; esac return fi ;; diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index 69c06aeb0dc..040fcc28d00 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -126,7 +126,10 @@ if [ "${1:-}" = "status" ]; then elif fm_harness_pid_alive "$old"; then echo "lock: held by live $(fm_lock_owner_label "$old")" elif fm_harness_pid_excludes "$old"; then - echo "lock: held by native owner with unconfirmed health $old" + case "$old" in + native:*) echo "lock: held by native owner with unconfirmed health $old" ;; + *) echo "lock: held by owner with unconfirmed health ($(fm_lock_holder_label "$old"))" ;; + esac else echo "lock: stale ($(fm_lock_holder_label "$old") dead or not a harness)" fi diff --git a/bin/fm-procevent-lib.sh b/bin/fm-procevent-lib.sh index 8d136f72c5f..47644d2940d 100644 --- a/bin/fm-procevent-lib.sh +++ b/bin/fm-procevent-lib.sh @@ -1255,6 +1255,7 @@ fm_procevent_result_extension_load() { # fm_procevent_digest_valid "$FM_PROCEVENT_RESULT_EXTENSION_BINDING_DIGEST" || return 2 } +# shellcheck disable=SC2034 # Public result consumed by sourcing callers. FM_PROCEVENT_INBOX_ERROR= fm_procevent_inbox_has_only_handled_history() { # local state=$1 inbox record name result id seq extension diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index f9c06f5cabf..17606664580 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -228,15 +228,21 @@ fm_win_normalize_command() { # printf '%s' "${path%.exe}" } -# Print the normalized executable path of live Windows process $1, or return 1. +# Print the normalized executable path of live Windows process $1. # Presence in `ps -W` is also this bridge's liveness test, because kill -0 cannot # answer that question across the namespace boundary. +# Returns 1 when the process is absent and 2 when the table cannot be queried. fm_win_command() { # - local winpid=$1 out + local winpid=$1 table out case "$winpid" in ''|*[!0-9]*) return 1 ;; esac - out=$(ps -W 2>/dev/null | awk -v w="$winpid" '$4 == w { for (i = 8; i <= NF; i++) printf "%s%s", (i > 8 ? " " : ""), $i; exit }') + if ! table=$(ps -W 2>/dev/null); then + return 2 + fi + if ! out=$(printf '%s\n' "$table" | awk -v w="$winpid" '$4 == w { for (i = 8; i <= NF; i++) printf "%s%s", (i > 8 ? " " : ""), $i; exit }'); then + return 2 + fi [ -n "$out" ] || return 1 fm_win_normalize_command "$out" } @@ -360,13 +366,13 @@ EOF # kill -0 cannot see across that boundary and would report a live harness as # dead - which would hand a running session's home to a second one. fm_harness_pid_alive() { - local pid=$1 comm args winpid + local pid=$1 comm args winpid owner_rc case "$pid" in native:*) fm_native_owner_state "$pid" return ;; esac if winpid=$(fm_win_untag_pid "$pid"); then - comm=$(fm_win_command "$winpid") || return 1 + if comm=$(fm_win_command "$winpid"); then :; else owner_rc=$?; return "$owner_rc"; fi fm_harness_process_matches "$comm" "$comm" return fi @@ -379,13 +385,9 @@ fm_harness_pid_alive() { # Test exclusion rather than positive health: return 0 for a live or unknown # owner identity, and 1 only when the owner is proven dead. fm_harness_pid_excludes() { - local pid=$1 owner_rc - case "$pid" in native:*) - if fm_native_owner_state "$pid"; then return 0; else owner_rc=$?; fi - [ "$owner_rc" -ne 1 ] - return ;; - esac - fm_harness_pid_alive "$pid" + local owner_rc + if fm_harness_pid_alive "$1"; then return 0; else owner_rc=$?; fi + [ "$owner_rc" -ne 1 ] } # True when state dir $1 holds the owner identity of this native session or ANY diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index c22ded97a78..26f9d992c7e 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -115,6 +115,7 @@ fm_supervision_status() { return 0 } +# shellcheck disable=SC2034 # Public result consumed by sourcing callers. FM_SUP_RESIDUAL_ERROR= fm_supervision_residual_inputs_absent() { # local state=$1 record count @@ -133,6 +134,10 @@ fm_supervision_residual_inputs_absent() { # return 1 fi done + if ! fm_terminal_outcome_pending_absent "$state"; then + FM_SUP_RESIDUAL_ERROR=${FM_TERMINAL_OUTCOME_ERROR:-"terminal outcome state is unresolved under $state/terminal-outcomes"} + return 1 + fi if ! fm_procevent_inbox_has_only_handled_history "$state"; then FM_SUP_RESIDUAL_ERROR=${FM_PROCEVENT_INBOX_ERROR:-"process-event result state is unresolved under $state/procevent-inbox"} return 1 diff --git a/bin/fm-terminal-outcome-lib.sh b/bin/fm-terminal-outcome-lib.sh new file mode 100644 index 00000000000..1f2db387733 --- /dev/null +++ b/bin/fm-terminal-outcome-lib.sh @@ -0,0 +1,49 @@ +# shellcheck shell=bash + +# shellcheck disable=SC2034 # Public result consumed by sourcing callers. +FM_TERMINAL_OUTCOME_ERROR= + +fm_terminal_outcome_pending_absent() { # + local state=$1 directory record name fingerprint + FM_TERMINAL_OUTCOME_ERROR= + directory="$state/terminal-outcomes" + if [ ! -e "$directory" ] && [ ! -L "$directory" ]; then + return 0 + fi + if [ ! -d "$directory" ] || [ -L "$directory" ]; then + FM_TERMINAL_OUTCOME_ERROR="terminal outcome state is unresolved at $directory" + return 1 + fi + for record in "$directory"/* "$directory"/.[!.]* "$directory"/..?*; do + if [ ! -e "$record" ] && [ ! -L "$record" ]; then + continue + fi + if [ ! -f "$record" ] || [ -L "$record" ]; then + FM_TERMINAL_OUTCOME_ERROR="terminal outcome state is unresolved at $record" + return 1 + fi + name=${record##*/} + fingerprint=${name%.*} + if [ "${#fingerprint}" -ne 32 ]; then + FM_TERMINAL_OUTCOME_ERROR="terminal outcome state is unrecognized at $record" + return 1 + fi + case "$fingerprint" in *[!A-Fa-f0-9]*) + FM_TERMINAL_OUTCOME_ERROR="terminal outcome state is unrecognized at $record" + return 1 + ;; + esac + case "$name" in + *.pending) + FM_TERMINAL_OUTCOME_ERROR="unresolved terminal outcome is present at $record" + return 1 + ;; + *.presented|*.reported) ;; + *) + FM_TERMINAL_OUTCOME_ERROR="terminal outcome state is unrecognized at $record" + return 1 + ;; + esac + done + return 0 +} diff --git a/bin/native-owner/admit.sh b/bin/native-owner/admit.sh index ed698f7d7c2..33fc7357a6e 100644 --- a/bin/native-owner/admit.sh +++ b/bin/native-owner/admit.sh @@ -29,6 +29,7 @@ fi . bin/fm-backlog-transition-lib.sh . bin/fm-pr-lib.sh . bin/fm-procevent-lib.sh +. bin/fm-terminal-outcome-lib.sh . bin/fm-supervision-lib.sh if ! fm_backlog_empty_fleet_preflight "$FM_HOME/state" "$FM_HOME/data"; then printf '%s\n' "${FM_BACKLOG_EMPTY_ERROR:-the home contains work-bearing records}" >&2 diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 642ebefc372..fd3f6428810 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -160,6 +160,28 @@ static int EnvironmentTests() { refused=false;try{EmptyFleet(turnEnded,true);using(var lease=new NativeHomeLease(turnEnded)){} }catch(InvalidOperationException){refused=true;} if(!refused||File.ReadAllText(turnEndedRecord)!="preserve\n"||File.Exists(Path.Combine(turnEnded,"owner-probe.json"))||File.Exists(Path.Combine(turnEndedState,".lock")))throw new InvalidOperationException("Residual turn-end work passed admission or caused lease activity"); Console.WriteLine("PASS: residual turn-end work is preserved without lease activity"); + foreach(string phase in new [] {"upstream","presentation"}) { + string outcomeHome=Path.Combine(directory,"pending-terminal-outcome-"+phase),outcomeState=Path.Combine(outcomeHome,"state"),outcomeDirectory=Path.Combine(outcomeState,"terminal-outcomes"),fingerprint=new string(phase=="upstream"?'a':'b',32),outcomeRecord=Path.Combine(outcomeDirectory,fingerprint+".pending"),outcomeBody="schema=fm-terminal-outcome.v1\nfingerprint="+fingerprint+"\nphase="+phase+"\n"; + Directory.CreateDirectory(outcomeDirectory);File.WriteAllText(outcomeRecord,outcomeBody); + foreach(bool launch in new [] {true,false}) { + refused=false;try{EmptyFleet(outcomeHome,launch);using(var lease=new NativeHomeLease(outcomeHome)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(outcomeRecord)!=outcomeBody||File.Exists(Path.Combine(outcomeHome,"owner-probe.json"))||File.Exists(Path.Combine(outcomeState,".lock")))throw new InvalidOperationException("Pending terminal outcome passed admission, changed, or caused lease activity"); + } + } + Console.WriteLine("PASS: upstream and presentation terminal outcomes remain unresolved and preserved"); + foreach(string terminalState in new [] {"presented","reported"}) { + string outcomeHome=Path.Combine(directory,"settled-terminal-outcome-"+terminalState),outcomeState=Path.Combine(outcomeHome,"state"),outcomeDirectory=Path.Combine(outcomeState,"terminal-outcomes"),fingerprint=new string(terminalState=="presented"?'c':'d',32),outcomeRecord=Path.Combine(outcomeDirectory,fingerprint+"."+terminalState),outcomeBody="schema=fm-terminal-outcome.v1\nfingerprint="+fingerprint+"\n"; + Directory.CreateDirectory(outcomeDirectory);File.WriteAllText(outcomeRecord,outcomeBody);EmptyFleet(outcomeHome,true);EmptyFleet(outcomeHome,false); + if(File.ReadAllText(outcomeRecord)!=outcomeBody||File.Exists(Path.Combine(outcomeHome,"owner-probe.json"))||File.Exists(Path.Combine(outcomeState,".lock")))throw new InvalidOperationException("Settled terminal outcome was refused, changed, or caused lease activity"); + } + Console.WriteLine("PASS: presented and reported terminal-outcome history remains admissible and unchanged"); + string unknownOutcomeHome=Path.Combine(directory,"unknown-terminal-outcome"),unknownOutcomeState=Path.Combine(unknownOutcomeHome,"state"),unknownOutcomeDirectory=Path.Combine(unknownOutcomeState,"terminal-outcomes"),unknownOutcomeRecord=Path.Combine(unknownOutcomeDirectory,new string('e',32)+".unknown"),unknownOutcomeBody="preserve unknown terminal state\n"; + Directory.CreateDirectory(unknownOutcomeDirectory);File.WriteAllText(unknownOutcomeRecord,unknownOutcomeBody); + foreach(bool launch in new [] {true,false}) { + refused=false;try{EmptyFleet(unknownOutcomeHome,launch);using(var lease=new NativeHomeLease(unknownOutcomeHome)){} }catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(unknownOutcomeRecord)!=unknownOutcomeBody||File.Exists(Path.Combine(unknownOutcomeHome,"owner-probe.json"))||File.Exists(Path.Combine(unknownOutcomeState,".lock")))throw new InvalidOperationException("Unknown terminal-outcome state passed admission, changed, or caused lease activity"); + } + Console.WriteLine("PASS: unknown terminal-outcome state is preserved and refused"); string pendingResultHome=Path.Combine(directory,"pending-process-event-result"),pendingResultState=Path.Combine(pendingResultHome,"state"),pendingResultInbox=Path.Combine(pendingResultState,"procevent-inbox"),pendingResult=Path.Combine(pendingResultInbox,"native-result.1.result"),pendingAdapter=Path.Combine(pendingResultInbox,"native-result.1.adapter"),pendingResultBody="preserve captured result\n"; Directory.CreateDirectory(pendingResultInbox);File.WriteAllText(pendingResult,pendingResultBody);File.WriteAllText(pendingAdapter,"lavish\n"); refused=false;try{EmptyFleet(pendingResultHome,true);using(var lease=new NativeHomeLease(pendingResultHome)){} }catch(InvalidOperationException){refused=true;} diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index e29a1ad0d80..923012ffd91 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -300,6 +300,7 @@ while [ "$#" -gt 0 ]; do esac done if [ "$mode" = W ]; then + [ "${FM_TEST_WIN_QUERY_EXIT:-0}" = 0 ] || exit "$FM_TEST_WIN_QUERY_EXIT" printf '%s\n' ' PID PPID PGID WINPID TTY UID STIME COMMAND' [ -n "${FM_TEST_WIN_TABLE:-}" ] && printf '%s\n' "$FM_TEST_WIN_TABLE" exit 0 @@ -345,6 +346,114 @@ test_windows_session_is_identified_from_its_published_pid() { pass "session-lock: a Windows session is identified from its published pid across the severed parent link" } +test_windows_query_outcomes_reach_every_owner_gate() { + local dir fakebin case_dir out rc + dir="$TMP_ROOT/win-query-outcomes" + fakebin=$(cygwin_fakebin "$dir") + + case_dir="$dir/present" + mkdir -p "$case_dir/state" + printf '%s\n' 'win:7204' > "$case_dir/state/.lock" + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE="$WIN_TABLE" "$ROOT/bin/fm-lock.sh" status) + [ "$out" = 'lock: held by live harness pid win:7204' ] || fail "present Windows owner was not live: $out" + if PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE="$WIN_TABLE" "$ROOT/bin/fm-lock.sh" native-admission-predicate >/dev/null 2>&1; then + fail "native admission accepted a present Windows owner" + fi + set +e + PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_CYG_PPID=700 FM_TEST_WIN_TABLE="$WIN_TABLE" "$ROOT/bin/fm-lock.sh" >/dev/null 2>&1 + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "ordinary acquisition replaced a present Windows owner" + [ "$(cat "$case_dir/state/.lock")" = 'win:7204' ] || fail "present Windows owner changed during acquisition" + + case_dir="$dir/absent" + mkdir -p "$case_dir/state" + printf '%s\n' 'win:7204' > "$case_dir/state/.lock" + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE='' "$ROOT/bin/fm-lock.sh" status) + [ "$out" = 'lock: stale (pid win:7204 dead or not a harness)' ] || fail "absent Windows owner was not stale: $out" + PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_TABLE='' "$ROOT/bin/fm-lock.sh" native-admission-predicate >/dev/null \ + || fail "native admission did not accept a proven-absent Windows owner" + PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_CYG_PPID=700 FM_TEST_WIN_TABLE='' "$ROOT/bin/fm-lock.sh" >/dev/null \ + || fail "ordinary acquisition did not replace a proven-absent Windows owner" + [ "$(cat "$case_dir/state/.lock")" = 700 ] || fail "proven-absent Windows owner was not replaced by the current harness" + + case_dir="$dir/failed" + mkdir -p "$case_dir/state" + printf '%s\n' 'win:7204' > "$case_dir/state/.lock" + rc=0 + FM_TEST_WIN_QUERY_EXIT=7 lib_eval "$fakebin" 'fm_harness_pid_alive win:7204' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "failed Windows query did not remain unknown: $rc" + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_QUERY_EXIT=7 "$ROOT/bin/fm-lock.sh" status) + [ "$out" = 'lock: held by owner with unconfirmed health (pid win:7204)' ] || fail "failed Windows query was not reported as unknown: $out" + if PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_QUERY_EXIT=7 "$ROOT/bin/fm-lock.sh" native-admission-predicate >/dev/null 2>&1; then + fail "native admission accepted an owner after a failed Windows query" + fi + set +e + PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_CYG_PPID=700 FM_TEST_WIN_QUERY_EXIT=7 "$ROOT/bin/fm-lock.sh" >/dev/null 2>&1 + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "ordinary acquisition replaced an owner after a failed Windows query" + [ "$(cat "$case_dir/state/.lock")" = 'win:7204' ] || fail "failed Windows query changed the recorded owner" + pass "session-lock: Windows query presence, absence, and failure remain distinct through status and acquisition" +} + +test_harness_detection_uses_shared_native_selection() { + local dir bindir fakebin home out + dir="$TMP_ROOT/native-harness-selection" + bindir="$dir/bin" + home="$dir/home" + mkdir -p "$bindir" "$home/state" "$home/config" + cp "$ROOT/bin/fm-harness.sh" "$ROOT/bin/fm-session-lock-lib.sh" \ + "$ROOT/bin/fm-cursor-lib.sh" "$ROOT/bin/fm-gemini-lib.sh" "$bindir/" + fakebin=$(cygwin_fakebin "$dir/fake") + cat > "$fakebin/cygpath" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "${@: -1}" +SH + cat > "$bindir/fm-native-owner.exe" <<'SH' +#!/usr/bin/env bash +[ "${1:-}" = owner ] && [ "${2:-}" = harness ] || exit 2 +case "${FM_TEST_NATIVE_VERDICT:-}" in codex) printf '%s\n' codex ;; *) exit 2 ;; esac +SH + chmod +x "$bindir/fm-harness.sh" "$bindir/fm-native-owner.exe" "$fakebin/cygpath" + + out=$(env -u CLAUDECODE -u GROK_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ + -u GEMINI_CLI -u ATLASSIAN_AGENT_TYPE -u ROVODEV_CLI -u FM_OMP_HARNESS \ + -u FM_PI_HARNESS -u FM_STATE_OVERRIDE PATH="$fakebin:$PATH" FM_HOME="$home" \ + PI_CODING_AGENT=true FM_TEST_NATIVE_VERDICT=codex "$bindir/fm-harness.sh") + [ "$out" = pi ] || fail "ordinary marker routing changed without a native record: $out" + + printf '%s\n' 'native:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' > "$home/state/.lock" + out=$(env -u CLAUDECODE -u GROK_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ + -u GEMINI_CLI -u ATLASSIAN_AGENT_TYPE -u ROVODEV_CLI -u FM_OMP_HARNESS \ + -u FM_PI_HARNESS -u FM_STATE_OVERRIDE PATH="$fakebin:$PATH" FM_HOME="$home" \ + PI_CODING_AGENT=true "$bindir/fm-harness.sh") + [ "$out" = unknown ] || fail "native lock without authenticated proof fell through to a marker: $out" + + rm "$home/state/.lock" + printf '%s\n' '{}' > "$home/owner-probe.json" + out=$(env -u CLAUDECODE -u GROK_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ + -u GEMINI_CLI -u ATLASSIAN_AGENT_TYPE -u ROVODEV_CLI -u FM_OMP_HARNESS \ + -u FM_PI_HARNESS -u FM_STATE_OVERRIDE PATH="$fakebin:$PATH" FM_HOME="$home" \ + PI_CODING_AGENT=true "$bindir/fm-harness.sh") + [ "$out" = unknown ] || fail "unusable native probe fell through to a marker: $out" + out=$(env -u CLAUDECODE -u GROK_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ + -u GEMINI_CLI -u ATLASSIAN_AGENT_TYPE -u ROVODEV_CLI -u FM_OMP_HARNESS \ + -u FM_PI_HARNESS -u FM_STATE_OVERRIDE PATH="$fakebin:$PATH" FM_HOME="$home" \ + PI_CODING_AGENT=true FM_TEST_NATIVE_VERDICT=codex "$bindir/fm-harness.sh") + [ "$out" = codex ] || fail "authenticated native probe did not select Codex: $out" + pass "harness detection delegates durable native selection without marker fallback" +} + test_windows_published_pid_is_confirmed_before_it_is_trusted() { local dir fakebin dir="$TMP_ROOT/win-unconfirmed" @@ -811,6 +920,8 @@ test_ordinary_paths_are_never_harness_processes test_harness_beyond_a_gap_never_owns_the_lock test_competing_version_named_session_is_seen_as_live test_windows_session_is_identified_from_its_published_pid +test_windows_query_outcomes_reach_every_owner_gate +test_harness_detection_uses_shared_native_selection test_windows_published_pid_is_confirmed_before_it_is_trusted test_windows_pid_is_never_resolved_as_a_cygwin_pid test_a_published_identity_is_accepted_by_the_gates_that_read_the_lock From 39591e42b00a08fe8d6d376adfea6728f7b2b413 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 21:22:38 +1200 Subject: [PATCH 47/61] no-mistakes(review): Support fallback receipts and repair source-aware lint --- bin/fm-procevent-lib.sh | 1 + bin/fm-supervision-lib.sh | 1 + bin/fm-terminal-outcome-lib.sh | 11 +++++++---- tests/fixtures/native-owner/NativeDriver.cs | 20 ++++++++++++++++++++ 4 files changed, 29 insertions(+), 4 deletions(-) diff --git a/bin/fm-procevent-lib.sh b/bin/fm-procevent-lib.sh index 47644d2940d..6f41053b5cd 100644 --- a/bin/fm-procevent-lib.sh +++ b/bin/fm-procevent-lib.sh @@ -1316,6 +1316,7 @@ fm_procevent_inbox_has_only_handled_history() { # fi ;; *) + # shellcheck disable=SC2034 # Public result consumed by sourcing callers. FM_PROCEVENT_INBOX_ERROR="process-event result history is malformed at $record" return 1 ;; diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index 26f9d992c7e..92506f1a584 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -139,6 +139,7 @@ fm_supervision_residual_inputs_absent() { # return 1 fi if ! fm_procevent_inbox_has_only_handled_history "$state"; then + # shellcheck disable=SC2034 # Public result consumed by sourcing callers. FM_SUP_RESIDUAL_ERROR=${FM_PROCEVENT_INBOX_ERROR:-"process-event result state is unresolved under $state/procevent-inbox"} return 1 fi diff --git a/bin/fm-terminal-outcome-lib.sh b/bin/fm-terminal-outcome-lib.sh index 1f2db387733..e6d78a41ce0 100644 --- a/bin/fm-terminal-outcome-lib.sh +++ b/bin/fm-terminal-outcome-lib.sh @@ -24,10 +24,13 @@ fm_terminal_outcome_pending_absent() { # fi name=${record##*/} fingerprint=${name%.*} - if [ "${#fingerprint}" -ne 32 ]; then - FM_TERMINAL_OUTCOME_ERROR="terminal outcome state is unrecognized at $record" - return 1 - fi + case "${#fingerprint}" in + 16|32) ;; + *) + FM_TERMINAL_OUTCOME_ERROR="terminal outcome state is unrecognized at $record" + return 1 + ;; + esac case "$fingerprint" in *[!A-Fa-f0-9]*) FM_TERMINAL_OUTCOME_ERROR="terminal outcome state is unrecognized at $record" return 1 diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index fd3f6428810..7d0482d638a 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -66,6 +66,15 @@ static void RunFirstmate(string role,bool shouldSucceed) { if((p.ExitCode==0)!=shouldSucceed) throw new IOException("Unexpected Firstmate operation result for "+role); } } + static void AcknowledgeTerminalOutcome(string home,string fingerprint) { + string script=Path.Combine(CodeRoot,"bin","fm-inactive-reconcile.sh"); + var start=BashHelper(script,"acknowledge "+Quote(fingerprint),home);start.RedirectStandardError=true; + using(var process=Process.Start(start)) { + var error=process.StandardError.ReadToEndAsync(); + if(!process.WaitForExit(30000)){process.Kill();throw new TimeoutException("Terminal outcome acknowledgement exceeded its bound");} + if(process.ExitCode!=0)throw new InvalidOperationException("Terminal outcome acknowledgement failed: "+error.Result.Trim()); + } + } static int Fixture() { foreach (string c in new [] {"root", "wrong-session", "wrong-home", "wrong-capability"}) Client(c); var child = Process.Start(new ProcessStartInfo(OwnExe, "client inherited-child") { UseShellExecute=false }); @@ -169,6 +178,17 @@ static int EnvironmentTests() { } } Console.WriteLine("PASS: upstream and presentation terminal outcomes remain unresolved and preserved"); + foreach(int fingerprintLength in new [] {32,16}) { + string outcomeHome=Path.Combine(directory,"acknowledged-terminal-outcome-"+fingerprintLength),outcomeState=Path.Combine(outcomeHome,"state"),outcomeDirectory=Path.Combine(outcomeState,"terminal-outcomes"),fingerprint=new string(fingerprintLength==32?'c':'d',fingerprintLength),pending=Path.Combine(outcomeDirectory,fingerprint+".pending"),presented=Path.Combine(outcomeDirectory,fingerprint+".presented"),outcomeBody="schema=fm-terminal-outcome.v1\nfingerprint="+fingerprint+"\ntask_id=fixture\nincarnation=fixture-1\nstate=done\noutcome_key=fixture-complete\norigin=direct\nphase=presentation\npr=\ncreated_epoch=1\nnotice_emitted=0\n"; + Directory.CreateDirectory(outcomeDirectory);File.WriteAllText(pending,outcomeBody); + refused=false;try{EmptyFleet(outcomeHome,true);}catch(InvalidOperationException){refused=true;} + if(!refused||File.ReadAllText(pending)!=outcomeBody)throw new InvalidOperationException("Pending producer-format terminal outcome passed admission or changed"); + AcknowledgeTerminalOutcome(outcomeHome,fingerprint); + if(File.Exists(pending)||!File.Exists(presented))throw new InvalidOperationException("Terminal outcome acknowledgement did not commit the presentation transition"); + EmptyFleet(outcomeHome,true);EmptyFleet(outcomeHome,false); + if(File.ReadAllText(presented)!=outcomeBody||File.Exists(Path.Combine(outcomeHome,"owner-probe.json"))||File.Exists(Path.Combine(outcomeState,".lock")))throw new InvalidOperationException("Acknowledged producer-format terminal outcome was refused, changed, or caused lease activity"); + } + Console.WriteLine("PASS: acknowledged 32- and 16-hex producer terminal outcomes remain admissible and unchanged"); foreach(string terminalState in new [] {"presented","reported"}) { string outcomeHome=Path.Combine(directory,"settled-terminal-outcome-"+terminalState),outcomeState=Path.Combine(outcomeHome,"state"),outcomeDirectory=Path.Combine(outcomeState,"terminal-outcomes"),fingerprint=new string(terminalState=="presented"?'c':'d',32),outcomeRecord=Path.Combine(outcomeDirectory,fingerprint+"."+terminalState),outcomeBody="schema=fm-terminal-outcome.v1\nfingerprint="+fingerprint+"\n"; Directory.CreateDirectory(outcomeDirectory);File.WriteAllText(outcomeRecord,outcomeBody);EmptyFleet(outcomeHome,true);EmptyFleet(outcomeHome,false); From 0e432cd3815607c12d024e4378d3f0b8beb0c4f9 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 21:45:53 +1200 Subject: [PATCH 48/61] no-mistakes(review): Preserve native owner uncertainty and admission boundaries --- bin/fm-lock.sh | 34 +++---- bin/fm-session-lock-lib.sh | 10 +-- bin/native-owner/NativeOperations.cs | 7 +- tests/fixtures/native-owner/NativeDriver.cs | 14 +++ tests/fm-session-lock-ancestry.test.sh | 99 +++++++++++++++++++-- 5 files changed, 136 insertions(+), 28 deletions(-) diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index 040fcc28d00..3e608ec79bd 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -35,6 +35,7 @@ fm_lock_holder_label() { } fm_lock_conflict_message() { + local owner_rc if ! fm_session_pid_valid "$1"; then printf 'error: session lock owner is unrecognized; operate read-only until resolved' return 0 @@ -45,8 +46,10 @@ fm_lock_conflict_message() { *) printf 'error: another live firstmate session holds the lock (pid %s); operate read-only until resolved' "$1" ;; esac return 0 + else + owner_rc=$? fi - if fm_harness_pid_excludes "$1"; then + if [ "$owner_rc" -ne 1 ]; then printf 'error: another firstmate session may hold the lock (%s); operate read-only until resolved' "$(fm_lock_holder_label "$1")" return 0 fi @@ -99,12 +102,8 @@ if [ "${1:-}" = "native-admission-predicate" ]; then fi ;; esac - if fm_harness_pid_excludes "$old"; then - if conflict=$(fm_lock_conflict_message "$old"); then - echo "$conflict" >&2 - else - echo "error: another firstmate session may hold the lock; native launch refused" >&2 - fi + if conflict=$(fm_lock_conflict_message "$old"); then + echo "$conflict" >&2 exit 1 fi exit 0 @@ -123,15 +122,20 @@ if [ "${1:-}" = "status" ]; then } if ! fm_session_pid_valid "$old"; then echo "lock: held by unrecognized owner with unknown health" - elif fm_harness_pid_alive "$old"; then - echo "lock: held by live $(fm_lock_owner_label "$old")" - elif fm_harness_pid_excludes "$old"; then - case "$old" in - native:*) echo "lock: held by native owner with unconfirmed health $old" ;; - *) echo "lock: held by owner with unconfirmed health ($(fm_lock_holder_label "$old"))" ;; - esac else - echo "lock: stale ($(fm_lock_holder_label "$old") dead or not a harness)" + if fm_harness_pid_alive "$old"; then + echo "lock: held by live $(fm_lock_owner_label "$old")" + else + owner_rc=$? + if [ "$owner_rc" -eq 1 ]; then + echo "lock: stale ($(fm_lock_holder_label "$old") dead or not a harness)" + else + case "$old" in + native:*) echo "lock: held by native owner with unconfirmed health $old" ;; + *) echo "lock: held by owner with unconfirmed health ($(fm_lock_holder_label "$old"))" ;; + esac + fi + fi fi exit 0 fi diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index 17606664580..627a3aae034 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -148,8 +148,9 @@ FM_WIN_PID_PREFIX='win:' # Native routing is selected by durable home records, never an inherited role. FM_NATIVE_OWNER_BIN="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-native-owner.exe" fm_native_owner_selected() { - local state="${FM_STATE_OVERRIDE:-${FM_HOME:-}/state}" value - [ -f "${state%/state}/owner-probe.json" ] && return 0 + local state="${FM_STATE_OVERRIDE:-${FM_HOME:-}/state}" probe value + probe="${state%/state}/owner-probe.json" + if [ -e "$probe" ] || [ -L "$probe" ]; then return 0; fi value=$(cat "$state/.lock" 2>/dev/null || true) case "$value" in native:*) return 0 ;; esac return 1 @@ -399,10 +400,9 @@ fm_harness_pid_excludes() { # lock, a malformed lock, a lock held by a harness outside this ancestry, or an # ancestry that cannot be resolved all refuse ownership. fm_session_lock_owned_by_self() { - local state=$1 lock_pid pids pid native_state + local state=$1 lock_pid pids pid if fm_win_boundary_applies && FM_STATE_OVERRIDE="$state" fm_native_owner_selected; then - native_state=$(cygpath -w "$state") || return 1 - MSYS2_ARG_CONV_EXCL='*' "$FM_NATIVE_OWNER_BIN" owner owns "$native_state" + FM_STATE_OVERRIDE="$state" fm_native_owner_call owns return fi lock_pid=$(cat "$state/.lock" 2>/dev/null || true) diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index 0a9086d2c18..b95ddb1deb0 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -46,9 +46,14 @@ static void OwnerAdmission(string home,bool launch,string[] provenDeadGeneration if(process.ExitCode!=0)throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing records were preserved: "+error.Result.Trim()); } } + static bool PathPresent(string name) { + try { File.GetAttributes(name);return true; } + catch(FileNotFoundException) { return false; } + catch(DirectoryNotFoundException) { return false; } + } internal static void EmptyFleet(string home,bool launch,string[] provenDeadGenerations=null) { string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); - if((Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || File.Exists(Path.Combine(home,"data","secondmates.md")) || File.Exists(Path.Combine(home,"data","projects.md")) || File.Exists(Path.Combine(home,".env")) || File.Exists(Path.Combine(home,"config","x-mode.env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); + if((Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || PathPresent(Path.Combine(home,"data","secondmates.md")) || PathPresent(Path.Combine(home,"data","projects.md")) || PathPresent(Path.Combine(home,".env")) || PathPresent(Path.Combine(home,"config","x-mode.env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); OwnerAdmission(home,launch,provenDeadGenerations); } static IntPtr FileHandle(string path, uint access, uint creation) { diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 7d0482d638a..8c3c99de96c 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -113,6 +113,20 @@ static int EnvironmentTests() { if(File.Exists(injected)||leaked.Length!=4)throw new InvalidOperationException("Denied inherited environment reached the native host"); foreach(string key in leaked)if(!key.StartsWith("FM_PROBE_",StringComparison.Ordinal))throw new InvalidOperationException("Unexpected inherited environment reached the native host"); Console.WriteLine("PASS: inherited Windows environment denylist is case-insensitive"); + foreach(string shape in new [] {"absent","regular","directory"}) { + string sentinelHome=Path.Combine(directory,"registry-sentinel-"+shape),registry=Path.Combine(sentinelHome,"data","projects.md"); + Directory.CreateDirectory(sentinelHome); + if(shape=="regular") { Directory.CreateDirectory(Path.GetDirectoryName(registry));File.WriteAllText(registry,"preserve registry\n"); } + if(shape=="directory") Directory.CreateDirectory(registry); + foreach(bool launch in new [] {true,false}) { + bool sentinelRefused=false;try{EmptyFleet(sentinelHome,launch);}catch(InvalidOperationException){sentinelRefused=true;} + if(sentinelRefused!=(shape!="absent"))throw new InvalidOperationException("Registry sentinel shape received the wrong admission result: "+shape); + } + if(shape=="regular"&&File.ReadAllText(registry)!="preserve registry\n")throw new InvalidOperationException("Regular registry sentinel changed during admission"); + if(shape=="directory"&&!Directory.Exists(registry))throw new InvalidOperationException("Directory registry sentinel changed during admission"); + if(File.Exists(Path.Combine(sentinelHome,"owner-probe.json"))||File.Exists(Path.Combine(sentinelHome,"state",".lock")))throw new InvalidOperationException("Registry sentinel admission acquired ownership"); + } + Console.WriteLine("PASS: absent registry is admitted while file and directory sentinels are preserved and refused"); string residual=Path.Combine(directory,"residual"),residualState=Path.Combine(residual,"state"),status=Path.Combine(residualState,"orphan.status"); Directory.CreateDirectory(residualState);File.WriteAllText(status,"preserve\n"); bool refused=false;try{EmptyFleet(residual,true);}catch(InvalidOperationException){refused=true;} diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index 923012ffd91..f38f6e4b09a 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -301,6 +301,13 @@ while [ "$#" -gt 0 ]; do done if [ "$mode" = W ]; then [ "${FM_TEST_WIN_QUERY_EXIT:-0}" = 0 ] || exit "$FM_TEST_WIN_QUERY_EXIT" + if [ -n "${FM_TEST_WIN_FAIL_ONCE_COUNTER:-}" ]; then + count=$(cat "$FM_TEST_WIN_FAIL_ONCE_COUNTER" 2>/dev/null || true) + count=${count:-0} + count=$((count + 1)) + printf '%s\n' "$count" > "$FM_TEST_WIN_FAIL_ONCE_COUNTER" + [ "$count" -ne 1 ] || exit 7 + fi printf '%s\n' ' PID PPID PGID WINPID TTY UID STIME COMMAND' [ -n "${FM_TEST_WIN_TABLE:-}" ] && printf '%s\n' "$FM_TEST_WIN_TABLE" exit 0 @@ -347,7 +354,7 @@ test_windows_session_is_identified_from_its_published_pid() { } test_windows_query_outcomes_reach_every_owner_gate() { - local dir fakebin case_dir out rc + local dir fakebin case_dir counter out rc dir="$TMP_ROOT/win-query-outcomes" fakebin=$(cygwin_fakebin "$dir") @@ -403,11 +410,40 @@ test_windows_query_outcomes_reach_every_owner_gate() { set -e [ "$rc" -ne 0 ] || fail "ordinary acquisition replaced an owner after a failed Windows query" [ "$(cat "$case_dir/state/.lock")" = 'win:7204' ] || fail "failed Windows query changed the recorded owner" + + case_dir="$dir/fail-once" + counter="$case_dir/query-count" + mkdir -p "$case_dir/state" + printf '%s\n' 'win:7204' > "$case_dir/state/.lock" + out=$(PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_FAIL_ONCE_COUNTER="$counter" "$ROOT/bin/fm-lock.sh" status) + [ "$out" = 'lock: held by owner with unconfirmed health (pid win:7204)' ] \ + || fail "transient Windows query failure was reclassified by status: $out" + [ "$(cat "$counter")" = 1 ] || fail "status queried one owner decision more than once" + + rm -f "$counter" + set +e + PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_WIN_FAIL_ONCE_COUNTER="$counter" "$ROOT/bin/fm-lock.sh" native-admission-predicate >/dev/null 2>&1 + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "native admission accepted an owner after a transient Windows query failure" + [ "$(cat "$counter")" = 1 ] || fail "native admission queried one owner decision more than once" + + rm -f "$counter" + set +e + PATH="$fakebin:$PATH" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + FM_TEST_CYG_PPID=700 FM_TEST_WIN_FAIL_ONCE_COUNTER="$counter" "$ROOT/bin/fm-lock.sh" >/dev/null 2>&1 + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "ordinary acquisition replaced an owner after a transient Windows query failure" + [ "$(cat "$case_dir/state/.lock")" = 'win:7204' ] || fail "transient Windows query failure changed the recorded owner" + [ "$(cat "$counter")" = 1 ] || fail "ordinary acquisition queried one owner decision more than once" pass "session-lock: Windows query presence, absence, and failure remain distinct through status and acquisition" } test_harness_detection_uses_shared_native_selection() { - local dir bindir fakebin home out + local dir bindir fakebin home out shape dir="$TMP_ROOT/native-harness-selection" bindir="$dir/bin" home="$dir/home" @@ -440,12 +476,20 @@ SH [ "$out" = unknown ] || fail "native lock without authenticated proof fell through to a marker: $out" rm "$home/state/.lock" + for shape in regular directory broken-link; do + case "$shape" in + regular) printf '%s\n' '{}' > "$home/owner-probe.json" ;; + directory) mkdir "$home/owner-probe.json" ;; + broken-link) ln -s "$home/missing-owner-probe" "$home/owner-probe.json" ;; + esac + out=$(env -u CLAUDECODE -u GROK_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ + -u GEMINI_CLI -u ATLASSIAN_AGENT_TYPE -u ROVODEV_CLI -u FM_OMP_HARNESS \ + -u FM_PI_HARNESS -u FM_STATE_OVERRIDE PATH="$fakebin:$PATH" FM_HOME="$home" \ + PI_CODING_AGENT=true "$bindir/fm-harness.sh") + [ "$out" = unknown ] || fail "$shape native probe fell through to a marker: $out" + case "$shape" in directory) rmdir "$home/owner-probe.json" ;; *) rm "$home/owner-probe.json" ;; esac + done printf '%s\n' '{}' > "$home/owner-probe.json" - out=$(env -u CLAUDECODE -u GROK_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ - -u GEMINI_CLI -u ATLASSIAN_AGENT_TYPE -u ROVODEV_CLI -u FM_OMP_HARNESS \ - -u FM_PI_HARNESS -u FM_STATE_OVERRIDE PATH="$fakebin:$PATH" FM_HOME="$home" \ - PI_CODING_AGENT=true "$bindir/fm-harness.sh") - [ "$out" = unknown ] || fail "unusable native probe fell through to a marker: $out" out=$(env -u CLAUDECODE -u GROK_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ -u GEMINI_CLI -u ATLASSIAN_AGENT_TYPE -u ROVODEV_CLI -u FM_OMP_HARNESS \ -u FM_PI_HARNESS -u FM_STATE_OVERRIDE PATH="$fakebin:$PATH" FM_HOME="$home" \ @@ -836,6 +880,46 @@ test_native_state_contract() ( pass "native identity routing preserves unknown exclusion without certifying health" ) +test_native_owns_uses_shared_dispatch() ( + local dir fakebin state ambient args rc expected + dir="$TMP_ROOT/native-owns-dispatch" + fakebin="$dir/fakebin" + state="$dir/selected/state" + ambient="$dir/ambient/state" + mkdir -p "$fakebin" "$state" "$ambient" + printf '%s\n' '{}' > "$dir/selected/owner-probe.json" + cat > "$fakebin/uname" <<'SH' +#!/usr/bin/env bash +printf '%s\n' MINGW64_NT-fixture +SH + cat > "$fakebin/cygpath" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "${@: -1}" +SH + cat > "$dir/fm-native-owner.exe" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$@" > "$FM_TEST_NATIVE_ARGS" +exit "$FM_TEST_NATIVE_EXIT" +SH + chmod +x "$fakebin/uname" "$fakebin/cygpath" "$dir/fm-native-owner.exe" + # shellcheck source=bin/fm-session-lock-lib.sh + . "$LIB" + FM_NATIVE_OWNER_BIN="$dir/fm-native-owner.exe" + FM_HOME="$dir/ambient" + FM_STATE_OVERRIDE="$ambient" + args="$dir/arguments" + for expected in 0 2; do + rc=0 + PATH="$fakebin:$PATH" FM_TEST_NATIVE_ARGS="$args" FM_TEST_NATIVE_EXIT="$expected" \ + fm_session_lock_owned_by_self "$state" || rc=$? + [ "$rc" -eq "$expected" ] || fail "native owns changed verifier exit $expected to $rc" + [ "$(wc -l < "$args" | tr -d ' ')" -eq 3 ] && [ "$(sed -n '1p' "$args")" = owner ] \ + && [ "$(sed -n '2p' "$args")" = owns ] && [ "$(sed -n '3p' "$args")" = "$state" ] \ + || fail "native owns did not preserve the selected state override" + done + pass "native self-ownership delegates through the shared owner command" +) + test_native_status_and_acquisition_behavior() { local dir bindir fakebin identity out rc dir="$TMP_ROOT/native-status" @@ -912,6 +996,7 @@ test_numeric_ancestry_interfaces_reject_native_identity() { } test_native_state_contract +test_native_owns_uses_shared_dispatch test_native_status_and_acquisition_behavior test_numeric_ancestry_interfaces_reject_native_identity test_version_named_session_is_identified_on_both_platforms From b27678661cd43706b9c59f770f4d80c51c74bf01 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 23:03:37 +1200 Subject: [PATCH 49/61] no-mistakes(review): Preserve notification offers and atomically publish lifetime markers --- bin/native-owner/codex-tool-gate.mjs | 14 ++++++-- tests/fixtures/native-owner/Build.ps1 | 2 ++ tests/fixtures/native-owner/NativeDriver.cs | 17 ++++++++- .../native-owner/OperationLifetimeTests.cs | 5 ++- .../fixtures/native-owner/fake-app-server.mjs | 11 ++++-- .../fixtures/native-owner/tool-gate.test.mjs | 36 +++++++++++++++++-- 6 files changed, 74 insertions(+), 11 deletions(-) diff --git a/bin/native-owner/codex-tool-gate.mjs b/bin/native-owner/codex-tool-gate.mjs index cb362abd2e2..32958b5285f 100644 --- a/bin/native-owner/codex-tool-gate.mjs +++ b/bin/native-owner/codex-tool-gate.mjs @@ -17,6 +17,14 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { const seen = new Set(); const retiredTurns = new Set(); const deny = reason => ({ success: false, value: { denied: reason } }); + const recordOffer = (turn, value) => { + let turnOffers = offers.get(turn); + if (!turnOffers) { + turnOffers = new Set(); + offers.set(turn, turnOffers); + } + turnOffers.add(value); + }; const valid = params => !closed && isAlive() && params.threadId === primaryThread && typeof params.turnId === 'string' && params.turnId === activeTurn; @@ -54,7 +62,7 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { // Redelivery is read-only: cancellation or a lost response must not // strand pending work or start another native check before handling it. if (receipt && !acknowledged) { - offers.set(params.turnId, receipt); + recordOffer(params.turnId, receipt); return { success: true, value: { ...offered } }; } } else if (params.tool === 'fm_notification_ack') { @@ -89,7 +97,7 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { acknowledged = false; offered = Object.freeze({ message: note.message, receipt, challenge, checkpointExit: note.checkpointExit }); if (!valid(params)) return deny('wrong-thread-turn-or-replay'); - offers.set(params.turnId, receipt); + recordOffer(params.turnId, receipt); return { success: true, value: { ...offered } }; } const result = await operate('ack', { receipt, observed: args.observed }); @@ -107,7 +115,7 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { } }, wasOffered(thread, turn, expectedReceipt) { - return thread === primaryThread && offers.get(turn) === expectedReceipt; + return thread === primaryThread && offers.get(turn)?.has(expectedReceipt) === true; }, }); } diff --git a/tests/fixtures/native-owner/Build.ps1 b/tests/fixtures/native-owner/Build.ps1 index 266b8323153..be0cae852aa 100644 --- a/tests/fixtures/native-owner/Build.ps1 +++ b/tests/fixtures/native-owner/Build.ps1 @@ -15,6 +15,8 @@ $binary = Join-Path $copy 'bin/SessionProbe.exe' $sources = @((Join-Path $repo 'bin/native-owner/NativeOwner.cs'), (Join-Path $repo 'bin/native-owner/NativeHomeLease.cs'), (Join-Path $PSScriptRoot 'NativeDriver.cs'), (Join-Path $repo 'bin/native-owner/NativeReceiptJournal.cs'), (Join-Path $PSScriptRoot 'ReceiptTests.cs')) $sources += @((Join-Path $repo 'bin/native-owner/NativeOperations.cs'), (Join-Path $repo 'bin/native-owner/NativeAcknowledgementEvidence.cs'), (Join-Path $repo 'bin/native-owner/NativeOperationLifetime.cs'), (Join-Path $PSScriptRoot 'OperationLifetimeTests.cs')) Add-Type -Path $sources -OutputAssembly $binary -OutputType ConsoleApplication -ReferencedAssemblies System.dll,System.Core.dll,System.Web.Extensions.dll +& $binary operation-lifetime-contention +if ($LASTEXITCODE -ne 0) { throw 'Operation marker publication tests failed' } & $binary receipt-tests $receiptExit = $LASTEXITCODE if ($receiptExit -ne 0) { throw 'Durable receipt lifecycle tests failed' } diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 8c3c99de96c..7fcb6a3ec5e 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -439,6 +439,12 @@ public static int Main(string[] args) { try { int ownerResult; if(args.Length==1 && args[0]=="receipt-tests") { ReceiptTests.Run();return TestOperationLifetime(); } + if(args.Length==1 && args[0]=="operation-lifetime-tests") return TestOperationLifetime(); + if(args.Length==1 && args[0]=="operation-lifetime-contention") { + string previous=Environment.GetEnvironmentVariable("FM_PROBE_MARKER_CONTENTION"); + try {Environment.SetEnvironmentVariable("FM_PROBE_MARKER_CONTENTION","1");return TestOperationLifetime();} + finally {Environment.SetEnvironmentVariable("FM_PROBE_MARKER_CONTENTION",previous);} + } if(args.Length==1 && args[0]=="environment-tests") return EnvironmentTests(); if(TryOwnerCommand(args,out ownerResult)) return ownerResult; if(args.Length==1 && args[0]=="privilege-probe") return Client("ordinary-sandbox-command",true); @@ -446,7 +452,16 @@ public static int Main(string[] args) { if(args.Length==2 && args[0]=="sleep") { int ms=int.Parse(args[1]); if(ms<0 || ms>15000) throw new ArgumentException("Sleep must be bounded"); Thread.Sleep(ms); return 0; } if(args.Length==2 && args[0]=="operation-parent") { using(var descendant=Process.Start(new ProcessStartInfo(OwnExe,"sleep 10000") {UseShellExecute=false})) { - File.WriteAllText(args[1],descendant.Id.ToString());descendant.WaitForExit();return descendant.ExitCode; + string staging=args[1]+".publishing"; + try { + using(var stream=new FileStream(staging,FileMode.CreateNew,FileAccess.Write,FileShare.None)) + using(var writer=new StreamWriter(stream,new UTF8Encoding(false))) { + writer.Write(descendant.Id.ToString());writer.Flush(); + if(Environment.GetEnvironmentVariable("FM_PROBE_MARKER_CONTENTION")=="1") System.Threading.Thread.Sleep(1000); + } + File.Move(staging,args[1]); + } finally {if(File.Exists(staging)) File.Delete(staging);} + descendant.WaitForExit();return descendant.ExitCode; } } if(args.Length==3 && args[0]=="lease-check") return LeaseCheck(args[1],args[2]); diff --git a/tests/fixtures/native-owner/OperationLifetimeTests.cs b/tests/fixtures/native-owner/OperationLifetimeTests.cs index 55f5079ac36..3db6514424f 100644 --- a/tests/fixtures/native-owner/OperationLifetimeTests.cs +++ b/tests/fixtures/native-owner/OperationLifetimeTests.cs @@ -36,7 +36,10 @@ static int TestOperationLifetime() { Console.WriteLine("PASS: operation "+(close ? "last-handle close" : "bounded stop")+" stops its deferred worker and preserves an independent process"); } finally { if(descendant!=null) descendant.Dispose(); - if(child.process!=IntPtr.Zero) { if(WaitForSingleObject(child.process,0)==WAIT_TIMEOUT) TerminateProcess(child.process,125);CloseHandle(child.thread);CloseHandle(child.process); } + if(child.process!=IntPtr.Zero) { + if(WaitForSingleObject(child.process,0)==WAIT_TIMEOUT) {TerminateProcess(child.process,125);WaitForSingleObject(child.process,3000);} + CloseHandle(child.thread);CloseHandle(child.process); + } if(job!=IntPtr.Zero) CloseHandle(job); if(File.Exists(marker)) File.Delete(marker); } diff --git a/tests/fixtures/native-owner/fake-app-server.mjs b/tests/fixtures/native-owner/fake-app-server.mjs index 2f84a58d383..d1cd2aabbf9 100644 --- a/tests/fixtures/native-owner/fake-app-server.mjs +++ b/tests/fixtures/native-owner/fake-app-server.mjs @@ -22,13 +22,18 @@ export function createFakeAppServer(scenario){ const frame=JSON.parse(line); if(frame.method==='initialized')return; if(frame.id===900){ - if(scenario==='success'){ + if(scenario==='success'||scenario==='success-then-next-check'){ const result=JSON.parse(frame.result.contentItems[0].text); toolStage='ack';call(901,'fm_notification_ack',{receipt:result.receipt,observed:result.challenge}); }else complete(); return; } - if(frame.id===901){complete();return;} + if(frame.id===901){ + if(scenario==='success-then-next-check'){toolStage='next-check';call(902,'fm_notification_check',{});} + else complete(); + return; + } + if(frame.id===902){complete();return;} if(frame.method==='initialize'){send({id:frame.id,result:{}});return;} if(frame.method==='config/read'){send({id:frame.id,result:{config:{features:{apps:false,plugins:false},mcp_servers:{}}}});return;} if(frame.method==='thread/start'){ @@ -43,7 +48,7 @@ export function createFakeAppServer(scenario){ if(scenario==='prose')complete(); else if(scenario==='denied'){toolStage='check';call(900,'fm_notification_check',{}, {namespace:'other'});} else if(scenario==='malformed'){toolStage='check';call(900,'fm_notification_check',null);} - else if(scenario==='success'){toolStage='check';call(900,'fm_notification_check',{});} + else if(scenario==='success'||scenario==='success-then-next-check'){toolStage='check';call(900,'fm_notification_check',{});} else fail(Error('unknown scenario '+scenario)); }); return; diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index 47bffffaaf4..e5792f52c26 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -5,7 +5,7 @@ import os from 'node:os'; import path from 'node:path'; import {PassThrough} from 'node:stream'; import {fileURLToPath} from 'node:url'; -import {createNotificationGate} from '../../../bin/native-owner/codex-tool-gate.mjs'; +import {confirmAutomaticNotificationOffer,createNotificationGate} from '../../../bin/native-owner/codex-tool-gate.mjs'; import {runCodexHost} from '../../../bin/native-owner/codex-host-runtime.mjs'; import {createFakeAppServer} from './fake-app-server.mjs'; const message={receipt:'receipt',challenge:'observed',message:'Controlled message',checkpointExit:124}; @@ -83,6 +83,21 @@ test('next cycle works without reauthorizing an earlier receipt',async()=>{ assert.equal((await gate.handle(ack({turnId:'turn2',callId:'second-ack',arguments:{receipt:'receipt2',observed:'observed'}}))).success,true); assert.equal(calls.length,4); }); +test('one turn retains each offered receipt without authorizing another turn',async()=>{ + let cycle=0; + const {gate,calls}=fixture(action=>action==='check'?{operationState:'delivered',notification:{...message,receipt:`receipt-${++cycle}`}}:{operationState:'acknowledged'}); + assert.equal((await gate.handle(check())).value.receipt,'receipt-1'); + assert.equal((await gate.handle(ack({callId:'ack-first',arguments:{receipt:'receipt-1',observed:'observed'}}))).success,true); + assert.equal((await gate.handle(check({callId:'check-second'}))).value.receipt,'receipt-2'); + gate.endTurn('primary','turn'); + assert.equal(confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt-1'),true); + assert.equal(confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt-2'),true); + assert.throws(()=>confirmAutomaticNotificationOffer(gate,'primary',{id:'turn',status:'completed'},'receipt-unoffered')); + assert.throws(()=>confirmAutomaticNotificationOffer(gate,'primary',{id:'wrong-turn',status:'completed'},'receipt-1')); + gate.beginTurn('primary','next'); + assert.equal((await gate.handle(check({turnId:'next',callId:'redeliver-second'}))).value.receipt,'receipt-2'); + assert.equal(calls.length,3); +}); test('closed gate cannot be rebound to another thread',()=>{ const {gate}=fixture();assert.throws(()=>gate.beginTurn('foreign','turn2'));gate.close();assert.throws(()=>gate.beginTurn('primary','turn2')); }); @@ -129,7 +144,8 @@ async function hostScenario(scenario,cycles=1){ if(action==='status')return {operationState:'ready'}; if(action==='result'){ if(ackPending){ackPending=false;cycle++;return {operationState:'acknowledged'};} - return {operationState:'delivered',notification:notification(Math.min(cycle,cycles-1))}; + const available=scenario==='success-then-next-check'?2:cycles; + return {operationState:'delivered',notification:notification(Math.min(cycle,available-1))}; } if(action==='ack'){ assert.deepEqual(extra,{receipt:notification(cycle).receipt,observed:'observed'}); @@ -144,12 +160,13 @@ async function hostScenario(scenario,cycles=1){ env:{...process.env,FM_PROBE_HOME:runtime,FM_PROBE_CODE_ROOT:repo,FM_HOME:home,FM_PROBE_SESSION:'session',FM_PROBE_NONCE:'nonce'}, input,output,error,native,mcpServerNames:[],spawnAppServer:()=>createFakeAppServer(scenario),installSignalHandlers:false, afterAutomaticTurn:({evidence})=>{ + if(scenario==='success-then-next-check')return false; if(evidence.automatic.length===cycles)setTimeout(()=>input.write('/quit\n'),20); return true; }, }); }catch(errorValue){failure=errorValue;} - return {result,failure,acknowledgements,shutdowns,afterShutdown,host:JSON.parse(fs.readFileSync(path.join(runtime,'host.json'),'utf8'))}; + return {result,failure,acknowledgements,shutdowns,afterShutdown,pendingReceipt:notification(cycle).receipt,host:JSON.parse(fs.readFileSync(path.join(runtime,'host.json'),'utf8'))}; } for(const scenario of ['prose','denied','malformed'])test(`actual host loop preserves a ${scenario} completed turn`,async()=>{ @@ -176,6 +193,19 @@ test('actual host loop suppresses only the exact successfully offered receipt',a ]); }); +test('actual host loop preserves the initiating offer when the turn reads the next receipt',async()=>{ + const result=await hostScenario('success-then-next-check'); + assert.equal(result.failure,undefined); + assert.equal(result.acknowledgements,1); + assert.equal(result.pendingReceipt,'receipt-2'); + assert.equal(result.shutdowns,1); + assert.equal(result.afterShutdown,0); + assert.deepEqual(result.result.automatic.map(item=>[item.receipt,item.outcome]),[['receipt','offered']]); + assert.deepEqual(result.host.tools.map(tool=>[tool.tool,tool.success]),[ + ['fm_notification_check',true],['fm_notification_ack',true],['fm_notification_check',true], + ]); +}); + test('actual host loop gives consecutive automatic turns distinct protocol identities',async()=>{ const result=await hostScenario('success',2); assert.equal(result.failure,undefined); From 544fe6869fe1aec7f31d1324a46de0bf2beddc75 Mon Sep 17 00:00:00 2001 From: Cristian Date: Fri, 18 Sep 2026 23:38:12 +1200 Subject: [PATCH 50/61] no-mistakes(review): Bind acknowledgements to current-turn notification offers --- bin/native-owner/codex-tool-gate.mjs | 9 ++++--- .../fixtures/native-owner/fake-app-server.mjs | 20 ++++++++++++-- .../fixtures/native-owner/tool-gate.test.mjs | 26 +++++++++++++++++++ 3 files changed, 49 insertions(+), 6 deletions(-) diff --git a/bin/native-owner/codex-tool-gate.mjs b/bin/native-owner/codex-tool-gate.mjs index 32958b5285f..68962166e2b 100644 --- a/bin/native-owner/codex-tool-gate.mjs +++ b/bin/native-owner/codex-tool-gate.mjs @@ -25,6 +25,8 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { } turnOffers.add(value); }; + const wasOffered = (thread, turn, expectedReceipt) => + thread === primaryThread && offers.get(turn)?.has(expectedReceipt) === true; const valid = params => !closed && isAlive() && params.threadId === primaryThread && typeof params.turnId === 'string' && params.turnId === activeTurn; @@ -67,7 +69,8 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { } } else if (params.tool === 'fm_notification_ack') { if (Object.keys(args).sort().join(',') !== 'observed,receipt' || !receipt || - args.receipt !== receipt || args.observed !== challenge) { + args.receipt !== receipt || args.observed !== challenge || + !wasOffered(params.threadId, params.turnId, receipt)) { return deny('wrong-receipt-or-unhandled-notification'); } if (acknowledged) return deny('receipt-already-consumed'); @@ -114,9 +117,7 @@ export function createNotificationGate({ primaryThread, operate, isAlive }) { busy = false; } }, - wasOffered(thread, turn, expectedReceipt) { - return thread === primaryThread && offers.get(turn)?.has(expectedReceipt) === true; - }, + wasOffered, }); } diff --git a/tests/fixtures/native-owner/fake-app-server.mjs b/tests/fixtures/native-owner/fake-app-server.mjs index d1cd2aabbf9..985213a2c98 100644 --- a/tests/fixtures/native-owner/fake-app-server.mjs +++ b/tests/fixtures/native-owner/fake-app-server.mjs @@ -5,7 +5,7 @@ import {createInterface} from 'node:readline'; export function createFakeAppServer(scenario){ const server=new EventEmitter(),input=new PassThrough(); server.stdin=input;server.stdout=new PassThrough();server.stderr=new PassThrough(); - let toolStage='',exited=false; + let toolStage='',exited=false,pendingAcknowledgement=null; const finish=()=>{ if(exited)return; exited=true;server.stdout.end();server.stderr.end(); @@ -14,13 +14,25 @@ export function createFakeAppServer(scenario){ const fail=error=>{server.stderr.write(error.message+'\n');server.emit('error',error);finish();}; const send=value=>server.stdout.write(JSON.stringify(value)+'\n'); const thread='primary';let turn='',turnNumber=0; - const complete=()=>send({method:'turn/completed',params:{threadId:thread,turn:{id:turn,status:'completed'}}}); + const complete=(status='completed')=>send({method:'turn/completed',params:{threadId:thread,turn:{id:turn,status}}}); const call=(id,tool,args,overrides={})=>send({id,method:'item/tool/call',params:{threadId:thread,turnId:turn,callId:'call-'+id,namespace:null,tool,arguments:args,...overrides}}); const lines=createInterface({input}); lines.on('line',line=>{ try { const frame=JSON.parse(line); if(frame.method==='initialized')return; + if(scenario==='interrupted-direct-ack'){ + if(frame.id===900){ + const result=JSON.parse(frame.result.contentItems[0].text); + pendingAcknowledgement={receipt:result.receipt,observed:result.challenge};complete('interrupted');return; + } + if(frame.id===901){toolStage='reread';call(902,'fm_notification_check',{});return;} + if(frame.id===902){ + const result=JSON.parse(frame.result.contentItems[0].text); + toolStage='ack';call(903,'fm_notification_ack',{receipt:result.receipt,observed:result.challenge});return; + } + if(frame.id===903){complete();return;} + } if(frame.id===900){ if(scenario==='success'||scenario==='success-then-next-check'){ const result=JSON.parse(frame.result.contentItems[0].text); @@ -49,6 +61,10 @@ export function createFakeAppServer(scenario){ else if(scenario==='denied'){toolStage='check';call(900,'fm_notification_check',{}, {namespace:'other'});} else if(scenario==='malformed'){toolStage='check';call(900,'fm_notification_check',null);} else if(scenario==='success'||scenario==='success-then-next-check'){toolStage='check';call(900,'fm_notification_check',{});} + else if(scenario==='interrupted-direct-ack'){ + if(turnNumber===1){toolStage='check';call(900,'fm_notification_check',{});} + else {toolStage='direct-ack';call(901,'fm_notification_ack',pendingAcknowledgement);} + } else fail(Error('unknown scenario '+scenario)); }); return; diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index e5792f52c26..41044d11343 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -124,6 +124,16 @@ test('notification arriving after cancellation remains available next turn',asyn const recovered=await gate.handle(check({turnId:'next',callId:'redelivery'})); assert.equal(recovered.success,true);assert.equal(recovered.value.receipt,'receipt');assert.equal(calls.length,1); }); +test('a later turn must reread a pending receipt before acknowledgement',async()=>{ + const {gate,calls}=fixture();assert.equal((await gate.handle(check())).success,true); + gate.endTurn('primary','turn');gate.beginTurn('primary','next'); + const denied=await gate.handle(ack({turnId:'next',callId:'direct-ack'})); + assert.deepEqual(denied,{success:false,value:{denied:'wrong-receipt-or-unhandled-notification'}});assert.equal(calls.length,1); + assert.equal((await gate.handle(check({turnId:'next',callId:'redelivery'}))).success,true);assert.equal(calls.length,1); + assert.equal((await gate.handle(ack({turnId:'next',callId:'recovered-ack'}))).success,true); + gate.endTurn('primary','next');assert.equal(confirmAutomaticNotificationOffer(gate,'primary',{id:'next',status:'completed'},'receipt'),true); + assert.deepEqual(calls,[['check'],['ack',{receipt:'receipt',observed:'observed'}]]); +}); test('operation failure is not reported as success and cannot be blindly retried',async()=>{ const {gate,calls}=fixture(()=>{throw Error('partial operation requires reconciliation');}); assert.equal((await gate.handle(check())).success,false);assert.equal((await gate.handle(check({callId:'retry'}))).success,false);assert.equal(calls.length,1); @@ -161,6 +171,7 @@ async function hostScenario(scenario,cycles=1){ input,output,error,native,mcpServerNames:[],spawnAppServer:()=>createFakeAppServer(scenario),installSignalHandlers:false, afterAutomaticTurn:({evidence})=>{ if(scenario==='success-then-next-check')return false; + if(scenario==='interrupted-direct-ack')return evidence.automatic.length<2; if(evidence.automatic.length===cycles)setTimeout(()=>input.write('/quit\n'),20); return true; }, @@ -206,6 +217,21 @@ test('actual host loop preserves the initiating offer when the turn reads the ne ]); }); +test('actual host loop requires a reread after an interrupted offer',async()=>{ + const result=await hostScenario('interrupted-direct-ack'); + assert.equal(result.failure,undefined); + assert.equal(result.acknowledgements,1); + assert.equal(result.shutdowns,1); + assert.equal(result.afterShutdown,0); + assert.deepEqual(result.result.automatic.map(item=>[item.status,item.outcome]),[ + ['interrupted','interrupted'],['completed','offered'], + ]); + assert.deepEqual(result.host.tools.map(tool=>[tool.tool,tool.success]),[ + ['fm_notification_check',true],['fm_notification_ack',false], + ['fm_notification_check',true],['fm_notification_ack',true], + ]); +}); + test('actual host loop gives consecutive automatic turns distinct protocol identities',async()=>{ const result=await hostScenario('success',2); assert.equal(result.failure,undefined); From 657f658a633d514bb7bc673862792e015a2edc06 Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 09:10:59 +1200 Subject: [PATCH 51/61] no-mistakes(review): Reject unsafe projects paths and document terminal outcomes --- bin/fm-terminal-outcome-lib.sh | 6 +++++ bin/native-owner/NativeOperations.cs | 10 ++++++++- tests/fixtures/native-owner/NativeDriver.cs | 25 +++++++++++++++++++++ 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/bin/fm-terminal-outcome-lib.sh b/bin/fm-terminal-outcome-lib.sh index e6d78a41ce0..53c25761995 100644 --- a/bin/fm-terminal-outcome-lib.sh +++ b/bin/fm-terminal-outcome-lib.sh @@ -1,4 +1,10 @@ # shellcheck shell=bash +# Shared terminal-outcome admission predicate. +# Usage: . bin/fm-terminal-outcome-lib.sh +# +# fm_terminal_outcome_pending_absent returns 0 when no unresolved +# terminal outcome is present and 1 with FM_TERMINAL_OUTCOME_ERROR set when the +# terminal-outcome state is unresolved or unrecognized. # shellcheck disable=SC2034 # Public result consumed by sourcing callers. FM_TERMINAL_OUTCOME_ERROR= diff --git a/bin/native-owner/NativeOperations.cs b/bin/native-owner/NativeOperations.cs index b95ddb1deb0..76ae452a9a2 100644 --- a/bin/native-owner/NativeOperations.cs +++ b/bin/native-owner/NativeOperations.cs @@ -51,9 +51,17 @@ static bool PathPresent(string name) { catch(FileNotFoundException) { return false; } catch(DirectoryNotFoundException) { return false; } } + static bool ProjectsOccupied(string name) { + FileAttributes attributes; + try { attributes=File.GetAttributes(name); } + catch(FileNotFoundException) { return false; } + catch(DirectoryNotFoundException) { return false; } + if((attributes&FileAttributes.ReparsePoint)!=0 || (attributes&FileAttributes.Directory)==0) return true; + return Directory.GetFileSystemEntries(name).Length!=0; + } internal static void EmptyFleet(string home,bool launch,string[] provenDeadGenerations=null) { string state=Path.Combine(home,"state"),projects=Path.Combine(home,"projects"); - if((Directory.Exists(projects)&&Directory.GetFileSystemEntries(projects).Length!=0) || PathPresent(Path.Combine(home,"data","secondmates.md")) || PathPresent(Path.Combine(home,"data","projects.md")) || PathPresent(Path.Combine(home,".env")) || PathPresent(Path.Combine(home,"config","x-mode.env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); + if(ProjectsOccupied(projects) || PathPresent(Path.Combine(home,"data","secondmates.md")) || PathPresent(Path.Combine(home,"data","projects.md")) || PathPresent(Path.Combine(home,".env")) || PathPresent(Path.Combine(home,"config","x-mode.env")) || (Directory.Exists(Path.Combine(state,"procevent"))&&Directory.GetFileSystemEntries(Path.Combine(state,"procevent")).Length!=0)) throw new InvalidOperationException("This experimental launcher requires an empty fleet; existing fleet records were preserved"); OwnerAdmission(home,launch,provenDeadGenerations); } static IntPtr FileHandle(string path, uint access, uint creation) { diff --git a/tests/fixtures/native-owner/NativeDriver.cs b/tests/fixtures/native-owner/NativeDriver.cs index 7fcb6a3ec5e..fa7716f2c9a 100644 --- a/tests/fixtures/native-owner/NativeDriver.cs +++ b/tests/fixtures/native-owner/NativeDriver.cs @@ -14,6 +14,7 @@ using System.Web.Script.Serialization; public static partial class NativeOwner { + [DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] [return: MarshalAs(UnmanagedType.I1)] static extern bool CreateSymbolicLink(string link,string target,uint flags); static string LogonSid() { using(var identity=WindowsIdentity.GetCurrent()) { foreach(var row in TokenGroups(identity.Token,2)) @@ -127,6 +128,30 @@ static int EnvironmentTests() { if(File.Exists(Path.Combine(sentinelHome,"owner-probe.json"))||File.Exists(Path.Combine(sentinelHome,"state",".lock")))throw new InvalidOperationException("Registry sentinel admission acquired ownership"); } Console.WriteLine("PASS: absent registry is admitted while file and directory sentinels are preserved and refused"); + foreach(string shape in new [] {"absent","empty-directory","regular-file","nonempty-directory","directory-link","broken-link"}) { + string projectsHome=Path.Combine(directory,"projects-path-"+shape),projects=Path.Combine(projectsHome,"projects"),target=Path.Combine(directory,"projects-target-"+shape),body="preserve occupied projects path\n"; + Directory.CreateDirectory(projectsHome); + if(shape=="empty-directory") Directory.CreateDirectory(projects); + if(shape=="regular-file") File.WriteAllText(projects,body); + if(shape=="nonempty-directory") { Directory.CreateDirectory(projects);File.WriteAllText(Path.Combine(projects,"unlanded.txt"),body); } + if(shape=="directory-link") { Directory.CreateDirectory(target);if(!CreateSymbolicLink(projects,target,3))throw new Win32Exception(Marshal.GetLastWin32Error(),"Directory symlink fixture failed"); } + if(shape=="broken-link"&&!CreateSymbolicLink(projects,target,3))throw new Win32Exception(Marshal.GetLastWin32Error(),"Broken directory symlink fixture failed"); + bool expected=shape=="absent"||shape=="empty-directory"; + foreach(bool launch in new [] {true,false}) { + bool projectsRefused=false;try{EmptyFleet(projectsHome,launch);}catch(InvalidOperationException){projectsRefused=true;} + if(projectsRefused==expected)throw new InvalidOperationException("Projects path shape received the wrong admission result: "+shape); + } + if(shape=="absent"&&(File.Exists(projects)||Directory.Exists(projects)))throw new InvalidOperationException("Absent projects path changed during admission"); + if(shape=="empty-directory"&&(!Directory.Exists(projects)||Directory.GetFileSystemEntries(projects).Length!=0))throw new InvalidOperationException("Empty projects directory changed during admission"); + if(shape=="regular-file"&&File.ReadAllText(projects)!=body)throw new InvalidOperationException("Regular projects path changed during admission"); + if(shape=="nonempty-directory"&&File.ReadAllText(Path.Combine(projects,"unlanded.txt"))!=body)throw new InvalidOperationException("Nonempty projects directory changed during admission"); + if((shape=="directory-link"||shape=="broken-link")&&(File.GetAttributes(projects)&FileAttributes.ReparsePoint)==0)throw new InvalidOperationException("Projects directory link changed during admission"); + if(shape=="directory-link"&&(!Directory.Exists(target)||Directory.GetFileSystemEntries(target).Length!=0))throw new InvalidOperationException("Projects directory link target changed during admission"); + if(shape=="broken-link"&&Directory.Exists(target))throw new InvalidOperationException("Broken projects directory link target changed during admission"); + if(shape=="directory-link"||shape=="broken-link") { Directory.CreateDirectory(target);File.WriteAllText(Path.Combine(projects,"target-check"),body);if(File.ReadAllText(Path.Combine(target,"target-check"))!=body)throw new InvalidOperationException("Projects directory link target changed during admission"); } + if(File.Exists(Path.Combine(projectsHome,"owner-probe.json"))||File.Exists(Path.Combine(projectsHome,"state",".lock")))throw new InvalidOperationException("Projects path admission acquired ownership"); + } + Console.WriteLine("PASS: only absent and ordinary empty projects paths are admitted unchanged"); string residual=Path.Combine(directory,"residual"),residualState=Path.Combine(residual,"state"),status=Path.Combine(residualState,"orphan.status"); Directory.CreateDirectory(residualState);File.WriteAllText(status,"preserve\n"); bool refused=false;try{EmptyFleet(residual,true);}catch(InvalidOperationException){refused=true;} From 21f36001a562238f450d702f3f7920f269a86e33 Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 09:42:52 +1200 Subject: [PATCH 52/61] no-mistakes(document): Clarify native launcher admission documentation --- bin/native-owner/NativeOwner.cs | 5 +++-- docs/native-windows-codex.md | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/bin/native-owner/NativeOwner.cs b/bin/native-owner/NativeOwner.cs index ece68aaff3e..305114c2a0c 100644 --- a/bin/native-owner/NativeOwner.cs +++ b/bin/native-owner/NativeOwner.cs @@ -1,5 +1,6 @@ -// Experimental native ownership core; no installed launcher is provided. -// The fixture driver registers scopes; association alone never grants ownership. +// Experimental ownership core for the explicit opt-in launcher; no runtime +// backend is installed. The controller registers scopes; association alone +// never grants ownership. using System; using System.Collections; using System.Collections.Generic; diff --git a/docs/native-windows-codex.md b/docs/native-windows-codex.md index 8bc2529befd..fc16e16a45d 100644 --- a/docs/native-windows-codex.md +++ b/docs/native-windows-codex.md @@ -60,7 +60,8 @@ FM_NATIVE_TEST_JQ_IMAGE= FM_LIVE_NATIVE_LAUNCHER=1 FM_LIVE ## Safety boundary and limits Only empty-fleet homes beneath the current user's Windows temporary directory are accepted. -Existing fleet metadata, projects, registrations, Relay configuration, process-event sources, non-temporary homes, and existing reparse-point ancestors are refused. +Existing fleet metadata, registrations, Relay configuration, process-event sources, non-temporary homes, and existing reparse-point ancestors are refused. +The `projects` path may be absent or an ordinary empty directory; populated, non-directory, and reparse-point forms are refused unchanged. The app-server thread is ephemeral, read-only, network-disabled, and approval-never; Apps, plugins, and configured MCP servers are disabled for this host and their effective catalogs are checked before readiness. Only controller-selected startup, notification check, and acknowledgement scripts receive registered native operation authority. The two notification tools are the model-facing path to those registered host operations, not the complete Codex tool catalog; ordinary tools cannot acquire native authority from the inherited endpoint, claims, callback identity, or session job. From b106d69f72e4dd3bf7fbc80fbfbe64562c424603 Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 11:40:46 +1200 Subject: [PATCH 53/61] no-mistakes(lint): Resolve source-aware ShellCheck diagnostics --- bin/backends/herdr.sh | 2 +- bin/fm-startup-network.sh | 2 +- bin/fm-wake-lib.sh | 15 ++++++++------- tests/fm-session-lock-ancestry.test.sh | 6 +++--- tests/fm-startup-network.test.sh | 2 ++ 5 files changed, 15 insertions(+), 12 deletions(-) diff --git a/bin/backends/herdr.sh b/bin/backends/herdr.sh index 88a8cb61492..39506993b44 100644 --- a/bin/backends/herdr.sh +++ b/bin/backends/herdr.sh @@ -3083,7 +3083,7 @@ fm_backend_herdr_composer_state() { # -> empty|pending|pending-unprove verdict=$(fm_composer_classify_screen "$caps" "$cap") if [ "$verdict" = need-identity ]; then if ! identity=$(fm_backend_herdr_composer_identity "$target" 2>/dev/null) || [ -z "$identity" ]; then - identity=probe-absent + identity='probe-absent' fi verdict=$(fm_composer_classify_screen "$caps" "$cap" '' "$identity") [ "$verdict" != need-identity ] || verdict=unknown diff --git a/bin/fm-startup-network.sh b/bin/fm-startup-network.sh index a3acd1bf141..7011b4cf21f 100755 --- a/bin/fm-startup-network.sh +++ b/bin/fm-startup-network.sh @@ -344,7 +344,7 @@ native_admission_predicate() { [ "$key" = startup-network ] || continue [ "$kind" = check ] || return 1 recognized=false - for state in done failed timeout; do + for state in 'done' failed timeout; do expected=$(fm_wake_startup_network_payload "$state") || return 1 [ "$payload" != "$expected" ] || recognized=true done diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 0dfd544f847..4be76e153fa 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -2239,6 +2239,7 @@ fm_wake_native_empty_fleet_preflight() { # [] else status=$? if [ "$status" -ne 1 ]; then + # shellcheck disable=SC2034 # Public result consumed by bin/native-owner/admit.sh. FM_WAKE_NATIVE_ADMISSION_ERROR="wake completion history is unrecognized at $history" return 1 fi @@ -2358,6 +2359,7 @@ fm_wake_ack_evidence_clear() { } fm_wake_ack_evidence_legacy_token() { + # shellcheck disable=SC2016 # Single quotes are deliberate: ${...} belongs to the Node snippet. node -e ' let input=""; process.stdin.setEncoding("utf8"); @@ -2460,12 +2462,12 @@ fm_wake_ack_evidence_capture() { while IFS= read -r id; do [ -n "$id" ] || continue note="$STATE/inbox/$id.note" - [ ! -e "$STATE/inbox/handled/$id.note" ] && [ ! -L "$STATE/inbox/handled/$id.note" ] \ - && fm_wake_ack_note_safe "$note" && digest=$(fm_wake_ack_hash "$note") || { - fm_lock_release "$FM_WAKE_QUEUE_LOCK" - fm_wake_ack_evidence_clear - return 1 - } + if [ -e "$STATE/inbox/handled/$id.note" ] || [ -L "$STATE/inbox/handled/$id.note" ] \ + || ! fm_wake_ack_note_safe "$note" || ! digest=$(fm_wake_ack_hash "$note"); then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_wake_ack_evidence_clear + return 1 + fi printf '%s\t%s\n' "$id" "$digest" >> "$FM_WAKE_ACK_EVIDENCE_NOTES.hashes" || { fm_lock_release "$FM_WAKE_QUEUE_LOCK" fm_wake_ack_evidence_clear @@ -2584,7 +2586,6 @@ fm_wake_ack_evidence_load() { # return 1 fi rm -f -- "$notes_derived" "$expected_notes" - FM_WAKE_ACK_EVIDENCE_TOKEN=$token } fm_wake_ack_evidence_native_recovery() { # diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index f38f6e4b09a..7c22ce9043d 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -849,9 +849,9 @@ test_e2e_daemon_parented_version_named_session_keeps_its_lock() { } test_native_state_contract() ( - # shellcheck source=bin/fm-session-lock-lib.sh + # shellcheck source=/dev/null . "$LIB" - local answer rc identity=native:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + local answer rc identity=native:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa FM_HOME FM_STATE_OVERRIDE fm_native_owner_call() { return "$answer"; } fm_session_pid_valid "$identity" || fail "valid native identity rejected" if fm_session_pid_valid native:123; then fail "malformed native identity accepted"; fi @@ -902,7 +902,7 @@ printf '%s\n' "$@" > "$FM_TEST_NATIVE_ARGS" exit "$FM_TEST_NATIVE_EXIT" SH chmod +x "$fakebin/uname" "$fakebin/cygpath" "$dir/fm-native-owner.exe" - # shellcheck source=bin/fm-session-lock-lib.sh + # shellcheck source=/dev/null . "$LIB" FM_NATIVE_OWNER_BIN="$dir/fm-native-owner.exe" FM_HOME="$dir/ambient" diff --git a/tests/fm-startup-network.test.sh b/tests/fm-startup-network.test.sh index 8633f76fbf3..4f830cb02b1 100755 --- a/tests/fm-startup-network.test.sh +++ b/tests/fm-startup-network.test.sh @@ -342,6 +342,7 @@ EOF assert_grep 'check startup-network' "$home/state/.wake-queue" \ "an actionable result did not reach the wake queue" ( + # shellcheck source=/dev/null FM_ROOT_OVERRIDE="$root" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" . "$root/bin/fm-wake-lib.sh" fm_wake_native_empty_fleet_preflight "$home/state" ) || fail "the native admission owner refused an actual startup completion wake" @@ -350,6 +351,7 @@ EOF sed 's/^state=done$/state=failed/' "$home/state/.startup-network.status" > "$status_tmp" mv "$status_tmp" "$home/state/.startup-network.status" ( + # shellcheck source=/dev/null FM_ROOT_OVERRIDE="$root" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" . "$root/bin/fm-wake-lib.sh" fm_wake_native_empty_fleet_preflight "$home/state" ) || fail "native admission tied a queued startup completion to mutable latest-run status" From 6f74dd7f23f180747f2ce04d35afd4eac8f8e1b6 Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 11:51:29 +1200 Subject: [PATCH 54/61] no-mistakes(lint): Preserve successful evidence loads after cleanup failure --- bin/fm-wake-lib.sh | 1 + tests/fixtures/native-owner/ReceiptTests.cs | 5 +++++ tests/fixtures/native-owner/zero-recovery.sh | 17 ++++++++++++++++- 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 4be76e153fa..62cc7d4a607 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -2586,6 +2586,7 @@ fm_wake_ack_evidence_load() { # return 1 fi rm -f -- "$notes_derived" "$expected_notes" + return 0 } fm_wake_ack_evidence_native_recovery() { # diff --git a/tests/fixtures/native-owner/ReceiptTests.cs b/tests/fixtures/native-owner/ReceiptTests.cs index 07f5ae68c8e..9fba5d90913 100644 --- a/tests/fixtures/native-owner/ReceiptTests.cs +++ b/tests/fixtures/native-owner/ReceiptTests.cs @@ -96,6 +96,11 @@ static Dictionary ZeroPayload(NativeHomeLease lease) { } public static int Run() { OwnerProbeLinkCase(); + Case("validated acknowledgement evidence survives scratch cleanup failure",lease=>{ + Targets(lease); + var payload=OwnerPayload(lease,"cleanup-failure"); + ZeroRecovery(lease,"load-cleanup-failure",(string)payload["ownerEvidence"]); + }); Case("one writer and unobserved acknowledgement refusal",lease=>{ Targets(lease); using(var journal=new NativeReceiptJournal(lease,A)) { diff --git a/tests/fixtures/native-owner/zero-recovery.sh b/tests/fixtures/native-owner/zero-recovery.sh index 0317fe92a72..e54db1c9eeb 100644 --- a/tests/fixtures/native-owner/zero-recovery.sh +++ b/tests/fixtures/native-owner/zero-recovery.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Usage: zero-recovery.sh present|acknowledge|append [generation] +# Usage: zero-recovery.sh present|acknowledge|append|load-cleanup-failure [generation-or-token] set -eu ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) export FM_HOME @@ -31,5 +31,20 @@ case "${1:-}" in . bin/fm-wake-lib.sh fm_wake_append check later-notification 'later notification remains pending' ;; + load-cleanup-failure) + token=${3:?} + load_rc=0 + . bin/fm-wake-lib.sh + rm() { + case "$*" in + *fm-wake-ack-derived.*fm-wake-ack-expected.*) return 1 ;; + *) command rm "$@" ;; + esac + } + fm_wake_ack_evidence_load "$token" || load_rc=$? + [ "$load_rc" = 0 ] + [ "$FM_WAKE_ACK_EVIDENCE_CUTOFF" = 1 ] + fm_wake_ack_evidence_clear + ;; *) exit 2 ;; esac From 2181d676108a54134cd20ce2e50364b51c706e2f Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 12:05:55 +1200 Subject: [PATCH 55/61] no-mistakes(lint): Annotate indirect ShellCheck consumers --- bin/fm-wake-lib.sh | 1 + tests/fm-session-lock-ancestry.test.sh | 2 ++ 2 files changed, 3 insertions(+) diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 62cc7d4a607..c78925f1756 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -2499,6 +2499,7 @@ fm_wake_ack_evidence_capture() { } > "$payload" || { rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; } encoded=$(base64 < "$payload" | tr -d '\r\n') || { rm -f -- "$payload"; fm_wake_ack_evidence_clear; return 1; } rm -f -- "$payload" + # shellcheck disable=SC2034 # Public capture output consumed by native-owner/ack-evidence.sh. FM_WAKE_ACK_EVIDENCE_TOKEN="v1.$encoded" } diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index 7c22ce9043d..852b06f2d48 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -852,6 +852,7 @@ test_native_state_contract() ( # shellcheck source=/dev/null . "$LIB" local answer rc identity=native:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa FM_HOME FM_STATE_OVERRIDE + # shellcheck disable=SC2329 # Mock invoked indirectly by fm_native_owner_state. fm_native_owner_call() { return "$answer"; } fm_session_pid_valid "$identity" || fail "valid native identity rejected" if fm_session_pid_valid native:123; then fail "malformed native identity accepted"; fi @@ -904,6 +905,7 @@ SH chmod +x "$fakebin/uname" "$fakebin/cygpath" "$dir/fm-native-owner.exe" # shellcheck source=/dev/null . "$LIB" + # shellcheck disable=SC2034 # Test input consumed by fm_native_owner_call. FM_NATIVE_OWNER_BIN="$dir/fm-native-owner.exe" FM_HOME="$dir/ambient" FM_STATE_OVERRIDE="$ambient" From 919b7565bec43a40b3c85b41be0e7e06275a80ac Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 12:39:22 +1200 Subject: [PATCH 56/61] no-mistakes(lint): Persist startup fixture overrides before sourcing --- tests/fm-startup-network.test.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/fm-startup-network.test.sh b/tests/fm-startup-network.test.sh index 4f830cb02b1..8c14c043ac8 100755 --- a/tests/fm-startup-network.test.sh +++ b/tests/fm-startup-network.test.sh @@ -342,8 +342,11 @@ EOF assert_grep 'check startup-network' "$home/state/.wake-queue" \ "an actionable result did not reach the wake queue" ( + export FM_ROOT_OVERRIDE="$root" + export FM_HOME="$home" + export FM_STATE_OVERRIDE="$home/state" # shellcheck source=/dev/null - FM_ROOT_OVERRIDE="$root" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" . "$root/bin/fm-wake-lib.sh" + . "$root/bin/fm-wake-lib.sh" fm_wake_native_empty_fleet_preflight "$home/state" ) || fail "the native admission owner refused an actual startup completion wake" @@ -351,8 +354,11 @@ EOF sed 's/^state=done$/state=failed/' "$home/state/.startup-network.status" > "$status_tmp" mv "$status_tmp" "$home/state/.startup-network.status" ( + export FM_ROOT_OVERRIDE="$root" + export FM_HOME="$home" + export FM_STATE_OVERRIDE="$home/state" # shellcheck source=/dev/null - FM_ROOT_OVERRIDE="$root" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" . "$root/bin/fm-wake-lib.sh" + . "$root/bin/fm-wake-lib.sh" fm_wake_native_empty_fleet_preflight "$home/state" ) || fail "native admission tied a queued startup completion to mutable latest-run status" From 09d19aaf038933e5b59206b1f393753ee8b2bc58 Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 12:54:58 +1200 Subject: [PATCH 57/61] no-mistakes(lint): Localize startup fixture environment overrides --- tests/fm-startup-network.test.sh | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/tests/fm-startup-network.test.sh b/tests/fm-startup-network.test.sh index 8c14c043ac8..74d2a7f02f0 100755 --- a/tests/fm-startup-network.test.sh +++ b/tests/fm-startup-network.test.sh @@ -342,9 +342,8 @@ EOF assert_grep 'check startup-network' "$home/state/.wake-queue" \ "an actionable result did not reach the wake queue" ( - export FM_ROOT_OVERRIDE="$root" - export FM_HOME="$home" - export FM_STATE_OVERRIDE="$home/state" + local FM_ROOT_OVERRIDE="$root" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" + export FM_ROOT_OVERRIDE FM_HOME FM_STATE_OVERRIDE # shellcheck source=/dev/null . "$root/bin/fm-wake-lib.sh" fm_wake_native_empty_fleet_preflight "$home/state" @@ -354,9 +353,8 @@ EOF sed 's/^state=done$/state=failed/' "$home/state/.startup-network.status" > "$status_tmp" mv "$status_tmp" "$home/state/.startup-network.status" ( - export FM_ROOT_OVERRIDE="$root" - export FM_HOME="$home" - export FM_STATE_OVERRIDE="$home/state" + local FM_ROOT_OVERRIDE="$root" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" + export FM_ROOT_OVERRIDE FM_HOME FM_STATE_OVERRIDE # shellcheck source=/dev/null . "$root/bin/fm-wake-lib.sh" fm_wake_native_empty_fleet_preflight "$home/state" From ead9b366da1ebce721a859fb4bcaf6488d4552fb Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 15:32:32 +1200 Subject: [PATCH 58/61] docs: describe native-owner detection precedence --- .agents/skills/harness-adapters/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.agents/skills/harness-adapters/SKILL.md b/.agents/skills/harness-adapters/SKILL.md index ca4f1233245..5b443ecc543 100644 --- a/.agents/skills/harness-adapters/SKILL.md +++ b/.agents/skills/harness-adapters/SKILL.md @@ -39,8 +39,9 @@ Muse, Gemini, and AGY are verified only for crewmate and scout work, never a sec ## Detection -`../../../bin/fm-harness.sh` prints firstmate's own harness from verified environment markers and process ancestry, and owns how they combine. -A marker names its harness, but a structural ancestor of a different harness outranks it, because a marker is ordinary environment state a child or a multiplexer can retain while ancestry is what proves who owns the process tree. +`../../../bin/fm-harness.sh` prints firstmate's own harness and owns how native-owner verification, verified environment markers, and process ancestry combine. +For the [experimental native Windows Codex launcher](../../../docs/native-windows-codex.md), durable home records select native-owner verification before marker and ancestry detection; an unverified or unsupported native-owner result remains `unknown`, without falling back to those other signals. +Outside that native path, a marker names its harness, but a structural ancestor of a different harness outranks it, because a marker is ordinary environment state a child or a multiplexer can retain while ancestry is what proves who owns the process tree. Only `FM_PI_HARNESS=pi-signed` at the launch boundary together with `PI_CODING_AGENT=true` selects Pi-signed; shared unmarked launcher ancestry remains Pi. omp publishes no marker of its own; `FM_OMP_HARNESS=omp` is Firstmate's launch marker and the anchored process name `omp` is its ancestry evidence, as `references/harness/omp.md` records. `../../../bin/fm-spawn.sh` owns worker marker establishment, while the README launch command owns the signed-primary boundary. From 203a69a3007ea5d06f70869f39277cdbde430903 Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 17:45:20 +1200 Subject: [PATCH 59/61] test: isolate native host fixture appdata on Linux --- tests/fixtures/native-owner/tool-gate.test.mjs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/fixtures/native-owner/tool-gate.test.mjs b/tests/fixtures/native-owner/tool-gate.test.mjs index 41044d11343..60a64801b0b 100644 --- a/tests/fixtures/native-owner/tool-gate.test.mjs +++ b/tests/fixtures/native-owner/tool-gate.test.mjs @@ -167,7 +167,7 @@ async function hostScenario(scenario,cycles=1){ let result,failure; try { result=await runCodexHost({ - env:{...process.env,FM_PROBE_HOME:runtime,FM_PROBE_CODE_ROOT:repo,FM_HOME:home,FM_PROBE_SESSION:'session',FM_PROBE_NONCE:'nonce'}, + env:{...process.env,APPDATA:path.join(area,'appdata'),FM_PROBE_HOME:runtime,FM_PROBE_CODE_ROOT:repo,FM_HOME:home,FM_PROBE_SESSION:'session',FM_PROBE_NONCE:'nonce'}, input,output,error,native,mcpServerNames:[],spawnAppServer:()=>createFakeAppServer(scenario),installSignalHandlers:false, afterAutomaticTurn:({evidence})=>{ if(scenario==='success-then-next-check')return false; From 755daafd3b9ceb648a12c3d712984d8414862d9e Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 21:16:26 +1200 Subject: [PATCH 60/61] no-mistakes(document): Document Windows owner identities and verification limits --- bin/fm-claude-stop-autoarm.sh | 21 +++++++-------- bin/fm-lock.sh | 37 ++++++++++++++------------- bin/fm-session-lock-lib.sh | 17 ++++++------ bin/fm-session-start.sh | 22 ++++++++-------- bin/fm-startup-network.sh | 13 +++++----- bin/fm-wake-lib.sh | 4 +-- docs/scripts.md | 4 ++- docs/sessionstart-nudge.md | 11 ++++---- docs/verification/runtime-backends.md | 4 +++ docs/verification/supervision.md | 3 ++- docs/watcher-continuity.md | 4 +-- 11 files changed, 76 insertions(+), 64 deletions(-) diff --git a/bin/fm-claude-stop-autoarm.sh b/bin/fm-claude-stop-autoarm.sh index ac164d49757..464f82efc22 100755 --- a/bin/fm-claude-stop-autoarm.sh +++ b/bin/fm-claude-stop-autoarm.sh @@ -11,14 +11,15 @@ # secondmate home) with AGENTS.md, bin/, and the effective state dir - the # exact fm-turnend-guard.sh scope. Child crew/scout worktrees stay inert. # - Identity: only when THIS session holds state/.lock, as -# bin/fm-session-lock-lib.sh decides it: the recorded pid is a harness -# ancestor, or a live lock was recorded under this same trusted Claude -# session id (which is what keeps a background session arming after its -# transient helper chain is recycled). -# When an existing numeric owner fails the shared harness-liveness predicate, -# the hook delegates guarded recovery to bin/fm-lock.sh and then re-verifies -# ownership. A live owner, missing lock, malformed lock, or unresolved -# ancestry remains inert, so a competing session never arms or rewakes. +# bin/fm-session-lock-lib.sh decides it: the recorded owner belongs to this +# session's verified identity set, or a live lock was recorded under this +# same trusted Claude session id (which is what keeps a background session +# arming after its transient helper chain is recycled). +# When an existing recognized owner is proven dead by the shared liveness +# predicate, the hook delegates guarded recovery to bin/fm-lock.sh and then +# re-verifies ownership. A live or unknown owner, missing lock, malformed +# lock, or unresolved identity remains inert, so a competing session never +# arms or rewakes. # - AFK: while state/.afk exists the away daemon owns the watcher and triage; # this hook exits 0 and NEVER rewakes the primary (checked again at # translation time so a mid-cycle AFK transition is honored). @@ -59,8 +60,8 @@ # until the synchronous guard has consumed its attended fail-open. # # The epoch ledger state/.claude-autoarm-epoch records the latest claim -# generation and outcome, and binds rewake outcomes to the session-lock pid and -# watcher recovery generation, so the synchronous Stop guard +# generation and outcome, and binds rewake outcomes to the session-lock owner +# identity and watcher recovery generation, so the synchronous Stop guard # (bin/fm-turnend-guard.sh --claude) can allow a stop whose recovery this hook # already owns, instead of forcing a duplicate continuation for the same event # epoch. The failure marker diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index e66ff7340c4..67e268c7192 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -3,24 +3,25 @@ # Writes a numeric local pid, a tagged Windows pid, or an opaque launch-bound # native owner identity. # -# Line 1 of state/.lock is the owning session's anchor pid, resolved by -# fm_session_lock_anchor_pid in bin/fm-session-lock-lib.sh: the harness (agent) -# process found by walking the shell's ancestry, which lives as long as the -# firstmate session - unlike the transient subshell PID of any one tool call, -# which is dead moments after it is written. For a Claude session that proves a -# trusted session id the anchor is CLAUDE_PID, the model-loop process, so a -# shared transient daemon or a front-end that outlives the session never keeps -# a dead session's lock alive. Line 1 keeps its whole-line pid format because -# every other reader takes the first line as the pid. +# Line 1 of state/.lock is the owning session's identity, resolved by +# fm_session_lock_anchor_pid in bin/fm-session-lock-lib.sh. Process-backed +# sessions record a verified harness identity that lives as long as the +# firstmate session, unlike the transient subshell PID of any one tool call; +# the experimental native launcher records its opaque generation instead. For +# a Claude session that proves a trusted session id the anchor is CLAUDE_PID, +# the model-loop process, so a shared transient daemon or a front-end that +# outlives the session never keeps a dead session's lock alive. Every reader +# consumes the complete first line as one owner identity. # # The trusted id itself is recorded beside the lock in state/.lock-session, a # sidecar written only here and only under the claim lock: refreshed on every # confirmed-own acquisition, including the early already-mine exit that waits # for the claim lock, removed when the acquiring session proves no trusted id, # and left byte-identical when it already names that id. A same-session -# confirmation never rewrites line 1 while the recorded pid is alive, because -# bin/fm-startup-network.sh compares that pid across its deferred sweeps; a dead -# recorded pid is reclaimed and rewritten to this session's anchor. +# confirmation never rewrites line 1 while the recorded owner is positively +# live, because bin/fm-startup-network.sh compares that identity across its +# deferred sweeps; a proven-dead recorded owner is reclaimed and rewritten to +# this session's anchor. # # Usage: fm-lock.sh acquire; exit 1 unless ownership is verified # fm-lock.sh status print holder and liveness; always exits 0 @@ -36,8 +37,8 @@ LOCK="$STATE/.lock" LOCK_SESSION="$STATE/.lock-session" # Harness identity (FM_HARNESS_RE, ancestry walk, holder liveness, trusted -# session id, anchor pid) is owned by the shared session-lock lib so the Claude -# Stop auto-arm applies the exact same identity contract. +# session id, owner identity) is owned by the shared session-lock lib so the +# Claude Stop auto-arm applies the exact same identity contract. # shellcheck source=bin/fm-session-lock-lib.sh . "$SCRIPT_DIR/fm-session-lock-lib.sh" @@ -271,8 +272,8 @@ publish_lock_session_or_die() { exit 1 } -# This session already holds the lock, recorded as pid $1. Line 1 stays exactly -# as recorded while that pid is alive; only the sidecar is refreshed, under the +# This session already holds the lock under identity $1. Line 1 stays exactly +# as recorded while that owner is live; only the sidecar is refreshed, under the # claim lock, so a /clear re-key inside the same process replaces the old id. # A same-session confirmation waits for the claim lock so the sidecar refresh # completes. After the wait, the lock is re-read and the sidecar is refreshed @@ -280,7 +281,7 @@ publish_lock_session_or_die() { # and the caller continues with the ordinary live-owner or reclaim path. The # prior-session-sweep-is-finishing refusal is a takeover rule and does not # apply here. -confirm_own_lock() { # +confirm_own_lock() { # local recorded waited=0 if [ "$CLAIM_LOCK_HELD" -ne 1 ]; then fm_lock_acquire_wait "$CLAIM_LOCK" @@ -341,7 +342,7 @@ if [ -e "$LOCK" ] || [ -L "$LOCK" ]; then fi fi fi -# The sidecar goes first: a fresh pid beside a previous session's id would let +# The sidecar goes first: a fresh owner beside a previous session's id would let # that session's resume own this lock. If the sidecar changes before line 1 is # written, a failure restores the previous sidecar. If line 1 is written but # not yet verified, a failure removes the sidecar and leaves the lock diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index 95346d72ff8..bde52872e37 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -470,14 +470,15 @@ fm_session_lock_same_session() { # [] [ "$recorded" = "$trusted" ] } -# Print the pid bin/fm-lock.sh records on lock line 1 for this session. For a -# Claude session with a trusted id that is CLAUDE_PID, the model-loop process: -# never the shared transient daemon and never a front-end that outlives the -# session, so "recorded pid dead" keeps meaning "session gone" instead of -# wedging a home behind a live daemon whose session died. A replaced background -# helper leaves a dead pid that its own session's next hook reclaims, because -# the sidecar still names that session. Every other session records the -# outermost pid of its contiguous run, exactly as before. +# Print the owner identity bin/fm-lock.sh records on lock line 1 for this +# session. The native route retains its opaque generation. For a Claude session +# with a trusted id the identity is CLAUDE_PID, the model-loop process: never the +# shared transient daemon and never a front-end that outlives the session, so a +# dead recorded process still means the session is gone instead of wedging a +# home behind a live daemon whose session died. A replaced background helper +# leaves a dead pid that its own session's next hook reclaims, because the +# sidecar still names that session. Every other process-backed session records +# the outermost pid of its verified identity set. fm_session_lock_anchor_pid() { local pids pids=$(fm_harness_ancestry_pids) || return 1 diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index e57ec0b480f..ae0098d1d28 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -203,19 +203,19 @@ # and a session that owns the lock is exactly the session that must # handle and acknowledge them. Lock acquisition still runs, because # ownership must be re-verified rather than assumed: fm-lock.sh -# already treats a lock owned through shared ancestry or a trusted -# same-session Claude id as its own, so the re-emit proceeds, while -# a lock another live session took meanwhile still produces the -# ordinary read-only path. +# already accepts authenticated native ownership, a member of the +# current session's verified identity set, or a trusted same-session +# Claude id, so the re-emit proceeds, while a lock another live +# session took meanwhile still produces the ordinary read-only path. # # --source The native session-open source, supplied only by # fm-sessionstart-run.sh. A genuine `startup` that owns the active # session lock records AGENTS.md's SHA-256 baseline only after the -# digest completion record is published, keyed to that lock's -# harness pid. No resume, clear, reset, compact, or other rebuild +# digest completion record is published, keyed to that lock's owner +# identity. No resume, clear, reset, compact, or other rebuild # creates or replaces it. Pi and pi-signed compaction are the only # supported stale-cache rebuild pair: a missing baseline, a baseline -# for another harness pid, or a changed hash causes the complete +# for another owner identity, or a changed hash causes the complete # current AGENTS.md to print before the bulky digest. The baseline # remains immutable so every later drifted compaction refreshes # again, while an equal baseline emits no instruction refresh. @@ -558,7 +558,7 @@ hash_file_sha256() { # The baseline describes instructions this true session started with, not the # most recently emitted instructions. It is intentionally immutable for this # lock owner: every later stale-context rebuild needs the current file again. -write_agents_baseline() { # +write_agents_baseline() { # local lock_pid=$1 agents_hash=$2 tmp [ -n "$lock_pid" ] && [ -n "$agents_hash" ] || return 1 tmp=$(mktemp "$STATE/.session-start-agents-baseline.XXXXXX" 2>/dev/null) || return 1 @@ -570,7 +570,7 @@ write_agents_baseline() { # return 1 } -agents_baseline_drifted() { # +agents_baseline_drifted() { # local lock_pid=$1 baseline_pid baseline_hash current_hash [ -f "$AGENTS_BASELINE_FILE" ] && [ ! -L "$AGENTS_BASELINE_FILE" ] || return 0 baseline_pid=$(sed -n '1p' "$AGENTS_BASELINE_FILE" 2>/dev/null || true) @@ -584,7 +584,7 @@ agents_baseline_drifted() { # # Only run-tier source pairs with both a stale native instruction cache and a # working Firstmate delivery path arrive here. Claude fresh-reads on reset, and # Codex has no tracked interactive reset delivery path. -agents_refresh_required() { # +agents_refresh_required() { # local lock_pid=$1 case "$PRIMARY_HARNESS:$SESSION_SOURCE" in pi:compact|pi-signed:compact) ;; @@ -593,7 +593,7 @@ agents_refresh_required() { # agents_baseline_drifted "$lock_pid" } -print_agents_refresh_if_required() { # +print_agents_refresh_if_required() { # local lock_pid=$1 agents_refresh_required "$lock_pid" || return 0 section "CURRENT AGENTS.md - INSTRUCTION REFRESH" diff --git a/bin/fm-startup-network.sh b/bin/fm-startup-network.sh index a9a649718e4..0db1681a7e4 100755 --- a/bin/fm-startup-network.sh +++ b/bin/fm-startup-network.sh @@ -78,7 +78,8 @@ # unrecognized state and 2 for invalid usage. # # STATE, all under this home's state/ and gitignored with it: -# .startup-network.status key=value record - generation, lock_pid, state, +# .startup-network.status key=value record - generation, lock_pid (the +# session-lock owner identity), state, # pid, started, finished, rc, locked, phases, and # whether the report was published. The single # source of truth for what ran and how it ended. @@ -207,7 +208,7 @@ phase_label() { # # --- start ------------------------------------------------------------------- -worker_covers_request() { # +worker_covers_request() { # local locked=$1 lock_pid=$2 [ "$locked" != 1 ] && return 0 [ "$(status_get lock_pid)" = "$lock_pid" ] \ @@ -305,13 +306,13 @@ EOF # The question is deliberately "does the lock still name the session that asked # for this work?", not "is that session still alive". The hazard being closed is # a SECOND session sweeping concurrently. A different session can take the lock -# only after the recorded holder is dead, when bin/fm-lock.sh rewrites that pid -# with its own anchor. An unchanged value therefore proves no one else owns the sweeps, which is +# only after the recorded holder is proven dead, when bin/fm-lock.sh rewrites +# that identity with its own anchor. An unchanged value therefore proves no one else owns the sweeps, which is # the whole guarantee. Requiring liveness instead would refuse to finish work # nobody else has claimed, and the sweeps are idempotent, so finishing it is # strictly better than abandoning it. A missing, unreadable, or replaced lock all # fail closed to the read-only probe. -lock_unchanged() { # +lock_unchanged() { # local expected=$1 current fm_session_pid_valid "$expected" || return 1 [ -f "$STATE/.lock" ] && [ ! -L "$STATE/.lock" ] || return 1 @@ -440,7 +441,7 @@ EOF await_delivery "$generation" "$state" } -cmd_run() { # +cmd_run() { # local locked=$1 lock_pid=$2 generation=$3 phases started budget out rc sweep_locked=0 downgraded=0 internal=0 lease_held=0 timings stage_started mkdir -p "$STATE" 2>/dev/null || return 1 started=$(now) diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index c78925f1756..d9feb35fa14 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1649,8 +1649,8 @@ fm_autoarm_claim_open() { # [grace] # The watcher runs only between turns; turn-end re-arms. # # Healthy means outcome=rewake with no exhausted-failure marker, bound to the -# current session-lock pid and current watcher recovery generation. The rewake -# ledger must also be at least as new as the last watcher beacon: a later beacon +# current session-lock owner identity and current watcher recovery generation. +# The rewake ledger must also be at least as new as the last watcher beacon: a later beacon # proves another between-turns watcher cycle has begun, so the rewake belongs to # an earlier handling turn. # diff --git a/docs/scripts.md b/docs/scripts.md index ef45d68dfa2..f2927ca58a4 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -44,7 +44,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-ensure-agents-md.sh` | Ensure a project's real `AGENTS.md`, its `CLAUDE.md` `@AGENTS.md` pointer, and self-governance guidance (explicit project mark documented in the helper's header and help) | | `fm-guard.sh` | Warn on primary-checkout tangles, main-session pending wakes, and unhealthy supervision | | `fm-primary-scope-lib.sh` | Shared marker-or-plain-checkout primary-home predicate for tracked hooks | -| `fm-session-lock-lib.sh` | Shared session-lock ownership from harness ancestry or a trusted Claude session id for fm-lock.sh and the Claude Stop auto-arm | +| `fm-session-lock-lib.sh` | Shared session-lock identity, liveness, and self-ownership classification for lock and guard consumers | | `fm-claude-stop-autoarm.sh` | Claude Stop `asyncRewake` hook owning tokenless watcher continuity with single-flight exit-2 rewake (docs/watcher-continuity.md) | | `fm-turnend-guard.sh` | Shared primary turn-end guard predicate so no turn ends blind (docs/turnend-guard.md) | | `fm-turnend-guard-grok.sh` | Grok Stop-hook adapter for the primary turn-end guard | @@ -99,6 +99,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-timeout-lib.sh` | Single owner of hard-bounded command execution and its fallback watchdog | | `fm-timing-lib.sh` | Single owner of the deferred network stage's per-step elapsed-time records, inert unless a run asks for them | | `fm-supervision-lib.sh` | Shared in-flight-work-without-fresh-watcher-beacon predicate | +| `fm-terminal-outcome-lib.sh` | Shared refusal predicate for unresolved inactive terminal outcomes | | `fm-ff-lib.sh` | Shared guarded fast-forward/reconcile helper for origin pulls and secondmate syncs, with durable divergence markers | | `fm-lock-lib.sh` | Shared "is this git lock provably abandoned?" proof used by teardown and fleet-sync | | `fm-config-inherit-lib.sh` | Shared primary-to-secondmate inherited local-material propagation and config-reread delivery | @@ -141,6 +142,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-teardown.sh` | Fail-closed teardown: return landed ship worktrees, require completed scout deliverables, retire secondmate homes | | `fm-harness.sh` | Detect the running harness, resolve crew or secondmate harness, model, and effort, and validate the native-only `ultra` effort | | `fm-lock.sh` | Per-home firstmate session lock | +| [`fm-native-codex.ps1`](native-windows-codex.md) | Build or explicitly launch the restricted experimental native Windows Codex candidate | | `fm-x-lib.sh` | Shared Relay config, relay, and reply-threading helpers | | `fm-x-poll.sh` | One bounded Relay poll: stash newly offered mentions and emit their once-only wake | | `fm-x-reply.sh` | Post or dry-run preview a composed Relay reply or follow-up | diff --git a/docs/sessionstart-nudge.md b/docs/sessionstart-nudge.md index 11018891ec1..86bb1a047da 100644 --- a/docs/sessionstart-nudge.md +++ b/docs/sessionstart-nudge.md @@ -32,9 +32,9 @@ This deliberately inverts the previous nudge matcher, which fired on `startup|re Compaction is covered where a tracked adapter delivers that source because a compacted session has lost exactly the digest it needs, and resume is excluded from the run because it restores that digest instead of losing it. Current harness ownership of the lock and its matching `state/.session-start-complete` record together are the idempotency interlock for the whole scheme. -The full digest clears that completion record after acquiring the lock and republishes the lock owner's pid only after every stage completes, so `clear` or `compact` cannot skip startup sweeps after a truncated run. -`bin/fm-lock.sh` treats a lock owned through either the shared ancestry verdict or a trusted same-session Claude id as this session's own, so a proven `clear` or `compact` re-emit re-verifies ownership and proceeds, while a lock another live session took meanwhile still produces the ordinary read-only digest. -On a run-tier harness only `resume`, `reload`, and `fork` are routed to the nudge wrapper, whose separate ancestry-only check normally stays silent when this process already holds the lock. +The full digest clears that completion record after acquiring the lock and republishes the lock owner's identity only after every stage completes, so `clear` or `compact` cannot skip startup sweeps after a truncated run. +`bin/fm-lock.sh` treats authenticated native ownership, an owner in the current session's verified identity set, or a trusted same-session Claude id as this session's own, so a proven `clear` or `compact` re-emit re-verifies ownership and proceeds, while a lock another live session took meanwhile still produces the ordinary read-only digest. +On a run-tier harness only `resume`, `reload`, and `fork` are routed to the nudge wrapper, whose separate local-ancestry or shared cross-boundary ownership check normally stays silent when this process already holds the lock. After a background Claude helper-chain recycle breaks that ancestry, the wrapper may emit a redundant nudge even though the shared same-session verdict still owns the lock; the requested session start remains idempotent. `bin/fm-session-start.sh --reemit` owns which work a re-emit skips, its true-start AGENTS.md baseline, and its supported stale-instruction refresh pairs; its header is the single owner of those mechanics. @@ -60,8 +60,9 @@ The Guard Predicates section of [`turnend-guard.md`](turnend-guard.md#guard-pred The nudge payload starts with U+2063 and the stable `FIRSTMATE_OP: ` label, carries the current `session-start` protocol kind, and retains exactly ``Run `bin/fm-session-start.sh` now, exactly once, before executing any other instructions.`` as its body. The Ahoy skill owns the rule that this marked operational input is never a captain-authored session boundary, including its narrow legacy compatibility cases, and its own step 0 helm check is the fallback that protects a nudge-tier harness whose first command is a skill. -Before printing, the nudge wrapper reads `state/.lock` and walks at most eight parents from its own pid in its own separate, hard-coded loop, independent of the shared sixteen-hop ancestry walk in `bin/fm-session-lock-lib.sh` that `bin/fm-lock.sh` uses for anchor selection and ownership, and independent of Pi's `lockOwnership()`. -If the lock names a live pid in that ancestry, session start already ran in this harness session and the wrapper stays silent. +Before printing, the nudge wrapper reads `state/.lock`; for a local numeric owner it walks at most eight parents from its own pid in a separate hard-coded loop, independent of the shared sixteen-hop identity walk in `bin/fm-session-lock-lib.sh` and independent of Pi's `lockOwnership()`. +For an opaque native owner or tagged Windows process identity it delegates to that shared library, because the local parent table cannot answer across either boundary. +If either route proves the lock belongs to this harness session, session start already ran and the wrapper stays silent. Every ordinary transport path in both wrappers exits 0, including malformed state and adapter errors, because a Claude SessionStart exit 2 blocks session initialization. The run wrapper's internal `--pi-prerequisite` mode uses silent exit 3 only for an intentional gate or scope stand-down, letting Pi distinguish ineligibility from an eligible empty native result without changing any harness hook's exit contract. A lock another session holds and a truncated digest therefore surface as digest text, while broken GitHub auth surfaces through the deferred network result inline or as a wake; none becomes a refusal to open the session. diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index dbd0e8c008c..42f5b351553 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -6,6 +6,10 @@ This record contains reusable version-scoped evidence for active runtime guarant The backend guides own current setup, safety boundaries, and limitations. Exact task chronology, branch names, temporary homes, local paths, process ids, thread ids, and delivery transcripts remain in private reports or PR evidence. +## Windows session-lock ownership + +The tagged Windows process bridge has portable deterministic coverage in `tests/fm-session-lock-ancestry.test.sh`, but Windows Claude ownership has not been verified. + ## Harness detection precedence `bin/fm-harness.sh` owns native-owner, marker, and ancestry precedence; the evidence below covers marker and ancestry only, not the experimental native candidate. diff --git a/docs/verification/supervision.md b/docs/verification/supervision.md index 6e5198fa2ab..28af15ac580 100644 --- a/docs/verification/supervision.md +++ b/docs/verification/supervision.md @@ -323,11 +323,12 @@ That inertness result is scoped to the builds it exercised: it did not establish The secondmate-home scope and manual-repair wake path were measured with Claude Code 2.1.207 on 2026-07-12, when a native background completion re-invoked the idle model with no human input. The current Stop-owned main/secondmate inclusion and child-worktree exclusion are covered deterministically by `tests/fm-claude-stop-autoarm.test.sh`. -Session-lock ownership in `bin/fm-session-lock-lib.sh` is decided against a session's whole contiguous harness ancestry rather than one chosen pid, so the Stop auto-arm reaches its lock owner wherever that owner sits: a pid of Claude Code's multi-level `bg-spare` hook worker chain, or an inner pid when a harness-named daemon parents the session. +On the POSIX path covered here, session-lock ownership in `bin/fm-session-lock-lib.sh` is decided against a session's whole contiguous harness ancestry rather than one chosen pid, so the Stop auto-arm reaches its lock owner wherever that owner sits: a pid of Claude Code's multi-level `bg-spare` hook worker chain, or an inner pid when a harness-named daemon parents the session. A background Claude session whose transient helper chain is recycled loses that contiguity while its recorded owner stays alive, so the library also accepts a trusted same-session id: `CLAUDE_CODE_SESSION_ID` counts only when `CLAUDE_PID` is a Claude-shaped member of the current run, it must equal the id `bin/fm-lock.sh` recorded in `state/.lock-session`, and the recorded pid must still be a live harness, while every weaker combination (no id, no sidecar, an untrusted id, a different id, a dead recorded pid) leaves the ancestry verdict unchanged. For such a session `bin/fm-lock.sh` records `CLAUDE_PID` on lock line 1 instead of the outermost chain pid, so a shared daemon or front-end that outlives the session never keeps a dead session's lock alive, and a same-session confirmation never rewrites a live line 1. Harness identity is read from the executable path and `argv[0]` as well as the command basename, because Claude Code's native installer names the per-session executable by its version (`.../share/claude/versions/2.1.220`): `ps -o comm=` reports that path on macOS and the bare version string on Linux, and neither basename names a harness. `tests/fm-session-lock-ancestry.test.sh` pins both platforms' reporting semantics behind a deterministic process table and runs the real Stop auto-arm in version-named, daemon-parented, and combined real process trees. +[`runtime-backends.md`](runtime-backends.md#windows-session-lock-ownership) owns the separate tagged-Windows coverage status. The same suite drives the ancestry and session-id signals apart in that table, asserting the divergence itself so no case is vacuous, and runs a real orphaned front-end, daemon, pty-host, and bg-spare tree whose daemon is ended mid-run: the same id keeps arming through the real `bin/fm-lock.sh`, `bin/fm-claude-stop-autoarm.sh`, and `bin/fm-turnend-guard.sh --claude` with lock line 1 and the sidecar untouched, a different id, an untrusted id, and no id each keep the live-owner refusal naming the recorded id, and the dead front-end is reclaimed onto the spare's pid rather than the outermost pty-host. `tests/fm-turnend-foreign-owner-repro.py` keeps the genuinely foreign live owner as the negative control and adds the same-id positive control. Both ran on 2026-09-18 on macOS with bash 3.2.57 as the fake harness interpreter: diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index 506f076e201..680c9c02c5d 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -14,8 +14,8 @@ omp's replacement follows the same generation-owner contract in `.omp/extensions Cursor's `.cursor/hooks.json` `stop` hook (`bin/fm-turnend-guard-cursor.sh`) owns routine tokenless re-arm for a Cursor primary by parking that awaited hook on `bin/fm-watch-arm.sh` and returning an actionable close as one follow-up; [`turnend-guard.md`](turnend-guard.md#harness-integrations) owns its Pi-host stand-down, loop bounds, and supersession baton. Claude's `.claude/settings.json` Stop `asyncRewake` hook (`bin/fm-claude-stop-autoarm.sh`) owns routine tokenless re-arm. The hook fires on every Stop, and an eligible primary with supervision need admits one home-scoped owner that foregrounds `bin/fm-watch-arm.sh` inside the hook-owned process tree. -A numeric session-lock owner that fails the shared `fm_harness_pid_alive` predicate is reclaimed through `bin/fm-lock.sh` before auto-arm state changes, while a live owner the session does not own, an absent lock, or a malformed lock keeps the competing hook inert. -Whether the session owns that lock is the shared `fm_session_lock_owned_by_self` verdict in `bin/fm-session-lock-lib.sh`, which accepts a recorded pid inside the current harness ancestry or a live lock recorded under this same trusted Claude session id, so a background session keeps arming after its transient helper chain is recycled. +A recognized session-lock owner that the shared `fm_harness_pid_alive` predicate proves dead is reclaimed through `bin/fm-lock.sh` before auto-arm state changes, while a live or unknown owner the session does not own, an absent lock, or a malformed lock keeps the competing hook inert. +Whether the session owns that lock is the shared `fm_session_lock_owned_by_self` verdict in `bin/fm-session-lock-lib.sh`, which accepts a recorded process identity in the current session's verified identity set or a live lock recorded under this same trusted Claude session id, so a background session keeps arming after its transient helper chain is recycled. [`turnend-guard.md`](turnend-guard.md#guard-predicates) owns the Claude guard's behavior when that live owner is genuinely another session. The stale-owner claim occurs only after the existing AFK and supervision-need gates pass. After each non-actionable arm close, the hook rechecks the identity-matched watcher lock and fresh beacon before retrying a bounded number of times. From ee9be1c8386eb4833ee630dc748782575c3babfc Mon Sep 17 00:00:00 2001 From: Cristian Date: Sat, 19 Sep 2026 21:56:46 +1200 Subject: [PATCH 61/61] no-mistakes(ci): Captain, fixed the pre-existing cleanup race in tests/fm-public-followup.test.sh. The fixture now stops and joins its scoped remote-worker tree before deleting temporary files, and reports failure if shutdown cannot be confirmed. This prevents the late writer that caused `Directory not empty`; `git diff --check` passes. Focused Bash execution was unavailable because managed Git Bash fails with Win32 error 5; host verification command: `FM_TEST_ONLY=test_remote_secondmate_loop_delivers_and_retires bash tests/fm-public-followup.test.sh`. Windows Claude ownership has not been verified. That run-scoped exception was not applied to this CI failure --- tests/fm-public-followup.test.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/fm-public-followup.test.sh b/tests/fm-public-followup.test.sh index c5551d99a0b..107542263ab 100755 --- a/tests/fm-public-followup.test.sh +++ b/tests/fm-public-followup.test.sh @@ -53,7 +53,18 @@ pf_test_cleanup() { fi if [ -f "$pid_file" ]; then pid=$(cat "$pid_file" 2>/dev/null) || pid= - [ -z "$pid" ] || kill "$pid" 2>/dev/null || true + if [ -n "$pid" ]; then + ( + # worker.pid names the serving child; stop its known isolated supervisor + # tree and wait for its cleanup before removing the fixture directory. + # shellcheck source=bin/fm-remote-job-lib.sh + . "$ROOT/bin/fm-remote-job-lib.sh" + fm_remote_job_stop_worker_tree "$pid" + ) || { + printf 'not ok - remote fixture worker did not stop before cleanup\n' >&2 + return 1 + } + fi fi fm_test_cleanup }