Repository navigation
[docs] Reposition from coding agents to agent harnesses #731
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
eeed712
[docs] Reposition from coding agents to agent harnesses
NiveditJain ca0064f
[docs] Reference the PR number in the changelog entries
NiveditJain 529f20b
[docs] Address review: harness classes, audit claims, guard scope
NiveditJain c2af86a
[docs] Note the audit-report accuracy fix in the changelog
NiveditJain c074939
[docs] Stop counting the Python SDK among the twelve harnesses
NiveditJain File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| // @vitest-environment node | ||
| /** | ||
| * Drift guard for the NUMBERS AND HARNESS NAMES in user-facing prose. | ||
| * | ||
| * The positioning rewrite put four counts into copy read by nearly everyone who | ||
| * meets this project — 12 harnesses, 40 built-in policies, 29 canonical events, | ||
| * and blocking verified on 12/12 PreToolUse and 8/12 Stop. Those counts are only | ||
| * worth printing flat, without hedging, because `enforcement-capability.ts` is a | ||
| * machine-readable matrix with the probed version on every row. A stale count | ||
| * turns that asset into a liability: an engineer who checks one number and finds | ||
| * it wrong has a reason to discount the whole claim. | ||
| * | ||
| * This is not hypothetical. Before this file existed, README.md advertised "30 | ||
| * built-in policies" in three places and the landing site's schema.org block said | ||
| * 39 — while the real count was 40, and had been for some time. Both were | ||
| * hand-maintained, nothing read them, so both rotted silently. Same failure mode | ||
| * __tests__/hooks/dogfood-configs.test.ts was written to catch for the dogfood | ||
| * configs. | ||
| * | ||
| * SCOPE: this repo only. The landing site (befailproof.ai) lives in a different | ||
| * repo and carries the same counts in app/layout.js and app/llms.txt/route.js; | ||
| * it needs its own check. See FailproofAI/agenteye#629. | ||
| */ | ||
| import { describe, it, expect } from "vitest"; | ||
| import { readFileSync } from "node:fs"; | ||
| import { resolve } from "node:path"; | ||
| import { INTEGRATION_TYPES, HOOK_EVENT_TYPES } from "@/src/hooks/types"; | ||
| import { ENFORCEMENT_CAPABILITY } from "@/src/hooks/enforcement-capability"; | ||
| import { BUILTIN_POLICIES } from "@/src/hooks/builtin-policies"; | ||
|
|
||
| const ROOT = process.cwd(); | ||
| const read = (p: string) => readFileSync(resolve(ROOT, p), "utf8"); | ||
|
|
||
| /** Every count asserted below, derived from source rather than restated. */ | ||
| const TRUTH = { | ||
| harnesses: INTEGRATION_TYPES.length, | ||
| policies: BUILTIN_POLICIES.length, | ||
| events: HOOK_EVENT_TYPES.length, | ||
| preToolUseBlocks: INTEGRATION_TYPES.filter( | ||
| (c) => ENFORCEMENT_CAPABILITY[c]?.PreToolUse === "block", | ||
| ).length, | ||
| stopBlocks: INTEGRATION_TYPES.filter((c) => ENFORCEMENT_CAPABILITY[c]?.Stop === "block").length, | ||
| }; | ||
|
|
||
| /** | ||
| * Files that state a count in prose a reader acts on. Deliberately short: this | ||
| * guards the strings with the most readers, not every mention in the tree. | ||
| */ | ||
| const COPY_FILES = ["README.md", "package.json", "docs/index.mdx"]; | ||
|
|
||
| describe("copy counts match source", () => { | ||
| it("the derived counts are what the copy claims", () => { | ||
| // Fails loudly if someone adds a harness or a policy without re-reading the | ||
| // copy. Update this block ONLY together with every string it guards. | ||
| expect(TRUTH).toEqual({ | ||
| harnesses: 12, | ||
| policies: 40, | ||
| events: 29, | ||
| preToolUseBlocks: 12, | ||
| stopBlocks: 8, | ||
| }); | ||
| }); | ||
|
|
||
| it.each(COPY_FILES)("%s states no stale policy count", (file) => { | ||
| const text = read(file); | ||
| const claimed = [...text.matchAll(/(\d+)\s+built-in polic/gi)].map((m) => Number(m[1])); | ||
| for (const n of claimed) expect(n).toBe(TRUTH.policies); | ||
| }); | ||
|
NiveditJain marked this conversation as resolved.
|
||
|
|
||
| it.each(COPY_FILES)("%s states no stale harness count", (file) => { | ||
| const text = read(file); | ||
| const claimed = [...text.matchAll(/(\d+)\s+(?:agent\s+)?harness(?:es)?\b/gi)].map((m) => | ||
| Number(m[1]), | ||
| ); | ||
| for (const n of claimed) expect(n).toBe(TRUTH.harnesses); | ||
| }); | ||
|
|
||
| it("README names only harnesses that ship", () => { | ||
| const text = read("README.md").toLowerCase(); | ||
| // Named on marketing surfaces but absent from INTEGRATION_TYPES. Keeping | ||
| // them out of the README is what makes the harness list checkable. | ||
| for (const absent of ["gemini cli", "langgraph", "deep agents"]) { | ||
| expect(text).not.toContain(absent); | ||
| } | ||
| }); | ||
|
|
||
| it("README names the two harnesses that prove the product is not coding-only", () => { | ||
| // The whole positioning rests on the harness list spanning classes. Hermes | ||
| // (Slack/Telegram gateway) and OpenClaw (self-hosted assistant) are the two | ||
| // shipped integrations that were never coding tools; if a future edit drops | ||
| // them from the hero, the breadth claim silently loses its evidence. | ||
| const text = read("README.md").toLowerCase(); | ||
| expect(text).toContain("hermes"); | ||
| expect(text).toContain("openclaw"); | ||
| }); | ||
|
|
||
| it("the translated READMEs are linked from the tree the pipeline maintains", () => { | ||
| // docs-old/i18n/ was frozen when the Mintlify site replaced it; the | ||
| // translation workflow writes docs/i18n/. Linking the frozen copies means | ||
| // non-English readers get a README that is never updated again. | ||
| expect(read("README.md")).not.toContain("docs-old/i18n/"); | ||
| }); | ||
| }); | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.