From f7dd4131f4e281fd0d6d9d2bb8e3e24a47fdf870 Mon Sep 17 00:00:00 2001 From: Claude Code Date: Sat, 1 Aug 2026 09:49:49 +0000 Subject: [PATCH 1/2] feat(access,cli): the integrity checks, ow check, and source state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes plan tasks 7.1-7.5, 7.7 and 6.1. Settles the design gap the plan deferred to group 7, and the gap was wider than the note there described. `listEntityPages` read only the top level of `wiki/`, while the plan's own layout files pages as `wiki/projects/`, `wiki/people/`, `wiki/topics/`. So a project that followed the documented layout had no pages at all as far as the store was concerned: nothing indexed, no orphan reported, `ow graph` empty, MCP serving nothing, every `[[link]]` reading as broken. The gate still validated writes to them, which is what kept it quiet. A page is now its slug wherever it sits under `wiki/` — a folder is organisation, a link is a name — recorded as `adr:0016-a-page-is-its-slug-wherever-it-sits`. Slug uniqueness is the one rule that needs, and it is a finding rather than a silent choice. Codewiki lives at `wiki/codewiki/`; a top-level `codewiki/` is no longer gated and is reported as misplaced. The checks (7.1-7.5) each answer a question the gate cannot, because every one is about a relationship between things rather than a single write. 7.4 needs no glossary file: a page's `title` is the canonical term and its `aliases` are the synonyms, so a separate file would be a second record of one fact. The scaffolded skill told the agent to "check the glossary" — a file that has never existed — and now says what is actually there. 6.1 derives state from disk rather than persisting it. The filesystem is already persisted and resumable: manifest.json, text.md, the pages, journal.json. A state file beside those is the same two-records problem. `ow check` (7.7) exits 0 clean / 1 could not run / 2 found errors, matching the `scc` contract this repo documents. Only errors fail: a source uploaded this morning that nothing cites yet must not turn CI red. From the two reviews on this branch: - Catastrophic backtracking in the codewiki citation regex. `/` was in both the segment class and the separator, so the pattern was ambiguous: ~26 slash segments took 1.6s, ~34 took minutes. Eighty bytes of page body — which an agent writes, possibly steered by a poisoned source in raw/ — wedged `ow check`, CI and the UI in a synchronous spin no try/catch interrupts. - `ow graph`, `ow search` and the MCP read tools all still assumed a page was at the top level, so the ADR's central claim was false in three places. MCP's was silent: a superseded page came back as `type: unknown, status: active`. - The gate started denying any page that wrote `[[changelog]]` or `[[index]]`, with a reason that read as a bug — those files exist in every scaffolded project and the skill tells the agent to use them. - `checkProject` created `wiki/` through `readIndex`, so a read wrote — and it is exported into the read-only surface the MCP process imports. - `checkVocabulary` blamed the wrong page when two pages claimed one alias, and told a page to stop writing its own title. Both are now reported as the conflict they are. - Reported lines pointed into the frontmatter; a citation on a heading line was not counted; a page's H1 title was reported as an uncited section; a file ending in a newline was counted one line too long, so a citation one line past the end was accepted. - Control characters from page content reached the terminal unescaped, letting an alias forge `ow check: no findings`. - `checkLinks` walked the wiki once per page; 3s over 800 pages. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_016iMM93Wk43o44V5J2AxPgL --- ...016-a-page-is-its-slug-wherever-it-sits.md | 79 +++ packages/access/src/check/checks.ts | 579 ++++++++++++++++ packages/access/src/check/findings.ts | 102 +++ packages/access/src/gate/gate.ts | 32 +- packages/access/src/index.ts | 42 +- packages/access/src/read.ts | 36 +- packages/access/src/skills.ts | 33 +- packages/access/src/sources/state.ts | 123 ++++ packages/access/src/store/index-write.ts | 7 +- packages/access/src/store/index.ts | 110 ++- packages/access/src/store/page.ts | 6 + packages/access/src/store/wikilinks.ts | 61 +- packages/access/tests/check.spec.ts | 630 ++++++++++++++++++ packages/access/tests/gate-decision.spec.ts | 63 +- packages/access/tests/sources-state.spec.ts | 156 +++++ packages/access/tests/store-index.spec.ts | 107 ++- packages/access/tests/store-wikilinks.spec.ts | 55 +- packages/cli/src/commands/check.ts | 88 +++ packages/cli/src/commands/graph.ts | 31 +- packages/cli/src/commands/search.ts | 15 +- packages/cli/src/hooks.ts | 22 +- packages/cli/src/main.ts | 18 + packages/cli/tests/check.spec.ts | 175 +++++ packages/cli/tests/hooks.spec.ts | 17 +- packages/cli/tests/queries.spec.ts | 54 +- packages/mcp/src/tools.ts | 23 +- packages/mcp/tests/mcp.spec.ts | 67 +- plans/open-wiki.md | 18 +- 28 files changed, 2635 insertions(+), 114 deletions(-) create mode 100644 docs/adr/0016-a-page-is-its-slug-wherever-it-sits.md create mode 100644 packages/access/src/check/checks.ts create mode 100644 packages/access/src/check/findings.ts create mode 100644 packages/access/src/sources/state.ts create mode 100644 packages/access/tests/check.spec.ts create mode 100644 packages/access/tests/sources-state.spec.ts create mode 100644 packages/cli/src/commands/check.ts create mode 100644 packages/cli/tests/check.spec.ts diff --git a/docs/adr/0016-a-page-is-its-slug-wherever-it-sits.md b/docs/adr/0016-a-page-is-its-slug-wherever-it-sits.md new file mode 100644 index 0000000..100b42f --- /dev/null +++ b/docs/adr/0016-a-page-is-its-slug-wherever-it-sits.md @@ -0,0 +1,79 @@ +--- +status: accepted +--- + +# 0016 · A page is its slug, wherever it sits under wiki/ + +## Context + +Two documents described the wiki's shape and did not agree, and the code +implemented a third thing. + +- The plan's directory diagram files pages by type: + `wiki/projects/*.md`, `wiki/people/*.md`, `wiki/topics/*.md`, + `wiki/codewiki/*.md`. +- The `wiki` skill scaffolded into every project tells the agent to write the + page at `wiki/.md`, flat. +- `listEntityPages` read only the **top level** of `wiki/`, and + `resolveWikilinks` resolved `[[target]]` by testing for `wiki/.md`. + +So a project that followed the plan's layout had no pages at all as far as the +store was concerned: nothing appeared in the index, no orphan was ever reported, +`ow graph` returned an empty list, the MCP read tools served nothing, and every +`[[link]]` to such a page read as broken. The gate still validated writes to +them, which is what kept the contradiction quiet — pages were being checked on +the way in and then losing their existence. + +Plan task 7.5 recorded a narrower version of this as a design gap to settle +here: it named codewiki, whose pages the gate accepted at the project's top +level while the skill's prose put them under `wiki/`. Building the integrity +checks is what showed the gap was not about codewiki. It was about every +subdirectory. + +Whatever settles it has to answer one question — what does `[[checkout]]` name? +— because the whole convention rests on that link resolving. + +## Decision + +**A page is addressed by its slug: the filename without `.md`, wherever the file +sits under `wiki/`.** A folder is organisation; a link is a name. + +- `wiki/checkout.md` and `wiki/topics/checkout.md` are both reached as + `[[checkout]]`. The flat layout the skill teaches and the typed layout the + plan draws are the same model, differently filed. +- **A slug is unique across the wiki.** `[[checkout]]` cannot mean two files. + This is reported as a finding (`page.duplicate-slug`), not resolved by + picking one — a link silently pointing at the wrong page is worse than a link + that says it is ambiguous. +- `index.md`, `changelog.md` and `log.md` are the wiki's own pages only at the + top of `wiki/`. A `wiki/topics/index.md` is an ordinary page called "index". +- **Codewiki lives at `wiki/codewiki/`.** The gate no longer treats a top-level + `codewiki/` as part of the wiki, because it is not: nothing indexes it, + nothing links it, nothing can cite it. `ow check` reports it as misplaced + rather than letting it look right. + +This is the model Obsidian uses, which is where `[[wikilink]]` comes from. It is +also the only one that leaves both existing documents true, which matters more +than usual here: the skill is scaffolded into projects and ages there, so a +convention that contradicts the code cannot be fixed by editing one file. + +## Consequences + +- Pages filed by type become visible: indexed, checked, walked by `ow graph`, + served over MCP. For a project that already followed the plan's layout this + looks like a sudden crop of `page.orphan` findings — they were always orphans; + nothing could see them. +- `resolveWikilinks` walks `wiki/` once per validated write instead of doing one + `existsSync` per link. It runs on the gate's hot path, so this is a real cost, + bounded by the number of pages — a few milliseconds for a wiki of hundreds. + If that ever stops being true, the answer is a cache with an invalidation + story, not a return to path-shaped links. +- Renaming a page still means fixing the links that pointed at it, but moving + one between folders now costs nothing — the slug did not change. That is the + property worth having, and it is why the alternative (path-shaped links like + `[[topics/checkout]]`) was rejected: it makes filing a decision that breaks + links, and it breaks the `^[a-z0-9-]+$` slug rule the page schema validates. +- Slug uniqueness is enforced by a check, not by the gate. A write that creates + a collision is accepted and then reported. Moving it into the gate would be + better — a collision would never land — and is worth doing when the gate can + afford the lookup it already does for wikilinks. diff --git a/packages/access/src/check/checks.ts b/packages/access/src/check/checks.ts new file mode 100644 index 0000000..b15852b --- /dev/null +++ b/packages/access/src/check/checks.ts @@ -0,0 +1,579 @@ +import { existsSync, readFileSync, readdirSync, statSync } from "node:fs"; +import { join } from "node:path"; +import { listPages, readIndex, isIndexed, CODEWIKI_DIR, type PageRef } from "../store/index.js"; +import { readFrontmatter, validatePage } from "../store/page.js"; +import { linkableSlugs, resolveWikilinks } from "../store/wikilinks.js"; +import { extractProvenanceLinks, resolveProvenance } from "../store/provenance.js"; +import { listSources } from "../sources/manifest.js"; +import { assertWithin } from "../paths.js"; +import type { Finding } from "./findings.js"; +import { safe, sortFindings } from "./findings.js"; + +/** + * The integrity checks (plan group 7). Each one answers a question the gate + * cannot: the gate sees a single write, and every check here is about a + * relationship *between* things — a link and its target, a page and the index, + * a citation and the file it points into, a word and the term it should be. + * + * They are pure reads. Nothing here repairs anything: a check that quietly + * fixed a page would be a write nobody reviewed, and the corrections are the + * agent's or the user's to make (7.6 shows them, 7.7 prints them). + */ + +/** A page read once, so ten checks do not read it ten times. */ +export interface LoadedPage extends PageRef { + text: string; + body: string; + frontmatter: Record | null; +} + +function loadPages(projectRoot: string): LoadedPage[] { + return listPages(projectRoot).map((ref) => { + const text = readFileSync(join(projectRoot, ref.path), "utf8"); + const block = readFrontmatter(text); + return { + ...ref, + text, + body: block?.body ?? text, + frontmatter: + block?.parsed && block.frontmatter && typeof block.frontmatter === "object" + ? (block.frontmatter as Record) + : null, + }; + }); +} + +/** + * The 1-based line of the *file* that `needle` first appears on in the page's + * body, or undefined. + * + * Searching the whole file found the slug or alias in `id:` or `title:` first + * and sent the reader into the frontmatter every time. Searching the body and + * adding the frontmatter's height gives the line the prose is actually on. The + * match is case-insensitive because the checks that produce these needles are. + */ +function lineInPage(page: LoadedPage, needle: string): number | undefined { + if (needle === "") return undefined; + const at = page.body.toLowerCase().indexOf(needle.toLowerCase()); + if (at < 0) return undefined; + const frontmatterLines = page.text.length - page.body.length; + const before = page.text.slice(0, frontmatterLines); + return before.split("\n").length - 1 + page.body.slice(0, at).split("\n").length; +} + +/** The 1-based line `needle` first appears on in `text`, or undefined. */ +function lineOf(text: string, needle: string): number | undefined { + const at = text.indexOf(needle); + if (at < 0) return undefined; + return text.slice(0, at).split("\n").length; +} + +/** + * 7.1 — broken wikilinks, orphan pages, and the rule that makes a slug mean one + * page. + */ +export function checkLinks(projectRoot: string, pages: LoadedPage[]): Finding[] { + const findings: Finding[] = []; + + // Built once for the whole run. `resolveWikilinks` walks `wiki/` when it is + // not handed a set, and calling it per page made the check quadratic — 3 + // seconds over 800 pages, where the pages are already in hand here. + const known = linkableSlugs(projectRoot); + + for (const page of pages) { + for (const issue of resolveWikilinks(projectRoot, page.body, page.slug, known)) { + findings.push({ + code: "wikilink.broken", + severity: "error", + page: page.path, + message: `${page.path}: ${safe(issue.reason)}`, + fix: "Write the page it names, or correct the link to a slug that exists. `ow graph` lists every page.", + // The target comes off the issue rather than being cut back out of the + // sentence, which produced garbage the moment the wording changed. + line: issue.target ? lineInPage(page, `[[${issue.target}`) : undefined, + }); + } + } + + const indexText = readIndex(projectRoot); + for (const page of pages) { + if (isIndexed(indexText, page.slug)) continue; + findings.push({ + code: "page.orphan", + severity: "error", + page: page.path, + message: `${page.path} is not reachable from wiki/index.md`, + fix: `Add a link to [[${page.slug}]] in wiki/index.md, under the section it belongs to.`, + }); + } + + // A slug names a page, so two files cannot share one: `[[checkout]]` would + // have no answer. Reported rather than resolved — picking one silently is how + // a link starts pointing at the wrong page. + const bySlug = new Map(); + for (const page of pages) { + const group = bySlug.get(page.slug); + if (group) group.push(page); + else bySlug.set(page.slug, [page]); + } + for (const [slug, group] of bySlug) { + if (group.length < 2) continue; + const where = group.map((p) => p.path).join(", "); + for (const page of group) { + findings.push({ + code: "page.duplicate-slug", + severity: "error", + page: page.path, + message: `the slug "${slug}" names ${group.length} pages (${where}), so [[${slug}]] is ambiguous`, + fix: "Rename all but one, and fix the links that pointed at it. A folder is organisation; the slug is the name.", + }); + } + } + + return findings; +} + +/** 7.2 — the changelog against the pages, and sources nothing cites. */ +export function checkRecords( + projectRoot: string, + pages: LoadedPage[], + citedSources: ReadonlySet, +): Finding[] { + const findings: Finding[] = []; + const changelogPath = join(projectRoot, "wiki", "changelog.md"); + const changelog = existsSync(changelogPath) ? readFileSync(changelogPath, "utf8") : ""; + const slugs = new Set(pages.map((p) => p.slug)); + + // Named in the changelog, gone from the wiki. + const named = new Set(); + for (const match of changelog.matchAll(/\[\[([^\]|#]+)/g)) { + const slug = match[1]?.trim(); + if (slug) named.add(slug); + } + for (const slug of named) { + if (slugs.has(slug)) continue; + findings.push({ + code: "changelog.missing-page", + // A warning, not an error, and the difference is not cosmetic: a page + // that was deliberately deleted leaves an entry naming it forever, and a + // changelog is a record — correcting it to point at nothing is not a + // thing the syntax can express. An error here would be permanently red + // with no action that clears it, which teaches a reader to ignore the + // whole report. + severity: "warning", + page: "wiki/changelog.md", + message: `wiki/changelog.md records [[${safe(slug)}]], which is not a page`, + fix: "If the page was renamed, name the new slug. If it was deleted, this entry is the record that it existed and the finding is expected — nothing to do.", + line: lineOf(changelog, `[[${slug}`), + }); + } + + // In the wiki, never recorded. A page that arrived without a changelog entry + // is a change nobody can find later. + for (const page of pages) { + if (named.has(page.slug)) continue; + findings.push({ + code: "changelog.unrecorded-page", + severity: "warning", + page: page.path, + message: `${page.path} is not mentioned anywhere in wiki/changelog.md`, + fix: `Add an entry under today's date recording what [[${page.slug}]] is for.`, + }); + } + + // A source in raw/ that no page rests on. This is the case that disappears + // from view on its own: nothing links to it, so nobody trips over it. + for (const id of listSources(projectRoot)) { + if (citedSources.has(id)) continue; + findings.push({ + code: "source.uncited", + severity: "warning", + source: id, + message: `raw/${id} is a source no page cites`, + fix: "Distil it into a page, or accept that it is not yet used — it stays in raw/ either way; sources are never deleted to tidy a report.", + }); + } + + return findings; +} + +/** 7.3 — provenance links that resolve to no source, or to no instant in one. */ +export function checkProvenance(projectRoot: string, pages: LoadedPage[]): Finding[] { + const findings: Finding[] = []; + for (const page of pages) { + for (const issue of resolveProvenance(projectRoot, linksOf(page))) { + findings.push({ + code: "provenance.unresolved", + severity: "error", + page: page.path, + message: `${page.path}: ${issue.reason}`, + fix: "Upload the source it names, or correct the citation. A citation that opens nothing is worse than none — if the source cannot be produced, the claim comes out with it.", + }); + } + } + return findings; +} + +/** The provenance links a page rests on: its frontmatter plus its prose. */ +function linksOf(page: LoadedPage): string[] { + const declared = Array.isArray(page.frontmatter?.["sources"]) + ? (page.frontmatter["sources"] as unknown[]).filter((s): s is string => typeof s === "string") + : []; + return [...new Set([...declared, ...extractProvenanceLinks(page.body)])]; +} + +/** + * Source id → the pages citing it. One read of the wiki answers both "which + * sources does nothing cite" (7.2) and "which pages cite this source" (6.4), + * and feeds every source's state (6.1). + */ +export function citedSourcePages(pages: LoadedPage[]): Map { + const citations = new Map(); + for (const page of pages) { + for (const link of linksOf(page)) { + const id = /^(?:src|rec):\/\/([^#]+)#/.exec(link)?.[1]; + if (id === undefined) continue; + const pagesForId = citations.get(id); + if (pagesForId) { + if (!pagesForId.includes(page.path)) pagesForId.push(page.path); + } else { + citations.set(id, [page.path]); + } + } + } + return citations; +} + +/** Every source id any page cites, for the uncited check of 7.2. */ +export function citedSourceIds(pages: LoadedPage[]): Set { + return new Set(citedSourcePages(pages).keys()); +} + +/** + * 7.4 — a synonym used where the project has a canonical term. + * + * The canonical terms are not a separate file: every page already declares its + * own in frontmatter — `title` is the term, `aliases` are the synonyms. Adding + * a `glossary.md` beside that would be a second record of one fact, and the + * copy is the one that goes stale. + * + * The page that *declares* an alias is exempt. That is where "Fenix, also known + * as the fenix platform" legitimately belongs, and flagging it would train a + * reader to ignore this check. + */ +export function checkVocabulary(pages: LoadedPage[]): Finding[] { + const findings: Finding[] = []; + + const titleOf = (page: LoadedPage): string | undefined => + typeof page.frontmatter?.["title"] === "string" ? page.frontmatter["title"] : undefined; + + // Every title, so an alias claiming a name another page already answers to is + // reported as the conflict it is rather than as noise on the victim. + const titles = new Map(); + for (const page of pages) { + const title = titleOf(page); + if (title) titles.set(title.toLowerCase(), page); + } + + // alias (lowercased) → the page that declares it + const canonical = new Map(); + for (const page of pages) { + const aliases = page.frontmatter?.["aliases"]; + const title = titleOf(page); + if (!Array.isArray(aliases) || title === undefined) continue; + for (const alias of aliases) { + if (typeof alias !== "string" || alias.trim() === "") continue; + const key = alias.toLowerCase(); + + // Two pages claiming one alias is the same ambiguity `page.duplicate-slug` + // exists for. Last-writer-wins silently picked one and then flagged the + // *declaring* page of the loser for writing its own alias. + const claimed = canonical.get(key); + if (claimed && claimed.path !== page.path) { + findings.push({ + code: "glossary.conflict", + severity: "error", + page: page.path, + message: `"${safe(alias)}" is an alias of both ${claimed.path} ("${safe(claimed.title)}") and ${page.path} ("${safe(title)}")`, + fix: "One name means one concept. Drop the alias from whichever page it does not belong to, or the term is genuinely two things and needs two names.", + }); + continue; + } + + // An alias that is some other page's title is the same conflict wearing a + // different hat, and left unreported it flagged that page every time it + // wrote its own name. + const owner = titles.get(key); + if (owner && owner.path !== page.path) { + findings.push({ + code: "glossary.conflict", + severity: "error", + page: page.path, + message: `${page.path} lists "${safe(alias)}" as an alias, but that is the title of ${owner.path}`, + fix: "Remove the alias, or rename the other page. A word cannot be both this page's synonym and that page's name.", + }); + continue; + } + + canonical.set(key, { slug: page.slug, title, path: page.path }); + } + } + if (canonical.size === 0) return findings; + + for (const page of pages) { + // Code is literal, and a wikilink to an alias is a legitimate way to reach + // the page — neither is prose using the wrong word. + const prose = page.body + .replace(/```[\s\S]*?```/g, "") + .replace(/`[^`\n]*`/g, "") + .replace(/\[\[[^\]]*\]\]/g, ""); + + for (const [alias, owner] of canonical) { + if (owner.path === page.path) continue; // its own definition + const pattern = new RegExp(`(?(); + + const linesIn = (target: string): number | null => { + const cached = lineCounts.get(target); + if (cached !== undefined) return cached; + let count: number | null = null; + try { + const file = assertWithin(projectRoot, join(projectRoot, target)); + const stat = existsSync(file) ? statSync(file) : null; + if (stat?.isFile() && stat.size <= MAX_CITED_FILE_BYTES) { + const lines = readFileSync(file, "utf8").split(/\r?\n/); + // A file ending in a newline splits into a trailing empty element that + // is not a line. Counting it accepts a citation one line past the end — + // exactly what this check exists to catch, on a file shaped the way + // almost every text file is. + if (lines[lines.length - 1] === "") lines.pop(); + count = lines.length; + } + } catch { + count = null; // outside the project: it cites nothing this can see + } + lineCounts.set(target, count); + return count; + }; + + for (const page of pages.filter((p) => p.codewiki)) { + for (const match of page.body.matchAll(CODEWIKI_CITATION)) { + const [whole, target, startText, endText] = match; + const start = Number(startText); + const end = endText === undefined ? start : Number(endText); + // A "path" this long is not one. The regex is linear now, but a bound on + // what reaches the filesystem costs nothing and keeps a pathological + // target from becoming a pathological stat. + const total = target!.length > MAX_CITATION_TARGET ? null : linesIn(target!); + + if (total === null) { + findings.push({ + code: "codewiki.citation-unresolved", + severity: "error", + page: page.path, + message: `${page.path}: ${safe(whole!)} points at ${safe(target!)}, which is not a file in this project`, + fix: "The file moved or was deleted. Point the citation at where the code lives now, or remove the section — prose that cites nothing has drifted free of the code it describes.", + line: lineInPage(page, whole!), + }); + continue; + } + if (end > total) { + findings.push({ + code: "codewiki.citation-past-end", + severity: "error", + page: page.path, + message: `${page.path}: ${safe(whole!)} runs past the end of ${safe(target!)}, which has ${total} lines`, + fix: "The file shrank under the citation. Re-read it and cite the lines the section is actually about.", + line: lineInPage(page, whole!), + }); + } + } + + // Every section cites something. A section that cites nothing is prose that + // has drifted free of the code it describes. + // Lines of the body, reported as lines of the file: every other finding is + // file-relative, and a reader following a body-relative one lands in the + // frontmatter. + const frontmatterLines = + page.text.slice(0, page.text.length - page.body.length).split("\n").length - 1; + const lines = page.body.split("\n"); + let heading: { title: string; line: number } | null = null; + let cited = false; + let inFence = false; + + const hasCitation = (line: string): boolean => { + CODEWIKI_CITATION.lastIndex = 0; + return CODEWIKI_CITATION.test(line); + }; + + const closeSection = (): void => { + if (heading && !cited) { + findings.push({ + code: "codewiki.section-uncited", + severity: "warning", + page: page.path, + message: `${page.path}: the section "${safe(heading.title)}" cites no lines`, + fix: "Add the citation the section is about, as [path/to/file.ts:12-40](). Do not narrate what a reader can see — if there is nothing to point at, the section does not belong here.", + line: heading.line, + }); + } + }; + + lines.forEach((line, i) => { + if (/^\s*```/.test(line)) inFence = !inFence; + if (inFence) return; + const level = /^(#{1,6})\s+\S/.exec(line); + if (level) { + closeSection(); + // A single `#` is the page's title, not a section of it — the skill's + // example shows `##` sections under one. Treating the title as a + // section fired on every conventionally written page. + if (level[1]!.length === 1) { + heading = null; + return; + } + heading = { + title: line.replace(/^#+\s+/, "").trim(), + line: frontmatterLines + i + 1, + }; + // A citation written on the heading line itself counts. Returning + // before looking produced "the section [code.ts:1-2]() cites no lines", + // quoting the citation it had just refused to see. + cited = hasCitation(line); + return; + } + if (hasCitation(line)) cited = true; + }); + closeSection(); + } + + // A `codewiki/` at the project root looks right and is not: codewiki lives + // under `wiki/`, and nothing outside `wiki/` is part of the wiki at all. + const stray = join(projectRoot, CODEWIKI_DIR); + if (existsSync(stray) && statSync(stray).isDirectory()) { + // Recursive, like the model three functions above: a stray + // `codewiki/area/x.md` is exactly as misplaced as `codewiki/x.md`. + const strayPages = readdirSync(stray, { recursive: true, withFileTypes: true }).filter( + (entry) => entry.isFile() && entry.name.endsWith(".md"), + ); + if (strayPages.length > 0) { + findings.push({ + code: "codewiki.misplaced", + severity: "error", + message: `${CODEWIKI_DIR}/ at the project root holds ${strayPages.length} page(s), but codewiki lives at wiki/${CODEWIKI_DIR}/`, + fix: `Move them to wiki/${CODEWIKI_DIR}/. Outside wiki/ they are not part of the wiki: nothing indexes them, nothing links them, and no write to them is validated.`, + }); + } + } + + return findings; +} + +/** The page schema, re-checked outside the gate — group 5's rules, after the fact. */ +export function checkSchema(pages: LoadedPage[]): Finding[] { + const findings: Finding[] = []; + for (const page of pages) { + const result = validatePage(page.text, page.slug); + if (result.ok) continue; + for (const issue of result.errors) { + findings.push({ + code: "page.invalid", + severity: "error", + page: page.path, + message: `${page.path}: ${issue.field ? `${issue.field}: ` : ""}${issue.reason}`, + fix: "Correct the frontmatter. `ow write` applies the same rules, and the gate would have refused this write — it arrived some other way.", + }); + } + } + return findings; +} + +export interface CheckReport { + findings: Finding[]; + /** How many pages and sources were looked at, so an empty report means something. */ + pages: number; + sources: number; +} + +/** Read every page once. Exported so a caller running several things over the + * wiki — the checks, the source states, a UI refresh — pays for one walk. */ +export function readWiki(projectRoot: string): LoadedPage[] { + return loadPages(projectRoot); +} + +/** + * Run every check over a project. One read of the wiki feeds all of them, so + * this is a single pass over the pages rather than one per check. + */ +export function checkProject(projectRoot: string): CheckReport { + const pages = loadPages(projectRoot); + const cited = citedSourceIds(pages); + + const findings = [ + ...checkSchema(pages), + ...checkLinks(projectRoot, pages), + ...checkRecords(projectRoot, pages, cited), + ...checkProvenance(projectRoot, pages), + ...checkVocabulary(pages), + ...checkCodewiki(projectRoot, pages), + ]; + + return { + findings: sortFindings(findings), + pages: pages.length, + sources: listSources(projectRoot).length, + }; +} diff --git a/packages/access/src/check/findings.ts b/packages/access/src/check/findings.ts new file mode 100644 index 0000000..1f26d18 --- /dev/null +++ b/packages/access/src/check/findings.ts @@ -0,0 +1,102 @@ +/** + * What a check reports (plan group 7). With the agent writing pages through the + * filesystem, this stops being hygiene and becomes the net of record: group 5 + * refuses a malformed write, but a page edited in another editor, written + * through the shell, or made wrong by a *later* change to something else never + * passes a gate at all. + * + * Every finding carries a **correction path**, not just a complaint. 7.6 shows + * these in the UI and 7.7 prints them for an agent, and both need to say what + * to do — a refusal a reader cannot act on becomes an attempt they repeat + * verbatim (the same reason 9.13 exists). + */ + +/** + * The kinds of finding. Stable strings: the UI groups by them, `ow check + * --json` prints them, and a CI job greps them, so they are an interface. + */ +export const FINDING_CODES = [ + // 7.1 — reachability and links + "wikilink.broken", + "page.orphan", + "page.duplicate-slug", + // 7.2 — the records, and sources nothing uses + "changelog.missing-page", + "changelog.unrecorded-page", + "source.uncited", + // 7.3 — provenance + "provenance.unresolved", + // 7.4 — vocabulary + "glossary.synonym", + "glossary.conflict", + // 7.5 — codewiki + "codewiki.citation-unresolved", + "codewiki.citation-past-end", + "codewiki.section-uncited", + "codewiki.misplaced", + // the page schema itself, re-checked outside the gate + "page.invalid", +] as const; + +export type FindingCode = (typeof FINDING_CODES)[number]; + +/** + * `error` is something wrong: a link that goes nowhere, a citation that does + * not resolve. `warning` is something that is probably wrong but that a project + * may legitimately choose — a source nobody has cited yet is the common one, + * on the day it was uploaded. + */ +export type Severity = "error" | "warning"; + +export interface Finding { + code: FindingCode; + severity: Severity; + /** What is wrong, in one sentence, naming the thing. */ + message: string; + /** What to do about it. Required: a finding nobody can act on is noise. */ + fix: string; + /** The page this is about, as a project-relative path. */ + page?: string; + /** The source id this is about. */ + source?: string; + /** The 1-based line of `page` the finding sits on, where there is one. */ + line?: number; +} + +/** + * Make a string from a page safe to put in a message. + * + * Every interpolated value below reaches a finding from somewhere an agent + * writes: a frontmatter alias, a heading, a link target, a filename. A + * double-quoted YAML scalar carries `\n`, `\r` and `\u001b` intact, and + * `ow check` writes its report straight to a terminal — so an alias holding a + * newline plus its own "ow check: no findings" line, or a `\r` and a cursor + * escape, can forge or erase the summary a human reads to decide whether the + * wiki is sound. The report is the artifact that decision rests on. + * + * Control characters go, and the text is bounded: a 10 KB alias should not + * flood the report either. + */ +export const MAX_QUOTED = 200; + +export function safe(text: string): string { + const stripped = text.replace(/\p{Cc}/gu, " ").trim(); + return stripped.length > MAX_QUOTED ? `${stripped.slice(0, MAX_QUOTED)}…` : stripped; +} + +/** Order findings the way a reader wants them: errors first, then by location. */ +export function sortFindings(findings: readonly Finding[]): Finding[] { + const rank = (f: Finding): number => (f.severity === "error" ? 0 : 1); + return [...findings].sort( + (a, b) => + rank(a) - rank(b) || + (a.page ?? a.source ?? "").localeCompare(b.page ?? b.source ?? "") || + (a.line ?? 0) - (b.line ?? 0) || + a.code.localeCompare(b.code), + ); +} + +/** True when anything found is an error rather than a warning. */ +export function hasErrors(findings: readonly Finding[]): boolean { + return findings.some((f) => f.severity === "error"); +} diff --git a/packages/access/src/gate/gate.ts b/packages/access/src/gate/gate.ts index 7acabbd..4fc8446 100644 --- a/packages/access/src/gate/gate.ts +++ b/packages/access/src/gate/gate.ts @@ -12,10 +12,12 @@ import { readFrontmatter, validatePage, NON_ENTITY_PAGES } from "../store/page.j * and either let it through unchanged, accept it completed, or refuse it with * reasons the writer can act on. * - * The gate owns the wiki and codewiki entity pages — the pages that carry the - * schema of 5.1. Everything else (sources, the non-entity `index.md`/`changelog` - * /`log.md`, files outside `wiki/` and `codewiki/`) is passed through; it is not - * the gate's to validate. The gate's own configuration (`.claude/`, `.mcp.json`, + * The gate owns the entity pages under `wiki/` — at any depth, codewiki + * included, since codewiki lives at `wiki/codewiki/` + * (`adr:0016-a-page-is-its-slug-wherever-it-sits`). Everything else (sources, + * the non-entity `index.md`/`changelog.md`/`log.md` at the top of `wiki/`, and + * anything outside `wiki/`) is passed through; it is not the gate's to + * validate. The gate's own configuration (`.claude/`, `.mcp.json`, * `CLAUDE.md`) is refused outright (9.6): a write path that reached it would * edit away its own restraint through a change that reads as documentation. * @@ -51,9 +53,23 @@ function gatedPageRel(projectRoot: string, filePath: string): string | null { if (rel === "" || rel.startsWith("..")) return null; const folded = rel.toLowerCase(); if (!folded.endsWith(".md")) return null; - if (!folded.startsWith("wiki/") && !folded.startsWith("codewiki/")) return null; - // Non-entity pages are themselves, not validated against the schema. - if ((NON_ENTITY_PAGES as readonly string[]).includes(basename(folded))) return null; + // `wiki/` and everything under it, codewiki included. A top-level + // `codewiki/` was gated too, which is the half of plan 7.5's design gap that + // pointed the wrong way: the scaffolded skill and the plan's layout both put + // codewiki at `wiki/codewiki/`, and a page outside `wiki/` is not part of the + // wiki at all — nothing indexes it, nothing links it, nothing can cite it. + // Gating it implied otherwise. `ow check` reports a stray top-level + // `codewiki/` as misplaced instead of pretending it is a wiki. + if (!folded.startsWith("wiki/")) return null; + // Non-entity pages are themselves, not validated against the schema. Only + // the three at the top of `wiki/`: a `wiki/topics/index.md` is a page called + // "index", not the wiki's index. + if ( + !folded.slice("wiki/".length).includes("/") && + (NON_ENTITY_PAGES as readonly string[]).includes(basename(folded)) + ) { + return null; + } return rel; } @@ -126,4 +142,4 @@ export function gateWrite(input: GateInput): GateDecision { if (reasons.length > 0) return { action: "deny", reasons }; return { action: "accept", content: completed }; -} \ No newline at end of file +} diff --git a/packages/access/src/index.ts b/packages/access/src/index.ts index e4b86b4..20abed9 100644 --- a/packages/access/src/index.ts +++ b/packages/access/src/index.ts @@ -81,6 +81,12 @@ export { } from "./sources/manifest.js"; export { registerSource, type RegisterInput } from "./sources/register.js"; export { deriveId, isIdTaken, EmptyNameError } from "./sources/id.js"; +export { + sourceState, + listSourceStates, + type SourceState, + type SourceStage, +} from "./sources/state.js"; export { uploadTextSource, writeSourceText, normaliseText } from "./sources/ingest.js"; export { uploadPdfSource, @@ -110,5 +116,39 @@ export { export { resolveProvenance, extractProvenanceLinks } from "./store/provenance.js"; export { completeFrontmatter } from "./store/complete.js"; export { recordWrite, type WriteEntry, type WriteAction } from "./store/record.js"; -export { listEntityPages, isIndexed, findOrphans, readIndex } from "./store/index.js"; +export { + listEntityPages, + listPages, + pagePath, + isIndexed, + findOrphans, + readIndex, + CODEWIKI_DIR, + type PageRef, +} from "./store/index.js"; export { registerInIndex } from "./store/index-write.js"; + +// The integrity checks (group 7) +export { + checkProject, + checkLinks, + checkRecords, + checkProvenance, + checkVocabulary, + checkCodewiki, + checkSchema, + citedSourceIds, + citedSourcePages, + readWiki, + type CheckReport, + type LoadedPage, +} from "./check/checks.js"; +export { + FINDING_CODES, + sortFindings, + hasErrors, + safe, + type Finding, + type FindingCode, + type Severity, +} from "./check/findings.js"; diff --git a/packages/access/src/read.ts b/packages/access/src/read.ts index 5e8ce5f..dc05f1a 100644 --- a/packages/access/src/read.ts +++ b/packages/access/src/read.ts @@ -11,7 +11,39 @@ * settings reads `ow.json`. Neither writes a page, so both belong here. */ export { resolveReal, isWithin, assertWithin, OutsideProjectError } from "./paths.js"; -export { listEntityPages, isIndexed, findOrphans, readIndex } from "./store/index.js"; +export { + listEntityPages, + listPages, + pagePath, + isIndexed, + findOrphans, + readIndex, + CODEWIKI_DIR, + type PageRef, +} from "./store/index.js"; +// The checks read only; the MCP process may run them. +export { + checkProject, + readWiki, + citedSourcePages, + type CheckReport, + type LoadedPage, +} from "./check/checks.js"; +export { + sourceState, + listSourceStates, + type SourceState, + type SourceStage, +} from "./sources/state.js"; +export { + FINDING_CODES, + sortFindings, + hasErrors, + safe, + type Finding, + type FindingCode, + type Severity, +} from "./check/findings.js"; export { readFrontmatter, validatePage, @@ -48,4 +80,4 @@ export { UnknownNameError, MovedProjectError, InvalidNameError, -} from "./registry.js"; \ No newline at end of file +} from "./registry.js"; diff --git a/packages/access/src/skills.ts b/packages/access/src/skills.ts index 2abaced..459965b 100644 --- a/packages/access/src/skills.ts +++ b/packages/access/src/skills.ts @@ -11,7 +11,7 @@ import { join } from "node:path"; * The `open-wiki-version` frontmatter marker lets `ow init` report staleness * instead of overwriting — the open question in that record. */ -export const SKILLS_VERSION = "0.1.0"; +export const SKILLS_VERSION = "0.2.0"; const WIKI_SKILL = `--- name: wiki @@ -45,12 +45,32 @@ Every entity page is markdown with YAML frontmatter: - **id** is \`type:slug\` — \`project:fenix\`, \`person:ana\`, \`topic:checkout\`, or a codewiki area. The filename is the slug. - **type** is \`project\`, \`person\`, \`topic\` or a codewiki page. +- **aliases** are the other names for this concept. They are what stops three + names for one thing appearing within a week: \`ow check\` reports any page + writing an alias where this page's **title** belongs. Put a synonym here once + and it is handled everywhere. - **status** is \`active\` unless the page is superseded, when it is \`superseded\` and \`superseded-by\` names the replacement. - **sources** lists the provenance links this page rests on. \`index.md\`, \`changelog.md\` and \`log.md\` are not entity pages; they are -themselves and are not validated against this schema. +themselves and are not validated against this schema. That is true of the three +at the top of \`wiki/\` only — a \`wiki/topics/index.md\` is an ordinary page +called "index". + +## Where a page lives + +**A page is its slug, wherever it sits under \`wiki/\`.** Write +\`wiki/checkout.md\` or file it as \`wiki/topics/checkout.md\`; either way it is +\`[[checkout]]\`, and moving it between folders breaks nothing. A folder is +organisation; a link is a name — \`adr:0016-a-page-is-its-slug-wherever-it-sits\`. + +The one rule this needs: **a slug names exactly one page.** Two files called +\`checkout.md\` in different folders make \`[[checkout]]\` ambiguous, and +\`ow check\` reports it rather than guessing. + +Codewiki pages go under \`wiki/codewiki/\`. Nothing outside \`wiki/\` is part of +the wiki at all. ## A claim @@ -81,7 +101,10 @@ reader sees. Both, or it is not supersession. skim looks authoritative and is not. 2. One page per concept. A source covering three concepts becomes three pages or three edits, not a mirror of the source. -3. Check the glossary; use the canonical term. +3. Use the project's own term. Every page's \`title\` is the canonical name for + its concept and its \`aliases\` are the names to avoid — there is no separate + glossary to consult, and adding one would be a second record of the same + fact. 4. Write the page at \`wiki/.md\`, link it from \`index.md\`, record it in \`changelog.md\`. 5. Run \`ow check\`. @@ -93,8 +116,8 @@ description: Narrate an area of this project's code in wiki/codewiki/, where eve open-wiki-version: ${SKILLS_VERSION} --- -\`codewiki/\` is prose that explains code, one page per area, with every section -citing the exact lines it is about: +\`wiki/codewiki/\` is prose that explains code, one page per area, with every +section citing the exact lines it is about: ## How the dispatcher routes diff --git a/packages/access/src/sources/state.ts b/packages/access/src/sources/state.ts new file mode 100644 index 0000000..12ef93a --- /dev/null +++ b/packages/access/src/sources/state.ts @@ -0,0 +1,123 @@ +import { existsSync, readFileSync, statSync } from "node:fs"; +import { join } from "node:path"; +import { assertWithin } from "../paths.js"; +import { listSources, readManifest, type SourceKind } from "./manifest.js"; + +/** + * Each source's state (plan 6.1): received, text ready, cited on a page. + * + * **Derived from disk, not persisted beside it.** "Persisted and resumable" is + * what the plan asks for, and the filesystem already is both: `manifest.json` + * says the source was received, `text.md` says the text is ready, the pages say + * what is cited, and `journal.json` says how far a transcription got. A state + * file next to those would be a second record of one fact — and the copy is the + * one that goes stale, which is the rule the plan applies to its own checklists + * (`routing.md`) and to the wiki's index. + * + * So a crash loses nothing and there is nothing to reconcile: the next read + * observes the same directory and reaches the same answer. + */ + +export type SourceStage = + /** The directory and manifest exist; nothing has been extracted yet. */ + | "received" + /** Transcription is part-done — a journal is present and unfinished (4.9). */ + | "transcribing" + /** `text.md` is written and the source is readable. */ + | "text-ready" + /** At least one page cites it. */ + | "cited" + /** It stopped, and the reason is worth showing rather than retrying blindly. */ + | "failed"; + +export interface SourceState { + id: string; + title: string; + kind: SourceKind; + stage: SourceStage; + /** True once `text.md` exists, whatever the stage says. */ + textReady: boolean; + /** The pages citing this source, as project-relative paths. */ + citedBy: string[]; + /** Why it stopped, when it did. */ + error?: string; + /** Chunks done / total, while a transcription is in flight (4.9, 6.3). */ + progress?: { done: number; total: number }; +} + +/** The transcription journal group 4 writes; read here, never written here. */ +interface Journal { + chunks?: Array<{ index?: number; done?: boolean; error?: string }>; + error?: string; +} + +function readJournal(projectRoot: string, id: string): Journal | null { + try { + const file = assertWithin(projectRoot, join(projectRoot, "raw", id, "journal.json")); + if (!existsSync(file) || !statSync(file).isFile()) return null; + return JSON.parse(readFileSync(file, "utf8")) as Journal; + } catch { + // A journal that will not parse is not a reason to hide the source; the + // stage falls back to what the rest of the directory says. + return null; + } +} + +/** + * The state of one source. `citedBy` is supplied by the caller because working + * it out means reading every page, and a listing of twenty sources must not do + * that twenty times. + */ +export function sourceState( + projectRoot: string, + id: string, + citedBy: readonly string[] = [], +): SourceState { + const manifest = readManifest(projectRoot, id); + const dir = assertWithin(projectRoot, join(projectRoot, "raw", id)); + const textReady = existsSync(join(dir, "text.md")); + + const journal = readJournal(projectRoot, id); + const chunks = journal?.chunks ?? []; + const done = chunks.filter((c) => c.done).length; + const failed = chunks.find((c) => c.error)?.error ?? journal?.error; + + const base = { + id, + title: manifest.title, + kind: manifest.kind, + textReady, + citedBy: [...citedBy], + }; + + if (failed !== undefined && !textReady) { + return { ...base, stage: "failed", error: failed, ...progressOf(chunks.length, done) }; + } + // `textReady` gates "cited" on purpose: a page citing a source whose text + // never landed is citing something nothing could have read, and reporting + // that as the last stage of the pipeline would hide it. The citation is still + // recorded in `citedBy`, so the caller can say both things. + if (textReady) return { ...base, stage: citedBy.length > 0 ? "cited" : "text-ready" }; + if (chunks.length > 0) { + return { ...base, stage: "transcribing", ...progressOf(chunks.length, done) }; + } + return { ...base, stage: "received" }; +} + +function progressOf(total: number, done: number): { progress?: { done: number; total: number } } { + return total > 0 ? { progress: { done, total } } : {}; +} + +/** + * Every source's state in one pass. `citations` maps a source id to the pages + * citing it — `citedSourcePages` in the check module builds it from one read of + * the wiki. + */ +export function listSourceStates( + projectRoot: string, + citations: ReadonlyMap = new Map(), +): SourceState[] { + return listSources(projectRoot) + .map((id) => sourceState(projectRoot, id, citations.get(id) ?? [])) + .sort((a, b) => a.id.localeCompare(b.id)); +} diff --git a/packages/access/src/store/index-write.ts b/packages/access/src/store/index-write.ts index 1b99fb8..db7cce4 100644 --- a/packages/access/src/store/index-write.ts +++ b/packages/access/src/store/index-write.ts @@ -1,4 +1,4 @@ -import { writeFileSync } from "node:fs"; +import { mkdirSync, writeFileSync } from "node:fs"; import { assertWithin } from "../paths.js"; import { readIndex, isIndexed, PAGES_SECTION } from "./index.js"; @@ -13,6 +13,9 @@ import { readIndex, isIndexed, PAGES_SECTION } from "./index.js"; export function registerInIndex(projectRoot: string, slug: string, title?: string): boolean { const file = assertWithin(projectRoot, `${projectRoot}/wiki/index.md`); const text = readIndex(projectRoot); + // `readIndex` creates nothing — it is a read. Making the directory is this + // side's job, because this is the side that writes. + mkdirSync(`${projectRoot}/wiki`, { recursive: true }); if (isIndexed(text, slug)) return false; const bullet = title ? `- [[${slug}]] — ${title}` : `- [[${slug}]]`; @@ -31,4 +34,4 @@ export function registerInIndex(projectRoot: string, slug: string, title?: strin } writeFileSync(file, next.endsWith("\n") ? next : `${next}\n`, "utf8"); return true; -} \ No newline at end of file +} diff --git a/packages/access/src/store/index.ts b/packages/access/src/store/index.ts index e9268e9..5b00c4d 100644 --- a/packages/access/src/store/index.ts +++ b/packages/access/src/store/index.ts @@ -1,5 +1,5 @@ -import { existsSync, readFileSync, readdirSync, mkdirSync } from "node:fs"; -import { join } from "node:path"; +import { existsSync, readFileSync, readdirSync } from "node:fs"; +import { join, posix, relative, sep } from "node:path"; import { assertWithin } from "../paths.js"; import { NON_ENTITY_PAGES } from "./page.js"; @@ -9,6 +9,24 @@ import { NON_ENTITY_PAGES } from "./page.js"; * topic; the store only guarantees reachability, and flags a page that has * become unreachable — one whose slug no link in the index points at. * + * **A page is addressed by its slug, wherever it sits under `wiki/`.** Settling + * that is plan task 7.5's job, and the gap turned out to be wider than the note + * there described — it was never only about codewiki: + * + * - the plan's directory diagram puts pages in `wiki/projects/`, + * `wiki/people/`, `wiki/topics/` and `wiki/codewiki/`; + * - the scaffolded `wiki` skill tells the agent to write `wiki/.md`, flat; + * - and this module read only the *top level* of `wiki/`, so every page written + * the way the diagram describes was invisible to the index, the orphan check, + * `ow graph` and the MCP read tools. + * + * Walking the tree and keying on the basename honours both readings: a folder + * is organisation, a link is a name — the model Obsidian uses, which is where + * `[[wikilink]]` comes from in the first place. The one rule it needs is that a + * slug is unique across the wiki, because `[[checkout]]` cannot mean two files. + * That is a group 7 finding (`page.duplicate-slug`), not something to resolve + * by silently picking one. + * * The write that registers a page in the index lives in `index-write.ts`, so the * read surface the MCP process imports (plan 9.9) pulls no write code. */ @@ -17,28 +35,92 @@ const INDEX_HEADER = "# Index\n\nEvery page is reachable from here; a page that is not is a page nobody will find again.\n\n"; const PAGES_SECTION = "## Pages"; -/** The slugs of every entity page at the top of `wiki/`. */ -export function listEntityPages(projectRoot: string): string[] { +/** Where codewiki pages live, relative to `wiki/` (plan 7.5). */ +export const CODEWIKI_DIR = "codewiki"; + +/** One entity page: the name it is linked by, and where it actually sits. */ +export interface PageRef { + /** The filename without `.md`. What a `[[wikilink]]` names. */ + slug: string; + /** Project-relative posix path, e.g. `wiki/codewiki/dispatch.md`. */ + path: string; + /** True when the page sits under `wiki/codewiki/`. */ + codewiki: boolean; +} + +/** + * Every entity page under `wiki/`, at any depth. `index.md`, `changelog.md` and + * `log.md` are excluded at the top level — they are the wiki's own pages, not + * entities. + * + * Symlinked directories are not followed. A link inside `wiki/` pointing + * elsewhere on disk would otherwise enumerate that elsewhere as project + * content, which is the escape `paths.ts` exists to stop. + */ +export function listPages(projectRoot: string): PageRef[] { const wiki = join(projectRoot, "wiki"); if (!existsSync(wiki)) return []; - const slugs: string[] = []; - for (const entry of readdirSync(wiki)) { - if (!entry.endsWith(".md")) continue; - if ((NON_ENTITY_PAGES as readonly string[]).includes(entry)) continue; - slugs.push(entry.slice(0, -3)); - } - return slugs; + + const pages: PageRef[] = []; + const walk = (dir: string): void => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.isSymbolicLink()) continue; + const full = join(dir, entry.name); + if (entry.isDirectory()) { + walk(full); + continue; + } + if (!entry.isFile() || !entry.name.endsWith(".md")) continue; + + const rel = relative(wiki, full).split(sep).join(posix.sep); + // Only the three at the very top are the wiki's own; a file called + // `index.md` inside `wiki/topics/` is an ordinary page named "index". + if (!rel.includes(posix.sep) && (NON_ENTITY_PAGES as readonly string[]).includes(rel)) { + continue; + } + pages.push({ + slug: entry.name.slice(0, -3), + path: `wiki/${rel}`, + codewiki: rel.startsWith(`${CODEWIKI_DIR}/`), + }); + } + }; + walk(wiki); + return pages.sort((a, b) => a.path.localeCompare(b.path)); +} + +/** The slugs of every entity page under `wiki/`, at any depth. */ +export function listEntityPages(projectRoot: string): string[] { + return listPages(projectRoot).map((page) => page.slug); +} + +/** The project-relative path of the page with this slug, or undefined. */ +export function pagePath(projectRoot: string, slug: string): string | undefined { + return listPages(projectRoot).find((page) => page.slug === slug)?.path; +} + +/** A slug reaches this from a filename, so it may carry regex metacharacters. */ +function escapeForRegExp(text: string): string { + return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); } /** True when `indexText` links to `slug` (as `[[slug]]`, `[[slug|...]]` or `[[slug#...]]`). */ export function isIndexed(indexText: string, slug: string): boolean { - return new RegExp(`\\[\\[${slug}(\\||#|\\]\\])`).test(indexText); + return new RegExp(`\\[\\[${escapeForRegExp(slug)}(\\||#|\\]\\])`).test(indexText); } -/** Read the index, creating an empty one (header only) when absent. */ +/** + * Read the index, returning an empty one (header only) when absent. + * + * **This creates nothing.** It used to `mkdir` `wiki/`, which made every + * caller a writer: `ow check` run in the wrong directory left a `wiki/` behind, + * and `checkProject` — which the read-only surface the MCP process imports now + * exports — quietly wrote to disk, against the guarantee of 9.9 that read-only + * is what that process *can* do. The directory is the write path's to create; + * `registerInIndex` does it there. + */ export function readIndex(projectRoot: string): string { const file = assertWithin(projectRoot, join(projectRoot, "wiki", "index.md")); - mkdirSync(join(projectRoot, "wiki"), { recursive: true }); return existsSync(file) ? readFileSync(file, "utf8") : INDEX_HEADER; } diff --git a/packages/access/src/store/page.ts b/packages/access/src/store/page.ts index 635952e..124db3f 100644 --- a/packages/access/src/store/page.ts +++ b/packages/access/src/store/page.ts @@ -57,6 +57,12 @@ const REQUIRED_KEYS = [ export interface PageIssue { field: string; reason: string; + /** + * The thing the issue is about — a wikilink's target, say. Carried so a + * caller can locate it without parsing it back out of `reason`, which + * silently produces garbage the first time the wording changes. + */ + target?: string; } export type PageValidation = diff --git a/packages/access/src/store/wikilinks.ts b/packages/access/src/store/wikilinks.ts index f35b26f..a972639 100644 --- a/packages/access/src/store/wikilinks.ts +++ b/packages/access/src/store/wikilinks.ts @@ -1,5 +1,5 @@ -import { existsSync } from "node:fs"; -import { join } from "node:path"; +import { listEntityPages } from "./index.js"; +import { NON_ENTITY_PAGES } from "./page.js"; import type { PageIssue } from "./page.js"; /** @@ -30,28 +30,67 @@ function targetOf(inner: string): string { /** * Scan `body` for wikilinks and return one issue per link that does not resolve - * to a page in `wiki/` (or to the page being written, `currentSlug`). Empty + * to a page under `wiki/` (or to the page being written, `currentSlug`). Empty * means every link resolves. + * + * A link names a **slug**, and the page carrying that slug may sit anywhere + * under `wiki/` — `wiki/checkout.md` or `wiki/topics/checkout.md` are the same + * page as far as `[[checkout]]` is concerned. Checking for `wiki/.md` + * directly, as this did, made every link to a page filed under a type + * subdirectory read as broken. See `store/index.ts` for why the model is this. */ export function resolveWikilinks( projectRoot: string, body: string, currentSlug: string, + known?: ReadonlySet, ): PageIssue[] { - const issues: PageIssue[] = []; + const targets = collectTargets(body, currentSlug); + if (targets.length === 0) return []; + + // One directory walk for the whole page rather than one `existsSync` per + // link. This runs on the gate's hot path, once per write — and a caller with + // many pages to check (`ow check`) passes the set in, so the walk happens + // once for the run rather than once per page. + const resolvable = known ?? linkableSlugs(projectRoot); + return targets + .filter((target) => !resolvable.has(target)) + .map((target) => ({ + field: "wikilink", + target, + reason: `[[${target}]] does not resolve to a page (no page with the slug "${target}" under wiki/)`, + })); +} + +/** + * Every slug a `[[link]]` may name: the entity pages, plus `index`, + * `changelog` and `log`. + * + * Those three are not entity pages — they carry no frontmatter and are not + * validated against the schema — but they are files that exist in every + * scaffolded project, and the skill tells the agent to record things in the + * changelog and to link from the index. Leaving them out of the link set denied + * any page mentioning `[[changelog]]`, with a reason that reads as a bug: "no + * page with the slug changelog under wiki/", when there is a file called + * exactly that. + */ +export function linkableSlugs(projectRoot: string): Set { + const slugs = new Set(listEntityPages(projectRoot)); + for (const name of NON_ENTITY_PAGES) slugs.add(name.replace(/\.md$/, "")); + return slugs; +} + +/** The distinct link targets in a body, minus the page's own slug. */ +function collectTargets(body: string, currentSlug: string): string[] { const seen = new Set(); + const targets: string[] = []; for (const match of withoutCode(body).matchAll(WIKILINK)) { const target = targetOf(match[1] ?? ""); if (target === "") continue; if (target === currentSlug) continue; if (seen.has(target)) continue; // report each broken link once seen.add(target); - if (!existsSync(join(projectRoot, "wiki", `${target}.md`))) { - issues.push({ - field: "wikilink", - reason: `[[${target}]] does not resolve to a page (no wiki/${target}.md)`, - }); - } + targets.push(target); } - return issues; + return targets; } diff --git a/packages/access/tests/check.spec.ts b/packages/access/tests/check.spec.ts new file mode 100644 index 0000000..64b45e8 --- /dev/null +++ b/packages/access/tests/check.spec.ts @@ -0,0 +1,630 @@ +import { describe, expect, it, beforeEach, afterEach } from "vitest"; +import { mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { checkProject, readWiki, citedSourcePages } from "../src/check/checks.js"; +import { hasErrors, sortFindings, type Finding } from "../src/check/findings.js"; + +function tempProject(): string { + const root = realpathSync(mkdtempSync(join(tmpdir(), "ow-check-"))); + mkdirSync(join(root, "wiki"), { recursive: true }); + mkdirSync(join(root, "raw"), { recursive: true }); + return root; +} + +/** Write a page with valid frontmatter, at any depth under `wiki/`. */ +function page( + root: string, + relPath: string, + body = "", + frontmatter: Record = {}, +): void { + const slug = relPath.split("/").pop()!.replace(/\.md$/, ""); + const fm = { + id: `topic:${slug}`, + type: "topic", + title: slug, + status: "active", + aliases: [], + updated: "2026-08-01", + sources: [], + "superseded-by": "", + ...frontmatter, + }; + const yaml = Object.entries(fm) + .map(([k, v]) => `${k}: ${JSON.stringify(v)}`) + .join("\n"); + const file = join(root, "wiki", relPath); + mkdirSync(dirname(file), { recursive: true }); + writeFileSync(file, `---\n${yaml}\n---\n${body}`, "utf8"); +} + +function index(root: string, slugs: string[]): void { + const links = slugs.map((s) => `- [[${s}]]`).join("\n"); + writeFileSync(join(root, "wiki", "index.md"), `# Index\n\n## Pages\n\n${links}\n`, "utf8"); +} + +function changelog(root: string, slugs: string[]): void { + const bullets = slugs.map((s) => `- Created [[${s}]].`).join("\n"); + writeFileSync( + join(root, "wiki", "changelog.md"), + `# Changelog\n\n## 2026-08-01\n\n${bullets}\n`, + "utf8", + ); +} + +function source(root: string, id: string, text = "# Source\n"): void { + const dir = join(root, "raw", id); + mkdirSync(dir, { recursive: true }); + writeFileSync( + join(dir, "manifest.json"), + JSON.stringify({ id, title: id, kind: "file", original: id }), + "utf8", + ); + writeFileSync(join(dir, "text.md"), text, "utf8"); +} + +/** The codes present in a report, for asserting on what a check found. */ +function codes(findings: Finding[]): string[] { + return findings.map((f) => f.code); +} + +describe("the integrity checks (group 7)", () => { + let root: string; + beforeEach(() => (root = tempProject())); + afterEach(() => rmSync(root, { recursive: true, force: true })); + + describe("a wiki with nothing wrong", () => { + it("reports nothing", () => { + source(root, "notes.md"); + page(root, "fenix.md", "Rests on src://notes.md#p1.", { + sources: ["src://notes.md#p1"], + }); + index(root, ["fenix"]); + changelog(root, ["fenix"]); + + const report = checkProject(root); + expect(report.findings).toEqual([]); + expect(report.pages).toBe(1); + expect(report.sources).toBe(1); + }); + }); + + describe("7.1 — links and reachability", () => { + it("reports a wikilink that resolves to no page", () => { + page(root, "fenix.md", "See [[nowhere]]."); + index(root, ["fenix"]); + changelog(root, ["fenix"]); + + const findings = checkProject(root).findings.filter((f) => f.code === "wikilink.broken"); + expect(findings).toHaveLength(1); + expect(findings[0]!.message).toContain("nowhere"); + expect(findings[0]!.page).toBe("wiki/fenix.md"); + expect(findings[0]!.fix.length).toBeGreaterThan(0); + }); + + it("resolves a link to a page filed under a type subdirectory", () => { + // The plan's layout puts pages in wiki/topics/ and wiki/people/; a link + // names the slug, not the path. + page(root, "topics/checkout.md"); + page(root, "fenix.md", "See [[checkout]]."); + index(root, ["fenix", "checkout"]); + changelog(root, ["fenix", "checkout"]); + + expect(codes(checkProject(root).findings)).not.toContain("wikilink.broken"); + }); + + it("reports a page nothing in the index reaches", () => { + page(root, "fenix.md"); + page(root, "orphaned.md"); + index(root, ["fenix"]); + changelog(root, ["fenix", "orphaned"]); + + const findings = checkProject(root).findings.filter((f) => f.code === "page.orphan"); + expect(findings).toHaveLength(1); + expect(findings[0]!.page).toBe("wiki/orphaned.md"); + }); + + it("sees a page in a subdirectory, which used to be invisible entirely", () => { + // Reading only the top level of wiki/ meant a page written the way the + // plan's layout describes was never indexed, never checked, and never + // returned by ow graph. + page(root, "topics/checkout.md"); + index(root, []); + changelog(root, []); + + const report = checkProject(root); + expect(report.pages).toBe(1); + expect(codes(report.findings)).toContain("page.orphan"); + }); + + it("reports two pages sharing one slug, because a link cannot mean both", () => { + page(root, "checkout.md"); + page(root, "topics/checkout.md"); + index(root, ["checkout"]); + changelog(root, ["checkout"]); + + const findings = checkProject(root).findings.filter((f) => f.code === "page.duplicate-slug"); + expect(findings).toHaveLength(2); // one against each file + expect(findings[0]!.message).toContain("ambiguous"); + }); + + it("does not treat wiki/index.md, changelog.md or log.md as pages", () => { + writeFileSync(join(root, "wiki", "log.md"), "# Log\n", "utf8"); + index(root, []); + changelog(root, []); + expect(checkProject(root).pages).toBe(0); + }); + + it("treats a nested file called index.md as an ordinary page", () => { + page(root, "topics/index.md"); + index(root, ["index"]); + changelog(root, ["index"]); + expect(checkProject(root).pages).toBe(1); + }); + }); + + describe("7.2 — the records, and sources nothing uses", () => { + it("reports a changelog entry naming a page that does not exist", () => { + page(root, "fenix.md"); + index(root, ["fenix"]); + changelog(root, ["fenix", "deleted-page"]); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "changelog.missing-page", + ); + expect(findings).toHaveLength(1); + expect(findings[0]!.message).toContain("deleted-page"); + }); + + it("reports a page the changelog never recorded", () => { + page(root, "fenix.md"); + index(root, ["fenix"]); + changelog(root, []); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "changelog.unrecorded-page", + ); + expect(findings).toHaveLength(1); + expect(findings[0]!.severity).toBe("warning"); + }); + + it("reports a source no page cites", () => { + source(root, "unused.md"); + index(root, []); + changelog(root, []); + + const findings = checkProject(root).findings.filter((f) => f.code === "source.uncited"); + expect(findings).toHaveLength(1); + expect(findings[0]!.source).toBe("unused.md"); + expect(findings[0]!.severity).toBe("warning"); + }); + + it("counts a source cited only in prose as cited", () => { + // The agent writes citations where the claims are; sources: is filled in + // for it. Either one means the source is used. + source(root, "notes.md"); + page(root, "fenix.md", "As src://notes.md#p1 says."); + index(root, ["fenix"]); + changelog(root, ["fenix"]); + + expect(codes(checkProject(root).findings)).not.toContain("source.uncited"); + }); + + it("never reports the inbox as an uncited source", () => { + mkdirSync(join(root, "raw", "_inbox"), { recursive: true }); + writeFileSync(join(root, "raw", "_inbox", "dropped.md"), "x", "utf8"); + index(root, []); + changelog(root, []); + + const findings = checkProject(root).findings.filter((f) => f.code === "source.uncited"); + expect(findings).toEqual([]); + }); + }); + + describe("7.3 — provenance", () => { + it("reports a citation pointing at no source", () => { + page(root, "fenix.md", "From src://missing.pdf#p3.", { sources: ["src://missing.pdf#p3"] }); + index(root, ["fenix"]); + changelog(root, ["fenix"]); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "provenance.unresolved", + ); + expect(findings).toHaveLength(1); + expect(findings[0]!.message).toContain("missing.pdf"); + }); + + it("reports a recording citation whose fragment is not an instant", () => { + const dir = join(root, "raw", "weekly-2026-08-01"); + mkdirSync(dir, { recursive: true }); + writeFileSync( + join(dir, "manifest.json"), + JSON.stringify({ + id: "weekly-2026-08-01", + title: "Weekly", + kind: "recording", + original: "", + }), + "utf8", + ); + page(root, "fenix.md", "", { sources: ["rec://weekly-2026-08-01#halfway"] }); + index(root, ["fenix"]); + changelog(root, ["fenix"]); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "provenance.unresolved", + ); + expect(findings).toHaveLength(1); + expect(findings[0]!.message).toContain("HH:MM"); + }); + }); + + describe("7.4 — one term per concept", () => { + it("reports a page using another page's alias instead of its title", () => { + page(root, "fenix.md", "", { title: "Fenix", aliases: ["fenix platform"] }); + page(root, "checkout.md", "Built on the fenix platform, mostly."); + index(root, ["fenix", "checkout"]); + changelog(root, ["fenix", "checkout"]); + + const findings = checkProject(root).findings.filter((f) => f.code === "glossary.synonym"); + expect(findings).toHaveLength(1); + expect(findings[0]!.page).toBe("wiki/checkout.md"); + expect(findings[0]!.message).toContain("Fenix"); + }); + + it("leaves the page that declares the alias alone", () => { + // "Fenix, also known as the fenix platform" is where the synonym belongs. + page(root, "fenix.md", "Also known as the fenix platform.", { + title: "Fenix", + aliases: ["fenix platform"], + }); + index(root, ["fenix"]); + changelog(root, ["fenix"]); + + expect(codes(checkProject(root).findings)).not.toContain("glossary.synonym"); + }); + + it("does not flag an alias inside a wikilink or inside code", () => { + page(root, "fenix.md", "", { title: "Fenix", aliases: ["fenix platform"] }); + page(root, "checkout.md", "See [[fenix platform]] and `fenix platform`."); + index(root, ["fenix", "checkout"]); + changelog(root, ["fenix", "checkout"]); + + expect(codes(checkProject(root).findings)).not.toContain("glossary.synonym"); + }); + + it("reports two pages claiming one alias as a conflict, not as noise on one of them", () => { + // Last-writer-wins silently picked one and then flagged the *declaring* + // page of the loser for writing its own alias. + page(root, "a-page.md", "", { title: "Alpha", aliases: ["widget"] }); + page(root, "b-page.md", "", { title: "Beta", aliases: ["widget"] }); + index(root, ["a-page", "b-page"]); + changelog(root, ["a-page", "b-page"]); + + const found = checkProject(root).findings; + expect(codes(found)).toContain("glossary.conflict"); + expect(codes(found)).not.toContain("glossary.synonym"); + }); + + it("reports an alias that is another page's title, rather than flagging that page", () => { + page(root, "fenix.md", "", { title: "Fenix", aliases: ["checkout"] }); + page(root, "checkout.md", "The checkout is where it happens.", { title: "checkout" }); + index(root, ["fenix", "checkout"]); + changelog(root, ["fenix", "checkout"]); + + const found = checkProject(root).findings; + expect(codes(found)).toContain("glossary.conflict"); + // The victim page must not be told to stop writing its own name. + expect(found.filter((f) => f.code === "glossary.synonym")).toEqual([]); + }); + + it("matches whole words only", () => { + page(root, "fenix.md", "", { title: "Fenix", aliases: ["ana"] }); + page(root, "checkout.md", "The banana analysis is unrelated."); + index(root, ["fenix", "checkout"]); + changelog(root, ["fenix", "checkout"]); + + expect(codes(checkProject(root).findings)).not.toContain("glossary.synonym"); + }); + }); + + describe("7.5 — codewiki", () => { + it("accepts a citation that resolves and sits inside the file", () => { + writeFileSync(join(root, "code.ts"), "a\nb\nc\nd\ne\n", "utf8"); + page(root, "codewiki/dispatch.md", "## How it routes\n\n[code.ts:2-4]()\n\nOne switch.\n"); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + const found = codes(checkProject(root).findings); + expect(found).not.toContain("codewiki.citation-unresolved"); + expect(found).not.toContain("codewiki.citation-past-end"); + }); + + it("reports a citation whose file is gone", () => { + page(root, "codewiki/dispatch.md", "## Routing\n\n[src/gone.ts:1-4]()\n"); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "codewiki.citation-unresolved", + ); + expect(findings).toHaveLength(1); + expect(findings[0]!.message).toContain("src/gone.ts"); + }); + + it("reports a citation running past the end of its file", () => { + writeFileSync(join(root, "code.ts"), "a\nb\nc\n", "utf8"); + page(root, "codewiki/dispatch.md", "## Routing\n\n[code.ts:2-99]()\n"); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "codewiki.citation-past-end", + ); + expect(findings).toHaveLength(1); + expect(findings[0]!.message).toContain("3 lines"); + }); + + it("counts the last line of a file that ends with a newline exactly once", () => { + // "a\nb\nc\n" is three lines. Splitting on \n yields four elements, and + // trusting that would accept [code.ts:1-4] on a three-line file. + writeFileSync(join(root, "code.ts"), "a\nb\nc\n", "utf8"); + page(root, "codewiki/edge.md", "## Edge\n\n[code.ts:1-4]()\n"); + index(root, ["edge"]); + changelog(root, ["edge"]); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "codewiki.citation-past-end", + ); + expect(findings).toHaveLength(1); + }); + + it("accepts a citation ending exactly on the last line", () => { + writeFileSync(join(root, "code.ts"), "a\nb\nc\n", "utf8"); + page(root, "codewiki/exact.md", "## Exact\n\n[code.ts:1-3]()\n"); + index(root, ["exact"]); + changelog(root, ["exact"]); + + expect(codes(checkProject(root).findings)).not.toContain("codewiki.citation-past-end"); + }); + + it("reports a section that cites nothing", () => { + writeFileSync(join(root, "code.ts"), "a\nb\n", "utf8"); + page( + root, + "codewiki/dispatch.md", + "## Routing\n\n[code.ts:1-2]()\n\n## Opinions\n\nNo citation here.\n", + ); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "codewiki.section-uncited", + ); + expect(findings).toHaveLength(1); + expect(findings[0]!.message).toContain("Opinions"); + }); + + it("refuses to read a citation pointing outside the project", () => { + page(root, "codewiki/dispatch.md", "## Routing\n\n[../../etc/passwd:1-2]()\n"); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + const findings = checkProject(root).findings.filter( + (f) => f.code === "codewiki.citation-unresolved", + ); + expect(findings).toHaveLength(1); + }); + + it("does not apply the codewiki rules to an ordinary wiki page", () => { + page(root, "fenix.md", "## A section with no citation\n\nProse.\n"); + index(root, ["fenix"]); + changelog(root, ["fenix"]); + + expect(codes(checkProject(root).findings)).not.toContain("codewiki.section-uncited"); + }); + + it("counts a citation written on the heading line itself", () => { + writeFileSync(join(root, "code.ts"), "a\nb\n", "utf8"); + page(root, "codewiki/dispatch.md", "## Alpha [code.ts:1-2]()\n\nProse.\n"); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + // It used to report `the section "Alpha [code.ts:1-2]()" cites no lines`, + // quoting the citation it had just refused to look at. + expect(codes(checkProject(root).findings)).not.toContain("codewiki.section-uncited"); + }); + + it("does not treat the page's own H1 title as an uncited section", () => { + writeFileSync(join(root, "code.ts"), "a\nb\n", "utf8"); + page( + root, + "codewiki/dispatch.md", + "# The dispatcher\n\nAn intro.\n\n## Routing\n\n[code.ts:1-2]()\n", + ); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + expect(codes(checkProject(root).findings)).not.toContain("codewiki.section-uncited"); + }); + + it("reports a stray codewiki page at any depth below the root", () => { + mkdirSync(join(root, "codewiki", "area"), { recursive: true }); + writeFileSync(join(root, "codewiki", "area", "x.md"), "# Stray\n", "utf8"); + index(root, []); + changelog(root, []); + + expect(codes(checkProject(root).findings)).toContain("codewiki.misplaced"); + }); + + it("reports a codewiki/ at the project root as misplaced", () => { + // Outside wiki/ it is not part of the wiki: nothing indexes it, nothing + // links it, and no write to it is validated. + mkdirSync(join(root, "codewiki"), { recursive: true }); + writeFileSync(join(root, "codewiki", "stray.md"), "# Stray\n", "utf8"); + index(root, []); + changelog(root, []); + + const findings = checkProject(root).findings.filter((f) => f.code === "codewiki.misplaced"); + expect(findings).toHaveLength(1); + expect(findings[0]!.fix).toContain("wiki/codewiki/"); + }); + }); + + describe("hostile page content", () => { + it("scans a pathological citation-shaped line in linear time", { timeout: 10_000 }, () => { + // `[/a/a/a/...]` with the `/` also inside the segment class made the + // pattern ambiguous: every partition of the run is a backtracking path, + // and ~26 segments took 1.6s, ~34 took minutes. That is a page body — an + // agent writes those, possibly steered by a poisoned source in raw/ — + // wedging `ow check`, CI and the UI in a spin no try/catch interrupts. + const payload = `[${"/a".repeat(60)}]`; + page(root, "codewiki/evil.md", `## Section\n\n${payload}\n`); + index(root, ["evil"]); + changelog(root, ["evil"]); + + const started = Date.now(); + checkProject(root); + expect(Date.now() - started).toBeLessThan(5000); + }); + + it("strips control characters out of a message, so a page cannot forge the report", () => { + // `ow check` writes findings to a terminal, and the report is what a + // human reads to decide whether the wiki is sound. A YAML double-quoted + // alias carries \n, \r and \u001b intact. + page(root, "fenix.md", "", { + title: "Fenix", + aliases: ["widget\n\now check: no findings (99 pages)\n\u001b[2K"], + }); + page(root, "checkout.md", "Uses a widget here."); + index(root, ["fenix", "checkout"]); + changelog(root, ["fenix", "checkout"]); + + for (const finding of checkProject(root).findings) { + expect(finding.message).not.toMatch(/\p{Cc}/u); + expect(finding.fix).not.toMatch(/\p{Cc}/u); + } + }); + + it("reports a very long alias without flooding the report", () => { + page(root, "fenix.md", "", { title: "Fenix", aliases: ["x".repeat(10_000)] }); + page(root, "checkout.md", `Mentions ${"x".repeat(10_000)} here.`); + index(root, ["fenix", "checkout"]); + changelog(root, ["fenix", "checkout"]); + + const synonym = checkProject(root).findings.find((f) => f.code === "glossary.synonym"); + expect(synonym!.message.length).toBeLessThan(600); + }); + }); + + describe("finding locations", () => { + it("reports the line in the file, not in the body", () => { + // Reported lines used to land in the frontmatter, because the slug and + // the alias appear in `id:` and `title:` before they appear in prose. + page(root, "target.md"); + page(root, "fenix.md", "\nsome prose\n\nSee [[nowhere]] here.\n"); + index(root, ["fenix", "target"]); + changelog(root, ["fenix", "target"]); + + const broken = checkProject(root).findings.find((f) => f.code === "wikilink.broken"); + const text = readFileSync(join(root, "wiki", "fenix.md"), "utf8"); + expect(text.split("\n")[broken!.line! - 1]).toContain("[[nowhere]]"); + }); + + it("reports a codewiki section's line in the file too", () => { + page(root, "codewiki/dispatch.md", "\n## Opinions\n\nNo citation here.\n"); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + const finding = checkProject(root).findings.find( + (f) => f.code === "codewiki.section-uncited", + ); + const text = readFileSync(join(root, "wiki", "codewiki", "dispatch.md"), "utf8"); + expect(text.split("\n")[finding!.line! - 1]).toContain("## Opinions"); + }); + }); + + describe("the page schema, re-checked after the fact", () => { + it("reports a page that never went through the gate", () => { + // A page written through the shell, or edited in another editor, passes + // no gate at all — group 7 is the only thing between it and permanence. + writeFileSync(join(root, "wiki", "raw-page.md"), "no frontmatter at all\n", "utf8"); + index(root, ["raw-page"]); + changelog(root, ["raw-page"]); + + const findings = checkProject(root).findings.filter((f) => f.code === "page.invalid"); + expect(findings.length).toBeGreaterThan(0); + }); + }); + + describe("citedSourcePages", () => { + it("maps a source to every page that cites it", () => { + source(root, "notes.md"); + page(root, "a.md", "src://notes.md#p1"); + page(root, "b.md", "", { sources: ["src://notes.md#p2"] }); + + const citations = citedSourcePages(readWiki(root)); + expect(citations.get("notes.md")?.sort()).toEqual(["wiki/a.md", "wiki/b.md"]); + }); + + it("lists a page once even when it cites the same source twice", () => { + source(root, "notes.md"); + page(root, "a.md", "src://notes.md#p1 and again src://notes.md#p2"); + + expect(citedSourcePages(readWiki(root)).get("notes.md")).toEqual(["wiki/a.md"]); + }); + }); + + describe("the report itself", () => { + it("puts errors before warnings", () => { + page(root, "fenix.md", "See [[nowhere]]."); + source(root, "unused.md"); + index(root, ["fenix"]); + changelog(root, []); + + const findings = checkProject(root).findings; + const firstWarning = findings.findIndex((f) => f.severity === "warning"); + const lastError = findings.map((f) => f.severity).lastIndexOf("error"); + expect(lastError).toBeLessThan(firstWarning); + }); + + it("gives every finding a correction path", () => { + page(root, "fenix.md", "See [[nowhere]]."); + source(root, "unused.md"); + index(root, []); + changelog(root, []); + + for (const finding of checkProject(root).findings) { + expect(finding.fix.length).toBeGreaterThan(0); + } + }); + + it("returns nothing at all for a project with no wiki", () => { + rmSync(join(root, "wiki"), { recursive: true, force: true }); + const report = checkProject(root); + expect(report.pages).toBe(0); + expect(report.findings).toEqual([]); + }); + }); + + describe("sortFindings / hasErrors", () => { + const error: Finding = { code: "page.orphan", severity: "error", message: "e", fix: "f" }; + const warning: Finding = { + code: "source.uncited", + severity: "warning", + message: "w", + fix: "f", + }; + + it("sorts errors first", () => { + expect(sortFindings([warning, error])[0]).toBe(error); + }); + + it("says whether anything is an error", () => { + expect(hasErrors([warning])).toBe(false); + expect(hasErrors([warning, error])).toBe(true); + expect(hasErrors([])).toBe(false); + }); + }); +}); diff --git a/packages/access/tests/gate-decision.spec.ts b/packages/access/tests/gate-decision.spec.ts index 71bd69b..8e799bf 100644 --- a/packages/access/tests/gate-decision.spec.ts +++ b/packages/access/tests/gate-decision.spec.ts @@ -27,22 +27,36 @@ describe("gateWrite — routing (9.5)", () => { afterEach(() => rmSync(root, { recursive: true, force: true })); it("refuses a write to the gate's own configuration (9.6)", () => { - const d = gateWrite({ projectRoot: root, filePath: ".claude/settings.json", content: "x", date: DATE }); + const d = gateWrite({ + projectRoot: root, + filePath: ".claude/settings.json", + content: "x", + date: DATE, + }); expect(d.action).toBe("deny"); if (d.action === "deny") expect(d.reasons.join(" ")).toContain("configuration"); }); it("passes through a non-gated path unchanged", () => { - expect(gateWrite({ projectRoot: root, filePath: "README.md", content: "hi\n", date: DATE })).toEqual({ + expect( + gateWrite({ projectRoot: root, filePath: "README.md", content: "hi\n", date: DATE }), + ).toEqual({ action: "allow", }); expect( - gateWrite({ projectRoot: root, filePath: "raw/doc.pdf/text.md", content: "hi\n", date: DATE }), + gateWrite({ + projectRoot: root, + filePath: "raw/doc.pdf/text.md", + content: "hi\n", + date: DATE, + }), ).toEqual({ action: "allow" }); }); it("passes through the non-entity pages under wiki/ (index, changelog, log)", () => { - expect(gateWrite({ projectRoot: root, filePath: "wiki/index.md", content: "# Index\n", date: DATE })).toEqual({ + expect( + gateWrite({ projectRoot: root, filePath: "wiki/index.md", content: "# Index\n", date: DATE }), + ).toEqual({ action: "allow", }); }); @@ -233,12 +247,44 @@ describe("gateWrite — path confinement (2.6, the review's HIGH finding)", () = // On Windows these name the same files as `wiki/fenix.md`. A pass-through // would land an unvalidated page in the wiki, so the gate must have an // opinion about all of them. - for (const filePath of ["Wiki/fenix.md", "wiki/fenix.MD", "WIKI/FENIX.MD", "CodeWiki/x.md"]) { + for (const filePath of ["Wiki/fenix.md", "wiki/fenix.MD", "WIKI/FENIX.MD"]) { + const d = gateWrite({ projectRoot: root, filePath, content: page(GOOD_FM), date: DATE }); + expect(d.action, filePath).not.toBe("allow"); + } + }); + + it("gates a codewiki page, which lives under wiki/", () => { + for (const filePath of ["wiki/codewiki/dispatch.md", "Wiki/CodeWiki/Dispatch.MD"]) { const d = gateWrite({ projectRoot: root, filePath, content: page(GOOD_FM), date: DATE }); expect(d.action, filePath).not.toBe("allow"); } }); + it("has no opinion about a top-level codewiki/, which is not the wiki", () => { + // Settling plan 7.5: the scaffolded skill and the plan's layout both put + // codewiki at `wiki/codewiki/`. A page outside `wiki/` is not part of the + // wiki at all — nothing indexes it, nothing links it, nothing can cite it — + // and gating it implied otherwise. `ow check` reports it as misplaced. + const d = gateWrite({ + projectRoot: root, + filePath: "codewiki/x.md", + content: page(GOOD_FM), + date: DATE, + }); + expect(d.action).toBe("allow"); + }); + + it("gates a nested page named index.md, which is not the wiki index", () => { + // Only the three at the top of wiki/ are the wiki's own pages. + const d = gateWrite({ + projectRoot: root, + filePath: "wiki/topics/index.md", + content: page(GOOD_FM.replace("id: t:fenix", "id: t:index")), + date: DATE, + }); + expect(d.action).not.toBe("allow"); + }); + it("takes the slug from the filename without the extension, whatever its case", () => { // `basename(rel, ".md")` leaves `.MD` attached, which would make the slug // `Fenix.MD` and fail the id/filename agreement for the wrong reason. @@ -260,9 +306,12 @@ describe("gateWrite — path confinement (2.6, the review's HIGH finding)", () = describe("formatDenial (9.13)", () => { it("renders a header and one bullet per reason", () => { - const text = formatDenial("wiki/fenix.md", ["updated must be a date", "[[x]] does not resolve"]); + const text = formatDenial("wiki/fenix.md", [ + "updated must be a date", + "[[x]] does not resolve", + ]); expect(text).toContain("wiki/fenix.md"); expect(text).toContain("- updated must be a date"); expect(text).toContain("- [[x]] does not resolve"); }); -}); \ No newline at end of file +}); diff --git a/packages/access/tests/sources-state.spec.ts b/packages/access/tests/sources-state.spec.ts new file mode 100644 index 0000000..86f2a13 --- /dev/null +++ b/packages/access/tests/sources-state.spec.ts @@ -0,0 +1,156 @@ +import { describe, expect, it, beforeEach, afterEach } from "vitest"; +import { mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { listSourceStates, sourceState } from "../src/sources/state.js"; +import { MissingSourceError } from "../src/sources/manifest.js"; + +function tempProject(): string { + const root = realpathSync(mkdtempSync(join(tmpdir(), "ow-state-"))); + mkdirSync(join(root, "raw"), { recursive: true }); + return root; +} + +/** A source directory, with only the parts a case needs. */ +function source( + root: string, + id: string, + parts: { kind?: "file" | "recording"; text?: string; journal?: unknown } = {}, +): void { + const dir = join(root, "raw", id); + mkdirSync(dir, { recursive: true }); + writeFileSync( + join(dir, "manifest.json"), + JSON.stringify({ id, title: `Title of ${id}`, kind: parts.kind ?? "file", original: id }), + "utf8", + ); + if (parts.text !== undefined) writeFileSync(join(dir, "text.md"), parts.text, "utf8"); + if (parts.journal !== undefined) { + writeFileSync(join(dir, "journal.json"), JSON.stringify(parts.journal), "utf8"); + } +} + +describe("source state (6.1)", () => { + let root: string; + beforeEach(() => (root = tempProject())); + afterEach(() => rmSync(root, { recursive: true, force: true })); + + describe("sourceState", () => { + it("is received when the manifest is there and nothing else is", () => { + source(root, "arch.pdf"); + const state = sourceState(root, "arch.pdf"); + expect(state.stage).toBe("received"); + expect(state.textReady).toBe(false); + expect(state.title).toBe("Title of arch.pdf"); + expect(state.kind).toBe("file"); + }); + + it("is text-ready once text.md exists and no page cites it", () => { + source(root, "arch.pdf", { text: "# Arch\n" }); + const state = sourceState(root, "arch.pdf"); + expect(state.stage).toBe("text-ready"); + expect(state.textReady).toBe(true); + expect(state.citedBy).toEqual([]); + }); + + it("is not cited while its text never landed, though it records the citation", () => { + // A page citing a source with no text.md cites something nothing could + // have read. Calling that the last stage of the pipeline hides it. + source(root, "arch.pdf"); + const state = sourceState(root, "arch.pdf", ["wiki/fenix.md"]); + expect(state.stage).toBe("received"); + expect(state.citedBy).toEqual(["wiki/fenix.md"]); + expect(state.textReady).toBe(false); + }); + + it("is cited once a page rests on it, and says which pages", () => { + source(root, "arch.pdf", { text: "# Arch\n" }); + const state = sourceState(root, "arch.pdf", ["wiki/fenix.md"]); + expect(state.stage).toBe("cited"); + expect(state.citedBy).toEqual(["wiki/fenix.md"]); + }); + + it("is transcribing while a journal is part-done, with the progress", () => { + source(root, "weekly", { + kind: "recording", + journal: { chunks: [{ done: true }, { done: true }, { done: false }] }, + }); + const state = sourceState(root, "weekly"); + expect(state.stage).toBe("transcribing"); + expect(state.progress).toEqual({ done: 2, total: 3 }); + }); + + it("is failed, with the reason, when a chunk stopped and no text landed", () => { + // A run abandoned partway is the case that keeps its WAV forever, so + // surfacing it is part of the retention story, not a nicety. + source(root, "weekly", { + kind: "recording", + journal: { chunks: [{ done: true }, { error: "groq: 401 unauthorized" }] }, + }); + const state = sourceState(root, "weekly"); + expect(state.stage).toBe("failed"); + expect(state.error).toContain("401"); + expect(state.progress).toEqual({ done: 1, total: 2 }); + }); + + it("is not failed once the text landed anyway", () => { + // A chunk that failed and was retried successfully leaves its error in + // the journal; the text is the thing that says it finished. + source(root, "weekly", { + kind: "recording", + text: "# Weekly\n", + journal: { chunks: [{ done: true, error: "transient" }] }, + }); + expect(sourceState(root, "weekly").stage).toBe("text-ready"); + }); + + it("falls back to the directory when the journal will not parse", () => { + const dir = join(root, "raw", "weekly"); + source(root, "weekly", { kind: "recording", text: "# Weekly\n" }); + writeFileSync(join(dir, "journal.json"), "{ not json", "utf8"); + expect(sourceState(root, "weekly").stage).toBe("text-ready"); + }); + + it("refuses an id that is not a source", () => { + expect(() => sourceState(root, "nothing")).toThrow(MissingSourceError); + }); + + it("refuses an id that escapes raw/", () => { + expect(() => sourceState(root, "../../elsewhere")).toThrow(); + }); + }); + + describe("listSourceStates", () => { + it("lists every source, in a stable order", () => { + source(root, "b.md", { text: "b" }); + source(root, "a.md"); + expect(listSourceStates(root).map((s) => s.id)).toEqual(["a.md", "b.md"]); + }); + + it("marks the ones a page cites, from the map it is handed", () => { + source(root, "a.md", { text: "a" }); + source(root, "b.md", { text: "b" }); + + const states = listSourceStates(root, new Map([["a.md", ["wiki/fenix.md"]]])); + expect(states.find((s) => s.id === "a.md")?.stage).toBe("cited"); + expect(states.find((s) => s.id === "b.md")?.stage).toBe("text-ready"); + }); + + it("never lists the inbox, which is a doorway and not a source", () => { + mkdirSync(join(root, "raw", "_inbox"), { recursive: true }); + source(root, "a.md"); + expect(listSourceStates(root).map((s) => s.id)).toEqual(["a.md"]); + }); + + it("returns nothing for a project with no sources", () => { + expect(listSourceStates(root)).toEqual([]); + }); + + it("survives a restart without reconciling anything", () => { + // The state is the directory, so a second read of the same disk reaches + // the same answer — there is nothing to persist and nothing to resume. + source(root, "a.md", { text: "a" }); + expect(listSourceStates(root)).toEqual(listSourceStates(root)); + }); + }); +}); diff --git a/packages/access/tests/store-index.spec.ts b/packages/access/tests/store-index.spec.ts index fac4799..8656acb 100644 --- a/packages/access/tests/store-index.spec.ts +++ b/packages/access/tests/store-index.spec.ts @@ -1,8 +1,23 @@ import { describe, expect, it, beforeEach, afterEach } from "vitest"; -import { mkdtempSync, mkdirSync, rmSync, readFileSync, writeFileSync } from "node:fs"; +import { + existsSync, + mkdtempSync, + mkdirSync, + rmSync, + readFileSync, + symlinkSync, + writeFileSync, +} from "node:fs"; import { tmpdir } from "node:os"; -import { join } from "node:path"; -import { listEntityPages, isIndexed, findOrphans } from "../src/store/index.js"; +import { dirname, join } from "node:path"; +import { + listEntityPages, + listPages, + pagePath, + isIndexed, + findOrphans, + readIndex, +} from "../src/store/index.js"; import { registerInIndex } from "../src/store/index-write.js"; function tempProject() { @@ -115,3 +130,89 @@ describe("findOrphans (5.7)", () => { expect(findOrphans(root)).toEqual(["fenix"]); }); }); + +describe("listPages / pagePath — a page is its slug wherever it sits (adr:0016)", () => { + let root: string; + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "ow-listpages-")); + mkdirSync(join(root, "wiki"), { recursive: true }); + }); + afterEach(() => rmSync(root, { recursive: true, force: true })); + + const write = (rel: string): void => { + const file = join(root, "wiki", rel); + mkdirSync(dirname(file), { recursive: true }); + writeFileSync(file, "# page\n", "utf8"); + }; + + it("finds a page at any depth, and says where it is", () => { + write("flat.md"); + write("topics/checkout.md"); + write("codewiki/dispatch.md"); + + expect(listPages(root).map((p) => [p.slug, p.path])).toEqual([ + ["dispatch", "wiki/codewiki/dispatch.md"], + ["flat", "wiki/flat.md"], + ["checkout", "wiki/topics/checkout.md"], + ]); + }); + + it("marks only the pages under codewiki/", () => { + write("topics/checkout.md"); + write("codewiki/dispatch.md"); + const byCodewiki = Object.fromEntries(listPages(root).map((p) => [p.slug, p.codewiki])); + expect(byCodewiki).toEqual({ checkout: false, dispatch: true }); + }); + + it("excludes the wiki's own three pages, at the top level only", () => { + write("index.md"); + write("changelog.md"); + write("log.md"); + write("topics/index.md"); + expect(listPages(root).map((p) => p.path)).toEqual(["wiki/topics/index.md"]); + }); + + it("does not follow a symlinked directory out of the project", () => { + const outside = mkdtempSync(join(tmpdir(), "ow-outside-")); + try { + writeFileSync(join(outside, "secret.md"), "# secret\n", "utf8"); + try { + symlinkSync(outside, join(root, "wiki", "escape")); + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + if (code === "EPERM" || code === "EACCES" || code === "ENOSYS") return; + throw err; + } + expect(listPages(root)).toEqual([]); + } finally { + rmSync(outside, { recursive: true, force: true }); + } + }); + + it("returns nothing when there is no wiki, and creates nothing", () => { + rmSync(join(root, "wiki"), { recursive: true, force: true }); + expect(listPages(root)).toEqual([]); + expect(existsSync(join(root, "wiki"))).toBe(false); + }); + + it("pagePath resolves a slug to where the file actually is", () => { + write("topics/checkout.md"); + expect(pagePath(root, "checkout")).toBe("wiki/topics/checkout.md"); + expect(pagePath(root, "nothing")).toBeUndefined(); + }); + + it("readIndex creates no directory — a read must not write", () => { + // checkProject is exported into the read-only surface the MCP process + // imports; a read that mkdirs makes every caller a writer, and `ow check` + // in the wrong directory would leave a wiki/ behind. + rmSync(join(root, "wiki"), { recursive: true, force: true }); + readIndex(root); + expect(existsSync(join(root, "wiki"))).toBe(false); + }); + + it("isIndexed treats a slug carrying regex metacharacters literally", () => { + expect(isIndexed("- [[a.b]]", "a.b")).toBe(true); + // `(.*)` as a slug would otherwise match any wikilink and mark itself indexed. + expect(isIndexed("- [[something]]", "(.*)")).toBe(false); + }); +}); diff --git a/packages/access/tests/store-wikilinks.spec.ts b/packages/access/tests/store-wikilinks.spec.ts index 11ad423..6c38959 100644 --- a/packages/access/tests/store-wikilinks.spec.ts +++ b/packages/access/tests/store-wikilinks.spec.ts @@ -1,7 +1,7 @@ import { describe, expect, it, beforeEach, afterEach } from "vitest"; import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { dirname, join } from "node:path"; import { resolveWikilinks } from "../src/store/wikilinks.js"; function tempProject() { @@ -77,3 +77,56 @@ describe("resolveWikilinks (5.3)", () => { expect(resolveWikilinks(root, body, "fenix")).toEqual([]); }); }); + +describe("resolveWikilinks — what a link may name (adr:0016)", () => { + let root: string; + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "ow-links2-")); + mkdirSync(join(root, "wiki"), { recursive: true }); + }); + afterEach(() => rmSync(root, { recursive: true, force: true })); + + const write = (rel: string): void => { + const file = join(root, "wiki", rel); + mkdirSync(dirname(file), { recursive: true }); + writeFileSync(file, "# page\n", "utf8"); + }; + + it("resolves a link to a page filed under a subdirectory", () => { + write("topics/checkout.md"); + expect(resolveWikilinks(root, "See [[checkout]].", "other")).toEqual([]); + }); + + it("resolves a link to the wiki's own index, changelog and log", () => { + // These are not entity pages, but they are files in every scaffolded + // project and the skill tells the agent to record things in them. Denying + // a page for mentioning [[changelog]] gave a reason that reads as a bug. + write("changelog.md"); + write("index.md"); + write("log.md"); + expect( + resolveWikilinks( + root, + "Recorded in the [[changelog]], linked from [[index]], see [[log]].", + "x", + ), + ).toEqual([]); + }); + + it("still reports a link that names nothing", () => { + const issues = resolveWikilinks(root, "See [[nowhere]].", "x"); + expect(issues).toHaveLength(1); + expect(issues[0]!.target).toBe("nowhere"); + }); + + it("carries the target on the issue, so nobody parses it back out of the reason", () => { + const issues = resolveWikilinks(root, "See [[a-b-c]].", "x"); + expect(issues[0]!.target).toBe("a-b-c"); + }); + + it("accepts a caller-supplied slug set instead of walking the wiki", () => { + // `ow check` builds the set once for the whole run; per-page walks made it + // quadratic — three seconds over eight hundred pages. + expect(resolveWikilinks(root, "See [[checkout]].", "x", new Set(["checkout"]))).toEqual([]); + }); +}); diff --git a/packages/cli/src/commands/check.ts b/packages/cli/src/commands/check.ts new file mode 100644 index 0000000..761440a --- /dev/null +++ b/packages/cli/src/commands/check.ts @@ -0,0 +1,88 @@ +import { checkProject, hasErrors, safe, type Finding } from "@open-wiki/access"; + +/** + * `ow check` — the integrity checks of group 7, without the application + * (plan 7.7). One implementation, three readers: an agent that just wrote a + * page, a CI job on a committed wiki, and 7.6's UI, which renders the same + * findings. + * + * The exit code is the contract, and it is the one this repo already uses for + * `scc`: `0` clean, `2` it ran and found something, `1` it could not run. A + * finding is an answer, not a crash — which is what lets a CI job treat `2` as + * "fix this" and `1` as "the check itself is broken". + */ + +export const CHECK_OK = 0; +export const CHECK_FAILED_TO_RUN = 1; +export const CHECK_FOUND = 2; + +export interface CheckOptions { + json: boolean; + /** Report warnings too. Errors are always reported. */ + warnings: boolean; +} + +export function parseCheckArgs(args: string[]): CheckOptions { + return { + json: args.includes("--json"), + // Warnings are on by default: "a source nobody cites" is exactly the thing + // that disappears from view on its own, so hiding it by default would + // defeat the check. `--errors-only` is for a CI job that wants a gate. + warnings: !args.includes("--errors-only"), + }; +} + +function renderText(findings: Finding[], counts: { pages: number; sources: number }): string { + if (findings.length === 0) { + return `ow check: no findings (${counts.pages} page(s), ${counts.sources} source(s))\n`; + } + + const lines: string[] = []; + for (const finding of findings) { + const where = finding.page ?? (finding.source ? `raw/${finding.source}` : ""); + const at = finding.line === undefined ? where : `${where}:${finding.line}`; + lines.push(`${finding.severity}: ${finding.code}`); + // Scrubbed again here, at the one place that writes to a terminal. The + // messages are built from page content, and a `\r` plus a cursor escape can + // overwrite the summary a human reads to decide whether the wiki is sound. + lines.push(` ${safe(finding.message)}`); + if (at) lines.push(` at ${safe(at)}`); + // The correction path, on every finding. A reader who cannot act on a + // report repeats the thing that caused it. + lines.push(` fix: ${safe(finding.fix)}`); + lines.push(""); + } + + const errors = findings.filter((f) => f.severity === "error").length; + const warnings = findings.length - errors; + lines.push( + `ow check: ${errors} error(s), ${warnings} warning(s) over ${counts.pages} page(s) and ${counts.sources} source(s)`, + ); + return lines.join("\n") + "\n"; +} + +export interface CheckOutput { + stdout: string; + code: number; +} + +/** + * Run the checks and render them. Returns the text and the exit code rather + * than writing and exiting, so a test can drive it — the same split `main.ts` + * makes for every other verb. + */ +export function runCheck(projectRoot: string, options: CheckOptions): CheckOutput { + const report = checkProject(projectRoot); + const findings = options.warnings + ? report.findings + : report.findings.filter((f) => f.severity === "error"); + + const stdout = options.json + ? JSON.stringify({ findings, pages: report.pages, sources: report.sources }, null, 2) + "\n" + : renderText(findings, report); + + // Only an error fails the check. A warning is something a project may + // legitimately choose — a source uploaded this morning that nothing cites yet + // must not turn a CI job red. + return { stdout, code: hasErrors(findings) ? CHECK_FOUND : CHECK_OK }; +} diff --git a/packages/cli/src/commands/graph.ts b/packages/cli/src/commands/graph.ts index 16747ed..83e0050 100644 --- a/packages/cli/src/commands/graph.ts +++ b/packages/cli/src/commands/graph.ts @@ -1,6 +1,12 @@ import { readFileSync } from "node:fs"; import { join } from "node:path"; -import { findOrphans, listEntityPages, readFrontmatter, readIndex } from "@open-wiki/access"; +import { + findOrphans, + listPages, + readFrontmatter, + readIndex, + type PageRef, +} from "@open-wiki/access"; /** * `ow graph [superseded|orphans|index]` — the structural queries (plan 9.12). @@ -8,16 +14,21 @@ import { findOrphans, listEntityPages, readFrontmatter, readIndex } from "@open- * records. Local queries print JSON for a harness to read. */ export function runGraph(projectRoot: string, sub: string | undefined): string { - const pages = listEntityPages(projectRoot); + // The refs, not just the slugs: a page is its slug wherever it sits under + // `wiki/` (`adr:0016`), so reading one means looking up where it is. Assuming + // `wiki/.md` threw ENOENT — a stack, not a sentence — the moment a + // project filed a page the way the plan's layout describes. + const refs = listPages(projectRoot); + const pages = refs.map((p) => p.slug); if (sub === "orphans") return JSON.stringify(findOrphans(projectRoot), null, 2); - if (sub === "superseded") return JSON.stringify(supersessionWalk(projectRoot, pages), null, 2); + if (sub === "superseded") return JSON.stringify(supersessionWalk(projectRoot, refs), null, 2); if (sub === "index") return JSON.stringify(indexState(projectRoot, pages), null, 2); // default: the whole structure return JSON.stringify( { pages, orphans: findOrphans(projectRoot), - superseded: supersessionWalk(projectRoot, pages), + superseded: supersessionWalk(projectRoot, refs), }, null, 2, @@ -30,10 +41,10 @@ interface SupersededEntry { updated: string; } -function supersessionWalk(projectRoot: string, pages: string[]): SupersededEntry[] { +function supersessionWalk(projectRoot: string, refs: PageRef[]): SupersededEntry[] { const out: SupersededEntry[] = []; - for (const slug of pages) { - const text = readPage(projectRoot, slug); + for (const { slug, path } of refs) { + const text = readPage(projectRoot, path); const block = readFrontmatter(text); if (!block || !block.parsed) continue; const fm = block.frontmatter as Record; @@ -53,6 +64,6 @@ function indexState(projectRoot: string, pages: string[]): string[] { return pages.filter((slug) => new RegExp(`\\[\\[${slug}(\\||#|\\]\\])`).test(indexText)); } -function readPage(projectRoot: string, slug: string): string { - return readFileSync(join(projectRoot, "wiki", `${slug}.md`), "utf8"); -} \ No newline at end of file +function readPage(projectRoot: string, relPath: string): string { + return readFileSync(join(projectRoot, relPath), "utf8"); +} diff --git a/packages/cli/src/commands/search.ts b/packages/cli/src/commands/search.ts index fb864ba..aa23f7d 100644 --- a/packages/cli/src/commands/search.ts +++ b/packages/cli/src/commands/search.ts @@ -1,6 +1,6 @@ import { readFileSync } from "node:fs"; import { join } from "node:path"; -import { listEntityPages, readFrontmatter } from "@open-wiki/access"; +import { listPages, readFrontmatter } from "@open-wiki/access"; /** * `ow search ` — lexical search over the wiki (plan 9.12). A scan over a @@ -11,14 +11,19 @@ import { listEntityPages, readFrontmatter } from "@open-wiki/access"; export function runSearch(projectRoot: string, query: string): string { const q = query.toLowerCase(); const results: Array<{ slug: string; title: string; matches: number }> = []; - for (const slug of listEntityPages(projectRoot)) { - const text = readFileSync(join(projectRoot, "wiki", `${slug}.md`), "utf8"); + // Read where the page actually is: a page is its slug wherever it sits under + // `wiki/` (`adr:0016`), and assuming the top level threw ENOENT on any + // project that filed pages by type. + for (const { slug, path } of listPages(projectRoot)) { + const text = readFileSync(join(projectRoot, path), "utf8"); const lower = text.toLowerCase(); const matches = countOccurrences(lower, q); if (matches === 0) continue; const block = readFrontmatter(text); const title = - block && block.parsed && typeof (block.frontmatter as Record)?.["title"] === "string" + block && + block.parsed && + typeof (block.frontmatter as Record)?.["title"] === "string" ? String((block.frontmatter as Record)["title"]) : slug; results.push({ slug, title, matches }); @@ -35,4 +40,4 @@ function countOccurrences(haystack: string, needle: string): number { i = haystack.indexOf(needle, i + needle.length); } return count; -} \ No newline at end of file +} diff --git a/packages/cli/src/hooks.ts b/packages/cli/src/hooks.ts index 650450b..b2a7d65 100644 --- a/packages/cli/src/hooks.ts +++ b/packages/cli/src/hooks.ts @@ -101,16 +101,20 @@ function titleOf(markdown: string): string | undefined { } /** A shell command that writes a wiki/codewiki page, and the path it targets. */ +// `wiki/` only, which covers `wiki/codewiki/`. A top-level `codewiki/` used to +// be listed here as well, and once the gate stopped treating it as part of the +// wiki (adr:0016) the two doors disagreed: this one flagged a shell write there +// and handed it to a gate that then said "allow". const SHELL_WRITE_TARGETS: ReadonlyArray = [ - />>?\s*((?:wiki|codewiki)\/[^\s;|&<>]*\.md)/, - /\btee\s+(?:-\S+\s+)*((?:wiki|codewiki)\/[^\s;|&<>]*\.md)/, - /\b(?:cp|mv|install)\s+(?:-\S+\s+)*\S+\s+((?:wiki|codewiki)\/[^\s;|&<>]*\.md)/, - /\bsed\s+-i\b[^|&]*?\s((?:wiki|codewiki)\/[^\s;|&<>]*\.md)/, + />>?\s*(wiki\/[^\s;|&<>]*\.md)/, + /\btee\s+(?:-\S+\s+)*(wiki\/[^\s;|&<>]*\.md)/, + /\b(?:cp|mv|install)\s+(?:-\S+\s+)*\S+\s+(wiki\/[^\s;|&<>]*\.md)/, + /\bsed\s+-i\b[^|&]*?\s(wiki\/[^\s;|&<>]*\.md)/, ]; export function detectShellWrite(command: string, _projectRoot: string): string | null { // Normalise backslashes to forward slashes first: the targets anchor on - // `(?:wiki|codewiki)/`, and on Windows a shell write may use `wiki\fenix.md`. + // `wiki/`, and on Windows a shell write may use `wiki\fenix.md`. const posix = command.replace(/\\/g, "/"); for (const re of SHELL_WRITE_TARGETS) { const m = re.exec(posix); @@ -211,11 +215,7 @@ export function runPreToolUse( }; } -export function runPostToolUse( - input: PostToolUseInput, - projectRoot: string, - date: string, -): null { +export function runPostToolUse(input: PostToolUseInput, projectRoot: string, date: string): null { if (input.tool_name !== "Write" && input.tool_name !== "Edit") return null; const s = readSidecar(projectRoot, input.tool_use_id); @@ -236,4 +236,4 @@ export function runPostToolUse( registerInIndex(projectRoot, s.slug, s.title); deleteSidecar(projectRoot, input.tool_use_id); return null; -} \ No newline at end of file +} diff --git a/packages/cli/src/main.ts b/packages/cli/src/main.ts index a4028f6..0dca30f 100644 --- a/packages/cli/src/main.ts +++ b/packages/cli/src/main.ts @@ -5,6 +5,7 @@ import { runGateCommand } from "./commands/gate.js"; import { runGraph } from "./commands/graph.js"; import { runSearch } from "./commands/search.js"; import { runConsultAdd } from "./commands/consult.js"; +import { parseCheckArgs, runCheck, CHECK_FAILED_TO_RUN } from "./commands/check.js"; import { today } from "./date.js"; /** @@ -23,6 +24,7 @@ Usage: ow init [--language ] [--name ] scaffold a project and install the gate ow write [--content | --file ] write a page through the gate (no-hook path) ow gate pre|post the hook handlers (read JSON on stdin) + ow check [--json] [--errors-only] the integrity checks; exit 2 means errors ow graph [superseded|orphans|index] structural queries, as JSON ow search lexical search over the wiki, as JSON ow consult add add a read-only consult of another project @@ -85,6 +87,22 @@ export async function main(argv: string[], projectRoot: string = process.cwd()): return runGateCommand(kind); } + case "check": { + try { + const { stdout, code } = runCheck(projectRoot, parseCheckArgs(argv.slice(1))); + process.stdout.write(stdout); + return code; + } catch (err) { + // Exit 1 is "the check could not run", which is a different thing from + // "the check ran and found something" (exit 2). A CI job acts on the + // difference. + process.stderr.write( + `ow check could not run: ${err instanceof Error ? err.message : String(err)}\n`, + ); + return CHECK_FAILED_TO_RUN; + } + } + case "graph": { process.stdout.write(runGraph(projectRoot, argv[1]) + "\n"); return 0; diff --git a/packages/cli/tests/check.spec.ts b/packages/cli/tests/check.spec.ts new file mode 100644 index 0000000..b837c4e --- /dev/null +++ b/packages/cli/tests/check.spec.ts @@ -0,0 +1,175 @@ +import { describe, expect, it, beforeEach, afterEach, vi } from "vitest"; +import { mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { + CHECK_FAILED_TO_RUN, + CHECK_FOUND, + CHECK_OK, + parseCheckArgs, + runCheck, +} from "../src/commands/check.js"; +import { main } from "../src/main.js"; + +function tempProject(): string { + const root = realpathSync(mkdtempSync(join(tmpdir(), "ow-cli-check-"))); + mkdirSync(join(root, "wiki"), { recursive: true }); + mkdirSync(join(root, "raw"), { recursive: true }); + writeFileSync(join(root, "wiki", "index.md"), "# Index\n", "utf8"); + writeFileSync(join(root, "wiki", "changelog.md"), "# Changelog\n", "utf8"); + return root; +} + +function page(root: string, relPath: string, body = ""): void { + const slug = relPath.split("/").pop()!.replace(/\.md$/, ""); + const file = join(root, "wiki", relPath); + mkdirSync(dirname(file), { recursive: true }); + writeFileSync( + file, + `---\nid: topic:${slug}\ntype: topic\ntitle: ${slug}\nstatus: active\naliases: []\n` + + `updated: 2026-08-01\nsources: []\nsuperseded-by: ""\n---\n${body}`, + "utf8", + ); +} + +function clean(root: string): void { + page(root, "fenix.md"); + writeFileSync(join(root, "wiki", "index.md"), "# Index\n\n- [[fenix]]\n", "utf8"); + writeFileSync( + join(root, "wiki", "changelog.md"), + "# Changelog\n\n## 2026-08-01\n\n- Created [[fenix]].\n", + "utf8", + ); +} + +describe("ow check (7.7)", () => { + let root: string; + beforeEach(() => (root = tempProject())); + afterEach(() => rmSync(root, { recursive: true, force: true })); + + describe("parseCheckArgs", () => { + it("reports warnings by default", () => { + // A source nobody cites is precisely the thing that disappears on its + // own; hiding it behind a flag would defeat the check. + expect(parseCheckArgs([])).toEqual({ json: false, warnings: true }); + }); + + it("takes --json and --errors-only", () => { + expect(parseCheckArgs(["--json"]).json).toBe(true); + expect(parseCheckArgs(["--errors-only"]).warnings).toBe(false); + }); + }); + + describe("runCheck", () => { + it("exits 0 and says so when there is nothing wrong", () => { + clean(root); + const { stdout, code } = runCheck(root, { json: false, warnings: true }); + expect(code).toBe(CHECK_OK); + expect(stdout).toContain("no findings"); + }); + + it("exits 2 when it found an error", () => { + page(root, "fenix.md", "See [[nowhere]]."); + const { stdout, code } = runCheck(root, { json: false, warnings: true }); + expect(code).toBe(CHECK_FOUND); + expect(stdout).toContain("wikilink.broken"); + }); + + it("prints the correction path for every finding", () => { + page(root, "fenix.md", "See [[nowhere]]."); + const { stdout } = runCheck(root, { json: false, warnings: true }); + // A refusal a reader cannot act on becomes an attempt they repeat. + expect(stdout).toContain("fix:"); + }); + + it("stays green on warnings alone, so an unused source does not fail CI", () => { + clean(root); + const dir = join(root, "raw", "unused.md"); + mkdirSync(dir, { recursive: true }); + writeFileSync( + join(dir, "manifest.json"), + JSON.stringify({ id: "unused.md", title: "Unused", kind: "file", original: "unused.md" }), + "utf8", + ); + + const { stdout, code } = runCheck(root, { json: false, warnings: true }); + expect(stdout).toContain("source.uncited"); + expect(code).toBe(CHECK_OK); + }); + + it("drops warnings under --errors-only", () => { + clean(root); + const dir = join(root, "raw", "unused.md"); + mkdirSync(dir, { recursive: true }); + writeFileSync( + join(dir, "manifest.json"), + JSON.stringify({ id: "unused.md", title: "Unused", kind: "file", original: "unused.md" }), + "utf8", + ); + + const { stdout } = runCheck(root, { json: false, warnings: false }); + expect(stdout).not.toContain("source.uncited"); + }); + + it("prints JSON a harness can read", () => { + page(root, "fenix.md", "See [[nowhere]]."); + const { stdout } = runCheck(root, { json: true, warnings: true }); + const parsed = JSON.parse(stdout); + expect(Array.isArray(parsed.findings)).toBe(true); + expect(parsed.findings[0]).toHaveProperty("code"); + expect(parsed.findings[0]).toHaveProperty("fix"); + expect(parsed.pages).toBe(1); + }); + }); + + describe("through the dispatch", () => { + it("runs as `ow check` and returns its exit code", async () => { + const write = vi.spyOn(process.stdout, "write").mockReturnValue(true); + try { + page(root, "fenix.md", "See [[nowhere]]."); + expect(await main(["check"], root)).toBe(CHECK_FOUND); + expect(write.mock.calls.join("")).toContain("wikilink.broken"); + } finally { + write.mockRestore(); + } + }); + + it("returns 0 on a clean project", async () => { + const write = vi.spyOn(process.stdout, "write").mockReturnValue(true); + try { + clean(root); + expect(await main(["check"], root)).toBe(CHECK_OK); + } finally { + write.mockRestore(); + } + }); + + it("returns 1, not 2, when the check could not run at all", async () => { + // A CI job acts on the difference: 2 is "fix your wiki", 1 is "the check + // itself is broken". Collapsing them sends people to the wrong place. + const write = vi.spyOn(process.stdout, "write").mockReturnValue(true); + const err = vi.spyOn(process.stderr, "write").mockReturnValue(true); + try { + // `wiki/` replaced by a file: the walk cannot run. + rmSync(join(root, "wiki"), { recursive: true, force: true }); + writeFileSync(join(root, "wiki"), "not a directory", "utf8"); + + expect(await main(["check"], root)).toBe(CHECK_FAILED_TO_RUN); + expect(err.mock.calls.join("")).toContain("could not run"); + } finally { + write.mockRestore(); + err.mockRestore(); + } + }); + + it("is listed in the usage text", async () => { + const write = vi.spyOn(process.stdout, "write").mockReturnValue(true); + try { + await main(["help"], root); + expect(write.mock.calls.join("")).toContain("ow check"); + } finally { + write.mockRestore(); + } + }); + }); +}); diff --git a/packages/cli/tests/hooks.spec.ts b/packages/cli/tests/hooks.spec.ts index c0d4e8a..d691082 100644 --- a/packages/cli/tests/hooks.spec.ts +++ b/packages/cli/tests/hooks.spec.ts @@ -37,9 +37,11 @@ describe("detectShellWrite (9.5)", () => { expect(detectShellWrite(`echo hi > wiki/fenix.md`, "/p")).toBe("wiki/fenix.md"); expect(detectShellWrite(`cat x >> wiki/fenix.md`, "/p")).toBe("wiki/fenix.md"); }); - it("flags cp/mv/tee/sed -i into wiki/ or codewiki/", () => { + it("flags cp/mv/tee/sed -i into wiki/, codewiki included", () => { expect(detectShellWrite(`cp /tmp/x wiki/fenix.md`, "/p")).toBe("wiki/fenix.md"); - expect(detectShellWrite(`mv /tmp/x codewiki/dispatch.md`, "/p")).toBe("codewiki/dispatch.md"); + expect(detectShellWrite(`mv /tmp/x wiki/codewiki/dispatch.md`, "/p")).toBe( + "wiki/codewiki/dispatch.md", + ); expect(detectShellWrite(`echo hi | tee wiki/fenix.md`, "/p")).toBe("wiki/fenix.md"); expect(detectShellWrite(`sed -i 's/a/b/' wiki/fenix.md`, "/p")).toBe("wiki/fenix.md"); }); @@ -54,7 +56,9 @@ describe("detectShellWrite (9.5)", () => { }); it("flags a shell write that uses Windows backslash separators", () => { expect(detectShellWrite(`echo hi > wiki\\fenix.md`, "/p")).toBe("wiki/fenix.md"); - expect(detectShellWrite(`cp /tmp/x codewiki\\dispatch.md`, "/p")).toBe("codewiki/dispatch.md"); + expect(detectShellWrite(`cp /tmp/x wiki\\codewiki\\dispatch.md`, "/p")).toBe( + "wiki/codewiki/dispatch.md", + ); }); }); @@ -189,7 +193,10 @@ describe("runPostToolUse — recording (9.5, 5.6, 5.7)", () => { it("records a modified page when the page already existed", () => { writeFileSync(join(root, "wiki", "fenix.md"), page(GOOD_FM.replace('""', "2026-08-01"))); preWrite(root, "wiki/fenix.md", page(GOOD_FM, "edited.\n"), "tu_m"); - writeFileSync(join(root, "wiki", "fenix.md"), page(GOOD_FM.replace('""', "2026-08-01"), "edited.\n")); + writeFileSync( + join(root, "wiki", "fenix.md"), + page(GOOD_FM.replace('""', "2026-08-01"), "edited.\n"), + ); runPostToolUse( { hook_event_name: "PostToolUse", @@ -220,4 +227,4 @@ describe("runPostToolUse — recording (9.5, 5.6, 5.7)", () => { ); expect(existsSync(join(root, "wiki", "log.md"))).toBe(false); }); -}); \ No newline at end of file +}); diff --git a/packages/cli/tests/queries.spec.ts b/packages/cli/tests/queries.spec.ts index f80810c..b7fdb19 100644 --- a/packages/cli/tests/queries.spec.ts +++ b/packages/cli/tests/queries.spec.ts @@ -25,7 +25,11 @@ function tempProject(): string { } function writePageFile(root: string, slug: string, frontmatter: string[], body: string): void { - writeFileSync(join(root, "wiki", `${slug}.md`), `---\n${frontmatter.join("\n")}\n---\n${body}`, "utf8"); + writeFileSync( + join(root, "wiki", `${slug}.md`), + `---\n${frontmatter.join("\n")}\n---\n${body}`, + "utf8", + ); } function active(slug: string, title: string): string[] { @@ -122,7 +126,12 @@ describe("ow search (9.12)", () => { let root: string; beforeEach(() => { root = tempProject(); - writePageFile(root, "fenix", active("fenix", "Fenix"), "Fenix is a rebuild. A rebuild twice over.\n"); + writePageFile( + root, + "fenix", + active("fenix", "Fenix"), + "Fenix is a rebuild. A rebuild twice over.\n", + ); }); afterEach(() => rmSync(root, { recursive: true, force: true })); @@ -150,7 +159,12 @@ describe("ow write — the pages the gate does not own", () => { it("writes a non-entity page through without validating it against the page schema", () => { // `index.md` is the agent's to curate, not the schema's to validate. - const result = runWrite(root, join(root, "wiki", "index.md"), "# Index\n\nCurated by hand.\n", DATE); + const result = runWrite( + root, + join(root, "wiki", "index.md"), + "# Index\n\nCurated by hand.\n", + DATE, + ); expect(result.ok).toBe(true); expect(readFileSync(join(root, "wiki", "index.md"), "utf8")).toContain("Curated by hand."); // Passed through, so it is not announced in the changelog as an entity page. @@ -175,3 +189,37 @@ describe("ow write — the pages the gate does not own", () => { expect(relativePath(root, "wiki/fenix.md")).toBe("wiki/fenix.md"); }); }); + +describe("graph and search over a page filed under a subdirectory (adr:0016)", () => { + let root: string; + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "ow-nested-q-")); + mkdirSync(join(root, "wiki", "topics"), { recursive: true }); + writeFileSync( + join(root, "wiki", "topics", "checkout.md"), + "---\nid: topic:checkout\ntype: topic\ntitle: Checkout\nstatus: superseded\n" + + 'aliases: []\nupdated: 2026-08-01\nsources: []\nsuperseded-by: "topic:pay"\n---\nBody about payments.\n', + "utf8", + ); + writeFileSync(join(root, "wiki", "index.md"), "# Index\n\n- [[checkout]]\n", "utf8"); + }); + afterEach(() => rmSync(root, { recursive: true, force: true })); + + it("walks supersession without throwing ENOENT", () => { + // Both commands resolved a slug as wiki/.md, so any project filing a + // page the way the plan's layout describes got a stack, not a sentence. + const parsed = JSON.parse(runGraph(root, "superseded")); + expect(parsed).toEqual([ + { slug: "checkout", "superseded-by": "topic:pay", updated: "2026-08-01" }, + ]); + }); + + it("lists the page in the default graph", () => { + expect(JSON.parse(runGraph(root, undefined)).pages).toEqual(["checkout"]); + }); + + it("searches its text", () => { + const results = JSON.parse(runSearch(root, "payments")); + expect(results).toEqual([{ slug: "checkout", title: "Checkout", matches: 1 }]); + }); +}); diff --git a/packages/mcp/src/tools.ts b/packages/mcp/src/tools.ts index e2b4305..1daf6fc 100644 --- a/packages/mcp/src/tools.ts +++ b/packages/mcp/src/tools.ts @@ -11,6 +11,7 @@ import { listSources, type PageFrontmatter, type SourceManifest, + pagePath as lookupPagePath, } from "@open-wiki/access/read"; /** @@ -103,13 +104,25 @@ export function readSourceText(projectRoot: string, id: string): string { return readFileSync(file, "utf8"); } -/** The confined path of a page file; throws if the slug escapes `wiki/`. */ +/** + * The confined path of a page file; throws if the slug escapes `wiki/`. + * + * A page is its slug wherever it sits under `wiki/` + * (`adr:0016-a-page-is-its-slug-wherever-it-sits`), so the file is looked up + * rather than assumed at the top level. Assuming it meant a page filed as + * `wiki/topics/checkout.md` was listed by `ow_index` and then served as "no + * page \"checkout\" under wiki/" — and its frontmatter read as `null`, so the + * index reported `type: unknown, status: active` for a page that might be + * superseded. That is worse than absent: it is confidently wrong. + */ function pagePath(projectRoot: string, slug: string): string { + const wikiDir = join(projectRoot, "wiki"); + const found = lookupPagePath(projectRoot, slug); // Confine to the `wiki/` directory, not just the project: a slug like // `../README` would otherwise resolve to `/README.md`, which is inside - // the project but is not a wiki page. The server serves `wiki/` only. - const wikiDir = join(projectRoot, "wiki"); - return assertWithin(wikiDir, join(wikiDir, `${slug}.md`)); + // the project but is not a wiki page. The server serves `wiki/` only. The + // fallback keeps "no such page" reachable for a slug that matches nothing. + return assertWithin(wikiDir, join(projectRoot, found ?? `wiki/${slug}.md`)); } /** The confined path of a source's `text.md`; throws if the id escapes `raw/`. */ @@ -127,4 +140,4 @@ function readPageFrontmatter(projectRoot: string, slug: string): PageFrontmatter return block.frontmatter as PageFrontmatter; } -export { OutsideProjectError }; \ No newline at end of file +export { OutsideProjectError }; diff --git a/packages/mcp/tests/mcp.spec.ts b/packages/mcp/tests/mcp.spec.ts index 36e3a6d..1998c9c 100644 --- a/packages/mcp/tests/mcp.spec.ts +++ b/packages/mcp/tests/mcp.spec.ts @@ -33,7 +33,12 @@ function writePage( frontmatter: Record, body: string, ): void { - const fm = "---\n" + Object.entries(frontmatter).map(([k, v]) => `${k}: ${v}`).join("\n") + "\n---\n\n"; + const fm = + "---\n" + + Object.entries(frontmatter) + .map(([k, v]) => `${k}: ${v}`) + .join("\n") + + "\n---\n\n"; writeFileSync(join(root, "wiki", `${slug}.md`), fm + body, "utf8"); } @@ -171,7 +176,12 @@ describe("MCP read tools — path confinement (9.9)", () => { mkdirSync(dir, { recursive: true }); writeFileSync( join(dir, "manifest.json"), - JSON.stringify({ id: "pending.pdf", title: "Pending", kind: "file", original: "pending.pdf" }) + "\n", + JSON.stringify({ + id: "pending.pdf", + title: "Pending", + kind: "file", + original: "pending.pdf", + }) + "\n", "utf8", ); const state = listSourcesState(root).find((s) => s.id === "pending.pdf"); @@ -212,7 +222,9 @@ describe("MCP read-only boundary (9.9)", () => { // Relative imports stay inside this package, so they cannot reach the write // path; everything else must be one of the read-only roots below. if (spec.startsWith("./") || spec.startsWith("../")) return true; - return ALLOWED.some((a) => spec === a || (a.endsWith(":") ? spec.startsWith(a) : spec.startsWith(a))); + return ALLOWED.some( + (a) => spec === a || (a.endsWith(":") ? spec.startsWith(a) : spec.startsWith(a)), + ); } function srcFiles(dir: string): string[] { @@ -243,14 +255,7 @@ describe("MCP read-only boundary (9.9)", () => { // If the read barrel leaked a write symbol, importing it would pull the // write path after all. Belt and braces: assert the known write symbols are // absent from the barrel's text. - const barrel = join( - __dirname, - "..", - "..", - "access", - "src", - "read.ts", - ); + const barrel = join(__dirname, "..", "..", "access", "src", "read.ts"); expect(existsSync(barrel)).toBe(true); const text = readFileSync(barrel, "utf8"); const writeExports = [ @@ -271,7 +276,10 @@ describe("MCP read-only boundary (9.9)", () => { "writeSourceText", ]; for (const w of writeExports) { - expect(new RegExp(`\\b${w}\\b`).test(text), `read barrel re-exports write symbol "${w}"`).toBe(false); + expect( + new RegExp(`\\b${w}\\b`).test(text), + `read barrel re-exports write symbol "${w}"`, + ).toBe(false); } }); }); @@ -295,4 +303,37 @@ describe("parseMcpArgs (9.7)", () => { it("refuses to start without a project name", () => { expect(() => parseMcpArgs(["--read-only"])).toThrow(/--project/); }); -}); \ No newline at end of file +}); + +describe("a page filed under a subdirectory (adr:0016)", () => { + let root: string; + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "ow-mcp-nested-")); + mkdirSync(join(root, "wiki", "topics"), { recursive: true }); + writeFileSync( + join(root, "wiki", "topics", "checkout.md"), + "---\nid: topic:checkout\ntype: topic\ntitle: Checkout\nstatus: superseded\n" + + 'aliases: []\nupdated: 2026-08-01\nsources: []\nsuperseded-by: "topic:pay"\n---\nBody.\n', + "utf8", + ); + writeFileSync(join(root, "wiki", "index.md"), "# Index\n\n- [[checkout]]\n", "utf8"); + }); + afterEach(() => rmSync(root, { recursive: true, force: true })); + + it("serves the page rather than reporting it missing", () => { + // It was listed by ow_index and then served as `no page "checkout" under + // wiki/` — the index and the reader disagreeing about the same project. + expect(readPageWhole(root, "checkout").content).toContain("Body."); + }); + + it("reads its frontmatter, so the index does not report it as unknown", () => { + // The frontmatter read degraded to null, so a superseded page came back as + // `type: unknown, status: active` — confidently wrong rather than absent. + const entry = indexStructure(root).find((p) => p.slug === "checkout"); + expect(entry).toMatchObject({ title: "Checkout", type: "topic", status: "superseded" }); + }); + + it("still refuses a slug that escapes wiki/", () => { + expect(() => readPageWhole(root, "../README")).toThrow(); + }); +}); diff --git a/plans/open-wiki.md b/plans/open-wiki.md index 878226c..b1d17d4 100644 --- a/plans/open-wiki.md +++ b/plans/open-wiki.md @@ -176,7 +176,8 @@ which is 9.5's problem. ## 6 — Source flow -- [ ] 6.1 (Unit) Model each source's state — received, text ready, cited on a page — persisted and resumable +- [x] 6.1 (Unit) Model each source's state — received, text ready, cited on a page — persisted and resumable + - **Derived, not persisted.** The filesystem already is both: `manifest.json` says received, `text.md` says the text is ready, the pages say what is cited, `journal.json` says how far a transcription got. A state file beside those would be a second record of one fact, and the copy is the one that goes stale — the rule this plan applies to its own checklists and to the wiki's index. So a crash loses nothing and there is nothing to reconcile: the next read observes the same directory and reaches the same answer, which is what "resumable" was asking for. - [ ] 6.2 (Unit) Sources screen: one row per source with its current state, what is missing, and the error when it stopped - [ ] 6.3 (Unit) A transcribe button on a stopped recording, with per-chunk progress and the option to redo only what failed - [ ] 6.4 (Unit) Show, for a source, which pages cite it, and navigate from there to the page @@ -189,14 +190,15 @@ which is 9.5's problem. With the agent writing through the filesystem, this stops being hygiene and becomes the net of record. -- [ ] 7.1 (Unit) Report broken wikilinks and orphan pages -- [ ] 7.2 (Unit) Report a desynchronised changelog and a source never cited -- [ ] 7.3 (Unit) Report a provenance link that does not resolve to an existing source or instant -- [ ] 7.4 (Unit) Report a synonym used where the project has a canonical term -- [ ] 7.5 (Unit) Report a codewiki citation that no longer resolves or runs past the end of its file — the check that comes with scaffolding codewiki, per `adr:0015-the-convention-ships-as-skills` - - Design gap a review surfaced, to settle here: the gate accepts `codewiki/*.md` (top-level) but `listEntityPages`/`findOrphans`/`indexStructure` read only top-level `wiki/`, so a codewiki page is accepted by the gate yet invisible to the index, the orphans check, and `ow_index`. The codewiki skill prose also says `wiki/codewiki/`, which the gate's `wiki/` branch would treat as a wiki page named for its basename. Wiring codewiki into the store is a slug/index model decision (a `codewiki/` slug breaks the `^[a-z0-9-]+$` slug regex), deferred to group 7 with the rest of the integrity checks; the MVP closes on `wiki/` entity pages and does not exercise codewiki. +- [x] 7.1 (Unit) Report broken wikilinks and orphan pages +- [x] 7.2 (Unit) Report a desynchronised changelog and a source never cited +- [x] 7.3 (Unit) Report a provenance link that does not resolve to an existing source or instant +- [x] 7.4 (Unit) Report a synonym used where the project has a canonical term +- [x] 7.5 (Unit) Report a codewiki citation that no longer resolves or runs past the end of its file — the check that comes with scaffolding codewiki, per `adr:0015-the-convention-ships-as-skills` + - **Settled** — `adr:0016-a-page-is-its-slug-wherever-it-sits`. The gap was wider than it looked: `listEntityPages` read only the *top level* of `wiki/`, so `wiki/projects/`, `wiki/people/` and `wiki/topics/` — the layout this plan's own diagram describes — were invisible too, not just codewiki. A page is now addressed by its slug wherever it sits under `wiki/`; a folder is organisation and a link is a name, which is what makes `[[wikilink]]` work in the first place. Slug uniqueness is the one rule that model needs, and it is a finding (`page.duplicate-slug`) rather than something resolved by silently picking one. Codewiki lives at `wiki/codewiki/`; a top-level `codewiki/` is no longer gated and is reported as misplaced. - [ ] 7.6 (Unit) Expose the checks in the UI, with the correction path described per finding -- [ ] 7.7 (Unit) Expose the same checks as `ow check`, so an agent and a CI job can run them without the application + - Deferred to group 8, which is where the UI is. The findings already carry the correction path — `fix` on every one — so this is rendering, not new checking. +- [x] 7.7 (Unit) Expose the same checks as `ow check`, so an agent and a CI job can run them without the application ## 8 — Application From 29610372358fb09f844d6ed87615a7a7393e243a Mon Sep 17 00:00:00 2001 From: Claude Code Date: Sat, 1 Aug 2026 10:19:04 +0000 Subject: [PATCH 2/2] fix(access): close the CodeRabbit findings on the checks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `listPages` folds case on `.md`, as the gate does. `gatedPageRel` lowercases before testing the extension, so it validates and accepts `wiki/fenix.MD` — and matching case-sensitively here meant that page was accepted by the gate and then invisible to the index, the orphan check, `ow graph` and MCP. That is the failure this addressing model exists to end, reappearing one level down. - A citation inside a fenced code block is an example, not a citation. A codewiki page documenting the citation form — which the skill's own prose does — failed `ow check` for its own sample. - The stray-codewiki walk no longer follows symlinked directories out of the project; `recursive: true` does. - `listSourceStates` keeps going when one manifest will not parse or a source vanishes mid-listing. A sources screen showing nothing because of one bad directory is worse than one showing the other nineteen. - `listSources` reads dirents rather than stat'ing each entry, so a dangling symlink under `raw/` no longer throws ENOENT and aborts the whole run — reachable now that `checkRecords` calls it. - `sourceState` confines against `raw/`, not merely the project: an id like `../wiki` stays inside the project and is still not a source. The test that claimed to cover this only exercised an id that left the project entirely, which proves less. - `safe()` on the remaining interpolated values, in `checkProvenance` and `checkSchema`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_016iMM93Wk43o44V5J2AxPgL --- packages/access/src/check/checks.ts | 44 ++++++++++++++++++--- packages/access/src/sources/manifest.ts | 15 ++++--- packages/access/src/sources/state.ts | 23 +++++++++-- packages/access/src/store/index.ts | 14 +++++-- packages/access/tests/check.spec.ts | 31 +++++++++++++++ packages/access/tests/sources-state.spec.ts | 16 +++++++- 6 files changed, 123 insertions(+), 20 deletions(-) diff --git a/packages/access/src/check/checks.ts b/packages/access/src/check/checks.ts index b15852b..742b424 100644 --- a/packages/access/src/check/checks.ts +++ b/packages/access/src/check/checks.ts @@ -61,6 +61,24 @@ function lineInPage(page: LoadedPage, needle: string): number | undefined { return before.split("\n").length - 1 + page.body.slice(0, at).split("\n").length; } +/** + * Blank out fenced code, keeping the line count intact so a reported line still + * points where the reader expects. + */ +function withoutFences(body: string): string { + let inFence = false; + return body + .split("\n") + .map((line) => { + if (/^\s*```/.test(line)) { + inFence = !inFence; + return ""; + } + return inFence ? "" : line; + }) + .join("\n"); +} + /** The 1-based line `needle` first appears on in `text`, or undefined. */ function lineOf(text: string, needle: string): number | undefined { const at = text.indexOf(needle); @@ -206,7 +224,7 @@ export function checkProvenance(projectRoot: string, pages: LoadedPage[]): Findi code: "provenance.unresolved", severity: "error", page: page.path, - message: `${page.path}: ${issue.reason}`, + message: `${page.path}: ${safe(issue.reason)}`, fix: "Upload the source it names, or correct the citation. A citation that opens nothing is worse than none — if the source cannot be produced, the claim comes out with it.", }); } @@ -410,7 +428,11 @@ export function checkCodewiki(projectRoot: string, pages: LoadedPage[]): Finding }; for (const page of pages.filter((p) => p.codewiki)) { - for (const match of page.body.matchAll(CODEWIKI_CITATION)) { + // Fences are examples, not citations. A codewiki page documenting the + // citation form — which the skill's own prose does — would otherwise fail + // `ow check` for the sample inside its fence. The section scan below + // already ignores fences, and so does `checkVocabulary`. + for (const match of withoutFences(page.body).matchAll(CODEWIKI_CITATION)) { const [whole, target, startText, endText] = match; const start = Number(startText); const end = endText === undefined ? start : Number(endText); @@ -506,9 +528,19 @@ export function checkCodewiki(projectRoot: string, pages: LoadedPage[]): Finding if (existsSync(stray) && statSync(stray).isDirectory()) { // Recursive, like the model three functions above: a stray // `codewiki/area/x.md` is exactly as misplaced as `codewiki/x.md`. - const strayPages = readdirSync(stray, { recursive: true, withFileTypes: true }).filter( - (entry) => entry.isFile() && entry.name.endsWith(".md"), - ); + // `recursive: true` follows symlinked directories, which would walk out of + // the project. Same rule as `listPages`: links are skipped, not followed. + const strayPages: string[] = []; + const walkStray = (dir: string): void => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.isSymbolicLink()) continue; + if (entry.isDirectory()) walkStray(join(dir, entry.name)); + else if (entry.isFile() && entry.name.toLowerCase().endsWith(".md")) { + strayPages.push(entry.name); + } + } + }; + walkStray(stray); if (strayPages.length > 0) { findings.push({ code: "codewiki.misplaced", @@ -533,7 +565,7 @@ export function checkSchema(pages: LoadedPage[]): Finding[] { code: "page.invalid", severity: "error", page: page.path, - message: `${page.path}: ${issue.field ? `${issue.field}: ` : ""}${issue.reason}`, + message: `${page.path}: ${safe(issue.field ? `${issue.field}: ${issue.reason}` : issue.reason)}`, fix: "Correct the frontmatter. `ow write` applies the same rules, and the gate would have refused this write — it arrived some other way.", }); } diff --git a/packages/access/src/sources/manifest.ts b/packages/access/src/sources/manifest.ts index 0824826..81db50c 100644 --- a/packages/access/src/sources/manifest.ts +++ b/packages/access/src/sources/manifest.ts @@ -1,4 +1,4 @@ -import { existsSync, readFileSync, readdirSync, statSync } from "node:fs"; +import { existsSync, readFileSync, readdirSync } from "node:fs"; import { join } from "node:path"; import { assertWithin, OutsideProjectError } from "../paths.js"; @@ -89,12 +89,15 @@ export function listSources(projectRoot: string): string[] { const raw = join(projectRoot, "raw"); if (!existsSync(raw)) return []; const ids: string[] = []; - for (const entry of readdirSync(raw)) { - if (entry === INBOX) continue; - const dir = join(raw, entry); - if (!statSync(dir).isDirectory()) continue; + for (const entry of readdirSync(raw, { withFileTypes: true })) { + if (entry.name === INBOX) continue; + // `withFileTypes` describes the entry itself, so a dangling symlink is + // reported rather than stat'd. `statSync` on one throws ENOENT, which used + // to abort the whole listing — and `ow check` with it. + if (!entry.isDirectory()) continue; + const dir = join(raw, entry.name); if (!existsSync(join(dir, "manifest.json"))) continue; - ids.push(entry); + ids.push(entry.name); } return ids; } diff --git a/packages/access/src/sources/state.ts b/packages/access/src/sources/state.ts index 12ef93a..1406adf 100644 --- a/packages/access/src/sources/state.ts +++ b/packages/access/src/sources/state.ts @@ -74,7 +74,12 @@ export function sourceState( citedBy: readonly string[] = [], ): SourceState { const manifest = readManifest(projectRoot, id); - const dir = assertWithin(projectRoot, join(projectRoot, "raw", id)); + // Confined against `raw/`, not merely the project: an id like `../wiki` stays + // inside the project and is still not a source. `readManifest` already roots + // at `raw/`; this has to agree with it or the two disagree about what an id + // may name. + const rawDir = join(projectRoot, "raw"); + const dir = assertWithin(rawDir, join(rawDir, id)); const textReady = existsSync(join(dir, "text.md")); const journal = readJournal(projectRoot, id); @@ -117,7 +122,17 @@ export function listSourceStates( projectRoot: string, citations: ReadonlyMap = new Map(), ): SourceState[] { - return listSources(projectRoot) - .map((id) => sourceState(projectRoot, id, citations.get(id) ?? [])) - .sort((a, b) => a.id.localeCompare(b.id)); + const states: SourceState[] = []; + for (const id of listSources(projectRoot)) { + try { + states.push(sourceState(projectRoot, id, citations.get(id) ?? [])); + } catch { + // `listSources` only checks that `manifest.json` exists. One that will + // not parse, or a source deleted between the listing and the read, must + // not take the other nineteen with it — a sources screen showing nothing + // because of one bad directory is worse than one showing nineteen. + continue; + } + } + return states.sort((a, b) => a.id.localeCompare(b.id)); } diff --git a/packages/access/src/store/index.ts b/packages/access/src/store/index.ts index 5b00c4d..c6a6cb6 100644 --- a/packages/access/src/store/index.ts +++ b/packages/access/src/store/index.ts @@ -70,18 +70,26 @@ export function listPages(projectRoot: string): PageRef[] { walk(full); continue; } - if (!entry.isFile() || !entry.name.endsWith(".md")) continue; + // Case-folded, like the gate. `gatedPageRel` lowercases before testing + // for `.md`, so it validates and accepts `wiki/fenix.MD` — and matching + // case-sensitively here meant that page was accepted by the gate and then + // invisible to the index, the orphan check, `ow graph` and MCP. That is + // the exact failure this addressing model exists to end, one level down. + if (!entry.isFile() || !entry.name.toLowerCase().endsWith(".md")) continue; const rel = relative(wiki, full).split(sep).join(posix.sep); // Only the three at the very top are the wiki's own; a file called // `index.md` inside `wiki/topics/` is an ordinary page named "index". - if (!rel.includes(posix.sep) && (NON_ENTITY_PAGES as readonly string[]).includes(rel)) { + if ( + !rel.includes(posix.sep) && + (NON_ENTITY_PAGES as readonly string[]).includes(rel.toLowerCase()) + ) { continue; } pages.push({ slug: entry.name.slice(0, -3), path: `wiki/${rel}`, - codewiki: rel.startsWith(`${CODEWIKI_DIR}/`), + codewiki: rel.toLowerCase().startsWith(`${CODEWIKI_DIR}/`), }); } }; diff --git a/packages/access/tests/check.spec.ts b/packages/access/tests/check.spec.ts index 64b45e8..acaab9d 100644 --- a/packages/access/tests/check.spec.ts +++ b/packages/access/tests/check.spec.ts @@ -156,6 +156,23 @@ describe("the integrity checks (group 7)", () => { expect(checkProject(root).pages).toBe(0); }); + it("sees a page whose extension is upper case, as the gate does", () => { + // The gate folds case before testing for `.md`, so it validates and + // accepts `wiki/fenix.MD`. Matching case-sensitively here meant that page + // was accepted and then invisible to everything downstream. + writeFileSync( + join(root, "wiki", "shouty.MD"), + "---\nid: topic:shouty\ntype: topic\ntitle: shouty\nstatus: active\naliases: []\n" + + 'updated: 2026-08-01\nsources: []\nsuperseded-by: ""\n---\n', + "utf8", + ); + index(root, ["shouty"]); + changelog(root, ["shouty"]); + + expect(checkProject(root).pages).toBe(1); + expect(codes(checkProject(root).findings)).not.toContain("page.orphan"); + }); + it("treats a nested file called index.md as an ordinary page", () => { page(root, "topics/index.md"); index(root, ["index"]); @@ -458,6 +475,20 @@ describe("the integrity checks (group 7)", () => { expect(codes(checkProject(root).findings)).toContain("codewiki.misplaced"); }); + it("does not resolve a citation shown inside a fenced code block", () => { + // A codewiki page documenting the citation form — which the skill's own + // prose does — would otherwise fail `ow check` for its own example. + page( + root, + "codewiki/dispatch.md", + "## Form\n\n```\n[path/to/file.ts:12-40]()\n```\n\nAnd prose.\n", + ); + index(root, ["dispatch"]); + changelog(root, ["dispatch"]); + + expect(codes(checkProject(root).findings)).not.toContain("codewiki.citation-unresolved"); + }); + it("reports a codewiki/ at the project root as misplaced", () => { // Outside wiki/ it is not part of the wiki: nothing indexes it, nothing // links it, and no write to it is validated. diff --git a/packages/access/tests/sources-state.spec.ts b/packages/access/tests/sources-state.spec.ts index 86f2a13..17ad2ba 100644 --- a/packages/access/tests/sources-state.spec.ts +++ b/packages/access/tests/sources-state.spec.ts @@ -115,7 +115,11 @@ describe("source state (6.1)", () => { expect(() => sourceState(root, "nothing")).toThrow(MissingSourceError); }); - it("refuses an id that escapes raw/", () => { + it("refuses an id that escapes raw/ but stays inside the project", () => { + // The interesting case is a single `..`: it leaves `raw/` while staying + // in the project, so confining against the project root would let it + // through. `../../elsewhere` leaves the project too and proves less. + expect(() => sourceState(root, "../wiki")).toThrow(); expect(() => sourceState(root, "../../elsewhere")).toThrow(); }); }); @@ -142,6 +146,16 @@ describe("source state (6.1)", () => { expect(listSourceStates(root).map((s) => s.id)).toEqual(["a.md"]); }); + it("keeps listing the rest when one manifest will not parse", () => { + // A sources screen showing nothing because of one bad directory is worse + // than one showing the other nineteen. + source(root, "good.md", { text: "g" }); + mkdirSync(join(root, "raw", "broken.md"), { recursive: true }); + writeFileSync(join(root, "raw", "broken.md", "manifest.json"), "{ not json", "utf8"); + + expect(listSourceStates(root).map((s) => s.id)).toEqual(["good.md"]); + }); + it("returns nothing for a project with no sources", () => { expect(listSourceStates(root)).toEqual([]); });