Skip to content

docs(internal): doc-truth pipeline design record (doc-truth PR 5/5) #7381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thisisjoshford merged 3 commits into from
Aug 11, 2026
Merged

docs(internal): doc-truth pipeline design record (doc-truth PR 5/5) #7381

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1c49eff
docs(internal): record the doc-truth pipeline design (issue #7317)
thisisjoshford Aug 7, 2026
fe005d0
Merge remote-tracking branch 'origin/main' into docs/doc-truth-design
thisisjoshford Aug 7, 2026
496a3be
docs(internal): record review hardening in the doc-truth design
thisisjoshford Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
209 changes: 209 additions & 0 deletions docs/internal/plans/2026-08-07-doc-truth-pipeline.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
# Doc-Truth Verification Pipeline

**Date:** 2026-08-07
**Status:** Implemented (phases 1–4); follow-ups specced below
**Answers:** issue #7317 "Proposal: Doc-Truth Verification Pipeline"
**PRs:** #7375 (drift fixes) → #7376 (docs reference gate) → #7378 (doc-fact
contract tests), #7379 (docs-live + changelog), this document

## 1. Problem

Public Mintlify docs (`docs/`) drifted from shipped behavior, and nothing
caught it. Confirmed live cases at the time of writing:

- `docs/extensions/building-a-tool.md` taught the retired
`reborn.extension_manifest.v2` authoring shape (`[[host_api]]` /
`[capability_provider.tools]`), which the v3 parser hard-rejects, and never
mentioned `origin_gate_matrix` — the field whose absence broke
1.0-era extensions on newer binaries.
- `docs/api/responses.mdx` claimed `temperature` was rejected (the router
accepts 0.0–2.0 and forwards it), claimed `model` must be `"default"` (any
well-formed ≤256-byte name passes), claimed `max_output_tokens` is rejected
(accepted-and-ignored by DTO policy), and omitted the required `model`
field from every request example.
- `docs/channels/building-a-channel.mdx` instructed contributors to edit two
files that no longer exist.

Root cause, in two parts:

1. **No deterministic coupling** between doc claims and code — the repo has
41 architecture ratchet tests pinning facts in `crates/`, and zero aimed
at `docs/`.
2. **Deploy-target mismatch** — the Mintlify GitHub App deployed `docs/` on
every push to `main`, while binaries ship from `ironclaw-v*` tags, so
even perfectly accurate docs describe unreleased behavior between
releases.

## 2. Decisions of record

- **Single doc tree, no Mintlify versions.** Versioning would multiply every
page across versions × the two locales (`en` + `zh`) on a weekly release
train. Instead the site tracks the **latest stable release**, and older
releases are served by each git tag's preserved `docs/` tree
(`https://github.com/nearai/ironclaw/tree/ironclaw-vX.Y.Z/docs`).
- **`docs-live` deployment branch.** Release automation force-points
`refs/heads/docs-live` at each stable release commit; the Mintlify
dashboard deploys from that branch (one-time out-of-repo change). Details
and runbook: `docs/internal/weekly-release-strategy.md` § Docs
publication.
- **Gates are deterministic only.** Pass/fail comes from string/table/route
assertions, never an LLM judgment (refinement 1 of the issue). LLM
assistance is reserved for the *non-blocking* fix-PR generator
(follow-up, §5).
- **Catch drift at PR time.** The static gates run in Code Style on every
PR (refinement 2); the release-time checks are a backstop, not the
primary defense.
- **Human-curated changelog** (`docs/changelog.mdx`), one `<Update>` entry
per stable release, landed on `main` **before** the Monday cut so the
candidate branch inherits it, and enforced by the cut script. Writing the
entry only on the frozen release branch would satisfy that week's gate and
then vanish: the branch is never merged back, so the next candidate — cut
from `main` — would ship a changelog missing the previous release.

## 3. Architecture — three layers

### 3.1 PR-time static gates (Code Style)

| Gate | What it pins | Where |
| --- | --- | --- |
| `scripts/ci/check-guidance.py` (docs surface) | Every backticked repo path in published pages, the `zh/` mirror, and `docs/reborn/contracts/` resolves against `git ls-files`. Mintlify link targets are a different namespace and deliberately unchecked; dated archives (`docs/internal/`, non-contract `docs/reborn/`) are excluded as classes. MDX comment form of the suppress marker: `{/* check-guidance: path-ok */}`. | `fast-checks` job, `has_guidance` trigger (probes pinned in `ws12_workflow_contracts.py`) |
| `crates/app/ironclaw_cli/tests/docs_cli_reference.rs` | `docs/using/cli.mdx` ↔ the real binary's `--help`, both directions, subcommand granularity, alias-aware, row-count floor. | `cargo test -p ironclaw` |
| `crates/extensions/ironclaw_extension_registry/tests/docs_manifest_schema_version.rs` | Zero retired `reborn.extension_manifest.v2` literals in published pages (fences included); `building-a-tool.md` names `MANIFEST_SCHEMA_VERSION_V3` and documents `origin_gate_matrix`. | `cargo test -p ironclaw_extension_registry` |
| `crates/product/ironclaw_openai_compat/tests/docs_responses_contract.rs` | The `doc-fact:responses-request-policy` marker block in `docs/api/responses.mdx`, driven behaviorally through the real router — the marker's values parameterize the assertions. | `cargo test -p ironclaw_openai_compat` |
| `scripts/ci/docs_publication_boundary.py` (pre-existing) | Publication fence: every page published or fenced; `.mintignore` frozen. | `docs-publication-boundary` job |

**The doc-fact marker convention.** A doc region delimited by
`{/* doc-fact:<name> ... */}` in `.mdx` (HTML comment in `.md`) holds
`key = value` lines that the owning crate's contract test parses and verifies
against code — invisible in the rendered page, adjacent to the prose it pins
so an editor changing the prose sees the contract. Tests live in the crate
that owns the truth (clap tree → `ironclaw_cli`; schema constant → the
registry; request policy → `ironclaw_openai_compat`), never in a generic
doc-checking engine that would re-encode the truth as strings and drift
itself.

**Gate reachability.** The affected-area planner
(`scripts/ci/reborn_pr_test_plan.py`) historically classified `docs/` as
having no Rust test surface, which the doc-fact tests falsify: a docs-only
PR would have selected zero crate tests and merged green, deferring the
failure to the next full run on someone else's change. The planner now
routes docs changes to the crates whose tests read them — any published-tree
docs change selects `ironclaw_extension_registry` (its sweep walks the whole
published tree), `docs/using/cli.mdx` additionally selects `ironclaw`, and
`docs/api/responses.mdx` additionally selects `ironclaw_openai_compat`.
Fenced trees (`docs/internal/`, `docs/reborn/`, drafts) keep the prose
classification: no cargo test reads them, and check-guidance covers their
path claims independently.
Comment on lines +85 to +96

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="docs/internal/plans/2026-08-07-doc-truth-pipeline.md"

echo "== file exists and line count =="
wc -l "$file" || true

echo "== relevant sections =="
sed -n '55,105p' "$file" | cat -n

echo "== check-guidance occurrences in repo =="
rg -n "check-guidance|check_guidance|check guidance" . -S || true

Repository: nearai/ironclaw

Length of output: 8235

Clarify the fenced-tree check-guidance coverage.

check-guidance.py excludes docs/internal/ and non-contract docs/reborn/, so the later claim that check-guidance independently covers those same fenced trees is inconsistent. State coverage as false, or list the fenced-tree exception precisely and note the exclusion from the docs coverage summary.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 85 - 96,
Revise the “Gate reachability” paragraph to accurately describe check-guidance
coverage for fenced trees: do not claim independent coverage for docs/internal/
or non-contract docs/reborn/ when they are excluded by check-guidance.py. State
coverage as false or precisely identify any exception, and exclude those trees
from the docs coverage summary.


### 3.2 Release-time (the cut and publish chain)

- **Changelog gate** — `scripts/ci/cut_ironclaw_release.py::
ensure_stable_changelog_entry`: a stable (non-rc) tag is refused when the
candidate commit's `docs/changelog.mdx` lacks the release's
`description="vX.Y.Z"` entry — an exact attribute match, so an rc-labeled
entry (`description="vX.Y.Z-rc.1"`) cannot satisfy the stable gate by
substring. Rc cuts exempt (hotfix flow unimpeded).
Comment on lines +100 to +105

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Enforce exactly one changelog entry per stable release.

The decision in lines 56-61 requires one <Update> per release. The described gate only checks for one exact description="vX.Y.Z" match. Duplicate entries can pass. Require exactly one matching entry and retain the release-candidate exemption.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 100 - 105,
Update the changelog gate described by ensure_stable_changelog_entry so stable
releases require exactly one matching description="vX.Y.Z" entry in
docs/changelog.mdx, rejecting both missing and duplicate entries. Preserve the
existing exact-match behavior and exemption for release-candidate cuts.

- **`publish-docs-live`** — job in `.github/workflows/ironclaw-release.yml`
after `host`, prerelease-guarded, force-updates `refs/heads/docs-live` to
the released commit via the refs API (bootstraps the branch on first
run). Forced by design: successive stable tags need not be
ancestor-related; the branch is a pointer, not a history. Two guards keep
the pointer honest: the prerelease check, and a newest-stable-tag check —
the job moves the branch only when the workflow's own tag is the highest
stable `ironclaw-v*` tag, so re-running an older release's workflow (a
routine move after a flaky artifact upload) cannot silently revert the
live site. Pinned in `ws12_workflow_contracts.py` `REQUIRED_MARKERS` so
cargo-dist regeneration cannot silently drop either the job or the guard.
Comment on lines +106 to +116

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate relevant files =="
fd -a 'AGENTS\.md|CONTRACT.md|README\.md|2026-08-07-doc-truth-pipeline\.md|ironclaw-release\.yml|ws12_workflow_contracts\.py|CLAUDE.md|.*rules.*' . | sed 's#^\./##' | head -200

echo
echo "== git status/stat =="
git status --short
git diff --stat

echo
echo "== internal doc relevant lines =="
if [ -f docs/internal/plans/2026-08-07-doc-truth-pipeline.md ]; then
  sed -n '80,135p' docs/internal/plans/2026-08-07-doc-truth-pipeline.md | cat -n
fi

echo
echo "== workflow and contract occurrences =="
rg -n "publish-docs-live|docs-live|NEWEST|newest-stable|stable tag|refs/heads/docs-live|REQUIRED_MARKERS|publish-docs" .github docs ws12_workflow_contracts.py 2>/dev/null | head -300

Repository: nearai/ironclaw

Length of output: 12411

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== target doc plan excerpts =="
sed -n '1,150p' docs/internal/plans/2026-08-07-doc-truth-pipeline.md | cat -n

echo
echo "== workflow occurrences =="
rg -n "publish-docs-live|docs-live|concurrency|cancel-in-progress|workflow_dispatch|workflow_call|workflow_run|ironclaw-v|stability|stable" .github workflows 2>/dev/null || true

echo
echo "== contract occurrences =="
for f in $(git ls-files | rg '(^|/)(AGENTS\.md|CLAUDE\.md|CONTRACT\.md|README\.md|ws12_workflow_contracts\.py)$' 2>/dev/null); do
  if rg -q "publish-docs-live|docs-live|REQUIRED_MARKERS|newest-stable-tag|stability" "$f" 2>/dev/null; then
    echo "--- $f"
    rg -n "publish-docs-live|docs-live|REQUIRED_MARKERS|newest-stable-tag|stability" "$f" | head -200
  fi
done

echo
echo "== list workflow files =="
fd -e yml -e yaml . .github . 2>/dev/null | head -200

echo
echo "== exact workflow snippets around publish job if present =="
for f in $(git ls-files .github/workflows | sort); do
  if rg -q "publish-docs-live|docs-live" "$f" 2>/dev/null; then
    echo "--- $f"
    rg -n -C 8 "publish-docs-lite|publish-docs-live|docs-live|workflow_concurrency|concurrency" "$f"
  fi
done

Repository: nearai/ironclaw

Length of output: 30447

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== exact workflow filenames =="
git ls-files '.github/workflows/*'

echo
echo "== ironclaw-release.yml =="
if [ -f .github/workflows/ironclaw-release.yml ]; then
  wc -l .github/workflows/ironclaw-release.yml
  sed -n '1,240p' .github/workflows/ironclaw-release.yml | cat -n
else
  echo "missing .github/workflows/ironclaw-release.yml"
fi

echo
echo "== scripts with docs-live / newest stable logic =="
rg -n "docs-live|newest|highest|stable tag|stable tags|refs/heads/docs-live|github\.ref_name|github\.ref|workflow_dispatch|workflow_call" scripts .github/workflows/cut-ironclaw-release.yml .github/workflows/ironclaw-release.yml 2>/dev/null || true

echo
echo "== ws12_workflow_contracts relevant marker =="
sed -n '1,80p' scripts/ci/ws12_workflow_contracts.py | cat -n

Repository: nearai/ironclaw

Length of output: 20569

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== ws12_marker_required checks =="
rg -n "REQUIRED_MARKERS|publish-docs-live|publish-docs-live|docs-live|highest-stable|newest-stable|workflow_jobs|jobs:" scripts/ci/ws12_workflow_contracts.py | head -250

echo
echo "== scripts/ci/ws12_workflow_contracts.py markers section around implementation =="
sed -n '70,120p' scripts/ci/ws12_workflow_contracts.py | cat -n

echo
echo "== workflow trigger / concurrency in release workflows =="
python3 - <<'PY'
from pathlib import Path
for p in [Path(".github/workflows/ironclaw-release.yml"), Path(".github/workflows/cut-ironclaw-release.yml")]:
    print(f"--- {p}")
    text = p.read_text()
    for i, line in enumerate(text.splitlines(), 1):
        if any(s in line for s in ["on:", "push:", "tags:", "workflow_dispatch:", "concurrency:", "cancel-in-progress:"]):
            print(f"{i}: {line}")
PY

echo
echo "== all occurrences across repo =="
rg -n "publish-docs-live|newest-stable-tag|highest stable|requires.*workflow|workflow_run|workflow_dispatch:|on:|concurrency:" .github/scripts docs/internal/plans/2026-08-07-doc-truth-pipeline.md scripts/ci/ws12_workflow_contracts.py 2>/dev/null | head -300

Repository: nearai/ironclaw

Length of output: 5292

Keep the docs-live pointer behind serialized release publication.

The current workflow shape allows multiple release instances to run for different stable tags. An older publish-docs-live run can update refs/heads/docs-live after a newer release has already pointed it forward; the newest-tag check only prevents older runs from overwriting a pre-flight check, not from overwriting a later production publish. Serialize release publishes per docs-live (for example via workflow concurrency keyed on that intent) or make the update itself an atomic precondition-check-then-write operation.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 106 - 116,
The publish-docs-live flow must serialize updates to refs/heads/docs-live so an
older release cannot overwrite a newer publication. Update the publish-docs-live
job in the workflow and its corresponding REQUIRED_MARKERS contract to add
concurrency keyed to docs-live publication, or enforce an atomic
precondition-check-then-write around the refs API update; preserve the existing
prerelease and newest-stable-tag guards.


### 3.3 Human checklist

`docs/internal/weekly-release-strategy.md`: the changelog entry lands on
`main` before the Monday cut (see §2 — the candidate inherits it and every
later candidate keeps it); Wednesday promotion verifies the deployed site
reflects the release (the changelog page is the probe); the "Docs
publication" section holds the Mintlify dashboard configuration, the
branch-protection shape for `docs-live` (restrict who can push, but allow
force pushes for the Actions actor — protection that blocks force pushes
breaks the automation it guards), the emergency manual repoint command, and
the **docs-hotfix recipe**: when a live page is wrong mid-week, publish a
commit whose tree is the released tag plus the docs fix and repoint
`docs-live` at it — never at `main`, which would republish unreleased
behavior. The dashboard repoint is the one out-of-repo configuration this
Comment on lines +128 to +131

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Persist documentation hotfixes into the next release source.

This recipe points docs-live at an untagged hotfix commit. The next stable publication force-points it to the tagged release commit, so the hotfix disappears unless it is merged or cherry-picked into the next release source. Make propagation a required step.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 128 - 131,
Update the docs-hotfix recipe to require propagating the hotfix commit into the
next release source by merging or cherry-picking it before the next stable
publication. Preserve the existing rule that docs-live points to the released
tag plus the documentation fix, not main.

pipeline cannot verify from CI; the post-promotion check is the
compensating control.
Comment on lines +120 to +133

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Use a release-unique deployment probe.

The changelog entry lands on main before the cut. A dashboard pointed at main can therefore render the expected entry while docs-live is misconfigured. Verify the deployed tree with an immutable release marker or an equivalent artifact-level check.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 120 - 133,
The Wednesday post-promotion check described in the “Docs publication” section
must use a release-unique deployment probe rather than only checking for the
changelog entry. Update the probe to verify an immutable release marker or
equivalent artifact-level evidence from the deployed tree, while preserving the
existing docs-hotfix and branch-targeting guidance.


## 4. What is enforced where

| Doc surface | Drift class | Gate |
| --- | --- | --- |
| Any published page, `zh/`, `docs/reborn/contracts/` | dead repo path in backticks | check-guidance docs surface |
| `docs/using/cli.mdx` | missing/retired subcommand rows | `docs_cli_reference.rs` |
| `docs/extensions/building-a-tool.md` + published tree | retired schema literal; missing v3/gate-matrix teaching | `docs_manifest_schema_version.rs` |
| `docs/api/responses.mdx` | request-policy claims (rejections, ranges, caps, unknown-field tolerance) | `docs_responses_contract.rs` |
| `docs/changelog.mdx` | stable release missing its entry | cut-script changelog gate |
| whole site | describing unreleased behavior | `docs-live` deployment branch |
| nav/fence integrity | unpublished-but-reachable pages | `docs_publication_boundary.py` |

## 5. Deferred follow-ups (specced, not built)

1. **Release-time live probe gate.** Extend
`scripts/ci/smoke-release-binary.py` `REQUIRED_EVIDENCE` with doc-derived
probes against the *exact packaged binary* (every documented subcommand
answers `--help`; a served instance's `/v1/responses` rejects
`tool_choice` with 400). Catches build-feature divergence that PR-tier
tests on the dev profile cannot. Alternative home: an e2e blackbox lane
scenario beside `tests/e2e/scenarios/test_reborn_blackbox_smoke.py`.
2. **LLM auto-fix docs PRs** — modeled on `openwiki-update.yml` (cron +
dispatch, `GH_RELEASES_MANAGER` app token, opens a PR, never auto-merge):
run the deterministic gates in `--json` mode, feed failures to a model
that drafts the docs-only fix PR. The deterministic gates remain the only
blocking authority; the generator only reduces fix friction (refinements
1 and 3 of the issue). MDX lint of generated output before the PR opens
(refinement 4) belongs in that workflow.
3. **CLI flag-level doc coverage** — deliberately out of the subcommand
gate today; flags churn fast enough that a naive gate would train people
to stop reading failures. Needs a curated flag-fact marker per command
group if demand appears.
4. **Mintlify internal-link integrity** — `mint broken-links` is documented
for local use but never runs in CI; wiring it (or a Python equivalent
over `docs.json` routes) into the docs-publication-boundary job closes
the link-rot class the reference gate deliberately skips.
5. **zh freshness policy** — the `zh/` mirror gets path-reference checking
but no translation-parity signal; a per-page `translated-at` frontmatter
plus a non-blocking staleness report is the cheap version.
6. **Contract-doc pinned-test claims** — `docs/reborn/contracts/` names
pinning tests whose functions have moved or been renamed (found while
fixing its dangling paths); an owner pass re-verifying those claims is
recorded here rather than half-fixed mechanically.

## 6. Risks and mitigations

- **Docs-only contributors hitting gate failures**: extraction is
inline-code-only with a curated not-a-path filter; failures name the file,
line, and token; the `path-ok` marker (HTML or MDX comment form) is the
documented escape hatch.
- **Historical archives**: excluded as classes, not seeded as debt — dated
plans and ADRs describe the tree as it stood; forcing them current would
rewrite history, and 705 of 709 pre-exclusion dangles sat there.
- **Docs-only PRs skipping the cargo gates**: closed by the planner routing
in §3.1 — without it the doc-fact tests only ran on full-scope events, so
the PRs most likely to break them merged green and the failure landed on
an unrelated change.
- **Changelog history loss**: closed by writing the entry on `main` before
the cut (§2); the gate alone cannot catch it because each cut only checks
its own version.
- **Workflow re-run reverting the site**: closed by the newest-stable-tag
guard in `publish-docs-live` (§3.2).
- **Branch protection breaking the forced update**: the runbook's
protection recommendation spells out the force-push allowance; protection
configured without it would 422 the repoint.
- **cargo-dist regeneration dropping the docs-live job**: `REQUIRED_MARKERS`
fails Code Style.
- **Dashboard repoint is invisible to CI**: weekly-checklist observable
(changelog probe) is the compensating control.
- **A wrong page live mid-week**: the runbook's docs-hotfix recipe (§3.3)
fixes the live site without waiting for the next stable release and
without republishing `main`.
- **First `docs-live` publication predating the drift fixes**: the branch
publishes the *tagged* tree, so the fixes go live at the first stable
release cut after #7375 merged.
Loading