Repository navigation
docs(internal): doc-truth pipeline design record (doc-truth PR 5/5) - #7381
Merged
Merged
Conversation
The as-built design for keeping the public Mintlify docs in sync with released binaries: the problem evidence, the decisions of record (single doc tree deploying from a docs-live branch; deterministic gates only; human-curated changelog), the three enforcement layers (PR-time static gates, release-time cut/publish automation, human checklist), a surface-to-gate enforcement table, the deferred follow-ups (release-time live probes on the packaged binary, an openwiki-style non-blocking LLM fix-PR generator, flag-level CLI coverage, link integrity, zh freshness, contract-doc pinned-test re-verification), and the risk register. Part of #7317 (doc-truth pipeline, PR 5 of 5). Companions: #7375 #7376 #7378 #7379. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This PR was not deployed automatically as @thisisjoshford does not have access to the Railway project. In order to get automatic PR deploys, please add @thisisjoshford to your workspace on Railway. |
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds a plan for the Doc-Truth Verification Pipeline. The plan defines pull-request and release gates, documentation publication checks, contract markers, enforced drift classes, deferred automation, and operational mitigations. ChangesDoc-Truth Verification Pipeline
Estimated code review effort: 1 (Trivial) | ~5 minutes 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
size: XS
Contributor
🧭 IronLoop Run · ReviewThis comment updates in place as the Run moves through its stages. 🟥 Final result · Could not complete
Automatic trigger · attempt 1 of 3 · failed after 6s IronLoop could not complete the review for this Run. Failure details
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an internal design record documenting the “Doc-Truth Verification Pipeline” intended to prevent published docs/ drift from shipped behavior (per #7317), including the motivating drift cases, decisions of record, enforcement layers, and deferred follow-ups.
Changes:
- Add an internal plan/design document describing the doc-truth pipeline architecture and enforcement layers.
- Capture “decisions of record” (e.g.,
docs-livedeployment model, deterministic gates, curated changelog) and a surface→gate mapping. - Document deferred follow-ups and a risk/mitigation register.
Suppressed comments (1)
docs/internal/plans/2026-08-07-doc-truth-pipeline.md:88
- This section names specific implementation details (
cut_ironclaw_release.py::ensure_stable_changelog_entry, apublish-docs-livejob in.github/workflows/ironclaw-release.yml, anddocs/changelog.mdx) that are not present in the current repository state. Consider rewriting these bullets to reference the owning PR (#7379) and describe the behavior without hard-coding symbol/job names that may change, or ensure this document is only merged after those changes land.
- **Changelog gate** — `scripts/ci/cut_ironclaw_release.py::
ensure_stable_changelog_entry`: a stable (non-rc) tag is refused when the
candidate commit's `docs/changelog.mdx` lacks the release's `vX.Y.Z`
entry. Rc cuts exempt (hotfix flow unimpeded).
- **`publish-docs-live`** — job in `.github/workflows/ironclaw-release.yml`
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Doc-Truth Verification Pipeline | ||
|
|
||
| **Date:** 2026-08-07 | ||
| **Status:** Implemented (phases 1–4); follow-ups specced below |
Post-implementation review found five gaps; this records their resolutions where the design doc already claims the guarantees: planner routing so doc-fact tests run on docs-only PRs, the changelog entry landing on main before the Monday cut, a newest-stable-tag guard on publish-docs-live, the force-push allowance in the docs-live branch-protection shape, and the docs-hotfix repoint recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md`:
- Around line 100-105: Update the changelog gate described by
ensure_stable_changelog_entry so stable releases require exactly one matching
description="vX.Y.Z" entry in docs/changelog.mdx, rejecting both missing and
duplicate entries. Preserve the existing exact-match behavior and exemption for
release-candidate cuts.
- Around line 120-133: The Wednesday post-promotion check described in the “Docs
publication” section must use a release-unique deployment probe rather than only
checking for the changelog entry. Update the probe to verify an immutable
release marker or equivalent artifact-level evidence from the deployed tree,
while preserving the existing docs-hotfix and branch-targeting guidance.
- Around line 128-131: Update the docs-hotfix recipe to require propagating the
hotfix commit into the next release source by merging or cherry-picking it
before the next stable publication. Preserve the existing rule that docs-live
points to the released tag plus the documentation fix, not main.
- Around line 85-96: Revise the “Gate reachability” paragraph to accurately
describe check-guidance coverage for fenced trees: do not claim independent
coverage for docs/internal/ or non-contract docs/reborn/ when they are excluded
by check-guidance.py. State coverage as false or precisely identify any
exception, and exclude those trees from the docs coverage summary.
- Around line 106-116: The publish-docs-live flow must serialize updates to
refs/heads/docs-live so an older release cannot overwrite a newer publication.
Update the publish-docs-live job in the workflow and its corresponding
REQUIRED_MARKERS contract to add concurrency keyed to docs-live publication, or
enforce an atomic precondition-check-then-write around the refs API update;
preserve the existing prerelease and newest-stable-tag guards.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 8e01df0b-fd3b-460a-b5b7-6bb4f5e9eedd
📒 Files selected for processing (1)
docs/internal/plans/2026-08-07-doc-truth-pipeline.md
Comment on lines
+85
to
+96
| **Gate reachability.** The affected-area planner | ||
| (`scripts/ci/reborn_pr_test_plan.py`) historically classified `docs/` as | ||
| having no Rust test surface, which the doc-fact tests falsify: a docs-only | ||
| PR would have selected zero crate tests and merged green, deferring the | ||
| failure to the next full run on someone else's change. The planner now | ||
| routes docs changes to the crates whose tests read them — any published-tree | ||
| docs change selects `ironclaw_extension_registry` (its sweep walks the whole | ||
| published tree), `docs/using/cli.mdx` additionally selects `ironclaw`, and | ||
| `docs/api/responses.mdx` additionally selects `ironclaw_openai_compat`. | ||
| Fenced trees (`docs/internal/`, `docs/reborn/`, drafts) keep the prose | ||
| classification: no cargo test reads them, and check-guidance covers their | ||
| path claims independently. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="docs/internal/plans/2026-08-07-doc-truth-pipeline.md"
echo "== file exists and line count =="
wc -l "$file" || true
echo "== relevant sections =="
sed -n '55,105p' "$file" | cat -n
echo "== check-guidance occurrences in repo =="
rg -n "check-guidance|check_guidance|check guidance" . -S || trueRepository: nearai/ironclaw
Length of output: 8235
Clarify the fenced-tree check-guidance coverage.
check-guidance.py excludes docs/internal/ and non-contract docs/reborn/, so the later claim that check-guidance independently covers those same fenced trees is inconsistent. State coverage as false, or list the fenced-tree exception precisely and note the exclusion from the docs coverage summary.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 85 - 96,
Revise the “Gate reachability” paragraph to accurately describe check-guidance
coverage for fenced trees: do not claim independent coverage for docs/internal/
or non-contract docs/reborn/ when they are excluded by check-guidance.py. State
coverage as false or precisely identify any exception, and exclude those trees
from the docs coverage summary.
Comment on lines
+100
to
+105
| - **Changelog gate** — `scripts/ci/cut_ironclaw_release.py:: | ||
| ensure_stable_changelog_entry`: a stable (non-rc) tag is refused when the | ||
| candidate commit's `docs/changelog.mdx` lacks the release's | ||
| `description="vX.Y.Z"` entry — an exact attribute match, so an rc-labeled | ||
| entry (`description="vX.Y.Z-rc.1"`) cannot satisfy the stable gate by | ||
| substring. Rc cuts exempt (hotfix flow unimpeded). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Enforce exactly one changelog entry per stable release.
The decision in lines 56-61 requires one <Update> per release. The described gate only checks for one exact description="vX.Y.Z" match. Duplicate entries can pass. Require exactly one matching entry and retain the release-candidate exemption.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 100 - 105,
Update the changelog gate described by ensure_stable_changelog_entry so stable
releases require exactly one matching description="vX.Y.Z" entry in
docs/changelog.mdx, rejecting both missing and duplicate entries. Preserve the
existing exact-match behavior and exemption for release-candidate cuts.
Comment on lines
+106
to
+116
| - **`publish-docs-live`** — job in `.github/workflows/ironclaw-release.yml` | ||
| after `host`, prerelease-guarded, force-updates `refs/heads/docs-live` to | ||
| the released commit via the refs API (bootstraps the branch on first | ||
| run). Forced by design: successive stable tags need not be | ||
| ancestor-related; the branch is a pointer, not a history. Two guards keep | ||
| the pointer honest: the prerelease check, and a newest-stable-tag check — | ||
| the job moves the branch only when the workflow's own tag is the highest | ||
| stable `ironclaw-v*` tag, so re-running an older release's workflow (a | ||
| routine move after a flaky artifact upload) cannot silently revert the | ||
| live site. Pinned in `ws12_workflow_contracts.py` `REQUIRED_MARKERS` so | ||
| cargo-dist regeneration cannot silently drop either the job or the guard. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate relevant files =="
fd -a 'AGENTS\.md|CONTRACT.md|README\.md|2026-08-07-doc-truth-pipeline\.md|ironclaw-release\.yml|ws12_workflow_contracts\.py|CLAUDE.md|.*rules.*' . | sed 's#^\./##' | head -200
echo
echo "== git status/stat =="
git status --short
git diff --stat
echo
echo "== internal doc relevant lines =="
if [ -f docs/internal/plans/2026-08-07-doc-truth-pipeline.md ]; then
sed -n '80,135p' docs/internal/plans/2026-08-07-doc-truth-pipeline.md | cat -n
fi
echo
echo "== workflow and contract occurrences =="
rg -n "publish-docs-live|docs-live|NEWEST|newest-stable|stable tag|refs/heads/docs-live|REQUIRED_MARKERS|publish-docs" .github docs ws12_workflow_contracts.py 2>/dev/null | head -300Repository: nearai/ironclaw
Length of output: 12411
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target doc plan excerpts =="
sed -n '1,150p' docs/internal/plans/2026-08-07-doc-truth-pipeline.md | cat -n
echo
echo "== workflow occurrences =="
rg -n "publish-docs-live|docs-live|concurrency|cancel-in-progress|workflow_dispatch|workflow_call|workflow_run|ironclaw-v|stability|stable" .github workflows 2>/dev/null || true
echo
echo "== contract occurrences =="
for f in $(git ls-files | rg '(^|/)(AGENTS\.md|CLAUDE\.md|CONTRACT\.md|README\.md|ws12_workflow_contracts\.py)$' 2>/dev/null); do
if rg -q "publish-docs-live|docs-live|REQUIRED_MARKERS|newest-stable-tag|stability" "$f" 2>/dev/null; then
echo "--- $f"
rg -n "publish-docs-live|docs-live|REQUIRED_MARKERS|newest-stable-tag|stability" "$f" | head -200
fi
done
echo
echo "== list workflow files =="
fd -e yml -e yaml . .github . 2>/dev/null | head -200
echo
echo "== exact workflow snippets around publish job if present =="
for f in $(git ls-files .github/workflows | sort); do
if rg -q "publish-docs-live|docs-live" "$f" 2>/dev/null; then
echo "--- $f"
rg -n -C 8 "publish-docs-lite|publish-docs-live|docs-live|workflow_concurrency|concurrency" "$f"
fi
doneRepository: nearai/ironclaw
Length of output: 30447
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== exact workflow filenames =="
git ls-files '.github/workflows/*'
echo
echo "== ironclaw-release.yml =="
if [ -f .github/workflows/ironclaw-release.yml ]; then
wc -l .github/workflows/ironclaw-release.yml
sed -n '1,240p' .github/workflows/ironclaw-release.yml | cat -n
else
echo "missing .github/workflows/ironclaw-release.yml"
fi
echo
echo "== scripts with docs-live / newest stable logic =="
rg -n "docs-live|newest|highest|stable tag|stable tags|refs/heads/docs-live|github\.ref_name|github\.ref|workflow_dispatch|workflow_call" scripts .github/workflows/cut-ironclaw-release.yml .github/workflows/ironclaw-release.yml 2>/dev/null || true
echo
echo "== ws12_workflow_contracts relevant marker =="
sed -n '1,80p' scripts/ci/ws12_workflow_contracts.py | cat -nRepository: nearai/ironclaw
Length of output: 20569
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== ws12_marker_required checks =="
rg -n "REQUIRED_MARKERS|publish-docs-live|publish-docs-live|docs-live|highest-stable|newest-stable|workflow_jobs|jobs:" scripts/ci/ws12_workflow_contracts.py | head -250
echo
echo "== scripts/ci/ws12_workflow_contracts.py markers section around implementation =="
sed -n '70,120p' scripts/ci/ws12_workflow_contracts.py | cat -n
echo
echo "== workflow trigger / concurrency in release workflows =="
python3 - <<'PY'
from pathlib import Path
for p in [Path(".github/workflows/ironclaw-release.yml"), Path(".github/workflows/cut-ironclaw-release.yml")]:
print(f"--- {p}")
text = p.read_text()
for i, line in enumerate(text.splitlines(), 1):
if any(s in line for s in ["on:", "push:", "tags:", "workflow_dispatch:", "concurrency:", "cancel-in-progress:"]):
print(f"{i}: {line}")
PY
echo
echo "== all occurrences across repo =="
rg -n "publish-docs-live|newest-stable-tag|highest stable|requires.*workflow|workflow_run|workflow_dispatch:|on:|concurrency:" .github/scripts docs/internal/plans/2026-08-07-doc-truth-pipeline.md scripts/ci/ws12_workflow_contracts.py 2>/dev/null | head -300Repository: nearai/ironclaw
Length of output: 5292
Keep the docs-live pointer behind serialized release publication.
The current workflow shape allows multiple release instances to run for different stable tags. An older publish-docs-live run can update refs/heads/docs-live after a newer release has already pointed it forward; the newest-tag check only prevents older runs from overwriting a pre-flight check, not from overwriting a later production publish. Serialize release publishes per docs-live (for example via workflow concurrency keyed on that intent) or make the update itself an atomic precondition-check-then-write operation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 106 - 116,
The publish-docs-live flow must serialize updates to refs/heads/docs-live so an
older release cannot overwrite a newer publication. Update the publish-docs-live
job in the workflow and its corresponding REQUIRED_MARKERS contract to add
concurrency keyed to docs-live publication, or enforce an atomic
precondition-check-then-write around the refs API update; preserve the existing
prerelease and newest-stable-tag guards.
Comment on lines
+120
to
+133
| `docs/internal/weekly-release-strategy.md`: the changelog entry lands on | ||
| `main` before the Monday cut (see §2 — the candidate inherits it and every | ||
| later candidate keeps it); Wednesday promotion verifies the deployed site | ||
| reflects the release (the changelog page is the probe); the "Docs | ||
| publication" section holds the Mintlify dashboard configuration, the | ||
| branch-protection shape for `docs-live` (restrict who can push, but allow | ||
| force pushes for the Actions actor — protection that blocks force pushes | ||
| breaks the automation it guards), the emergency manual repoint command, and | ||
| the **docs-hotfix recipe**: when a live page is wrong mid-week, publish a | ||
| commit whose tree is the released tag plus the docs fix and repoint | ||
| `docs-live` at it — never at `main`, which would republish unreleased | ||
| behavior. The dashboard repoint is the one out-of-repo configuration this | ||
| pipeline cannot verify from CI; the post-promotion check is the | ||
| compensating control. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Use a release-unique deployment probe.
The changelog entry lands on main before the cut. A dashboard pointed at main can therefore render the expected entry while docs-live is misconfigured. Verify the deployed tree with an immutable release marker or an equivalent artifact-level check.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 120 - 133,
The Wednesday post-promotion check described in the “Docs publication” section
must use a release-unique deployment probe rather than only checking for the
changelog entry. Update the probe to verify an immutable release marker or
equivalent artifact-level evidence from the deployed tree, while preserving the
existing docs-hotfix and branch-targeting guidance.
Comment on lines
+128
to
+131
| the **docs-hotfix recipe**: when a live page is wrong mid-week, publish a | ||
| commit whose tree is the released tag plus the docs fix and repoint | ||
| `docs-live` at it — never at `main`, which would republish unreleased | ||
| behavior. The dashboard repoint is the one out-of-repo configuration this |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Persist documentation hotfixes into the next release source.
This recipe points docs-live at an untagged hotfix commit. The next stable publication force-points it to the tagged release commit, so the hotfix disappears unless it is merged or cherry-picked into the next release source. Make propagation a required step.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/internal/plans/2026-08-07-doc-truth-pipeline.md` around lines 128 - 131,
Update the docs-hotfix recipe to require propagating the hotfix commit into the
next release source by merging or cherry-picking it before the next stable
publication. Preserve the existing rule that docs-live points to the released
tag plus the documentation fix, not main.
8 of 9 tasks
serrrfirat
approved these changes
Aug 8, 2026
7 tasks done
l3ocifer
pushed a commit
to l3ocifer/frick-ironclaw
that referenced
this pull request
Sep 3, 2026
…earai#7381) * docs(internal): record the doc-truth pipeline design (issue nearai#7317) The as-built design for keeping the public Mintlify docs in sync with released binaries: the problem evidence, the decisions of record (single doc tree deploying from a docs-live branch; deterministic gates only; human-curated changelog), the three enforcement layers (PR-time static gates, release-time cut/publish automation, human checklist), a surface-to-gate enforcement table, the deferred follow-ups (release-time live probes on the packaged binary, an openwiki-style non-blocking LLM fix-PR generator, flag-level CLI coverage, link integrity, zh freshness, contract-doc pinned-test re-verification), and the risk register. Part of nearai#7317 (doc-truth pipeline, PR 5 of 5). Companions: nearai#7375 nearai#7376 nearai#7378 nearai#7379. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(internal): record review hardening in the doc-truth design Post-implementation review found five gaps; this records their resolutions where the design doc already claims the guarantees: planner routing so doc-fact tests run on docs-only PRs, the changelog entry landing on main before the Monday cut, a newest-stable-tag guard on publish-docs-live, the force-push allowance in the docs-live branch-protection shape, and the docs-hotfix repoint recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/internal/plans/2026-08-07-doc-truth-pipeline.md: the as-built design answering issue Proposal: Doc-Truth Verification Pipeline #7317 — problem evidence (the three confirmed drift cases plus the deploy-target mismatch root cause), decisions of record (no Mintlify versions;docs-livedeployment branch; deterministic-only gates; curated changelog), the three enforcement layers, a surface→gate enforcement table, deferred follow-ups (release-time live probes viasmoke-release-binary.pyevidence keys, an openwiki-style non-blocking LLM fix-PR generator, CLI flag-level coverage, Mintlify link integrity, zh freshness, contract-doc pinned-test re-verification), and the risk register.Change Type
Linked Issue
Closes #7317 (the design record is the answer; the implementation landed/lands in #7375, #7376, #7378, #7379).
Validation
python3 scripts/ci/docs_publication_boundary.py—docs/internal/is fenced, no nav entry neededpython3 scripts/ci/check-guidance.py— cleanTest Strategy
User behavior: none — internal design document only.
Risk areas: none.
Tests added or updated: Not applicable: documentation-only; the enforcement the document describes is tested in its companion PRs.
What the tests prove: n/a.
Commands run: the two gates above.
Security Impact
None.
Reborn Trust-Boundary Checklist
N/A — internal documentation.
🤖 Generated with Claude Code
Review hardening (496a3be)
The design record now documents the resolutions of five gaps found in a post-implementation review: planner routing so the doc-fact tests run on docs-only PRs (#7378), the changelog entry landing on
mainbefore the Monday cut, the newest-stable-tag guard onpublish-docs-live, the force-push allowance in the docs-live branch-protection shape, and the docs-hotfix repoint recipe (#7379).