feat: daily docs-sync bot keeping kilo-docs in sync with merged PRs - #12512
Conversation
Adds a scheduled workflow that keeps packages/kilo-docs in sync with PRs merged to Kilo-Org/cloud and Kilo-Org/kilocode: - watermark.mjs derives the processing window from the bot's own PR body marker (self-healing, no external state; 72h fallback, 14d cap) - collect.mjs queries merged PRs via the GitHub API and applies a deterministic pre-filter (bots, chores, docs-only PRs) - triage.mjs classifies PRs in chunks of 25 with kilo run; failed chunks degrade to unclassified instead of failing the run - edit.mjs updates docs in batches of 5 PRs with kilo run, bounded per batch; failures surface as skipped entries in the PR body - verify runs the kilo-docs build + test suite; one LLM fix pass on failure; still-red becomes a draft PR - upsert-pr.mjs maintains one rolling auto-docs PR (appends while open, fresh branch after merge), with a 15-file draft cap and a machine-readable processed-through watermark Also adds docs-sync.yml to the workflow allowlist in script/check-workflows.ts.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Incremental review of commit All previously reported findings in this PR (unscoped agent write access, watermark trust, cross-chunk triage contamination, retry coverage, PR-body interpolation, file-list classification, JSON array extraction, etc.) are in unchanged files/lines for this commit and remain tracked on their existing resolved threads — not repeated here. Files Reviewed (1 file changed in this commit)
Previous Review Summaries (5 snapshots, latest commit 8b7da4d)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 8b7da4d)Status: No Issues Found | Recommendation: Merge Incremental review of commit
No new issues were introduced by this commit. All other previously reported findings in this PR (unscoped agent write access, unauthenticated watermark trust, cross-chunk triage contamination, limited retry coverage, missing per-PR error isolation, force-reset branch handling, unescaped PR-body interpolation, truncated file-list classification, JSON array extraction) are unchanged by this commit, in unchanged files, and remain tracked on their existing threads — not repeated here. Files Reviewed (2 files changed in this commit)
Previous review (commit 4c68610)Status: 1 Issue Found | Recommendation: Address before merge Incremental review of commit Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (1 file changed in this commit)
Fix these issues in Kilo Cloud Previous review (commit e37cabb)Status: 1 Issue Found | Recommendation: Address before merge Incremental review of commit Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (1 file changed in this commit)
Fix these issues in Kilo Cloud Previous review (commit d1f66fe)Status: 1 Issue Found | Recommendation: Address before merge Incremental review of commit Overview
Issue Details (click to expand)WARNING
Files Reviewed (3 files changed in this commit)
Fix these issues in Kilo Cloud Previous review (commit 405ed74)Status: 9 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
WARNING
SUGGESTION
Files Reviewed (13 files)
Reviewed by claude-sonnet-5 · Input: 22 · Output: 3.7K · Cached: 456.9K Review guidance: REVIEW.md from base branch |
- message positional must come before flags: --file is multi-value and consumes a trailing message as a file path (File not found) - authenticate via the existing KILO_API_KEY repo secret (the kilo provider reads it natively); drop the DOCS_SYNC_KILO_CONFIG config secret requirement - fix default model IDs: gateway provider id is kilo/, not kilocode/ - include stderr tail in triage/edit failure logs
kilo run prints the assistant message twice (streaming render + final summary), so stdout can contain the same JSON array back-to-back. Parse the largest valid trailing array instead of slicing first-to-last bracket. Verified against real chunked triage output.
- edit.mjs: unambiguous summary file path in the batch prompt and a fallback read when the agent drops the docs-sync-out/ prefix, so real edits never report as skipped - prepare-branch.mjs: use the open auto-docs PR's actual head.ref instead of assuming docs/auto-sync - upsert-pr.mjs: compute the 15-file draft cap on the cumulative PR diff (origin/main...HEAD), not just the latest commit
Security: - sanitize HTML-comment sequences out of agent-generated PR body values so a crafted value cannot forge section markers or the watermark - draft any PR whose diff touches non-content files in packages/kilo-docs (outside pages/ and lib/nav/) — build-executable changes force human review before merge - on merge conflict, keep the conflicted rolling branch untouched (preserving human commits) and continue on a fresh dated branch that links the old PR Resilience: - retry GitHub API calls on network errors and 5xx, not just 403 rate limits - isolate per-PR collect failures instead of aborting the run - trust watermark markers only on bot-authored PRs and clamp future dates loudly - validate chunk triage entries belong to their chunk before the shared dedupe - use changed_files for files_total and skip docs-only classification on truncated (300+) file lists - pipe stderr in the edit pass so failure warnings carry the real CLI error
- escape pipe characters in changeRow actions (same as skippedRow) - sanitize agent-chosen file paths before they land in draftReasons and the PR body (residual marker-forgery path via filenames) - log expected fetch misses in prepare-branch instead of silent catches
|
@iscekic Documentation is often not properly reviewed (especially if this will run after a large merge). If somebody would manage to poison the docs and get through the review, what credentials will be exposed to a potential injection attack? |
@marius-kilocode You mean a prompt attack on the agent making the doc update? I'm guessing any secret we have available to CI. |
Release and dependency bots ship user-facing changes (e.g. JetBrains release PRs from kilo-maintainer[bot]). The auto-docs label check and docs-only path filter remain as the loop guards.
…ilo-Org#12512) * feat: daily docs-sync bot workflow (Kilo CLI) Adds a scheduled workflow that keeps packages/kilo-docs in sync with PRs merged to Kilo-Org/cloud and Kilo-Org/kilocode: - watermark.mjs derives the processing window from the bot's own PR body marker (self-healing, no external state; 72h fallback, 14d cap) - collect.mjs queries merged PRs via the GitHub API and applies a deterministic pre-filter (bots, chores, docs-only PRs) - triage.mjs classifies PRs in chunks of 25 with kilo run; failed chunks degrade to unclassified instead of failing the run - edit.mjs updates docs in batches of 5 PRs with kilo run, bounded per batch; failures surface as skipped entries in the PR body - verify runs the kilo-docs build + test suite; one LLM fix pass on failure; still-red becomes a draft PR - upsert-pr.mjs maintains one rolling auto-docs PR (appends while open, fresh branch after merge), with a 15-file draft cap and a machine-readable processed-through watermark Also adds docs-sync.yml to the workflow allowlist in script/check-workflows.ts. * fix: correct kilo run invocation and auth - message positional must come before flags: --file is multi-value and consumes a trailing message as a file path (File not found) - authenticate via the existing KILO_API_KEY repo secret (the kilo provider reads it natively); drop the DOCS_SYNC_KILO_CONFIG config secret requirement - fix default model IDs: gateway provider id is kilo/, not kilocode/ - include stderr tail in triage/edit failure logs * fix: handle kilo run double-printed assistant output kilo run prints the assistant message twice (streaming render + final summary), so stdout can contain the same JSON array back-to-back. Parse the largest valid trailing array instead of slicing first-to-last bracket. Verified against real chunked triage output. * fix: reviewer-pass robustness fixes - edit.mjs: unambiguous summary file path in the batch prompt and a fallback read when the agent drops the docs-sync-out/ prefix, so real edits never report as skipped - prepare-branch.mjs: use the open auto-docs PR's actual head.ref instead of assuming docs/auto-sync - upsert-pr.mjs: compute the 15-file draft cap on the cumulative PR diff (origin/main...HEAD), not just the latest commit * fix: address Kilobot review findings Security: - sanitize HTML-comment sequences out of agent-generated PR body values so a crafted value cannot forge section markers or the watermark - draft any PR whose diff touches non-content files in packages/kilo-docs (outside pages/ and lib/nav/) — build-executable changes force human review before merge - on merge conflict, keep the conflicted rolling branch untouched (preserving human commits) and continue on a fresh dated branch that links the old PR Resilience: - retry GitHub API calls on network errors and 5xx, not just 403 rate limits - isolate per-PR collect failures instead of aborting the run - trust watermark markers only on bot-authored PRs and clamp future dates loudly - validate chunk triage entries belong to their chunk before the shared dedupe - use changed_files for files_total and skip docs-only classification on truncated (300+) file lists - pipe stderr in the edit pass so failure warnings carry the real CLI error * fix: address second Kilobot review round - escape pipe characters in changeRow actions (same as skippedRow) - sanitize agent-chosen file paths before they land in draftReasons and the PR body (residual marker-forgery path via filenames) - log expected fetch misses in prepare-branch instead of silent catches * feat: keep bot-authored PRs in the docs-sync digest Release and dependency bots ship user-facing changes (e.g. JetBrains release PRs from kilo-maintainer[bot]). The auto-docs label check and docs-only path filter remain as the loop guards.
3 participants
What
Adds
.github/workflows/docs-sync.yml: a daily (07:00 UTC) + manually dispatchable workflow that keepspackages/kilo-docsin sync with features merged to Kilo-Org/cloud and this repo, by running Kilo CLI headless and maintaining one rolling PR for human review.How it works
<!-- docs-sync: processed-through ... -->): latest openauto-docsPR, else last merged one, else 72h fallback, capped at 14 days. No external state; missed or failed runs self-heal on the next run. Manual replays via thesincedispatch input.auto-docs-labeled PRs (loop guard),chore|test|ci|build|docs|style|refactor|reverttitles, and docs-only-path PRs. Bot-authored PRs are kept (release/dependency bots ship user-facing changes). Emits a slim digest (for triage) and a full digest with patch excerpts (for the edit pass).packages/kilo-docsfollowing its ownAGENTS.md+STYLE_GUIDE.md(Markdoc tags, nav updates, redirect rules), with bounded context per batch. Batch failures surface as skipped entries in the PR body; nothing fails silently.kilo-docsbuild + test suite (content-integrity, headings, sitemap, redirects). One LLM fix pass on failure; still red → the PR opens as a draft.docs/auto-sync; while a PR is open the next run appends to it (changes/skipped tables merge via marker sections, humans' commits on the branch are preserved); after a merge a fresh branch starts. Draft when more than 15 files change or verification fails. Every change row links the merged PR it documents.Files
.github/workflows/docs-sync.yml— the pipeline (also added to thescript/check-workflows.tsallowlist).github/docs-sync/*.mjs— dependency-free Node helpers:watermark,collect,triage,filter-worthy,edit,prepare-branch,upsert-pr,extract-json,lib.github/docs-sync/triage-prompt.md,edit-prompt.md— the agent prompts; tune these to adjust qualityManual steps (after merge)
dry_run=trueandsinceset to ~7 days back. Download thedocs-sync-outartifact, readtriage.json, and tune.github/docs-sync/triage-prompt.mdif the verdicts look wrong. No branches or PRs are created in dry-run mode.dry_run=falseand review the rolling PR it opens. The 07:00 UTC cron activates automatically once this lands on the default branch.auto-docs.ymland theDOC_WEBHOOK_*secrets from both repos (left in place intentionally for now; @iscekic will do this manually).DOCS_SYNC_TRIAGE_MODELand/orDOCS_SYNC_EDIT_MODELto change the default (kilo/moonshotai/kimi-k3; the edit pass runs at--variant high). A cheaper model for triage is the obvious first knob.No other setup needed, already verified in this repo:
KILO_API_KEYsecret, which the CLI'skiloprovider reads natively (no config file or extra secret required).can_approve_pull_request_reviews: true).POSTHOG_API_KEYexists (verify step uses it for the docs build, same asdocs-build.yml).Testing
collect.mjsrun live against both repos: ~130 merged PRs in a 3.5-day window → 116 kept after the deterministic pre-filter.kilo/moonshotai/kimi-k3: 116 PRs in 5 chunks → 116 entries, 25 docs-worthy, 0 failed chunks. Verdict quality hand-checked (accurate mobile/JetBrains/CLI feature classifications with sound reasons).triage.mjsandedit.mjsfailure paths validated with a shimmedkilobinary: retries, degrade-on-failure, coverage-fill, and dedupe all behave (permanently failing chunks/batches become visible "skipped" rows, never silent).upsert-pr.mjsbody rendering round-trip tested (marker sections merge without duplicates, watermark marker updates, draft reasons render).check-workflows.ts,check-opencode-annotations.ts,check-forbidden-strings.ts; the prompt files contain no GFM tables (table-padding check not triggered).kilo runverified working on Linux (fresh install in anode:24container) and macOS.CLI invocation notes (baked into the code)
--fileis multi-value and consumes a trailing message as a file path (File not found). All call sites usekilo run "<msg>" ... -f fileordering.kilo runprints the assistant message twice (streaming render + final summary), so raw stdout can contain the same JSON array back-to-back —extract-json.mjsparses the largest valid trailing array.kilo runfrom inside another running Kilo harness inheritsKILO_RUN_ID/KILO_SERVER_*env and fails withSession not found— scrub the env or use a clean shell locally; CI runners are unaffected.Notes
auto-docslabel (auto-created on first run) and thedocs/branch prefix per kilo-docs conventions.