ci: harden Mintlify docs automation (AI triage classifier + scoped generator)#2405
Conversation
chelojimenez
temporarily deployed
to
preview-pr-2405
GitHub Actions
Inactive
dosubot
Bot
added
the
size:XS
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b91b20ae1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| permissions: | ||
| contents: write # enable auto-merge | ||
| pull-requests: write # close / label / comment | ||
| id-token: write # claude-code-action auth | ||
| actions: read # read CI status |
There was a problem hiding this comment.
In the triage workflow I checked, the GITHUB_TOKEN permissions explicitly omit issues, so unspecified permissions are none; GitHub's Labels REST API documents Create a label as requiring Issues repository permission with write access. Because the very next step runs gh label create ... --force, every Mintlify PR run will fail before Claude can merge/close/review anything unless this job grants issues: write.
Useful? React with 👍 / 👎.
| on: | ||
| pull_request: | ||
| types: [opened, reopened] |
There was a problem hiding this comment.
This workflow is intended to close stale/non-mergeable Mintlify PRs, but GitHub Actions docs state that workflows do not run on pull_request activity when the PR has a merge conflict. For the conflicted bot PRs called out by the classifier's own CLOSE rules, this opened/reopened pull_request trigger means the triage job never starts, so those stale PRs will remain unprocessed.
Useful? React with 👍 / 👎.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
|
||
| on: | ||
| pull_request: | ||
| types: [opened, reopened] |
There was a problem hiding this comment.
No triage on PR pushes
Medium Severity
The triage workflow only listens for opened and reopened, not synchronize. Mintlify PRs updated by new commits (conflict fixes, bot revisions) no longer re-run classification or re-enable auto-merge, unlike the removed auto-merge workflow.
Reviewed by Cursor Bugbot for commit 9b91b20. Configure here.
Internal previewPreview URL: https://mcp-inspector-pr-2405.up.railway.app |
|
Warning Review limit reached
More reviews will be available in 10 minutes and 7 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
WalkthroughThe PR removes the previous auto-merge workflow and adds a Claude-driven triage workflow that runs only on newly opened Mintlify bot PRs, verifies mergeability, and can either run Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/mintlify-triage.yml:
- Line 20: Replace the conditional gate that uses github.actor with the PR
author identity by changing the job-level if condition that currently reads "if:
github.actor == 'mintlify[bot]'" to use github.event.pull_request.user.login ==
'mintlify[bot]' so the workflow checks the pull request author (PR author
boundary) rather than the event actor; keep the rest of the logic and formatting
of the mintlify-triage job intact.
- Around line 29-32: Update the workflow to pin mutable action tags to immutable
commit SHAs: replace uses: actions/checkout@v4 and uses:
anthropics/claude-code-action@v1 with their respective full commit SHAs so the
actions are deterministic; additionally, add with: persist-credentials: false to
the actions/checkout invocation (alongside fetch-depth) to prevent leaving
authenticated git credentials in the runner when the job has
contents/pull-requests: write.
- Around line 22-26: Remove the permissive Bash(gh api:*) entry from the
workflow's allowed-tools so the action cannot perform arbitrary GitHub API
calls; update the allowed-tools list to only include the specific commands
already intended (e.g., Bash(gh pr*) and Bash(git*)) and ensure any references
to Bash(gh api:*) in the YAML are deleted or replaced with the more specific gh
pr* entries to preserve the intended single-action boundary.
In @.github/workflows/update-docs.yml:
- Around line 73-81: The workflow currently always calls the Mintlify API on
merged PRs and can re-introduce docs-only bot PRs; add an early skip before the
Mintlify API call (the step that invokes the Mintlify API / "Call Mintlify API"
step) to exit when the PR author is "mintlify[bot]" or when the changed files
are entirely under docs/; implement this by adding a conditional or a small
check step (using actions/checkout + a git diff of the merge commit or
actions/github-script to list changed files) that sets an output/flag and then
guard the Mintlify API step with if: > checks (e.g., if: ${{
github.event.pull_request.user.login != 'mintlify[bot]' && !inputs.docs_only
}}), so the Mintlify API is skipped for bot PRs or docs-only changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 67430db0-3d01-4d39-b1c6-14a605b9e8c2
📒 Files selected for processing (3)
.github/workflows/auto-merge-mintlify.yml.github/workflows/mintlify-triage.yml.github/workflows/update-docs.yml
💤 Files with no reviewable changes (1)
- .github/workflows/auto-merge-mintlify.yml
| - name: Checkout repository | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 1 # full file tree (code + docs) so feature greps work |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".github/workflows/mintlify-triage.yml"
echo "== File =="
wc -l "$FILE"
echo
echo "== Uses lines =="
rg -n "^\s*uses:" "$FILE" || true
echo
echo "== Lines 1-120 =="
nl -ba "$FILE" | sed -n '1,120p'Repository: MCPJam/inspector
Length of output: 298
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".github/workflows/mintlify-triage.yml"
echo "== Full file (numbered) =="
cat -n "$FILE"
echo
echo "== Permissions occurrences =="
rg -n "permissions:" "$FILE" || true
echo
echo "== Checkout step block (lines containing checkout until blank line or next - name/uses) =="
# crude but sufficient for small file
awk '
/uses: actions\/checkout@/ {start=1}
start {print}
start && NR>1 && NF==0 {exit}
' "$FILE"Repository: MCPJam/inspector
Length of output: 5171
🌐 Web query:
actions/checkout v4 persist-credentials default write token .git/config
💡 Result:
In actions/checkout (v4 and later), the persist-credentials input defaults to true [1][2][3]. While earlier iterations of the action directly wrote the authentication token into the local.git/config file, recent updates have improved credential security by storing them in a separate file within the runner's temporary directory ($RUNNER_TEMP) [3][4][5][6]. The action then references this file using git config include.path mechanisms rather than embedding the token directly into the repository's primary configuration [6]. Key points regarding this behavior: - Default behavior: The action automatically configures the token to enable authenticated git commands (like git fetch or git push) during the job [7][8][3]. - Security: Credentials are kept in a separate file under $RUNNER_TEMP, not directly inside.git/config [3][4][5][6]. - Cleanup: The token and the associated temporary configuration file are removed during post-job cleanup [7][8][3]. - Opt-out: You can disable this behavior by setting persist-credentials: false in your workflow step [7][8][3].
Citations:
- 1: https://github.com/actions/checkout/blob/v4/action.yml
- 2: https://github.com/actions/checkout/blob/v4.1.1/action.yml
- 3: https://github.com/actions/checkout
- 4: https://github.com/actions/checkout/tree/v6.0.2?tab=readme-ov-file
- 5: https://github.com/marketplace/actions/checkout
- 6: actions/checkout@069c695
- 7: https://github.com/actions/checkout/blob/v4/README.md
- 8: https://github.com/actions/checkout/?tab=readme-ov-file
Pin third-party actions to commit SHAs and disable checkout credential persistence
actions/checkout@v4andanthropics/claude-code-action@v1are on mutable tags—pin both to full commit SHAs.actions/checkoutdefaultspersist-credentialstotrue; since this job hascontents/pull-requests: write, setwith: persist-credentials: falseon checkout to avoid leaving authenticated git credentials configured during the run.
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 29-32: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 30-30: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/mintlify-triage.yml around lines 29 - 32, Update the
workflow to pin mutable action tags to immutable commit SHAs: replace uses:
actions/checkout@v4 and uses: anthropics/claude-code-action@v1 with their
respective full commit SHAs so the actions are deterministic; additionally, add
with: persist-credentials: false to the actions/checkout invocation (alongside
fetch-depth) to prevent leaving authenticated git credentials in the runner when
the job has contents/pull-requests: write.
9b91b20 to
7b97cd4
Compare
chelojimenez
temporarily deployed
to
preview-pr-2405
GitHub Actions
Inactive
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7b97cd4. Configure here.
|
|
||
| Investigate first (read-only): | ||
| - Diff + metadata: `gh pr diff ${{ github.event.pull_request.number }}` and `gh pr view ${{ github.event.pull_request.number }} --json files,mergeable,mergeStateStatus,title,body`. A non-mergeable / conflicting PR usually means it targets a page that was renamed or deleted (stale). | ||
| - Confirm any feature the PR documents actually exists in the code: `git grep -i "<symbol or exact UI string>" -- mcpjam-inspector/ sdk/`. If you cannot find it, treat the feature as non-existent. |
There was a problem hiding this comment.
Triage skips CLI code paths
Medium Severity
The triage prompt tells the classifier to verify features only under mcpjam-inspector/ and sdk/, but many user-facing docs describe cli/ behavior. Legitimate bot PRs can be closed as documenting non-existent features when symbols live only in cli/.
Reviewed by Cursor Bugbot for commit 7b97cd4. Configure here.
|
|
||
| MERGE only if you can confirm ALL of: the PR is mergeable; every target page still exists; the feature is present in the code; the content is user-facing and accurate; and it is not a duplicate or editing anything under `docs/contributing/**`. | ||
|
|
||
| CLOSE in every other case — stale (deleted/renamed page or not mergeable); backend/internal content (Convex, internal HTTP endpoints/routes, environment variables, secrets/keys, server-side persistence or runtime, internal source-file paths); a feature you cannot find in the code; a duplicate; a clear factual error versus the code; OR anything you are not confident enough to merge. Closing is safe and cheap — the bot regenerates a fresh PR when the code changes again. |
There was a problem hiding this comment.
Early mergeable check closes PRs
Medium Severity
Triage runs immediately on opened and requires mergeable to MERGE, while GitHub often reports mergeable as unknown right after open. With a close bias, valid docs PRs can be closed before mergeability is computed.
Reviewed by Cursor Bugbot for commit 7b97cd4. Configure here.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/mintlify-triage.yml:
- Around line 45-60: The workflow removed the REVIEW fallback: update the
allowed-tools string (the --allowed-tools entry) to permit labeling (e.g.,
include Bash(gh pr label:* or Bash(gh pr edit:*)) and modify the prompt decision
block (the prompt: | text that currently forces MERGE or CLOSE) to add a third
branch "REVIEW" when the classifier is unsure; implement the exact action by
adding a new EXECUTE line that runs a gh command to apply the needs-human-review
label (for example `gh pr label ${{ github.event.pull_request.number }} --add
needs-human-review` or `gh pr edit ... --add-label needs-human-review`) and
ensure the instructions state to choose MERGE, CLOSE, or REVIEW (label then
stop) so ambiguous-but-possibly-correct PRs are escalated rather than
auto-closed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f5b65270-a953-452e-a82d-98aca49dac54
📒 Files selected for processing (3)
.github/workflows/auto-merge-mintlify.yml.github/workflows/mintlify-triage.yml.github/workflows/update-docs.yml
💤 Files with no reviewable changes (1)
- .github/workflows/auto-merge-mintlify.yml
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/update-docs.yml
| --allowed-tools "Bash(gh pr view:*),Bash(gh pr diff:*),Bash(gh pr merge:*),Bash(gh pr close:*),Bash(gh pr comment:*),Bash(git grep:*),Bash(git log:*),Bash(git show:*),Bash(git ls-files:*),Grep,Glob,Read" | ||
| prompt: | | ||
| You are the documentation-PR triager for ${{ github.repository }}. PR #${{ github.event.pull_request.number }} was opened by the Mintlify docs bot. Decide and execute EXACTLY ONE outcome: MERGE or CLOSE. Bias toward CLOSE — only MERGE when you can positively verify the PR is good. Everything published must be user-facing, accurate, and free of backend/internal detail. Do not ask questions; take the action yourself with the gh CLI. | ||
|
|
||
| Investigate first (read-only): | ||
| - Diff + metadata: `gh pr diff ${{ github.event.pull_request.number }}` and `gh pr view ${{ github.event.pull_request.number }} --json files,mergeable,mergeStateStatus,title,body`. A non-mergeable / conflicting PR usually means it targets a page that was renamed or deleted (stale). | ||
| - Confirm any feature the PR documents actually exists in the code: `git grep -i "<symbol or exact UI string>" -- mcpjam-inspector/ sdk/`. If you cannot find it, treat the feature as non-existent. | ||
| - Check whether the content is already documented: `git grep -i "<phrase>" -- docs/`. | ||
|
|
||
| MERGE only if you can confirm ALL of: the PR is mergeable; every target page still exists; the feature is present in the code; the content is user-facing and accurate; and it is not a duplicate or editing anything under `docs/contributing/**`. | ||
|
|
||
| CLOSE in every other case — stale (deleted/renamed page or not mergeable); backend/internal content (Convex, internal HTTP endpoints/routes, environment variables, secrets/keys, server-side persistence or runtime, internal source-file paths); a feature you cannot find in the code; a duplicate; a clear factual error versus the code; OR anything you are not confident enough to merge. Closing is safe and cheap — the bot regenerates a fresh PR when the code changes again. | ||
|
|
||
| Execute exactly one, then stop: | ||
| - MERGE → `gh pr merge ${{ github.event.pull_request.number }} --auto --squash` | ||
| - CLOSE → `gh pr close ${{ github.event.pull_request.number }} --comment "<one concise sentence on why>"` |
There was a problem hiding this comment.
Restore the promised REVIEW fallback.
The workflow now forces every uncertain case into CLOSE, but the PR objective for this automation explicitly calls for a third outcome: label needs-human-review when the classifier cannot decide. As written, ambiguous-but-possibly-correct docs PRs will be auto-closed instead of escalated, which changes the contract and makes the triage materially harsher than intended. Please add the REVIEW path back to both the tool allowlist and the prompt’s decision rules.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/mintlify-triage.yml around lines 45 - 60, The workflow
removed the REVIEW fallback: update the allowed-tools string (the
--allowed-tools entry) to permit labeling (e.g., include Bash(gh pr label:* or
Bash(gh pr edit:*)) and modify the prompt decision block (the prompt: | text
that currently forces MERGE or CLOSE) to add a third branch "REVIEW" when the
classifier is unsure; implement the exact action by adding a new EXECUTE line
that runs a gh command to apply the needs-human-review label (for example `gh pr
label ${{ github.event.pull_request.number }} --add needs-human-review` or `gh
pr edit ... --add-label needs-human-review`) and ensure the instructions state
to choose MERGE, CLOSE, or REVIEW (label then stop) so
ambiguous-but-possibly-correct PRs are escalated rather than auto-closed.
- Replace auto-merge-of-everything with a Claude triage classifier (mintlify-triage.yml): verifies each mintlify[bot] PR against the code, then MERGE (auto-merge, CI-gated) when it can confirm the docs are accurate / user-facing / clean, else CLOSE (stale, backend-internal, inaccurate, duplicate, or unverifiable). Biased to close; the bot regenerates a fresh PR when the code changes again. - Restrict the generator (update-docs.yml) to user-facing docs: never touch docs/contributing/**, never document backend internals, ground content in the merged diff. Skip the bot's own merged PRs to prevent a generate/merge loop. - Security/correctness from review: gate on PR author (not spoofable github.actor), drop persisted git credentials and the broad gh api tool, least-privilege permissions, no label step. https://claude.ai/code/session_01LExdyi7zaPZZRz9dvjU9Yp
chelojimenez
force-pushed
the
claude/laughing-allen-XrwLo
branch
from
7b97cd4 to
5bda049
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5bda049a01
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| MERGE only if you can confirm ALL of: the PR is mergeable; every target page still exists; the feature is present in the code; the content is user-facing and accurate; and it is not a duplicate or editing anything under `docs/contributing/**`. | ||
|
|
||
| CLOSE in every other case — stale (deleted/renamed page or not mergeable); backend/internal content (Convex, internal HTTP endpoints/routes, environment variables, secrets/keys, server-side persistence or runtime, internal source-file paths); a feature you cannot find in the code; a duplicate; a clear factual error versus the code; OR anything you are not confident enough to merge. Closing is safe and cheap — the bot regenerates a fresh PR when the code changes again. |
There was a problem hiding this comment.
Because this workflow runs immediately on pull_request.opened, gh pr view --json mergeable,mergeStateStatus can still report an unknown/non-mergeable value for a clean PR while GitHub computes the test merge commit; GitHub's Pulls API docs state that mergeable may be null while a background job computes it and the request should be retried after giving it time. With the current CLOSE rule, a newly opened Mintlify PR can be closed as stale before GitHub has finished computing mergeability, so the triager should wait/retry or distinguish UNKNOWN/null from an actual conflict.
Useful? React with 👍 / 👎.
1 participant
Hardens the Mintlify docs automation so the 74-PR backlog can't rebuild. Two changes:
1. AI triage classifier (
mintlify-triage.yml) — replaces the naive auto-mergeThe old
auto-merge-mintlify.ymlrangh pr merge --autoon every bot PR with no content check — so accurate-but-stale PRs piled up (conflicts/link-rot kept auto-merge from completing) and anything that happened to pass CI could merge unverified.The replacement runs a Claude classifier (
anthropics/claude-code-action@v1, sameANTHROPIC_API_KEYasclaude.yml) on eachmintlify[bot]PR. It verifies the PR against the actual code, then takes exactly one action:--auto --squash, so CI still gates the actual merge) — only when it's mergeable, the target pages exist, the feature is confirmed in the code, and it's accurate + non-duplicate.needs-human-reviewwhen it can't safely decide (the conservative default).Safeguards: it never merges a feature it can't find in the code or anything that looks backend-internal;
--automeans broken docs still can't land; humans can override any decision.2. Scoped generator (
update-docs.yml)The generator prompt previously told the bot to also write
docs/contributing/docs — the source of the backend-internals leaks. New hard rules: only user-facing docs, never touchdocs/contributing/**, never document backend/internal mechanics (Convex, internal routes, env vars, secrets, source paths), ground every statement in the merged diff (no invented features/numbers), and target the current page structure instead of recreating renamed/removed pages.Note: this grants the classifier merge/close authority on bot PRs, so it's worth a read before merging. It only takes effect once on
main. The remaining prevention items (Mintlify-dashboard config: grounding generation in merged diffs, pinning the bot to current page names) are outside the repo.Generated by Claude Code
Note
Medium Risk
Grants an automated agent merge/close authority on bot documentation PRs; misclassification could close good PRs or queue bad ones for auto-merge, though
--autostill requires CI and scope is limited tomintlify[bot].Overview
Replaces blind auto-merge of every
mintlify[bot]PR with a Claude triage workflow that verifies each docs PR against the repo (diff, mergeability,git grepin app code) and executes exactly one outcome: enable auto-merge only when content is user-facing, accurate, and non-duplicate, or close with a short comment otherwise (bias to close). Triage gates on PR author (mintlify[bot]), notgithub.actor, and only runs onopened.Auto-update Documentation now skips when a merged PR is authored by the bot (breaks generate → merge → regenerate loops) and tightens the Mintlify agent prompt: user-facing docs only, never
docs/contributing/**, no backend/internal details, diff-grounded facts, and updates aligned to the currentdocs/docs.jsonstructure instead of resurrecting removed pages.Reviewed by Cursor Bugbot for commit 5bda049. Bugbot is set up for automated code reviews on this repo. Configure here.