Repository navigation
docs(mcp-review): Context7 A/B eval findings + tool-name fix - #847
Conversation
Captures the 2026-06-20 controlled A/B (AC #2/#3/#4 of #816): Context7 connects and its tools are permitted, the agent adopts it spontaneously on library PRs and skips it on non-library diffs (zero overhead), and the review-quality delta vs base is ~0 on mainstream libs (base model already current to Jan 2026) — value concentrates on post-cutoff/long-tail deps plus auditable grounding. Latency: +~30s per library review, parity otherwise. Also fixes a stale tool name: the live Context7 server exposes resolve-library-id and query-docs (not get-library-docs). The repo variable uses the mcp__context7__* wildcard so production was unaffected, but the doc named a non-existent tool. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015VRMMWmqW9nW81jygmea3e
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
More reviews will be available in 58 minutes and 50 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe MCP-powered review initiative doc is updated to correct the ChangesContext7 MCP pilot documentation update
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related issues
Possibly related PRs
Suggested labels
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the MCP-powered review documentation to specify the correct Context7 tools and adds a detailed evaluation section from a local A/B test. The feedback identifies an inaccuracy regarding the latency tracking capabilities of scripts/pr_review_health.sh and suggests clarifying that live run durations should be monitored via GitHub Actions run history instead.
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-20T19:36:35Z. |
pr_review_health.sh exits early on all-green runs and only analyzes failed-run logs, so it does not compile latency deltas for successful runs. Point live-run duration tracking at GitHub Actions run history instead. Addresses gemini-code-assist review on #847. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015VRMMWmqW9nW81jygmea3e
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-20T19:38:28Z. |
Dev-Lead — fix-bot-comment (no-changes)Agent reasoning |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Dev-Lead — waiting on PR blockers (intent: fix-reviews)PR: #847 |
Dev-Lead — fix-reviews (no-changes)Agent reasoning |
donpetry-bot
left a comment
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: LOW
Reviewed commit: 9bed4457223311bf0245970fc49494cbe9e242f9
Review mode: triage-approved (single reviewer)
Summary
Docs-only update to docs/initiatives/mcp-powered-review.md (+52/-21, 1 file): records the 2026-06-20 Context7 A/B evaluation (AC #2/#3/#4 of #816) and fixes a stale tool name (get-library-docs -> query-docs). No code, config, workflow, or dependency changes.
Linked issue analysis
No formal closingIssuesReferences, but the PR body and commits tie this to #816's eval ACs (#2/#3/#4). The doc substantively records the requested evaluation findings (connectivity, graceful degradation, spontaneous adoption, correctness/latency deltas) and a rollout recommendation, which is exactly what those ACs asked to capture.
Findings
- No security-relevant changes: documentation only, no auth/secrets/crypto/migrations/CI-config touched. CONTEXT7_API_KEY is mentioned only as a header name (zero-auth pilot); no secret value present.
- Verified the one factual claim added by the latest commit (9bed445): scripts/pr_review_health.sh:85-89 does exit 0 on all-green runs and only downloads/analyzes failed-run logs, so the doc's AC#4 latency-tracking note is accurate.
- The prior gemini-code-assist COMMENTED review (on 39cbade, re: latency tracking) was addressed by commit 9bed445; coderabbitai subsequently APPROVED the head SHA. No unresolved review threads or unanswered human questions.
- Secret-scan MCP (run_secret_scanning) was not runnable in this context (permission not granted); the gitleaks CI check passed and the diff contains no secret material.
CI status
All gating checks green: Lint, ShellCheck, CodeQL, Analyze (actions/python), SonarCloud, Secret scan (gitleaks), unit-tests, agent-shield/AgentShield, holdout-guard, review/review, Compile agentic workflows — all SUCCESS. dependency-audit ecosystem jobs SKIPPED (no matching ecosystems). The CANCELLED entries are dev-lead agent automation (dispatch/ci-relay), not gating CI. mergeStateStatus BLOCKED is due to the required org-leads human review, which this verdict feeds.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
|
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-20T21:46:50Z. |
1 similar comment
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-20T21:46:50Z. |
3 participants
Summary
Follow-up to #826 (which wired the zero-auth Context7 MCP server into the PR-review runtime). This records the evaluation asked for by #816's AC #2/#3/#4 and fixes a stale tool name in the doc.
What was tested
A controlled A/B run locally (not via the
pr-review/stableCI path — that path is still pinned to the pre-Context7 config; promotingstableis the production rollout and remains a human decision). The eval replicatedengine.sh's deep-tier invocation exactly (--mcp-config .github/review-mcp.json --strict-mcp-config --allowed-tools "…,mcp__context7__*", modelclaude-opus-4-8), isolating Context7 as the single variable across three inlined PR diffs reviewed off vs on:useQueryoptionsFindings
resolve-library-id+query-docsconnect;is_error=falseon every call; correct grounded answer with doc citationInterpretation: plumbing works and the agent uses Context7 with good judgment, but the quality lift on mainstream libs is ≈0 because the base model is already current to Jan 2026. The marginal value concentrates on post-cutoff / long-tail / private deps and auditable citations — which this repo's Bash/YAML PR profile rarely hits.
Also fixed
The live Context7 server exposes exactly two tools —
resolve-library-idandquery-docs(verified via a directtools/listJSON-RPC call). The doc previously named a non-existentget-library-docstool. Production was unaffected because the repo variable uses themcp__context7__*wildcard, but the doc is now accurate.Rollout recommendation (unchanged gate)
Keep Context7 wired — zero-cost when irrelevant, auditable grounding when relevant. Favorable enough to promote to
pr-review/stableso live reviews exercise it, with honest expectations (near-zero lift on this repo; strong case on library-heavy repos). Promotingstableand any downstream rollout stays a separate, explicit human decision.Closes the eval tasks on #816.
🤖 Generated with Claude Code
Generated by Claude Code
Summary by CodeRabbit