Repository navigation
feat: implement issue #737 — [Phase 1] Instrument auto-rebase: agentic-conflict-resolution rate + fan-out CI-run volume report - #755
Conversation
…c-conflict-resolution rate + fan-out CI-run volume report
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
More reviews will be available in 28 minutes and 53 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the auto_rebase_health.sh script and its corresponding BATS unit tests to measure agentic conflict-resolution rates and estimate fan-out CI-run volume, as well as updating AGENTS.md to document this new workflow. The review feedback suggests a robustness improvement in the jq filter to safely handle null or missing PR authors, and an optimization to use pure Bash arithmetic instead of spawning an external awk process in fmt_rate.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Dev-Lead — fix-reviews (applied)Changes committed and pushed. |
|
Dev-Lead — waiting on PR blockers (intent: review-changes)PR: #755 |
|
Note @don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically. |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
donpetry-bot
left a comment
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: 29ce63880e2e5d00bd55f1f68be9e75457972da0
Review mode: triage-approved (single reviewer)
Summary
Adds a daily Auto-rebase Health report: a new no-LLM workflow (.github/workflows/auto-rebase-health.yml), the pure-helper + network-I/O script scripts/auto_rebase_health.sh, a 140-line bats suite, an AGENTS.md repo-specific-exception note, and the lint.yml bats wiring. +484/−0 across 5 files. Confirming the triage low-risk assessment.
Linked issue analysis
Closes #737 and meets all five acceptance criteria:
- Counts conflict sentinels (
<!-- auto-rebase-conflict:) and dev-leadintent=rebaseresponses over a configurable lookback (summarize_sentinels). - Quantifies auto-rebase fan-out from
auto-rebase.ymlrun telemetry and labels it an estimate (summarize_runs/estimate_fanout). - Recurring daily schedule (07:00 UTC) +
workflow_dispatchwithlookback_days; surfaced via step summary and a pinned tracking-issue comment. - Documented as a Workflow Files exception in AGENTS.md, consistent with the existing exception notes.
- shellcheck-clean (CI green) with bats coverage of the counting/aggregation/rendering logic, added to the lint.yml bats list.
Findings
GitHub Actions surface reviewed closely (the only sensitive area):
- Safe triggers (
schedule+workflow_dispatch) — nopull_request_targetand no untrusted-PR checkout/execution. lookback_daysflows through anenv:block rather than being interpolated into arun:line, andworkflow_dispatchrequires write access — no script-injection vector.- Least-privilege
permissions(contents:read, issues:write, pull-requests:read, actions:read);github.tokenonly, no secrets/PAT used in the workflow. - Third-party actions SHA-pinned (checkout v6.0.3, github-script v9.0.0).
- Script uses
set -euo pipefail,jq --arg(no interpolation), null-safe.author?.login // "", and a zero-denominator guard infmt_rate.
Prior review feedback resolved: gemini-code-assist's two suggestions (null-safe jq author handling and pure-bash arithmetic in fmt_rate) are reflected in fix commit 29ce638; the CodeRabbit review was dismissed and dev-lead reported no further changes needed. No unresolved threads or unanswered human-reviewer questions.
CI status
All checks green. CodeQL (actions/python), SonarCloud, Gitleaks, AgentShield, ShellCheck, bats, unit-tests, and the latest review / review run all SUCCESS. The single CANCELLED "review / review" entry is an earlier concurrency-superseded run of the same check, followed by a SUCCESS re-run — no genuinely failing checks. mergeStateStatus is BLOCKED only on the required org-leads human review.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
* feat(personas): enforce the eval count gate; read evals.path OR evals.paths Companion to petry-projects/.github#779. Two things, both #755: 1. Read the new evals shape. `evals` now carries `path` (one set) OR `paths` (several, for a persona that wraps multiple skills — pr-review's deep-review + triage). check_evals handles both and requires dev/ + holdout/ splits in every set. Backward-compatible: qa-lead's single `path` still validates against both the live main schema and the #779 branch schema. 2. Make principle 5 partially real (finding 5). "No persona reaches stable without an eval gate" was documented but unenforced. Now: once a persona's status reaches its `evals.required_before` ring, each held-out set MUST carry >= `min_cases` real cases (blank lines ignored). A draft is exempt — draft is exactly when seed cases are placeholders. So a persona can no longer be promoted past required_before on the 5 synthetic starters. The SCORED gate (running the cases against a judge and requiring a pass rate) needs an eval harness and is deliberately NOT here — it is a separate follow-up. What is hermetically checkable (the held-out COUNT at promotion) is enforced now; what needs a runtime is tracked, not faked. Verification - bats tests/test_validate_personas.bats: 22/22 (16 + 6 new: paths all-splits, paths one-split-missing, draft-not-enforced, stable-enforced-fails, enough-cases-passes, blank-lines-ignored). - validate-personas.py against BOTH the live main schema and the #779 branch schema: OK (qa-lead is draft, so the count gate does not bite). Merge after #779 (the validator fetches the schema from .github main; until #779 lands, `evals.paths` is not yet permitted there — but qa-lead uses `path`, so main stays green either way). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(reviews): address review comments [skip ci-relay] --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
2 participants
Closes #737
Implemented by dev-lead agent. Please review.