Repository navigation
fix(budget): exempt the Dependabot merge pipeline, count marked agent comments, and set the cap from measured PR peaks (#2224) - #2229
Conversation
…rked agent comments re-arming the budget (#2224) The per-PR automation budget (#926) decided who is automated from the login alone. That was wrong in two directions. On a PR authored by dependabot[bot], Dependabot's own commit and the dependabot-automerge App's update-branch merge commits and re-approvals were counted as agent activity. A Dependabot PR waiting behind a busy main reached the cap after three or four branch updates, was labelled needs-human-review and lost auto-merge (broodminder-data#176, held 36 hours). Those events are now flagged `mechanical` and neither count nor reset. Both keys are required: any other bot on a Dependabot PR, and the App on anyone else's PR, count as before. dev-lead, the personas and auto-rebase post as the owner account, so each of their comments read as a human interaction and reset the budget. On #176 dev-lead's hold notice took the count from 10 to 0 a minute after the escalation. A comment or review whose body carries one of our automation markers is now flagged `agent_marked` and never resets the budget. It does not count either; only bot logins count. gather_pr_automation_events sets the flags, so every consumer inherits the rule. No event is dropped, because the stall and merge-ready scans read .when from all of them. An unreadable PR author exempts nothing. Commits, unmarked comments and review-thread replies made under a human login still read as human. That limit is documented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 32 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces corrections to the PR automation budget system (#2224). It ensures that Dependabot merge pipeline events on Dependabot-authored PRs do not count towards or reset the budget, and that agent comments or reviews carrying automation markers do not reset the budget even when posted under a human owner account. These changes are implemented in scripts/lib/pr-automation-budget.sh, documented in docs/agentic-interaction-model.md, and verified with new unit tests in tests/test_pr_automation_budget.bats. There are no review comments to assess, and I have no feedback to provide.
![codeant-ai[bot]](https://avatars.githubusercontent.com/in/646884?s=60&v=4){kind=small}
CodeAnt Nitpicks1 code suggestion1. Because
|
CodeAnt PR Risk: High Risk
Assessed commit: |
QA Lead — test-risk advisoryRisk tier: MEDIUM. This changes the runaway guard that every agent checks before it writes, on both stable channels, and the What I'd shore up (highest leverage first):
Escalate? No. The spoofing gap is narrow: it needs a Dependabot-authored PR, and an agent's comments and reviews still count. Fix it in this PR rather than block on it. Advisory only — I comment, I do not change code. Ground: BMAD Test Architecture |
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-10T23:10:40Z. |
Dev-Lead — review-changes (applied)Changes committed and pushed. Requested items addressed:
|
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-10T23:30:23Z. |
|
No description provided. |
|
No description provided. |
|
No description provided. |
… humans quote keywords The agent_marked regex was too loose with [^>]*, allowing arbitrary text after keywords. This caused humans quoting marker patterns to be incorrectly tagged as agents, preventing them from resetting the automation budget. Fixed by requiring proper delimiters (space, colon, or dash) after keywords without built-in delimiters (pr-review-agent, pr-review-claim, dev-lead, dependency-advisory, pr-automation-budget). Updated test 57 to use proper delimiter patterns instead of artificial 'x y=z' suffixes, reflecting the more restrictive regex that prevents false positives while still matching all valid marker formats. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
|
No description provided. |
|
No description provided. |
|
No description provided. |
|
Rate-limit notice: CodeRabbit's free OSS review capacity has been reached. No code findings are reported in this comment — it is a pure operational notice. |
Dev-Lead — fix-bot-comment (no-changes)Agent reasoning |
The agent_marked jq function was using test($re) which matches the regex anywhere in the body. This caused false positives when a human quoted one of our automation markers (e.g., in a code example), incorrectly flagging the human's comment as agent-authored and preventing budget reset. Fixed by stripping backtick-enclosed content before testing, so quoted markers are not detected. This allows humans to quote markers when discussing code without triggering the false positive. Fixes the CodeAnt finding in #2229 (#2224 false-positive fix). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry
changed the title
5f10f49 to
df4600a
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
|
No description provided. |
|
No description provided. |
|
No description provided. |
…ed PR peaks (#2224) Agent comments posted under the owner login now count toward the per-PR automation budget as well as no longer resetting it. Before this, dev-lead, persona and auto-rebase activity was invisible to the breaker. The cap of 10 was never set against that activity, so MAX_PR_AUTOMATION_CYCLES goes to 50, chosen from measurement to block only extreme cases. Sample: 332 agent-authored PRs (every one created in the org from 2026-09-10 to 2026-10-10, plus those still open). Peak count between two human interactions: median 10, p95 25, p99 35; highest on a PR that merged, 35; one PR (93) above 50. At a cap of 10 with marked comments counted, 50.9% of the sample would have been escalated at least once; at 50, 0.3%. #860 peaks at 1,487 under the same rule and reaches 50 within ten minutes of its ack storm. Stopping marked comments resetting the budget while keeping the cap at 10, as the first commit on this branch did, would by itself have raised the share of PRs escalated at least once from 4.8% to 16.6%. The runaway detector's RUNAWAY_MAX_CYCLES goes from 10 to 25 (half the cap, about p95) so the daily report still names a PR well before the breaker stops it. Tests that pinned the old cap now follow the default: the threshold cases, the two #860 storm scenarios (scaled to the cap in force), three fixtures that built exactly ten bot events, and the detector's threshold cases. docs: review-thread replies do reset the budget. Each reply arrives as a review with an empty body, and that review is collected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…get-mechanical-and-agent-events
|
No description provided. |
…er match is not anchored (#2224) Review feedback on #2229 questioned whether a review-thread reply can reset the budget, since /pulls/{pr}/comments is never read. It can: the reply arrives on /pulls/{pr}/reviews as a COMMENTED review with an empty body, which is collected and carries no marker. A gather-level test now pins that known limit. The marker regex stays unanchored. In the comments of the 54 open PRs, 495 of 2,300 marker-carrying agent comments have the marker somewhere other than the start of the body, so anchoring would read a fifth of agent comments as human. Quoted markers are handled by dropping backtick spans, as dev-lead's commit did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No description provided. |
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T04:23:22Z. |
Test suite newly red — human attention neededThe |
Superseded — dev-lead is now withholding action under the `dev-lead:hands-off` label. Click to expand the prior `needs-human-review` hold notice.dev-lead is withholding action on this item. It is labeled To re-enable automated pickup: remove the |
don-petry
added
dev-lead:hands-off
…get-mechanical-and-agent-events
|
dev-lead is withholding action on this item. It is labeled To re-enable automated pickup: remove the |
|
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T15:42:59Z. |
2 participants
Summary
The per-PR automation budget (#926) decided who is "automated" from the login alone. This PR corrects that in the two places where the login is misleading, and resets the cap from measurement so that only extreme cases are blocked.
dependabot[bot], events bydependabot[bot]and the dependabot-automerge App are flaggedmechanical. They neither count nor reset.agent_marked. It counts, and it never resets the budget.MAX_PR_AUTOMATION_CYCLESgoes from 10 to 50. The runaway detector'sRUNAWAY_MAX_CYCLESgoes from 10 to 25.gather_pr_automation_eventssets both flags andcompute_pr_automation_cycleshonours them, so every consumer inherits the rule with no change of its own.Closes #2224
Problem
Both problems showed on petry-projects/broodminder-data#176, a green Dependabot PR that sat unmerged for 36 hours.
dependabot-rebasetook the count to 10 (each one adds the App's merge commit and its re-approval). pr-review then posted the exhaustion comment, addedneeds-human-reviewand disabled auto-merge.dependabot-rebaseonly merges PRs with auto-merge on, so the PR stalled. A branch update is caused by a push tomain, not by an agent answering an agent.don-petry, was read as a human interaction and took the count to 0. The next pr-review run went past the budget gate. A machine action reset the budget, which Runaway PR #860: agents self-triggering — add per-PR churn breaker, human-gated cap resets, and self-trigger guards #926 says must never happen.Fixing the second problem makes agent activity visible to the breaker for the first time, and the cap of 10 was never set against it. The first version of this PR only stopped marked comments resetting the budget and kept the cap at 10. Measured over each PR's life, that alone would have raised the share of PRs escalated at least once from 4.8% to 16.6%. The maintainer decided on 2026-10-10 that marked comments should count and that the cap should be raised from actual metrics.
Calibration
332 agent-authored PRs: every PR created in the org from 2026-09-10 to 2026-10-10, plus every PR still open. The metric is the peak of the running count over the PR's life.
mainThe issue has the full tables and the two limits of the measurement.
Risk
Medium. This is the #860 runaway guard, and every agent consults it before writing.
mechanicalflag needs both a PR authored bydependabot[bot]and an event login inPR_BUDGET_MECHANICAL_LOGINS. Any other bot on a Dependabot PR still counts.open >48h with agent churnreason will now also name PRs whose only recent activity is marked agent comments.gather_pr_automation_eventsmakes one more API call per check (the PR author), and merges its three lists on stdin so a long history is not placed on jq's command line.Not changed, and stated in the code and in
docs/agentic-interaction-model.md: a commit, an unmarked comment, or a review with no marker in its body (which includes review-thread replies) that an agent makes under the owner login still reads as a human interaction and resets the budget. The calibration is measured with those resets in place. If that gap is closed later, the counts rise and the cap must be measured again.Test plan
bats tests/test_pr_automation_budget.bats tests/test_pr_runaway_regression.bats tests/test_pr_runaway_detect.bats: 102 pass, 1 fail. The failure ispr_age_hours computes whole hours against an injected now, which fails the same way on unmodifiedmainon macOS (BSDdate)._MAINTAINER_GATE_AGENT_MARKERS; six cases throughgather_pr_automation_eventswith a stubbedgh; the default cap is 50.tests/test_pr_automation_budget.batsfollowMAX_PR_AUTOMATION_CYCLESinstead of the literal 10; the two storms intests/test_pr_runaway_regression.batsare scaled to the cap in force (they were 12 events against a cap of 10) and still exhaust and escalate once; three fixtures that built exactly ten bot events (test_persona_pr_pregate.bats,test_dev_lead_retry_pr_escalation.bats,test_dev_lead_resume.bats) build fifty; the detector's threshold cases intests/test_pr_runaway_detect.batsmove from 10/11 to 25/26.shellcheck --severity=warning -xon both libraries: clean.mainon this machine (GNU/BSDdatedifferences and similar).tests/dev-lead/unit/test_fix_reviews.batshit my local 300-second limit before finishing. CI on Linux is the authoritative run.Rollback
Revert this PR. No state is written and no marker or label format changes. Without a revert: set
MAX_PR_AUTOMATION_CYCLESin the calling workflow to change the cap, orPR_BUDGET_MECHANICAL_LOGINSto an empty value to switch off the Dependabot exemption.Monitoring
[budget] PR #N in <repo>: K Dependabot-pipeline event(s) left out of the automation budget (#2224)and… marker-carrying agent event(s) under a human login counted as automated (#2224).<!-- pr-automation-budget exhausted -->comment on a Dependabot PR;dependabot-rebaseno longer logsSkipping — auto-merge not enabledfor a green one; budget escalations on agent PRs stay rare.pr-review/v1-stableanddev-lead/v139-stable(the library ships on both channels), or the daily runaway report listing many PRs forautomated cycles.Interaction contract
🤖 Generated with Claude Code