ci(#6668): add exponential backoff to dispatch polling - #6669
Conversation
Replace fixed 5-second poll intervals in E2E behaviour test dispatch detection with exponential backoff (2s initial, doubling to 30s cap, 5-minute total timeout). This makes dispatch polling more tolerant of transient GitHub Actions API delays that caused flaky "was not dispatched" and "no recent workflow runs found" failures. The change applies to WaitForWorkflow, WaitForHarnessAgent, and WaitForFailedHarnessAgent. CountHarnessDispatches retains a fixed 5-second interval via countSettlePoll since it polls for run settlement, not dispatch detection. Closes #6668
|
🤖 Finished Review · ✅ Success · Started 9:46 PM UTC · Completed 10:05 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.43 |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Risk Assessment: moderate (2/5) DetailsModerate risk driven primarily by Tier 2 git history signals: the githubactions CI driver is a high-churn file with 11+ commits in 30 days, 5 distinct authors, and 8 fix-related commits in 90 days, indicating an active stabilization area. However, the PR itself is well-scoped (2 files, 161 lines, 50% test ratio), touches no protected paths, security-sensitive files, CI workflows, or dependencies, and is authored by a known bot. The change aligns precisely with the linked issue scope and is trivially revertible. |
ReviewFindingsLow
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
fullsend-ai-review
Bot
added
the
ready-for-merge
|
🤖 Finished Retro · ✅ Success · Started 7:53 PM UTC · Completed 8:05 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.71 |
Retro: PR #6669 —
|
| Time (UTC) | Event | Cost |
|---|---|---|
| 08-26 21:26 | Retro agent files #6668 | — |
| 08-26 21:26–21:32 | Triage (run): identified related issues, applied ready-to-code |
$0.96 |
| 08-26 21:32–21:44 | Code (run): implemented backoff, opened PR #6669 | $3.56 |
| 08-26 21:45–22:05 | Review (run): approved with 2 low findings | $6.43 |
| 08-27 11:53 | Human review (rh-hemartin): approved | — |
| 08-27 19:51 | Merged by waynesun09 | — |
| Total agent cost | $10.95 |
Observations
Code agent debugging mid-run (positive signal): The code agent hit a test failure during development — a monotonicity assertion broke when the backoff interval (30s cap) transitioned into the completion-wait loop (15s pollInterval). The agent correctly diagnosed the issue and fixed the test to only assert on dispatch-phase intervals. This demonstrates good iterative debugging capability.
Review agent challenger filtering (positive signal): The review sub-agents initially produced 6 findings. The challenger sub-agent filtered 2 as false positives (a naming-consistency finding and a dead-code finding already addressed in the diff) and downgraded 2 to info-level (filtered by severity threshold). The final 2 low findings were both relevant.
Review sub-agent model failures (known issue): 3 of 5 review sub-agents failed because claude-sonnet-4-5@20250929 was unavailable, then retried on opus. This inflated review cost. Already tracked in #6666 (stale sonnet model version mapping inflates review sub-agent costs ~5x) and #1771 (review sub-agents should fall back to an available model).
Duplicate review dispatch (known issue): Two review runs triggered simultaneously on PR open; one was cancelled. This pattern is extensively tracked across #766, #4960, #5139, and the consolidation meta-issue #5817.
Autonomy alignment: The review agent approved and the human approved without comments — review judgment was aligned for this class of change (test infrastructure resilience fix, well-scoped, high test ratio). Relevant tracking in #6210 (criteria for review agent automated approvals).
Proposals
No new proposals. All identified improvement areas are already covered by existing open issues.
2 participants
Summary
Add exponential backoff to E2E behaviour test dispatch polling to reduce flaky failures caused by transient GitHub Actions API delays. The dispatch detection loop now starts polling at 2-second intervals, doubles each iteration up to a 30-second cap, with a 5-minute total timeout window.
Related Issue
Changes
dispatchPoll(5s) ×dispatchMaxTry(48) constants with exponential backoff constants:dispatchPollInit(2s),dispatchPollMax(30s),dispatchTimeout(5min)nextBackoff()helper that doubles the interval up to a capWaitForWorkflowfrom count-based to deadline-based loop with backoffWaitForHarnessAgentandWaitForFailedHarnessAgentto use backoff intervalscountSettlePollconstant forCountHarnessDispatches(retains fixed 5s interval since it polls for run settlement, not dispatch detection)TestNextBackoffunit test andTestDispatchPollingBackoffintegration test verifying exponential interval growthTestDispatchDetectionWindow_AtLeast4Minutesto validatedispatchTimeoutTesting
pkg/behaviourtest/drivers/ci/githubactions/passTestNextBackoffverifies backoff doubling and cap behaviorTestDispatchPollingBackoffverifies thatWaitForWorkflowuses exponentially growing intervals (2s → 4s → 8s → 16s → 30s)go vetpassesChecklist
!for breaking changes)Closes #6668
Post-script verification
agent/6668-dispatch-backoff)6c44cb2dfcd83b9bca5f7c9641b3ab5063d68457..HEAD)