chore(ci): increase max_turns to 45

[ralphbean]

Trying to reduce flakes. This went up after Adam introduced the Skill use fix.

[qodo-code-review]

PR Summary by Qodo

Increase triage eval max_turns to 45 to reduce flakes

⚙️ Configuration changes 🕐 Less than 5 minutes

AI Description

• Increase triage eval case max_turns from 35 to 45.
• Reduce flakiness after recent Skill-use behavior changes.

Diagram

graph TD
  A["Case annotations.yaml"] -->|"max_turns=45"| B["Eval harness"] -->|"triage conversation"| C{{"LLM"}}

  subgraph Legend
    direction LR
    _file["Config file"] ~~~ _svc["Service/process"] ~~~ _ext{{"External"}}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Fix underlying flake source (prompt/tooling regression)

• ➕ Addresses root cause instead of increasing time/cost budget
• ➕ Avoids longer runtimes masking future regressions
• ➖ May require deeper investigation and changes across prompts/tools
• ➖ Longer lead time than a config-only tweak

2. Add targeted retry/backoff in eval runner

• ➕ Reduces intermittent failures without expanding max_turns
• ➕ Can be scoped to known flaky steps
• ➖ Can hide legitimate deterministic failures
• ➖ Adds complexity and may complicate result interpretation

Recommendation: Raising max_turns is a reasonable, low-risk short-term mitigation for flakes caused by longer conversations after the Skill-use fix. If flakes persist or runtime/cost rises, follow up with root-cause debugging (preferred) rather than continuing to increase turn limits.

Files changed (4) +4 / -4

Other (4) +4 / -4

annotations.yamlIncrease max_turns for bug URL-encoding triage case +1/-1

Increase max_turns for bug URL-encoding triage case

• Bumps the case max_turns limit from 35 to 45 to allow more interaction steps before termination. Intended to reduce flaky failures due to longer tool/skill usage sequences.

eval/triage/cases/001-bug-url-encoding/annotations.yaml

annotations.yamlIncrease max_turns for needs-info vague crash triage case +1/-1

Increase max_turns for needs-info vague crash triage case

• Bumps the case max_turns limit from 35 to 45 to reduce flakiness from running out of turns. Keeps cost cap unchanged.

eval/triage/cases/002-needs-info-vague-crash/annotations.yaml

annotations.yamlIncrease max_turns for feature request triage case +1/-1

Increase max_turns for feature request triage case

• Bumps the case max_turns limit from 35 to 45 to accommodate longer back-and-forth. No other annotation fields are changed.

eval/triage/cases/003-feature-request/annotations.yaml

annotations.yamlIncrease max_turns for duplicate issue triage case +1/-1

Increase max_turns for duplicate issue triage case

• Bumps the case max_turns limit from 35 to 45 to reduce flaky timeouts/early termination. Cost budget remains the same.

eval/triage/cases/004-duplicate-issue/annotations.yaml

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:37 PM UTC · Completed 7:45 PM UTC

Commit: 3e8dd08 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.58

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Tip of the day

💡 Did you know, you can switch off images and animations for a plain-text comment

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[fullsend-ai-review]

Looks good to me

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 7:56 PM UTC · Completed 8:06 PM UTC

Commit: 3e8dd08 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.59

[fullsend-ai-retro]

Retro: PR #6562 — chore(ci): increase max_turns to 45

This was a clean, fast workflow. Human-authored PR by ralphbean bumping max_turns from 35 to 45 in 4 triage eval annotations.yaml files to reduce flakes after a Skill-use behavior fix increased agent turn counts. Created at 19:35 UTC, merged at 19:54 UTC — 19 minutes end-to-end. Review agent approved ("Looks good to me"), human (ascerra) approved, all CI passed.

Evidence for existing open issues

No novel proposals — all findings map to issues already being tracked:

• agents#586 / fullsend#987: Issue #6561 (filed by the retro agent on the prior PR fix(#6475): raise triage eval max_turns from 30 to 35 #6476) proposed raising max_turns to 40, but this PR went directly to 45. Issue Raise triage eval max_turns from 35 to 40 for additional headroom #6561 is now stale and could cause a regression if a code agent implements it literally. This is concrete evidence for the superseded-issue detection gap tracked in agents#586 and the staleness-check proposal in fullsend#987. Note: Raise triage eval max_turns from 35 to 40 for additional headroom #6561 currently has ready-for-triage (not ready-to-code), so the immediate risk is low.

• fullsend#4060 / agents#513: The review agent spent $1.58 and ~10 minutes (Opus model, 25 tool calls, 3 sub-agents dispatched) reviewing an 8-line config-only diff. It correctly classified the change as "mechanical / value-only" but still dispatched correctness and style-conventions sub-agents, plus a challenger. The correctness sub-agent returned zero findings; the challenger had nothing to evaluate. This is direct evidence for #4060 (reduce sub-agent dispatch for mechanical changes) and agents#513 (short-circuit to lightweight mode for mechanical bumps).

• agents#379 / fullsend#1771: The style-conventions sub-agent failed silently because claude-sonnet-4-5@20250929 was unavailable on the Vertex deployment. The failure was swallowed as an info-level finding below the reporting threshold. Evidence for agents#379 (handle model unavailability with fallback or alerting) and fullsend#1771 (fall back to an available model).

• fullsend#2981: The challenger sub-agent was dispatched despite all prior sub-agents producing zero findings above threshold. Evidence for #2981 (skip challenger when all findings are below severity threshold).

What went well

• Fast turnaround: 19 minutes from PR creation to merge for a clear config change.
• Review agent correctly identified the change type and did not generate false positives.
• CI pipeline ran cleanly — all checks passed.
• The human reviewer's approval aligned with the agent's assessment.