Skip to content

test(#5491): merge triage sufficient + sandbox URL-block scenarios - #5499

Merged
ifireball merged 1 commit into
mainfrom
agent/5491-merge-triage-scenarios
Jul 23, 2026
Merged

test(#5491): merge triage sufficient + sandbox URL-block scenarios#5499
ifireball merged 1 commit into
mainfrom
agent/5491-merge-triage-scenarios

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

Merge two triage behaviour scenarios that each paid a full GHA workflow wait into a single scenario with a combined dummy table, halving the wall-clock cost of triage behaviour testing.

Related Issue

Closes #5491 (part of #5490 — fewer positive GHA round-trips for the behaviour suite).

Changes

  • Combined the "Triage applies ready-to-code on sufficient issue" and "Sandbox blocks disallowed outbound URL" scenarios into one scenario in e2e/behaviour/features/triage/triage.feature
  • The merged scenario's dummy table performs a disallowed url_get (expect fail) then writes the sufficient triage fixture (expect succeed)
  • After a single workflow wait, asserts all three outcomes: URL op failure, fixture op success, and ready-to-code label presence

Testing

  • go build -tags behaviour ./e2e/behaviour/... passes
  • go vet ./pkg/behaviourtest/... passes
  • go test ./pkg/behaviourtest/... passes
  • All three original assertions preserved in merged scenario
  • Full behaviour suite run on pool org (CI)

Checklist

  • PR title follows Conventional Commits (correct type, no breaking change)
  • No secrets in diff
  • Change is scoped to this issue only (no dispatch.feature, fork, or framework changes)

Closes #5491

Post-script verification

  • Branch is not main/master (agent/5491-merge-triage-scenarios)
  • Secret scan passed (gitleaks — c9c7540c2995d9a294ede00c2444d2eb600e4852..HEAD)
  • PR body secret scan passed (gitleaks — no-git)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

Combine two triage scenarios that each paid a full GHA workflow
wait into a single scenario with a combined dummy table. The
merged scenario performs a disallowed url_get (expect fail) and
writes the sufficient triage fixture (expect succeed), then
asserts all three outcomes after a single workflow wait: URL op
failure, fixture op success, and ready-to-code label presence.

Pre-commit could not run in sandbox (network restriction).

Closes #5491
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team as a code owner July 23, 2026 04:54
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Jul 23, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 23, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:55 AM UTC · Completed 5:01 AM UTC
Commit: e06387c · View workflow run →

@github-actions

Copy link
Copy Markdown

Site preview

Preview: https://ab27131c-site.fullsend-ai.workers.dev

Commit: e06387cfb61e7efd27dfad065b1dc6828d4b89f2

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

Copy link
Copy Markdown

Looks good to me

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Jul 23, 2026
@ifireball ifireball added the ok-to-test Allow e2e CI to run after maintainer review (must be re-applied after each push) label Jul 23, 2026
@ifireball
ifireball added this pull request to the merge queue Jul 23, 2026
Merged via the queue into main with commit e196835 Jul 23, 2026
39 of 40 checks passed
@ifireball
ifireball deleted the agent/5491-merge-triage-scenarios branch July 23, 2026 06:02
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jul 23, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 6:05 AM UTC · Completed 6:14 AM UTC
Commit: e06387c · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #5499 — merge triage sufficient + sandbox URL-block scenarios

Verdict: clean workflow, no new proposals. All improvement angles are already tracked by existing open issues.

Timeline

Time (UTC) Event
04:34 ifireball creates issue #5491 (part of epic #5490)
04:35 /fs-triage triggered with epic-scoped guidance
04:40 Triage agent completes — correctly labels ready-to-code (run 29979909160)
04:45 /fs-code triggered by human with scope constraints
04:54 Code agent completes — PR #5499 created in 9.5 min, $2.24 (run 29980333206)
04:55 Review agent auto-triggered
05:02 Review agent approves — 3 sub-agents (correctness/Opus, style/Sonnet, intent/Sonnet), zero findings, $1.58 (run 29980735100)
05:54 Human (ifireball) approves — zero comments
06:03 PR merged

Total wall-clock: ~88 min. Agent execution: ~22 min. Agent cost: ~$3.82. Rework cycles: 0.

What went well

  1. First-time correct code. The code agent produced a clean 4-addition/13-deletion change in one file that precisely matched the issue's acceptance criteria. No fix cycles needed.
  2. Appropriate review depth. Despite the brief "Looks good to me" comment, the review agent dispatched three specialized sub-agents that verified correctness (dummy runtime iteration semantics), style (Gherkin conventions), and intent coherence (scope alignment with test(behaviour): merge triage sufficient + sandbox URL-block scenarios #5491). Zero findings across all dimensions was the correct outcome for this change.
  3. Human–agent alignment. The human reviewer approved with zero additional findings, confirming the review agent's assessment was complete.
  4. Effective prompt caching. The code agent achieved 2.3M cache-read tokens vs 107 input tokens; the review agent achieved 587K cache-read vs 17 input tokens. Both leveraged caching well.

Evidence for existing issues (no new proposals needed)

  • #2842 (use cheaper model for trivial bot-authored PRs): The review agent's correctness sub-agent used Opus on a 17-line test-only change. Sonnet would likely have reached the same zero-finding verdict at lower cost.
  • #4532 and #5048 (autonomy readiness for test-only PRs): This PR adds another data point — review agent fully covered human review on a test-only single-file change with zero delta. The 52-minute gap between agent approval and human approval represents the main inefficiency, recoverable only through increased autonomy.
  • #3226 (skip retro for trivial bot-authored PRs): While Skip retro agent for bot-authored trivial PRs with no agent interaction #3226 targets PRs with no agent interaction (e.g., dependabot), this clean agent workflow — zero rework, zero review findings, zero human delta — is a borderline case where retro cost ($5+) outweighs insight. However, retros on clean workflows do provide positive autonomy-readiness signal, so the value trade-off is more nuanced than Skip retro agent for bot-authored trivial PRs with no agent interaction #3226's scenario.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test Allow e2e CI to run after maintainer review (must be re-applied after each push) ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(behaviour): merge triage sufficient + sandbox URL-block scenarios

1 participant