Skip to content

fix(#6518): restore config.yaml agents between pool slot leases - #6521

Merged
ifireball merged 1 commit into
mainfrom
agent/6518-restore-config-agents
Aug 23, 2026
Merged

fix(#6518): restore config.yaml agents between pool slot leases#6521
ifireball merged 1 commit into
mainfrom
agent/6518-restore-config-agents

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

  • Add snapshot/restore logic for the agents section of .fullsend/config.yaml in CleanupScenario, following the existing pattern used for kill_switch, runtime, and allowed_remote_resources.
  • Call snapshotAgents() before every cfg.SetAgents() call across dispatch.go, url_dispatch.go, and base_dispatch.go so the install-time agent list is captured before any scenario modifies it.
  • RestoreAgents() is called during CleanupScenario teardown to reset agents to their pre-scenario values, preventing harness registrations and custom agent entries from leaking across pool slot leases within a run.

Context

Pool test-repo-NN slots are reused across behaviour scenarios. CleanupScenario restored kill_switch, runtime, and allowed_remote_resources but did not restore the agents list. This caused URL host prefixes and custom agent entries from one scenario to persist into later scenarios on the same slot, leading to flaky test behavior (e.g., allowlist-negative scenarios passing incorrectly on reused slots).

Testing

  • Unit tests for snapshotAgents, RestoreAgents, and CleanupScenario agent restore added in cleanup_test.go and url_dispatch_test.go
  • resetScenarioWorld test updated to verify new fields are cleared
  • All existing tests in pkg/behaviourtest/... pass
  • gofmt, go vet pass; pre-commit could not run due to sandbox network restrictions (post-script runs authoritative check)

Closes #6518

Post-script verification

  • Branch is not main/master (agent/6518-restore-config-agents)
  • Secret scan passed (gitleaks — 997021c226272037a8cb69b32c73e14f26073b5e..HEAD)
  • PR body secret scan passed (gitleaks — no-git)

CleanupScenario already restored kill_switch, runtime, and
allowed_remote_resources between behaviour test scenarios, but did
not restore the agents list. This caused custom agent entries
(local or URL-sourced) registered by one scenario to leak into
later scenarios that reused the same pool slot.

Add snapshot/restore logic for the agents section of config.yaml,
following the existing pattern used for allowed_remote_resources:

- Add AgentsOverridden/AgentsOriginal fields to world.World
- Add snapshotAgents() called before every cfg.SetAgents() in
  dispatch.go, url_dispatch.go, and base_dispatch.go
- Add RestoreAgents() called by CleanupScenario during teardown
- Reset both fields in resetScenarioWorld between scenarios

Closes #6518
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team as a code owner August 23, 2026 13:09
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Aug 23, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:12 PM UTC · Completed 1:23 PM UTC

Commit: 87de5cc · View workflow run →

@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.48780% with 8 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/behaviourtest/steps/url_dispatch.go 76.47% 4 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

Copy link
Copy Markdown

Looks good to me

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Aug 23, 2026
@ifireball
ifireball added this pull request to the merge queue Aug 23, 2026
Merged via the queue into main with commit 68da58c Aug 23, 2026
65 of 66 checks passed
@ifireball
ifireball deleted the agent/6518-restore-config-agents branch August 23, 2026 13:30
@fullsend-ai-retro

fullsend-ai-retro Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 1:32 PM UTC · Completed 1:52 PM UTC

Commit: 87de5cc · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #6521 — restore config.yaml agents between pool slot leases

Workflow outcome: Merged cleanly in ~1h45m from issue creation. Well-scoped fix adding snapshot/restore logic for the agents section in CleanupScenario, following existing patterns. 277 additions across 9 files with 9 new unit tests. All CI passed, both agent and human reviewers approved.

Timeline

Time (UTC) Event
11:45 Issue #6518 created by ifireball
11:46–11:50 Triage agent labeled issue as blocked (on #6517)
12:26 Human triggered /fs-triage after #6517 was resolved
12:32 Re-triage removed blocked label, added ready-to-code
12:32–12:43 Code run #1 (32639667792): 76 turns, $4.84 — completed all implementation but never committed
12:57 Human triggered /fs-code retry
12:58–13:09 Code run #2 (32640871937): 82 turns, $5.47 — succeeded, created PR
13:10–13:23 Review agent approved with "Looks good to me"
13:29 Human (ifireball) approved
13:30 Merged

Main finding: first code run wasted $4.84 on completed-but-uncommitted work

The first code agent run completed ALL implementation work correctly — edited 9 files, wrote tests, all tests passed, staged files — but the model emitted end_turn after step 9 (running tests) without proceeding to step 10 (commit). This was not a timeout: the model voluntarily stopped generating. The post-script found no commits and reported "no changed files — nothing to do."

Contributing factors:

  • ~30 of 76 turns spent on context gathering before writing code
  • ~6 turns spent working around pre-commit HTTP 403 errors (sandbox network restrictions prevented fetching remote hook repos)
  • Broken time budget tracking (AGENT_START env var didn't persist across isolated Bash calls, producing nonsensical remaining-time calculations)
  • Commit was sequenced as the final step (step 10), making it the most vulnerable to any premature stop

The second run succeeded because it used git commit --no-verify to bypass the sandbox-blocked pre-commit hook after manually verifying all lint/test/format checks.

Existing issues providing coverage (no new proposals needed)

  • fullsend-ai/agents#773 — checkpoint commits before timeout. This retro provides evidence that the failure mode also occurs without timeout (model end_turn), broadening the justification. The proposed fix (commit earlier in the workflow) would prevent both timeout and non-timeout cases.
  • fullsend-ai/agents#821 — commit-existence gate before declaring success. Would have caught the empty-commit outcome.
  • fullsend-ai/fullsend#5075 — code agent reports success when time budget exhausted without producing code. The workflow reported success despite no code being produced. This retro shows the same reporting gap occurs for non-timeout empty outcomes.
  • fullsend-ai/agents#281 — pre-disable git hooks inside sandbox. Would have saved ~6 turns in the first run and eliminated the commit-blocking hook failure in both runs.

Review quality

The review agent approved a 277-line, 9-file change with "Looks good to me" and no inline comments. The human reviewer (who authored the issue) also approved without comments. Given the PR followed an established snapshot/restore pattern, was well-tested, and all CI passed (80.49% patch coverage), this level of review depth appears appropriate for the change class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(behaviour): CleanupScenario does not restore config.yaml agents or allowlist

1 participant