fix(canary): reframe smoke prompt to give GPT-4o explicit permission to echo - #2507
Merged
Merged
Conversation
…to echo Canary started flaking 2026-05-01 22:11 with model-refusal replies: - "I'm unable to do that." - "I'm unable to fulfill that request. Can I assist you with anything else?" - "I'm unable to reply with responses that don't allow me to fulfill tasks…" 3 fails / 10 recent runs ≈ 30% flake. Trigger: 2026-04-30's Platform Capabilities preamble (#2332) added the directive "Use them proactively" to the top of every system prompt. Combined with the heavy A2A + HMA tool docs further down, the model reads the contrived bare-echo prompt ("Reply with exactly: PONG") as out-of-role and intermittently refuses. Real user prompts don't hit this — only the synthetic smoke prompt does, so the right fix is in the canary's prompt phrasing, not the platform's system prompt (which is correctly priming agents toward tool use). New phrasing explicitly tells the model "this is a smoke test" and "no tools or memory are needed" so it has permission to comply. Also updates the child workspace's CHILD_PONG prompt with the same framing — same failure mode would have hit it once full-mode runs again. No code change to system prompt, no test infra change. Just two prompt strings + a load-bearing comment so future readers don't trim back to the brittle phrasing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
requested a review
from hongmingwang-moleculeai
as a code owner
May 2, 2026 06:53
HongmingWang-Rabbit
enabled auto-merge
May 2, 2026 06:53
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
1. cp_provisioner.go: only forward kind when == WorkspaceKindPlatform. Ordinary workspaces always have kind='workspace' from the DB COALESCE, so we must suppress it to keep the omitempty contract honest. Test updated to pass Kind='workspace' (real-world value) instead of empty string (DB-failure-path only). 2. workspace_provision.go ~323: log kind lookup errors at ERROR level instead of silently swallowing them. A transient DB error during a concierge restart was downgrading to the plain image with no trace. Non-existent rows (sql.ErrNoRows) are still silently ignored as expected for new workspaces. Closes #2507. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Canary smoke test went from ~100% pass to 30% flake starting 2026-05-01 22:11 with model-refusal replies:
3 fails / 10 recent runs.
Root cause
The Platform Capabilities preamble (#2332, 2026-04-30) added `"Use them proactively"` to the top of every system prompt. Combined with the heavy A2A + HMA tool docs further down, GPT-4o reads the canary's contrived bare-echo prompt (`"Reply with exactly: PONG"`) as out-of-role and intermittently refuses.
Real user prompts don't hit this — only the synthetic smoke prompt does. So the platform's prompt is correct (we want agents primed toward tool use); the fix belongs in the canary.
Change
Two strings: parent + child smoke prompts now read
Plus a load-bearing comment so a future reader doesn't trim back to the brittle phrasing.
Test plan
Out of scope (separate concerns)
🤖 Generated with Claude Code