Skip to content

Send a question's chosen preview back to Claude Code (annotations) - #24356

Merged
teamleaderleo merged 7 commits into
feat-cmux-nextfrom
aff-ask-annotations
Oct 26, 2026
Merged

teamleaderleo merged 7 commits into
feat-cmux-nextfrom
aff-ask-annotations

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Leo's ask, his words: "ask user question tool and its various states need looking at allegedly there's some sort of vibe-coded thing that agents can do to have prettier ask question situations...? idk but we could look into that/research that then support it..."

Leo-ask: ask-question-states

What was wrong

Claude Code's AskUserQuestion lets an agent attach a preview (a mockup or code sample) to each option. The answer the host sends back may carry annotations (question text -> {preview?, notes?}), and the agent then reads "Q"="A" selected preview: <mockup>, so it knows which mockup the person picked. cmux sent only answers. The card never sent the chosen preview, the pane relay refused any annotations param, and acpmux would have dropped it anyway.

The fix

  • Card (question/model.ts reply): for a Claude Code ask, each single-select item whose chosen option has a preview adds annotations[prompt] = {preview}. Other answers, multi-selects, the Chief and Codex send nothing new. App.tsx and direct.ts pass it through to _acpmux/permission_respond.
  • Pane relay, in both Swift AcpmuxPaneMethods and cmux-agent-pane-policy, from the same policy.json: annotations is a known param. It goes only beside answers, to a pending question the daemon sent, keyed by that question's own items, and each value is {preview?, notes?} strings of at most 64 KiB.
  • acpmux (questions::check_annotations, respond_permission_annotated):
    • It takes annotations only from the person's own connection, and only with answers. A phone's signed answer or a session's answer can't carry them.
    • For Claude Code's AskUserQuestion, a preview must match one of that question's option previews as normalized, and the agent gets its own original text back, since Claude Code refuses a preview it didn't write. Notes must be 1 to 8192 bytes. Any other field, or any other question, is refused.
    • For any other question, the annotations are dropped, so the answer still goes through.
  • Protocol IR, generated TS types and acpmux-schema.json document the param.

Not in this PR: notes UI (the protocol carries notes now; the card doesn't offer a field yet). HTML previews, markdown descriptions, answered and skipped summaries, and MCP elicitation belong to the transcript-parity lane.

Relay analysis

This adds one param to an existing method of the agent pane relay (`_acpmux/permission_respond`). It doesn't add a method, and it changes nothing remote: `RemoteAcpmuxWire` still passes only the option.

  • Command or content execution: none. The value is checked text that ends up in Claude Code's own tool input, and Claude Code validates it again (`admitCardAnswer`).
  • Unowned objects: the relay accepts it only for a question the daemon sent this pane, keyed by that question's items. acpmux accepts it only from the person's connection, for the pending prompt, beside answers.
  • Local-state exposure: none. Nothing is read back. A preview has to be text the agent itself wrote, and notes are capped at 8192 bytes.

Test

Red at the test-only commit 3fc192c, green at the fix:

  • bun test src/agent-session/acpmux/question/model.test.ts src/agent-session/acpmux/question/record.test.ts (from webviews/, on cmux16): red 11 pass / 1 fail ("a Claude answer sends the chosen option's preview"), green 12 pass / 0 fail.
  • acpmux questions_annotations* (tests/hub_integration/questions.rs: carried with the agent's own preview text; refused when the preview isn't the agent's, the key or field is unknown, notes are empty or too long, or answers are missing; dropped for a question that isn't Claude's) and the policy cases in tests/cases/check.json run on the fleet with cmux-tui-rust-check.sh test. Results are below.
  • cmux-ci compile-check --app at the head: see below.

Observed / Not run

Observed: the webview tests above on cmux16. Also at the head on cmux16: question/, needsYou/ and test/question-card-submit.test.tsx all green, tsc --noEmit clean, and prepush-gate.py reports no new violations.

Not run: a live Claude Code run that reads the annotation back (the fake agent echoes the tool input instead), and the Swift relay unit tests (the shared policy cases cover the same rule in Rust; the Swift harness doesn't read questions state yet).

No capture: no visible UI change; the card looks the same and sends one more field.

Gallery: agent-pane.question-card

Changelog

Changed: When you answer an agent's question that has option previews, Claude Code now learns which preview you picked.

Adopted and rebased (2026-10-25)

The stale-pr-reconciler, word for word: "These PRs in your area have no live owner (Leo, 2026-10-25: "these should never have been this stale for this long"). Adopt each: pr-owner claim N (in that repo), rebase onto its base, an independent review at the new head, then land it".

  • Squashed onto feat-cmux-next (it was 3,151 commits behind). Previews are now {text, format} objects on base; the card already reads preview.text, and check_annotations compares against the trimmed raw option text, so the agent still gets its own text back. The only conflict was where the new function sits in hub/questions.rs.
  • Codex review (gpt-6-astra xhigh) round 1, BLOCK: 1. a selected preview over 65,536 UTF-8 bytes rode along as an annotation, and both pane relays then refused the whole answer. Fixed: the card leaves out a preview over the relay's bound (MAX_ANNOTATION_BYTES, the same as policy.json maximum_annotation_bytes), and the answer still goes through. Test first: a preview over the relay's 65,536-byte bound is left out, and the answer still goes (32,768 x "é" is exactly the bound; one byte more, and 65,537 ASCII bytes, are left out).
  • Round 2: SHIP at f7c5b23. Round 3 (a vp fmt reformat of that test only): SHIP at 079ae3e.
  • Rebased again onto 714af0c, past acpmux: chat multiplayer A: person author, provenance, answeredBy, queue.not_author #25268 (chat multiplayer: author, answeredBy). respond_permission_annotated now returns (pending, answered_by, annotations), and the request handler keeps base's answerer.author.
  • Round 4 (fresh reviewer), BLOCK: 1. the annotation checks ran before the pending lookup, so a losing annotated answer from a non-person connection got person_required instead of -32002 with the first answer's answeredBy/outcome. Fixed by moving them to after the lookup. Test first: a_losing_answer_is_told_who_answered_first now also sends an annotated losing answer (red at 2eddd47, green at 2551dc9, hub_integration questions/permission/author 40 pass).
  • Round 5: SHIP at 2551dc9. The Release compile (Xcode 26) passed at that head. The Swift test reds are in untouched areas (password manager route, icon family, sidebar rail), and the generated-files red is base's open-remote CLI name (cmux-next: one verb to open a persistent remote terminal (Linux VMs too): "open <host> in a bottom split over ssh" #25092).

Tests on the rebased branch (fast-checks on Blacksmith)

  • Red at the test commit 2838a9b: questions_annotations_carry... and ..._refused_and_the_ask_stays fail; policy full_check_order fails on the new cases.
  • Green at fb4cc7c: acpmux --test hub_integration questions 8 pass, --lib questions 5 pass, cmux-agent-pane-policy parity 16 pass; clippy -D warnings and fmt clean. cmux-agent-pane-policy --test typescript the_allowlist_is_what_the_page_sends fails the same way on base de67de9 (_acpmux/mcp/reconnect), so it isn't from this PR.
  • Webviews: red at a898660 (the bound test fails), green at f7c5b23: question/, needsYou/, test/question-card-submit.test.tsx 46 pass, tsc --noEmit clean.
  • Base reds this PR shows too: Script tests (release-compile-derived-data.test.sh "bench OTHER_SWIFT_FLAGS", fails on base 5e6cb13); gallery health agent-pane.composer#file-not-attached (broken on other PRs too).

No timers or observers added. No capture: the card looks the same and sends one more field.

Agent session: big-red
🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 079ae3e: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr24356 -f ref=aff-ask-annotations -f platform=macos -f backend_mode=local

Gallery before/after for 2551dc9: 16 states rendered, 0 images differ, 5 more touched states not rendered (picked by the Gallery: line in the PR body). Images are in the run's artifact, not on this public PR.

gh run download 40279999965 -R manaflow-ai/cmux -n gallery-advisory-24356-2551dc935a6a3db410425a083d3f7d0bf9eaa1d0 (then read gallery-review.json)

States
State Theme Status Diff Picked by
agent-pane.question-card/pending-single Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/pending-single Apple System Colors unchanged 0% pr-body
agent-pane.question-card/folder-trust Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/folder-trust Apple System Colors unchanged 0% pr-body
agent-pane.question-card/folder-trust-picked Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/folder-trust-picked Apple System Colors unchanged 0% pr-body
agent-pane.question-card/pending-multi Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/pending-multi Apple System Colors unchanged 0% pr-body
agent-pane.question-card/pending-other-typing Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/pending-other-typing Apple System Colors unchanged 0% pr-body
agent-pane.question-card/pending-with-preview Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/pending-with-preview Apple System Colors unchanged 0% pr-body
agent-pane.question-card/pending-with-preview-hover Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/pending-with-preview-hover Apple System Colors unchanged 0% pr-body
agent-pane.question-card/pending-with-preview-fenced Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/pending-with-preview-fenced Apple System Colors unchanged 0% pr-body
agent-pane.question-card/pending-4-questions Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/pending-4-questions Apple System Colors unchanged 0% pr-body
agent-pane.question-card/codex-user-input Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/codex-user-input Apple System Colors unchanged 0% pr-body
agent-pane.question-card/acp-interactive Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/acp-interactive Apple System Colors unchanged 0% pr-body
agent-pane.question-card/chief-asks Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/chief-asks Apple System Colors unchanged 0% pr-body
agent-pane.question-card/reply-refused Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/reply-refused Apple System Colors unchanged 0% pr-body
agent-pane.question-card/answered Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/answered Apple System Colors unchanged 0% pr-body
agent-pane.question-card/answered-remote Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/answered-remote Apple System Colors unchanged 0% pr-body
agent-pane.question-card/answered-by-session Apple System Colors Light unchanged 0% pr-body
agent-pane.question-card/answered-by-session Apple System Colors unchanged 0% pr-body

Pick other states with a Gallery: entry.id, entry.id/variant line in the PR body.

Probably yours: accent-glass-invariants.test.ts fails here but not on main's latest full suite. Seen on other PRs too (likely flaky): claimPaint.test.tsx, claimPaint.test.tsx, agent-pane-agents-pill.test.ts and 12 more.

CI failed on 079ae3ee5a (run 40263423131 attempt 1): 2 code, 1 unknown.

Job Verdict Why
web / react-apps-base-check (4) code a web test failed
web / react-apps-base-check (1) code a web test failed
web / react-apps-check unknown no known signature; failed step: Check webview test shards

Failures:

  • seen on other PRs claimPaint.test.tsx a pooled Settings page claimed by the host reports its paint once its first screen is in: failed (job)
  • seen on other PRs claimPaint.test.tsx a Settings page loaded fresh reports its first paint once its first screen is in: failed (job)
  • new in this PR accent-glass-invariants.test.ts no orange or system accent and no opaque glass on 2140 gallery stages: failed (job)
  • seen on other PRs agent-pane-agents-pill.test.ts by default the running agents sit in the tray above the composer, not in the transcript: failed (job)
  • seen on other PRs agent-pane-step-rows.test.ts every step is one 26px row on one rhythm, across transcript rows: failed (job)
  • seen on other PRs chat-environment-scroll-stability.test.tsx long-chat scroll leaves the Left chip mounted and stable, while own cwd changes update it: failed (job)
  • seen on other PRs chat-environment-scroll-stability.test.tsx long-chat scroll leaves the Right chip mounted and stable, while own cwd changes update it: failed (job)
  • seen on other PRs chat-inspector-overlay-toggle.test.ts chromium, header: the toggle is above the open overlay and a click on it closes the Inspector: failed (job)
  • seen on other PRs agent-pane-background-tasks.test.ts the summary card lists the chat's background tasks: running ones first, the finished folded: failed (job)
  • seen on other PRs agent-pane-hairline-focus.test.ts composer at 1x: focus is the resting hairline a step brighter, no ring: failed (job)
  • ... 6 more in the job logs
Matched log lines
web / react-apps-base-check (4): (fail) a pooled Settings page claimed by the host reports its paint once its first screen is in [175.97ms]
web / react-apps-base-check (1): (fail) the summary card lists the chat's background tasks: running ones first, the finished folded [941.92ms]

Not re-run automatically: web / react-apps-base-check (4), web / react-apps-base-check (1), web / react-apps-check are not machine failures.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 714af0c to compare; do not merge until green; see lane-rules

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 22, 2026
@blacksmith-sh

This comment has been minimized.

@blacksmith-sh

blacksmith-sh Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Found 45 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmpxhvgec95/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpyxk26ofz/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL
OCKING_CHECKS'/
bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL
OCKING_CHECKS'
View Logs
bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL
OCKING_CHECKS'/
bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL
OCKING_CHECKS'
View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
test/agent-pane-background-tasks.test.ts/agent-pane-background-tasks.test View Logs

...and 35 more test failures. View all on Blacksmith

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-ix34d created for this PR by the cmux coordinator.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Orphaned: no activity for 13 h, and no lane holds a lease on it. Adopted by the orphan sweep: merged feat-cmux-next in at 96079b6. A later pass queues it for the merge hub if it is reviewed, small and red only on base reds.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next crash-class rules / cmux-next crash-class rules is red on 13 of 13 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next guards / nightly-guards / workflow-guard-tests / release-notary is red on 17 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › Checks job structure is red on 17 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by hb-acpmux on big-red · session

teamleaderleo and others added 5 commits October 26, 2026 04:58
Claude Code's AskUserQuestion answer may carry annotations (question
text -> {preview, notes}); the agent then reads which mockup the person
picked. The card never sends them, the pane relay refuses them and
acpmux drops them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d41-f5e8-7631-b4aa-a544e3ef96a8&host=cosmic-gray
The card adds annotations beside a Claude Code answer: for each
single-select item, the chosen option's preview. The pane relay
(Swift and cmux-agent-pane-policy) lets annotations through only beside
answers, keyed by the question's own items, {preview?, notes?} strings.
acpmux takes them only from the person's own connection and with
answers; for Claude Code's AskUserQuestion a preview must be one of
that question's option previews as normalized, the agent gets its own
text back (Claude Code refuses a preview it did not write), and notes
are 1 to 8192 bytes. Any other question drops them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d41-f5e8-7631-b4aa-a544e3ef96a8&host=cosmic-gray
…red)

Codex review: a selected preview over 65,536 UTF-8 bytes rides as an
annotation, and both pane relays refuse the whole answer for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The card sends a chosen preview as an annotation only up to the pane
relay's 65,536 UTF-8 bytes (policy.json maximum_annotation_bytes); a
longer one is left out, so the answer itself still goes through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next fast gates / post cmux-next/fast-gates is red on 16 of 18 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

teamleaderleo and others added 2 commits October 26, 2026 05:09
Codex review: annotations were checked before the pending lookup, so a
second answer from a non-person connection got person_required instead of
-32002 with the first answer's answeredBy and outcome.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A losing answer that carries annotations is told who answered first
(-32002 with answeredBy and outcome), as any other losing answer is; the
person and answers-only rules for annotations still apply to a pending ask.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

high-signal review link / link is red on 14 of 18 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › Script tests (pin-cmux-tui, embed-cef, ensure-cef store) is red on 7 of 11 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo
teamleaderleo merged commit e3f2015 into feat-cmux-next Oct 26, 2026
228 of 245 checks passed
@teamleaderleo
teamleaderleo deleted the aff-ask-annotations branch October 26, 2026 06:28
teamleaderleo added a commit that referenced this pull request Oct 26, 2026
…et cover exactly what the person picked (fails: annotations are refused without the person, the daemon verb refuses them, the ticket names no answers)

Codex review of #25424: the remote wire dropped a question's annotations
(the chosen preview, #24356), and the gesture ticket named no answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Oct 26, 2026
… under the person's proof, and its ticket binds the answers

Codex review of #25424:
- The remote wire, the session daemon's client and agent-session-permission
  now carry a question's annotations (Claude Code's chosen preview and
  notes, #24356) beside its answers. acpmux counts annotations from a
  connection that is not the person only under the person's decision proof
  over them: the answers proof gains a seventh part, the annotations'
  digest, when the answer has annotations. A phone's signed answer still
  never carries them.
- A permission answer's gesture ticket now names the question's answers and
  annotations, compared by their canonical JSON, so a ticket the page took
  for one answer never redeems or gets signed for another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

This branch had an error being deployed

1 failed and 3 inactive deployments
artifacts — 2551dc93 Deployed Oct 26, 2026 by teamleaderleo via add the Windows daemon to the tree #55091
sentry-dsyms — 2551dc93 Deployed Oct 26, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #55091
cmux-tui-merge-tree — 2551dc93 Deployed Oct 26, 2026 by teamleaderleo via publish the merge tree #6171
ci-routing-agent — 2551dc93 Deployed Oct 26, 2026 by teamleaderleo via namespace autofix #3061
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant