Repository navigation
feat(voice): hand a sentence naming no offered label to Clark's turn with the focused widget's actions - #600
Merged
Merged
Conversation
…with the focused widget's actions A sentence spoken while a widget that offers actions is focused, and that names none of their labels, now goes to Clark's voice turn instead of being refused. The turn's data lists the focused widget's offered actions (binding id, label, description, input schema) as inert package words, and Clark can perform one only through perform_widget_action, under the same schema, execution policy and host card as a typed request. A labelled sentence keeps the direct spoken press; a page that cannot perform keeps the refusal.
…arts, and hand every unmatched sentence to Clark - A spoken sentence that names none of the focused widget's labels now goes to Clark's turn whatever the widget offers; it is refused only when no agent is wired. - The widget's offered actions are read when the turn starts (dataAtStart, resolved after the turn claims the conversation), and left out if the widget was closed or another one focused meanwhile. - A description is listed only from a running definition with the instance's version and digest that declares the action with the label and schema its binding recorded. - The package's label, description and input schema share one quoted, inert renderer with the perform tool's list. - The spoken approval question and the widget refusals follow the person's language. - The fixture model takes a spoken turn's binding from the turn's data, so the browser journey fails if the data is dropped.
…he widget id in offered-action listings - A dataAtStart hook that throws gives no data instead of leaving the conversation's claimed turn unreleased. - The widget id is quoted and made inert in the perform tool's list and in the focused widget's context. - The voice runner's no-focus and action-gone refusals follow the person's language, and with no agent wired an unmatched sentence is refused whether or not a widget is focused. - A closed voice session's turn gets no widget data. - The renderer and the fixture model share one binding key constant. - Tests: a throwing hook frees the conversation, typed words carrying start-time data are never steered, event-driven waits in place of sleeps, a hostile widget id, the no-focus refusal and the closed session.
Contributor
Author
|
Review attestation: ready to merge at A push to this PR makes this attestation stale; the new head needs its own review. |
Contributor
Author
|
Review attestation: ready to merge at A push to this PR makes this attestation stale; the new head needs its own review. |
…poken-action # Conflicts: # apps/runtime/src/bootstrap/voice-bootstrap.ts # docs/manifest.json
Contributor
Author
|
Review attestation: ready to merge at A push to this PR makes this attestation stale; the new head needs its own review. |
Contributor
Author
|
Review attestation: ready to merge at A push to this PR makes this attestation stale; the new head needs its own review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #444
Summary
Implements the decided option 3 for label-free spoken matching.
Before this change, a sentence spoken while a widget was focused that matched none of its offered labels was refused by the voice session ("Tôi chưa rõ bạn muốn làm gì với widget đang mở…"). It never reached Clark's turn, so "format this as a percentage" with the spreadsheet focused did nothing.
Now, when the focused widget offers
actions.perform@1actions and the session's page sentwidgetPerform: 1:perform_widget_action;The host adds no matching heuristics and packages declare no phrasings. A sentence that says a label keeps the direct spoken press. Without a page that can perform, or when the widget offers no performable actions, the sentence is refused by naming what the widget offers, as before.
Code:
apps/runtime/src/widget-perform-tool.ts:focusedWidgetActionsContextandFOCUSED_WIDGET_NOTE.apps/runtime/src/voice-session.ts: new injectedfocusedWidgetContextoption. An unmatched sentence is handed toanswerwithwidgetContextwhen the conditions above hold.apps/runtime/src/bootstrap/voice-bootstrap.ts: the inlineansweris extracted asanswerSpokenSentence(behaviour unchanged), which adds the note and the data. The new option is wired.Tests
apps/runtime/test/widget-perform.spec.ts, new describe "a spoken sentence that names none of the focused widget's offered actions". It uses a real voice socket and the node's own wiring; only the provider and the model are scripted, and the scripted model calls the realperform_widget_actiontool:widgetPerformis refused and reaches no turn;apps/web/e2e/widget-perform.spec.ts, "a range is formatted when the person says so out loud without naming the action's label". It fails without the wiring (B2 stays0.25) and passes with it.corepack pnpm verifyandcorepack pnpm invariants.Docs impact
docs/widget-development.mdand.vi.md: the voice paragraph now describes the label-free handoff in place of "not matched yet".docs/widgets-and-extensions.mdand.vi.md: the spreadsheet bullet.docs/conformance-traceability.md: the evidence.docs/manifest.json: digests.docs/cli.htmlandvi/docs/cli.html, the text-editor note) still say label-free sentences are not matched yet and need a follow-up.🤖 Generated with Claude Code