feat(studio): let an agent edit text and styles, guarded by miguel-heygen · Pull Request #3518 · heygen-com/hyperframes

miguel-heygen · 2026-08-27T00:42:12Z

What

studio_set_text and studio_set_style, the first tools that change the composition.

Also plumbs the write-blocked state, which was the blocker for shipping any write at all.

Stacked on #3517.

Why

Four read tools shipped before this one on purpose. An agent that can see the composition but not touch it is only half useful, but a tool that writes without a guard is worse than none, because Studio has several paths where a failed write reports success.

How

Both act on the current selection and take no handle. That is forced rather than chosen: the handlers read the ambient React selection, and applyDomSelection only schedules a state update, so selecting and committing inside one call would write to whatever was selected before. Select first, then edit.

The write guard. domEditSaveQueuePaused and the external-file conflict both lived on App and were unreachable from the tool surface, so canWrite was optimistic and the code said so in a comment. They now derive into a single writeBlockedReason on the shell context: one field, one owner, conflict taking precedence because resolving it is what unblocks the queue.

That guard matters more than it looks. Both states are banners in Studio with no lock behind them, so nothing else was stopping a programmatic write from landing on top of a conflict the user had been asked to adjudicate.

Three things the tools refuse to fake:

They check the outcome, not the absence of a throw. Awaiting the handler proves nothing here; the tagged outcome added in #3510 is what proves the write landed.

A partial style result is reported as partial. handleDomStyleCommit is one property per call, so N properties are N commits. The result carries applied and rejected maps rather than one boolean that would have to pick a side.

Style commits run sequentially, never concurrently. Two commits racing through Studio's client-side read-modify-write can record undo entries that both claim the same starting content.

Every decline reason maps to a hint naming what to do instead, so a refusal routes the agent rather than just stopping it.

Test plan

Unit tests added/updated
Manual testing performed
Documentation updated (if applicable)

14 tests in contentTools.test.ts:

Text writes and reports the result, including changed: false when it already said that.
A blocked write never reaches the handler, asserted for both the conflict and paused-queue reasons.
A declined commit is not reported as success, which is the whole point of the outcome plumbing.
Decline reasons map to hints naming the alternative tool.
Concurrency is measured, not assumed: the sequential-commit test tracks in-flight count and asserts it never exceeds 1.
Partial style success reports applied and rejected separately.
Empty and non-object styles are rejected without committing.

Full package suite 4577 passing across 412 files. bunx tsc --noEmit, bunx oxlint and bunx fallow audit --fail-on-issues all clean.

Still no browser verification anywhere in this stack. That is the final unit.

Note

The push for this branch raced a full disk; the commit and remote ref landed, the local upstream config did not and was repaired afterwards. Nothing in the diff is affected.

Adds `studio_select` and `studio_seek`, so an agent and the human are looking at the same element and the same instant. Selecting reveals the inspector, exactly as a click does, which is what makes the agent's move visible. Selection is shared state, not a per-call argument, and that is forced rather than chosen. Most of Studio's edit handlers read the ambient React selection, and `applyDomSelection` only schedules a state update, so selecting and committing inside ONE call would write to whatever was selected before. Two tool calls are separated by a render, so the contract is select first, then act. That is also how a human works: click, then type. `studio_seek` uses `requestSeek`, not `setCurrentTime`. The latter only moves the timeline's displayed number and leaves the composition where it was. Two things the tools refuse to fake: Seek does not clamp. `seek()` already clamps against the adapter's duration, which can differ from the store's, and clamping again would give that invariant two owners that can disagree. The tool reports where the playhead actually landed instead, read back afterwards. `requestSeek` is fire-and-forget, so it cannot report that no adapter was mounted to receive it. The tool compares the playhead before and after and fails rather than claiming a seek that never happened. Select separates three failures that a single message would have merged: the preview is not mounted yet (wait), no element matches the handle (re-read), and the element cannot be selected (try a neighbour). The agent's next move differs for each, so collapsing them would cost it a round trip or a retry loop.

Renders the composition to a PNG at a given time and returns the URL. This is what turns the tool set from a remote control into a loop: author a change, capture the instant it affects, look, adjust. No agent can judge motion from source, because "what does this look like at 2.4 seconds" is not a question a file answers. Reuses Studio's existing capture endpoint via `buildFrameCaptureUrl` rather than inventing a second one. Two things this does not fake: It reports the time the playhead LANDED on, not the time requested. The player clamps, so those differ at the ends, and attaching the wrong time to a frame is how an agent draws a confident wrong conclusion about motion. It waits before capturing, by default 150ms. The frame is rendered from the file on disk, and the render cache is cleared by a file watcher with a 40ms write-stability threshold, so a capture that beats the watcher renders the PRE-edit composition. That exact staleness was a real bug here once. An agent reading a stale frame as "my edit failed" would thrash, so the wait is on by default, `settleMs` makes it tunable, and the tool description names the failure rather than leaving it to be rediscovered. It probes with HEAD before returning, so a URL that 404s comes back as a failure with a hint instead of as a link the agent cannot render.

Everything about one element in one call: resolved styles, text fields, box, data attributes, GSAP animations, and what the element will and will not accept. The point is to prevent a failed write rather than to satisfy curiosity. `can.reasonIfDisabled` is passed through verbatim from Studio's own capabilities, so an agent that reads first should never attempt an edit the element would refuse. Three things it refuses to get wrong: Animations are reported ONLY for the current selection, because that is the only element Studio parses them for. Attributing them to any other element would be reporting the wrong element's motion, which is worse than reporting none. When a handle names something else the field is empty and `animationEditingBlocked` says why. `animationEditingBlocked` also carries the two states where animation editing is off entirely, multiple timelines and an unsupported timeline pattern. Both live on the selection context. Learning them from a read costs one call; learning them from a failed write costs a retry loop. Inspecting a handle does NOT change what is selected. It is a read, and stealing the human's selection would be a side effect they did not ask for. There is a test asserting `applySelection` is never called. Nothing selected and no handle given is a failure, not an empty result. An empty result would assert "this element has nothing", which is a different and false claim.

The first tools that change the composition. Both act on the current selection and take no handle, which is forced rather than chosen: the handlers read the ambient React selection, and `applyDomSelection` only schedules a state update, so selecting and committing inside one call would write to whatever was selected before. Select first, then edit. Also plumbs the write-blocked state, which was the blocker for shipping any write at all. `domEditSaveQueuePaused` and the external-file conflict both lived on App and were unreachable from the tool surface, so `canWrite` was optimistic and a comment said so. They now derive into a single `writeBlockedReason` on the shell context: one field, one owner, conflict taking precedence because resolving it is what unblocks the queue. That guard matters more than it looks. Both states are BANNERS in Studio with no lock behind them, so nothing else was stopping a programmatic write from landing on top of a conflict the user had been asked to adjudicate. Three things the tools refuse to fake: They check the outcome, not the absence of a throw. Studio has several paths where a failed commit resolves anyway, so awaiting the handler proves nothing. The tagged outcome added earlier is what proves the write landed. A partial style result is reported as partial. `handleDomStyleCommit` is one property per call, so N properties are N commits; the result carries `applied` and `rejected` maps rather than a single boolean that would have to pick a side. Style commits run sequentially, never concurrently. Two commits racing through Studio's client-side read-modify-write can record undo entries that both claim the same starting content. There is a test that measures concurrency rather than trusting the loop. Every decline reason maps to a hint naming what to do instead, so a refusal routes the agent rather than just stopping it.

…3517) Everything about one element in one call: resolved styles, text fields, box, data attributes, GSAP animations, and what the element will and will not accept. The point is to prevent a failed write rather than to satisfy curiosity. `can.reasonIfDisabled` is passed through verbatim from Studio's own capabilities, so an agent that reads first should never attempt an edit the element would refuse. Three things it refuses to get wrong: Animations are reported ONLY for the current selection, because that is the only element Studio parses them for. Attributing them to any other element would be reporting the wrong element's motion, which is worse than reporting none. When a handle names something else the field is empty and `animationEditingBlocked` says why. `animationEditingBlocked` also carries the two states where animation editing is off entirely, multiple timelines and an unsupported timeline pattern. Both live on the selection context. Learning them from a read costs one call; learning them from a failed write costs a retry loop. Inspecting a handle does NOT change what is selected. It is a read, and stealing the human's selection would be a side effect they did not ask for. There is a test asserting `applySelection` is never called. Nothing selected and no handle given is a failure, not an empty result. An empty result would assert "this element has nothing", which is a different and false claim.

Resolve conflicts from squash-merged #3515 (selection tools). Branch retains frame + inspect tools from the stack. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Resolve conflicts from squash-merged #3515 (selection tools). Branch retains the full WebMCP tool stack from the PR chain. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

`studio_transform` does what a drag does, and then checks. The box in the result is READ BACK after the write, never echoed from the request, and `applied` lists what actually took effect. That is not belt-and-braces. The plan for this unit said to re-derive the geometry handlers' behaviour rather than trust any description of them, and doing that turned up three different behaviours behind one interface. The handlers on `DomEditActionsValue` are the GSAP-AWARE wrappers, aliased in `useDomEditSession.ts:534-538`, not the CSS ones in `useDomGeometryCommits.ts` that an earlier note in this workstream described. `handleGsapAwarePathOffsetCommit` and `handleGsapAwareRotationCommit` are `if (gsapCommitMutation) { ...intercept... }` with no else branch. Their own comments say the absence is deliberate: position and rotation are written as GSAP code and there is no CSS fallback to write to. So they can return having done nothing. `handleGsapAwareBoxSizeCommit` is not like the other two. It runs through `runGestureTransaction` with separate scale and width/height routes, so resize works more generally. Reading back is what turns that middle case from a silent lie into a reported one. A move that did nothing comes back in `unchanged` with a reason. Three smaller decisions: Operations re-read between each other, so a move is judged against the box AFTER a resize in the same call. Comparing against the original would credit the resize's change to the move. Rotation is reported as dispatched, not verified. `rotate` is an individual transform property and does not appear in the computed transform, so there is no honest box-derived signal, and claiming one would be worse than saying so. x pairs with y and width pairs with height. Accepting one alone would mean inventing the other from the current value, which moves the element somewhere the caller did not ask for. The pairing rule and its minimum live in one `parsePair` helper rather than as four separate branches.

vanceingalls

Style holds up under adversarial read. Guard model is single-owner (writeBlockedReason derived on the shell context with conflict-first precedence over paused-queue), and it is enforced BEFORE dispatch, so both banner states now behave as locks for programmatic writes. Outcome-tagged writes (not "no throw") plus the partial-success applied / rejected maps plus measured sequential dispatch (in-flight counter asserted in tests) match the tagged-outcome contract set up in #3510 and preserve undo integrity through N-property style commits.

Schema shape mirrors #3519: additionalProperties: false, explicit typeof guards for text and styles, non-object / null / array styles refused, and empty-entries refused so an "empty commit" cannot fake success. Decline reasons are mapped to actionable hints (routes the agent instead of just stopping it), and geometry-property is refused with a pointer to the transform tools.

The intentional decision to route through handleDomTextCommit / handleDomStyleCommit rather than add a second sanitizer at the tool boundary is consistent with the "route to Studio's real editing paths, don't reinvent them" premise across the stack — the guarded surface here is around conflict-state, outcome verification, and partial application, not value filtering.

Reviewed head be6fc0b

Review by Via

somanshreddy

Second-eyes pass (alongside @via), focused on the guarded-write mechanism at be6fc0b0. Direct manual review — I deliberately did not spawn a full second automated pass because the PR is DIRTY and mid-stack; the diff that finally ships will differ from this head (see the state note), so a line-by-line pass belongs after the rebase. What I verified is the core mechanism, which is stable regardless.

Core write mechanism — sound, verified at source

The guard closes the real gap. writeBlockedReason = externalFileConflict ? "…" : domEditSaveQueuePaused (useStudioContextValue.ts) — conflict takes precedence, derived into one shell-context field, wired into the tool via getWriteBlockedReason (StudioAgentTools.tsx) and checked before dispatch (guardWrite). Both states were banners with no lock behind them, so this is what actually stops a programmatic write landing on a conflict the user is still adjudicating. Correct and well-motivated.
Sequential style commits are genuinely sequential. for (…) { await deps.setStyle(…) } — in-flight never exceeds 1, and the comment names the failure it prevents (two client-side read-modify-write commits recording undo entries that both claim the same starting content). The concurrency test that tracks in-flight count is the right shape.
Outcome-checked, not throw-checked. fromOutcome keys on outcome.ok and maps each decline reason to an actionable hint; partial style results report applied/rejected separately rather than collapsing to one boolean. Honest.
Interleaved human/agent selection race (the class-of-bug from the #3515 review): it's an inherent, documented property of the shared-selection design — the guard reads the current selection at dispatch and doesn't lock it, and the file's own docstring calls out "select first, then edit, two calls separated by a render." I agree with @rames D Jusso that this is an integration-test target, not a blocker.

State / hygiene — worth resolving before the final pass (my main finding)

This PR is DIRTY and the diff is +2746/-249 across 75 files, not the ~+999/-70 the write surface actually is. Two things are tangled in:

Unrelated render-pipeline changes ride along — producer/artifactTransaction (+177), core/colorGrading, engine/frameCapture, cli/snapshot, release notes v0.8.17–20. These aren't the WebMCP write tools; they read as diff-noise from a base (feat/studio-webmcp-frame) that's behind main.
contentTools.ts here is the pre-fix version — field = … : undefined, the exact studio_set_text field-resolution bug that #3521 fixed and that has already merged to main. So a rebase onto main will conflict in contentTools.ts, and merging this head as-is would reintroduce the child-field write bug (empty patch → server 400 persist-failed).

Recommend rebasing onto current main first: the diff should shrink to the real write surface and contentTools.ts should re-converge with #3521's field resolver. That reconverged diff is what I'd want a final pass to read. No concern with the mechanism itself — it's the packaging.

(Not blocking on my end — this is second eyes; @via holds the primary pass. Tests weren't run locally; per the PR body the package suite is green and the guard/concurrency tests are mutation-meaningful.)

Resolve conflicts from parallel squash merges in the stack. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The base branch was changed.

…ntent # Conflicts: # packages/studio/src/webmcp/StudioAgentTools.tsx # packages/studio/src/webmcp/useStudioAgentTools.test.tsx # packages/studio/src/webmcp/useStudioAgentTools.ts

vanceingalls

R2 verify at 8078783 — clean post-#3516-merge base retarget. PR-surface (packages/studio/src/webmcp/**) is byte-identical to prior stamp head be6fc0b — GitHub compare be6fc0b…80787832 shows ahead-by-7 with zero studio files changed; the 7 upstream commits touch packages/core/runtime and packages/player only (main advanced, not the PR). Now +882/-7 across 10 files, all studio-webmcp/App/Context — matches the guard-mechanism scope I signed off on. (Miga cited +861/-7; actual is +882/-7, but the delta is upstream-only, not on this PR's surface.) CI green at HEAD. Stamping.

Review by Via

somanshreddy

Re-review at the reconverged head 8078783233 (base now main, diff +882/-7 vs main). This is the full line-pass I deferred when the PR was mid-rebase/DIRTY — done now on the clean merge base. Manual pass (not codex).

The revert risk I flagged earlier is cleared. contentTools.ts and transformTools.ts don't exist on main, so they're introduced fresh here — there's no prior main version for the reconverge to have reverted. The 7 deletions are all benign wiring/test-count churn (context lines in useStudioAgentTools, the StudioAgentToolsDeps interface extension, and the toHaveBeenCalledTimes(5) assertions superseded by 8), not a lost fix.

The write tools are correctly guarded and anti-vacuous:

guardWrite/guard check getWriteBlockedReason() then a live selection before any dispatch, and return typed blocked/invalid failures.
studioSetText: the optional field resolves to undefined when absent (→ the element's own text, per the schema description) — correct, not an empty-patch; changed is computed against the captured before.
studioSetStyle: rejects an empty/non-string-valued styles object ("an empty commit would report success having done nothing") and commits sequentially, one property per call, with a comment explaining why (racing client-side read-modify-write would corrupt undo entries). Collects applied/rejected.
studioTransform: paired-field validation (must-be-given-together / min), and fails blocked when applied.length === 0 rather than reporting a no-op success.
Registration: 3 new tools wired through runToolBody; the count assertion moved 5→8 and a not-ready case asserts registerTool isn't called. contentTools.test.ts (+201) and transformTools.test.ts (+179) exercise the guard/anti-vacuity/outcome paths.

No blocking findings — the reconverged diff is clean and matches the stable-core mechanism I verified on the prior head.

Verdict: approve-pending-CI. Standard checks are still running at this SHA (Typecheck/Test/CLI smoke pending, not red). I'll flip this to an approval the moment CI is green — ping me when it lands or I'll catch it, and I won't stamp over a pending required check.

The squash merge of #3518 re-introduced the old assertion that document.modelContext is absent. The polyfill from #3514 installs it as a fallback — that is expected behavior. Same fix as #3532: remove the assertion, keep the boot-cleanly contract.

* feat(studio): let an agent drive Studio's selection and playhead Adds `studio_select` and `studio_seek`, so an agent and the human are looking at the same element and the same instant. Selecting reveals the inspector, exactly as a click does, which is what makes the agent's move visible. Selection is shared state, not a per-call argument, and that is forced rather than chosen. Most of Studio's edit handlers read the ambient React selection, and `applyDomSelection` only schedules a state update, so selecting and committing inside ONE call would write to whatever was selected before. Two tool calls are separated by a render, so the contract is select first, then act. That is also how a human works: click, then type. `studio_seek` uses `requestSeek`, not `setCurrentTime`. The latter only moves the timeline's displayed number and leaves the composition where it was. Two things the tools refuse to fake: Seek does not clamp. `seek()` already clamps against the adapter's duration, which can differ from the store's, and clamping again would give that invariant two owners that can disagree. The tool reports where the playhead actually landed instead, read back afterwards. `requestSeek` is fire-and-forget, so it cannot report that no adapter was mounted to receive it. The tool compares the playhead before and after and fails rather than claiming a seek that never happened. Select separates three failures that a single message would have merged: the preview is not mounted yet (wait), no element matches the handle (re-read), and the element cannot be selected (try a neighbour). The agent's next move differs for each, so collapsing them would cost it a round trip or a retry loop. * feat(studio): give an agent eyes with studio_frame Renders the composition to a PNG at a given time and returns the URL. This is what turns the tool set from a remote control into a loop: author a change, capture the instant it affects, look, adjust. No agent can judge motion from source, because "what does this look like at 2.4 seconds" is not a question a file answers. Reuses Studio's existing capture endpoint via `buildFrameCaptureUrl` rather than inventing a second one. Two things this does not fake: It reports the time the playhead LANDED on, not the time requested. The player clamps, so those differ at the ends, and attaching the wrong time to a frame is how an agent draws a confident wrong conclusion about motion. It waits before capturing, by default 150ms. The frame is rendered from the file on disk, and the render cache is cleared by a file watcher with a 40ms write-stability threshold, so a capture that beats the watcher renders the PRE-edit composition. That exact staleness was a real bug here once. An agent reading a stale frame as "my edit failed" would thrash, so the wait is on by default, `settleMs` makes it tunable, and the tool description names the failure rather than leaving it to be rediscovered. It probes with HEAD before returning, so a URL that 404s comes back as a failure with a hint instead of as a link the agent cannot render. * feat(studio): add studio_inspect, so an agent reads before it writes Everything about one element in one call: resolved styles, text fields, box, data attributes, GSAP animations, and what the element will and will not accept. The point is to prevent a failed write rather than to satisfy curiosity. `can.reasonIfDisabled` is passed through verbatim from Studio's own capabilities, so an agent that reads first should never attempt an edit the element would refuse. Three things it refuses to get wrong: Animations are reported ONLY for the current selection, because that is the only element Studio parses them for. Attributing them to any other element would be reporting the wrong element's motion, which is worse than reporting none. When a handle names something else the field is empty and `animationEditingBlocked` says why. `animationEditingBlocked` also carries the two states where animation editing is off entirely, multiple timelines and an unsupported timeline pattern. Both live on the selection context. Learning them from a read costs one call; learning them from a failed write costs a retry loop. Inspecting a handle does NOT change what is selected. It is a read, and stealing the human's selection would be a side effect they did not ask for. There is a test asserting `applySelection` is never called. Nothing selected and no handle given is a failure, not an empty result. An empty result would assert "this element has nothing", which is a different and false claim. * feat(studio): let an agent edit text and styles, guarded The first tools that change the composition. Both act on the current selection and take no handle, which is forced rather than chosen: the handlers read the ambient React selection, and `applyDomSelection` only schedules a state update, so selecting and committing inside one call would write to whatever was selected before. Select first, then edit. Also plumbs the write-blocked state, which was the blocker for shipping any write at all. `domEditSaveQueuePaused` and the external-file conflict both lived on App and were unreachable from the tool surface, so `canWrite` was optimistic and a comment said so. They now derive into a single `writeBlockedReason` on the shell context: one field, one owner, conflict taking precedence because resolving it is what unblocks the queue. That guard matters more than it looks. Both states are BANNERS in Studio with no lock behind them, so nothing else was stopping a programmatic write from landing on top of a conflict the user had been asked to adjudicate. Three things the tools refuse to fake: They check the outcome, not the absence of a throw. Studio has several paths where a failed commit resolves anyway, so awaiting the handler proves nothing. The tagged outcome added earlier is what proves the write landed. A partial style result is reported as partial. `handleDomStyleCommit` is one property per call, so N properties are N commits; the result carries `applied` and `rejected` maps rather than a single boolean that would have to pick a side. Style commits run sequentially, never concurrently. Two commits racing through Studio's client-side read-modify-write can record undo entries that both claim the same starting content. There is a test that measures concurrency rather than trusting the loop. Every decline reason maps to a hint naming what to do instead, so a refusal routes the agent rather than just stopping it. * feat(studio): move, resize and rotate, verified by reading back `studio_transform` does what a drag does, and then checks. The box in the result is READ BACK after the write, never echoed from the request, and `applied` lists what actually took effect. That is not belt-and-braces. The plan for this unit said to re-derive the geometry handlers' behaviour rather than trust any description of them, and doing that turned up three different behaviours behind one interface. The handlers on `DomEditActionsValue` are the GSAP-AWARE wrappers, aliased in `useDomEditSession.ts:534-538`, not the CSS ones in `useDomGeometryCommits.ts` that an earlier note in this workstream described. `handleGsapAwarePathOffsetCommit` and `handleGsapAwareRotationCommit` are `if (gsapCommitMutation) { ...intercept... }` with no else branch. Their own comments say the absence is deliberate: position and rotation are written as GSAP code and there is no CSS fallback to write to. So they can return having done nothing. `handleGsapAwareBoxSizeCommit` is not like the other two. It runs through `runGestureTransaction` with separate scale and width/height routes, so resize works more generally. Reading back is what turns that middle case from a silent lie into a reported one. A move that did nothing comes back in `unchanged` with a reason. Three smaller decisions: Operations re-read between each other, so a move is judged against the box AFTER a resize in the same call. Comparing against the original would credit the resize's change to the move. Rotation is reported as dispatched, not verified. `rotate` is an individual transform property and does not appear in the computed transform, so there is no honest box-derived signal, and claiming one would be worse than saying so. x pairs with y and width pairs with height. Accepting one alone would mean inventing the other from the current value, which moves the element somewhere the caller did not ask for. The pairing rule and its minimum live in one `parsePair` helper rather than as four separate branches. * feat(studio): let an agent author motion Four tools: add an animation, change its duration/ease/position, add a keyframe, delete it. This is the capability that makes the tool set worth having, because motion is the one thing an agent cannot judge or author from source. These are deliberately less confident than the rest of the set, and the reason is the handlers underneath them: `handleGsapAddAnimation(method)` takes only a method. Its insert position comes from the live playhead, not the caller, and the call is `void ...catch()` so it returns nothing. `handleGsapAddKeyframeBatch` returns a promise but catches its own failure, so awaiting proves the call finished, not that it landed. `handleGsapDeleteAnimation` discards its promise entirely. `handleGsapUpdateMeta` is the one honest signal. It returns a boolean. U8 handled the same problem by reading the result back. That does not work here: the animation list comes from React state that only refreshes on a render, and no render happens inside one tool call. Rather than fake a verification with a frame-timer, these report what was DISPATCHED and the descriptions tell the agent to call studio_inspect to see the result. Saying "I asked for this" is honest; saying "this happened" would not be. Three consequences worth stating: `studio_add_animation` takes no position. The handler reads the playhead, so accepting one would report a number that had no effect. It reports where the playhead actually was and tells the agent to seek first. `studio_update_animation` rules out the no-selection case BEFORE dispatch. The handler answers `false` for both "nothing selected" and "the write failed", so eliminating one is what makes the other legible. Keyframe percent and properties are validated in the tool, because nothing in the platform checks input against the declared schema. * feat(studio): add studio_inspect, so an agent reads before it writes (#3517) Everything about one element in one call: resolved styles, text fields, box, data attributes, GSAP animations, and what the element will and will not accept. The point is to prevent a failed write rather than to satisfy curiosity. `can.reasonIfDisabled` is passed through verbatim from Studio's own capabilities, so an agent that reads first should never attempt an edit the element would refuse. Three things it refuses to get wrong: Animations are reported ONLY for the current selection, because that is the only element Studio parses them for. Attributing them to any other element would be reporting the wrong element's motion, which is worse than reporting none. When a handle names something else the field is empty and `animationEditingBlocked` says why. `animationEditingBlocked` also carries the two states where animation editing is off entirely, multiple timelines and an unsupported timeline pattern. Both live on the selection context. Learning them from a read costs one call; learning them from a failed write costs a retry loop. Inspecting a handle does NOT change what is selected. It is a read, and stealing the human's selection would be a side effect they did not ask for. There is a test asserting `applySelection` is never called. Nothing selected and no handle given is a failure, not an empty result. An empty result would assert "this element has nothing", which is a different and false claim. * feat(studio): move, resize and rotate, verified by reading back (#3519) `studio_transform` does what a drag does, and then checks. The box in the result is READ BACK after the write, never echoed from the request, and `applied` lists what actually took effect. That is not belt-and-braces. The plan for this unit said to re-derive the geometry handlers' behaviour rather than trust any description of them, and doing that turned up three different behaviours behind one interface. The handlers on `DomEditActionsValue` are the GSAP-AWARE wrappers, aliased in `useDomEditSession.ts:534-538`, not the CSS ones in `useDomGeometryCommits.ts` that an earlier note in this workstream described. `handleGsapAwarePathOffsetCommit` and `handleGsapAwareRotationCommit` are `if (gsapCommitMutation) { ...intercept... }` with no else branch. Their own comments say the absence is deliberate: position and rotation are written as GSAP code and there is no CSS fallback to write to. So they can return having done nothing. `handleGsapAwareBoxSizeCommit` is not like the other two. It runs through `runGestureTransaction` with separate scale and width/height routes, so resize works more generally. Reading back is what turns that middle case from a silent lie into a reported one. A move that did nothing comes back in `unchanged` with a reason. Three smaller decisions: Operations re-read between each other, so a move is judged against the box AFTER a resize in the same call. Comparing against the original would credit the resize's change to the move. Rotation is reported as dispatched, not verified. `rotate` is an individual transform property and does not appear in the computed transform, so there is no honest box-derived signal, and claiming one would be worse than saying so. x pairs with y and width pairs with height. Accepting one alone would mean inventing the other from the current value, which moves the element somewhere the caller did not ask for. The pairing rule and its minimum live in one `parsePair` helper rather than as four separate branches. * docs: document Studio's WebMCP agent tools, proven end-to-end in a browser (#3521) * docs: document Studio's WebMCP agent tools Adds `guides/webmcp`, under Developers > Agent setup. Its first job is to defuse a name collision. `guides/mcp` already exists and covers HeyGen's HOSTED MCP connector, which builds a video from a chat. This page is about an agent working inside Studio on a composition already open in front of you. Different feature, confusingly similar name, so the page says what it is not before it says what it is. Written to DOCS_GUIDELINES: one-sentence intro, outcome before implementation, real values rather than placeholders, and three callouts. The three things a reader most needs are the ones easiest to get wrong: The API is `document.modelContext`, not `navigator.modelContext`. Most published examples use the second, which is a polyfill compatibility shim rather than a spec member, so feature-detecting it misleads. Select first, then edit. Most editing tools act on the current selection, and an agent that skips it gets an error rather than a wrong-element write. Leave Studio visible. Some of Studio's write paths report failure through a toast rather than a return value, so the human is the one who sees it. That is a real property of the co-pilot design, not a nicety, so the page says it plainly. Verified with `npx mint validate` and `npx mint broken-links --check-redirects`, both passing. * fix(studio): target the text field that exists, not one named self Found by running the tools end to end in a browser, which is the only way it could have been found: the unit tests mock `setText`, so they never crossed the boundary where this breaks. An element's text usually lives in a CHILD field, keyed like `self:0:h1` or `child:0:h1`. `studio_set_text` passed no field key, so `buildNextDomTextFields` planned zero operations, the request went out with an empty patch, and the server answered: POST /api/projects/<id>/file-mutations/patch-element -> 400 {"error":"target and operations required"} Which surfaced as `persist-failed`. The tool was telling the truth, so the reporting work in the earlier PRs did its job, but the failure looked like a server problem and was not. The tool now resolves the field: the one the caller named, or the element's single field when it has exactly one. An element with several fields is asked to name one; an element with none is reported blocked. Naming a field the element does not have is rejected with the list of the ones it does have, rather than silently writing nowhere. Four regression tests, including the exact `child:0:h1` shape that failed. One existing assertion changed: it expected the field to be `undefined`, which is precisely the bug, so it now expects the resolved key. Also documents two things the browser run surfaced, both real and neither a defect: registration is asynchronous, so a caller reading `getTools()` too early sees a partial list; and the tools that act on the current selection need a render between the select and the edit, which a real agent gets for free because its calls arrive as separate messages. * docs: give the agent-tools kill switch instructions that work The page told readers to set agentToolsEnabled in Studio's preferences. Nothing writes that flag: it is read in useStudioAgentTools and parsed in studioUiPreferences, but there is no settings UI and no toggle, so the instruction could not be followed. Replace it with the localStorage write that actually flips it, and spell out the merge, since overwriting the key drops every other stored preference. * docs: do not promise a per-call permission prompt we have not verified The page said the browser asks before any agent calls a tool. Prompt granularity is browser-specific and unsettled during the origin trial, and we have not observed it on the native path. Say what holds, that access is gated, and name the part that is still moving. * fix(studio): re-apply WebMCP test polyfill fix (#3532 regression) The squash merge of #3518 re-introduced the old assertion that document.modelContext is absent. The polyfill from #3514 installs it as a fallback — that is expected behavior. Same fix as #3532: remove the assertion, keep the boot-cleanly contract. --------- Co-authored-by: miga-heygen <miguel.sierra_miga@heygen.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(studio): let an agent edit text and styles, guarded (heygen-com#3518) * feat(studio): let an agent drive Studio's selection and playhead Adds `studio_select` and `studio_seek`, so an agent and the human are looking at the same element and the same instant. Selecting reveals the inspector, exactly as a click does, which is what makes the agent's move visible. Selection is shared state, not a per-call argument, and that is forced rather than chosen. Most of Studio's edit handlers read the ambient React selection, and `applyDomSelection` only schedules a state update, so selecting and committing inside ONE call would write to whatever was selected before. Two tool calls are separated by a render, so the contract is select first, then act. That is also how a human works: click, then type. `studio_seek` uses `requestSeek`, not `setCurrentTime`. The latter only moves the timeline's displayed number and leaves the composition where it was. Two things the tools refuse to fake: Seek does not clamp. `seek()` already clamps against the adapter's duration, which can differ from the store's, and clamping again would give that invariant two owners that can disagree. The tool reports where the playhead actually landed instead, read back afterwards. `requestSeek` is fire-and-forget, so it cannot report that no adapter was mounted to receive it. The tool compares the playhead before and after and fails rather than claiming a seek that never happened. Select separates three failures that a single message would have merged: the preview is not mounted yet (wait), no element matches the handle (re-read), and the element cannot be selected (try a neighbour). The agent's next move differs for each, so collapsing them would cost it a round trip or a retry loop. * feat(studio): give an agent eyes with studio_frame Renders the composition to a PNG at a given time and returns the URL. This is what turns the tool set from a remote control into a loop: author a change, capture the instant it affects, look, adjust. No agent can judge motion from source, because "what does this look like at 2.4 seconds" is not a question a file answers. Reuses Studio's existing capture endpoint via `buildFrameCaptureUrl` rather than inventing a second one. Two things this does not fake: It reports the time the playhead LANDED on, not the time requested. The player clamps, so those differ at the ends, and attaching the wrong time to a frame is how an agent draws a confident wrong conclusion about motion. It waits before capturing, by default 150ms. The frame is rendered from the file on disk, and the render cache is cleared by a file watcher with a 40ms write-stability threshold, so a capture that beats the watcher renders the PRE-edit composition. That exact staleness was a real bug here once. An agent reading a stale frame as "my edit failed" would thrash, so the wait is on by default, `settleMs` makes it tunable, and the tool description names the failure rather than leaving it to be rediscovered. It probes with HEAD before returning, so a URL that 404s comes back as a failure with a hint instead of as a link the agent cannot render. * feat(studio): add studio_inspect, so an agent reads before it writes Everything about one element in one call: resolved styles, text fields, box, data attributes, GSAP animations, and what the element will and will not accept. The point is to prevent a failed write rather than to satisfy curiosity. `can.reasonIfDisabled` is passed through verbatim from Studio's own capabilities, so an agent that reads first should never attempt an edit the element would refuse. Three things it refuses to get wrong: Animations are reported ONLY for the current selection, because that is the only element Studio parses them for. Attributing them to any other element would be reporting the wrong element's motion, which is worse than reporting none. When a handle names something else the field is empty and `animationEditingBlocked` says why. `animationEditingBlocked` also carries the two states where animation editing is off entirely, multiple timelines and an unsupported timeline pattern. Both live on the selection context. Learning them from a read costs one call; learning them from a failed write costs a retry loop. Inspecting a handle does NOT change what is selected. It is a read, and stealing the human's selection would be a side effect they did not ask for. There is a test asserting `applySelection` is never called. Nothing selected and no handle given is a failure, not an empty result. An empty result would assert "this element has nothing", which is a different and false claim. * feat(studio): let an agent edit text and styles, guarded The first tools that change the composition. Both act on the current selection and take no handle, which is forced rather than chosen: the handlers read the ambient React selection, and `applyDomSelection` only schedules a state update, so selecting and committing inside one call would write to whatever was selected before. Select first, then edit. Also plumbs the write-blocked state, which was the blocker for shipping any write at all. `domEditSaveQueuePaused` and the external-file conflict both lived on App and were unreachable from the tool surface, so `canWrite` was optimistic and a comment said so. They now derive into a single `writeBlockedReason` on the shell context: one field, one owner, conflict taking precedence because resolving it is what unblocks the queue. That guard matters more than it looks. Both states are BANNERS in Studio with no lock behind them, so nothing else was stopping a programmatic write from landing on top of a conflict the user had been asked to adjudicate. Three things the tools refuse to fake: They check the outcome, not the absence of a throw. Studio has several paths where a failed commit resolves anyway, so awaiting the handler proves nothing. The tagged outcome added earlier is what proves the write landed. A partial style result is reported as partial. `handleDomStyleCommit` is one property per call, so N properties are N commits; the result carries `applied` and `rejected` maps rather than a single boolean that would have to pick a side. Style commits run sequentially, never concurrently. Two commits racing through Studio's client-side read-modify-write can record undo entries that both claim the same starting content. There is a test that measures concurrency rather than trusting the loop. Every decline reason maps to a hint naming what to do instead, so a refusal routes the agent rather than just stopping it. * feat(studio): add studio_inspect, so an agent reads before it writes (heygen-com#3517) Everything about one element in one call: resolved styles, text fields, box, data attributes, GSAP animations, and what the element will and will not accept. The point is to prevent a failed write rather than to satisfy curiosity. `can.reasonIfDisabled` is passed through verbatim from Studio's own capabilities, so an agent that reads first should never attempt an edit the element would refuse. Three things it refuses to get wrong: Animations are reported ONLY for the current selection, because that is the only element Studio parses them for. Attributing them to any other element would be reporting the wrong element's motion, which is worse than reporting none. When a handle names something else the field is empty and `animationEditingBlocked` says why. `animationEditingBlocked` also carries the two states where animation editing is off entirely, multiple timelines and an unsupported timeline pattern. Both live on the selection context. Learning them from a read costs one call; learning them from a failed write costs a retry loop. Inspecting a handle does NOT change what is selected. It is a read, and stealing the human's selection would be a side effect they did not ask for. There is a test asserting `applySelection` is never called. Nothing selected and no handle given is a failure, not an empty result. An empty result would assert "this element has nothing", which is a different and false claim. * feat(studio): move, resize and rotate, verified by reading back (heygen-com#3519) `studio_transform` does what a drag does, and then checks. The box in the result is READ BACK after the write, never echoed from the request, and `applied` lists what actually took effect. That is not belt-and-braces. The plan for this unit said to re-derive the geometry handlers' behaviour rather than trust any description of them, and doing that turned up three different behaviours behind one interface. The handlers on `DomEditActionsValue` are the GSAP-AWARE wrappers, aliased in `useDomEditSession.ts:534-538`, not the CSS ones in `useDomGeometryCommits.ts` that an earlier note in this workstream described. `handleGsapAwarePathOffsetCommit` and `handleGsapAwareRotationCommit` are `if (gsapCommitMutation) { ...intercept... }` with no else branch. Their own comments say the absence is deliberate: position and rotation are written as GSAP code and there is no CSS fallback to write to. So they can return having done nothing. `handleGsapAwareBoxSizeCommit` is not like the other two. It runs through `runGestureTransaction` with separate scale and width/height routes, so resize works more generally. Reading back is what turns that middle case from a silent lie into a reported one. A move that did nothing comes back in `unchanged` with a reason. Three smaller decisions: Operations re-read between each other, so a move is judged against the box AFTER a resize in the same call. Comparing against the original would credit the resize's change to the move. Rotation is reported as dispatched, not verified. `rotate` is an individual transform property and does not appear in the computed transform, so there is no honest box-derived signal, and claiming one would be worse than saying so. x pairs with y and width pairs with height. Accepting one alone would mean inventing the other from the current value, which moves the element somewhere the caller did not ask for. The pairing rule and its minimum live in one `parsePair` helper rather than as four separate branches. --------- Co-authored-by: miga-heygen <miguel.sierra_miga@heygen.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * build: upgrade stack and dependencies --------- Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com> Co-authored-by: miga-heygen <miguel.sierra_miga@heygen.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This was referenced Aug 27, 2026

feat(studio): move, resize and rotate, verified by reading back #3519

Merged

docs: document Studio's WebMCP agent tools, proven end-to-end in a browser #3521

Merged

miguel-heygen marked this pull request as draft August 27, 2026 01:33

miguel-heygen added 4 commits August 27, 2026 00:40

miguel-heygen force-pushed the feat/studio-webmcp-inspect branch from bc098b8 to 1478adf Compare August 27, 2026 04:48

miguel-heygen force-pushed the feat/studio-webmcp-content branch from 953a529 to f766c84 Compare August 27, 2026 04:48

Base automatically changed from feat/studio-webmcp-inspect to feat/studio-webmcp-frame August 30, 2026 05:12

miga-heygen and others added 2 commits August 30, 2026 05:22

Merge main into feat/studio-webmcp-frame

0fffa66

Resolve conflicts from squash-merged #3515 (selection tools). Branch retains frame + inspect tools from the stack. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge main into feat/studio-webmcp-content

3caca52

Resolve conflicts from squash-merged #3515 (selection tools). Branch retains the full WebMCP tool stack from the PR chain. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

miga-heygen marked this pull request as ready for review August 30, 2026 05:25

vanceingalls previously approved these changes Aug 30, 2026

View reviewed changes

somanshreddy reviewed Aug 30, 2026

View reviewed changes

Merge feat/studio-webmcp-frame into feat/studio-webmcp-content

72cb259

Resolve conflicts from parallel squash merges in the stack. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Base automatically changed from feat/studio-webmcp-frame to main August 30, 2026 17:00

Merge remote-tracking branch 'origin/main' into feat/studio-webmcp-co…

8078783

…ntent # Conflicts: # packages/studio/src/webmcp/StudioAgentTools.tsx # packages/studio/src/webmcp/useStudioAgentTools.test.tsx # packages/studio/src/webmcp/useStudioAgentTools.ts

miga-heygen enabled auto-merge (squash) August 31, 2026 03:08

vanceingalls approved these changes Aug 31, 2026

View reviewed changes

somanshreddy reviewed Aug 31, 2026

View reviewed changes

miga-heygen merged commit f84b4c2 into main Aug 31, 2026
51 of 53 checks passed

miga-heygen deleted the feat/studio-webmcp-content branch August 31, 2026 07:47

terencecho mentioned this pull request Aug 31, 2026

feat(studio): let an agent author motion #3520

Merged

3 tasks

jrusso1020 mentioned this pull request Aug 31, 2026

docs(changelog): weekly digest 2026-08-24–2026-08-31 #3563

Merged

Navigation Menu

Uh oh!

feat(studio): let an agent edit text and styles, guarded - #3518

feat(studio): let an agent edit text and styles, guarded#3518
miga-heygen merged 10 commits into
mainfrom
feat/studio-webmcp-content

miguel-heygen commented Aug 27, 2026

Uh oh!

vanceingalls left a comment

Uh oh!

somanshreddy left a comment

Uh oh!

vanceingalls left a comment

Uh oh!

somanshreddy left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

miguel-heygen commented Aug 27, 2026

What

Why

How

Test plan

Note

Uh oh!

vanceingalls left a comment

Choose a reason for hiding this comment

Uh oh!

somanshreddy left a comment

Choose a reason for hiding this comment

Core write mechanism — sound, verified at source

State / hygiene — worth resolving before the final pass (my main finding)

Uh oh!

vanceingalls left a comment

Choose a reason for hiding this comment

Uh oh!

somanshreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants