Skip to content

fix(studio): stabilize previews and normalize clip audio - #3280

Closed
miguel-heygen wants to merge 1 commit into
mainfrom
magi/fix-studio-audio-timeout
Closed

fix(studio): stabilize previews and normalize clip audio#3280
miguel-heygen wants to merge 1 commit into
mainfrom
magi/fix-studio-audio-timeout

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

What

Fixes the two real issue families reproduced from the reported screenshots and exact saved project:

  1. Studio could not author gain above unity, so a genuinely quiet clip could not be raised to match a reference.
  2. Agent-launched previews could lose their foreground listener when the launcher session ended, leaving an open Studio tab pointed at a dead localhost server.

It also adds hyperframes normalize-audio so agents can measure two local authored clips with integrated LUFS and safely write the target clip's matched data-volume.

Reproduction findings

  • The original source mismatch was real: the first clip measured -13.8 LUFS and the second -33.7 LUFS.
  • The later saved derivatives did not retain that direction. Before this correction, the exact saved project rendered the first clip at -15.5 LUFS and the second at -11.8 LUFS; both were authored at data-volume="1", so equal fader values still did not mean equal perceived loudness.
  • Browser refresh did not kill Studio. Repeated refresh exposed a listener that the agent-owned foreground process had already lost.

How

  • Added shared audio gain/dB conversion helpers and one +12 dB ceiling across Studio, Web Audio, runtime media, the engine mixer, and FFmpeg render.
  • Converted both Studio media property panels to a nonlinear -infinity / -60 dB ... 0 dB ... +12 dB fader with unity at the center.
  • Kept native HTMLMediaElement.volume clamped while applying above-unity gain through Web Audio.
  • Added a detached preview lifecycle with readiness probing, per-project state/logs, custom-port discovery and reuse, stale-record cleanup, process-birth validation, and ownership-checked stop.
  • Made bare preview select managed mode for non-TTY/agent sessions while retaining foreground behavior for interactive terminals.
  • Added one-line JSON contracts for start/status/list/stop/kill-all and exact URL-encoded Studio deep links.
  • Added normalize-audio --reference <id> --target <id> [--write], which:
    • measures the exact local clip bytes and authored media ranges with FFmpeg EBU R128;
    • accounts for both clips' existing authored gains;
    • changes only the selected target's data-volume;
    • is dry-run by default;
    • rejects remote/out-of-project sources, malformed values, clipping, muted references, and boosts above Studio's ceiling.
  • Updated CLI and installed-skill guidance for preview lifecycle and measured loudness matching.

Exact-project correction

The private saved project was updated in place with the first clip as the reference. The second clip's data-volume changed from 1 to 0.645654 (-3.8 dB). A fresh full render measured both authored segments at -15.5 LUFS. Re-downloading the authoritative saved project and rerunning normalize-audio reports it matched within the default 0.5 LU tolerance.

Verification

  • Full CLI suite: 187 files passed, 1 skipped; 2,694 tests passed, 3 skipped.
  • Skills suite: 466/466 passed; manifest and skill mirrors are in sync.
  • CLI typecheck, Oxlint, Oxfmt, package build, tracked-artifact checks, and Fallow gate passed.
  • New normalizer unit suite: 19/19 passed, including parsing, path confinement, malformed authoring, clipping, gain ceiling, and source-preserving writes.
  • Exact corrected saved project: lint/check passed (one existing Google Fonts advisory only), full 26.922-second render completed, and both segments measured -15.5 LUFS.
  • Exact corrected saved project loaded on its real Studio route and survived 8/8 reloads with no browser errors.
  • Packed CLI lifecycle acceptance covered managed start/reuse/status/list/stop, foreground TTY behavior, JSON purity, custom ports, ownership adversaries, process-tree cleanup, and no remaining listener/state.

Compatibility is preserved: existing data-volume="1" clips remain at unity, interactive human preview remains foreground by default, and normalization never guesses which clip should define the mix.

@mintlify

mintlify Bot commented Aug 15, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
hyperframes 🟢 Ready View Preview Aug 15, 2026, 2:18 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

Persist agent preview processes safely and support above-unity gain in preview and render.

Add a measured LUFS normalizer for matching local authored clips.
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Closing this — every piece of it has landed or moved somewhere better, and nothing is dropped.

I rebased it onto current main first rather than close it blind. That collapsed it from +3050/-331 to +944/-26, which is the measurement this decision rests on: two thirds was already merged, and almost all of the remainder duplicates PRs that are still open.

Where each piece went:

Piece Where it is now
Preview ownership records, signal-only-what-the-OS-owns merged — #3307, #3308
One JSON document per preview lifecycle op merged — #3309
Audio gain ceiling, the probe shim, the mixer and transport carrying it merged — #3333
Studio delete / marquee / preview-reload fixes merged — #3339
Volume fader precision open — #3305
normalize-audio open — #3306, which is ~115 lines further along than the draft here
Background preview modes, studioProxyEnv, vite.preview-config, templates open — #3310, byte-identical to what was left here
el.volume clamp in the sandbox bridge + the two gain tests new — #3349

That last row is the only thing that was genuinely unlanded, and it turned out to matter: raising the ceiling to 12 dB made data-volume legal up to ~3.98, but the sandbox bridge assigns the product straight to el.volume, which throws IndexSizeError above 1 — inside a loop, so one boosted clip stops the volume control working for every clip after it. Verified in Chrome and reproduced in the suite. It was written here when the ceiling was, and got stranded while the rest shipped in pieces. #3349 carries it with a test that reds without the fix.

Merging this as-is would have conflicted with #3306 and #3310 and re-landed an older normalize-audio, so closing is the honest end state rather than a write-off. The rebased branch is pushed if anyone wants to read what was left.

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Superseded — see the breakdown above.

miguel-heygen added a commit that referenced this pull request Aug 20, 2026
…x bridge (#3349)

Authoring a clip above unity gain throws at runtime today.

## What breaks

`MAX_AUDIO_GAIN_DB = 12` makes `data-volume` legal up to ~3.98. The sandbox runtime's volume bridge assigns the product straight to the element:

```ts
el.volume = clipVolume * volume;   // init.ts, onSetVolume
```

`HTMLMediaElement.volume` is spec-pinned to [0,1] and **throws `IndexSizeError`** outside it — verified in Chrome, and the test DOM agrees:

```
el.volume = 2  →  IndexSizeError: Failed to set the 'volume' property...
```

The throw lands inside a `for` loop over every media element, so it takes the rest of the loop with it: every clip after the boosted one keeps whatever volume it already had, while `state.bridgeVolume` says the change was applied. A composition with one boosted clip stops responding to the volume control for every clip authored after it.

## The fix

Clamp what the element receives. That is not lossy, because the element was never where the boost lived — the transport gets the authored gain unclamped, and this PR pins that half too:

- `syncRuntimeMedia` hands `onElementVolume` both the element's clamped volume **and** the authored gain, so the transport can have the boost the element cannot hold.
- `setElementVolume` keeps that gain on the per-element node, clamped only to `MAX_AUDIO_GAIN`.

Those two paths already worked; they were untested, and they are the reason clamping the element is the right half to clamp.

## Tests

- `init.test.ts` — a boosted clip followed by a quieter one, both seeded with sentinels, then the real `set-volume` control message. Asserts the boosted element lands at 1 **and** that the clip after it still gets its own volume, which is what a throw mid-loop strands.
- `media.test.ts` — the transport receives the authored gain while the element stays legal.
- `webAudioTransport.test.ts` — the per-element gain node keeps a boost above unity.

All three mutation-checked: removing the clamp reds the first, and clamping the gain at either transport seam reds the others.

## Provenance

This is the last unlanded piece of #3280. That PR was rebased onto current `main` and collapsed from +3050 to +944, of which everything except these lines is either already merged (#3308, #3309, #3333, #3339) or duplicated by the open #3306 and #3310. Cutting it out separately because the throw is live on `main` now and shouldn't wait behind a PR that is otherwise redundant.
felipecaldas pushed a commit to felipecaldas/hyperframes that referenced this pull request Aug 20, 2026
…x bridge (heygen-com#3349)

Authoring a clip above unity gain throws at runtime today.

## What breaks

`MAX_AUDIO_GAIN_DB = 12` makes `data-volume` legal up to ~3.98. The sandbox runtime's volume bridge assigns the product straight to the element:

```ts
el.volume = clipVolume * volume;   // init.ts, onSetVolume
```

`HTMLMediaElement.volume` is spec-pinned to [0,1] and **throws `IndexSizeError`** outside it — verified in Chrome, and the test DOM agrees:

```
el.volume = 2  →  IndexSizeError: Failed to set the 'volume' property...
```

The throw lands inside a `for` loop over every media element, so it takes the rest of the loop with it: every clip after the boosted one keeps whatever volume it already had, while `state.bridgeVolume` says the change was applied. A composition with one boosted clip stops responding to the volume control for every clip authored after it.

## The fix

Clamp what the element receives. That is not lossy, because the element was never where the boost lived — the transport gets the authored gain unclamped, and this PR pins that half too:

- `syncRuntimeMedia` hands `onElementVolume` both the element's clamped volume **and** the authored gain, so the transport can have the boost the element cannot hold.
- `setElementVolume` keeps that gain on the per-element node, clamped only to `MAX_AUDIO_GAIN`.

Those two paths already worked; they were untested, and they are the reason clamping the element is the right half to clamp.

## Tests

- `init.test.ts` — a boosted clip followed by a quieter one, both seeded with sentinels, then the real `set-volume` control message. Asserts the boosted element lands at 1 **and** that the clip after it still gets its own volume, which is what a throw mid-loop strands.
- `media.test.ts` — the transport receives the authored gain while the element stays legal.
- `webAudioTransport.test.ts` — the per-element gain node keeps a boost above unity.

All three mutation-checked: removing the clamp reds the first, and clamping the gain at either transport seam reds the others.

## Provenance

This is the last unlanded piece of heygen-com#3280. That PR was rebased onto current `main` and collapsed from +3050 to +944, of which everything except these lines is either already merged (heygen-com#3308, heygen-com#3309, heygen-com#3333, heygen-com#3339) or duplicated by the open heygen-com#3306 and heygen-com#3310. Cutting it out separately because the throw is live on `main` now and shouldn't wait behind a PR that is otherwise redundant.

(cherry picked from commit 9140c0e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant