fix(studio): stabilize previews and normalize clip audio - #3280
fix(studio): stabilize previews and normalize clip audio#3280miguel-heygen wants to merge 1 commit into
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
7368231 to
bf41590
Compare
bf41590 to
e3fd173
Compare
e3fd173 to
6cb48c0
Compare
6cb48c0 to
33efa91
Compare
Persist agent preview processes safely and support above-unity gain in preview and render. Add a measured LUFS normalizer for matching local authored clips.
33efa91 to
2869bab
Compare
|
Closing this — every piece of it has landed or moved somewhere better, and nothing is dropped. I rebased it onto current Where each piece went:
That last row is the only thing that was genuinely unlanded, and it turned out to matter: raising the ceiling to 12 dB made Merging this as-is would have conflicted with #3306 and #3310 and re-landed an older |
|
Superseded — see the breakdown above. |
…x bridge (#3349) Authoring a clip above unity gain throws at runtime today. ## What breaks `MAX_AUDIO_GAIN_DB = 12` makes `data-volume` legal up to ~3.98. The sandbox runtime's volume bridge assigns the product straight to the element: ```ts el.volume = clipVolume * volume; // init.ts, onSetVolume ``` `HTMLMediaElement.volume` is spec-pinned to [0,1] and **throws `IndexSizeError`** outside it — verified in Chrome, and the test DOM agrees: ``` el.volume = 2 → IndexSizeError: Failed to set the 'volume' property... ``` The throw lands inside a `for` loop over every media element, so it takes the rest of the loop with it: every clip after the boosted one keeps whatever volume it already had, while `state.bridgeVolume` says the change was applied. A composition with one boosted clip stops responding to the volume control for every clip authored after it. ## The fix Clamp what the element receives. That is not lossy, because the element was never where the boost lived — the transport gets the authored gain unclamped, and this PR pins that half too: - `syncRuntimeMedia` hands `onElementVolume` both the element's clamped volume **and** the authored gain, so the transport can have the boost the element cannot hold. - `setElementVolume` keeps that gain on the per-element node, clamped only to `MAX_AUDIO_GAIN`. Those two paths already worked; they were untested, and they are the reason clamping the element is the right half to clamp. ## Tests - `init.test.ts` — a boosted clip followed by a quieter one, both seeded with sentinels, then the real `set-volume` control message. Asserts the boosted element lands at 1 **and** that the clip after it still gets its own volume, which is what a throw mid-loop strands. - `media.test.ts` — the transport receives the authored gain while the element stays legal. - `webAudioTransport.test.ts` — the per-element gain node keeps a boost above unity. All three mutation-checked: removing the clamp reds the first, and clamping the gain at either transport seam reds the others. ## Provenance This is the last unlanded piece of #3280. That PR was rebased onto current `main` and collapsed from +3050 to +944, of which everything except these lines is either already merged (#3308, #3309, #3333, #3339) or duplicated by the open #3306 and #3310. Cutting it out separately because the throw is live on `main` now and shouldn't wait behind a PR that is otherwise redundant.
…x bridge (heygen-com#3349) Authoring a clip above unity gain throws at runtime today. ## What breaks `MAX_AUDIO_GAIN_DB = 12` makes `data-volume` legal up to ~3.98. The sandbox runtime's volume bridge assigns the product straight to the element: ```ts el.volume = clipVolume * volume; // init.ts, onSetVolume ``` `HTMLMediaElement.volume` is spec-pinned to [0,1] and **throws `IndexSizeError`** outside it — verified in Chrome, and the test DOM agrees: ``` el.volume = 2 → IndexSizeError: Failed to set the 'volume' property... ``` The throw lands inside a `for` loop over every media element, so it takes the rest of the loop with it: every clip after the boosted one keeps whatever volume it already had, while `state.bridgeVolume` says the change was applied. A composition with one boosted clip stops responding to the volume control for every clip authored after it. ## The fix Clamp what the element receives. That is not lossy, because the element was never where the boost lived — the transport gets the authored gain unclamped, and this PR pins that half too: - `syncRuntimeMedia` hands `onElementVolume` both the element's clamped volume **and** the authored gain, so the transport can have the boost the element cannot hold. - `setElementVolume` keeps that gain on the per-element node, clamped only to `MAX_AUDIO_GAIN`. Those two paths already worked; they were untested, and they are the reason clamping the element is the right half to clamp. ## Tests - `init.test.ts` — a boosted clip followed by a quieter one, both seeded with sentinels, then the real `set-volume` control message. Asserts the boosted element lands at 1 **and** that the clip after it still gets its own volume, which is what a throw mid-loop strands. - `media.test.ts` — the transport receives the authored gain while the element stays legal. - `webAudioTransport.test.ts` — the per-element gain node keeps a boost above unity. All three mutation-checked: removing the clamp reds the first, and clamping the gain at either transport seam reds the others. ## Provenance This is the last unlanded piece of heygen-com#3280. That PR was rebased onto current `main` and collapsed from +3050 to +944, of which everything except these lines is either already merged (heygen-com#3308, heygen-com#3309, heygen-com#3333, heygen-com#3339) or duplicated by the open heygen-com#3306 and heygen-com#3310. Cutting it out separately because the throw is live on `main` now and shouldn't wait behind a PR that is otherwise redundant. (cherry picked from commit 9140c0e)
What
Fixes the two real issue families reproduced from the reported screenshots and exact saved project:
It also adds
hyperframes normalize-audioso agents can measure two local authored clips with integrated LUFS and safely write the target clip's matcheddata-volume.Reproduction findings
-13.8 LUFSand the second-33.7 LUFS.-15.5 LUFSand the second at-11.8 LUFS; both were authored atdata-volume="1", so equal fader values still did not mean equal perceived loudness.How
+12 dBceiling across Studio, Web Audio, runtime media, the engine mixer, and FFmpeg render.-infinity / -60 dB ... 0 dB ... +12 dBfader with unity at the center.HTMLMediaElement.volumeclamped while applying above-unity gain through Web Audio.normalize-audio --reference <id> --target <id> [--write], which:data-volume;Exact-project correction
The private saved project was updated in place with the first clip as the reference. The second clip's
data-volumechanged from1to0.645654(-3.8 dB). A fresh full render measured both authored segments at-15.5 LUFS. Re-downloading the authoritative saved project and rerunningnormalize-audioreports it matched within the default0.5 LUtolerance.Verification
-15.5 LUFS.Compatibility is preserved: existing
data-volume="1"clips remain at unity, interactive human preview remains foreground by default, and normalization never guesses which clip should define the mix.