Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sample playback support for Web exports #91382

Merged
merged 1 commit into from
Jun 19, 2024

Conversation

adamscott
Copy link
Member

@adamscott adamscott commented Apr 30, 2024

⚠️ Here be dragons ⚠️

When this PR will be merged for 4.3, this feature will be marked as experimental and could change in 4.4.

Based on these PRs

tl;dr

This PR adds (back) the concept of samples to the Godot Engine. Currently, the PR enables only the web platform to play samples.

It principally fixes #87329, as that issue would plague any non-threaded web releases with crackling audio.

Example

Single-threaded web example using streams
(old way)
Single-threaded web example using samples
(this PR)
https://adamscott.github.io/2d-platformer-demo-main-thread/ https://adamscott.github.io/2d-platformer-demo-main-thread-samples/

Introduction

Godot uses streaming to mix game audio. Each active stream is registered and then the engine mix on-the-fly the needed audio frames together to output audio based on the audio latency parameter. It works very well on modern platforms.

Samples are another way to handle sound instead of mixing streams. Instead of handling mixing sound and music by the game processes, it relies on off-loading it to the host system. While it doesn't permit full access to the mixing apparatus, it's super useful on systems that don't have a lot of processing power.

To use samples, you register a sample, and then tell the system to play it when needed. And to stop it. It's like a music player, you set the file, then you click on play. You don't control how the software do it, but you know it does.

Godot used to have samples back in Godot 1 and 2, especially to support platforms like the PSP, and the web (thanks to the Web Audio API).

As newer console platforms let developers handle their own mixing logic and that SharedArrayBuffers were introduced in browsers (permitting WebWorkers (web threads) to share memory with the game), samples support was dropped from Godot. Everything was fine.

Anyway, the implementation was somewhat lacking. You had to specifically want to play samples, you couldn't use common nodes to play both streams and samples.

The problem

But on the web platform, Spectre and Meltdown happened. And it completely changed where SharedArrayBuffers were able to be used. Enter "cross-origin isolated" websites, where it's impossible to contact other websites or display ads, and complicating hosting of simple games, greatly reducing the appeal for our web builds.

Hence the work on #85939 in order to compile Godot to run on the main thread. This enables exporting Godot games on the browser without having to cross-origin isolate your website. Unfortunately, this brought an unexpected issue: software mixing is pretty much incompatible with single-threaded games. Especially running on older/less powerful hardware.

Wanna hear for yourself? Try the single-threaded platformer demo (without this PR applied) on your phone or on a computer that doesn't have a great CPU.

The investigation

My colleague @Faless and I considered every solution imaginable: augmenting latency for the web, traced the processes on the web and on mobile and refactor the AudioWorklet processing the audio. But alas. Nothing substantial could have been done.

The only solution we found was to resort to Web Audio samples.

And it's not uncommon for web game engines. We were the uncommon ones not using web audio samples. So, a few weeks ago, I began work on this PR.

The sole requirement: seamlessness

My main focus was to reuse as possible as many features that already exist. It means that in order to play samples, I wanted the UX to keep as close as possible to existing tools.

Godot strives itself to offer the same experience for every target that it exports to. Imagine making the developer choose between having samples for the web export and streaming nodes for the rest. And having to manually add or remove nodes based on the platform with scripts.

This has such a big impact that it's a clear no go for us. We don't want that poor UX.

The solution

My solution is a big hack. (But it does works wonderfully.)

The idea is to reuse all the existing stream nodes and systems. And make the stream elements capable of producing samples.

This story begins with the new project setting audio/general/default_playback_type (hidden currently in the advanced options). Usually, it should stay with the value "Stream", as normally, that's how Godot works currently. But the magic happens with audio/general/default_playback_type.web set as "Sample".

That's because AudioStreamPlayer, AudioStreamPlayer2D and AudioStreamPlayer3D now have a new property called playback_type, which is set by default to... "Default". That's where the magic happens! On standard exports, the nodes will be defined as "Stream", but on web exports, "Sample" will be used instead!

The magic operates behind the scenes though.

The man behind the curtain

Essentially, when a stream is considered a "sample", it doesn't get mixed at all in the mixing phase. Instead, it relies on callbacks by the StreamPlayer nodes.

The StreamPlayer nodes, when their play() method is called, are calling internally AudioServer::start_sample_playback(). All the AudioServer does is to call AudioDriver::start_sample_playback(). If the driver doesn't implement that function, it just doesn't play any sound. But if it does, the driver can now tell the backend to play that sound.

The same thing happens for stop, pause, etc. You can even update the sample, like when the position of the node changes!

Isn't this fascinating?

Registering samples

Before playing the samples, it's important to register them first.

If played without previous registration, the player will make sure to register it first. Though, it's recommended to register manually streams. That's because, on single threaded games, memory transfer is synchronous, so it may make your game stutter. You register a stream as a sample by calling this method:

# optional step
const my_stream_resource = preload("res://assets/my_stream.wav")

AudioServer.register_stream_as_sample(my_stream_resource)

Under the hood, Godot will call the mix() method of the stream playback for the entire duration of the clip. This makes it so that it's possible to play any type of sound media that Godot supports (WAV, mp3, ogg vorbis).

Is it really seamless, though?

These demos were exported to the web (single-threaded) using samples without ever touching the project nodes, resources, nor files.

Demo title Playable link Source
2D Platformer Play Link
Dodge the Creeps Play Link
3D Platformer Play Link
Truck Town Play Link
Hell of Mirrors Play Link
Catburglar Play Link

Bugs yet to fix before merge

  • Buses don't chain properly
  • Autoplay doesn't work right now.
  • Fix issues with sample rate.
  • AnimationPlayer cannot play samples.
  • Advanced audio importers fail to show / infinite loop (the problem came and go without any of my input)
  • Only forward loop is supported (fix may not make it to the final release) looping is kinda a little broken right now

Known limitations

  • Effects don't apply (will certainly not be part of the initial release and GDScript based effects cannot be used)

Technical diagrams

Registering and playing samples

sequenceDiagram
    participant Script
    participant AudioStreamPlayer2D
    participant AudioStreamPlayerInternal
    participant AudioServer
    participant AudioDriverWeb
    participant JavaScriptAudioLibrary

    Script ->> AudioStreamPlayer2D: set_stream(Ref<AudioStream>)
    AudioStreamPlayer2D ->> AudioStreamPlayerInternal: set_stream(Ref<AudioStream>)

    Script ->> AudioServer: register_stream_as_sample(Ref<AudioStream>)
    activate AudioServer
    AudioServer ->> AudioServer: register_sample(Ref<AudioSample>)
    AudioServer ->> AudioDriverWeb: register_sample(Ref<AudioSample>)
    deactivate AudioServer
    AudioDriverWeb ->> JavaScriptAudioLibrary: godot_audio_sample_register_stream()

    Script ->> AudioStreamPlayer2D: play()
    activate AudioStreamPlayer2D
    AudioStreamPlayer2D ->> AudioStreamPlayerInternal: play_basic()
    activate AudioStreamPlayerInternal
    opt _is_sample() && stream->can_be_sampled() && stream_playback->sample_playback is null
        AudioStreamPlayerInternal ->> AudioServer: is_stream_registered_as_sample(Ref<AudioStream>)
        activate AudioServer
        deactivate AudioServer
        AudioServer ->> AudioStreamPlayerInternal: 
        opt stream is not registered
            AudioStreamPlayerInternal ->> AudioServer: register_stream_as_sample(Ref<AudioStream>)
            activate AudioServer
            AudioServer ->> AudioServer: register_sample(Ref<AudioSample>)
            AudioServer ->> AudioDriverWeb: register_sample(Ref<AudioSample>)
            deactivate AudioServer
            AudioDriverWeb ->> JavaScriptAudioLibrary: godot_audio_sample_register_stream()
        end
        AudioStreamPlayerInternal ->> AudioStreamPlayerInternal: set stream_playback->sample_playback
    end
    AudioStreamPlayerInternal ->> AudioStreamPlayer2D: Ref<AudioStreamPlayback>
    
    deactivate AudioStreamPlayerInternal

    opt stream playback sample exists
        AudioStreamPlayer2D ->> AudioStreamPlayer2D: Update stream_playback->sample_playback for specific AudioStreamPlayer2D stuff
        AudioStreamPlayer2D ->> AudioServer: start_sample_playback(Ref<AudioSamplePlayback>)
        deactivate AudioStreamPlayer2D

        AudioServer ->> AudioDriverWeb: start_sample_playback(Ref<AudioSamplePlayback>)
        AudioDriverWeb ->> JavaScriptAudioLibrary: godot_audio_sample_start()
    end
Loading

Samples and streams

classDiagram
    namespace AudioStreams {
        class AudioStream
        class AudioStreamWAV
        class AudioStreamPlayback
        class AudioStreamPlaybackWAV
    }
    namespace AudioSamples {
        class AudioSample
        class AudioSamplePlayback
    }

    %% Inheritance
    AudioStream <|-- AudioStreamWAV
    AudioStreamPlayback <|-- AudioStreamPlaybackWAV

    %% Links
    AudioStream ..|> AudioSample
    AudioStreamWAV ..|> AudioSample

    AudioStreamPlayback ..|> AudioSamplePlayback

    %% Cardinality
    AudioStreamPlaybackWAV "1" --> "many" AudioSamplePlayback

    AudioSample "1" --> "many" AudioStream
    AudioSamplePlayback "1" --> "many" AudioStream

    %% Classes
    class AudioStream {
        +can_be_sampled()*
        +get_sample()
    }

    class AudioStreamWAV {
        +can_be_sampled()
        +get_sample()
    }

    class AudioStreamPlayback {
        +set_is_sample()*
        +get_is_sample()*
        +set_sample_playback()*
        +get_sample_playback()*
    }

    class AudioStreamPlaybackWAV {
        -bool _is_sample
	    -Ref~AudioSamplePlayback~ sample_playback

        +set_is_sample()
        +get_is_sample()
        +set_sample_playback()
        +get_sample_playback()
    }

    class AudioSample {
        +Ref~AudioStream~stream
        +data
        +num_channels
        +sample_rate
        +loop_mode
        +loop_begin
        +loop_end
    }

    class AudioSamplePlayback {
        +Ref~AudioStream~stream
        +float offset
        +float volume_db
        +PositionMode position_mode
        +Vector3 position
        +StringName bus
    }
Loading

Fixes

Fixes #87329

modules/minimp3/audio_stream_mp3.h Outdated Show resolved Hide resolved
modules/vorbis/audio_stream_ogg_vorbis.h Outdated Show resolved Hide resolved
scene/audio/audio_stream_player_internal.h Outdated Show resolved Hide resolved
scene/resources/audio_stream_wav.h Outdated Show resolved Hide resolved
servers/audio/audio_stream.h Outdated Show resolved Hide resolved
servers/audio_server.cpp Show resolved Hide resolved
servers/audio_server.cpp Show resolved Hide resolved
servers/audio_server.cpp Show resolved Hide resolved
servers/audio/audio_stream.h Outdated Show resolved Hide resolved
servers/audio_server.h Outdated Show resolved Hide resolved
@adamscott adamscott marked this pull request as ready for review May 1, 2024 13:08
@adamscott adamscott requested review from a team as code owners May 1, 2024 13:08
@adamscott adamscott force-pushed the sample-player branch 8 times, most recently from 6115ee4 to 4ff18d6 Compare May 1, 2024 20:20
@adamscott adamscott marked this pull request as draft May 1, 2024 22:15
@akien-mga

This comment was marked as resolved.

@adamscott adamscott force-pushed the sample-player branch 2 times, most recently from 2371978 to 3de8154 Compare May 2, 2024 14:49
@AThousandShips
Copy link
Member

AThousandShips commented Jun 13, 2024

Ouch it seems that the bindings won't work correctly generating string name arguments because of syntax, I don't think we can solve that in this so might be better to make the default empty and have it default to using Master if the name is empty, or similar

We already have cases with &"" for parameters but no other cases, just for default members, so this would need some rework of the bindings that might take some time

I'll see if I can write a fix on the godot-cpp side

@AThousandShips
Copy link
Member

AThousandShips commented Jun 13, 2024

Got a fix that should work to help with that one, pushing to godot-cpp right now and we can see if we can't get it working

Done:

If approved it'd have to be cherry picked for 4.2 though for CI to finish here

@dsnopek
Copy link
Contributor

dsnopek commented Jun 14, 2024

PR godotengine/godot-cpp#1487 was cherry-picked, so tests here should pass. I just kicked off another test run, so we'll see!

Copy link
Member

@akien-mga akien-mga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me overall, let's go!

platform/web/audio_driver_web.cpp Show resolved Hide resolved
platform/web/audio_driver_web.cpp Outdated Show resolved Hide resolved
scene/resources/audio_stream_polyphonic.cpp Outdated Show resolved Hide resolved
doc/classes/AudioStreamPlayback.xml Outdated Show resolved Hide resolved
@adamscott adamscott force-pushed the sample-player branch 3 times, most recently from 9554d2e to 9102c13 Compare June 17, 2024 15:03
@Calinou
Copy link
Member

Calinou commented Jun 17, 2024

https://adamscott.github.io/2d-platformer-demo-main-thread-samples/ now works great on both Firefox and Chromium, although I can still see the error message in the console every time I pick up a coin. The coin pickup sound still plays correctly though.

doc/classes/ProjectSettings.xml Outdated Show resolved Hide resolved
doc/classes/AudioServer.xml Outdated Show resolved Hide resolved
doc/classes/AudioServer.xml Outdated Show resolved Hide resolved
@Maran23
Copy link
Contributor

Maran23 commented Jun 18, 2024

Sorry for the delay. Tested right now with then newest code, I got another error now when unpausing the game (and therefore all audio as well):
image
image

@akien-mga akien-mga merged commit 19bf77f into godotengine:master Jun 19, 2024
16 checks passed
@akien-mga
Copy link
Member

Awesome work @adamscott!

Let's merge now to get wider testing, and fix the remaining issues in follow up PRs.

@adamscott
Copy link
Member Author

Sorry for the delay. Tested right now with then newest code, I got another error now when unpausing the game (and therefore all audio as well): image image

I'll create an issue in the tracker. I'll try to reproduce this on my side ASAP.

@JunShiozawa
Copy link

@adamscott
I will try to backport this PR to Godot 3.x.

I need WebGL 1.0, so

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Release Blocker
Development

Successfully merging this pull request may close these issues.

Cracking audio with Godot 4 no-threads Web builds