Audio: Write to device's buffer directly without temporary buffers. #90013

kus04e4ek · 2024-03-29T10:14:19Z

Closes Add support for more output types in AudioDriver godot-proposals#8797
Closes Bring the compatibility methods from AudioDriverWASAPI into AudioDriver godot-proposals#8815
Fixes Godot does not listen to microphones with over 2 channels #82823 (will take the first 2 channels) (didn't test, but the code should account for that)

Also, a little bit of a clean-up.

TODO:

Implement for the input buffer;
Test on MacOS/iOS. (I don't have them, so would appreciate some help)

kus04e4ek · 2024-03-29T11:01:43Z

drivers/xaudio2/audio_driver_xaudio2.h

@@ -36,7 +36,6 @@
 #include "core/templates/safe_refcount.h"
 #include "servers/audio_server.h"

-#include <mmsystem.h>


Maybe it's needed for MSVC?

kus04e4ek · 2024-03-29T11:02:01Z

platform/android/audio_driver_opensl.cpp

@@ -30,11 +30,6 @@

 #include "audio_driver_opensl.h"

-#include <string.h>


Probably not needed? Android builds always use clang

kus04e4ek · 2024-03-29T11:03:07Z

platform/web/audio_driver_web.h

@@ -54,7 +54,8 @@ class AudioDriverWeb : public AudioDriver {

 	int buffer_length = 0;
 	int mix_rate = 0;
-	int channel_count = 0;
+	int channels = 0;


Changed the name to match other audio drivers.

kus04e4ek · 2024-03-29T11:05:52Z

servers/audio_server.h

-	virtual SpeakerMode get_speaker_mode() const = 0;
+	virtual int get_total_channels() const = 0;


It could be better to implement new SpeakerMode values instead of get_total_channels?

kus04e4ek · 2024-03-29T11:11:43Z

servers/audio_server.cpp

+
+					if (k == cs - 1 && stride == (uintptr_t)cs * 2 - 1) {
+						// Downmix.
+						_write_sample(p_buffer, idx, (int32_t)(((int64_t)vl2 + (int64_t)vr2) / 2));


Maybe it would be better to pass _write_sample as an argument or make it a virtual method in AudioDriver? It looks like all existing drivers don't need it though

drivers/pulseaudio/audio_driver_pulseaudio.cpp

drivers/wasapi/audio_driver_wasapi.cpp

drivers/coreaudio/audio_driver_coreaudio.cpp

fire · 2024-03-29T14:05:25Z

How would you suggest testing this?

kus04e4ek · 2024-03-29T16:06:01Z

How would you suggest testing this?

Just using AudioStreamPlayer and switching driver in the settings.
On Linux, ALSA and PulseAudio always use 16-bit integers.
On Android, OpenSL always uses 16-bit integers.
On MacOS and iOS, CoreAudio always uses ~~floats~~ 16-bit integers.
On Web, Worklet always uses floats.
On Windows, XAudio2 always uses 16-bit integers, WASAPI uses the most fitting format, format can be obtained only through debugging, there might be different formats for different output devices.
MovieWriter always uses 32-bit integers. Also setting channels can be useful for testing too.
AudioDriverDummy uses no buffer, but still processes audio.

I checked downsampling by modifying PulseAudio code like this:

Patch

diff --git a/drivers/pulseaudio/audio_driver_pulseaudio.cpp b/drivers/pulseaudio/audio_driver_pulseaudio.cpp
index 2e7a3155da1..166ee9472a8 100644
--- a/drivers/pulseaudio/audio_driver_pulseaudio.cpp
+++ b/drivers/pulseaudio/audio_driver_pulseaudio.cpp
@@ -200,6 +200,8 @@ Error AudioDriverPulseAudio::init_output_device() {
 		return err;
 	}
 
+	pa_channel_map_init_mono(&pa_map);
+
 	if (pa_map.channels <= 8) {
 		channels = pa_map.channels;
 	} else {

kus04e4ek · 2024-04-19T12:42:10Z

Web audio is broken on master too. (#90906)
MovieWriter is not recording any video (#90932) and corrupts audio when using any SpeakerMode other than Stereo (can't reproduce it anymore?) on master too.
Apart from that (and the fact that I didn't test on MacOS and iOS), this PR seems fully functional.

kus04e4ek · 2024-04-19T13:09:49Z

servers/audio_server.cpp

+		case AudioDriver::BUFFER_FORMAT_INTEGER_32:
+			// (1u << (32 - 1)) - 1 can't be stored in float correctly.
+			((int32_t *)p_buffer)[p_idx] = (double)p_sample * ((1u << (32 - 1)) - 1);


It is different from the implementation that was before:

godot/servers/audio_server.cpp

Lines 288 to 291 in 9bc49a6

float l = CLAMP(buf[from + j].left, -1.0, 1.0);

int32_t vl = l * ((1 << 20) - 1);

int32_t vl2 = (vl < 0 ? -1 : 1) * (ABS(vl) << 11);

*dest = vl2;

But I'm not sure why it was like this, all I can find on the Internet is how to convert from float to 16-bit integer and it's just to multiply a float to match the range of the number (eg. -1.0 is the minimum value that can fit in 16-bit integer and 1.0 is the maximum), so I expect it to be the same with 32-bit integers?
Also, used everywhere ((1u << (*bits* - 1)) - 1), though it would be better to use (1u << (*bits* - 1)) for negative numbers and ((1u << (*bits* - 1)) - 1) for positive ones. But I think it's really hard to hear the difference so kept it like this for a small optimization (which would be good as audio callbacks really should run as fast as possible).

kus04e4ek · 2024-04-19T13:11:32Z

servers/audio_server.cpp

+		case AudioDriver::BUFFER_FORMAT_INTEGER_24: {
+			int32_t sample = int32_t(((int8_t *)p_buffer)[p_idx * 3 + 2]) << 16;
+			sample |= int32_t(((int8_t *)p_buffer)[p_idx * 3 + 1]) << 8;
+			sample |= int32_t(((int8_t *)p_buffer)[p_idx * 3 + 0]);
+
+			return (float)sample / (1u << (24 - 1));


A sign will probably be wrong, but unsure how to fix that.

kus04e4ek · 2024-04-19T13:11:55Z

servers/audio_server.cpp

+		case AudioDriver::BUFFER_FORMAT_INTEGER_24: {
+			int32_t sample = p_sample * ((1u << (24 - 1)) - 1);
+			int32_t sign = p_sample < 0 ? -1 : 1;
+
+			((int8_t *)p_buffer)[p_idx * 3 + 2] = sign * (ABS(sample) >> 16);
+			((int8_t *)p_buffer)[p_idx * 3 + 1] = sign * (ABS(sample) >> 8);
+			((int8_t *)p_buffer)[p_idx * 3 + 0] = sample;


Same as the above.

kus04e4ek · 2024-04-19T13:14:21Z

platform/web/js/libs/library_godot_audio.js

 				const inb = GodotRuntime.heapSub(HEAPF32, p_in_buf, p_in_size);
 				const input = event.inputBuffer;
+				const input_channels = input.numberOfChannels;
 				if (GodotAudio.input) {
-					const inlen = input.getChannelData(0).length;
-					for (let ch = 0; ch < 2; ch++) {
+					for (let ch = 0; ch < input_channels; ch++) {
 						const data = input.getChannelData(ch);
-						for (let s = 0; s < inlen; s++) {
-							inb[s * 2 + ch] = data[s];
+						for (let sample = 0; sample < data.length; sample++) {
+							inb[sample * input_channels + ch] = data[sample];


Might not be a good idea to pass all of the channels of the input buffer, but thought that downmixing code should be kept in the same place for all AudioDrivers.

Faless · 2024-04-19T13:53:39Z

Web audio is broken on master too.

Can you expand on this? If you use threaded export audio should work fine.

kus04e4ek · 2024-04-19T16:09:14Z

Web audio is broken on master too.

Can you expand on this? If you use threaded export audio should work fine.

Created an Issue: Web audio is distorted when using headphones #90906.

kus04e4ek · 2024-04-29T12:20:55Z

drivers/SCsub

+if env["platform"] == "windows" and not env.msvc:
+    SConscript("backtrace/SCsub")


Moved out of Sounds drivers section

kus04e4ek · 2024-04-29T12:23:39Z

drivers/SCsub

+if env["platform"] in ["ios", "macos"]:
+    SConscript("coreaudio/SCsub")
+
+if env["platform"] == "linuxbsd":
+    if env["alsa"]:
+        SConscript("alsa/SCsub")
+    if env["pulseaudio"]:
+        SConscript("pulseaudio/SCsub")
+
 if env["platform"] == "windows":
+    if env["xaudio2"]:
+        SConscript("xaudio2/SCsub")
    SConscript("wasapi/SCsub")


I think it's better this way, as no empty files will be compiled. Maybe it would be better to add checks like this to other drivers? Also should I delete checks like #ifdef ALSA_ENABLED in files?

kus04e4ek · 2024-04-29T12:25:14Z

drivers/xaudio2/audio_driver_xaudio2.cpp


 	int latency = Engine::get_singleton()->get_audio_output_latency();
-	buffer_size = closest_power_of_2(latency * mix_rate / 1000);


I don't think power of 2 is required, can't find anything about it online

kus04e4ek · 2024-04-29T12:26:47Z

drivers/xaudio2/audio_driver_xaudio2.h

-	WAVEFORMATEX wave_format = { 0 };
+	WAVEFORMATEX wave_format;


Fixes g++ warning

kus04e4ek · 2024-04-29T12:36:23Z

platform/web/audio_driver_web.cpp

+	if (unlikely(audio_context.input_channels == 0)) {
+		audio_context.input_channels = godot_audio_get_input_channels();
+	}


Don't really know if there is a better way to initialize input_channels

kus04e4ek · 2024-04-29T12:47:52Z

servers/audio_server.h

 	unsigned int input_position = 0;
 	unsigned int input_size = 0;

-	void audio_server_process(int p_frames, int32_t *p_buffer, bool p_update_mix_time = true);
+	void audio_server_process(int p_frames, void *p_buffer, bool p_active = true, bool p_update_mix_time = true);


Not really sure about p_active, it tells whether or not to process audio or just fill buffer with 0. I think it should be an argument, because AudioDriverOpenSL and AudioDriverCoreAudio pass false if callback is unable to lock mutex, it doesn't feel right to have a function like is_active because of that. input_process has something simmilar to this, if you pass nullptr as a buffer (it's needed for AudioDriverWASAPI), internall buffer will be filled with 0, we can't pass nullptr in audio_server_process, as we still need to fill a buffer. It's just a simplification to not call memset and to not reapeat the same code

kus04e4ek · 2024-07-06T16:29:38Z

Marked as a draft, cause I want to separate some changes into different PRs/commits

nyabinary · 2024-10-02T16:23:03Z

Marked as a draft, cause I want to separate some changes into different PRs/commits

Any updates on this?

kus04e4ek · 2024-10-06T09:11:54Z

Marked as a draft, cause I want to separate some changes into different PRs/commits

Any updates on this?

Sadly no, have no time right now :/

kus04e4ek requested review from a team as code owners March 29, 2024 10:14

AThousandShips added enhancement topic:audio labels Mar 29, 2024

AThousandShips added this to the 4.x milestone Mar 29, 2024

kus04e4ek force-pushed the audio-fix branch from 6fbf194 to e6f62c3 Compare March 29, 2024 11:16

kus04e4ek commented Mar 29, 2024

View reviewed changes

kus04e4ek force-pushed the audio-fix branch from e6f62c3 to f35f1e4 Compare March 29, 2024 11:24

kus04e4ek force-pushed the audio-fix branch from f35f1e4 to 01a3669 Compare April 19, 2024 12:41

kus04e4ek force-pushed the audio-fix branch from 01a3669 to bcf65b5 Compare April 19, 2024 12:48

kus04e4ek commented Apr 19, 2024

View reviewed changes

kus04e4ek force-pushed the audio-fix branch 2 times, most recently from e86d4df to b1d4f24 Compare April 19, 2024 13:29

kus04e4ek mentioned this pull request Apr 20, 2024

Godot does not listen to microphones with over 2 channels #82823

Open

kus04e4ek force-pushed the audio-fix branch from b1d4f24 to 453bc79 Compare April 20, 2024 11:25

kus04e4ek closed this Apr 20, 2024

kus04e4ek force-pushed the audio-fix branch from 453bc79 to 9bc49a6 Compare April 20, 2024 11:43

kus04e4ek reopened this Apr 20, 2024

kus04e4ek force-pushed the audio-fix branch 3 times, most recently from 4320fea to 620e598 Compare April 29, 2024 12:17

kus04e4ek requested a review from a team as a code owner April 29, 2024 12:17

kus04e4ek commented Apr 29, 2024

View reviewed changes

Audio: Write to device's buffer directly without temporary buffers.

f39ef50

kus04e4ek force-pushed the audio-fix branch from 620e598 to f39ef50 Compare May 3, 2024 14:24

kus04e4ek mentioned this pull request Jun 6, 2024

Replace the current audio driver with Oboe on Android #86776

Draft

13 tasks

kus04e4ek marked this pull request as draft July 6, 2024 16:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Audio: Write to device's buffer directly without temporary buffers. #90013

Audio: Write to device's buffer directly without temporary buffers. #90013

kus04e4ek commented Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

fire commented Mar 29, 2024

kus04e4ek commented Mar 29, 2024 •

edited

Loading

kus04e4ek commented Apr 19, 2024 •

edited

Loading

kus04e4ek Apr 19, 2024

kus04e4ek Apr 19, 2024

kus04e4ek Apr 19, 2024

kus04e4ek Apr 19, 2024 •

edited

Loading

Faless commented Apr 19, 2024

kus04e4ek commented Apr 19, 2024

kus04e4ek Apr 29, 2024

kus04e4ek Apr 29, 2024 •

edited

Loading

kus04e4ek Apr 29, 2024

kus04e4ek Apr 29, 2024

kus04e4ek Apr 29, 2024

kus04e4ek Apr 29, 2024

kus04e4ek commented Jul 6, 2024

nyabinary commented Oct 2, 2024

kus04e4ek commented Oct 6, 2024

		@@ -30,11 +30,6 @@

		#include "audio_driver_opensl.h"

		#include <string.h>

		virtual SpeakerMode get_speaker_mode() const = 0;
		virtual int get_total_channels() const = 0;

	float l = CLAMP(buf[from + j].left, -1.0, 1.0);
	int32_t vl = l * ((1 << 20) - 1);
	int32_t vl2 = (vl < 0 ? -1 : 1) * (ABS(vl) << 11);
	*dest = vl2;

		if env["platform"] == "windows" and not env.msvc:
		SConscript("backtrace/SCsub")


		int latency = Engine::get_singleton()->get_audio_output_latency();
		buffer_size = closest_power_of_2(latency * mix_rate / 1000);

Audio: Write to device's buffer directly without temporary buffers. #90013

Are you sure you want to change the base?

Audio: Write to device's buffer directly without temporary buffers. #90013

Conversation

kus04e4ek commented Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

kus04e4ek Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

kus04e4ek Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

kus04e4ek Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

kus04e4ek Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

fire commented Mar 29, 2024

kus04e4ek commented Mar 29, 2024 • edited Loading

kus04e4ek commented Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kus04e4ek Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

Faless commented Apr 19, 2024

kus04e4ek commented Apr 19, 2024

Choose a reason for hiding this comment

kus04e4ek Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kus04e4ek commented Jul 6, 2024

nyabinary commented Oct 2, 2024

kus04e4ek commented Oct 6, 2024

kus04e4ek commented Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek commented Mar 29, 2024 •

edited

Loading

kus04e4ek commented Apr 19, 2024 •

edited

Loading

kus04e4ek Apr 19, 2024 •

edited

Loading

kus04e4ek Apr 29, 2024 •

edited

Loading