Added audio buffer

[hegde95]

closes #225

This work adds sound into the state space of vizdoom, we can now pass audio buffers synchronously as a NumPy array to the state variable. To enable this set game.set_soft_sound_enabled(True). enabling this slows down sync frame rate by a bit, but this gives researchers the option to tap into a lesser-seen state space.

[Miffyli]

@mwydmuch

These madlads @hegde95 and @alex-petrenko did it and added sound buffer support for ViZDoom. The code has been tested to work with nice results (see here).

There are some tweaks @hegde95 is still adding, but once that is done, can you also take glance over it to see if it checks out? I will also add a Python example on using this.

[mwydmuch]

Hi @hegde95 and @alex-petrenko, thank you very much for this PR, this is really amazing! I will take a more detailed look at the code this week, for now, I just replaced OpenAL requirement with OpenAL Soft, since basic MacOS and Windows implementations don't have required extensions.

[Miffyli]

I updated the example to be more compact and specific to the sound buffers (along with example of storing audio).

Major comments:

• OpenAL Soft seems to allow running the code on Ubuntu 20.04 (previously crashed), but I can not hear any audio with any settings. Can somebody confirm they get the same?
• audio_buffer exists and is populated even when game.set_soft_sound_enabled(False) is set. Preferably with this setting things should be as before
• Change set_soft_sound_enabled to set_sound_buffer_enabled for consistency
• Change set_sound_observation_num_frames to set_sound_buffer_size (I think this is clearer this way)

[mwydmuch]

@Miffyli, I'm sorry, my previous commit broke the Ubuntu version, I've fixed this, but...:

• Even with OpenAL Soft on Mac, the process crashes because "ALC_SOFT_loopback" is not available (openal-info shows it is) :/
• On Ubuntu 20.04, there is a problem with OpenAL Soft 1.19 (Error when running examples/python/basic.py #472), this can be fixed by installing a newer version from the source, however after doing it, sound_buffer.py produce silent file basic_sound.wav (normal sound works) :(

[mwydmuch]

@hegde95 and @alex-petrenko, I agree with @Miffyli comments and I would only add one: if "ALC_SOFT_loopback" and "ALC_EXT_thread_local_context" is not available, the error message should be sent to the controller using VIZ_MQSend(VIZ_MSG_CODE_DOOM_ERROR, <error message>); But the priority is to make it work on newer Ubuntu versions and to stop sound buffer from being populated when the option is disabled. I will also try to work on this in the following days, but unfortunately, my time is a bit limited.

[Miffyli]

sound_buffer.py produce silent file basic_sound.wav (normal sound works) :(

Oh that is interesting. For me (before your updates) the normal sound did not work either (it ran the game, I just could not hear anything). At least we are taking steps to right direction :D

[Miffyli]

@hegde95 and @mwydmuch a small poke, to avoid leaving this PR all alone :)

Quick test with 20.04 original OpenAL still yields no luck. Did not try to install newer OpenAL system-wide in case it breaks something. Conda has OpenAL Soft 1.20 in their packages, which could (maybe?) fix this but seems to require more setup to point compilers to the conda library.

[mwydmuch]

@Miffyli, @hegde95, I'm currently on vacation. I will be back on 26 July, and then for sure, I will work on It :)

[hegde95]

The last three changes address #391

[alex-petrenko]

@hegde95 thank you for adding these!

[mwydmuch]

Hi guys, I tested it again on different configurations. It seems that after upgrading OpenAL Soft to 1.21 (from source), the current version of this branch works as expected on Ubuntu 20.04 and on Ubuntu 21.04. It also works on Windows 10 (with alext.h header added). I still have a problem with Mac, but I guess it's related to building. I think we can merge this after improving the nameing of few methods. I would like to make it more consistent with the rest of the API. Also, the "sound" and "audio" is used interchangeably right now. I think we should decide to use one of these words. Personally, I think "audio" fits the context better. Thus I would propose to change the current API to (and also rename some internals):

• State.audio_buffer
• is/set_soft_sound_enabled -> is/set_audio_buffer_enabled
• get/set_sound_sampling_freq -> get/set_audio_sampling_freq
• get/set_sound_observation_num_frames -> get/set_audio_buffer_size

What do you think?
I can take care of these changes.
It would also be nice to add short descriptions of these methods to doc/DoomGame.md

[Miffyli]

@mwydmuch

Can you provide instructions on how you updated the OpenAL package? I assume it was system-wide update? It would be nice to have instructions (or just a pointer that "go here, follow instructions") on this somewhere in case somebody runs into same problems :D. I can also try this on couple of machines (the results have been odd all around tbh).

I agree with the naming, sounds good!

[mwydmuch]

Sure @Miffyli, my way may not be entirely correct, since I'm tinkering with paths a bit. The paths should probably be setup during project configuration, but right I'm not sure how to do it.

git clone https://github.com/kcat/openal-soft.git
cd openal-soft
git checkout 1.21.1 # Latest tagged version
cd build
cmake ..
make -j
sudo make install 

This will install to /usr/local/, which is ok, except that libopenal.so is expected to be in /usr/lib/x86_64-linux-gnu instead of /usr/local/lib, that is why I add the link manually:

sudo rm /usr/lib/x86_64-linux-gnu/libopenal.so # If already installed from apt
sudo rm /usr/lib/x86_64-linux-gnu/libopenal.so.1
sudo ln -s /usr/local/lib/libopenal.so /usr/lib/x86_64-linux-gnu/libopenal.so
sudo ln -s /usr/local/lib/libopenal.so.1 /usr/lib/x86_64-linux-gnu/libopenal.so.1

[alex-petrenko]

Hi Marek! I think the naming suggestions are great, let's change it! Regarding the openal version thing, does this mean that everyone will need to install this custom version? Or does it work out of the box on the new Ubuntu versions?

…

On Thu, Jul 29, 2021, 12:07 Marek Wydmuch ***@***.***> wrote: Sure @Miffyli <https://github.com/Miffyli>, my way maybe not entarily correct, since I'm tinkering with paths a bit. The paths should probaly be setup during project configuration, but right I'm not sure how to do it. git clone https://github.com/kcat/openal-soft.git cd openal-soft git checkout 1.21.1 # Latest tagged version cd build cmake .. make -j sudo make install This will install to /usr/local/, which is ok, except that libopenal.so is expected to be in /usr/lib/x86_64-linux-gnu instead of /usr/local/lib, that is why I add link manually: sudo rm /usr/lib/x86_64-linux-gnu/libopenal.so # If allready instaled from apt sudo rm /usr/lib/x86_64-linux-gnu/libopenal.so.1 sudo ln -s /usr/local/lib/libopenal.so /usr/lib/x86_64-linux-gnu/libopenal.so sudo ln -s /usr/local/lib/libopenal.so.1 /usr/lib/x86_64-linux-gnu/libopenal.so.1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#486 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJ6HLYMDVTCE47JEYMOFCDT2GRHTANCNFSM472N6MSQ> .

[mwydmuch]

There is the same problem with Ubuntu 20.04 and 21.04 since they are using the same OpenAL Soft versions (1.19.1). I did some more research, and found that efx extension is responsible for the error. One can disable it in ViZDoom using game.add_game_args("+snd_efx 0"), then it works with default OpenAL Soft. Possibly, it disables some more advanced audio effects like echo.

[Miffyli]

I gave the suggestions a quick try but no luck. Installing OpenAL Soft led to some confused stuff with sudo ("no permissions"), and even doing the awful thing of installing/running things in sudo, I was given a buffer of silent audio :( . I am running KDE Neon which might change results, and this might be too complex to fix over all distros right here.

I added some notes in example to help people spot this and possibly fix it.

[mwydmuch]

Ok, I've made few more small changes. The amount of memory allocated for the audio buffer now depends on the value passed by set_audio_buffer_size, the hardcoded limit of 4 frames was removed. I also fixed some issues on MacOS (probably not in a very elegant way, but now it works). I'm ok with merging this now.

[mwydmuch]

@Miffyli both of them failed? game.add_game_args("+snd_efx 0") too? That's a bit disappointing :/ @mihahauke could you also try to test it?

[Miffyli]

Yeah, sadly both of the approaches failed :(. However, I have one more silly thing I could try out: I did not have any audio devices attached to the machine (remote workstation), so it might be OS's settings that mute audio (maybe). I will try running it with headset attached, and also test if I can hear sound without audio buffers.

@hegde95 can you check if changes are ok to you so we can merge this? :)

[Miffyli]

@mwydmuch Now the things work! After several re-installations of openal-soft the things now work. I think removing the old openal-soft first and then installing this new one eventually did the trick (I used checkinstall tool on Ubuntu, that might have also had an effect). Now I can hear sounds stored in the audio file, but still can not hear sounds when feeding them to speakers.

Edit: I got hold of a fresh-er Ubuntu 20.04 (vanilla, not KDE neon like my workstation), and there the game.add_game_args("+snd_efx 0") fix worked fine, no installation needed! The code crashed to the good ol' Assertion with stuff about gain, but this parameter fixed it.

For anybody reading and trying to get sound in ViZDoom working, try out:

• If ViZDoom crashes on init, try doom_game.add_game_args("+snd_efx 0"). Note that this might remove some audio effects like reverberation.
• If ViZDoom runs without audio, try Marek's steps here.
• If that fails, try uninstalling openal you have on your system and then repeating installation.

@hegde95 will check this soon for validity, and then we can merge it. Splendid job 👍

[mwydmuch]

@Miffyli great to hear that! Can't wait to merge it :)

[Miffyli]

@hegde95 I added explicit "thank you"s to you in Readme for audio buffer. Give 👍 if that is ok to you :)

[mwydmuch]

I'm going through the PR again, I've noticed that you modified bots.cfg, I reverted these changes, since this file should serve as an example, so I would like to keep it simple. I'm not sure why you made this change, perhaps you would like to have these bots for your sample-factory. That's why quickly added the CVAR variable that allows loading custom bots config:

game.add_game_args("+viz_bots_path ../../scenarios/perfect_bots.cfg")

(added this line and comment to cig_multiplayer_bots.py).

[hegde95]

Hey sorry for the delay, I was able to look at things and they seem fine! Merge away!

[mwydmuch]

Ok, merged. Thank you very much @hegde95, @alex-petrenko, and @Miffyli for your contributions, I will prepare a new release soon :)