Support reading the audio track of the video file as #MP3

[bohning]

Issue type: Feature request

Actual behaviour

USDX supports reading the audio track from the video file by setting, e.g.

#MP3:videofile.mp4
#VIDEO:videofile.mp4

If the video contains audio and is synchronous (this is probably the default), then a single video file with audio track is sufficient without the need to extract the audio track from the video file into a separate mp3 file.

UltraStar Play currently does not support this, resulting in an error:

[WRN] Unsupported audio format: videofile.mp4

Expected behaviour

It would be great if that feature was supported by UltraStar Play as well.

[achimmihca]

Sadly, getting an AudioClip from an mp4 file is not as easy with Unity as I expected.
I asked on their forums how to do it (if it can be done at all): https://forum.unity.com/threads/get-audioclip-from-mp4.1212537/

[bohning]

Interesting, I would as well have thought that this would be a rather easy feature to implement. Thanks for following up on it in the unity forums.

[achimmihca]

Can't we create a Temp folder and conversion it?

This would be (a) the most complicated and (b) inefficient approach.

• (a) because there is no conversion tool that we could ship or assume on all target platforms. Thus, we would have to write a converter ourself. Or the feature would be only available on some platforms with some configurations which IMO is not worth the effort.
• (b) because instead of extracting audio once by the song creator, it is extracted by every user again. Wasted energy.
  • Further, there would still be a redundant audio file, wasting memory.
  • Plus the initial conversion would take some time, which results in bad user experience.

Thus, I argue that it's better to use a separate audio file in the first place if we cannot use the video file's audio "on the fly".

[achimmihca]

I close this as won't fix.
It is not supported by Unity and we do not want to implement media file decoding.

Workaround: Songs have to be created with a separate audio file.

[achimmihca]

To be more specific: it is not supported to get an AudioClip from a video file.

The AudioClip is expected in the current implementation. However, it might be possible to add an abstraction and make it work with a VideoPlayer as well (one VideoPlayer for the video, other VideoPlayer for the audio).

Need to look into this again.