Streaming ogg/opus in real time

[michel538]

As you suggested I am bringing one discussion into a distinct one.

My current knowledge of opus/ogg is limited at this moment. Some of my wording could be misleading because non-applicable.

The application:

• stream opus audios to an opus decoder
• audios chunk remains opus.
• streaming should be real time - wait between streaming each chunk the duration of the audio chunk.
• chunk duration should be estimated.

Thanks Matthew for your suggestion.

Suggestion:

I would recommend that you consider OpusFileStream.

OpusFileStream.get_buffer() or OpusFileStream.get_buffer_as_array() will get you the next "chunk" of PCM (decoded audio).

The duration of that PCM can be easily calculated. First find the length of the chuck in samples, then divide it by 48,000 (which is the number of samples per second for Opus-encoded audio). That gives you the length in seconds. Something like the following:

array = stream.get_buffer_as_array()
length_samples = len(array)
samples_per_second = stream.frequency
length_seconds = length_samples / samples_per_second

Question:

• is the "array" already a pcm?
• if so is there a way to get ogg chunk?

Thanks

Michel

[mattgwwalker]

Hi Michel,

I'm still not entirely clear on your specific application. However it sounds a lot like an audio-conferencing application.

With audio conferencing you would send Opus-encoded audio between, say, two participants. The sender records audio, which is broken into "chunks" as required by the Opus specification. The chucks' duration must be one of 2.5, 5, 10, 20, 40, or 60 ms. The sender then sends the Opus-encoded audio to the sender via an established network connection (typically UDP over IP). The receiver receives the Opus-encoded "chunk" and decodes it into PCM, which it then plays back. The receiver cannot entirely rely on the "chunks" arriving at the correct time (due, for example, to network delays) so a small jitter buffer is often implemented by the receiver. Note that this technique does not require Ogg encapsulation as the UDP framing may fill that role.

If you are interested in implementing such a system, then the class OpusBufferedEncoder would be helpful. It would be used by the sender to compress the audio into Opus-encoded "chunks". OpusDecoder would be used by the receiver to decode the audio into PCM. There is no need to "estimate" the duration of a chunk; it is known at the time of encoding and also when decoded.

If however you already have Opus-encoded audio from another application, and it is responsible for streaming the audio, then you would only need to write a "receiver". In this case I would recommend that you consider the class OpusFileStream.

To answer your two specific questions:

• Is the "array" already a pcm? Yes, array would be PCM.
• If so is there a way to get ogg chunk? By "ogg chunk" do you mean that you are wanting a sender to read an OggOpus file and obtain the Opus-encoded frame so that it could send the Opus-encoded audio over the wire without re-compression? If that's the objective then you could achieve this using PyOgg's ctypes interface to the Ogg library. However the current classes, such as OpusFile do not offer such functionality.

I hope that helps,

Matthew

[michel538]

Thanks Matthew,

My application is only the "sender" in your response above.

• define opus chunk duration.
• send opus chunk to receiver.
• wait "opus chunk duration" before sending next chunk.

You have shared how to get duration:

    buf = stream.get_buffer_as_array()
    length_samples = len(buf)
    samples_per_second = stream.frequency
    length_seconds = length_samples / samples_per_second

Having a pcm chunk (as numpy array) in "buf", how can I make "buf" back into a opus "chunk"?

You suggest the use of "OpusBufferedEncoder". I have played with it

    encoder=OpusBufferedEncoder()
    encoder.set_channels(2)
    encoder.set_sampling_frequency(48000)
    encoder.set_frame_size(20)
    encoder.buffered_encode(buf)

return:

pcm_ctypes = Buffer.from_buffer(pcm_bytes) TypeError: underlying buffer is not writable

"buf" type is probaby not correctly handle in my example. If you could help I would aprreciate.

Thanks

Michel

[mattgwwalker]

Oh wow, sorry for the ridiculously late reply!

You can make a buffer writable by coping it with a call to bytearray. So something like:

encoder.buffered_encode(bytearray(buf))

The reason writable buffers are required is complicated and to do with Python optimizing away copies of non-writable bytes.