Skip to content

Commit

Permalink
Custom audio modes handling (#260)
Browse files Browse the repository at this point in the history
* Audio types feature

* docs

* cleanup

* audio docs
  • Loading branch information
davidliu authored Sep 5, 2023
1 parent f2c1a53 commit 265b5d0
Show file tree
Hide file tree
Showing 12 changed files with 334 additions and 56 deletions.
51 changes: 30 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,15 @@
- [Docs](#docs)
- [Installation](#installation)
- [Usage](#usage)
- [Permissions](#permissions)
- [Publishing camera and microphone](#publishing-camera-and-microphone)
- [Sharing screen](#sharing-screen)
- [Rendering subscribed tracks](#rendering-subscribed-tracks)
- [Audio modes](#audio-modes)
- [@FlowObservable](#flowobservable)
- [Permissions](#permissions)
- [Publishing camera and microphone](#publishing-camera-and-microphone)
- [Sharing screen](#sharing-screen)
- [Rendering subscribed tracks](#rendering-subscribed-tracks)
- [Audio modes](#audio-modes)
- [@FlowObservable](#flowobservable)
- [Sample App](#sample-app)
- [Dev Environment](#dev-environment)
- [Optional (Dev convenience)](#optional-dev-convenience)
- [Optional (Dev convenience)](#optional-dev-convenience)

## Docs

Expand Down Expand Up @@ -67,7 +67,9 @@ subprojects {
### Permissions

LiveKit relies on the `RECORD_AUDIO` and `CAMERA` permissions to use the microphone and camera.
These permission must be requested at runtime. Reference the [sample app](https://github.com/livekit/client-sdk-android/blob/4e76e36e0d9f895c718bd41809ab5ff6c57aabd4/sample-app-compose/src/main/java/io/livekit/android/composesample/MainActivity.kt#L134) for an example.
These permission must be requested at runtime. Reference
the [sample app](https://github.com/livekit/client-sdk-android/blob/4e76e36e0d9f895c718bd41809ab5ff6c57aabd4/sample-app-compose/src/main/java/io/livekit/android/composesample/MainActivity.kt#L134)
for an example.

### Publishing camera and microphone

Expand Down Expand Up @@ -174,22 +176,27 @@ for the full implementation.

### Audio modes

WebRTC utilizes an audio module to interface with the device's audio input and output. By default, the audio module is configured for two-way communications.
By default, the audio is configured for two-way communications.

If you are building a livestreaming or music app, you can make the following tweaks to improve playback quality:
If you are building a livestreaming or media playback focus app, you can use the preset
`MediaAudioType` when creating the `Room` object for better audio quality.

```kt
WebRTCModuleOptions options = WebRTCModuleOptions.getInstance();
AudioDeviceModule adm = JavaAudioDeviceModule.builder(this)
.setAudioAttributes(AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_MEDIA)
.setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
.build())
.setUseStereoOutput(true)
.build();
options.audioDeviceModule = adm;
val room = LiveKit.create(
appContext = application,
overrides = LiveKitOverrides(
audioOptions = AudioOptions(
audioOutputType = AudioType.MediaAudioType()
)
)
)
```

Note: audio routing becomes automatically handled by the system and cannot be manually controlled.

For more control over the specific audio attributes and modes, a `CustomAudioType` can be
passed instead.

### `@FlowObservable`

Properties marked with `@FlowObservable` can be accessed as a Kotlin Flow to observe changes
Expand All @@ -205,11 +212,13 @@ coroutineScope.launch {

## Sample App

**Note**: If you wish to run the sample apps directly from this repo, please consult the [Dev Environment instructions](#dev-environment).
**Note**: If you wish to run the sample apps directly from this repo, please consult
the [Dev Environment instructions](#dev-environment).

We have a basic quickstart sample
app [here](https://github.com/livekit/client-sdk-android/blob/main/sample-app-basic), showing how to
connect to a room, publish your device's audio/video, and display the video of one remote participant.
connect to a room, publish your device's audio/video, and display the video of one remote
participant.

There are two more full featured video conferencing sample apps:

Expand Down
2 changes: 1 addition & 1 deletion livekit-android-sdk/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ dependencies {
implementation 'org.jetbrains.kotlinx:kotlinx-serialization-json:1.1.0'
api 'io.github.webrtc-sdk:android:104.5112.10'
api "com.squareup.okhttp3:okhttp:4.10.0"
api 'com.github.davidliu:audioswitch:1689af118f69dcd8c8dc95e5a711dd0a7a626e69'
api 'com.github.davidliu:audioswitch:7b55cec426227a75be25b0d7ad8537d4aede2a2a'
implementation "androidx.annotation:annotation:1.4.0"
implementation "androidx.core:core:${versions.androidx_core}"
implementation "com.google.protobuf:protobuf-javalite:${versions.protobuf}"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@

package io.livekit.android

import android.media.AudioAttributes
import android.media.AudioManager
import io.livekit.android.audio.AudioFocusHandler
import io.livekit.android.audio.AudioHandler
import io.livekit.android.audio.AudioSwitchHandler
import io.livekit.android.audio.NoAudioHandler
import okhttp3.OkHttpClient
import org.webrtc.VideoDecoderFactory
Expand All @@ -25,14 +29,45 @@ import org.webrtc.audio.AudioDeviceModule
import org.webrtc.audio.JavaAudioDeviceModule

/**
* Overrides to replace LiveKit internally used component with custom implementations.
* Overrides to replace LiveKit internally used components with custom implementations.
*/
data class LiveKitOverrides(
/**
* Override the [OkHttpClient] used by the library.
*/
val okHttpClient: OkHttpClient? = null,

/**
* Override the [VideoEncoderFactory] used by the library.
*/
val videoEncoderFactory: VideoEncoderFactory? = null,

/**
* Override the [VideoDecoderFactory] used by the library.
*/
val videoDecoderFactory: VideoDecoderFactory? = null,

val audioOptions: AudioOptions? = null,
)


class AudioOptions(
/**
* Override the default output [AudioType].
*
* This affects the audio routing and how the audio is handled. Default is [AudioType.CallAudioType].
*
* Note: if [audioHandler] is also passed, the values from [audioOutputType] will not be reflected in it,
* and must be set manually.
*/
val audioOutputType: AudioType? = null,
/**
* Override the default [AudioHandler].
*
* Use [NoAudioHandler] to turn off automatic audio handling.
*/
val audioHandler: AudioHandler? = null,

/**
* Override the default [AudioDeviceModule].
*/
Expand All @@ -44,22 +79,47 @@ data class LiveKitOverrides(
* Not used if [audioDeviceModule] is provided.
*/
val javaAudioDeviceModuleCustomizer: ((builder: JavaAudioDeviceModule.Builder) -> Unit)? = null,
)

sealed class AudioType(val audioMode: Int, val audioAttributes: AudioAttributes, val audioStreamType: Int) {
/**
* Override the [VideoEncoderFactory] used by the library.
* An audio type for general media playback usage (i.e. listener-only use cases).
*
* Audio routing is handled automatically by the system in normal media mode,
* and bluetooth microphones may not work on some devices.
*
* The default [AudioHandler] for this type is [AudioFocusHandler].
*/
val videoEncoderFactory: VideoEncoderFactory? = null,
class MediaAudioType : AudioType(
AudioManager.MODE_NORMAL,
AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_MEDIA)
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
.build(),
AudioManager.STREAM_MUSIC
)

/**
* Override the [VideoDecoderFactory] used by the library.
* An audio type for calls (i.e. participating in the call or publishing local microphone).
*
* Audio routing can be manually controlled.
*
* The default [AudioHandler] for this type is [AudioSwitchHandler].
*/
val videoDecoderFactory: VideoDecoderFactory? = null,
class CallAudioType : AudioType(
AudioManager.MODE_IN_COMMUNICATION,
AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_VOICE_COMMUNICATION)
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
.build(),
AudioManager.STREAM_VOICE_CALL
)

/**
* Override the default [AudioHandler].
* An audio type that takes in a user-defined [AudioAttributes] and audio stream type.
*
* Use [NoAudioHandler] to turn off automatic audio handling.
* The default [AudioHandler] for this type is [AudioFocusHandler].
*/

val audioHandler: AudioHandler? = null
)
class CustomAudioType(audioMode: Int, audioAttributes: AudioAttributes, audioStreamType: Int) :
AudioType(audioMode, audioAttributes, audioStreamType)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
/*
* Copyright 2023 LiveKit, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package io.livekit.android.audio

import android.content.Context
import android.media.AudioAttributes
import android.media.AudioFocusRequest
import android.media.AudioManager
import android.os.Build
import androidx.annotation.RequiresApi
import javax.inject.Inject
import javax.inject.Singleton

/**
* A basic [AudioHandler] that manages audio focus while started.
*/
@Singleton
open class AudioFocusHandler
@Inject
constructor(context: Context) : AudioHandler {

/**
* The audio focus mode to use while started.
*
* Defaults to [AudioManager.AUDIOFOCUS_GAIN].
*/
var focusMode: Int = AudioManager.AUDIOFOCUS_GAIN

/**
* The audio stream type to use when requesting audio focus on pre-O devices.
*
* Defaults to [AudioManager.STREAM_MUSIC].
*
* Refer to this [compatibility table](https://source.android.com/docs/core/audio/attributes#compatibility)
* to ensure that your values match between android versions.
*/
var audioStreamType: Int = AudioManager.STREAM_MUSIC

/**
* The audio attribute usage type to use when requesting audio focus on devices O and beyond.
*
* Defaults to [AudioAttributes.USAGE_MEDIA].
*
* Refer to this [compatibility table](https://source.android.com/docs/core/audio/attributes#compatibility)
* to ensure that your values match between android versions.
*/
var audioAttributeUsageType: Int = AudioAttributes.USAGE_MEDIA

/**
* The audio attribute content type to use when requesting audio focus on devices O and beyond.
*
* Defaults to [AudioAttributes.CONTENT_TYPE_SPEECH].
*
* Refer to this [compatibility table](https://source.android.com/docs/core/audio/attributes#compatibility)
* to ensure that your values match between android versions.
*/
var audioAttributeContentType: Int = AudioAttributes.CONTENT_TYPE_SPEECH

private val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager
private var audioRequest: AudioFocusRequest? = null
private var audioFocusListener = AudioManager.OnAudioFocusChangeListener {
onAudioFocusChangeListener?.onAudioFocusChange(it)
}

var onAudioFocusChangeListener: AudioManager.OnAudioFocusChangeListener? = null

override fun start() {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
audioRequest = createAudioRequest()
audioRequest?.let { audioManager.requestAudioFocus(it) }
} else {
@Suppress("DEPRECATION")
audioManager.requestAudioFocus(audioFocusListener, audioStreamType, focusMode)
}
}

override fun stop() {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
audioRequest?.let { audioManager.abandonAudioFocusRequest(it) }
audioRequest = null
} else {
@Suppress("DEPRECATION")
audioManager.abandonAudioFocus(onAudioFocusChangeListener)
}
}

@RequiresApi(Build.VERSION_CODES.O)
open fun createAudioRequest(): AudioFocusRequest {
return AudioFocusRequest.Builder(focusMode)
.setOnAudioFocusChangeListener(audioFocusListener)
.setAudioAttributes(
AudioAttributes.Builder()
.setUsage(audioAttributeUsageType)
.setContentType(audioAttributeContentType)
.build()
)
.build()
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package io.livekit.android.audio

/**
* Interface for handling android audio routing.
* Interface for handling android audio.
*/
interface AudioHandler {
/**
Expand Down
Loading

0 comments on commit 265b5d0

Please sign in to comment.