Picture in Picture support for VideoView and SwiftUI VideoView

[VatamanuBogdan]

Is your feature request related to a problem? Please describe.

I read about how to add Picture in Picture in a video call application and there are two steps that you have to follow to add PiP support:

1. You have to create an AVPictureInPictureController that has an AVSampleBufferDisplayLayer attached to it where the buffer contains the video content that will be shown on the PiP window.
2. Make sure that the AVSampleBufferDisplayLayer from the previous step is added as a subview to the view hierarchy otherwise the PiP will not work

I've tried to add PiP support to my LiveKit application, the implementation makes use of avSampleBufferDisplayLayer VideoView property by using sampleBuffer rendering mode. It worked but it has some flaws. The problem is caused by the internal renderer invalidation that happens here, the renderer will be recreated and thus the avSampleBufferDisplayLayer will be changed, this way the AVPictureInPictureController won't point to the right buffer and PiP won't work.

Describe the solution you'd like

I think that the best way to enable Picture in Picture support in LiveKit would be to add to the VideoView an extra SampleBufferVideoRenderer that will have the purpose of exposing to the users its internal AVSampleBufferDisplayLayer (a new property on VideoView that is named pictureInPictureDisplayLayer). Another aspect of this implementation is that the pictureInPictureDisplayLayer must not be destroyed whenever the track state changes instead it should be flushed.

The new overhead added to VideoView should be on user demand. I suggest adding a new boolean property pictureInPictureEnabled on VideoView that will handle the creation of pictureInPictureDisplayLayer.

Describe alternatives you've considered

Other alternative solutions, would be:

1. Stop invalidating the renderer whenever the track state changes and flush the old content of SampleBufferVideoRenderer
2. Add a new PictureInPictureRenderer renderer class that exposes the implementation of SampleBufferVideoRenderer as public, this way the library users will be able to attach it to a VideoTrack and add Picture in Picture support by themselves without the usage of VideoView.

Additional context

I would like to help with this new feature if you agree and create a PR with one of the solutions that were proposed above.

[jestha166]

Hi @VatamanuBogdan , were you able to create the pip view when the app goes in background?

[VatamanuBogdan]

Hi, @jestha166, sorry for the late response.
Yes, I was able to create the PiP view by using avSampleBufferDisplayLayer property that is not null when the rendering mode is set to sample buffer.

All you have to do is to create an AVPictureInPictureController:

let contentSource = AVPictureInPictureController.ContentSource(
            /* Where videoView it's your view that renders in sampleBuffer mode */
            sampleBufferDisplayLayer: videoView.avSampleBufferDisplayLayer,
            playbackDelegate: self
)

let pipController = AVPictureInPictureController(contentSource: contentSource)

Then you only have to start the pip controller when your app goes in background using:

pipController.startPictureInPicture()

and stop it when the app return in foreground:

pipController.stopPictureInPicture()