Composition Player gets stuck during sequence transition when using MultipleInputVideoGraph

[AradiPatrik]

Version

Media3 main branch

More version details

Top of main branch as of Aug 25 2025

Devices that reproduce the issue

All devices

Devices that do not reproduce the issue

none

Reproducible in the demo app?

Yes

Reproduction steps

DefaultVideoCompositor deadlock causes multi-sequence compositions to freeze during media item transitions

Summary

When using MultipleInputVideoGraph with sequences containing multiple media items, the DefaultVideoCompositor deadlocks when one sequence transitions between items, causing the entire composition to freeze.

1. Open demo-composition app
2. Select "2x2 grid" layout
3. Delete all items from video sequence
4. Add the first sample video 8 times (10-second video)
5. Start playback

repro.mov

Expected: All 4 sequences play their 2 videos (20 seconds total) smoothly
Actual: Video freezes after first 10 seconds, audio continues, timer keeps counting indefinitely, seeking breaks

Root Cause

DefaultVideoCompositor.getFramesToComposite() blocks ALL compositing when ANY input has empty frame queue:

for (int i = 0; i < inputSources.size(); i++) {
  if (inputSources.valueAt(i).frameInfos.isEmpty()) {
    return ImmutableList.of();  // Blocks all compositing
  }
}

During media item transitions, the transitioning sequence's frame queue becomes temporarily empty while the decoder reinitializes. This prevents the compositor from producing any output frames, causing other sequences to never reach their end-of-stream signal.

Example timing from debugging:

• Sequence transitions at 10s (decoder release → empty queue → reinitialization)
• Compositor stops producing output during this ~200ms gap
• Other sequences need their final ~120ms of frames to reach end-of-stream
• These final frames are never composited due to the blocking condition
• Sequences cannot signal end-of-stream (lastOutputFrame < finalFrame)
• Permanent deadlock

Validate Root Cause

Early End-of-Stream Signal

Modify PlaybackVideoGraphWrapper to signal end-of-stream slightly before the actual end to avoid the deadlock:

// In onOutputFrameAvailableForRendering()
boolean isLastFrame = finalFramePresentationTimeUs != C.TIME_UNSET
    && framePresentationTimeUs >= finalFramePresentationTimeUs - TOLERANCE_US;  // Add tolerance
if (isLastFrame) {
  signalEndOfVideoGraphOutputStream();
}

This ensures sequences can transition even if the last few milliseconds aren't composited.
This completely fixed the issue on my part but feels hacky.

Proposed Solutions

Option 1: Reuse Last Frame

Modify getFramesToComposite() to use the last available frame when an input is temporarily empty:

if (inputSource.frameInfos.isEmpty() && lastFrameInfo != null) {
  framesToComposite.add(lastFrameInfo);  // Use last frame instead of blocking
}

Implemented this approach in the PR.

Option 2: Skip Empty Inputs

Allow compositing with available inputs only, temporarily excluding empty ones.
This would result in black/empty frames during preview which is not what we want in my opinion.

Additional Notes

• Issue occurs consistently when any sequence transitions while others are mid-playback
• The PlaybackVideoGraphWrapper correctly tracks presentation times but cannot progress due to missing compositor output
• Single sequence compositions work correctly as they don't trigger the blocking condition

Expected result

See description above

Actual result

See description above

Media

reproducible in demo-composition using the included media

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[droid-girl]

Hi @AradiPatrik ,
The team is currently working to enable transitions and MultipleInputVideoGraph APIs.
There are multiple limitations with MultipleInputVideoGraph and CompositionPlayer, for example seeking or transitions are not fully supported yet.
We are actively working on enabling transitions in CompositionPlayer.