fix: pause & resume recording fix on iOS

[sroske]

What

When pausing and resuming record on iOS the paused timestamp should now be used in adjusting the final record timestamp. So when recording, then pausing, then resuming record the resulting video will correctly "jump ahead" and skip the time during the pause.

Changes

I implemented the Swift version (https://stackoverflow.com/q/61547473) of the original Objective-C sample (http://www.gdcl.co.uk/2013/02/20/iPhone-Pause.html) intended to correctly adjust the timestamp when pausing record.

Specifically the CameraView RecordVideo extension was modified to track the offset calculated when the record process is first paused. Some ivars were added to the CameraView class to help track this.

Tested on

• iPhone 14 Pro
• iPad Mini

Related issues

• Fixes 🐛 Video pausing does not work correctly on iOS #973
• Fixes ✨ Pausing and resuming record #562

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
react-native-vision-camera	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Mar 28, 2023 at 6:26PM (UTC)

[mrousavy]

Hey - this also seems rather complex. I think ideally AVAssetWriter should understand those timestamp diffs, maybe we should sync to the audio sessions master clock instead of the other way around?

[e-vural]

@mrousavy Hi. That problem still continue. Any solution do you suggest?

[mrousavy]

Can you please create a detailed issue report so I can take a look at that?

[e-vural]

Hi @mrousavy . Sorry for late answer.

1. Start Recording wait 5 seconds
2. Pause and wait 5 seconds
3. Resume Recording wait 5 seconds
4. Stop Recording

Normally video duration should be 10 seconds but its 15 seconds. Audio finished after 10 seconds but video still continue. I guess , if i press pause , video recording continue with paused frame view. So, when i pause and play
video and audio sync is broken.

[devadnaan]

Hi @mrousavy . Sorry for late answer.

1. Start Recording wait 5 seconds
2. Pause and wait 5 seconds
3. Resume Recording wait 5 seconds
4. Stop Recording

Normally video duration should be 10 seconds but its 15 seconds. Audio finished after 10 seconds but video still continue. I guess , if i press pause , video recording continue with paused frame view. So, when i pause and play video and audio sync is broken.

Yep same issue works fine in android but not in ios, @mrousavy can you please check ?

[mrousavy]

Hey all!

I just spent a few days on thinking about a battleproof timestamp synchronization solution, and I came up with a great idea.
I built a TrackTimeline helper class which represents a video or audio track - it can be started & stopped, paused & resumed, and even supports nesting pauses without issues.

• The total duration of the video is summed up from the difference between the first and the last actually written timestamps, minus the total duration of all pauses between a video. No more incorrect video.duration! 🥳
• Whereas before I just had a 4 second timeout if no frames arrive, I now just wait twice the frame latency (a few milliseconds) to ensure no frames are left out at maximum! 🎉
• A video can be stopped while it is paused without any issues, as a pause call is taken into consideration before stopping 💪
• A video file's session now exactly starts at the start() timestamp, and ends at the exact timestamp of the last video frame - this ensures there can never be any blank frames in the video, even if the audio track is longer 🤩

This was really complex to built as I had to synchronize timestamps between capture sessions, and the entire thing is a producer model - a video buffer can come like a second or so later than the audio buffer, but I need to make sure the video track starts before the audio track starts, and ends after the audio track ends - that's a huge brainf*ck! 🤯😅

There's also no helper APIs for this on iOS, and it looks like no other Camera framework (not even native Swift/ObjC iOS Camera libraries) support this - they all break when timestamps have a delay (e.g. video stabilization enabled) (or dont even support delays at all) ; so I had to build the thing myself.

Check out this PR and try if it fixes the issue for you; #2948

Thanks! ❤️