video capture causes GStreamer-CRITICAL error on Apple Silicon (M1)

[SableRaf]

Description

Attempting to initialize camera capture on Apple Silicon causes a critical error in GStreamer. On an Intel Mac, the same issue does not happen. Note: this issue has been mentioned on the forum here.

Expected Behavior

The camera capture should be initiated without errors.

Current Behavior

A critical error is raised and the camera capture can not be initiated. See below:

Processing video library using bundled GStreamer 1.20.3
Scanning GStreamer plugins... Done.
Available cameras:
FaceTime HD Camera

(Processing core video:4365): GStreamer-CRITICAL **: 13:31:26.506: gst_bin_add_many: assertion 'GST_IS_ELEMENT (element_1)' failed

(Processing core video:4365): GStreamer-CRITICAL **: 13:31:26.513: gst_element_link_pads_full: assertion 'GST_IS_ELEMENT (dest)' failed

Steps to Reproduce

1. Use an M1 MacBook Pro
2. Try running the capture example from the reference
3. See errors in the console

Below is a stripped down version of the example that still triggers the error. The error doesn't happen if you comment out cam = new Capture(this, cameras[0]);

import processing.video.*;

Capture cam;

void setup() {
  size(640, 480);

  String[] cameras = Capture.list();
  
  if (cameras.length == 0) {
    println("There are no cameras available for capture.");
    exit();
  } else {
    println("Available cameras:");
    for (int i = 0; i < cameras.length; i++) {
      println(cameras[i]);
    }

    cam = new Capture(this, cameras[0]);
    //cam.start();     
  }      
}

Your Environment

• Processing version: 4.0.1
• Operating System and OS version: MacOS 12.5.1 Monterey
• Other info: M1 Max CPU

Possible Causes / Solutions

Unknown

[brad-miller]

until video library is update to 2.2 release (assuming your using 2.2.1)
try this
cam = new Capture(this, 640, 480, "pipeline:avfvideosrc device-index=0 ! video/x-raw, width=640, height=480, framerate=30/1")

[SableRaf]

Thanks for the tip! In a post on the Processing forum, @neilcsmith-net suggested the following for cross platform compatibility. It worked for me on macOS.

cam = new Capture(this, 640, 480, "pipeline:autovideosrc ! video/x-raw, width=640, height=480, framerate=30/1");

[SableRaf]

The following works too (also suggested by @neilcsmith-net)

cam = new Capture(this, 640, 480, "pipeline:autovideosrc");

Note: this one needs some tweaks to avoid the image getting stretched (see below). I'm not sure how to do that so if you have a solution, let me know. Then we can just fix the code examples.

[neilcsmith-net]

@SableRaf actually, looking at this that's already the default https://github.com/processing/processing-video/blob/master/src/processing/video/Capture.java#L496

Should just leave the whole Capture.list() bit out of examples IMO. Underlying code obviously has issues on some platforms. Not sure whether that's in GStreamer, our library, or the Video library. Will try and do some testing of device lookup.

For the stretching, this is likely due to missing pixel-aspect-ratio in the derived Caps string at https://github.com/processing/processing-video/blob/master/src/processing/video/Capture.java#L460 as opposed to the alternative code at https://github.com/praxis-live/praxiscore/blob/master/praxiscore-video-gstreamer/src/main/java/org/praxislive/video/gstreamer/components/PImageSink.java#L49 This is needed if the dimensions are set after the videoscale element.

cc @codeanticode

[jjwilliams01]

I am using a variety of UVC camera models. Over the last few releases, they have all stopped working with the GettingStartedCapture example. They will work with the cam = new Capture(this, 640, 480, "pipeline:autovideosrc"); as long as the camera is the only one (Mac Mini). (Thanks SableRaf!).

The error I am receiving consistently is BaseSrc: [avfvideosrc0] : Internal data stream error. I assume this is a pipeline format error, but have not seen documentation on how to change it.

The camera modules I am using are UVC compliant, are from about a dozen different manufacturers and are recognized by normal Mac apps. They only fail in Processing. The problem is identical on Mac Silicon and Mac Intel. I have also tested in Sonoma, Ventura and Monterey.

Advice anyone? I am (respectfully) calling this a bug since it fails with the included Processing video capture example, and I am desperate for a workaround.

[joshmiller2010]

Not sure if this is of any help, but I had the same issue with, and no previous fixes worked. In the past I've been able to get around this error with OBS Virtual Camera, but no longer. But the app Camera Graph (in the mac os app store) works similarly to OBS and I was able to share the camera through there and connect to the virtual camera in Processing. For me the pipeline:autovideosrc didn't work either with Sonoma and Processing 4.3 (intel or silicon version).

[PeteHaughie]

The following works too (also suggested by @neilcsmith-net)

cam = new Capture(this, 640, 480, "pipeline:autovideosrc");

Note: this one needs some tweaks to avoid the image getting stretched (see below). I'm not sure how to do that so if you have a solution, let me know. Then we can just fix the code examples.

Hi Raph, this issue has reared its ugly head again in MacOS 14.5 and I found that the only thing that actually worked was cam = new Capture(this, 640, 480, "pipeline:autovideosrc");

[ostermuellerj]

More context for this solution here (how to select inputs, etc)
https://wp.nyu.edu/shanghai-ima-interaction-lab/how-to-capture-camera-in-catalina/