Add PipeWire camera feeds

[j20001970]

This PR uses PipeWire to provide camera feeds on Linux systems.

The benefits of using PipeWire for camera devices includes not having to use V4L2 kernel API directly, better integrated for desktop environments (camera indicator on KDE Plasma for example), and sandboxed camera access is also possible by using XDG desktop portal.

Camera feed driver on linuxbsd will be selected at runtime, and fallback to V4L2 driver if failed to initialize PipeWire.

TODO:

• Use XDG desktop portal to request camera permission when setting monitoring_feeds to true (supersede Support OS::request_permission("CAMERA") on linuxbsd using XDG Desktop Portal #100396)
• Use sowrap
• Select PipeWire or V4L2 driver at runtime
• Add support for planar YUV 420 format (Done but no hardware to test)
• Test with more camera feed demo projects
   Tested with the following shader:

shader_type canvas_item;

const mat3 yuv2rgb_bt709 = mat3(
	vec3(1.00000, 1.00000, 1.00000),
	vec3(0.00000, -0.21482, 2.12798),
	vec3(1.28033, -0.38059, 0.00000)
);

uniform sampler2D texture_yuy2;

void fragment() {
	vec2 size = TEXTURE_PIXEL_SIZE;
	vec2 UV_u = UV - floor(mod(UV / size, 2)) * vec2(1, 0) * size;
	vec2 UV_v = UV + (vec2(1, 0) - floor(mod(UV / size, 2))) * vec2(1, 0) * size;
	float y = texture(texture_yuy2, UV).r;
	float u = texture(texture_yuy2, UV_u).g;
	float v = texture(texture_yuy2, UV_v).g;
	vec3 rgb = yuv2rgb_bt709 * vec3(y, u - 0.5, v - 0.5);
	COLOR = vec4(rgb, 1.0);
}

 Tested with GDMP demo vision tasks without enabling camera feed addon.
 Tested with godot-camerafeed-demo.

[deralmas]

some directions on this one would be appreciated, I could not get dynload-wrapper working

I think I can give a hand :D

What issues are you having exactly?

[j20001970]

some directions on this one would be appreciated, I could not get dynload-wrapper working

I think I can give a hand :D

What issues are you having exactly?

I tried to generate sowrap from pipewire.h with the following command:

../dynload-wrapper/generate-wrapper.py --include ./thirdparty/linuxbsd_headers/pipewire/src/pipewire/pipewire.h --sys-include ./thirdparty/linuxbsd_headers/pipewire/src/pipewire/pipewire.h --include-dir ./thirdparty/linuxbsd_headers/pipewire/src --include-dir ./thirdparty/linuxbsd_headers/pipewire/spa/include --soname libpipewire-0.3.so.0 --init-name pipewire --output-header ./platform/linuxbsd/pipewre-so_wrap.h --output-implementation ./platform/linuxbsd/pipewire-so_wrap.c

But there are lots of parse errors, mostly coming from SPA headers like:

pycparser.plyparser.ParseError: ./thirdparty/linuxbsd_headers/pipewire/spa/include/spa/utils/hook.h:469:32: before: )

Since SPA is header-only, is there any ways to prevent dynload-wrapper from parsing SPA headers while PipeWire is using them? I tried both --omit-prefix and --ignore-headers options but that didn't work.

[deralmas]

Hi, I investigated the issue and I think I found a solution :D

We parse C code through pycparser, which by design does not support compiler extensions like __typeof__. Luckily, as recommended by the author, there's an addon called pycparserext. You'll have to install it with pip install pycparserext

In addition to that, the fake libc is missing the definition of locale_t.

I addressed those issues in a very rough patch to dynload-wrapper which you can try below. I haven't tried building the files but at a glance they seem correct.

Note that, according to --help, --sys-include is meant to be specified as if you were including something in the code, not a path (i.e. <pipewire/pipewire.h>).

I did this invocation:

../dynload-wrapper/generate-wrapper.py \
	--include-dir ./thirdparty/linuxbsd_headers/pipewire/src/ \
	--include-dir ./thirdparty/linuxbsd_headers/pipewire/spa/include/ \
	--include ./thirdparty/linuxbsd_headers/pipewire/src/pipewire/pipewire.h  \
	--soname libpipewire-0.3.so.0 \
	--init-name pipewire \
	--sys-include '<pipewire/pipewire.h>' \
	--output-header ./platform/linuxbsd/pipewire-so_wrap.h \
	--output-implementation ./platform/linuxbsd/pipewire-so_wrap.c

With this dynload-wrapper patch:

diff --git a/fake_libc_include/_fake_typedefs.h b/fake_libc_include/_fake_typedefs.h
index 3442dc1..031f33f 100644
--- a/fake_libc_include/_fake_typedefs.h
+++ b/fake_libc_include/_fake_typedefs.h
@@ -176,4 +176,6 @@ typedef struct xcb_connection_t xcb_connection_t;
 typedef uint32_t xcb_window_t;
 typedef uint32_t xcb_visualid_t;
 
+typedef struct locale_t locale_t;
+
 #endif
diff --git a/generate-wrapper.py b/generate-wrapper.py
index 62b6294..a122823 100755
--- a/generate-wrapper.py
+++ b/generate-wrapper.py
@@ -29,6 +29,7 @@ import textwrap
 from datetime import datetime
 
 try:
+    import pycparser
     from pycparser import c_parser, c_ast, parse_file, c_generator
     from pycparser.c_ast import Decl, FuncDecl, PtrDecl
 except:
@@ -36,6 +37,14 @@ except:
     print("Try installing it with pip install pycparser or using your distributions package manager.")
     sys.exit(1)
 
+try:
+    import pycparserext
+    from pycparserext import ext_c_parser, ext_c_generator
+    from pycparserext.ext_c_parser import FuncDeclExt
+except:
+    print("pycparserext not found.")
+    sys.exit(1)
+
 VERSION="0.7"
 URL="https://github.com/hpvb/dynload-wrapper"
 NOW=datetime.now().strftime("%Y-%m-%d %H:%M:%S")
@@ -43,7 +52,8 @@ PROGNAME=os.path.basename(sys.argv[0])
 FLAGS=""
 
 def stringify_declaration(t):
-    generator = c_generator.CGenerator()
+    #generator = c_generator.CGenerator()
+    generator = ext_c_generator.GnuCGenerator()
     return generator.visit(t)
 
 def get_name(t):
@@ -73,14 +83,17 @@ def parse_header(filename, omit_prefix, initname, ignore_headers = [], ignore_al
             cpp_args.append(include_dir)
 
     print(f"cpp_args: {cpp_args}")
-    ast = parse_file(filename, use_cpp=True, cpp_path='gcc', cpp_args=cpp_args)
+    text = pycparser.preprocess_file(filename, cpp_path='gcc', cpp_args=cpp_args)
+    #parser = c_parser.CParser()
+    parser = ext_c_parser.GnuCParser()
+    ast = parser.parse(text)
 
     functions = []
     sym_definitions = []
 
     for ext in ast.ext:
         if isinstance(ext, Decl):
-            if not isinstance(ext.type, FuncDecl):
+            if not isinstance(ext.type, FuncDeclExt):
                 continue
 
             skip = False

Again please excuse the crudity of this model. I didn't have time to build it to scale or paint polish it. ;)

Lastly, for some reason I get some debug lextab.py and yacctab.py files. I don't have time to figure out why so please remember to delete them before committing.

HTH :D

CC @hpvb (I'll make a PR soon™! Cleaned up of course)

[j20001970]

Hi, I investigated the issue and I think I found a solution :D

We parse C code through pycparser, which by design does not support compiler extensions like __typeof__. Luckily, as recommended by the author, there's an addon called pycparserext. You'll have to install it with pip install pycparserext

In addition to that, the fake libc is missing the definition of locale_t.

I addressed those issues in a very rough patch to dynload-wrapper which you can try below. I haven't tried building the files but at a glance they seem correct.

Note that, according to --help, --sys-include is meant to be specified as if you were including something in the code, not a path (i.e. <pipewire/pipewire.h>).

I did this invocation:

../dynload-wrapper/generate-wrapper.py \
	--include-dir ./thirdparty/linuxbsd_headers/pipewire/src/ \
	--include-dir ./thirdparty/linuxbsd_headers/pipewire/spa/include/ \
	--include ./thirdparty/linuxbsd_headers/pipewire/src/pipewire/pipewire.h  \
	--soname libpipewire-0.3.so.0 \
	--init-name pipewire \
	--sys-include '<pipewire/pipewire.h>' \
	--output-header ./platform/linuxbsd/pipewire-so_wrap.h \
	--output-implementation ./platform/linuxbsd/pipewire-so_wrap.c

With this dynload-wrapper patch:

diff --git a/fake_libc_include/_fake_typedefs.h b/fake_libc_include/_fake_typedefs.h
index 3442dc1..031f33f 100644
--- a/fake_libc_include/_fake_typedefs.h
+++ b/fake_libc_include/_fake_typedefs.h
@@ -176,4 +176,6 @@ typedef struct xcb_connection_t xcb_connection_t;
 typedef uint32_t xcb_window_t;
 typedef uint32_t xcb_visualid_t;
 
+typedef struct locale_t locale_t;
+
 #endif
diff --git a/generate-wrapper.py b/generate-wrapper.py
index 62b6294..a122823 100755
--- a/generate-wrapper.py
+++ b/generate-wrapper.py
@@ -29,6 +29,7 @@ import textwrap
 from datetime import datetime
 
 try:
+    import pycparser
     from pycparser import c_parser, c_ast, parse_file, c_generator
     from pycparser.c_ast import Decl, FuncDecl, PtrDecl
 except:
@@ -36,6 +37,14 @@ except:
     print("Try installing it with pip install pycparser or using your distributions package manager.")
     sys.exit(1)
 
+try:
+    import pycparserext
+    from pycparserext import ext_c_parser, ext_c_generator
+    from pycparserext.ext_c_parser import FuncDeclExt
+except:
+    print("pycparserext not found.")
+    sys.exit(1)
+
 VERSION="0.7"
 URL="https://github.com/hpvb/dynload-wrapper"
 NOW=datetime.now().strftime("%Y-%m-%d %H:%M:%S")
@@ -43,7 +52,8 @@ PROGNAME=os.path.basename(sys.argv[0])
 FLAGS=""
 
 def stringify_declaration(t):
-    generator = c_generator.CGenerator()
+    #generator = c_generator.CGenerator()
+    generator = ext_c_generator.GnuCGenerator()
     return generator.visit(t)
 
 def get_name(t):
@@ -73,14 +83,17 @@ def parse_header(filename, omit_prefix, initname, ignore_headers = [], ignore_al
             cpp_args.append(include_dir)
 
     print(f"cpp_args: {cpp_args}")
-    ast = parse_file(filename, use_cpp=True, cpp_path='gcc', cpp_args=cpp_args)
+    text = pycparser.preprocess_file(filename, cpp_path='gcc', cpp_args=cpp_args)
+    #parser = c_parser.CParser()
+    parser = ext_c_parser.GnuCParser()
+    ast = parser.parse(text)
 
     functions = []
     sym_definitions = []
 
     for ext in ast.ext:
         if isinstance(ext, Decl):
-            if not isinstance(ext.type, FuncDecl):
+            if not isinstance(ext.type, FuncDeclExt):
                 continue
 
             skip = False

Again please excuse the crudity of this model. I didn't have time to build it to scale or paint polish it. ;)

Lastly, for some reason I get some debug lextab.py and yacctab.py files. I don't have time to figure out why so please remember to delete them before committing.

HTH :D

CC @hpvb (I'll make a PR soon™! Cleaned up of course)

I managed to generate sowrap for pipewire with pycparseext pip package installed and dynload-wrapper patch applied, as well as workaround __int128 parse error by commenting out the <sys/mount.h> include in line 16 from pipewire/utils.h:
pycparser.plyparser.ParseError: /usr/include/linux/types.h:12:20: before: __int128

You saved my day and this PR, thank you very much!

[j20001970]

Please let me know if the commits need to be squashed.

[AThousandShips]

Please do squash them, we require single commits

[j20001970]

f7ee552f8908b1875db917097222d13ad5815374 removed unused proxy callback.

[j20001970]

CI now reports -Werror=missing-field-initializers on PipeWire SPA headers, is there any solutions to this?
Edit: Nvm, solved the problem by appending -Wno-error=missing-field-initializers flag to camera source.

[shiena]

@j20001970
I tested the PipeWire camera module. The PipeWire package in Ubuntu 24.04.3 is version 1.0.5, while the PipeWire version added in this pull request is 1.2.6, so it doesn't work. Therefore, it would be safer to add a version check like the following:

diff --git a/modules/camera/register_types.cpp b/modules/camera/register_types.cpp
index f598fa4226..aaeac49cd9 100644
--- a/modules/camera/register_types.cpp
+++ b/modules/camera/register_types.cpp
@@ -60,8 +60,15 @@ void initialize_camera_module(ModuleInitializationLevel p_level) {
        int dylibloader_verbose = 0;
 #endif // defined(DEBUG_ENABLED)
        if (initialize_pipewire(dylibloader_verbose) == 0) {
-               print_verbose("CameraServer: Using PipeWire driver.");
-               CameraServer::make_default<CameraPipeWire>();
+               // pw_check_library_version is available since 0.3.75.
+               // If the function is not available or version is too old, fall back to V4L2.
+               if (pw_check_library_version_dylibloader_wrapper_pipewire && pw_check_library_version(PW_MAJOR, PW_MINOR, PW_MICRO)) {
+                       print_verbose("CameraServer: Using PipeWire driver.");
+                       CameraServer::make_default<CameraPipeWire>();
+               } else {
+                       print_verbose(vformat("CameraServer: PipeWire version too old (%s), falling back to V4L2 driver.", pw_get_library_version()));
+                       CameraServer::make_default<CameraLinux>();
+               }
        } else {
                print_verbose("CameraServer: Using V4L2 driver.");
                CameraServer::make_default<CameraLinux>();

[j20001970]

What's changed in bc2cfd0432a8b9514c170b103409974001cb4791:

• Adapt FreeDesktopPortalDesktop changes introduced in 1a3a254
• Add PipeWire version check as suggested by @shiena
• Change PipeWire headers version from 1.2.6 to 1.0.0

[j20001970]

5c2a132aadc09a2a2901ec149b0ec402f2855564 added MJPEG format support. Please note that MJPEG format might not work as expected at the moment, since the stream from webcams might contain extraneous bytes and jpeg_turbo_load_image_from_buffer consider such warning as error (this might be fixed by #113359).

And some of the code has changed since last approval, should I re-request the review?

[j20001970]

47e4b48c718f7066e37e311aed60baa8964dc9d6 added PW_KEY_APP_NAME property when connecting PipeWire instance to display user-friendly application name using the webcam.

[j20001970]

e633f2d1453dc1a7fe0d20560c86d93beab01bfd fixed pw_context_connect_fd crash when using XDG camera portal, and changed DBus timeout from infinite to default in relevant calls.

EDIT: Accidentely changed timeout in FreeDesktopPortalDesktop::send_request as well, fixed in a0e9a182b80de91375451fa7ba79c97aadb6dce3

[Calinou]

I did one more test on Fedora 43 with an Elgato Facecam 4K camera in V4L2 mode:

master	This PR

MJPEG, YUY2 and NV12 formats all display correctly in their highest resolutions (2160p, 1440p and 2160p respectively). Framerate is pretty smooth, although 4K60 doesn't seem to be available (this webcam is able to do it, but it might be Windows-only). I made sure to test under good lighting conditions.

Some camera indicators can get stuck on "Creating" in KDE's applet if you exit the app while switching formats:

Also, running in --verbose mode shows the camera being connected, disconnecting then reconnecting immediately after:

CameraServer: Registered camera Elgato Facecam 4K (V4L2) with ID 1 and position 0 at index 0
CameraServer: Removed camera Elgato Facecam 4K (V4L2) with ID 1 and position 0
CameraServer: Registered camera Elgato Facecam 4K (V4L2) with ID 1 and position 0 at index 0

The camera indicator doesn't show up in master, which means PipeWire is being used even if the camera name ends with a (V4L2) suffix. I also don't see a warning about missing PipeWire development libraries when compiling.

[j20001970]

The PR currently has a bug that the camera feeds will fail to start streaming on systems using PipeWire 1.6.1 (maybe 1.6.0) onward. The stream from PipeWire side will fail with error message like error use input buffers: -22 (invalid argument). Tested on Arch running KDE (desktop) and GNOME (laptop) desktop environment.

Strangely, PipeWire tutorial 5 from official documentation also seems stopped working (the program does not report captured frame byte size), while some webcam capture applications using PipeWire like OBS still work as expected. So it is likely that there are still something the PR does not handle well. Please do not merge this until the issue is solved.

[j20001970]

The PR currently has a bug that the camera feeds will fail to start streaming on systems using PipeWire 1.6.1 (maybe 1.6.0)...

d7c64a8 have fixed aforementioned issue by specifying buffer datatype when connecting the stream. The PR should now work on newer PipeWire version. on_stream_state_changed callback is also added to notify stream error.

Now to respond the review feedback from @Calinou (thanks for the review!):

Also, running in --verbose mode shows the camera being connected, disconnecting then reconnecting immediately after:

When CameraPipeWire::set_monitoring_feeds is set to false, all feeds created by CameraPipeWire will be removed from the server. And in godot-camerafeed-demo, reloading camera list will first stop monitoring feeds then start monitoring again. So the behaviour is kind of expected with the following order:

1. Registering feeds by calling CameraServer.get_singleton()->set_monitoring_feeds(true) from CameraTexture constructor.
2. Removing feeds by setting CameraServer.monitoring_feeds to false in _reload_camera_list
3. Registering feeds again by setting CameraServer.monitoring_feeds to true in _reload_camera_list

Some camera indicators can get stuck on "Creating" in KDE's applet if you exit the app while switching formats:

I can't seem to reproduce the problem, does that mean exiting app need to happen after a new format is selected, as well as before the feed start providing frames with the new format?

Framerate is pretty smooth, although 4K60 doesn't seem to be available

I'm not sure if Windows could provide webcam resolution higher than the specs. I took a look at the Elgato website, it states the video resolution up to 2160p60 (MJPG only) when connected by USB3.0. So both the master branch and this PR should support formats listed in the specs by the look of the screenshots.