Add CameraFeed support for Web

[shiena]

fixed: godotengine/godot-proposals#12493

Current Limitation:
The platform/web/js/libs/library_godot_camera.js library includes certain functionalities that are inherently asynchronous. We are currently investigating how to execute these functions synchronously, as this is a requirement for our project, but haven't yet found a solution.
I was able to build with JSPI and call asynchronous functions synchronously in Chrome. However, JSPI only works in Chrome and Edge 137.
https://webassembly.org/features/
https://learn.microsoft.com/en-us/microsoft-edge/web-platform/release-notes/137#javascript-promise-integration-jspi-in-webassembly

Supported browsers:

• Chrome v137.0.7151.69
• Firefox ESR v128.11.0esr
• Safari v18.5(20621.2.5.11.8)

[shiena]

I've tried enabling Asyncify as follows, which I believe should fix it, but the build is still failing.

diff --git a/platform/web/SCsub b/platform/web/SCsub
index cf872e65ff..5bb105e1cd 100644
--- a/platform/web/SCsub
+++ b/platform/web/SCsub
@@ -106,6 +106,8 @@ else:
     sys_env.Append(LIBS=["idbfs.js"])
     build = sys_env.add_program(build_targets, web_files + ["web_runtime.cpp"])
 
+sys_env.Append(LINKFLAGS=["-sASYNCIFY=1"])
+
 sys_env.Depends(build[0], sys_env["JS_LIBS"])
 sys_env.Depends(build[0], sys_env["JS_PRE"])
 sys_env.Depends(build[0], sys_env["JS_POST"])

cache:INFO: generating system asset: symbol_lists/8eab473934c10451394fc82e8912c6b3712c9ff9.json... (this will be cached in "C:\Users\shien\dev\emsdk\upstream\emscripten\cache\symbol_lists\8eab473934c10451394fc82e8912c6b3712c9ff9.json" for subsequent builds)
cache:INFO:  - ok
unexpected expression type
UNREACHABLE executed at C:\b\s\w\ir\cache\builder\emscripten-releases\binaryen\src\passes\Asyncify.cpp:1142!
em++: error: 'C:/Users/shien/dev/emsdk/upstream\bin\wasm-opt --strip-target-features --post-emscripten -Os --low-memory-unused --asyncify --pass-arg=asyncify-propagate-addlist --pass-arg=asyncify-imports@env.invoke_*,env.__asyncjs__*,*.fd_sync,*.emscripten_promise_await,*.emscripten_idb_load,*.emscripten_idb_store,*.emscripten_idb_delete,*.emscripten_idb_exists,*.emscripten_idb_clear,*.emscripten_idb_load_blob,*.emscripten_idb_store_blob,*.emscripten_sleep,*.emscripten_wget_data,*.emscripten_scan_registers,*.emscripten_lazy_load_code,*._load_secondary_module,*.emscripten_fiber_swap,*.SDL_Delay --zero-filled-memory --pass-arg=directize-initial-contents-immutable --no-stack-ir bin\godot.web.template_debug.wasm32.nothreads.wasm -o bin\godot.web.template_debug.wasm32.nothreads.wasm -g --mvp-features --enable-bulk-memory --enable-exception-handling --enable-multivalue --enable-mutable-globals --enable-reference-types --enable-sign-ext --enable-simd' failed (returned 3221226505)
scons: *** [bin\godot.web.template_debug.wasm32.nothreads.js] Error 1

[Calinou]

Asyncify makes binaries a lot larger, so we need to be able to use it selectively only for functions that need it.

cc @adamscott

[shiena]

Ready for review.
The demo site requires Chrome 137+ due to its use of JSPI.
I wanted to use ASYNCIFY=1 in SCsub but ran into errors, so I'm currently using ASYNCIFY=2 for JSPI.

Demo site: https://shiena.github.io/godot-camerafeed-demo/
Demo source code: https://github.com/shiena/godot-camerafeed-demo

[Faless]

The demo site requires Chrome 137+ due to its use of JSPI.
I wanted to use ASYNCIFY=1 in SCsub but ran into errors, so I'm currently using ASYNCIFY=2 for JSPI.

This means it can't be merged, we at least want support the latest version of Chrome, Firefox and iOS (and we also try to support the latest version of Firefox ESR, but that has been on and off in the past).

That said, it should be possible to implement the API by relying on callbacks instead of requiring asyncify.

[Faless]

We shouldn't use the EM_*_JS functions, as they break dynamic linking.

[shiena]

Sorry for the late reply! This has already been addressed — EM_ASYNC_JS has been replaced with extern declarations in godot_camera.h + library_godot_camera.js, following the same pattern as the other platform libraries.

[shiena]

That said, it should be possible to implement the API by relying on callbacks instead of requiring asyncify.

I'm trying to get a list of CameraFeed objects using CameraServer.feeds() after setting CameraServer.monitoring_feeds = true in GDScript. Once I have a CameraFeed, I then want to retrieve its available formats using CameraFeed.formats.

However, both of these methods return a JavaScript Promise, which makes using callbacks inherently slow.
If there's an alternative approach, I'd be happy to try it out immediately.

[shiena]

@Faless
Removed Asyncify dependency and confirmed functionality in the following browsers.

• Chrome v137.0.7151.69
• Firefox ESR v128.11.0esr
• Safari v18.5(20621.2.5.11.8)

However, please note the following limitations:

• Retrieving the list of camera devices with CameraServer.monitoring_feeds = true is an asynchronous operation, so it requires a few frames until the state becomes true.
  https://github.com/shiena/godot-camerafeed-demo/blob/main/scripts/Main.gd#L51-L56
• Firefox ESR does not support MediaStreamTrack.getCapabilities(), which means CameraFeed.formats and related APIs will not function.

[Benjamin-Dobell]

Not a maintainer, but I think there's a few things that will need to be changed. That said, I have tested this and it does indeed work as is, and has unblocked me in the interim, so thank you very much! 🙏

[Benjamin-Dobell]

This needs to be swapped out for:

driver = memnew(CameraDriverWeb);

Otherwise, we get errors like:

[Browser Error] memory access out of bounds
  RuntimeError: memory access out of bounds
      at $dlfree (wasm://wasm/godot.web.template_debug.wasm32.dlink.wasm-0220781e:1:886272)
      at $Memory::free_static(void*, bool) (wasm://wasm/godot.side.web.template_debug.wasm32.dlink.wasm-88ec8f42:1:55343982)
      at $CameraWeb::_cleanup() (wasm://wasm/godot.side.web.template_debug.wasm32.dlink.wasm-88ec8f42:1:9515425)
      at $CameraWeb::~CameraWeb() (wasm://wasm/godot.side.web.template_debug.wasm32.dlink.wasm-88ec8f42:1:9515608)
      at $Main::cleanup(bool) (wasm://wasm/godot.side.web.template_debug.wasm32.dlink.wasm-88ec8f42:1:3893745)

[shiena]

Fixed in 833ccaf38df7fbea00e022ae0c56eaf0d18434fe.

[Benjamin-Dobell]

I suspect this will be a blocker to getting this merged. The other camera implementations make use of a fixed number of buffers. Allocating a new buffer every frame will lead to significant GC churn on the main thread.

Admittedly, making this work nicely with transferable objects is probably a bit more complex than just maintaining bunch of shared buffers shared by threads with locking.

It think the best way to do it would be something like maintain 4 buffers. When a frame is copied into a buffer, we post/transfer it to the main thread as long as we have at least 1 buffer left remaining in the worker after the transfer. Otherwise, we don't post the frame (yet).

The main thread receives buffers similar to now, and always keeps minimum 1 buffer on its side as well (except for when the first image is yet to be received).

When the main thread/context has:

• More than 1 buffer on its side.
• Already processed the head buffer (frame callback).

then is posts the head buffer back to the worker.

However, this still leads to GC as wrapper references to those transferred buffers are allocated as the buffers ping pong back and forth. The peak memory usage is constrained though, so that's a pretty significant win on its lonesome.

The ideal implementation would detect availability of SharedArrayBuffer and when available use it in conjunction with Atomics.

[shiena]

Addressed in 40a3f5e46c43fbac2c575a09ac8af59b9a2475cd. Three changes to reduce per-frame GC pressure:

• Introduced a transferable buffer pool (4 pre-allocated ArrayBuffers) in the worker, ping-ponged back from the main thread after use.
• Cached the Wasm heap pointer for frame data copy to avoid per-frame malloc/free.
• Cached facingMode at stream start instead of calling getVideoTracks() + getSettings() every frame.

With these changes, the JS heap stays flat over 60 seconds at 1024×768 @ 30fps. The SharedArrayBuffer upgrade can be explored as a follow-up.

[adamscott]

I'm gonna take a look at this PR today or tomorrow.

[Benjamin-Dobell]

@shiena Thanks for making those changes. I'd made some similar changes locally but adopted yours about 24 hours ago. Working great 🙏

[Benjamin-Dobell]

❤️

[shiena]

A note on the camera frame capture strategy. Browser support for the APIs we'd like to use for pulling pixels out of a getUserMedia() stream is uneven, so this PR implements several capture paths and picks the best one available at runtime.

In order of preference:

1. WebCodecs in a Worker — MediaStreamTrackProcessor produces VideoFrames, which are transferred to a dedicated Worker that extracts RGBA pixels via VideoFrame.copyTo(). No canvas, no main-thread pixel work. Used on modern Chromium.

2. Canvas 2D in a Worker — createImageBitmap() on the <video> element produces a transferable bitmap; the Worker draws it to an OffscreenCanvas and reads pixels with getImageData(). Used when Workers are available but WebCodecs isn't (e.g. current Firefox).

3. WebCodecs on the main thread — same VideoFrame.copyTo() pipeline as (1), but without offloading. Used when WebCodecs exists but Workers / OffscreenCanvas / createImageBitmap don't.

4. Canvas 2D on the main thread — classic drawImage(video) + getImageData() driven by requestAnimationFrame. Last resort for browsers that lack both of the above (e.g. Safari without WebCodecs).

The tier is chosen once by feature detection when the stream starts. On top of that, each tier also falls back to a lower tier at runtime if its API unexpectedly throws (e.g. VideoFrame.copyTo() can fail even when the API is present), so a misdetected environment degrades gracefully instead of breaking the feed.