ImageSegmenter not working on Firefox due to WebGL format incompatibility

[lebaudantoine]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Any

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

135.0.1

Programming Language and version

Typescript

MediaPipe version

0.10.9

Bazel version

No response

Solution

SelfieSegmenter

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

The MediaPipe ImageSegmenter task doesn't work on Firefox browsers using delegate 'GPU'

Describe the expected behaviour

The delegate 'GPU' should be supported on Firefox browsers

Standalone code/steps you may have used to try to get what you need

Will describe it in comment.

Other info / Complete Logs

WebGL warning: readPixels: Format and type RED/FLOAT incompatible with this R32F attachment. This framebuffer requires either RGBA/FLOAT or getParameter(IMPLEMENTATION_COLOR_READ_FORMAT/_TYPE) RGBA/FLOAT. 29\
After reporting 32, no further warnings will be reported for this WebGL context.

[lebaudantoine]

Bug

The MediaPipe ImageSegmenter task doesn't work properly on Firefox browsers. Several issues are reporting it. #4501
When attempting to use it, Firefox shows the following WebGL warning:

WebGL warning: readPixels: Format and type RED/FLOAT incompatible with this R32F attachment. This framebuffer requires either RGBA/FLOAT or getParameter(IMPLEMENTATION_COLOR_READ_FORMAT/_TYPE) RGBA/FLOAT. 29\
After reporting 32, no further warnings will be reported for this WebGL context.

This causes the GPU delegate to fail on Firefox, forcing fallback to CPU inference which results in poor performance with video streams.

Root Cause

The issue occurs in the convertToFloat32Array() method in mask.ts (line 288)(

mediapipe/mediapipe/tasks/web/vision/core/mask.ts

Line 288 in eecf8a5

private convertToFloat32Array(): Float32Array {

).

For most browsers, the code uses gl.RED, gl.FLOAT format when reading pixels:

gl.readPixels(0, 0, this.width, this.height, gl.RED, gl.FLOAT, float32Array);

But Firefox's WebGL implementation doesn't support this format combination with R32F attachments. It requires using gl.RGBA, gl.FLOAT instead, as indicated by the warning message.

Observation

The code already has a special case for iOS that uses the RGBA format and then extracts manually the red channel:

if (isIOS()) {
  // WebKit on iOS only supports gl.HALF_FLOAT for single channel reads...
  const outputArray = new Float32Array(this.width * this.height * 4);
  gl.readPixels(
      0, 0, this.width, this.height, gl.RGBA, gl.FLOAT, outputArray);
  for (let i = 0, j = 0; i < float32Array.length; ++i, j += 4) {
    float32Array[i] = outputArray[j];
  }
}

Proposed Solution

Add a Firefox detection similar to the iOS detection, and use the same RGBA workaround for Firefox browsers. I've tested this approach by monkey-patching the function locally, and it resolves the issue completely, allowing the GPU delegate to work properly on Firefox.

Suggested implementation:

1. Add an isFirefox() detection function in [platform_utils.ts](https://github.com/google-ai-edge/mediapipe/blob/eecf8a5696d5e47edbd3b6e99a6064c99f9ab1a4/mediapipe/web/graph_runner/platform_utils.ts)
2. Modify the condition in convertToFloat32Array() to check for both iOS and Firefox: if (isIOS() || isFirefox())

Expected Benefits

This change would make MediaPipe ImageSegmenter usable with GPU acceleration on Firefox, significantly improving performance for Firefox users without requiring any changes to application code.