GPU support for macOS Metal?

[yiyinglai]

OS Platform and Distribution

macOS Somona 14.6

Compiler version

clang 15.0.0

Programming Language and version

C++(demo code)

Installed using virtualenv? pip? Conda?(if python)

No response

MediaPipe version

No response

Bazel version

No response

XCode and Tulsi versions (if iOS)

No response

Android SDK and NDK versions (if android)

No response

Android AAR (if android)

None

OpenCV version (if running on desktop)

No response

Describe the problem

no member named 'GetCVPixelBufferRef' in namespace 'mediapipe' when building desktop hand tracking gpu

Complete Logs

Following this guide https://ai.google.dev/edge/mediapipe/framework/getting_started/cpp, I can run desktop hand tracking cpu with no issue, but desktop hand tracking gpu could not be compiled, logs:

(base) yiyinglai@MacBook-Air mediapipe % bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
  mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu
WARNING: /Users/yiyinglai/workspace/github/mediapipe/mediapipe/framework/BUILD:69:24: in cc_library rule //mediapipe/framework:calculator_cc_proto: target '//mediapipe/framework:calculator_cc_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /Users/yiyinglai/workspace/github/mediapipe/mediapipe/framework/tool/BUILD:204:24: in cc_library rule //mediapipe/framework/tool:field_data_cc_proto: target '//mediapipe/framework/tool:field_data_cc_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
INFO: Analyzed target //mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: From Linking mediapipe/modules/palm_detection/palm_detection_model_loader_graph_text_to_binary_graph [for tool]:
ld: warning: ignoring duplicate libraries: '-lc++', '-lm', '-lpthread'
INFO: From Linking mediapipe/modules/hand_landmark/hand_landmark_tracking_gpu_graph_text_to_binary_graph [for tool]:
ld: warning: ignoring duplicate libraries: '-lc++', '-lm', '-lpthread'
INFO: From Linking mediapipe/modules/hand_landmark/palm_detection_detection_to_roi_graph_text_to_binary_graph [for tool]:
ld: warning: ignoring duplicate libraries: '-lc++', '-lm', '-lpthread'
INFO: From Linking mediapipe/graphs/hand_tracking/subgraphs/hand_renderer_gpu_graph_text_to_binary_graph [for tool]:
ld: warning: ignoring duplicate libraries: '-lc++', '-lm', '-lpthread'
INFO: From Linking mediapipe/modules/palm_detection/palm_detection_gpu_graph_text_to_binary_graph [for tool]:
ld: warning: ignoring duplicate libraries: '-lc++', '-lm', '-lpthread'
INFO: From Linking mediapipe/modules/hand_landmark/hand_landmark_model_loader_graph_text_to_binary_graph [for tool]:
ld: warning: ignoring duplicate libraries: '-lc++', '-lm', '-lpthread'
INFO: From Linking mediapipe/modules/hand_landmark/hand_landmark_gpu_graph_text_to_binary_graph [for tool]:
ld: warning: ignoring duplicate libraries: '-lc++', '-lm', '-lpthread'
INFO: From Linking mediapipe/modules/hand_landmark/hand_landmark_landmarks_to_roi_graph_text_to_binary_graph [for tool]:
ld: warning: ignoring duplicate libraries: '-lc++', '-lm', '-lpthread'
ERROR: /Users/yiyinglai/workspace/github/mediapipe/mediapipe/gpu/BUILD:945:11: Compiling mediapipe/gpu/MPPMetalHelper.cc failed: (Exit 1): wrapped_clang_pp failed: error executing command (from target //mediapipe/gpu:MPPMetalHelper) external/local_config_cc/wrapped_clang_pp '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG ... (remaining 78 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
mediapipe/gpu/MPPMetalHelper.cc:140:46: error: no member named 'GetCVPixelBufferRef' in namespace 'mediapipe'
  CVPixelBufferRef pixel_buffer = mediapipe::GetCVPixelBufferRef(gpuBuffer);
                                  ~~~~~~~~~~~^
mediapipe/gpu/MPPMetalHelper.cc:188:18: error: no member named 'GetCVPixelBufferRef' in namespace 'mediapipe'
      mediapipe::GetCVPixelBufferRef(gpuBuffer), NULL, metalPixelFormat, width,
      ~~~~~~~~~~~^
2 errors generated.
Error in child process '/usr/bin/xcrun'. 1
Target //mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 356.539s, Critical Path: 353.42s
INFO: 680 processes: 129 internal, 551 darwin-sandbox.
FAILED: Build did NOT complete successfully

[yiyinglai]

Figured out how to build it, you need to add --copt -DMEDIAPIPE_GPU_BUFFER_USE_CV_PIXEL_BUFFER so that GetCVPixelBufferRef is implemented for gpu (https://ai.google.dev/edge/mediapipe/framework/getting_started/cpp documentation needs update)

bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 --copt -DMEDIAPIPE_GPU_BUFFER_USE_CV_PIXEL_BUFFER \
  mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu

However, it didn't run successfully

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_gpu.pbtxt

(base) yiyinglai@MacBook-Air mediapipe % GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \                                  
  --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_gpu.pbtxt
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1727542503.354913 8438947 demo_run_graph_main_gpu.cc:54] Get calculator graph config contents: # MediaPipe graph that performs multi-hand tracking with TensorFlow Lite on GPU.
# Used in the examples in
# mediapipe/examples/android/src/java/com/mediapipe/apps/handtrackinggpu.

# GPU image. (GpuBuffer)
input_stream: "input_video"

# GPU image. (GpuBuffer)
output_stream: "output_video"
# Collection of detected/predicted hands, each represented as a list of
# landmarks. (std::vector<NormalizedLandmarkList>)
output_stream: "hand_landmarks"

# Generates side packet cotaining max number of hands to detect/track.
node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:num_hands"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 2 }
    }
  }
}

# Detects/tracks hand landmarks.
node {
  calculator: "HandLandmarkTrackingGpu"
  input_stream: "IMAGE:input_video"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:hand_landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:hand_rects_from_landmarks"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:hand_rects_from_palm_detections"
}

# Subgraph that renders annotations and overlays them on top of the input
# images (see hand_renderer_gpu.pbtxt).
node {
  calculator: "HandRendererSubgraph"
  input_stream: "IMAGE:input_video"
  input_stream: "DETECTIONS:palm_detections"
  input_stream: "LANDMARKS:hand_landmarks"
  input_stream: "HANDEDNESS:handedness"
  input_stream: "NORM_RECTS:0:hand_rects_from_palm_detections"
  input_stream: "NORM_RECTS:1:hand_rects_from_landmarks"
  output_stream: "IMAGE:output_video"
}

I0000 00:00:1727542503.357008 8438947 demo_run_graph_main_gpu.cc:60] Initialize the calculator graph.
I0000 00:00:1727542503.363917 8438947 demo_run_graph_main_gpu.cc:64] Initialize the GPU.
I0000 00:00:1727542503.714582 8438947 gl_context.cc:357] GL version: 2.1 (2.1 Metal - 88.1), renderer: Apple M2
I0000 00:00:1727542503.714864 8438947 demo_run_graph_main_gpu.cc:70] Initialize the camera or load the video.
2024-09-29 00:55:04.730 hand_tracking_gpu[60945:8438947] WARNING: AVCaptureDeviceTypeExternal is deprecated for Continuity Cameras. Please use AVCaptureDeviceTypeContinuityCamera and add NSCameraUseContinuityCameraDeviceType to your Info.plist.
I0000 00:00:1727542507.677030 8438947 demo_run_graph_main_gpu.cc:91] Start running the calculator graph.
I0000 00:00:1727542507.677950 8438947 demo_run_graph_main_gpu.cc:96] Start grabbing and processing frames.
INFO: Created TensorFlow Lite delegate for Metal.
E0000 00:00:1727542513.340863 8439035 shader_util.cc:99] Failed to compile shader:
 1 #version 330 
 2 #ifdef GL_ES 
 3 #define DEFAULT_PRECISION(p, t) precision p t; 
 4 #else 
 5 #define DEFAULT_PRECISION(p, t) 
 6 #define lowp 
 7 #define mediump 
 8 #define highp 
 9 #endif  // defined(GL_ES) 
10 #if __VERSION__ < 130
11 #define in attribute
12 #define out varying
13 #endif  // __VERSION__ < 130
14 in vec4 position; in mediump vec4 texture_coordinate; out mediump vec2 sample_coordinate; void main() { gl_Position = position; sample_coordinate = texture_coordinate.xy; }
E0000 00:00:1727542513.340918 8439035 shader_util.cc:106] Error message: ERROR: 0:1: '' :  version '330' is not supported

I0000 00:00:1727542513.341694 8438947 demo_run_graph_main_gpu.cc:188] Shutting down.
E0000 00:00:1727542513.381401 8438947 demo_run_graph_main_gpu.cc:199] Failed to run the graph: CalculatorGraph::Run() failed: 
Calculator::Process() for node "handrenderersubgraph__AnnotationOverlayCalculator" failed: ; RET_CHECK failure (mediapipe/calculators/util/annotation_overlay_calculator.cc:732) program_Problem initializing the program.

[TravisLiao]

I have the same issue, are there any new updates? @yiyinglai @kuaashish