Google Meets Background Segmentation Model?

[kirawi]

tensorflow/tfjs#4177

Would it be possible to get a .pb of the raw tflite model listed there, assuming it truly is Apache licensed?

[PINTO0309]

I'm just going to do a reverse conversion like the other models in MediaPipe, so I can probably convert it to .pb.
google-ai-edge/mediapipe#245

I'll try it tonight.

[simon-lanf]

@PINTO0309 that would be a real christmas miracle.

[PINTO0309]

I decided to convert the model while optimizing it, so it will take some time.

[simon-lanf]

Take your time :)

[PINTO0309]

I have generated and committed models for .pb, .tflite float32/float16, INT8, EdgeTPU, TFJS, TF-TRT, CoreML, and OpenVINO IR for testing. However, I was so exhausted that I did not create a test program to test it. I would be very happy if you could test it with your help. 😃
https://github.com/PINTO0309/PINTO_model_zoo/tree/master/082_MediaPipe_Meet_Segmentation

[ldenoue]

I didn’t see the CoreML model: is it somewhere?

[PINTO0309]

@ldenoue
https://github.com/PINTO0309/PINTO_model_zoo/blob/main/082_MediaPipe_Meet_Segmentation/download_full_144x256.sh

[ldenoue]

@PINTO0309 thanks, I saw it now! It works as is, but I am not able to convert my images to the required input.

So I've tried using coremltools to change the input from float32[1,144,256,3] to an RGB image:

import coremltools.proto.FeatureTypes_pb2 as ft 
spec = coremltools.utils.load_spec("model_coreml_float32.mlmodel")
input = spec.description.input[0]
input.type.imageType.colorSpace = ft.ImageFeatureType.RGB
input.type.imageType.height = 144 
input.type.imageType.width = 256
coremltools.utils.save_spec(spec, "model_coreml_float32_rgb_input.mlmodel")

Then I get a compilation error in XCode coremlc: error: compiler error: Espresso exception: "Invalid blob shape": Cannot broadcast blobs

Any idea how to fix this?
Or how could I transform my input image (UIImage or CGImage) into an MLMultiArray required by the mlmodel that you generated?

Really great repo, thanks for sharing all your experiments.

Laurent

[ldenoue]

@PINTO0309 any idea why CoreML model complains once input is converted to RGB?

I noticed that your CoreML model finishes with AddBroadcastable followed by Deconvolution, but after I used coremltools to change the input into RGB, the exported CoreML model finishes with Deconvolution + AddBroadcastable (reversed).

Any idea on how to make the CoreML work with RGB input/output?