Fold "FP16 weights -> Dequantize -> FP32 -> Conv/MatMul..." to "FP32 weights -> Conv/MatMul..."

[zhenhuaw-me]

ONNX quantization doesn't take FP16 as a quantized data type, therefore nearly all FP16 quantized TFLite models are unsupported (see this FAQ).

We recommend user switch to full integer quantization. But if the user cannot do that (for example no TensorFlow model available), we can fold "FP16 weights -> Dequantize -> FP32 -> Conv/MatMul..." to "FP32 weights -> Conv/MatMul..." to workaround this issue.

[paulgavrikov]

It would be awesome if you could implement this! Unfortunately, the mediapipe models often rely on fp16 weights.

[ram95014]

I would second @paulgavrikov and request the fold from fp16 to fp32. Thanks. If that is not possible or would take a long time, it would be great if you can guide me to do this.

[zhenhuaw-me]

@paulgavrikov @ram95014 Thanks for your feedback! I am very glad about that. This should be possible and should not take too much time but I was not working on it. It would be great if you can help!

In general, it could be divided into three steps:

1. Parse and build the graph just like ONNX can support FP16 as a quantization type. In this stage, we will have many patterns like FP16 weights -> Dequantize -> FP32 Tensor -> Conv/MatMul which will become illegal if convert to ONNX directly. I think we can support this stage currently but I have not tried that locally.
2. Walk the graph and fold FP16 weights. We can introduce a new pass like layout propagation or handling quantization - may be name like foldFP16Weights. Search the FP16 weights -> Dequantize -> FP32 Tensor -> Conv/MatMul in the graph (iterate the operators and check), for each pattern P:
   1. Cast the weights of the FP16 weights to FP32 directly, including changing Tensor.data, Tensor.dtype and etc.
   2. Detach the FP16 weights (which are actually FP32 weights now) and the Dequantize operator, and the FP32 Tensor and the Conv/MatMul operator.
   3. Attach the casted FP16 weights (which are actually FP32 weights now) to the Conv/MatMul operator.
   4. Remove the Dequantize and FP32 Tensor from the graph.
3. Recollect the tensors and operator like other graph operator passes.

I would suggest starting by adding a new operator to tflite2onnx to understand the code. They are pretty easy, you may find some in merged PRs.

It would be great if we can bring it up in the next minor release. Let's prioritize it!

[zhenhuaw-me]

@paulgavrikov @ram95014 This functionality has been enabled, please try out with the latest code.
If anything looks wrong, please open issues and link to this one. Thanks!

[mikkelmedm]

Hi, should this be fixed? I am still having issues

[zhenhuaw-me]

@mikkelmedm It has been fixed and protected by this test. What's the error you have?

[mikkelmedm]

Running it on a MediaPipe model I am getting an error like "FP16 is not tested, and might not work properly"