mxnet --> onnx converted insightface arcface model generates incorrect inference results

[cyrusbehr]

I am trying to convert the Insightface arcface LResNet100E-IR,ArcFace@ms1m-refine-v2 mxnet model to work with onnx.

Based on this issue, it looks like mxnet only supports up to onnx v1.3.0. I am therefore using the following library verions:

mxnet==1.7.0.post1
onnx==1.3.0
onnxruntime==1.6.0

It looks like out of the box the arcface model is not support by onnx, but I came across this issue which which links a script for properly converting the arcface model to onnx.

Running the script converts the model from mxnet format to onnx format successfully, however the output is no longer correct.
Here is my mxnet script I use for running inference (using a pre-aligned face chip):

#############################
# Inference with mxnet
#############################
import numpy as np
import cv2
import mxnet as mx
from collections import namedtuple

import pkg_resources
print("mxnet version:", pkg_resources.get_distribution("mxnet").version)

Batch = namedtuple('Batch', ['data'])
def getEmbedding(face_chip):
    face_chip = np.swapaxes(face_chip, 0, 2)
    face_chip = np.swapaxes(face_chip, 1, 2)
    face_chip = face_chip[np.newaxis, :]
    array = mx.nd.array(face_chip)
    mod.forward(Batch([array]))
    out = mod.get_outputs()[0].asnumpy()
    return out


prefix = "./models/insightface/model"
sym, arg, aux = mx.model.load_checkpoint(prefix, 0)
ctx = mx.cpu()
mod = mx.mod.Module(symbol=sym, context=ctx, label_names=None)
mod.bind(for_training=False, data_shapes=[('data', (1,3,112,112))])
mod.set_params(arg, aux)

img = cv2.imread("face_chip.jpg")

img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)


res = getEmbedding(img)
print(res[0][0])

I print the first element of the feature vector, which is: 2.216425

Now here is my script for running inference with onnxruntime:

#############################
# Inference with onnx runtime
#############################
import json
import numpy as np
import cv2
import onnx
import onnxruntime
from onnx import numpy_helper

import pkg_resources
print("onnx-runtime version:", pkg_resources.get_distribution("onnxruntime").version)
print("onnx version:", pkg_resources.get_distribution("onnx").version)


model="./models/insightface/arcface_r100.onnx"
 
#Preprocess the image
img = cv2.imread("face_chip.jpg")

img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img.resize((1, 3, 112, 112))
data = json.dumps({'data': img.tolist()})
data = np.array(json.loads(data)['data']).astype('float32')
 
session = onnxruntime.InferenceSession(model, None)
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name
 
result = session.run([output_name], {input_name: data})
print(result[0][0][0])

Again, I print the first element of the feature vector, which is now: -0.19618489.

This is clearly not correct. What have I done wrong? Has the conversion failed?

[nttstar]

onnx.version == '1.2.1' should work well.
The listed pre-trained models may be too load to support onnx conversion, I'm not sure.

[SthPhoenix]

@cyrusbehr it looks like you are using different code for image preprocessing for mxnet version and ONNX version.
Try using same preprocessing in both cases, something like this:

img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = np.transpose(img, (2, 0, 1))
img = np.expand_dims(img, axis=0)
img = img.astype(np.float32)

You can also check my scripts here: insight_tester.py
But don't forget to change 'trt' backend to 'onnx'

[cyrusbehr]

@SthPhoenix that did the trick thank you.

[eeric]

图像张量不减均值除方差吗？

[SthPhoenix]

图像张量不减均值除方差吗？

That's true for newer models trained with pytorch

[eeric]

图像张量不减均值除方差吗？

That's true for newer models trained with pytorch

no pytorch, while tensor was minus and mul in mxnet, because model-symbol.json of mxnet was marked as below:

{
  "op": "_minus_scalar", 
  "name": "_minusscalar0", 
  "attrs": {"scalar": "127.5"}, 
  "inputs": [[1, 0, 0]]
}, 
{
  "op": "_mul_scalar", 
  "name": "_mulscalar0", 
  "attrs": {"scalar": "0.0078125"}, 
  "inputs": [[2, 0, 0]]
},