[BUG] dai.node.NeuralNetwork Output tensor contains almost all zeros

[htangcreaxon]

HI, I have no clues why the NeuralNetwork node's output tensor comes with almost all zero with using openvino pretrained model ssdlite_mobilenet_v2 and my pytorch custom model. Before converting to the blob, I have ensure both can either run on onnxruntime or openvino.

Check if issue already exists

• I did Google it (e.g. error xy github luxonis depthai)
• I have checked troubleshooting in documentation. Unfortunately, no related topics so far...

Describe the bug
dai.node.NeuralNetwork output tensor from nnData.getLayerFp16("<layer_name>") contains almost all zeros either on TensorFlow model or Pytorch ssdlite_mobilenetv2 model.

print(np.array(lo_detectionOutput).reshape((100, 7)))
# terminal prompt  ( ssdlite_mobilenet_v2 openvino pretrained )
[
[-1.  0.  0.  0.  0.  0.  0.] 
  ...
  [ 0.  0.  0.  0.  0.  0.  0.]
  ...
 [ 0.  0.  0.  0.  0.  0.  0.]
]

Minimal Reproducible Example
Well, I am performing the sanity check for the pretrained detection model ssdlite_mobilenet_v2 from openvino to blob with using dai.node.NeuralNetwork instead of the example from the mobilenetv2Network docs which uses dai.node.MobileNetDetectionNetwork. I referenced from the two following tutorial code examples for the test.

1. Video & MobilenetSSD: I'm testing the inference on video input
2. gen2-face-detection/main.py: since the ssdlite is structrally different from ssd and dai.node.NeuralNetwork supposes to be compatible with any openvino model. This example is showing how to decode the nnData ouput tensor. For the dai.node.MobileNetDetectionNetwork, it returns ImgDetections which contains [label, confidence, x1, y1, x2, y2] in that example. Therefore, I was thinking dai.node.NeuralNetwork is more suitable for ssdlite in this case. correct me if I'm wrong here.

To reproduce this, since both tensorflow and pytorch comes with the same output array with almost all zeros. I'm here tor provide the flow that reproduce the behaviour I have.
Firstly, I pull the pretrained model from the openvino model zoo - ssdlite_mobilenet_v2. To run the model on oak, I converted it with the following options. (from tensorflow to openvino firtst)
refer : conversion to openvino #194

mo --framework=tf \
--data_type=FP16 \
--output_dir=public/ssdlite_mobilenet_v2/FP16 \
--model_name=ssdlite_mobilenet_v2 '--input_shape=[1, 300, 300, 3]' \
--input=image_tensor \
'--scale_values=image_tensor[255]' \
--reverse_input_channels \
--transformations_config=/opt/conda/lib/python3.8/site-packages/openvino/tools/mo/front/tf/ssd_v2_support.json \
--input_model=public/ssdlite_mobilenet_v2/ssdlite_mobilenet_v2_coco_2018_05_09/frozen_inference_graph.pb \
--layout "NHWC->NCHW" 
--tensorflow_object_detection_api_pipeline_config=public/ssdlite_mobilenet_v2/ssdlite_mobilenet_v2_coco_2018_05_09/pipeline.config

After that, I converted it to blob format with the following options

python -m blobconverter \
--openvino-xml ssdlite_mobilenet_v2.xml \
--openvino-bin ssdlite_mobilenet_v2.bin \
--shaves 6 -dt FP16 -v 2022.1
Downloading /home/htang/.cache/blobconverter/ssdlite_mobilenet_v2_openvino_2022.1_6shave.blob...
[==================================================]
Done

To test the result, I simply modify the original tutorial code from the Video & MobilenetSSD
Testing Snippet:

from time import monotonic
from pathlib import Path
import argparse
import sys
import cv2
import depthai as dai
import numpy as np


parser = argparse.ArgumentParser("test nn with inference on video")
parser.add_argument("blob_model", type=str, help="path to the blob model file")
parser.add_argument("video", type=str, help="path to the video file")
LABELS = [ "vehicle", "person" ]
args   = vars(parser.parse_args())

def main(args):
    blob_path    = f'{Path(args.get("blob_model")).resolve().absolute()}'
    video_source = f'{Path(args.get("video")).resolve().absolute()}'

    # create pipeline
    pipeline = dai.Pipeline()
    nn       = pipeline.create(dai.node.NeuralNetwork)
    xinFrame = pipeline.create(dai.node.XLinkIn)
    nnOut    = pipeline.create(dai.node.XLinkOut)

    # node properties
    xinFrame.setStreamName("inFrame")
    nnOut.setStreamName("nn") 
    nn.setBlobPath(blob_path)
    nn.setNumInferenceThreads(2)
    nn.input.setBlocking(False)

    # node connection
    xinFrame.out.link(nn.input)
    nn.out.link(nnOut.input)

    with dai.Device(pipeline) as device :
        inQ  = device.getInputQueue(name = "inFrame") # -> type : depthai.DataInputQueue
        detQ = device.getOutputQueue(name = "nn")     # -> type : depthai.DataOutputQueue
        assert device.isPipelineRunning, "pipeline not running please check the pipeline instantiation or node connection"


        frame = None
        #to_planar = lambda arr, shape: cv2.resize(arr, shape).transpose(2, 0, 1).flatten()

        cap   = cv2.VideoCapture(video_source)
        while cap.isOpened():
            rect, frame = cap.read()
            if not rect : break
            img = dai.ImgFrame()
            img.setData(frame)
            img.setTimestamp(monotonic()) 
            img.setWidth(300)
            img.setHeight(300)
            inQ.send(img)

            inDet = detQ.get()            
            
            
            print(f'[tensorflow pretrained blob]: {inDet.getAllLayerNames()}')
            # -> ['DetectionOutput']
            lo_detectionOutput = inDet.getLayerFp16("DetectionOutput")
            
            if frame is not None:
                cv2.imshow("test", frame)
                print(np.array(lo_detectionOutput).reshape((100, 7)))   
            if cv2.waitKey(1) == ord('q'): break

if __name__ == '__main__':
    main(args)

Expected behavior
Suppose I was getting the decoded tensor output from the nnData.getLayerFp16("DetectionOutput") following the format mentioned in the openvino's repo i.e
with np.array(lo_detectionOutput).reshape((100, 7)), the detection will be :
[image_id, label, conf, x_min, y_min, x_max, y_max]

Screenshots
I think most of the error can be revolved with only terminal prompt since I still cannot get the correct expected result...

Pipeline Graph
I haven't installed this plugin and my pipeline is relatively simple. Just link the input frame node to the dai.node.NeuralNetworknode.

Attach system log
The PC I run the snippet script

{
    "architecture": "64bit",
    "machine": "x86_64",
    "platform": "macOS-10.16-x86_64-i386-64bit",
    "processor": "i386",
    "python_build": "main Nov 24 2022 08:29:02",
    "python_compiler": "Clang 14.0.6 ",
    "python_implementation": "CPython",
    "python_version": "3.9.15",
    "release": "21.2.0",
    "system": "Darwin",
    "version": "Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64",
    "win32_ver": "",
    "packages": [
        "certifi @ file:///private/var/folders/sy/f16zz6x50xz3113nwtb9bvq00000gp/T/abs_477u68wvzm/croot/certifi_1671487773341/work/certifi",
        "charset-normalizer==2.1.1",
        "depthai==2.19.1.0",
        "idna==3.4",
        "numpy==1.24.1",
        "opencv-python==4.7.0.68",
        "pip==22.3.1",
        "PyYAML==6.0",
        "requests==2.28.1",
        "setuptools==65.5.0",
        "urllib3==1.26.13",
        "wheel==0.37.1"
    ],
    "usb": [
        "NoLib"
    ],
    "uname": [
        "Darwin ws11.local 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64 i386"
    ]
}
System info gathered successfully - saved as "log_system_information.json"

Additional context
However, I compiled the models on another PC and I don' think that matters.
That's all I have...
Thank you in advance

[daniilpastukhov]

Hello @htangcreaxon, thank you for reporting the issue.

I'll be looking into this issue within next days.

Best regards, Daniil

[Erol444]

Has this been resolved @daniilpastukhov ?