TensorRT EP failed to create engine from network.

[xlg-go]

Describe the issue

I have a YOLOv8 detection model deployed using .NET with TensorRT as the provider. However, when I execute it, I encounter an error. Previously, when using TensorRT version 8, there were no issues.

[ErrorCode:ShapeInferenceNotRegistered] Non-zero status code returned while running TRTKernel_graph_torch_jit_4528351051880633562_0 node. Name:'TensorrtExecutionProvider_TRTKernel_graph_torch_jit_4528351051880633562_0_0' Status Message: TensorRT EP failed to create engine from network.

To reproduce

1. build onnxruntime from src code [branch main]
2. build nuget
3. install nuget
4. publish .net to unbuntu20.04

Urgency

I am very anxious.

Platform

Linux

OS Version

Ubuntu20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

Microsoft.ML.OnnxRuntime.TensorRT.1.19.0-dev-20240719-0951-9140d9b1ff.nupkg

ONNX Runtime API

C#

Architecture

X64

Execution Provider

TensorRT

Execution Provider Library Version

cuda12.5.1, cudnn9.2.1.18, tensorrt10.2.0.19

GPU

NVIDIA Quadro P6000 (Pascal)

[xlg-go]

@yf711

Could you please help me?

[yf711]

Hi @xlg-go
could you share a minimal repro case? Including the model you are using and sample code/input/output.
also, could you share the parameter you used to build ort nuget pkg from source?

[xlg-go]

Hi @xlg-go could you share a minimal repro case? Including the model you are using and sample code/input/output. also, could you share the parameter you used to build ort nuget pkg from source?

./build.sh --parallel --build_shared_lib --skip_submodule_sync --allow_running_as_root --skip_tests --config Release --update --build --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES='50;52;53;60;61;62;70;72;75;80;86;87;89;90'" --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu --use_tensorrt --tensorrt_home /usr/lib/x86_64-linux-gnu --build_nuget

A demo will be provided later.

thank you very much~

[xlg-go]

Hi @xlg-go could you share a minimal repro case? Including the model you are using and sample code/input/output. also, could you share the parameter you used to build ort nuget pkg from source?

Hi @yf711

here is src demo, thanks a lot! The *.so file is very large, would it be convenient to use yours?
I have replaced the model file with an OCR model. The YOLOv8 model is too large, but the error is similar.
When I execute "dotnet ML.OnnxRuntime.Test.dll", the following error occurs.
2024-07-22 15:18:52.698542292 [E:onnxruntime:CSharpOnnxRuntime, tensorrt_execution_provider.h:88 log] [2024-07-22 07:18:52 ERROR] IBuilder::buildSerializedNetwork: Error Code 1: Internal Error (Unsupported SM: 0x601) 2024-07-22 15:18:52.698631427 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running TRTKernel_graph_Model from PaddlePaddle._9493788502571759918_0 node. Name:'TensorrtExecutionProvider_TRTKernel_graph_Model from PaddlePaddle._9493788502571759918_0_0' Status Message: TensorRT EP failed to create engine from network.

ML.OnnxRuntime.Test.zip

or py code:
model file: The model file is located within the ML.OnnxRuntime.Test.zip archive.

import onnxruntime as ort
import numpy as np


def main():
    # ort.set_default_logger_severity(0)
    trt_ep_options = {
        "device_id": 0,
        "trt_engine_cache_enable": True,
        "trt_engine_cache_path": './trt_catch_dir',
    }
    sess = ort.InferenceSession(
        "./mlmodel/red_hw_score.onnx",
        providers=[
            ("TensorrtExecutionProvider", trt_ep_options),
            "CUDAExecutionProvider",
        ],
    )

    batch_size, chanel, height, width = 1, 3, 128, 256

    inputs = {
        "x": np.zeros(
            (batch_size, chanel, height, width), dtype=np.float32
        ),
    }

    # while True:
    outputs = sess.run(None, inputs)
    print(outputs)


if __name__ == '__main__':
    main()

[yf711]

Quick question, are you using pascal GPU, such as GTX 1080? Pascal GPU architecture is deprecated on TRT8.6

NVIDIA/TensorRT#3826 this issue matched your error code Unsupported SM: 0x601

[xlg-go]

Quick question, are you using pascal GPU, such as GTX 1080? Pascal GPU architecture is deprecated on TRT8.6

NVIDIA/TensorRT#3826 this issue matched your error code Unsupported SM: 0x601

Thank you for taking the time to address my query.
The GPU in question is indeed a Quadro P6000, which utilises the Pascal architecture.