Fail to get INT8 engine with Deepstream7.1

[SiewKee-Lim]

Hi, I tried to get yolov8 INT8 engine file by following steps from https://github.com/marcoslucianops/DeepStream-Yolo/blob/master/docs/INT8Calibration.md.
However, I getting this error "ERROR: [TRT]: [checkSanity.cpp::checkLinks::218] Error Code 2: Internal Error (Assertion item.second != nullptr failed. region should have been removed from Graph::regions) Segmentation fault (core dumped)" when running command "deepstream-app -c deepstream_app_config.txt". Could you help me on this issue?

Here are the configuration that I have run with:

1. Deepstream 7.1
2. TensorRT 10.3
3. CUDA 12.6

[RX-02333]

same issue,and I found ultralytics/ultralytics#15806 says that downgrade TensorRT to 8.6.1.6 works,so maybe yolo int8 models are incompatible with TensorRT 10.X...

[marcoslucianops]

I didn't get issues running INT8 calibration with DeepStream 7.1 here

[ietamaher]

same issue, Building the TensorRT Engine

ERROR: [TRT]: [checkSanity.cpp::checkLinks::218] Error Code 2: Internal Error (Assertion item.second != nullptr failed. region should have been removed from Graph::regions)

same config as above.
Deepstream 7.1
TensorRT 10.3
CUDA 12.6
Runing on jetson AGX Orin
Opencv 4.10

Ps
I tested few weeks ago on jetson xavier nx with no issues

[marcoslucianops]

Which model are you using? Can you send the full log?

[ietamaher]

I'm running Yolov8. the issue is present only for INT8 Calibration.

rapit@ubuntu:~/DeepStream-Yolo$ deepstream-app -c deepstream_app_config.txt
Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
WARNING: Deserialize engine failed because file path: /home/rapit/DeepStream-Yolo/model_b1_gpu0_int8.engine open error
0:00:00.178367820 2848 0xaaaafd225c70 WARN nvinfer gstnvinfer.cpp:681:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2080> [UID = 1]: deserialize engine from file :/home/rapit/DeepStream-Yolo/model_b1_gpu0_int8.engine failed
0:00:00.178429612 2848 0xaaaafd225c70 WARN nvinfer gstnvinfer.cpp:681:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2185> [UID = 1]: deserialize backend context from engine from file :/home/rapit/DeepStream-Yolo/model_b1_gpu0_int8.engine failed, try rebuild
0:00:00.178449676 2848 0xaaaafd225c70 INFO nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2106> [UID = 1]: Trying to create engine from model files
WARNING: INT8 calibration file not specified/accessible. INT8 calibration can be done through setDynamicRange API in 'NvDsInferCreateNetwork' implementation

Building the TensorRT Engine

File does not exist: /home/rapit/DeepStream-Yolo/calib.table
ERROR: [TRT]: [checkSanity.cpp::checkLinks::218] Error Code 2: Internal Error (Assertion item.second != nullptr failed. region should have been removed from Graph::regions)
Segmentation fault (core dumped)

---

[primary-gie]
enable=1
gpu-id=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_yoloV8.txt

---

GNU nano 6.2 config_infer_primary_yoloV8.txt
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
onnx-file=yolov8m.pt.onnx
model-engine-file=model_b1_gpu0_int8.engine
int8-calib-file=calib.table
labelfile-path=labels.txt
batch-size=1
network-mode=1
num-detected-classes=80
interval=0
gie-unique-id=1
process-mode=1
network-type=0
cluster-mode=2
maintain-aspect-ratio=1
symmetric-padding=1
#workspace-size=2000
parse-bbox-func-name=NvDsInferParseYolo
#parse-bbox-func-name=NvDsInferParseYoloCuda
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
engine-create-func-name=NvDsInferYoloCudaEngineGet

[class-attrs-all]
nms-iou-threshold=0.45
pre-cluster-threshold=0.25
topk=300

[marcoslucianops]

Looks like it's a issue with the TRT 10.3 for Jetson boards. I don't have a Orin to debug, so it's hard to check this issue.

[Foglia-m]

Hi, I'm putting my issue in this thread because it is related, I'm running YOLOX model without int8 calibration and I also have an issue on jetson orin board with TRT 10.3:

ERROR: [TRT]: IBuilder: :buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node /0/backbone/stem/conv/conv/Conv.)
Segmentation fault (core dumped)

Can this error help in any way? Should we try implementing lower versions? Unfortunately, this requires reflash of the board.

I'll try in the next days on a desktop gpu to see how it runs.

[marcoslucianops]

@Foglia-m did you convert the pth model with the utils/export_yolox.py?

[Foglia-m]

@marcoslucianops Yes I did! I also trained a yolov8 and managed to build the tensorrt engine on deepstream.

For the yolox I also tried setting a higher opset number when converting to onnx but it did not work out either.

[marcoslucianops]

@Foglia-m I don't have Orin board to test, so it's hard to debug this issue.

[Foglia-m]

@marcoslucianops I did manage to build the tensor engine using deepstream on a laptop, but I'm still failing on the jetson.