Enable TRT provider option configuration for C# (updated version) #7808

chilo-ms · 2021-05-24T09:47:17Z

Description: Enable TRT provider option configuration for C#

Use key-value strings to configure TRT EP provider options and create session options with TRT EP specific configurations:

        var trtProviderOptions = new OrtTensorRTProviderOptions();
        var providerOptionsDict = new Dictionary<string, string>();
        providerOptionsDict["device_id"] = "0";
        providerOptionsDict["trt_int8_enable"] = "1";
        providerOptionsDict["trt_int8_calibration_table_name"] = "C:\calibration.flatbuffers";
        trtProviderOptions.UpdateOptions(providerOptionsDict);
        ....
        options.AppendExecutionProvider_Tensorrt(trtProviderOptions);

After configuration, call GetOptions() to get provider options as c# string to check

        var resultProviderOptionsDict = new Dictionary<string, string>();
        ProviderOptionsValueHelper.StringToDict(trtProviderOptions.GetOptions(), resultProviderOptionsDict);

Note: This PR is the modification of #7179

…rosoft/onnxruntime into c_sharp_trt_provider_options

chilo-ms · 2021-05-24T10:31:27Z

/azp run orttraining-linux-ci-pipeline,orttraining-mac-ci-pipeline,orttraining-linux-gpu-ci-pipeline,centos7_cpu,Linux CPU Minimal Build E2E CI Pipeline,Linux Nuphar CI Pipeline,MacOS NoContribops CI Pipeline,Linux OpenVINO CI Pipeline,orttraining-distributed, orttraining-amd-gpu-ci-pipeline

azure-pipelines · 2021-05-24T10:31:32Z

Pull request contains merge conflicts.

yuslepukhin · 2021-05-24T16:46:07Z

onnxruntime/core/providers/shared_library/provider_host_api.h


  virtual void* GetInfo() { return nullptr; }  // Returns a provider specific information interface if it exists
+
+  virtual const ProviderOptions GetProviderOptions() { return {}; }  // Returns a provider specific information interface if it exists


const

redundant

yuslepukhin · 2021-05-24T16:47:35Z

onnxruntime/core/providers/shared_library/provider_host_api.h

 // Copyright (c) Microsoft Corporation. All rights reserved.
 // Licensed under the MIT License.
+//
+#include "core/framework/provider_options.h"


Can you forward declare it here? As a reminder, you can use forward declarations in the folowing cases:

pointer 2) reference 3) return by value

Is it forward declare here?
In provider_options.h, I think they are all complete type:

using ProviderOptions = std::unordered_map<std::string, std::string>; using ProviderOptionsVector = std::vector; using ProviderOptionsMap = std::unordered_map<std::string, ProviderOptions>;

ProviderOptions can be forward declared in this header. Under onnxruntime you can say
struct/class ProviderOptions.

yuslepukhin · 2021-05-24T16:48:20Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+  /**
+  * Use this API to create the configuration of a TensorRT Execution Provider.
+  */
+  ORT_API2_STATUS(CreateTensorRTProviderOptions, _Outptr_ OrtTensorRTProviderOptions** out);


Please, document all the params and state how to destroy the instance when done.

added. and move the definitions to tensorrt_provider_factory.h

yuslepukhin · 2021-05-24T16:49:19Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+  * Get configuration of a TensorRT Execution Provider.
+  * \param allocator - a ptr to an instance of OrtAllocator obtained with CreateAllocator() or GetAllocatorWithDefaultOptions()
+  *                      the specified allocator will be used to allocate continuous buffers for output strings and lengths.
+  * \param ptr - is a null terminated string allocated using 'allocator'. The caller is responsible for freeing it.


is a null terminated string allocated using 'allocator'. The caller is responsible for freeing it.

UTF-8 string, state that one must use the same allocator to free it.

added. and move the definitions to tensorrt_provider_factory.h

include/onnxruntime/core/session/onnxruntime_c_api.h

pranavsharma

See comment.

onnxruntime/core/providers/tensorrt/tensorrt_provider_factory.cc

onnxruntime/core/framework/provider_bridge_ort.cc

include/onnxruntime/core/session/onnxruntime_c_api.h

pranavsharma · 2021-06-22T06:47:34Z

onnxruntime/core/providers/tensorrt/tensorrt_provider_factory.cc

-      trt_options.trt_int8_calibration_table_name = new char[internal_options.int8_calibration_table_name.size() + 1];
-      strcpy((char*)trt_options.trt_int8_calibration_table_name, internal_options.int8_calibration_table_name.c_str());
+      trt_options.trt_int8_calibration_table_name = new char[str_size + 1];
+      strncpy((char*)trt_options.trt_int8_calibration_table_name, internal_options.int8_calibration_table_name.c_str(), str_size + 1);


strncpy won't add the null char. I would change this to strncy(dest, src, str_size); and then dest[str_size] = '\0';

I was wondering why strncpy won't add the null char?
In this case, the size of src is always same as str_size, so strncpy will always copy the null char of src and then stop due to str_size + 1 is reached. (Also, the c_str() returns char array with null char, so src contains null char at the end)

https://en.cppreference.com/w/c/string/byte/strncpy

I think it's better to be explicit.

Agree to be explicit in order to avoid potential null termination bug.

onnxruntime/core/framework/provider_bridge_ort.cc

chilo-ms added 19 commits May 17, 2021 02:20

prepare for C# to configure provider options

7c2d4c6

add c# code

1921ec4

revert modification

9ec0f35

Add update provider info configuration in trt ep side

6ac01ca

fix bugs

de6cb01

fix bug for compiler error C2259

c80a02a

Add c# test

a2b8984

fix bug

681c319

fix bug

a67fcf3

Properly deal with string

ace27d2

Add c# api for accepting trt provider options

12b7cdc

fix bug

a763a64

Merge branch 'c_sharp_trt_provider_options' of https://github.com/mic…

557724b

…rosoft/onnxruntime into c_sharp_trt_provider_options

Modify C# test

b628bf5

add shared lib test

9677dfc

Add get provider options functionality

3e9d013

clean up

d8c18aa

clean up

69d37e8

fix bug

d40122b

chilo-ms requested review from jywu-msft, stevenlix and yuslepukhin May 24, 2021 09:47

chilo-ms requested a review from a team as a code owner May 24, 2021 09:47

Merge branch 'master' into c_sharp_trt_provider_options

b456e6c

yuslepukhin reviewed May 24, 2021

View reviewed changes

chilo-ms added 3 commits June 8, 2021 06:59

fix accidental bug

7a5f903

Add API_IMPL_BEGIN/API_IMPL_END

5e3f600

clean up

3ee5b20

jywu-msft previously approved these changes Jun 10, 2021

View reviewed changes

RyanUnderhill previously approved these changes Jun 11, 2021

View reviewed changes

pranavsharma reviewed Jun 11, 2021

View reviewed changes

include/onnxruntime/core/session/onnxruntime_c_api.h Outdated Show resolved Hide resolved

pranavsharma suggested changes Jun 11, 2021

View reviewed changes

code refactor

b312090

chilo-ms dismissed stale reviews from RyanUnderhill and jywu-msft via b312090 June 15, 2021 09:48

pranavsharma reviewed Jun 15, 2021

View reviewed changes

onnxruntime/core/providers/tensorrt/tensorrt_provider_factory.cc Outdated Show resolved Hide resolved

pranavsharma reviewed Jun 15, 2021

View reviewed changes

onnxruntime/core/framework/provider_bridge_ort.cc Show resolved Hide resolved

pranavsharma reviewed Jun 15, 2021

View reviewed changes

include/onnxruntime/core/session/onnxruntime_c_api.h Outdated Show resolved Hide resolved

chilo-ms and others added 4 commits June 17, 2021 21:37

code refactor

1f6280e

Merge branch 'master' into c_sharp_trt_provider_options

c48c9ba

fix CI fail

41a8dee

fix bug

3649fa9

chilo-ms requested a review from pranavsharma June 22, 2021 03:40

pranavsharma reviewed Jun 22, 2021

View reviewed changes

onnxruntime/core/framework/provider_bridge_ort.cc Outdated Show resolved Hide resolved

chilo-ms added 2 commits June 22, 2021 02:29

use string append

2ca2a19

Change the code to better handle strncpy and string append

a2456af

pranavsharma approved these changes Jun 25, 2021

View reviewed changes

chilo-ms merged commit 9107525 into master Jun 25, 2021

chilo-ms deleted the c_sharp_trt_provider_options branch June 25, 2021 10:21

hariharans29 mentioned this pull request Sep 17, 2021

How to apply "gpu_mem_limit" to CUDA Execution Provider in C#? #8995

Closed

hariharans29 mentioned this pull request Jan 4, 2022

Enable CUDA provider option configuration for C# #10188

Merged

chilo-ms mentioned this pull request Jan 15, 2022

Add TensorRT timing cache feature #10297

Closed

chilo-ms mentioned this pull request Feb 1, 2022

Make user capable of adding new field in OrtTensorRTProviderOptionsV2 as new provider option #10450

Merged

jywu-msft mentioned this pull request Mar 29, 2022

OpenVINO-EP v4.0 Release PR with OpenVINO 2022.1 #11025

Merged


		virtual void* GetInfo() { return nullptr; } // Returns a provider specific information interface if it exists

		virtual const ProviderOptions GetProviderOptions() { return {}; } // Returns a provider specific information interface if it exists

Enable TRT provider option configuration for C# (updated version) #7808

Enable TRT provider option configuration for C# (updated version) #7808

Uh oh!

Conversation

chilo-ms commented May 24, 2021

Uh oh!

chilo-ms commented May 24, 2021

Uh oh!

azure-pipelines bot commented May 24, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin May 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pranavsharma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms Jun 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pranavsharma Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

yuslepukhin May 24, 2021 •

edited

Loading

chilo-ms Jun 22, 2021 •

edited

Loading

pranavsharma Jun 23, 2021 •

edited

Loading

chilo-ms Jun 23, 2021 •

edited

Loading