Add support in C# to configure a CUDA EP instance #6291

hariharans29 · 2021-01-08T13:13:19Z

Description: Add support in C# to configure a CUDA EP instance

Support for configuring a CUDA EP instance was added to C/C++/Python APIs recently. This change adds support for doing the same via the C# API.

TODO: Make changes to the C/C++ API section of the documentation once #6253 is checked-in

Motivation and Context
C# -> C API parity

hariharans29 · 2021-01-08T13:15:08Z

csharp/src/Microsoft.ML.OnnxRuntime/InferenceSession.cs

-        /// <param name="names">names to convert to zero terminated utf8 and pin</param>
-        /// <param name="cleanupList">list to add pinned memory to for later disposal</param>
-        /// <returns></returns>
-        private IntPtr[] ConvertNamesToUtf8<T>(IReadOnlyCollection<T> inputs, NameExtractor<T> extractor,


Moved to a utility class to share code

hariharans29 · 2021-01-08T13:18:50Z

csharp/src/Microsoft.ML.OnnxRuntime/SessionOptions.cs

+        /// <param name="cudaProviderOptions">CUDA EP provider options to configure the CUDA EP instance</param>
+        public void AppendExecutionProvider_CUDA(OrtCUDAProviderOptions cudaProviderOptions)
+        {
+            NativeApiStatus.VerifySuccess(NativeMethods.SessionOptionsAppendExecutionProvider_CUDA(handle, cudaProviderOptions.Handle));


Main interface being made available to a user - configure and append a CUDA EP to SessionOptions

hariharans29 · 2021-01-08T13:19:51Z

include/onnxruntime/core/session/onnxruntime_c_api.h

-/// <summary>
-/// Options for the CUDA provider that are passed to SessionOptionsAppendExecutionProvider_CUDA
-/// </summary>
-typedef struct OrtCUDAProviderOptions {


Moved this struct to an internal header - this introduces only source incompatibility and ABI should still be preserved

can we do this? what if a user is using SessionOptionsAppendExecutionProvider_CUDA() and directly using this struct?
also, if this can be moved, then how about moving OrtCudnnConvAlgoSearch out as well?

In reply to: 553939121 [](ancestors = 553939121)

Can we do this ?
I believe we can. In their app, if people are constructing OrtCUDAProviderOptions and are calling SessionOptionsAppendExecutionProvider_CUDA() by passing in a pointer to the constructed instance, that app should still work with the latest version of ORT, wouldn't it ? (i.e.) ABI is preserved. They won't be able to re-build their app with the latest ORT binaries because the latest header will now be missing the definition of OrtCUDAProviderOptions , so source compatibility doesn't exist - something we never guaranteed. They would have to use the API to create an instance of OrtCUDAProviderOptions to pass into SessionOptionsAppendExecutionProvider_CUDA(). We are trying to remove ambiguity in the way OrtCUDAProviderOptions can be constructed. We did something similar in Expose knobs to create and share (CPU) allocators across sessions in C# and Python #5634 for OrtArenaCfg.
What do you think about this ?
CC: @pranavsharma

Can we move OrtCudnnConvAlgoSearch out as well ?
Yes, I think so, moved it out.

SessionOptionsAppendExecutionProvider_CUDA takes a pointer to OrtCUDAProviderOptions. The position and the signature of this function is not changing between v6 and v7. Hence in both old and new DLL cases, the correct function will be called. If the user is using the old header with the new DLL, she supplies the OrtCUDAProviderOptions ptr and all is good. New header with old DLL won't work with the same code (obviously) - unlikely scenario as there's no reason a user would upgrade without using the new DLL.

hariharans29 · 2021-01-08T13:20:13Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+  /**
+  * Use this API to create the configuration of a CUDA Execution Provider
+  */
+  ORT_API2_STATUS(CreateCUDAProviderOptions, _Outptr_ OrtCUDAProviderOptions** out);


Expose an API to create native OrtCUDAProviderOptions to be invoked from C#

hariharans29 · 2021-01-08T13:20:56Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+  /**
+  * Use this API to set the appropriate configuration knobs of a CUDA Execution Provider
+  * Please refer to the following on different key/value pairs to configure a CUDA EP and their meaning:
+  * https://github.com/microsoft/onnxruntime/blob/gh-pages/docs/reference/execution-providers/CUDA-ExecutionProvider.md


The doc file will be checked-in in #6253

hariharans29 · 2021-01-08T13:21:33Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+  * Please refer to the following on different key/value pairs to configure a CUDA EP and their meaning:
+  * https://github.com/microsoft/onnxruntime/blob/gh-pages/docs/reference/execution-providers/CUDA-ExecutionProvider.md
+  */
+  ORT_API2_STATUS(UpdateCUDAProviderOptions, _Inout_ OrtCUDAProviderOptions* cuda_provider_options,


Can provide options as key/value pairs - the same way it is done in Python

edgchen1 · 2021-01-08T22:31:27Z

onnxruntime/test/perftest/ort_test_session.cc

+    cuda_options_values.push_back("kNextPowerOfTwo");
+
+    // cuda mem limit
+    cuda_options_values.push_back(std::to_string(std::numeric_limits<size_t>::max()).c_str());


std::to_string(std::numeric_limits<size_t>::max()).c_str() [](start = 34, length = 58)

this appears to store a pointer to memory within a temporary variable.

Gosh - yes, thanks for spotting that. I created a new var that holds the string equivalent of std::numeric_limits<size_t>::max() and its lifetime should now be the lifetime of this method and passed in the pointer to memory held by it to cuda_options_values.

edgchen1 · 2021-01-08T22:39:41Z

csharp/src/Microsoft.ML.OnnxRuntime/NativeMethods.cs

+        public delegate IntPtr /*(OrtStatus*)*/DSessionOptionsAppendExecutionProvider_CUDA(
+                                               IntPtr /*(OrtSessionOptions*)*/ options,
+                                               IntPtr /*(const OrtCUDAProviderOptions*)*/ cudaProviderOptions);
+        public static DSessionOptionsAppendExecutionProvider_CUDA SessionOptionsAppendExecutionProvider_CUDA;


SessionOptionsAppendExecutionProvider_CUDA [](start = 66, length = 42)

this function's name looks different from the other append EP functions. can it be more consistent?

That is because they are fundamentally different. OrtSessionOptionsAppendExecutionProvider_CUDA is a symbol available in the shared library if the CUDA EP is built. SessionOptionsAppendExecutionProvider_CUDA is available via the C API struct always (whether built with CUDA support or not).

Also NativeMethods is an "internal concept". If you take a look at the public interface in SessionOptions.cs - the method made available to the user is consistent. There are two overloads of AppendExecutionProvider_CUDA - one calls the OrtSessionOptionsAppendExecutionProvider_CUDA and the other calls SessionOptionsAppendExecutionProvider_CUDA and all these details are abstracted from the user.

Unfortunately, I couldn't think of a way to make the naming here more consistent given that we already have a OrtSessionOptionsAppendExecutionProvider_CUDA() defined here. Open to suggestions though.

edgchen1 · 2021-01-08T22:53:32Z

csharp/src/Microsoft.ML.OnnxRuntime/ProviderOptions.cs

+        /// <summary>
+        /// Updates  the configuration knobs of OrtCUDAProviderOptions that will eventually be used to configure a CUDA EP
+        /// Please refer to the following on different key/value pairs to configure a CUDA EP and their meaning:
+        /// https://github.com/microsoft/onnxruntime/blob/gh-pages/docs/reference/execution-providers/CUDA-ExecutionProvider.md


https://github.com/microsoft/onnxruntime/blob/gh-pages/docs/reference/execution-providers/CUDA-ExecutionProvider.md [](start = 12, length = 115)

FYI, i believe this doc page will be published at https://www.onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html. @natke would that be a more stable link to reference?

That seems right. I will use the link where it will be published.

edgchen1 · 2021-01-08T22:54:48Z

csharp/src/Microsoft.ML.OnnxRuntime/ProviderOptions.cs

+        /// <param name="keys">keys of all the configuration knobs of a CUDA Execution Provider</param>
+        /// <param name="values">values of all the configuration knobs of a CUDA Execution Provider (must match number of keys)</param>
+
+        public void UpdateOptions(string[] keys, string[] values)


string[] keys, string[] values [](start = 34, length = 30)

would a Dictionary<string, string> be a friendlier way to pass the config options?

edgchen1 · 2021-01-08T23:17:09Z

include/onnxruntime/core/framework/provider_options.h

+/// This is CUDA provider specific but needs to live in a header that is build-flavor agnostic
+/// Options for the CUDA provider that are passed to SessionOptionsAppendExecutionProvider_CUDA
+/// </summary>
+typedef struct OrtCUDAProviderOptions {


OrtCUDAProviderOptions [](start = 15, length = 22)

perhaps there is some overloaded naming of "provider options". i'd suggest putting this in a separate header (cuda_provider_options.h?) so we can keep the includes more finely-grained, i.e. only including OrtCUDAProviderOptions where it's needed. if you can come up with a better name for the map<string, string> used to store provider option configs that'd be great too.

Good suggestion. I moved the struct to cuda_provider_options.h, but it can't be moved to a cuda folder and has to remain in the same folder as provider_options.h because it needs to be available even in non-CUDA builds.

I didn't quite understand the second part of the comment - the one about map<string, string>

it's fine to leave it as ProviderOptions (the map<string, string>). we use "ProviderOptions" in the names of that typedef and the EP struct so i was just thinking it would be nice if the names were more different.

edgchen1 · 2021-01-08T23:18:32Z

onnxruntime/core/session/onnxruntime_c_api.cc

+                    size_t num_keys) {
+  API_IMPL_BEGIN
+#ifdef USE_CUDA
+  std::unordered_map<std::string, std::string> provider_options_map;


std::unordered_map<std::string, std::string [](start = 2, length = 43)

aka ProviderOptions

edgchen1 · 2021-01-08T23:23:24Z

onnxruntime/core/session/onnxruntime_c_api.cc

+        provider_options_values == nullptr || provider_options_values[i][0] == '\0') {
+      return OrtApis::CreateStatus(ORT_INVALID_ARGUMENT, "key/value cannot be empty");
+    }
+    provider_options_map[std::string(provider_options_keys[i])] = std::string(provider_options_values[i]);


std::string [](start = 25, length = 11)

nit: how about provider_options_map[provider_options_keys[i]] = provider_options_values[i];

edgchen1 · 2021-01-08T23:33:34Z

i wonder if we should just move to supporting EP creation with config as string key-value pairs from the C API. would be nice to have a uniform way of specifying config for all EPs, and a single place to validate values and specify defaults. and fewer configuration structs in the public API. what do you think?

pranavsharma · 2021-01-11T21:04:50Z

include/onnxruntime/core/session/onnxruntime_c_api.h

-/// <summary>
-/// Options for the CUDA provider that are passed to SessionOptionsAppendExecutionProvider_CUDA
-/// </summary>
-typedef struct OrtCUDAProviderOptions {


SessionOptionsAppendExecutionProvider_CUDA takes a pointer to OrtCUDAProviderOptions. The position and the signature of this function is not changing between v6 and v7. Hence in both old and new DLL cases, the correct function will be called. If the user is using the old header with the new DLL, she supplies the OrtCUDAProviderOptions ptr and all is good. New header with old DLL won't work with the same code (obviously) - unlikely scenario as there's no reason a user would upgrade without using the new DLL.

pranavsharma · 2021-01-11T21:15:30Z

include/onnxruntime/core/framework/cuda_provider_options.h

+
+#pragma once
+
+typedef enum OrtCudnnConvAlgoSearch {


do we need the typedef?

edgchen1 · 2021-01-12T18:39:09Z

i wonder if we should just move to supporting EP creation with config as string key-value pairs from the C API. would be nice to have a uniform way of specifying config for all EPs, and a single place to validate values and specify defaults. and fewer configuration structs in the public API. what do you think?

@hariharans29, @pranavsharma
could we have a single EP creation function like this?

SessionOptionsAppendExecutionProvider(
  OrtSessionOptions* session_options,
  const char* ep_name,
  size_t ep_config_count, const char* const* ep_config_keys, const char* const* ep_config_values)

pranavsharma · 2021-01-12T20:03:11Z

i wonder if we should just move to supporting EP creation with config as string key-value pairs from the C API. would be nice to have a uniform way of specifying config for all EPs, and a single place to validate values and specify defaults. and fewer configuration structs in the public API. what do you think?

@hariharans29, @pranavsharma
could we have a single EP creation function like this?
SessionOptionsAppendExecutionProvider(
  OrtSessionOptions* session_options,
  const char* ep_name,
  size_t ep_config_count, const char* const* ep_config_keys, const char* const* ep_config_values)

Your proposal is fine, albeit a bit less convenient for users since they now have to worry about serializing the values. Not sure about the single place of validation; I don't think you'll able to validate much centrally besides matching the number of keys and values. Even before I didn't want EP specific config options to be exposed in the main C API header since it's unnecessary stuff that users get even when they're not using that specific EP, for e.g. see how we've both CUDA and OpenVino options exposed in c_api.h.

As a side note, we also have config keys in session options that we could use given that all the EP append functions accept a ptr to session options.

yuslepukhin · 2021-02-08T18:57:16Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+
+  /**
+  * Use this API to set the appropriate configuration knobs of a CUDA Execution Provider
+  * Please refer to the following on different key/value pairs to configure a CUDA EP and their meaning:


Please refer to the following on [](start = 4, length = 32)

Sure, but we need to specifically state the format and the content for the params. Strings are UTF8, do they contain zero terminators, etc.

yuslepukhin · 2021-02-08T18:58:05Z

onnxruntime/core/session/onnxruntime_c_api.cc

+    if (provider_options_keys[i] == nullptr || provider_options_keys[i][0] == '\0' ||
+        provider_options_values == nullptr || provider_options_values[i][0] == '\0') {
+      return OrtApis::CreateStatus(ORT_INVALID_ARGUMENT, "key/value cannot be empty");
+    }


Perhaps, we can somehow add an index of the problematic pair?

yuslepukhin · 2021-02-08T19:00:18Z

onnxruntime/test/perftest/ort_test_session.cc

+    OrtCUDAProviderOptions* cuda_options;
+    Ort::ThrowOnError(api.CreateCUDAProviderOptions(&cuda_options));
+    std::unique_ptr<OrtCUDAProviderOptions, decltype(api.ReleaseCUDAProviderOptions)> rel_cuda_options(cuda_options, api.ReleaseCUDAProviderOptions);
+


Would not be easier to provide a C++ interface that would make sure that we don't leak?

yuslepukhin · 2021-02-08T19:00:47Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+  /**
+  * Use this API to create the configuration of a CUDA Execution Provider
+  */
+  ORT_API2_STATUS(CreateCUDAProviderOptions, _Outptr_ OrtCUDAProviderOptions** out);


I vote for C++ interface

yuslepukhin · 2021-02-08T19:03:01Z

csharp/src/Microsoft.ML.OnnxRuntime/NativeOnnxValueHelper.cs


 using Microsoft.ML.OnnxRuntime.Tensors;
 using System;
+using System.Linq;


System.Linq; [](start = 6, length = 12)

Check if this is really needed

yuslepukhin · 2021-05-11T20:14:40Z

csharp/src/Microsoft.ML.OnnxRuntime/NativeOnnxValueHelper.cs


 using Microsoft.ML.OnnxRuntime.Tensors;
 using System;
+using System.Linq;


System.Linq;

Is this needed?

I'm referencing the C# code here.
I think is for ElementAt() and if I don't include System.Linq, I will get:
error CS1061: 'IReadOnlyCollection' does not contain a definition for 'ElementAt' and no accessible extension method 'ElementAt' accepting a first argument of type 'IReadOnlyCollection' could be found

yuslepukhin · 2021-05-11T20:19:25Z

csharp/test/Microsoft.ML.OnnxRuntime.Tests/InferenceTest.cs

+                providerOptionsDict["cudnn_conv_algo_search"] = "HEURISTIC";
+                providerOptionsDict["do_copy_in_default_stream"] = "1";
+
+                cudaProviderOptions.UpdateOptions(providerOptionsDict);


cudaProviderOptions.UpdateOptions(providerOptionsDict);

Is there any way to make sure that they actually had effect? Also can we add some negative test?

yuslepukhin · 2021-05-11T20:21:53Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+  ORT_API2_STATUS(CreateCUDAProviderOptions, _Outptr_ OrtCUDAProviderOptions** out);
+
+  /**
+  * Use this API to set the appropriate configuration knobs of a CUDA Execution Provider


se this API to set the ap

The encoding must be mentioned. Also we should always document Doxygen params, although reference to external doc is also helpful.

yuslepukhin · 2021-05-11T20:22:50Z

I am not seeing C++ API which is usually super helpful.

HectorSVC · 2021-05-20T03:58:28Z

onnxruntime/core/session/onnxruntime_c_api.cc

+  ProviderOptions provider_options_map;
+  for (size_t i = 0; i != num_keys; ++i) {
+    if (provider_options_keys[i] == nullptr || provider_options_keys[i][0] == '\0' ||
+        provider_options_values == nullptr || provider_options_values[i][0] == '\0') {


provider_options_values

provider_options_values[i]

stale · 2022-04-16T08:53:41Z

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

hariharans29 added 5 commits January 7, 2021 18:12

Initial commit

5d14b08

More changes

36aa7b4

Merge remote-tracking branch 'origin/master' into hari/cSharpApi

94e4603

More changes

9c74b15

More changes

6b8ca35

hariharans29 requested a review from a team as a code owner January 8, 2021 13:13

hariharans29 commented Jan 8, 2021

View reviewed changes

edgchen1 reviewed Jan 8, 2021

View reviewed changes

hariharans29 added 3 commits January 10, 2021 21:39

PR feedback

2fc1a8b

Fix build

1e28cfa

Fix build

4b054a1

pranavsharma reviewed Jan 11, 2021

View reviewed changes

yuslepukhin reviewed Feb 8, 2021

View reviewed changes

hariharans29 mentioned this pull request Apr 5, 2021

Enable TRT provider option configuration for C# #7179

Closed

chilo-ms mentioned this pull request Apr 12, 2021

Enable CUDA provider option configuration for C# #7315

Open

yuslepukhin reviewed May 11, 2021

View reviewed changes

HectorSVC reviewed May 20, 2021

View reviewed changes

hariharans29 mentioned this pull request Sep 17, 2021

How to apply "gpu_mem_limit" to CUDA Execution Provider in C#? #8995

Closed

callbarian mentioned this pull request Nov 11, 2021

C# adding option for cudnn_conv_algo_search : DEFAULT #9730

Closed

stale bot added the stale issues that have not been addressed in a while; categorized by a bot label Apr 16, 2022

hariharans29 closed this Aug 26, 2022

hariharans29 deleted the hari/cSharpApi branch August 26, 2022 21:23

Add support in C# to configure a CUDA EP instance #6291

Add support in C# to configure a CUDA EP instance #6291

Uh oh!

Conversation

hariharans29 commented Jan 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edgchen1 Jan 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hariharans29 Jan 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hariharans29 Jan 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hariharans29 Jan 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edgchen1 commented Jan 8, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edgchen1 commented Jan 12, 2021

Uh oh!

pranavsharma commented Jan 12, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

hariharans29 commented Jan 8, 2021 •

edited

Loading

edgchen1 Jan 8, 2021 •

edited

Loading

hariharans29 Jan 9, 2021 •

edited

Loading

hariharans29 Jan 9, 2021 •

edited

Loading

hariharans29 Jan 9, 2021 •

edited

Loading