Skip to content

Enable cuda graph in TensorRT EP #10423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
stevenlix wants to merge 9 commits into from
Closed

Enable cuda graph in TensorRT EP #10423

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
04c9fd7
add cuda graph support
stevenlix Jan 21, 2022
634c2e3
add provider option for cuda graph
Jan 21, 2022
2489976
fix issues
stevenlix Jan 27, 2022
1e71901
clean up
Jan 28, 2022
a330b5a
clean up
stevenlix Jan 28, 2022
3e2e847
fix issues
stevenlix Jan 28, 2022
6427a24
fixed pointer issue
stevenlix Jan 29, 2022
f17e84c
merge master
stevenlix Feb 24, 2022
d09a39d
fix issues
stevenlix Mar 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions include/onnxruntime/core/providers/tensorrt/tensorrt_provider_options.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,5 @@ struct OrtTensorRTProviderOptionsV2 {
int trt_engine_decryption_enable; // enable engine decryption. Default 0 = false, nonzero = true
const char* trt_engine_decryption_lib_path; // specify engine decryption library path
int trt_force_sequential_engine_build; // force building TensorRT engine sequentially. Default 0 = false, nonzero = true
int trt_cuda_graph_enable; // enable cuda graph. Default 0 = false, nonzero = true
};
1 change: 1 addition & 0 deletions include/onnxruntime/core/session/onnxruntime_c_api.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,7 @@ typedef struct OrtTensorRTProviderOptions {
int trt_engine_decryption_enable; // enable engine decryption. Default 0 = false, nonzero = true
const char* trt_engine_decryption_lib_path; // specify engine decryption library path
int trt_force_sequential_engine_build; // force building TensorRT engine sequentially. Default 0 = false, nonzero = true
int trt_cuda_graph_enable; // enable cuda graph. Default 0 = false, nonzero = true

@hariharans29 hariharans29 Jan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the struct get versioned with the addition of a new option ?

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we agreed we shouldn't be updating provider structs in c api anymore (for the very reason that Hari brings up about versioning)
and instead only updating the opaque struct OrtTensorRTProviderOptionsV2
+@chilo-ms FYI

@hariharans29 hariharans29 Jan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think it should become OrtTensorRTProviderOptionsV3 if V2 has shipped with the previous ORT release (This was my understanding)

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think it should become OrtTensorRTProviderOptionsV3 if V2 has shipped with the previous ORT release (This was my understanding)

we don't need to update the version of the opaque struct when adding fields right? since it's only accessed via api and not directly. if the newly added field can't be represented as a string, then we would need to add another api to access those.

@hariharans29 hariharans29 Jan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC from the time we were discussing it, we need to version it for any mutation (addition or removal of fields). If we add support for a new field without versioning it, doesn't the UpdateTensorRTProviderOptions API behave differently in ORT 1.10 (where the V2 struct won't support the new field) and in ORT 1.11 (where the V2 struct will support the new field) ?

@chilo-ms chilo-ms Jan 28, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a consensus on using key-value strings for provider options that can be represented by string but we didn't explicitly say we support versioning. But we require error reporting for undocumented config keys.

@stevenlix, here is my "enable timing cache" PR, you can reference it to add new field to opaque struct OrtTensorRTProviderOptionsV2

@pranavsharma pranavsharma Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No versioning of struct is required beyond V2. The availability or the unavailability of APIs to manipulate the V2 struct provides the versioning.

@hariharans29 hariharans29 Jan 29, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The availability or the unavailability of APIs to manipulate the V2 struct provides the versioning." - What APIs exists to manipulate the V2 struct will be the same in ORT 1.10 and ORT 1.11 won't they ? It is just that the UpdateTensorRTProviderOptions() API will additionally support one more key (enable_cuda_graph) in 1.11 (which obviously won't be supported in the released 1.10).

@jywu-mysoft jywu-mysoft Jan 30, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, one uses CreateTensorRTProviderOptions(), UpdateTensorRTProviderOptions() to create and update the struct.
The second api deals with strings, and will recognize a new string key "enable_cuda_graph" in ort 1.11
api signatures don't change.

@stevenlix stevenlix Feb 1, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

talked to @chilo-ms offline. He is going to make a separate PR from the timing cache PR for OrtTensorRTProviderOptionsV2. After the PR merged, I will add cuda_graph option in OrtTensorRTProviderOptionsV2.

} OrtTensorRTProviderOptions;

/** \brief MIGraphX Provider Options
Expand Down
51 changes: 43 additions & 8 deletions onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,7 @@ TensorrtExecutionProvider::TensorrtExecutionProvider(const TensorrtExecutionProv
engine_decryption_lib_path_ = info.engine_decryption_lib_path;
}
force_sequential_engine_build_ = info.force_sequential_engine_build;
cuda_graph_enable_ = info.cuda_graph_enable;
} else {
const std::string max_partition_iterations_env = onnxruntime::GetEnvironmentVar(tensorrt_env_vars::kMaxPartitionIterations);
if (!max_partition_iterations_env.empty()) {
Expand Down Expand Up @@ -519,6 +520,11 @@ TensorrtExecutionProvider::TensorrtExecutionProvider(const TensorrtExecutionProv
if (!force_sequential_engine_build_env.empty()) {
force_sequential_engine_build_ = (std::stoi(force_sequential_engine_build_env) == 0 ? false : true);
}

const std::string cuda_graph_enable_env = onnxruntime::GetEnvironmentVar(tensorrt_env_vars::kCUDAGraphEnable);
if (!cuda_graph_enable_env.empty()) {
cuda_graph_enable_ = (std::stoi(cuda_graph_enable_env) == 0 ? false : true);
}
}

// Validate setting
Expand Down Expand Up @@ -579,7 +585,8 @@ TensorrtExecutionProvider::TensorrtExecutionProvider(const TensorrtExecutionProv
<< ", trt_cache_path: " << cache_path_
<< ", trt_engine_decryption_enable: " << engine_decryption_enable_
<< ", trt_engine_decryption_lib_path: " << engine_decryption_lib_path_
<< ", trt_force_sequential_engine_build: " << force_sequential_engine_build_;
<< ", trt_force_sequential_engine_build: " << force_sequential_engine_build_
<< ", trt_cuda_graph_enable: " << cuda_graph_enable_;
}

TensorrtExecutionProvider::~TensorrtExecutionProvider() {
Expand Down Expand Up @@ -1159,7 +1166,10 @@ std::unique_lock<OrtMutex> TensorrtExecutionProvider::GetEngineBuildLock() const

common::Status TensorrtExecutionProvider::Compile(const std::vector<Node*>& fused_nodes,
std::vector<NodeComputeInfo>& node_compute_funcs) {
for (const auto* fused_node : fused_nodes) {
size_t fused_nodes_size = fused_nodes.size();
cuda_graphs_.reserve(fused_nodes_size);
for (size_t node_idx = 0; node_idx < fused_nodes_size; node_idx++) {
const auto* fused_node = fused_nodes[node_idx];

@hariharans29 hariharans29 Jan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, how does the TRT EP handle control flow nodes ? I fear we must explicitly not support using cuda graphs for models with control flow nodes as the graph captured for one input may not the same graph required for another input (because of the dynamic graph branching).

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed. we should explicitly exclude graphs with loops/conditionals.

@stevenlix stevenlix Jan 28, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may highlight the constrains for dynamic shape cases in document, so that users can choose to enable cuda graph or not.

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be nice to enforce constraints (doesn't support dynamic shapes and dynamic graphs) in code rather than punting to user/documentation. Let's see if there's a reasonable balance that can be achieved here.

@stevenlix stevenlix Feb 2, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a little bit tricky. Dynamic shape model is okay, but when input shapes of incoming data change, cuda graph needs to be recaptured, so the check has to be done in runtime. There is an API to update executable graph, but I haven't seen any APIs that can check existing cuda graph's profile, and we can't afford to update graph for every enqueue.

// Build map from input name to its index in input definitions
std::unordered_map<std::string, size_t> input_map;
const auto& input_defs = fused_node->InputDefs();
Expand Down Expand Up @@ -1407,6 +1417,7 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector<Node*>& fuse
contexts_.emplace(fused_node->Name(), std::move(trt_context));
builders_.emplace(fused_node->Name(), std::move(trt_builder));
networks_.emplace(fused_node->Name(), std::move(trt_network));
cuda_graph_instances_.emplace(fused_node->Name(), cuda_graphs_[node_idx]);
input_info_[fused_node->Name()].push_back(input_indexes);
output_info_[fused_node->Name()].push_back(output_indexes);
output_info_[fused_node->Name()].push_back(output_types);
Expand All @@ -1421,8 +1432,9 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector<Node*>& fuse
&engines_[context->node_name], &contexts_[context->node_name], &builders_[context->node_name],
&networks_[context->node_name], input_info_[context->node_name], output_info_[context->node_name],
input_shape_ranges_[context->node_name], &tensorrt_mu_, fp16_enable_, int8_enable_, int8_calibration_cache_available_,
dla_enable_, dla_core_, &max_workspace_size_, trt_node_name_with_precision, engine_cache_enable_, cache_path_,
runtime_.get(), nullptr, allocator_, dynamic_range_map, engine_decryption_enable_, engine_decryption_, engine_encryption_};
dla_enable_, dla_core_, cuda_graph_enable_, nullptr, cuda_graph_instances_[context->node_name], &max_workspace_size_,
trt_node_name_with_precision, engine_cache_enable_, cache_path_, runtime_.get(), nullptr, allocator_,
dynamic_range_map, engine_decryption_enable_, engine_decryption_, engine_encryption_};
*state = p.release();
return 0;
};
Expand All @@ -1445,6 +1457,8 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector<Node*>& fuse
auto trt_engine = trt_state->engine->get();
auto trt_context = trt_state->context->get();
auto trt_profile = &(trt_state->trt_profile);
auto cuda_graph_ptr = &(trt_state->cuda_graph_ptr);

auto alloc = trt_state->scratch_allocator;
int num_inputs = static_cast<int>(input_indexes.size());
int num_outputs = static_cast<int>(output_indexes.size());
Expand Down Expand Up @@ -2006,10 +2020,31 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector<Node*>& fuse
}
}

// Run TRT inference
if (!trt_context->enqueueV2(&buffers[0], stream, nullptr)) {
return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, "TensorRT EP execution context enqueue failed.");
}
// Run TRT inference
if (trt_state->cuda_graph_enable)
{
if (*cuda_graph_ptr == nullptr) {
Comment thread
hariharans29 marked this conversation as resolved.
cudaGraph_t graph;
*cuda_graph_ptr = &(trt_state->cuda_graph_instance);
//warm up for cuda graph capturing
if (!trt_context->enqueueV2(&buffers[0], stream, nullptr)) {

@hariharans29 hariharans29 Jan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does enqueueV2() synchronize with the GPU before returning ? If not, we may have to wait for the warm-up tasks queued on the stream to finish before the stream capture...

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this warm up even needed?

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like it's for handling a known issue with dynamic shapes? (please add a comment)

@stevenlix stevenlix Jan 28, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warm-up is needed to do initialization (flushing any old context) before graph capturing according to Nvidia. CUDA graph still has issue in some dynamic shape cases.

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the note section of the api doc for enqueuev2 https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_execution_context.html#a2f4429652736e8ef6e19f433400108c7
seems to allude to dynamic shapes can work if you call enqueuev2() once before graph capture? is it similar to what you are doing?

@stevenlix stevenlix Jan 28, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If shape changed, cuda graph needs to be recaptured, which is not desired because the capturing happens in inference.

return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, "TensorRT EP execution context enqueue failed.");
}
CUDA_CALL_THROW(cudaStreamBeginCapture(stream, cudaStreamCaptureModeRelaxed));

@hariharans29 hariharans29 Jan 31, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If certain operators can't be supported by the TensorRT and CUDA EPs, does the CPU EP come into play ? If so, the same comment as this- #9978 (comment)

@stevenlix stevenlix Feb 1, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cuda graph capturing in this PR is only for TensorRT subgraphs. Unsupported ops can still fall back to CUDA/CPU EPs.

@hariharans29 hariharans29 Feb 2, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood now, thanks.

One question though: Let us say the model looks like this: TensorRT subgraph 1 -> CPU op -> TensorRT subgraph 2 and you are capturing the graphs for both the TRT subgraphs, will the necessary synchronization logic before the CPU op happen even with the cuda graph setup ?

@stevenlix stevenlix Feb 2, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the case is the same with and without cuda graph, isn't it? CPU needs to wait until TRT subgraph1 produces its result. That's interesting if there is a better cuda graph setup that can avoid the wait.

if (!trt_context->enqueueV2(&buffers[0], stream, nullptr)) {
return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, "TensorRT EP execution context enqueue failed.");
}
CUDA_CALL_THROW(cudaStreamEndCapture(stream, &graph));
CUDA_CALL_THROW(cudaStreamSynchronize(stream));
CUDA_CALL_THROW(cudaGraphInstantiate(*cuda_graph_ptr, graph, NULL, NULL, 0));
CUDA_CALL_THROW(cudaGraphDestroy(graph));

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

safer to use unique_ptr right?
otherwise in error/exception paths you may not hit this destroy.

}

@jywu-mysoft jywu-mysoft Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens to graph? do you need to destroy it? else we leak memory?

cudaGraphLaunch(**cuda_graph_ptr, stream);
} else {
if (!trt_context->enqueueV2(&buffers[0], stream, nullptr)) {
return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, "TensorRT EP execution context enqueue failed.");
}
}

// Cast INT64 input to INT32 because TensorRT doesn't fully support INT64
for (size_t i = 0, end = output_binding_names.size(); i < end; ++i) {
Expand Down
7 changes: 7 additions & 0 deletions onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ static const std::string kCachePath = "ORT_TENSORRT_CACHE_PATH";
static const std::string kDecryptionEnable = "ORT_TENSORRT_ENGINE_DECRYPTION_ENABLE";
static const std::string kDecryptionLibPath = "ORT_TENSORRT_ENGINE_DECRYPTION_LIB_PATH";
static const std::string kForceSequentialEngineBuild= "ORT_TENSORRT_FORCE_SEQUENTIAL_ENGINE_BUILD";
static const std::string kCUDAGraphEnable = "ORT_TENSORRT_CUDA_GRAPH_ENABLE";
// Old env variable for backward compatibility
static const std::string kEngineCachePath = "ORT_TENSORRT_ENGINE_CACHE_PATH";
} // namespace tensorrt_env_vars
Expand Down Expand Up @@ -96,6 +97,9 @@ struct TensorrtFuncState {
bool int8_calibration_cache_available;
bool dla_enable;
int dla_core;
bool cuda_graph_enable;
cudaGraphExec_t* cuda_graph_ptr = nullptr;
cudaGraphExec_t cuda_graph_instance;
size_t* max_workspace_size_ptr = nullptr;
std::string trt_node_name_with_precision;
bool engine_cache_enable;
Expand Down Expand Up @@ -167,6 +171,8 @@ class TensorrtExecutionProvider : public IExecutionProvider {
bool engine_decryption_enable_ = false;
int (*engine_decryption_)(const char*, char*, size_t*);
int (*engine_encryption_)(const char*, char*, size_t);
bool cuda_graph_enable_ = false;
std::vector<cudaGraphExec_t> cuda_graphs_;

std::unordered_map<std::string, tensorrt_ptr::unique_pointer<nvonnxparser::IParser>> parsers_;
std::unordered_map<std::string, tensorrt_ptr::unique_pointer<nvinfer1::ICudaEngine>> engines_;
Expand All @@ -176,6 +182,7 @@ class TensorrtExecutionProvider : public IExecutionProvider {
std::unordered_map<std::string, std::vector<std::unordered_map<std::string, size_t>>> input_info_;
std::unordered_map<std::string, std::vector<std::unordered_map<std::string, size_t>>> output_info_;
std::unordered_map<std::string, std::unordered_map<std::string, std::unordered_map<size_t, std::pair<int64_t, int64_t>>>> input_shape_ranges_;
std::unordered_map<std::string, cudaGraphExec_t> cuda_graph_instances_;

/**Get IndexedSubGraph based on node list of the subgraph*/
std::unique_ptr<IndexedSubGraph> GetSubGraph(SubGraph_t graph_nodes_index,
Expand Down
4 changes: 4 additions & 0 deletions onnxruntime/core/providers/tensorrt/tensorrt_execution_provider_info.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ constexpr const char* kCachePath = "trt_engine_cache_path";
constexpr const char* kDecryptionEnable = "trt_engine_decryption_enable";
constexpr const char* kDecryptionLibPath = "trt_engine_decryption_lib_path";
constexpr const char* kForceSequentialEngineBuild = "trt_force_sequential_engine_build";
constexpr const char* kCUDAGraphEnable = "trt_cuda_graph_enable";
} // namespace provider_option_names
} // namespace tensorrt

Expand Down Expand Up @@ -63,6 +64,7 @@ TensorrtExecutionProviderInfo TensorrtExecutionProviderInfo::FromProviderOptions
.AddAssignmentToReference(tensorrt::provider_option_names::kDecryptionEnable, info.engine_decryption_enable)
.AddAssignmentToReference(tensorrt::provider_option_names::kDecryptionLibPath, info.engine_decryption_lib_path)
.AddAssignmentToReference(tensorrt::provider_option_names::kForceSequentialEngineBuild, info.force_sequential_engine_build)
.AddAssignmentToReference(tensorrt::provider_option_names::kCUDAGraphEnable, info.cuda_graph_enable)
.Parse(options));

return info;
Expand All @@ -87,6 +89,7 @@ ProviderOptions TensorrtExecutionProviderInfo::ToProviderOptions(const TensorrtE
{tensorrt::provider_option_names::kDecryptionEnable, MakeStringWithClassicLocale(info.engine_decryption_enable)},
{tensorrt::provider_option_names::kDecryptionLibPath, MakeStringWithClassicLocale(info.engine_decryption_lib_path)},
{tensorrt::provider_option_names::kForceSequentialEngineBuild, MakeStringWithClassicLocale(info.force_sequential_engine_build)},
{tensorrt::provider_option_names::kCUDAGraphEnable, MakeStringWithClassicLocale(info.cuda_graph_enable)},
};
return options;
}
Expand Down Expand Up @@ -115,6 +118,7 @@ ProviderOptions TensorrtExecutionProviderInfo::ToProviderOptions(const OrtTensor
{tensorrt::provider_option_names::kDecryptionEnable, MakeStringWithClassicLocale(info.trt_engine_decryption_enable)},
{tensorrt::provider_option_names::kDecryptionLibPath, kDecryptionLibPath_},
{tensorrt::provider_option_names::kForceSequentialEngineBuild, MakeStringWithClassicLocale(info.trt_force_sequential_engine_build)},
{tensorrt::provider_option_names::kCUDAGraphEnable, MakeStringWithClassicLocale(info.trt_cuda_graph_enable)},
};
return options;
}
Expand Down
1 change: 1 addition & 0 deletions onnxruntime/core/providers/tensorrt/tensorrt_execution_provider_info.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ struct TensorrtExecutionProviderInfo {
bool engine_decryption_enable{false};
std::string engine_decryption_lib_path{""};
bool force_sequential_engine_build{false};
bool cuda_graph_enable{false};

static TensorrtExecutionProviderInfo FromProviderOptions(const ProviderOptions& options);
static ProviderOptions ToProviderOptions(const TensorrtExecutionProviderInfo& info);
Expand Down
2 changes: 2 additions & 0 deletions onnxruntime/core/providers/tensorrt/tensorrt_provider_factory.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ struct Tensorrt_Provider : Provider {
info.engine_decryption_enable = options.trt_engine_decryption_enable != 0;
info.engine_decryption_lib_path = options.trt_engine_decryption_lib_path == nullptr ? "" : options.trt_engine_decryption_lib_path;
info.force_sequential_engine_build = options.trt_force_sequential_engine_build != 0;
info.cuda_graph_enable = options.trt_cuda_graph_enable != 0;
return std::make_shared<TensorrtProviderFactory>(info);
}

Expand Down Expand Up @@ -134,6 +135,7 @@ struct Tensorrt_Provider : Provider {
}

trt_options.trt_force_sequential_engine_build = internal_options.force_sequential_engine_build;
trt_options.trt_cuda_graph_enable = internal_options.cuda_graph_enable;
}

ProviderOptions GetProviderOptions(const void* provider_options) override {
Expand Down
1 change: 1 addition & 0 deletions onnxruntime/core/session/provider_bridge_ort.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1466,6 +1466,7 @@ ORT_API_STATUS_IMPL(OrtApis::CreateTensorRTProviderOptions, _Outptr_ OrtTensorRT
(*out)->trt_engine_decryption_enable = false;
(*out)->trt_engine_decryption_lib_path = nullptr;
(*out)->trt_force_sequential_engine_build = false;
(*out)->trt_cuda_graph_enable = false;
return nullptr;
#else
ORT_UNUSED_PARAMETER(out);
Expand Down
9 changes: 9 additions & 0 deletions onnxruntime/python/onnxruntime_pybind_state.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,7 @@ std::unique_ptr<IExecutionProvider> CreateExecutionProviderInstance(
nullptr,
0,
nullptr,
0,
0};
for (auto option : it->second) {
if (option.first == "device_id") {
Expand Down Expand Up @@ -500,6 +501,14 @@ std::unique_ptr<IExecutionProvider> CreateExecutionProviderInstance(
} else {
ORT_THROW("[ERROR] [TensorRT] The value for the key 'trt_force_sequential_engine_build' should be a boolean i.e. 'True' or 'False'. Default value is False.\n");
}
} else if (option.first == "trt_cuda_graph_enable") {
if (option.second == "True" || option.second == "true") {
params.trt_cuda_graph_enable = true;
} else if (option.second == "False" || option.second == "false") {
params.trt_cuda_graph_enable = false;
} else {
ORT_THROW("[ERROR] [TensorRT] The value for the key 'trt_cuda_graph_enable' should be a boolean i.e. 'True' or 'False'. Default value is False.\n");
}
} else {
ORT_THROW("Invalid TensorRT EP option: ", option.first);
}
Expand Down
1 change: 1 addition & 0 deletions onnxruntime/python/tools/tensorrt/perf/mem_test/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ std::unique_ptr<OrtTensorRTProviderOptions> get_default_trt_provider_options() {
tensorrt_options->trt_engine_decryption_enable = false;
tensorrt_options->trt_engine_decryption_lib_path = "";
tensorrt_options->trt_force_sequential_engine_build = false;
tensorrt_options->trt_cuda_graph_enable = false;

return tensorrt_options;
}
Expand Down
1 change: 1 addition & 0 deletions onnxruntime/test/perftest/command_args_parser.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ namespace perftest {
"\t [TensorRT only] [trt_engine_cache_enable]: Enable engine caching.\n"
"\t [TensorRT only] [trt_engine_cache_path]: Specify engine cache path.\n"
"\t [TensorRT only] [trt_force_sequential_engine_build]: Force TensorRT engines to be built sequentially.\n"
"\t [TensorRT only] [trt_cuda_graph_enable]: Enable CUDA graph.\n"
"\t [Usage]: -e <provider_name> -i '<key1>|<value1> <key2>|<value2>'\n\n"
"\t [Example] [For TensorRT EP] -e tensorrt -i 'trt_fp16_enable|true trt_int8_enable|true trt_int8_calibration_table_name|calibration.flatbuffers trt_int8_use_native_calibration_table|false trt_force_sequential_engine_build|false'\n"
"\t-h: help\n");
Expand Down
12 changes: 11 additions & 1 deletion onnxruntime/test/perftest/ort_test_session.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ OnnxRuntimeTestSession::OnnxRuntimeTestSession(Ort::Env& env, std::random_device
bool trt_engine_decryption_enable = false;
std::string trt_engine_decryption_lib_path = "";
bool trt_force_sequential_engine_build = false;
bool trt_cuda_graph_enable = false;

#ifdef _MSC_VER
std::string ov_string = ToMBString(performance_test_config.run_config.ep_runtime_config_string);
Expand Down Expand Up @@ -205,8 +206,16 @@ OnnxRuntimeTestSession::OnnxRuntimeTestSession(Ort::Env& env, std::random_device
} else {
ORT_THROW("[ERROR] [TensorRT] The value for the key 'trt_force_sequential_engine_build' should be a boolean i.e. true or false. Default value is false.\n");
}
} else if (key == "trt_cuda_graph_enable") {
if (value == "true" || value == "True") {
trt_cuda_graph_enable = true;
} else if (value == "false" || value == "False") {
trt_cuda_graph_enable = false;
} else {
ORT_THROW("[ERROR] [TensorRT] The value for the key 'trt_cuda_graph_enable' should be a boolean i.e. true or false. Default value is false.\n");
}
} else {
ORT_THROW("[ERROR] [TensorRT] wrong key type entered. Choose from the following runtime key options that are available for TensorRT. ['device_id', 'trt_max_partition_iterations', 'trt_min_subgraph_size', 'trt_max_workspace_size', 'trt_fp16_enable', 'trt_int8_enable', 'trt_int8_calibration_table_name', 'trt_int8_use_native_calibration_table', 'trt_dla_enable', 'trt_dla_core', 'trt_dump_subgraphs', 'trt_engine_cache_enable', 'trt_engine_cache_path', 'trt_engine_decryption_enable', 'trt_engine_decryption_lib_path', 'trt_force_sequential_engine_build'] \n");
ORT_THROW("[ERROR] [TensorRT] wrong key type entered. Choose from the following runtime key options that are available for TensorRT. ['device_id', 'trt_max_partition_iterations', 'trt_min_subgraph_size', 'trt_max_workspace_size', 'trt_fp16_enable', 'trt_int8_enable', 'trt_int8_calibration_table_name', 'trt_int8_use_native_calibration_table', 'trt_dla_enable', 'trt_dla_core', 'trt_dump_subgraphs', 'trt_engine_cache_enable', 'trt_engine_cache_path', 'trt_engine_decryption_enable', 'trt_engine_decryption_lib_path', 'trt_force_sequential_engine_build', 'trt_cuda_graph_enable'] \n");
}
}
OrtTensorRTProviderOptions tensorrt_options;
Expand All @@ -228,6 +237,7 @@ OnnxRuntimeTestSession::OnnxRuntimeTestSession(Ort::Env& env, std::random_device
tensorrt_options.trt_engine_decryption_enable = trt_engine_decryption_enable;
tensorrt_options.trt_engine_decryption_lib_path = trt_engine_decryption_lib_path.c_str();
tensorrt_options.trt_force_sequential_engine_build = trt_force_sequential_engine_build;
tensorrt_options.trt_cuda_graph_enable = trt_cuda_graph_enable;
session_options.AppendExecutionProvider_TensorRT(tensorrt_options);

OrtCUDAProviderOptions cuda_options;
Expand Down
1 change: 1 addition & 0 deletions onnxruntime/test/providers/cpu/model_tests.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,7 @@ TEST_P(ModelTest, Run) {
nullptr,
0,
nullptr,
0,
0};
ASSERT_STATUS_OK(session_object.RegisterExecutionProvider(TensorrtExecutionProviderWithOptions(&params)));
} else {
Expand Down
1 change: 1 addition & 0 deletions onnxruntime/test/util/default_providers.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ std::unique_ptr<IExecutionProvider> DefaultTensorrtExecutionProvider() {
nullptr,
0,
nullptr,
0,
0};
if (auto factory = CreateExecutionProviderFactory_Tensorrt(&params))
return factory->CreateProvider();
Expand Down