Skip to content

[Doc] Performance tuning of convolution heavy models with the CUDA EP #9916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pranavsharma merged 2 commits into from
Dec 3, 2021
Merged

[Doc] Performance tuning of convolution heavy models with the CUDA EP #9916

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d276e43
Initial commit
hariharans29 Dec 2, 2021
6bde7f6
More changes
hariharans29 Dec 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions docs/performance/tune-performance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,7 @@ import onnxruntime as rt

so = rt.SessionOptions()
so.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL
session = rt.InferenceSession(model, sess_options=so)
session.set_providers(['CUDAExecutionProvider'])
session = rt.InferenceSession(model, sess_options=so, providers=['CUDAExecutionProvider'])
Copy link
Member Author

@hariharans29 hariharans29 Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting 1.10, providers needs to be explicitly provided at session construction time

```

## Which Execution Provider will provide the best performance?
Expand Down Expand Up @@ -281,6 +280,24 @@ https://github.com/microsoft/onnxruntime/blob/master/docs/python/inference/api_s
* C#
https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.Tests/OrtIoBindingAllocationTest.cs

### Convolution heavy models and the CUDA EP
ORT leverages CuDNN for convolution operations and the first step in this process is to determine which "optimal" convolution algorithm to use while performing the convolution operation for the given input configuration (input shape, filter shape, etc.) in each `Conv` node . This sub-step involves querying CuDNN for a "workspace" memory size and have this allocated so that CuDNN can use this auxiliary memory while determining the "optimal" convolution algorithm to use. By default, ORT clamps the workspace size to 32 MB which may lead to a sub-optimal convolution algorithm getting picked by CuDNN. To allow ORT to allocate the maximum possible workspace as determined by CuDNN, a provider option needs to get set (as shown below). Keep in mind that using this flag may increase the peak memory usage by a factor (sometimes a few GBs) but this does help CuDNN pick the best convolution algorithm for the given input. We have found that this is an important flag to use while using an fp16 model as this allows CuDNN to pick tensor core algorithms for the convolution operations (if the hardware supports tensor core operations). This flag may or may not result in performance gains for other data types (`float` and `double`).

* Python
```
providers = [("CUDAExecutionProvider", {"cudnn_conv_use_max_workspace": '1'})]
sess_options = ort.SessionOptions()
sess = ort.InferenceSession("my_conv_heavy_fp16_model.onnx", sess_options = sess_options, providers=providers)
options = sess.get_provider_options()
cuda_options = options['CUDAExecutionProvider']
cuda_options['cudnn_conv_use_max_workspace'] = '1'
sess.set_providers(['CUDAExecutionProvider'], [cuda_options])
```
* C/C++
Support for this provider option will be added in upcoming releases.

* C#
Support for this provider option will be added in upcoming releases.

## Troubleshooting performance issues

Expand Down