Repository navigation
BUILD: Add configurable CUDA device debug info flag - #800
michal-shalev wants to merge 5 commits into
Conversation
Signed-off-by: Michal Shalev <mshalev@nvidia.com>
|
👋 Hi michal-shalev! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
Signed-off-by: Michal Shalev <mshalev@nvidia.com>
| if not get_option('enable_cuda_debug') | ||
| add_project_arguments('-dopt=on', language: 'cuda') | ||
| endif |
There was a problem hiding this comment.
| if not get_option('enable_cuda_debug') | |
| add_project_arguments('-dopt=on', language: 'cuda') | |
| endif | |
| if get_option('enable_cuda_debug') | |
| add_project_arguments('-G', language: 'cuda') | |
| endif |
When -G is not specified, -dopt=on is implicit.
There was a problem hiding this comment.
I checked here:
https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/ 4.2.3.3
And also compiled with and without add_project_arguments('-dopt=on', language: 'cuda') to verify.
There was a problem hiding this comment.
but as @rakhmets mentioned
4.2.3.12. --dopt kind (-dopt)
Enable device code optimization. When specified along with -G, enables limited debug information generation for optimized device code (currently, only line number information). When -G is not specified, -dopt=on is implicit.
So, should we just add -G for debug mode and do not add it for release?
There was a problem hiding this comment.
I think the issue is that -G is added by meson.
So, in debug build it will be -G. But I think the purpose of the PR is:
- debug build
-G -dopt=on - debug + cuda debug:
-G
There was a problem hiding this comment.
why do we need a separate debug option for cuda? I'd just make sure we have -G for debug and we do not have it for release
There was a problem hiding this comment.
debug is the default build type, and it's not documented anywhere, out of the box users can get very bad performance because of this "-G" flag
Signed-off-by: Michal Shalev <mshalev@nvidia.com>
| option('cudapath_inc', type: 'string', value: '', description: 'Include path for CUDA') | ||
| option('cudapath_lib', type: 'string', value: '', description: 'Library path for CUDA') | ||
| option('cudapath_stub', type: 'string', value: '', description: 'Extra Stub path for CUDA') | ||
| option('enable_cuda_debug', type: 'boolean', value: false, description: 'Enable CUDA debug mode (disables -dopt=on optimization)') |
There was a problem hiding this comment.
Could we tie this to the buildtype (debug/release) instead of adding a new option?
There was a problem hiding this comment.
yes and maybe we need to build NIXL in release mode by default
|
Closing this PR following #869 |
This branch was previously deployed
5 participants
What?
Add configurable CUDA device debug info flag with
cuda_enable_debugmeson option.Why?
Meson automatically adds the -G flag for CUDA debug builds, which significantly degrades performance. The -G flag generates device debug info and turns off optimizations unless -dopt=on is specified.
Reference: https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/ 4.2.3.3
How?