Add Macro to remove asserts in hip kernel

[jerryyin]

HIP does not currently support assert() calls inside kernels. Instead of removing assert calls which are all over our code base, This pull request pick the gpu_device_functions.h that is the dependency to hip kernels to add the macro. This way, "assert()" macro is redefined to nop, and dependency kernels will pick up the re-definition in the build process. The macro is used to convert assert() calls to nop, so that we can create tf-rocm debug builds.

The reason why the issue does not exist in a release build is due to asserts being expanded to NOP in release build. This is done by having NDEBUG set.

After this update together with rocPRIM PR 51, a debug build of TensorFlow-Rocm can be achieved by:
$ bazel build --config=rocm --copt=-g --compilation_mode=dbg --strip=never //tensorflow/tools/pip_package_build_pip_package

[ROCm-Apps-Test]

Can one of the admins verify this patch?

[ROCm-Apps-Test]

Can one of the admins verify this patch?

[ROCm-Apps-Test]

Can one of the admins verify this patch?

[deven-amd]

@parallelo, can you add @jerryyin to the cool-kids list so that his PRs can trigger CI automatically :)

thanks

[whchung]

jenkins: ok to test

[deven-amd]

@jerryyin

interesting solution to the assert problem. I had run into the same issue when I was trying to do a debug build. I remember running trying out something similar (i.e. redefining assert to be a no-op, though not in the same file as you), but it never completely worked for me.

eventually I had taken the nuclear option, and added +define+NDEBUG as "always on" compile option (just like TENSORFLOW_USE_ROCM) to get around the assert issue. At that point I had run into the rocprim issue (which you have identified the fix for :) ). I never checked in the change to disable asserts since I could not get the debug build to work.

[parallelo]

FYI - @jerryyin has been added to the CI admin list

[jerryyin]

@jerryyin

interesting solution to the assert problem. I had run into the same issue when I was trying to do a debug build. I remember running trying out something similar (i.e. redefining assert to be a no-op, though not in the same file as you), but it never completely worked for me.

eventually I had taken the nuclear option, and added +define+NDEBUG as "always on" compile option (just like TENSORFLOW_USE_ROCM) to get around the assert issue. At that point I had run into the rocprim issue (which you have identified the fix for :) ). I never checked in the change to disable asserts since I could not get the debug build to work.

Thanks! I got lucky in chasing down some of the HIP kernel call chains and down to this file. When I add the re-definition, all the linker issue just immediately go away at once. For me, I have to address the issue in building a debug build so that I can do operator level debugging. (The workaround to add symbols only to stream executor isn't enough to me)

[whchung]

LGTM

[whchung]

@jerryyin Thanks for submitting this PR, as well as ROCm/rocPRIM#51 . Should the PR in rocPRIM gets merged, please submit another PR to revise workspace.bzl to help move the pointer for rocPRIM

[jerryyin]

@jerryyin Thanks for submitting this PR, as well as ROCmSoftwarePlatform/rocPRIM#51 . Should the PR in rocPRIM gets merged, please submit another PR to revise workspace.bzl to help move the pointer for rocPRIM

@whchung Will do!