Repository navigation
Conversation
pemeliya
marked this pull request as ready for review
August 7, 2026 12:07
Member
|
Should be realigned in light of: |
Member
Author
ouch, if that PR is not merged yet, I would go with this fix still. I mean, this is supposed to be a temporary band-aid for ROCM to unblock our CI. Once we figure out a persistent fix, we can remove that. The point is, #46965 changed several files to just forward GpuComputeCapability to the function. Yet, ROCM version from RocmComputeCapability won't be enough for us to decide if RCCL supports symmetric memory or not - so it would only help to differentiate between CUDA and ROCM runtimes. Therefore, I think we could go with a more concise fix from this PR. |
Member
|
Lets ask @PatriosTheGreat since he has the most context on this change. |
pemeliya
force-pushed
the
pemeliya/rccl_disable_symm_memory
branch
from
August 11, 2026 08:26
410be4e to
80bd7ca
Compare
runtime check skipped symm memory test for rocm Revert "skipped symm memory test for rocm" This reverts commit 30d805a. disabled commwindowregister call
pemeliya
force-pushed
the
pemeliya/rccl_disable_symm_memory
branch
from
August 11, 2026 16:33
4952877 to
3e23a78
Compare
PatriosTheGreat
approved these changes
Aug 11, 2026
Member
Author
|
After converstations with @PatriosTheGreat , having a stub RcclSymmetricMemory should suffice |
This was referenced Aug 11, 2026
copybara-service Bot
pushed a commit
that referenced
this pull request
Aug 11, 2026
Imported from GitHub PR #46895 For now, we disable symmetric memory support on ROCM platform which was enabled by default in 7a57c69 This is a hot-fix, later on we can add an RCCL version check for that. Still, symmetric memory can be enforced on ROCM with **--xla_gpu_experimental_enable_nccl_symmetric_buffers=true** XLA flag. This can be used instead of this fix: #46878 Copybara import of the project: -- 3e23a78 by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: disable rccl symm memory runtime check skipped symm memory test for rocm Revert "skipped symm memory test for rocm" This reverts commit 30d805a. disabled commwindowregister call -- ae6d9dc by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: revert -- c682659 by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: revert Merging this change closes #46895 FUTURE_COPYBARA_INTEGRATE_REVIEW=#46895 from ROCm:pemeliya/rccl_disable_symm_memory c682659 PiperOrigin-RevId: 962837161
copybara-service Bot
pushed a commit
to tensorflow/tensorflow
that referenced
this pull request
Aug 11, 2026
Imported from GitHub PR openxla/xla#46895 For now, we disable symmetric memory support on ROCM platform which was enabled by default in openxla/xla@7a57c69 This is a hot-fix, later on we can add an RCCL version check for that. Still, symmetric memory can be enforced on ROCM with **--xla_gpu_experimental_enable_nccl_symmetric_buffers=true** XLA flag. This can be used instead of this fix: openxla/xla#46878 Copybara import of the project: -- 3e23a7877e70c81fa7cf997a66e4526d2c539ccf by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: disable rccl symm memory runtime check skipped symm memory test for rocm Revert "skipped symm memory test for rocm" This reverts commit 30d805ac6fbf9e83c217936fde63973d68f829db. disabled commwindowregister call -- ae6d9dc8b2db762cde70bac3596e20144e34187f by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: revert -- c682659d74d43cfb9cfa82aa935dc7d0f00c6f5f by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: revert Merging this change closes #46895 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#46895 from ROCm:pemeliya/rccl_disable_symm_memory c682659d74d43cfb9cfa82aa935dc7d0f00c6f5f PiperOrigin-RevId: 962837161
copybara-service Bot
pushed a commit
to tensorflow/tensorflow
that referenced
this pull request
Aug 11, 2026
Imported from GitHub PR openxla/xla#46895 For now, we disable symmetric memory support on ROCM platform which was enabled by default in openxla/xla@7a57c69 This is a hot-fix, later on we can add an RCCL version check for that. Still, symmetric memory can be enforced on ROCM with **--xla_gpu_experimental_enable_nccl_symmetric_buffers=true** XLA flag. This can be used instead of this fix: openxla/xla#46878 Copybara import of the project: -- 3e23a7877e70c81fa7cf997a66e4526d2c539ccf by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: disable rccl symm memory runtime check skipped symm memory test for rocm Revert "skipped symm memory test for rocm" This reverts commit 30d805ac6fbf9e83c217936fde63973d68f829db. disabled commwindowregister call -- ae6d9dc8b2db762cde70bac3596e20144e34187f by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: revert -- c682659d74d43cfb9cfa82aa935dc7d0f00c6f5f by Pavel Emeliyanenko <pavel.emeliyanenko@amd.com>: revert Merging this change closes #46895 PiperOrigin-RevId: 962907538
copybara-service Bot
pushed a commit
to tensorflow/tensorflow
that referenced
this pull request
Aug 11, 2026
Reverts ab2db4a FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#46895 from ROCm:pemeliya/rccl_disable_symm_memory c682659d74d43cfb9cfa82aa935dc7d0f00c6f5f PiperOrigin-RevId: 949784741
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For now, we disable symmetric memory support on ROCM platform which was enabled by default in 7a57c69
This is a hot-fix, later on we can add an RCCL version check for that.
Still, symmetric memory can be enforced on ROCM with --xla_gpu_experimental_enable_nccl_symmetric_buffers=true XLA flag.
This can be used instead of this fix: #46878