Skip to content

Workaround intermittent segfault in libcudf_streaming UCX tests - #24703

Closed
mhaseeb123 wants to merge 1 commit into
NVIDIA:mainfrom
mhaseeb123:fix-streaming-ucxx-shuffler-segv
Closed

mhaseeb123 wants to merge 1 commit into
NVIDIA:mainfrom
mhaseeb123:fix-streaming-ucxx-shuffler-segv

Conversation

@mhaseeb123

@mhaseeb123 mhaseeb123 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR disables UCX dmabuf (UCX_CUDA_COPY_DMABUF=no, unless already set) in the libcudf_streaming UCXX test environment. Since UCX 1.22, memory type detection exports a dmabuf for every new device buffer, which intermittently segfaults inside cuMemGetHandleForAddressRange on the rapidsmpf progress thread (seen mostly on arm64 CI). The tests don't need dmabuf.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Oct 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the libcudf Affects libcudf (C++/CUDA) code. label Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e21d857

@mhaseeb123 mhaseeb123 added bug Something isn't working 2 - In Progress Currently a work in progress non-breaking Non-breaking change labels Oct 19, 2026
@mhaseeb123 mhaseeb123 changed the title Fix intermittent segfault in libcudf_streaming UCXX tests Investigate intermittent segfault in libcudf_streaming UCXX tests Oct 20, 2026
@mhaseeb123 mhaseeb123 added 5 - Ready to Merge Testing and reviews complete, ready to merge DO NOT MERGE Hold off on merging; see PR for details and removed 2 - In Progress Currently a work in progress 5 - Ready to Merge Testing and reviews complete, ready to merge labels Oct 20, 2026
@mhaseeb123
mhaseeb123 force-pushed the fix-streaming-ucxx-shuffler-segv branch from e21d857 to c42f774 Compare October 20, 2026 01:16
@copy-pr-bot

copy-pr-bot Bot commented Oct 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@mhaseeb123

Copy link
Copy Markdown
Contributor Author

/ok to test c42f774

@mhaseeb123 mhaseeb123 changed the title Investigate intermittent segfault in libcudf_streaming UCXX tests Fix intermittent segfault in libcudf_streaming UCXX tests Oct 20, 2026
@mhaseeb123
mhaseeb123 marked this pull request as ready for review October 20, 2026 01:16
@mhaseeb123
mhaseeb123 requested a review from a team as a code owner October 20, 2026 01:16
@mhaseeb123
mhaseeb123 requested a review from madsbk October 20, 2026 01:16
@mhaseeb123
mhaseeb123 marked this pull request as draft October 20, 2026 01:16
@mhaseeb123
mhaseeb123 marked this pull request as ready for review October 20, 2026 01:17
@mhaseeb123 mhaseeb123 added 3 - Ready for Review Ready for review by team and removed DO NOT MERGE Hold off on merging; see PR for details labels Oct 20, 2026
@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/cudf/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: e71f8fe9-da3a-46c1-831a-c699c5fa1dcd
📥 Commits

Reviewing files that changed from the base of the PR and between 87d2d78 and c42f774.

📒 Files selected for processing: 1
  • cpp/libcudf_streaming/tests/main/ucxx.cpp

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

📝 Summary

Summary by CodeRabbit

  • Tests
    • Updated test setup to disable UCX CUDA dmabuf exports by default.
    • Preserves any existing user-provided configuration for this setting.

The UCXX test environment now sets UCX_CUDA_COPY_DMABUF to "no" when it is not already defined. The setting occurs before CUDA and MPI initialization.

Changes: UCX test environment setup

Layer / File(s) Summary
Set the UCX environment default
cpp/libcudf_streaming/tests/main/ucxx.cpp
UCXXEnvironment::SetUp sets UCX_CUDA_COPY_DMABUF to "no" only when the variable is not already defined. The file includes <cstdlib> for environment-variable handling.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~5 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to c42f7

No actionable merge-blocking risk remains.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description check Passed The description explains the UCX dmabuf change, the intermittent segmentation fault, and the affected test environment.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly identifies the main change: mitigating an intermittent segmentation fault in the libcudf_streaming UCX tests.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@bdice bdice left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a bug in UCX. We ought to isolate and report it.

I had a different solution in 688c279. What do you think about that? I think I prefer setting env vars in CI scripts rather than test harnesses unless there are benefits I’m overlooking.

@mhaseeb123

mhaseeb123 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

This feels like a bug in UCX. We ought to isolate and report it.

🎯 The env var export here is a workaround, not a fix.

I had a different solution in 688c279. What do you think about that? I think I prefer setting env vars in CI scripts rather than test harnesses unless there are benefits I’m overlooking.

That works. Let me know if I should close this PR (in favor of another) or just adapt your commit in here.

@mhaseeb123 mhaseeb123 changed the title Fix intermittent segfault in libcudf_streaming UCXX tests Workaround intermittent segfault in libcudf_streaming UCX tests Oct 20, 2026
@mhaseeb123

Copy link
Copy Markdown
Contributor Author

See: rapidsai/ucxx#784

@mhaseeb123 mhaseeb123 closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3 - Ready for Review Ready for review by team bug Something isn't working libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants