Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pinning for pyarrow in wheels #17018

Merged
merged 3 commits into from
Oct 9, 2024

Conversation

vyasr
Copy link
Contributor

@vyasr vyasr commented Oct 8, 2024

Description

We have recently observed a number of seg faults in our Python tests. From some investigation, the error comes from the import of pyarrow loading the bundled libarrow.so, and in particular when that library runs a jemalloc function background_thread_entry. We have observed similar (but not identical) errors in the past that have to do with as-yet unsolved problems in the way that arrow handles multi-threaded environments. The error is currently only observed on arm runners and with pyarrow 17.0.0. In my tests the error is highly sensitive to everything from import order to unrelated code segments, suggesting a race condition, some form of memory corruption, or perhaps symbol resolution errors at runtime. As a result, I have had limited success in drilling down further into specific causes, especially since attempts to rebuild libarrow.so also squash the error and I therefore cannot use debug symbols. From some offline discussion we decided that avoiding the problematic version is a sufficient fix for now. Due to the sensitivity, I am simply skipping 17.0.0 in this PR. I suspect that future builds of pyarrow will also usually not exhibit this bug (although it may recur occasionally on specific versions of pyarrow). Therefore, rather than lowering the upper bound I would prefer to allow us to float and see if and when this problem reappears. Since our DFG+RBB combination for wheel builds does not yet support any matrix entry other than cuda, I'm using environment markers to specify the constraint rather than a matrix entry in dependencies.yaml.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@vyasr vyasr added bug Something isn't working non-breaking Non-breaking change labels Oct 8, 2024
@vyasr vyasr self-assigned this Oct 8, 2024
@vyasr vyasr requested a review from a team as a code owner October 8, 2024 19:31
@vyasr vyasr requested a review from KyleFromNVIDIA October 8, 2024 19:31
@github-actions github-actions bot added Python Affects Python cuDF API. pylibcudf Issues specific to the pylibcudf package labels Oct 8, 2024
@vyasr vyasr requested a review from a team as a code owner October 8, 2024 22:39
@github-actions github-actions bot added the cudf.polars Issues specific to cudf.polars label Oct 8, 2024
@vyasr
Copy link
Contributor Author

vyasr commented Oct 9, 2024

/merge

@rapids-bot rapids-bot bot merged commit ded4dd2 into rapidsai:branch-24.12 Oct 9, 2024
104 checks passed
@vyasr vyasr deleted the fix/pin_arm_pyarrow branch October 9, 2024 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cudf.polars Issues specific to cudf.polars non-breaking Non-breaking change pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants