ARROW-4582: [Python/C++] Acquire the GIL on Py_INCREF #3655
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
Would You mind adding a test for it? |
Member
Author
|
@kszucs I have no idea on how to test this. It's a race condition that occured really seldomly. |
Codecov Report
@@ Coverage Diff @@
## master #3655 +/- ##
==========================================
+ Coverage 87.76% 87.79% +0.02%
==========================================
Files 689 689
Lines 83984 84014 +30
Branches 1081 1081
==========================================
+ Hits 73712 73759 +47
+ Misses 10157 10144 -13
+ Partials 115 111 -4
Continue to review full report at Codecov.
|
Member
Author
|
Green AppVeyor build: https://ci.appveyor.com/project/xhochy/arrow/builds/22398318 |
Member
|
@xhochy nice catch!! Taking a closer look |
xhochy
added a commit
to xhochy/arrow
that referenced
this pull request
Feb 19, 2019
Minimal reproducing example:
```
import dask
import pandas as pd
import pyarrow as pa
import numpy as np
def segfault_me(df):
pa.Table.from_pandas(df, nthreads=1)
while True:
df = pd.DataFrame(
{"P": np.arange(0, 10), "L": np.arange(0, 10), "TARGET": np.arange(10, 20)}
)
dask.compute([
dask.delayed(segfault_me)(df),
dask.delayed(segfault_me)(df),
dask.delayed(segfault_me)(df),
dask.delayed(segfault_me)(df),
dask.delayed(segfault_me)(df),
])
```
Segfaults are more likely when run in AddressSanitizer or otherwise slow system with many cores. It is important that always the same df is passed into the functions.
The issue was that the reference count of the underlying NumPy array was increased at the same time by multiple threads. The decrease happend then with a GIL, so the array was sometimes destroyed while still used.
Author: Korn, Uwe <[email protected]>
Closes apache#3655 from xhochy/ARROW-4582 and squashes the following commits:
7f9838d <Korn, Uwe> docker-compose run clang-format
3d6e5ee <Korn, Uwe> ARROW-4582: Acquire the GIL on Py_INCREF
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Minimal reproducing example:
Segfaults are more likely when run in AddressSanitizer or otherwise slow system with many cores. It is important that always the same df is passed into the functions.
The issue was that the reference count of the underlying NumPy array was increased at the same time by multiple threads. The decrease happend then with a GIL, so the array was sometimes destroyed while still used.