Skip to content

Commit 52d52a1

Browse files
ELHoussineTsrowen
authored andcommitted
[SPARK-40376][PYTHON] Avoid Numpy deprecation warning
### What changes were proposed in this pull request? Use `bool` instead of `np.bool` as `np.bool` will be deprecated (see: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations) Using `np.bool` generates this warning: ``` UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation. 3070E `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. 3071E Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations ``` ### Why are the changes needed? Deprecation soon: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. ### Does this PR introduce _any_ user-facing change? The warning will be suppressed ### How was this patch tested? Existing tests should suffice. Closes #37817 from ELHoussineT/patch-1. Authored-by: ELHoussineT <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent 11eb66e commit 52d52a1

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

python/pyspark/sql/pandas/conversion.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -295,7 +295,7 @@ def _to_corrected_pandas_type(dt: DataType) -> Optional[Type]:
295295
elif type(dt) == DoubleType:
296296
return np.float64
297297
elif type(dt) == BooleanType:
298-
return np.bool # type: ignore[attr-defined]
298+
return bool
299299
elif type(dt) == TimestampType:
300300
return np.datetime64
301301
elif type(dt) == TimestampNTZType:

0 commit comments

Comments
 (0)