[SPARK-40376][PYTHON] Avoid Numpy deprecation warning

[ELHoussineT]

What changes were proposed in this pull request?

Use bool instead of np.bool as np.bool will be deprecated (see: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations)

Using np.bool generates this warning:

UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.
3070E                     `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
3071E                   Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

Why are the changes needed?

Deprecation soon: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations.

Does this PR introduce any user-facing change?

The warning will be suppressed

How was this patch tested?

Existing tests should suffice.

[srowen]

Seems fine, though is this just an alias for previous versions of numpy that are currently supported too?

[ELHoussineT]

Seems fine, though is this just an alias for previous versions of numpy that are currently supported too?

Correct.

[itholic]

Can we add a [PYTHON] tag for the title ?

Also check the https://spark.apache.org/contributing.html out when you find some time.

[itholic]

Would you check the "Workflow run detection failed" in https://github.com/apache/spark/pull/37817/checks?check_run_id=8226916282 for enabling Github Actions ??

[itholic]

I think How was this patch tested? in the PR description should contain how do we verify this patch within Apache Spark code base.

In this case, we can just simply mention like: Using the existing test for example.

[itholic]

The PR description usually started from "What changes were proposed in this pull request?"

So, can we put the description

"""
Using np.bool generates this warning:

UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.
3070E                     `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
3071E                   Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

"""

into "What changes were proposed in this pull request?" ??

e.g.

What changes were proposed in this pull request?

Using np.bool generates this warning:

UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.
3070E                     `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
3071E                   Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

Use bool instead of np.bool as np.bool will be deprecated (see: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations)

Why are the changes needed?

...

[itholic]

Looks fine otherwise.

Thanks for the your first contribution to Apache Spark!

[ELHoussineT]

@itholic

Can we add a [PYTHON] tag for the title ?

Done

Also check the https://spark.apache.org/contributing.html out when you find some time.

I am sorry, you are right.

Would you check the "Workflow run detection failed" in https://github.com/apache/spark/pull/37817/checks?check_run_id=8226916282 for enabling Github Actions ??

I did that and the build in my fork went through: https://github.com/ELHoussineT/spark/actions/workflows/build_main.yml
But the actions in the PR are still red, thoughts?

I think How was this patch tested? in the PR description should contain how do we verify this patch within Apache Spark code base.

Updated.

So, can we put the description into "What changes were proposed in this pull request?" ??

Done

Thanks for the your first contribution to Apache Spark!

Its a tiny one! You're welcome :)

[srowen]

Hm, try pushing an empty commit?

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Oh, remove the type ignore comment:

annotations failed mypy checks:
python/pyspark/sql/pandas/conversion.py:298: error: unused "type: ignore" comment
Found 1 error in 1 file (checked 339 source files)

[srowen]

Ping @ELHoussineT

[ELHoussineT]

@srowen Sorry for the late reply.

Updated, let's see if it will go through.

[srowen]

Merged to master

[itholic]

Cool !

[joaoleveiga]

Hello all. Is this only going to be released in PySpark 3.4?

I looked at the branch-3.3 code and failed to see this change, if I saw correctly.

Thanks

[srowen]

It's in 3.4, not 3.3, yes: https://github.com/apache/spark/blob/v3.4.0-rc1/python/pyspark/sql/pandas/conversion.py#L301
See the JIRA

[aimtsou]

@srowen: Although this is causing an issue:

If you try to build your own docker image of Spark including pyspark while trying to be compliant with Databricks you will observe that Databricks Runtime 12.1 and 12.2(which is currently in beta), both support officially until Spark 3.3.1 (while current version is 3.3.2).

Actually all of the LTS versions in the support matrix are not EOLed and since numpy 1.20.0 was released in 01/2021, which means that most spark compliant versions carry this bug. If you try to use Pandas by using toPandas() you end up with the numpy error, consequently being blocked from upgrading your spark versions.

Is there any chance of back-porting this commit into previous pyspark versions?

[srowen]

This is just a deprecation warning, not an error, right? I don't see a particular urgency here.
I don't think this is related to Databricks, particularly, either - Databricks can do what it likes with patches, etc. It will have a runtime based on 3.4 shortly after it's released.

[aimtsou]

Hi @srowen,

Thank you for your very prompt reply.

You are not correct about the error, after 1.20.0 it creates an attribute error

attr = 'bool'

    def __getattr__(attr):
        # Warn for expired attributes, and return a dummy function
        # that always raises an exception.
        import warnings
        try:
            msg = __expired_functions__[attr]
        except KeyError:
            pass
        else:
            warnings.warn(msg, DeprecationWarning, stacklevel=2)
    
            def _expired(*args, **kwds):
                raise RuntimeError(msg)
    
            return _expired
    
        # Emit warnings for deprecated attributes
        try:
            val, msg = __deprecated_attrs__[attr]
        except KeyError:
            pass
        else:
            warnings.warn(msg, DeprecationWarning, stacklevel=2)
            return val
    
        if attr in __future_scalars__:
            # And future warnings for those that will change, but also give
            # the AttributeError
            warnings.warn(
                f"In the future `np.{attr}` will be defined as the "
                "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
    
        if attr in __former_attrs__:
>           raise AttributeError(__former_attrs__[attr])
E           AttributeError: module 'numpy' has no attribute 'bool'.
E           `np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
E           The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
E               https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

/usr/local/lib/python3.9/site-packages/numpy/__init__.py:305: AttributeError

This is the end of an error, coming after calling the function toPandas() from my tests:

/usr/local/lib/python3.9/site-packages/<my-pkg>/unit/test_case_runner.py:26: in run_test
    self.assert_df_are_equal(expected_df, actual)
/usr/local/lib/python3.9/site-packages/<my-pkg>/unit/test_case_runner.py:58: in assert_df_are_equal
    self.handler.compare_df(result, expected, config=self.compare_config)
/usr/local/lib/python3.9/site-packages/<my-pkg>/spark_test_handler.py:38: in compare_df
    actual_pd = actual.toPandas().sort_values(by=sort_columns, ignore_index=True)
/usr/local/lib/python3.9/site-packages/pyspark/sql/pandas/conversion.py:216: in toPandas
    pandas_type = PandasConversionMixin._to_corrected_pandas_type(field.dataType)
/usr/local/lib/python3.9/site-packages/pyspark/sql/pandas/conversion.py:298: in _to_corrected_pandas_type
    return np.bool  # type: ignore[attr-defined]

And the error comes from the numpy in the system which gets called by the _to_corrected_pandas_type function inside pyspark.

I agree about the comments on databricks but as shown above this does not work on Spark 3.3.1 without taking into consideration if you want to be compliant with Databricks.

[srowen]

Well, I think we're talking about numpy 1.20 here, not >1.20. You're correct that you therefore would not use the latest versions of numpy with Spark 3.3, but would work with 3.4. If that presents a significant problem during the lifetime of Spark 3.3, sure I think that's a decent argument to back-port. Do you know what version of numpy actually removed this ? if it not a recent removal, yeah I think we should back port this simple change

[aimtsou]

Yes we agree that users can limit their numpy system installation to < 1.20.0, if they use Spark 3.3

I will have to check and test the different versions but I believe according to the notes from numpy, should be from the numpy 1.20.0 1,2. I will have to verify it to be sure though.

Well numpy 1.20.0 was released in 01/2021 which makes it 2 year old but the final decision is up to you.

[srowen]

Looks like 1.22 removed it actually. That's still not recent. Yeah I think this is worth back porting

[srowen]

Also merged to 3.3

[aimtsou]

Thank you @srowen, really appreciated

[joaoleveiga]

Also merged to 3.3

Thank you so much! Here I was assuming I would pick up this thread on monday but you delivered it 😄

Cheers