GH-35040: [Python] Pyarrow scalar cast should use compute kernel

[danepitkin]

Rationale for this change

Scalar cast should use the computer kernel just like Arrays, instead of its own custom implementation.

Are these changes tested?

Added test cases for GH-35370, GH-34901, and GH-35040

Are there any user-facing changes?

The Scalar.cast() API is enhanced and backwards compatible.

• Closes: [C++] Change Scalar::CastTo to do safe cast by default and allow to specify cast options ? #35040

[github-actions]

• Closes: [C++] Change Scalar::CastTo to do safe cast by default and allow to specify cast options ? #35040

[github-actions]

⚠️ GitHub issue #35040 has been automatically assigned in GitHub to PR creator.

[danepitkin]

Build error is unrelated. The error exists on main (and in other PRs).

[AlenkaF]

LGTM +1

Just a small nit though: I would also add a test for casting float to int, using default safe option and an unsafe casting option. Not sure about the memory pool though.

Yes, the failing tests are not related, I have created an issue for it: #35490

[danepitkin]

LGTM +1

Just a small nit though: I would also add a test for casting float to int, using default safe option and an unsafe casting option. Not sure about the memory pool though.

Yes, the failing tests are not related, I have created an issue for it: #35490

Great idea! I will add this test.

[AlenkaF]

The failure in the MacOS build could now be corrected with a rebase.

[AlenkaF]

Thank you for the update Dane!
I think this is ready, will merge 👍

[danepitkin]

We can also close #34901 and #35370 now!

I'll take care of this.

[ursabot]

Benchmark runs are scheduled for baseline = e7a885d and contender = 053b5ee. 053b5ee is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️2.15% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.27% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.72% ⬆️0.27%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 053b5eec ec2-t3-xlarge-us-east-2
[Finished] 053b5eec test-mac-arm
[Finished] 053b5eec ursa-i9-9960x
[Finished] 053b5eec ursa-thinkcentre-m75q
[Finished] e7a885d8 ec2-t3-xlarge-us-east-2
[Finished] e7a885d8 test-mac-arm
[Finished] e7a885d8 ursa-i9-9960x
[Finished] e7a885d8 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[mroeschke]

xref #36677

I don't think the new safe keyword fully covers past cases where numeric scalars could be cast to temporal scalars

>>> import pyarrow as pa
>>> pa.scalar(1.1).cast(pa.time32("s"))
<pyarrow.Time32Scalar: datetime.time(0, 0, 1)>
>>> pa.__version__
'12.0.1'

>>> import pyarrow as pa
>>> pa.scalar(1.1).cast(pa.time32("s"), safe=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/scalar.pxi", line 91, in pyarrow.lib.Scalar.cast
  File "/opt/miniconda3/envs/pyarrow-pandas-dev/lib/python3.11/site-packages/pyarrow/compute.py", line 403, in cast
    return call_function("cast", [arr], options, memory_pool)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_compute.pyx", line 572, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 367, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from double to time32 using function cast_time32
>>> pa.__version__
'13.0.0.dev497'