Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] RecordBatchReader cast() errors introduced in pyarrow 16 #41884

Closed
davlee1972 opened this issue May 29, 2024 · 2 comments
Closed

[Python] RecordBatchReader cast() errors introduced in pyarrow 16 #41884

davlee1972 opened this issue May 29, 2024 · 2 comments

Comments

@davlee1972
Copy link

Describe the bug, including details regarding any error messages, version, and platform.

RecordBatchReader.cast() was added in pyarrow 16..

Error: pyarrow.lib.ArrowTypeError: Field 0 cannot be cast from date32[day] to date32[day]

>>> import pyarrow as pa
>>> import datetime
>>>
>>> t = pa.Table.from_pylist([{"abc": datetime.date.today()}])
>>> tb = t.to_reader()
>>> tb.cast(t.schema)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow\\ipc.pxi", line 800, in pyarrow.lib.RecordBatchReader.cast
  File "pyarrow\\error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\\error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: Field 0 cannot be cast from date32[day] to date32[day]

In my use case:

I'm trying to cast the results of a database query from:
Field 0 is a string, Field 1 is a date32[day], Field 2 is a double
to
Field 0 is a string, Field 1 is a date32[day], Field 2 is a float

But I'm getting Field 1 cast error on dates..

Component(s)

Python

@davlee1972 davlee1972 changed the title RecordBatchReader cast() errors introduced in pyarrow 16 [Python] RecordBatchReader cast() errors introduced in pyarrow 16 May 29, 2024
@jorisvandenbossche jorisvandenbossche added this to the 17.0.0 milestone Jun 7, 2024
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Jun 11, 2024
@jorisvandenbossche
Copy link
Member

@davlee1972 thanks for the report! It seems the cast from date32 to date32 is not explicitly included in the table of cast kernels, and therefore the CanCast fails.

Working on a fix in #42098

jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Jun 13, 2024
jorisvandenbossche added a commit that referenced this issue Jun 18, 2024
…qual schema for all types (#42098)

### Rationale for this change

The cast from `date32` to `date32` is not included in the table of cast functions.

### What changes are included in this PR?

Ensure we don't error if the types are equal in the initialization of the `CastingRecordBatchReader`

### Are these changes tested?

Yes

### Are there any user-facing changes?

No
* GitHub Issue: #41884

Lead-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Felipe Oliveira Carvalho <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
@jorisvandenbossche
Copy link
Member

Issue resolved by pull request 42098
#42098

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants