Skip to content

Inconsistent and incorrect struct field coercion #14396

@alamb

Description

@alamb

Describe the bug

When coercing structs with different types DataFusion is inconsistent in its behavior. Sometimes it errors and in other times it is inconsistent

To Reproduce

Consider two structs with two fields

create table t as values
(
 { 'foo': 'baz' }, 
 { 'xxx': arrow_cast('blarg', 'Utf8View') } -- column4 has List of Struct w/ Utf8View and a different field name
);

They can be coerced to the same field using UNION,

> select column1 from t UNION ALL select column2 from t;
+-------------+
| column1     |
+-------------+
| {c0: baz}   |
| {c0: blarg} |
+-------------+
2 row(s) fetched.
Elapsed 0.018 seconds.

Likewise with CASE they are coerced to the same value

> select CASE WHEN 1=2 THEN column1 ELSE column2 END from t ;
+-----------------------------------------------------------------+
| CASE WHEN Int64(1) = Int64(2) THEN t.column1 ELSE t.column2 END |
+-----------------------------------------------------------------+
| {c0: blarg}                                                     |
+-----------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.010 seconds.

Expected behavior

I believe the field names will be different after

However I expect all of the above queries to fail as the field names are incorrect

I expect the following cases to work:

  1. Field names are the same but in different order
  2. Field names are the same but the value types need to be coerced themselves (see case on Type Coercion fails for List with inner type struct which has large/view types #14154)

Additional context

@jayzhan211 suggests: https://github.com/apache/datafusion/pull/14384/files#r1937492704

Yes, I think type union resolution is the correct on for CASE

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions