-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
When coercing structs with different types DataFusion is inconsistent in its behavior. Sometimes it errors and in other times it is inconsistent
To Reproduce
Consider two structs with two fields
create table t as values
(
{ 'foo': 'baz' },
{ 'xxx': arrow_cast('blarg', 'Utf8View') } -- column4 has List of Struct w/ Utf8View and a different field name
);They can be coerced to the same field using UNION,
> select column1 from t UNION ALL select column2 from t;
+-------------+
| column1 |
+-------------+
| {c0: baz} |
| {c0: blarg} |
+-------------+
2 row(s) fetched.
Elapsed 0.018 seconds.
Likewise with CASE they are coerced to the same value
> select CASE WHEN 1=2 THEN column1 ELSE column2 END from t ;
+-----------------------------------------------------------------+
| CASE WHEN Int64(1) = Int64(2) THEN t.column1 ELSE t.column2 END |
+-----------------------------------------------------------------+
| {c0: blarg} |
+-----------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.010 seconds.Expected behavior
I believe the field names will be different after
Casecoercion of Structs loses field names #14383- Type Coercion fails for List with inner type struct which has large/view types #14154
However I expect all of the above queries to fail as the field names are incorrect
I expect the following cases to work:
- Field names are the same but in different order
- Field names are the same but the value types need to be coerced themselves (see case on Type Coercion fails for List with inner type struct which has large/view types #14154)
Additional context
@jayzhan211 suggests: https://github.com/apache/datafusion/pull/14384/files#r1937492704
Yes, I think type union resolution is the correct on for CASE
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working