Fix dereference operations for union type in Hive Connector

[groupcache4321]

Description

Support dereferencing using field names such as "field0", "field1", "tag" for Hive's Union Type.

Additional context and related issues

#15017,
#3483,
e071da4

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive Connector
*  Fix queries referencing nested fields in union typed columns. ({issue}`15278`)

[phd3]

I'm not sure I understand this part ... Can you please elaborate what's the issue here?

[groupcache4321]

The nested, nested column I created: union<struct>> get interpreted as union<> for Avro type.

[groupcache4321]

this issue is unrelated to the fix implementation. Due to this I can't test nested nested case with Avro.

[phd3]

I see. since union types typically have multiple fields, curious if we have something like union<int, <struct<int, int>> , what is it read as?

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: yluan.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[phd3]

some comments, but overall looks good.
please look at build failure and also git author details - currently it's showing cla not signed.

[phd3]

I see. since union types typically have multiple fields, curious if we have something like union<int, <struct<int, int>> , what is it read as?

[phd3]

I tried this out for avro. looks like when we've just one type within uniontype, hive tries to be smart about it in terms of table schema by removing the union but then fails insertion.

e.g. this fails

use default;
drop table complexavrobase;
create table complexavrobase (c0 uniontype<struct<a:int,b:string,c:uniontype<string,int>>>, c1 int) stored as AVRO;
insert into complexavrobase SELECT create_union(0, named_struct('a', 1, 'b', 'structval', 'c', create_union(1, 'd', 5))), 8 FROM (select 'ignore') ignore;

But if we actually have a union of more than one type - then it succeeds.

use default;
drop table complexavro;
create table complexavro (c0 uniontype<int,string,struct<a:int,b:string,c:uniontype<string,int>>>, c1 int) stored as AVRO;
insert into complexavro SELECT create_union(0, 100,  'first', named_struct('a', 1, 'b', 'structval', 'c', create_union(0, 'd', 5))), 8 from (select 'ignore') ignore;
insert into complexavro SELECT create_union(1, 100,  'first', named_struct('a', 1, 'b', 'structval', 'c', create_union(0, 'd', 5))), 8 from (select 'ignore') ignore;
SELECT * FROM complexavro;
insert into complexavro SELECT create_union(2, 100,  'first', named_struct('a', 1, 'b', 'structval', 'c', create_union(0, 'd', 5))), 8 from (select 'ignore') ignore;

However, hive fails to read back results after this. But IMO Trino should still be able to read back. Could you please modify the test so that we've coverage for both ?

[groupcache4321]

e.g. this fails

I would keep the original test as well, since there is no constraint specifying union type must have more than one type.

[groupcache4321]

pushed, please take a further look!

[groupcache4321]

Summary of bugs we found using Hive inserting union columns for AVRO format table:

1. When inserting values into nested Union<Struct<>> whereas Union only has one field that is struct, we found Union level were ignored. I.e. when "show create table", the schema is correct, but when inserting, Hive complains that schema mismatch.
2. When inserting values into nested Union<Struct<Union<int, string >>, we found we have to insert using create_union(tagId, string, int) instead of create_union(tagId, int string)