Support reading uniontype as struct from Avro/ORC Hive tables

[lxynov]

Reading uniontypes by converting them into structs. Take type
"uniontype<int, double>" as an example:

1. It will be regarded as "struct<tag int, field0 int, field1 string>"
2. Data {1: 'hello'}, {0: 312}, {1: 'world'} will be read as [1, NULL,
   'hello'], [0, 312, NULL], [1, NULL, 'world']

Writing into uniontypes remains unsupported.

(Note: support for Parquet is not added because Parquet itself doesn't support union types yet.)

Closes #1751

[electrum]

Starting the names at “field1” would be more consistent with SQL one-based numbering.

[findepi]

Add a comment explaining whyh this is storage format dependent

[lxynov]

Thanks, I've added such a comment

[lxynov]

Starting the names at “field1” would be more consistent with SQL one-based numbering.

@electrum Thanks for the suggestion, but the tag field starts at zero in Hive. If we start the names at "field1", we'll also need to make tag start at 1 to reduce confusion? But this will be inconsistent with Hive. What do you think?

[electrum]

That makes sense. Let’s keep it consistent with Hive.

[dain]

Looking good. One comment about using dictionary block instead of copying the data. Also, some union tests to AbstractTestHiveFileFormats?

[dain]

Instead of copying the data use a dictionary block.

[lxynov]

@dain Thanks for the pointer, but I didn't find a good way to handle NULLs. Is there a way to append a NULL to the raw block so that it can serve as a dictionary?

[dain]

Ah, yes. This is the same problem we had with unnest. In that case, we scanned the block for a null and if present, we used that; otherwise we copied. We can leave this for now.

[dain]

Let me know when it is updated

[lxynov]

@dain Thanks for the review! I've rebased it to the latest master.

Also, some union tests to AbstractTestHiveFileFormats?

AbstractTestHiveFileFormats uses serializeObject to build write objects so union types will be written as structs eventually. So I didn't add these tests. Only a product test TestReadUniontype was added which uses HQLs to write union types.

[dain]

Ah, yes. This is the same problem we had with unnest. In that case, we scanned the block for a null and if present, we used that; otherwise we copied. We can leave this for now.