[GOBBLIN-1774] Util for detecting non optional uniontype columns based on Hive Table metadata by homatthew · Pull Request #3632 · apache/gobblin

homatthew · 2023-01-31T22:32:48Z

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-1774] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1774

Description

Here are some details about my PR, including screenshots (if applicable):

Problem Statement:

Non-optional union: A union type with more than one non-null option type
e.g. [“int”, “string”] or [“null”, “int”, “string”]
Alternate phrasing:
A schema where there 's a union type of >= 2 branches which are non null.
[int, null] - no
[int, string] - yes
[int, string, null] yes

Iceberg, unlike Hive, does not support non optional uniontypes (i.e. uniontypes that can be null). These are supported as struct types instead.
- Allow reading non-optional unions as struct of optional fields iceberg#189
- Add support for union record type iceberg#1046
This util will be used across GMIP and compaction to handle separate edge cases for non optional unions, so special workarounds are necessary to have datasets with both heterogeneous schemas (e.g. a seamless view that contains legacy schemas and new iceberg compatible schemas that use struct).
Iceberg uses a similar pattern to Trino to represent complex uniontypes as struct. See Support reading uniontype as struct from Avro/ORC Hive tables trinodb/trino#3483

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

codecov-commenter · 2023-01-31T22:52:00Z

Codecov Report

Merging #3632 (12318bf) into master (667d797) will increase coverage by 0.04%.
The diff coverage is 83.33%.

@@             Coverage Diff              @@
##             master    #3632      +/-   ##
============================================
+ Coverage     46.56%   46.61%   +0.04%     
- Complexity    10666    10707      +41     
============================================
  Files          2133     2133              
  Lines         83541    83612      +71     
  Branches       9288     9299      +11     
============================================
+ Hits          38905    38977      +72     
  Misses        41074    41074              
+ Partials       3562     3561       -1

Impacted Files	Coverage Δ
...che/gobblin/hive/metastore/HiveMetaStoreUtils.java	`55.55% <83.33%> (+9.20%)`	⬆️
...lin/util/filesystem/FileSystemInstrumentation.java	`85.71% <0.00%> (-7.15%)`	⬇️
...service/modules/orchestration/DagManagerUtils.java	`83.47% <0.00%> (-6.09%)`	⬇️
...che/gobblin/data/management/copy/CopyableFile.java	`73.33% <0.00%> (-2.01%)`	⬇️
...in/service/modules/orchestration/Orchestrator.java	`52.73% <0.00%> (-0.88%)`	⬇️
...blin/service/monitoring/KafkaJobStatusMonitor.java	`45.98% <0.00%> (-0.29%)`	⬇️
...lin/service/monitoring/SpecStoreChangeMonitor.java	`0.00% <0.00%> (ø)`
...ervice/modules/flow/BaseFlowToJobSpecCompiler.java	`55.00% <0.00%> (ø)`
...ervice/monitoring/DagActionStoreChangeMonitor.java	`0.00% <0.00%> (ø)`
...lin/data/management/copy/ManifestBasedDataset.java	`51.78% <0.00%> (ø)`
... and 12 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

...ive-registration/src/test/java/org/apache/gobblin/hive/metastore/HiveMetaStoreUtilsTest.java