Separate classes under hive legacy package to new hivelink module #87

funcheetah · 2022-01-21T23:55:41Z

This PR separates classes under hive legacy package to new hivelink module

wmoustafa · 2022-01-25T18:26:05Z

build.gradle

  }
 }

+project(':iceberg-hiveberg') {


Are we modifying the root build.gradle? Why?

Could you also clarify where these dependencies are copied from?

We are moving classes from legacy package to new module here as demo. This modification is needed to add a new module. If we move them to a new code repo, this change can be removed. All the dependencies are from hive-metastore compileOnly and testCompile dependencies, which are necessary and cannot be pulled by pulling hive-metastore module

wmoustafa · 2022-01-25T18:28:57Z

build.gradle

      compile project(':iceberg-arrow')
      compile project(':iceberg-hive-metastore')
      compile project(':iceberg-spark')
+      compile project(':iceberg-hiveberg')


Is this an example of a circular dependency?

This is not. This is needed since LegacyHiveTable used in Reader and SparkBatchScan is moved to iceberg-hiveberg module.

wmoustafa

Thanks @funcheetah for this PR. This is a great start.
Summary of offline discussion:

Most of the remaining Iceberg changes are upgrading access modifiers (which is reasonable since we are extending the classes).
Right now the dependency direction is from iceberg-spark to hiveberg. Assuming that direction is somewhat risky since the dependency should be the other way around. The only reason for having it this right now is that Reader and SparkBatchScan classes contain references to classes in hiveberg but there should be a way to overcome that by extending those classes and their calling chain (classes that instantiate them) up to a an SPI class, or converting them to SPI. @funcheetah to investigate that. Reader has been shown to be a simple example of extending since it called from the IcebergSource class, which implements a Spark SPI.
We should develop a diff metric that reflects how close the code base is after each change to upstream Iceberg.

wmoustafa · 2022-01-25T18:29:52Z

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogs.java

    return CATALOG_CACHE.get(metastoreUri, uri -> new HiveCatalog(conf));
  }

-  public static HiveCatalog loadLegacyCatalog(Configuration conf) {


Is this a backward incompatible change? Do you have a plan on how to roll it out with Spark?

This is a backward incompatible change. It is not used directly from spark but from a wrapper layer. We can change the wrapper layer to accommodate this change.

This reverts commit 2e8b743.

wmoustafa

Given the package name change, could you please go through the deployment scenario and make sure deployment will be smooth for both Spark 2.3 and Spark 3.1? cc: @ljfgem since he considered similar backward compatibility issues in Coral.

wmoustafa · 2022-03-04T11:15:51Z

hivelink/src/test/java/org/apache/iceberg/hivelink/ScriptRunner.java

+package org.apache.iceberg.hivelink;
+


This likely should be com.linkedin.hivelink.

I agree. As it is still a module in iceberg, we can keep it for now and change it to com.linkedin.hivelink once moved to a new repo.

funcheetah · 2022-03-07T22:37:38Z

Given the package name change, could you please go through the deployment scenario and make sure deployment will be smooth for both Spark 2.3 and Spark 3.1? cc: @ljfgem since he considered similar backward compatibility issues in Coral.

I think it should be fine as the dependencies change are captured on our wrapper layer.

…nkedin#87) * separate class under legacy to new hiveberg module * fix build * remove hiveberg dependency in iceberg-spark2 module * Revert "remove hiveberg dependency in iceberg-spark2 module" This reverts commit 2e8b743. * rename hiveberg module to hivelink Co-authored-by: Wenye Zhang <[email protected]>

* Hive Catalog: Add a hive catalog that does not override existing Hive metadata (#10) Add custom hive catalog to not override existing Hive metadata Fail early with a proper exception if the metadata file is not existing Simplify CustomHiveCatalog (#22) * Shading: Add a iceberg-runtime shaded module (#12) * ORC: Add test for reading files without Iceberg IDs (#16) * Hive Metadata Scan: Support reading tables with only Hive metadata (#23, #24, #25, #26) - Support for non string partition columns (#24) - Support for Hive tables without avro.schema.literal (#25) - Hive Metadata Scan: Notify ScanEvent listeners on planning (#35) - Hive Metadata Scan: Do not use table snapshot summary for estimating statistics (#37) - Hive Metadata Scan: Return empty statistics (#49) - Hive Metadata Scan: Do not throw an exception on dangling partitions; log warning message (#50) - Hive Metadata Scan: Fix pushdown of non-partition predicates within NOT (#51) Co-authored-by: Ratandeep Ratti <[email protected]> Co-authored-by: Kuai Yu <[email protected]> Co-authored-by: Walaa Eldin Moustafa <[email protected]> * Row level filtering: Allow table scans to pass a row level filter for ORC files - ORC: Support NameMapping with row-level filtering (#53) * Hive: Made Predicate Pushdown dynamic based on the Hive Version * Hive: Fix uppercase bug and determine catalog from table properties (#38) * Hive: Return lowercase fieldname from IcebergRecordStructField * Hive: Determine catalog from table property * Hive: Fix schema not forwarded to SerDe on MR jobs (#45) (#47) * Hive: Use Hive table location in HiveIcebergSplit * Hive: Fix schema not passed to Serde * Hive: Refactor tests for tables with unqualified location URI Co-authored-by: Shardul Mahadik <[email protected]> * Hive Metadata Scan: Support case insensitive name mapping (#52) * Hive Metadata Scan: Merge Hive and Avro schemas to fix datatype inconsistencies (#57) Hive Metadata Scan: Fix Hive primitive to Avro logical type conversion (#58) Hive Metadata Scan: Fix support for Hive timestamp type (#61) Co-authored-by: Raymond Zhang <[email protected]> Co-authored-by: Shardul Mahadik <[email protected]> Fix HasDuplicateLowercaseColumnNames's visit method to use a new visi… (#67) * Fix HasDuplicateLowercaseColumnNames's visit method to use a new visitor instance every time * Trigger CI (cherry picked from commit b90e838) * Stop using serdeToFileFormat to unblock formats other than Avro or Orc (#64) * Stop using serdeToFileFormat to unblock formats other than Avro or Orc * Fix style check * Do not delete metadata location when HMS has been successfully updated (#68) (cherry picked from commit 766407e) * Support reading Avro complex union types (#73) Co-authored-by: Wenye Zhang <[email protected]> * [#2039] Support default value semantic for AVRO (#75) (cherry picked from commit c18f4c4) * Support hive non string partition cols (#78) * Support non-string hive type partition columns in LegacyHiveTableScan * Leverage eval against partition filter expression to filter non-string columns * Support default value read for ORC format in spark (#76) * Support default value read for ORC format in spark * Refactor common code for ReadBuilder for both non-vectorized and vectorized read * Fix code style issue * Add special handling of ROW_POSITION metadata column * Add corner case check for partition field * Use BaseDataReader.convertConstant to convert constants, and expand its functionality to support nested-type contants such as array/map/struct * Support nested type default value for vectorized read * Support deeply nested type default value for vectorized read * Support reading ORC complex union types (#74) * Support reading orc complex union types * add more tests * support union in VectorizedSparkOrcReaders and improve tests * support union in VectorizedSparkOrcReaders and improve tests - continued * fix checkstyle Co-authored-by: Wenye Zhang <[email protected]> * Support avro.schema.literal/hive union types in Hive legacy table to Iceberg conversion (#80) * Fix ORC schema visitors to support reading ORC files with deeply nest… (#81) * Fix ORC schema visitors to support reading ORC files with deeply nested union type schema * Added test for vectorized read * Disable avro validation for default values Co-authored-by: Shenoda Guirguis <[email protected]> * Fix spark avro reader reading union schema data (#83) * Fix spark avro reader to read correctly structured nested data values * Make sure field-id mapping is correctly maintained given arbitrary nested schema that contains union * Avro: Change union read schema from hive to trino (#84) * [LI] Avro: Refactor union-to-struct schema - Part 1. changes to support reading Avro * ORC: Change union read schema from hive to trino (#85) * [LI] ORC: Refactor union-to-struct schema - Part 2. changes to support reading ORC * Change Hive type to Iceberg type conversion for union * Recorder hive table properties to align the avro.schema.literal placement contract (#86) * [#2039] Support default value semantic for AVRO (cherry picked from commit c18f4c4) * reverting commits 2c59857 and f362aed (#88) Co-authored-by: Shenoda Guirguis <[email protected]> * logically patching PR 2328 on HiveMetadataPreservingTableOperations * Support timestamp as partition type (#91) * Support timestamp in partition types * Address comment * Separate classes under hive legacy package to new hivelink module (#87) * separate class under legacy to new hiveberg module * fix build * remove hiveberg dependency in iceberg-spark2 module * Revert "remove hiveberg dependency in iceberg-spark2 module" This reverts commit 2e8b743. * rename hiveberg module to hivelink Co-authored-by: Wenye Zhang <[email protected]> * [LI] Align default value validation align with avro semantics in terms of nullable (nested) fields (#92) * Align default value validation align with avro semantics in terms of nullable (nested) fields * Allow setting null as default value for nested fields in record default * [LI][Spark][Avro] read avro union using decoder instead of directly returning v… (#94) * [LI][Spark] read avro union using decoder instead of directly returning value * Add a comment for the schema * Improve the logging when the deserailzed index is invalid to read the symbol from enum (#96) * Move custom hive catalog to hivelink-core (#99) * Handle non-nullable union of single type for Avro (#98) * Handle non-nullable union of single type Co-authored-by: Wenye Zhang <[email protected]> * Handle null default in nested type default value situations (#100) * Move 'Hive Metadata Scan: Support case insensitive name mapping' (PR 52) to hivelink-core (#102) * Remove activeSparkSession (#103) * Disable default value preserving (#106) * Disable default value preserving * [LI][Avro] Do not reorder elements inside a Avro union schema (#93) * handle single type union properly in AvroSchemaVisitor for deep nested schema (#107) * Handle non-nullable union of single type for ORC spark non-vectorized reader (#104) * Handle single type union for non-vectorized reader * [Avro] Retain the type of field while copying the default values. (#109) * Retain the type of field while copying the default values. * [Hivelink] Refactor support hive non string partition cols to rid of … (#110) * [Hivelink] Refactor support hive non string partition cols to rid of Iceberg-oss code changes * Release automation overhaul: Sonatype Nexus, Shipkit and GH Actions (#101) * Add scm and developer info (#111) * [Core] Fix and refactor schema parser (#112) * [Core] Fix/Refactor SchemaParser to fix multiple bugs * Enhance the UT for testing required fields with default values (#113) * Enhance the UT for testing required fields with default values * Addressed review comments * Addressed review comment * Support single type union for ORC-vectorization reader (#114) * Support single type union for ORC-vectorization reader * Support single type union for ORC-vectorization reader Co-authored-by: Yiqiang Ding <[email protected]> * Refactor HMS code upon cherry-pick * Check for schema corruption and fix it on commit (#117) * Check for schema corruption and fix it on commit * ORC: Handle query where select and filter only uses default value col… (#118) * ORC: Handle query where select and filter only use default value columns * Set ORC columns and fix case-sensitivity issue with schema check (#119) * Hive: Return null for currentSnapshot() (#121) * Hive: Return null for currentSnapshot() * Handle snapshots() * Fix MergeHiveSchemaWithAvro to make it copy full Avro schema attributes (#120) * Fix MergeHiveSchemaWithAvro to make it copy full Avro schema attributes * Add logic to derive partition column id from partition.column.ids pro… (#122) * Add logic to derive partition column id from partition.column.ids property * Do not push down filter to ORC for union type schema (#123) * Bug fix: MergeHiveSchemaWithAvro should retain avro properties for li… (#125) * Bug fix: MergeHiveSchemaWithAvro should retain avro properties for list and map when they are nullable * LinkedIn rebase draft * Refactor hivelink 1 * Make hivelink module test all pass * Make spark 2.4 module work * Fix mr module * Make spark 3.1 module work * Fix TestSparkMetadataColumns * Minor fix for spark 2.4 * Update default spark version to 3.1 * Update java ci to only run spark 2.4 and 3.1 * Minor fix HiveTableOperations * Adapt github CI to 0.14.x branch * Fix mr module checkstyle * Fix checkstyle for orc module * Fix spark2.4 checkstyle * Refactor catalog loading logic using CatalogUtil * Minor change to CI/release Co-authored-by: Shardul Mahadik <[email protected]> Co-authored-by: Ratandeep Ratti <[email protected]> Co-authored-by: Shardul Mahadik <[email protected]> Co-authored-by: Kuai Yu <[email protected]> Co-authored-by: Walaa Eldin Moustafa <[email protected]> Co-authored-by: Sushant Raikar <[email protected]> Co-authored-by: ZihanLi58 <[email protected]> Co-authored-by: Wenye Zhang <[email protected]> Co-authored-by: Wenye Zhang <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Lei Sun <[email protected]> Co-authored-by: Jiefan <[email protected]> Co-authored-by: yiqiangin <[email protected]> Co-authored-by: Malini Mahalakshmi Venkatachari <[email protected]> Co-authored-by: Yiqiang Ding <[email protected]> Co-authored-by: Yiqiang Ding <[email protected]> Co-authored-by: Jack Moseley <[email protected]>

* Hive Catalog: Add a hive catalog that does not override existing Hive metadata (linkedin#10) Add custom hive catalog to not override existing Hive metadata Fail early with a proper exception if the metadata file is not existing Simplify CustomHiveCatalog (linkedin#22) * Shading: Add a iceberg-runtime shaded module (linkedin#12) * ORC: Add test for reading files without Iceberg IDs (linkedin#16) * Hive Metadata Scan: Support reading tables with only Hive metadata (linkedin#23, linkedin#24, linkedin#25, linkedin#26) - Support for non string partition columns (linkedin#24) - Support for Hive tables without avro.schema.literal (linkedin#25) - Hive Metadata Scan: Notify ScanEvent listeners on planning (linkedin#35) - Hive Metadata Scan: Do not use table snapshot summary for estimating statistics (linkedin#37) - Hive Metadata Scan: Return empty statistics (linkedin#49) - Hive Metadata Scan: Do not throw an exception on dangling partitions; log warning message (linkedin#50) - Hive Metadata Scan: Fix pushdown of non-partition predicates within NOT (linkedin#51) Co-authored-by: Ratandeep Ratti <[email protected]> Co-authored-by: Kuai Yu <[email protected]> Co-authored-by: Walaa Eldin Moustafa <[email protected]> * Row level filtering: Allow table scans to pass a row level filter for ORC files - ORC: Support NameMapping with row-level filtering (linkedin#53) * Hive: Made Predicate Pushdown dynamic based on the Hive Version * Hive: Fix uppercase bug and determine catalog from table properties (linkedin#38) * Hive: Return lowercase fieldname from IcebergRecordStructField * Hive: Determine catalog from table property * Hive: Fix schema not forwarded to SerDe on MR jobs (linkedin#45) (linkedin#47) * Hive: Use Hive table location in HiveIcebergSplit * Hive: Fix schema not passed to Serde * Hive: Refactor tests for tables with unqualified location URI Co-authored-by: Shardul Mahadik <[email protected]> * Hive Metadata Scan: Support case insensitive name mapping (linkedin#52) * Hive Metadata Scan: Merge Hive and Avro schemas to fix datatype inconsistencies (linkedin#57) Hive Metadata Scan: Fix Hive primitive to Avro logical type conversion (linkedin#58) Hive Metadata Scan: Fix support for Hive timestamp type (linkedin#61) Co-authored-by: Raymond Zhang <[email protected]> Co-authored-by: Shardul Mahadik <[email protected]> Fix HasDuplicateLowercaseColumnNames's visit method to use a new visi… (linkedin#67) * Fix HasDuplicateLowercaseColumnNames's visit method to use a new visitor instance every time * Trigger CI (cherry picked from commit b90e838) * Stop using serdeToFileFormat to unblock formats other than Avro or Orc (linkedin#64) * Stop using serdeToFileFormat to unblock formats other than Avro or Orc * Fix style check * Do not delete metadata location when HMS has been successfully updated (linkedin#68) (cherry picked from commit 766407e) * Support reading Avro complex union types (linkedin#73) Co-authored-by: Wenye Zhang <[email protected]> * [#2039] Support default value semantic for AVRO (linkedin#75) (cherry picked from commit c18f4c4) * Support hive non string partition cols (linkedin#78) * Support non-string hive type partition columns in LegacyHiveTableScan * Leverage eval against partition filter expression to filter non-string columns * Support default value read for ORC format in spark (linkedin#76) * Support default value read for ORC format in spark * Refactor common code for ReadBuilder for both non-vectorized and vectorized read * Fix code style issue * Add special handling of ROW_POSITION metadata column * Add corner case check for partition field * Use BaseDataReader.convertConstant to convert constants, and expand its functionality to support nested-type contants such as array/map/struct * Support nested type default value for vectorized read * Support deeply nested type default value for vectorized read * Support reading ORC complex union types (linkedin#74) * Support reading orc complex union types * add more tests * support union in VectorizedSparkOrcReaders and improve tests * support union in VectorizedSparkOrcReaders and improve tests - continued * fix checkstyle Co-authored-by: Wenye Zhang <[email protected]> * Support avro.schema.literal/hive union types in Hive legacy table to Iceberg conversion (linkedin#80) * Fix ORC schema visitors to support reading ORC files with deeply nest… (linkedin#81) * Fix ORC schema visitors to support reading ORC files with deeply nested union type schema * Added test for vectorized read * Disable avro validation for default values Co-authored-by: Shenoda Guirguis <[email protected]> * Fix spark avro reader reading union schema data (linkedin#83) * Fix spark avro reader to read correctly structured nested data values * Make sure field-id mapping is correctly maintained given arbitrary nested schema that contains union * Avro: Change union read schema from hive to trino (linkedin#84) * [LI] Avro: Refactor union-to-struct schema - Part 1. changes to support reading Avro * ORC: Change union read schema from hive to trino (linkedin#85) * [LI] ORC: Refactor union-to-struct schema - Part 2. changes to support reading ORC * Change Hive type to Iceberg type conversion for union * Recorder hive table properties to align the avro.schema.literal placement contract (linkedin#86) * [#2039] Support default value semantic for AVRO (cherry picked from commit c18f4c4) * reverting commits 2c59857 and f362aed (linkedin#88) Co-authored-by: Shenoda Guirguis <[email protected]> * logically patching PR 2328 on HiveMetadataPreservingTableOperations * Support timestamp as partition type (linkedin#91) * Support timestamp in partition types * Address comment * Separate classes under hive legacy package to new hivelink module (linkedin#87) * separate class under legacy to new hiveberg module * fix build * remove hiveberg dependency in iceberg-spark2 module * Revert "remove hiveberg dependency in iceberg-spark2 module" This reverts commit 2e8b743. * rename hiveberg module to hivelink Co-authored-by: Wenye Zhang <[email protected]> * [LI] Align default value validation align with avro semantics in terms of nullable (nested) fields (linkedin#92) * Align default value validation align with avro semantics in terms of nullable (nested) fields * Allow setting null as default value for nested fields in record default * [LI][Spark][Avro] read avro union using decoder instead of directly returning v… (linkedin#94) * [LI][Spark] read avro union using decoder instead of directly returning value * Add a comment for the schema * Improve the logging when the deserailzed index is invalid to read the symbol from enum (linkedin#96) * Move custom hive catalog to hivelink-core (linkedin#99) * Handle non-nullable union of single type for Avro (linkedin#98) * Handle non-nullable union of single type Co-authored-by: Wenye Zhang <[email protected]> * Handle null default in nested type default value situations (linkedin#100) * Move 'Hive Metadata Scan: Support case insensitive name mapping' (PR 52) to hivelink-core (linkedin#102) * Remove activeSparkSession (linkedin#103) * Disable default value preserving (linkedin#106) * Disable default value preserving * [LI][Avro] Do not reorder elements inside a Avro union schema (linkedin#93) * handle single type union properly in AvroSchemaVisitor for deep nested schema (linkedin#107) * Handle non-nullable union of single type for ORC spark non-vectorized reader (linkedin#104) * Handle single type union for non-vectorized reader * [Avro] Retain the type of field while copying the default values. (linkedin#109) * Retain the type of field while copying the default values. * [Hivelink] Refactor support hive non string partition cols to rid of … (linkedin#110) * [Hivelink] Refactor support hive non string partition cols to rid of Iceberg-oss code changes * Release automation overhaul: Sonatype Nexus, Shipkit and GH Actions (linkedin#101) * Add scm and developer info (linkedin#111) * [Core] Fix and refactor schema parser (linkedin#112) * [Core] Fix/Refactor SchemaParser to fix multiple bugs * Enhance the UT for testing required fields with default values (linkedin#113) * Enhance the UT for testing required fields with default values * Addressed review comments * Addressed review comment * Support single type union for ORC-vectorization reader (linkedin#114) * Support single type union for ORC-vectorization reader * Support single type union for ORC-vectorization reader Co-authored-by: Yiqiang Ding <[email protected]> * Refactor HMS code upon cherry-pick * Check for schema corruption and fix it on commit (linkedin#117) * Check for schema corruption and fix it on commit * ORC: Handle query where select and filter only uses default value col… (linkedin#118) * ORC: Handle query where select and filter only use default value columns * Set ORC columns and fix case-sensitivity issue with schema check (linkedin#119) * Hive: Return null for currentSnapshot() (linkedin#121) * Hive: Return null for currentSnapshot() * Handle snapshots() * Fix MergeHiveSchemaWithAvro to make it copy full Avro schema attributes (linkedin#120) * Fix MergeHiveSchemaWithAvro to make it copy full Avro schema attributes * Add logic to derive partition column id from partition.column.ids pro… (linkedin#122) * Add logic to derive partition column id from partition.column.ids property * Do not push down filter to ORC for union type schema (linkedin#123) * Bug fix: MergeHiveSchemaWithAvro should retain avro properties for li… (linkedin#125) * Bug fix: MergeHiveSchemaWithAvro should retain avro properties for list and map when they are nullable * LinkedIn rebase draft * Refactor hivelink 1 * Make hivelink module test all pass * Make spark 2.4 module work * Fix mr module * Make spark 3.1 module work * Fix TestSparkMetadataColumns * Minor fix for spark 2.4 * Update default spark version to 3.1 * Update java ci to only run spark 2.4 and 3.1 * Minor fix HiveTableOperations * Adapt github CI to 0.14.x branch * Fix mr module checkstyle * Fix checkstyle for orc module * Fix spark2.4 checkstyle * Refactor catalog loading logic using CatalogUtil * Minor change to CI/release Co-authored-by: Shardul Mahadik <[email protected]> Co-authored-by: Ratandeep Ratti <[email protected]> Co-authored-by: Shardul Mahadik <[email protected]> Co-authored-by: Kuai Yu <[email protected]> Co-authored-by: Walaa Eldin Moustafa <[email protected]> Co-authored-by: Sushant Raikar <[email protected]> Co-authored-by: ZihanLi58 <[email protected]> Co-authored-by: Wenye Zhang <[email protected]> Co-authored-by: Wenye Zhang <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Lei Sun <[email protected]> Co-authored-by: Jiefan <[email protected]> Co-authored-by: yiqiangin <[email protected]> Co-authored-by: Malini Mahalakshmi Venkatachari <[email protected]> Co-authored-by: Yiqiang Ding <[email protected]> Co-authored-by: Yiqiang Ding <[email protected]> Co-authored-by: Jack Moseley <[email protected]>

* Rebase LI-Iceberg changes on top of Apache Iceberg 1.0.0 release * Hive Catalog: Add a hive catalog that does not override existing Hive metadata (#10) Add custom hive catalog to not override existing Hive metadata Fail early with a proper exception if the metadata file is not existing Simplify CustomHiveCatalog (#22) * Shading: Add a iceberg-runtime shaded module (#12) * ORC: Add test for reading files without Iceberg IDs (#16) * Hive Metadata Scan: Support reading tables with only Hive metadata (#23, #24, #25, #26) - Support for non string partition columns (#24) - Support for Hive tables without avro.schema.literal (#25) - Hive Metadata Scan: Notify ScanEvent listeners on planning (#35) - Hive Metadata Scan: Do not use table snapshot summary for estimating statistics (#37) - Hive Metadata Scan: Return empty statistics (#49) - Hive Metadata Scan: Do not throw an exception on dangling partitions; log warning message (#50) - Hive Metadata Scan: Fix pushdown of non-partition predicates within NOT (#51) Co-authored-by: Ratandeep Ratti <[email protected]> Co-authored-by: Kuai Yu <[email protected]> Co-authored-by: Walaa Eldin Moustafa <[email protected]> * Row level filtering: Allow table scans to pass a row level filter for ORC files - ORC: Support NameMapping with row-level filtering (#53) * Hive: Made Predicate Pushdown dynamic based on the Hive Version * Hive: Fix uppercase bug and determine catalog from table properties (#38) * Hive: Return lowercase fieldname from IcebergRecordStructField * Hive: Determine catalog from table property * Hive: Fix schema not forwarded to SerDe on MR jobs (#45) (#47) * Hive: Use Hive table location in HiveIcebergSplit * Hive: Fix schema not passed to Serde * Hive: Refactor tests for tables with unqualified location URI Co-authored-by: Shardul Mahadik <[email protected]> * Hive Metadata Scan: Support case insensitive name mapping (#52) * Hive Metadata Scan: Merge Hive and Avro schemas to fix datatype inconsistencies (#57) Hive Metadata Scan: Fix Hive primitive to Avro logical type conversion (#58) Hive Metadata Scan: Fix support for Hive timestamp type (#61) Co-authored-by: Raymond Zhang <[email protected]> Co-authored-by: Shardul Mahadik <[email protected]> Fix HasDuplicateLowercaseColumnNames's visit method to use a new visi… (#67) * Fix HasDuplicateLowercaseColumnNames's visit method to use a new visitor instance every time * Trigger CI (cherry picked from commit b90e838) * Stop using serdeToFileFormat to unblock formats other than Avro or Orc (#64) * Stop using serdeToFileFormat to unblock formats other than Avro or Orc * Fix style check * Do not delete metadata location when HMS has been successfully updated (#68) (cherry picked from commit 766407e) * Support reading Avro complex union types (#73) Co-authored-by: Wenye Zhang <[email protected]> * [#2039] Support default value semantic for AVRO (#75) (cherry picked from commit c18f4c4) * Support hive non string partition cols (#78) * Support non-string hive type partition columns in LegacyHiveTableScan * Leverage eval against partition filter expression to filter non-string columns * Support default value read for ORC format in spark (#76) * Support default value read for ORC format in spark * Refactor common code for ReadBuilder for both non-vectorized and vectorized read * Fix code style issue * Add special handling of ROW_POSITION metadata column * Add corner case check for partition field * Use BaseDataReader.convertConstant to convert constants, and expand its functionality to support nested-type contants such as array/map/struct * Support nested type default value for vectorized read * Support deeply nested type default value for vectorized read * Support reading ORC complex union types (#74) * Support reading orc complex union types * add more tests * support union in VectorizedSparkOrcReaders and improve tests * support union in VectorizedSparkOrcReaders and improve tests - continued * fix checkstyle Co-authored-by: Wenye Zhang <[email protected]> * Support avro.schema.literal/hive union types in Hive legacy table to Iceberg conversion (#80) * Fix ORC schema visitors to support reading ORC files with deeply nest… (#81) * Fix ORC schema visitors to support reading ORC files with deeply nested union type schema * Added test for vectorized read * Disable avro validation for default values Co-authored-by: Shenoda Guirguis <[email protected]> * Fix spark avro reader reading union schema data (#83) * Fix spark avro reader to read correctly structured nested data values * Make sure field-id mapping is correctly maintained given arbitrary nested schema that contains union * Avro: Change union read schema from hive to trino (#84) * [LI] Avro: Refactor union-to-struct schema - Part 1. changes to support reading Avro * ORC: Change union read schema from hive to trino (#85) * [LI] ORC: Refactor union-to-struct schema - Part 2. changes to support reading ORC * Change Hive type to Iceberg type conversion for union * Recorder hive table properties to align the avro.schema.literal placement contract (#86) * [#2039] Support default value semantic for AVRO (cherry picked from commit c18f4c4) * reverting commits 2c59857 and f362aed (#88) Co-authored-by: Shenoda Guirguis <[email protected]> * logically patching PR 2328 on HiveMetadataPreservingTableOperations * Support timestamp as partition type (#91) * Support timestamp in partition types * Address comment * Separate classes under hive legacy package to new hivelink module (#87) * separate class under legacy to new hiveberg module * fix build * remove hiveberg dependency in iceberg-spark2 module * Revert "remove hiveberg dependency in iceberg-spark2 module" This reverts commit 2e8b743. * rename hiveberg module to hivelink Co-authored-by: Wenye Zhang <[email protected]> * [LI] Align default value validation align with avro semantics in terms of nullable (nested) fields (#92) * Align default value validation align with avro semantics in terms of nullable (nested) fields * Allow setting null as default value for nested fields in record default * [LI][Spark][Avro] read avro union using decoder instead of directly returning v… (#94) * [LI][Spark] read avro union using decoder instead of directly returning value * Add a comment for the schema * Improve the logging when the deserailzed index is invalid to read the symbol from enum (#96) * Move custom hive catalog to hivelink-core (#99) * Handle non-nullable union of single type for Avro (#98) * Handle non-nullable union of single type Co-authored-by: Wenye Zhang <[email protected]> * Handle null default in nested type default value situations (#100) * Move 'Hive Metadata Scan: Support case insensitive name mapping' (PR 52) to hivelink-core (#102) * Remove activeSparkSession (#103) * Disable default value preserving (#106) * Disable default value preserving * [LI][Avro] Do not reorder elements inside a Avro union schema (#93) * handle single type union properly in AvroSchemaVisitor for deep nested schema (#107) * Handle non-nullable union of single type for ORC spark non-vectorized reader (#104) * Handle single type union for non-vectorized reader * [Avro] Retain the type of field while copying the default values. (#109) * Retain the type of field while copying the default values. * [Hivelink] Refactor support hive non string partition cols to rid of … (#110) * [Hivelink] Refactor support hive non string partition cols to rid of Iceberg-oss code changes * Release automation overhaul: Sonatype Nexus, Shipkit and GH Actions (#101) * Add scm and developer info (#111) * [Core] Fix and refactor schema parser (#112) * [Core] Fix/Refactor SchemaParser to fix multiple bugs * Enhance the UT for testing required fields with default values (#113) * Enhance the UT for testing required fields with default values * Addressed review comments * Addressed review comment * Support single type union for ORC-vectorization reader (#114) * Support single type union for ORC-vectorization reader * Support single type union for ORC-vectorization reader Co-authored-by: Yiqiang Ding <[email protected]> * Refactor HMS code upon cherry-pick * Check for schema corruption and fix it on commit (#117) * Check for schema corruption and fix it on commit * ORC: Handle query where select and filter only uses default value col… (#118) * ORC: Handle query where select and filter only use default value columns * Set ORC columns and fix case-sensitivity issue with schema check (#119) * Hive: Return null for currentSnapshot() (#121) * Hive: Return null for currentSnapshot() * Handle snapshots() * Fix MergeHiveSchemaWithAvro to make it copy full Avro schema attributes (#120) * Fix MergeHiveSchemaWithAvro to make it copy full Avro schema attributes * Add logic to derive partition column id from partition.column.ids pro… (#122) * Add logic to derive partition column id from partition.column.ids property * Do not push down filter to ORC for union type schema (#123) * Bug fix: MergeHiveSchemaWithAvro should retain avro properties for li… (#125) * Bug fix: MergeHiveSchemaWithAvro should retain avro properties for list and map when they are nullable * LinkedIn rebase draft * Refactor hivelink 1 * Make hivelink module test all pass * Make spark 2.4 module work * Fix mr module * Make spark 3.1 module work * Fix TestSparkMetadataColumns * Minor fix for spark 2.4 * Update default spark version to 3.1 * Update java ci to only run spark 2.4 and 3.1 * Minor fix HiveTableOperations * Adapt github CI to 0.14.x branch * Fix mr module checkstyle * Fix checkstyle for orc module * Fix spark2.4 checkstyle * Refactor catalog loading logic using CatalogUtil * Minor change to CI/release Co-authored-by: Shardul Mahadik <[email protected]> Co-authored-by: Ratandeep Ratti <[email protected]> Co-authored-by: Shardul Mahadik <[email protected]> Co-authored-by: Kuai Yu <[email protected]> Co-authored-by: Walaa Eldin Moustafa <[email protected]> Co-authored-by: Sushant Raikar <[email protected]> Co-authored-by: ZihanLi58 <[email protected]> Co-authored-by: Wenye Zhang <[email protected]> Co-authored-by: Wenye Zhang <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Lei Sun <[email protected]> Co-authored-by: Jiefan <[email protected]> Co-authored-by: yiqiangin <[email protected]> Co-authored-by: Malini Mahalakshmi Venkatachari <[email protected]> Co-authored-by: Yiqiang Ding <[email protected]> Co-authored-by: Yiqiang Ding <[email protected]> Co-authored-by: Jack Moseley <[email protected]> * Add flink 1.14 artifacts for release Co-authored-by: Shardul Mahadik <[email protected]> Co-authored-by: Ratandeep Ratti <[email protected]> Co-authored-by: Shardul Mahadik <[email protected]> Co-authored-by: Kuai Yu <[email protected]> Co-authored-by: Walaa Eldin Moustafa <[email protected]> Co-authored-by: Sushant Raikar <[email protected]> Co-authored-by: ZihanLi58 <[email protected]> Co-authored-by: Wenye Zhang <[email protected]> Co-authored-by: Wenye Zhang <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Shenoda Guirguis <[email protected]> Co-authored-by: Lei Sun <[email protected]> Co-authored-by: Jiefan <[email protected]> Co-authored-by: yiqiangin <[email protected]> Co-authored-by: Malini Mahalakshmi Venkatachari <[email protected]> Co-authored-by: Yiqiang Ding <[email protected]> Co-authored-by: Yiqiang Ding <[email protected]> Co-authored-by: Jack Moseley <[email protected]>

Wenye Zhang added 2 commits January 20, 2022 17:07

separate class under legacy to new hiveberg module

ea9fc5b

fix build

76bda7b

github-actions bot added BUILD HIVE SPARK labels Jan 21, 2022

wmoustafa reviewed Jan 25, 2022

View reviewed changes

wmoustafa reviewed Jan 26, 2022

View reviewed changes

Wenye Zhang added 3 commits January 27, 2022 13:35

remove hiveberg dependency in iceberg-spark2 module

2e8b743

Revert "remove hiveberg dependency in iceberg-spark2 module"

2681f10

This reverts commit 2e8b743.

rename hiveberg module to hivelink

36dae56

funcheetah changed the title ~~[Reference PR] Separate classes under hive legacy package to new hiveberg module~~ Separate classes under hive legacy package to new hivelink module Mar 3, 2022

wmoustafa reviewed Mar 4, 2022

View reviewed changes

wmoustafa approved these changes Mar 7, 2022

View reviewed changes

funcheetah merged commit fcf50f1 into linkedin:li-0.11.x Mar 7, 2022

Separate classes under hive legacy package to new hivelink module #87

Separate classes under hive legacy package to new hivelink module #87

Uh oh!

Conversation

funcheetah commented Jan 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wmoustafa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wmoustafa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

funcheetah commented Mar 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

funcheetah commented Jan 21, 2022 •

edited

Loading