[HUDI-3549] Removing dependency on "spark-avro" #4955

Merged

nsivabalan merged 33 commits intoapache:masterfrom

onehouseinc:ak/spk-avr-shd-fix

Mar 29, 2022

Commits on Mar 28, 2022

Removing spark-avro bundling from Hudi by default;
Alexey Kudinkin
committed
Scaffolded Spark 3.2.x hierarchy:
Alexey Kudinkin
committed
Moved `ExpressionCodeGen`,`ExpressionPayload` into `hudi-spark` module
Alexey Kudinkin
committed
Bootstrapped
Alexey Kudinkin
committed
Limited access to `SchemaConverters`, `AvroUtils` to `spark.sql` package
Alexey Kudinkin
committed
Fixing compilation for "hudi-spark3" module
Alexey Kudinkin
committed
Down-scoped `AvroUtils` to be Spark-version specific
Alexey Kudinkin
committed
Bootstrapped Spark 3.1.x Avro serializer/deserializer hierarchy
Alexey Kudinkin
committed
Bootstrapped Spark 2.4.x Avro serializer/deserializer hierarchy
Alexey Kudinkin
committed
Removed "spark-avro" dependency from all modules
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Reverting accidental change
Alexey Kudinkin
committed
Fixed `AvroDeserializer` to stay compatible w/ both Spark 3.2.1 and 3.2.0
Alexey Kudinkin
committed
Cleaned up all "spark-avro" refs
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Untethered bespoke Column Stats index from the write path
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing tests dependent on `SparkAdapter`
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Inlined all methods renamed in `DataSourceUtils`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Reverting incorrect change
Alexey Kudinkin
committed
Tidying up after rebase
Alexey Kudinkin
committed
Removing spark-avro from Hudi examples
Alexey Kudinkin
committed
Fixed compilation for Spark 3.2.0
Alexey Kudinkin
committed
Sync'd Avro Serializer/Deserializer hierarchy to Spark 3.1.3
Alexey Kudinkin
committed
Bumped 3.1.x branch to rely on Spark 3.1.3
Alexey Kudinkin
committed
Relocate "org.apache.spark.sql.avro" to avoid potential collisions w/ "spark-avro"
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-3549] Removing dependency on "spark-avro" #4955

[HUDI-3549] Removing dependency on "spark-avro" #4955
nsivabalan merged 33 commits intoapache:masterfrom
onehouseinc:ak/spk-avr-shd-fix

Commits on Mar 28, 2022

Removing spark-avro bundling from Hudi by default;

Scaffolded Spark 3.2.x hierarchy:

Moved `ExpressionCodeGen`,`ExpressionPayload` into `hudi-spark` module

Bootstrapped

Limited access to `SchemaConverters`, `AvroUtils` to `spark.sql` package

Fixing compilation for "hudi-spark3" module

Down-scoped `AvroUtils` to be Spark-version specific

Bootstrapped Spark 3.1.x Avro serializer/deserializer hierarchy

Bootstrapped Spark 2.4.x Avro serializer/deserializer hierarchy

Removed "spark-avro" dependency from all modules

Tidying up

Fixing compilation

Missing license

Reverting accidental change

Fixed `AvroDeserializer` to stay compatible w/ both Spark 3.2.1 and 3.2.0

Cleaned up all "spark-avro" refs

Tidying up

Untethered bespoke Column Stats index from the write path

Fixing compilation

Fixing tests dependent on `SparkAdapter`

Fixing compilation

Tidying up

Inlined all methods renamed in `DataSourceUtils`

Tidying up

`lint`

Fixing compilation

Reverting incorrect change

Tidying up after rebase

Removing spark-avro from Hudi examples

Fixed compilation for Spark 3.2.0

Sync'd Avro Serializer/Deserializer hierarchy to Spark 3.1.3

Bumped 3.1.x branch to rely on Spark 3.1.3

Relocate "org.apache.spark.sql.avro" to avoid potential collisions w/ "spark-avro"