[HUDI-4690][HUDI-4503] Cleaning up Hudi custom Spark `Rule`s#6361

Closed

alexeykudinkin wants to merge 130 commits intoapache:masterfrom

onehouseinc:ak/spk-sql-clup

Commits on Jan 5, 2023

`isHoodieTable` > `resolvesToHoodieTable`; Make sure predicate only matches already resolved tables
Alexey Kudinkin
committed
Cleaned up unnecessary `HoodieCatalystPlanUtils` overrides
Alexey Kudinkin
committed
Added `resolveHoodieTable` fetching `CatalogTable` from the resolved `LogicalPlan`;
Alexey Kudinkin
committed
Fixed `TimeTravelRelation` implementation to become `LeafNode` instead of `Command`;
Alexey Kudinkin
committed
Converted `TimeTravelRelation` to `UnaryNode`;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Extracted `TimeTravelRelation` resolution rule to `HoodieSpark3Analysis`
Alexey Kudinkin
committed
Fixing compilation for Spark 3.1
Alexey Kudinkin
committed
(Temporarily) Disabling test
Alexey Kudinkin
committed
Fixing `MergeIntoTable` statement to make sure source/target outputs are properly aligned
Alexey Kudinkin
committed
Make sure Hudi rules only applicable to resolved constructs
Alexey Kudinkin
committed
Fixed `MergeIntoHoodieTableCommand` to reference proper table identifier;
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Rebased Hudi `*Index` commands to bear resolved `CatalogTable` instead of `tableId`;
Alexey Kudinkin
committed
Rebased `Delete`/`Update` commands onto using already resolved tables
Alexey Kudinkin
committed
Fixing invalid `MERGE INTO` statements in tests
Alexey Kudinkin
committed
Cleaning up dead code
Alexey Kudinkin
committed
Extracted `HoodieExtendedParserInterface` to consolidate all extensions to Spark's `ParserInterface`;
Alexey Kudinkin
committed
Cleaned up Spark 3 specific rules
Alexey Kudinkin
committed
Cleaned up `InsertIntoStatement` matching;
Alexey Kudinkin
committed
Fixed fallback from V2 to V1 for `InsertIntoStatement`;
Alexey Kudinkin
committed
Avoid unnecessary `DataFrame` de-referencing
Alexey Kudinkin
committed
Fixing test queries
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Added `AvroSchemaEvolutionUtils.reconcileFieldNamesCasing`;
Alexey Kudinkin
committed
Make sure `HoodieSparkSqlWriter` reconciles Schema's field-name casing (transitioning from Spark to Avro);
Alexey Kudinkin
committed
Make sure `InternalSchema` can handle case-sensitive use-cases
Alexey Kudinkin
committed
Revisited `HoodieSparkUtils.createRdd` seq to appropriately handle the case-mismatching field-names (since Spark is case-insensitive, while Avro is not);
Alexey Kudinkin
committed
Clean up `MergeIntoHoodieTableCommand`
Alexey Kudinkin
committed
Cleaned up `HoodieAnalysis`
Alexey Kudinkin
committed
Revisited `MergeIntoHoodieTableCommand` impl to address plentiful issues
Alexey Kudinkin
committed
Fixed invalid assignment expressions binding;
Alexey Kudinkin
committed
Fixing invalid equality check
Alexey Kudinkin
committed
Fixed resolving pre-combine field attribute to fail w/ proper error description
Alexey Kudinkin
committed
Fixed source dataset reshaping seq to always override primary/pre-combine columns, unless these are already contained w/in
Alexey Kudinkin
committed
Restricted permitted merge-condition experssions to simple attribute-reference ones,
Alexey Kudinkin
committed
Fixed Cast pattern-matching to accommodate for Spark >= 3.2 appropriately
Alexey Kudinkin
committed
Cleaned up Deleting actions to reuse same code as Updating/Inserting ones
Alexey Kudinkin
committed
Added validation of the schema to-be-expected by `ExpressionPayload` against the one it obtains (to make troubleshooting easier)
Alexey Kudinkin
committed
Cleaned up duplicated methods
Alexey Kudinkin
committed
Make schema validations to be gated by config;
Alexey Kudinkin
committed
Fixed `MergeIntoHoodieTableCommand` to properly bind expressions against amended source dataset
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Fixed handling of deleting actions
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Gate validation t/h standalone internal config;
Alexey Kudinkin
committed
Abstracted meta-field's `Metadata` instantiation w/in `SparkAdapter`
Alexey Kudinkin
committed
After rebsae cleanup
Alexey Kudinkin
committed
Abstracting `HoodieLogicalRelation` in `hudi-spark3-common`
Alexey Kudinkin
committed
Moved `createCatalystMetadataForMetaField` from `HoodieCatalystPlansUtils` to `SparkAdapter`
Alexey Kudinkin
committed
Bifurcated `HoodieCatalystPlanUtils` for all Spark 3.x versions;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Bubbled `resolveHoodieTable` down to Spark >= 3.2 adapters
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
`HoodieSpark3Analysis` > `HoodieSpark32PlusAnalysis`
Alexey Kudinkin
committed
Extracted rules resolving/folding `HoodieLogicalRelation` into `HoodieSpark3Analysis` (to make these compatible w/ Spark 3.1)
Alexey Kudinkin
committed
Missing license;
Alexey Kudinkin
committed
Fixing init seq
Alexey Kudinkin
committed
Disable rules incompatible w/ Spark 3.1
Alexey Kudinkin
committed
Missing `resolveHoodieTable` override properly handling `HoodieLogicalRelation`s
Alexey Kudinkin
committed
Fixing error messages for Spark 3.1
Alexey Kudinkin
committed
Exempt transforms run w/in analysis phase
Alexey Kudinkin
committed
Fixed folding of the `HoodieLogicalRelation`s to properly restore full output of the `LogicalRelation`
Alexey Kudinkin
committed
Fixing invalid cloning seq
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Addressing rebase artifacts
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Rebasing `ExpressionPayload` to rely on `HoodieAvroSerializer`
Alexey Kudinkin
committed
Fixing compilation (again)
Alexey Kudinkin
committed
Restored missing config overrides
Alexey Kudinkin
committed
Fixed missing source table's Avro schema being appended to write-opts
Alexey Kudinkin
committed
Simplify of MIT handling to make sure it provides for consistent behavior (check HUDI-4946 for more details)
Alexey Kudinkin
committed
Fixed meta-fields filtering for Spark < 3.2;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Cleaning up `attributeEquals` utility
Alexey Kudinkin
committed
Allow partial assignments for updating clauses
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Reverting rebase artifacts
Alexey Kudinkin
committed
Fixing more tests
Alexey Kudinkin
committed
Revisited canonicalization seq in `HoodieSparkSqlWriter` to also reconcile field-name casing
Alexey Kudinkin
committed
Tidying up utils
Alexey Kudinkin
committed
Handle superfluous casting appropriately in MIT
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Revisited source dataset reshaping to make sure casing of primary-key and pre-combine-key are properly
Alexey Kudinkin
committed
Reference target table attributes instead of ones from the condition
Alexey Kudinkin
committed
Reverting unnecessary changes
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Set "spark.testing" property
Alexey Kudinkin
committed
Fixed MIT-specific configs to not be overridable by any other configuration
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixing tests (one more time)
Alexey Kudinkin
committed
Fixing presence check for the case-insensitive case
Alexey Kudinkin
committed
Re-enabling tests
Alexey Kudinkin
committed
Enable adapting resolution rules for Spark 2.x (previously Spark 3.x only)
Alexey Kudinkin
committed
Merged Hudi's implementation resolution rules into `ResolveImplementations` one
Alexey Kudinkin
committed
Ported Spark SQL Merge Into statement resolution logic (from Spark 3.1)
Alexey Kudinkin
committed
Fixed tests error messages for Spark 2.x
Alexey Kudinkin
committed
Fixed Spark 2 `ResolveReferences` rule to use appropriate resolver
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Reverting mering of rules resolving Hudi impelementations
Alexey Kudinkin
committed
Fixed Spark 2 resolution of Merge Into statement containing `update *`/`insert *` expressions
Alexey Kudinkin
committed
Added `HoodieSparkClientTestBase` properly injecting `HoodieSparkSessionExtensions`
Alexey Kudinkin
committed
Simplified handling of the meta-fields to avoid `HoodieLogicalRelationAdapter`
Alexey Kudinkin
committed
Cleaned up `HoodieLogicalRelationAdapter`
Alexey Kudinkin
committed
Broke out `ResolveImplementationsEarly` from `ResolveImplementations`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Make sure meta-fields are stripped out from both receiving/providing ends
Alexey Kudinkin
committed
Cleaned up `DeleteFromTableCommand`
Alexey Kudinkin
committed
Cleaned up `UpdateTableCommand`
Alexey Kudinkin
committed
Fixed `resolveHoodieTable` to be able to resolve t/h projections and filters
Alexey Kudinkin
committed
Fixing MIT and UT to project out meta-field attributes instead of eliminating them
Alexey Kudinkin
committed
Fixed IIS to project out meta-fields from both receiving/providing ends as well
Alexey Kudinkin
committed
Missing casting in `UpdateTable`
Alexey Kudinkin
committed
Cleaning up unnecessary meta-fields manipulations
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed

Commits on Jan 6, 2023

Fixed `AdaptIngestionTargetLogicalRelations` to apply only to resolved targets
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed Spark 2.x resolution logic for MIT to always manually resolve the source to avoid race-condition,
Alexey Kudinkin
committed
Revert "Fixed Spark 2.x resolution logic for MIT to always manually resolve the source to avoid race-condition,"
Alexey Kudinkin
committed
Avoiding hijacking of the resolution of the `MergeIntoTable` by default catch-all rule in vanilla `ResolveReferences`
Alexey Kudinkin
committed
Removing outdated test
Alexey Kudinkin
committed
Make sure MIT target and source tables could be processed independently, as otherwise assignments might
Alexey Kudinkin
committed

Commits on Jan 7, 2023

Revisited `AdaptIngestionTargetLogicalRelations` to
Alexey Kudinkin
committed
Fixing `TimeTravelRelation` to keep it as non-resolved node (to avoid nodes on top of it being resolved)
Alexey Kudinkin
committed
Fixed Hudi TVF to be properly resolved for Spark >= 3.2
Alexey Kudinkin
committed