[HUDI-4690][HUDI-4503] Cleaning up Hudi custom Spark Rules
#6361
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alexeykudinkin
force-pushed
the
ak/spk-sql-clup
branch
4 times, most recently
from
8d8cda2 to
24501af
Compare
alexeykudinkin
changed the title
RulesRules
alexeykudinkin
force-pushed
the
ak/spk-sql-clup
branch
from
c182b7e to
bf943f5
Compare
5 tasks
alexeykudinkin
changed the title
RulesRules
alexeykudinkin
changed the title
RulesRules
alexeykudinkin
force-pushed
the
ak/spk-sql-clup
branch
3 times, most recently
from
88f855d to
d37c994
Compare
alexeykudinkin
changed the title
RulesRules
alexeykudinkin
force-pushed
the
ak/spk-sql-clup
branch
2 times, most recently
from
bf18e26 to
1f3b814
Compare
yihua
added
priority:critical
alexeykudinkin
force-pushed
the
ak/spk-sql-clup
branch
5 times, most recently
from
533eb9b to
392b7ce
Compare
alexeykudinkin
force-pushed
the
ak/spk-sql-clup
branch
from
392b7ce to
844bce7
Compare
alexeykudinkin
force-pushed
the
ak/spk-sql-clup
branch
from
844bce7 to
c69c3db
Compare
alexeykudinkin
commented
Sep 16, 2022
| createRdd(df, structName, recordNamespace, toScalaOption(targetSchema)) | ||
| } | ||
|
|
||
| if (latestTableSchema.isPresent && reconcileToLatestSchema) { |
Contributor
Author
There was a problem hiding this comment.
This conditional got removed as useless
added 12 commits
January 4, 2023 19:54
…he source to avoid race-condition, w/ Spark's `ResolveReferences` rule
…esolve the source to avoid race-condition," This reverts commit ebca9c4.
…lt catch-all rule in vanilla `ResolveReferences`
alexeykudinkin
force-pushed
the
ak/spk-sql-clup
branch
from
4230038 to
04d1820
Compare
added 2 commits
January 6, 2023 09:28
…ly, as otherwise assignments might get resolved before we do their adaptation; Handle unresolved plans in `resolveHudiTable`
alexeykudinkin
commented
Jan 7, 2023
| def customResolutionRules: Seq[RuleBuilder] = { | ||
| val rules: ListBuffer[RuleBuilder] = ListBuffer( | ||
| // Default rules | ||
| session => HoodieResolveReferences(session), |
Contributor
Author
There was a problem hiding this comment.
All custom resolution logic has been removed (we now rely on Spark to resolve all of its own commands)
added 3 commits
January 6, 2023 23:06
- Apply to producing side based on whether output contains meta-fields (rather than whether it resolves to Hudi table) - Handle producing side in one pass (ie both resolved/unresolved)
… nodes on top of it being resolved)
Collaborator
Contributor
Author
|
CI is green:
|
alexeykudinkin
added
priority:critical
alexeykudinkin
added
priority:blocker
4 tasks
Contributor
Author
|
Closing this in favor of rebased & squashed #7871 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
This PR cleans up considerable amount of Spark's (internal) resolution logic that has been copied over into Hudi components, while in reality there's no actual need for that.
Changelog: TBA
Issues this will be addressing:
Impact
None
Contributor's checklist