[HUDI-3130] Fixing Hive getSchema for RT tables addressing different partitions having different schemas #4468
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nsivabalan
reviewed
Dec 29, 2021
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/AbstractRealtimeRecordReader.java
Outdated
Show resolved
Hide resolved
codope
changed the title
nsivabalan
changed the title
nsivabalan
force-pushed
the
hiveSchema
branch
2 times, most recently
from
5933187 to
2461ceb
Compare
nsivabalan
force-pushed
the
hiveSchema
branch
from
c032433 to
256f7d8
Compare
Contributor
|
@codope : good to review the patch. I have pushed commits to address feedback. |
nsivabalan
added
status:triaged
codope
reviewed
Feb 28, 2022
Member
codope
left a comment
codope
left a comment
There was a problem hiding this comment.
Looks good. One minor comment. Also, it would be good to add a test for multi-partitioned table with schema evolution for one of the partition.
...-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
Outdated
Show resolved
Hide resolved
Contributor
|
sure. will add a test for schema evolution |
Contributor
|
@codope : Added a test for schema evolution across partitions. you can take a look. |
Collaborator
codope
approved these changes
Mar 6, 2022
| * @throws Exception | ||
| */ | ||
| public Schema getTableAvroSchema() throws Exception { | ||
| return getTableAvroSchema(metaClient.getTableConfig().populateMetaFields()); |
Contributor
There was a problem hiding this comment.
@aditiwari01 @codope folks, can you please elaborate why this has been changed? Why are we assuming that Table's schema should have meta-fields?
vingov
pushed a commit
to vingov/hudi
that referenced
this pull request
Apr 3, 2022
…partitions having different schemas (apache#4468) * Fixing Hive getSchema for RT tables * Addressing feedback * temp diff * fixing tests after spark datasource read support for metadata table is merged to master * Adding multi-partition schema evolution tests to HoodieRealTimeRecordReader Co-authored-by: Aditya Tiwari <[email protected]> Co-authored-by: sivabalan <[email protected]>
stayrascal
pushed a commit
to stayrascal/hudi
that referenced
this pull request
Apr 12, 2022
…partitions having different schemas (apache#4468) * Fixing Hive getSchema for RT tables * Addressing feedback * temp diff * fixing tests after spark datasource read support for metadata table is merged to master * Adding multi-partition schema evolution tests to HoodieRealTimeRecordReader Co-authored-by: Aditya Tiwari <[email protected]> Co-authored-by: sivabalan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the pull request
Fixing Hive getSchema for RT tables
Refer issue for more details: #2802
Verify this pull request:
Unit tests to be explored. I have tested on local and cluster spark-hive.
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.