[MINOR] Support column type evolution for Hive #7129
Conversation
|
@xiarixiaoyao please help to review. |
xiarixiaoyao
changed the title
| </dependency> | ||
|
|
||
| <!-- Hive --> | ||
| <dependency> |
There was a problem hiding this comment.
why we need hive dependency here?
There was a problem hiding this comment.
changed. now dont need to add hive dependency
There was a problem hiding this comment.
This is still showing up. Can you remove this dependency as the change is in hudi-hadoop-mr package and hive-exec is already defined as dependency in hudi-hadoop-mr /pom.xml
| @UseFileSplitsFromInputFormat | ||
| public class HoodieParquetInputFormat extends HoodieParquetInputFormatBase { | ||
| public class HoodieParquetInputFormat extends HoodieParquetInputFormatBase implements | ||
| SelfDescribingInputFormatInterface { |
There was a problem hiding this comment.
it will be better to implements SelfDescribingInputFormatInterface by HoodieParquetInputFormatBase
let's move this change to HoodieParquetInputFormatBase
public abstract class HoodieParquetInputFormatBase extends MapredParquetInputFormat implements Configurable, SelfDescribingInputFormatInterface
leesf
left a comment
leesf
left a comment
There was a problem hiding this comment.
LGTM, Please the CI failure and fix
nsivabalan
added
priority:high
|
Because of SelfDescribingInputFormatInterface not exist in hive 1.x, so i have to add dependency in hudi-spark-common. Otherwise, the SelfDescribingInputFormatInterface class cannot be found if Hive 1.x is used during hudi-spark-common compilation. |
|
@hudi-bot run azure |
|
@hudi-bot run azure |
|
@hudi-bot run azure |
|
LGTM as well, though I am still curious to understand in detail how does this interface |
In hive FetechOpertor, will judge needConversion by the inputformot whether implement SelfDescribingInputFormatInterface |
| <artifactId>hive-common</artifactId> | ||
| <version>${hive.version}</version> | ||
| </dependency> | ||
| <dependency> |
There was a problem hiding this comment.
Can you remove this dependency as the change is in hudi-hadoop-mr package and hive-exec is already defined as dependency in hudi-hadoop-mr /pom.xml
| </dependency> | ||
|
|
||
| <!-- Hive --> | ||
| <dependency> |
There was a problem hiding this comment.
This is still showing up. Can you remove this dependency as the change is in hudi-hadoop-mr package and hive-exec is already defined as dependency in hudi-hadoop-mr /pom.xml
|
@fsilent : Thanks a lot for the PR. Can you fix the package dependencies and also add a jira ticket in the description. |
10 participants
Change Logs
#6989 cannot support column type evolution.
class that implements SelfDescribingInputFormatInterface,
hive will try to do colum type evolution by itself
Impact
support type change
Risk level (write none, low medium or high below)
low.
Documentation Update
N/A
Contributor's checklist