[HUDI-4767] Fix non partition table in hudi-flink ignore KEYGEN_CLASS… #6567
Conversation
| FlinkOptions.KEYGEN_CLASS_NAME.key(), NonpartitionedAvroKeyGenerator.class.getName()); | ||
| return; | ||
| if (StringUtils.isNullOrEmpty(conf.get(FlinkOptions.KEYGEN_CLASS_NAME))) { | ||
| // tweak the key gen class if possible |
There was a problem hiding this comment.
Hi, what kind of keygen clazz do you want to configure for non-partitioned table then ?
There was a problem hiding this comment.
Hi @danny0405 I want to configure org.apache.hudi.keygen.ComplexAvroKeyGenerator for non partition in hudi-flink side, which not only follow partitions, because in spark side, use complex key as default, but flink can not assign complex for non partition.
There was a problem hiding this comment.
In #5815, we have fixed the spark sql to use NonpartitionedKeyGenerator for non partitioned table.
There was a problem hiding this comment.
But user cannot assign keygen_class seems not friendly.
There was a problem hiding this comment.
Generally that's true, but non-partitioned table is a special case and hudi configure the keygen clazz transparently for user.
There was a problem hiding this comment.
Hudi configure keygen clazz auto is great, so the option should not exists, once configured but not effect, is it strange？The code in spark has changed to follow hudi-partition way, but in historical data, if the layout of non-partitioned table with complex key by spark, the only chance for hudi-flink is to configure keygen.
There was a problem hiding this comment.
That's true, but we better use the right key gen clazz for better performance
There was a problem hiding this comment.
Of course, shorter key will gain better performance, but this option should also take effect, right？
…_NAME in conf bug
yihua
added
priority:medium
yihua
left a comment
yihua
left a comment
There was a problem hiding this comment.
Closing the PR as the behavior is expected.
|
Feel free to reopen if still necessary. |
4 participants
Change Logs
Add null or empty judge in setupHoodieKeyOptions of org.apache.hudi.table.HoodieTableFactory class
Impact
User configure KEYGEN_CLASS_NAME of hudi-flink module will take effect.
Risk level: none | low | medium | high
none
Contributor's checklist