-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SUPPORT] current state of promotion type support #11599
Comments
@jonvex Any insights here? |
Hive sync has more limited type promotion like you described. We have some documentation here https://hudi.apache.org/docs/schema_evolution but as far as I know, this has only been tested for spark |
Hive support looks like much broader than the hivesync implementation see bottom of page https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types
Consequently glue metastore should be the same.
I will propose a fix for hive sync then.
|
@parisni That would be very helpful, thanks for the contribution~ |
@jonvex
|
definitely, the same spark written with vanilla spark 3.3.4 :
something wrong w/ emr apparently |
also turns out glue sync likely does not update fields, only sql.schema:
IN:
EDIT: Found the issue. Will propose a patch. |
I tested athena support for promotion type. It works fine, until a partition get mixed with multiple types (int/float); eg: after an upsert. The reason is in athena each partition has it's own schema, and it has to be updated with the promoted type. Otherwise such error will occur:
or
I guess we should reconsider approach proposed here #9071 |
FYI amazon redshift spectrum likely does not support all of promotion types. At least
|
here is a small recap from my tests @jonvex @danny0405 . "No" means a select query ordering on the column will fail at runtime
|
hudi 0.14.1
spark 3.3.x
I did not find any mention of hudi promotion type in the documentation
apparently promotion type is not supported #5519 :
Now from my tests below:
So:
The text was updated successfully, but these errors were encountered: