[HUDI-3935] Adding config to fallback to enabled Partition Values extraction from Partition path #5377
Conversation
Added docs
…sFromPartitionPath` flags
alexeykudinkin
changed the title
Spark32HoodieParquetFileFormat not being compatible w/ Spark 3.2.0
alexeykudinkin
force-pushed
the
ak/ts-kgen-cfg-fb
branch
from
08b5e4f to
e63b627
Compare
alexeykudinkin
changed the title
| return this; | ||
| } | ||
|
|
||
| public PropertyBuilder setDropPartitionColumnsWhenWrite(Boolean dropPartitionColumnsWhenWrite) { |
There was a problem hiding this comment.
having "write" in the name makes is clear. If not, one could read it as "should drop partition columns when reading". So, I feel we can leave it as is.
alexeykudinkin
changed the title
| } | ||
|
|
||
| implicit def convert[T, U](prop: ConfigProperty[T])(implicit converter: T => U): ConfigProperty[U] = { | ||
| checkState(prop.hasDefaultValue) |
There was a problem hiding this comment.
this could implicitly break when add a new config with no default. i see this improves code quality but we should avoid nice-to-have changes in the last min patch before release.
There was a problem hiding this comment.
If this will break it will break when the class is loaded, meaning that all the tests using class would be broken, which is very easy to diagnose
…raction from Partition path (#5377)
|
Hi @alexeykudinkin |
|
@TengHuo correct |
|
Got it, thanks a lot |
6 participants
Tips
What is the purpose of the pull request
#5364 made extraction of values for partition columns from partition path became configurable, disabling it by default,
since by default Hudi persists partition columns in the data file which could be fetched directly instead of parsing partition values from partition path.
This PR adds a fallback configuration allowing to control whether partition values should be parsed from the partition path (which is default Spark behavior).
Brief change log
shouldExtractPartitionValuesFromPartitionPathflagsEXTRACT_PARTITION_VALUES_FROM_PARTITION_PATHVerify this pull request
This pull request is already covered by existing tests, such as (please describe tests).
This change added tests and can be verified as follows:
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.