Skip to content

[HUDI-4998] Inference of META_SYNC_PARTITION_EXTRACTOR_CLASS does not work with META_SYNC_PARTITION_FIELDS #6899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xushiyan merged 3 commits into from
Oct 20, 2022
Merged

[HUDI-4998] Inference of META_SYNC_PARTITION_EXTRACTOR_CLASS does not work with META_SYNC_PARTITION_FIELDS #6899

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9b4e7ab
[HUDI-4998] Inference of META_SYNC_PARTITION_EXTRACTOR_CLASS does not…
xicm Oct 9, 2022
34dc3eb
update default value of META_SYNC_PARTITION_EXTRACTOR_CLASS
xicm Oct 10, 2022
87b19d8
Infer META_SYNC_PARTITION_EXTRACTOR_CLASS from META_SYNC_PARTITION_FI…
xicm Oct 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 10 additions & 16 deletions hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/HoodieSyncConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,28 +94,22 @@ public class HoodieSyncConfig extends HoodieConfig {

public static final ConfigProperty<String> META_SYNC_PARTITION_EXTRACTOR_CLASS = ConfigProperty
.key("hoodie.datasource.hive_sync.partition_extractor_class")
.defaultValue("org.apache.hudi.hive.MultiPartKeysValueExtractor")
.defaultValue("org.apache.hudi.hive.NonPartitionedExtractor")
.withInferFunction(cfg -> {
Option<String> partitionFieldsOpt = Option.ofNullable(cfg.getString(HoodieTableConfig.PARTITION_FIELDS))
.or(() -> Option.ofNullable(cfg.getString(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME)));
if (!partitionFieldsOpt.isPresent()) {
Option<String> partitionFieldsOpt = Option.ofNullable(cfg.getString(META_SYNC_PARTITION_FIELDS));
if (!partitionFieldsOpt.isPresent() || StringUtils.isNullOrEmpty(partitionFieldsOpt.get())) {
return Option.empty();
}
String partitionFields = partitionFieldsOpt.get();
if (StringUtils.nonEmpty(partitionFields)) {
int numOfPartFields = partitionFields.split(",").length;
if (numOfPartFields == 1) {
if (cfg.contains(HIVE_STYLE_PARTITIONING_ENABLE)
&& cfg.getString(HIVE_STYLE_PARTITIONING_ENABLE).equals("true")) {
return Option.of("org.apache.hudi.hive.HiveStylePartitionValueExtractor");
} else {
return Option.of("org.apache.hudi.hive.SinglePartPartitionValueExtractor");
}
int numOfPartFields = partitionFieldsOpt.get().split(",").length;
if (numOfPartFields == 1) {
if (cfg.contains(HIVE_STYLE_PARTITIONING_ENABLE)
&& cfg.getString(HIVE_STYLE_PARTITIONING_ENABLE).equals("true")) {
return Option.of("org.apache.hudi.hive.HiveStylePartitionValueExtractor");
} else {
return Option.of("org.apache.hudi.hive.MultiPartKeysValueExtractor");
return Option.of("org.apache.hudi.hive.SinglePartPartitionValueExtractor");
}
} else {
return Option.of("org.apache.hudi.hive.NonPartitionedExtractor");
return Option.of("org.apache.hudi.hive.MultiPartKeysValueExtractor");
}
})
.withDocumentation("Class which implements PartitionValueExtractor to extract the partition values, "
Expand Down
2 changes: 1 addition & 1 deletion ...sync/hudi-sync-common/src/test/java/org/apache/hudi/sync/common/TestHoodieSyncConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ void testInferPartitionFields() {
void testInferPartitionExtractorClass() {
Properties props0 = new Properties();
HoodieSyncConfig config0 = new HoodieSyncConfig(props0, new Configuration());
assertEquals("org.apache.hudi.hive.MultiPartKeysValueExtractor",
assertEquals("org.apache.hudi.hive.NonPartitionedExtractor",
config0.getStringOrDefault(META_SYNC_PARTITION_EXTRACTOR_CLASS),
String.format("should get default value due to absence of both %s and %s",
HoodieTableConfig.PARTITION_FIELDS.key(), KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key()));
Expand Down