[HUDI-5275] Fix reading data using the HoodieHiveCatalog will cause the Spark write to fail

[waywtdcc]

Change Logs

Fix reading data using the HoodieHiveCatalog will cause the Spark write to fail.
The original hoodiehivecatalog read the table only according to the attributes of the hive table, but not according to the hoodie.properties file, resulting in an error reading the table attributes. Changed hoodie.datasource.write hive_ style_ The partitioning attribute is inconsistent with the real attribute

Impact

Fix reading data using the HoodieHiveCatalog will cause the Spark write to fail.

Risk level (write none, low medium or high below)

low

Documentation Update

• Fix reading data using the HoodieHiveCatalog will cause the Spark write to fail

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

Do we need to put the explicit table options here ? Which function needs this config and where we miss it then ?

[danny0405]

We can read the table config through StreamerUtil.createMetaClient(xx).getTableConfig()

We actually get the original hive table params first here: parameters = hiveTable.getParameters(), you mean there are options missing for spark ?

[waywtdcc]

Yes, after Spark created the table, there are some table properties in .hoodie/hoodie.properties, such as hoodie.datasource.write.hive_style_partitioning. These attributes are not included in the hive table attribute.

[danny0405]

But we only append hive options here, curious why this can cause error ?

[waywtdcc]

Because spark uses the hoodie.datasource.write.hive_style_partitioning property to be true when creating hudi non-partitioned tables, and records hoodie.datasource.write.hive_style_partitioning=true in hoodie.properties. But this property is not seen in hoodiehivecatalog, and inferred It is also inferred that it is wrong, so hoodie.datasource.write.hive_style_partitioning is assigned a value of false and uploaded to hive options. As a result, spark has two different hoodie.datasource.write.hive_style_partitioning attribute values when viewing the table, thus reporting an error.

[waywtdcc]

if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
Path hoodieTablePath = new Path(path);
boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
.map(fileStatus -> fileStatus.getPath().getName())
.filter(f -> !f.equals(".hoodie") && !f.equals("default"))
.anyMatch(FilePathUtils::isHiveStylePartitioning);
parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
}

--- The params in this code are not inferred according to the hoodie.properties property file, resulting in an inference error.

[danny0405]

Got you, i think we should not put extra options like FlinkOptions.HIVE_STYLE_PARTITIONING here, the right fix is to supplement the table config options for the flink read/write path, I have applied a patch actually.

[hudi-bot]

CI report:

• c165528 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[codope]

Could you please add a test for this scenario?

[waywtdcc]

@danny0405 Hello, do you have any comments on this PR?

[danny0405]

Yeah, I have reviewed and applied a path:
5275.zip

The idea to fix is: we better not add extra options like FlinkOptions.HIVE_STYLE_PARTITIONING when reading the table, a more proper way to fix the issue is to merge the table config options for reader/writer path.

Can you apply the patch and add a test case in TestHoodieTableFactory to test 2 cases (to test that the write config options always have higher priority):

1. the table source merges the built-in table config that is not defined in write config.
2. the table source can not override the existing write config if the option value changes.

[danny0405]

Hi, gentle ping :)

[waywtdcc]

Sorry, just saw it. Is it better to replace loadFromHoodiePropertieFile with StreamerUtil.getTableConfig here?

[danny0405]

Because there is no response for a long time and the code freeze is very near, I worked based on the path and would land it once the CI passed: #7666

Close this one instead.