[HUDI-5242] Do not fail Meta sync in Deltastreamer when inline table service fails

[jonvex]

Change Logs

After the files are written, table services like clustering and compaction can fail. This causes the sync to the metaserver to not happen. This pr adds a config that when set to false, the deltastreamer will not fail and the sync to the metaserver will occur. A warning will be logged with the exception that occurred. To use this new behavior, set hoodie.fail.writes.on.inline.table.service.exception to false.

How I tested:

First, in inlineClustering:BaseHoodieWriteClient.java, add the following code before the return

if (true) {
      throw new HoodieException("CLUSTER FAILURE");
}

Test not setting the config:

1. Run the test testInlineClustering:TestHoodieDeltaStreamer.java
2. Verify the tests pass and search the logs for "CLUSTER FAILURE"

Test setting the config:

1. in writeCommonPropsToFile:HoodieDeltaStreamerTestBase.java add the line props.setProperty("hoodie.fail.writes.on.inline.table.service.exception","true");
2. Run the test testInlineClustering:TestHoodieDeltaStreamer.java
3. Verify the test failed with the hoodie exception "CLUSTER FAILURE"

Impact

For users that retry delta sync on failure, they will continue adding one commit each retry. New partitions will not show up in the metaserver.

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

A new configuration was created and an appropriate description was added

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[nsivabalan]

good you were able to test it out.
can we write a functional test for this as well.

[nsivabalan]

can you fix the PR desc w/ right config key

[codope]

@jonvex Can you please check the test failures?

[jonvex]

All tests should pass now. I thought we wanted the new behavior on by default but it was a miscommunication.

[nsivabalan]

we can land once the CI succeeds.

[hudi-bot]

CI report:

• 59d5fb6 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build