[HUDI-5007] Prevent Hudi from reading the entire timeline's when perf…

[voonhous]

…orming a LATEST streaming read

Change Logs

Prevent Hudi from performing unnecessary file scans when performing a stream read from the latest instant.

Impact

No public API changed, changing the access level of a private method from private to public so that it can be used for testing

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

Nice catch, i have reviewed and applied a patch here:
5007.patch.zip

Can you also add a test class in TestInputFormat and test the input splits logic for streaming read in this case ?

You can take a reference from TestInputFormat#testReadSkipCompaction

[voonhous]

Can you also add a test class in TestInputFormat and test the input splits logic for streaming read in this case ?

You can take a reference from TestInputFormat#testReadSkipCompaction

Apologies for the late reply, given that the read behaviour is correct, the only way we can test this fix is to make the IncrementalInputSplits#getInstantRange method public.

This bug is caused by IncrementalInputSplits#getInstantRange not producing the correct result, causing unnecessary file checks, blocking the first read cycle.

The IncrementalInputSplits.inputSplits method is still able to produce the right result, allowing the right records to be read out.

Hence, I believe a test should be performed on the generation of HoodieInstants to read via the IncrementalInputSplits#getInstantRange method instead of the read behaviour like the rest of the tests in the TestInputFormat class are doing.

To do this, one will have to make the method public or use Reflection to access this class, I am not sure if loading the class in via reflection is the right way to approach this test.

Do you have any suggestions?

[voonhous]

@danny0405 Added the required tests to validate this fix, can you please to take a look at it again?

Thank you.

[voonhous]

@danny0405 Added the changes that you have suggested. Thank you.

[hudi-bot]

CI report:

• bdf960d UNKNOWN
• fcf16d5 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

The failed test case in module hudi-utilities should not be affected by this patch, would merge it soon ~

[danny0405]

+1