[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests #3719

zhangyue19921010 · 2021-09-26T09:30:09Z

What is the purpose of the pull request

Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests.

Cache HoodieTableFileSystemView at baseDir level

The same as HoodieTableMetaClient

Here is the test result based on S3 hudi table
240 partitions and 2400 data files.

Condition	All Requests	Get Requests	List Requests	4xx errors
Disable metadata table	17280	14144	1322	1568
Disable metadata table + this patch tuning	11495	9839	796	620

Condition	All Requests	Get Requests	List Requests	4xx errors
Enable metadata table	21024	13980	3198	3438
Enable metadata table + this patch tuning	13399	10485	1093	1298

I also verify the query result, and it works fine.

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

hudi-bot · 2021-09-26T09:35:20Z

CI report:

82b6fa3 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run travis re-run the last Travis build
@hudi-bot run azure re-run the last Azure build

leesf · 2021-09-26T14:04:46Z

I am a little curious about why the requests in Enable metadata table is larger than Disable metadata table, and also what the query response time difference between enable/disable metadata table?

zhangyue19921010 · 2021-09-27T06:45:19Z

I am a little curious about why the requests in Enable metadata table is larger than Disable metadata table, and also what the query response time difference between enable/disable metadata table?

I think metadata table feature let hudi query internal hudi MOR table to do file list instead of listing the files in partitions so that can improve list performance. But as we know a query-hudi-table-action need more requests compared a single list action. Maybe this can explain why the requests in Enable metadata table is larger than Disable metadata table

zhangyue19921010 · 2021-09-27T06:48:21Z

As for performance improved because of hudi metadata table, I got the follow answers:

Condition	Processing Time
Disable metadata table	143s
Enable metadata table	120s

based on S3 hudi table 240 partitions and 2400 data files.

zhangyue19921010 · 2021-09-27T06:48:49Z

By the way, thanks for your attention @leesf :)

leesf · 2021-09-28T15:13:59Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java


-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());


here the fsView would never get updated once put into cache, even the writer commits new file, the new file will not appears in the fsView, I think it may lead to wrong results in flink streaming read, right? cc @danny0405

Flink streaming reader does not use this code path.

it is in flink incremental read code path. still a problem?

Thanks for your review.
As we can see HoodieROTablePathFilter already create and cache HoodieTableMetaClient at baseDir level, also setLoadActiveTimelineOnLoad(true) which will create an active timeline in singleton mode.
So that IMO no matter we cache the fsView or not, any new created files will not appear in current hoodieROTablePathFilter.

Now we cached the fsView using above cached meta client and cached active timeline. Maybe can have no bad effect but can reduce unnecessary init action.

Feel free to correct me if I am wrong :)

danny0405 · 2021-09-29T02:30:53Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java

+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());
+          if (null == fsView) {
+            fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext, metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+            hoodieTableFileSystemViewCache.put(baseDir.toString(), fsView);


Use computeIfAbsent to simplify the logic.

Sure thing. Changed.

danny0405 · 2021-09-29T02:31:44Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java

-        } finally {
-          if (fsView != null) {
-            fsView.close();
-          }


How should the cached views been closed ?

fsView.close() will do

@Override public void close() { closed = true; super.reset(); partitionToFileGroupsMap = null; fgIdToPendingCompaction = null; fgIdToBootstrapBaseFile = null; fgIdToReplaceInstants = null; }

Because of we recycling fsView, we can't close it. Although we only create one fsView for each baseDir. And will cause no memory leak maybe.

zhangyue19921010 · 2021-10-08T10:04:35Z

Hi @vinothchandar I noticed that you are the author of HoodieROTableFilter.java. Would you mind take a look at your convince?

Very appreciate it if you could give me a hand :)

leesf

@vinothchandar do you have time to take a final pass here.

nsivabalan · 2021-10-12T18:18:07Z

@zhangyue19921010 : wrt your comment on perf difference (enabling and disabling metadata), was it a read query that you benchmarked? If yes, did you also enable metadata when you queried? CC @xushiyan

zhangyue19921010 · 2021-10-13T07:46:06Z

@zhangyue19921010 : wrt your comment on perf difference (enabling and disabling metadata), was it a read query that you benchmarked? If yes, did you also enable metadata when you queried? CC @xushiyan

@nsivabalan Thanks a lot for your review. Yep, I use a query with meta-data enabled at query side to do the benchmark.

zhuoluoy · 2023-03-10T02:29:19Z

Actually, for legacy MapReduce. This patch is very important. Without this patch, HoodiROTablePathFilter will be thousands times slower.

…SystemView, aiming to reduce unnecessary list/get requests (apache#3719)

cache hoodieTableFileSystemView

f4d2fc3

leesf reviewed Sep 28, 2021

View reviewed changes

danny0405 reviewed Sep 29, 2021

View reviewed changes

code review

82b6fa3

zhangyue19921010 requested review from danny0405 and leesf October 8, 2021 09:58

leesf approved these changes Oct 11, 2021

View reviewed changes

nsivabalan self-assigned this Oct 12, 2021

nsivabalan approved these changes Oct 22, 2021

View reviewed changes

nsivabalan merged commit 1e285dc into apache:master Oct 22, 2021

h7kanna mentioned this pull request Dec 2, 2021

[SUPPORT] NullPointerException in HoodieROTablePathFilter Hudi 0.10.0 #4188

Closed

Junyewu mentioned this pull request Dec 9, 2022

[SUPPORT] With HoodieROTablePathFilter is too slow load normal parquets in hudi release #7417

Open

southernriver pushed a commit to southernriver/hudi that referenced this pull request Mar 31, 2023

[HUDI-2489] Tuning HoodieROTablePathFilter by caching hoodieTableFile…

af6e110

…SystemView, aiming to reduce unnecessary list/get requests (apache#3719)

[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests #3719

[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests #3719

Uh oh!

Conversation

zhangyue19921010 commented Sep 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the pull request

Verify this pull request

Committer checklist

Uh oh!

hudi-bot commented Sep 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

leesf commented Sep 26, 2021

Uh oh!

zhangyue19921010 commented Sep 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhangyue19921010 commented Sep 27, 2021

Uh oh!

zhangyue19921010 commented Sep 27, 2021

Uh oh!

leesf Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny0405 Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

leesf Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

danny0405 Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

danny0405 Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 Sep 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 commented Oct 8, 2021

Uh oh!

leesf left a comment

Choose a reason for hiding this comment

Uh oh!

nsivabalan commented Oct 12, 2021

Uh oh!

zhangyue19921010 commented Oct 13, 2021

Uh oh!

zhuoluoy commented Mar 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zhangyue19921010 commented Sep 26, 2021 •

edited

Loading

hudi-bot commented Sep 26, 2021 •

edited

Loading

zhangyue19921010 commented Sep 27, 2021 •

edited

Loading

leesf Sep 28, 2021 •

edited

Loading

zhangyue19921010 Sep 29, 2021 •

edited

Loading