Skip to content

[HUDI-7105] support filesystem view configuable to avoid clean oom #10116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Askwang wants to merge 2 commits into from
Closed

[HUDI-7105] support filesystem view configuable to avoid clean oom #10116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
21d2553
filesystem view configuable
Askwang Nov 16, 2023
ed40a83
fix config not valid
Askwang Nov 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ...-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ public FileSystemViewStorageConfig getRemoteFileSystemViewConfig() {
.withRemoteTimelineInitialRetryIntervalMs(writeConfig.getClientSpecifiedViewStorageConfig().getRemoteTimelineInitialRetryIntervalMs())
.withRemoteTimelineClientMaxRetryIntervalMs(writeConfig.getClientSpecifiedViewStorageConfig().getRemoteTimelineClientMaxRetryIntervalMs())
.withRemoteTimelineClientRetryExceptions(writeConfig.getClientSpecifiedViewStorageConfig().getRemoteTimelineClientRetryExceptions())
.withRemoteFileSystemViewFirst(writeConfig.getClientSpecifiedViewStorageConfig().isRemoteViewFirst())
.build();
}

Expand Down
9 changes: 7 additions & 2 deletions hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,6 @@ public static FileSystemViewManager createViewManager(final HoodieEngineContext
return new FileSystemViewManager(context, config, (metaClient, viewConfig) -> createRemoteFileSystemView(conf,
viewConfig, metaClient));
case REMOTE_FIRST:
LOG.info("Creating remote first table view");
return new FileSystemViewManager(context, config, (metaClient, viewConfig) -> {
RemoteHoodieTableFileSystemView remoteFileSystemView =
createRemoteFileSystemView(conf, viewConfig, metaClient);
Expand All @@ -279,7 +278,13 @@ public static FileSystemViewManager createViewManager(final HoodieEngineContext
throw new IllegalArgumentException("Secondary Storage type can only be in-memory or spillable. Was :"
+ viewConfig.getSecondaryStorageType());
}
return new PriorityBasedFileSystemView(remoteFileSystemView, secondaryView);
if (config.isRemoteViewFirst()) {
LOG.info("Creating remote table view first");
return new PriorityBasedFileSystemView(remoteFileSystemView, secondaryView);
} else {
LOG.info("Creating secondary table view first");
Copy link
Contributor

@danny0405 danny0405 Nov 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @zhedoubushishi , who have also encountered OOM for async cleaning.

return new PriorityBasedFileSystemView(secondaryView, remoteFileSystemView);
}
});
default:
throw new IllegalArgumentException("Unknown file system view type :" + config.getStorageType());
Expand Down
15 changes: 15 additions & 0 deletions hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewStorageConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,12 @@ public class FileSystemViewStorageConfig extends HoodieConfig {
.withDocumentation("Config to control whether backup needs to be configured if clients were not able to reach"
+ " timeline service.");

public static final ConfigProperty<String> REMOTE_VIEW_FIRST = ConfigProperty
.key("hoodie.filesystem.view.remote.first")
.defaultValue("true")
.markAdvanced()
.withDocumentation("Controls whether or not the file system view is remote first or secondary first.");

public static FileSystemViewStorageConfig.Builder newBuilder() {
return new Builder();
}
Expand Down Expand Up @@ -274,6 +280,10 @@ public String getRocksdbBasePath() {
return getString(ROCKSDB_BASE_PATH);
}

public boolean isRemoteViewFirst() {
return getBoolean(REMOTE_VIEW_FIRST);
}

/**
* The builder used to build {@link FileSystemViewStorageConfig}.
*/
Expand Down Expand Up @@ -378,6 +388,11 @@ public Builder withEnableBackupForRemoteFileSystemView(boolean enable) {
return this;
}

public Builder withRemoteFileSystemViewFirst(boolean enable) {
fileSystemViewStorageConfig.setValue(REMOTE_VIEW_FIRST, Boolean.toString(enable));
return this;
}

public FileSystemViewStorageConfig build() {
fileSystemViewStorageConfig.setDefaults(FileSystemViewStorageConfig.class.getName());
// Validations
Expand Down