Skip to content

[MINOR] Use direct marker for spark when timeline server is disabled #7272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leesf merged 1 commit into from
Nov 23, 2022
Merged

[MINOR] Use direct marker for spark when timeline server is disabled #7272

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2794,7 +2794,12 @@ public HoodieWriteConfig build() {
private String getDefaultMarkersType(EngineType engineType) {
switch (engineType) {
case SPARK:
return MarkerType.TIMELINE_SERVER_BASED.toString();
if (writeConfig.isEmbeddedTimelineServerEnabled()) {
return MarkerType.TIMELINE_SERVER_BASED.toString();
} else {
LOG.warn("Embedded timeline server is disabled, fallback to use direct marker type for spark");
return MarkerType.DIRECT.toString();
Copy link
Copy Markdown
Contributor

@leesf leesf Nov 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could log.warn here.

Copy link
Copy Markdown
Contributor Author

@stream2000 stream2000 Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Copy Markdown
Contributor

@yihua yihua Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic of turning off the timeline-server-based markers is already covered by WriteMarkersFactory.

}
case FLINK:
case JAVA:
// Timeline-server-based marker is not supported for Flink and Java engines
Expand Down