[HUDI-1072] Add replace metadata file to timeline #1853

satishkotha · 2020-07-20T22:59:42Z

What is the purpose of the pull request

This is part of work required for RFC-18 and RFC-19. Add replace action to valid actions in the timeline.

To keep the diff small and get feedback, i am sending just the structure of metadata. For examples of how this will be used, see POC here satishkotha@45d8c26.

I'm happy to bring in other changes from POC if you think we can push large change.

Brief change log

Add replace action to valid actions in the timeline
Add replace metadata file format

Verify this pull request

This change added unit tests

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

n3nash · 2020-07-22T01:10:10Z

hudi-common/src/main/avro/HoodieReplaceMetadata.avsc

+ "fields": [
+     {"name": "totalFilesReplaced", "type": "int"},
+     {"name": "command", "type": "string"},
+     {"name": "partitionMetadata", "type": {


Does the partitionMetadata contain partitionPath -> new file groups ?

This can be obtained from related commit file. Every t.replace has a corresponding t.[delta]commit (we decided to write two files to reduce overhead of reading all commit files in the query path)

Plus, it helps on query side to keep replace metadata files small. I also didn't want to repeat same information in two places.

Let me know if you think its important to store new file group info here.

High level Question : To make sure we are all on the same page : Is this metadata enough to achieve clustering ? Do you foresee any changes that needs to happen to this metadata to support clustering ? The PR mentions that this is for both clustering and overwrite. Hence, asking this question.

n3nash · 2020-07-22T01:12:00Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java

  }

+  /**
+   * Transition Clean State from inflight to Committed.


s/Clean/Replace

will update. thanks

n3nash · 2020-07-22T01:13:24Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java

      INFLIGHT_DELTA_COMMIT_EXTENSION, REQUESTED_DELTA_COMMIT_EXTENSION, SAVEPOINT_EXTENSION,
      INFLIGHT_SAVEPOINT_EXTENSION, CLEAN_EXTENSION, REQUESTED_CLEAN_EXTENSION, INFLIGHT_CLEAN_EXTENSION,
-      INFLIGHT_COMPACTION_EXTENSION, REQUESTED_COMPACTION_EXTENSION, INFLIGHT_RESTORE_EXTENSION, RESTORE_EXTENSION));
+      INFLIGHT_COMPACTION_EXTENSION, REQUESTED_COMPACTION_EXTENSION, INFLIGHT_RESTORE_EXTENSION, RESTORE_EXTENSION,


Since the intention of this PR is to simply introduce a new action, do we need to change the VALID_ACTION in this PR ? Do the other methods you implemented in the ActiveTimeline require this ? I'm wondering if we can avoid this change in case we want to land this PR incrementally

I added this for tests. But can split this and related test out into a different review

Its better to avoid this for rollout purpose. In case, this PR gets landed before the next and a release cut, then we need to worry about ordering of rollout between readers and writers.

n3nash · 2020-07-22T01:15:04Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java

+   *
+   * @return
+   */
+  HoodieTimeline getCompletedAndReplaceTimeline();


May be you have used this API in a different PR, but where exactly are we going to use this specific method ?

yes, this is used to build filesystem view in #1859, can move this to that PR

n3nash

High level, introducing replace action changes seem fine to me, interested in learning how old_file_group -> new_file_group mapping is stored and accessed. Yet to review the other PRs if they have this info..

satishkotha · 2020-07-22T03:28:37Z

High level, introducing replace action changes seem fine to me, interested in learning how old_file_group -> new_file_group mapping is stored and accessed. Yet to review the other PRs if they have this info..

This is not 1:1 mapping. So this is not being stored in replace metadata file. We can get new file groups created from corresponding commit metadata file. As far as I can see, we only need this information for auditing, debugging. So I'm planning to build this with CLI task. Let me know if you see this useful in other scenarios.

bvaradar · 2020-07-27T19:40:37Z

hudi-common/src/main/avro/HoodieReplaceMetadata.avsc

+ "fields": [
+     {"name": "totalFilesReplaced", "type": "int"},
+     {"name": "command", "type": "string"},
+     {"name": "partitionMetadata", "type": {


High level Question : To make sure we are all on the same page : Is this metadata enough to achieve clustering ? Do you foresee any changes that needs to happen to this metadata to support clustering ? The PR mentions that this is for both clustering and overwrite. Hence, asking this question.

bvaradar · 2020-07-27T19:42:46Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java

      INFLIGHT_DELTA_COMMIT_EXTENSION, REQUESTED_DELTA_COMMIT_EXTENSION, SAVEPOINT_EXTENSION,
      INFLIGHT_SAVEPOINT_EXTENSION, CLEAN_EXTENSION, REQUESTED_CLEAN_EXTENSION, INFLIGHT_CLEAN_EXTENSION,
-      INFLIGHT_COMPACTION_EXTENSION, REQUESTED_COMPACTION_EXTENSION, INFLIGHT_RESTORE_EXTENSION, RESTORE_EXTENSION));
+      INFLIGHT_COMPACTION_EXTENSION, REQUESTED_COMPACTION_EXTENSION, INFLIGHT_RESTORE_EXTENSION, RESTORE_EXTENSION,


Its better to avoid this for rollout purpose. In case, this PR gets landed before the next and a release cut, then we need to worry about ordering of rollout between readers and writers.

bvaradar · 2020-07-27T19:45:20Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java

  String[] VALID_ACTIONS_IN_TIMELINE = {COMMIT_ACTION, DELTA_COMMIT_ACTION,
      CLEAN_ACTION, SAVEPOINT_ACTION, RESTORE_ACTION, ROLLBACK_ACTION,
-      COMPACTION_ACTION};
+      COMPACTION_ACTION, REPLACE_ACTION};


same comment as above

bvaradar · 2020-07-27T19:49:42Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java

+   *
+   * @return
+   */
+  HoodieTimeline getCompletedAndReplaceTimeline();


Does the return timeline contains only replace timeline ? The naming is confusing to me. How about getValidReplaceTimeline or anything else to reflect the intention

vinothchandar · 2020-08-31T16:06:32Z

@satishkotha is this WIP?

satishkotha · 2020-08-31T23:45:17Z

This has been replaced by #2048

[HUDI-1072] Add replace metadata file to timeline

bc94669

vinothchandar assigned bvaradar Jul 21, 2020

Add replace action file names

0dea192

satishkotha mentioned this pull request Jul 22, 2020

[WIP] [HUDI-1072] Use replace metadata file to filter excluded files in views #1859

Closed

5 tasks

n3nash reviewed Jul 22, 2020

View reviewed changes

bvaradar reviewed Jul 27, 2020

View reviewed changes

satishkotha closed this Aug 31, 2020

[HUDI-1072] Add replace metadata file to timeline #1853

[HUDI-1072] Add replace metadata file to timeline #1853

Uh oh!

Conversation

satishkotha commented Jul 20, 2020

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n3nash left a comment

Choose a reason for hiding this comment

Uh oh!

satishkotha commented Jul 22, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinothchandar commented Aug 31, 2020

Uh oh!

satishkotha commented Aug 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants