-
Notifications
You must be signed in to change notification settings - Fork 3.4k
API: add an action API for rewrite deletes #2841
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 11 commits
a2eee90
ed51fc7
0697cf5
9537fa1
c68303b
6014276
597e8f9
82c8118
bc1a0b9
dee5962
70b7162
dcb3201
a6a2540
6d0853b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| package org.apache.iceberg.actions; | ||
|
|
||
| import org.apache.iceberg.expressions.Expression; | ||
|
|
||
| /** | ||
| * An action for converting the equality delete files to position delete files. | ||
| */ | ||
| public interface ConvertEqualityDeleteFiles | ||
| extends SnapshotUpdate<ConvertEqualityDeleteFiles, ConvertEqualityDeleteFiles.Result> { | ||
|
|
||
| /** | ||
| * A row filter for finding the equality deletes to convert. | ||
| * | ||
| * @param expression An iceberg expression used to find deletes. | ||
| * @return this for method chaining | ||
| */ | ||
| ConvertEqualityDeleteFiles rowFilter(Expression expression); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since we have just one filter, there is no need to add "row" in the method name. Let's change it back to
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
|
|
||
| /** | ||
| * A partition filter for finding the equality deletes to convert. | ||
| * | ||
| * @param expression An iceberg expression used to find deletes. | ||
| * @return this for method chaining | ||
| */ | ||
| ConvertEqualityDeleteFiles partitionFilter(Expression expression); | ||
|
|
||
| /** | ||
| * The action result that contains a summary of the execution. | ||
| */ | ||
| interface Result { | ||
| /** | ||
| * Returns the count of the deletes that been converted. | ||
| */ | ||
| int convertedEqualityDeleteFilesCount(); | ||
|
|
||
| /** | ||
| * Returns the count of the added position delete files. | ||
| */ | ||
| int addedPositionDeleteFilesCount(); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| package org.apache.iceberg.actions; | ||
|
|
||
| import org.apache.iceberg.expressions.Expression; | ||
|
|
||
| /** | ||
| * An action for rewriting the position delete files. | ||
|
chenjunjiedada marked this conversation as resolved.
Outdated
|
||
| * <p> | ||
| * Generally used for optimizing the sizing and layout of position delete files within a table. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: should be "size" not "sizing" |
||
| */ | ||
| public interface RewritePositionDeleteFiles | ||
| extends SnapshotUpdate<RewritePositionDeleteFiles, RewritePositionDeleteFiles.Result> { | ||
|
|
||
| /** | ||
| * bin-pack the position deletes. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should be capitalized. |
||
| * | ||
| * @return this for method chaining | ||
| */ | ||
| RewritePositionDeleteFiles binPackPositionDeletes(); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why add
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Make sense to remove this, |
||
|
|
||
| /** | ||
| * A row filter for finding deletes to rewrite. | ||
| * | ||
| * @param expression An iceberg expression used to find deletes. | ||
| * @return this for method chaining | ||
| */ | ||
| RewritePositionDeleteFiles rowFilter(Expression expression); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that we plan to convert this to a partition filter using an inclusive projection like the equality delete rewrite, correct? @jackye1995, is there a use case for matching files and then using stats to filter out position delete files that don't match? I can't think of one. |
||
|
|
||
| /** | ||
| * A partition filter for finding deletes to rewrite. | ||
| * | ||
| * @param expression An iceberg expression used to find deletes. | ||
| * @return this for method chaining | ||
| */ | ||
| RewritePositionDeleteFiles partitionFilter(Expression expression); | ||
|
|
||
| /** | ||
| * The action result that contains a summary of the execution. | ||
| */ | ||
| interface Result { | ||
| /** | ||
| * Returns the count of the position deletes that been rewritten. | ||
| */ | ||
| int rewrittenDeleteFilesCount(); | ||
|
|
||
| /** | ||
| * Returns the count of the added delete files. | ||
| */ | ||
| int addedDeleteFilesCount(); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| package org.apache.iceberg.actions; | ||
|
|
||
| import java.util.Map; | ||
| import java.util.Set; | ||
| import org.apache.iceberg.DeleteFile; | ||
| import org.apache.iceberg.FileScanTask; | ||
| import org.apache.iceberg.Table; | ||
|
|
||
| /** | ||
| * A strategy for the action to convert equality delete to position deletes. | ||
| */ | ||
| public interface ConvertEqualityDeleteStrategy { | ||
|
|
||
| /** | ||
| * Returns the name of this convert deletes strategy | ||
| */ | ||
| String name(); | ||
|
|
||
| /** | ||
| * Returns the table being modified by this convert strategy | ||
| */ | ||
| Table table(); | ||
|
|
||
| /** | ||
| * Returns a set of options which this convert strategy can use. This is an allowed-list and any options not | ||
| * specified here will be rejected at runtime. | ||
| */ | ||
| Set<String> validOptions(); | ||
|
|
||
| /** | ||
| * Sets options to be used with this strategy | ||
| */ | ||
| RewriteDeleteStrategy options(Map<String, String> options); | ||
|
|
||
| /** | ||
| * Select the delete files to convert. | ||
| * | ||
| * @return iterable of original delete file to be converted. | ||
| */ | ||
| Iterable<DeleteFile> selectDeleteFiles(); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the data file rewrite strategy, the equivalent method accepts an
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In my POC implementation, there is a local variable that holds the iterable. Accepting the |
||
|
|
||
| /** | ||
| * Define how to convert the deletes. | ||
| * | ||
| * @param deleteFilesToConvert a group of files to be converted together | ||
| * @return iterable of delete files used to replace the original delete files. | ||
| */ | ||
| Iterable<DeleteFile> convertDeleteFiles(Iterable<DeleteFile> deleteFilesToConvert); | ||
|
|
||
| /** | ||
| * Groups delete files into lists which will be processed in a single executable unit. Each group will end up being | ||
| * committed as an independent set of changes. This creates the jobs which will eventually be run as by the underlying | ||
| * Action. | ||
| * | ||
| * @param dataFiles iterable of data files that contain the DeleteFile to be converted | ||
| * @return iterable of lists of FileScanTasks which will be processed together | ||
| */ | ||
| Iterable<Iterable<FileScanTask>> planDeleteFileGroups(Iterable<FileScanTask> dataFiles); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: The order of methods doesn't match
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. will update. |
||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| package org.apache.iceberg.actions; | ||
|
|
||
| import java.util.Map; | ||
| import java.util.Set; | ||
| import org.apache.iceberg.DeleteFile; | ||
| import org.apache.iceberg.Table; | ||
|
|
||
| public interface RewriteDeleteStrategy { | ||
|
|
||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we also need the methods similar to
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't add For
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| /** | ||
| * Returns the name of this rewrite deletes strategy | ||
| */ | ||
| String name(); | ||
|
|
||
| /** | ||
| * Returns the table being modified by this rewrite strategy | ||
| */ | ||
| Table table(); | ||
|
|
||
| /** | ||
| * Returns a set of options which this rewrite strategy can use. This is an allowed-list and any options not | ||
| * specified here will be rejected at runtime. | ||
| */ | ||
| Set<String> validOptions(); | ||
|
|
||
| /** | ||
| * Sets options to be used with this strategy | ||
| */ | ||
| RewriteDeleteStrategy options(Map<String, String> options); | ||
|
|
||
| /** | ||
| * Select the delete files to rewrite. | ||
| * | ||
| * @return iterable of original delete file to be replaced. | ||
| */ | ||
| Iterable<DeleteFile> selectDeleteFiles(); | ||
|
|
||
| /** | ||
| * Define how to rewrite the deletes. | ||
| * | ||
| * @param deleteFilesToRewrite a group of files to be rewritten together | ||
| * @return iterable of delete files used to replace the original delete files. | ||
| */ | ||
| Iterable<DeleteFile> rewriteDeleteFiles(Iterable<DeleteFile> deleteFilesToRewrite); | ||
|
|
||
| /** | ||
| * Groups into lists which will be processed in a single executable unit. Each group will end up being | ||
| * committed as an independent set of changes. This creates the jobs which will eventually be run as by the underlying | ||
| * Action. | ||
| * | ||
| * @param deleteFiles iterable of DeleteFile to be rewritten | ||
| * @return iterable of lists of FileScanTasks which will be processed together | ||
| */ | ||
| Iterable<Iterable<DeleteFile>> planDeleteFileGroups(Iterable<DeleteFile> deleteFiles); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. wondering why here it is also wondering if both interfaces in this PR can use
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is one difference between the rewrite position delete and the convert, it is about the planning. |
||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that this should mention that this filter will be converted to a partition filter with an inclusive projection and that matching delete files will be rewritten. It should be clear that it isn't possible to match individual deletes, only whole files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will add doc.